Commit graph

244 commits

Author SHA1 Message Date
Eelco Dolstra 763db9879c Store the channel redirects in S3 2019-01-24 19:07:26 +01:00
Eelco Dolstra 4a3ae75e4f Fix build 2019-01-24 19:06:58 +01:00
Eelco Dolstra 8dfa9f37c8
Merge pull request #22 from samueldr/fix/bttf-guard
Fixes channel going back in time due to incomplete change.
2018-08-30 16:06:37 +02:00
Samuel Dionne-Riel d313d2bd51 Fixes channel going back in time due to incomplete change.
The change in d3d33d5b25 changed how the
locally saved channel information was saved on the local machine.

 * d3d33d5b25 (diff-682bf482f6dd273f1c3c49afd99b4660R192)

The changeset did *not* change the guard clause, which now made it act
like this:

 1. Try readlink on the file
 2. (bail as it's undefined)

The new behaviour is as more or less expected:

 1. Reads the file if available
 2. Split the URL components
 3. Keep the last bit

This can give us an empty string, which will compare to `-1` thus
fulfilling the need.
2018-08-25 23:18:47 -04:00
Eelco Dolstra e9587f9388 d3g5gsiof5omrk.cloudfront.net -> releases.nixos.org 2018-07-19 16:01:55 +02:00
Eelco Dolstra 97d9e3ea01
Merge pull request #17 from samueldr/sha256sum-check
Adds filename to sha256 file for use with `sha256 -c`
2018-06-13 17:41:06 +02:00
Samuel Dionne-Riel 2181f3c02a Adds filename to sha256 file for use with sha256 -c
The manpage says:

> The  sums  are  computed as described in FIPS-180-2.  When checking, the input should be a former
> output of this program.  The default mode is to print a line with checksum, **a space, a  character
> indicating  input  mode ('*' for binary, ' ' for text or where binary is insignificant),** and name
> for each FILE.

By adding the filename to the generated sha256 file, a user would be
able to download an iso, the sha256 file, then use `sha256 -c
[file].iso.sha256` to verify the file.

* * *

This fixes nixos-homepage#224
2018-06-06 18:11:16 -04:00
Eelco Dolstra 9f96e1e25d
Improve error message 2018-02-06 16:45:13 +01:00
Eelco Dolstra 6a653c367e
Fix build 2018-01-04 15:33:22 +01:00
Eelco Dolstra c3a9ba32ee
Sync with nixUnstable 2017-07-21 12:48:15 +02:00
Eelco Dolstra c6428851dc
mirror-nixos-branch: Update the Git repo last 2017-07-21 12:48:14 +02:00
Eelco Dolstra 2f7994e3de
Remove bad check 2017-07-07 19:42:20 +02:00
Eelco Dolstra b6c049019a
mirror-nixos-branch.pl: Call index-debuginfo 2017-07-07 17:12:25 +02:00
Eelco Dolstra fcd6250fd0
mirror-channel.pl: Remove
This is no longer used, since Hydra uploads directly to S3 now. Also,
"nix copy" can copy to S3 now.
2017-07-07 17:06:21 +02:00
Eelco Dolstra a5e125330c
generate-programs-index: Check whether symlink targets exist
Fixes https://github.com/NixOS/nixpkgs/issues/26661
2017-07-07 16:45:45 +02:00
Eelco Dolstra 74c917454c
Add a command for indexing debug info files by build ID
For every file "lib/debug/.build-id/<build-id>.debug" in every store
path ending in -debug in a release, this command creates an S3 key
"debuginfo/<build-id>" specifying the location of the NAR that
contains the debug info file for that build ID. Thus,
https://cache.nixos.org/debuginfo allows debug info files to be looked
up by build ID. This allows debug info files to be fetched
automatically by dwarffs [1].

One minor issue is that in theory, multiple store paths could provide
debug info files for a particular build ID. So then when we garbage
collect a store path from the binary cache, we migth invalidate a
debug info link even though there are non-GC'ed store paths that
provide the file. Ah well.

I could have used HTTP redirects, but they're kind of a pain to
configure with S3.

[1] https://github.com/edolstra/dwarffs
2017-07-07 15:33:27 +02:00
Eelco Dolstra a91fb566a9
Factor out the file cache 2017-07-06 14:53:52 +02:00
Eelco Dolstra 3daca6b592
Update generate-programs-index to handle .ls 2017-04-18 16:41:11 +02:00
Eelco Dolstra dd0d2d2ca6
Update build 2017-04-18 15:26:38 +02:00
Eelco Dolstra a5368b6239 Merge pull request #12 from primeos/master
Fix the displayed hash name (SHA-512 -> SHA-256)
2017-03-31 12:04:05 +02:00
Michael Weiss c3e8e3badb Fix the displayed hash name (SHA-512 -> SHA-256)
The HTML pages for each channel are showing the SHA-256 hash of each
file but the table header actually referred to it as an "SHA-512 hash".
2017-03-31 06:39:24 +02:00
Eelco Dolstra 1c07d6dd08
Lock the Git repo 2016-12-05 11:50:17 +01:00
Eelco Dolstra 2fd4d6beea Vacuum programs.sqlite
This cuts a few hundred KiB from the database.
2016-11-10 18:27:04 +01:00
Eelco Dolstra 6883275653 Ignore package listings with malformed UTF-8
For example, https://cache.nixos.org/5q3s4vfwwd9n9499yg11ydziwxgn644m.ls.xz
contains the string "bokm<E5>l.alias". This causes nlohmann::json to
barf.
2016-11-10 18:10:08 +01:00
Eelco Dolstra 2176c939fc Compute FS closures in parallel
This is much faster.
2016-11-10 18:10:08 +01:00
Eelco Dolstra b74e03a870 Revert "Disable programs.sqlite generation"
This reverts commit 7f2bbc884c.
2016-11-10 18:10:08 +01:00
Eelco Dolstra ba19471e5b Use the .ls.xz listings generated by Hydra to speed up building programs.sqlite
This reduces the time to build the index for the regular (non-small) channel from ~10 hours to ~4 minutes.

Fixes #9.
2016-11-10 18:10:03 +01:00
Eelco Dolstra 6391c4dcec Fix check for release existance
We no longer have a github-link file, so this check always failed. As
a result, we were constantly re-uploading releases.
2016-10-21 20:10:53 +02:00
Eelco Dolstra 038c8ee8d4 Update the channels repo early
This way, users of the nixpkgs-channels repo don't have to wait for
all the slow stuff to finish (like index generation and copying of
ISOs).
2016-10-21 19:22:05 +02:00
Eelco Dolstra 7f2bbc884c Disable programs.sqlite generation 2016-10-21 18:27:28 +02:00
Eelco Dolstra eea0729481 Generate a simple directory listing 2016-09-29 11:50:14 +02:00
Eelco Dolstra 6cb926311e Only update the release link if the release has changed 2016-09-29 10:55:11 +02:00
Eelco Dolstra fcf5a77736 Add an empty file at the release prefix for nix-channel compatibility
nix-channel expects the prefix to exist (not return a 404).
2016-09-26 19:50:36 +02:00
Eelco Dolstra d3d33d5b25 Store releases in S3
Fixes #7.
2016-09-26 18:06:51 +02:00
Eelco Dolstra becf1f2ae3 Atomically update channels/.htaccess
Otherwise we might clobber the redirects if the disk is full, and in
any case there is a time window in which .htaccess is in an
inconsistent state.

Fixes https://github.com/NixOS/nixpkgs/issues/18544.
2016-09-12 23:20:12 +02:00
Eelco Dolstra 8a0ed20e6d Improve error message 2016-08-22 17:26:27 +02:00
Eelco Dolstra 76ca1da3ee Add $out/bin to $PATH 2016-08-22 17:18:44 +02:00
Eelco Dolstra c981c37438 Add README 2016-08-22 17:18:29 +02:00
Eelco Dolstra ab086b22ef Revive programs.sqlite generation
Fixes #4.
2016-08-11 14:41:26 +02:00
Eelco Dolstra 8f7fbdaff7 Die tabs die 2016-08-08 17:32:24 +02:00
Eelco Dolstra 72501d545d Remove obsolete binary patch generation scripts 2016-04-13 11:42:52 +02:00
Eelco Dolstra 852e1b6bf2 Drop 32-bit images (except iso-minimal) 2016-03-17 12:21:13 +01:00
Eelco Dolstra 000a9f2766 Workaround for https://github.com/NixOS/nix/issues/792 2016-03-07 11:12:24 +01:00
Eelco Dolstra 373d8b7b38 Fix NixOS 15.09 channel generation 2016-03-07 10:51:46 +01:00
Eelco Dolstra f46cb14bb7 Handle nixpkgs-unstable channel updates 2016-03-04 20:45:35 +01:00
Eelco Dolstra a6ab71c2f9 Update .htaccess redirects to latest NixOS ISOs 2016-03-04 20:27:15 +01:00
Eelco Dolstra 18af18f351 Fix Git path 2016-03-03 18:29:05 +01:00
Eelco Dolstra a058826231 Record top-level store paths of each release
We need this to do garbage collection on cache.nixos.org now that
MANIFEST is gone.
2016-03-03 15:59:25 +01:00
Eelco Dolstra f7beae78eb Rewrite mirror-nixos-branch in perl; handle new Hydra situation
We're no longer on a machine that contains the actual store paths of
the release, so we have to fetch them.
2016-03-01 20:03:10 +01:00
Eelco Dolstra 4de7a4a85c Remove tabs 2016-03-01 11:12:36 +01:00