Commit graph

256 commits

Author SHA1 Message Date
Eelco Dolstra 0df25e1e8f
Merge pull request #28 from NixOS/back-in-time-exit-code
mirror-nixos-branch.pl: exit 0 if the channel would go back in time
2019-11-15 17:06:16 +01:00
Graham Christensen 7ee70fd479
mirror-nixos-branch.pl: exit 0 if the channel would go back in time
Prometheus now has a configured alert if an update job fails for a
while. Going back in time is not so unusual:

1. a hydra evaluation completes with some failed jobs
2. the channel advances to this commit
3. an admin restarts a failed job
4. the channel would go backward because the evaluation is complete

Since it is not a problem, make it a non-problematic exit code.
2019-11-15 11:01:34 -05:00
Eelco Dolstra fac8ec969e
Revert "Namespace channel pushes in nixpkgs"
This reverts commit 38d2339ca2.
2019-10-29 12:29:31 +01:00
Eelco Dolstra 1856e2be31
Merge pull request #27 from NixOS/grahamc-patch-1
Namespace channel pushes in nixpkgs
2019-10-29 12:28:26 +01:00
Graham Christensen 38d2339ca2
Namespace channel pushes in nixpkgs
PRs to NixOS are a bit confused about if they should PR to release-* or nixos-* or nixpkgs-*. I think namespacing the refs will go a long way in making it more obvious.
2019-10-29 12:17:25 +01:00
Eelco Dolstra 6165ae664d Push to the nixpkgs repo
Fixes https://github.com/NixOS/nixpkgs/issues/71176.
2019-10-17 19:35:28 +02:00
Graham Christensen 3e89acae6a
Merge pull request #24 from samueldr/fix/wrong-hash-format
mirror: sets hash format to base16
2019-04-30 18:26:52 -04:00
Samuel Dionne-Riel 81dfa0d65a mirror: sets hash format to base16
From the nix 2.2 changelog:

> nix hash-file and nix hash-path now print hashes in SRI
> format by default. They also use SHA-256 by default
> instead of SHA-512 because that's what we use most of the
> time in Nixpkgs.
2019-04-30 18:24:33 -04:00
Graham Christensen d7a903d1c2
Merge pull request #23 from samueldr/fix/json-error-handling
file-cache: use the right exception type for parse error
2019-01-30 22:58:33 -05:00
Samuel Dionne-Riel 0effeee537 file-cache: use the right exception type for parse error
```
error: [json.exception.parse_error.101] parse error at 195: syntax error - invalid string: ill-formed UTF-8 byte; last read: '"bokml'; expected string literal
```

This is most probably caused by the upgrade in nixos-org-configurations/1b74e019b166d37a1510a99aeea86a0fa656854a
that, in turn, caused nlohmann_json to go from 2.1.0 -> 3.2.0

I haven't found concrete evidence in the changelog as to a change
changing the exception type. Though, the timing for failures matches
with this update.

Furthermore, once this fix is applied, launching the update locally
works fine.
2019-01-30 20:56:40 -05:00
Eelco Dolstra 90fcc7be84 Rsync channel redirects to nixos.org 2019-01-24 23:25:12 +01:00
Eelco Dolstra 7f650ec13e Revert "Store the channel redirects in S3"
This reverts commit 763db9879c.
2019-01-24 23:00:11 +01:00
Eelco Dolstra 763db9879c Store the channel redirects in S3 2019-01-24 19:07:26 +01:00
Eelco Dolstra 4a3ae75e4f Fix build 2019-01-24 19:06:58 +01:00
Eelco Dolstra 8dfa9f37c8
Merge pull request #22 from samueldr/fix/bttf-guard
Fixes channel going back in time due to incomplete change.
2018-08-30 16:06:37 +02:00
Samuel Dionne-Riel d313d2bd51 Fixes channel going back in time due to incomplete change.
The change in d3d33d5b25 changed how the
locally saved channel information was saved on the local machine.

 * d3d33d5b25 (diff-682bf482f6dd273f1c3c49afd99b4660R192)

The changeset did *not* change the guard clause, which now made it act
like this:

 1. Try readlink on the file
 2. (bail as it's undefined)

The new behaviour is as more or less expected:

 1. Reads the file if available
 2. Split the URL components
 3. Keep the last bit

This can give us an empty string, which will compare to `-1` thus
fulfilling the need.
2018-08-25 23:18:47 -04:00
Eelco Dolstra e9587f9388 d3g5gsiof5omrk.cloudfront.net -> releases.nixos.org 2018-07-19 16:01:55 +02:00
Eelco Dolstra 97d9e3ea01
Merge pull request #17 from samueldr/sha256sum-check
Adds filename to sha256 file for use with `sha256 -c`
2018-06-13 17:41:06 +02:00
Samuel Dionne-Riel 2181f3c02a Adds filename to sha256 file for use with sha256 -c
The manpage says:

> The  sums  are  computed as described in FIPS-180-2.  When checking, the input should be a former
> output of this program.  The default mode is to print a line with checksum, **a space, a  character
> indicating  input  mode ('*' for binary, ' ' for text or where binary is insignificant),** and name
> for each FILE.

By adding the filename to the generated sha256 file, a user would be
able to download an iso, the sha256 file, then use `sha256 -c
[file].iso.sha256` to verify the file.

* * *

This fixes nixos-homepage#224
2018-06-06 18:11:16 -04:00
Eelco Dolstra 9f96e1e25d
Improve error message 2018-02-06 16:45:13 +01:00
Eelco Dolstra 6a653c367e
Fix build 2018-01-04 15:33:22 +01:00
Eelco Dolstra c3a9ba32ee
Sync with nixUnstable 2017-07-21 12:48:15 +02:00
Eelco Dolstra c6428851dc
mirror-nixos-branch: Update the Git repo last 2017-07-21 12:48:14 +02:00
Eelco Dolstra 2f7994e3de
Remove bad check 2017-07-07 19:42:20 +02:00
Eelco Dolstra b6c049019a
mirror-nixos-branch.pl: Call index-debuginfo 2017-07-07 17:12:25 +02:00
Eelco Dolstra fcd6250fd0
mirror-channel.pl: Remove
This is no longer used, since Hydra uploads directly to S3 now. Also,
"nix copy" can copy to S3 now.
2017-07-07 17:06:21 +02:00
Eelco Dolstra a5e125330c
generate-programs-index: Check whether symlink targets exist
Fixes https://github.com/NixOS/nixpkgs/issues/26661
2017-07-07 16:45:45 +02:00
Eelco Dolstra 74c917454c
Add a command for indexing debug info files by build ID
For every file "lib/debug/.build-id/<build-id>.debug" in every store
path ending in -debug in a release, this command creates an S3 key
"debuginfo/<build-id>" specifying the location of the NAR that
contains the debug info file for that build ID. Thus,
https://cache.nixos.org/debuginfo allows debug info files to be looked
up by build ID. This allows debug info files to be fetched
automatically by dwarffs [1].

One minor issue is that in theory, multiple store paths could provide
debug info files for a particular build ID. So then when we garbage
collect a store path from the binary cache, we migth invalidate a
debug info link even though there are non-GC'ed store paths that
provide the file. Ah well.

I could have used HTTP redirects, but they're kind of a pain to
configure with S3.

[1] https://github.com/edolstra/dwarffs
2017-07-07 15:33:27 +02:00
Eelco Dolstra a91fb566a9
Factor out the file cache 2017-07-06 14:53:52 +02:00
Eelco Dolstra 3daca6b592
Update generate-programs-index to handle .ls 2017-04-18 16:41:11 +02:00
Eelco Dolstra dd0d2d2ca6
Update build 2017-04-18 15:26:38 +02:00
Eelco Dolstra a5368b6239 Merge pull request #12 from primeos/master
Fix the displayed hash name (SHA-512 -> SHA-256)
2017-03-31 12:04:05 +02:00
Michael Weiss c3e8e3badb Fix the displayed hash name (SHA-512 -> SHA-256)
The HTML pages for each channel are showing the SHA-256 hash of each
file but the table header actually referred to it as an "SHA-512 hash".
2017-03-31 06:39:24 +02:00
Eelco Dolstra 1c07d6dd08
Lock the Git repo 2016-12-05 11:50:17 +01:00
Eelco Dolstra 2fd4d6beea Vacuum programs.sqlite
This cuts a few hundred KiB from the database.
2016-11-10 18:27:04 +01:00
Eelco Dolstra 6883275653 Ignore package listings with malformed UTF-8
For example, https://cache.nixos.org/5q3s4vfwwd9n9499yg11ydziwxgn644m.ls.xz
contains the string "bokm<E5>l.alias". This causes nlohmann::json to
barf.
2016-11-10 18:10:08 +01:00
Eelco Dolstra 2176c939fc Compute FS closures in parallel
This is much faster.
2016-11-10 18:10:08 +01:00
Eelco Dolstra b74e03a870 Revert "Disable programs.sqlite generation"
This reverts commit 7f2bbc884c.
2016-11-10 18:10:08 +01:00
Eelco Dolstra ba19471e5b Use the .ls.xz listings generated by Hydra to speed up building programs.sqlite
This reduces the time to build the index for the regular (non-small) channel from ~10 hours to ~4 minutes.

Fixes #9.
2016-11-10 18:10:03 +01:00
Eelco Dolstra 6391c4dcec Fix check for release existance
We no longer have a github-link file, so this check always failed. As
a result, we were constantly re-uploading releases.
2016-10-21 20:10:53 +02:00
Eelco Dolstra 038c8ee8d4 Update the channels repo early
This way, users of the nixpkgs-channels repo don't have to wait for
all the slow stuff to finish (like index generation and copying of
ISOs).
2016-10-21 19:22:05 +02:00
Eelco Dolstra 7f2bbc884c Disable programs.sqlite generation 2016-10-21 18:27:28 +02:00
Eelco Dolstra eea0729481 Generate a simple directory listing 2016-09-29 11:50:14 +02:00
Eelco Dolstra 6cb926311e Only update the release link if the release has changed 2016-09-29 10:55:11 +02:00
Eelco Dolstra fcf5a77736 Add an empty file at the release prefix for nix-channel compatibility
nix-channel expects the prefix to exist (not return a 404).
2016-09-26 19:50:36 +02:00
Eelco Dolstra d3d33d5b25 Store releases in S3
Fixes #7.
2016-09-26 18:06:51 +02:00
Eelco Dolstra becf1f2ae3 Atomically update channels/.htaccess
Otherwise we might clobber the redirects if the disk is full, and in
any case there is a time window in which .htaccess is in an
inconsistent state.

Fixes https://github.com/NixOS/nixpkgs/issues/18544.
2016-09-12 23:20:12 +02:00
Eelco Dolstra 8a0ed20e6d Improve error message 2016-08-22 17:26:27 +02:00
Eelco Dolstra 76ca1da3ee Add $out/bin to $PATH 2016-08-22 17:18:44 +02:00
Eelco Dolstra c981c37438 Add README 2016-08-22 17:18:29 +02:00