Commit graph

166 commits

Author SHA1 Message Date
Rok Garbas d1fb4bf67a
Upload channels artifacts (or links to those artifacts) to download.nixos.org
This will enable us to move nixos.org website to other location as
described in nixos/nixos-homepage#326
2020-02-17 13:16:22 +01:00
Eelco Dolstra a04700b949
Merge pull request #25 from worldofpeace/plasma5-rename
rename iso_graphical to iso_plasma5
2020-01-30 14:32:42 +01:00
worldofpeace 751409599d rename iso_graphical to iso_plasma5 2020-01-30 08:28:20 -05:00
Eelco Dolstra 91f79b69a3
Fix adding nix to $PATH 2020-01-03 16:13:08 +01:00
Graham Christensen 579cbfc508
Merge pull request #31 from samueldr/fix/2020-01-02-mirror-issue-2
mirror-branch: Always exit with 1 on nix-instantiate error
2020-01-02 22:08:31 -05:00
Samuel Dionne-Riel c4a61dfe8a mirror-branch: Always exit with 1 on nix-instantiate error
Resolving the values and being fancy is harder than the actual benefits
would give out.

Return all the info we have, and let the pager'd person deal with the
data.
2020-01-02 22:04:43 -05:00
Graham Christensen 5146d2cb25
Merge pull request #30 from samueldr/fix/2020-01-02-mirror-issue
mirror-branch: Bails when failing to run nix-instantiate
2020-01-02 21:24:53 -05:00
Samuel Dionne-Riel cac10f5360 mirror-branch: Bails when failing to run nix-instantiate
A more durable solution would likely be to make a generic function that
wraps backticks and fails when $? is non-zero, rather than doing it
in-situ. There are other backtick uses that may be problematic
2020-01-02 21:19:58 -05:00
Eelco Dolstra 059ee08f5c Use nix-master 2019-12-30 21:51:21 +01:00
Eelco Dolstra aec148383a Convert to flake 2019-12-30 21:34:49 +01:00
Eelco Dolstra 0df25e1e8f
Merge pull request #28 from NixOS/back-in-time-exit-code
mirror-nixos-branch.pl: exit 0 if the channel would go back in time
2019-11-15 17:06:16 +01:00
Graham Christensen 7ee70fd479
mirror-nixos-branch.pl: exit 0 if the channel would go back in time
Prometheus now has a configured alert if an update job fails for a
while. Going back in time is not so unusual:

1. a hydra evaluation completes with some failed jobs
2. the channel advances to this commit
3. an admin restarts a failed job
4. the channel would go backward because the evaluation is complete

Since it is not a problem, make it a non-problematic exit code.
2019-11-15 11:01:34 -05:00
Eelco Dolstra fac8ec969e
Revert "Namespace channel pushes in nixpkgs"
This reverts commit 38d2339ca2.
2019-10-29 12:29:31 +01:00
Eelco Dolstra 1856e2be31
Merge pull request #27 from NixOS/grahamc-patch-1
Namespace channel pushes in nixpkgs
2019-10-29 12:28:26 +01:00
Graham Christensen 38d2339ca2
Namespace channel pushes in nixpkgs
PRs to NixOS are a bit confused about if they should PR to release-* or nixos-* or nixpkgs-*. I think namespacing the refs will go a long way in making it more obvious.
2019-10-29 12:17:25 +01:00
Eelco Dolstra 6165ae664d Push to the nixpkgs repo
Fixes https://github.com/NixOS/nixpkgs/issues/71176.
2019-10-17 19:35:28 +02:00
Graham Christensen 3e89acae6a
Merge pull request #24 from samueldr/fix/wrong-hash-format
mirror: sets hash format to base16
2019-04-30 18:26:52 -04:00
Samuel Dionne-Riel 81dfa0d65a mirror: sets hash format to base16
From the nix 2.2 changelog:

> nix hash-file and nix hash-path now print hashes in SRI
> format by default. They also use SHA-256 by default
> instead of SHA-512 because that's what we use most of the
> time in Nixpkgs.
2019-04-30 18:24:33 -04:00
Graham Christensen d7a903d1c2
Merge pull request #23 from samueldr/fix/json-error-handling
file-cache: use the right exception type for parse error
2019-01-30 22:58:33 -05:00
Samuel Dionne-Riel 0effeee537 file-cache: use the right exception type for parse error
```
error: [json.exception.parse_error.101] parse error at 195: syntax error - invalid string: ill-formed UTF-8 byte; last read: '"bokml'; expected string literal
```

This is most probably caused by the upgrade in nixos-org-configurations/1b74e019b166d37a1510a99aeea86a0fa656854a
that, in turn, caused nlohmann_json to go from 2.1.0 -> 3.2.0

I haven't found concrete evidence in the changelog as to a change
changing the exception type. Though, the timing for failures matches
with this update.

Furthermore, once this fix is applied, launching the update locally
works fine.
2019-01-30 20:56:40 -05:00
Eelco Dolstra 90fcc7be84 Rsync channel redirects to nixos.org 2019-01-24 23:25:12 +01:00
Eelco Dolstra 7f650ec13e Revert "Store the channel redirects in S3"
This reverts commit 763db9879c.
2019-01-24 23:00:11 +01:00
Eelco Dolstra 763db9879c Store the channel redirects in S3 2019-01-24 19:07:26 +01:00
Eelco Dolstra 4a3ae75e4f Fix build 2019-01-24 19:06:58 +01:00
Eelco Dolstra 8dfa9f37c8
Merge pull request #22 from samueldr/fix/bttf-guard
Fixes channel going back in time due to incomplete change.
2018-08-30 16:06:37 +02:00
Samuel Dionne-Riel d313d2bd51 Fixes channel going back in time due to incomplete change.
The change in d3d33d5b25 changed how the
locally saved channel information was saved on the local machine.

 * d3d33d5b25 (diff-682bf482f6dd273f1c3c49afd99b4660R192)

The changeset did *not* change the guard clause, which now made it act
like this:

 1. Try readlink on the file
 2. (bail as it's undefined)

The new behaviour is as more or less expected:

 1. Reads the file if available
 2. Split the URL components
 3. Keep the last bit

This can give us an empty string, which will compare to `-1` thus
fulfilling the need.
2018-08-25 23:18:47 -04:00
Eelco Dolstra e9587f9388 d3g5gsiof5omrk.cloudfront.net -> releases.nixos.org 2018-07-19 16:01:55 +02:00
Eelco Dolstra 97d9e3ea01
Merge pull request #17 from samueldr/sha256sum-check
Adds filename to sha256 file for use with `sha256 -c`
2018-06-13 17:41:06 +02:00
Samuel Dionne-Riel 2181f3c02a Adds filename to sha256 file for use with sha256 -c
The manpage says:

> The  sums  are  computed as described in FIPS-180-2.  When checking, the input should be a former
> output of this program.  The default mode is to print a line with checksum, **a space, a  character
> indicating  input  mode ('*' for binary, ' ' for text or where binary is insignificant),** and name
> for each FILE.

By adding the filename to the generated sha256 file, a user would be
able to download an iso, the sha256 file, then use `sha256 -c
[file].iso.sha256` to verify the file.

* * *

This fixes nixos-homepage#224
2018-06-06 18:11:16 -04:00
Eelco Dolstra 9f96e1e25d
Improve error message 2018-02-06 16:45:13 +01:00
Eelco Dolstra 6a653c367e
Fix build 2018-01-04 15:33:22 +01:00
Eelco Dolstra c3a9ba32ee
Sync with nixUnstable 2017-07-21 12:48:15 +02:00
Eelco Dolstra c6428851dc
mirror-nixos-branch: Update the Git repo last 2017-07-21 12:48:14 +02:00
Eelco Dolstra 2f7994e3de
Remove bad check 2017-07-07 19:42:20 +02:00
Eelco Dolstra b6c049019a
mirror-nixos-branch.pl: Call index-debuginfo 2017-07-07 17:12:25 +02:00
Eelco Dolstra fcd6250fd0
mirror-channel.pl: Remove
This is no longer used, since Hydra uploads directly to S3 now. Also,
"nix copy" can copy to S3 now.
2017-07-07 17:06:21 +02:00
Eelco Dolstra a5e125330c
generate-programs-index: Check whether symlink targets exist
Fixes https://github.com/NixOS/nixpkgs/issues/26661
2017-07-07 16:45:45 +02:00
Eelco Dolstra 74c917454c
Add a command for indexing debug info files by build ID
For every file "lib/debug/.build-id/<build-id>.debug" in every store
path ending in -debug in a release, this command creates an S3 key
"debuginfo/<build-id>" specifying the location of the NAR that
contains the debug info file for that build ID. Thus,
https://cache.nixos.org/debuginfo allows debug info files to be looked
up by build ID. This allows debug info files to be fetched
automatically by dwarffs [1].

One minor issue is that in theory, multiple store paths could provide
debug info files for a particular build ID. So then when we garbage
collect a store path from the binary cache, we migth invalidate a
debug info link even though there are non-GC'ed store paths that
provide the file. Ah well.

I could have used HTTP redirects, but they're kind of a pain to
configure with S3.

[1] https://github.com/edolstra/dwarffs
2017-07-07 15:33:27 +02:00
Eelco Dolstra a91fb566a9
Factor out the file cache 2017-07-06 14:53:52 +02:00
Eelco Dolstra 3daca6b592
Update generate-programs-index to handle .ls 2017-04-18 16:41:11 +02:00
Eelco Dolstra dd0d2d2ca6
Update build 2017-04-18 15:26:38 +02:00
Eelco Dolstra a5368b6239 Merge pull request #12 from primeos/master
Fix the displayed hash name (SHA-512 -> SHA-256)
2017-03-31 12:04:05 +02:00
Michael Weiss c3e8e3badb Fix the displayed hash name (SHA-512 -> SHA-256)
The HTML pages for each channel are showing the SHA-256 hash of each
file but the table header actually referred to it as an "SHA-512 hash".
2017-03-31 06:39:24 +02:00
Eelco Dolstra 1c07d6dd08
Lock the Git repo 2016-12-05 11:50:17 +01:00
Eelco Dolstra 2fd4d6beea Vacuum programs.sqlite
This cuts a few hundred KiB from the database.
2016-11-10 18:27:04 +01:00
Eelco Dolstra 6883275653 Ignore package listings with malformed UTF-8
For example, https://cache.nixos.org/5q3s4vfwwd9n9499yg11ydziwxgn644m.ls.xz
contains the string "bokm<E5>l.alias". This causes nlohmann::json to
barf.
2016-11-10 18:10:08 +01:00
Eelco Dolstra 2176c939fc Compute FS closures in parallel
This is much faster.
2016-11-10 18:10:08 +01:00
Eelco Dolstra b74e03a870 Revert "Disable programs.sqlite generation"
This reverts commit 7f2bbc884c.
2016-11-10 18:10:08 +01:00
Eelco Dolstra ba19471e5b Use the .ls.xz listings generated by Hydra to speed up building programs.sqlite
This reduces the time to build the index for the regular (non-small) channel from ~10 hours to ~4 minutes.

Fixes #9.
2016-11-10 18:10:03 +01:00
Eelco Dolstra 6391c4dcec Fix check for release existance
We no longer have a github-link file, so this check always failed. As
a result, we were constantly re-uploading releases.
2016-10-21 20:10:53 +02:00