From e30d1daf26bfd4a4647bb8c55c7643f7308e4f88 Mon Sep 17 00:00:00 2001 From: Robert Hensing Date: Wed, 16 Jun 2021 19:06:41 +0200 Subject: [PATCH 001/112] installer: Jeeze -> Oh no Even if it doesn't offend me or probably most people, this word can be considered mildly blasphemous. --- scripts/install-multi-user.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/install-multi-user.sh b/scripts/install-multi-user.sh index e1046c19c..b06847ffe 100644 --- a/scripts/install-multi-user.sh +++ b/scripts/install-multi-user.sh @@ -329,7 +329,7 @@ finish_fail() { finish_cleanup failure < Date: Sat, 11 Jun 2022 13:30:51 -0500 Subject: [PATCH 002/112] doc: add install test info to hacking.md --- doc/manual/src/contributing/hacking.md | 64 +++++++++++++++++++++++++- 1 file changed, 63 insertions(+), 1 deletion(-) diff --git a/doc/manual/src/contributing/hacking.md b/doc/manual/src/contributing/hacking.md index 59ce5cac7..9a371afa7 100644 --- a/doc/manual/src/contributing/hacking.md +++ b/doc/manual/src/contributing/hacking.md @@ -83,7 +83,7 @@ by: $ nix develop ``` -## Testing +## Testing Nix Nix comes with three different flavors of tests: unit, functional and integration. @@ -108,3 +108,65 @@ These tests include everything that needs to interact with external services or Because these tests are expensive and require more than what the standard github-actions setup provides, they only run on the master branch (on ). You can run them manually with `nix build .#hydraJobs.tests.{testName}` or `nix-build -A hydraJobs.tests.{testName}` + +## Testing the install scripts + +Testing the install scripts has traditionally been tedious, but you can now do this much more easily via the GitHub Actions CI runs (at least for platforms that Github Actions supports). + +If you've already pushed to a fork of Nix on GitHub before, you may have noticed that the CI workflows in your fork list skipped "installer" and "installer_test" jobs. Once your Nix fork is set up correctly, pushing to it will also run these jobs. +- The `installer` job will generate installers for these platforms: x86_64-linux, armv6l-linux, armv7l-linux, x86_64-darwin. While this installer is in your Cachix cache, you can use it for manual testing on any of these platforms. +- the `installer_test` job will try to use this installer and run a trivial Nix command on `ubuntu-latest` and `macos-latest`. + +### One-time setup +1. Have a GitHub account with a fork of the Nix repo. +2. At cachix.org: + - Create or log in to an account. + - Create a Cachix cache using the format `-nix-install-tests`. + - Navigate to the new cache > Settings > Auth Tokens. + - Generate a new cachix auth token and copy the generated value. +4. At github.com: + - Navigate to your Nix fork > Settings > Secrets > Actions > New repository secret. + - Name the secret `CACHIX_AUTH_TOKEN` + - Paste the copied value of the Cachix cache auth token. + +### Using the CI-generated installer for manual testing + +After the CI run completes, you can check the output to extract the installer url: +1. Click into the detailed view of the CI run. +2. Click into any `installer_test` run (the URL you're here to extract will be the same in all of them). +3. Click into the `Run cachix/install-nix-action@v...` step and click the detail triangle next to the first log line (it will also be `Run cachix/install-nix-action@v...`) +4. Copy the install_url +5. To generate an install command, plug this install_url and your github username into this template: + + ```console + sh <(curl -L ) --tarball-url-prefix https://-nix-install-tests.cachix.org/serve + ``` + + From 64404220f54a36d3457433580ab8d78cf016572d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Sol=C3=A8ne=20Rapenne?= Date: Wed, 20 Jul 2022 14:53:03 +0200 Subject: [PATCH 003/112] nix shell: document how to invoke multiple commands from the command line --- src/nix/shell.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/nix/shell.md b/src/nix/shell.md index 90b81fb2f..161fdeb8d 100644 --- a/src/nix/shell.md +++ b/src/nix/shell.md @@ -23,6 +23,12 @@ R""( Hi everybody! ``` +* Run multiple commands in a shell environment: + + ```console + # nix shell nixpkgs#gnumake -c /bin/sh -c "cd src && make" + ``` + * Run GNU Hello in a chroot store: ```console From 92bae33ca5db60e729ce07156ebf1c06cf865cc8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Sol=C3=A8ne=20Rapenne?= Date: Thu, 21 Jul 2022 14:25:07 +0200 Subject: [PATCH 004/112] nix shell: example shouldn't use an absolute path for the shell Co-authored-by: Eelco Dolstra --- src/nix/shell.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/nix/shell.md b/src/nix/shell.md index 161fdeb8d..9fa1031f5 100644 --- a/src/nix/shell.md +++ b/src/nix/shell.md @@ -26,7 +26,7 @@ R""( * Run multiple commands in a shell environment: ```console - # nix shell nixpkgs#gnumake -c /bin/sh -c "cd src && make" + # nix shell nixpkgs#gnumake -c sh -c "cd src && make" ``` * Run GNU Hello in a chroot store: From a5be5e01200a12cc34d0e3a2e3f964d5c95208b9 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 4 Aug 2022 14:07:06 -0700 Subject: [PATCH 005/112] doc/manual: define {local,remote} store, binary cache, substituter Nix veterans intuitively know what the following terms mean. They are used in several places in the nix documentation, but never defined: - local store - remote store - binary cache - substituter In particular, I found the last two terms to be confusingly similar. Let's give definitions for them. --- doc/manual/src/SUMMARY.md.in | 1 + .../src/package-management/terminology.md | 27 +++++++++++++++++++ 2 files changed, 28 insertions(+) create mode 100644 doc/manual/src/package-management/terminology.md diff --git a/doc/manual/src/SUMMARY.md.in b/doc/manual/src/SUMMARY.md.in index a47d39f31..f8da2247b 100644 --- a/doc/manual/src/SUMMARY.md.in +++ b/doc/manual/src/SUMMARY.md.in @@ -22,6 +22,7 @@ - [Garbage Collector Roots](package-management/garbage-collector-roots.md) - [Channels](package-management/channels.md) - [Sharing Packages Between Machines](package-management/sharing-packages.md) + - [Terminology](package-management/terminology.md) - [Serving a Nix store via HTTP](package-management/binary-cache-substituter.md) - [Copying Closures via SSH](package-management/copy-closure.md) - [Serving a Nix store via SSH](package-management/ssh-substituter.md) diff --git a/doc/manual/src/package-management/terminology.md b/doc/manual/src/package-management/terminology.md new file mode 100644 index 000000000..28e2a1f0b --- /dev/null +++ b/doc/manual/src/package-management/terminology.md @@ -0,0 +1,27 @@ +# Terminology + +A *local store* exists on the local filesystem of the machine where +Nix is invoked. The `/nix/store` directory is one example of a +local store. You can use other local stores by passing the +`--store` flag to `nix`. + +A *remote store* is a store which exists anywhere other than the +local filesystem. One example is the `/nix/store` directory on +another machine, accessed via `ssh` or served by the `nix-serve` +Perl script. + +A *binary cache* is a remote store which is not the local store of +any machine. Examples of binary caches include S3 buckets and the +[NixOS binary cache](https://cache.nixos.org). Binary caches use a +disk layout that is different from local stores; in particular, they +keep metadata and signatures in `.narinfo` files rather than in +`/nix/var/nix/db`. + +A *substituter* is a store other than `/nix/store` from which nix will +copy the realisation of a derivation instead of building it. Nix will +not copy a realisation from a remote store unless one of the following +is true: + +- the realisation is signed by one of the `trusted-public-key`s +- the substituter is in the `trusted-substituters` list +- the `no-require-sigs` option has been set to disable signature checking From 56d4fc194ba90ee4e559a07a895f6bf4a61ef462 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 4 Aug 2022 14:14:24 -0700 Subject: [PATCH 006/112] fourth trust condition: FODs --- doc/manual/src/package-management/terminology.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/manual/src/package-management/terminology.md b/doc/manual/src/package-management/terminology.md index 28e2a1f0b..1722a9fc8 100644 --- a/doc/manual/src/package-management/terminology.md +++ b/doc/manual/src/package-management/terminology.md @@ -25,3 +25,5 @@ is true: - the realisation is signed by one of the `trusted-public-key`s - the substituter is in the `trusted-substituters` list - the `no-require-sigs` option has been set to disable signature checking +- the derivation is a fixed-output derivation + From 8f44d24c525160b2ddef5e18a4af4ce667e23e9f Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 4 Aug 2022 14:19:25 -0700 Subject: [PATCH 007/112] !fixup whitespace --- doc/manual/src/package-management/terminology.md | 1 - 1 file changed, 1 deletion(-) diff --git a/doc/manual/src/package-management/terminology.md b/doc/manual/src/package-management/terminology.md index 1722a9fc8..d800bafc1 100644 --- a/doc/manual/src/package-management/terminology.md +++ b/doc/manual/src/package-management/terminology.md @@ -26,4 +26,3 @@ is true: - the substituter is in the `trusted-substituters` list - the `no-require-sigs` option has been set to disable signature checking - the derivation is a fixed-output derivation - From 62674659ed7b7fc6a2c884f52df2474e344400f8 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 4 Aug 2022 14:21:17 -0700 Subject: [PATCH 008/112] !fixup capitalize Nix --- doc/manual/src/package-management/terminology.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/src/package-management/terminology.md b/doc/manual/src/package-management/terminology.md index d800bafc1..8dc2ede33 100644 --- a/doc/manual/src/package-management/terminology.md +++ b/doc/manual/src/package-management/terminology.md @@ -17,7 +17,7 @@ disk layout that is different from local stores; in particular, they keep metadata and signatures in `.narinfo` files rather than in `/nix/var/nix/db`. -A *substituter* is a store other than `/nix/store` from which nix will +A *substituter* is a store other than `/nix/store` from which Nix will copy the realisation of a derivation instead of building it. Nix will not copy a realisation from a remote store unless one of the following is true: From 1b97f3872ed70d9ad5d19d27dc56b42ba4d26382 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 4 Aug 2022 14:22:14 -0700 Subject: [PATCH 009/112] !fixup: transposed characters --- doc/manual/src/package-management/terminology.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/src/package-management/terminology.md b/doc/manual/src/package-management/terminology.md index 8dc2ede33..6ad0f6833 100644 --- a/doc/manual/src/package-management/terminology.md +++ b/doc/manual/src/package-management/terminology.md @@ -22,7 +22,7 @@ copy the realisation of a derivation instead of building it. Nix will not copy a realisation from a remote store unless one of the following is true: -- the realisation is signed by one of the `trusted-public-key`s +- the realisation is signed by one of the `trusted-public-keys` - the substituter is in the `trusted-substituters` list - the `no-require-sigs` option has been set to disable signature checking - the derivation is a fixed-output derivation From aae771cad26a3803ef0a0855c782823d22949cf3 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Fri, 5 Aug 2022 10:12:46 -0700 Subject: [PATCH 010/112] !implement https://github.com/NixOS/nix/pull/6870#discussion_r938912244 --- doc/manual/src/package-management/terminology.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/doc/manual/src/package-management/terminology.md b/doc/manual/src/package-management/terminology.md index 6ad0f6833..241bb6c5a 100644 --- a/doc/manual/src/package-management/terminology.md +++ b/doc/manual/src/package-management/terminology.md @@ -18,11 +18,10 @@ keep metadata and signatures in `.narinfo` files rather than in `/nix/var/nix/db`. A *substituter* is a store other than `/nix/store` from which Nix will -copy the realisation of a derivation instead of building it. Nix will -not copy a realisation from a remote store unless one of the following -is true: +copy a store path instead of building it. Nix will not copy a store +path from a remote store unless one of the following is true: -- the realisation is signed by one of the `trusted-public-keys` +- the store object is signed by one of the `trusted-public-keys` - the substituter is in the `trusted-substituters` list - the `no-require-sigs` option has been set to disable signature checking -- the derivation is a fixed-output derivation +- the store object is the realisation of a fixed-output derivation From 2eb74c918dc7dc04ed36b3fdcd95406007d97690 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Fri, 5 Aug 2022 10:13:41 -0700 Subject: [PATCH 011/112] derivations do not need to be signed --- doc/manual/src/package-management/terminology.md | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/manual/src/package-management/terminology.md b/doc/manual/src/package-management/terminology.md index 241bb6c5a..4b9e68de9 100644 --- a/doc/manual/src/package-management/terminology.md +++ b/doc/manual/src/package-management/terminology.md @@ -24,4 +24,5 @@ path from a remote store unless one of the following is true: - the store object is signed by one of the `trusted-public-keys` - the substituter is in the `trusted-substituters` list - the `no-require-sigs` option has been set to disable signature checking +- the store object is a derivation - the store object is the realisation of a fixed-output derivation From 66a93a76b9842ac18188b91f5a30c4ac4f2b6118 Mon Sep 17 00:00:00 2001 From: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com> Date: Fri, 5 Aug 2022 17:15:37 +0000 Subject: [PATCH 012/112] Update doc/manual/src/package-management/terminology.md Co-authored-by: Attila Gulyas --- .../src/package-management/terminology.md | 24 ++++++++++++------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/doc/manual/src/package-management/terminology.md b/doc/manual/src/package-management/terminology.md index 4b9e68de9..564667f93 100644 --- a/doc/manual/src/package-management/terminology.md +++ b/doc/manual/src/package-management/terminology.md @@ -1,14 +1,22 @@ # Terminology -A *local store* exists on the local filesystem of the machine where -Nix is invoked. The `/nix/store` directory is one example of a -local store. You can use other local stores by passing the -`--store` flag to `nix`. +From the perspective of the location where Nix is +invoked1, the Nix store can be referred to +as a "_local_" or a "_remote_" one: -A *remote store* is a store which exists anywhere other than the -local filesystem. One example is the `/nix/store` directory on -another machine, accessed via `ssh` or served by the `nix-serve` -Perl script. +\[1]: Where "invoking Nix" means an executing a Nix core +action/operation on a Nix store. For example, using any CLI +commands from the `NixOS/nix` implementation. + ++ A *local store* exists on the local filesystem of + the machine where Nix is invoked. You can use other + local stores by passing the `--store` flag to the + `nix` command. + ++ A *remote store* exists anywhere other than the + local filesystem. One example is the `/nix/store` + directory on another machine, accessed via `ssh` or + served by the `nix-serve` Perl script. A *binary cache* is a remote store which is not the local store of any machine. Examples of binary caches include S3 buckets and the From d5506aa71200425b65cc1777077478f5ff8d2aff Mon Sep 17 00:00:00 2001 From: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com> Date: Fri, 5 Aug 2022 17:19:52 +0000 Subject: [PATCH 013/112] Update doc/manual/src/package-management/terminology.md Co-authored-by: Attila Gulyas --- doc/manual/src/package-management/terminology.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/doc/manual/src/package-management/terminology.md b/doc/manual/src/package-management/terminology.md index 564667f93..b3e9ea040 100644 --- a/doc/manual/src/package-management/terminology.md +++ b/doc/manual/src/package-management/terminology.md @@ -18,12 +18,10 @@ commands from the `NixOS/nix` implementation. directory on another machine, accessed via `ssh` or served by the `nix-serve` Perl script. -A *binary cache* is a remote store which is not the local store of -any machine. Examples of binary caches include S3 buckets and the -[NixOS binary cache](https://cache.nixos.org). Binary caches use a -disk layout that is different from local stores; in particular, they -keep metadata and signatures in `.narinfo` files rather than in -`/nix/var/nix/db`. +A *binary cache* is a specialized Nix store whose metadata and +signatures are kept in `.narinfo` files rather than in the Nix +database. Examples of binary caches include S3 buckets and the +[NixOS binary cache](https://cache.nixos.org). A *substituter* is a store other than `/nix/store` from which Nix will copy a store path instead of building it. Nix will not copy a store From 4de95f7f565df71d8ebddb7434e2b0feb49a833b Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Fri, 5 Aug 2022 10:33:48 -0700 Subject: [PATCH 014/112] gesture at explanation of why binary caches exist --- doc/manual/src/package-management/terminology.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/doc/manual/src/package-management/terminology.md b/doc/manual/src/package-management/terminology.md index b3e9ea040..493f5f03e 100644 --- a/doc/manual/src/package-management/terminology.md +++ b/doc/manual/src/package-management/terminology.md @@ -18,10 +18,12 @@ commands from the `NixOS/nix` implementation. directory on another machine, accessed via `ssh` or served by the `nix-serve` Perl script. -A *binary cache* is a specialized Nix store whose metadata and -signatures are kept in `.narinfo` files rather than in the Nix -database. Examples of binary caches include S3 buckets and the -[NixOS binary cache](https://cache.nixos.org). +A *binary cache* is a Nix store which uses a different format: its +metadata and signatures are kept in `.narinfo` files rather than in a +Nix database. This different format simplifies serving store objects +over the network, but cannot host builds. Examples of binary caches +include S3 buckets and the [NixOS binary +cache](https://cache.nixos.org). A *substituter* is a store other than `/nix/store` from which Nix will copy a store path instead of building it. Nix will not copy a store From 1d3b92e80ca1564bf2c5ee207df707e215188633 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Fri, 5 Aug 2022 10:39:43 -0700 Subject: [PATCH 015/112] move package-management/terminology into glossary.md --- doc/manual/src/SUMMARY.md.in | 1 - doc/manual/src/glossary.md | 37 +++++++++++++++++++ .../src/package-management/terminology.md | 36 ------------------ 3 files changed, 37 insertions(+), 37 deletions(-) delete mode 100644 doc/manual/src/package-management/terminology.md diff --git a/doc/manual/src/SUMMARY.md.in b/doc/manual/src/SUMMARY.md.in index f8da2247b..a47d39f31 100644 --- a/doc/manual/src/SUMMARY.md.in +++ b/doc/manual/src/SUMMARY.md.in @@ -22,7 +22,6 @@ - [Garbage Collector Roots](package-management/garbage-collector-roots.md) - [Channels](package-management/channels.md) - [Sharing Packages Between Machines](package-management/sharing-packages.md) - - [Terminology](package-management/terminology.md) - [Serving a Nix store via HTTP](package-management/binary-cache-substituter.md) - [Copying Closures via SSH](package-management/copy-closure.md) - [Serving a Nix store via SSH](package-management/ssh-substituter.md) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index aa0ac78cb..f4c51588d 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -11,6 +11,32 @@ The location in the file system where store objects live. Typically `/nix/store`. + From the perspective of the location where Nix is + invoked1, the Nix store can be referred to + as a "_local_" or a "_remote_" one: + + \[1]: Where "invoking Nix" means an executing a Nix core + action/operation on a Nix store. For example, using any CLI + commands from the `NixOS/nix` implementation. + + + A *local store* exists on the local filesystem of + the machine where Nix is invoked. You can use other + local stores by passing the `--store` flag to the + `nix` command. + + + A *remote store* exists anywhere other than the + local filesystem. One example is the `/nix/store` + directory on another machine, accessed via `ssh` or + served by the `nix-serve` Perl script. + + - [binary cache]{#binary-cache}\ + A *binary cache* is a Nix store which uses a different format: its + metadata and signatures are kept in `.narinfo` files rather than in a + Nix database. This different format simplifies serving store objects + over the network, but cannot host builds. Examples of binary caches + include S3 buckets and the [NixOS binary + cache](https://cache.nixos.org). + - [store path]{#gloss-store-path}\ The location in the file system of a store object, i.e., an immediate child of the Nix store directory. @@ -29,6 +55,17 @@ store object by downloading a pre-built version of the store object from some server. + - [substituter]{#gloss-substituter}\ + A *substituter* is a store other than `/nix/store` from which Nix will + copy a store path instead of building it. Nix will not copy a store + path from a remote store unless one of the following is true: + + - the store object is signed by one of the `trusted-public-keys` + - the substituter is in the `trusted-substituters` list + - the `no-require-sigs` option has been set to disable signature checking + - the store object is a derivation + - the store object is the realisation of a fixed-output derivation + - [purity]{#gloss-purity}\ The assumption that equal Nix derivations when run always produce the same output. This cannot be guaranteed in general (e.g., a diff --git a/doc/manual/src/package-management/terminology.md b/doc/manual/src/package-management/terminology.md deleted file mode 100644 index 493f5f03e..000000000 --- a/doc/manual/src/package-management/terminology.md +++ /dev/null @@ -1,36 +0,0 @@ -# Terminology - -From the perspective of the location where Nix is -invoked1, the Nix store can be referred to -as a "_local_" or a "_remote_" one: - -\[1]: Where "invoking Nix" means an executing a Nix core -action/operation on a Nix store. For example, using any CLI -commands from the `NixOS/nix` implementation. - -+ A *local store* exists on the local filesystem of - the machine where Nix is invoked. You can use other - local stores by passing the `--store` flag to the - `nix` command. - -+ A *remote store* exists anywhere other than the - local filesystem. One example is the `/nix/store` - directory on another machine, accessed via `ssh` or - served by the `nix-serve` Perl script. - -A *binary cache* is a Nix store which uses a different format: its -metadata and signatures are kept in `.narinfo` files rather than in a -Nix database. This different format simplifies serving store objects -over the network, but cannot host builds. Examples of binary caches -include S3 buckets and the [NixOS binary -cache](https://cache.nixos.org). - -A *substituter* is a store other than `/nix/store` from which Nix will -copy a store path instead of building it. Nix will not copy a store -path from a remote store unless one of the following is true: - -- the store object is signed by one of the `trusted-public-keys` -- the substituter is in the `trusted-substituters` list -- the `no-require-sigs` option has been set to disable signature checking -- the store object is a derivation -- the store object is the realisation of a fixed-output derivation From b5d85f0922e0f4a9585a281f6d938ec67cd07349 Mon Sep 17 00:00:00 2001 From: "Travis A. Everett" Date: Fri, 5 Aug 2022 13:49:18 -0500 Subject: [PATCH 016/112] Apply suggestions from code review Co-authored-by: Valentin Gagarin --- doc/manual/src/contributing/hacking.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/doc/manual/src/contributing/hacking.md b/doc/manual/src/contributing/hacking.md index 9a371afa7..86c6522f2 100644 --- a/doc/manual/src/contributing/hacking.md +++ b/doc/manual/src/contributing/hacking.md @@ -113,11 +113,12 @@ You can run them manually with `nix build .#hydraJobs.tests.{testName}` or `nix- Testing the install scripts has traditionally been tedious, but you can now do this much more easily via the GitHub Actions CI runs (at least for platforms that Github Actions supports). -If you've already pushed to a fork of Nix on GitHub before, you may have noticed that the CI workflows in your fork list skipped "installer" and "installer_test" jobs. Once your Nix fork is set up correctly, pushing to it will also run these jobs. -- The `installer` job will generate installers for these platforms: x86_64-linux, armv6l-linux, armv7l-linux, x86_64-darwin. While this installer is in your Cachix cache, you can use it for manual testing on any of these platforms. +If you've already pushed to a fork of Nix on GitHub before, you may have noticed that the CI workflows in your fork list skipped `installer` and `installer_test` jobs. Once your Nix fork is set up correctly, pushing to it will also run these jobs. +- The `installer` job will generate installers for these platforms: `x86_64-linux`, `armv6l-linux`, `armv7l-linux`, `x86_64-darwin`. While this installer is in your Cachix cache, you can use it for manual testing on any of these platforms. - the `installer_test` job will try to use this installer and run a trivial Nix command on `ubuntu-latest` and `macos-latest`. ### One-time setup + 1. Have a GitHub account with a fork of the Nix repo. 2. At cachix.org: - Create or log in to an account. From 9b7bd2dd1fc83f6df449fce3967a95cb098ca4b2 Mon Sep 17 00:00:00 2001 From: "Travis A. Everett" Date: Mon, 8 Aug 2022 10:04:27 -0500 Subject: [PATCH 017/112] Apply suggestions from code review Co-authored-by: Valentin Gagarin --- doc/manual/src/contributing/hacking.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/doc/manual/src/contributing/hacking.md b/doc/manual/src/contributing/hacking.md index 86c6522f2..d8a8c8591 100644 --- a/doc/manual/src/contributing/hacking.md +++ b/doc/manual/src/contributing/hacking.md @@ -114,7 +114,13 @@ You can run them manually with `nix build .#hydraJobs.tests.{testName}` or `nix- Testing the install scripts has traditionally been tedious, but you can now do this much more easily via the GitHub Actions CI runs (at least for platforms that Github Actions supports). If you've already pushed to a fork of Nix on GitHub before, you may have noticed that the CI workflows in your fork list skipped `installer` and `installer_test` jobs. Once your Nix fork is set up correctly, pushing to it will also run these jobs. -- The `installer` job will generate installers for these platforms: `x86_64-linux`, `armv6l-linux`, `armv7l-linux`, `x86_64-darwin`. While this installer is in your Cachix cache, you can use it for manual testing on any of these platforms. +- The `installer` job will generate installers for these platforms: + - `x86_64-linux` + - `armv6l-linux` + - `armv7l-linux` + - `x86_64-darwin`. + + While this installer is in your Cachix cache, you can use it for manual testing on any of these platforms. - the `installer_test` job will try to use this installer and run a trivial Nix command on `ubuntu-latest` and `macos-latest`. ### One-time setup From bac1e1bf8c359b5e6831c3974a05bdce867775a5 Mon Sep 17 00:00:00 2001 From: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com> Date: Mon, 8 Aug 2022 17:52:31 +0000 Subject: [PATCH 018/112] Update doc/manual/src/glossary.md Co-authored-by: Valentin Gagarin --- doc/manual/src/glossary.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index f4c51588d..77de58965 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -12,13 +12,9 @@ `/nix/store`. From the perspective of the location where Nix is - invoked1, the Nix store can be referred to + invoked, the Nix store can be referred to as a "_local_" or a "_remote_" one: - \[1]: Where "invoking Nix" means an executing a Nix core - action/operation on a Nix store. For example, using any CLI - commands from the `NixOS/nix` implementation. - + A *local store* exists on the local filesystem of the machine where Nix is invoked. You can use other local stores by passing the `--store` flag to the From d8e54d19f71f78540dd967b2e42be6a5d8a0b1bb Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Mon, 22 Aug 2022 12:50:48 +0200 Subject: [PATCH 019/112] Revert "Merge pull request #6420 from nix-community/doc-what-is-nix" This reverts commit 81e101345fda2a8651c470f08b364a1ca6fa37cf, reversing changes made to 7d1280bbaf7f4cd142c2259dec620c42bf6f96fd. --- doc/manual/src/SUMMARY.md.in | 8 - doc/manual/src/architecture/architecture.md | 79 --------- doc/manual/src/architecture/store/fso.md | 69 -------- doc/manual/src/architecture/store/path.md | 105 ------------ doc/manual/src/architecture/store/store.md | 151 ------------------ .../store/store/build-system-terminology.md | 32 ---- .../src/architecture/store/store/closure.md | 29 ---- 7 files changed, 473 deletions(-) delete mode 100644 doc/manual/src/architecture/architecture.md delete mode 100644 doc/manual/src/architecture/store/fso.md delete mode 100644 doc/manual/src/architecture/store/path.md delete mode 100644 doc/manual/src/architecture/store/store.md delete mode 100644 doc/manual/src/architecture/store/store/build-system-terminology.md delete mode 100644 doc/manual/src/architecture/store/store/closure.md diff --git a/doc/manual/src/SUMMARY.md.in b/doc/manual/src/SUMMARY.md.in index 8fbb59716..084c8f442 100644 --- a/doc/manual/src/SUMMARY.md.in +++ b/doc/manual/src/SUMMARY.md.in @@ -59,14 +59,6 @@ @manpages@ - [Files](command-ref/files.md) - [nix.conf](command-ref/conf-file.md) - - [Glossary](glossary.md) - [Contributing](contributing/contributing.md) - [Hacking](contributing/hacking.md) diff --git a/doc/manual/src/architecture/architecture.md b/doc/manual/src/architecture/architecture.md deleted file mode 100644 index 41deb07af..000000000 --- a/doc/manual/src/architecture/architecture.md +++ /dev/null @@ -1,79 +0,0 @@ -# Architecture - -*(This chapter is unstable and a work in progress. Incoming links may rot.)* - -This chapter describes how Nix works. -It should help users understand why Nix behaves as it does, and it should help developers understand how to modify Nix and how to write similar tools. - -## Overview - -Nix consists of [hierarchical layers][layer-architecture]. - -``` -+-----------------------------------------------------------------+ -| Nix | -| [ commmand line interface ]------, | -| | | | -| evaluates | | -| | manages | -| V | | -| [ configuration language ] | | -| | | | -| +-----------------------------|-------------------V-----------+ | -| | store evaluates to | | -| | | | | -| | referenced by V builds | | -| | [ build input ] ---> [ build plan ] ---> [ build result ] | | -| | | | -| +-------------------------------------------------------------+ | -+-----------------------------------------------------------------+ -``` - -At the top is the [command line interface](../command-ref/command-ref.md), translating from invocations of Nix executables to interactions with the underlying layers. - -Below that is the [Nix expression language](../expressions/expression-language.md), a [purely functional][purely-functional-programming] configuration language. -It is used to compose expressions which ultimately evaluate to self-contained *build plans*, used to derive *build results* from referenced *build inputs*. - -The command line and Nix language are what users interact with most. - -> **Note** -> The Nix language itself does not have a notion of *packages* or *configurations*. -> As far as we are concerned here, the inputs and results of a build plan are just data. - -Underlying these is the [Nix store](./store/store.md), a mechanism to keep track of build plans, data, and references between them. -It can also execute build plans to produce new data. - -A build plan is a series of *build tasks*. -Each build task has a special build input which is used as *build instructions*. -The result of a build task can be input to another build task. - -``` -+-----------------------------------------------------------------------------------------+ -| store | -| ................................................. | -| : build plan : | -| : : | -| [ build input ]-----instructions-, : | -| : | : | -| : v : | -| [ build input ]----------->[ build task ]--instructions-, : | -| : | : | -| : | : | -| : v : | -| : [ build task ]----->[ build result ] | -| [ build input ]-----instructions-, ^ : | -| : | | : | -| : v | : | -| [ build input ]----------->[ build task ]---------------' : | -| : ^ : | -| : | : | -| [ build input ]------------------' : | -| : : | -| : : | -| :...............................................: | -| | -+-----------------------------------------------------------------------------------------+ -``` - -[layer-architecture]: https://en.m.wikipedia.org/wiki/Multitier_architecture#Layers -[purely-functional-programming]: https://en.m.wikipedia.org/wiki/Purely_functional_programming diff --git a/doc/manual/src/architecture/store/fso.md b/doc/manual/src/architecture/store/fso.md deleted file mode 100644 index e0eb69f60..000000000 --- a/doc/manual/src/architecture/store/fso.md +++ /dev/null @@ -1,69 +0,0 @@ -# File System Object - -The Nix store uses a simple file system model for the data it holds in [store objects](store.md#store-object). - -Every file system object is one of the following: - - - File: an executable flag, and arbitrary data for contents - - Directory: mapping of names to child file system objects - - [Symbolic link][symlink]: may point anywhere. - -We call a store object's outermost file system object the *root*. - - data FileSystemObject - = File { isExecutable :: Bool, contents :: Bytes } - | Directory { entries :: Map FileName FileSystemObject } - | SymLink { target :: Path } - -Examples: - -- a directory with contents - - /nix/store/-hello-2.10 - ├── bin - │   └── hello - └── share - ├── info - │   └── hello.info - └── man - └── man1 - └── hello.1.gz - -- a directory with relative symlink and other contents - - /nix/store/-go-1.16.9 - ├── bin -> share/go/bin - ├── nix-support/ - └── share/ - -- a directory with absolute symlink - - /nix/store/d3k...-nodejs - └── nix_node -> /nix/store/f20...-nodejs-10.24. - -A bare file or symlink can be a root file system object. -Examples: - - /nix/store/-hello-2.10.tar.gz - - /nix/store/4j5...-pkg-config-wrapper-0.29.2-doc -> /nix/store/i99...-pkg-config-0.29.2-doc - -Symlinks pointing outside of their own root or to a store object without a matching reference are allowed, but might not function as intended. -Examples: - -- an arbitrarily symlinked file may change or not exist at all - - /nix/store/-foo - └── foo -> /home/foo - -- if a symlink to a store path was not automatically created by Nix, it may be invalid or get invalidated when the store object is deleted - - /nix/store/-bar - └── bar -> /nix/store/abc...-foo - -Nix file system objects do not support [hard links][hardlink]: -each file system object which is not the root has exactly one parent and one name. -However, as store objects are immutable, an underlying file system can use hard links for optimization. - -[symlink]: https://en.m.wikipedia.org/wiki/Symbolic_link -[hardlink]: https://en.m.wikipedia.org/wiki/Hard_link diff --git a/doc/manual/src/architecture/store/path.md b/doc/manual/src/architecture/store/path.md deleted file mode 100644 index 663f04f46..000000000 --- a/doc/manual/src/architecture/store/path.md +++ /dev/null @@ -1,105 +0,0 @@ -# Store Path - -Nix implements [references](store.md#reference) to [store objects](store.md#store-object) as *store paths*. - -Store paths are pairs of - -- a 20-byte [digest](#digest) for identification -- a symbolic name for people to read. - -Example: - -- digest: `b6gvzjyb2pg0kjfwrjmg1vfhh54ad73z` -- name: `firefox-33.1` - -It is rendered to a file system path as the concatenation of - - - [store directory](#store-directory) - - path-separator (`/`) - - [digest](#digest) rendered in a custom variant of [base-32](https://en.m.wikipedia.org/wiki/Base32) (20 arbitrary bytes become 32 ASCII characters) - - hyphen (`-`) - - name - -Example: - - /nix/store/b6gvzjyb2pg0kjfwrjmg1vfhh54ad73z-firefox-33.1 - |--------| |------------------------------| |----------| - store directory digest name - -## Store Directory - -Every [store](./store.md) has a store directory. - -If the store has a [file system representation](./store.md#files-and-processes), this directory contains the store’s [file system objects](#file-system-object), which can be addressed by [store paths](#store-path). - -This means a store path is not just derived from the referenced store object itself, but depends on the store the store object is in. - -> **Note** -> The store directory defaults to `/nix/store`, but is in principle arbitrary. - -It is important which store a given store object belongs to: -Files in the store object can contain store paths, and processes may read these paths. -Nix can only guarantee [referential integrity](store/closure.md) if store paths do not cross store boundaries. - -Therefore one can only copy store objects to a different store if - -- the source and target stores' directories match - - or - -- the store object in question has no references, that is, contains no store paths. - -One cannot copy a store object to a store with a different store directory. -Instead, it has to be rebuilt, together with all its dependencies. -It is in general not enough to replace the store directory string in file contents, as this may render executables unusable by invalidating their internal offsets or checksums. - -# Digest - -In a [store path](#store-path), the [digest][digest] is the output of a [cryptographic hash function][hash] of either all *inputs* involved in building the referenced store object or its actual *contents*. - -Store objects are therefore said to be either [input-addressed](#input-addressing) or [content-addressed](#content-addressing). - -> **Historical Note** -> The 20 byte restriction is because originally digests were [SHA-1][sha-1] hashes. -> Nix now uses [SHA-256][sha-256], and longer hashes are still reduced to 20 bytes for compatibility. - -[digest]: https://en.m.wiktionary.org/wiki/digest#Noun -[hash]: https://en.m.wikipedia.org/wiki/Cryptographic_hash_function -[sha-1]: https://en.m.wikipedia.org/wiki/SHA-1 -[sha-256]: https://en.m.wikipedia.org/wiki/SHA-256 - -### Reference scanning - -When a new store object is built, Nix scans its file contents for store paths to construct its set of references. - -The special format of a store path's [digest](#digest) allows reliably detecting it among arbitrary data. -Nix uses the [closure](store.md#closure) of build inputs to derive the list of allowed store paths, to avoid false positives. - -This way, scanning files captures run time dependencies without the user having to declare them explicitly. -Doing it at build time and persisting references in the store object avoids repeating this time-consuming operation. - -> **Note** -> In practice, it is sometimes still necessary for users to declare certain dependencies explicitly, if they are to be preserved in the build result's closure. -This depends on the specifics of the software to build and run. -> -> For example, Java programs are compressed after compilation, which obfuscates any store paths they may refer to and prevents Nix from automatically detecting them. - -## Input Addressing - -Input addressing means that the digest derives from how the store object was produced, namely its build inputs and build plan. - -To compute the hash of a store object one needs a deterministic serialisation, i.e., a binary string representation which only changes if the store object changes. - -Nix has a custom serialisation format called Nix Archive (NAR) - -Store object references of this sort can *not* be validated from the content of the store object. -Rather, a cryptographic signature has to be used to indicate that someone is vouching for the store object really being produced from a build plan with that digest. - -## Content Addressing - -Content addressing means that the digest derives from the store object's contents, namely its file system objects and references. -If one knows content addressing was used, one can recalculate the reference and thus verify the store object. - -Content addressing is currently only used for the special cases of source files and "fixed-output derivations", where the contents of a store object are known in advance. -Content addressing of build results is still an [experimental feature subject to some restrictions](https://github.com/tweag/rfcs/blob/cas-rfc/rfcs/0062-content-addressed-paths.md). - diff --git a/doc/manual/src/architecture/store/store.md b/doc/manual/src/architecture/store/store.md deleted file mode 100644 index 08b6701d5..000000000 --- a/doc/manual/src/architecture/store/store.md +++ /dev/null @@ -1,151 +0,0 @@ -# Store - -A Nix store is a collection of *store objects* with references between them. -It supports operations to manipulate that collection. - -The following concept map is a graphical outline of this chapter. -Arrows indicate suggested reading order. - -``` - ,--------------[ store ]----------------, - | | | - v v v - [ store object ] [ closure ]--, [ operations ] - | | | | | | - v | | v v | - [ files and processes ] | | [ garbage collection ] | - / \ | | | - v v | v v -[ file system object ] [ store path ] | [ derivation ]--->[ building ] - | ^ | | | - v | v v | - [ digest ]----' [ reference scanning ]<------------' - / \ - v v -[ input addressing ] [ content addressing ] -``` - -## Store Object - -A store object can hold - -- arbitrary *data* -- *references* to other store objects. - -Store objects can be build inputs, build results, or build tasks. - -Store objects are [immutable][immutable-object]: once created, they do not change until they are deleted. - -## Reference - -A store object reference is an [opaque][opaque-data-type], [unique identifier][unique-identifier]: -The only way to obtain references is by adding or building store objects. -A reference will always point to exactly one store object. - -## Operations - -A Nix store can *add*, *retrieve*, and *delete* store objects. - - [ data ] - | - V - [ store ] ---> add ----> [ store' ] - | - V - [ reference ] - - - - [ reference ] - | - V - [ store ] ---> get - | - V - [ store object ] - - - - [ reference ] - | - V - [ store ] --> delete --> [ store' ] - - -It can *perform builds*, that is, create new store objects by transforming build inputs into build outputs, using instructions from the build tasks. - - - [ reference ] - | - V - [ store ] --> build --(maybe)--> [ store' ] - | - V - [ reference ] - - -As it keeps track of references, it can [garbage-collect][garbage-collection] unused store objects. - - - [ store ] --> collect garbage --> [ store' ] - -## Files and Processes - -Nix maps between its store model and the [Unix paradigm][unix-paradigm] of [files and processes][file-descriptor], by encoding immutable store objects and opaque identifiers as file system primitives: files and directories, and paths. -That allows processes to resolve references contained in files and thus access the contents of store objects. - -Store objects are therefore implemented as the pair of - - - a [file system object](fso.md) for data - - a set of [store paths](path.md) for references. - -[unix-paradigm]: https://en.m.wikipedia.org/wiki/Everything_is_a_file -[file-descriptor]: https://en.m.wikipedia.org/wiki/File_descriptor - -The following diagram shows a radical simplification of how Nix interacts with the operating system: -It uses files as build inputs, and build outputs are files again. -On the operating system, files can be run as processes, which in turn operate on files. -A build function also amounts to an operating system process (not depicted). - -``` -+-----------------------------------------------------------------+ -| Nix | -| [ commmand line interface ]------, | -| | | | -| evaluates | | -| | manages | -| V | | -| [ configuration language ] | | -| | | | -| +-----------------------------|-------------------V-----------+ | -| | store evaluates to | | -| | | | | -| | referenced by V builds | | -| | [ build input ] ---> [ build plan ] ---> [ build result ] | | -| | ^ | | | -| +---------|----------------------------------------|----------+ | -+-----------|----------------------------------------|------------+ - | | - file system object store path - | | -+-----------|----------------------------------------|------------+ -| operating system +------------+ | | -| '------------ | | <-----------' | -| | file | | -| ,-- | | <-, | -| | +------------+ | | -| execute as | | read, write, execute | -| | +------------+ | | -| '-> | process | --' | -| +------------+ | -+-----------------------------------------------------------------+ -``` - -There exist different types of stores, which all follow this model. -Examples: -- store on the local file system -- remote store accessible via SSH -- binary cache store accessible via HTTP - -To make store objects accessible to processes, stores ultimately have to expose store objects through the file system. - diff --git a/doc/manual/src/architecture/store/store/build-system-terminology.md b/doc/manual/src/architecture/store/store/build-system-terminology.md deleted file mode 100644 index eefbaa630..000000000 --- a/doc/manual/src/architecture/store/store/build-system-terminology.md +++ /dev/null @@ -1,32 +0,0 @@ -# A [Rosetta stone][rosetta-stone] for build system terminology - -The Nix store's design is comparable to other build systems. -Usage of terms is, for historic reasons, not entirely consistent within the Nix ecosystem, and still subject to slow change. - -The following translation table points out similarities and equivalent terms, to help clarify their meaning and inform consistent use in the future. - -| generic build system | Nix | [Bazel][bazel] | [Build Systems à la Carte][bsalc] | programming language | -| -------------------------------- | ---------------- | -------------------------------------------------------------------- | --------------------------------- | ------------------------ | -| data (build input, build result) | store object | [artifact][bazel-artifact] | value | value | -| build instructions | builder | ([depends on action type][bazel-actions]) | function | function | -| build task | derivation | [action][bazel-action] | `Task` | [thunk][thunk] | -| build plan | derivation graph | [action graph][bazel-action-graph], [build graph][bazel-build-graph] | `Tasks` | [call graph][call-graph] | -| build | build | build | application of `Build` | evaluation | -| persistence layer | store | [action cache][bazel-action-cache] | `Store` | heap | - -All of these systems share features of [declarative programming][declarative-programming] languages, a key insight first put forward by Eelco Dolstra et al. in [Imposing a Memory Management Discipline on Software Deployment][immdsd] (2004), elaborated in his PhD thesis [The Purely Functional Software Deployment Model][phd-thesis] (2006), and further refined by Andrey Mokhov et al. in [Build Systems à la Carte][bsalc] (2018). - -[rosetta-stone]: https://en.m.wikipedia.org/wiki/Rosetta_Stone -[bazel]: https://bazel.build/start/bazel-intro -[bazel-artifact]: https://bazel.build/reference/glossary#artifact -[bazel-actions]: https://docs.bazel.build/versions/main/skylark/lib/actions.html -[bazel-action]: https://bazel.build/reference/glossary#action -[bazel-action-graph]: https://bazel.build/reference/glossary#action-graph -[bazel-build-graph]: https://bazel.build/reference/glossary#build-graph -[bazel-action-cache]: https://bazel.build/reference/glossary#action-cache -[thunk]: https://en.m.wikipedia.org/wiki/Thunk -[call-graph]: https://en.m.wikipedia.org/wiki/Call_graph -[declarative-programming]: https://en.m.wikipedia.org/wiki/Declarative_programming -[immdsd]: https://edolstra.github.io/pubs/immdsd-icse2004-final.pdf -[phd-thesis]: https://edolstra.github.io/pubs/phd-thesis.pdf -[bsalc]: https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf diff --git a/doc/manual/src/architecture/store/store/closure.md b/doc/manual/src/architecture/store/store/closure.md deleted file mode 100644 index 065b95ffc..000000000 --- a/doc/manual/src/architecture/store/store/closure.md +++ /dev/null @@ -1,29 +0,0 @@ -# Closure - -Nix stores ensure [referential integrity][referential-integrity]: for each store object in the store, all the store objects it references must also be in the store. - -The set of all store objects reachable by following references from a given initial set of store objects is called a *closure*. - -Adding, building, copying and deleting store objects must be done in a way that preserves referential integrity: - -- A newly added store object cannot have references, unless it is a build task. - -- Build results must only refer to store objects in the closure of the build inputs. - - Building a store object will add appropriate references, according to the build task. - -- Store objects being copied must refer to objects already in the destination store. - - Recursive copying must either proceed in dependency order or be atomic. - -- We can only safely delete store objects which are not reachable from any reference still in use. - - - -[referential-integrity]: https://en.m.wikipedia.org/wiki/Referential_integrity -[garbage-collection]: https://en.m.wikipedia.org/wiki/Garbage_collection_(computer_science) -[immutable-object]: https://en.m.wikipedia.org/wiki/Immutable_object -[opaque-data-type]: https://en.m.wikipedia.org/wiki/Opaque_data_type -[unique-identifier]: https://en.m.wikipedia.org/wiki/Unique_identifier - - From 520587b9a0327194371146f4dd25c9227a2c79e2 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 13:38:07 -0700 Subject: [PATCH 020/112] glossary: local store: clarify --- doc/manual/src/glossary.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index 77de58965..d653a2ae4 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -15,10 +15,10 @@ invoked, the Nix store can be referred to as a "_local_" or a "_remote_" one: - + A *local store* exists on the local filesystem of + + A *local store* exists on the filesystem of the machine where Nix is invoked. You can use other local stores by passing the `--store` flag to the - `nix` command. + `nix` command. Local stores can be used for building derivations. + A *remote store* exists anywhere other than the local filesystem. One example is the `/nix/store` From 2812682ebee9d4419ba89690177b31564ce5ba77 Mon Sep 17 00:00:00 2001 From: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com> Date: Thu, 1 Sep 2022 20:40:39 +0000 Subject: [PATCH 021/112] Update doc/manual/src/glossary.md Co-authored-by: John Ericson --- doc/manual/src/glossary.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index d653a2ae4..f072f35e1 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -52,7 +52,7 @@ from some server. - [substituter]{#gloss-substituter}\ - A *substituter* is a store other than `/nix/store` from which Nix will + A *substituter* is an additional store from which Nix will copy a store path instead of building it. Nix will not copy a store path from a remote store unless one of the following is true: From 9cb84121435e8ca6a51950b9d96a3d3be47c809e Mon Sep 17 00:00:00 2001 From: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com> Date: Thu, 1 Sep 2022 20:41:04 +0000 Subject: [PATCH 022/112] Update doc/manual/src/glossary.md Co-authored-by: John Ericson --- doc/manual/src/glossary.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index f072f35e1..91865c807 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -53,7 +53,7 @@ - [substituter]{#gloss-substituter}\ A *substituter* is an additional store from which Nix will - copy a store path instead of building it. Nix will not copy a store + copy store objects it doesn't have. Nix will not copy a store path from a remote store unless one of the following is true: - the store object is signed by one of the `trusted-public-keys` From 41153f30bd5ca1bd9fa10d18da7a6b5b78a94087 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 13:54:09 -0700 Subject: [PATCH 023/112] glossary: substituter: merge output-addressed cases --- doc/manual/src/glossary.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index 91865c807..608beb8f8 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -59,8 +59,9 @@ - the store object is signed by one of the `trusted-public-keys` - the substituter is in the `trusted-substituters` list - the `no-require-sigs` option has been set to disable signature checking - - the store object is a derivation - - the store object is the realisation of a fixed-output derivation + - the store object is *output-addressed*; this includes + derivations, the outputs of content-addressed derivations, and + the outputs of fixed-output derivations. - [purity]{#gloss-purity}\ The assumption that equal Nix derivations when run always produce From 1f56b5d77247d89a15a2c16ba2f5d1d672c835e8 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 14:09:06 -0700 Subject: [PATCH 024/112] doc/manual: un-inline definitions from `substitute` --- doc/manual/src/glossary.md | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index 608beb8f8..8dff4646c 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -7,6 +7,14 @@ translated into low-level *store derivations* (implicitly by `nix-env` and `nix-build`, or explicitly by `nix-instantiate`). + - [content-addressed derivation]{#gloss-fixed-output-derivation} + FIXME + + - [fixed-output derivation]{#gloss-fixed-output-derivation} + A derivation which includes the `__outHash` attribute; the output + of such derivations must exactly match the hash. All fixed-output + derivations are [content-addressed derivations](#gloss-content-addressed-derivation). + - [store]{#gloss-store}\ The location in the file system where store objects live. Typically `/nix/store`. @@ -44,6 +52,16 @@ derivation outputs (objects produced by running a build action), or derivations (files describing a build action). + - [input-addressed store object]{#gloss-input-addressed-store-object}\ + Store objects produced by building a + non-[content-addressed](#gloss-content-addressed-derivation) + derivation. + + - [output-addressed store object]{#gloss-output-addressed-store-object}\ + A store object whose store path hashes its content. This + includes derivations and the outputs of + [content-addressed derivations](#gloss-content-addressed-derivation) + - [substitute]{#gloss-substitute}\ A substitute is a command invocation stored in the Nix database that describes how to build a store object, bypassing the normal build @@ -59,9 +77,7 @@ - the store object is signed by one of the `trusted-public-keys` - the substituter is in the `trusted-substituters` list - the `no-require-sigs` option has been set to disable signature checking - - the store object is *output-addressed*; this includes - derivations, the outputs of content-addressed derivations, and - the outputs of fixed-output derivations. + - the store object is [output-addressed](#gloss-output-addressed-store-object) - [purity]{#gloss-purity}\ The assumption that equal Nix derivations when run always produce From 0a98d564b3d8d195c023429e5f7faf63e20b5d93 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 14:41:27 -0700 Subject: [PATCH 025/112] glossary: resolve FIXME in #gloss-fixed-output-derivation --- doc/manual/src/glossary.md | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index 8dff4646c..473aef03c 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -7,13 +7,14 @@ translated into low-level *store derivations* (implicitly by `nix-env` and `nix-build`, or explicitly by `nix-instantiate`). - - [content-addressed derivation]{#gloss-fixed-output-derivation} - FIXME + - [content-addressed derivation]{#gloss-content-addressed-derivation}\ + A derivation which has the + [`__contentAddressed`](language/advanced-attributes.md#contentAddressed) + attribute set to `true`. - - [fixed-output derivation]{#gloss-fixed-output-derivation} - A derivation which includes the `__outHash` attribute; the output - of such derivations must exactly match the hash. All fixed-output - derivations are [content-addressed derivations](#gloss-content-addressed-derivation). + - [fixed-output derivation]{#gloss-fixed-output-derivation}\ + A derivation which includes the `__outputHash` attribute; the output + of such derivations must exactly match the hash. - [store]{#gloss-store}\ The location in the file system where store objects live. Typically @@ -59,8 +60,10 @@ - [output-addressed store object]{#gloss-output-addressed-store-object}\ A store object whose store path hashes its content. This - includes derivations and the outputs of - [content-addressed derivations](#gloss-content-addressed-derivation) + includes derivations, the outputs of + [content-addressed derivations](#gloss-content-addressed-derivation), + and the outputs of + [fixed-output derivations](#gloss-fixed-output-derivation). - [substitute]{#gloss-substitute}\ A substitute is a command invocation stored in the Nix database that From 1b2b8c39fd64871b3df261c05ed001fcd1057a6c Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 14:47:17 -0700 Subject: [PATCH 026/112] fix link to language/advanced-attributes.md#adv-attr-contentAddressed --- doc/manual/src/glossary.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index 473aef03c..bf1d0cf05 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -9,7 +9,7 @@ - [content-addressed derivation]{#gloss-content-addressed-derivation}\ A derivation which has the - [`__contentAddressed`](language/advanced-attributes.md#contentAddressed) + [`__contentAddressed`](language/advanced-attributes.md#adv-attr-contentAddressed) attribute set to `true`. - [fixed-output derivation]{#gloss-fixed-output-derivation}\ From def4fb9a0f73046efbf9fdb4f1e35898fb27ca34 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 14:47:33 -0700 Subject: [PATCH 027/112] __outputHash: add link --- doc/manual/src/glossary.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index bf1d0cf05..d61cfc823 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -13,8 +13,8 @@ attribute set to `true`. - [fixed-output derivation]{#gloss-fixed-output-derivation}\ - A derivation which includes the `__outputHash` attribute; the output - of such derivations must exactly match the hash. + A derivation which includes the + [`__outputHash`](language/advanced-attributes.md#adv-attr-outputHash) attribute. - [store]{#gloss-store}\ The location in the file system where store objects live. Typically From 8139bbe2ba767458cba4158627ee3d58f4a35d7d Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 15:01:07 -0700 Subject: [PATCH 028/112] implement https://github.com/NixOS/nix/pull/6870#pullrequestreview-1093700220 --- doc/manual/src/glossary.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index d61cfc823..b30633833 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -74,8 +74,8 @@ - [substituter]{#gloss-substituter}\ A *substituter* is an additional store from which Nix will - copy store objects it doesn't have. Nix will not copy a store - path from a remote store unless one of the following is true: + copy store objects it doesn't have. Nix will copy a store + path from a remote store only if one of the following is true: - the store object is signed by one of the `trusted-public-keys` - the substituter is in the `trusted-substituters` list From 57f12df5e4cde436566d3c4f0226d329e6eedf1a Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 15:09:10 -0700 Subject: [PATCH 029/112] input-addressed store object: include FODOs --- doc/manual/src/glossary.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index b30633833..89a9b94a1 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -54,8 +54,9 @@ derivations (files describing a build action). - [input-addressed store object]{#gloss-input-addressed-store-object}\ - Store objects produced by building a - non-[content-addressed](#gloss-content-addressed-derivation) + A store object produced by building a + non-[content-addressed](#gloss-content-addressed-derivation), + non-[fixed-output](#gloss-fixed-output-derivation), derivation. - [output-addressed store object]{#gloss-output-addressed-store-object}\ From d5e064d8162e377556dc9daba99868085561a080 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 17:46:31 -0700 Subject: [PATCH 030/112] glossary: fix broken link --- doc/manual/src/glossary.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index 89a9b94a1..1aebdaa67 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -9,7 +9,7 @@ - [content-addressed derivation]{#gloss-content-addressed-derivation}\ A derivation which has the - [`__contentAddressed`](language/advanced-attributes.md#adv-attr-contentAddressed) + [`__contentAddressed`](language/advanced-attributes.md#adv-attr-__contentAddressed) attribute set to `true`. - [fixed-output derivation]{#gloss-fixed-output-derivation}\ From 887e922be29d37d377ef766c7fe7a2103f43ca21 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 17:47:13 -0700 Subject: [PATCH 031/112] glossary: outputHash, not __outputHash --- doc/manual/src/glossary.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index 1aebdaa67..73e209103 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -14,7 +14,7 @@ - [fixed-output derivation]{#gloss-fixed-output-derivation}\ A derivation which includes the - [`__outputHash`](language/advanced-attributes.md#adv-attr-outputHash) attribute. + [`outputHash`](language/advanced-attributes.md#adv-attr-outputHash) attribute. - [store]{#gloss-store}\ The location in the file system where store objects live. Typically From f6c750e8b2f299e9876fd8f2578f9093682f6d7f Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 17:48:34 -0700 Subject: [PATCH 032/112] glossary: remove extraneous comma --- doc/manual/src/glossary.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index 73e209103..7ba595ba0 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -56,7 +56,7 @@ - [input-addressed store object]{#gloss-input-addressed-store-object}\ A store object produced by building a non-[content-addressed](#gloss-content-addressed-derivation), - non-[fixed-output](#gloss-fixed-output-derivation), + non-[fixed-output](#gloss-fixed-output-derivation) derivation. - [output-addressed store object]{#gloss-output-addressed-store-object}\ From 59dc8346ca53f49ccdbbd6709b12a479376d1464 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 17:51:56 -0700 Subject: [PATCH 033/112] move substituter signature-checking conditions to configuration file documentation --- doc/manual/src/glossary.md | 9 ++------- src/libstore/globals.hh | 8 ++++++++ 2 files changed, 10 insertions(+), 7 deletions(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index 7ba595ba0..6bf041e7c 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -75,13 +75,8 @@ - [substituter]{#gloss-substituter}\ A *substituter* is an additional store from which Nix will - copy store objects it doesn't have. Nix will copy a store - path from a remote store only if one of the following is true: - - - the store object is signed by one of the `trusted-public-keys` - - the substituter is in the `trusted-substituters` list - - the `no-require-sigs` option has been set to disable signature checking - - the store object is [output-addressed](#gloss-output-addressed-store-object) + copy store objects it doesn't have. For details, see the + [`substituters` option](command-ref/conf-file.html#conf-substituters). - [purity]{#gloss-purity}\ The assumption that equal Nix derivations when run always produce diff --git a/src/libstore/globals.hh b/src/libstore/globals.hh index d7f351166..a659036e2 100644 --- a/src/libstore/globals.hh +++ b/src/libstore/globals.hh @@ -613,6 +613,14 @@ public: are tried based on their Priority value, which each substituter can set independently. Lower value means higher priority. The default is `https://cache.nixos.org`, with a Priority of 40. + + Nix will copy a store path from a remote store only if one + of the following is true: + + - the store object is signed by one of the `trusted-public-keys` + - the substituter is in the `trusted-substituters` list + - the `no-require-sigs` option has been set to disable signature checking + - the store object is [output-addressed](#gloss-output-addressed-store-object) )", {"binary-caches"}}; From 1ab913467ef8e9ff946e64bd31841775d743b2d6 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 17:54:23 -0700 Subject: [PATCH 034/112] linkify mention of other options --- src/libstore/globals.hh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/libstore/globals.hh b/src/libstore/globals.hh index a659036e2..a4db3bf08 100644 --- a/src/libstore/globals.hh +++ b/src/libstore/globals.hh @@ -617,10 +617,10 @@ public: Nix will copy a store path from a remote store only if one of the following is true: - - the store object is signed by one of the `trusted-public-keys` - - the substituter is in the `trusted-substituters` list - - the `no-require-sigs` option has been set to disable signature checking - - the store object is [output-addressed](#gloss-output-addressed-store-object) + - the store object is signed by one of the [`trusted-public-keys`](#conf-trusted-public-keys) + - the substituter is in the [`trusted-substituters`](#conf-trusted-substituters) list + - the [`require-sigs`](#conf-require-sigs) option has been set to `false` + - the store object is [output-addressed](glossary.md#gloss-output-addressed-store-object) )", {"binary-caches"}}; From e6f5352e71a1811eb2eb3bfb989e109de590c7a7 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 18:27:00 -0700 Subject: [PATCH 035/112] #binary-cache -> #gloss-binary-cache --- doc/manual/src/glossary.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index 6bf041e7c..a34b8a60c 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -34,7 +34,7 @@ directory on another machine, accessed via `ssh` or served by the `nix-serve` Perl script. - - [binary cache]{#binary-cache}\ + - [binary cache]{#gloss-binary-cache}\ A *binary cache* is a Nix store which uses a different format: its metadata and signatures are kept in `.narinfo` files rather than in a Nix database. This different format simplifies serving store objects From e90f2fcfc71ca997f254c86f8ed12fc143374752 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Thu, 1 Sep 2022 18:28:05 -0700 Subject: [PATCH 036/112] glossary: add entry for `chroot store` (used 11 times in nix) --- doc/manual/src/glossary.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index a34b8a60c..70a0eb994 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -34,6 +34,9 @@ directory on another machine, accessed via `ssh` or served by the `nix-serve` Perl script. + - [chroot store]{#gloss-chroot-store}\ + A local store whose canonical path is anything other than `/nix/store`. + - [binary cache]{#gloss-binary-cache}\ A *binary cache* is a Nix store which uses a different format: its metadata and signatures are kept in `.narinfo` files rather than in a From 1f041ac54f43093e4f4df1caa630d491ff51c3f8 Mon Sep 17 00:00:00 2001 From: Andrew Brooks Date: Fri, 2 Sep 2022 18:32:35 -0500 Subject: [PATCH 037/112] Prevent tempdir from being GC-ed before addToStoreFromDump has renamed it This fixes issue 6823 by placing the tempdir used in LocalStore::addToStoreFromDump outside the Nix store, where automatic GC is no longer a concern. --- src/libstore/local-store.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/libstore/local-store.cc b/src/libstore/local-store.cc index a272e4301..6abd52683 100644 --- a/src/libstore/local-store.cc +++ b/src/libstore/local-store.cc @@ -1388,7 +1388,7 @@ StorePath LocalStore::addToStoreFromDump(Source & source0, std::string_view name StringSource dumpSource { dump }; ChainSource bothSource { dumpSource, source }; - auto tempDir = createTempDir(realStoreDir, "add"); + auto tempDir = createTempDir("", "add"); delTempDir = std::make_unique(tempDir); tempPath = tempDir + "/x"; From 6ce2e96c88c71e40303980eb5793aa6ae4a5a333 Mon Sep 17 00:00:00 2001 From: Jonathan Ringer Date: Tue, 6 Sep 2022 08:18:13 -0700 Subject: [PATCH 038/112] Docs: Add nix develop --command entry Add example of nix develop being used to execuate a series of script commands. This is common when doing things like CI/CD, and should be represented in the official documentation. Also useful for people looking for the 'nix develop' equivalent of 'nix-shell --run'. Related: - https://github.com/NixOS/nix/issues/6908 - https://github.com/NixOS/nix/issues/6908#issuecomment-1229266853 --- src/nix/develop.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/nix/develop.md b/src/nix/develop.md index e036ec6b9..4e8542d1b 100644 --- a/src/nix/develop.md +++ b/src/nix/develop.md @@ -66,6 +66,12 @@ R""( `nixpkgs#glibc` in `~/my-glibc` and want to compile another package against it. +* Run a series of script commands: + + ```console + # nix develop --command bash -c "mkdir build && cmake .. && make" + ``` + # Description `nix develop` starts a `bash` shell that provides an interactive build From 27be54ca533933db8c3e0cde4b213abf10dd5237 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Tue, 6 Sep 2022 18:27:39 +0200 Subject: [PATCH 039/112] nix develop: Ignore stdenv's $SHELL Stdenv sets this to a bash that doesn't have readline/completion support, so running 'nix (develop|shell)' inside a 'nix develop' gives you a crippled shell. So let's just ignore the derivation's $SHELL. This could break interactive use of build phases that use $SHELL, but they appear to be fairly rare. --- src/nix/develop.cc | 1 + 1 file changed, 1 insertion(+) diff --git a/src/nix/develop.cc b/src/nix/develop.cc index ba7ba7c25..4de109754 100644 --- a/src/nix/develop.cc +++ b/src/nix/develop.cc @@ -246,6 +246,7 @@ struct Common : InstallableCommand, MixProfile "NIX_LOG_FD", "NIX_REMOTE", "PPID", + "SHELL", "SHELLOPTS", "SSL_CERT_FILE", // FIXME: only want to ignore /no-cert-file.crt "TEMP", From ece12a97d9c7e0024ebddb9e5eb0c919a9efb694 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Tue, 6 Sep 2022 19:20:31 +0200 Subject: [PATCH 040/112] lockfile -> lock file for consistency --- src/libexpr/flake/flake.cc | 4 ++-- src/libexpr/flake/lockfile.cc | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/src/libexpr/flake/flake.cc b/src/libexpr/flake/flake.cc index 105e76bc6..119c556ac 100644 --- a/src/libexpr/flake/flake.cc +++ b/src/libexpr/flake/flake.cc @@ -483,12 +483,12 @@ LockedFlake lockFlake( } else if (auto follows = std::get_if<1>(&i.second)) { if (! trustLock) { // It is possible that the flake has changed, - // so we must confirm all the follows that are in the lockfile are also in the flake. + // so we must confirm all the follows that are in the lock file are also in the flake. auto overridePath(inputPath); overridePath.push_back(i.first); auto o = overrides.find(overridePath); // If the override disappeared, we have to refetch the flake, - // since some of the inputs may not be present in the lockfile. + // since some of the inputs may not be present in the lock file. if (o == overrides.end()) { mustRefetch = true; // There's no point populating the rest of the fake inputs, diff --git a/src/libexpr/flake/lockfile.cc b/src/libexpr/flake/lockfile.cc index 60b52d578..629d2e669 100644 --- a/src/libexpr/flake/lockfile.cc +++ b/src/libexpr/flake/lockfile.cc @@ -36,7 +36,7 @@ LockedNode::LockedNode(const nlohmann::json & json) , isFlake(json.find("flake") != json.end() ? (bool) json["flake"] : true) { if (!lockedRef.input.isLocked()) - throw Error("lockfile contains mutable lock '%s'", + throw Error("lock file contains mutable lock '%s'", fetchers::attrsToJSON(lockedRef.input.toAttrs())); } From 84fe75a12a085c6b4b8d4ac65a048f569de1252b Mon Sep 17 00:00:00 2001 From: Andrew Brooks Date: Tue, 6 Sep 2022 17:48:00 -0500 Subject: [PATCH 041/112] Keep created temp dirs inside store, but protect from GC Implements the approach suggested by feedback on PR #6994, where tempdir paths are created in the store (now with an exclusive lock). As part of this work, the currently-broken and unused `createTempDirInStore` function is updated to create an exclusive lock on the temp directory in the store. The GC now makes a non-blocking attempt to lock any store directories that "look like" the temp directories created by this function, and if it can't acquire one, ignores the directory. --- src/libstore/gc.cc | 12 ++++++++++++ src/libstore/local-store.cc | 29 +++++++++++++++++++---------- src/libstore/local-store.hh | 2 +- 3 files changed, 32 insertions(+), 11 deletions(-) diff --git a/src/libstore/gc.cc b/src/libstore/gc.cc index 4c1a82279..6cd7efbc9 100644 --- a/src/libstore/gc.cc +++ b/src/libstore/gc.cc @@ -619,6 +619,18 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results) Path path = storeDir + "/" + std::string(baseName); Path realPath = realStoreDir + "/" + std::string(baseName); + /* There may be temp directories in the store that are still in use + by another process. We need to be sure that we can acquire an + exclusive lock before deleting them. */ + AutoCloseFD tmpDirFd; + if (baseName.rfind("add-", 0) == 0) { + tmpDirFd = open(realPath.c_str(), O_RDONLY | O_DIRECTORY); + if (tmpDirFd.get() == -1 || !lockFile(tmpDirFd.get(), ltWrite, false)) { + debug("skipping locked tempdir '%s'", realPath); + return; + } + } + printInfo("deleting '%1%'", path); results.paths.insert(path); diff --git a/src/libstore/local-store.cc b/src/libstore/local-store.cc index 6abd52683..5ee451da3 100644 --- a/src/libstore/local-store.cc +++ b/src/libstore/local-store.cc @@ -1382,13 +1382,15 @@ StorePath LocalStore::addToStoreFromDump(Source & source0, std::string_view name std::unique_ptr delTempDir; Path tempPath; + Path tempDir; + AutoCloseFD tempDirFd; if (!inMemory) { /* Drain what we pulled so far, and then keep on pulling */ StringSource dumpSource { dump }; ChainSource bothSource { dumpSource, source }; - auto tempDir = createTempDir("", "add"); + std::tie(tempDir, tempDirFd) = createTempDirInStore(); delTempDir = std::make_unique(tempDir); tempPath = tempDir + "/x"; @@ -1431,6 +1433,7 @@ StorePath LocalStore::addToStoreFromDump(Source & source0, std::string_view name } else { /* Move the temporary path we restored above. */ moveFile(tempPath, realPath); + tempDirFd.close(); } /* For computing the nar hash. In recursive SHA-256 mode, this @@ -1507,18 +1510,24 @@ StorePath LocalStore::addTextToStore( /* Create a temporary directory in the store that won't be - garbage-collected. */ -Path LocalStore::createTempDirInStore() + garbage-collected until the returned FD is closed. */ +std::pair LocalStore::createTempDirInStore() { - Path tmpDir; + Path tmpDirFn; + AutoCloseFD tmpDirFd; + bool lockedByUs = false; do { /* There is a slight possibility that `tmpDir' gets deleted by - the GC between createTempDir() and addTempRoot(), so repeat - until `tmpDir' exists. */ - tmpDir = createTempDir(realStoreDir); - addTempRoot(parseStorePath(tmpDir)); - } while (!pathExists(tmpDir)); - return tmpDir; + the GC between createTempDir() and when we acquire a lock on it. + We'll repeat until 'tmpDir' exists and we've locked it. */ + tmpDirFn = createTempDir(realStoreDir, "add"); + tmpDirFd = open(tmpDirFn.c_str(), O_RDONLY | O_DIRECTORY); + if (tmpDirFd.get() < 0) { + continue; + } + lockedByUs = lockFile(tmpDirFd.get(), ltWrite, true); + } while (!pathExists(tmpDirFn) || !lockedByUs); + return {tmpDirFn, std::move(tmpDirFd)}; } diff --git a/src/libstore/local-store.hh b/src/libstore/local-store.hh index 70d225be3..bd0ce1fe6 100644 --- a/src/libstore/local-store.hh +++ b/src/libstore/local-store.hh @@ -256,7 +256,7 @@ private: void findRuntimeRoots(Roots & roots, bool censor); - Path createTempDirInStore(); + std::pair createTempDirInStore(); void checkDerivationOutputs(const StorePath & drvPath, const Derivation & drv); From 7852609999cdcbc056ef47b530dcb253bd2c0697 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Mon, 12 Sep 2022 11:27:25 +0200 Subject: [PATCH 042/112] issue template: add feature label each change should be an improvement, a label for that is redundant. --- .github/ISSUE_TEMPLATE/feature_request.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md index 392ed30c6..4fe86d5ec 100644 --- a/.github/ISSUE_TEMPLATE/feature_request.md +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -2,7 +2,7 @@ name: Feature request about: Suggest an idea for this project title: '' -labels: improvement +labels: feature assignees: '' --- From 565d888e0f6a2c66ee7b10f6fe6a97f79fa51732 Mon Sep 17 00:00:00 2001 From: Andrew Brooks Date: Mon, 12 Sep 2022 11:33:23 -0500 Subject: [PATCH 043/112] Address PR feedback on #6694 --- src/libstore/gc.cc | 5 ++--- src/libstore/local-store.cc | 3 +-- 2 files changed, 3 insertions(+), 5 deletions(-) diff --git a/src/libstore/gc.cc b/src/libstore/gc.cc index 6cd7efbc9..9ef8972f3 100644 --- a/src/libstore/gc.cc +++ b/src/libstore/gc.cc @@ -622,9 +622,8 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results) /* There may be temp directories in the store that are still in use by another process. We need to be sure that we can acquire an exclusive lock before deleting them. */ - AutoCloseFD tmpDirFd; - if (baseName.rfind("add-", 0) == 0) { - tmpDirFd = open(realPath.c_str(), O_RDONLY | O_DIRECTORY); + if (baseName.find("tmp-", 0) == 0) { + AutoCloseFD tmpDirFd = open(realPath.c_str(), O_RDONLY | O_DIRECTORY); if (tmpDirFd.get() == -1 || !lockFile(tmpDirFd.get(), ltWrite, false)) { debug("skipping locked tempdir '%s'", realPath); return; diff --git a/src/libstore/local-store.cc b/src/libstore/local-store.cc index 5ee451da3..0b07cde34 100644 --- a/src/libstore/local-store.cc +++ b/src/libstore/local-store.cc @@ -1433,7 +1433,6 @@ StorePath LocalStore::addToStoreFromDump(Source & source0, std::string_view name } else { /* Move the temporary path we restored above. */ moveFile(tempPath, realPath); - tempDirFd.close(); } /* For computing the nar hash. In recursive SHA-256 mode, this @@ -1520,7 +1519,7 @@ std::pair LocalStore::createTempDirInStore() /* There is a slight possibility that `tmpDir' gets deleted by the GC between createTempDir() and when we acquire a lock on it. We'll repeat until 'tmpDir' exists and we've locked it. */ - tmpDirFn = createTempDir(realStoreDir, "add"); + tmpDirFn = createTempDir(realStoreDir, "tmp"); tmpDirFd = open(tmpDirFn.c_str(), O_RDONLY | O_DIRECTORY); if (tmpDirFd.get() < 0) { continue; From c6ff33ff5c83a546fc6e82055aa04abfe41011dc Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Tue, 13 Sep 2022 15:29:13 +0200 Subject: [PATCH 044/112] RunPager: Stop the progress bar In particular, the progress bar was interfering with 'less' rendering in '--help' (e.g. run 'nix --help' and hit '/' to search). --- src/libmain/shared.cc | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/libmain/shared.cc b/src/libmain/shared.cc index 52b75f757..c1cf38565 100644 --- a/src/libmain/shared.cc +++ b/src/libmain/shared.cc @@ -4,6 +4,7 @@ #include "gc-store.hh" #include "util.hh" #include "loggers.hh" +#include "progress-bar.hh" #include #include @@ -422,6 +423,8 @@ RunPager::RunPager() if (!pager) pager = getenv("PAGER"); if (pager && ((std::string) pager == "" || (std::string) pager == "cat")) return; + stopProgressBar(); + Pipe toPager; toPager.create(); From d365cced4fadbbc63f0c39902a7091e1a34c34de Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Tue, 13 Sep 2022 16:58:32 +0200 Subject: [PATCH 045/112] Trim option descriptions This removes unintended blank lines in Markdown when the description is a multiline string literal. --- src/libutil/args.cc | 6 +++--- src/nix/main.cc | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/src/libutil/args.cc b/src/libutil/args.cc index 44b63f0f6..753980fd4 100644 --- a/src/libutil/args.cc +++ b/src/libutil/args.cc @@ -216,7 +216,7 @@ nlohmann::json Args::toJSON() if (flag->shortName) j["shortName"] = std::string(1, flag->shortName); if (flag->description != "") - j["description"] = flag->description; + j["description"] = trim(flag->description); j["category"] = flag->category; if (flag->handler.arity != ArityAny) j["arity"] = flag->handler.arity; @@ -237,7 +237,7 @@ nlohmann::json Args::toJSON() } auto res = nlohmann::json::object(); - res["description"] = description(); + res["description"] = trim(description()); res["flags"] = std::move(flags); res["args"] = std::move(args); auto s = doc(); @@ -379,7 +379,7 @@ nlohmann::json MultiCommand::toJSON() auto j = command->toJSON(); auto cat = nlohmann::json::object(); cat["id"] = command->category(); - cat["description"] = categories[command->category()]; + cat["description"] = trim(categories[command->category()]); j["category"] = std::move(cat); cmds[name] = std::move(j); } diff --git a/src/nix/main.cc b/src/nix/main.cc index f434e9655..e0155cd5d 100644 --- a/src/nix/main.cc +++ b/src/nix/main.cc @@ -325,7 +325,7 @@ void mainWrapped(int argc, char * * argv) std::cout << "attrs\n"; break; } for (auto & s : *completions) - std::cout << s.completion << "\t" << s.description << "\n"; + std::cout << s.completion << "\t" << trim(s.description) << "\n"; } }); From 8ebdbeb2574ab3a8b6dbd9826451d9f26ca5ad3e Mon Sep 17 00:00:00 2001 From: Ana Hobden Date: Wed, 7 Sep 2022 11:58:25 -0700 Subject: [PATCH 046/112] Add fish suport to installer Before this patch, installing Nix using the Fish shell did not work because Fish wasn't configured to add Nix to the PATH. Some options in #1512 offered workarounds, but they typically involve extra plugins or packages. This patch adds native, out-of-the-box support for the Fish shell. Note that Fish supports a `conf.d` directory, which is intended for exactly use cases like this: software projects distributing shell snippets. This patch takes advantage of it. The installer doesn't append any Nix loader behavior to any Fish config file. Because of that, the uninstall process is smooth and a reinstall obliterates the existing nix.fish files that we place instead of bothering the user with a backup / manual removal. Both single-user and multi-user cases are covered. It has been tested on Ubuntu, and a Mac with MacPorts, homebrew, and the Fish installer pkg. Closes #1512 Co-authored-by: Graham Christensen --- .gitignore | 2 ++ scripts/install-multi-user.sh | 47 +++++++++++++++++++++++++++++ scripts/install-nix-from-closure.sh | 29 +++++++++++++++--- scripts/local.mk | 2 ++ scripts/nix-profile-daemon.fish.in | 35 +++++++++++++++++++++ scripts/nix-profile.fish.in | 35 +++++++++++++++++++++ scripts/nix-profile.sh.in | 1 - 7 files changed, 145 insertions(+), 6 deletions(-) create mode 100644 scripts/nix-profile-daemon.fish.in create mode 100644 scripts/nix-profile.fish.in diff --git a/.gitignore b/.gitignore index 0c1b89ace..8e0db013f 100644 --- a/.gitignore +++ b/.gitignore @@ -27,6 +27,8 @@ perl/Makefile.config # /scripts/ /scripts/nix-profile.sh /scripts/nix-profile-daemon.sh +/scripts/nix-profile.fish +/scripts/nix-profile-daemon.fish # /src/libexpr/ /src/libexpr/lexer-tab.cc diff --git a/scripts/install-multi-user.sh b/scripts/install-multi-user.sh index 9a990275c..a39339050 100644 --- a/scripts/install-multi-user.sh +++ b/scripts/install-multi-user.sh @@ -37,6 +37,19 @@ readonly PROFILE_TARGETS=("/etc/bashrc" "/etc/profile.d/nix.sh" "/etc/zshrc" "/e readonly PROFILE_BACKUP_SUFFIX=".backup-before-nix" readonly PROFILE_NIX_FILE="$NIX_ROOT/var/nix/profiles/default/etc/profile.d/nix-daemon.sh" +# Fish has different syntax than zsh/bash, treat it separate +readonly PROFILE_FISH_SUFFIX="conf.d/nix.fish" +readonly PROFILE_FISH_PREFIXES=( + # each of these are common values of $__fish_sysconf_dir, + # under which Fish will look for a file named + # $PROFILE_FISH_SUFFIX. + "/etc/fish" # standard + "/usr/local/etc/fish" # their installer .pkg for macOS + "/opt/homebrew/etc/fish" # homebrew + "/opt/local/etc/fish" # macports +) +readonly PROFILE_NIX_FILE_FISH="$NIX_ROOT/var/nix/profiles/default/etc/profile.d/nix-daemon.fish" + readonly NIX_INSTALLED_NIX="@nix@" readonly NIX_INSTALLED_CACERT="@cacert@" #readonly NIX_INSTALLED_NIX="/nix/store/j8dbv5w6jl34caywh2ygdy88knx1mdf7-nix-2.3.6" @@ -828,6 +841,19 @@ fi EOF } +# Fish has differing syntax +fish_source_lines() { + cat <&2 - printf '\nif [ -e %s ]; then . %s; fi # added by Nix installer\n' "$p" "$p" >> "$fn" + printf '\nif [ -e %s ]; then . %s; fi # added by Nix installer\n' "$p_sh" "$p_sh" >> "$fn" fi added=1 + p=${p_sh} break fi done for i in .zshenv .zshrc; do fn="$HOME/$i" if [ -w "$fn" ]; then - if ! grep -q "$p" "$fn"; then + if ! grep -q "$p_sh" "$fn"; then echo "modifying $fn..." >&2 - printf '\nif [ -e %s ]; then . %s; fi # added by Nix installer\n' "$p" "$p" >> "$fn" + printf '\nif [ -e %s ]; then . %s; fi # added by Nix installer\n' "$p_sh" "$p_sh" >> "$fn" fi added=1 + p=${p_sh} break fi done + + if [ -d "$HOME/.config/fish" ]; then + fishdir=$HOME/.config/fish/conf.d + if [ ! -d "$fishdir" ]; then + mkdir -p "$fishdir" + fi + + fn="$fishdir/nix.fish" + echo "placing $fn..." >&2 + printf '\nif test -e %s; . %s; end # added by Nix installer\n' "$p_fish" "$p_fish" > "$fn" + added=1 + p=${p_fish} + fi +else + p=${p_sh} fi if [ -z "$added" ]; then diff --git a/scripts/local.mk b/scripts/local.mk index b8477178e..46255e432 100644 --- a/scripts/local.mk +++ b/scripts/local.mk @@ -6,6 +6,8 @@ noinst-scripts += $(nix_noinst_scripts) profiledir = $(sysconfdir)/profile.d $(eval $(call install-file-as, $(d)/nix-profile.sh, $(profiledir)/nix.sh, 0644)) +$(eval $(call install-file-as, $(d)/nix-profile.fish, $(profiledir)/nix.fish, 0644)) $(eval $(call install-file-as, $(d)/nix-profile-daemon.sh, $(profiledir)/nix-daemon.sh, 0644)) +$(eval $(call install-file-as, $(d)/nix-profile-daemon.fish, $(profiledir)/nix-daemon.fish, 0644)) clean-files += $(nix_noinst_scripts) diff --git a/scripts/nix-profile-daemon.fish.in b/scripts/nix-profile-daemon.fish.in new file mode 100644 index 000000000..56d851a9c --- /dev/null +++ b/scripts/nix-profile-daemon.fish.in @@ -0,0 +1,35 @@ +# Only execute this file once per shell. +if test -n "$__ETC_PROFILE_NIX_SOURCED" + return +end + +set __ETC_PROFILE_NIX_SOURCED 1 + +set --export NIX_PROFILES "@localstatedir@/nix/profiles/default $HOME/.nix-profile" + +# Set $NIX_SSL_CERT_FILE so that Nixpkgs applications like curl work. +if test -n "$NIX_SSH_CERT_FILE" + : # Allow users to override the NIX_SSL_CERT_FILE +else if test -e /etc/ssl/certs/ca-certificates.crt # NixOS, Ubuntu, Debian, Gentoo, Arch + set --export NIX_SSL_CERT_FILE /etc/ssl/certs/ca-certificates.crt +else if test -e /etc/ssl/ca-bundle.pem # openSUSE Tumbleweed + set --export NIX_SSL_CERT_FILE /etc/ssl/ca-bundle.pem +else if test -e /etc/ssl/certs/ca-bundle.crt # Old NixOS + set --export NIX_SSL_CERT_FILE /etc/ssl/certs/ca-bundle.crt +else if test -e /etc/pki/tls/certs/ca-bundle.crt # Fedora, CentOS + set --export NIX_SSL_CERT_FILE /etc/pki/tls/certs/ca-bundle.crt +else if test -e "$NIX_LINK/etc/ssl/certs/ca-bundle.crt" # fall back to cacert in Nix profile + set --export NIX_SSL_CERT_FILE "$NIX_LINK/etc/ssl/certs/ca-bundle.crt" +else if test -e "$NIX_LINK/etc/ca-bundle.crt" # old cacert in Nix profile + set --export NIX_SSL_CERT_FILE "$NIX_LINK/etc/ca-bundle.crt" +else + # Fall back to what is in the nix profiles, favouring whatever is defined last. + for i in $NIX_PROFILES + if test -e "$i/etc/ssl/certs/ca-bundle.crt" + set --export NIX_SSL_CERT_FILE "$i/etc/ssl/certs/ca-bundle.crt" + end + end +end + +fish_add_path --prepend --global "@localstatedir@/nix/profiles/default/bin" +fish_add_path --prepend --global "$HOME/.nix-profile/bin" diff --git a/scripts/nix-profile.fish.in b/scripts/nix-profile.fish.in new file mode 100644 index 000000000..59d247771 --- /dev/null +++ b/scripts/nix-profile.fish.in @@ -0,0 +1,35 @@ +if test -n "$HOME" && test -n "$USER" + + # Set up the per-user profile. + + set NIX_LINK $HOME/.nix-profile + + # Set up environment. + # This part should be kept in sync with nixpkgs:nixos/modules/programs/environment.nix + set --export NIX_PROFILES "@localstatedir@/nix/profiles/default $HOME/.nix-profile" + + # Set $NIX_SSL_CERT_FILE so that Nixpkgs applications like curl work. + if test -n "$NIX_SSH_CERT_FILE" + : # Allow users to override the NIX_SSL_CERT_FILE + else if test -e /etc/ssl/certs/ca-certificates.crt # NixOS, Ubuntu, Debian, Gentoo, Arch + set --export NIX_SSL_CERT_FILE /etc/ssl/certs/ca-certificates.crt + else if test -e /etc/ssl/ca-bundle.pem # openSUSE Tumbleweed + set --export NIX_SSL_CERT_FILE /etc/ssl/ca-bundle.pem + else if test -e /etc/ssl/certs/ca-bundle.crt # Old NixOS + set --export NIX_SSL_CERT_FILE /etc/ssl/certs/ca-bundle.crt + else if test -e /etc/pki/tls/certs/ca-bundle.crt # Fedora, CentOS + set --export NIX_SSL_CERT_FILE /etc/pki/tls/certs/ca-bundle.crt + else if test -e "$NIX_LINK/etc/ssl/certs/ca-bundle.crt" # fall back to cacert in Nix profile + set --export NIX_SSL_CERT_FILE "$NIX_LINK/etc/ssl/certs/ca-bundle.crt" + else if test -e "$NIX_LINK/etc/ca-bundle.crt" # old cacert in Nix profile + set --export NIX_SSL_CERT_FILE "$NIX_LINK/etc/ca-bundle.crt" + end + + # Only use MANPATH if it is already set. In general `man` will just simply + # pick up `.nix-profile/share/man` because is it close to `.nix-profile/bin` + # which is in the $PATH. For more info, run `manpath -d`. + set --export --prepend --path MANPATH "$NIX_LINK/share/man" + + fish_add_path --prepend --global "$NIX_LINK/bin" + set --erase NIX_LINK +end diff --git a/scripts/nix-profile.sh.in b/scripts/nix-profile.sh.in index 45cbcbe74..5636085d4 100644 --- a/scripts/nix-profile.sh.in +++ b/scripts/nix-profile.sh.in @@ -1,7 +1,6 @@ if [ -n "$HOME" ] && [ -n "$USER" ]; then # Set up the per-user profile. - # This part should be kept in sync with nixpkgs:nixos/modules/programs/shell.nix NIX_LINK=$HOME/.nix-profile From 7194c87dce39d89868b3bc25790fefb56f7fefae Mon Sep 17 00:00:00 2001 From: Ana Hobden Date: Mon, 12 Sep 2022 09:46:06 -0700 Subject: [PATCH 047/112] Add installer_test matrix for shells Signed-off-by: Ana Hobden --- .github/workflows/ci.yml | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 86b5dfd2e..628d1d192 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -81,7 +81,14 @@ jobs: with: install_url: '${{needs.installer.outputs.installerURL}}' install_options: "--tarball-url-prefix https://${{ env.CACHIX_NAME }}.cachix.org/serve" - - run: nix-instantiate -E 'builtins.currentTime' --eval + - run: sudo apt install fish zsh + if: matrix.os == 'ubuntu-latest' + - run: brew install fish + if: matrix.os == 'macos-latest' + - run: exec bash -c "nix-instantiate -E 'builtins.currentTime' --eval" + - run: exec sh -c "nix-instantiate -E 'builtins.currentTime' --eval" + - run: exec zsh -c "nix-instantiate -E 'builtins.currentTime' --eval" + - run: exec fish -c "nix-instantiate -E 'builtins.currentTime' --eval" docker_push_image: needs: [check_secrets, tests] From fae3b4fe8abc2b307a583e396a24d7899bb21451 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Wed, 14 Sep 2022 15:40:43 +0200 Subject: [PATCH 048/112] Add an installer test This runs the installer in a QEMU VM. Unlike the old installer test that ran inside a declaratively built RedHat/Debian image, this uses an image from Vagrant. --- flake.nix | 5 + tests/installer/default.nix | 136 +++++++++++++++++++++++++++ tests/installer/vagrant_insecure_key | 27 ++++++ 3 files changed, 168 insertions(+) create mode 100644 tests/installer/default.nix create mode 100644 tests/installer/vagrant_insecure_key diff --git a/flake.nix b/flake.nix index cdb81179a..ec64719d1 100644 --- a/flake.nix +++ b/flake.nix @@ -546,6 +546,11 @@ # againstLatestStable = testNixVersions pkgs pkgs.nix pkgs.nixStable; } "touch $out"); + installerTests = import ./tests/installer { + binaryTarballs = self.hydraJobs.binaryTarball; + inherit nixpkgsFor; + }; + }; checks = forAllSystems (system: { diff --git a/tests/installer/default.nix b/tests/installer/default.nix new file mode 100644 index 000000000..39911aeb2 --- /dev/null +++ b/tests/installer/default.nix @@ -0,0 +1,136 @@ +{ binaryTarballs +, nixpkgsFor +}: + +let + + installScripts = { + install-default = { + script = '' + set -eux + + tar -xf ./nix.tar.xz + mv ./nix-* nix + ./nix/install --no-channel-add + ''; + }; + + install-force-no-daemon = { + script = '' + set -eux + + tar -xf ./nix.tar.xz + mv ./nix-* nix + ./nix/install --no-daemon + ''; + }; + + install-force-daemon = { + script = '' + set -eux + + tar -xf ./nix.tar.xz + mv ./nix-* nix + ./nix/install --daemon + ''; + }; + }; + + images = { + + "ubuntu-14-04" = { + image = import { + url = https://app.vagrantup.com/ubuntu/boxes/trusty64/versions/20190514.0.0/providers/virtualbox.box; + hash = "sha256-iUUXyRY8iW7DGirb0zwGgf1fRbLA7wimTJKgP7l/OQ8="; + }; + rootDisk = "box-disk1.vmdk"; + system = "x86_64-linux"; + }; + + "ubuntu-16-04" = { + image = import { + url = https://app.vagrantup.com/ubuntu/boxes/xenial64/versions/20211001.0.0/providers/virtualbox.box; + hash = "sha256-JCc0wd9vaSzCU8coByVtb/oDTAXYBPnORwEShS4oj4U="; + }; + rootDisk = "ubuntu-xenial-16.04-cloudimg.vmdk"; + system = "x86_64-linux"; + }; + + "ubuntu-22-10" = { + image = import { + url = https://app.vagrantup.com/ubuntu/boxes/kinetic64/versions/20220910.0.0/providers/virtualbox.box; + hash = "sha256-/IXr+Apyx2dqX6Gj4SoNtQ/5v1eKKopwzFgozAq6GFY="; + }; + rootDisk = "ubuntu-kinetic-22.10-cloudimg.vmdk"; + system = "x86_64-linux"; + }; + + }; + + makeTest = imageName: testName: + let image = images.${imageName}; in + with nixpkgsFor.${image.system}; + runCommand + "installer-test-${imageName}-${testName}" + { buildInputs = [ qemu_kvm openssh ]; + image = image.image; + installScript = installScripts.${testName}.script; + binaryTarball = binaryTarballs.${system}; + } + '' + echo "Unpacking Vagrant box..." + tar xvf $image + + qemu-img create -b ./${image.rootDisk} -F vmdk -f qcow2 ./disk.qcow2 + + echo "Starting qemu..." + qemu-kvm -m 4096 -nographic \ + -drive id=disk1,file=./disk.qcow2,if=virtio \ + -netdev user,id=net0,restrict=yes,hostfwd=tcp::20022-:22 -device virtio-net-pci,netdev=net0 & + qemu_pid=$! + trap "kill $qemu_pid" EXIT + + if ! [ -e ./vagrant_insecure_key ]; then + cp ${./vagrant_insecure_key} vagrant_insecure_key + fi + + chmod 0400 ./vagrant_insecure_key + + ssh_opts="-o StrictHostKeyChecking=no -o PubkeyAcceptedKeyTypes=+ssh-rsa -i ./vagrant_insecure_key" + ssh="ssh -p 20022 -q $ssh_opts vagrant@localhost" + + echo "Waiting for SSH..." + for ((i = 0; i < 120; i++)); do + echo "[ssh] Trying to connect..." + if $ssh -- true; then + echo "[ssh] Connected!" + break + fi + if ! kill -0 $qemu_pid; then + echo "qemu died unexpectedly" + exit 1 + fi + sleep 1 + done + + echo "Copying installer..." + scp -P 20022 $ssh_opts $binaryTarball/nix-*.tar.xz vagrant@localhost:nix.tar.xz + + echo "Running installer..." + $ssh "$installScript" + + echo "Testing Nix installation..." + # FIXME: should update ~/.bashrc. + $ssh "source ~/.profile; nix-env --version" + + echo "Done!" + touch $out + ''; + +in + +{ + ubuntu-14-04.install-default = makeTest "ubuntu-14-04" "install-default"; + #ubuntu-16-04.install-default = makeTest "ubuntu-16-04" "install-default"; + #ubuntu-22-10.install-default = makeTest "ubuntu-22-10" "install-default"; +} diff --git a/tests/installer/vagrant_insecure_key b/tests/installer/vagrant_insecure_key new file mode 100644 index 000000000..7d6a08390 --- /dev/null +++ b/tests/installer/vagrant_insecure_key @@ -0,0 +1,27 @@ +-----BEGIN RSA PRIVATE KEY----- +MIIEogIBAAKCAQEA6NF8iallvQVp22WDkTkyrtvp9eWW6A8YVr+kz4TjGYe7gHzI +w+niNltGEFHzD8+v1I2YJ6oXevct1YeS0o9HZyN1Q9qgCgzUFtdOKLv6IedplqoP +kcmF0aYet2PkEDo3MlTBckFXPITAMzF8dJSIFo9D8HfdOV0IAdx4O7PtixWKn5y2 +hMNG0zQPyUecp4pzC6kivAIhyfHilFR61RGL+GPXQ2MWZWFYbAGjyiYJnAmCP3NO +Td0jMZEnDkbUvxhMmBYSdETk1rRgm+R4LOzFUGaHqHDLKLX+FIPKcF96hrucXzcW +yLbIbEgE98OHlnVYCzRdK8jlqm8tehUc9c9WhQIBIwKCAQEA4iqWPJXtzZA68mKd +ELs4jJsdyky+ewdZeNds5tjcnHU5zUYE25K+ffJED9qUWICcLZDc81TGWjHyAqD1 +Bw7XpgUwFgeUJwUlzQurAv+/ySnxiwuaGJfhFM1CaQHzfXphgVml+fZUvnJUTvzf +TK2Lg6EdbUE9TarUlBf/xPfuEhMSlIE5keb/Zz3/LUlRg8yDqz5w+QWVJ4utnKnK +iqwZN0mwpwU7YSyJhlT4YV1F3n4YjLswM5wJs2oqm0jssQu/BT0tyEXNDYBLEF4A +sClaWuSJ2kjq7KhrrYXzagqhnSei9ODYFShJu8UWVec3Ihb5ZXlzO6vdNQ1J9Xsf +4m+2ywKBgQD6qFxx/Rv9CNN96l/4rb14HKirC2o/orApiHmHDsURs5rUKDx0f9iP +cXN7S1uePXuJRK/5hsubaOCx3Owd2u9gD6Oq0CsMkE4CUSiJcYrMANtx54cGH7Rk +EjFZxK8xAv1ldELEyxrFqkbE4BKd8QOt414qjvTGyAK+OLD3M2QdCQKBgQDtx8pN +CAxR7yhHbIWT1AH66+XWN8bXq7l3RO/ukeaci98JfkbkxURZhtxV/HHuvUhnPLdX +3TwygPBYZFNo4pzVEhzWoTtnEtrFueKxyc3+LjZpuo+mBlQ6ORtfgkr9gBVphXZG +YEzkCD3lVdl8L4cw9BVpKrJCs1c5taGjDgdInQKBgHm/fVvv96bJxc9x1tffXAcj +3OVdUN0UgXNCSaf/3A/phbeBQe9xS+3mpc4r6qvx+iy69mNBeNZ0xOitIjpjBo2+ +dBEjSBwLk5q5tJqHmy/jKMJL4n9ROlx93XS+njxgibTvU6Fp9w+NOFD/HvxB3Tcz +6+jJF85D5BNAG3DBMKBjAoGBAOAxZvgsKN+JuENXsST7F89Tck2iTcQIT8g5rwWC +P9Vt74yboe2kDT531w8+egz7nAmRBKNM751U/95P9t88EDacDI/Z2OwnuFQHCPDF +llYOUI+SpLJ6/vURRbHSnnn8a/XG+nzedGH5JGqEJNQsz+xT2axM0/W/CRknmGaJ +kda/AoGANWrLCz708y7VYgAtW2Uf1DPOIYMdvo6fxIB5i9ZfISgcJ/bbCUkFrhoH ++vq/5CIWxCPp0f85R4qxxQ5ihxJ0YDQT9Jpx4TMss4PSavPaBH3RXow5Ohe+bYoQ +NE5OgEXk2wVfZczCZpigBKbKZHNYcelXtTt/nP3rsCuGcM4h53s= +-----END RSA PRIVATE KEY----- From 0a8e666dd6d18ede4b5cd648e19d5950ee19f095 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Wed, 14 Sep 2022 18:40:16 +0200 Subject: [PATCH 049/112] Add Fedora 36 --- tests/installer/default.nix | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/tests/installer/default.nix b/tests/installer/default.nix index 39911aeb2..ab3ef62f0 100644 --- a/tests/installer/default.nix +++ b/tests/installer/default.nix @@ -65,6 +65,16 @@ let system = "x86_64-linux"; }; + + "fedora-36" = { + image = import { + url = https://app.vagrantup.com/generic/boxes/fedora36/versions/4.1.12/providers/libvirt.box; + hash = "sha256-rxPgnDnFkTDwvdqn2CV3ZUo3re9AdPtSZ9SvOHNvaks="; + }; + rootDisk = "box.img"; + system = "x86_64-linux"; + }; + }; makeTest = imageName: testName: @@ -78,10 +88,12 @@ let binaryTarball = binaryTarballs.${system}; } '' - echo "Unpacking Vagrant box..." + echo "Unpacking Vagrant box $image..." tar xvf $image - qemu-img create -b ./${image.rootDisk} -F vmdk -f qcow2 ./disk.qcow2 + image_type=$(qemu-img info ${image.rootDisk} | sed 's/file format: \(.*\)/\1/; t; d') + + qemu-img create -b ./${image.rootDisk} -F "$image_type" -f qcow2 ./disk.qcow2 echo "Starting qemu..." qemu-kvm -m 4096 -nographic \ @@ -121,7 +133,7 @@ let echo "Testing Nix installation..." # FIXME: should update ~/.bashrc. - $ssh "source ~/.profile; nix-env --version" + $ssh "source ~/.bash_profile || source ~/.bash_login || source ~/.profile || true; nix-env --version" echo "Done!" touch $out @@ -133,4 +145,5 @@ in ubuntu-14-04.install-default = makeTest "ubuntu-14-04" "install-default"; #ubuntu-16-04.install-default = makeTest "ubuntu-16-04" "install-default"; #ubuntu-22-10.install-default = makeTest "ubuntu-22-10" "install-default"; + fedora-36.install-default = makeTest "fedora-36" "install-default"; } From 906c947ee8f2478d27e5eda649f44716e952d8a6 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Wed, 14 Sep 2022 18:53:30 +0200 Subject: [PATCH 050/112] Enable daemon installation test on Fedora --- tests/installer/default.nix | 28 ++++++++++++++++++---------- 1 file changed, 18 insertions(+), 10 deletions(-) diff --git a/tests/installer/default.nix b/tests/installer/default.nix index ab3ef62f0..8b6bde73f 100644 --- a/tests/installer/default.nix +++ b/tests/installer/default.nix @@ -7,8 +7,6 @@ let installScripts = { install-default = { script = '' - set -eux - tar -xf ./nix.tar.xz mv ./nix-* nix ./nix/install --no-channel-add @@ -17,8 +15,6 @@ let install-force-no-daemon = { script = '' - set -eux - tar -xf ./nix.tar.xz mv ./nix-* nix ./nix/install --no-daemon @@ -27,15 +23,15 @@ let install-force-daemon = { script = '' - set -eux - tar -xf ./nix.tar.xz mv ./nix-* nix - ./nix/install --daemon + ./nix/install --daemon --no-channel-add ''; }; }; + disableSELinux = "sudo setenforce 0"; + images = { "ubuntu-14-04" = { @@ -65,7 +61,6 @@ let system = "x86_64-linux"; }; - "fedora-36" = { image = import { url = https://app.vagrantup.com/generic/boxes/fedora36/versions/4.1.12/providers/libvirt.box; @@ -73,6 +68,7 @@ let }; rootDisk = "box.img"; system = "x86_64-linux"; + postBoot = disableSELinux; }; }; @@ -84,6 +80,7 @@ let "installer-test-${imageName}-${testName}" { buildInputs = [ qemu_kvm openssh ]; image = image.image; + postBoot = image.postBoot or ""; installScript = installScripts.${testName}.script; binaryTarball = binaryTarballs.${system}; } @@ -125,15 +122,25 @@ let sleep 1 done + if [[ -n $postBoot ]]; then + echo "Running post-boot commands..." + $ssh "set -ex; $postBoot" + fi + echo "Copying installer..." scp -P 20022 $ssh_opts $binaryTarball/nix-*.tar.xz vagrant@localhost:nix.tar.xz echo "Running installer..." - $ssh "$installScript" + $ssh "set -eux; $installScript" echo "Testing Nix installation..." # FIXME: should update ~/.bashrc. - $ssh "source ~/.bash_profile || source ~/.bash_login || source ~/.profile || true; nix-env --version" + $ssh " + set -ex + source ~/.bash_profile || source ~/.bash_login || source ~/.profile || true + nix-env --version + nix --extra-experimental-features nix-command store ping + " echo "Done!" touch $out @@ -146,4 +153,5 @@ in #ubuntu-16-04.install-default = makeTest "ubuntu-16-04" "install-default"; #ubuntu-22-10.install-default = makeTest "ubuntu-22-10" "install-default"; fedora-36.install-default = makeTest "fedora-36" "install-default"; + fedora-36.install-force-daemon = makeTest "fedora-36" "install-force-daemon"; } From cc6e31231547fc64c89c6682316f2bab03db6879 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Wed, 14 Sep 2022 19:44:41 +0200 Subject: [PATCH 051/112] Get Ubuntu 22.10 to work --- tests/installer/default.nix | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/tests/installer/default.nix b/tests/installer/default.nix index 8b6bde73f..9b2a34bdb 100644 --- a/tests/installer/default.nix +++ b/tests/installer/default.nix @@ -85,6 +85,8 @@ let binaryTarball = binaryTarballs.${system}; } '' + shopt -s nullglob + echo "Unpacking Vagrant box $image..." tar xvf $image @@ -92,10 +94,19 @@ let qemu-img create -b ./${image.rootDisk} -F "$image_type" -f qcow2 ./disk.qcow2 + extra_qemu_opts= + + # Add the config disk, required by the Ubuntu images. + config_drive=$(echo *configdrive.vmdk || true) + if [[ -n $config_drive ]]; then + extra_qemu_opts+=" -drive id=disk2,file=$config_drive,if=virtio" + fi + echo "Starting qemu..." qemu-kvm -m 4096 -nographic \ -drive id=disk1,file=./disk.qcow2,if=virtio \ - -netdev user,id=net0,restrict=yes,hostfwd=tcp::20022-:22 -device virtio-net-pci,netdev=net0 & + -netdev user,id=net0,restrict=yes,hostfwd=tcp::20022-:22 -device virtio-net-pci,netdev=net0 \ + $extra_qemu_opts & qemu_pid=$! trap "kill $qemu_pid" EXIT @@ -137,7 +148,13 @@ let # FIXME: should update ~/.bashrc. $ssh " set -ex - source ~/.bash_profile || source ~/.bash_login || source ~/.profile || true + + # FIXME: get rid of this; ideally ssh should just work. + source ~/.bash_profile || true + source ~/.bash_login || true + source ~/.profile || true + source /etc/bashrc || true + nix-env --version nix --extra-experimental-features nix-command store ping " @@ -151,7 +168,8 @@ in { ubuntu-14-04.install-default = makeTest "ubuntu-14-04" "install-default"; #ubuntu-16-04.install-default = makeTest "ubuntu-16-04" "install-default"; - #ubuntu-22-10.install-default = makeTest "ubuntu-22-10" "install-default"; + ubuntu-22-10.install-default = makeTest "ubuntu-22-10" "install-default"; + ubuntu-22-10.install-force-daemon = makeTest "ubuntu-22-10" "install-force-daemon"; fedora-36.install-default = makeTest "fedora-36" "install-default"; fedora-36.install-force-daemon = makeTest "fedora-36" "install-force-daemon"; } From 02af02854d41b390957300bac778139bc1c6b5c2 Mon Sep 17 00:00:00 2001 From: Matthew Kenigsberg Date: Wed, 14 Sep 2022 15:35:56 -0600 Subject: [PATCH 052/112] dockerImage: fix root shell Currently root's shell is set to a path that does not exist; this change sets it to the correct path to bash --- docker.nix | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docker.nix b/docker.nix index e95caf274..bb2b4e7ff 100644 --- a/docker.nix +++ b/docker.nix @@ -33,7 +33,7 @@ let root = { uid = 0; - shell = "/bin/bash"; + shell = "${pkgs.bashInteractive}/bin/bash"; home = "/root"; gid = 0; }; From fe958a682d293dec5f27d0c161833b453370d755 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 15 Sep 2022 11:42:10 +0200 Subject: [PATCH 053/112] Test building --- tests/installer/default.nix | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/tests/installer/default.nix b/tests/installer/default.nix index 9b2a34bdb..d0707018f 100644 --- a/tests/installer/default.nix +++ b/tests/installer/default.nix @@ -146,7 +146,7 @@ let echo "Testing Nix installation..." # FIXME: should update ~/.bashrc. - $ssh " + $ssh < \$out"]; }') + [[ \$(cat \$out) = foobar ]] + EOF echo "Done!" touch $out From 29aaec1e593f1837a73779f243ed0ec4220f7ea8 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 15 Sep 2022 13:15:26 +0200 Subject: [PATCH 054/112] Make cross product of images and tests --- tests/installer/default.nix | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/tests/installer/default.nix b/tests/installer/default.nix index d0707018f..72ad764cc 100644 --- a/tests/installer/default.nix +++ b/tests/installer/default.nix @@ -168,11 +168,9 @@ let in -{ - ubuntu-14-04.install-default = makeTest "ubuntu-14-04" "install-default"; - #ubuntu-16-04.install-default = makeTest "ubuntu-16-04" "install-default"; - ubuntu-22-10.install-default = makeTest "ubuntu-22-10" "install-default"; - ubuntu-22-10.install-force-daemon = makeTest "ubuntu-22-10" "install-force-daemon"; - fedora-36.install-default = makeTest "fedora-36" "install-default"; - fedora-36.install-force-daemon = makeTest "fedora-36" "install-force-daemon"; -} +builtins.mapAttrs (imageName: image: + { ${image.system} = builtins.mapAttrs (testName: test: + makeTest imageName testName + ) installScripts; + } +) images From 5c8cdb9b60e0e8d24458a15577e4be3aaa16b600 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 15 Sep 2022 13:19:46 +0200 Subject: [PATCH 055/112] Add Ubuntu 22.04 LTS --- tests/installer/default.nix | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/tests/installer/default.nix b/tests/installer/default.nix index 72ad764cc..17b0fd4ea 100644 --- a/tests/installer/default.nix +++ b/tests/installer/default.nix @@ -52,6 +52,15 @@ let system = "x86_64-linux"; }; + "ubuntu-22-04" = { + image = import { + url = https://app.vagrantup.com/generic/boxes/ubuntu2204/versions/4.1.12/providers/libvirt.box; + hash = "sha256-HNll0Qikw/xGIcogni5lz01vUv+R3o8xowP2EtqjuUQ="; + }; + rootDisk = "box.img"; + system = "x86_64-linux"; + }; + "ubuntu-22-10" = { image = import { url = https://app.vagrantup.com/ubuntu/boxes/kinetic64/versions/20220910.0.0/providers/virtualbox.box; From ef714aa8a566bbdb30919ffd45b8a1fd8e2bc484 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 15 Sep 2022 13:25:26 +0200 Subject: [PATCH 056/112] Remove pre-release Ubuntu 22.10 --- tests/installer/default.nix | 9 --------- 1 file changed, 9 deletions(-) diff --git a/tests/installer/default.nix b/tests/installer/default.nix index 17b0fd4ea..a2cbbcbb8 100644 --- a/tests/installer/default.nix +++ b/tests/installer/default.nix @@ -61,15 +61,6 @@ let system = "x86_64-linux"; }; - "ubuntu-22-10" = { - image = import { - url = https://app.vagrantup.com/ubuntu/boxes/kinetic64/versions/20220910.0.0/providers/virtualbox.box; - hash = "sha256-/IXr+Apyx2dqX6Gj4SoNtQ/5v1eKKopwzFgozAq6GFY="; - }; - rootDisk = "ubuntu-kinetic-22.10-cloudimg.vmdk"; - system = "x86_64-linux"; - }; - "fedora-36" = { image = import { url = https://app.vagrantup.com/generic/boxes/fedora36/versions/4.1.12/providers/libvirt.box; From 503f31e2a0de3192ccc572cc17a6dd02863ebec8 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 15 Sep 2022 13:28:03 +0200 Subject: [PATCH 057/112] Use libvirt image --- tests/installer/default.nix | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/installer/default.nix b/tests/installer/default.nix index a2cbbcbb8..39e3c8d26 100644 --- a/tests/installer/default.nix +++ b/tests/installer/default.nix @@ -45,10 +45,10 @@ let "ubuntu-16-04" = { image = import { - url = https://app.vagrantup.com/ubuntu/boxes/xenial64/versions/20211001.0.0/providers/virtualbox.box; - hash = "sha256-JCc0wd9vaSzCU8coByVtb/oDTAXYBPnORwEShS4oj4U="; + url = https://app.vagrantup.com/generic/boxes/ubuntu1604/versions/4.1.12/providers/libvirt.box; + hash = "sha256-lO4oYQR2tCh5auxAYe6bPOgEqOgv3Y3GC1QM1tEEEU8="; }; - rootDisk = "ubuntu-xenial-16.04-cloudimg.vmdk"; + rootDisk = "box.img"; system = "x86_64-linux"; }; From a96ad2ab25ea054b9d1c473ce2f692bd1f83402b Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 15 Sep 2022 14:51:10 +0200 Subject: [PATCH 058/112] Add RHEL 7/8 --- tests/installer/default.nix | 34 +++++++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/tests/installer/default.nix b/tests/installer/default.nix index 39e3c8d26..d31e2a949 100644 --- a/tests/installer/default.nix +++ b/tests/installer/default.nix @@ -71,6 +71,38 @@ let postBoot = disableSELinux; }; + # Currently fails with 'error while loading shared libraries: + # libsodium.so.23: cannot stat shared object: Invalid argument'. + /* + "rhel-6" = { + image = import { + url = https://app.vagrantup.com/generic/boxes/rhel6/versions/4.1.12/providers/libvirt.box; + hash = "sha256-QwzbvRoRRGqUCQptM7X/InRWFSP2sqwRt2HaaO6zBGM="; + }; + rootDisk = "box.img"; + system = "x86_64-linux"; + }; + */ + + "rhel-7" = { + image = import { + url = https://app.vagrantup.com/generic/boxes/rhel7/versions/4.1.12/providers/libvirt.box; + hash = "sha256-b4afnqKCO9oWXgYHb9DeQ2berSwOjS27rSd9TxXDc/U="; + }; + rootDisk = "box.img"; + system = "x86_64-linux"; + }; + + "rhel-8" = { + image = import { + url = https://app.vagrantup.com/generic/boxes/rhel8/versions/4.1.12/providers/libvirt.box; + hash = "sha256-zFOPjSputy1dPgrQRixBXmlyN88cAKjJ21VvjSWUCUY="; + }; + rootDisk = "box.img"; + system = "x86_64-linux"; + postBoot = disableSELinux; + }; + }; makeTest = imageName: testName: @@ -116,7 +148,7 @@ let chmod 0400 ./vagrant_insecure_key - ssh_opts="-o StrictHostKeyChecking=no -o PubkeyAcceptedKeyTypes=+ssh-rsa -i ./vagrant_insecure_key" + ssh_opts="-o StrictHostKeyChecking=no -o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedKeyTypes=+ssh-rsa -i ./vagrant_insecure_key" ssh="ssh -p 20022 -q $ssh_opts vagrant@localhost" echo "Waiting for SSH..." From ba04b5b1d74c285e12fc3d24524cb8f30f108767 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 15 Sep 2022 14:51:44 +0200 Subject: [PATCH 059/112] Disable Ubuntu 14.04 --- tests/installer/default.nix | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/installer/default.nix b/tests/installer/default.nix index d31e2a949..a3a7f85f9 100644 --- a/tests/installer/default.nix +++ b/tests/installer/default.nix @@ -34,6 +34,7 @@ let images = { + /* "ubuntu-14-04" = { image = import { url = https://app.vagrantup.com/ubuntu/boxes/trusty64/versions/20190514.0.0/providers/virtualbox.box; @@ -42,6 +43,7 @@ let rootDisk = "box-disk1.vmdk"; system = "x86_64-linux"; }; + */ "ubuntu-16-04" = { image = import { From 3dd313a7c20772ef34af4a43fb3673df1e7d00cf Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 15 Sep 2022 15:50:52 +0200 Subject: [PATCH 060/112] Add RHEL 9 --- tests/installer/default.nix | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/tests/installer/default.nix b/tests/installer/default.nix index a3a7f85f9..eab103562 100644 --- a/tests/installer/default.nix +++ b/tests/installer/default.nix @@ -105,6 +105,17 @@ let postBoot = disableSELinux; }; + "rhel-9" = { + image = import { + url = https://app.vagrantup.com/generic/boxes/rhel9/versions/4.1.12/providers/libvirt.box; + hash = "sha256-vL/FbB3kK1rcSaR627nWmScYGKGk4seSmAdq6N5diMg="; + }; + rootDisk = "box.img"; + system = "x86_64-linux"; + postBoot = disableSELinux; + extraQemuOpts = "-cpu Westmere-v2"; + }; + }; makeTest = imageName: testName: @@ -128,7 +139,7 @@ let qemu-img create -b ./${image.rootDisk} -F "$image_type" -f qcow2 ./disk.qcow2 - extra_qemu_opts= + extra_qemu_opts="${image.extraQemuOpts}" # Add the config disk, required by the Ubuntu images. config_drive=$(echo *configdrive.vmdk || true) From 0d4bf9c4d836f8e8570ad1d39245a5835ef4aaf1 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 15 Sep 2022 15:56:46 +0200 Subject: [PATCH 061/112] Fix evaluation --- tests/installer/default.nix | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/installer/default.nix b/tests/installer/default.nix index eab103562..c118937a6 100644 --- a/tests/installer/default.nix +++ b/tests/installer/default.nix @@ -139,7 +139,7 @@ let qemu-img create -b ./${image.rootDisk} -F "$image_type" -f qcow2 ./disk.qcow2 - extra_qemu_opts="${image.extraQemuOpts}" + extra_qemu_opts="${image.extraQemuOpts or ""}" # Add the config disk, required by the Ubuntu images. config_drive=$(echo *configdrive.vmdk || true) From 84fb036062b879c454188a2a4f7123720a6eb9be Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Thu, 15 Sep 2022 18:16:17 +0200 Subject: [PATCH 062/112] add issue template for missing or incorrect documentation this allows anyone to create labelled issues for easy filtering. --- .../ISSUE_TEMPLATE/missing_documentation.md | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 .github/ISSUE_TEMPLATE/missing_documentation.md diff --git a/.github/ISSUE_TEMPLATE/missing_documentation.md b/.github/ISSUE_TEMPLATE/missing_documentation.md new file mode 100644 index 000000000..84868814f --- /dev/null +++ b/.github/ISSUE_TEMPLATE/missing_documentation.md @@ -0,0 +1,28 @@ +--- +name: Missing or incorrect documentation +about: +title: '' +labels: 'documentation' +assignees: '' + +--- + +## Problem + + + +## Checklist + + + +- [ ] checked [latest Nix manual]\ ([source]) +- [ ] checked [open documentation issues and pull requests] for possible duplicates + +[latest Nix manual]: https://nixos.org/manual/nix/unstable/ +[source]: https://github.com/NixOS/nix/tree/master/doc/manual/src +[open documentation issues and pull requests]: https://github.com/NixOS/nix/labels/documentation + +## Proposal + + + From 47fa1087c8864654f6cfae84e25ac6db318ed1d4 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Fri, 16 Sep 2022 09:36:20 +0200 Subject: [PATCH 063/112] Update doc/manual/src/contributing/hacking.md --- doc/manual/src/contributing/hacking.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/manual/src/contributing/hacking.md b/doc/manual/src/contributing/hacking.md index d8a8c8591..628744bf2 100644 --- a/doc/manual/src/contributing/hacking.md +++ b/doc/manual/src/contributing/hacking.md @@ -142,8 +142,8 @@ After the CI run completes, you can check the output to extract the installer ur 1. Click into the detailed view of the CI run. 2. Click into any `installer_test` run (the URL you're here to extract will be the same in all of them). 3. Click into the `Run cachix/install-nix-action@v...` step and click the detail triangle next to the first log line (it will also be `Run cachix/install-nix-action@v...`) -4. Copy the install_url -5. To generate an install command, plug this install_url and your github username into this template: +4. Copy the value of `install_url` +5. To generate an install command, plug this `install_url` and your GitHub username into this template: ```console sh <(curl -L ) --tarball-url-prefix https://-nix-install-tests.cachix.org/serve From 0a4bd9fe88807cbae51b8b8b51c4897a76991d20 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Fri, 16 Sep 2022 09:36:30 +0200 Subject: [PATCH 064/112] Update doc/manual/src/contributing/hacking.md --- doc/manual/src/contributing/hacking.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/src/contributing/hacking.md b/doc/manual/src/contributing/hacking.md index 628744bf2..9e4e679e6 100644 --- a/doc/manual/src/contributing/hacking.md +++ b/doc/manual/src/contributing/hacking.md @@ -138,7 +138,7 @@ If you've already pushed to a fork of Nix on GitHub before, you may have noticed ### Using the CI-generated installer for manual testing -After the CI run completes, you can check the output to extract the installer url: +After the CI run completes, you can check the output to extract the installer URL: 1. Click into the detailed view of the CI run. 2. Click into any `installer_test` run (the URL you're here to extract will be the same in all of them). 3. Click into the `Run cachix/install-nix-action@v...` step and click the detail triangle next to the first log line (it will also be `Run cachix/install-nix-action@v...`) From 1ae974120a24f70eba12e073dbba4b7bac73eedf Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Fri, 16 Sep 2022 09:36:37 +0200 Subject: [PATCH 065/112] Update doc/manual/src/contributing/hacking.md --- doc/manual/src/contributing/hacking.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/src/contributing/hacking.md b/doc/manual/src/contributing/hacking.md index 9e4e679e6..7f3905d38 100644 --- a/doc/manual/src/contributing/hacking.md +++ b/doc/manual/src/contributing/hacking.md @@ -121,7 +121,7 @@ If you've already pushed to a fork of Nix on GitHub before, you may have noticed - `x86_64-darwin`. While this installer is in your Cachix cache, you can use it for manual testing on any of these platforms. -- the `installer_test` job will try to use this installer and run a trivial Nix command on `ubuntu-latest` and `macos-latest`. +- The `installer_test` job will try to use this installer and run a trivial Nix command on `ubuntu-latest` and `macos-latest`. ### One-time setup From dc8c0b173c2cc5bd0fe4273f741fda5591ba4133 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Fri, 16 Sep 2022 09:36:55 +0200 Subject: [PATCH 066/112] Update doc/manual/src/contributing/hacking.md --- doc/manual/src/contributing/hacking.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/src/contributing/hacking.md b/doc/manual/src/contributing/hacking.md index 7f3905d38..f4aeda871 100644 --- a/doc/manual/src/contributing/hacking.md +++ b/doc/manual/src/contributing/hacking.md @@ -130,7 +130,7 @@ If you've already pushed to a fork of Nix on GitHub before, you may have noticed - Create or log in to an account. - Create a Cachix cache using the format `-nix-install-tests`. - Navigate to the new cache > Settings > Auth Tokens. - - Generate a new cachix auth token and copy the generated value. + - Generate a new Cachix auth token and copy the generated value. 4. At github.com: - Navigate to your Nix fork > Settings > Secrets > Actions > New repository secret. - Name the secret `CACHIX_AUTH_TOKEN` From 875a99eaa483850e7794a495102ce0c97658d89f Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Fri, 16 Sep 2022 09:41:27 +0200 Subject: [PATCH 067/112] fix markdown rendering quirk markdown would interpret parentheses as belonging to the first link without escaping. --- .github/ISSUE_TEMPLATE/missing_documentation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/ISSUE_TEMPLATE/missing_documentation.md b/.github/ISSUE_TEMPLATE/missing_documentation.md index 84868814f..8ded9f063 100644 --- a/.github/ISSUE_TEMPLATE/missing_documentation.md +++ b/.github/ISSUE_TEMPLATE/missing_documentation.md @@ -15,7 +15,7 @@ assignees: '' -- [ ] checked [latest Nix manual]\ ([source]) +- [ ] checked [latest Nix manual] \([source]) - [ ] checked [open documentation issues and pull requests] for possible duplicates [latest Nix manual]: https://nixos.org/manual/nix/unstable/ From b3550d9179611692a4e27fbe4e5f493f4e8713e3 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Fri, 16 Sep 2022 00:47:54 -0700 Subject: [PATCH 068/112] libexpr/fetchurl.nix: allow __impure fetch This commit adds an optional `__impure` parameter to fetchurl.nix, which allows the caller to use `libfetcher`'s fetcher in an impure derivation. This allows nixpkgs' patch-normalizing fetcher (fetchpatch) to be rewritten to use nix's internal fetchurl, thereby eliminating the awkward "you can't use fetchpatch here" banners scattered all over the place. See also: https://github.com/NixOS/nixpkgs/pull/188587 --- src/libexpr/fetchurl.nix | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/src/libexpr/fetchurl.nix b/src/libexpr/fetchurl.nix index 02531103b..38815fcc4 100644 --- a/src/libexpr/fetchurl.nix +++ b/src/libexpr/fetchurl.nix @@ -12,13 +12,13 @@ , executable ? false , unpack ? false , name ? baseNameOf (toString url) +, __impure ? false }: -derivation { +derivation ({ builder = "builtin:fetchurl"; # New-style output content requirements. - inherit outputHashAlgo outputHash; outputHashMode = if unpack || executable then "recursive" else "flat"; inherit name url executable unpack; @@ -38,4 +38,6 @@ derivation { # To make "nix-prefetch-url" work. urls = [ url ]; -} +} // (if __impure + then { inherit __impure; } + else { inherit outputHashAlgo outputHash; })) From 673fd21b7c12b3b0a7fd7e0c9c78caefd8906836 Mon Sep 17 00:00:00 2001 From: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com> Date: Fri, 16 Sep 2022 08:51:14 +0000 Subject: [PATCH 069/112] Update src/libexpr/fetchurl.nix Co-authored-by: Eelco Dolstra --- src/libexpr/fetchurl.nix | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/libexpr/fetchurl.nix b/src/libexpr/fetchurl.nix index 38815fcc4..b487e959a 100644 --- a/src/libexpr/fetchurl.nix +++ b/src/libexpr/fetchurl.nix @@ -12,7 +12,7 @@ , executable ? false , unpack ? false , name ? baseNameOf (toString url) -, __impure ? false +, impure ? false }: derivation ({ From fb985f855c3bba09703bfb0ad7618ab881c2b0c4 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Fri, 16 Sep 2022 01:52:20 -0700 Subject: [PATCH 070/112] fetchurl.nix: change other use of __impure --- src/libexpr/fetchurl.nix | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/libexpr/fetchurl.nix b/src/libexpr/fetchurl.nix index b487e959a..9d1b61d7f 100644 --- a/src/libexpr/fetchurl.nix +++ b/src/libexpr/fetchurl.nix @@ -38,6 +38,6 @@ derivation ({ # To make "nix-prefetch-url" work. urls = [ url ]; -} // (if __impure - then { inherit __impure; } +} // (if impure + then { __impure = true; } else { inherit outputHashAlgo outputHash; })) From ad5b09423aa7e5112de06667e28e78a9e4d1ff33 Mon Sep 17 00:00:00 2001 From: Adam Joseph Date: Fri, 16 Sep 2022 01:59:24 -0700 Subject: [PATCH 071/112] release-notes/rl-next.md: note new argument to fetchurl.nix --- doc/manual/src/release-notes/rl-next.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/doc/manual/src/release-notes/rl-next.md b/doc/manual/src/release-notes/rl-next.md index 78ae99f4b..68f7d1a9d 100644 --- a/doc/manual/src/release-notes/rl-next.md +++ b/doc/manual/src/release-notes/rl-next.md @@ -1,2 +1,7 @@ # Release X.Y (202?-??-??) +* `` now accepts an additional argument `impure` which + defaults to `false`. If it is set to `true`, the `hash` and `sha256` + arguments will be ignored and the resulting derivation will have + `__impure` set to `true`, making it an impure derivation. + From 4bd52bf6c4c88e2f8a9b703d75c3db5ad062353c Mon Sep 17 00:00:00 2001 From: "Travis A. Everett" Date: Sat, 17 Sep 2022 13:20:11 -0500 Subject: [PATCH 072/112] Apply suggestions from code review Co-authored-by: Valentin Gagarin --- doc/manual/src/contributing/hacking.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/doc/manual/src/contributing/hacking.md b/doc/manual/src/contributing/hacking.md index f4aeda871..5fad34763 100644 --- a/doc/manual/src/contributing/hacking.md +++ b/doc/manual/src/contributing/hacking.md @@ -85,8 +85,6 @@ $ nix develop ## Testing Nix -Nix comes with three different flavors of tests: unit, functional and integration. - ### Unit-tests The unit-tests for each Nix library (`libexpr`, `libstore`, etc..) are defined From 84bdb0e3ade70be722087b95beb7f460e0d3da8d Mon Sep 17 00:00:00 2001 From: "Travis A. Everett" Date: Sun, 18 Sep 2022 12:58:28 -0500 Subject: [PATCH 073/112] address review feedback Mainly: - Try to triangulate between narrative that framed this as a new/easy process and the need for a reference that will not quickly grow stale. - Fix a ~continuity issue where the text was talking about "your Cachix cache" before saying that you'd need to make a Cachix cache to enable the installer tests. - Adopt suggestion on titling, and nest subtitles in the installer test section. --- doc/manual/src/contributing/hacking.md | 28 +++++++++++++------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/doc/manual/src/contributing/hacking.md b/doc/manual/src/contributing/hacking.md index 5fad34763..f67660ab2 100644 --- a/doc/manual/src/contributing/hacking.md +++ b/doc/manual/src/contributing/hacking.md @@ -83,7 +83,7 @@ by: $ nix develop ``` -## Testing Nix +## Running tests ### Unit-tests @@ -107,21 +107,21 @@ Because these tests are expensive and require more than what the standard github You can run them manually with `nix build .#hydraJobs.tests.{testName}` or `nix-build -A hydraJobs.tests.{testName}` -## Testing the install scripts +### Installer tests -Testing the install scripts has traditionally been tedious, but you can now do this much more easily via the GitHub Actions CI runs (at least for platforms that Github Actions supports). +With just a little one-time setup, the Nix repository's GitHub Actions continuous integration (CI) workflow can easily test the installer each time you push to a branch. -If you've already pushed to a fork of Nix on GitHub before, you may have noticed that the CI workflows in your fork list skipped `installer` and `installer_test` jobs. Once your Nix fork is set up correctly, pushing to it will also run these jobs. -- The `installer` job will generate installers for these platforms: +Creating a Cachix cache for your installer tests and adding its authorization token to GitHub enables [two installer-specific jobs in the CI workflow](https://github.com/NixOS/nix/blob/88a45d6149c0e304f6eb2efcc2d7a4d0d569f8af/.github/workflows/ci.yml#L50-L91): + +- The `installer` job generates installers for the platforms below and uploads them to your Cachix cache: - `x86_64-linux` - `armv6l-linux` - `armv7l-linux` - - `x86_64-darwin`. - - While this installer is in your Cachix cache, you can use it for manual testing on any of these platforms. -- The `installer_test` job will try to use this installer and run a trivial Nix command on `ubuntu-latest` and `macos-latest`. + - `x86_64-darwin` -### One-time setup +- The `installer_test` job (which runs on `ubuntu-latest` and `macos-latest`) will try to install Nix with the cached installer and run a trivial Nix command. + +#### One-time setup 1. Have a GitHub account with a fork of the Nix repo. 2. At cachix.org: @@ -129,12 +129,12 @@ If you've already pushed to a fork of Nix on GitHub before, you may have noticed - Create a Cachix cache using the format `-nix-install-tests`. - Navigate to the new cache > Settings > Auth Tokens. - Generate a new Cachix auth token and copy the generated value. -4. At github.com: +3. At github.com: - Navigate to your Nix fork > Settings > Secrets > Actions > New repository secret. - - Name the secret `CACHIX_AUTH_TOKEN` + - Name the secret `CACHIX_AUTH_TOKEN`. - Paste the copied value of the Cachix cache auth token. -### Using the CI-generated installer for manual testing +#### Using the CI-generated installer for manual testing After the CI run completes, you can check the output to extract the installer URL: 1. Click into the detailed view of the CI run. @@ -147,7 +147,7 @@ After the CI run completes, you can check the output to extract the installer UR sh <(curl -L ) --tarball-url-prefix https://-nix-install-tests.cachix.org/serve ``` -