lix-project/lix
20
55
Fork 31
Code Issues 345 Code Review (Gerrit) Projects 3 Releases Wiki Activity
lix/tests/tarball.sh

65 lines
2.8 KiB
Bash
Raw Normal View History

Add tarball tests
2015-06-01 14:18:23 +00:00
source common.sh
clearStore
Test the NAR info cache
2016-05-30 12:53:57 +00:00
rm -rf $TEST_HOME
Fix tarball test The tarball cache is stored in $HOME, so the test should set up its own $HOME.
2015-06-02 11:20:43 +00:00
Add tarball tests
2015-06-01 14:18:23 +00:00
tarroot=$TEST_ROOT/tarball
rm -rf $tarroot
mkdir -p $tarroot
cp dependencies.nix $tarroot/default.nix
cp config.nix dependencies.builder*.sh $tarroot/
Tarball trees: Propagate lastModified This makes them behave consistently with GitHub/GitLab flakes.
2023-08-22 17:29:25 +00:00
touch -d '@1000000000' $tarroot $tarroot/*
Add tarball tests
2015-06-01 14:18:23 +00:00
Move 'nix hash-*' and 'nix to-*' to 'nix hash' From the 'nix' UX review.
2020-07-24 16:44:43 +00:00
hash=$(nix hash path $tarroot)
Backport libfetchers from the flakes branch This provides a pluggable mechanism for defining new fetchers. It adds a builtin function 'fetchTree' that generalizes existing fetchers like 'fetchGit', 'fetchMercurial' and 'fetchTarball'. 'fetchTree' takes a set of attributes, e.g. fetchTree { type = "git"; url = "https://example.org/repo.git"; ref = "some-branch"; rev = "abcdef..."; } The existing fetchers are just wrappers around this. Note that the input attributes to fetchTree are the same as flake input specifications and flake lock file entries. All fetchers share a common cache stored in ~/.cache/nix/fetcher-cache-v1.sqlite. This replaces the ad hoc caching mechanisms in fetchGit and download.cc (e.g. ~/.cache/nix/{tarballs,git-revs*}). This also adds support for Git worktrees (c169ea59049f861aaba429f48b828d0820b74d1d).
2020-03-30 14:04:18 +00:00
Simplify tarball test
2019-12-13 16:26:58 +00:00
test_tarball() {
local ext="$1"
local compressor="$2"
Add tarball tests
2015-06-01 14:18:23 +00:00
Simplify tarball test
2019-12-13 16:26:58 +00:00
tarball=$TEST_ROOT/tarball.tar$ext
Fix tar invocation on FreeBSD tar(1) on FreeBSD does not use standard output or input when the -f flag is not provided. Instead, it defaults to /dev/sa0 on FreeBSD. Make this tar invocation a bit more robust and explicitly tell tar(1) to use standard output. This is one of the issues discovered while porting Nix to FreeBSD. It has been tested and committed locally to FreeBSD ports: https://svnweb.freebsd.org/ports/head/sysutils/nix/Makefile?revision=550026&view=markup#l108
2020-09-28 13:23:21 +00:00
(cd $TEST_ROOT && tar cf - tarball) | $compressor > $tarball
Add tarball tests
2015-06-01 14:18:23 +00:00
Harden tests' bash Use `set -u` and `set -o pipefail` to catch accidental mistakes and failures more strongly. - `set -u` catches the use of undefined variables - `set -o pipefail` catches failures (like `set -e`) earlier in the pipeline. This makes the tests a bit more robust. It is nice to read code not worrying about these spurious success paths (via uncaught) errors undermining the tests. Indeed, I caught some bugs doing this. There are a few tests where we run a command that should fail, and then search its output to make sure the failure message is one that we expect. Before, since the `grep` was the last command in the pipeline the exit code of those failing programs was silently ignored. Now with `set -o pipefail` it won't be, and we have to do something so the expected failure doesn't accidentally fail the test. To do that we use `expect` and a new `expectStderr` to check for the exact failing exit code. See the comments on each for why. `grep -q` is replaced with `grepQuiet`, see the comments on that function for why. `grep -v` when we just want the exit code is replaced with `grepInverse, see the comments on that function for why. `grep -q -v` together is, surprise surprise, replaced with `grepQuietInverse`, which is both combined. Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2021-12-09 15:26:46 +00:00
nix-env -f file://$tarball -qa --out-path | grepQuiet dependencies
Add tarball tests
2015-06-01 14:18:23 +00:00
Simplify tarball test
2019-12-13 16:26:58 +00:00
nix-build -o $TEST_ROOT/result file://$tarball
Add tarball tests
2015-06-01 14:18:23 +00:00
Simplify tarball test
2019-12-13 16:26:58 +00:00
nix-build -o $TEST_ROOT/result '<foo>' -I foo=file://$tarball
Make the search path lazier with non-fatal errors Thus, -I / $NIX_PATH entries are now downloaded only when they are needed for evaluation. An error to download an entry is a non-fatal warning (just like non-existant paths). This does change the semantics of builtins.nixPath, which now returns the original, rather than resulting path. E.g., before we had [ { path = "/nix/store/hgm3yxf1lrrwa3z14zpqaj5p9vs0qklk-nixexprs.tar.xz"; prefix = "nixpkgs"; } ... ] but now [ { path = "https://nixos.org/channels/nixos-16.03/nixexprs.tar.xz"; prefix = "nixpkgs"; } ... ] Fixes #792.
2016-04-14 13:32:24 +00:00
Simplify tarball test
2019-12-13 16:26:58 +00:00
nix-build -o $TEST_ROOT/result -E "import (fetchTarball file://$tarball)"
fetchTree: Do not re-fetch paths already present
2022-02-27 14:25:22 +00:00
# Do not re-fetch paths already present
nix-build -o $TEST_ROOT/result -E "import (fetchTarball { url = file:///does-not-exist/must-remain-unused/$tarball; sha256 = \"$hash\"; })"
Make the search path lazier with non-fatal errors Thus, -I / $NIX_PATH entries are now downloaded only when they are needed for evaluation. An error to download an entry is a non-fatal warning (just like non-existant paths). This does change the semantics of builtins.nixPath, which now returns the original, rather than resulting path. E.g., before we had [ { path = "/nix/store/hgm3yxf1lrrwa3z14zpqaj5p9vs0qklk-nixexprs.tar.xz"; prefix = "nixpkgs"; } ... ] but now [ { path = "https://nixos.org/channels/nixos-16.03/nixexprs.tar.xz"; prefix = "nixpkgs"; } ... ] Fixes #792.
2016-04-14 13:32:24 +00:00
Merge legacy `fetchGit`-builtin with the generic `fetchTree`-function The original idea was to implement a git-fetcher in Nix's core that supports content hashes[1]. In #3549[2] it has been suggested to actually use `fetchTree` for this since it's a fairly generic wrapper over the new fetcher-API[3] and already supports content-hashes. This patch implements a new git-fetcher based on `fetchTree` by incorporating the following changes: * Removed the original `fetchGit`-implementation and replaced it with an alias on the `fetchTree` implementation. * Ensured that the `git`-fetcher from `libfetchers` always computes a content-hash and returns an "empty" revision on dirty trees (the latter one is needed to retain backwards-compatibility). * The hash-mismatch error in the fetcher-API exits with code 102 as it usually happens whenever a hash-mismatch is detected by Nix. * Removed the `flakes`-feature-flag: I didn't see a reason why this API is so tightly coupled to the flakes-API and at least `fetchGit` should remain usable without any feature-flags. * It's only possible to specify a `narHash` for a `git`-tree if either a `ref` or a `rev` is given[4]. * It's now possible to specify an URL without a protocol. If it's missing, `file://` is automatically added as it was the case in the original `fetchGit`-implementation. [1] https://github.com/NixOS/nix/pull/3216 [2] https://github.com/NixOS/nix/pull/3549#issuecomment-625194383 [3] https://github.com/NixOS/nix/pull/3459 [4] https://github.com/NixOS/nix/pull/3216#issuecomment-553956703
2020-04-29 20:39:58 +00:00
nix-build -o $TEST_ROOT/result -E "import (fetchTree file://$tarball)"
nix-build -o $TEST_ROOT/result -E "import (fetchTree { type = \"tarball\"; url = file://$tarball; })"
nix-build -o $TEST_ROOT/result -E "import (fetchTree { type = \"tarball\"; url = file://$tarball; narHash = \"$hash\"; })"
fetchTree: Do not re-fetch paths already present
2022-02-27 14:25:22 +00:00
# Do not re-fetch paths already present
nix-build -o $TEST_ROOT/result -E "import (fetchTree { type = \"tarball\"; url = file:///does-not-exist/must-remain-unused/$tarball; narHash = \"$hash\"; })"
Harden tests' bash Use `set -u` and `set -o pipefail` to catch accidental mistakes and failures more strongly. - `set -u` catches the use of undefined variables - `set -o pipefail` catches failures (like `set -e`) earlier in the pipeline. This makes the tests a bit more robust. It is nice to read code not worrying about these spurious success paths (via uncaught) errors undermining the tests. Indeed, I caught some bugs doing this. There are a few tests where we run a command that should fail, and then search its output to make sure the failure message is one that we expect. Before, since the `grep` was the last command in the pipeline the exit code of those failing programs was silently ignored. Now with `set -o pipefail` it won't be, and we have to do something so the expected failure doesn't accidentally fail the test. To do that we use `expect` and a new `expectStderr` to check for the exact failing exit code. See the comments on each for why. `grep -q` is replaced with `grepQuiet`, see the comments on that function for why. `grep -v` when we just want the exit code is replaced with `grepInverse, see the comments on that function for why. `grep -q -v` together is, surprise surprise, replaced with `grepQuietInverse`, which is both combined. Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2021-12-09 15:26:46 +00:00
expectStderr 102 nix-build -o $TEST_ROOT/result -E "import (fetchTree { type = \"tarball\"; url = file://$tarball; narHash = \"sha256-xdKv2pq/IiwLSnBBJXW8hNowI4MrdZfW+SYqDQs7Tzc=\"; })" | grep 'NAR hash mismatch in input'
Merge legacy `fetchGit`-builtin with the generic `fetchTree`-function The original idea was to implement a git-fetcher in Nix's core that supports content hashes[1]. In #3549[2] it has been suggested to actually use `fetchTree` for this since it's a fairly generic wrapper over the new fetcher-API[3] and already supports content-hashes. This patch implements a new git-fetcher based on `fetchTree` by incorporating the following changes: * Removed the original `fetchGit`-implementation and replaced it with an alias on the `fetchTree` implementation. * Ensured that the `git`-fetcher from `libfetchers` always computes a content-hash and returns an "empty" revision on dirty trees (the latter one is needed to retain backwards-compatibility). * The hash-mismatch error in the fetcher-API exits with code 102 as it usually happens whenever a hash-mismatch is detected by Nix. * Removed the `flakes`-feature-flag: I didn't see a reason why this API is so tightly coupled to the flakes-API and at least `fetchGit` should remain usable without any feature-flags. * It's only possible to specify a `narHash` for a `git`-tree if either a `ref` or a `rev` is given[4]. * It's now possible to specify an URL without a protocol. If it's missing, `file://` is automatically added as it was the case in the original `fetchGit`-implementation. [1] https://github.com/NixOS/nix/pull/3216 [2] https://github.com/NixOS/nix/pull/3549#issuecomment-625194383 [3] https://github.com/NixOS/nix/pull/3459 [4] https://github.com/NixOS/nix/pull/3216#issuecomment-553956703
2020-04-29 20:39:58 +00:00
Tarball trees: Propagate lastModified This makes them behave consistently with GitHub/GitLab flakes.
2023-08-22 17:29:25 +00:00
[[ $(nix eval --impure --expr "(fetchTree file://$tarball).lastModified") = 1000000000 ]]
Merge legacy `fetchGit`-builtin with the generic `fetchTree`-function The original idea was to implement a git-fetcher in Nix's core that supports content hashes[1]. In #3549[2] it has been suggested to actually use `fetchTree` for this since it's a fairly generic wrapper over the new fetcher-API[3] and already supports content-hashes. This patch implements a new git-fetcher based on `fetchTree` by incorporating the following changes: * Removed the original `fetchGit`-implementation and replaced it with an alias on the `fetchTree` implementation. * Ensured that the `git`-fetcher from `libfetchers` always computes a content-hash and returns an "empty" revision on dirty trees (the latter one is needed to retain backwards-compatibility). * The hash-mismatch error in the fetcher-API exits with code 102 as it usually happens whenever a hash-mismatch is detected by Nix. * Removed the `flakes`-feature-flag: I didn't see a reason why this API is so tightly coupled to the flakes-API and at least `fetchGit` should remain usable without any feature-flags. * It's only possible to specify a `narHash` for a `git`-tree if either a `ref` or a `rev` is given[4]. * It's now possible to specify an URL without a protocol. If it's missing, `file://` is automatically added as it was the case in the original `fetchGit`-implementation. [1] https://github.com/NixOS/nix/pull/3216 [2] https://github.com/NixOS/nix/pull/3549#issuecomment-625194383 [3] https://github.com/NixOS/nix/pull/3459 [4] https://github.com/NixOS/nix/pull/3216#issuecomment-553956703
2020-04-29 20:39:58 +00:00
nix-instantiate --strict --eval -E "!((import (fetchTree { type = \"tarball\"; url = file://$tarball; narHash = \"$hash\"; })) ? submodules)" >&2
nix-instantiate --strict --eval -E "!((import (fetchTree { type = \"tarball\"; url = file://$tarball; narHash = \"$hash\"; })) ? submodules)" 2>&1 | grep 'true'
Backport libfetchers from the flakes branch This provides a pluggable mechanism for defining new fetchers. It adds a builtin function 'fetchTree' that generalizes existing fetchers like 'fetchGit', 'fetchMercurial' and 'fetchTarball'. 'fetchTree' takes a set of attributes, e.g. fetchTree { type = "git"; url = "https://example.org/repo.git"; ref = "some-branch"; rev = "abcdef..."; } The existing fetchers are just wrappers around this. Note that the input attributes to fetchTree are the same as flake input specifications and flake lock file entries. All fetchers share a common cache stored in ~/.cache/nix/fetcher-cache-v1.sqlite. This replaces the ad hoc caching mechanisms in fetchGit and download.cc (e.g. ~/.cache/nix/{tarballs,git-revs*}). This also adds support for Git worktrees (c169ea59049f861aaba429f48b828d0820b74d1d).
2020-03-30 14:04:18 +00:00
Simplify tarball test
2019-12-13 16:26:58 +00:00
nix-instantiate --eval -E '1 + 2' -I fnord=file://no-such-tarball.tar$ext
nix-instantiate --eval -E 'with <fnord/xyzzy>; 1 + 2' -I fnord=file://no-such-tarball$ext
(! nix-instantiate --eval -E '<fnord/xyzzy> 1' -I fnord=file://no-such-tarball$ext)
Initial gzip support Closes #3256
2019-12-13 08:29:33 +00:00
Simplify tarball test
2019-12-13 16:26:58 +00:00
nix-instantiate --eval -E '<fnord/config.nix>' -I fnord=file://no-such-tarball$ext -I fnord=.
Forbid the `name` attribute for fetchTree We need to support it for the “old” fetch* functions for backwards compatibility, but we don’t need it for fetchTree (as it’s a new function). Given that changing the `name` messes-up the content hashing, we can just forbid passing a custom `name` argument to it
2021-07-08 11:29:58 +00:00
# Ensure that the `name` attribute isn’t accepted as that would mess
# with the content-addressing
(! nix-instantiate --eval -E "fetchTree { type = \"tarball\"; url = file://$tarball; narHash = \"$hash\"; name = \"foo\"; }")
Simplify tarball test
2019-12-13 16:26:58 +00:00
}
Initial gzip support Closes #3256
2019-12-13 08:29:33 +00:00
Simplify tarball test
2019-12-13 16:26:58 +00:00
test_tarball '' cat
test_tarball .xz xz
test_tarball .gz gzip
Validate tarball components
2019-12-13 17:11:37 +00:00
rm -rf $TEST_ROOT/tmp
mkdir -p $TEST_ROOT/tmp
(! TMPDIR=$TEST_ROOT/tmp XDG_RUNTIME_DIR=$TEST_ROOT/tmp nix-env -f file://$(pwd)/bad.tar.xz -qa --out-path)
(! [ -e $TEST_ROOT/tmp/bad ])
Reference in a new issue Copy permalink