lix-project/lix-releng-staging
0
0
Fork 0
Code Packages Activity
lix-releng-staging/tests/binary-cache.sh
John Ericson ea0adfc582 Get rid of .drv special-casing for store path installables 
The release notes document the change in behavior, I don't include it
here so there is no risk to it getting out of sync.

> Motivation

>> Plumbing CLI should be simple

Store derivation installations are intended as "plumbing": very simple
utilities for advanced users and scripts, and not what regular users
interact with. (Similarly, regular Git users will use branch and tag
names not explicit hashes for most things.)

The plumbing CLI should prize simplicity over convenience; that is its
raison d'etre. If the user provides a path, we should treat it the same
way not caring what sort of path it is.

>> Scripting

This is especially important for the scripting use-case. when arbitrary
paths are sent to e.g. `nix copy` and the script author wants consistent
behavior regardless of what those store paths are. Otherwise the script
author needs to be careful to filter out `.drv` ones, and then run `nix
copy` again with those paths and `--derivation`. That is not good!

>> Surprisingly low impact

Only two lines in the tests need changing, showing that the impact of
this is pretty light.

Many command, like `nix log` will continue to work with just the
derivation passed as before. This because we used to:

- Special case the drv path and replace it with it's outputs (what this
  gets rid of).

- Turn those output path *back* into the original drv path.

Now we just skip that entire round trip!

> Context

Issue #7261 lays out a broader vision for getting rid of `--derivation`,
and has this as one of its dependencies. But we can do this with or
without that.

`Installable::toDerivations` is changed to handle the case of a
`DerivedPath::Opaque` ending in `.drv`, which is new: it simply doesn't
need to do any extra work in that case. On this basis, commands like
`nix {show-derivation,log} /nix/store/...-foo.drv` still work as before,
as described above.

When testing older daemons, the post-build-hook will be run against the
old CLI, so we need the old version of the post-build-hook to support
that use-case.

Co-authored-by: Travis A. Everett <travis.a.everett@gmail.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-02-28 17:07:05 -05:00

287 lines
8.3 KiB
Bash
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

source common.sh
needLocalStore "'--no-require-sigs' can’t be used with the daemon"
# We can produce drvs directly into the binary cache
clearStore
clearCacheCache
nix-instantiate --store "file://$cacheDir" dependencies.nix
# Create the binary cache.
clearStore
clearCache
outPath=$(nix-build dependencies.nix --no-out-link)
nix copy --to file://$cacheDir $outPath
# Test copying build logs to the binary cache.
nix log --store file://$cacheDir $outPath 2>&1 | grep 'is not available'
nix store copy-log --to file://$cacheDir $outPath
nix log --store file://$cacheDir $outPath | grep FOO
rm -rf $TEST_ROOT/var/log/nix
nix log $outPath 2>&1 | grep 'is not available'
nix log --substituters file://$cacheDir $outPath | grep FOO
# Test copying build logs from the binary cache.
nix store copy-log --from file://$cacheDir $(nix-store -qd $outPath)^'*'
nix log $outPath | grep FOO
basicDownloadTests() {
# No uploading tests bcause upload with force HTTP doesn't work.
# By default, a binary cache doesn't support "nix-env -qas", but does
# support installation.
clearStore
clearCacheCache
nix-env --substituters "file://$cacheDir" -f dependencies.nix -qas \* | grep -- "---"
nix-store --substituters "file://$cacheDir" --no-require-sigs -r $outPath
[ -x $outPath/program ]
# But with the right configuration, "nix-env -qas" should also work.
clearStore
clearCacheCache
echo "WantMassQuery: 1" >> $cacheDir/nix-cache-info
nix-env --substituters "file://$cacheDir" -f dependencies.nix -qas \* | grep -- "--S"
nix-env --substituters "file://$cacheDir" -f dependencies.nix -qas \* | grep -- "--S"
x=$(nix-env -f dependencies.nix -qas \* --prebuilt-only)
[ -z "$x" ]
nix-store --substituters "file://$cacheDir" --no-require-sigs -r $outPath
nix-store --check-validity $outPath
nix-store -qR $outPath | grep input-2
echo "WantMassQuery: 0" >> $cacheDir/nix-cache-info
}
# Test LocalBinaryCacheStore.
basicDownloadTests
# Test HttpBinaryCacheStore.
export _NIX_FORCE_HTTP=1
basicDownloadTests
# Test whether Nix notices if the NAR doesn't match the hash in the NAR info.
clearStore
nar=$(ls $cacheDir/nar/*.nar.xz | head -n1)
mv $nar $nar.good
mkdir -p $TEST_ROOT/empty
nix-store --dump $TEST_ROOT/empty | xz > $nar
nix-build --substituters "file://$cacheDir" --no-require-sigs dependencies.nix -o $TEST_ROOT/result 2>&1 | tee $TEST_ROOT/log
grep -q "hash mismatch" $TEST_ROOT/log
mv $nar.good $nar
# Test whether this unsigned cache is rejected if the user requires signed caches.
clearStore
clearCacheCache
if nix-store --substituters "file://$cacheDir" -r $outPath; then
echo "unsigned binary cache incorrectly accepted"
exit 1
fi
# Test whether fallback works if a NAR has disappeared. This does not require --fallback.
clearStore
mv $cacheDir/nar $cacheDir/nar2
nix-build --substituters "file://$cacheDir" --no-require-sigs dependencies.nix -o $TEST_ROOT/result
mv $cacheDir/nar2 $cacheDir/nar
# Test whether fallback works if a NAR is corrupted. This does require --fallback.
clearStore
mv $cacheDir/nar $cacheDir/nar2
mkdir $cacheDir/nar
for i in $(cd $cacheDir/nar2 && echo *); do touch $cacheDir/nar/$i; done
(! nix-build --substituters "file://$cacheDir" --no-require-sigs dependencies.nix -o $TEST_ROOT/result)
nix-build --substituters "file://$cacheDir" --no-require-sigs dependencies.nix -o $TEST_ROOT/result --fallback
rm -rf $cacheDir/nar
mv $cacheDir/nar2 $cacheDir/nar
# Test whether building works if the binary cache contains an
# incomplete closure.
clearStore
rm -v $(grep -l "StorePath:.*dependencies-input-2" $cacheDir/*.narinfo)
nix-build --substituters "file://$cacheDir" --no-require-sigs dependencies.nix -o $TEST_ROOT/result 2>&1 | tee $TEST_ROOT/log
grep -q "copying path.*input-0" $TEST_ROOT/log
grep -q "copying path.*input-2" $TEST_ROOT/log
grep -q "copying path.*top" $TEST_ROOT/log
# Idem, but without cached .narinfo.
clearStore
clearCacheCache
nix-build --substituters "file://$cacheDir" --no-require-sigs dependencies.nix -o $TEST_ROOT/result 2>&1 | tee $TEST_ROOT/log
grep -q "don't know how to build" $TEST_ROOT/log
grep -q "building.*input-1" $TEST_ROOT/log
grep -q "building.*input-2" $TEST_ROOT/log
grep -q "copying path.*input-0" $TEST_ROOT/log
grep -q "copying path.*top" $TEST_ROOT/log
# Create a signed binary cache.
clearCache
clearCacheCache
nix key generate-secret --key-name test.nixos.org-1 > $TEST_ROOT/sk1
publicKey=$(nix key convert-secret-to-public < $TEST_ROOT/sk1)
nix key generate-secret --key-name test.nixos.org-1 > $TEST_ROOT/sk2
badKey=$(nix key convert-secret-to-public < $TEST_ROOT/sk2)
nix key generate-secret --key-name foo.nixos.org-1 > $TEST_ROOT/sk3
otherKey=$(nix key convert-secret-to-public < $TEST_ROOT/sk3)
_NIX_FORCE_HTTP= nix copy --to file://$cacheDir?secret-key=$TEST_ROOT/sk1 $outPath
# Downloading should fail if we don't provide a key.
clearStore
clearCacheCache
(! nix-store -r $outPath --substituters "file://$cacheDir")
# And it should fail if we provide an incorrect key.
clearStore
clearCacheCache
(! nix-store -r $outPath --substituters "file://$cacheDir" --trusted-public-keys "$badKey")
# It should succeed if we provide the correct key.
nix-store -r $outPath --substituters "file://$cacheDir" --trusted-public-keys "$otherKey $publicKey"
# It should fail if we corrupt the .narinfo.
clearStore
cacheDir2=$TEST_ROOT/binary-cache-2
rm -rf $cacheDir2
cp -r $cacheDir $cacheDir2
for i in $cacheDir2/*.narinfo; do
grep -v References $i > $i.tmp
mv $i.tmp $i
done
clearCacheCache
(! nix-store -r $outPath --substituters "file://$cacheDir2" --trusted-public-keys "$publicKey")
# If we provide a bad and a good binary cache, it should succeed.
nix-store -r $outPath --substituters "file://$cacheDir2 file://$cacheDir" --trusted-public-keys "$publicKey"
unset _NIX_FORCE_HTTP
# Test 'nix verify --all' on a binary cache.
nix store verify -vvvvv --all --store file://$cacheDir --no-trust
# Test local NAR caching.
narCache=$TEST_ROOT/nar-cache
rm -rf $narCache
mkdir $narCache
[[ $(nix store cat --store "file://$cacheDir?local-nar-cache=$narCache" $outPath/foobar) = FOOBAR ]]
rm -rfv "$cacheDir/nar"
[[ $(nix store cat --store "file://$cacheDir?local-nar-cache=$narCache" $outPath/foobar) = FOOBAR ]]
(! nix store cat --store file://$cacheDir $outPath/foobar)
# Test NAR listing generation.
clearCache
outPath=$(nix-build --no-out-link -E '
with import ./config.nix;
mkDerivation {
name = "nar-listing";
buildCommand = "mkdir $out; echo foo > $out/bar; ln -s xyzzy $out/link";
}
')
nix copy --to file://$cacheDir?write-nar-listing=1 $outPath
diff -u \
<(jq -S < $cacheDir/$(basename $outPath | cut -c1-32).ls) \
<(echo '{"version":1,"root":{"type":"directory","entries":{"bar":{"type":"regular","size":4,"narOffset":232},"link":{"type":"symlink","target":"xyzzy"}}}}' | jq -S)
# Test debug info index generation.
clearCache
outPath=$(nix-build --no-out-link -E '
with import ./config.nix;
mkDerivation {
name = "debug-info";
buildCommand = "mkdir -p $out/lib/debug/.build-id/02; echo foo > $out/lib/debug/.build-id/02/623eda209c26a59b1a8638ff7752f6b945c26b.debug";
}
')
nix copy --to "file://$cacheDir?index-debug-info=1&compression=none" $outPath
diff -u \
<(cat $cacheDir/debuginfo/02623eda209c26a59b1a8638ff7752f6b945c26b.debug | jq -S) \
<(echo '{"archive":"../nar/100vxs724qr46phz8m24iswmg9p3785hsyagz0kchf6q6gf06sw6.nar","member":"lib/debug/.build-id/02/623eda209c26a59b1a8638ff7752f6b945c26b.debug"}' | jq -S)
# Test against issue https://github.com/NixOS/nix/issues/3964
#
expr='
with import ./config.nix;
mkDerivation {
name = "multi-output";
buildCommand = "mkdir -p $out; echo foo > $doc; echo $doc > $out/docref";
outputs = ["out" "doc"];
}
'
outPath=$(nix-build --no-out-link -E "$expr")
docPath=$(nix-store -q --references $outPath)
# $ nix-store -q --tree $outPath
# ...-multi-output
# +---...-multi-output-doc
nix copy --to "file://$cacheDir" $outPath
hashpart() {
basename "$1" | cut -c1-32
}
# break the closure of out by removing doc
rm $cacheDir/$(hashpart $docPath).narinfo
nix-store --delete $outPath $docPath
# -vvv is the level that logs during the loop
timeout 60 nix-build --no-out-link -E "$expr" --option substituters "file://$cacheDir" \
--option trusted-binary-caches "file://$cacheDir" --no-require-sigs
View git blame Copy permalink