lix/tests/local.mk
John Ericson c11836126b Harden tests' bash
Use `set -u` and `set -o pipefail` to catch accidental mistakes and
failures more strongly.

 - `set -u` catches the use of undefined variables
 - `set -o pipefail` catches failures (like `set -e`) earlier in the
   pipeline.

This makes the tests a bit more robust. It is nice to read code not
worrying about these spurious success paths (via uncaught) errors
undermining the tests. Indeed, I caught some bugs doing this.

There are a few tests where we run a command that should fail, and then
search its output to make sure the failure message is one that we
expect. Before, since the `grep` was the last command in the pipeline
the exit code of those failing programs was silently ignored. Now with
`set -o pipefail` it won't be, and we have to do something so the
expected failure doesn't accidentally fail the test.

To do that we use `expect` and a new `expectStderr` to check for the
exact failing exit code. See the comments on each for why.

`grep -q` is replaced with `grepQuiet`, see the comments on that
function for why.

`grep -v` when we just want the exit code is replaced with `grepInverse,
see the comments on that function for why.

`grep -q -v` together is, surprise surprise, replaced with
`grepQuietInverse`, which is both combined.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-03-08 10:26:30 -05:00

141 lines
2.9 KiB
Makefile

nix_tests = \
test-infra.sh \
init.sh \
flakes/flakes.sh \
flakes/run.sh \
flakes/mercurial.sh \
flakes/circular.sh \
flakes/init.sh \
flakes/inputs.sh \
flakes/follow-paths.sh \
flakes/bundle.sh \
flakes/check.sh \
flakes/unlocked-override.sh \
flakes/absolute-paths.sh \
flakes/build-paths.sh \
ca/gc.sh \
gc.sh \
remote-store.sh \
lang.sh \
fetchMercurial.sh \
gc-auto.sh \
user-envs.sh \
user-envs-migration.sh \
binary-cache.sh \
multiple-outputs.sh \
ca/build.sh \
ca/new-build-cmd.sh \
nix-build.sh \
gc-concurrent.sh \
repair.sh \
fixed.sh \
export-graph.sh \
timeout.sh \
fetchGitRefs.sh \
gc-runtime.sh \
tarball.sh \
fetchGit.sh \
fetchurl.sh \
fetchPath.sh \
fetchTree-file.sh \
simple.sh \
referrers.sh \
optimise-store.sh \
substitute-with-invalid-ca.sh \
ca/concurrent-builds.sh \
signing.sh \
ca/build-with-garbage-path.sh \
hash.sh \
gc-non-blocking.sh \
check.sh \
ca/substitute.sh \
nix-shell.sh \
ca/signatures.sh \
ca/nix-shell.sh \
ca/nix-copy.sh \
check-refs.sh \
build-remote-input-addressed.sh \
secure-drv-outputs.sh \
restricted.sh \
fetchGitSubmodules.sh \
flakes/search-root.sh \
ca/duplicate-realisation-in-closure.sh \
readfile-context.sh \
nix-channel.sh \
recursive.sh \
dependencies.sh \
check-reqs.sh \
build-remote-content-addressed-fixed.sh \
build-remote-content-addressed-floating.sh \
nar-access.sh \
pure-eval.sh \
eval.sh \
ca/post-hook.sh \
repl.sh \
ca/repl.sh \
ca/recursive.sh \
binary-cache-build-remote.sh \
search.sh \
logging.sh \
export.sh \
config.sh \
add.sh \
local-store.sh \
filter-source.sh \
misc.sh \
dump-db.sh \
linux-sandbox.sh \
build-dry.sh \
structured-attrs.sh \
shell.sh \
brotli.sh \
zstd.sh \
compression-levels.sh \
nix-copy-ssh.sh \
post-hook.sh \
function-trace.sh \
flakes/config.sh \
fmt.sh \
eval-store.sh \
why-depends.sh \
ca/why-depends.sh \
import-derivation.sh \
ca/import-derivation.sh \
nix_path.sh \
case-hack.sh \
placeholders.sh \
ssh-relay.sh \
plugins.sh \
build.sh \
build-delete.sh \
output-normalization.sh \
ca/nix-run.sh \
selfref-gc.sh ca/selfref-gc.sh \
db-migration.sh \
bash-profile.sh \
pass-as-file.sh \
describe-stores.sh \
nix-profile.sh \
suggestions.sh \
store-ping.sh \
fetchClosure.sh \
completions.sh \
flakes/show.sh \
impure-derivations.sh \
path-from-hash-part.sh \
toString-path.sh
ifeq ($(HAVE_LIBCPUID), 1)
nix_tests += compute-levels.sh
endif
install-tests += $(foreach x, $(nix_tests), tests/$(x))
clean-files += $(d)/tests/common/vars-and-functions.sh $(d)/config.nix $(d)/ca/config.nix
test-deps += tests/common/vars-and-functions.sh tests/config.nix tests/ca/config.nix
ifeq ($(BUILD_SHARED_LIBS), 1)
test-deps += tests/plugins/libplugintest.$(SO_EXT)
endif