Put functional tests in tests/functional

I think it is bad for these reasons when `tests/` contains a mix of
functional and integration tests

 - Concepts is harder to understand, the documentation makes a good
   unit vs functional vs integration distinction, but when the
   integration tests are just two subdirs within `tests/` this is not
   clear.

 - Source filtering in the `flake.nix` is more complex. We need to
   filter out some of the dirs from `tests/`, rather than simply pick
   the dirs we want and take all of them. This is a good sign the
   structure of what we are trying to do is not matching the structure
   of the files.

With this change we have a clean:
```shell-session
$ git show 'HEAD:tests'
tree HEAD:tests

functional/
installer/
nixos/
```

(cherry picked from commit 68c81c737571794f7246db53fb4774e94fcf4b7e)
This commit is contained in:
John Ericson 2023-10-05 12:12:18 -04:00
parent 7242521265
commit 30dcc19d1f
598 changed files with 93 additions and 93 deletions

2
.github/labeler.yml vendored
View file

@ -20,4 +20,4 @@
# Unit tests # Unit tests
- src/*/tests/**/* - src/*/tests/**/*
# Functional and integration tests # Functional and integration tests
- tests/**/* - tests/functional/**/*

34
.gitignore vendored
View file

@ -79,24 +79,24 @@ perl/Makefile.config
/src/build-remote/build-remote /src/build-remote/build-remote
# /tests/ # /tests/functional/
/tests/test-tmp /tests/functional/test-tmp
/tests/common/vars-and-functions.sh /tests/functional/common/vars-and-functions.sh
/tests/result* /tests/functional/result*
/tests/restricted-innocent /tests/functional/restricted-innocent
/tests/shell /tests/functional/shell
/tests/shell.drv /tests/functional/shell.drv
/tests/config.nix /tests/functional/config.nix
/tests/ca/config.nix /tests/functional/ca/config.nix
/tests/dyn-drv/config.nix /tests/functional/dyn-drv/config.nix
/tests/repl-result-out /tests/functional/repl-result-out
/tests/test-libstoreconsumer/test-libstoreconsumer /tests/functional/test-libstoreconsumer/test-libstoreconsumer
# /tests/lang/ # /tests/functional/lang/
/tests/lang/*.out /tests/functional/lang/*.out
/tests/lang/*.out.xml /tests/functional/lang/*.out.xml
/tests/lang/*.err /tests/functional/lang/*.err
/tests/lang/*.ast /tests/functional/lang/*.ast
/perl/lib/Nix/Config.pm /perl/lib/Nix/Config.pm
/perl/lib/Nix/Store.cc /perl/lib/Nix/Store.cc

View file

@ -52,7 +52,7 @@ Check out the [security policy](https://github.com/NixOS/nix/security/policy).
- [ ] Fixes an [idea approved](https://github.com/NixOS/nix/labels/idea%20approved) issue - [ ] Fixes an [idea approved](https://github.com/NixOS/nix/labels/idea%20approved) issue
- [ ] Tests, as appropriate: - [ ] Tests, as appropriate:
- Functional tests [`tests/**.sh`](./tests) - Functional tests [`tests/functional/**.sh`](./tests/functional)
- Unit tests [`src/*/tests`](./src/) - Unit tests [`src/*/tests`](./src/)
- Integration tests [`tests/nixos/*`](./tests/nixos) - Integration tests [`tests/nixos/*`](./tests/nixos)
- [ ] User documentation in the [manual](..doc/manual/src) - [ ] User documentation in the [manual](..doc/manual/src)

View file

@ -26,11 +26,11 @@ makefiles += \
src/libutil/tests/local.mk \ src/libutil/tests/local.mk \
src/libstore/tests/local.mk \ src/libstore/tests/local.mk \
src/libexpr/tests/local.mk \ src/libexpr/tests/local.mk \
tests/local.mk \ tests/functional/local.mk \
tests/ca/local.mk \ tests/functional/ca/local.mk \
tests/dyn-drv/local.mk \ tests/functional/dyn-drv/local.mk \
tests/test-libstoreconsumer/local.mk \ tests/functional/test-libstoreconsumer/local.mk \
tests/plugins/local.mk tests/functional/plugins/local.mk
else else
makefiles += \ makefiles += \
mk/disable-tests.mk mk/disable-tests.mk

View file

@ -12,7 +12,7 @@ Finer-grained filtering is also possible using the [--gtest_filter](https://goog
## Functional tests ## Functional tests
The functional tests reside under the `tests` directory and are listed in `tests/local.mk`. The functional tests reside under the `tests/functional` directory and are listed in `tests/functional/local.mk`.
Each test is a bash script. Each test is a bash script.
### Running the whole test suite ### Running the whole test suite
@ -21,8 +21,8 @@ The whole test suite can be run with:
```shell-session ```shell-session
$ make install && make installcheck $ make install && make installcheck
ran test tests/foo.sh... [PASS] ran test tests/functional/foo.sh... [PASS]
ran test tests/bar.sh... [PASS] ran test tests/functional/bar.sh... [PASS]
... ...
``` ```
@ -30,14 +30,14 @@ ran test tests/bar.sh... [PASS]
Sometimes it is useful to group related tests so they can be easily run together without running the entire test suite. Sometimes it is useful to group related tests so they can be easily run together without running the entire test suite.
Each test group is in a subdirectory of `tests`. Each test group is in a subdirectory of `tests`.
For example, `tests/ca/local.mk` defines a `ca` test group for content-addressed derivation outputs. For example, `tests/functional/ca/local.mk` defines a `ca` test group for content-addressed derivation outputs.
That test group can be run like this: That test group can be run like this:
```shell-session ```shell-session
$ make ca.test-group -j50 $ make ca.test-group -j50
ran test tests/ca/nix-run.sh... [PASS] ran test tests/functional/ca/nix-run.sh... [PASS]
ran test tests/ca/import-derivation.sh... [PASS] ran test tests/functional/ca/import-derivation.sh... [PASS]
... ...
``` ```
@ -56,21 +56,21 @@ install-tests-groups += $(test-group-name)
Individual tests can be run with `make`: Individual tests can be run with `make`:
```shell-session ```shell-session
$ make tests/${testName}.sh.test $ make tests/functional/${testName}.sh.test
ran test tests/${testName}.sh... [PASS] ran test tests/functional/${testName}.sh... [PASS]
``` ```
or without `make`: or without `make`:
```shell-session ```shell-session
$ ./mk/run-test.sh tests/${testName}.sh $ ./mk/run-test.sh tests/functional/${testName}.sh
ran test tests/${testName}.sh... [PASS] ran test tests/functional/${testName}.sh... [PASS]
``` ```
To see the complete output, one can also run: To see the complete output, one can also run:
```shell-session ```shell-session
$ ./mk/debug-test.sh tests/${testName}.sh $ ./mk/debug-test.sh tests/functional/${testName}.sh
+ foo + foo
output from foo output from foo
+ bar + bar
@ -105,7 +105,7 @@ edit it like so:
Then, running the test with `./mk/debug-test.sh` will drop you into GDB once the script reaches that point: Then, running the test with `./mk/debug-test.sh` will drop you into GDB once the script reaches that point:
```shell-session ```shell-session
$ ./mk/debug-test.sh tests/${testName}.sh $ ./mk/debug-test.sh tests/functional/${testName}.sh
... ...
+ gdb blash blub + gdb blash blub
GNU gdb (GDB) 12.1 GNU gdb (GDB) 12.1
@ -124,9 +124,11 @@ This technique is to include the exact output/behavior of a former version of Ni
For example, this technique is used for the language tests, to check both the printed final value if evaluation was successful, and any errors and warnings encountered. For example, this technique is used for the language tests, to check both the printed final value if evaluation was successful, and any errors and warnings encountered.
It is frequently useful to regenerate the expected output. It is frequently useful to regenerate the expected output.
To do that, rerun the failed test with `_NIX_TEST_ACCEPT=1`. To do that, rerun the failed test(s) with `_NIX_TEST_ACCEPT=1`.
(At least, this is the convention we've used for `tests/lang.sh`. For example:
If we add more characterization testing we should always strive to be consistent.) ```bash
_NIX_TEST_ACCEPT=1 make tests/functional/lang.sh.test
```
An interesting situation to document is the case when these tests are "overfitted". An interesting situation to document is the case when these tests are "overfitted".
The language tests are, again, an example of this. The language tests are, again, an example of this.

View file

@ -58,9 +58,7 @@
nixSrc = fileset.toSource { nixSrc = fileset.toSource {
root = ./.; root = ./.;
fileset = fileset.intersect baseFiles ( fileset = fileset.intersect baseFiles (fileset.unions [
fileset.difference
(fileset.unions [
./.version ./.version
./boehmgc-coroutine-sp-fallback.diff ./boehmgc-coroutine-sp-fallback.diff
./bootstrap.sh ./bootstrap.sh
@ -74,19 +72,13 @@
./mk ./mk
./precompiled-headers.h ./precompiled-headers.h
./src ./src
./tests ./tests/functional
./COPYING ./COPYING
./scripts/local.mk ./scripts/local.mk
(fileset.fileFilter (f: lib.strings.hasPrefix "nix-profile" f.name) ./scripts) (fileset.fileFilter (f: lib.strings.hasPrefix "nix-profile" f.name) ./scripts)
# TODO: do we really need README.md? It doesn't seem used in the build. # TODO: do we really need README.md? It doesn't seem used in the build.
./README.md ./README.md
]) ]);
(fileset.unions [
# Removed file sets
./tests/nixos
./tests/installer
])
);
}; };
# Memoize nixpkgs for different platforms for efficiency. # Memoize nixpkgs for different platforms for efficiency.

View file

@ -1,11 +1,15 @@
test_dir=tests/functional
test=$(echo -n "$test" | sed -e "s|^$test_dir/||")
TESTS_ENVIRONMENT=("TEST_NAME=${test%.*}" 'NIX_REMOTE=') TESTS_ENVIRONMENT=("TEST_NAME=${test%.*}" 'NIX_REMOTE=')
: ${BASH:=/usr/bin/env bash} : ${BASH:=/usr/bin/env bash}
init_test () { init_test () {
cd tests && env "${TESTS_ENVIRONMENT[@]}" $BASH -e init.sh 2>/dev/null > /dev/null cd "$test_dir" && env "${TESTS_ENVIRONMENT[@]}" $BASH -e init.sh 2>/dev/null > /dev/null
} }
run_test_proper () { run_test_proper () {
cd $(dirname $test) && env "${TESTS_ENVIRONMENT[@]}" $BASH -e $(basename $test) cd "$test_dir/$(dirname $test)" && env "${TESTS_ENVIRONMENT[@]}" $BASH -e $(basename $test)
} }

View file

@ -776,7 +776,7 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
} }
}; };
/* Synchronisation point for testing, see tests/gc-concurrent.sh. */ /* Synchronisation point for testing, see tests/functional/gc-concurrent.sh. */
if (auto p = getEnv("_NIX_TEST_GC_SYNC")) if (auto p = getEnv("_NIX_TEST_GC_SYNC"))
readFile(*p); readFile(*p);

View file

@ -30,7 +30,7 @@ extern std::regex refRegex;
/// Instead of defining what a good Git Ref is, we define what a bad Git Ref is /// Instead of defining what a good Git Ref is, we define what a bad Git Ref is
/// This is because of the definition of a ref in refs.c in https://github.com/git/git /// This is because of the definition of a ref in refs.c in https://github.com/git/git
/// See tests/fetchGitRefs.sh for the full definition /// See tests/functional/fetchGitRefs.sh for the full definition
const static std::string badGitRefRegexS = "//|^[./]|/\\.|\\.\\.|[[:cntrl:][:space:]:?^~\[]|\\\\|\\*|\\.lock$|\\.lock/|@\\{|[/.]$|^@$|^$"; const static std::string badGitRefRegexS = "//|^[./]|/\\.|\\.\\.|[[:cntrl:][:space:]:?^~\[]|\\\\|\\*|\\.lock$|\\.lock/|@\\{|[/.]$|^@$|^$";
extern std::regex badGitRefRegex; extern std::regex badGitRefRegex;

View file

@ -1,6 +1,6 @@
source common.sh source common.sh
sed -e "s|@localstatedir@|$TEST_ROOT/profile-var|g" -e "s|@coreutils@|$coreutils|g" < ../scripts/nix-profile.sh.in > $TEST_ROOT/nix-profile.sh sed -e "s|@localstatedir@|$TEST_ROOT/profile-var|g" -e "s|@coreutils@|$coreutils|g" < ../../scripts/nix-profile.sh.in > $TEST_ROOT/nix-profile.sh
user=$(whoami) user=$(whoami)
rm -rf $TEST_HOME $TEST_ROOT/profile-var rm -rf $TEST_HOME $TEST_ROOT/profile-var

View file

@ -6,7 +6,7 @@ unset NIX_STATE_DIR
remoteDir=$TEST_ROOT/remote remoteDir=$TEST_ROOT/remote
# Note: ssh{-ng}://localhost bypasses ssh. See tests/build-remote.sh for # Note: ssh{-ng}://localhost bypasses ssh. See tests/functional/build-remote.sh for
# more details. # more details.
nix-build $file -o $TEST_ROOT/result --max-jobs 0 \ nix-build $file -o $TEST_ROOT/result --max-jobs 0 \
--arg busybox $busybox \ --arg busybox $busybox \

View file

@ -25,4 +25,4 @@ clean-files += \
$(d)/config.nix $(d)/config.nix
test-deps += \ test-deps += \
tests/ca/config.nix tests/functional/ca/config.nix

View file

@ -6,7 +6,7 @@ COMMON_VARS_AND_FUNCTIONS_SH_SOURCED=1
export PS4='+(${BASH_SOURCE[0]-$0}:$LINENO) ' export PS4='+(${BASH_SOURCE[0]-$0}:$LINENO) '
export TEST_ROOT=$(realpath ${TMPDIR:-/tmp}/nix-test)/${TEST_NAME:-default} export TEST_ROOT=$(realpath ${TMPDIR:-/tmp}/nix-test)/${TEST_NAME:-default/tests\/functional//}
export NIX_STORE_DIR export NIX_STORE_DIR
if ! NIX_STORE_DIR=$(readlink -f $TEST_ROOT/store 2> /dev/null); then if ! NIX_STORE_DIR=$(readlink -f $TEST_ROOT/store 2> /dev/null); then
# Maybe the build directory is symlinked. # Maybe the build directory is symlinked.

View file

@ -12,4 +12,4 @@ clean-files += \
$(d)/config.nix $(d)/config.nix
test-deps += \ test-deps += \
tests/dyn-drv/config.nix tests/functional/dyn-drv/config.nix

Some files were not shown because too many files have changed in this diff Show more