Merge pull request #9517 from NixOS/2.18-flatten-tests

[Backport 2.18-maintanence] Backport test source layout reorgs
This commit is contained in:
John Ericson 2023-12-01 13:53:13 -05:00 committed by GitHub
commit 211b6e1855
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
655 changed files with 479 additions and 386 deletions

2
.github/labeler.yml vendored
View file

@ -20,4 +20,4 @@
# Unit tests
- src/*/tests/**/*
# Functional and integration tests
- tests/**/*
- tests/functional/**/*

40
.gitignore vendored
View file

@ -41,14 +41,14 @@ perl/Makefile.config
/src/libexpr/parser-tab.hh
/src/libexpr/parser-tab.output
/src/libexpr/nix.tbl
/src/libexpr/tests/libnixexpr-tests
/tests/unit/libexpr/libnixexpr-tests
# /src/libstore/
*.gen.*
/src/libstore/tests/libnixstore-tests
/tests/unit/libstore/libnixstore-tests
# /src/libutil/
/src/libutil/tests/libnixutil-tests
/tests/unit/libutil/libnixutil-tests
/src/nix/nix
@ -79,24 +79,24 @@ perl/Makefile.config
/src/build-remote/build-remote
# /tests/
/tests/test-tmp
/tests/common/vars-and-functions.sh
/tests/result*
/tests/restricted-innocent
/tests/shell
/tests/shell.drv
/tests/config.nix
/tests/ca/config.nix
/tests/dyn-drv/config.nix
/tests/repl-result-out
/tests/test-libstoreconsumer/test-libstoreconsumer
# /tests/functional/
/tests/functional/test-tmp
/tests/functional/common/vars-and-functions.sh
/tests/functional/result*
/tests/functional/restricted-innocent
/tests/functional/shell
/tests/functional/shell.drv
/tests/functional/config.nix
/tests/functional/ca/config.nix
/tests/functional/dyn-drv/config.nix
/tests/functional/repl-result-out
/tests/functional/test-libstoreconsumer/test-libstoreconsumer
# /tests/lang/
/tests/lang/*.out
/tests/lang/*.out.xml
/tests/lang/*.err
/tests/lang/*.ast
# /tests/functional/lang/
/tests/functional/lang/*.out
/tests/functional/lang/*.out.xml
/tests/functional/lang/*.err
/tests/functional/lang/*.ast
/perl/lib/Nix/Config.pm
/perl/lib/Nix/Store.cc

View file

@ -52,7 +52,7 @@ Check out the [security policy](https://github.com/NixOS/nix/security/policy).
- [ ] Fixes an [idea approved](https://github.com/NixOS/nix/labels/idea%20approved) issue
- [ ] Tests, as appropriate:
- Functional tests [`tests/**.sh`](./tests)
- Functional tests [`tests/functional/**.sh`](./tests/functional)
- Unit tests [`src/*/tests`](./src/)
- Integration tests [`tests/nixos/*`](./tests/nixos)
- [ ] User documentation in the [manual](..doc/manual/src)

View file

@ -23,14 +23,17 @@ makefiles = \
ifeq ($(tests), yes)
makefiles += \
src/libutil/tests/local.mk \
src/libstore/tests/local.mk \
src/libexpr/tests/local.mk \
tests/local.mk \
tests/ca/local.mk \
tests/dyn-drv/local.mk \
tests/test-libstoreconsumer/local.mk \
tests/plugins/local.mk
tests/unit/libutil/local.mk \
tests/unit/libutil-support/local.mk \
tests/unit/libstore/local.mk \
tests/unit/libstore-support/local.mk \
tests/unit/libexpr/local.mk \
tests/unit/libexpr-support/local.mk \
tests/functional/local.mk \
tests/functional/ca/local.mk \
tests/functional/dyn-drv/local.mk \
tests/functional/test-libstoreconsumer/local.mk \
tests/functional/plugins/local.mk
else
makefiles += \
mk/disable-tests.mk

View file

@ -39,17 +39,21 @@ INPUT = \
src/libcmd \
src/libexpr \
src/libexpr/flake \
src/libexpr/tests \
src/libexpr/tests/value \
tests/unit/libexpr \
tests/unit/libexpr/value \
tests/unit/libexpr/test \
tests/unit/libexpr/test/value \
src/libexpr/value \
src/libfetchers \
src/libmain \
src/libstore \
src/libstore/build \
src/libstore/builtins \
src/libstore/tests \
tests/unit/libstore \
tests/unit/libstore/test \
src/libutil \
src/libutil/tests \
tests/unit/libutil \
tests/unit/libutil/test \
src/nix \
src/nix-env \
src/nix-store

View file

@ -3,16 +3,28 @@
## Unit-tests
The unit-tests for each Nix library (`libexpr`, `libstore`, etc..) are defined
under `src/{library_name}/tests` using the
under `tests/unit/{library_name}/tests` using the
[googletest](https://google.github.io/googletest/) and
[rapidcheck](https://github.com/emil-e/rapidcheck) frameworks.
You can run the whole testsuite with `make check`, or the tests for a specific component with `make libfoo-tests_RUN`.
Finer-grained filtering is also possible using the [--gtest_filter](https://google.github.io/googletest/advanced.html#running-a-subset-of-the-tests) command-line option, or the `GTEST_FILTER` environment variable.
### Unit test support libraries
There are headers and code which are not just used to test the library in question, but also downstream libraries.
For example, we do [property testing] with the [rapidcheck] library.
This requires writing `Arbitrary` "instances", which are used to describe how to generate values of a given type for the sake of running property tests.
Because types contain other types, `Arbitrary` "instances" for some type are not just useful for testing that type, but also any other type that contains it.
Downstream types frequently contain upstream types, so it is very important that we share arbitrary instances so that downstream libraries' property tests can also use them.
It is important that these testing libraries don't contain any actual tests themselves.
On some platforms they would be run as part of every test executable that uses them, which is redundant.
On other platforms they wouldn't be run at all.
## Functional tests
The functional tests reside under the `tests` directory and are listed in `tests/local.mk`.
The functional tests reside under the `tests/functional` directory and are listed in `tests/functional/local.mk`.
Each test is a bash script.
### Running the whole test suite
@ -21,8 +33,8 @@ The whole test suite can be run with:
```shell-session
$ make install && make installcheck
ran test tests/foo.sh... [PASS]
ran test tests/bar.sh... [PASS]
ran test tests/functional/foo.sh... [PASS]
ran test tests/functional/bar.sh... [PASS]
...
```
@ -30,14 +42,14 @@ ran test tests/bar.sh... [PASS]
Sometimes it is useful to group related tests so they can be easily run together without running the entire test suite.
Each test group is in a subdirectory of `tests`.
For example, `tests/ca/local.mk` defines a `ca` test group for content-addressed derivation outputs.
For example, `tests/functional/ca/local.mk` defines a `ca` test group for content-addressed derivation outputs.
That test group can be run like this:
```shell-session
$ make ca.test-group -j50
ran test tests/ca/nix-run.sh... [PASS]
ran test tests/ca/import-derivation.sh... [PASS]
ran test tests/functional/ca/nix-run.sh... [PASS]
ran test tests/functional/ca/import-derivation.sh... [PASS]
...
```
@ -56,21 +68,21 @@ install-tests-groups += $(test-group-name)
Individual tests can be run with `make`:
```shell-session
$ make tests/${testName}.sh.test
ran test tests/${testName}.sh... [PASS]
$ make tests/functional/${testName}.sh.test
ran test tests/functional/${testName}.sh... [PASS]
```
or without `make`:
```shell-session
$ ./mk/run-test.sh tests/${testName}.sh
ran test tests/${testName}.sh... [PASS]
$ ./mk/run-test.sh tests/functional/${testName}.sh
ran test tests/functional/${testName}.sh... [PASS]
```
To see the complete output, one can also run:
```shell-session
$ ./mk/debug-test.sh tests/${testName}.sh
$ ./mk/debug-test.sh tests/functional/${testName}.sh
+ foo
output from foo
+ bar
@ -105,7 +117,7 @@ edit it like so:
Then, running the test with `./mk/debug-test.sh` will drop you into GDB once the script reaches that point:
```shell-session
$ ./mk/debug-test.sh tests/${testName}.sh
$ ./mk/debug-test.sh tests/functional/${testName}.sh
...
+ gdb blash blub
GNU gdb (GDB) 12.1
@ -124,9 +136,11 @@ This technique is to include the exact output/behavior of a former version of Ni
For example, this technique is used for the language tests, to check both the printed final value if evaluation was successful, and any errors and warnings encountered.
It is frequently useful to regenerate the expected output.
To do that, rerun the failed test with `_NIX_TEST_ACCEPT=1`.
(At least, this is the convention we've used for `tests/lang.sh`.
If we add more characterization testing we should always strive to be consistent.)
To do that, rerun the failed test(s) with `_NIX_TEST_ACCEPT=1`.
For example:
```bash
_NIX_TEST_ACCEPT=1 make tests/functional/lang.sh.test
```
An interesting situation to document is the case when these tests are "overfitted".
The language tests are, again, an example of this.

View file

@ -58,35 +58,28 @@
nixSrc = fileset.toSource {
root = ./.;
fileset = fileset.intersect baseFiles (
fileset.difference
(fileset.unions [
./.version
./boehmgc-coroutine-sp-fallback.diff
./bootstrap.sh
./configure.ac
./doc
./local.mk
./m4
./Makefile
./Makefile.config.in
./misc
./mk
./precompiled-headers.h
./src
./tests
./COPYING
./scripts/local.mk
(fileset.fileFilter (f: lib.strings.hasPrefix "nix-profile" f.name) ./scripts)
# TODO: do we really need README.md? It doesn't seem used in the build.
./README.md
])
(fileset.unions [
# Removed file sets
./tests/nixos
./tests/installer
])
);
fileset = fileset.intersect baseFiles (fileset.unions [
./.version
./boehmgc-coroutine-sp-fallback.diff
./bootstrap.sh
./configure.ac
./doc
./local.mk
./m4
./Makefile
./Makefile.config.in
./misc
./mk
./precompiled-headers.h
./src
./tests/functional
./tests/unit
./COPYING
./scripts/local.mk
(fileset.fileFilter (f: lib.strings.hasPrefix "nix-profile" f.name) ./scripts)
# TODO: do we really need README.md? It doesn't seem used in the build.
./README.md
]);
};
# Memoize nixpkgs for different platforms for efficiency.

View file

@ -1,11 +1,15 @@
test_dir=tests/functional
test=$(echo -n "$test" | sed -e "s|^$test_dir/||")
TESTS_ENVIRONMENT=("TEST_NAME=${test%.*}" 'NIX_REMOTE=')
: ${BASH:=/usr/bin/env bash}
init_test () {
cd tests && env "${TESTS_ENVIRONMENT[@]}" $BASH -e init.sh 2>/dev/null > /dev/null
cd "$test_dir" && env "${TESTS_ENVIRONMENT[@]}" $BASH -e init.sh 2>/dev/null > /dev/null
}
run_test_proper () {
cd $(dirname $test) && env "${TESTS_ENVIRONMENT[@]}" $BASH -e $(basename $test)
cd "$test_dir/$(dirname $test)" && env "${TESTS_ENVIRONMENT[@]}" $BASH -e $(basename $test)
}

View file

@ -1,19 +0,0 @@
check: libexpr-tests_RUN
programs += libexpr-tests
libexpr-tests_NAME := libnixexpr-tests
libexpr-tests_DIR := $(d)
libexpr-tests_INSTALL_DIR :=
libexpr-tests_SOURCES := \
$(wildcard $(d)/*.cc) \
$(wildcard $(d)/value/*.cc)
libexpr-tests_CXXFLAGS += -I src/libexpr -I src/libutil -I src/libstore -I src/libexpr/tests -I src/libfetchers
libexpr-tests_LIBS = libstore-tests libutils-tests libexpr libutil libstore libfetchers
libexpr-tests_LDFLAGS := -lrapidcheck $(GTEST_LIBS) -lgmock

View file

@ -776,7 +776,7 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
}
};
/* Synchronisation point for testing, see tests/gc-concurrent.sh. */
/* Synchronisation point for testing, see tests/functional/gc-concurrent.sh. */
if (auto p = getEnv("_NIX_TEST_GC_SYNC"))
readFile(*p);

View file

@ -1,29 +0,0 @@
check: libstore-tests-exe_RUN
programs += libstore-tests-exe
libstore-tests-exe_NAME = libnixstore-tests
libstore-tests-exe_DIR := $(d)
libstore-tests-exe_INSTALL_DIR :=
libstore-tests-exe_LIBS = libstore-tests
libstore-tests-exe_LDFLAGS := $(GTEST_LIBS)
libraries += libstore-tests
libstore-tests_NAME = libnixstore-tests
libstore-tests_DIR := $(d)
libstore-tests_INSTALL_DIR :=
libstore-tests_SOURCES := $(wildcard $(d)/*.cc)
libstore-tests_CXXFLAGS += -I src/libstore -I src/libutil
libstore-tests_LIBS = libutil-tests libstore libutil
libstore-tests_LDFLAGS := -lrapidcheck $(GTEST_LIBS)

View file

@ -1,29 +0,0 @@
check: libutil-tests_RUN
programs += libutil-tests
libutil-tests-exe_NAME = libnixutil-tests
libutil-tests-exe_DIR := $(d)
libutil-tests-exe_INSTALL_DIR :=
libutil-tests-exe_LIBS = libutil-tests
libutil-tests-exe_LDFLAGS := $(GTEST_LIBS)
libraries += libutil-tests
libutil-tests_NAME = libnixutil-tests
libutil-tests_DIR := $(d)
libutil-tests_INSTALL_DIR :=
libutil-tests_SOURCES := $(wildcard $(d)/*.cc)
libutil-tests_CXXFLAGS += -I src/libutil
libutil-tests_LIBS = libutil
libutil-tests_LDFLAGS := -lrapidcheck $(GTEST_LIBS)

View file

@ -30,7 +30,7 @@ extern std::regex refRegex;
/// Instead of defining what a good Git Ref is, we define what a bad Git Ref is
/// This is because of the definition of a ref in refs.c in https://github.com/git/git
/// See tests/fetchGitRefs.sh for the full definition
/// See tests/functional/fetchGitRefs.sh for the full definition
const static std::string badGitRefRegexS = "//|^[./]|/\\.|\\.\\.|[[:cntrl:][:space:]:?^~\[]|\\\\|\\*|\\.lock$|\\.lock/|@\\{|[/.]$|^@$|^$";
extern std::regex badGitRefRegex;

View file

@ -1,6 +1,6 @@
source common.sh
sed -e "s|@localstatedir@|$TEST_ROOT/profile-var|g" -e "s|@coreutils@|$coreutils|g" < ../scripts/nix-profile.sh.in > $TEST_ROOT/nix-profile.sh
sed -e "s|@localstatedir@|$TEST_ROOT/profile-var|g" -e "s|@coreutils@|$coreutils|g" < ../../scripts/nix-profile.sh.in > $TEST_ROOT/nix-profile.sh
user=$(whoami)
rm -rf $TEST_HOME $TEST_ROOT/profile-var

View file

@ -6,7 +6,7 @@ unset NIX_STATE_DIR
remoteDir=$TEST_ROOT/remote
# Note: ssh{-ng}://localhost bypasses ssh. See tests/build-remote.sh for
# Note: ssh{-ng}://localhost bypasses ssh. See tests/functional/build-remote.sh for
# more details.
nix-build $file -o $TEST_ROOT/result --max-jobs 0 \
--arg busybox $busybox \

View file

@ -25,4 +25,4 @@ clean-files += \
$(d)/config.nix
test-deps += \
tests/ca/config.nix
tests/functional/ca/config.nix

View file

@ -6,7 +6,7 @@ COMMON_VARS_AND_FUNCTIONS_SH_SOURCED=1
export PS4='+(${BASH_SOURCE[0]-$0}:$LINENO) '
export TEST_ROOT=$(realpath ${TMPDIR:-/tmp}/nix-test)/${TEST_NAME:-default}
export TEST_ROOT=$(realpath ${TMPDIR:-/tmp}/nix-test)/${TEST_NAME:-default/tests\/functional//}
export NIX_STORE_DIR
if ! NIX_STORE_DIR=$(readlink -f $TEST_ROOT/store 2> /dev/null); then
# Maybe the build directory is symlinked.

View file

@ -12,4 +12,4 @@ clean-files += \
$(d)/config.nix
test-deps += \
tests/dyn-drv/config.nix
tests/functional/dyn-drv/config.nix

Some files were not shown because too many files have changed in this diff Show more