doc/testing.md: Rewrite some outdated sections for meson and current source layout

Change-Id: Ia23f82c9a564b55bd799afbda59c28c9b0a65c13
This commit is contained in:
jade 2024-06-10 21:31:30 -07:00
parent 5f6eb6eb44
commit 211f79d4a2
2 changed files with 53 additions and 43 deletions

View file

@ -12,44 +12,51 @@ The unit tests are defined using the [googletest] and [rapidcheck] frameworks.
> An example of some files, demonstrating much of what is described below > An example of some files, demonstrating much of what is described below
> >
> ``` > ```
> src
> ├── libexpr
> │ ├── value/context.hh
> │ ├── value/context.cc
> │ │
> │ …
> └── tests
> │ ├── value/context.hh
> │ ├── value/context.cc
> │ │
> │ …
> │
> ├── unit-test-data
> │ ├── libstore
> │ │ ├── worker-protocol/content-address.bin
> │ │ …
> │ …
> … > …
> ├── src
> │   ├── libexpr
> │   │   ├── …
> │   │   ├── value
> │   │   │   ├── context.cc
> │   │   │   └── context.hh
> │ … …
> ├── tests
> │   …
> │   └── unit
> │   ├── libcmd
> │   │   └── args.cc
> │   ├── libexpr
> │   │   ├── …
> │   │   └── value
> │   │   ├── context.cc
> │   │   └── print.cc
> │   ├── libexpr-support
> │   │   └── tests
> │   │   ├── libexpr.hh
> │   │   └── value
> │   │   ├── context.cc
> │   │   └── context.hh
> │   ├── libstore
> │   │   ├── common-protocol.cc
> │   │   ├── data
> │   │   │   ├── libstore
> │   │   │   │   ├── common-protocol
> │   │   │   │   │   ├── content-address.bin
> │   │   │   │   │   ├── drv-output.bin
> … … … … … …
> ``` > ```
<!-- FIXME(Lix): this might get renamed to liblixexpr, etc? --> The unit tests for each Lix library (`liblixexpr`, `liblixstore`, etc..) live inside a directory `src/${library_shortname}/tests` within the directory for the library (`src/${library_shortname}`).
The unit tests for each Lix library (`libnixexpr`, `libnixstore`, etc..) live inside a directory `src/${library_shortname}/tests` within the directory for the library (`src/${library_shortname}`). The data is in `tests/unit/LIBNAME/data/LIBNAME`, with one subdir per library, with the same name as where the code goes.
For example, `liblixstore` code is in `src/libstore`, and its test data is in `tests/unit/libstore/data/libstore`.
The data is in `unit-test-data`, with one subdir per library, with the same name as where the code goes. The path to the unit test data directory is passed to the unit test executable with the environment variable `_NIX_TEST_UNIT_DATA`.
For example, `libnixstore` code is in `src/libstore`, and its test data is in `unit-test-data/libstore`.
The path to the `unit-test-data` directory is passed to the unit test executable with the environment variable `_NIX_TEST_UNIT_DATA`.
> **Note**
> Due to the way googletest works, downstream unit test executables will actually include and re-run upstream library tests.
> Therefore it is important that the same value for `_NIX_TEST_UNIT_DATA` be used with the tests for each library.
> That is why we have the test data nested within a single `unit-test-data` directory.
### Running tests ### Running tests
You can run the whole testsuite with `just test` (see justfile for exact invocation of meson), and if you want to run just one test suite, use `just test --suite installcheck functional-init` where `installcheck` is the name of the test suite in this case and `functional-init` is the name of the test. You can run the whole testsuite with `just test` (see justfile for exact invocation of meson), and if you want to run just one test suite, use `just test --suite installcheck functional-init` where `installcheck` is the name of the test suite in this case and `functional-init` is the name of the test.
To get a list of tests, use `meson test -C build --list`. To get a list of tests, use `meson test -C build --list` (or `just test --list` for short).
For `installcheck` specifically, first run `just install` before running the test suite (this is due to meson limitations that don't let us put a dependency on installing before doing the test). For `installcheck` specifically, first run `just install` before running the test suite (this is due to meson limitations that don't let us put a dependency on installing before doing the test).
@ -74,20 +81,27 @@ See [below](#characterization-testing-1) for a broader discussion of characteriz
Like with the functional characterization, `_NIX_TEST_ACCEPT=1` is also used. Like with the functional characterization, `_NIX_TEST_ACCEPT=1` is also used.
For example: For example:
```shell-session ```shell-session
$ _NIX_TEST_ACCEPT=1 make libstore-tests-exe_RUN $ _NIX_TEST_ACCEPT=1 just test --suite check libstore-unit-tests
... ...
[ SKIPPED ] WorkerProtoTest.string_read ../tests/unit/libstore/common-protocol.cc:27: Skipped
[ SKIPPED ] WorkerProtoTest.string_write Cannot read golden master because another test is also updating it
[ SKIPPED ] WorkerProtoTest.storePath_read
[ SKIPPED ] WorkerProtoTest.storePath_write ../tests/unit/libstore/common-protocol.cc:62: Skipped
Updating golden master
../tests/unit/libstore/common-protocol.cc:27: Skipped
Cannot read golden master because another test is also updating it
../tests/unit/libstore/common-protocol.cc:62: Skipped
Updating golden master
... ...
``` ```
will regenerate the "golden master" expected result for the `libnixstore` characterization tests. will regenerate the "golden master" expected result for the `liblixstore` characterization tests.
The characterization tests will mark themselves "skipped" since they regenerated the expected result instead of actually testing anything. The characterization tests will mark themselves "skipped" since they regenerated the expected result instead of actually testing anything.
## Functional tests ## Functional tests
The functional tests reside under the `tests/functional` directory and are listed in `tests/functional/local.mk`. The functional tests reside under the `tests/functional` directory and are listed in `tests/functional/meson.build`.
Each test is a bash script. Each test is a bash script.
### Running the whole test suite ### Running the whole test suite
@ -185,7 +199,7 @@ edit it like so:
``` ```
<div class="warning"> <div class="warning">
FIXME(meson): the command here may be incorrect for meson. FIXME(meson): the command here is incorrect for meson and this whole functionality may need rebuilding.
</div> </div>
Then, running the test with `./mk/debug-test.sh` will drop you into GDB once the script reaches that point: Then, running the test with `./mk/debug-test.sh` will drop you into GDB once the script reaches that point:
@ -209,15 +223,11 @@ This technique is to include the exact output/behavior of a former version of Ni
For example, this technique is used for the language tests, to check both the printed final value if evaluation was successful, and any errors and warnings encountered. For example, this technique is used for the language tests, to check both the printed final value if evaluation was successful, and any errors and warnings encountered.
<div class="warning">
FIXME(meson): this is incorrect for meson. `_NIX_TEST_ACCEPT=1` is still valid but the test invocation needs to change.
</div>
It is frequently useful to regenerate the expected output. It is frequently useful to regenerate the expected output.
To do that, rerun the failed test(s) with `_NIX_TEST_ACCEPT=1`. To do that, rerun the failed test(s) with `_NIX_TEST_ACCEPT=1`.
For example: For example:
```bash ```bash
_NIX_TEST_ACCEPT=1 make tests/functional/lang.sh.test _NIX_TEST_ACCEPT=1 just test --suite installcheck -v functional-lang
``` ```
An interesting situation to document is the case when these tests are "overfitted". An interesting situation to document is the case when these tests are "overfitted".

View file

@ -229,7 +229,7 @@ This release has the following new features:
<https://cache.nixos.org/> or <ssh://machine>. The following store <https://cache.nixos.org/> or <ssh://machine>. The following store
types are supported: types are supported:
- `LocalStore` (stori URI `local` or an absolute path) and the - `LocalStore` (store URI `local` or an absolute path) and the
misnamed `RemoteStore` (`daemon`) provide access to a local Nix misnamed `RemoteStore` (`daemon`) provide access to a local Nix
store, the latter via the Nix daemon. You can use `auto` or the store, the latter via the Nix daemon. You can use `auto` or the
empty string to auto-select a local or daemon store depending on empty string to auto-select a local or daemon store depending on