forked from lix-project/lix
doc/testing.md: Rewrite some outdated sections for meson and current source layout
Change-Id: Ia23f82c9a564b55bd799afbda59c28c9b0a65c13
This commit is contained in:
parent
5f6eb6eb44
commit
211f79d4a2
|
@ -12,44 +12,51 @@ The unit tests are defined using the [googletest] and [rapidcheck] frameworks.
|
||||||
> An example of some files, demonstrating much of what is described below
|
> An example of some files, demonstrating much of what is described below
|
||||||
>
|
>
|
||||||
> ```
|
> ```
|
||||||
> src
|
|
||||||
> ├── libexpr
|
|
||||||
> │ ├── value/context.hh
|
|
||||||
> │ ├── value/context.cc
|
|
||||||
> │ │
|
|
||||||
> │ …
|
|
||||||
> └── tests
|
|
||||||
> │ ├── value/context.hh
|
|
||||||
> │ ├── value/context.cc
|
|
||||||
> │ │
|
|
||||||
> │ …
|
|
||||||
> │
|
|
||||||
> ├── unit-test-data
|
|
||||||
> │ ├── libstore
|
|
||||||
> │ │ ├── worker-protocol/content-address.bin
|
|
||||||
> │ │ …
|
|
||||||
> │ …
|
|
||||||
> …
|
> …
|
||||||
|
> ├── src
|
||||||
|
> │ ├── libexpr
|
||||||
|
> │ │ ├── …
|
||||||
|
> │ │ ├── value
|
||||||
|
> │ │ │ ├── context.cc
|
||||||
|
> │ │ │ └── context.hh
|
||||||
|
> │ … …
|
||||||
|
> ├── tests
|
||||||
|
> │ …
|
||||||
|
> │ └── unit
|
||||||
|
> │ ├── libcmd
|
||||||
|
> │ │ └── args.cc
|
||||||
|
> │ ├── libexpr
|
||||||
|
> │ │ ├── …
|
||||||
|
> │ │ └── value
|
||||||
|
> │ │ ├── context.cc
|
||||||
|
> │ │ └── print.cc
|
||||||
|
> │ ├── libexpr-support
|
||||||
|
> │ │ └── tests
|
||||||
|
> │ │ ├── libexpr.hh
|
||||||
|
> │ │ └── value
|
||||||
|
> │ │ ├── context.cc
|
||||||
|
> │ │ └── context.hh
|
||||||
|
> │ ├── libstore
|
||||||
|
> │ │ ├── common-protocol.cc
|
||||||
|
> │ │ ├── data
|
||||||
|
> │ │ │ ├── libstore
|
||||||
|
> │ │ │ │ ├── common-protocol
|
||||||
|
> │ │ │ │ │ ├── content-address.bin
|
||||||
|
> │ │ │ │ │ ├── drv-output.bin
|
||||||
|
> … … … … … …
|
||||||
> ```
|
> ```
|
||||||
|
|
||||||
<!-- FIXME(Lix): this might get renamed to liblixexpr, etc? -->
|
The unit tests for each Lix library (`liblixexpr`, `liblixstore`, etc..) live inside a directory `src/${library_shortname}/tests` within the directory for the library (`src/${library_shortname}`).
|
||||||
|
|
||||||
The unit tests for each Lix library (`libnixexpr`, `libnixstore`, etc..) live inside a directory `src/${library_shortname}/tests` within the directory for the library (`src/${library_shortname}`).
|
The data is in `tests/unit/LIBNAME/data/LIBNAME`, with one subdir per library, with the same name as where the code goes.
|
||||||
|
For example, `liblixstore` code is in `src/libstore`, and its test data is in `tests/unit/libstore/data/libstore`.
|
||||||
The data is in `unit-test-data`, with one subdir per library, with the same name as where the code goes.
|
The path to the unit test data directory is passed to the unit test executable with the environment variable `_NIX_TEST_UNIT_DATA`.
|
||||||
For example, `libnixstore` code is in `src/libstore`, and its test data is in `unit-test-data/libstore`.
|
|
||||||
The path to the `unit-test-data` directory is passed to the unit test executable with the environment variable `_NIX_TEST_UNIT_DATA`.
|
|
||||||
|
|
||||||
> **Note**
|
|
||||||
> Due to the way googletest works, downstream unit test executables will actually include and re-run upstream library tests.
|
|
||||||
> Therefore it is important that the same value for `_NIX_TEST_UNIT_DATA` be used with the tests for each library.
|
|
||||||
> That is why we have the test data nested within a single `unit-test-data` directory.
|
|
||||||
|
|
||||||
### Running tests
|
### Running tests
|
||||||
|
|
||||||
You can run the whole testsuite with `just test` (see justfile for exact invocation of meson), and if you want to run just one test suite, use `just test --suite installcheck functional-init` where `installcheck` is the name of the test suite in this case and `functional-init` is the name of the test.
|
You can run the whole testsuite with `just test` (see justfile for exact invocation of meson), and if you want to run just one test suite, use `just test --suite installcheck functional-init` where `installcheck` is the name of the test suite in this case and `functional-init` is the name of the test.
|
||||||
|
|
||||||
To get a list of tests, use `meson test -C build --list`.
|
To get a list of tests, use `meson test -C build --list` (or `just test --list` for short).
|
||||||
|
|
||||||
For `installcheck` specifically, first run `just install` before running the test suite (this is due to meson limitations that don't let us put a dependency on installing before doing the test).
|
For `installcheck` specifically, first run `just install` before running the test suite (this is due to meson limitations that don't let us put a dependency on installing before doing the test).
|
||||||
|
|
||||||
|
@ -74,20 +81,27 @@ See [below](#characterization-testing-1) for a broader discussion of characteriz
|
||||||
Like with the functional characterization, `_NIX_TEST_ACCEPT=1` is also used.
|
Like with the functional characterization, `_NIX_TEST_ACCEPT=1` is also used.
|
||||||
For example:
|
For example:
|
||||||
```shell-session
|
```shell-session
|
||||||
$ _NIX_TEST_ACCEPT=1 make libstore-tests-exe_RUN
|
$ _NIX_TEST_ACCEPT=1 just test --suite check libstore-unit-tests
|
||||||
...
|
...
|
||||||
[ SKIPPED ] WorkerProtoTest.string_read
|
../tests/unit/libstore/common-protocol.cc:27: Skipped
|
||||||
[ SKIPPED ] WorkerProtoTest.string_write
|
Cannot read golden master because another test is also updating it
|
||||||
[ SKIPPED ] WorkerProtoTest.storePath_read
|
|
||||||
[ SKIPPED ] WorkerProtoTest.storePath_write
|
../tests/unit/libstore/common-protocol.cc:62: Skipped
|
||||||
|
Updating golden master
|
||||||
|
|
||||||
|
../tests/unit/libstore/common-protocol.cc:27: Skipped
|
||||||
|
Cannot read golden master because another test is also updating it
|
||||||
|
|
||||||
|
../tests/unit/libstore/common-protocol.cc:62: Skipped
|
||||||
|
Updating golden master
|
||||||
...
|
...
|
||||||
```
|
```
|
||||||
will regenerate the "golden master" expected result for the `libnixstore` characterization tests.
|
will regenerate the "golden master" expected result for the `liblixstore` characterization tests.
|
||||||
The characterization tests will mark themselves "skipped" since they regenerated the expected result instead of actually testing anything.
|
The characterization tests will mark themselves "skipped" since they regenerated the expected result instead of actually testing anything.
|
||||||
|
|
||||||
## Functional tests
|
## Functional tests
|
||||||
|
|
||||||
The functional tests reside under the `tests/functional` directory and are listed in `tests/functional/local.mk`.
|
The functional tests reside under the `tests/functional` directory and are listed in `tests/functional/meson.build`.
|
||||||
Each test is a bash script.
|
Each test is a bash script.
|
||||||
|
|
||||||
### Running the whole test suite
|
### Running the whole test suite
|
||||||
|
@ -185,7 +199,7 @@ edit it like so:
|
||||||
```
|
```
|
||||||
|
|
||||||
<div class="warning">
|
<div class="warning">
|
||||||
FIXME(meson): the command here may be incorrect for meson.
|
FIXME(meson): the command here is incorrect for meson and this whole functionality may need rebuilding.
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
Then, running the test with `./mk/debug-test.sh` will drop you into GDB once the script reaches that point:
|
Then, running the test with `./mk/debug-test.sh` will drop you into GDB once the script reaches that point:
|
||||||
|
@ -209,15 +223,11 @@ This technique is to include the exact output/behavior of a former version of Ni
|
||||||
|
|
||||||
For example, this technique is used for the language tests, to check both the printed final value if evaluation was successful, and any errors and warnings encountered.
|
For example, this technique is used for the language tests, to check both the printed final value if evaluation was successful, and any errors and warnings encountered.
|
||||||
|
|
||||||
<div class="warning">
|
|
||||||
FIXME(meson): this is incorrect for meson. `_NIX_TEST_ACCEPT=1` is still valid but the test invocation needs to change.
|
|
||||||
</div>
|
|
||||||
|
|
||||||
It is frequently useful to regenerate the expected output.
|
It is frequently useful to regenerate the expected output.
|
||||||
To do that, rerun the failed test(s) with `_NIX_TEST_ACCEPT=1`.
|
To do that, rerun the failed test(s) with `_NIX_TEST_ACCEPT=1`.
|
||||||
For example:
|
For example:
|
||||||
```bash
|
```bash
|
||||||
_NIX_TEST_ACCEPT=1 make tests/functional/lang.sh.test
|
_NIX_TEST_ACCEPT=1 just test --suite installcheck -v functional-lang
|
||||||
```
|
```
|
||||||
|
|
||||||
An interesting situation to document is the case when these tests are "overfitted".
|
An interesting situation to document is the case when these tests are "overfitted".
|
||||||
|
|
|
@ -229,7 +229,7 @@ This release has the following new features:
|
||||||
<https://cache.nixos.org/> or <ssh://machine>. The following store
|
<https://cache.nixos.org/> or <ssh://machine>. The following store
|
||||||
types are supported:
|
types are supported:
|
||||||
|
|
||||||
- `LocalStore` (stori URI `local` or an absolute path) and the
|
- `LocalStore` (store URI `local` or an absolute path) and the
|
||||||
misnamed `RemoteStore` (`daemon`) provide access to a local Nix
|
misnamed `RemoteStore` (`daemon`) provide access to a local Nix
|
||||||
store, the latter via the Nix daemon. You can use `auto` or the
|
store, the latter via the Nix daemon. You can use `auto` or the
|
||||||
empty string to auto-select a local or daemon store depending on
|
empty string to auto-select a local or daemon store depending on
|
||||||
|
|
Loading…
Reference in a new issue