functional2/lang: redesign of toml interface #851

New issue

Closed

opened 2025-06-08 21:59:21 +00:00 by commentatorforall · 1 comment

commentatorforall commented

2025-06-08 21:59:21 +00:00

Member

Issue description

Currently, when using a test.toml file to declare a lang test, all runners are always run on all in files.
This is inconvenient to unusable, when e.g. a singular in file requires flags, or some of the tests should run and others should fail.

Additionally, the current format of naming a test introduces some boilerplate that can be avoided.

Proposed solution

To avoid name duplication, base declaration should move from a dictionary to a list.
i.e. instead of declaring a test as [test-name] ... it would be declared using [[test]] ....

Providing a name should be optional. If not given the runner-name should be used.
the combination of the name and the in file suffix (default: ''; "in-my-test" -> '-my-test) must be unique. If it is not unique using the default values, the name attribute must be set.

A new optional flag should be added to mark a test as being a matrix test or not, defaulting to non-matrix.
(matrix tests being run on multiple files, while non matrix tests run on singular ones)

In addition to that, an optional in attribute should be added, stating on what nix file to run.
For single file tests, it must be a singular string, defaulting to in.nix when not provided.
For matrix tests, it must be a list of strings, defaulting to all present in-$suffix.nix files.

the flags and extra-files attributes should be kept as-is

cc: @piegames

## Issue description Currently, when using a `test.toml` file to declare a lang test, all runners are always run on all in files. This is inconvenient to unusable, when e.g. a singular in file requires flags, or some of the tests should run and others should fail. Additionally, the current format of naming a test introduces some boilerplate that can be avoided. ## Proposed solution To avoid name duplication, base declaration should move from a dictionary to a list. i.e. instead of declaring a test as `[test-name] ...` it would be declared using `[[test]] ...`. Providing a name should be optional. If not given the runner-name should be used. the combination of the name and the in file suffix (default: ''; "in-my-test" -> '-my-test) must be unique. If it is not unique using the default values, the name attribute must be set. A new optional flag should be added to mark a test as being a matrix test or not, defaulting to non-matrix. (matrix tests being run on multiple files, while non matrix tests run on singular ones) In addition to that, an optional `in` attribute should be added, stating on what nix file to run. For single file tests, it must be a singular string, defaulting to `in.nix` when not provided. For matrix tests, it must be a list of strings, defaulting to all present `in-$suffix.nix` files. the `flags` and `extra-files` attributes should be kept as-is cc: @piegames

commentatorforall added the

devx

testing

labels

2025-06-08 21:59:21 +00:00

commentatorforall self-assigned this

2025-06-08 21:59:21 +00:00

commentatorforall added this to the functional2 project

2025-06-08 21:59:21 +00:00

lix-bot commented

2025-06-09 17:49:33 +00:00

Member

This issue was mentioned on Gerrit on the following CLs:

commit message in cl/3297 ("tests/functional2/lang: improve toml design")

This issue was mentioned on Gerrit on the following CLs: * commit message in [cl/3297](https://gerrit.lix.systems/c/lix/+/3297) ("tests/functional2/lang: improve toml design")