jade commented

Owner

Currently there is not yet a way to dump e.g. Nix files into the test-dir in functional2. The ideal situation is that the test temp-dir has precisely declared contents so that a given test does not have any influence on the source tree.

Currently functional2 runs directly from the source tree and unlike the old shell tests, it does not need a copy of the test files in build/.

What would be nice is some kind of little DSL for declaring what is wanted in the test-dir and either copying from local paths or declaring the file contents inline, as the case may be.

Currently there is not yet a way to dump e.g. Nix files into the test-dir in functional2. The ideal situation is that the test temp-dir has precisely declared contents so that a given test does not have any influence on the source tree. Currently functional2 runs directly from the source tree and unlike the old shell tests, it does not need a copy of the test files in build/. What would be nice is some kind of little DSL for declaring what is wanted in the test-dir and either copying from local paths or declaring the file contents inline, as the case may be.

jade added the

testing

E/help wanted

labels

2024-12-10 19:55:07 +00:00

commentatorforall commented

2025-04-10 18:05:45 +00:00

Member

I'm interested in working on this issue, but am struggling to understand the target structure:
from what i understand it should look like the following:
test in python declares file contents or filepaths (of files in the test/functional2 directory)
a fixure(?) copies them into the temp dir, to ensure those and only those files are present
the test runs on files in the temp dir

is my understanding correct?

Additionally, how does are the tests supposed to look like in the end? similar to f, where one has a nix file and result declaration and pytest grabs those via parameters into a test case or will all nix stuff be written within python? (NOTE: i have only looked at and touched the functional/lang tests so far and don't really know how the other tests look like)

Outside of this scope, but on functional2 in general: why is print() used in some places instead of pytest functionality or a logger?

I'm interested in working on this issue, but am struggling to understand the target structure: from what i understand it should look like the following: test in python declares file contents or filepaths (of files in the test/functional2 directory) a fixure(?) copies them into the temp dir, to ensure those and only those files are present the test runs on files in the temp dir is my understanding correct? Additionally, how does are the tests supposed to look like in the end? similar to f, where one has a nix file and result declaration and pytest grabs those via parameters into a test case or will all nix stuff be written within python? (NOTE: i have only looked at and touched the functional/lang tests so far and don't really know how the other tests look like) Outside of this scope, but on functional2 in general: why is print() used in some places instead of pytest functionality or a logger?

jade commented

2025-04-10 19:44:02 +00:00

Author

Owner

So the current setup is that tests receive a tempdir from pytest (and there's a bug I need to file about that, deletion fails due to it not deleting things with readonly permissions). Tests run with a ~unknown working directory (I think it is the testsuite dir but idk), then explicitly start processes with a working directory of that temp dir.

The request here is something like copy_fixture from flake-compat, so https://git.lix.systems/lix-project/flake-compat/src/branch/main/tests/testlib/__init__.py#L14, but it should be able to either specify:

File from path
File with contents

And then enact that specification on the temp directory. Probably the api would look something like:

def test_foo(tmpdir: Path):
    test_files(tmpdir,
        Fixture('some-fixture-name-in-tests-functional2-fixtures'),
        File('filename', 'contents', executable=True),
        CopyFile('filename', new_filename='foo'),
        Template('filename.template', 'new_filename', {'blah': 'new_value'}),
    )

Yeah, the idea is that the test receives exactly the files given and can't see any ones it doesn't declare since they aren't copied into the tempdir.

Additionally, how does are the tests supposed to look like in the end? similar to f, where one has a nix file and result declaration and pytest grabs those via parameters into a test case or will all nix stuff be written within python? (NOTE: i have only looked at and touched the functional/lang tests so far and don't really know how the other tests look like)

I am not sure; for the general case, I think that we are doing non-uniform things with the nix CLI, so it's going to look pretty similar to the original functional test suite. For a replacement for the evaluation tests where there's a pile of nix code that is evaluated in a uniform way, that one will have some kind of looping or pytest parameterization.

There should still be separate nix files for non trivial things, so that we get good LSP help, etc, IMO.

As for why print() is used, I thought it was captured by pytest; basically the idea is that it's caught by the testsuite and if a test fails, its output will include that information. Purely laziness that it's not a logger or something else; it can be changed if you think it is a good idea.

So the current setup is that tests receive a tempdir from pytest (and there's a bug I need to file about that, deletion fails due to it not deleting things with readonly permissions). Tests run with a ~unknown working directory (I think it is the testsuite dir but idk), then explicitly start processes with a working directory of that temp dir. The request here is something like copy_fixture from flake-compat, so <https://git.lix.systems/lix-project/flake-compat/src/branch/main/tests/testlib/__init__.py#L14>, but it should be able to either specify: - File from path - File with contents And then enact that specification on the temp directory. Probably the api would look something like: ```python def test_foo(tmpdir: Path): test_files(tmpdir, Fixture('some-fixture-name-in-tests-functional2-fixtures'), File('filename', 'contents', executable=True), CopyFile('filename', new_filename='foo'), Template('filename.template', 'new_filename', {'blah': 'new_value'}), ) ``` Yeah, the idea is that the test receives exactly the files given and can't see any ones it doesn't declare since they aren't copied into the tempdir. > Additionally, how does are the tests supposed to look like in the end? similar to f, where one has a nix file and result declaration and pytest grabs those via parameters into a test case or will all nix stuff be written within python? (NOTE: i have only looked at and touched the functional/lang tests so far and don't really know how the other tests look like) I am not sure; for the *general* case, I think that we are doing non-uniform things with the nix CLI, so it's going to look pretty similar to the original functional test suite. For a replacement for the evaluation tests where there's a pile of nix code that is evaluated in a uniform way, that one will have some kind of looping or pytest parameterization. There should still be separate nix files for non trivial things, so that we get good LSP help, etc, IMO. As for why print() is used, I thought it was captured by pytest; basically the idea is that it's caught by the testsuite and if a test fails, its output will include that information. Purely laziness that it's not a logger or something else; it can be changed if you think it is a good idea.

piegames added this to the functional2 project

2025-04-17 13:21:42 +00:00

helle commented

2025-04-19 21:09:00 +00:00

Member

While we just implemented a way to generate files for another test in functional2/ we realised that fixing this should be a reasonably high priority (combined with some other outstanding bugs that make functional2/ not quite ready to port everything over to)

We could propose a syntax to define the file based fixtures and probably can also implement this reasonably quickly, but if commentatorforall wants to take this on that is fine as well, in which case can we please assign it to you?

Also yes, we probably want to use a logger, so that printed output from the test suite gets split from printed output from lix when the final error is produced (but is also still available interleaved we believe earlier in the output file)

While we just implemented a way to generate files for another test in `functional2/` we realised that fixing this should be a reasonably high priority (combined with some other outstanding bugs that make `functional2/` not quite ready to port everything over to) We could propose a syntax to define the file based fixtures and probably can also implement this reasonably quickly, but if commentatorforall wants to take this on that is fine as well, in which case can we please assign it to you? Also yes, we probably want to use a logger, so that printed output from the test suite gets split from printed output from lix when the final error is produced (but is also still available interleaved we believe earlier in the output file)

commentatorforall self-assigned this

2025-04-20 08:35:07 +00:00

lix-bot commented

2025-04-22 07:47:22 +00:00

Member

This issue was mentioned on Gerrit on the following CLs:

commit message in cl/3047 ("Added functions for copying files into tempdir for declarative testing without side effects")

This issue was mentioned on Gerrit on the following CLs: * commit message in [cl/3047](https://gerrit.lix.systems/c/lix/+/3047) ("Added functions for copying files into tempdir for declarative testing without side effects")

helle commented

2025-04-22 11:41:09 +00:00

Member

I may have an alternative implementation we frustrated coded in the last few days (before we noticed you had self-assigned it, sorry), it would have one of the following syntaxes, this may be more powerful then the proposed one by jade.

Code implementing these exists, though does need some polishing before publishing and some discussion on the finalised functionality that we want from this.

Current presumption is that you want CopyFile, CopyDirectory and Template relative to where the test is with Template using format templating, File to create a file from a Python string and Symlink to create a symlink within the tree, then there is Fixture, which uses a shared Fixture from the functional2/fixtures directory (is this a file or directory or either and should this just be an option on the other types and is templating applied needs to be discussed) and lastly, Directory, which is a subdirectory, which itself can contain the same things as the main one

The types once the class is instantiated and the instance of the class itself inherits from Path so they can be treated as filenames, etc, etc

# Baseline variant with late instantiation, only uses some basic metaclass tricks to read out
# annotations, relies on dicts being ordered (guaranteed in modern Python) for instantiation
# but that is about it
#
# Limitations include that filenames cannot include dots on the left hand side, hence filename and
# extension attributes existing on the classes to allow you to specify alternative ones
@pytest.fixture
def file_fixture_simplest(file_fixtures_config: FileFixturesConfig):
    # Create the SubDirectory's class ahead of time and include it later
    class SubDirectory(FileFixturesConfig):
        file_in_subdir: File() = "hello world"

    class MyFileFixture(FileFixtures):
        fixturename: Fixture('some-fixture-name-in-tests-functional2-fixtures')
        # A directory needs to be a seperate class (of the same type, FileFixtures) and included
        # with the annotation Directory(), this so that it will be in the ordered annotations and
        # hence will be ordered in the actual instatiation
        sub_directory: Directory() = SubDirectory
        # Cannot represent the proper filename on the left handside, so pass it as a parameter
        foo: CopyFile('random_file', filename="foo.txt.nix", mode=0o755)
        # Same, but we shorthand the common case of just wanting to add an extension
        filename: File(executable=True, extension="txt") = "contents"
        new_filename: Template('filename.template') = {'blah': 'new_value'}
        sub_copy: CopyDirectory("dir/")
        symlink: Symlink("foo.txt.nix")

    return MyFileFixture(config=file_fixtures_config)

Alternatively the following two extensions exist, but they rely on more Python code magic:

# Variant with subdirectories being included directly inline
#
# This hooks into the creation of the locals of the class to track when they are assigned and
# make sure that they all are represented in a list to be iterated over when instantiating them
# as the previous version relied on annotations for that, which the sub_directory would not have
@pytest.fixture
def file_fixture_with_subdir_inline(file_fixtures_config: FileFixturesConfig):
    class MyFileFixture(FileFixtures):
        fixturename: Fixture('some-fixture-name-in-tests-functional2-fixtures')
        # the sub_directory is now inline and is included as an instantiated attribute in the
        # instantiated object of this still, it will also take filename= and other per FileFixture
        # config markers
        class sub_directory(Directory, filename="sub_directory"):
            file_in_subdir: File() = "hello world"
        # Cannot represent the proper filename on the left handside, so pass it as a parameter
        foo: CopyFile('random_file', filename="foo.txt.nix", mode=0o755)
        # Same, but we shorthand the common case of just wanting to add an extension
        filename: File(executable=True, extension="txt") = "contents"
        new_filename: Template('filename.template') = {'blah': 'new_value'}
        sub_copy: CopyDirectory("dir/")
        symlink: Symlink("foo.txt.nix")

    return MyFileFixture(config=file_fixtures_config)

As mentioned in the comments, this does use the AST, so may be undesirable for like, Python stability, otoh, as long as cpython exists and is the one we use and the underlying modules do, this will likely continue to work

# Moderately cursed Python that tracks assignments to local variables, parses (AST) and then
# deparses annotations for files with extensions and creates dynamic temporary objects on the fly
# to fake files with extensions being there
#
# It does still allow for filename= and extension= attributes for filenames still not representable
# as python attribute names even if split at dots
@pytest.fixture
def file_fixture_with_file_extensions(file_fixtures_config: FileFixturesConfig):
    class MyFileFixture(FileFixtures):
        fixturename: Fixture('some-fixture-name-in-tests-functional2-fixtures')
        # Still can do this as with the previous example
        class sub_directory(Directory):
            file_in_subdir: File() = "hello world"
        # Can directly represent
        foo.txt.nix: CopyFile('random_file', mode=0o755)
        # As well as
        filename.txt: File(executable=True) = "contents"
        new_filename: Template('filename.template') = {'blah': 'new_value'}
        sub_copy: CopyDirectory("dir/")
        symlink: Symlink("foo.txt.nix")

    return MyFileFixture(config=file_fixtures_config)

I may have an alternative implementation we frustrated coded in the last few days (before we noticed you had self-assigned it, sorry), it would have one of the following syntaxes, this may be more powerful then the proposed one by jade. Code implementing these exists, though does need some polishing before publishing and some discussion on the finalised functionality that we want from this. Current presumption is that you want `CopyFile`, `CopyDirectory` and `Template` relative to where the test is with `Template` using format templating, `File` to create a file from a Python string and Symlink to create a symlink within the tree, then there is `Fixture`, which uses a shared Fixture from the `functional2/fixtures directory` (is this a file or directory or either and should this just be an option on the other types and is templating applied needs to be discussed) and lastly, `Directory`, which is a subdirectory, which itself can contain the same things as the main one The types once the class is instantiated and the instance of the class itself inherits from `Path` so they can be treated as filenames, etc, etc ```python # Baseline variant with late instantiation, only uses some basic metaclass tricks to read out # annotations, relies on dicts being ordered (guaranteed in modern Python) for instantiation # but that is about it # # Limitations include that filenames cannot include dots on the left hand side, hence filename and # extension attributes existing on the classes to allow you to specify alternative ones @pytest.fixture def file_fixture_simplest(file_fixtures_config: FileFixturesConfig): # Create the SubDirectory's class ahead of time and include it later class SubDirectory(FileFixturesConfig): file_in_subdir: File() = "hello world" class MyFileFixture(FileFixtures): fixturename: Fixture('some-fixture-name-in-tests-functional2-fixtures') # A directory needs to be a seperate class (of the same type, FileFixtures) and included # with the annotation Directory(), this so that it will be in the ordered annotations and # hence will be ordered in the actual instatiation sub_directory: Directory() = SubDirectory # Cannot represent the proper filename on the left handside, so pass it as a parameter foo: CopyFile('random_file', filename="foo.txt.nix", mode=0o755) # Same, but we shorthand the common case of just wanting to add an extension filename: File(executable=True, extension="txt") = "contents" new_filename: Template('filename.template') = {'blah': 'new_value'} sub_copy: CopyDirectory("dir/") symlink: Symlink("foo.txt.nix") return MyFileFixture(config=file_fixtures_config) ``` Alternatively the following two extensions exist, but they rely on more Python code magic: ```python # Variant with subdirectories being included directly inline # # This hooks into the creation of the locals of the class to track when they are assigned and # make sure that they all are represented in a list to be iterated over when instantiating them # as the previous version relied on annotations for that, which the sub_directory would not have @pytest.fixture def file_fixture_with_subdir_inline(file_fixtures_config: FileFixturesConfig): class MyFileFixture(FileFixtures): fixturename: Fixture('some-fixture-name-in-tests-functional2-fixtures') # the sub_directory is now inline and is included as an instantiated attribute in the # instantiated object of this still, it will also take filename= and other per FileFixture # config markers class sub_directory(Directory, filename="sub_directory"): file_in_subdir: File() = "hello world" # Cannot represent the proper filename on the left handside, so pass it as a parameter foo: CopyFile('random_file', filename="foo.txt.nix", mode=0o755) # Same, but we shorthand the common case of just wanting to add an extension filename: File(executable=True, extension="txt") = "contents" new_filename: Template('filename.template') = {'blah': 'new_value'} sub_copy: CopyDirectory("dir/") symlink: Symlink("foo.txt.nix") return MyFileFixture(config=file_fixtures_config) ``` As mentioned in the comments, this does use the AST, so may be undesirable for like, Python stability, otoh, as long as cpython exists and is the one we use and the underlying modules do, this will likely continue to work ```python # Moderately cursed Python that tracks assignments to local variables, parses (AST) and then # deparses annotations for files with extensions and creates dynamic temporary objects on the fly # to fake files with extensions being there # # It does still allow for filename= and extension= attributes for filenames still not representable # as python attribute names even if split at dots @pytest.fixture def file_fixture_with_file_extensions(file_fixtures_config: FileFixturesConfig): class MyFileFixture(FileFixtures): fixturename: Fixture('some-fixture-name-in-tests-functional2-fixtures') # Still can do this as with the previous example class sub_directory(Directory): file_in_subdir: File() = "hello world" # Can directly represent foo.txt.nix: CopyFile('random_file', mode=0o755) # As well as filename.txt: File(executable=True) = "contents" new_filename: Template('filename.template') = {'blah': 'new_value'} sub_copy: CopyDirectory("dir/") symlink: Symlink("foo.txt.nix") return MyFileFixture(config=file_fixtures_config) ```

commentatorforall commented

2025-04-23 08:31:49 +00:00

Member

Currently there are three different suggested syntax styles:

def test_some_stuff():
    test_files(tmpdir,
        Fixture('some-fixture-name-in-tests-functional2-fixtures'),
        File('filename', 'contents', executable=True),
        CopyFile('filename', new_filename='foo'),
        Template('filename.template', 'new_filename', {'blah': 'new_value'}),
    )
    ...

by jade: calling a function, providing the target directory and the files (current implementation of the MR)
Pros: could be parametrized by taking the files as parameters
Cons: not a fixure itself, maybe not transpartent on how to use

fixture style by helle: having an own fixure for test, providing filename: filetype/content information about files as attributes of a class (subdirs being inner classes)
I haven't quite understand on how a test using this would look like, please clarify/provide an example
Pros: (due to lack of understanding not known)
Cons: representation of filenames including extension is difficult/not possible, each test needs to provide its own fixture(?)

{ 'a': { 'b': File('c', mode=0400) } }

usage:

@pytest.mark.parametrize('with_files', [{'a.txt': 'content'}, {'second_test_run.txt': 'different content'}], indirect=['with_files'])
def test_some_stuff(with_files):
    ...

by Eldrich Horrors: Dictionary style as parameters for a fixture
with valuetype Dict being directories and valuetype other being file content/content-provider
Pros: only a single fixture as a provider, IMO structure easily visible, can be parametrerizable for reexecution with different files
Cons: unclear how to copy entire directories (for a functional/lang test usecase of having a bunch of files)

Things basically agreed upon: Filepaths should be relative and not from CWD

Is there a way to reach a consensus on how the api should look like?

Currently there are three different suggested syntax styles: ```py def test_some_stuff(): test_files(tmpdir, Fixture('some-fixture-name-in-tests-functional2-fixtures'), File('filename', 'contents', executable=True), CopyFile('filename', new_filename='foo'), Template('filename.template', 'new_filename', {'blah': 'new_value'}), ) ... ``` by jade: calling a function, providing the target directory and the files (current implementation of the MR) Pros: could be parametrized by taking the files as parameters Cons: not a fixure itself, maybe not transpartent on how to use --- fixture style by helle: having an own fixure for test, providing `filename: filetype/content` information about files as attributes of a class (subdirs being inner classes) I haven't quite understand on how a test using this would look like, please clarify/provide an example Pros: (due to lack of understanding not known) Cons: representation of filenames including extension is difficult/not possible, each test needs to provide its own fixture(?) --- ```py { 'a': { 'b': File('c', mode=0400) } } ``` usage: ```py @pytest.mark.parametrize('with_files', [{'a.txt': 'content'}, {'second_test_run.txt': 'different content'}], indirect=['with_files']) def test_some_stuff(with_files): ... ``` by Eldrich Horrors: Dictionary style as parameters for a fixture with valuetype Dict being directories and valuetype other being file content/content-provider Pros: only a single fixture as a provider, IMO structure easily visible, can be parametrerizable for reexecution with different files Cons: unclear how to copy entire directories (for a functional/lang test usecase of having a bunch of files) --- Things basically agreed upon: Filepaths should be relative and not from CWD Is there a way to reach a consensus on how the api should look like?

pennae commented

2025-04-23 10:33:15 +00:00

Owner

the function jade asked for can (and should) exist either way. iwrc pytest fixture are little more than context managers with some of dynamic language bullshit wrapped around them to make instantiation work, so chances are a fixture function can just be reused to dynamically add files. if we do this the dictionary parametrization approach is probably easier to use in practice. helle's approach is more explicit about what's happening, but to us it also feels like the additional specification is mostly noise—our set of operations here is small enough, and a dict structure sufficiently reminiscent of directory tree rendering, that a simpler fixture would be totally adequate. (if that's not true in the long run we can still change it later too!)

👍 2

helle commented

2025-04-26 14:47:21 +00:00

Member

sorry for missing the e-mail for replies to this, feel free to in the future poke me on Matrix, we don't have the best of work flow at the moment

and sorry for writing this long text, there is a tl;dr, but we felt like we should give a full response even with the conclusion we make, lol

some of the questions and ideas:

my proposal is used as followed:

based on one of the above syntaxes

def test_foo(file_fixture: FileFixtures):
    # Implements all the behaviour of a (Posix)Path
    str(file_fixture) == "tmp_path"
    contents = (file_fixture / "filename").read_text 
    # etc

    # Each attribute itself is also a (Posix)Path
    with file_fixture.foo.open() as f:
        pass
    # and any other operations on it work

If you only need it for one test, you can just define it inline for that test

def test_foo(file_fixtures_config: FileFixturesConfig):
    class TinyFixture(FileFixturesConfig):
        foo = File(executable=True, extension="txt") = "contents"
    fixture = TinyFixture(file_fixtures_config)

    str(fixture.foo) == "<path>/foo.txt" # etc

fixture or inline

jade's proposal also very easily works as a fixture, so that is not a reason
against it, we genuinely don't mind the syntax that much

@pytest.fixture
def file_fixture(tmpdir: Path):
    test_files(tmpdir,
            File('filename', 'contents', executable=True)
    )
    return tmpdir

file extensions and attributes

the reason we went with attributes and hence accept that in the simpelest (and
least fragile) implementation you can't directly have filenames with
extensions on the left hand side is that one of the things we have in the past
often had to do was refer to the files that have been created inside the python
code

so like being able to pass in the full file path as an argument to commands
would be as simple as str(file_fixture.foo), which also allows completion of
the attributes in IDEs and such as opposed to referring to tmp_path / "what_was_that_name"

the fact that we need a filename= and shorthand extension= (or maybe ext=)
parameter to get the completion on the part that "mattered" felt acceptable

related to this, with jade's syntax, a handle to the file (in this case the filename)
is not the most obvious thing to read at least to us, when you quickly glance at the
creation of test files, pennae's syntax does at least have this property of obvious
readability when properly formatted

addressing other things then contents and setting things like modes with Eldritch Horror's solution

you can pass in objects or the result of a function into the dictionary and
handle those in with_files to instantiate things from existing files, or copy
directories and also to apply special options like setting modes, similar to
what my version does

this version also does actually allow you to share the fixture between tests
(both with or without reinstantiating it by setting scope correctly), by making
it itself a fixture and returning the with_files

tl;dr

possibly leaning towards jade's syntax because of the small scope we honestly
deal with if we are diligent about formatting, possibly with test_dir's return
value both being a Path itself and chainable

one big benefit of such a syntax is honestly the ability to just dynamically
add files within the code, mine is more rigid in that and would need yet
further bits to do that (and that syntax may end up less obvious, like an add
or just outright attribute setting being handled)

sources

one partially open question is where the source for these files and templates
lives

we would suggest for simple files not shared, to have them within the same
directory as the test they belong with (we need to be okay with deeper
subdirectories in that case) and a fixtures/ directory under the top level
test directory for ones that will be shared with many tests potentially,
but just having a global and local fixtures/ directory is also an option

we would at least make sure that the usual case has the test files pretty
close to the tests, as otherwise the code organisation is not logical to
anyone new to the code base

footnote

to get hold of the correct paths, there may be a better option then asking
meson for it, by if you include the fixture: request: pytest.FixtureRequest
you can obtain request.path.parent for the path the test is in and
request.session.path for the root of the pytests, this is more robust then
relying on the cwd being set correctly

the downside is that it needs to be handed to test_files somewhere

(this is what the not shown fixture file_fixtures_config does in my
implementation, it grabs those two paths plus the tmp_path to work on)

def file_fixtures_config(tmp_path: Path, request: pytest.FixtureRequest) -> filefixture.FileFixturesConfig:
    return filefixture.FileFixturesConfig(request.path.parent,
                                          request.session.path,
                                          tmp_path)

sorry for missing the e-mail for replies to this, feel free to in the future poke me on Matrix, we don't have the best of work flow at the moment and sorry for writing this long text, there is a tl;dr, but we felt like we should give a full response even with the conclusion we make, lol # some of the questions and ideas: ## my proposal is used as followed: based on one of the above syntaxes ```python def test_foo(file_fixture: FileFixtures): # Implements all the behaviour of a (Posix)Path str(file_fixture) == "tmp_path" contents = (file_fixture / "filename").read_text # etc # Each attribute itself is also a (Posix)Path with file_fixture.foo.open() as f: pass # and any other operations on it work ``` If you only need it for one test, you can just define it inline for that test ```python def test_foo(file_fixtures_config: FileFixturesConfig): class TinyFixture(FileFixturesConfig): foo = File(executable=True, extension="txt") = "contents" fixture = TinyFixture(file_fixtures_config) str(fixture.foo) == "<path>/foo.txt" # etc ``` ## fixture or inline jade's proposal also very easily works as a fixture, so that is not a reason against it, we genuinely don't mind the syntax that much ```python @pytest.fixture def file_fixture(tmpdir: Path): test_files(tmpdir, File('filename', 'contents', executable=True) ) return tmpdir ``` ## file extensions and attributes the reason we went with attributes and hence accept that in the simpelest (and least fragile) implementation you can't directly have filenames with extensions on the left hand side is that one of the things we have in the past often had to do was refer to the files that have been created inside the python code so like being able to pass in the full file path as an argument to commands would be as simple as `str(file_fixture.foo)`, which also allows completion of the attributes in IDEs and such as opposed to referring to `tmp_path / "what_was_that_name"` the fact that we need a `filename=` and shorthand `extension=` (or maybe `ext=`) parameter to get the completion on the part that "mattered" felt acceptable related to this, with jade's syntax, a handle to the file (in this case the filename) is not the most obvious thing to read at least to us, when you quickly glance at the creation of test files, pennae's syntax does at least have this property of obvious readability when properly formatted ## addressing other things then contents and setting things like modes with Eldritch Horror's solution you can pass in objects or the result of a function into the dictionary and handle those in with_files to instantiate things from existing files, or copy directories and also to apply special options like setting modes, similar to what my version does this version also does actually allow you to share the fixture between tests (both with or without reinstantiating it by setting scope correctly), by making it itself a fixture and returning the with_files ## tl;dr possibly leaning towards jade's syntax because of the small scope we honestly deal with if we are diligent about formatting, possibly with test_dir's return value both being a Path itself and chainable one big benefit of such a syntax is honestly the ability to just dynamically add files within the code, mine is more rigid in that and would need yet further bits to do that (and that syntax may end up less obvious, like an add or just outright attribute setting being handled) ## sources one partially open question is where the source for these files and templates lives we would suggest for simple files not shared, to have them within the same directory as the test they belong with (we need to be okay with deeper subdirectories in that case) and a fixtures/ directory under the top level test directory for ones that will be shared with many tests potentially, but just having a global and local fixtures/ directory is also an option we would at least make sure that the usual case has the test files pretty close to the tests, as otherwise the code organisation is not logical to anyone new to the code base ## footnote to get hold of the correct paths, there may be a better option then asking meson for it, by if you include the fixture: `request: pytest.FixtureRequest` you can obtain `request.path.parent` for the path the test is in and `request.session.path` for the root of the pytests, this is more robust then relying on the cwd being set correctly the downside is that it needs to be handed to `test_files` somewhere (this is what the not shown fixture `file_fixtures_config` does in my implementation, it grabs those two paths plus the tmp_path to work on) ```python def file_fixtures_config(tmp_path: Path, request: pytest.FixtureRequest) -> filefixture.FileFixturesConfig: return filefixture.FileFixturesConfig(request.path.parent, request.session.path, tmp_path) ```

helle commented

2025-04-26 15:03:13 +00:00

Member

oh, a note on meta testing (testing of these fixtures)

we should make sure the fixtures work with varying scopes, ie, both per test scoping, but also scope="module" which is quite a common thing to need

oh, a note on meta testing (testing of these fixtures) we should make sure the fixtures work with varying scopes, ie, both per test scoping, but also scope="module" which is quite a common thing to need

helle commented

2025-04-26 15:11:31 +00:00

Member

and an open question (also mentioned on gerrit)

in jade's syntax, should we allow someone to do:

test_files(tmpdir,
    File('subdir/filename', 'contents')
)

as currently there is no clean way to do subdirectories of any shape other than by copying a full subdir tree (or does anyone know a better syntax)

and an open question (also mentioned on gerrit) in jade's syntax, should we allow someone to do: ```python test_files(tmpdir, File('subdir/filename', 'contents') ) ``` as currently there is no clean way to do subdirectories of any shape other than by copying a full subdir tree (or does anyone know a better syntax)

jade commented

2025-04-26 17:29:16 +00:00

Author

Owner

Could we have Dir('name', *contents) and then forbid multi component names (or only allow them in Dir, to avoid having multiple places writing into a directory)?

Could we have `Dir('name', *contents)` and then forbid multi component names (or only allow them in Dir, to avoid having multiple places writing into a directory)?

commentatorforall commented

2025-04-26 17:31:23 +00:00

Member

i would just move over to @pennae s approach of using a dictionary as the content definition. That way, sub directories are just handled by using a dictionary as the value

commentatorforall commented

2025-04-26 17:32:59 +00:00

Member

i.e.

with_files(tmpdir,
    {
        "file.txt": File("content"),
        "sub_directory": {
            "other_file.nix": Copyfile("within/assets/file.nix"),
            },
})

i.e. ```py with_files(tmpdir, { "file.txt": File("content"), "sub_directory": { "other_file.nix": Copyfile("within/assets/file.nix"), }, })

kfearsoff commented

2025-04-26 19:14:19 +00:00

Member

I don't have much of a stake, but using dictionary to represent a filesystem tree sounds very nice. It's basically like expressing the directory tree as a JSON.

helle commented

2025-04-27 10:42:14 +00:00

Member

yes, the dictionary approach makes the left hand side of what file it will belong to very obvious, that solves the same issue of readability we were addressing, while the rhs being a class or function like class, should make the operation done to create it very clear, I approve of this idea :3

ideally it will be usable in both parameters for a fixture or inline with a function, but if we can start with inline as a fixture we can contribute the other style if wanted later on

I think we all seem to agree on the shared parameter set of executable=True and mode=0onnn, are there others needed there?

then that leaves afaik the exact list of operations needed and where things are sourced from to be solidly defined?

yes, the dictionary approach makes the left hand side of what file it will belong to very obvious, that solves the same issue of readability we were addressing, while the rhs being a class or function like class, should make the operation done to create it very clear, I approve of this idea :3 ideally it will be usable in both parameters for a fixture or inline with a function, but if we can start with inline as a fixture we can contribute the other style if wanted later on I think we all seem to agree on the shared parameter set of `executable=True` and `mode=0onnn`, are there others needed there? then that leaves afaik the exact list of operations needed and where things are sourced from to be solidly defined?

commentatorforall commented

2025-04-27 11:11:20 +00:00

Member

As for where files are sourced, i would say we should stick with the current implementation/suggestion: in an assets folder next to the corresponding test files, though id agree that paths should be relative (to the test file) and not absolute from tests/functional2

commentatorforall commented

2025-04-27 12:43:27 +00:00

Member

As for operations, i'd say

File("content in here") # content defined inline, optionally add a mode parameter
CopyFile("path/to/assets/file") # copy a file from the test directory
Template("path/to/template/file", {"key": "values to be inserted"}) # Initialization of a template; should we error if not all variables are being set?
CopyTree("path/to/asset/tree") # I am unsure if we should have this too, if we decide to implement something like functional lang where we just test a whole directory, alternatively one could pass the file list as a parametrization to only have the neccesary files for the one test and even have the different files show up as their own test entry, which might be preferred

As for operations, i'd say ```py File("content in here") # content defined inline, optionally add a mode parameter CopyFile("path/to/assets/file") # copy a file from the test directory Template("path/to/template/file", {"key": "values to be inserted"}) # Initialization of a template; should we error if not all variables are being set? CopyTree("path/to/asset/tree") # I am unsure if we should have this too, if we decide to implement something like functional lang where we just test a whole directory, alternatively one could pass the file list as a parametrization to only have the neccesary files for the one test and even have the different files show up as their own test entry, which might be preferred

helle commented

2025-04-27 13:31:47 +00:00

Member

would probably for completeness still include a Symlink operation, given that there are potentially some tests where symlink vs actual file are relevant from having skimmed the old tests

and yes, Template's should demand full initialisation, which iirc the Python format language does by default ("{}" style stuff)

would probably for completeness still include a Symlink operation, given that there are potentially some tests where symlink vs actual file are relevant from having skimmed the old tests and yes, Template's should demand full initialisation, which iirc the Python format language does by default ("{}" style stuff)

commentatorforall commented

2025-04-27 13:41:33 +00:00

Member

great idea, text.format works here. It does not throw an error, when too many values were given / not all values were used. should we ignore that too, or try to throw an error ourselves in that case?
e.g. template: "this is a text with one {value}" and trying to fill in {"value": 123, "some_other_thing_which_isnt_used": "aaaaaa"} doesn't error or warn or anything

great idea, `text.format` works here. It does not throw an error, when too many values were given / not all values were used. should we ignore that too, or try to throw an error ourselves in that case? e.g. template: `"this is a text with one {value}"` and trying to fill in `{"value": 123, "some_other_thing_which_isnt_used": "aaaaaa"}` doesn't error or warn or anything

helle commented

2025-04-27 14:32:52 +00:00

Member

we just quickly wrote an implementation for that, feel free to use or modify as needed

formatter.txt

3 KiB

commentatorforall commented

2025-04-27 17:35:10 +00:00

Member

about file scoping: the scope of a fixture needs to be defined at the fixture itself (using @pytest.fixture(scope="moduel") etc)
hence user defined scoping isn't possible currently.
if requested, one could do with_files_scope, functions for each scope and then the user can use the adequate function, but unsure if i would like that approach

about file scoping: the scope of a fixture needs to be defined at the fixture itself (using `@pytest.fixture(scope="moduel")` etc) hence user defined scoping isn't possible currently. if requested, one could do `with_files_scope`, functions for each scope and then the user can use the adequate function, but unsure if i would like that approach

jade commented

2025-04-30 04:52:37 +00:00

Author

Owner

text.format using single {} is really annoying if you're templating nix scripts which is one of the main things we'd do with it. you might want something like substituteAll in nixpkgs instead, where you replace these @FOO@ things and take all the logic out of the template. mostly we don't care too much about escaping either because it's mostly substituting paths.

text.format using single {} is really annoying if you're templating nix scripts which is one of the main things we'd do with it. you might want something like substituteAll in nixpkgs instead, where you replace these `@FOO@` things and take all the logic out of the template. mostly we don't care too much about escaping either because it's mostly substituting paths.