lix-project/lix

Fork 54

functional2: add expect-test support #595

New issue

Closed

opened 2024-12-05 22:19:05 +00:00 by jade · 5 comments

jade commented

2024-12-05 22:19:05 +00:00

Owner

This will require some adjustment of the expect-test library for python to support parameterization and maybe add slightly better pytest support, since the current way it works is kind of ugly.

Also of note, there is a pytest-expecttest thing; please do not use it; it captures stdout/stderr for the expectations (why?!), whereas the unittest one it was taken from has a much more reasonable design.

The point of this is that we can have really nice inline test-cases for one-liners that snapshot the exact output, sort of like repl characterization.

(n.b. hmm, repl characterization could have totally been a python thing. maybe it will become one later. maybe not. it is not a big deal regardless)

Upstream bug: https://github.com/pytorch/expecttest/issues/25

This will require some adjustment of the expect-test library for python to support parameterization and maybe add slightly better pytest support, since the current way it works is kind of ugly. Also of note, there is a pytest-expecttest thing; please do not use it; it captures stdout/stderr for the expectations (why?!), whereas the unittest one it was taken from has a much more reasonable design. The point of this is that we can have really nice inline test-cases for one-liners that snapshot the exact output, sort of like repl characterization. (n.b. hmm, repl characterization could have totally been a python thing. maybe it will become one later. maybe not. it is not a big deal regardless) Upstream bug: https://github.com/pytorch/expecttest/issues/25

jade added the

testing

devx

labels

2024-12-05 22:19:05 +00:00

lix-bot commented

2024-12-05 22:39:27 +00:00

Member

This issue was mentioned on Gerrit on the following CLs:

commit message in cl/2279 ("tests/functional2: add terminal code eater")
commit message in cl/3145 ("tests/functional2: add snapshot fixture")

This issue was mentioned on Gerrit on the following CLs: * commit message in [cl/2279](https://gerrit.lix.systems/c/lix/+/2279) ("tests/functional2: add terminal code eater") * commit message in [cl/3145](https://gerrit.lix.systems/c/lix/+/3145) ("tests/functional2: add snapshot fixture")

jade referenced this issue from a commit

2024-12-05 23:33:54 +00:00

tests/functional2: add terminal code eater

lix-project referenced this issue from a commit

2024-12-11 02:28:33 +00:00

tests/functional2: add terminal code eater

jade self-assigned this

2025-01-22 18:13:50 +00:00

piegames added this to the functional2 project

2025-04-17 13:24:42 +00:00

commentatorforall commented

2025-05-10 17:59:08 +00:00

Member

ok, reading through different packages of snapshot/exepct testing, i have concluded the following opinion(s):

i am against the usage of print() and then checking against stdout, 1. because its kinda ugly and will likely interfere with logging 2. it'll most likely capture the entire stdout 3. might complicate things about running nix things
i personally prefer a assert snapshot == some_file_or_value over snapshot.assert_match(some_file_or_value)
the path to the snapshot files should be adjustable for seemless integration

ok, reading through different packages of snapshot/exepct testing, i have concluded the following opinion(s): - i am against the usage of print() and then checking against stdout, 1. because its kinda ugly and will likely interfere with logging 2. it'll most likely capture the entire stdout 3. might complicate things about running nix things - i personally prefer a `assert snapshot == some_file_or_value` over `snapshot.assert_match(some_file_or_value)` - the path to the snapshot files should be adjustable for seemless integration

commentatorforall commented

2025-05-10 18:14:44 +00:00

Member

looking at more existing plugins and their repos, most of them are either not maintiained or barely (latest release i saw of any one was >=1year old)
and then they are either missing features or are just not configurable in the slightest
i'd almost argue to hack something ourselves with the basic functionallity we need

looking at more existing plugins and their repos, most of them are either not maintiained or barely (latest release i saw of *any* one was >=1year old) and then they are either missing features or are just not configurable in the slightest i'd almost argue to hack something ourselves with the basic functionallity we need

piegames referenced this issue

2025-05-10 18:15:28 +00:00

functional2/lang infrastructure #825

jade commented

2025-05-10 19:32:53 +00:00

Author

Owner

fyi https://github.com/pytorch/expecttest/issues/25 is now fixed and that library looks generally very good now

piegames commented

2025-05-20 12:28:31 +00:00

Member

That's why I'd like to move forward with that implementation for now to unblock at least functional/lang, and we can always add a more fully-featured expecttest library later on should we need more advanced features for certain tests.

I'd like to merge https://gerrit.lix.systems/c/lix/+/3145 soon-ish, if nobody opposes. It adds a custom snapshot test runner, but at 90 lines of code including docs I don't see the maintenance costs as too high. Of course, it is limited to file-based comparison and won't be able to update Python strings and stuff, but maybe that's a feature. That's why I'd like to move forward with that implementation for now to unblock at least functional/lang, and we can always add a more fully-featured `expecttest` library later on should we need more advanced features for certain tests.

👍 1

lix-project referenced this issue from a commit

2025-05-21 19:29:51 +00:00

tests/functional2: add snapshot fixture

lix-project closed this issue

2025-05-21 19:29:51 +00:00