lix-project/lix

Fork 65

Lix JSON parser silently accepts duplicate object keys #1162

New issue

Open

opened 2026-03-20 00:01:03 +00:00 by sky1e · 6 comments

sky1e commented

2026-03-20 00:01:03 +00:00

Member

Describe the bug

Lix's JSON parser used by builtins.fromJSON silently accepts duplicate object keys, discarding all values but the last.

Steps To Reproduce

nix eval --expr 'builtins.fromJSON '"''"'{"a":"a","a":"b"}'"''"
> { a = "b" }

Expected behavior

Such unusual JSON input should be rejected, or if not, the current behavior should be at least tested for.

`nix --version` output

nix (Lix, like Nix) 2.94.0
System type: x86_64-linux
Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux
Features: gc, signed-caches
System configuration file: /etc/nix/nix.conf
User configuration files: /home/skye/.config/nix/nix.conf:/nix/store/0q6ydqfjgwhlsczwi5jk4ds1dpxdnrns-plasma-workspace-6.6.2/etc/xdg/nix/nix.conf:/nix/store/1r1bfsknakc5kfz0gmf0i3h9jqlqqcnn-kglobalacceld-6.6.2/etc/xdg/nix/nix.conf:/nix/store/qrw4yjb7king22w52ivrjaa5d1vkx9zc-baloo-6.24.0/etc/xdg/nix/nix.conf:/home/skye/.config/kdedefaults/nix/nix.conf:/home/skye/.config/kdedefaults/nix/nix.conf:/etc/xdg/nix/nix.conf:/home/skye/.local/share/flatpak/exports/etc/xdg/nix/nix.conf:/var/lib/flatpak/exports/etc/xdg/nix/nix.conf:/home/skye/.nix-profile/etc/xdg/nix/nix.conf:/nix/profile/etc/xdg/nix/nix.conf:/home/skye/.local/state/nix/profile/etc/xdg/nix/nix.conf:/etc/profiles/per-user/skye/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf
Store directory: /nix/store
State directory: /nix/var/nix
Data directory: /nix/store/0zqbw17irvj0c71wn6cjbmcg2v04gzvd-lix-2.94.0/share

Additional context

The existing behavior is defined by JSONObjectState in lix/libexpr/json-to-value.cc

     class JSONObjectState : public JSONState {
         using JSONState::JSONState;
         GcMap<Symbol, Value> attrs;
         Symbol _key;
         std::unique_ptr<JSONState> resolve(EvalState & state) override
         {
             auto attrs2 = state.ctx.buildBindings(attrs.size());
             for (auto & i : attrs)
                 attrs2.insert(i.first, i.second);
             parent->value() = {NewValueAs::attrs, attrs2.alreadySorted()};
             return std::move(parent);
         }
         void add() override
         {
             attrs.insert_or_assign(_key, value());
             v = nullptr;
         }
     public:
         void key(string_t & name, EvalState & state)
         {
             _key = state.ctx.symbols.create(name);
         }
     };

## Describe the bug Lix's JSON parser used by `builtins.fromJSON` silently accepts duplicate object keys, discarding all values but the last. ## Steps To Reproduce ``` nix eval --expr 'builtins.fromJSON '"''"'{"a":"a","a":"b"}'"''" > { a = "b" } ``` ## Expected behavior Such unusual JSON input should be rejected, or if not, the current behavior should be at least tested for. ## `nix --version` output ``` nix (Lix, like Nix) 2.94.0 System type: x86_64-linux Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux Features: gc, signed-caches System configuration file: /etc/nix/nix.conf User configuration files: /home/skye/.config/nix/nix.conf:/nix/store/0q6ydqfjgwhlsczwi5jk4ds1dpxdnrns-plasma-workspace-6.6.2/etc/xdg/nix/nix.conf:/nix/store/1r1bfsknakc5kfz0gmf0i3h9jqlqqcnn-kglobalacceld-6.6.2/etc/xdg/nix/nix.conf:/nix/store/qrw4yjb7king22w52ivrjaa5d1vkx9zc-baloo-6.24.0/etc/xdg/nix/nix.conf:/home/skye/.config/kdedefaults/nix/nix.conf:/home/skye/.config/kdedefaults/nix/nix.conf:/etc/xdg/nix/nix.conf:/home/skye/.local/share/flatpak/exports/etc/xdg/nix/nix.conf:/var/lib/flatpak/exports/etc/xdg/nix/nix.conf:/home/skye/.nix-profile/etc/xdg/nix/nix.conf:/nix/profile/etc/xdg/nix/nix.conf:/home/skye/.local/state/nix/profile/etc/xdg/nix/nix.conf:/etc/profiles/per-user/skye/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf Store directory: /nix/store State directory: /nix/var/nix Data directory: /nix/store/0zqbw17irvj0c71wn6cjbmcg2v04gzvd-lix-2.94.0/share ``` ## Additional context The existing behavior is defined by `JSONObjectState` in `lix/libexpr/json-to-value.cc` https://git.lix.systems/lix-project/lix/src/commit/774f95759908891d9f9bcd2dd3271f5c148359d7/lix/libexpr/json-to-value.cc#L37-L59

sky1e added the

bug

label

2026-03-20 00:01:03 +00:00

sky1e commented

2026-03-20 00:12:51 +00:00

Author

Member

Since I'm already in the weeds in the JSON parser I'm happy to drive a fix, but that first requires determining how this will be fixed. This would be a breaking change. Would we want to warn first before making it an error? Would we need to add a flag to re-enable old behavior?

tom-hubrecht commented

2026-03-20 08:56:28 +00:00

Member

Maybe it would be best to add it to the deprecated features, adding a warning for now that can be turned to a hard error in a future release

qyriad added the

labels

2026-03-20 10:23:05 +00:00

piegames commented

2026-03-20 10:26:33 +00:00

Owner

People argue on Matrix that the current behavior is desired, on the basis of the JSON spec being fucked and other JSON parsers behaving the same. In that case, the current behavior should be documented and tested.

qyriad commented

2026-03-20 10:29:46 +00:00

Owner

We strongly believe Lix should not ever hard error on this. Duplicate keys in JSON are a known failure of the spec (among plenty others), and, for better or for worse, many other popular JSON parser implementations also accept duplicate keys: Python's standard json module, jq, and serde_json all accept duplicate keys (and discard all but the last value).

No, to us, the question is not whether or not Lix should accept duplicate keys, but what Lix should do with them.

We strongly believe Lix should *not* ever hard error on this. Duplicate keys in JSON are a [known failure](https://stackoverflow.com/questions/21832701/does-json-syntax-allow-duplicate-keys-in-an-object) of the spec ([among plenty others](https://seriot.ch/software/parsing_json.html)), and, for better or for worse, many other popular JSON parser implementations also accept duplicate keys: [Python's standard `json` module](https://stackoverflow.com/questions/14902299/json-loads-allows-duplicate-keys-in-a-dictionary-overwriting-the-first-value), [jq](https://github.com/jqlang/jq/issues/1795), and [`serde_json`](https://github.com/serde-rs/json/issues/762) all accept duplicate keys (and discard all but the last value). No, to us, the question is not whether or not Lix should accept duplicate keys, but what Lix should *do* with them.

qyriad added the

docs

Context

RFD

labels

2026-03-20 10:30:14 +00:00

pennae commented

2026-03-20 11:52:04 +00:00

Owner

agreed, lix should not emit any errors here. opt-in warnings (once we have an eval warnings system) sound like a good idea though, but even then we can't emit them everywhere without either support from nlohmann or using our own object reconstruction for all json parsing. given that our behavior is consistent with the rest of the ecosystem we could also do nothing at all and still be largely fine.

agreed, lix should not emit any errors here. opt-in warnings (once we have an eval warnings system) sound like a good idea though, but even then we can't emit them everywhere without either support from nlohmann or using our own object reconstruction for *all* json parsing. given that our behavior is consistent with the rest of the ecosystem we could also do nothing at all and still be largely fine.

lix-bot commented

2026-04-08 21:28:07 +00:00

Member

This issue was mentioned on Gerrit on the following CLs:

commit message in cl/5457 ("Add test for duplicate JSON keys for builtins.fromJSON")

This issue was mentioned on Gerrit on the following CLs: * commit message in [cl/5457](https://gerrit.lix.systems/c/lix/+/5457) ("Add test for duplicate JSON keys for builtins.fromJSON")