2022-03-21 10:31:01 +00:00
|
|
|
#include "primops.hh"
|
|
|
|
#include "store-api.hh"
|
2022-03-22 20:14:58 +00:00
|
|
|
#include "make-content-addressed.hh"
|
2022-03-22 21:47:33 +00:00
|
|
|
#include "url.hh"
|
2022-03-21 10:31:01 +00:00
|
|
|
|
|
|
|
namespace nix {
|
|
|
|
|
2022-03-04 18:31:59 +00:00
|
|
|
static void prim_fetchClosure(EvalState & state, const PosIdx pos, Value * * args, Value & v)
|
2022-03-21 10:31:01 +00:00
|
|
|
{
|
2023-01-19 12:23:04 +00:00
|
|
|
state.forceAttrs(*args[0], pos, "while evaluating the argument passed to builtins.fetchClosure");
|
2022-03-21 10:31:01 +00:00
|
|
|
|
2022-03-21 13:34:45 +00:00
|
|
|
std::optional<std::string> fromStoreUrl;
|
|
|
|
std::optional<StorePath> fromPath;
|
2022-03-22 20:14:58 +00:00
|
|
|
bool toCA = false;
|
|
|
|
std::optional<StorePath> toPath;
|
2022-03-21 10:31:01 +00:00
|
|
|
|
|
|
|
for (auto & attr : *args[0]->attrs) {
|
2022-03-05 13:40:24 +00:00
|
|
|
const auto & attrName = state.symbols[attr.name];
|
|
|
|
|
|
|
|
if (attrName == "fromPath") {
|
Use `std::set<StringContextElem>` not `PathSet` for string contexts
Motivation
`PathSet` is not correct because string contexts have other forms
(`Built` and `DrvDeep`) that are not rendered as plain store paths.
Instead of wrongly using `PathSet`, or "stringly typed" using
`StringSet`, use `std::std<StringContextElem>`.
-----
In support of this change, `NixStringContext` is now defined as
`std::std<StringContextElem>` not `std:vector<StringContextElem>`. The
old definition was just used by a `getContext` method which was only
used by the eval cache. It can be deleted altogether since the types are
now unified and the preexisting `copyContext` function already suffices.
Summarizing the previous paragraph:
Old:
- `value/context.hh`: `NixStringContext = std::vector<StringContextElem>`
- `value.hh`: `NixStringContext Value::getContext(...)`
- `value.hh`: `copyContext(...)`
New:
- `value/context.hh`: `NixStringContext = std::set<StringContextElem>`
- `value.hh`: `copyContext(...)`
----
The string representation of string context elements no longer contains
the store dir. The diff of `src/libexpr/tests/value/context.cc` should
make clear what the new representation is, so we recommend reviewing
that file first. This was done for two reasons:
Less API churn:
`Value::mkString` and friends did not take a `Store` before. But if
`NixStringContextElem::{parse, to_string}` *do* take a store (as they
did before), then we cannot have the `Value` functions use them (in
order to work with the fully-structured `NixStringContext`) without
adding that argument.
That would have been a lot of churn of threading the store, and this
diff is already large enough, so the easier and less invasive thing to
do was simply make the element `parse` and `to_string` functions not
take the `Store` reference, and the easiest way to do that was to simply
drop the store dir.
Space usage:
Dropping the `/nix/store/` (or similar) from the internal representation
will safe space in the heap of the Nix programming being interpreted. If
the heap contains many strings with non-trivial contexts, the saving
could add up to something significant.
----
The eval cache version is bumped.
The eval cache serialization uses `NixStringContextElem::{parse,
to_string}`, and since those functions are changed per the above, that
means the on-disk representation is also changed.
This is simply done by changing the name of the used for the eval cache
from `eval-cache-v4` to eval-cache-v5`.
----
To avoid some duplication `EvalCache::mkPathString` is added to abstract
over the simple case of turning a store path to a string with just that
string in the context.
Context
This PR picks up where #7543 left off. That one introduced the fully
structured `NixStringContextElem` data type, but kept `PathSet context`
as an awkward middle ground between internal `char[][]` interpreter heap
string contexts and `NixStringContext` fully parsed string contexts.
The infelicity of `PathSet context` was specifically called out during
Nix team group review, but it was agreeing that fixing it could be left
as future work. This is that future work.
A possible follow-up step would be to get rid of the `char[][]`
evaluator heap representation, too, but it is not yet clear how to do
that. To use `NixStringContextElem` there we would need to get the STL
containers to GC pointers in the GC build, and I am not sure how to do
that.
----
PR #7543 effectively is writing the inverse of a `mkPathString`,
`mkOutputString`, and one more such function for the `DrvDeep` case. I
would like that PR to have property tests ensuring it is actually the
inverse as expected.
This PR sets things up nicely so that reworking that PR to be in that
more elegant and better tested way is possible.
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-01-29 01:31:10 +00:00
|
|
|
NixStringContext context;
|
2023-01-19 12:23:04 +00:00
|
|
|
fromPath = state.coerceToStorePath(attr.pos, *attr.value, context,
|
|
|
|
"while evaluating the 'fromPath' attribute passed to builtins.fetchClosure");
|
2022-03-21 10:31:01 +00:00
|
|
|
}
|
|
|
|
|
2022-03-05 13:40:24 +00:00
|
|
|
else if (attrName == "toPath") {
|
2022-03-04 18:31:59 +00:00
|
|
|
state.forceValue(*attr.value, attr.pos);
|
2022-03-22 20:14:58 +00:00
|
|
|
toCA = true;
|
|
|
|
if (attr.value->type() != nString || attr.value->string.s != std::string("")) {
|
Use `std::set<StringContextElem>` not `PathSet` for string contexts
Motivation
`PathSet` is not correct because string contexts have other forms
(`Built` and `DrvDeep`) that are not rendered as plain store paths.
Instead of wrongly using `PathSet`, or "stringly typed" using
`StringSet`, use `std::std<StringContextElem>`.
-----
In support of this change, `NixStringContext` is now defined as
`std::std<StringContextElem>` not `std:vector<StringContextElem>`. The
old definition was just used by a `getContext` method which was only
used by the eval cache. It can be deleted altogether since the types are
now unified and the preexisting `copyContext` function already suffices.
Summarizing the previous paragraph:
Old:
- `value/context.hh`: `NixStringContext = std::vector<StringContextElem>`
- `value.hh`: `NixStringContext Value::getContext(...)`
- `value.hh`: `copyContext(...)`
New:
- `value/context.hh`: `NixStringContext = std::set<StringContextElem>`
- `value.hh`: `copyContext(...)`
----
The string representation of string context elements no longer contains
the store dir. The diff of `src/libexpr/tests/value/context.cc` should
make clear what the new representation is, so we recommend reviewing
that file first. This was done for two reasons:
Less API churn:
`Value::mkString` and friends did not take a `Store` before. But if
`NixStringContextElem::{parse, to_string}` *do* take a store (as they
did before), then we cannot have the `Value` functions use them (in
order to work with the fully-structured `NixStringContext`) without
adding that argument.
That would have been a lot of churn of threading the store, and this
diff is already large enough, so the easier and less invasive thing to
do was simply make the element `parse` and `to_string` functions not
take the `Store` reference, and the easiest way to do that was to simply
drop the store dir.
Space usage:
Dropping the `/nix/store/` (or similar) from the internal representation
will safe space in the heap of the Nix programming being interpreted. If
the heap contains many strings with non-trivial contexts, the saving
could add up to something significant.
----
The eval cache version is bumped.
The eval cache serialization uses `NixStringContextElem::{parse,
to_string}`, and since those functions are changed per the above, that
means the on-disk representation is also changed.
This is simply done by changing the name of the used for the eval cache
from `eval-cache-v4` to eval-cache-v5`.
----
To avoid some duplication `EvalCache::mkPathString` is added to abstract
over the simple case of turning a store path to a string with just that
string in the context.
Context
This PR picks up where #7543 left off. That one introduced the fully
structured `NixStringContextElem` data type, but kept `PathSet context`
as an awkward middle ground between internal `char[][]` interpreter heap
string contexts and `NixStringContext` fully parsed string contexts.
The infelicity of `PathSet context` was specifically called out during
Nix team group review, but it was agreeing that fixing it could be left
as future work. This is that future work.
A possible follow-up step would be to get rid of the `char[][]`
evaluator heap representation, too, but it is not yet clear how to do
that. To use `NixStringContextElem` there we would need to get the STL
containers to GC pointers in the GC build, and I am not sure how to do
that.
----
PR #7543 effectively is writing the inverse of a `mkPathString`,
`mkOutputString`, and one more such function for the `DrvDeep` case. I
would like that PR to have property tests ensuring it is actually the
inverse as expected.
This PR sets things up nicely so that reworking that PR to be in that
more elegant and better tested way is possible.
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-01-29 01:31:10 +00:00
|
|
|
NixStringContext context;
|
2023-01-19 12:23:04 +00:00
|
|
|
toPath = state.coerceToStorePath(attr.pos, *attr.value, context,
|
|
|
|
"while evaluating the 'toPath' attribute passed to builtins.fetchClosure");
|
2022-03-22 20:14:58 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2022-03-05 13:40:24 +00:00
|
|
|
else if (attrName == "fromStore")
|
2023-01-19 12:23:04 +00:00
|
|
|
fromStoreUrl = state.forceStringNoCtx(*attr.value, attr.pos,
|
|
|
|
"while evaluating the 'fromStore' attribute passed to builtins.fetchClosure");
|
2022-03-21 10:31:01 +00:00
|
|
|
|
|
|
|
else
|
|
|
|
throw Error({
|
2022-03-05 13:40:24 +00:00
|
|
|
.msg = hintfmt("attribute '%s' isn't supported in call to 'fetchClosure'", attrName),
|
2022-03-04 18:31:59 +00:00
|
|
|
.errPos = state.positions[pos]
|
2022-03-21 10:31:01 +00:00
|
|
|
});
|
|
|
|
}
|
|
|
|
|
2022-03-21 13:34:45 +00:00
|
|
|
if (!fromPath)
|
2022-03-21 10:31:01 +00:00
|
|
|
throw Error({
|
2022-03-21 13:34:45 +00:00
|
|
|
.msg = hintfmt("attribute '%s' is missing in call to 'fetchClosure'", "fromPath"),
|
2022-03-04 18:31:59 +00:00
|
|
|
.errPos = state.positions[pos]
|
2022-03-21 10:31:01 +00:00
|
|
|
});
|
|
|
|
|
2022-03-21 13:34:45 +00:00
|
|
|
if (!fromStoreUrl)
|
2022-03-21 10:31:01 +00:00
|
|
|
throw Error({
|
2022-03-21 13:34:45 +00:00
|
|
|
.msg = hintfmt("attribute '%s' is missing in call to 'fetchClosure'", "fromStore"),
|
2022-03-04 18:31:59 +00:00
|
|
|
.errPos = state.positions[pos]
|
2022-03-21 10:31:01 +00:00
|
|
|
});
|
|
|
|
|
2022-03-22 21:47:33 +00:00
|
|
|
auto parsedURL = parseURL(*fromStoreUrl);
|
|
|
|
|
2022-03-22 22:19:21 +00:00
|
|
|
if (parsedURL.scheme != "http" &&
|
|
|
|
parsedURL.scheme != "https" &&
|
|
|
|
!(getEnv("_NIX_IN_TEST").has_value() && parsedURL.scheme == "file"))
|
2022-03-22 21:47:33 +00:00
|
|
|
throw Error({
|
|
|
|
.msg = hintfmt("'fetchClosure' only supports http:// and https:// stores"),
|
2022-03-04 18:31:59 +00:00
|
|
|
.errPos = state.positions[pos]
|
2022-03-22 21:47:33 +00:00
|
|
|
});
|
|
|
|
|
2022-04-06 09:52:51 +00:00
|
|
|
if (!parsedURL.query.empty())
|
|
|
|
throw Error({
|
|
|
|
.msg = hintfmt("'fetchClosure' does not support URL query parameters (in '%s')", *fromStoreUrl),
|
2022-03-04 18:31:59 +00:00
|
|
|
.errPos = state.positions[pos]
|
2022-04-06 09:52:51 +00:00
|
|
|
});
|
|
|
|
|
2022-03-22 21:47:33 +00:00
|
|
|
auto fromStore = openStore(parsedURL.to_string());
|
2022-03-21 10:31:01 +00:00
|
|
|
|
2022-03-22 20:14:58 +00:00
|
|
|
if (toCA) {
|
2022-03-22 21:01:20 +00:00
|
|
|
if (!toPath || !state.store->isValidPath(*toPath)) {
|
|
|
|
auto remappings = makeContentAddressed(*fromStore, *state.store, { *fromPath });
|
|
|
|
auto i = remappings.find(*fromPath);
|
|
|
|
assert(i != remappings.end());
|
|
|
|
if (toPath && *toPath != i->second)
|
|
|
|
throw Error({
|
|
|
|
.msg = hintfmt("rewriting '%s' to content-addressed form yielded '%s', while '%s' was expected",
|
|
|
|
state.store->printStorePath(*fromPath),
|
|
|
|
state.store->printStorePath(i->second),
|
|
|
|
state.store->printStorePath(*toPath)),
|
2022-03-04 18:31:59 +00:00
|
|
|
.errPos = state.positions[pos]
|
2022-03-22 21:01:20 +00:00
|
|
|
});
|
|
|
|
if (!toPath)
|
|
|
|
throw Error({
|
|
|
|
.msg = hintfmt(
|
|
|
|
"rewriting '%s' to content-addressed form yielded '%s'; "
|
|
|
|
"please set this in the 'toPath' attribute passed to 'fetchClosure'",
|
|
|
|
state.store->printStorePath(*fromPath),
|
|
|
|
state.store->printStorePath(i->second)),
|
2022-03-04 18:31:59 +00:00
|
|
|
.errPos = state.positions[pos]
|
2022-03-22 21:01:20 +00:00
|
|
|
});
|
|
|
|
}
|
2022-03-22 20:14:58 +00:00
|
|
|
} else {
|
2022-04-06 09:52:51 +00:00
|
|
|
if (!state.store->isValidPath(*fromPath))
|
|
|
|
copyClosure(*fromStore, *state.store, RealisedPath::Set { *fromPath });
|
2022-03-22 20:14:58 +00:00
|
|
|
toPath = fromPath;
|
|
|
|
}
|
2022-03-21 10:31:01 +00:00
|
|
|
|
2022-03-21 13:00:54 +00:00
|
|
|
/* In pure mode, require a CA path. */
|
|
|
|
if (evalSettings.pureEval) {
|
2022-03-22 20:14:58 +00:00
|
|
|
auto info = state.store->queryPathInfo(*toPath);
|
2022-03-21 13:00:54 +00:00
|
|
|
if (!info->isContentAddressed(*state.store))
|
|
|
|
throw Error({
|
|
|
|
.msg = hintfmt("in pure mode, 'fetchClosure' requires a content-addressed path, which '%s' isn't",
|
2022-03-22 20:14:58 +00:00
|
|
|
state.store->printStorePath(*toPath)),
|
2022-03-04 18:31:59 +00:00
|
|
|
.errPos = state.positions[pos]
|
2022-03-21 13:00:54 +00:00
|
|
|
});
|
|
|
|
}
|
2022-03-21 10:31:01 +00:00
|
|
|
|
Use `std::set<StringContextElem>` not `PathSet` for string contexts
Motivation
`PathSet` is not correct because string contexts have other forms
(`Built` and `DrvDeep`) that are not rendered as plain store paths.
Instead of wrongly using `PathSet`, or "stringly typed" using
`StringSet`, use `std::std<StringContextElem>`.
-----
In support of this change, `NixStringContext` is now defined as
`std::std<StringContextElem>` not `std:vector<StringContextElem>`. The
old definition was just used by a `getContext` method which was only
used by the eval cache. It can be deleted altogether since the types are
now unified and the preexisting `copyContext` function already suffices.
Summarizing the previous paragraph:
Old:
- `value/context.hh`: `NixStringContext = std::vector<StringContextElem>`
- `value.hh`: `NixStringContext Value::getContext(...)`
- `value.hh`: `copyContext(...)`
New:
- `value/context.hh`: `NixStringContext = std::set<StringContextElem>`
- `value.hh`: `copyContext(...)`
----
The string representation of string context elements no longer contains
the store dir. The diff of `src/libexpr/tests/value/context.cc` should
make clear what the new representation is, so we recommend reviewing
that file first. This was done for two reasons:
Less API churn:
`Value::mkString` and friends did not take a `Store` before. But if
`NixStringContextElem::{parse, to_string}` *do* take a store (as they
did before), then we cannot have the `Value` functions use them (in
order to work with the fully-structured `NixStringContext`) without
adding that argument.
That would have been a lot of churn of threading the store, and this
diff is already large enough, so the easier and less invasive thing to
do was simply make the element `parse` and `to_string` functions not
take the `Store` reference, and the easiest way to do that was to simply
drop the store dir.
Space usage:
Dropping the `/nix/store/` (or similar) from the internal representation
will safe space in the heap of the Nix programming being interpreted. If
the heap contains many strings with non-trivial contexts, the saving
could add up to something significant.
----
The eval cache version is bumped.
The eval cache serialization uses `NixStringContextElem::{parse,
to_string}`, and since those functions are changed per the above, that
means the on-disk representation is also changed.
This is simply done by changing the name of the used for the eval cache
from `eval-cache-v4` to eval-cache-v5`.
----
To avoid some duplication `EvalCache::mkPathString` is added to abstract
over the simple case of turning a store path to a string with just that
string in the context.
Context
This PR picks up where #7543 left off. That one introduced the fully
structured `NixStringContextElem` data type, but kept `PathSet context`
as an awkward middle ground between internal `char[][]` interpreter heap
string contexts and `NixStringContext` fully parsed string contexts.
The infelicity of `PathSet context` was specifically called out during
Nix team group review, but it was agreeing that fixing it could be left
as future work. This is that future work.
A possible follow-up step would be to get rid of the `char[][]`
evaluator heap representation, too, but it is not yet clear how to do
that. To use `NixStringContextElem` there we would need to get the STL
containers to GC pointers in the GC build, and I am not sure how to do
that.
----
PR #7543 effectively is writing the inverse of a `mkPathString`,
`mkOutputString`, and one more such function for the `DrvDeep` case. I
would like that PR to have property tests ensuring it is actually the
inverse as expected.
This PR sets things up nicely so that reworking that PR to be in that
more elegant and better tested way is possible.
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-01-29 01:31:10 +00:00
|
|
|
state.mkStorePathString(*toPath, v);
|
2022-03-21 10:31:01 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
static RegisterPrimOp primop_fetchClosure({
|
|
|
|
.name = "__fetchClosure",
|
|
|
|
.args = {"args"},
|
|
|
|
.doc = R"(
|
2022-03-22 22:31:48 +00:00
|
|
|
Fetch a Nix store closure from a binary cache, rewriting it into
|
|
|
|
content-addressed form. For example,
|
|
|
|
|
|
|
|
```nix
|
|
|
|
builtins.fetchClosure {
|
|
|
|
fromStore = "https://cache.nixos.org";
|
|
|
|
fromPath = /nix/store/r2jd6ygnmirm2g803mksqqjm4y39yi6i-git-2.33.1;
|
|
|
|
toPath = /nix/store/ldbhlwhh39wha58rm61bkiiwm6j7211j-git-2.33.1;
|
|
|
|
}
|
|
|
|
```
|
|
|
|
|
|
|
|
fetches `/nix/store/r2jd...` from the specified binary cache,
|
|
|
|
and rewrites it into the content-addressed store path
|
|
|
|
`/nix/store/ldbh...`.
|
|
|
|
|
|
|
|
If `fromPath` is already content-addressed, or if you are
|
|
|
|
allowing impure evaluation (`--impure`), then `toPath` may be
|
|
|
|
omitted.
|
|
|
|
|
|
|
|
To find out the correct value for `toPath` given a `fromPath`,
|
|
|
|
you can use `nix store make-content-addressed`:
|
|
|
|
|
|
|
|
```console
|
2022-03-22 22:40:04 +00:00
|
|
|
# nix store make-content-addressed --from https://cache.nixos.org /nix/store/r2jd6ygnmirm2g803mksqqjm4y39yi6i-git-2.33.1
|
2022-03-22 22:31:48 +00:00
|
|
|
rewrote '/nix/store/r2jd6ygnmirm2g803mksqqjm4y39yi6i-git-2.33.1' to '/nix/store/ldbhlwhh39wha58rm61bkiiwm6j7211j-git-2.33.1'
|
|
|
|
```
|
|
|
|
|
|
|
|
This function is similar to `builtins.storePath` in that it
|
|
|
|
allows you to use a previously built store path in a Nix
|
|
|
|
expression. However, it is more reproducible because it requires
|
|
|
|
specifying a binary cache from which the path can be fetched.
|
|
|
|
Also, requiring a content-addressed final store path avoids the
|
|
|
|
need for users to configure binary cache public keys.
|
2022-03-25 13:04:18 +00:00
|
|
|
|
|
|
|
This function is only available if you enable the experimental
|
|
|
|
feature `fetch-closure`.
|
2022-03-21 10:31:01 +00:00
|
|
|
)",
|
|
|
|
.fun = prim_fetchClosure,
|
2022-03-25 13:04:18 +00:00
|
|
|
.experimentalFeature = Xp::FetchClosure,
|
2022-03-21 10:31:01 +00:00
|
|
|
});
|
|
|
|
|
|
|
|
}
|