lix/src/libexpr/get-drvs.cc

413 lines
13 KiB
C++
Raw Normal View History

#include "get-drvs.hh"
#include "util.hh"
#include "eval-inline.hh"
#include "derivations.hh"
#include "store-api.hh"
#include "path-with-outputs.hh"
2013-11-19 13:09:03 +00:00
#include <cstring>
#include <regex>
2013-11-19 13:09:03 +00:00
namespace nix {
DrvInfo::DrvInfo(EvalState & state, std::string attrPath, Bindings * attrs)
: state(&state), attrs(attrs), attrPath(std::move(attrPath))
2017-07-17 17:02:56 +00:00
{
}
DrvInfo::DrvInfo(EvalState & state, ref<Store> store, const std::string & drvPathWithOutputs)
: state(&state), attrs(nullptr), attrPath("")
{
auto [drvPath, selectedOutputs] = parsePathWithOutputs(*store, drvPathWithOutputs);
this->drvPath = drvPath;
auto drv = store->derivationFromPath(drvPath);
name = drvPath.name();
if (selectedOutputs.size() > 1)
throw Error("building more than one derivation output is not supported, in '%s'", drvPathWithOutputs);
outputName =
selectedOutputs.empty()
? getOr(drv.env, "outputName", "out")
: *selectedOutputs.begin();
auto i = drv.outputs.find(outputName);
if (i == drv.outputs.end())
throw Error("derivation '%s' does not have output '%s'", store->printStorePath(drvPath), outputName);
auto & [outputName, output] = *i;
outPath = {output.path(*store, drv.name, outputName)};
}
std::string DrvInfo::queryName() const
2017-07-17 17:02:56 +00:00
{
if (name == "" && attrs) {
auto i = attrs->find(state->sName);
if (i == attrs->end()) throw TypeError("derivation name missing");
name = state->forceStringNoCtx(*i->value, noPos, "while evaluating the 'name' attribute of a derivation");
2017-07-17 17:02:56 +00:00
}
return name;
}
std::string DrvInfo::querySystem() const
2017-07-17 17:02:56 +00:00
{
if (system == "" && attrs) {
auto i = attrs->find(state->sSystem);
system = i == attrs->end() ? "unknown" : state->forceStringNoCtx(*i->value, i->pos, "while evaluating the 'system' attribute of a derivation");
2017-07-17 17:02:56 +00:00
}
return system;
}
std::optional<StorePath> DrvInfo::queryDrvPath() const
{
if (!drvPath && attrs) {
2013-11-19 13:09:03 +00:00
Bindings::iterator i = attrs->find(state->sDrvPath);
Use `std::set<StringContextElem>` not `PathSet` for string contexts Motivation `PathSet` is not correct because string contexts have other forms (`Built` and `DrvDeep`) that are not rendered as plain store paths. Instead of wrongly using `PathSet`, or "stringly typed" using `StringSet`, use `std::std<StringContextElem>`. ----- In support of this change, `NixStringContext` is now defined as `std::std<StringContextElem>` not `std:vector<StringContextElem>`. The old definition was just used by a `getContext` method which was only used by the eval cache. It can be deleted altogether since the types are now unified and the preexisting `copyContext` function already suffices. Summarizing the previous paragraph: Old: - `value/context.hh`: `NixStringContext = std::vector<StringContextElem>` - `value.hh`: `NixStringContext Value::getContext(...)` - `value.hh`: `copyContext(...)` New: - `value/context.hh`: `NixStringContext = std::set<StringContextElem>` - `value.hh`: `copyContext(...)` ---- The string representation of string context elements no longer contains the store dir. The diff of `src/libexpr/tests/value/context.cc` should make clear what the new representation is, so we recommend reviewing that file first. This was done for two reasons: Less API churn: `Value::mkString` and friends did not take a `Store` before. But if `NixStringContextElem::{parse, to_string}` *do* take a store (as they did before), then we cannot have the `Value` functions use them (in order to work with the fully-structured `NixStringContext`) without adding that argument. That would have been a lot of churn of threading the store, and this diff is already large enough, so the easier and less invasive thing to do was simply make the element `parse` and `to_string` functions not take the `Store` reference, and the easiest way to do that was to simply drop the store dir. Space usage: Dropping the `/nix/store/` (or similar) from the internal representation will safe space in the heap of the Nix programming being interpreted. If the heap contains many strings with non-trivial contexts, the saving could add up to something significant. ---- The eval cache version is bumped. The eval cache serialization uses `NixStringContextElem::{parse, to_string}`, and since those functions are changed per the above, that means the on-disk representation is also changed. This is simply done by changing the name of the used for the eval cache from `eval-cache-v4` to eval-cache-v5`. ---- To avoid some duplication `EvalCache::mkPathString` is added to abstract over the simple case of turning a store path to a string with just that string in the context. Context This PR picks up where #7543 left off. That one introduced the fully structured `NixStringContextElem` data type, but kept `PathSet context` as an awkward middle ground between internal `char[][]` interpreter heap string contexts and `NixStringContext` fully parsed string contexts. The infelicity of `PathSet context` was specifically called out during Nix team group review, but it was agreeing that fixing it could be left as future work. This is that future work. A possible follow-up step would be to get rid of the `char[][]` evaluator heap representation, too, but it is not yet clear how to do that. To use `NixStringContextElem` there we would need to get the STL containers to GC pointers in the GC build, and I am not sure how to do that. ---- PR #7543 effectively is writing the inverse of a `mkPathString`, `mkOutputString`, and one more such function for the `DrvDeep` case. I would like that PR to have property tests ensuring it is actually the inverse as expected. This PR sets things up nicely so that reworking that PR to be in that more elegant and better tested way is possible. Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-01-29 01:31:10 +00:00
NixStringContext context;
if (i == attrs->end())
drvPath = {std::nullopt};
else
drvPath = {state->coerceToStorePath(i->pos, *i->value, context, "while evaluating the 'drvPath' attribute of a derivation")};
}
return drvPath.value_or(std::nullopt);
}
StorePath DrvInfo::requireDrvPath() const
{
if (auto drvPath = queryDrvPath())
return *drvPath;
throw Error("derivation does not contain a 'drvPath' attribute");
}
StorePath DrvInfo::queryOutPath() const
{
2020-08-07 19:09:26 +00:00
if (!outPath && attrs) {
2013-11-19 13:09:03 +00:00
Bindings::iterator i = attrs->find(state->sOutPath);
Use `std::set<StringContextElem>` not `PathSet` for string contexts Motivation `PathSet` is not correct because string contexts have other forms (`Built` and `DrvDeep`) that are not rendered as plain store paths. Instead of wrongly using `PathSet`, or "stringly typed" using `StringSet`, use `std::std<StringContextElem>`. ----- In support of this change, `NixStringContext` is now defined as `std::std<StringContextElem>` not `std:vector<StringContextElem>`. The old definition was just used by a `getContext` method which was only used by the eval cache. It can be deleted altogether since the types are now unified and the preexisting `copyContext` function already suffices. Summarizing the previous paragraph: Old: - `value/context.hh`: `NixStringContext = std::vector<StringContextElem>` - `value.hh`: `NixStringContext Value::getContext(...)` - `value.hh`: `copyContext(...)` New: - `value/context.hh`: `NixStringContext = std::set<StringContextElem>` - `value.hh`: `copyContext(...)` ---- The string representation of string context elements no longer contains the store dir. The diff of `src/libexpr/tests/value/context.cc` should make clear what the new representation is, so we recommend reviewing that file first. This was done for two reasons: Less API churn: `Value::mkString` and friends did not take a `Store` before. But if `NixStringContextElem::{parse, to_string}` *do* take a store (as they did before), then we cannot have the `Value` functions use them (in order to work with the fully-structured `NixStringContext`) without adding that argument. That would have been a lot of churn of threading the store, and this diff is already large enough, so the easier and less invasive thing to do was simply make the element `parse` and `to_string` functions not take the `Store` reference, and the easiest way to do that was to simply drop the store dir. Space usage: Dropping the `/nix/store/` (or similar) from the internal representation will safe space in the heap of the Nix programming being interpreted. If the heap contains many strings with non-trivial contexts, the saving could add up to something significant. ---- The eval cache version is bumped. The eval cache serialization uses `NixStringContextElem::{parse, to_string}`, and since those functions are changed per the above, that means the on-disk representation is also changed. This is simply done by changing the name of the used for the eval cache from `eval-cache-v4` to eval-cache-v5`. ---- To avoid some duplication `EvalCache::mkPathString` is added to abstract over the simple case of turning a store path to a string with just that string in the context. Context This PR picks up where #7543 left off. That one introduced the fully structured `NixStringContextElem` data type, but kept `PathSet context` as an awkward middle ground between internal `char[][]` interpreter heap string contexts and `NixStringContext` fully parsed string contexts. The infelicity of `PathSet context` was specifically called out during Nix team group review, but it was agreeing that fixing it could be left as future work. This is that future work. A possible follow-up step would be to get rid of the `char[][]` evaluator heap representation, too, but it is not yet clear how to do that. To use `NixStringContextElem` there we would need to get the STL containers to GC pointers in the GC build, and I am not sure how to do that. ---- PR #7543 effectively is writing the inverse of a `mkPathString`, `mkOutputString`, and one more such function for the `DrvDeep` case. I would like that PR to have property tests ensuring it is actually the inverse as expected. This PR sets things up nicely so that reworking that PR to be in that more elegant and better tested way is possible. Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-01-29 01:31:10 +00:00
NixStringContext context;
2020-08-07 19:09:26 +00:00
if (i != attrs->end())
outPath = state->coerceToStorePath(i->pos, *i->value, context, "while evaluating the output path of a derivation");
}
2020-08-07 19:09:26 +00:00
if (!outPath)
throw UnimplementedError("CA derivations are not yet supported");
return *outPath;
}
DrvInfo::Outputs DrvInfo::queryOutputs(bool withPaths, bool onlyOutputsToInstall)
{
if (outputs.empty()) {
/* Get the outputs list. */
2013-11-19 13:09:03 +00:00
Bindings::iterator i;
if (attrs && (i = attrs->find(state->sOutputs)) != attrs->end()) {
state->forceList(*i->value, i->pos, "while evaluating the 'outputs' attribute of a derivation");
/* For each output... */
for (auto elem : i->value->listItems()) {
std::string output(state->forceStringNoCtx(*elem, i->pos, "while evaluating the name of an output of a derivation"));
if (withPaths) {
/* Evaluate the corresponding set. */
Bindings::iterator out = attrs->find(state->symbols.create(output));
if (out == attrs->end()) continue; // FIXME: throw error?
state->forceAttrs(*out->value, i->pos, "while evaluating an output of a derivation");
/* And evaluate its outPath attribute. */
Bindings::iterator outPath = out->value->attrs->find(state->sOutPath);
if (outPath == out->value->attrs->end()) continue; // FIXME: throw error?
Use `std::set<StringContextElem>` not `PathSet` for string contexts Motivation `PathSet` is not correct because string contexts have other forms (`Built` and `DrvDeep`) that are not rendered as plain store paths. Instead of wrongly using `PathSet`, or "stringly typed" using `StringSet`, use `std::std<StringContextElem>`. ----- In support of this change, `NixStringContext` is now defined as `std::std<StringContextElem>` not `std:vector<StringContextElem>`. The old definition was just used by a `getContext` method which was only used by the eval cache. It can be deleted altogether since the types are now unified and the preexisting `copyContext` function already suffices. Summarizing the previous paragraph: Old: - `value/context.hh`: `NixStringContext = std::vector<StringContextElem>` - `value.hh`: `NixStringContext Value::getContext(...)` - `value.hh`: `copyContext(...)` New: - `value/context.hh`: `NixStringContext = std::set<StringContextElem>` - `value.hh`: `copyContext(...)` ---- The string representation of string context elements no longer contains the store dir. The diff of `src/libexpr/tests/value/context.cc` should make clear what the new representation is, so we recommend reviewing that file first. This was done for two reasons: Less API churn: `Value::mkString` and friends did not take a `Store` before. But if `NixStringContextElem::{parse, to_string}` *do* take a store (as they did before), then we cannot have the `Value` functions use them (in order to work with the fully-structured `NixStringContext`) without adding that argument. That would have been a lot of churn of threading the store, and this diff is already large enough, so the easier and less invasive thing to do was simply make the element `parse` and `to_string` functions not take the `Store` reference, and the easiest way to do that was to simply drop the store dir. Space usage: Dropping the `/nix/store/` (or similar) from the internal representation will safe space in the heap of the Nix programming being interpreted. If the heap contains many strings with non-trivial contexts, the saving could add up to something significant. ---- The eval cache version is bumped. The eval cache serialization uses `NixStringContextElem::{parse, to_string}`, and since those functions are changed per the above, that means the on-disk representation is also changed. This is simply done by changing the name of the used for the eval cache from `eval-cache-v4` to eval-cache-v5`. ---- To avoid some duplication `EvalCache::mkPathString` is added to abstract over the simple case of turning a store path to a string with just that string in the context. Context This PR picks up where #7543 left off. That one introduced the fully structured `NixStringContextElem` data type, but kept `PathSet context` as an awkward middle ground between internal `char[][]` interpreter heap string contexts and `NixStringContext` fully parsed string contexts. The infelicity of `PathSet context` was specifically called out during Nix team group review, but it was agreeing that fixing it could be left as future work. This is that future work. A possible follow-up step would be to get rid of the `char[][]` evaluator heap representation, too, but it is not yet clear how to do that. To use `NixStringContextElem` there we would need to get the STL containers to GC pointers in the GC build, and I am not sure how to do that. ---- PR #7543 effectively is writing the inverse of a `mkPathString`, `mkOutputString`, and one more such function for the `DrvDeep` case. I would like that PR to have property tests ensuring it is actually the inverse as expected. This PR sets things up nicely so that reworking that PR to be in that more elegant and better tested way is possible. Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-01-29 01:31:10 +00:00
NixStringContext context;
outputs.emplace(output, state->coerceToStorePath(outPath->pos, *outPath->value, context, "while evaluating an output path of a derivation"));
} else
outputs.emplace(output, std::nullopt);
}
2013-11-19 13:09:03 +00:00
} else
outputs.emplace("out", withPaths ? std::optional{queryOutPath()} : std::nullopt);
}
if (!onlyOutputsToInstall || !attrs)
return outputs;
Bindings::iterator i;
if (attrs && (i = attrs->find(state->sOutputSpecified)) != attrs->end() && state->forceBool(*i->value, i->pos, "while evaluating the 'outputSpecified' attribute of a derivation")) {
Outputs result;
auto out = outputs.find(queryOutputName());
if (out == outputs.end())
throw Error("derivation does not have output '%s'", queryOutputName());
result.insert(*out);
return result;
}
else {
/* Check for `meta.outputsToInstall` and return `outputs` reduced to that. */
const Value * outTI = queryMeta("outputsToInstall");
if (!outTI) return outputs;
auto errMsg = Error("this derivation has bad 'meta.outputsToInstall'");
/* ^ this shows during `nix-env -i` right under the bad derivation */
if (!outTI->isList()) throw errMsg;
Outputs result;
for (auto elem : outTI->listItems()) {
if (elem->type() != nString) throw errMsg;
auto out = outputs.find(elem->string.s);
if (out == outputs.end()) throw errMsg;
result.insert(*out);
}
return result;
}
}
std::string DrvInfo::queryOutputName() const
{
if (outputName == "" && attrs) {
2013-11-19 13:09:03 +00:00
Bindings::iterator i = attrs->find(state->sOutputName);
outputName = i != attrs->end() ? state->forceStringNoCtx(*i->value, noPos, "while evaluating the output name of a derivation") : "";
}
return outputName;
}
2013-11-19 13:09:03 +00:00
Bindings * DrvInfo::getMeta()
{
2013-11-19 13:09:03 +00:00
if (meta) return meta;
if (!attrs) return 0;
Bindings::iterator a = attrs->find(state->sMeta);
if (a == attrs->end()) return 0;
state->forceAttrs(*a->value, a->pos, "while evaluating the 'meta' attribute of a derivation");
2013-11-19 13:09:03 +00:00
meta = a->value->attrs;
return meta;
}
2013-11-19 13:09:03 +00:00
StringSet DrvInfo::queryMetaNames()
{
2013-11-19 13:09:03 +00:00
StringSet res;
if (!getMeta()) return res;
2015-07-17 17:24:28 +00:00
for (auto & i : *meta)
res.emplace(state->symbols[i.name]);
2013-11-19 13:09:03 +00:00
return res;
}
bool DrvInfo::checkMeta(Value & v)
{
2022-02-03 23:31:33 +00:00
state->forceValue(v, [&]() { return v.determinePos(noPos); });
if (v.type() == nList) {
for (auto elem : v.listItems())
if (!checkMeta(*elem)) return false;
return true;
}
else if (v.type() == nAttrs) {
Bindings::iterator i = v.attrs->find(state->sOutPath);
if (i != v.attrs->end()) return false;
2015-07-17 17:24:28 +00:00
for (auto & i : *v.attrs)
if (!checkMeta(*i.value)) return false;
return true;
}
else return v.type() == nInt || v.type() == nBool || v.type() == nString ||
v.type() == nFloat;
}
Value * DrvInfo::queryMeta(const std::string & name)
{
2013-11-19 13:09:03 +00:00
if (!getMeta()) return 0;
Bindings::iterator a = meta->find(state->symbols.create(name));
if (a == meta->end() || !checkMeta(*a->value)) return 0;
2013-11-19 13:09:03 +00:00
return a->value;
}
std::string DrvInfo::queryMetaString(const std::string & name)
2013-11-19 13:09:03 +00:00
{
Value * v = queryMeta(name);
if (!v || v->type() != nString) return "";
2013-11-19 13:09:03 +00:00
return v->string.s;
}
NixInt DrvInfo::queryMetaInt(const std::string & name, NixInt def)
2013-11-19 13:09:03 +00:00
{
Value * v = queryMeta(name);
if (!v) return def;
if (v->type() == nInt) return v->integer;
if (v->type() == nString) {
2013-11-19 13:09:03 +00:00
/* Backwards compatibility with before we had support for
integer meta fields. */
2021-01-08 11:22:21 +00:00
if (auto n = string2Int<NixInt>(v->string.s))
return *n;
2013-11-19 13:09:03 +00:00
}
return def;
}
NixFloat DrvInfo::queryMetaFloat(const std::string & name, NixFloat def)
{
Value * v = queryMeta(name);
if (!v) return def;
if (v->type() == nFloat) return v->fpoint;
if (v->type() == nString) {
/* Backwards compatibility with before we had support for
float meta fields. */
2021-01-08 11:22:21 +00:00
if (auto n = string2Float<NixFloat>(v->string.s))
return *n;
}
return def;
}
2013-11-19 13:09:03 +00:00
bool DrvInfo::queryMetaBool(const std::string & name, bool def)
2013-11-19 13:09:03 +00:00
{
Value * v = queryMeta(name);
if (!v) return def;
if (v->type() == nBool) return v->boolean;
if (v->type() == nString) {
2013-11-19 13:09:03 +00:00
/* Backwards compatibility with before we had support for
Boolean meta fields. */
if (strcmp(v->string.s, "true") == 0) return true;
if (strcmp(v->string.s, "false") == 0) return false;
}
return def;
}
void DrvInfo::setMeta(const std::string & name, Value * v)
2013-11-19 13:09:03 +00:00
{
getMeta();
auto attrs = state->buildBindings(1 + (meta ? meta->size() : 0));
auto sym = state->symbols.create(name);
if (meta)
for (auto i : *meta)
if (i.name != sym)
attrs.insert(i);
if (v) attrs.insert(sym, v);
meta = attrs.finish();
2010-03-31 19:12:08 +00:00
}
2010-04-01 12:04:57 +00:00
/* Cache for already considered attrsets. */
2022-02-21 15:28:23 +00:00
typedef std::set<Bindings *> Done;
/* Evaluate value `v'. If it evaluates to a set of type `derivation',
then put information about it in `drvs' (unless it's already in `done').
The result boolean indicates whether it makes sense
for the caller to recursively search for derivations in `v'. */
2010-03-31 15:38:03 +00:00
static bool getDerivation(EvalState & state, Value & v,
const std::string & attrPath, DrvInfos & drvs, Done & done,
bool ignoreAssertionFailures)
{
try {
2022-02-03 23:31:33 +00:00
state.forceValue(v, [&]() { return v.determinePos(noPos); });
if (!state.isDerivation(v)) return true;
/* Remove spurious duplicates (e.g., a set like `rec { x =
derivation {...}; y = x;}'. */
if (!done.insert(v.attrs).second) return false;
2017-07-17 17:02:56 +00:00
DrvInfo drv(state, attrPath, v.attrs);
2017-07-17 17:02:56 +00:00
drv.queryName();
drvs.push_back(drv);
2017-07-17 17:02:56 +00:00
return false;
} catch (AssertionError & e) {
if (ignoreAssertionFailures) return false;
throw;
}
}
2019-02-12 12:43:32 +00:00
std::optional<DrvInfo> getDerivation(EvalState & state, Value & v,
bool ignoreAssertionFailures)
{
2010-04-01 12:04:57 +00:00
Done done;
DrvInfos drvs;
getDerivation(state, v, "", drvs, done, ignoreAssertionFailures);
if (drvs.size() != 1) return {};
return std::move(drvs.front());
}
static std::string addToPath(const std::string & s1, const std::string & s2)
{
return s1.empty() ? s2 : s1 + "." + s2;
}
static std::regex attrRegex("[A-Za-z_][A-Za-z0-9-_+]*");
static void getDerivations(EvalState & state, Value & vIn,
const std::string & pathPrefix, Bindings & autoArgs,
DrvInfos & drvs, Done & done,
bool ignoreAssertionFailures)
{
Value v;
state.autoCallFunction(autoArgs, vIn, v);
/* Process the expression. */
if (!getDerivation(state, v, pathPrefix, drvs, done, ignoreAssertionFailures)) ;
else if (v.type() == nAttrs) {
/* !!! undocumented hackery to support combining channels in
nix-env.cc. */
bool combineChannels = v.attrs->find(state.symbols.create("_combineChannels")) != v.attrs->end();
2008-08-25 14:15:56 +00:00
/* Consider the attributes in sorted order to get more
deterministic behaviour in nix-env operations (e.g. when
there are names clashes between derivations, the derivation
bound to the attribute with the "lower" name should take
precedence). */
for (auto & i : v.attrs->lexicographicOrder(state.symbols)) {
debug("evaluating attribute '%1%'", state.symbols[i->name]);
if (!std::regex_match(std::string(state.symbols[i->name]), attrRegex))
continue;
std::string pathPrefix2 = addToPath(pathPrefix, state.symbols[i->name]);
2008-08-25 14:15:56 +00:00
if (combineChannels)
getDerivations(state, *i->value, pathPrefix2, autoArgs, drvs, done, ignoreAssertionFailures);
else if (getDerivation(state, *i->value, pathPrefix2, drvs, done, ignoreAssertionFailures)) {
/* If the value of this attribute is itself a set,
should we recurse into it? => Only if it has a
`recurseForDerivations = true' attribute. */
if (i->value->type() == nAttrs) {
2020-06-18 11:44:40 +00:00
Bindings::iterator j = i->value->attrs->find(state.sRecurseForDerivations);
if (j != i->value->attrs->end() && state.forceBool(*j->value, j->pos, "while evaluating the attribute `recurseForDerivations`"))
getDerivations(state, *i->value, pathPrefix2, autoArgs, drvs, done, ignoreAssertionFailures);
}
}
}
}
else if (v.type() == nList) {
for (auto [n, elem] : enumerate(v.listItems())) {
std::string pathPrefix2 = addToPath(pathPrefix, fmt("%d", n));
if (getDerivation(state, *elem, pathPrefix2, drvs, done, ignoreAssertionFailures))
getDerivations(state, *elem, pathPrefix2, autoArgs, drvs, done, ignoreAssertionFailures);
}
}
2010-03-31 15:38:03 +00:00
else throw TypeError("expression does not evaluate to a derivation (or a set or list of those)");
}
void getDerivations(EvalState & state, Value & v, const std::string & pathPrefix,
Bindings & autoArgs, DrvInfos & drvs, bool ignoreAssertionFailures)
{
2010-04-01 12:04:57 +00:00
Done done;
getDerivations(state, v, pathPrefix, autoArgs, drvs, done, ignoreAssertionFailures);
}
}