forked from lix-project/lix
Merge remote-tracking branch 'origin/master' into coerce-string
This commit is contained in:
commit
ca5c3e86ab
|
@ -72,6 +72,7 @@
|
||||||
- [CLI guideline](contributing/cli-guideline.md)
|
- [CLI guideline](contributing/cli-guideline.md)
|
||||||
- [Release Notes](release-notes/release-notes.md)
|
- [Release Notes](release-notes/release-notes.md)
|
||||||
- [Release X.Y (202?-??-??)](release-notes/rl-next.md)
|
- [Release X.Y (202?-??-??)](release-notes/rl-next.md)
|
||||||
|
- [Release 2.7 (2022-03-07)](release-notes/rl-2.7.md)
|
||||||
- [Release 2.6 (2022-01-24)](release-notes/rl-2.6.md)
|
- [Release 2.6 (2022-01-24)](release-notes/rl-2.6.md)
|
||||||
- [Release 2.5 (2021-12-13)](release-notes/rl-2.5.md)
|
- [Release 2.5 (2021-12-13)](release-notes/rl-2.5.md)
|
||||||
- [Release 2.4 (2021-11-01)](release-notes/rl-2.4.md)
|
- [Release 2.4 (2021-11-01)](release-notes/rl-2.4.md)
|
||||||
|
|
|
@ -110,7 +110,7 @@ default, set it to `-`.
|
||||||
7. A comma-separated list of *mandatory features*. A machine will only
|
7. A comma-separated list of *mandatory features*. A machine will only
|
||||||
be used to build a derivation if all of the machine’s mandatory
|
be used to build a derivation if all of the machine’s mandatory
|
||||||
features appear in the derivation’s `requiredSystemFeatures`
|
features appear in the derivation’s `requiredSystemFeatures`
|
||||||
attribute..
|
attribute.
|
||||||
|
|
||||||
8. The (base64-encoded) public host key of the remote machine. If omitted, SSH
|
8. The (base64-encoded) public host key of the remote machine. If omitted, SSH
|
||||||
will use its regular known-hosts file. Specifically, the field is calculated
|
will use its regular known-hosts file. Specifically, the field is calculated
|
||||||
|
|
33
doc/manual/src/release-notes/rl-2.7.md
Normal file
33
doc/manual/src/release-notes/rl-2.7.md
Normal file
|
@ -0,0 +1,33 @@
|
||||||
|
# Release X.Y (2022-03-07)
|
||||||
|
|
||||||
|
* Nix will now make some helpful suggestions when you mistype
|
||||||
|
something on the command line. For instance, if you type `nix build
|
||||||
|
nixpkgs#thunderbrd`, it will suggest `thunderbird`.
|
||||||
|
|
||||||
|
* A number of "default" flake output attributes have been
|
||||||
|
renamed. These are:
|
||||||
|
|
||||||
|
* `defaultPackage.<system>` → `packages.<system>.default`
|
||||||
|
* `defaultApps.<system>` → `apps.<system>.default`
|
||||||
|
* `defaultTemplate` → `templates.default`
|
||||||
|
* `defaultBundler.<system>` → `bundlers.<system>.default`
|
||||||
|
* `overlay` → `overlays.default`
|
||||||
|
* `devShell.<system>` → `devShells.<system>.default`
|
||||||
|
|
||||||
|
The old flake output attributes still work, but `nix flake check`
|
||||||
|
will warn about them.
|
||||||
|
|
||||||
|
* Breaking API change: `nix bundle` now supports bundlers of the form
|
||||||
|
`bundler.<system>.<name>= derivation: another-derivation;`. This
|
||||||
|
supports additional functionality to inspect evaluation information
|
||||||
|
during bundling. A new
|
||||||
|
[repository](https://github.com/NixOS/bundlers) has various bundlers
|
||||||
|
implemented.
|
||||||
|
|
||||||
|
* `nix store ping` now reports the version of the remote Nix daemon.
|
||||||
|
|
||||||
|
* `nix flake {init,new}` now display information about which files have been
|
||||||
|
created.
|
||||||
|
|
||||||
|
* Templates can now define a `welcomeText` attribute, which is printed out by
|
||||||
|
`nix flake {init,new} --template <template>`.
|
|
@ -1,28 +1,3 @@
|
||||||
# Release X.Y (202?-??-??)
|
# Release X.Y (202?-??-??)
|
||||||
|
|
||||||
* A number of "default" flake output attributes have been
|
* Various nix commands can now read expressions from stdin with `--file -`.
|
||||||
renamed. These are:
|
|
||||||
|
|
||||||
* `defaultPackage.<system>` → `packages.<system>.default`
|
|
||||||
* `defaultApps.<system>` → `apps.<system>.default`
|
|
||||||
* `defaultTemplate` → `templates.default`
|
|
||||||
* `defaultBundler.<system>` → `bundlers.<system>.default`
|
|
||||||
* `overlay` → `overlays.default`
|
|
||||||
* `devShell.<system>` → `devShells.<system>.default`
|
|
||||||
|
|
||||||
The old flake output attributes still work, but `nix flake check`
|
|
||||||
will warn about them.
|
|
||||||
|
|
||||||
* `nix bundle` breaking API change now supports bundlers of the form
|
|
||||||
`bundler.<system>.<name>= derivation: another-derivation;`. This supports
|
|
||||||
additional functionality to inspect evaluation information during bundling. A
|
|
||||||
new [repository](https://github.com/NixOS/bundlers) has various bundlers
|
|
||||||
implemented.
|
|
||||||
|
|
||||||
* `nix store ping` now reports the version of the remote Nix daemon.
|
|
||||||
|
|
||||||
* `nix flake {init,new}` now display information about which files have been
|
|
||||||
created.
|
|
||||||
|
|
||||||
* Templates can now define a `welcomeText` attribute, which is printed out by
|
|
||||||
`nix flake {init,new} --template <template>`.
|
|
||||||
|
|
|
@ -1,5 +1,6 @@
|
||||||
[Unit]
|
[Unit]
|
||||||
Description=Nix Daemon
|
Description=Nix Daemon
|
||||||
|
Documentation=man:nix-daemon https://nixos.org/manual
|
||||||
RequiresMountsFor=@storedir@
|
RequiresMountsFor=@storedir@
|
||||||
RequiresMountsFor=@localstatedir@
|
RequiresMountsFor=@localstatedir@
|
||||||
ConditionPathIsReadWrite=@localstatedir@/nix/daemon-socket
|
ConditionPathIsReadWrite=@localstatedir@/nix/daemon-socket
|
||||||
|
|
|
@ -12,6 +12,7 @@
|
||||||
#include "eval-cache.hh"
|
#include "eval-cache.hh"
|
||||||
#include "url.hh"
|
#include "url.hh"
|
||||||
#include "registry.hh"
|
#include "registry.hh"
|
||||||
|
#include "build-result.hh"
|
||||||
|
|
||||||
#include <regex>
|
#include <regex>
|
||||||
#include <queue>
|
#include <queue>
|
||||||
|
@ -133,7 +134,9 @@ SourceExprCommand::SourceExprCommand()
|
||||||
addFlag({
|
addFlag({
|
||||||
.longName = "file",
|
.longName = "file",
|
||||||
.shortName = 'f',
|
.shortName = 'f',
|
||||||
.description = "Interpret installables as attribute paths relative to the Nix expression stored in *file*.",
|
.description =
|
||||||
|
"Interpret installables as attribute paths relative to the Nix expression stored in *file*. "
|
||||||
|
"If *file* is the character -, then a Nix expression will be read from standard input.",
|
||||||
.category = installablesCategory,
|
.category = installablesCategory,
|
||||||
.labels = {"file"},
|
.labels = {"file"},
|
||||||
.handler = {&file},
|
.handler = {&file},
|
||||||
|
@ -272,9 +275,9 @@ void completeFlakeRefWithFragment(
|
||||||
auto attr = root->findAlongAttrPath(attrPath);
|
auto attr = root->findAlongAttrPath(attrPath);
|
||||||
if (!attr) continue;
|
if (!attr) continue;
|
||||||
|
|
||||||
for (auto & attr2 : attr->getAttrs()) {
|
for (auto & attr2 : (*attr)->getAttrs()) {
|
||||||
if (hasPrefix(attr2, lastAttr)) {
|
if (hasPrefix(attr2, lastAttr)) {
|
||||||
auto attrPath2 = attr->getAttrPath(attr2);
|
auto attrPath2 = (*attr)->getAttrPath(attr2);
|
||||||
/* Strip the attrpath prefix. */
|
/* Strip the attrpath prefix. */
|
||||||
attrPath2.erase(attrPath2.begin(), attrPath2.begin() + attrPathPrefix.size());
|
attrPath2.erase(attrPath2.begin(), attrPath2.begin() + attrPathPrefix.size());
|
||||||
completions->add(flakeRefS + "#" + concatStringsSep(".", attrPath2));
|
completions->add(flakeRefS + "#" + concatStringsSep(".", attrPath2));
|
||||||
|
@ -568,15 +571,22 @@ std::tuple<std::string, FlakeRef, InstallableValue::DerivationInfo> InstallableF
|
||||||
auto cache = openEvalCache(*state, lockedFlake);
|
auto cache = openEvalCache(*state, lockedFlake);
|
||||||
auto root = cache->getRoot();
|
auto root = cache->getRoot();
|
||||||
|
|
||||||
|
Suggestions suggestions;
|
||||||
|
|
||||||
for (auto & attrPath : getActualAttrPaths()) {
|
for (auto & attrPath : getActualAttrPaths()) {
|
||||||
debug("trying flake output attribute '%s'", attrPath);
|
debug("trying flake output attribute '%s'", attrPath);
|
||||||
|
|
||||||
auto attr = root->findAlongAttrPath(
|
auto attrOrSuggestions = root->findAlongAttrPath(
|
||||||
parseAttrPath(*state, attrPath),
|
parseAttrPath(*state, attrPath),
|
||||||
true
|
true
|
||||||
);
|
);
|
||||||
|
|
||||||
if (!attr) continue;
|
if (!attrOrSuggestions) {
|
||||||
|
suggestions += attrOrSuggestions.getSuggestions();
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
auto attr = *attrOrSuggestions;
|
||||||
|
|
||||||
if (!attr->isDerivation())
|
if (!attr->isDerivation())
|
||||||
throw Error("flake output attribute '%s' is not a derivation", attrPath);
|
throw Error("flake output attribute '%s' is not a derivation", attrPath);
|
||||||
|
@ -591,7 +601,7 @@ std::tuple<std::string, FlakeRef, InstallableValue::DerivationInfo> InstallableF
|
||||||
return {attrPath, lockedFlake->flake.lockedRef, std::move(drvInfo)};
|
return {attrPath, lockedFlake->flake.lockedRef, std::move(drvInfo)};
|
||||||
}
|
}
|
||||||
|
|
||||||
throw Error("flake '%s' does not provide attribute %s",
|
throw Error(suggestions, "flake '%s' does not provide attribute %s",
|
||||||
flakeRef, showAttrPaths(getActualAttrPaths()));
|
flakeRef, showAttrPaths(getActualAttrPaths()));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -610,17 +620,24 @@ std::pair<Value *, Pos> InstallableFlake::toValue(EvalState & state)
|
||||||
|
|
||||||
auto emptyArgs = state.allocBindings(0);
|
auto emptyArgs = state.allocBindings(0);
|
||||||
|
|
||||||
|
Suggestions suggestions;
|
||||||
|
|
||||||
for (auto & attrPath : getActualAttrPaths()) {
|
for (auto & attrPath : getActualAttrPaths()) {
|
||||||
try {
|
try {
|
||||||
auto [v, pos] = findAlongAttrPath(state, attrPath, *emptyArgs, *vOutputs);
|
auto [v, pos] = findAlongAttrPath(state, attrPath, *emptyArgs, *vOutputs);
|
||||||
state.forceValue(*v, pos);
|
state.forceValue(*v, pos);
|
||||||
return {v, pos};
|
return {v, pos};
|
||||||
} catch (AttrPathNotFound & e) {
|
} catch (AttrPathNotFound & e) {
|
||||||
|
suggestions += e.info().suggestions;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
throw Error("flake '%s' does not provide attribute %s",
|
throw Error(
|
||||||
flakeRef, showAttrPaths(getActualAttrPaths()));
|
suggestions,
|
||||||
|
"flake '%s' does not provide attribute %s",
|
||||||
|
flakeRef,
|
||||||
|
showAttrPaths(getActualAttrPaths())
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
std::vector<std::pair<std::shared_ptr<eval_cache::AttrCursor>, std::string>>
|
std::vector<std::pair<std::shared_ptr<eval_cache::AttrCursor>, std::string>>
|
||||||
|
@ -635,7 +652,7 @@ InstallableFlake::getCursors(EvalState & state)
|
||||||
|
|
||||||
for (auto & attrPath : getActualAttrPaths()) {
|
for (auto & attrPath : getActualAttrPaths()) {
|
||||||
auto attr = root->findAlongAttrPath(parseAttrPath(state, attrPath));
|
auto attr = root->findAlongAttrPath(parseAttrPath(state, attrPath));
|
||||||
if (attr) res.push_back({attr, attrPath});
|
if (attr) res.push_back({*attr, attrPath});
|
||||||
}
|
}
|
||||||
|
|
||||||
return res;
|
return res;
|
||||||
|
@ -680,7 +697,10 @@ std::vector<std::shared_ptr<Installable>> SourceExprCommand::parseInstallables(
|
||||||
auto state = getEvalState();
|
auto state = getEvalState();
|
||||||
auto vFile = state->allocValue();
|
auto vFile = state->allocValue();
|
||||||
|
|
||||||
if (file)
|
if (file == "-") {
|
||||||
|
auto e = state->parseStdin();
|
||||||
|
state->eval(e, *vFile);
|
||||||
|
} else if (file)
|
||||||
state->evalFile(lookupFileArg(*state, *file), *vFile);
|
state->evalFile(lookupFileArg(*state, *file), *vFile);
|
||||||
else {
|
else {
|
||||||
auto e = state->parseExprFromString(*expr, absPath("."));
|
auto e = state->parseExprFromString(*expr, absPath("."));
|
||||||
|
@ -755,8 +775,7 @@ BuiltPaths getBuiltPaths(ref<Store> evalStore, ref<Store> store, const DerivedPa
|
||||||
throw Error(
|
throw Error(
|
||||||
"the derivation '%s' doesn't have an output named '%s'",
|
"the derivation '%s' doesn't have an output named '%s'",
|
||||||
store->printStorePath(bfd.drvPath), output);
|
store->printStorePath(bfd.drvPath), output);
|
||||||
if (settings.isExperimentalFeatureEnabled(
|
if (settings.isExperimentalFeatureEnabled(Xp::CaDerivations)) {
|
||||||
Xp::CaDerivations)) {
|
|
||||||
auto outputId =
|
auto outputId =
|
||||||
DrvOutput{outputHashes.at(output), output};
|
DrvOutput{outputHashes.at(output), output};
|
||||||
auto realisation =
|
auto realisation =
|
||||||
|
@ -802,12 +821,33 @@ BuiltPaths Installable::build(
|
||||||
pathsToBuild.insert(pathsToBuild.end(), b.begin(), b.end());
|
pathsToBuild.insert(pathsToBuild.end(), b.begin(), b.end());
|
||||||
}
|
}
|
||||||
|
|
||||||
if (mode == Realise::Nothing || mode == Realise::Derivation)
|
switch (mode) {
|
||||||
|
case Realise::Nothing:
|
||||||
|
case Realise::Derivation:
|
||||||
printMissing(store, pathsToBuild, lvlError);
|
printMissing(store, pathsToBuild, lvlError);
|
||||||
else if (mode == Realise::Outputs)
|
|
||||||
store->buildPaths(pathsToBuild, bMode, evalStore);
|
|
||||||
|
|
||||||
return getBuiltPaths(evalStore, store, pathsToBuild);
|
return getBuiltPaths(evalStore, store, pathsToBuild);
|
||||||
|
case Realise::Outputs: {
|
||||||
|
BuiltPaths res;
|
||||||
|
for (auto & buildResult : store->buildPathsWithResults(pathsToBuild, bMode, evalStore)) {
|
||||||
|
if (!buildResult.success())
|
||||||
|
buildResult.rethrow();
|
||||||
|
std::visit(overloaded {
|
||||||
|
[&](const DerivedPath::Built & bfd) {
|
||||||
|
std::map<std::string, StorePath> outputs;
|
||||||
|
for (auto & path : buildResult.builtOutputs)
|
||||||
|
outputs.emplace(path.first.outputName, path.second.outPath);
|
||||||
|
res.push_back(BuiltPath::Built { bfd.drvPath, outputs });
|
||||||
|
},
|
||||||
|
[&](const DerivedPath::Opaque & bo) {
|
||||||
|
res.push_back(BuiltPath::Opaque { bo.path });
|
||||||
|
},
|
||||||
|
}, buildResult.path.raw());
|
||||||
|
}
|
||||||
|
return res;
|
||||||
|
}
|
||||||
|
default:
|
||||||
|
assert(false);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
BuiltPaths Installable::toBuiltPaths(
|
BuiltPaths Installable::toBuiltPaths(
|
||||||
|
|
|
@ -74,8 +74,14 @@ std::pair<Value *, Pos> findAlongAttrPath(EvalState & state, const std::string &
|
||||||
throw Error("empty attribute name in selection path '%1%'", attrPath);
|
throw Error("empty attribute name in selection path '%1%'", attrPath);
|
||||||
|
|
||||||
Bindings::iterator a = v->attrs->find(state.symbols.create(attr));
|
Bindings::iterator a = v->attrs->find(state.symbols.create(attr));
|
||||||
if (a == v->attrs->end())
|
if (a == v->attrs->end()) {
|
||||||
throw AttrPathNotFound("attribute '%1%' in selection path '%2%' not found", attr, attrPath);
|
std::set<std::string> attrNames;
|
||||||
|
for (auto & attr : *v->attrs)
|
||||||
|
attrNames.insert(attr.name);
|
||||||
|
|
||||||
|
auto suggestions = Suggestions::bestMatches(attrNames, attr);
|
||||||
|
throw AttrPathNotFound(suggestions, "attribute '%1%' in selection path '%2%' not found", attr, attrPath);
|
||||||
|
}
|
||||||
v = &*a->value;
|
v = &*a->value;
|
||||||
pos = *a->pos;
|
pos = *a->pos;
|
||||||
}
|
}
|
||||||
|
|
|
@ -406,6 +406,16 @@ Value & AttrCursor::forceValue()
|
||||||
return v;
|
return v;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
Suggestions AttrCursor::getSuggestionsForAttr(Symbol name)
|
||||||
|
{
|
||||||
|
auto attrNames = getAttrs();
|
||||||
|
std::set<std::string> strAttrNames;
|
||||||
|
for (auto & name : attrNames)
|
||||||
|
strAttrNames.insert(std::string(name));
|
||||||
|
|
||||||
|
return Suggestions::bestMatches(strAttrNames, name);
|
||||||
|
}
|
||||||
|
|
||||||
std::shared_ptr<AttrCursor> AttrCursor::maybeGetAttr(Symbol name, bool forceErrors)
|
std::shared_ptr<AttrCursor> AttrCursor::maybeGetAttr(Symbol name, bool forceErrors)
|
||||||
{
|
{
|
||||||
if (root->db) {
|
if (root->db) {
|
||||||
|
@ -446,6 +456,11 @@ std::shared_ptr<AttrCursor> AttrCursor::maybeGetAttr(Symbol name, bool forceErro
|
||||||
return nullptr;
|
return nullptr;
|
||||||
//throw TypeError("'%s' is not an attribute set", getAttrPathStr());
|
//throw TypeError("'%s' is not an attribute set", getAttrPathStr());
|
||||||
|
|
||||||
|
for (auto & attr : *v.attrs) {
|
||||||
|
if (root->db)
|
||||||
|
root->db->setPlaceholder({cachedValue->first, attr.name});
|
||||||
|
}
|
||||||
|
|
||||||
auto attr = v.attrs->get(name);
|
auto attr = v.attrs->get(name);
|
||||||
|
|
||||||
if (!attr) {
|
if (!attr) {
|
||||||
|
@ -464,7 +479,7 @@ std::shared_ptr<AttrCursor> AttrCursor::maybeGetAttr(Symbol name, bool forceErro
|
||||||
cachedValue2 = {root->db->setPlaceholder({cachedValue->first, name}), placeholder_t()};
|
cachedValue2 = {root->db->setPlaceholder({cachedValue->first, name}), placeholder_t()};
|
||||||
}
|
}
|
||||||
|
|
||||||
return std::make_shared<AttrCursor>(
|
return make_ref<AttrCursor>(
|
||||||
root, std::make_pair(shared_from_this(), name), attr->value, std::move(cachedValue2));
|
root, std::make_pair(shared_from_this(), name), attr->value, std::move(cachedValue2));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -473,27 +488,31 @@ std::shared_ptr<AttrCursor> AttrCursor::maybeGetAttr(std::string_view name)
|
||||||
return maybeGetAttr(root->state.symbols.create(name));
|
return maybeGetAttr(root->state.symbols.create(name));
|
||||||
}
|
}
|
||||||
|
|
||||||
std::shared_ptr<AttrCursor> AttrCursor::getAttr(Symbol name, bool forceErrors)
|
ref<AttrCursor> AttrCursor::getAttr(Symbol name, bool forceErrors)
|
||||||
{
|
{
|
||||||
auto p = maybeGetAttr(name, forceErrors);
|
auto p = maybeGetAttr(name, forceErrors);
|
||||||
if (!p)
|
if (!p)
|
||||||
throw Error("attribute '%s' does not exist", getAttrPathStr(name));
|
throw Error("attribute '%s' does not exist", getAttrPathStr(name));
|
||||||
return p;
|
return ref(p);
|
||||||
}
|
}
|
||||||
|
|
||||||
std::shared_ptr<AttrCursor> AttrCursor::getAttr(std::string_view name)
|
ref<AttrCursor> AttrCursor::getAttr(std::string_view name)
|
||||||
{
|
{
|
||||||
return getAttr(root->state.symbols.create(name));
|
return getAttr(root->state.symbols.create(name));
|
||||||
}
|
}
|
||||||
|
|
||||||
std::shared_ptr<AttrCursor> AttrCursor::findAlongAttrPath(const std::vector<Symbol> & attrPath, bool force)
|
OrSuggestions<ref<AttrCursor>> AttrCursor::findAlongAttrPath(const std::vector<Symbol> & attrPath, bool force)
|
||||||
{
|
{
|
||||||
auto res = shared_from_this();
|
auto res = shared_from_this();
|
||||||
for (auto & attr : attrPath) {
|
for (auto & attr : attrPath) {
|
||||||
res = res->maybeGetAttr(attr, force);
|
auto child = res->maybeGetAttr(attr, force);
|
||||||
if (!res) return {};
|
if (!child) {
|
||||||
|
auto suggestions = res->getSuggestionsForAttr(attr);
|
||||||
|
return OrSuggestions<ref<AttrCursor>>::failed(suggestions);
|
||||||
}
|
}
|
||||||
return res;
|
res = child;
|
||||||
|
}
|
||||||
|
return ref(res);
|
||||||
}
|
}
|
||||||
|
|
||||||
std::string AttrCursor::getString()
|
std::string AttrCursor::getString()
|
||||||
|
|
|
@ -94,15 +94,17 @@ public:
|
||||||
|
|
||||||
std::string getAttrPathStr(Symbol name) const;
|
std::string getAttrPathStr(Symbol name) const;
|
||||||
|
|
||||||
|
Suggestions getSuggestionsForAttr(Symbol name);
|
||||||
|
|
||||||
std::shared_ptr<AttrCursor> maybeGetAttr(Symbol name, bool forceErrors = false);
|
std::shared_ptr<AttrCursor> maybeGetAttr(Symbol name, bool forceErrors = false);
|
||||||
|
|
||||||
std::shared_ptr<AttrCursor> maybeGetAttr(std::string_view name);
|
std::shared_ptr<AttrCursor> maybeGetAttr(std::string_view name);
|
||||||
|
|
||||||
std::shared_ptr<AttrCursor> getAttr(Symbol name, bool forceErrors = false);
|
ref<AttrCursor> getAttr(Symbol name, bool forceErrors = false);
|
||||||
|
|
||||||
std::shared_ptr<AttrCursor> getAttr(std::string_view name);
|
ref<AttrCursor> getAttr(std::string_view name);
|
||||||
|
|
||||||
std::shared_ptr<AttrCursor> findAlongAttrPath(const std::vector<Symbol> & attrPath, bool force = false);
|
OrSuggestions<ref<AttrCursor>> findAlongAttrPath(const std::vector<Symbol> & attrPath, bool force = false);
|
||||||
|
|
||||||
std::string getString();
|
std::string getString();
|
||||||
|
|
||||||
|
|
|
@ -24,6 +24,81 @@ LocalNoInlineNoReturn(void throwTypeError(const Pos & pos, const char * s, const
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
/* Note: Various places expect the allocated memory to be zeroed. */
|
||||||
|
[[gnu::always_inline]]
|
||||||
|
inline void * allocBytes(size_t n)
|
||||||
|
{
|
||||||
|
void * p;
|
||||||
|
#if HAVE_BOEHMGC
|
||||||
|
p = GC_MALLOC(n);
|
||||||
|
#else
|
||||||
|
p = calloc(n, 1);
|
||||||
|
#endif
|
||||||
|
if (!p) throw std::bad_alloc();
|
||||||
|
return p;
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
[[gnu::always_inline]]
|
||||||
|
Value * EvalState::allocValue()
|
||||||
|
{
|
||||||
|
#if HAVE_BOEHMGC
|
||||||
|
/* We use the boehm batch allocator to speed up allocations of Values (of which there are many).
|
||||||
|
GC_malloc_many returns a linked list of objects of the given size, where the first word
|
||||||
|
of each object is also the pointer to the next object in the list. This also means that we
|
||||||
|
have to explicitly clear the first word of every object we take. */
|
||||||
|
if (!*valueAllocCache) {
|
||||||
|
*valueAllocCache = GC_malloc_many(sizeof(Value));
|
||||||
|
if (!*valueAllocCache) throw std::bad_alloc();
|
||||||
|
}
|
||||||
|
|
||||||
|
/* GC_NEXT is a convenience macro for accessing the first word of an object.
|
||||||
|
Take the first list item, advance the list to the next item, and clear the next pointer. */
|
||||||
|
void * p = *valueAllocCache;
|
||||||
|
*valueAllocCache = GC_NEXT(p);
|
||||||
|
GC_NEXT(p) = nullptr;
|
||||||
|
#else
|
||||||
|
void * p = allocBytes(sizeof(Value));
|
||||||
|
#endif
|
||||||
|
|
||||||
|
nrValues++;
|
||||||
|
return (Value *) p;
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
[[gnu::always_inline]]
|
||||||
|
Env & EvalState::allocEnv(size_t size)
|
||||||
|
{
|
||||||
|
nrEnvs++;
|
||||||
|
nrValuesInEnvs += size;
|
||||||
|
|
||||||
|
Env * env;
|
||||||
|
|
||||||
|
#if HAVE_BOEHMGC
|
||||||
|
if (size == 1) {
|
||||||
|
/* see allocValue for explanations. */
|
||||||
|
if (!*env1AllocCache) {
|
||||||
|
*env1AllocCache = GC_malloc_many(sizeof(Env) + sizeof(Value *));
|
||||||
|
if (!*env1AllocCache) throw std::bad_alloc();
|
||||||
|
}
|
||||||
|
|
||||||
|
void * p = *env1AllocCache;
|
||||||
|
*env1AllocCache = GC_NEXT(p);
|
||||||
|
GC_NEXT(p) = nullptr;
|
||||||
|
env = (Env *) p;
|
||||||
|
} else
|
||||||
|
#endif
|
||||||
|
env = (Env *) allocBytes(sizeof(Env) + size * sizeof(Value *));
|
||||||
|
|
||||||
|
env->type = Env::Plain;
|
||||||
|
|
||||||
|
/* We assume that env->values has been cleared by the allocator; maybeThunk() and lookupVar fromWith expect this. */
|
||||||
|
|
||||||
|
return *env;
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
[[gnu::always_inline]]
|
||||||
void EvalState::forceValue(Value & v, const Pos & pos)
|
void EvalState::forceValue(Value & v, const Pos & pos)
|
||||||
{
|
{
|
||||||
forceValue(v, [&]() { return pos; });
|
forceValue(v, [&]() { return pos; });
|
||||||
|
@ -52,6 +127,7 @@ void EvalState::forceValue(Value & v, Callable getPos)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
[[gnu::always_inline]]
|
||||||
inline void EvalState::forceAttrs(Value & v, const Pos & pos, const std::string_view & errorCtx)
|
inline void EvalState::forceAttrs(Value & v, const Pos & pos, const std::string_view & errorCtx)
|
||||||
{
|
{
|
||||||
forceAttrs(v, [&]() { return pos; }, errorCtx);
|
forceAttrs(v, [&]() { return pos; }, errorCtx);
|
||||||
|
@ -59,6 +135,7 @@ inline void EvalState::forceAttrs(Value & v, const Pos & pos, const std::string_
|
||||||
|
|
||||||
|
|
||||||
template <typename Callable>
|
template <typename Callable>
|
||||||
|
[[gnu::always_inline]]
|
||||||
inline void EvalState::forceAttrs(Value & v, Callable getPos, const std::string_view & errorCtx)
|
inline void EvalState::forceAttrs(Value & v, Callable getPos, const std::string_view & errorCtx)
|
||||||
{
|
{
|
||||||
try {
|
try {
|
||||||
|
@ -74,6 +151,7 @@ inline void EvalState::forceAttrs(Value & v, Callable getPos, const std::string_
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
[[gnu::always_inline]]
|
||||||
inline void EvalState::forceList(Value & v, const Pos & pos, const std::string_view & errorCtx)
|
inline void EvalState::forceList(Value & v, const Pos & pos, const std::string_view & errorCtx)
|
||||||
{
|
{
|
||||||
try {
|
try {
|
||||||
|
@ -87,18 +165,5 @@ inline void EvalState::forceList(Value & v, const Pos & pos, const std::string_v
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Note: Various places expect the allocated memory to be zeroed. */
|
|
||||||
inline void * allocBytes(size_t n)
|
|
||||||
{
|
|
||||||
void * p;
|
|
||||||
#if HAVE_BOEHMGC
|
|
||||||
p = GC_MALLOC(n);
|
|
||||||
#else
|
|
||||||
p = calloc(n, 1);
|
|
||||||
#endif
|
|
||||||
if (!p) throw std::bad_alloc();
|
|
||||||
return p;
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
|
@ -63,9 +63,15 @@ static char * dupString(const char * s)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
static char * dupStringWithLen(const char * s, size_t size)
|
// When there's no need to write to the string, we can optimize away empty
|
||||||
|
// string allocations.
|
||||||
|
// This function handles makeImmutableStringWithLen(null, 0) by returning the
|
||||||
|
// empty string.
|
||||||
|
static const char * makeImmutableStringWithLen(const char * s, size_t size)
|
||||||
{
|
{
|
||||||
char * t;
|
char * t;
|
||||||
|
if (size == 0)
|
||||||
|
return "";
|
||||||
#if HAVE_BOEHMGC
|
#if HAVE_BOEHMGC
|
||||||
t = GC_STRNDUP(s, size);
|
t = GC_STRNDUP(s, size);
|
||||||
#else
|
#else
|
||||||
|
@ -75,6 +81,10 @@ static char * dupStringWithLen(const char * s, size_t size)
|
||||||
return t;
|
return t;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static inline const char * makeImmutableString(std::string_view s) {
|
||||||
|
return makeImmutableStringWithLen(s.data(), s.size());
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
RootValue allocRootValue(Value * v)
|
RootValue allocRootValue(Value * v)
|
||||||
{
|
{
|
||||||
|
@ -439,8 +449,10 @@ EvalState::EvalState(
|
||||||
, regexCache(makeRegexCache())
|
, regexCache(makeRegexCache())
|
||||||
#if HAVE_BOEHMGC
|
#if HAVE_BOEHMGC
|
||||||
, valueAllocCache(std::allocate_shared<void *>(traceable_allocator<void *>(), nullptr))
|
, valueAllocCache(std::allocate_shared<void *>(traceable_allocator<void *>(), nullptr))
|
||||||
|
, env1AllocCache(std::allocate_shared<void *>(traceable_allocator<void *>(), nullptr))
|
||||||
#else
|
#else
|
||||||
, valueAllocCache(std::make_shared<void *>(nullptr))
|
, valueAllocCache(std::make_shared<void *>(nullptr))
|
||||||
|
, env1AllocCache(std::make_shared<void *>(nullptr))
|
||||||
#endif
|
#endif
|
||||||
, baseEnv(allocEnv(128))
|
, baseEnv(allocEnv(128))
|
||||||
, staticBaseEnv(false, 0)
|
, staticBaseEnv(false, 0)
|
||||||
|
@ -702,9 +714,18 @@ LocalNoInlineNoReturn(void throwTypeErrorWithTrace(const Pos & pos, const char *
|
||||||
}).addTrace(p2, s3);
|
}).addTrace(p2, s3);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
LocalNoInlineNoReturn(void throwEvalError(const Pos & pos, const Suggestions & suggestions, const char * s, const std::string & s2))
|
||||||
|
{
|
||||||
|
throw EvalError(ErrorInfo {
|
||||||
|
.msg = hintfmt(s, s2),
|
||||||
|
.errPos = pos,
|
||||||
|
.suggestions = suggestions,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
LocalNoInlineNoReturn(void throwEvalError(const Pos & pos, const char * s, const std::string & s2))
|
LocalNoInlineNoReturn(void throwEvalError(const Pos & pos, const char * s, const std::string & s2))
|
||||||
{
|
{
|
||||||
throw EvalError({
|
throw EvalError(ErrorInfo {
|
||||||
.msg = hintfmt(s, s2),
|
.msg = hintfmt(s, s2),
|
||||||
.errPos = pos
|
.errPos = pos
|
||||||
});
|
});
|
||||||
|
@ -727,6 +748,37 @@ LocalNoInlineNoReturn(void throwEvalError(const Pos & p1, const char * s, const
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
LocalNoInlineNoReturn(void throwTypeError(const Pos & pos, const char * s))
|
||||||
|
{
|
||||||
|
throw TypeError({
|
||||||
|
.msg = hintfmt(s),
|
||||||
|
.errPos = pos
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
LocalNoInlineNoReturn(void throwTypeError(const Pos & pos, const char * s, const ExprLambda & fun, const Symbol & s2))
|
||||||
|
{
|
||||||
|
throw TypeError({
|
||||||
|
.msg = hintfmt(s, fun.showNamePos(), s2),
|
||||||
|
.errPos = pos
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
LocalNoInlineNoReturn(void throwTypeError(const Pos & pos, const Suggestions & suggestions, const char * s, const ExprLambda & fun, const Symbol & s2))
|
||||||
|
{
|
||||||
|
throw TypeError(ErrorInfo {
|
||||||
|
.msg = hintfmt(s, fun.showNamePos(), s2),
|
||||||
|
.errPos = pos,
|
||||||
|
.suggestions = suggestions,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
LocalNoInlineNoReturn(void throwTypeError(const char * s, const Value & v))
|
||||||
|
{
|
||||||
|
throw TypeError(s, showType(v));
|
||||||
|
}
|
||||||
|
|
||||||
LocalNoInlineNoReturn(void throwAssertionError(const Pos & pos, const char * s, const std::string & s1))
|
LocalNoInlineNoReturn(void throwAssertionError(const Pos & pos, const char * s, const std::string & s1))
|
||||||
{
|
{
|
||||||
throw AssertionError({
|
throw AssertionError({
|
||||||
|
@ -764,7 +816,7 @@ LocalNoInline(void addErrorTrace(Error & e, const Pos & pos, const char * s, con
|
||||||
|
|
||||||
void Value::mkString(std::string_view s)
|
void Value::mkString(std::string_view s)
|
||||||
{
|
{
|
||||||
mkString(dupStringWithLen(s.data(), s.size()));
|
mkString(makeImmutableString(s));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@ -795,7 +847,7 @@ void Value::mkStringMove(const char * s, const PathSet & context)
|
||||||
|
|
||||||
void Value::mkPath(std::string_view s)
|
void Value::mkPath(std::string_view s)
|
||||||
{
|
{
|
||||||
mkPath(dupStringWithLen(s.data(), s.size()));
|
mkPath(makeImmutableString(s));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@ -825,42 +877,6 @@ inline Value * EvalState::lookupVar(Env * env, const ExprVar & var, bool noEval)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
Value * EvalState::allocValue()
|
|
||||||
{
|
|
||||||
/* We use the boehm batch allocator to speed up allocations of Values (of which there are many).
|
|
||||||
GC_malloc_many returns a linked list of objects of the given size, where the first word
|
|
||||||
of each object is also the pointer to the next object in the list. This also means that we
|
|
||||||
have to explicitly clear the first word of every object we take. */
|
|
||||||
if (!*valueAllocCache) {
|
|
||||||
*valueAllocCache = GC_malloc_many(sizeof(Value));
|
|
||||||
if (!*valueAllocCache) throw std::bad_alloc();
|
|
||||||
}
|
|
||||||
|
|
||||||
/* GC_NEXT is a convenience macro for accessing the first word of an object.
|
|
||||||
Take the first list item, advance the list to the next item, and clear the next pointer. */
|
|
||||||
void * p = *valueAllocCache;
|
|
||||||
GC_PTR_STORE_AND_DIRTY(&*valueAllocCache, GC_NEXT(p));
|
|
||||||
GC_NEXT(p) = nullptr;
|
|
||||||
|
|
||||||
nrValues++;
|
|
||||||
auto v = (Value *) p;
|
|
||||||
return v;
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
Env & EvalState::allocEnv(size_t size)
|
|
||||||
{
|
|
||||||
nrEnvs++;
|
|
||||||
nrValuesInEnvs += size;
|
|
||||||
Env * env = (Env *) allocBytes(sizeof(Env) + size * sizeof(Value *));
|
|
||||||
env->type = Env::Plain;
|
|
||||||
|
|
||||||
/* We assume that env->values has been cleared by the allocator; maybeThunk() and lookupVar fromWith expect this. */
|
|
||||||
|
|
||||||
return *env;
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
void EvalState::mkList(Value & v, size_t size)
|
void EvalState::mkList(Value & v, size_t size)
|
||||||
{
|
{
|
||||||
v.mkList(size);
|
v.mkList(size);
|
||||||
|
@ -1230,8 +1246,15 @@ void ExprSelect::eval(EvalState & state, Env & env, Value & v)
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
state.forceAttrs(*vAttrs, pos, "While selecting an attribute");
|
state.forceAttrs(*vAttrs, pos, "While selecting an attribute");
|
||||||
if ((j = vAttrs->attrs->find(name)) == vAttrs->attrs->end())
|
if ((j = vAttrs->attrs->find(name)) == vAttrs->attrs->end()) {
|
||||||
throwEvalError(pos, "attribute '%1%' missing", name);
|
std::set<std::string> allAttrNames;
|
||||||
|
for (auto & attr : *vAttrs->attrs)
|
||||||
|
allAttrNames.insert(attr.name);
|
||||||
|
throwEvalError(
|
||||||
|
pos,
|
||||||
|
Suggestions::bestMatches(allAttrNames, name),
|
||||||
|
"attribute '%1%' missing", name);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
vAttrs = j->value;
|
vAttrs = j->value;
|
||||||
pos2 = j->pos;
|
pos2 = j->pos;
|
||||||
|
@ -1360,7 +1383,11 @@ void EvalState::callFunction(Value & fun, size_t nrArgs, Value * * args, Value &
|
||||||
user. */
|
user. */
|
||||||
for (auto & i : *args[0]->attrs)
|
for (auto & i : *args[0]->attrs)
|
||||||
if (!lambda.formals->has(i.name)) {
|
if (!lambda.formals->has(i.name)) {
|
||||||
|
std::set<std::string> formalNames;
|
||||||
|
for (auto & formal : lambda.formals->formals)
|
||||||
|
formalNames.insert(formal.name);
|
||||||
throwTypeErrorWithTrace(lambda.pos,
|
throwTypeErrorWithTrace(lambda.pos,
|
||||||
|
Suggestions::bestMatches(formalNames, i.name),
|
||||||
"Function '%1%' called with unexpected argument '%2%'", prettyLambdaName(lambda), i.name,
|
"Function '%1%' called with unexpected argument '%2%'", prettyLambdaName(lambda), i.name,
|
||||||
pos, "from call site");
|
pos, "from call site");
|
||||||
}
|
}
|
||||||
|
|
|
@ -133,9 +133,14 @@ private:
|
||||||
/* Cache used by prim_match(). */
|
/* Cache used by prim_match(). */
|
||||||
std::shared_ptr<RegexCache> regexCache;
|
std::shared_ptr<RegexCache> regexCache;
|
||||||
|
|
||||||
|
#if HAVE_BOEHMGC
|
||||||
/* Allocation cache for GC'd Value objects. */
|
/* Allocation cache for GC'd Value objects. */
|
||||||
std::shared_ptr<void *> valueAllocCache;
|
std::shared_ptr<void *> valueAllocCache;
|
||||||
|
|
||||||
|
/* Allocation cache for size-1 Env objects. */
|
||||||
|
std::shared_ptr<void *> env1AllocCache;
|
||||||
|
#endif
|
||||||
|
|
||||||
public:
|
public:
|
||||||
|
|
||||||
EvalState(
|
EvalState(
|
||||||
|
@ -347,8 +352,8 @@ public:
|
||||||
void autoCallFunction(Bindings & args, Value & fun, Value & res);
|
void autoCallFunction(Bindings & args, Value & fun, Value & res);
|
||||||
|
|
||||||
/* Allocation primitives. */
|
/* Allocation primitives. */
|
||||||
Value * allocValue();
|
inline Value * allocValue();
|
||||||
Env & allocEnv(size_t size);
|
inline Env & allocEnv(size_t size);
|
||||||
|
|
||||||
Value * allocAttr(Value & vAttrs, const Symbol & name);
|
Value * allocAttr(Value & vAttrs, const Symbol & name);
|
||||||
Value * allocAttr(Value & vAttrs, std::string_view name);
|
Value * allocAttr(Value & vAttrs, std::string_view name);
|
||||||
|
@ -509,3 +514,5 @@ extern EvalSettings evalSettings;
|
||||||
static const std::string corepkgsPrefix{"/__corepkgs__/"};
|
static const std::string corepkgsPrefix{"/__corepkgs__/"};
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#include "eval-inline.hh"
|
||||||
|
|
|
@ -102,7 +102,7 @@ StorePath DrvInfo::queryOutPath() const
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
DrvInfo::Outputs DrvInfo::queryOutputs(bool onlyOutputsToInstall)
|
DrvInfo::Outputs DrvInfo::queryOutputs(bool withPaths, bool onlyOutputsToInstall)
|
||||||
{
|
{
|
||||||
if (outputs.empty()) {
|
if (outputs.empty()) {
|
||||||
/* Get the ‘outputs’ list. */
|
/* Get the ‘outputs’ list. */
|
||||||
|
@ -112,9 +112,11 @@ DrvInfo::Outputs DrvInfo::queryOutputs(bool onlyOutputsToInstall)
|
||||||
|
|
||||||
/* For each output... */
|
/* For each output... */
|
||||||
for (auto elem : i->value->listItems()) {
|
for (auto elem : i->value->listItems()) {
|
||||||
|
std::string output(state->forceStringNoCtx(*elem, *i->pos, "While evaluating the name of one output of a DrvInfo"));
|
||||||
|
|
||||||
|
if (withPaths) {
|
||||||
/* Evaluate the corresponding set. */
|
/* Evaluate the corresponding set. */
|
||||||
std::string name(state->forceStringNoCtx(*elem, *i->pos, "While evaluating the name of one output of a DrvInfo"));
|
Bindings::iterator out = attrs->find(state->symbols.create(output));
|
||||||
Bindings::iterator out = attrs->find(state->symbols.create(name));
|
|
||||||
if (out == attrs->end()) continue; // FIXME: throw error?
|
if (out == attrs->end()) continue; // FIXME: throw error?
|
||||||
state->forceAttrs(*out->value, *i->pos, "While evaluating the description of a DrvInfo output");
|
state->forceAttrs(*out->value, *i->pos, "While evaluating the description of a DrvInfo output");
|
||||||
|
|
||||||
|
@ -122,10 +124,12 @@ DrvInfo::Outputs DrvInfo::queryOutputs(bool onlyOutputsToInstall)
|
||||||
Bindings::iterator outPath = out->value->attrs->find(state->sOutPath);
|
Bindings::iterator outPath = out->value->attrs->find(state->sOutPath);
|
||||||
if (outPath == out->value->attrs->end()) continue; // FIXME: throw error?
|
if (outPath == out->value->attrs->end()) continue; // FIXME: throw error?
|
||||||
PathSet context;
|
PathSet context;
|
||||||
outputs.emplace(name, state->coerceToStorePath(*outPath->pos, *outPath->value, context, "While evaluating the outPath of an output path of a DrvInfo"));
|
outputs.emplace(output, state->coerceToStorePath(*outPath->pos, *outPath->value, context, "While evaluating the outPath of an output path of a DrvInfo"));
|
||||||
|
} else
|
||||||
|
outputs.emplace(output, std::nullopt);
|
||||||
}
|
}
|
||||||
} else
|
} else
|
||||||
outputs.emplace("out", queryOutPath());
|
outputs.emplace("out", withPaths ? std::optional{queryOutPath()} : std::nullopt);
|
||||||
}
|
}
|
||||||
if (!onlyOutputsToInstall || !attrs)
|
if (!onlyOutputsToInstall || !attrs)
|
||||||
return outputs;
|
return outputs;
|
||||||
|
|
|
@ -13,7 +13,7 @@ namespace nix {
|
||||||
struct DrvInfo
|
struct DrvInfo
|
||||||
{
|
{
|
||||||
public:
|
public:
|
||||||
typedef std::map<std::string, StorePath> Outputs;
|
typedef std::map<std::string, std::optional<StorePath>> Outputs;
|
||||||
|
|
||||||
private:
|
private:
|
||||||
EvalState * state;
|
EvalState * state;
|
||||||
|
@ -46,8 +46,9 @@ public:
|
||||||
StorePath requireDrvPath() const;
|
StorePath requireDrvPath() const;
|
||||||
StorePath queryOutPath() const;
|
StorePath queryOutPath() const;
|
||||||
std::string queryOutputName() const;
|
std::string queryOutputName() const;
|
||||||
/** Return the list of outputs. The "outputs to install" are determined by `meta.outputsToInstall`. */
|
/** Return the unordered map of output names to (optional) output paths.
|
||||||
Outputs queryOutputs(bool onlyOutputsToInstall = false);
|
* The "outputs to install" are determined by `meta.outputsToInstall`. */
|
||||||
|
Outputs queryOutputs(bool withPaths = true, bool onlyOutputsToInstall = false);
|
||||||
|
|
||||||
StringSet queryMetaNames();
|
StringSet queryMetaNames();
|
||||||
Value * queryMeta(const std::string & name);
|
Value * queryMeta(const std::string & name);
|
||||||
|
|
|
@ -22,14 +22,13 @@ MakeError(RestrictedPathError, Error);
|
||||||
|
|
||||||
struct Pos
|
struct Pos
|
||||||
{
|
{
|
||||||
FileOrigin origin;
|
|
||||||
Symbol file;
|
Symbol file;
|
||||||
unsigned int line, column;
|
uint32_t line;
|
||||||
|
FileOrigin origin:2;
|
||||||
Pos() : origin(foString), line(0), column(0) { }
|
uint32_t column:30;
|
||||||
Pos(FileOrigin origin, const Symbol & file, unsigned int line, unsigned int column)
|
Pos() : line(0), origin(foString), column(0) { };
|
||||||
: origin(origin), file(file), line(line), column(column) { }
|
Pos(FileOrigin origin, const Symbol & file, uint32_t line, uint32_t column)
|
||||||
|
: file(file), line(line), origin(origin), column(column) { };
|
||||||
operator bool() const
|
operator bool() const
|
||||||
{
|
{
|
||||||
return line != 0;
|
return line != 0;
|
||||||
|
|
|
@ -1205,7 +1205,18 @@ static void prim_derivationStrict(EvalState & state, const Pos & pos, Value * *
|
||||||
// hash per output.
|
// hash per output.
|
||||||
auto hashModulo = hashDerivationModulo(*state.store, Derivation(drv), true);
|
auto hashModulo = hashDerivationModulo(*state.store, Derivation(drv), true);
|
||||||
std::visit(overloaded {
|
std::visit(overloaded {
|
||||||
[&](Hash & h) {
|
[&](const DrvHash & drvHash) {
|
||||||
|
auto & h = drvHash.hash;
|
||||||
|
switch (drvHash.kind) {
|
||||||
|
case DrvHash::Kind::Deferred:
|
||||||
|
for (auto & i : outputs) {
|
||||||
|
drv.outputs.insert_or_assign(i,
|
||||||
|
DerivationOutput {
|
||||||
|
.output = DerivationOutputDeferred{},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
case DrvHash::Kind::Regular:
|
||||||
for (auto & i : outputs) {
|
for (auto & i : outputs) {
|
||||||
auto outPath = state.store->makeOutputPath(i, h, drvName);
|
auto outPath = state.store->makeOutputPath(i, h, drvName);
|
||||||
drv.env[i] = state.store->printStorePath(outPath);
|
drv.env[i] = state.store->printStorePath(outPath);
|
||||||
|
@ -1216,21 +1227,15 @@ static void prim_derivationStrict(EvalState & state, const Pos & pos, Value * *
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
break;
|
||||||
|
}
|
||||||
},
|
},
|
||||||
[&](CaOutputHashes &) {
|
[&](const CaOutputHashes &) {
|
||||||
// Shouldn't happen as the toplevel derivation is not CA.
|
// Shouldn't happen as the toplevel derivation is not CA.
|
||||||
assert(false);
|
assert(false);
|
||||||
},
|
},
|
||||||
[&](DeferredHash &) {
|
|
||||||
for (auto & i : outputs) {
|
|
||||||
drv.outputs.insert_or_assign(i,
|
|
||||||
DerivationOutput {
|
|
||||||
.output = DerivationOutputDeferred{},
|
|
||||||
});
|
|
||||||
}
|
|
||||||
},
|
},
|
||||||
},
|
hashModulo.raw());
|
||||||
hashModulo);
|
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
@ -29,7 +29,7 @@ struct FetchSettings : public Config
|
||||||
|
|
||||||
* Github: the token value is the OAUTH-TOKEN string obtained
|
* Github: the token value is the OAUTH-TOKEN string obtained
|
||||||
as the Personal Access Token from the Github server (see
|
as the Personal Access Token from the Github server (see
|
||||||
https://docs.github.com/en/developers/apps/authorizing-oath-apps).
|
https://docs.github.com/en/developers/apps/building-oauth-apps/authorizing-oauth-apps).
|
||||||
|
|
||||||
* Gitlab: the token value is either the OAuth2 token or the
|
* Gitlab: the token value is either the OAuth2 token or the
|
||||||
Personal Access Token (these are different types tokens
|
Personal Access Token (these are different types tokens
|
||||||
|
|
|
@ -222,22 +222,46 @@ struct GitInputScheme : InputScheme
|
||||||
if (!input.getRef() && !input.getRev() && isLocal) {
|
if (!input.getRef() && !input.getRev() && isLocal) {
|
||||||
bool clean = false;
|
bool clean = false;
|
||||||
|
|
||||||
/* Check whether this repo has any commits. There are
|
auto env = getEnv();
|
||||||
probably better ways to do this. */
|
// Set LC_ALL to C: because we rely on the error messages from git rev-parse to determine what went wrong
|
||||||
auto gitDir = actualUrl + "/.git";
|
// that way unknown errors can lead to a failure instead of continuing through the wrong code path
|
||||||
auto commonGitDir = chomp(runProgram(
|
env["LC_ALL"] = "C";
|
||||||
"git",
|
|
||||||
true,
|
|
||||||
{ "-C", actualUrl, "rev-parse", "--git-common-dir" }
|
|
||||||
));
|
|
||||||
if (commonGitDir != ".git")
|
|
||||||
gitDir = commonGitDir;
|
|
||||||
|
|
||||||
bool haveCommits = !readDirectory(gitDir + "/refs/heads").empty();
|
/* Check whether HEAD points to something that looks like a commit,
|
||||||
|
since that is the refrence we want to use later on. */
|
||||||
|
auto result = runProgram(RunOptions {
|
||||||
|
.program = "git",
|
||||||
|
.args = { "-C", actualUrl, "--git-dir=.git", "rev-parse", "--verify", "--no-revs", "HEAD^{commit}" },
|
||||||
|
.environment = env,
|
||||||
|
.mergeStderrToStdout = true
|
||||||
|
});
|
||||||
|
auto exitCode = WEXITSTATUS(result.first);
|
||||||
|
auto errorMessage = result.second;
|
||||||
|
|
||||||
|
if (errorMessage.find("fatal: not a git repository") != std::string::npos) {
|
||||||
|
throw Error("'%s' is not a Git repository", actualUrl);
|
||||||
|
} else if (errorMessage.find("fatal: Needed a single revision") != std::string::npos) {
|
||||||
|
// indicates that the repo does not have any commits
|
||||||
|
// we want to proceed and will consider it dirty later
|
||||||
|
} else if (exitCode != 0) {
|
||||||
|
// any other errors should lead to a failure
|
||||||
|
throw Error("getting the HEAD of the Git tree '%s' failed with exit code %d:\n%s", actualUrl, exitCode, errorMessage);
|
||||||
|
}
|
||||||
|
|
||||||
|
bool hasHead = exitCode == 0;
|
||||||
try {
|
try {
|
||||||
if (haveCommits) {
|
if (hasHead) {
|
||||||
runProgram("git", true, { "-C", actualUrl, "diff-index", "--quiet", "HEAD", "--" });
|
// Using git diff is preferrable over lower-level operations here,
|
||||||
|
// because its conceptually simpler and we only need the exit code anyways.
|
||||||
|
auto gitDiffOpts = Strings({ "-C", actualUrl, "diff", "HEAD", "--quiet"});
|
||||||
|
if (!submodules) {
|
||||||
|
// Changes in submodules should only make the tree dirty
|
||||||
|
// when those submodules will be copied as well.
|
||||||
|
gitDiffOpts.emplace_back("--ignore-submodules");
|
||||||
|
}
|
||||||
|
gitDiffOpts.emplace_back("--");
|
||||||
|
runProgram("git", true, gitDiffOpts);
|
||||||
|
|
||||||
clean = true;
|
clean = true;
|
||||||
}
|
}
|
||||||
} catch (ExecError & e) {
|
} catch (ExecError & e) {
|
||||||
|
@ -282,7 +306,7 @@ struct GitInputScheme : InputScheme
|
||||||
// modified dirty file?
|
// modified dirty file?
|
||||||
input.attrs.insert_or_assign(
|
input.attrs.insert_or_assign(
|
||||||
"lastModified",
|
"lastModified",
|
||||||
haveCommits ? std::stoull(runProgram("git", true, { "-C", actualUrl, "log", "-1", "--format=%ct", "--no-show-signature", "HEAD" })) : 0);
|
hasHead ? std::stoull(runProgram("git", true, { "-C", actualUrl, "log", "-1", "--format=%ct", "--no-show-signature", "HEAD" })) : 0);
|
||||||
|
|
||||||
return {std::move(storePath), input};
|
return {std::move(storePath), input};
|
||||||
}
|
}
|
||||||
|
|
|
@ -1,5 +1,6 @@
|
||||||
#include "fetchers.hh"
|
#include "fetchers.hh"
|
||||||
#include "store-api.hh"
|
#include "store-api.hh"
|
||||||
|
#include "archive.hh"
|
||||||
|
|
||||||
namespace nix::fetchers {
|
namespace nix::fetchers {
|
||||||
|
|
||||||
|
@ -80,8 +81,9 @@ struct PathInputScheme : InputScheme
|
||||||
// nothing to do
|
// nothing to do
|
||||||
}
|
}
|
||||||
|
|
||||||
std::pair<StorePath, Input> fetch(ref<Store> store, const Input & input) override
|
std::pair<StorePath, Input> fetch(ref<Store> store, const Input & _input) override
|
||||||
{
|
{
|
||||||
|
Input input(_input);
|
||||||
std::string absPath;
|
std::string absPath;
|
||||||
auto path = getStrAttr(input.attrs, "path");
|
auto path = getStrAttr(input.attrs, "path");
|
||||||
|
|
||||||
|
@ -111,9 +113,15 @@ struct PathInputScheme : InputScheme
|
||||||
if (storePath)
|
if (storePath)
|
||||||
store->addTempRoot(*storePath);
|
store->addTempRoot(*storePath);
|
||||||
|
|
||||||
if (!storePath || storePath->name() != "source" || !store->isValidPath(*storePath))
|
time_t mtime = 0;
|
||||||
|
if (!storePath || storePath->name() != "source" || !store->isValidPath(*storePath)) {
|
||||||
// FIXME: try to substitute storePath.
|
// FIXME: try to substitute storePath.
|
||||||
storePath = store->addToStore("source", absPath);
|
auto src = sinkToSource([&](Sink & sink) {
|
||||||
|
mtime = dumpPathAndGetMtime(absPath, sink, defaultPathFilter);
|
||||||
|
});
|
||||||
|
storePath = store->addToStoreFromDump(*src, "source");
|
||||||
|
}
|
||||||
|
input.attrs.insert_or_assign("lastModified", uint64_t(mtime));
|
||||||
|
|
||||||
return {std::move(*storePath), input};
|
return {std::move(*storePath), input};
|
||||||
}
|
}
|
||||||
|
|
|
@ -1,6 +1,7 @@
|
||||||
#include "globals.hh"
|
#include "globals.hh"
|
||||||
#include "shared.hh"
|
#include "shared.hh"
|
||||||
#include "store-api.hh"
|
#include "store-api.hh"
|
||||||
|
#include "gc-store.hh"
|
||||||
#include "util.hh"
|
#include "util.hh"
|
||||||
#include "loggers.hh"
|
#include "loggers.hh"
|
||||||
|
|
||||||
|
|
|
@ -2,6 +2,7 @@
|
||||||
|
|
||||||
#include "crypto.hh"
|
#include "crypto.hh"
|
||||||
#include "store-api.hh"
|
#include "store-api.hh"
|
||||||
|
#include "log-store.hh"
|
||||||
|
|
||||||
#include "pool.hh"
|
#include "pool.hh"
|
||||||
|
|
||||||
|
@ -28,7 +29,9 @@ struct BinaryCacheStoreConfig : virtual StoreConfig
|
||||||
"other than -1 which we reserve to indicate Nix defaults should be used"};
|
"other than -1 which we reserve to indicate Nix defaults should be used"};
|
||||||
};
|
};
|
||||||
|
|
||||||
class BinaryCacheStore : public virtual BinaryCacheStoreConfig, public virtual Store
|
class BinaryCacheStore : public virtual BinaryCacheStoreConfig,
|
||||||
|
public virtual Store,
|
||||||
|
public virtual LogStore
|
||||||
{
|
{
|
||||||
|
|
||||||
private:
|
private:
|
||||||
|
|
|
@ -28,6 +28,7 @@ struct BuildResult
|
||||||
LogLimitExceeded,
|
LogLimitExceeded,
|
||||||
NotDeterministic,
|
NotDeterministic,
|
||||||
ResolvesToAlreadyValid,
|
ResolvesToAlreadyValid,
|
||||||
|
NoSubstituters,
|
||||||
} status = MiscFailure;
|
} status = MiscFailure;
|
||||||
std::string errorMsg;
|
std::string errorMsg;
|
||||||
|
|
||||||
|
@ -63,15 +64,26 @@ struct BuildResult
|
||||||
non-determinism.) */
|
non-determinism.) */
|
||||||
bool isNonDeterministic = false;
|
bool isNonDeterministic = false;
|
||||||
|
|
||||||
|
/* The derivation we built or the store path we substituted. */
|
||||||
|
DerivedPath path;
|
||||||
|
|
||||||
|
/* For derivations, a mapping from the names of the wanted outputs
|
||||||
|
to actual paths. */
|
||||||
DrvOutputs builtOutputs;
|
DrvOutputs builtOutputs;
|
||||||
|
|
||||||
/* The start/stop times of the build (or one of the rounds, if it
|
/* The start/stop times of the build (or one of the rounds, if it
|
||||||
was repeated). */
|
was repeated). */
|
||||||
time_t startTime = 0, stopTime = 0;
|
time_t startTime = 0, stopTime = 0;
|
||||||
|
|
||||||
bool success() {
|
bool success()
|
||||||
|
{
|
||||||
return status == Built || status == Substituted || status == AlreadyValid || status == ResolvesToAlreadyValid;
|
return status == Built || status == Substituted || status == AlreadyValid || status == ResolvesToAlreadyValid;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
void rethrow()
|
||||||
|
{
|
||||||
|
throw Error("%s", errorMsg);
|
||||||
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
|
@ -66,7 +66,7 @@ namespace nix {
|
||||||
|
|
||||||
DerivationGoal::DerivationGoal(const StorePath & drvPath,
|
DerivationGoal::DerivationGoal(const StorePath & drvPath,
|
||||||
const StringSet & wantedOutputs, Worker & worker, BuildMode buildMode)
|
const StringSet & wantedOutputs, Worker & worker, BuildMode buildMode)
|
||||||
: Goal(worker)
|
: Goal(worker, DerivedPath::Built { .drvPath = drvPath, .outputs = wantedOutputs })
|
||||||
, useDerivation(true)
|
, useDerivation(true)
|
||||||
, drvPath(drvPath)
|
, drvPath(drvPath)
|
||||||
, wantedOutputs(wantedOutputs)
|
, wantedOutputs(wantedOutputs)
|
||||||
|
@ -85,7 +85,7 @@ DerivationGoal::DerivationGoal(const StorePath & drvPath,
|
||||||
|
|
||||||
DerivationGoal::DerivationGoal(const StorePath & drvPath, const BasicDerivation & drv,
|
DerivationGoal::DerivationGoal(const StorePath & drvPath, const BasicDerivation & drv,
|
||||||
const StringSet & wantedOutputs, Worker & worker, BuildMode buildMode)
|
const StringSet & wantedOutputs, Worker & worker, BuildMode buildMode)
|
||||||
: Goal(worker)
|
: Goal(worker, DerivedPath::Built { .drvPath = drvPath, .outputs = wantedOutputs })
|
||||||
, useDerivation(false)
|
, useDerivation(false)
|
||||||
, drvPath(drvPath)
|
, drvPath(drvPath)
|
||||||
, wantedOutputs(wantedOutputs)
|
, wantedOutputs(wantedOutputs)
|
||||||
|
@ -135,7 +135,7 @@ void DerivationGoal::killChild()
|
||||||
void DerivationGoal::timedOut(Error && ex)
|
void DerivationGoal::timedOut(Error && ex)
|
||||||
{
|
{
|
||||||
killChild();
|
killChild();
|
||||||
done(BuildResult::TimedOut, ex);
|
done(BuildResult::TimedOut, {}, ex);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@ -182,7 +182,7 @@ void DerivationGoal::loadDerivation()
|
||||||
trace("loading derivation");
|
trace("loading derivation");
|
||||||
|
|
||||||
if (nrFailed != 0) {
|
if (nrFailed != 0) {
|
||||||
done(BuildResult::MiscFailure, Error("cannot build missing derivation '%s'", worker.store.printStorePath(drvPath)));
|
done(BuildResult::MiscFailure, {}, Error("cannot build missing derivation '%s'", worker.store.printStorePath(drvPath)));
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -217,26 +217,18 @@ void DerivationGoal::haveDerivation()
|
||||||
for (auto & [outputName, outputHash] : outputHashes)
|
for (auto & [outputName, outputHash] : outputHashes)
|
||||||
initialOutputs.insert({
|
initialOutputs.insert({
|
||||||
outputName,
|
outputName,
|
||||||
InitialOutput{
|
InitialOutput {
|
||||||
.wanted = true, // Will be refined later
|
.wanted = true, // Will be refined later
|
||||||
.outputHash = outputHash
|
.outputHash = outputHash
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
/* Check what outputs paths are not already valid. */
|
/* Check what outputs paths are not already valid. */
|
||||||
checkPathValidity();
|
auto [allValid, validOutputs] = checkPathValidity();
|
||||||
bool allValid = true;
|
|
||||||
for (auto & [_, status] : initialOutputs) {
|
|
||||||
if (!status.wanted) continue;
|
|
||||||
if (!status.known || !status.known->isValid()) {
|
|
||||||
allValid = false;
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/* If they are all valid, then we're done. */
|
/* If they are all valid, then we're done. */
|
||||||
if (allValid && buildMode == bmNormal) {
|
if (allValid && buildMode == bmNormal) {
|
||||||
done(BuildResult::AlreadyValid);
|
done(BuildResult::AlreadyValid, std::move(validOutputs));
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -277,7 +269,7 @@ void DerivationGoal::outputsSubstitutionTried()
|
||||||
trace("all outputs substituted (maybe)");
|
trace("all outputs substituted (maybe)");
|
||||||
|
|
||||||
if (nrFailed > 0 && nrFailed > nrNoSubstituters + nrIncompleteClosure && !settings.tryFallback) {
|
if (nrFailed > 0 && nrFailed > nrNoSubstituters + nrIncompleteClosure && !settings.tryFallback) {
|
||||||
done(BuildResult::TransientFailure,
|
done(BuildResult::TransientFailure, {},
|
||||||
Error("some substitutes for the outputs of derivation '%s' failed (usually happens due to networking issues); try '--fallback' to build derivation from source ",
|
Error("some substitutes for the outputs of derivation '%s' failed (usually happens due to networking issues); try '--fallback' to build derivation from source ",
|
||||||
worker.store.printStorePath(drvPath)));
|
worker.store.printStorePath(drvPath)));
|
||||||
return;
|
return;
|
||||||
|
@ -301,23 +293,17 @@ void DerivationGoal::outputsSubstitutionTried()
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
checkPathValidity();
|
auto [allValid, validOutputs] = checkPathValidity();
|
||||||
size_t nrInvalid = 0;
|
|
||||||
for (auto & [_, status] : initialOutputs) {
|
|
||||||
if (!status.wanted) continue;
|
|
||||||
if (!status.known || !status.known->isValid())
|
|
||||||
nrInvalid++;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (buildMode == bmNormal && nrInvalid == 0) {
|
if (buildMode == bmNormal && allValid) {
|
||||||
done(BuildResult::Substituted);
|
done(BuildResult::Substituted, std::move(validOutputs));
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
if (buildMode == bmRepair && nrInvalid == 0) {
|
if (buildMode == bmRepair && allValid) {
|
||||||
repairClosure();
|
repairClosure();
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
if (buildMode == bmCheck && nrInvalid > 0)
|
if (buildMode == bmCheck && !allValid)
|
||||||
throw Error("some outputs of '%s' are not valid, so checking is not possible",
|
throw Error("some outputs of '%s' are not valid, so checking is not possible",
|
||||||
worker.store.printStorePath(drvPath));
|
worker.store.printStorePath(drvPath));
|
||||||
|
|
||||||
|
@ -409,7 +395,7 @@ void DerivationGoal::repairClosure()
|
||||||
}
|
}
|
||||||
|
|
||||||
if (waitees.empty()) {
|
if (waitees.empty()) {
|
||||||
done(BuildResult::AlreadyValid);
|
done(BuildResult::AlreadyValid, assertPathValidity());
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -423,7 +409,7 @@ void DerivationGoal::closureRepaired()
|
||||||
if (nrFailed > 0)
|
if (nrFailed > 0)
|
||||||
throw Error("some paths in the output closure of derivation '%s' could not be repaired",
|
throw Error("some paths in the output closure of derivation '%s' could not be repaired",
|
||||||
worker.store.printStorePath(drvPath));
|
worker.store.printStorePath(drvPath));
|
||||||
done(BuildResult::AlreadyValid);
|
done(BuildResult::AlreadyValid, assertPathValidity());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@ -434,7 +420,7 @@ void DerivationGoal::inputsRealised()
|
||||||
if (nrFailed != 0) {
|
if (nrFailed != 0) {
|
||||||
if (!useDerivation)
|
if (!useDerivation)
|
||||||
throw Error("some dependencies of '%s' are missing", worker.store.printStorePath(drvPath));
|
throw Error("some dependencies of '%s' are missing", worker.store.printStorePath(drvPath));
|
||||||
done(BuildResult::DependencyFailed, Error(
|
done(BuildResult::DependencyFailed, {}, Error(
|
||||||
"%s dependencies of derivation '%s' failed to build",
|
"%s dependencies of derivation '%s' failed to build",
|
||||||
nrFailed, worker.store.printStorePath(drvPath)));
|
nrFailed, worker.store.printStorePath(drvPath)));
|
||||||
return;
|
return;
|
||||||
|
@ -523,10 +509,11 @@ void DerivationGoal::inputsRealised()
|
||||||
state = &DerivationGoal::tryToBuild;
|
state = &DerivationGoal::tryToBuild;
|
||||||
worker.wakeUp(shared_from_this());
|
worker.wakeUp(shared_from_this());
|
||||||
|
|
||||||
result = BuildResult();
|
buildResult = BuildResult { .path = buildResult.path };
|
||||||
}
|
}
|
||||||
|
|
||||||
void DerivationGoal::started() {
|
void DerivationGoal::started()
|
||||||
|
{
|
||||||
auto msg = fmt(
|
auto msg = fmt(
|
||||||
buildMode == bmRepair ? "repairing outputs of '%s'" :
|
buildMode == bmRepair ? "repairing outputs of '%s'" :
|
||||||
buildMode == bmCheck ? "checking outputs of '%s'" :
|
buildMode == bmCheck ? "checking outputs of '%s'" :
|
||||||
|
@ -588,19 +575,12 @@ void DerivationGoal::tryToBuild()
|
||||||
omitted, but that would be less efficient.) Note that since we
|
omitted, but that would be less efficient.) Note that since we
|
||||||
now hold the locks on the output paths, no other process can
|
now hold the locks on the output paths, no other process can
|
||||||
build this derivation, so no further checks are necessary. */
|
build this derivation, so no further checks are necessary. */
|
||||||
checkPathValidity();
|
auto [allValid, validOutputs] = checkPathValidity();
|
||||||
bool allValid = true;
|
|
||||||
for (auto & [_, status] : initialOutputs) {
|
|
||||||
if (!status.wanted) continue;
|
|
||||||
if (!status.known || !status.known->isValid()) {
|
|
||||||
allValid = false;
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if (buildMode != bmCheck && allValid) {
|
if (buildMode != bmCheck && allValid) {
|
||||||
debug("skipping build of derivation '%s', someone beat us to it", worker.store.printStorePath(drvPath));
|
debug("skipping build of derivation '%s', someone beat us to it", worker.store.printStorePath(drvPath));
|
||||||
outputLocks.setDeletion(true);
|
outputLocks.setDeletion(true);
|
||||||
done(BuildResult::AlreadyValid);
|
done(BuildResult::AlreadyValid, std::move(validOutputs));
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -626,7 +606,7 @@ void DerivationGoal::tryToBuild()
|
||||||
/* Yes, it has started doing so. Wait until we get
|
/* Yes, it has started doing so. Wait until we get
|
||||||
EOF from the hook. */
|
EOF from the hook. */
|
||||||
actLock.reset();
|
actLock.reset();
|
||||||
result.startTime = time(0); // inexact
|
buildResult.startTime = time(0); // inexact
|
||||||
state = &DerivationGoal::buildDone;
|
state = &DerivationGoal::buildDone;
|
||||||
started();
|
started();
|
||||||
return;
|
return;
|
||||||
|
@ -830,8 +810,8 @@ void DerivationGoal::buildDone()
|
||||||
|
|
||||||
debug("builder process for '%s' finished", worker.store.printStorePath(drvPath));
|
debug("builder process for '%s' finished", worker.store.printStorePath(drvPath));
|
||||||
|
|
||||||
result.timesBuilt++;
|
buildResult.timesBuilt++;
|
||||||
result.stopTime = time(0);
|
buildResult.stopTime = time(0);
|
||||||
|
|
||||||
/* So the child is gone now. */
|
/* So the child is gone now. */
|
||||||
worker.childTerminated(this);
|
worker.childTerminated(this);
|
||||||
|
@ -876,11 +856,11 @@ void DerivationGoal::buildDone()
|
||||||
|
|
||||||
/* Compute the FS closure of the outputs and register them as
|
/* Compute the FS closure of the outputs and register them as
|
||||||
being valid. */
|
being valid. */
|
||||||
registerOutputs();
|
auto builtOutputs = registerOutputs();
|
||||||
|
|
||||||
StorePathSet outputPaths;
|
StorePathSet outputPaths;
|
||||||
for (auto & [_, path] : finalOutputs)
|
for (auto & [_, output] : buildResult.builtOutputs)
|
||||||
outputPaths.insert(path);
|
outputPaths.insert(output.outPath);
|
||||||
runPostBuildHook(
|
runPostBuildHook(
|
||||||
worker.store,
|
worker.store,
|
||||||
*logger,
|
*logger,
|
||||||
|
@ -890,7 +870,7 @@ void DerivationGoal::buildDone()
|
||||||
|
|
||||||
if (buildMode == bmCheck) {
|
if (buildMode == bmCheck) {
|
||||||
cleanupPostOutputsRegisteredModeCheck();
|
cleanupPostOutputsRegisteredModeCheck();
|
||||||
done(BuildResult::Built);
|
done(BuildResult::Built, std::move(builtOutputs));
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -911,6 +891,8 @@ void DerivationGoal::buildDone()
|
||||||
outputLocks.setDeletion(true);
|
outputLocks.setDeletion(true);
|
||||||
outputLocks.unlock();
|
outputLocks.unlock();
|
||||||
|
|
||||||
|
done(BuildResult::Built, std::move(builtOutputs));
|
||||||
|
|
||||||
} catch (BuildError & e) {
|
} catch (BuildError & e) {
|
||||||
outputLocks.unlock();
|
outputLocks.unlock();
|
||||||
|
|
||||||
|
@ -930,14 +912,13 @@ void DerivationGoal::buildDone()
|
||||||
BuildResult::PermanentFailure;
|
BuildResult::PermanentFailure;
|
||||||
}
|
}
|
||||||
|
|
||||||
done(st, e);
|
done(st, {}, e);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
done(BuildResult::Built);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
void DerivationGoal::resolvedFinished() {
|
void DerivationGoal::resolvedFinished()
|
||||||
|
{
|
||||||
assert(resolvedDrvGoal);
|
assert(resolvedDrvGoal);
|
||||||
auto resolvedDrv = *resolvedDrvGoal->drv;
|
auto resolvedDrv = *resolvedDrvGoal->drv;
|
||||||
|
|
||||||
|
@ -950,6 +931,8 @@ void DerivationGoal::resolvedFinished() {
|
||||||
if (realWantedOutputs.empty())
|
if (realWantedOutputs.empty())
|
||||||
realWantedOutputs = resolvedDrv.outputNames();
|
realWantedOutputs = resolvedDrv.outputNames();
|
||||||
|
|
||||||
|
DrvOutputs builtOutputs;
|
||||||
|
|
||||||
for (auto & wantedOutput : realWantedOutputs) {
|
for (auto & wantedOutput : realWantedOutputs) {
|
||||||
assert(initialOutputs.count(wantedOutput) != 0);
|
assert(initialOutputs.count(wantedOutput) != 0);
|
||||||
assert(resolvedHashes.count(wantedOutput) != 0);
|
assert(resolvedHashes.count(wantedOutput) != 0);
|
||||||
|
@ -966,10 +949,11 @@ void DerivationGoal::resolvedFinished() {
|
||||||
signRealisation(newRealisation);
|
signRealisation(newRealisation);
|
||||||
worker.store.registerDrvOutput(newRealisation);
|
worker.store.registerDrvOutput(newRealisation);
|
||||||
outputPaths.insert(realisation->outPath);
|
outputPaths.insert(realisation->outPath);
|
||||||
|
builtOutputs.emplace(realisation->id, *realisation);
|
||||||
} else {
|
} else {
|
||||||
// If we don't have a realisation, then it must mean that something
|
// If we don't have a realisation, then it must mean that something
|
||||||
// failed when building the resolved drv
|
// failed when building the resolved drv
|
||||||
assert(!result.success());
|
assert(!buildResult.success());
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -981,7 +965,7 @@ void DerivationGoal::resolvedFinished() {
|
||||||
);
|
);
|
||||||
|
|
||||||
auto status = [&]() {
|
auto status = [&]() {
|
||||||
auto resolvedResult = resolvedDrvGoal->getResult();
|
auto & resolvedResult = resolvedDrvGoal->buildResult;
|
||||||
switch (resolvedResult.status) {
|
switch (resolvedResult.status) {
|
||||||
case BuildResult::AlreadyValid:
|
case BuildResult::AlreadyValid:
|
||||||
return BuildResult::ResolvesToAlreadyValid;
|
return BuildResult::ResolvesToAlreadyValid;
|
||||||
|
@ -990,7 +974,7 @@ void DerivationGoal::resolvedFinished() {
|
||||||
}
|
}
|
||||||
}();
|
}();
|
||||||
|
|
||||||
done(status);
|
done(status, std::move(builtOutputs));
|
||||||
}
|
}
|
||||||
|
|
||||||
HookReply DerivationGoal::tryBuildHook()
|
HookReply DerivationGoal::tryBuildHook()
|
||||||
|
@ -1100,7 +1084,7 @@ HookReply DerivationGoal::tryBuildHook()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
void DerivationGoal::registerOutputs()
|
DrvOutputs DerivationGoal::registerOutputs()
|
||||||
{
|
{
|
||||||
/* When using a build hook, the build hook can register the output
|
/* When using a build hook, the build hook can register the output
|
||||||
as valid (by doing `nix-store --import'). If so we don't have
|
as valid (by doing `nix-store --import'). If so we don't have
|
||||||
|
@ -1109,21 +1093,7 @@ void DerivationGoal::registerOutputs()
|
||||||
We can only early return when the outputs are known a priori. For
|
We can only early return when the outputs are known a priori. For
|
||||||
floating content-addressed derivations this isn't the case.
|
floating content-addressed derivations this isn't the case.
|
||||||
*/
|
*/
|
||||||
for (auto & [outputName, optOutputPath] : worker.store.queryPartialDerivationOutputMap(drvPath)) {
|
return assertPathValidity();
|
||||||
if (!wantOutput(outputName, wantedOutputs))
|
|
||||||
continue;
|
|
||||||
if (!optOutputPath)
|
|
||||||
throw BuildError(
|
|
||||||
"output '%s' from derivation '%s' does not have a known output path",
|
|
||||||
outputName, worker.store.printStorePath(drvPath));
|
|
||||||
auto & outputPath = *optOutputPath;
|
|
||||||
if (!worker.store.isValidPath(outputPath))
|
|
||||||
throw BuildError(
|
|
||||||
"output '%s' from derivation '%s' is supposed to be at '%s' but that path is not valid",
|
|
||||||
outputName, worker.store.printStorePath(drvPath), worker.store.printStorePath(outputPath));
|
|
||||||
|
|
||||||
finalOutputs.insert_or_assign(outputName, outputPath);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
Path DerivationGoal::openLogFile()
|
Path DerivationGoal::openLogFile()
|
||||||
|
@ -1175,16 +1145,17 @@ bool DerivationGoal::isReadDesc(int fd)
|
||||||
return fd == hook->builderOut.readSide.get();
|
return fd == hook->builderOut.readSide.get();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
void DerivationGoal::handleChildOutput(int fd, std::string_view data)
|
void DerivationGoal::handleChildOutput(int fd, std::string_view data)
|
||||||
{
|
{
|
||||||
if (isReadDesc(fd))
|
// local & `ssh://`-builds are dealt with here.
|
||||||
|
auto isWrittenToLog = isReadDesc(fd);
|
||||||
|
if (isWrittenToLog)
|
||||||
{
|
{
|
||||||
logSize += data.size();
|
logSize += data.size();
|
||||||
if (settings.maxLogSize && logSize > settings.maxLogSize) {
|
if (settings.maxLogSize && logSize > settings.maxLogSize) {
|
||||||
killChild();
|
killChild();
|
||||||
done(
|
done(
|
||||||
BuildResult::LogLimitExceeded,
|
BuildResult::LogLimitExceeded, {},
|
||||||
Error("%s killed after writing more than %d bytes of log output",
|
Error("%s killed after writing more than %d bytes of log output",
|
||||||
getName(), settings.maxLogSize));
|
getName(), settings.maxLogSize));
|
||||||
return;
|
return;
|
||||||
|
@ -1207,7 +1178,16 @@ void DerivationGoal::handleChildOutput(int fd, std::string_view data)
|
||||||
if (hook && fd == hook->fromHook.readSide.get()) {
|
if (hook && fd == hook->fromHook.readSide.get()) {
|
||||||
for (auto c : data)
|
for (auto c : data)
|
||||||
if (c == '\n') {
|
if (c == '\n') {
|
||||||
handleJSONLogMessage(currentHookLine, worker.act, hook->activities, true);
|
auto json = parseJSONMessage(currentHookLine);
|
||||||
|
if (json) {
|
||||||
|
auto s = handleJSONLogMessage(*json, worker.act, hook->activities, true);
|
||||||
|
// ensure that logs from a builder using `ssh-ng://` as protocol
|
||||||
|
// are also available to `nix log`.
|
||||||
|
if (s && !isWrittenToLog && logSink && (*json)["type"] == resBuildLogLine) {
|
||||||
|
auto f = (*json)["fields"];
|
||||||
|
(*logSink)((f.size() > 0 ? f.at(0).get<std::string>() : "") + "\n");
|
||||||
|
}
|
||||||
|
}
|
||||||
currentHookLine.clear();
|
currentHookLine.clear();
|
||||||
} else
|
} else
|
||||||
currentHookLine += c;
|
currentHookLine += c;
|
||||||
|
@ -1264,10 +1244,12 @@ OutputPathMap DerivationGoal::queryDerivationOutputMap()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
void DerivationGoal::checkPathValidity()
|
std::pair<bool, DrvOutputs> DerivationGoal::checkPathValidity()
|
||||||
{
|
{
|
||||||
bool checkHash = buildMode == bmRepair;
|
bool checkHash = buildMode == bmRepair;
|
||||||
auto wantedOutputsLeft = wantedOutputs;
|
auto wantedOutputsLeft = wantedOutputs;
|
||||||
|
DrvOutputs validOutputs;
|
||||||
|
|
||||||
for (auto & i : queryPartialDerivationOutputMap()) {
|
for (auto & i : queryPartialDerivationOutputMap()) {
|
||||||
InitialOutput & info = initialOutputs.at(i.first);
|
InitialOutput & info = initialOutputs.at(i.first);
|
||||||
info.wanted = wantOutput(i.first, wantedOutputs);
|
info.wanted = wantOutput(i.first, wantedOutputs);
|
||||||
|
@ -1284,26 +1266,28 @@ void DerivationGoal::checkPathValidity()
|
||||||
: PathStatus::Corrupt,
|
: PathStatus::Corrupt,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
if (settings.isExperimentalFeatureEnabled(Xp::CaDerivations)) {
|
|
||||||
auto drvOutput = DrvOutput{initialOutputs.at(i.first).outputHash, i.first};
|
auto drvOutput = DrvOutput{initialOutputs.at(i.first).outputHash, i.first};
|
||||||
|
if (settings.isExperimentalFeatureEnabled(Xp::CaDerivations)) {
|
||||||
if (auto real = worker.store.queryRealisation(drvOutput)) {
|
if (auto real = worker.store.queryRealisation(drvOutput)) {
|
||||||
info.known = {
|
info.known = {
|
||||||
.path = real->outPath,
|
.path = real->outPath,
|
||||||
.status = PathStatus::Valid,
|
.status = PathStatus::Valid,
|
||||||
};
|
};
|
||||||
} else if (info.known && info.known->status == PathStatus::Valid) {
|
} else if (info.known && info.known->isValid()) {
|
||||||
// We know the output because it' a static output of the
|
// We know the output because it's a static output of the
|
||||||
// derivation, and the output path is valid, but we don't have
|
// derivation, and the output path is valid, but we don't have
|
||||||
// its realisation stored (probably because it has been built
|
// its realisation stored (probably because it has been built
|
||||||
// without the `ca-derivations` experimental flag)
|
// without the `ca-derivations` experimental flag).
|
||||||
worker.store.registerDrvOutput(
|
worker.store.registerDrvOutput(
|
||||||
Realisation{
|
Realisation {
|
||||||
drvOutput,
|
drvOutput,
|
||||||
info.known->path,
|
info.known->path,
|
||||||
}
|
}
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if (info.wanted && info.known && info.known->isValid())
|
||||||
|
validOutputs.emplace(drvOutput, Realisation { drvOutput, info.known->path });
|
||||||
}
|
}
|
||||||
// If we requested all the outputs via the empty set, we are always fine.
|
// If we requested all the outputs via the empty set, we are always fine.
|
||||||
// If we requested specific elements, the loop above removes all the valid
|
// If we requested specific elements, the loop above removes all the valid
|
||||||
|
@ -1312,24 +1296,50 @@ void DerivationGoal::checkPathValidity()
|
||||||
throw Error("derivation '%s' does not have wanted outputs %s",
|
throw Error("derivation '%s' does not have wanted outputs %s",
|
||||||
worker.store.printStorePath(drvPath),
|
worker.store.printStorePath(drvPath),
|
||||||
concatStringsSep(", ", quoteStrings(wantedOutputsLeft)));
|
concatStringsSep(", ", quoteStrings(wantedOutputsLeft)));
|
||||||
|
|
||||||
|
bool allValid = true;
|
||||||
|
for (auto & [_, status] : initialOutputs) {
|
||||||
|
if (!status.wanted) continue;
|
||||||
|
if (!status.known || !status.known->isValid()) {
|
||||||
|
allValid = false;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return { allValid, validOutputs };
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
void DerivationGoal::done(BuildResult::Status status, std::optional<Error> ex)
|
DrvOutputs DerivationGoal::assertPathValidity()
|
||||||
{
|
{
|
||||||
result.status = status;
|
auto [allValid, validOutputs] = checkPathValidity();
|
||||||
|
if (!allValid)
|
||||||
|
throw Error("some outputs are unexpectedly invalid");
|
||||||
|
return validOutputs;
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
void DerivationGoal::done(
|
||||||
|
BuildResult::Status status,
|
||||||
|
DrvOutputs builtOutputs,
|
||||||
|
std::optional<Error> ex)
|
||||||
|
{
|
||||||
|
buildResult.status = status;
|
||||||
if (ex)
|
if (ex)
|
||||||
result.errorMsg = ex->what();
|
// FIXME: strip: "error: "
|
||||||
amDone(result.success() ? ecSuccess : ecFailed, ex);
|
buildResult.errorMsg = ex->what();
|
||||||
if (result.status == BuildResult::TimedOut)
|
amDone(buildResult.success() ? ecSuccess : ecFailed, ex);
|
||||||
|
if (buildResult.status == BuildResult::TimedOut)
|
||||||
worker.timedOut = true;
|
worker.timedOut = true;
|
||||||
if (result.status == BuildResult::PermanentFailure)
|
if (buildResult.status == BuildResult::PermanentFailure)
|
||||||
worker.permanentFailure = true;
|
worker.permanentFailure = true;
|
||||||
|
|
||||||
mcExpectedBuilds.reset();
|
mcExpectedBuilds.reset();
|
||||||
mcRunningBuilds.reset();
|
mcRunningBuilds.reset();
|
||||||
|
|
||||||
if (result.success()) {
|
if (buildResult.success()) {
|
||||||
|
assert(!builtOutputs.empty());
|
||||||
|
buildResult.builtOutputs = std::move(builtOutputs);
|
||||||
if (status == BuildResult::Built)
|
if (status == BuildResult::Built)
|
||||||
worker.doneBuilds++;
|
worker.doneBuilds++;
|
||||||
} else {
|
} else {
|
||||||
|
@ -1343,7 +1353,7 @@ void DerivationGoal::done(BuildResult::Status status, std::optional<Error> ex)
|
||||||
if (traceBuiltOutputsFile != "") {
|
if (traceBuiltOutputsFile != "") {
|
||||||
std::fstream fs;
|
std::fstream fs;
|
||||||
fs.open(traceBuiltOutputsFile, std::fstream::out);
|
fs.open(traceBuiltOutputsFile, std::fstream::out);
|
||||||
fs << worker.store.printStorePath(drvPath) << "\t" << result.toString() << std::endl;
|
fs << worker.store.printStorePath(drvPath) << "\t" << buildResult.toString() << std::endl;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
@ -2,7 +2,6 @@
|
||||||
|
|
||||||
#include "parsed-derivations.hh"
|
#include "parsed-derivations.hh"
|
||||||
#include "lock.hh"
|
#include "lock.hh"
|
||||||
#include "build-result.hh"
|
|
||||||
#include "store-api.hh"
|
#include "store-api.hh"
|
||||||
#include "pathlocks.hh"
|
#include "pathlocks.hh"
|
||||||
#include "goal.hh"
|
#include "goal.hh"
|
||||||
|
@ -105,20 +104,8 @@ struct DerivationGoal : public Goal
|
||||||
typedef void (DerivationGoal::*GoalState)();
|
typedef void (DerivationGoal::*GoalState)();
|
||||||
GoalState state;
|
GoalState state;
|
||||||
|
|
||||||
/* The final output paths of the build.
|
|
||||||
|
|
||||||
- For input-addressed derivations, always the precomputed paths
|
|
||||||
|
|
||||||
- For content-addressed derivations, calcuated from whatever the hash
|
|
||||||
ends up being. (Note that fixed outputs derivations that produce the
|
|
||||||
"wrong" output still install that data under its true content-address.)
|
|
||||||
*/
|
|
||||||
OutputPathMap finalOutputs;
|
|
||||||
|
|
||||||
BuildMode buildMode;
|
BuildMode buildMode;
|
||||||
|
|
||||||
BuildResult result;
|
|
||||||
|
|
||||||
/* The current round, if we're building multiple times. */
|
/* The current round, if we're building multiple times. */
|
||||||
size_t curRound = 1;
|
size_t curRound = 1;
|
||||||
|
|
||||||
|
@ -153,8 +140,6 @@ struct DerivationGoal : public Goal
|
||||||
/* Add wanted outputs to an already existing derivation goal. */
|
/* Add wanted outputs to an already existing derivation goal. */
|
||||||
void addWantedOutputs(const StringSet & outputs);
|
void addWantedOutputs(const StringSet & outputs);
|
||||||
|
|
||||||
BuildResult getResult() { return result; }
|
|
||||||
|
|
||||||
/* The states. */
|
/* The states. */
|
||||||
void getDerivation();
|
void getDerivation();
|
||||||
void loadDerivation();
|
void loadDerivation();
|
||||||
|
@ -176,7 +161,7 @@ struct DerivationGoal : public Goal
|
||||||
|
|
||||||
/* Check that the derivation outputs all exist and register them
|
/* Check that the derivation outputs all exist and register them
|
||||||
as valid. */
|
as valid. */
|
||||||
virtual void registerOutputs();
|
virtual DrvOutputs registerOutputs();
|
||||||
|
|
||||||
/* Open a log file and a pipe to it. */
|
/* Open a log file and a pipe to it. */
|
||||||
Path openLogFile();
|
Path openLogFile();
|
||||||
|
@ -211,8 +196,17 @@ struct DerivationGoal : public Goal
|
||||||
std::map<std::string, std::optional<StorePath>> queryPartialDerivationOutputMap();
|
std::map<std::string, std::optional<StorePath>> queryPartialDerivationOutputMap();
|
||||||
OutputPathMap queryDerivationOutputMap();
|
OutputPathMap queryDerivationOutputMap();
|
||||||
|
|
||||||
/* Return the set of (in)valid paths. */
|
/* Update 'initialOutputs' to determine the current status of the
|
||||||
void checkPathValidity();
|
outputs of the derivation. Also returns a Boolean denoting
|
||||||
|
whether all outputs are valid and non-corrupt, and a
|
||||||
|
'DrvOutputs' structure containing the valid and wanted
|
||||||
|
outputs. */
|
||||||
|
std::pair<bool, DrvOutputs> checkPathValidity();
|
||||||
|
|
||||||
|
/* Aborts if any output is not valid or corrupt, and otherwise
|
||||||
|
returns a 'DrvOutputs' structure containing the wanted
|
||||||
|
outputs. */
|
||||||
|
DrvOutputs assertPathValidity();
|
||||||
|
|
||||||
/* Forcibly kill the child process, if any. */
|
/* Forcibly kill the child process, if any. */
|
||||||
virtual void killChild();
|
virtual void killChild();
|
||||||
|
@ -223,6 +217,7 @@ struct DerivationGoal : public Goal
|
||||||
|
|
||||||
void done(
|
void done(
|
||||||
BuildResult::Status status,
|
BuildResult::Status status,
|
||||||
|
DrvOutputs builtOutputs = {},
|
||||||
std::optional<Error> ex = {});
|
std::optional<Error> ex = {});
|
||||||
|
|
||||||
StorePathSet exportReferences(const StorePathSet & storePaths);
|
StorePathSet exportReferences(const StorePathSet & storePaths);
|
||||||
|
|
|
@ -6,8 +6,12 @@
|
||||||
|
|
||||||
namespace nix {
|
namespace nix {
|
||||||
|
|
||||||
DrvOutputSubstitutionGoal::DrvOutputSubstitutionGoal(const DrvOutput& id, Worker & worker, RepairFlag repair, std::optional<ContentAddress> ca)
|
DrvOutputSubstitutionGoal::DrvOutputSubstitutionGoal(
|
||||||
: Goal(worker)
|
const DrvOutput & id,
|
||||||
|
Worker & worker,
|
||||||
|
RepairFlag repair,
|
||||||
|
std::optional<ContentAddress> ca)
|
||||||
|
: Goal(worker, DerivedPath::Opaque { StorePath::dummy })
|
||||||
, id(id)
|
, id(id)
|
||||||
{
|
{
|
||||||
state = &DrvOutputSubstitutionGoal::init;
|
state = &DrvOutputSubstitutionGoal::init;
|
||||||
|
@ -32,7 +36,7 @@ void DrvOutputSubstitutionGoal::init()
|
||||||
|
|
||||||
void DrvOutputSubstitutionGoal::tryNext()
|
void DrvOutputSubstitutionGoal::tryNext()
|
||||||
{
|
{
|
||||||
trace("Trying next substituter");
|
trace("trying next substituter");
|
||||||
|
|
||||||
if (subs.size() == 0) {
|
if (subs.size() == 0) {
|
||||||
/* None left. Terminate this goal and let someone else deal
|
/* None left. Terminate this goal and let someone else deal
|
||||||
|
@ -119,7 +123,7 @@ void DrvOutputSubstitutionGoal::realisationFetched()
|
||||||
void DrvOutputSubstitutionGoal::outPathValid()
|
void DrvOutputSubstitutionGoal::outPathValid()
|
||||||
{
|
{
|
||||||
assert(outputInfo);
|
assert(outputInfo);
|
||||||
trace("Output path substituted");
|
trace("output path substituted");
|
||||||
|
|
||||||
if (nrFailed > 0) {
|
if (nrFailed > 0) {
|
||||||
debug("The output path of the derivation output '%s' could not be substituted", id.to_string());
|
debug("The output path of the derivation output '%s' could not be substituted", id.to_string());
|
||||||
|
|
|
@ -47,43 +47,51 @@ void Store::buildPaths(const std::vector<DerivedPath> & reqs, BuildMode buildMod
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
std::vector<BuildResult> Store::buildPathsWithResults(
|
||||||
|
const std::vector<DerivedPath> & reqs,
|
||||||
|
BuildMode buildMode,
|
||||||
|
std::shared_ptr<Store> evalStore)
|
||||||
|
{
|
||||||
|
Worker worker(*this, evalStore ? *evalStore : *this);
|
||||||
|
|
||||||
|
Goals goals;
|
||||||
|
for (const auto & br : reqs) {
|
||||||
|
std::visit(overloaded {
|
||||||
|
[&](const DerivedPath::Built & bfd) {
|
||||||
|
goals.insert(worker.makeDerivationGoal(bfd.drvPath, bfd.outputs, buildMode));
|
||||||
|
},
|
||||||
|
[&](const DerivedPath::Opaque & bo) {
|
||||||
|
goals.insert(worker.makePathSubstitutionGoal(bo.path, buildMode == bmRepair ? Repair : NoRepair));
|
||||||
|
},
|
||||||
|
}, br.raw());
|
||||||
|
}
|
||||||
|
|
||||||
|
worker.run(goals);
|
||||||
|
|
||||||
|
std::vector<BuildResult> results;
|
||||||
|
|
||||||
|
for (auto & i : goals)
|
||||||
|
results.push_back(i->buildResult);
|
||||||
|
|
||||||
|
return results;
|
||||||
|
}
|
||||||
|
|
||||||
BuildResult Store::buildDerivation(const StorePath & drvPath, const BasicDerivation & drv,
|
BuildResult Store::buildDerivation(const StorePath & drvPath, const BasicDerivation & drv,
|
||||||
BuildMode buildMode)
|
BuildMode buildMode)
|
||||||
{
|
{
|
||||||
Worker worker(*this, *this);
|
Worker worker(*this, *this);
|
||||||
auto goal = worker.makeBasicDerivationGoal(drvPath, drv, {}, buildMode);
|
auto goal = worker.makeBasicDerivationGoal(drvPath, drv, {}, buildMode);
|
||||||
|
|
||||||
BuildResult result;
|
|
||||||
|
|
||||||
try {
|
try {
|
||||||
worker.run(Goals{goal});
|
worker.run(Goals{goal});
|
||||||
result = goal->getResult();
|
return goal->buildResult;
|
||||||
} catch (Error & e) {
|
} catch (Error & e) {
|
||||||
result.status = BuildResult::MiscFailure;
|
return BuildResult {
|
||||||
result.errorMsg = e.msg();
|
.status = BuildResult::MiscFailure,
|
||||||
}
|
.errorMsg = e.msg(),
|
||||||
// XXX: Should use `goal->queryPartialDerivationOutputMap()` once it's
|
.path = DerivedPath::Built { .drvPath = drvPath },
|
||||||
// extended to return the full realisation for each output
|
};
|
||||||
auto staticDrvOutputs = drv.outputsAndOptPaths(*this);
|
};
|
||||||
auto outputHashes = staticOutputHashes(*this, drv);
|
|
||||||
for (auto & [outputName, staticOutput] : staticDrvOutputs) {
|
|
||||||
auto outputId = DrvOutput{outputHashes.at(outputName), outputName};
|
|
||||||
if (staticOutput.second)
|
|
||||||
result.builtOutputs.insert_or_assign(
|
|
||||||
outputId,
|
|
||||||
Realisation{ outputId, *staticOutput.second}
|
|
||||||
);
|
|
||||||
if (settings.isExperimentalFeatureEnabled(Xp::CaDerivations) && !derivationHasKnownOutputPaths(drv.type())) {
|
|
||||||
auto realisation = this->queryRealisation(outputId);
|
|
||||||
if (realisation)
|
|
||||||
result.builtOutputs.insert_or_assign(
|
|
||||||
outputId,
|
|
||||||
*realisation
|
|
||||||
);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return result;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|
|
@ -2,6 +2,7 @@
|
||||||
|
|
||||||
#include "types.hh"
|
#include "types.hh"
|
||||||
#include "store-api.hh"
|
#include "store-api.hh"
|
||||||
|
#include "build-result.hh"
|
||||||
|
|
||||||
namespace nix {
|
namespace nix {
|
||||||
|
|
||||||
|
@ -55,10 +56,15 @@ struct Goal : public std::enable_shared_from_this<Goal>
|
||||||
/* Whether the goal is finished. */
|
/* Whether the goal is finished. */
|
||||||
ExitCode exitCode;
|
ExitCode exitCode;
|
||||||
|
|
||||||
|
/* Build result. */
|
||||||
|
BuildResult buildResult;
|
||||||
|
|
||||||
/* Exception containing an error message, if any. */
|
/* Exception containing an error message, if any. */
|
||||||
std::optional<Error> ex;
|
std::optional<Error> ex;
|
||||||
|
|
||||||
Goal(Worker & worker) : worker(worker)
|
Goal(Worker & worker, DerivedPath path)
|
||||||
|
: worker(worker)
|
||||||
|
, buildResult { .path = std::move(path) }
|
||||||
{
|
{
|
||||||
nrFailed = nrNoSubstituters = nrIncompleteClosure = 0;
|
nrFailed = nrNoSubstituters = nrIncompleteClosure = 0;
|
||||||
exitCode = ecBusy;
|
exitCode = ecBusy;
|
||||||
|
|
|
@ -1,4 +1,5 @@
|
||||||
#include "local-derivation-goal.hh"
|
#include "local-derivation-goal.hh"
|
||||||
|
#include "gc-store.hh"
|
||||||
#include "hook-instance.hh"
|
#include "hook-instance.hh"
|
||||||
#include "worker.hh"
|
#include "worker.hh"
|
||||||
#include "builtins.hh"
|
#include "builtins.hh"
|
||||||
|
@ -193,7 +194,7 @@ void LocalDerivationGoal::tryLocalBuild() {
|
||||||
outputLocks.unlock();
|
outputLocks.unlock();
|
||||||
buildUser.reset();
|
buildUser.reset();
|
||||||
worker.permanentFailure = true;
|
worker.permanentFailure = true;
|
||||||
done(BuildResult::InputRejected, e);
|
done(BuildResult::InputRejected, {}, e);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -755,7 +756,7 @@ void LocalDerivationGoal::startBuilder()
|
||||||
if (tcsetattr(builderOut.writeSide.get(), TCSANOW, &term))
|
if (tcsetattr(builderOut.writeSide.get(), TCSANOW, &term))
|
||||||
throw SysError("putting pseudoterminal into raw mode");
|
throw SysError("putting pseudoterminal into raw mode");
|
||||||
|
|
||||||
result.startTime = time(0);
|
buildResult.startTime = time(0);
|
||||||
|
|
||||||
/* Fork a child to build the package. */
|
/* Fork a child to build the package. */
|
||||||
|
|
||||||
|
@ -1127,7 +1128,7 @@ struct RestrictedStoreConfig : virtual LocalFSStoreConfig
|
||||||
/* A wrapper around LocalStore that only allows building/querying of
|
/* A wrapper around LocalStore that only allows building/querying of
|
||||||
paths that are in the input closures of the build or were added via
|
paths that are in the input closures of the build or were added via
|
||||||
recursive Nix calls. */
|
recursive Nix calls. */
|
||||||
struct RestrictedStore : public virtual RestrictedStoreConfig, public virtual LocalFSStore
|
struct RestrictedStore : public virtual RestrictedStoreConfig, public virtual LocalFSStore, public virtual GcStore
|
||||||
{
|
{
|
||||||
ref<LocalStore> next;
|
ref<LocalStore> next;
|
||||||
|
|
||||||
|
@ -1259,6 +1260,16 @@ struct RestrictedStore : public virtual RestrictedStoreConfig, public virtual Lo
|
||||||
}
|
}
|
||||||
|
|
||||||
void buildPaths(const std::vector<DerivedPath> & paths, BuildMode buildMode, std::shared_ptr<Store> evalStore) override
|
void buildPaths(const std::vector<DerivedPath> & paths, BuildMode buildMode, std::shared_ptr<Store> evalStore) override
|
||||||
|
{
|
||||||
|
for (auto & result : buildPathsWithResults(paths, buildMode, evalStore))
|
||||||
|
if (!result.success())
|
||||||
|
result.rethrow();
|
||||||
|
}
|
||||||
|
|
||||||
|
std::vector<BuildResult> buildPathsWithResults(
|
||||||
|
const std::vector<DerivedPath> & paths,
|
||||||
|
BuildMode buildMode = bmNormal,
|
||||||
|
std::shared_ptr<Store> evalStore = nullptr) override
|
||||||
{
|
{
|
||||||
assert(!evalStore);
|
assert(!evalStore);
|
||||||
|
|
||||||
|
@ -1272,25 +1283,12 @@ struct RestrictedStore : public virtual RestrictedStoreConfig, public virtual Lo
|
||||||
throw InvalidPath("cannot build '%s' in recursive Nix because path is unknown", req.to_string(*next));
|
throw InvalidPath("cannot build '%s' in recursive Nix because path is unknown", req.to_string(*next));
|
||||||
}
|
}
|
||||||
|
|
||||||
next->buildPaths(paths, buildMode);
|
auto results = next->buildPathsWithResults(paths, buildMode);
|
||||||
|
|
||||||
for (auto & path : paths) {
|
for (auto & result : results) {
|
||||||
auto p = std::get_if<DerivedPath::Built>(&path);
|
for (auto & [outputName, output] : result.builtOutputs) {
|
||||||
if (!p) continue;
|
newPaths.insert(output.outPath);
|
||||||
auto & bfd = *p;
|
newRealisations.insert(output);
|
||||||
auto drv = readDerivation(bfd.drvPath);
|
|
||||||
auto drvHashes = staticOutputHashes(*this, drv);
|
|
||||||
auto outputs = next->queryDerivationOutputMap(bfd.drvPath);
|
|
||||||
for (auto & [outputName, outputPath] : outputs)
|
|
||||||
if (wantOutput(outputName, bfd.outputs)) {
|
|
||||||
newPaths.insert(outputPath);
|
|
||||||
if (settings.isExperimentalFeatureEnabled(Xp::CaDerivations)) {
|
|
||||||
auto thisRealisation = next->queryRealisation(
|
|
||||||
DrvOutput{drvHashes.at(outputName), outputName}
|
|
||||||
);
|
|
||||||
assert(thisRealisation);
|
|
||||||
newRealisations.insert(*thisRealisation);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -1300,6 +1298,8 @@ struct RestrictedStore : public virtual RestrictedStoreConfig, public virtual Lo
|
||||||
goal.addDependency(path);
|
goal.addDependency(path);
|
||||||
for (auto & real : Realisation::closure(*next, newRealisations))
|
for (auto & real : Realisation::closure(*next, newRealisations))
|
||||||
goal.addedDrvOutputs.insert(real.id);
|
goal.addedDrvOutputs.insert(real.id);
|
||||||
|
|
||||||
|
return results;
|
||||||
}
|
}
|
||||||
|
|
||||||
BuildResult buildDerivation(const StorePath & drvPath, const BasicDerivation & drv,
|
BuildResult buildDerivation(const StorePath & drvPath, const BasicDerivation & drv,
|
||||||
|
@ -1340,6 +1340,12 @@ struct RestrictedStore : public virtual RestrictedStoreConfig, public virtual Lo
|
||||||
next->queryMissing(allowed, willBuild, willSubstitute,
|
next->queryMissing(allowed, willBuild, willSubstitute,
|
||||||
unknown, downloadSize, narSize);
|
unknown, downloadSize, narSize);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
virtual std::optional<std::string> getBuildLog(const StorePath & path) override
|
||||||
|
{ return std::nullopt; }
|
||||||
|
|
||||||
|
virtual void addBuildLog(const StorePath & path, std::string_view log) override
|
||||||
|
{ unsupported("addBuildLog"); }
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
||||||
|
@ -2068,7 +2074,7 @@ void LocalDerivationGoal::runChild()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
void LocalDerivationGoal::registerOutputs()
|
DrvOutputs LocalDerivationGoal::registerOutputs()
|
||||||
{
|
{
|
||||||
/* When using a build hook, the build hook can register the output
|
/* When using a build hook, the build hook can register the output
|
||||||
as valid (by doing `nix-store --import'). If so we don't have
|
as valid (by doing `nix-store --import'). If so we don't have
|
||||||
|
@ -2077,10 +2083,8 @@ void LocalDerivationGoal::registerOutputs()
|
||||||
We can only early return when the outputs are known a priori. For
|
We can only early return when the outputs are known a priori. For
|
||||||
floating content-addressed derivations this isn't the case.
|
floating content-addressed derivations this isn't the case.
|
||||||
*/
|
*/
|
||||||
if (hook) {
|
if (hook)
|
||||||
DerivationGoal::registerOutputs();
|
return DerivationGoal::registerOutputs();
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
std::map<std::string, ValidPathInfo> infos;
|
std::map<std::string, ValidPathInfo> infos;
|
||||||
|
|
||||||
|
@ -2203,6 +2207,8 @@ void LocalDerivationGoal::registerOutputs()
|
||||||
|
|
||||||
std::reverse(sortedOutputNames.begin(), sortedOutputNames.end());
|
std::reverse(sortedOutputNames.begin(), sortedOutputNames.end());
|
||||||
|
|
||||||
|
OutputPathMap finalOutputs;
|
||||||
|
|
||||||
for (auto & outputName : sortedOutputNames) {
|
for (auto & outputName : sortedOutputNames) {
|
||||||
auto output = drv->outputs.at(outputName);
|
auto output = drv->outputs.at(outputName);
|
||||||
auto & scratchPath = scratchOutputs.at(outputName);
|
auto & scratchPath = scratchOutputs.at(outputName);
|
||||||
|
@ -2339,6 +2345,7 @@ void LocalDerivationGoal::registerOutputs()
|
||||||
};
|
};
|
||||||
|
|
||||||
ValidPathInfo newInfo = std::visit(overloaded {
|
ValidPathInfo newInfo = std::visit(overloaded {
|
||||||
|
|
||||||
[&](const DerivationOutputInputAddressed & output) {
|
[&](const DerivationOutputInputAddressed & output) {
|
||||||
/* input-addressed case */
|
/* input-addressed case */
|
||||||
auto requiredFinalPath = output.path;
|
auto requiredFinalPath = output.path;
|
||||||
|
@ -2358,6 +2365,7 @@ void LocalDerivationGoal::registerOutputs()
|
||||||
newInfo0.references.insert(newInfo0.path);
|
newInfo0.references.insert(newInfo0.path);
|
||||||
return newInfo0;
|
return newInfo0;
|
||||||
},
|
},
|
||||||
|
|
||||||
[&](const DerivationOutputCAFixed & dof) {
|
[&](const DerivationOutputCAFixed & dof) {
|
||||||
auto newInfo0 = newInfoFromCA(DerivationOutputCAFloating {
|
auto newInfo0 = newInfoFromCA(DerivationOutputCAFloating {
|
||||||
.method = dof.hash.method,
|
.method = dof.hash.method,
|
||||||
|
@ -2380,14 +2388,17 @@ void LocalDerivationGoal::registerOutputs()
|
||||||
}
|
}
|
||||||
return newInfo0;
|
return newInfo0;
|
||||||
},
|
},
|
||||||
[&](DerivationOutputCAFloating dof) {
|
|
||||||
|
[&](DerivationOutputCAFloating & dof) {
|
||||||
return newInfoFromCA(dof);
|
return newInfoFromCA(dof);
|
||||||
},
|
},
|
||||||
|
|
||||||
[&](DerivationOutputDeferred) -> ValidPathInfo {
|
[&](DerivationOutputDeferred) -> ValidPathInfo {
|
||||||
// No derivation should reach that point without having been
|
// No derivation should reach that point without having been
|
||||||
// rewritten first
|
// rewritten first
|
||||||
assert(false);
|
assert(false);
|
||||||
},
|
},
|
||||||
|
|
||||||
}, output.output);
|
}, output.output);
|
||||||
|
|
||||||
/* FIXME: set proper permissions in restorePath() so
|
/* FIXME: set proper permissions in restorePath() so
|
||||||
|
@ -2498,11 +2509,12 @@ void LocalDerivationGoal::registerOutputs()
|
||||||
}
|
}
|
||||||
|
|
||||||
if (buildMode == bmCheck) {
|
if (buildMode == bmCheck) {
|
||||||
// In case of FOD mismatches on `--check` an error must be thrown as this is also
|
/* In case of fixed-output derivations, if there are
|
||||||
// a source for non-determinism.
|
mismatches on `--check` an error must be thrown as this is
|
||||||
|
also a source for non-determinism. */
|
||||||
if (delayedException)
|
if (delayedException)
|
||||||
std::rethrow_exception(delayedException);
|
std::rethrow_exception(delayedException);
|
||||||
return;
|
return assertPathValidity();
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Apply output checks. */
|
/* Apply output checks. */
|
||||||
|
@ -2514,7 +2526,7 @@ void LocalDerivationGoal::registerOutputs()
|
||||||
assert(prevInfos.size() == infos.size());
|
assert(prevInfos.size() == infos.size());
|
||||||
for (auto i = prevInfos.begin(), j = infos.begin(); i != prevInfos.end(); ++i, ++j)
|
for (auto i = prevInfos.begin(), j = infos.begin(); i != prevInfos.end(); ++i, ++j)
|
||||||
if (!(*i == *j)) {
|
if (!(*i == *j)) {
|
||||||
result.isNonDeterministic = true;
|
buildResult.isNonDeterministic = true;
|
||||||
Path prev = worker.store.printStorePath(i->second.path) + checkSuffix;
|
Path prev = worker.store.printStorePath(i->second.path) + checkSuffix;
|
||||||
bool prevExists = keepPreviousRound && pathExists(prev);
|
bool prevExists = keepPreviousRound && pathExists(prev);
|
||||||
hintformat hint = prevExists
|
hintformat hint = prevExists
|
||||||
|
@ -2552,7 +2564,7 @@ void LocalDerivationGoal::registerOutputs()
|
||||||
|
|
||||||
if (curRound < nrRounds) {
|
if (curRound < nrRounds) {
|
||||||
prevInfos = std::move(infos);
|
prevInfos = std::move(infos);
|
||||||
return;
|
return {};
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Remove the .check directories if we're done. FIXME: keep them
|
/* Remove the .check directories if we're done. FIXME: keep them
|
||||||
|
@ -2587,17 +2599,24 @@ void LocalDerivationGoal::registerOutputs()
|
||||||
means it's safe to link the derivation to the output hash. We must do
|
means it's safe to link the derivation to the output hash. We must do
|
||||||
that for floating CA derivations, which otherwise couldn't be cached,
|
that for floating CA derivations, which otherwise couldn't be cached,
|
||||||
but it's fine to do in all cases. */
|
but it's fine to do in all cases. */
|
||||||
|
DrvOutputs builtOutputs;
|
||||||
|
|
||||||
|
for (auto & [outputName, newInfo] : infos) {
|
||||||
|
auto thisRealisation = Realisation {
|
||||||
|
.id = DrvOutput {
|
||||||
|
initialOutputs.at(outputName).outputHash,
|
||||||
|
outputName
|
||||||
|
},
|
||||||
|
.outPath = newInfo.path
|
||||||
|
};
|
||||||
if (settings.isExperimentalFeatureEnabled(Xp::CaDerivations)) {
|
if (settings.isExperimentalFeatureEnabled(Xp::CaDerivations)) {
|
||||||
for (auto& [outputName, newInfo] : infos) {
|
|
||||||
auto thisRealisation = Realisation{
|
|
||||||
.id = DrvOutput{initialOutputs.at(outputName).outputHash,
|
|
||||||
outputName},
|
|
||||||
.outPath = newInfo.path};
|
|
||||||
signRealisation(thisRealisation);
|
signRealisation(thisRealisation);
|
||||||
worker.store.registerDrvOutput(thisRealisation);
|
worker.store.registerDrvOutput(thisRealisation);
|
||||||
}
|
}
|
||||||
|
builtOutputs.emplace(thisRealisation.id, thisRealisation);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
return builtOutputs;
|
||||||
}
|
}
|
||||||
|
|
||||||
void LocalDerivationGoal::signRealisation(Realisation & realisation)
|
void LocalDerivationGoal::signRealisation(Realisation & realisation)
|
||||||
|
@ -2606,7 +2625,7 @@ void LocalDerivationGoal::signRealisation(Realisation & realisation)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
void LocalDerivationGoal::checkOutputs(const std::map<Path, ValidPathInfo> & outputs)
|
void LocalDerivationGoal::checkOutputs(const std::map<std::string, ValidPathInfo> & outputs)
|
||||||
{
|
{
|
||||||
std::map<Path, const ValidPathInfo &> outputsByPath;
|
std::map<Path, const ValidPathInfo &> outputsByPath;
|
||||||
for (auto & output : outputs)
|
for (auto & output : outputs)
|
||||||
|
@ -2678,8 +2697,8 @@ void LocalDerivationGoal::checkOutputs(const std::map<Path, ValidPathInfo> & out
|
||||||
for (auto & i : *value) {
|
for (auto & i : *value) {
|
||||||
if (worker.store.isStorePath(i))
|
if (worker.store.isStorePath(i))
|
||||||
spec.insert(worker.store.parseStorePath(i));
|
spec.insert(worker.store.parseStorePath(i));
|
||||||
else if (finalOutputs.count(i))
|
else if (outputs.count(i))
|
||||||
spec.insert(finalOutputs.at(i));
|
spec.insert(outputs.at(i).path);
|
||||||
else throw BuildError("derivation contains an illegal reference specifier '%s'", i);
|
else throw BuildError("derivation contains an illegal reference specifier '%s'", i);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
@ -169,7 +169,7 @@ struct LocalDerivationGoal : public DerivationGoal
|
||||||
|
|
||||||
/* Check that the derivation outputs all exist and register them
|
/* Check that the derivation outputs all exist and register them
|
||||||
as valid. */
|
as valid. */
|
||||||
void registerOutputs() override;
|
DrvOutputs registerOutputs() override;
|
||||||
|
|
||||||
void signRealisation(Realisation &) override;
|
void signRealisation(Realisation &) override;
|
||||||
|
|
||||||
|
|
|
@ -6,7 +6,7 @@
|
||||||
namespace nix {
|
namespace nix {
|
||||||
|
|
||||||
PathSubstitutionGoal::PathSubstitutionGoal(const StorePath & storePath, Worker & worker, RepairFlag repair, std::optional<ContentAddress> ca)
|
PathSubstitutionGoal::PathSubstitutionGoal(const StorePath & storePath, Worker & worker, RepairFlag repair, std::optional<ContentAddress> ca)
|
||||||
: Goal(worker)
|
: Goal(worker, DerivedPath::Opaque { storePath })
|
||||||
, storePath(storePath)
|
, storePath(storePath)
|
||||||
, repair(repair)
|
, repair(repair)
|
||||||
, ca(ca)
|
, ca(ca)
|
||||||
|
@ -24,6 +24,13 @@ PathSubstitutionGoal::~PathSubstitutionGoal()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
void PathSubstitutionGoal::done(ExitCode result, BuildResult::Status status)
|
||||||
|
{
|
||||||
|
buildResult.status = status;
|
||||||
|
amDone(result);
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
void PathSubstitutionGoal::work()
|
void PathSubstitutionGoal::work()
|
||||||
{
|
{
|
||||||
(this->*state)();
|
(this->*state)();
|
||||||
|
@ -38,7 +45,7 @@ void PathSubstitutionGoal::init()
|
||||||
|
|
||||||
/* If the path already exists we're done. */
|
/* If the path already exists we're done. */
|
||||||
if (!repair && worker.store.isValidPath(storePath)) {
|
if (!repair && worker.store.isValidPath(storePath)) {
|
||||||
amDone(ecSuccess);
|
done(ecSuccess, BuildResult::AlreadyValid);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -65,7 +72,7 @@ void PathSubstitutionGoal::tryNext()
|
||||||
/* Hack: don't indicate failure if there were no substituters.
|
/* Hack: don't indicate failure if there were no substituters.
|
||||||
In that case the calling derivation should just do a
|
In that case the calling derivation should just do a
|
||||||
build. */
|
build. */
|
||||||
amDone(substituterFailed ? ecFailed : ecNoSubstituters);
|
done(substituterFailed ? ecFailed : ecNoSubstituters, BuildResult::NoSubstituters);
|
||||||
|
|
||||||
if (substituterFailed) {
|
if (substituterFailed) {
|
||||||
worker.failedSubstitutions++;
|
worker.failedSubstitutions++;
|
||||||
|
@ -163,7 +170,9 @@ void PathSubstitutionGoal::referencesValid()
|
||||||
|
|
||||||
if (nrFailed > 0) {
|
if (nrFailed > 0) {
|
||||||
debug("some references of path '%s' could not be realised", worker.store.printStorePath(storePath));
|
debug("some references of path '%s' could not be realised", worker.store.printStorePath(storePath));
|
||||||
amDone(nrNoSubstituters > 0 || nrIncompleteClosure > 0 ? ecIncompleteClosure : ecFailed);
|
done(
|
||||||
|
nrNoSubstituters > 0 || nrIncompleteClosure > 0 ? ecIncompleteClosure : ecFailed,
|
||||||
|
BuildResult::DependencyFailed);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -268,7 +277,7 @@ void PathSubstitutionGoal::finished()
|
||||||
|
|
||||||
worker.updateProgress();
|
worker.updateProgress();
|
||||||
|
|
||||||
amDone(ecSuccess);
|
done(ecSuccess, BuildResult::Substituted);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|
|
@ -53,6 +53,8 @@ struct PathSubstitutionGoal : public Goal
|
||||||
/* Content address for recomputing store path */
|
/* Content address for recomputing store path */
|
||||||
std::optional<ContentAddress> ca;
|
std::optional<ContentAddress> ca;
|
||||||
|
|
||||||
|
void done(ExitCode result, BuildResult::Status status);
|
||||||
|
|
||||||
public:
|
public:
|
||||||
PathSubstitutionGoal(const StorePath & storePath, Worker & worker, RepairFlag repair = NoRepair, std::optional<ContentAddress> ca = std::nullopt);
|
PathSubstitutionGoal(const StorePath & storePath, Worker & worker, RepairFlag repair = NoRepair, std::optional<ContentAddress> ca = std::nullopt);
|
||||||
~PathSubstitutionGoal();
|
~PathSubstitutionGoal();
|
||||||
|
|
|
@ -3,6 +3,9 @@
|
||||||
#include "worker-protocol.hh"
|
#include "worker-protocol.hh"
|
||||||
#include "build-result.hh"
|
#include "build-result.hh"
|
||||||
#include "store-api.hh"
|
#include "store-api.hh"
|
||||||
|
#include "store-cast.hh"
|
||||||
|
#include "gc-store.hh"
|
||||||
|
#include "log-store.hh"
|
||||||
#include "path-with-outputs.hh"
|
#include "path-with-outputs.hh"
|
||||||
#include "finally.hh"
|
#include "finally.hh"
|
||||||
#include "archive.hh"
|
#include "archive.hh"
|
||||||
|
@ -531,6 +534,25 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
case wopBuildPathsWithResults: {
|
||||||
|
auto drvs = readDerivedPaths(*store, clientVersion, from);
|
||||||
|
BuildMode mode = bmNormal;
|
||||||
|
mode = (BuildMode) readInt(from);
|
||||||
|
|
||||||
|
/* Repairing is not atomic, so disallowed for "untrusted"
|
||||||
|
clients. */
|
||||||
|
if (mode == bmRepair && !trusted)
|
||||||
|
throw Error("repairing is not allowed because you are not in 'trusted-users'");
|
||||||
|
|
||||||
|
logger->startWork();
|
||||||
|
auto results = store->buildPathsWithResults(drvs, mode);
|
||||||
|
logger->stopWork();
|
||||||
|
|
||||||
|
worker_proto::write(*store, to, results);
|
||||||
|
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
case wopBuildDerivation: {
|
case wopBuildDerivation: {
|
||||||
auto drvPath = store->parseStorePath(readString(from));
|
auto drvPath = store->parseStorePath(readString(from));
|
||||||
BasicDerivation drv;
|
BasicDerivation drv;
|
||||||
|
@ -623,9 +645,12 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
|
||||||
|
|
||||||
case wopAddIndirectRoot: {
|
case wopAddIndirectRoot: {
|
||||||
Path path = absPath(readString(from));
|
Path path = absPath(readString(from));
|
||||||
|
|
||||||
logger->startWork();
|
logger->startWork();
|
||||||
store->addIndirectRoot(path);
|
auto & gcStore = require<GcStore>(*store);
|
||||||
|
gcStore.addIndirectRoot(path);
|
||||||
logger->stopWork();
|
logger->stopWork();
|
||||||
|
|
||||||
to << 1;
|
to << 1;
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
@ -640,7 +665,8 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
|
||||||
|
|
||||||
case wopFindRoots: {
|
case wopFindRoots: {
|
||||||
logger->startWork();
|
logger->startWork();
|
||||||
Roots roots = store->findRoots(!trusted);
|
auto & gcStore = require<GcStore>(*store);
|
||||||
|
Roots roots = gcStore.findRoots(!trusted);
|
||||||
logger->stopWork();
|
logger->stopWork();
|
||||||
|
|
||||||
size_t size = 0;
|
size_t size = 0;
|
||||||
|
@ -671,7 +697,8 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
|
||||||
logger->startWork();
|
logger->startWork();
|
||||||
if (options.ignoreLiveness)
|
if (options.ignoreLiveness)
|
||||||
throw Error("you are not allowed to ignore liveness");
|
throw Error("you are not allowed to ignore liveness");
|
||||||
store->collectGarbage(options, results);
|
auto & gcStore = require<GcStore>(*store);
|
||||||
|
gcStore.collectGarbage(options, results);
|
||||||
logger->stopWork();
|
logger->stopWork();
|
||||||
|
|
||||||
to << results.paths << results.bytesFreed << 0 /* obsolete */;
|
to << results.paths << results.bytesFreed << 0 /* obsolete */;
|
||||||
|
@ -928,11 +955,12 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
|
||||||
logger->startWork();
|
logger->startWork();
|
||||||
if (!trusted)
|
if (!trusted)
|
||||||
throw Error("you are not privileged to add logs");
|
throw Error("you are not privileged to add logs");
|
||||||
|
auto & logStore = require<LogStore>(*store);
|
||||||
{
|
{
|
||||||
FramedSource source(from);
|
FramedSource source(from);
|
||||||
StringSink sink;
|
StringSink sink;
|
||||||
source.drainInto(sink);
|
source.drainInto(sink);
|
||||||
store->addBuildLog(path, sink.s);
|
logStore.addBuildLog(path, sink.s);
|
||||||
}
|
}
|
||||||
logger->stopWork();
|
logger->stopWork();
|
||||||
to << 1;
|
to << 1;
|
||||||
|
|
|
@ -510,7 +510,7 @@ static const DrvHashModulo pathDerivationModulo(Store & store, const StorePath &
|
||||||
*/
|
*/
|
||||||
DrvHashModulo hashDerivationModulo(Store & store, const Derivation & drv, bool maskOutputs)
|
DrvHashModulo hashDerivationModulo(Store & store, const Derivation & drv, bool maskOutputs)
|
||||||
{
|
{
|
||||||
bool isDeferred = false;
|
auto kind = DrvHash::Kind::Regular;
|
||||||
/* Return a fixed hash for fixed-output derivations. */
|
/* Return a fixed hash for fixed-output derivations. */
|
||||||
switch (drv.type()) {
|
switch (drv.type()) {
|
||||||
case DerivationType::CAFixed: {
|
case DerivationType::CAFixed: {
|
||||||
|
@ -526,7 +526,7 @@ DrvHashModulo hashDerivationModulo(Store & store, const Derivation & drv, bool m
|
||||||
return outputHashes;
|
return outputHashes;
|
||||||
}
|
}
|
||||||
case DerivationType::CAFloating:
|
case DerivationType::CAFloating:
|
||||||
isDeferred = true;
|
kind = DrvHash::Kind::Deferred;
|
||||||
break;
|
break;
|
||||||
case DerivationType::InputAddressed:
|
case DerivationType::InputAddressed:
|
||||||
break;
|
break;
|
||||||
|
@ -537,21 +537,20 @@ DrvHashModulo hashDerivationModulo(Store & store, const Derivation & drv, bool m
|
||||||
/* For other derivations, replace the inputs paths with recursive
|
/* For other derivations, replace the inputs paths with recursive
|
||||||
calls to this function. */
|
calls to this function. */
|
||||||
std::map<std::string, StringSet> inputs2;
|
std::map<std::string, StringSet> inputs2;
|
||||||
for (auto & i : drv.inputDrvs) {
|
for (auto & [drvPath, inputOutputs0] : drv.inputDrvs) {
|
||||||
const auto & res = pathDerivationModulo(store, i.first);
|
// Avoid lambda capture restriction with standard / Clang
|
||||||
|
auto & inputOutputs = inputOutputs0;
|
||||||
|
const auto & res = pathDerivationModulo(store, drvPath);
|
||||||
std::visit(overloaded {
|
std::visit(overloaded {
|
||||||
// Regular non-CA derivation, replace derivation
|
// Regular non-CA derivation, replace derivation
|
||||||
[&](const Hash & drvHash) {
|
[&](const DrvHash & drvHash) {
|
||||||
inputs2.insert_or_assign(drvHash.to_string(Base16, false), i.second);
|
kind |= drvHash.kind;
|
||||||
},
|
inputs2.insert_or_assign(drvHash.hash.to_string(Base16, false), inputOutputs);
|
||||||
[&](const DeferredHash & deferredHash) {
|
|
||||||
isDeferred = true;
|
|
||||||
inputs2.insert_or_assign(deferredHash.hash.to_string(Base16, false), i.second);
|
|
||||||
},
|
},
|
||||||
// CA derivation's output hashes
|
// CA derivation's output hashes
|
||||||
[&](const CaOutputHashes & outputHashes) {
|
[&](const CaOutputHashes & outputHashes) {
|
||||||
std::set<std::string> justOut = { "out" };
|
std::set<std::string> justOut = { "out" };
|
||||||
for (auto & output : i.second) {
|
for (auto & output : inputOutputs) {
|
||||||
/* Put each one in with a single "out" output.. */
|
/* Put each one in with a single "out" output.. */
|
||||||
const auto h = outputHashes.at(output);
|
const auto h = outputHashes.at(output);
|
||||||
inputs2.insert_or_assign(
|
inputs2.insert_or_assign(
|
||||||
|
@ -559,15 +558,24 @@ DrvHashModulo hashDerivationModulo(Store & store, const Derivation & drv, bool m
|
||||||
justOut);
|
justOut);
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
}, res);
|
}, res.raw());
|
||||||
}
|
}
|
||||||
|
|
||||||
auto hash = hashString(htSHA256, drv.unparse(store, maskOutputs, &inputs2));
|
auto hash = hashString(htSHA256, drv.unparse(store, maskOutputs, &inputs2));
|
||||||
|
|
||||||
if (isDeferred)
|
return DrvHash { .hash = hash, .kind = kind };
|
||||||
return DeferredHash { hash };
|
}
|
||||||
else
|
|
||||||
return hash;
|
|
||||||
|
void operator |= (DrvHash::Kind & self, const DrvHash::Kind & other) noexcept
|
||||||
|
{
|
||||||
|
switch (other) {
|
||||||
|
case DrvHash::Kind::Regular:
|
||||||
|
break;
|
||||||
|
case DrvHash::Kind::Deferred:
|
||||||
|
self = other;
|
||||||
|
break;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@ -575,20 +583,15 @@ std::map<std::string, Hash> staticOutputHashes(Store & store, const Derivation &
|
||||||
{
|
{
|
||||||
std::map<std::string, Hash> res;
|
std::map<std::string, Hash> res;
|
||||||
std::visit(overloaded {
|
std::visit(overloaded {
|
||||||
[&](const Hash & drvHash) {
|
[&](const DrvHash & drvHash) {
|
||||||
for (auto & outputName : drv.outputNames()) {
|
for (auto & outputName : drv.outputNames()) {
|
||||||
res.insert({outputName, drvHash});
|
res.insert({outputName, drvHash.hash});
|
||||||
}
|
|
||||||
},
|
|
||||||
[&](const DeferredHash & deferredHash) {
|
|
||||||
for (auto & outputName : drv.outputNames()) {
|
|
||||||
res.insert({outputName, deferredHash.hash});
|
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
[&](const CaOutputHashes & outputHashes) {
|
[&](const CaOutputHashes & outputHashes) {
|
||||||
res = outputHashes;
|
res = outputHashes;
|
||||||
},
|
},
|
||||||
}, hashDerivationModulo(store, drv, true));
|
}, hashDerivationModulo(store, drv, true).raw());
|
||||||
return res;
|
return res;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -738,7 +741,7 @@ static void rewriteDerivation(Store & store, BasicDerivation & drv, const String
|
||||||
auto hashModulo = hashDerivationModulo(store, Derivation(drv), true);
|
auto hashModulo = hashDerivationModulo(store, Derivation(drv), true);
|
||||||
for (auto & [outputName, output] : drv.outputs) {
|
for (auto & [outputName, output] : drv.outputs) {
|
||||||
if (std::holds_alternative<DerivationOutputDeferred>(output.output)) {
|
if (std::holds_alternative<DerivationOutputDeferred>(output.output)) {
|
||||||
Hash h = std::get<Hash>(hashModulo);
|
auto & h = hashModulo.requireNoFixedNonDeferred();
|
||||||
auto outPath = store.makeOutputPath(outputName, h, drv.name);
|
auto outPath = store.makeOutputPath(outputName, h, drv.name);
|
||||||
drv.env[outputName] = store.printStorePath(outPath);
|
drv.env[outputName] = store.printStorePath(outPath);
|
||||||
output = DerivationOutput {
|
output = DerivationOutput {
|
||||||
|
@ -751,31 +754,51 @@ static void rewriteDerivation(Store & store, BasicDerivation & drv, const String
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const Hash & DrvHashModulo::requireNoFixedNonDeferred() const {
|
||||||
|
auto * drvHashOpt = std::get_if<DrvHash>(&raw());
|
||||||
|
assert(drvHashOpt);
|
||||||
|
assert(drvHashOpt->kind == DrvHash::Kind::Regular);
|
||||||
|
return drvHashOpt->hash;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool tryResolveInput(
|
||||||
|
Store & store, StorePathSet & inputSrcs, StringMap & inputRewrites,
|
||||||
|
const StorePath & inputDrv, const StringSet & inputOutputs)
|
||||||
|
{
|
||||||
|
auto inputDrvOutputs = store.queryPartialDerivationOutputMap(inputDrv);
|
||||||
|
|
||||||
|
auto getOutput = [&](const std::string & outputName) {
|
||||||
|
auto & actualPathOpt = inputDrvOutputs.at(outputName);
|
||||||
|
if (!actualPathOpt)
|
||||||
|
warn("output %s of input %s missing, aborting the resolving",
|
||||||
|
outputName,
|
||||||
|
store.printStorePath(inputDrv)
|
||||||
|
);
|
||||||
|
return actualPathOpt;
|
||||||
|
};
|
||||||
|
|
||||||
|
for (auto & outputName : inputOutputs) {
|
||||||
|
auto actualPathOpt = getOutput(outputName);
|
||||||
|
if (!actualPathOpt) return false;
|
||||||
|
auto actualPath = *actualPathOpt;
|
||||||
|
inputRewrites.emplace(
|
||||||
|
downstreamPlaceholder(store, inputDrv, outputName),
|
||||||
|
store.printStorePath(actualPath));
|
||||||
|
inputSrcs.insert(std::move(actualPath));
|
||||||
|
}
|
||||||
|
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
std::optional<BasicDerivation> Derivation::tryResolve(Store & store) {
|
std::optional<BasicDerivation> Derivation::tryResolve(Store & store) {
|
||||||
BasicDerivation resolved { *this };
|
BasicDerivation resolved { *this };
|
||||||
|
|
||||||
// Input paths that we'll want to rewrite in the derivation
|
// Input paths that we'll want to rewrite in the derivation
|
||||||
StringMap inputRewrites;
|
StringMap inputRewrites;
|
||||||
|
|
||||||
for (auto & input : inputDrvs) {
|
for (auto & [inputDrv, inputOutputs] : inputDrvs)
|
||||||
auto inputDrvOutputs = store.queryPartialDerivationOutputMap(input.first);
|
if (!tryResolveInput(store, resolved.inputSrcs, inputRewrites, inputDrv, inputOutputs))
|
||||||
StringSet newOutputNames;
|
|
||||||
for (auto & outputName : input.second) {
|
|
||||||
auto actualPathOpt = inputDrvOutputs.at(outputName);
|
|
||||||
if (!actualPathOpt) {
|
|
||||||
warn("output %s of input %s missing, aborting the resolving",
|
|
||||||
outputName,
|
|
||||||
store.printStorePath(input.first)
|
|
||||||
);
|
|
||||||
return std::nullopt;
|
return std::nullopt;
|
||||||
}
|
|
||||||
auto actualPath = *actualPathOpt;
|
|
||||||
inputRewrites.emplace(
|
|
||||||
downstreamPlaceholder(store, input.first, outputName),
|
|
||||||
store.printStorePath(actualPath));
|
|
||||||
resolved.inputSrcs.insert(std::move(actualPath));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
rewriteDerivation(store, resolved, inputRewrites);
|
rewriteDerivation(store, resolved, inputRewrites);
|
||||||
|
|
||||||
|
|
|
@ -175,13 +175,43 @@ std::string outputPathName(std::string_view drvName, std::string_view outputName
|
||||||
// whose output hashes are always known since they are fixed up-front.
|
// whose output hashes are always known since they are fixed up-front.
|
||||||
typedef std::map<std::string, Hash> CaOutputHashes;
|
typedef std::map<std::string, Hash> CaOutputHashes;
|
||||||
|
|
||||||
struct DeferredHash { Hash hash; };
|
struct DrvHash {
|
||||||
|
Hash hash;
|
||||||
|
|
||||||
|
enum struct Kind {
|
||||||
|
// Statically determined derivations.
|
||||||
|
// This hash will be directly used to compute the output paths
|
||||||
|
Regular,
|
||||||
|
// Floating-output derivations (and their dependencies).
|
||||||
|
Deferred,
|
||||||
|
};
|
||||||
|
|
||||||
|
Kind kind;
|
||||||
|
};
|
||||||
|
|
||||||
|
void operator |= (DrvHash::Kind & self, const DrvHash::Kind & other) noexcept;
|
||||||
|
|
||||||
typedef std::variant<
|
typedef std::variant<
|
||||||
Hash, // regular DRV normalized hash
|
// Regular normalized derivation hash, and whether it was deferred (because
|
||||||
CaOutputHashes, // Fixed-output derivation hashes
|
// an ancestor derivation is a floating content addressed derivation).
|
||||||
DeferredHash // Deferred hashes for floating outputs drvs and their dependencies
|
DrvHash,
|
||||||
> DrvHashModulo;
|
// Fixed-output derivation hashes
|
||||||
|
CaOutputHashes
|
||||||
|
> DrvHashModuloRaw;
|
||||||
|
|
||||||
|
struct DrvHashModulo : DrvHashModuloRaw {
|
||||||
|
using Raw = DrvHashModuloRaw;
|
||||||
|
using Raw::Raw;
|
||||||
|
|
||||||
|
/* Get hash, throwing if it is per-output CA hashes or a
|
||||||
|
deferred Drv hash.
|
||||||
|
*/
|
||||||
|
const Hash & requireNoFixedNonDeferred() const;
|
||||||
|
|
||||||
|
inline const Raw & raw() const {
|
||||||
|
return static_cast<const Raw &>(*this);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
/* Returns hashes with the details of fixed-output subderivations
|
/* Returns hashes with the details of fixed-output subderivations
|
||||||
expunged.
|
expunged.
|
||||||
|
|
|
@ -11,6 +11,21 @@ nlohmann::json DerivedPath::Opaque::toJSON(ref<Store> store) const {
|
||||||
return res;
|
return res;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
nlohmann::json DerivedPath::Built::toJSON(ref<Store> store) const {
|
||||||
|
nlohmann::json res;
|
||||||
|
res["drvPath"] = store->printStorePath(drvPath);
|
||||||
|
// Fallback for the input-addressed derivation case: We expect to always be
|
||||||
|
// able to print the output paths, so let’s do it
|
||||||
|
auto knownOutputs = store->queryPartialDerivationOutputMap(drvPath);
|
||||||
|
for (const auto& output : outputs) {
|
||||||
|
if (knownOutputs.at(output))
|
||||||
|
res["outputs"][output] = store->printStorePath(knownOutputs.at(output).value());
|
||||||
|
else
|
||||||
|
res["outputs"][output] = nullptr;
|
||||||
|
}
|
||||||
|
return res;
|
||||||
|
}
|
||||||
|
|
||||||
nlohmann::json BuiltPath::Built::toJSON(ref<Store> store) const {
|
nlohmann::json BuiltPath::Built::toJSON(ref<Store> store) const {
|
||||||
nlohmann::json res;
|
nlohmann::json res;
|
||||||
res["drvPath"] = store->printStorePath(drvPath);
|
res["drvPath"] = store->printStorePath(drvPath);
|
||||||
|
@ -35,16 +50,22 @@ StorePathSet BuiltPath::outPaths() const
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
nlohmann::json derivedPathsWithHintsToJSON(const BuiltPaths & buildables, ref<Store> store) {
|
template<typename T>
|
||||||
|
nlohmann::json stuffToJSON(const std::vector<T> & ts, ref<Store> store) {
|
||||||
auto res = nlohmann::json::array();
|
auto res = nlohmann::json::array();
|
||||||
for (const BuiltPath & buildable : buildables) {
|
for (const T & t : ts) {
|
||||||
std::visit([&res, store](const auto & buildable) {
|
std::visit([&res, store](const auto & t) {
|
||||||
res.push_back(buildable.toJSON(store));
|
res.push_back(t.toJSON(store));
|
||||||
}, buildable.raw());
|
}, t.raw());
|
||||||
}
|
}
|
||||||
return res;
|
return res;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
nlohmann::json derivedPathsWithHintsToJSON(const BuiltPaths & buildables, ref<Store> store)
|
||||||
|
{ return stuffToJSON<BuiltPath>(buildables, store); }
|
||||||
|
nlohmann::json derivedPathsToJSON(const DerivedPaths & paths, ref<Store> store)
|
||||||
|
{ return stuffToJSON<DerivedPath>(paths, store); }
|
||||||
|
|
||||||
|
|
||||||
std::string DerivedPath::Opaque::to_string(const Store & store) const {
|
std::string DerivedPath::Opaque::to_string(const Store & store) const {
|
||||||
return store.printStorePath(path);
|
return store.printStorePath(path);
|
||||||
|
|
|
@ -45,6 +45,7 @@ struct DerivedPathBuilt {
|
||||||
|
|
||||||
std::string to_string(const Store & store) const;
|
std::string to_string(const Store & store) const;
|
||||||
static DerivedPathBuilt parse(const Store & store, std::string_view);
|
static DerivedPathBuilt parse(const Store & store, std::string_view);
|
||||||
|
nlohmann::json toJSON(ref<Store> store) const;
|
||||||
};
|
};
|
||||||
|
|
||||||
using _DerivedPathRaw = std::variant<
|
using _DerivedPathRaw = std::variant<
|
||||||
|
@ -119,5 +120,6 @@ typedef std::vector<DerivedPath> DerivedPaths;
|
||||||
typedef std::vector<BuiltPath> BuiltPaths;
|
typedef std::vector<BuiltPath> BuiltPaths;
|
||||||
|
|
||||||
nlohmann::json derivedPathsWithHintsToJSON(const BuiltPaths & buildables, ref<Store> store);
|
nlohmann::json derivedPathsWithHintsToJSON(const BuiltPaths & buildables, ref<Store> store);
|
||||||
|
nlohmann::json derivedPathsToJSON(const DerivedPaths & , ref<Store> store);
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
84
src/libstore/gc-store.hh
Normal file
84
src/libstore/gc-store.hh
Normal file
|
@ -0,0 +1,84 @@
|
||||||
|
#pragma once
|
||||||
|
|
||||||
|
#include "store-api.hh"
|
||||||
|
|
||||||
|
|
||||||
|
namespace nix {
|
||||||
|
|
||||||
|
|
||||||
|
typedef std::unordered_map<StorePath, std::unordered_set<std::string>> Roots;
|
||||||
|
|
||||||
|
|
||||||
|
struct GCOptions
|
||||||
|
{
|
||||||
|
/* Garbage collector operation:
|
||||||
|
|
||||||
|
- `gcReturnLive': return the set of paths reachable from
|
||||||
|
(i.e. in the closure of) the roots.
|
||||||
|
|
||||||
|
- `gcReturnDead': return the set of paths not reachable from
|
||||||
|
the roots.
|
||||||
|
|
||||||
|
- `gcDeleteDead': actually delete the latter set.
|
||||||
|
|
||||||
|
- `gcDeleteSpecific': delete the paths listed in
|
||||||
|
`pathsToDelete', insofar as they are not reachable.
|
||||||
|
*/
|
||||||
|
typedef enum {
|
||||||
|
gcReturnLive,
|
||||||
|
gcReturnDead,
|
||||||
|
gcDeleteDead,
|
||||||
|
gcDeleteSpecific,
|
||||||
|
} GCAction;
|
||||||
|
|
||||||
|
GCAction action{gcDeleteDead};
|
||||||
|
|
||||||
|
/* If `ignoreLiveness' is set, then reachability from the roots is
|
||||||
|
ignored (dangerous!). However, the paths must still be
|
||||||
|
unreferenced *within* the store (i.e., there can be no other
|
||||||
|
store paths that depend on them). */
|
||||||
|
bool ignoreLiveness{false};
|
||||||
|
|
||||||
|
/* For `gcDeleteSpecific', the paths to delete. */
|
||||||
|
StorePathSet pathsToDelete;
|
||||||
|
|
||||||
|
/* Stop after at least `maxFreed' bytes have been freed. */
|
||||||
|
uint64_t maxFreed{std::numeric_limits<uint64_t>::max()};
|
||||||
|
};
|
||||||
|
|
||||||
|
|
||||||
|
struct GCResults
|
||||||
|
{
|
||||||
|
/* Depending on the action, the GC roots, or the paths that would
|
||||||
|
be or have been deleted. */
|
||||||
|
PathSet paths;
|
||||||
|
|
||||||
|
/* For `gcReturnDead', `gcDeleteDead' and `gcDeleteSpecific', the
|
||||||
|
number of bytes that would be or was freed. */
|
||||||
|
uint64_t bytesFreed = 0;
|
||||||
|
};
|
||||||
|
|
||||||
|
|
||||||
|
struct GcStore : public virtual Store
|
||||||
|
{
|
||||||
|
inline static std::string operationName = "Garbage collection";
|
||||||
|
|
||||||
|
/* Add an indirect root, which is merely a symlink to `path' from
|
||||||
|
/nix/var/nix/gcroots/auto/<hash of `path'>. `path' is supposed
|
||||||
|
to be a symlink to a store path. The garbage collector will
|
||||||
|
automatically remove the indirect root when it finds that
|
||||||
|
`path' has disappeared. */
|
||||||
|
virtual void addIndirectRoot(const Path & path) = 0;
|
||||||
|
|
||||||
|
/* Find the roots of the garbage collector. Each root is a pair
|
||||||
|
(link, storepath) where `link' is the path of the symlink
|
||||||
|
outside of the Nix store that point to `storePath'. If
|
||||||
|
'censor' is true, privacy-sensitive information about roots
|
||||||
|
found in /proc is censored. */
|
||||||
|
virtual Roots findRoots(bool censor) = 0;
|
||||||
|
|
||||||
|
/* Perform a garbage collection. */
|
||||||
|
virtual void collectGarbage(const GCOptions & options, GCResults & results) = 0;
|
||||||
|
};
|
||||||
|
|
||||||
|
}
|
|
@ -678,7 +678,8 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
|
||||||
alive.insert(start);
|
alive.insert(start);
|
||||||
try {
|
try {
|
||||||
StorePathSet closure;
|
StorePathSet closure;
|
||||||
computeFSClosure(*path, closure);
|
computeFSClosure(*path, closure,
|
||||||
|
/* flipDirection */ false, gcKeepOutputs, gcKeepDerivations);
|
||||||
for (auto & p : closure)
|
for (auto & p : closure)
|
||||||
alive.insert(p);
|
alive.insert(p);
|
||||||
} catch (InvalidPath &) { }
|
} catch (InvalidPath &) { }
|
||||||
|
@ -841,7 +842,8 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
|
||||||
if (unlink(path.c_str()) == -1)
|
if (unlink(path.c_str()) == -1)
|
||||||
throw SysError("deleting '%1%'", path);
|
throw SysError("deleting '%1%'", path);
|
||||||
|
|
||||||
results.bytesFreed += st.st_size;
|
/* Do not accound for deleted file here. Rely on deletePath()
|
||||||
|
accounting. */
|
||||||
}
|
}
|
||||||
|
|
||||||
struct stat st;
|
struct stat st;
|
||||||
|
|
|
@ -279,7 +279,7 @@ public:
|
||||||
|
|
||||||
conn->to.flush();
|
conn->to.flush();
|
||||||
|
|
||||||
BuildResult status;
|
BuildResult status { .path = DerivedPath::Built { .drvPath = drvPath } };
|
||||||
status.status = (BuildResult::Status) readInt(conn->from);
|
status.status = (BuildResult::Status) readInt(conn->from);
|
||||||
conn->from >> status.errorMsg;
|
conn->from >> status.errorMsg;
|
||||||
|
|
||||||
|
@ -317,7 +317,7 @@ public:
|
||||||
|
|
||||||
conn->to.flush();
|
conn->to.flush();
|
||||||
|
|
||||||
BuildResult result;
|
BuildResult result { .path = DerivedPath::Opaque { StorePath::dummy } };
|
||||||
result.status = (BuildResult::Status) readInt(conn->from);
|
result.status = (BuildResult::Status) readInt(conn->from);
|
||||||
|
|
||||||
if (!result.success()) {
|
if (!result.success()) {
|
||||||
|
|
|
@ -1,6 +1,8 @@
|
||||||
#pragma once
|
#pragma once
|
||||||
|
|
||||||
#include "store-api.hh"
|
#include "store-api.hh"
|
||||||
|
#include "gc-store.hh"
|
||||||
|
#include "log-store.hh"
|
||||||
|
|
||||||
namespace nix {
|
namespace nix {
|
||||||
|
|
||||||
|
@ -23,7 +25,10 @@ struct LocalFSStoreConfig : virtual StoreConfig
|
||||||
"physical path to the Nix store"};
|
"physical path to the Nix store"};
|
||||||
};
|
};
|
||||||
|
|
||||||
class LocalFSStore : public virtual LocalFSStoreConfig, public virtual Store
|
class LocalFSStore : public virtual LocalFSStoreConfig,
|
||||||
|
public virtual Store,
|
||||||
|
public virtual GcStore,
|
||||||
|
public virtual LogStore
|
||||||
{
|
{
|
||||||
public:
|
public:
|
||||||
|
|
||||||
|
|
|
@ -701,8 +701,8 @@ void LocalStore::checkDerivationOutputs(const StorePath & drvPath, const Derivat
|
||||||
[&](const DerivationOutputInputAddressed & doia) {
|
[&](const DerivationOutputInputAddressed & doia) {
|
||||||
if (!h) {
|
if (!h) {
|
||||||
// somewhat expensive so we do lazily
|
// somewhat expensive so we do lazily
|
||||||
auto temp = hashDerivationModulo(*this, drv, true);
|
auto h0 = hashDerivationModulo(*this, drv, true);
|
||||||
h = std::get<Hash>(temp);
|
h = h0.requireNoFixedNonDeferred();
|
||||||
}
|
}
|
||||||
StorePath recomputed = makeOutputPath(i.first, *h, drvName);
|
StorePath recomputed = makeOutputPath(i.first, *h, drvName);
|
||||||
if (doia.path != recomputed)
|
if (doia.path != recomputed)
|
||||||
|
|
|
@ -5,6 +5,7 @@
|
||||||
#include "pathlocks.hh"
|
#include "pathlocks.hh"
|
||||||
#include "store-api.hh"
|
#include "store-api.hh"
|
||||||
#include "local-fs-store.hh"
|
#include "local-fs-store.hh"
|
||||||
|
#include "gc-store.hh"
|
||||||
#include "sync.hh"
|
#include "sync.hh"
|
||||||
#include "util.hh"
|
#include "util.hh"
|
||||||
|
|
||||||
|
@ -43,7 +44,7 @@ struct LocalStoreConfig : virtual LocalFSStoreConfig
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
||||||
class LocalStore : public virtual LocalStoreConfig, public virtual LocalFSStore
|
class LocalStore : public virtual LocalStoreConfig, public virtual LocalFSStore, public virtual GcStore
|
||||||
{
|
{
|
||||||
private:
|
private:
|
||||||
|
|
||||||
|
|
21
src/libstore/log-store.hh
Normal file
21
src/libstore/log-store.hh
Normal file
|
@ -0,0 +1,21 @@
|
||||||
|
#pragma once
|
||||||
|
|
||||||
|
#include "store-api.hh"
|
||||||
|
|
||||||
|
|
||||||
|
namespace nix {
|
||||||
|
|
||||||
|
struct LogStore : public virtual Store
|
||||||
|
{
|
||||||
|
inline static std::string operationName = "Build log storage and retrieval";
|
||||||
|
|
||||||
|
/* Return the build log of the specified store path, if available,
|
||||||
|
or null otherwise. */
|
||||||
|
virtual std::optional<std::string> getBuildLog(const StorePath & path) = 0;
|
||||||
|
|
||||||
|
virtual void addBuildLog(const StorePath & path, std::string_view log) = 0;
|
||||||
|
|
||||||
|
static LogStore & require(Store & store);
|
||||||
|
};
|
||||||
|
|
||||||
|
}
|
|
@ -1,6 +1,7 @@
|
||||||
#include "serialise.hh"
|
#include "serialise.hh"
|
||||||
#include "util.hh"
|
#include "util.hh"
|
||||||
#include "path-with-outputs.hh"
|
#include "path-with-outputs.hh"
|
||||||
|
#include "gc-store.hh"
|
||||||
#include "remote-fs-accessor.hh"
|
#include "remote-fs-accessor.hh"
|
||||||
#include "build-result.hh"
|
#include "build-result.hh"
|
||||||
#include "remote-store.hh"
|
#include "remote-store.hh"
|
||||||
|
@ -90,6 +91,35 @@ void write(const Store & store, Sink & out, const DrvOutput & drvOutput)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
BuildResult read(const Store & store, Source & from, Phantom<BuildResult> _)
|
||||||
|
{
|
||||||
|
auto path = worker_proto::read(store, from, Phantom<DerivedPath> {});
|
||||||
|
BuildResult res { .path = path };
|
||||||
|
res.status = (BuildResult::Status) readInt(from);
|
||||||
|
from
|
||||||
|
>> res.errorMsg
|
||||||
|
>> res.timesBuilt
|
||||||
|
>> res.isNonDeterministic
|
||||||
|
>> res.startTime
|
||||||
|
>> res.stopTime;
|
||||||
|
res.builtOutputs = worker_proto::read(store, from, Phantom<DrvOutputs> {});
|
||||||
|
return res;
|
||||||
|
}
|
||||||
|
|
||||||
|
void write(const Store & store, Sink & to, const BuildResult & res)
|
||||||
|
{
|
||||||
|
worker_proto::write(store, to, res.path);
|
||||||
|
to
|
||||||
|
<< res.status
|
||||||
|
<< res.errorMsg
|
||||||
|
<< res.timesBuilt
|
||||||
|
<< res.isNonDeterministic
|
||||||
|
<< res.startTime
|
||||||
|
<< res.stopTime;
|
||||||
|
worker_proto::write(store, to, res.builtOutputs);
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
std::optional<StorePath> read(const Store & store, Source & from, Phantom<std::optional<StorePath>> _)
|
std::optional<StorePath> read(const Store & store, Source & from, Phantom<std::optional<StorePath>> _)
|
||||||
{
|
{
|
||||||
auto s = readString(from);
|
auto s = readString(from);
|
||||||
|
@ -746,17 +776,24 @@ static void writeDerivedPaths(RemoteStore & store, ConnectionHandle & conn, cons
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
void RemoteStore::buildPaths(const std::vector<DerivedPath> & drvPaths, BuildMode buildMode, std::shared_ptr<Store> evalStore)
|
void RemoteStore::copyDrvsFromEvalStore(
|
||||||
|
const std::vector<DerivedPath> & paths,
|
||||||
|
std::shared_ptr<Store> evalStore)
|
||||||
{
|
{
|
||||||
if (evalStore && evalStore.get() != this) {
|
if (evalStore && evalStore.get() != this) {
|
||||||
/* The remote doesn't have a way to access evalStore, so copy
|
/* The remote doesn't have a way to access evalStore, so copy
|
||||||
the .drvs. */
|
the .drvs. */
|
||||||
RealisedPath::Set drvPaths2;
|
RealisedPath::Set drvPaths2;
|
||||||
for (auto & i : drvPaths)
|
for (auto & i : paths)
|
||||||
if (auto p = std::get_if<DerivedPath::Built>(&i))
|
if (auto p = std::get_if<DerivedPath::Built>(&i))
|
||||||
drvPaths2.insert(p->drvPath);
|
drvPaths2.insert(p->drvPath);
|
||||||
copyClosure(*evalStore, *this, drvPaths2);
|
copyClosure(*evalStore, *this, drvPaths2);
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
void RemoteStore::buildPaths(const std::vector<DerivedPath> & drvPaths, BuildMode buildMode, std::shared_ptr<Store> evalStore)
|
||||||
|
{
|
||||||
|
copyDrvsFromEvalStore(drvPaths, evalStore);
|
||||||
|
|
||||||
auto conn(getConnection());
|
auto conn(getConnection());
|
||||||
conn->to << wopBuildPaths;
|
conn->to << wopBuildPaths;
|
||||||
|
@ -773,6 +810,91 @@ void RemoteStore::buildPaths(const std::vector<DerivedPath> & drvPaths, BuildMod
|
||||||
readInt(conn->from);
|
readInt(conn->from);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
std::vector<BuildResult> RemoteStore::buildPathsWithResults(
|
||||||
|
const std::vector<DerivedPath> & paths,
|
||||||
|
BuildMode buildMode,
|
||||||
|
std::shared_ptr<Store> evalStore)
|
||||||
|
{
|
||||||
|
copyDrvsFromEvalStore(paths, evalStore);
|
||||||
|
|
||||||
|
std::optional<ConnectionHandle> conn_(getConnection());
|
||||||
|
auto & conn = *conn_;
|
||||||
|
|
||||||
|
if (GET_PROTOCOL_MINOR(conn->daemonVersion) >= 34) {
|
||||||
|
conn->to << wopBuildPathsWithResults;
|
||||||
|
writeDerivedPaths(*this, conn, paths);
|
||||||
|
conn->to << buildMode;
|
||||||
|
conn.processStderr();
|
||||||
|
return worker_proto::read(*this, conn->from, Phantom<std::vector<BuildResult>> {});
|
||||||
|
} else {
|
||||||
|
// Avoid deadlock.
|
||||||
|
conn_.reset();
|
||||||
|
|
||||||
|
// Note: this throws an exception if a build/substitution
|
||||||
|
// fails, but meh.
|
||||||
|
buildPaths(paths, buildMode, evalStore);
|
||||||
|
|
||||||
|
std::vector<BuildResult> results;
|
||||||
|
|
||||||
|
for (auto & path : paths) {
|
||||||
|
std::visit(
|
||||||
|
overloaded {
|
||||||
|
[&](const DerivedPath::Opaque & bo) {
|
||||||
|
results.push_back(BuildResult {
|
||||||
|
.status = BuildResult::Substituted,
|
||||||
|
.path = bo,
|
||||||
|
});
|
||||||
|
},
|
||||||
|
[&](const DerivedPath::Built & bfd) {
|
||||||
|
BuildResult res {
|
||||||
|
.status = BuildResult::Built,
|
||||||
|
.path = bfd,
|
||||||
|
};
|
||||||
|
|
||||||
|
OutputPathMap outputs;
|
||||||
|
auto drv = evalStore->readDerivation(bfd.drvPath);
|
||||||
|
auto outputHashes = staticOutputHashes(*evalStore, drv); // FIXME: expensive
|
||||||
|
auto drvOutputs = drv.outputsAndOptPaths(*this);
|
||||||
|
for (auto & output : bfd.outputs) {
|
||||||
|
if (!outputHashes.count(output))
|
||||||
|
throw Error(
|
||||||
|
"the derivation '%s' doesn't have an output named '%s'",
|
||||||
|
printStorePath(bfd.drvPath), output);
|
||||||
|
auto outputId =
|
||||||
|
DrvOutput{outputHashes.at(output), output};
|
||||||
|
if (settings.isExperimentalFeatureEnabled(Xp::CaDerivations)) {
|
||||||
|
auto realisation =
|
||||||
|
queryRealisation(outputId);
|
||||||
|
if (!realisation)
|
||||||
|
throw Error(
|
||||||
|
"cannot operate on an output of unbuilt "
|
||||||
|
"content-addressed derivation '%s'",
|
||||||
|
outputId.to_string());
|
||||||
|
res.builtOutputs.emplace(realisation->id, *realisation);
|
||||||
|
} else {
|
||||||
|
// If ca-derivations isn't enabled, assume that
|
||||||
|
// the output path is statically known.
|
||||||
|
assert(drvOutputs.count(output));
|
||||||
|
assert(drvOutputs.at(output).second);
|
||||||
|
res.builtOutputs.emplace(
|
||||||
|
outputId,
|
||||||
|
Realisation {
|
||||||
|
.id = outputId,
|
||||||
|
.outPath = *drvOutputs.at(output).second
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
results.push_back(res);
|
||||||
|
}
|
||||||
|
},
|
||||||
|
path.raw());
|
||||||
|
}
|
||||||
|
|
||||||
|
return results;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
BuildResult RemoteStore::buildDerivation(const StorePath & drvPath, const BasicDerivation & drv,
|
BuildResult RemoteStore::buildDerivation(const StorePath & drvPath, const BasicDerivation & drv,
|
||||||
BuildMode buildMode)
|
BuildMode buildMode)
|
||||||
|
@ -782,7 +904,7 @@ BuildResult RemoteStore::buildDerivation(const StorePath & drvPath, const BasicD
|
||||||
writeDerivation(conn->to, *this, drv);
|
writeDerivation(conn->to, *this, drv);
|
||||||
conn->to << buildMode;
|
conn->to << buildMode;
|
||||||
conn.processStderr();
|
conn.processStderr();
|
||||||
BuildResult res;
|
BuildResult res { .path = DerivedPath::Built { .drvPath = drvPath } };
|
||||||
res.status = (BuildResult::Status) readInt(conn->from);
|
res.status = (BuildResult::Status) readInt(conn->from);
|
||||||
conn->from >> res.errorMsg;
|
conn->from >> res.errorMsg;
|
||||||
if (GET_PROTOCOL_MINOR(conn->daemonVersion) >= 29) {
|
if (GET_PROTOCOL_MINOR(conn->daemonVersion) >= 29) {
|
||||||
|
|
|
@ -4,6 +4,8 @@
|
||||||
#include <string>
|
#include <string>
|
||||||
|
|
||||||
#include "store-api.hh"
|
#include "store-api.hh"
|
||||||
|
#include "gc-store.hh"
|
||||||
|
#include "log-store.hh"
|
||||||
|
|
||||||
|
|
||||||
namespace nix {
|
namespace nix {
|
||||||
|
@ -29,7 +31,10 @@ struct RemoteStoreConfig : virtual StoreConfig
|
||||||
|
|
||||||
/* FIXME: RemoteStore is a misnomer - should be something like
|
/* FIXME: RemoteStore is a misnomer - should be something like
|
||||||
DaemonStore. */
|
DaemonStore. */
|
||||||
class RemoteStore : public virtual RemoteStoreConfig, public virtual Store
|
class RemoteStore : public virtual RemoteStoreConfig,
|
||||||
|
public virtual Store,
|
||||||
|
public virtual GcStore,
|
||||||
|
public virtual LogStore
|
||||||
{
|
{
|
||||||
public:
|
public:
|
||||||
|
|
||||||
|
@ -96,6 +101,11 @@ public:
|
||||||
|
|
||||||
void buildPaths(const std::vector<DerivedPath> & paths, BuildMode buildMode, std::shared_ptr<Store> evalStore) override;
|
void buildPaths(const std::vector<DerivedPath> & paths, BuildMode buildMode, std::shared_ptr<Store> evalStore) override;
|
||||||
|
|
||||||
|
std::vector<BuildResult> buildPathsWithResults(
|
||||||
|
const std::vector<DerivedPath> & paths,
|
||||||
|
BuildMode buildMode,
|
||||||
|
std::shared_ptr<Store> evalStore) override;
|
||||||
|
|
||||||
BuildResult buildDerivation(const StorePath & drvPath, const BasicDerivation & drv,
|
BuildResult buildDerivation(const StorePath & drvPath, const BasicDerivation & drv,
|
||||||
BuildMode buildMode) override;
|
BuildMode buildMode) override;
|
||||||
|
|
||||||
|
@ -170,6 +180,9 @@ private:
|
||||||
|
|
||||||
std::atomic_bool failed{false};
|
std::atomic_bool failed{false};
|
||||||
|
|
||||||
|
void copyDrvsFromEvalStore(
|
||||||
|
const std::vector<DerivedPath> & paths,
|
||||||
|
std::shared_ptr<Store> evalStore);
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
||||||
|
|
|
@ -52,6 +52,10 @@ public:
|
||||||
bool sameMachine() override
|
bool sameMachine() override
|
||||||
{ return false; }
|
{ return false; }
|
||||||
|
|
||||||
|
// FIXME extend daemon protocol, move implementation to RemoteStore
|
||||||
|
std::optional<std::string> getBuildLog(const StorePath & path) override
|
||||||
|
{ unsupported("getBuildLog"); }
|
||||||
|
|
||||||
private:
|
private:
|
||||||
|
|
||||||
struct Connection : RemoteStore::Connection
|
struct Connection : RemoteStore::Connection
|
||||||
|
|
|
@ -76,59 +76,6 @@ enum AllowInvalidFlag : bool { DisallowInvalid = false, AllowInvalid = true };
|
||||||
const uint32_t exportMagic = 0x4558494e;
|
const uint32_t exportMagic = 0x4558494e;
|
||||||
|
|
||||||
|
|
||||||
typedef std::unordered_map<StorePath, std::unordered_set<std::string>> Roots;
|
|
||||||
|
|
||||||
|
|
||||||
struct GCOptions
|
|
||||||
{
|
|
||||||
/* Garbage collector operation:
|
|
||||||
|
|
||||||
- `gcReturnLive': return the set of paths reachable from
|
|
||||||
(i.e. in the closure of) the roots.
|
|
||||||
|
|
||||||
- `gcReturnDead': return the set of paths not reachable from
|
|
||||||
the roots.
|
|
||||||
|
|
||||||
- `gcDeleteDead': actually delete the latter set.
|
|
||||||
|
|
||||||
- `gcDeleteSpecific': delete the paths listed in
|
|
||||||
`pathsToDelete', insofar as they are not reachable.
|
|
||||||
*/
|
|
||||||
typedef enum {
|
|
||||||
gcReturnLive,
|
|
||||||
gcReturnDead,
|
|
||||||
gcDeleteDead,
|
|
||||||
gcDeleteSpecific,
|
|
||||||
} GCAction;
|
|
||||||
|
|
||||||
GCAction action{gcDeleteDead};
|
|
||||||
|
|
||||||
/* If `ignoreLiveness' is set, then reachability from the roots is
|
|
||||||
ignored (dangerous!). However, the paths must still be
|
|
||||||
unreferenced *within* the store (i.e., there can be no other
|
|
||||||
store paths that depend on them). */
|
|
||||||
bool ignoreLiveness{false};
|
|
||||||
|
|
||||||
/* For `gcDeleteSpecific', the paths to delete. */
|
|
||||||
StorePathSet pathsToDelete;
|
|
||||||
|
|
||||||
/* Stop after at least `maxFreed' bytes have been freed. */
|
|
||||||
uint64_t maxFreed{std::numeric_limits<uint64_t>::max()};
|
|
||||||
};
|
|
||||||
|
|
||||||
|
|
||||||
struct GCResults
|
|
||||||
{
|
|
||||||
/* Depending on the action, the GC roots, or the paths that would
|
|
||||||
be or have been deleted. */
|
|
||||||
PathSet paths;
|
|
||||||
|
|
||||||
/* For `gcReturnDead', `gcDeleteDead' and `gcDeleteSpecific', the
|
|
||||||
number of bytes that would be or was freed. */
|
|
||||||
uint64_t bytesFreed = 0;
|
|
||||||
};
|
|
||||||
|
|
||||||
|
|
||||||
enum BuildMode { bmNormal, bmRepair, bmCheck };
|
enum BuildMode { bmNormal, bmRepair, bmCheck };
|
||||||
|
|
||||||
struct BuildResult;
|
struct BuildResult;
|
||||||
|
@ -485,6 +432,16 @@ public:
|
||||||
BuildMode buildMode = bmNormal,
|
BuildMode buildMode = bmNormal,
|
||||||
std::shared_ptr<Store> evalStore = nullptr);
|
std::shared_ptr<Store> evalStore = nullptr);
|
||||||
|
|
||||||
|
/* Like `buildPaths()`, but return a vector of `BuildResult`s
|
||||||
|
corresponding to each element in `paths`. Note that in case of
|
||||||
|
a build/substitution error, this function won't throw an
|
||||||
|
exception, but return a `BuildResult` containing an error
|
||||||
|
message. */
|
||||||
|
virtual std::vector<BuildResult> buildPathsWithResults(
|
||||||
|
const std::vector<DerivedPath> & paths,
|
||||||
|
BuildMode buildMode = bmNormal,
|
||||||
|
std::shared_ptr<Store> evalStore = nullptr);
|
||||||
|
|
||||||
/* Build a single non-materialized derivation (i.e. not from an
|
/* Build a single non-materialized derivation (i.e. not from an
|
||||||
on-disk .drv file).
|
on-disk .drv file).
|
||||||
|
|
||||||
|
@ -531,26 +488,6 @@ public:
|
||||||
virtual void addTempRoot(const StorePath & path)
|
virtual void addTempRoot(const StorePath & path)
|
||||||
{ debug("not creating temporary root, store doesn't support GC"); }
|
{ debug("not creating temporary root, store doesn't support GC"); }
|
||||||
|
|
||||||
/* Add an indirect root, which is merely a symlink to `path' from
|
|
||||||
/nix/var/nix/gcroots/auto/<hash of `path'>. `path' is supposed
|
|
||||||
to be a symlink to a store path. The garbage collector will
|
|
||||||
automatically remove the indirect root when it finds that
|
|
||||||
`path' has disappeared. */
|
|
||||||
virtual void addIndirectRoot(const Path & path)
|
|
||||||
{ unsupported("addIndirectRoot"); }
|
|
||||||
|
|
||||||
/* Find the roots of the garbage collector. Each root is a pair
|
|
||||||
(link, storepath) where `link' is the path of the symlink
|
|
||||||
outside of the Nix store that point to `storePath'. If
|
|
||||||
'censor' is true, privacy-sensitive information about roots
|
|
||||||
found in /proc is censored. */
|
|
||||||
virtual Roots findRoots(bool censor)
|
|
||||||
{ unsupported("findRoots"); }
|
|
||||||
|
|
||||||
/* Perform a garbage collection. */
|
|
||||||
virtual void collectGarbage(const GCOptions & options, GCResults & results)
|
|
||||||
{ unsupported("collectGarbage"); }
|
|
||||||
|
|
||||||
/* Return a string representing information about the path that
|
/* Return a string representing information about the path that
|
||||||
can be loaded into the database using `nix-store --load-db' or
|
can be loaded into the database using `nix-store --load-db' or
|
||||||
`nix-store --register-validity'. */
|
`nix-store --register-validity'. */
|
||||||
|
@ -668,14 +605,6 @@ public:
|
||||||
*/
|
*/
|
||||||
StorePathSet exportReferences(const StorePathSet & storePaths, const StorePathSet & inputPaths);
|
StorePathSet exportReferences(const StorePathSet & storePaths, const StorePathSet & inputPaths);
|
||||||
|
|
||||||
/* Return the build log of the specified store path, if available,
|
|
||||||
or null otherwise. */
|
|
||||||
virtual std::optional<std::string> getBuildLog(const StorePath & path)
|
|
||||||
{ return std::nullopt; }
|
|
||||||
|
|
||||||
virtual void addBuildLog(const StorePath & path, std::string_view log)
|
|
||||||
{ unsupported("addBuildLog"); }
|
|
||||||
|
|
||||||
/* Hack to allow long-running processes like hydra-queue-runner to
|
/* Hack to allow long-running processes like hydra-queue-runner to
|
||||||
occasionally flush their path info cache. */
|
occasionally flush their path info cache. */
|
||||||
void clearPathInfoCache()
|
void clearPathInfoCache()
|
||||||
|
|
16
src/libstore/store-cast.hh
Normal file
16
src/libstore/store-cast.hh
Normal file
|
@ -0,0 +1,16 @@
|
||||||
|
#pragma once
|
||||||
|
|
||||||
|
#include "store-api.hh"
|
||||||
|
|
||||||
|
namespace nix {
|
||||||
|
|
||||||
|
template<typename T>
|
||||||
|
T & require(Store & store)
|
||||||
|
{
|
||||||
|
auto * castedStore = dynamic_cast<T *>(&store);
|
||||||
|
if (!castedStore)
|
||||||
|
throw UsageError("%s not supported by store '%s'", T::operationName, store.getUri());
|
||||||
|
return *castedStore;
|
||||||
|
}
|
||||||
|
|
||||||
|
}
|
|
@ -9,7 +9,7 @@ namespace nix {
|
||||||
#define WORKER_MAGIC_1 0x6e697863
|
#define WORKER_MAGIC_1 0x6e697863
|
||||||
#define WORKER_MAGIC_2 0x6478696f
|
#define WORKER_MAGIC_2 0x6478696f
|
||||||
|
|
||||||
#define PROTOCOL_VERSION (1 << 8 | 33)
|
#define PROTOCOL_VERSION (1 << 8 | 34)
|
||||||
#define GET_PROTOCOL_MAJOR(x) ((x) & 0xff00)
|
#define GET_PROTOCOL_MAJOR(x) ((x) & 0xff00)
|
||||||
#define GET_PROTOCOL_MINOR(x) ((x) & 0x00ff)
|
#define GET_PROTOCOL_MINOR(x) ((x) & 0x00ff)
|
||||||
|
|
||||||
|
@ -57,6 +57,7 @@ typedef enum {
|
||||||
wopQueryRealisation = 43,
|
wopQueryRealisation = 43,
|
||||||
wopAddMultipleToStore = 44,
|
wopAddMultipleToStore = 44,
|
||||||
wopAddBuildLog = 45,
|
wopAddBuildLog = 45,
|
||||||
|
wopBuildPathsWithResults = 46,
|
||||||
} WorkerOp;
|
} WorkerOp;
|
||||||
|
|
||||||
|
|
||||||
|
@ -91,6 +92,7 @@ MAKE_WORKER_PROTO(, ContentAddress);
|
||||||
MAKE_WORKER_PROTO(, DerivedPath);
|
MAKE_WORKER_PROTO(, DerivedPath);
|
||||||
MAKE_WORKER_PROTO(, Realisation);
|
MAKE_WORKER_PROTO(, Realisation);
|
||||||
MAKE_WORKER_PROTO(, DrvOutput);
|
MAKE_WORKER_PROTO(, DrvOutput);
|
||||||
|
MAKE_WORKER_PROTO(, BuildResult);
|
||||||
|
|
||||||
MAKE_WORKER_PROTO(template<typename T>, std::vector<T>);
|
MAKE_WORKER_PROTO(template<typename T>, std::vector<T>);
|
||||||
MAKE_WORKER_PROTO(template<typename T>, std::set<T>);
|
MAKE_WORKER_PROTO(template<typename T>, std::set<T>);
|
||||||
|
|
|
@ -64,11 +64,12 @@ static void dumpContents(const Path & path, off_t size,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
static void dump(const Path & path, Sink & sink, PathFilter & filter)
|
static time_t dump(const Path & path, Sink & sink, PathFilter & filter)
|
||||||
{
|
{
|
||||||
checkInterrupt();
|
checkInterrupt();
|
||||||
|
|
||||||
auto st = lstat(path);
|
auto st = lstat(path);
|
||||||
|
time_t result = st.st_mtime;
|
||||||
|
|
||||||
sink << "(";
|
sink << "(";
|
||||||
|
|
||||||
|
@ -103,7 +104,10 @@ static void dump(const Path & path, Sink & sink, PathFilter & filter)
|
||||||
for (auto & i : unhacked)
|
for (auto & i : unhacked)
|
||||||
if (filter(path + "/" + i.first)) {
|
if (filter(path + "/" + i.first)) {
|
||||||
sink << "entry" << "(" << "name" << i.first << "node";
|
sink << "entry" << "(" << "name" << i.first << "node";
|
||||||
dump(path + "/" + i.second, sink, filter);
|
auto tmp_mtime = dump(path + "/" + i.second, sink, filter);
|
||||||
|
if (tmp_mtime > result) {
|
||||||
|
result = tmp_mtime;
|
||||||
|
}
|
||||||
sink << ")";
|
sink << ")";
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
@ -114,13 +118,20 @@ static void dump(const Path & path, Sink & sink, PathFilter & filter)
|
||||||
else throw Error("file '%1%' has an unsupported type", path);
|
else throw Error("file '%1%' has an unsupported type", path);
|
||||||
|
|
||||||
sink << ")";
|
sink << ")";
|
||||||
|
|
||||||
|
return result;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
void dumpPath(const Path & path, Sink & sink, PathFilter & filter)
|
time_t dumpPathAndGetMtime(const Path & path, Sink & sink, PathFilter & filter)
|
||||||
{
|
{
|
||||||
sink << narVersionMagic1;
|
sink << narVersionMagic1;
|
||||||
dump(path, sink, filter);
|
return dump(path, sink, filter);
|
||||||
|
}
|
||||||
|
|
||||||
|
void dumpPath(const Path & path, Sink & sink, PathFilter & filter)
|
||||||
|
{
|
||||||
|
dumpPathAndGetMtime(path, sink, filter);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|
|
@ -48,6 +48,10 @@ namespace nix {
|
||||||
void dumpPath(const Path & path, Sink & sink,
|
void dumpPath(const Path & path, Sink & sink,
|
||||||
PathFilter & filter = defaultPathFilter);
|
PathFilter & filter = defaultPathFilter);
|
||||||
|
|
||||||
|
/* Same as `void dumpPath()`, but returns the last modified date of the path */
|
||||||
|
time_t dumpPathAndGetMtime(const Path & path, Sink & sink,
|
||||||
|
PathFilter & filter = defaultPathFilter);
|
||||||
|
|
||||||
void dumpString(std::string_view s, Sink & sink);
|
void dumpString(std::string_view s, Sink & sink);
|
||||||
|
|
||||||
/* FIXME: fix this API, it sucks. */
|
/* FIXME: fix this API, it sucks. */
|
||||||
|
|
|
@ -328,8 +328,13 @@ MultiCommand::MultiCommand(const Commands & commands_)
|
||||||
completions->add(name);
|
completions->add(name);
|
||||||
}
|
}
|
||||||
auto i = commands.find(s);
|
auto i = commands.find(s);
|
||||||
if (i == commands.end())
|
if (i == commands.end()) {
|
||||||
throw UsageError("'%s' is not a recognised command", s);
|
std::set<std::string> commandNames;
|
||||||
|
for (auto & [name, _] : commands)
|
||||||
|
commandNames.insert(name);
|
||||||
|
auto suggestions = Suggestions::bestMatches(commandNames, s);
|
||||||
|
throw UsageError(suggestions, "'%s' is not a recognised command", s);
|
||||||
|
}
|
||||||
command = {s, i->second()};
|
command = {s, i->second()};
|
||||||
command->second->parent = this;
|
command->second->parent = this;
|
||||||
}}
|
}}
|
||||||
|
|
|
@ -281,6 +281,13 @@ std::ostream & showErrorInfo(std::ostream & out, const ErrorInfo & einfo, bool s
|
||||||
oss << "\n";
|
oss << "\n";
|
||||||
}
|
}
|
||||||
|
|
||||||
|
auto suggestions = einfo.suggestions.trim();
|
||||||
|
if (! suggestions.suggestions.empty()){
|
||||||
|
oss << "Did you mean " <<
|
||||||
|
suggestions.trim() <<
|
||||||
|
"?" << std::endl;
|
||||||
|
}
|
||||||
|
|
||||||
// traces
|
// traces
|
||||||
if (!einfo.traces.empty()) {
|
if (!einfo.traces.empty()) {
|
||||||
unsigned int count = 0;
|
unsigned int count = 0;
|
||||||
|
|
|
@ -1,5 +1,6 @@
|
||||||
#pragma once
|
#pragma once
|
||||||
|
|
||||||
|
#include "suggestions.hh"
|
||||||
#include "ref.hh"
|
#include "ref.hh"
|
||||||
#include "types.hh"
|
#include "types.hh"
|
||||||
#include "fmt.hh"
|
#include "fmt.hh"
|
||||||
|
@ -53,6 +54,7 @@ typedef enum {
|
||||||
lvlVomit
|
lvlVomit
|
||||||
} Verbosity;
|
} Verbosity;
|
||||||
|
|
||||||
|
/* adjust Pos::origin bit width when adding stuff here */
|
||||||
typedef enum {
|
typedef enum {
|
||||||
foFile,
|
foFile,
|
||||||
foStdin,
|
foStdin,
|
||||||
|
@ -112,6 +114,8 @@ struct ErrorInfo {
|
||||||
std::optional<ErrPos> errPos;
|
std::optional<ErrPos> errPos;
|
||||||
std::list<Trace> traces;
|
std::list<Trace> traces;
|
||||||
|
|
||||||
|
Suggestions suggestions;
|
||||||
|
|
||||||
static std::optional<std::string> programName;
|
static std::optional<std::string> programName;
|
||||||
};
|
};
|
||||||
|
|
||||||
|
@ -143,6 +147,11 @@ public:
|
||||||
: err { .level = lvlError, .msg = hintfmt(fs, args...) }
|
: err { .level = lvlError, .msg = hintfmt(fs, args...) }
|
||||||
{ }
|
{ }
|
||||||
|
|
||||||
|
template<typename... Args>
|
||||||
|
BaseError(const Suggestions & sug, const Args & ... args)
|
||||||
|
: err { .level = lvlError, .msg = hintfmt(args...), .suggestions = sug }
|
||||||
|
{ }
|
||||||
|
|
||||||
BaseError(hintformat hint)
|
BaseError(hintformat hint)
|
||||||
: err { .level = lvlError, .msg = hint }
|
: err { .level = lvlError, .msg = hint }
|
||||||
{ }
|
{ }
|
||||||
|
|
|
@ -1,14 +1,13 @@
|
||||||
#pragma once
|
#pragma once
|
||||||
|
|
||||||
#include <functional>
|
|
||||||
|
|
||||||
/* A trivial class to run a function at the end of a scope. */
|
/* A trivial class to run a function at the end of a scope. */
|
||||||
|
template<typename Fn>
|
||||||
class Finally
|
class Finally
|
||||||
{
|
{
|
||||||
private:
|
private:
|
||||||
std::function<void()> fun;
|
Fn fun;
|
||||||
|
|
||||||
public:
|
public:
|
||||||
Finally(std::function<void()> fun) : fun(fun) { }
|
Finally(Fn fun) : fun(std::move(fun)) { }
|
||||||
~Finally() { fun(); }
|
~Finally() { fun(); }
|
||||||
};
|
};
|
||||||
|
|
|
@ -266,14 +266,21 @@ static Logger::Fields getFields(nlohmann::json & json)
|
||||||
return fields;
|
return fields;
|
||||||
}
|
}
|
||||||
|
|
||||||
bool handleJSONLogMessage(const std::string & msg,
|
std::optional<nlohmann::json> parseJSONMessage(const std::string & msg)
|
||||||
const Activity & act, std::map<ActivityId, Activity> & activities, bool trusted)
|
|
||||||
{
|
{
|
||||||
if (!hasPrefix(msg, "@nix ")) return false;
|
if (!hasPrefix(msg, "@nix ")) return std::nullopt;
|
||||||
|
|
||||||
try {
|
try {
|
||||||
auto json = nlohmann::json::parse(std::string(msg, 5));
|
return nlohmann::json::parse(std::string(msg, 5));
|
||||||
|
} catch (std::exception & e) {
|
||||||
|
printError("bad JSON log message from builder: %s", e.what());
|
||||||
|
}
|
||||||
|
return std::nullopt;
|
||||||
|
}
|
||||||
|
|
||||||
|
bool handleJSONLogMessage(nlohmann::json & json,
|
||||||
|
const Activity & act, std::map<ActivityId, Activity> & activities,
|
||||||
|
bool trusted)
|
||||||
|
{
|
||||||
std::string action = json["action"];
|
std::string action = json["action"];
|
||||||
|
|
||||||
if (action == "start") {
|
if (action == "start") {
|
||||||
|
@ -304,13 +311,18 @@ bool handleJSONLogMessage(const std::string & msg,
|
||||||
logger->log((Verbosity) json["level"], msg);
|
logger->log((Verbosity) json["level"], msg);
|
||||||
}
|
}
|
||||||
|
|
||||||
} catch (std::exception & e) {
|
|
||||||
printError("bad JSON log message from builder: %s", e.what());
|
|
||||||
}
|
|
||||||
|
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
bool handleJSONLogMessage(const std::string & msg,
|
||||||
|
const Activity & act, std::map<ActivityId, Activity> & activities, bool trusted)
|
||||||
|
{
|
||||||
|
auto json = parseJSONMessage(msg);
|
||||||
|
if (!json) return false;
|
||||||
|
|
||||||
|
return handleJSONLogMessage(*json, act, activities, trusted);
|
||||||
|
}
|
||||||
|
|
||||||
Activity::~Activity()
|
Activity::~Activity()
|
||||||
{
|
{
|
||||||
try {
|
try {
|
||||||
|
|
|
@ -4,6 +4,8 @@
|
||||||
#include "error.hh"
|
#include "error.hh"
|
||||||
#include "config.hh"
|
#include "config.hh"
|
||||||
|
|
||||||
|
#include <nlohmann/json_fwd.hpp>
|
||||||
|
|
||||||
namespace nix {
|
namespace nix {
|
||||||
|
|
||||||
typedef enum {
|
typedef enum {
|
||||||
|
@ -166,6 +168,12 @@ Logger * makeSimpleLogger(bool printBuildLogs = true);
|
||||||
|
|
||||||
Logger * makeJSONLogger(Logger & prevLogger);
|
Logger * makeJSONLogger(Logger & prevLogger);
|
||||||
|
|
||||||
|
std::optional<nlohmann::json> parseJSONMessage(const std::string & msg);
|
||||||
|
|
||||||
|
bool handleJSONLogMessage(nlohmann::json & json,
|
||||||
|
const Activity & act, std::map<ActivityId, Activity> & activities,
|
||||||
|
bool trusted);
|
||||||
|
|
||||||
bool handleJSONLogMessage(const std::string & msg,
|
bool handleJSONLogMessage(const std::string & msg,
|
||||||
const Activity & act, std::map<ActivityId, Activity> & activities,
|
const Activity & act, std::map<ActivityId, Activity> & activities,
|
||||||
bool trusted);
|
bool trusted);
|
||||||
|
|
114
src/libutil/suggestions.cc
Normal file
114
src/libutil/suggestions.cc
Normal file
|
@ -0,0 +1,114 @@
|
||||||
|
#include "suggestions.hh"
|
||||||
|
#include "ansicolor.hh"
|
||||||
|
#include "util.hh"
|
||||||
|
#include <algorithm>
|
||||||
|
|
||||||
|
namespace nix {
|
||||||
|
|
||||||
|
int levenshteinDistance(std::string_view first, std::string_view second)
|
||||||
|
{
|
||||||
|
// Implementation borrowed from
|
||||||
|
// https://en.wikipedia.org/wiki/Levenshtein_distance#Iterative_with_two_matrix_rows
|
||||||
|
|
||||||
|
int m = first.size();
|
||||||
|
int n = second.size();
|
||||||
|
|
||||||
|
auto v0 = std::vector<int>(n+1);
|
||||||
|
auto v1 = std::vector<int>(n+1);
|
||||||
|
|
||||||
|
for (auto i = 0; i <= n; i++)
|
||||||
|
v0[i] = i;
|
||||||
|
|
||||||
|
for (auto i = 0; i < m; i++) {
|
||||||
|
v1[0] = i+1;
|
||||||
|
|
||||||
|
for (auto j = 0; j < n; j++) {
|
||||||
|
auto deletionCost = v0[j+1] + 1;
|
||||||
|
auto insertionCost = v1[j] + 1;
|
||||||
|
auto substitutionCost = first[i] == second[j] ? v0[j] : v0[j] + 1;
|
||||||
|
v1[j+1] = std::min({deletionCost, insertionCost, substitutionCost});
|
||||||
|
}
|
||||||
|
|
||||||
|
std::swap(v0, v1);
|
||||||
|
}
|
||||||
|
|
||||||
|
return v0[n];
|
||||||
|
}
|
||||||
|
|
||||||
|
Suggestions Suggestions::bestMatches (
|
||||||
|
std::set<std::string> allMatches,
|
||||||
|
std::string query)
|
||||||
|
{
|
||||||
|
std::set<Suggestion> res;
|
||||||
|
for (const auto & possibleMatch : allMatches) {
|
||||||
|
res.insert(Suggestion {
|
||||||
|
.distance = levenshteinDistance(query, possibleMatch),
|
||||||
|
.suggestion = possibleMatch,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return Suggestions { res };
|
||||||
|
}
|
||||||
|
|
||||||
|
Suggestions Suggestions::trim(int limit, int maxDistance) const
|
||||||
|
{
|
||||||
|
std::set<Suggestion> res;
|
||||||
|
|
||||||
|
int count = 0;
|
||||||
|
|
||||||
|
for (auto & elt : suggestions) {
|
||||||
|
if (count >= limit || elt.distance > maxDistance)
|
||||||
|
break;
|
||||||
|
count++;
|
||||||
|
res.insert(elt);
|
||||||
|
}
|
||||||
|
|
||||||
|
return Suggestions{res};
|
||||||
|
}
|
||||||
|
|
||||||
|
std::string Suggestion::to_string() const
|
||||||
|
{
|
||||||
|
return ANSI_WARNING + filterANSIEscapes(suggestion) + ANSI_NORMAL;
|
||||||
|
}
|
||||||
|
|
||||||
|
std::string Suggestions::to_string() const
|
||||||
|
{
|
||||||
|
switch (suggestions.size()) {
|
||||||
|
case 0:
|
||||||
|
return "";
|
||||||
|
case 1:
|
||||||
|
return suggestions.begin()->to_string();
|
||||||
|
default: {
|
||||||
|
std::string res = "one of ";
|
||||||
|
auto iter = suggestions.begin();
|
||||||
|
res += iter->to_string(); // Iter can’t be end() because the container isn’t null
|
||||||
|
iter++;
|
||||||
|
auto last = suggestions.end(); last--;
|
||||||
|
for ( ; iter != suggestions.end() ; iter++) {
|
||||||
|
res += (iter == last) ? " or " : ", ";
|
||||||
|
res += iter->to_string();
|
||||||
|
}
|
||||||
|
return res;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Suggestions & Suggestions::operator+=(const Suggestions & other)
|
||||||
|
{
|
||||||
|
suggestions.insert(
|
||||||
|
other.suggestions.begin(),
|
||||||
|
other.suggestions.end()
|
||||||
|
);
|
||||||
|
return *this;
|
||||||
|
}
|
||||||
|
|
||||||
|
std::ostream & operator<<(std::ostream & str, const Suggestion & suggestion)
|
||||||
|
{
|
||||||
|
return str << suggestion.to_string();
|
||||||
|
}
|
||||||
|
|
||||||
|
std::ostream & operator<<(std::ostream & str, const Suggestions & suggestions)
|
||||||
|
{
|
||||||
|
return str << suggestions.to_string();
|
||||||
|
}
|
||||||
|
|
||||||
|
}
|
102
src/libutil/suggestions.hh
Normal file
102
src/libutil/suggestions.hh
Normal file
|
@ -0,0 +1,102 @@
|
||||||
|
#pragma once
|
||||||
|
|
||||||
|
#include "comparator.hh"
|
||||||
|
#include "types.hh"
|
||||||
|
#include <set>
|
||||||
|
|
||||||
|
namespace nix {
|
||||||
|
|
||||||
|
int levenshteinDistance(std::string_view first, std::string_view second);
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A potential suggestion for the cli interface.
|
||||||
|
*/
|
||||||
|
class Suggestion {
|
||||||
|
public:
|
||||||
|
int distance; // The smaller the better
|
||||||
|
std::string suggestion;
|
||||||
|
|
||||||
|
std::string to_string() const;
|
||||||
|
|
||||||
|
GENERATE_CMP(Suggestion, me->distance, me->suggestion)
|
||||||
|
};
|
||||||
|
|
||||||
|
class Suggestions {
|
||||||
|
public:
|
||||||
|
std::set<Suggestion> suggestions;
|
||||||
|
|
||||||
|
std::string to_string() const;
|
||||||
|
|
||||||
|
Suggestions trim(
|
||||||
|
int limit = 5,
|
||||||
|
int maxDistance = 2
|
||||||
|
) const;
|
||||||
|
|
||||||
|
static Suggestions bestMatches (
|
||||||
|
std::set<std::string> allMatches,
|
||||||
|
std::string query
|
||||||
|
);
|
||||||
|
|
||||||
|
Suggestions& operator+=(const Suggestions & other);
|
||||||
|
};
|
||||||
|
|
||||||
|
std::ostream & operator<<(std::ostream & str, const Suggestion &);
|
||||||
|
std::ostream & operator<<(std::ostream & str, const Suggestions &);
|
||||||
|
|
||||||
|
// Either a value of type `T`, or some suggestions
|
||||||
|
template<typename T>
|
||||||
|
class OrSuggestions {
|
||||||
|
public:
|
||||||
|
using Raw = std::variant<T, Suggestions>;
|
||||||
|
|
||||||
|
Raw raw;
|
||||||
|
|
||||||
|
T* operator ->()
|
||||||
|
{
|
||||||
|
return &**this;
|
||||||
|
}
|
||||||
|
|
||||||
|
T& operator *()
|
||||||
|
{
|
||||||
|
return std::get<T>(raw);
|
||||||
|
}
|
||||||
|
|
||||||
|
operator bool() const noexcept
|
||||||
|
{
|
||||||
|
return std::holds_alternative<T>(raw);
|
||||||
|
}
|
||||||
|
|
||||||
|
OrSuggestions(T t)
|
||||||
|
: raw(t)
|
||||||
|
{
|
||||||
|
}
|
||||||
|
|
||||||
|
OrSuggestions()
|
||||||
|
: raw(Suggestions{})
|
||||||
|
{
|
||||||
|
}
|
||||||
|
|
||||||
|
static OrSuggestions<T> failed(const Suggestions & s)
|
||||||
|
{
|
||||||
|
auto res = OrSuggestions<T>();
|
||||||
|
res.raw = s;
|
||||||
|
return res;
|
||||||
|
}
|
||||||
|
|
||||||
|
static OrSuggestions<T> failed()
|
||||||
|
{
|
||||||
|
return OrSuggestions<T>::failed(Suggestions{});
|
||||||
|
}
|
||||||
|
|
||||||
|
const Suggestions & getSuggestions()
|
||||||
|
{
|
||||||
|
static Suggestions noSuggestions;
|
||||||
|
if (const auto & suggestions = std::get_if<Suggestions>(&raw))
|
||||||
|
return *suggestions;
|
||||||
|
else
|
||||||
|
return noSuggestions;
|
||||||
|
}
|
||||||
|
|
||||||
|
};
|
||||||
|
|
||||||
|
}
|
43
src/libutil/tests/suggestions.cc
Normal file
43
src/libutil/tests/suggestions.cc
Normal file
|
@ -0,0 +1,43 @@
|
||||||
|
#include "suggestions.hh"
|
||||||
|
#include <gtest/gtest.h>
|
||||||
|
|
||||||
|
namespace nix {
|
||||||
|
|
||||||
|
struct LevenshteinDistanceParam {
|
||||||
|
std::string s1, s2;
|
||||||
|
int distance;
|
||||||
|
};
|
||||||
|
|
||||||
|
class LevenshteinDistanceTest :
|
||||||
|
public testing::TestWithParam<LevenshteinDistanceParam> {
|
||||||
|
};
|
||||||
|
|
||||||
|
TEST_P(LevenshteinDistanceTest, CorrectlyComputed) {
|
||||||
|
auto params = GetParam();
|
||||||
|
|
||||||
|
ASSERT_EQ(levenshteinDistance(params.s1, params.s2), params.distance);
|
||||||
|
ASSERT_EQ(levenshteinDistance(params.s2, params.s1), params.distance);
|
||||||
|
}
|
||||||
|
|
||||||
|
INSTANTIATE_TEST_SUITE_P(LevenshteinDistance, LevenshteinDistanceTest,
|
||||||
|
testing::Values(
|
||||||
|
LevenshteinDistanceParam{"foo", "foo", 0},
|
||||||
|
LevenshteinDistanceParam{"foo", "", 3},
|
||||||
|
LevenshteinDistanceParam{"", "", 0},
|
||||||
|
LevenshteinDistanceParam{"foo", "fo", 1},
|
||||||
|
LevenshteinDistanceParam{"foo", "oo", 1},
|
||||||
|
LevenshteinDistanceParam{"foo", "fao", 1},
|
||||||
|
LevenshteinDistanceParam{"foo", "abc", 3}
|
||||||
|
)
|
||||||
|
);
|
||||||
|
|
||||||
|
TEST(Suggestions, Trim) {
|
||||||
|
auto suggestions = Suggestions::bestMatches({"foooo", "bar", "fo", "gao"}, "foo");
|
||||||
|
auto onlyOne = suggestions.trim(1);
|
||||||
|
ASSERT_EQ(onlyOne.suggestions.size(), 1);
|
||||||
|
ASSERT_TRUE(onlyOne.suggestions.begin()->suggestion == "fo");
|
||||||
|
|
||||||
|
auto closest = suggestions.trim(999, 2);
|
||||||
|
ASSERT_EQ(closest.suggestions.size(), 3);
|
||||||
|
}
|
||||||
|
}
|
|
@ -406,8 +406,29 @@ static void _deletePath(int parentfd, const Path & path, uint64_t & bytesFreed)
|
||||||
throw SysError("getting status of '%1%'", path);
|
throw SysError("getting status of '%1%'", path);
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!S_ISDIR(st.st_mode) && st.st_nlink == 1)
|
if (!S_ISDIR(st.st_mode)) {
|
||||||
|
/* We are about to delete a file. Will it likely free space? */
|
||||||
|
|
||||||
|
switch (st.st_nlink) {
|
||||||
|
/* Yes: last link. */
|
||||||
|
case 1:
|
||||||
bytesFreed += st.st_size;
|
bytesFreed += st.st_size;
|
||||||
|
break;
|
||||||
|
/* Maybe: yes, if 'auto-optimise-store' or manual optimisation
|
||||||
|
was performed. Instead of checking for real let's assume
|
||||||
|
it's an optimised file and space will be freed.
|
||||||
|
|
||||||
|
In worst case we will double count on freed space for files
|
||||||
|
with exactly two hardlinks for unoptimised packages.
|
||||||
|
*/
|
||||||
|
case 2:
|
||||||
|
bytesFreed += st.st_size;
|
||||||
|
break;
|
||||||
|
/* No: 3+ links. */
|
||||||
|
default:
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
if (S_ISDIR(st.st_mode)) {
|
if (S_ISDIR(st.st_mode)) {
|
||||||
/* Make the directory accessible. */
|
/* Make the directory accessible. */
|
||||||
|
@ -682,7 +703,14 @@ std::string drainFD(int fd, bool block, const size_t reserveSize)
|
||||||
|
|
||||||
void drainFD(int fd, Sink & sink, bool block)
|
void drainFD(int fd, Sink & sink, bool block)
|
||||||
{
|
{
|
||||||
int saved;
|
// silence GCC maybe-uninitialized warning in finally
|
||||||
|
int saved = 0;
|
||||||
|
|
||||||
|
if (!block) {
|
||||||
|
saved = fcntl(fd, F_GETFL);
|
||||||
|
if (fcntl(fd, F_SETFL, saved | O_NONBLOCK) == -1)
|
||||||
|
throw SysError("making file descriptor non-blocking");
|
||||||
|
}
|
||||||
|
|
||||||
Finally finally([&]() {
|
Finally finally([&]() {
|
||||||
if (!block) {
|
if (!block) {
|
||||||
|
@ -691,12 +719,6 @@ void drainFD(int fd, Sink & sink, bool block)
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
if (!block) {
|
|
||||||
saved = fcntl(fd, F_GETFL);
|
|
||||||
if (fcntl(fd, F_SETFL, saved | O_NONBLOCK) == -1)
|
|
||||||
throw SysError("making file descriptor non-blocking");
|
|
||||||
}
|
|
||||||
|
|
||||||
std::vector<unsigned char> buf(64 * 1024);
|
std::vector<unsigned char> buf(64 * 1024);
|
||||||
while (1) {
|
while (1) {
|
||||||
checkInterrupt();
|
checkInterrupt();
|
||||||
|
|
|
@ -325,8 +325,7 @@ static void main_nix_build(int argc, char * * argv)
|
||||||
|
|
||||||
state->printStats();
|
state->printStats();
|
||||||
|
|
||||||
auto buildPaths = [&](const std::vector<StorePathWithOutputs> & paths0) {
|
auto buildPaths = [&](const std::vector<DerivedPath> & paths) {
|
||||||
auto paths = toDerivedPaths(paths0);
|
|
||||||
/* Note: we do this even when !printMissing to efficiently
|
/* Note: we do this even when !printMissing to efficiently
|
||||||
fetch binary cache data. */
|
fetch binary cache data. */
|
||||||
uint64_t downloadSize, narSize;
|
uint64_t downloadSize, narSize;
|
||||||
|
@ -348,7 +347,7 @@ static void main_nix_build(int argc, char * * argv)
|
||||||
auto & drvInfo = drvs.front();
|
auto & drvInfo = drvs.front();
|
||||||
auto drv = evalStore->derivationFromPath(drvInfo.requireDrvPath());
|
auto drv = evalStore->derivationFromPath(drvInfo.requireDrvPath());
|
||||||
|
|
||||||
std::vector<StorePathWithOutputs> pathsToBuild;
|
std::vector<DerivedPath> pathsToBuild;
|
||||||
RealisedPath::Set pathsToCopy;
|
RealisedPath::Set pathsToCopy;
|
||||||
|
|
||||||
/* Figure out what bash shell to use. If $NIX_BUILD_SHELL
|
/* Figure out what bash shell to use. If $NIX_BUILD_SHELL
|
||||||
|
@ -370,7 +369,10 @@ static void main_nix_build(int argc, char * * argv)
|
||||||
throw Error("the 'bashInteractive' attribute in <nixpkgs> did not evaluate to a derivation");
|
throw Error("the 'bashInteractive' attribute in <nixpkgs> did not evaluate to a derivation");
|
||||||
|
|
||||||
auto bashDrv = drv->requireDrvPath();
|
auto bashDrv = drv->requireDrvPath();
|
||||||
pathsToBuild.push_back({bashDrv});
|
pathsToBuild.push_back(DerivedPath::Built {
|
||||||
|
.drvPath = bashDrv,
|
||||||
|
.outputs = {},
|
||||||
|
});
|
||||||
pathsToCopy.insert(bashDrv);
|
pathsToCopy.insert(bashDrv);
|
||||||
shellDrv = bashDrv;
|
shellDrv = bashDrv;
|
||||||
|
|
||||||
|
@ -382,17 +384,24 @@ static void main_nix_build(int argc, char * * argv)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Build or fetch all dependencies of the derivation.
|
// Build or fetch all dependencies of the derivation.
|
||||||
for (const auto & input : drv.inputDrvs)
|
for (const auto & [inputDrv0, inputOutputs] : drv.inputDrvs) {
|
||||||
|
// To get around lambda capturing restrictions in the
|
||||||
|
// standard.
|
||||||
|
const auto & inputDrv = inputDrv0;
|
||||||
if (std::all_of(envExclude.cbegin(), envExclude.cend(),
|
if (std::all_of(envExclude.cbegin(), envExclude.cend(),
|
||||||
[&](const std::string & exclude) {
|
[&](const std::string & exclude) {
|
||||||
return !std::regex_search(store->printStorePath(input.first), std::regex(exclude));
|
return !std::regex_search(store->printStorePath(inputDrv), std::regex(exclude));
|
||||||
}))
|
}))
|
||||||
{
|
{
|
||||||
pathsToBuild.push_back({input.first, input.second});
|
pathsToBuild.push_back(DerivedPath::Built {
|
||||||
pathsToCopy.insert(input.first);
|
.drvPath = inputDrv,
|
||||||
|
.outputs = inputOutputs
|
||||||
|
});
|
||||||
|
pathsToCopy.insert(inputDrv);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
for (const auto & src : drv.inputSrcs) {
|
for (const auto & src : drv.inputSrcs) {
|
||||||
pathsToBuild.push_back({src});
|
pathsToBuild.push_back(DerivedPath::Opaque{src});
|
||||||
pathsToCopy.insert(src);
|
pathsToCopy.insert(src);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -543,7 +552,7 @@ static void main_nix_build(int argc, char * * argv)
|
||||||
|
|
||||||
else {
|
else {
|
||||||
|
|
||||||
std::vector<StorePathWithOutputs> pathsToBuild;
|
std::vector<DerivedPath> pathsToBuild;
|
||||||
std::vector<std::pair<StorePath, std::string>> pathsToBuildOrdered;
|
std::vector<std::pair<StorePath, std::string>> pathsToBuildOrdered;
|
||||||
RealisedPath::Set drvsToCopy;
|
RealisedPath::Set drvsToCopy;
|
||||||
|
|
||||||
|
@ -556,7 +565,7 @@ static void main_nix_build(int argc, char * * argv)
|
||||||
if (outputName == "")
|
if (outputName == "")
|
||||||
throw Error("derivation '%s' lacks an 'outputName' attribute", store->printStorePath(drvPath));
|
throw Error("derivation '%s' lacks an 'outputName' attribute", store->printStorePath(drvPath));
|
||||||
|
|
||||||
pathsToBuild.push_back({drvPath, {outputName}});
|
pathsToBuild.push_back(DerivedPath::Built{drvPath, {outputName}});
|
||||||
pathsToBuildOrdered.push_back({drvPath, {outputName}});
|
pathsToBuildOrdered.push_back({drvPath, {outputName}});
|
||||||
drvsToCopy.insert(drvPath);
|
drvsToCopy.insert(drvPath);
|
||||||
|
|
||||||
|
|
|
@ -1,4 +1,6 @@
|
||||||
#include "store-api.hh"
|
#include "store-api.hh"
|
||||||
|
#include "store-cast.hh"
|
||||||
|
#include "gc-store.hh"
|
||||||
#include "profiles.hh"
|
#include "profiles.hh"
|
||||||
#include "shared.hh"
|
#include "shared.hh"
|
||||||
#include "globals.hh"
|
#include "globals.hh"
|
||||||
|
@ -80,10 +82,11 @@ static int main_nix_collect_garbage(int argc, char * * argv)
|
||||||
// Run the actual garbage collector.
|
// Run the actual garbage collector.
|
||||||
if (!dryRun) {
|
if (!dryRun) {
|
||||||
auto store = openStore();
|
auto store = openStore();
|
||||||
|
auto & gcStore = require<GcStore>(*store);
|
||||||
options.action = GCOptions::gcDeleteDead;
|
options.action = GCOptions::gcDeleteDead;
|
||||||
GCResults results;
|
GCResults results;
|
||||||
PrintFreed freed(true, results);
|
PrintFreed freed(true, results);
|
||||||
store->collectGarbage(options, results);
|
gcStore.collectGarbage(options, results);
|
||||||
}
|
}
|
||||||
|
|
||||||
return 0;
|
return 0;
|
||||||
|
|
|
@ -128,7 +128,12 @@ static void getAllExprs(EvalState & state,
|
||||||
if (hasSuffix(attrName, ".nix"))
|
if (hasSuffix(attrName, ".nix"))
|
||||||
attrName = std::string(attrName, 0, attrName.size() - 4);
|
attrName = std::string(attrName, 0, attrName.size() - 4);
|
||||||
if (!seen.insert(attrName).second) {
|
if (!seen.insert(attrName).second) {
|
||||||
printError("warning: name collision in input Nix expressions, skipping '%1%'", path2);
|
std::string suggestionMessage = "";
|
||||||
|
if (path2.find("channels") != std::string::npos && path.find("channels") != std::string::npos) {
|
||||||
|
suggestionMessage = fmt("\nsuggestion: remove '%s' from either the root channels or the user channels", attrName);
|
||||||
|
}
|
||||||
|
printError("warning: name collision in input Nix expressions, skipping '%1%'"
|
||||||
|
"%2%", path2, suggestionMessage);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
/* Load the expression on demand. */
|
/* Load the expression on demand. */
|
||||||
|
@ -918,12 +923,17 @@ static void queryJSON(Globals & globals, std::vector<DrvInfo> & elems, bool prin
|
||||||
pkgObj.attr("pname", drvName.name);
|
pkgObj.attr("pname", drvName.name);
|
||||||
pkgObj.attr("version", drvName.version);
|
pkgObj.attr("version", drvName.version);
|
||||||
pkgObj.attr("system", i.querySystem());
|
pkgObj.attr("system", i.querySystem());
|
||||||
|
pkgObj.attr("outputName", i.queryOutputName());
|
||||||
|
|
||||||
if (printOutPath) {
|
{
|
||||||
DrvInfo::Outputs outputs = i.queryOutputs();
|
DrvInfo::Outputs outputs = i.queryOutputs(printOutPath);
|
||||||
JSONObject outputObj = pkgObj.object("outputs");
|
JSONObject outputObj = pkgObj.object("outputs");
|
||||||
for (auto & j : outputs)
|
for (auto & j : outputs) {
|
||||||
outputObj.attr(j.first, globals.state->store->printStorePath(j.second));
|
if (j.second)
|
||||||
|
outputObj.attr(j.first, globals.state->store->printStorePath(*j.second));
|
||||||
|
else
|
||||||
|
outputObj.attr(j.first, nullptr);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if (printMeta) {
|
if (printMeta) {
|
||||||
|
@ -1052,6 +1062,7 @@ static void opQuery(Globals & globals, Strings opFlags, Strings opArgs)
|
||||||
/* Print the desired columns, or XML output. */
|
/* Print the desired columns, or XML output. */
|
||||||
if (jsonOutput) {
|
if (jsonOutput) {
|
||||||
queryJSON(globals, elems, printOutPath, printMeta);
|
queryJSON(globals, elems, printOutPath, printMeta);
|
||||||
|
cout << '\n';
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -1154,13 +1165,16 @@ static void opQuery(Globals & globals, Strings opFlags, Strings opArgs)
|
||||||
columns.push_back(drvPath ? store.printStorePath(*drvPath) : "-");
|
columns.push_back(drvPath ? store.printStorePath(*drvPath) : "-");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (xmlOutput)
|
||||||
|
attrs["outputName"] = i.queryOutputName();
|
||||||
|
|
||||||
if (printOutPath && !xmlOutput) {
|
if (printOutPath && !xmlOutput) {
|
||||||
DrvInfo::Outputs outputs = i.queryOutputs();
|
DrvInfo::Outputs outputs = i.queryOutputs();
|
||||||
std::string s;
|
std::string s;
|
||||||
for (auto & j : outputs) {
|
for (auto & j : outputs) {
|
||||||
if (!s.empty()) s += ';';
|
if (!s.empty()) s += ';';
|
||||||
if (j.first != "out") { s += j.first; s += "="; }
|
if (j.first != "out") { s += j.first; s += "="; }
|
||||||
s += store.printStorePath(j.second);
|
s += store.printStorePath(*j.second);
|
||||||
}
|
}
|
||||||
columns.push_back(s);
|
columns.push_back(s);
|
||||||
}
|
}
|
||||||
|
@ -1174,17 +1188,15 @@ static void opQuery(Globals & globals, Strings opFlags, Strings opArgs)
|
||||||
}
|
}
|
||||||
|
|
||||||
if (xmlOutput) {
|
if (xmlOutput) {
|
||||||
if (printOutPath || printMeta) {
|
|
||||||
XMLOpenElement item(xml, "item", attrs);
|
XMLOpenElement item(xml, "item", attrs);
|
||||||
if (printOutPath) {
|
DrvInfo::Outputs outputs = i.queryOutputs(printOutPath);
|
||||||
DrvInfo::Outputs outputs = i.queryOutputs();
|
|
||||||
for (auto & j : outputs) {
|
for (auto & j : outputs) {
|
||||||
XMLAttrs attrs2;
|
XMLAttrs attrs2;
|
||||||
attrs2["name"] = j.first;
|
attrs2["name"] = j.first;
|
||||||
attrs2["path"] = store.printStorePath(j.second);
|
if (j.second)
|
||||||
|
attrs2["path"] = store.printStorePath(*j.second);
|
||||||
xml.writeEmptyElement("output", attrs2);
|
xml.writeEmptyElement("output", attrs2);
|
||||||
}
|
}
|
||||||
}
|
|
||||||
if (printMeta) {
|
if (printMeta) {
|
||||||
StringSet metaNames = i.queryMetaNames();
|
StringSet metaNames = i.queryMetaNames();
|
||||||
for (auto & j : metaNames) {
|
for (auto & j : metaNames) {
|
||||||
|
@ -1237,8 +1249,6 @@ static void opQuery(Globals & globals, Strings opFlags, Strings opArgs)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
} else
|
|
||||||
xml.writeEmptyElement("item", attrs);
|
|
||||||
} else
|
} else
|
||||||
table.push_back(columns);
|
table.push_back(columns);
|
||||||
|
|
||||||
|
|
|
@ -56,7 +56,7 @@ bool createUserEnv(EvalState & state, DrvInfos & elems,
|
||||||
output paths, and optionally the derivation path, as well
|
output paths, and optionally the derivation path, as well
|
||||||
as the meta attributes. */
|
as the meta attributes. */
|
||||||
std::optional<StorePath> drvPath = keepDerivations ? i.queryDrvPath() : std::nullopt;
|
std::optional<StorePath> drvPath = keepDerivations ? i.queryDrvPath() : std::nullopt;
|
||||||
DrvInfo::Outputs outputs = i.queryOutputs(true);
|
DrvInfo::Outputs outputs = i.queryOutputs(true, true);
|
||||||
StringSet metaNames = i.queryMetaNames();
|
StringSet metaNames = i.queryMetaNames();
|
||||||
|
|
||||||
auto attrs = state.buildBindings(7 + outputs.size());
|
auto attrs = state.buildBindings(7 + outputs.size());
|
||||||
|
@ -76,15 +76,15 @@ bool createUserEnv(EvalState & state, DrvInfos & elems,
|
||||||
for (const auto & [m, j] : enumerate(outputs)) {
|
for (const auto & [m, j] : enumerate(outputs)) {
|
||||||
(vOutputs.listElems()[m] = state.allocValue())->mkString(j.first);
|
(vOutputs.listElems()[m] = state.allocValue())->mkString(j.first);
|
||||||
auto outputAttrs = state.buildBindings(2);
|
auto outputAttrs = state.buildBindings(2);
|
||||||
outputAttrs.alloc(state.sOutPath).mkString(state.store->printStorePath(j.second));
|
outputAttrs.alloc(state.sOutPath).mkString(state.store->printStorePath(*j.second));
|
||||||
attrs.alloc(j.first).mkAttrs(outputAttrs);
|
attrs.alloc(j.first).mkAttrs(outputAttrs);
|
||||||
|
|
||||||
/* This is only necessary when installing store paths, e.g.,
|
/* This is only necessary when installing store paths, e.g.,
|
||||||
`nix-env -i /nix/store/abcd...-foo'. */
|
`nix-env -i /nix/store/abcd...-foo'. */
|
||||||
state.store->addTempRoot(j.second);
|
state.store->addTempRoot(*j.second);
|
||||||
state.store->ensurePath(j.second);
|
state.store->ensurePath(*j.second);
|
||||||
|
|
||||||
references.insert(j.second);
|
references.insert(*j.second);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Copy the meta attributes.
|
// Copy the meta attributes.
|
||||||
|
|
|
@ -3,6 +3,9 @@
|
||||||
#include "dotgraph.hh"
|
#include "dotgraph.hh"
|
||||||
#include "globals.hh"
|
#include "globals.hh"
|
||||||
#include "build-result.hh"
|
#include "build-result.hh"
|
||||||
|
#include "store-cast.hh"
|
||||||
|
#include "gc-store.hh"
|
||||||
|
#include "log-store.hh"
|
||||||
#include "local-store.hh"
|
#include "local-store.hh"
|
||||||
#include "monitor-fd.hh"
|
#include "monitor-fd.hh"
|
||||||
#include "serve-protocol.hh"
|
#include "serve-protocol.hh"
|
||||||
|
@ -428,11 +431,12 @@ static void opQuery(Strings opFlags, Strings opArgs)
|
||||||
store->computeFSClosure(
|
store->computeFSClosure(
|
||||||
args, referrers, true, settings.gcKeepOutputs, settings.gcKeepDerivations);
|
args, referrers, true, settings.gcKeepOutputs, settings.gcKeepDerivations);
|
||||||
|
|
||||||
Roots roots = store->findRoots(false);
|
auto & gcStore = require<GcStore>(*store);
|
||||||
|
Roots roots = gcStore.findRoots(false);
|
||||||
for (auto & [target, links] : roots)
|
for (auto & [target, links] : roots)
|
||||||
if (referrers.find(target) != referrers.end())
|
if (referrers.find(target) != referrers.end())
|
||||||
for (auto & link : links)
|
for (auto & link : links)
|
||||||
cout << fmt("%1% -> %2%\n", link, store->printStorePath(target));
|
cout << fmt("%1% -> %2%\n", link, gcStore.printStorePath(target));
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -472,13 +476,15 @@ static void opReadLog(Strings opFlags, Strings opArgs)
|
||||||
{
|
{
|
||||||
if (!opFlags.empty()) throw UsageError("unknown flag");
|
if (!opFlags.empty()) throw UsageError("unknown flag");
|
||||||
|
|
||||||
|
auto & logStore = require<LogStore>(*store);
|
||||||
|
|
||||||
RunPager pager;
|
RunPager pager;
|
||||||
|
|
||||||
for (auto & i : opArgs) {
|
for (auto & i : opArgs) {
|
||||||
auto path = store->followLinksToStorePath(i);
|
auto path = logStore.followLinksToStorePath(i);
|
||||||
auto log = store->getBuildLog(path);
|
auto log = logStore.getBuildLog(path);
|
||||||
if (!log)
|
if (!log)
|
||||||
throw Error("build log of derivation '%s' is not available", store->printStorePath(path));
|
throw Error("build log of derivation '%s' is not available", logStore.printStorePath(path));
|
||||||
std::cout << *log;
|
std::cout << *log;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
@ -588,20 +594,22 @@ static void opGC(Strings opFlags, Strings opArgs)
|
||||||
|
|
||||||
if (!opArgs.empty()) throw UsageError("no arguments expected");
|
if (!opArgs.empty()) throw UsageError("no arguments expected");
|
||||||
|
|
||||||
|
auto & gcStore = require<GcStore>(*store);
|
||||||
|
|
||||||
if (printRoots) {
|
if (printRoots) {
|
||||||
Roots roots = store->findRoots(false);
|
Roots roots = gcStore.findRoots(false);
|
||||||
std::set<std::pair<Path, StorePath>> roots2;
|
std::set<std::pair<Path, StorePath>> roots2;
|
||||||
// Transpose and sort the roots.
|
// Transpose and sort the roots.
|
||||||
for (auto & [target, links] : roots)
|
for (auto & [target, links] : roots)
|
||||||
for (auto & link : links)
|
for (auto & link : links)
|
||||||
roots2.emplace(link, target);
|
roots2.emplace(link, target);
|
||||||
for (auto & [link, target] : roots2)
|
for (auto & [link, target] : roots2)
|
||||||
std::cout << link << " -> " << store->printStorePath(target) << "\n";
|
std::cout << link << " -> " << gcStore.printStorePath(target) << "\n";
|
||||||
}
|
}
|
||||||
|
|
||||||
else {
|
else {
|
||||||
PrintFreed freed(options.action == GCOptions::gcDeleteDead, results);
|
PrintFreed freed(options.action == GCOptions::gcDeleteDead, results);
|
||||||
store->collectGarbage(options, results);
|
gcStore.collectGarbage(options, results);
|
||||||
|
|
||||||
if (options.action != GCOptions::gcDeleteDead)
|
if (options.action != GCOptions::gcDeleteDead)
|
||||||
for (auto & i : results.paths)
|
for (auto & i : results.paths)
|
||||||
|
@ -625,9 +633,11 @@ static void opDelete(Strings opFlags, Strings opArgs)
|
||||||
for (auto & i : opArgs)
|
for (auto & i : opArgs)
|
||||||
options.pathsToDelete.insert(store->followLinksToStorePath(i));
|
options.pathsToDelete.insert(store->followLinksToStorePath(i));
|
||||||
|
|
||||||
|
auto & gcStore = require<GcStore>(*store);
|
||||||
|
|
||||||
GCResults results;
|
GCResults results;
|
||||||
PrintFreed freed(true, results);
|
PrintFreed freed(true, results);
|
||||||
store->collectGarbage(options, results);
|
gcStore.collectGarbage(options, results);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|
|
@ -52,15 +52,26 @@ struct CmdBuild : InstallablesCommand, MixDryRun, MixJSON, MixProfile
|
||||||
|
|
||||||
void run(ref<Store> store) override
|
void run(ref<Store> store) override
|
||||||
{
|
{
|
||||||
|
if (dryRun) {
|
||||||
|
std::vector<DerivedPath> pathsToBuild;
|
||||||
|
|
||||||
|
for (auto & i : installables) {
|
||||||
|
auto b = i->toDerivedPaths();
|
||||||
|
pathsToBuild.insert(pathsToBuild.end(), b.begin(), b.end());
|
||||||
|
}
|
||||||
|
printMissing(store, pathsToBuild, lvlError);
|
||||||
|
if (json)
|
||||||
|
logger->cout("%s", derivedPathsToJSON(pathsToBuild, store).dump());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
auto buildables = Installable::build(
|
auto buildables = Installable::build(
|
||||||
getEvalStore(), store,
|
getEvalStore(), store,
|
||||||
dryRun ? Realise::Derivation : Realise::Outputs,
|
Realise::Outputs,
|
||||||
installables, buildMode);
|
installables, buildMode);
|
||||||
|
|
||||||
if (json) logger->cout("%s", derivedPathsWithHintsToJSON(buildables, store).dump());
|
if (json) logger->cout("%s", derivedPathsWithHintsToJSON(buildables, store).dump());
|
||||||
|
|
||||||
if (dryRun) return;
|
|
||||||
|
|
||||||
if (outLink != "")
|
if (outLink != "")
|
||||||
if (auto store2 = store.dynamic_pointer_cast<LocalFSStore>())
|
if (auto store2 = store.dynamic_pointer_cast<LocalFSStore>())
|
||||||
for (const auto & [_i, buildable] : enumerate(buildables)) {
|
for (const auto & [_i, buildable] : enumerate(buildables)) {
|
||||||
|
|
|
@ -206,7 +206,8 @@ static StorePath getDerivationEnvironment(ref<Store> store, ref<Store> evalStore
|
||||||
output.second = { .output = DerivationOutputInputAddressed { .path = StorePath::dummy } };
|
output.second = { .output = DerivationOutputInputAddressed { .path = StorePath::dummy } };
|
||||||
drv.env[output.first] = "";
|
drv.env[output.first] = "";
|
||||||
}
|
}
|
||||||
Hash h = std::get<0>(hashDerivationModulo(*evalStore, drv, true));
|
auto h0 = hashDerivationModulo(*evalStore, drv, true);
|
||||||
|
const Hash & h = h0.requireNoFixedNonDeferred();
|
||||||
|
|
||||||
for (auto & output : drv.outputs) {
|
for (auto & output : drv.outputs) {
|
||||||
auto outPath = store->makeOutputPath(output.first, h, drv.name);
|
auto outPath = store->makeOutputPath(output.first, h, drv.name);
|
||||||
|
|
|
@ -706,9 +706,14 @@ struct CmdFlakeInitCommon : virtual Args, EvalCommand
|
||||||
|
|
||||||
auto [cursor, attrPath] = installable.getCursor(*evalState);
|
auto [cursor, attrPath] = installable.getCursor(*evalState);
|
||||||
|
|
||||||
auto templateDir = cursor->getAttr("path")->getString();
|
auto templateDirAttr = cursor->getAttr("path");
|
||||||
|
auto templateDir = templateDirAttr->getString();
|
||||||
|
|
||||||
assert(store->isInStore(templateDir));
|
if (!store->isInStore(templateDir))
|
||||||
|
throw TypeError(
|
||||||
|
"'%s' was not found in the Nix store\n"
|
||||||
|
"If you've set '%s' to a string, try using a path instead.",
|
||||||
|
templateDir, templateDirAttr->getAttrPathStr());
|
||||||
|
|
||||||
std::vector<Path> files;
|
std::vector<Path> files;
|
||||||
|
|
||||||
|
|
|
@ -2,6 +2,7 @@
|
||||||
#include "common-args.hh"
|
#include "common-args.hh"
|
||||||
#include "shared.hh"
|
#include "shared.hh"
|
||||||
#include "store-api.hh"
|
#include "store-api.hh"
|
||||||
|
#include "log-store.hh"
|
||||||
#include "progress-bar.hh"
|
#include "progress-bar.hh"
|
||||||
|
|
||||||
using namespace nix;
|
using namespace nix;
|
||||||
|
@ -34,17 +35,24 @@ struct CmdLog : InstallableCommand
|
||||||
|
|
||||||
RunPager pager;
|
RunPager pager;
|
||||||
for (auto & sub : subs) {
|
for (auto & sub : subs) {
|
||||||
|
auto * logSubP = dynamic_cast<LogStore *>(&*sub);
|
||||||
|
if (!logSubP) {
|
||||||
|
printInfo("Skipped '%s' which does not support retrieving build logs", sub->getUri());
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
auto & logSub = *logSubP;
|
||||||
|
|
||||||
auto log = std::visit(overloaded {
|
auto log = std::visit(overloaded {
|
||||||
[&](const DerivedPath::Opaque & bo) {
|
[&](const DerivedPath::Opaque & bo) {
|
||||||
return sub->getBuildLog(bo.path);
|
return logSub.getBuildLog(bo.path);
|
||||||
},
|
},
|
||||||
[&](const DerivedPath::Built & bfd) {
|
[&](const DerivedPath::Built & bfd) {
|
||||||
return sub->getBuildLog(bfd.drvPath);
|
return logSub.getBuildLog(bfd.drvPath);
|
||||||
},
|
},
|
||||||
}, b.raw());
|
}, b.raw());
|
||||||
if (!log) continue;
|
if (!log) continue;
|
||||||
stopProgressBar();
|
stopProgressBar();
|
||||||
printInfo("got build log for '%s' from '%s'", installable->what(), sub->getUri());
|
printInfo("got build log for '%s' from '%s'", installable->what(), logSub.getUri());
|
||||||
std::cout << *log;
|
std::cout << *log;
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
|
@ -25,6 +25,7 @@ extern "C" {
|
||||||
#include "eval-inline.hh"
|
#include "eval-inline.hh"
|
||||||
#include "attr-path.hh"
|
#include "attr-path.hh"
|
||||||
#include "store-api.hh"
|
#include "store-api.hh"
|
||||||
|
#include "log-store.hh"
|
||||||
#include "common-eval-args.hh"
|
#include "common-eval-args.hh"
|
||||||
#include "get-drvs.hh"
|
#include "get-drvs.hh"
|
||||||
#include "derivations.hh"
|
#include "derivations.hh"
|
||||||
|
@ -526,9 +527,16 @@ bool NixRepl::processLine(std::string line)
|
||||||
bool foundLog = false;
|
bool foundLog = false;
|
||||||
RunPager pager;
|
RunPager pager;
|
||||||
for (auto & sub : subs) {
|
for (auto & sub : subs) {
|
||||||
auto log = sub->getBuildLog(drvPath);
|
auto * logSubP = dynamic_cast<LogStore *>(&*sub);
|
||||||
|
if (!logSubP) {
|
||||||
|
printInfo("Skipped '%s' which does not support retrieving build logs", sub->getUri());
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
auto & logSub = *logSubP;
|
||||||
|
|
||||||
|
auto log = logSub.getBuildLog(drvPath);
|
||||||
if (log) {
|
if (log) {
|
||||||
printInfo("got build log for '%s' from '%s'", drvPathRaw, sub->getUri());
|
printInfo("got build log for '%s' from '%s'", drvPathRaw, logSub.getUri());
|
||||||
logger->writeToStdout(*log);
|
logger->writeToStdout(*log);
|
||||||
foundLog = true;
|
foundLog = true;
|
||||||
break;
|
break;
|
||||||
|
|
|
@ -1,6 +1,8 @@
|
||||||
#include "command.hh"
|
#include "command.hh"
|
||||||
#include "shared.hh"
|
#include "shared.hh"
|
||||||
#include "store-api.hh"
|
#include "store-api.hh"
|
||||||
|
#include "store-cast.hh"
|
||||||
|
#include "log-store.hh"
|
||||||
#include "sync.hh"
|
#include "sync.hh"
|
||||||
#include "thread-pool.hh"
|
#include "thread-pool.hh"
|
||||||
|
|
||||||
|
@ -26,7 +28,10 @@ struct CmdCopyLog : virtual CopyCommand, virtual InstallablesCommand
|
||||||
|
|
||||||
void run(ref<Store> srcStore) override
|
void run(ref<Store> srcStore) override
|
||||||
{
|
{
|
||||||
|
auto & srcLogStore = require<LogStore>(*srcStore);
|
||||||
|
|
||||||
auto dstStore = getDstStore();
|
auto dstStore = getDstStore();
|
||||||
|
auto & dstLogStore = require<LogStore>(*dstStore);
|
||||||
|
|
||||||
StorePathSet drvPaths;
|
StorePathSet drvPaths;
|
||||||
|
|
||||||
|
@ -35,8 +40,8 @@ struct CmdCopyLog : virtual CopyCommand, virtual InstallablesCommand
|
||||||
drvPaths.insert(drvPath);
|
drvPaths.insert(drvPath);
|
||||||
|
|
||||||
for (auto & drvPath : drvPaths) {
|
for (auto & drvPath : drvPaths) {
|
||||||
if (auto log = srcStore->getBuildLog(drvPath))
|
if (auto log = srcLogStore.getBuildLog(drvPath))
|
||||||
dstStore->addBuildLog(drvPath, *log);
|
dstLogStore.addBuildLog(drvPath, *log);
|
||||||
else
|
else
|
||||||
throw Error("build log for '%s' is not available", srcStore->printStorePath(drvPath));
|
throw Error("build log for '%s' is not available", srcStore->printStorePath(drvPath));
|
||||||
}
|
}
|
||||||
|
|
|
@ -2,6 +2,8 @@
|
||||||
#include "common-args.hh"
|
#include "common-args.hh"
|
||||||
#include "shared.hh"
|
#include "shared.hh"
|
||||||
#include "store-api.hh"
|
#include "store-api.hh"
|
||||||
|
#include "store-cast.hh"
|
||||||
|
#include "gc-store.hh"
|
||||||
|
|
||||||
using namespace nix;
|
using namespace nix;
|
||||||
|
|
||||||
|
@ -32,12 +34,14 @@ struct CmdStoreDelete : StorePathsCommand
|
||||||
|
|
||||||
void run(ref<Store> store, std::vector<StorePath> && storePaths) override
|
void run(ref<Store> store, std::vector<StorePath> && storePaths) override
|
||||||
{
|
{
|
||||||
|
auto & gcStore = require<GcStore>(*store);
|
||||||
|
|
||||||
for (auto & path : storePaths)
|
for (auto & path : storePaths)
|
||||||
options.pathsToDelete.insert(path);
|
options.pathsToDelete.insert(path);
|
||||||
|
|
||||||
GCResults results;
|
GCResults results;
|
||||||
PrintFreed freed(true, results);
|
PrintFreed freed(true, results);
|
||||||
store->collectGarbage(options, results);
|
gcStore.collectGarbage(options, results);
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
|
@ -2,6 +2,8 @@
|
||||||
#include "common-args.hh"
|
#include "common-args.hh"
|
||||||
#include "shared.hh"
|
#include "shared.hh"
|
||||||
#include "store-api.hh"
|
#include "store-api.hh"
|
||||||
|
#include "store-cast.hh"
|
||||||
|
#include "gc-store.hh"
|
||||||
|
|
||||||
using namespace nix;
|
using namespace nix;
|
||||||
|
|
||||||
|
@ -33,10 +35,12 @@ struct CmdStoreGC : StoreCommand, MixDryRun
|
||||||
|
|
||||||
void run(ref<Store> store) override
|
void run(ref<Store> store) override
|
||||||
{
|
{
|
||||||
|
auto & gcStore = require<GcStore>(*store);
|
||||||
|
|
||||||
options.action = dryRun ? GCOptions::gcReturnDead : GCOptions::gcDeleteDead;
|
options.action = dryRun ? GCOptions::gcReturnDead : GCOptions::gcDeleteDead;
|
||||||
GCResults results;
|
GCResults results;
|
||||||
PrintFreed freed(options.action == GCOptions::gcDeleteDead, results);
|
PrintFreed freed(options.action == GCOptions::gcDeleteDead, results);
|
||||||
store->collectGarbage(options, results);
|
gcStore.collectGarbage(options, results);
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
|
@ -50,3 +50,22 @@ nix build -f dependencies.nix -o $RESULT --dry-run
|
||||||
nix build -f dependencies.nix -o $RESULT
|
nix build -f dependencies.nix -o $RESULT
|
||||||
|
|
||||||
[[ -h $RESULT ]]
|
[[ -h $RESULT ]]
|
||||||
|
|
||||||
|
###################################################
|
||||||
|
# Check the JSON output
|
||||||
|
clearStore
|
||||||
|
clearCache
|
||||||
|
|
||||||
|
RES=$(nix build -f dependencies.nix --dry-run --json)
|
||||||
|
|
||||||
|
if [[ -z "$NIX_TESTS_CA_BY_DEFAULT" ]]; then
|
||||||
|
echo "$RES" | jq '.[0] | [
|
||||||
|
(.drvPath | test("'$NIX_STORE_DIR'.*\\.drv")),
|
||||||
|
(.outputs.out | test("'$NIX_STORE_DIR'"))
|
||||||
|
] | all'
|
||||||
|
else
|
||||||
|
echo "$RES" | jq '.[0] | [
|
||||||
|
(.drvPath | test("'$NIX_STORE_DIR'.*\\.drv")),
|
||||||
|
.outputs.out == null
|
||||||
|
] | all'
|
||||||
|
fi
|
||||||
|
|
|
@ -11,13 +11,13 @@ let
|
||||||
args = ["sh" "-e" args.builder or (builtins.toFile "builder-${args.name}.sh" "if [ -e .attrs.sh ]; then source .attrs.sh; fi; eval \"$buildCommand\"")];
|
args = ["sh" "-e" args.builder or (builtins.toFile "builder-${args.name}.sh" "if [ -e .attrs.sh ]; then source .attrs.sh; fi; eval \"$buildCommand\"")];
|
||||||
outputHashMode = "recursive";
|
outputHashMode = "recursive";
|
||||||
outputHashAlgo = "sha256";
|
outputHashAlgo = "sha256";
|
||||||
} // removeAttrs args ["builder" "meta"])
|
} // removeAttrs args ["builder" "meta" "passthru"])
|
||||||
// { meta = args.meta or {}; };
|
// { meta = args.meta or {}; passthru = args.passthru or {}; };
|
||||||
|
|
||||||
input1 = mkDerivation {
|
input1 = mkDerivation {
|
||||||
shell = busybox;
|
shell = busybox;
|
||||||
name = "build-remote-input-1";
|
name = "build-remote-input-1";
|
||||||
buildCommand = "echo FOO > $out";
|
buildCommand = "echo hi-input1; echo FOO > $out";
|
||||||
requiredSystemFeatures = ["foo"];
|
requiredSystemFeatures = ["foo"];
|
||||||
outputHash = "sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc=";
|
outputHash = "sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc=";
|
||||||
};
|
};
|
||||||
|
@ -25,7 +25,7 @@ let
|
||||||
input2 = mkDerivation {
|
input2 = mkDerivation {
|
||||||
shell = busybox;
|
shell = busybox;
|
||||||
name = "build-remote-input-2";
|
name = "build-remote-input-2";
|
||||||
buildCommand = "echo BAR > $out";
|
buildCommand = "echo hi; echo BAR > $out";
|
||||||
requiredSystemFeatures = ["bar"];
|
requiredSystemFeatures = ["bar"];
|
||||||
outputHash = "sha256-XArauVH91AVwP9hBBQNlkX9ccuPpSYx9o0zeIHb6e+Q=";
|
outputHash = "sha256-XArauVH91AVwP9hBBQNlkX9ccuPpSYx9o0zeIHb6e+Q=";
|
||||||
};
|
};
|
||||||
|
@ -34,6 +34,7 @@ let
|
||||||
shell = busybox;
|
shell = busybox;
|
||||||
name = "build-remote-input-3";
|
name = "build-remote-input-3";
|
||||||
buildCommand = ''
|
buildCommand = ''
|
||||||
|
echo hi-input3
|
||||||
read x < ${input2}
|
read x < ${input2}
|
||||||
echo $x BAZ > $out
|
echo $x BAZ > $out
|
||||||
'';
|
'';
|
||||||
|
@ -46,6 +47,7 @@ in
|
||||||
mkDerivation {
|
mkDerivation {
|
||||||
shell = busybox;
|
shell = busybox;
|
||||||
name = "build-remote";
|
name = "build-remote";
|
||||||
|
passthru = { inherit input1 input2 input3; };
|
||||||
buildCommand =
|
buildCommand =
|
||||||
''
|
''
|
||||||
read x < ${input1}
|
read x < ${input1}
|
||||||
|
|
|
@ -9,20 +9,20 @@ let
|
||||||
inherit system;
|
inherit system;
|
||||||
builder = busybox;
|
builder = busybox;
|
||||||
args = ["sh" "-e" args.builder or (builtins.toFile "builder-${args.name}.sh" "if [ -e .attrs.sh ]; then source .attrs.sh; fi; eval \"$buildCommand\"")];
|
args = ["sh" "-e" args.builder or (builtins.toFile "builder-${args.name}.sh" "if [ -e .attrs.sh ]; then source .attrs.sh; fi; eval \"$buildCommand\"")];
|
||||||
} // removeAttrs args ["builder" "meta"])
|
} // removeAttrs args ["builder" "meta" "passthru"])
|
||||||
// { meta = args.meta or {}; };
|
// { meta = args.meta or {}; passthru = args.passthru or {}; };
|
||||||
|
|
||||||
input1 = mkDerivation {
|
input1 = mkDerivation {
|
||||||
shell = busybox;
|
shell = busybox;
|
||||||
name = "build-remote-input-1";
|
name = "build-remote-input-1";
|
||||||
buildCommand = "echo FOO > $out";
|
buildCommand = "echo hi-input1; echo FOO > $out";
|
||||||
requiredSystemFeatures = ["foo"];
|
requiredSystemFeatures = ["foo"];
|
||||||
};
|
};
|
||||||
|
|
||||||
input2 = mkDerivation {
|
input2 = mkDerivation {
|
||||||
shell = busybox;
|
shell = busybox;
|
||||||
name = "build-remote-input-2";
|
name = "build-remote-input-2";
|
||||||
buildCommand = "echo BAR > $out";
|
buildCommand = "echo hi; echo BAR > $out";
|
||||||
requiredSystemFeatures = ["bar"];
|
requiredSystemFeatures = ["bar"];
|
||||||
};
|
};
|
||||||
|
|
||||||
|
@ -30,6 +30,7 @@ let
|
||||||
shell = busybox;
|
shell = busybox;
|
||||||
name = "build-remote-input-3";
|
name = "build-remote-input-3";
|
||||||
buildCommand = ''
|
buildCommand = ''
|
||||||
|
echo hi-input3
|
||||||
read x < ${input2}
|
read x < ${input2}
|
||||||
echo $x BAZ > $out
|
echo $x BAZ > $out
|
||||||
'';
|
'';
|
||||||
|
@ -41,6 +42,7 @@ in
|
||||||
mkDerivation {
|
mkDerivation {
|
||||||
shell = busybox;
|
shell = busybox;
|
||||||
name = "build-remote";
|
name = "build-remote";
|
||||||
|
passthru = { inherit input1 input2 input3; };
|
||||||
buildCommand =
|
buildCommand =
|
||||||
''
|
''
|
||||||
read x < ${input1}
|
read x < ${input1}
|
||||||
|
|
|
@ -54,6 +54,14 @@ nix path-info --store $TEST_ROOT/machine3 --all \
|
||||||
| grep -v builder-build-remote-input-2.sh \
|
| grep -v builder-build-remote-input-2.sh \
|
||||||
| grep builder-build-remote-input-3.sh
|
| grep builder-build-remote-input-3.sh
|
||||||
|
|
||||||
|
|
||||||
|
# Temporarily disabled because of https://github.com/NixOS/nix/issues/6209
|
||||||
|
if [[ -z "$CONTENT_ADDRESSED" ]]; then
|
||||||
|
for i in input1 input3; do
|
||||||
|
nix log --store $TEST_ROOT/machine0 --file "$file" --arg busybox $busybox passthru."$i" | grep hi-$i
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
|
||||||
# Behavior of keep-failed
|
# Behavior of keep-failed
|
||||||
out="$(nix-build 2>&1 failing.nix \
|
out="$(nix-build 2>&1 failing.nix \
|
||||||
--builders "$(join_by '; ' "${builders[@]}")" \
|
--builders "$(join_by '; ' "${builders[@]}")" \
|
||||||
|
|
6
tests/ca/build-dry.sh
Normal file
6
tests/ca/build-dry.sh
Normal file
|
@ -0,0 +1,6 @@
|
||||||
|
source common.sh
|
||||||
|
|
||||||
|
export NIX_TESTS_CA_BY_DEFAULT=1
|
||||||
|
|
||||||
|
cd .. && source build-dry.sh
|
||||||
|
|
5
tests/eval.nix
Normal file
5
tests/eval.nix
Normal file
|
@ -0,0 +1,5 @@
|
||||||
|
{
|
||||||
|
int = 123;
|
||||||
|
str = "foo";
|
||||||
|
attr.foo = "bar";
|
||||||
|
}
|
29
tests/eval.sh
Normal file
29
tests/eval.sh
Normal file
|
@ -0,0 +1,29 @@
|
||||||
|
source common.sh
|
||||||
|
|
||||||
|
clearStore
|
||||||
|
|
||||||
|
testStdinHeredoc=$(nix eval -f - <<EOF
|
||||||
|
{
|
||||||
|
bar = 3 + 1;
|
||||||
|
foo = 2 + 2;
|
||||||
|
}
|
||||||
|
EOF
|
||||||
|
)
|
||||||
|
[[ $testStdinHeredoc == '{ bar = 4; foo = 4; }' ]]
|
||||||
|
|
||||||
|
nix eval --expr 'assert 1 + 2 == 3; true'
|
||||||
|
|
||||||
|
[[ $(nix eval int -f "./eval.nix") == 123 ]]
|
||||||
|
[[ $(nix eval str -f "./eval.nix") == '"foo"' ]]
|
||||||
|
[[ $(nix eval str --raw -f "./eval.nix") == 'foo' ]]
|
||||||
|
[[ $(nix eval attr -f "./eval.nix") == '{ foo = "bar"; }' ]]
|
||||||
|
[[ $(nix eval attr --json -f "./eval.nix") == '{"foo":"bar"}' ]]
|
||||||
|
[[ $(nix eval int -f - < "./eval.nix") == 123 ]]
|
||||||
|
|
||||||
|
|
||||||
|
nix-instantiate --eval -E 'assert 1 + 2 == 3; true'
|
||||||
|
[[ $(nix-instantiate -A int --eval "./eval.nix") == 123 ]]
|
||||||
|
[[ $(nix-instantiate -A str --eval "./eval.nix") == '"foo"' ]]
|
||||||
|
[[ $(nix-instantiate -A attr --eval "./eval.nix") == '{ foo = "bar"; }' ]]
|
||||||
|
[[ $(nix-instantiate -A attr --eval --json "./eval.nix") == '{"foo":"bar"}' ]]
|
||||||
|
[[ $(nix-instantiate -A int --eval - < "./eval.nix") == 123 ]]
|
|
@ -11,7 +11,7 @@ repo=$TEST_ROOT/git
|
||||||
|
|
||||||
export _NIX_FORCE_HTTP=1
|
export _NIX_FORCE_HTTP=1
|
||||||
|
|
||||||
rm -rf $repo ${repo}-tmp $TEST_HOME/.cache/nix $TEST_ROOT/worktree $TEST_ROOT/shallow
|
rm -rf $repo ${repo}-tmp $TEST_HOME/.cache/nix $TEST_ROOT/worktree $TEST_ROOT/shallow $TEST_ROOT/minimal
|
||||||
|
|
||||||
git init $repo
|
git init $repo
|
||||||
git -C $repo config user.email "foobar@example.com"
|
git -C $repo config user.email "foobar@example.com"
|
||||||
|
@ -147,8 +147,13 @@ path3=$(nix eval --impure --raw --expr "(builtins.fetchGit $repo).outPath")
|
||||||
# (check dirty-tree handling was used)
|
# (check dirty-tree handling was used)
|
||||||
[[ $(nix eval --impure --raw --expr "(builtins.fetchGit $repo).rev") = 0000000000000000000000000000000000000000 ]]
|
[[ $(nix eval --impure --raw --expr "(builtins.fetchGit $repo).rev") = 0000000000000000000000000000000000000000 ]]
|
||||||
[[ $(nix eval --impure --raw --expr "(builtins.fetchGit $repo).shortRev") = 0000000 ]]
|
[[ $(nix eval --impure --raw --expr "(builtins.fetchGit $repo).shortRev") = 0000000 ]]
|
||||||
|
# Making a dirty tree clean again and fetching it should
|
||||||
|
# record correct revision information. See: #4140
|
||||||
|
echo world > $repo/hello
|
||||||
|
[[ $(nix eval --impure --raw --expr "(builtins.fetchGit $repo).rev") = $rev2 ]]
|
||||||
|
|
||||||
# Committing shouldn't change store path, or switch to using 'master'
|
# Committing shouldn't change store path, or switch to using 'master'
|
||||||
|
echo dev > $repo/hello
|
||||||
git -C $repo commit -m 'Bla5' -a
|
git -C $repo commit -m 'Bla5' -a
|
||||||
path4=$(nix eval --impure --raw --expr "(builtins.fetchGit $repo).outPath")
|
path4=$(nix eval --impure --raw --expr "(builtins.fetchGit $repo).outPath")
|
||||||
[[ $(cat $path4/hello) = dev ]]
|
[[ $(cat $path4/hello) = dev ]]
|
||||||
|
@ -170,6 +175,14 @@ NIX=$(command -v nix)
|
||||||
path5=$(nix eval --impure --raw --expr "(builtins.fetchGit { url = $repo; ref = \"dev\"; }).outPath")
|
path5=$(nix eval --impure --raw --expr "(builtins.fetchGit { url = $repo; ref = \"dev\"; }).outPath")
|
||||||
[[ $path3 = $path5 ]]
|
[[ $path3 = $path5 ]]
|
||||||
|
|
||||||
|
# Fetching from a repo with only a specific revision and no branches should
|
||||||
|
# not fall back to copying files and record correct revision information. See: #5302
|
||||||
|
mkdir $TEST_ROOT/minimal
|
||||||
|
git -C $TEST_ROOT/minimal init
|
||||||
|
git -C $TEST_ROOT/minimal fetch $repo $rev2
|
||||||
|
git -C $TEST_ROOT/minimal checkout $rev2
|
||||||
|
[[ $(nix eval --impure --raw --expr "(builtins.fetchGit { url = $TEST_ROOT/minimal; }).rev") = $rev2 ]]
|
||||||
|
|
||||||
# Fetching a shallow repo shouldn't work by default, because we can't
|
# Fetching a shallow repo shouldn't work by default, because we can't
|
||||||
# return a revCount.
|
# return a revCount.
|
||||||
git clone --depth 1 file://$repo $TEST_ROOT/shallow
|
git clone --depth 1 file://$repo $TEST_ROOT/shallow
|
||||||
|
@ -193,3 +206,11 @@ rev4_nix=$(nix eval --impure --raw --expr "(builtins.fetchGit { url = \"file://$
|
||||||
# The name argument should be handled
|
# The name argument should be handled
|
||||||
path9=$(nix eval --impure --raw --expr "(builtins.fetchGit { url = \"file://$repo\"; ref = \"HEAD\"; name = \"foo\"; }).outPath")
|
path9=$(nix eval --impure --raw --expr "(builtins.fetchGit { url = \"file://$repo\"; ref = \"HEAD\"; name = \"foo\"; }).outPath")
|
||||||
[[ $path9 =~ -foo$ ]]
|
[[ $path9 =~ -foo$ ]]
|
||||||
|
|
||||||
|
# should fail if there is no repo
|
||||||
|
rm -rf $repo/.git
|
||||||
|
(! nix eval --impure --raw --expr "(builtins.fetchGit \"file://$repo\").outPath")
|
||||||
|
|
||||||
|
# should succeed for a repo without commits
|
||||||
|
git init $repo
|
||||||
|
path10=$(nix eval --impure --raw --expr "(builtins.fetchGit \"file://$repo\").outPath")
|
||||||
|
|
6
tests/fetchPath.sh
Normal file
6
tests/fetchPath.sh
Normal file
|
@ -0,0 +1,6 @@
|
||||||
|
source common.sh
|
||||||
|
|
||||||
|
touch foo -t 202211111111
|
||||||
|
# We only check whether 2022-11-1* **:**:** is the last modified date since
|
||||||
|
# `lastModified` is transformed into UTC in `builtins.fetchTarball`.
|
||||||
|
[[ "$(nix eval --impure --raw --expr "(builtins.fetchTree \"path://$PWD/foo\").lastModifiedDate")" =~ 2022111.* ]]
|
|
@ -21,6 +21,7 @@ nix_tests = \
|
||||||
tarball.sh \
|
tarball.sh \
|
||||||
fetchGit.sh \
|
fetchGit.sh \
|
||||||
fetchurl.sh \
|
fetchurl.sh \
|
||||||
|
fetchPath.sh \
|
||||||
simple.sh \
|
simple.sh \
|
||||||
referrers.sh \
|
referrers.sh \
|
||||||
optimise-store.sh \
|
optimise-store.sh \
|
||||||
|
@ -52,6 +53,7 @@ nix_tests = \
|
||||||
build-remote-content-addressed-floating.sh \
|
build-remote-content-addressed-floating.sh \
|
||||||
nar-access.sh \
|
nar-access.sh \
|
||||||
pure-eval.sh \
|
pure-eval.sh \
|
||||||
|
eval.sh \
|
||||||
ca/post-hook.sh \
|
ca/post-hook.sh \
|
||||||
repl.sh \
|
repl.sh \
|
||||||
ca/repl.sh \
|
ca/repl.sh \
|
||||||
|
@ -92,8 +94,9 @@ nix_tests = \
|
||||||
bash-profile.sh \
|
bash-profile.sh \
|
||||||
pass-as-file.sh \
|
pass-as-file.sh \
|
||||||
describe-stores.sh \
|
describe-stores.sh \
|
||||||
store-ping.sh \
|
nix-profile.sh \
|
||||||
nix-profile.sh
|
suggestions.sh \
|
||||||
|
store-ping.sh
|
||||||
|
|
||||||
ifeq ($(HAVE_LIBCPUID), 1)
|
ifeq ($(HAVE_LIBCPUID), 1)
|
||||||
nix_tests += compute-levels.sh
|
nix_tests += compute-levels.sh
|
||||||
|
|
44
tests/suggestions.sh
Normal file
44
tests/suggestions.sh
Normal file
|
@ -0,0 +1,44 @@
|
||||||
|
source common.sh
|
||||||
|
|
||||||
|
clearStore
|
||||||
|
|
||||||
|
cd "$TEST_HOME"
|
||||||
|
|
||||||
|
cat <<EOF > flake.nix
|
||||||
|
{
|
||||||
|
outputs = a: {
|
||||||
|
packages.$system = {
|
||||||
|
foo = 1;
|
||||||
|
fo1 = 1;
|
||||||
|
fo2 = 1;
|
||||||
|
fooo = 1;
|
||||||
|
foooo = 1;
|
||||||
|
fooooo = 1;
|
||||||
|
fooooo1 = 1;
|
||||||
|
fooooo2 = 1;
|
||||||
|
fooooo3 = 1;
|
||||||
|
fooooo4 = 1;
|
||||||
|
fooooo5 = 1;
|
||||||
|
fooooo6 = 1;
|
||||||
|
};
|
||||||
|
};
|
||||||
|
}
|
||||||
|
EOF
|
||||||
|
|
||||||
|
# Probable typo in the requested attribute path. Suggest some close possibilities
|
||||||
|
NIX_BUILD_STDERR_WITH_SUGGESTIONS=$(! nix build .\#fob 2>&1 1>/dev/null)
|
||||||
|
[[ "$NIX_BUILD_STDERR_WITH_SUGGESTIONS" =~ "Did you mean one of fo1, fo2, foo or fooo?" ]] || \
|
||||||
|
fail "The nix build stderr should suggest the three closest possiblities"
|
||||||
|
|
||||||
|
# None of the possible attributes is close to `bar`, so shouldn’t suggest anything
|
||||||
|
NIX_BUILD_STDERR_WITH_NO_CLOSE_SUGGESTION=$(! nix build .\#bar 2>&1 1>/dev/null)
|
||||||
|
[[ ! "$NIX_BUILD_STDERR_WITH_NO_CLOSE_SUGGESTION" =~ "Did you mean" ]] || \
|
||||||
|
fail "The nix build stderr shouldn’t suggest anything if there’s nothing relevant to suggest"
|
||||||
|
|
||||||
|
NIX_EVAL_STDERR_WITH_SUGGESTIONS=$(! nix build --impure --expr '(builtins.getFlake (builtins.toPath ./.)).packages.'$system'.fob' 2>&1 1>/dev/null)
|
||||||
|
[[ "$NIX_EVAL_STDERR_WITH_SUGGESTIONS" =~ "Did you mean one of fo1, fo2, foo or fooo?" ]] || \
|
||||||
|
fail "The evaluator should suggest the three closest possiblities"
|
||||||
|
|
||||||
|
NIX_EVAL_STDERR_WITH_SUGGESTIONS=$(! nix build --impure --expr '({ foo }: foo) { foo = 1; fob = 2; }' 2>&1 1>/dev/null)
|
||||||
|
[[ "$NIX_EVAL_STDERR_WITH_SUGGESTIONS" =~ "Did you mean foo?" ]] || \
|
||||||
|
fail "The evaluator should suggest the three closest possiblities"
|
|
@ -17,6 +17,16 @@ outPath10=$(nix-env -f ./user-envs.nix -qa --out-path --no-name '*' | grep foo-1
|
||||||
drvPath10=$(nix-env -f ./user-envs.nix -qa --drv-path --no-name '*' | grep foo-1.0)
|
drvPath10=$(nix-env -f ./user-envs.nix -qa --drv-path --no-name '*' | grep foo-1.0)
|
||||||
[ -n "$outPath10" -a -n "$drvPath10" ]
|
[ -n "$outPath10" -a -n "$drvPath10" ]
|
||||||
|
|
||||||
|
# Query with json
|
||||||
|
nix-env -f ./user-envs.nix -qa --json | jq -e '.[] | select(.name == "bar-0.1") | [
|
||||||
|
.outputName == "out",
|
||||||
|
.outputs.out == null
|
||||||
|
] | all'
|
||||||
|
nix-env -f ./user-envs.nix -qa --json --out-path | jq -e '.[] | select(.name == "bar-0.1") | [
|
||||||
|
.outputName == "out",
|
||||||
|
(.outputs.out | test("'$NIX_STORE_DIR'.*-0\\.1"))
|
||||||
|
] | all'
|
||||||
|
|
||||||
# Query descriptions.
|
# Query descriptions.
|
||||||
nix-env -f ./user-envs.nix -qa '*' --description | grep -q silly
|
nix-env -f ./user-envs.nix -qa '*' --description | grep -q silly
|
||||||
rm -rf $HOME/.nix-defexpr
|
rm -rf $HOME/.nix-defexpr
|
||||||
|
|
Loading…
Reference in a new issue