lix-project/lix
18
39
Fork 15
Code Issues 288 Code Review (Gerrit) Projects 1 Releases Wiki Activity

Merge remote-tracking branch 'upstream/master' into hash-always-has-type

Browse source
This commit is contained in:
John Ericson 2020-07-27 14:22:45 +00:00
parent 5ea817dace d7c0f094cb
commit 699fc89b39
35 changed files with 542 additions and 522 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
Makefile.config.in
View file

@ -19,6 +19,7 @@ LIBLZMA_LIBS = @LIBLZMA_LIBS@
OPENSSL_LIBS = @OPENSSL_LIBS@ OPENSSL_LIBS = @OPENSSL_LIBS@
PACKAGE_NAME = @PACKAGE_NAME@ PACKAGE_NAME = @PACKAGE_NAME@
PACKAGE_VERSION = @PACKAGE_VERSION@ PACKAGE_VERSION = @PACKAGE_VERSION@
SHELL = @bash@
SODIUM_LIBS = @SODIUM_LIBS@ SODIUM_LIBS = @SODIUM_LIBS@
SQLITE3_LIBS = @SQLITE3_LIBS@ SQLITE3_LIBS = @SQLITE3_LIBS@
bash = @bash@ bash = @bash@

25
README.md
View file

@ -12,7 +12,7 @@ for more details.
On Linux and macOS the easiest way to Install Nix is to run the following shell command On Linux and macOS the easiest way to Install Nix is to run the following shell command
(as a user other than root): (as a user other than root):
``` ```console
$ curl -L https://nixos.org/nix/install | sh $ curl -L https://nixos.org/nix/install | sh
``` ```
@ -20,27 +20,8 @@ Information on additional installation methods is available on the [Nix download
## Building And Developing ## Building And Developing
### Building Nix See our [Hacking guide](https://hydra.nixos.org/job/nix/master/build.x86_64-linux/latest/download-by-type/doc/manual#chap-hacking) in our manual for instruction on how to
build nix from source with nix-build or how to get a development environment.
You can build Nix using one of the targets provided by [release.nix](./release.nix):
```
$ nix-build ./release.nix -A build.aarch64-linux
$ nix-build ./release.nix -A build.x86_64-darwin
$ nix-build ./release.nix -A build.i686-linux
$ nix-build ./release.nix -A build.x86_64-linux
```
### Development Environment
You can use the provided `shell.nix` to get a working development environment:
```
$ nix-shell
$ ./bootstrap.sh
$ ./configure
$ make
```
## Additional Resources ## Additional Resources

119
doc/manual/expressions/builder-syntax.xml
View file

@ -1,119 +0,0 @@
<section xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id='sec-builder-syntax'>
<title>Builder Syntax</title>
<example xml:id='ex-hello-builder'><title>Build script for GNU Hello
(<filename>builder.sh</filename>)</title>
<programlisting>
source $stdenv/setup <co xml:id='ex-hello-builder-co-1' />
PATH=$perl/bin:$PATH <co xml:id='ex-hello-builder-co-2' />
tar xvfz $src <co xml:id='ex-hello-builder-co-3' />
cd hello-*
./configure --prefix=$out <co xml:id='ex-hello-builder-co-4' />
make <co xml:id='ex-hello-builder-co-5' />
make install</programlisting>
</example>
<para><xref linkend='ex-hello-builder' /> shows the builder referenced
from Hello's Nix expression (stored in
<filename>pkgs/applications/misc/hello/ex-1/builder.sh</filename>).
The builder can actually be made a lot shorter by using the
<emphasis>generic builder</emphasis> functions provided by
<varname>stdenv</varname>, but here we write out the build steps to
elucidate what a builder does. It performs the following
steps:</para>
<calloutlist>
<callout arearefs='ex-hello-builder-co-1'>
<para>When Nix runs a builder, it initially completely clears the
environment (except for the attributes declared in the
derivation). For instance, the <envar>PATH</envar> variable is
empty<footnote><para>Actually, it's initialised to
<filename>/path-not-set</filename> to prevent Bash from setting it
to a default value.</para></footnote>. This is done to prevent
undeclared inputs from being used in the build process. If for
example the <envar>PATH</envar> contained
<filename>/usr/bin</filename>, then you might accidentally use
<filename>/usr/bin/gcc</filename>.</para>
<para>So the first step is to set up the environment. This is
done by calling the <filename>setup</filename> script of the
standard environment. The environment variable
<envar>stdenv</envar> points to the location of the standard
environment being used. (It wasn't specified explicitly as an
attribute in <xref linkend='ex-hello-nix' />, but
<varname>mkDerivation</varname> adds it automatically.)</para>
</callout>
<callout arearefs='ex-hello-builder-co-2'>
<para>Since Hello needs Perl, we have to make sure that Perl is in
the <envar>PATH</envar>. The <envar>perl</envar> environment
variable points to the location of the Perl package (since it
was passed in as an attribute to the derivation), so
<filename><replaceable>$perl</replaceable>/bin</filename> is the
directory containing the Perl interpreter.</para>
</callout>
<callout arearefs='ex-hello-builder-co-3'>
<para>Now we have to unpack the sources. The
<varname>src</varname> attribute was bound to the result of
fetching the Hello source tarball from the network, so the
<envar>src</envar> environment variable points to the location in
the Nix store to which the tarball was downloaded. After
unpacking, we <command>cd</command> to the resulting source
directory.</para>
<para>The whole build is performed in a temporary directory
created in <varname>/tmp</varname>, by the way. This directory is
removed after the builder finishes, so there is no need to clean
up the sources afterwards. Also, the temporary directory is
always newly created, so you don't have to worry about files from
previous builds interfering with the current build.</para>
</callout>
<callout arearefs='ex-hello-builder-co-4'>
<para>GNU Hello is a typical Autoconf-based package, so we first
have to run its <filename>configure</filename> script. In Nix
every package is stored in a separate location in the Nix store,
for instance
<filename>/nix/store/9a54ba97fb71b65fda531012d0443ce2-hello-2.1.1</filename>.
Nix computes this path by cryptographically hashing all attributes
of the derivation. The path is passed to the builder through the
<envar>out</envar> environment variable. So here we give
<filename>configure</filename> the parameter
<literal>--prefix=$out</literal> to cause Hello to be installed in
the expected location.</para>
</callout>
<callout arearefs='ex-hello-builder-co-5'>
<para>Finally we build Hello (<literal>make</literal>) and install
it into the location specified by <envar>out</envar>
(<literal>make install</literal>).</para>
</callout>
</calloutlist>
<para>If you are wondering about the absence of error checking on the
result of various commands called in the builder: this is because the
shell script is evaluated with Bash's <option>-e</option> option,
which causes the script to be aborted if any command fails without an
error check.</para>
</section>

45
doc/manual/hacking.xml
View file

@ -4,18 +4,37 @@
<title>Hacking</title> <title>Hacking</title>
<para>This section provides some notes on how to hack on Nix. To get <para>This section provides some notes on how to hack on Nix. To get
the latest version of Nix from GitHub: the latest version of Nix from GitHub:
<screen> <screen>
$ git clone git://github.com/NixOS/nix.git $ git clone https://github.com/NixOS/nix.git
$ cd nix $ cd nix
</screen> </screen>
</para> </para>
<para>To build it and its dependencies: <para>To build Nix for the current operating system/architecture use
<screen> <screen>
$ nix-build release.nix -A build.x86_64-linux $ nix-build
</screen> </screen>
or if you have a flakes-enabled nix:
<screen>
$ nix build
</screen>
This will build <literal>defaultPackage</literal> attribute defined in the <literal>flake.nix</literal> file.
To build for other platforms add one of the following suffixes to it: aarch64-linux,
i686-linux, x86_64-darwin, x86_64-linux.
i.e.
<screen>
nix-build -A defaultPackage.x86_64-linux
</screen>
</para> </para>
<para>To build all dependencies and start a shell in which all <para>To build all dependencies and start a shell in which all
@ -27,13 +46,27 @@ $ nix-shell
To build Nix itself in this shell: To build Nix itself in this shell:
<screen> <screen>
[nix-shell]$ ./bootstrap.sh [nix-shell]$ ./bootstrap.sh
[nix-shell]$ configurePhase [nix-shell]$ ./configure $configureFlags
[nix-shell]$ make [nix-shell]$ make -j $NIX_BUILD_CORES
</screen> </screen>
To install it in <literal>$(pwd)/inst</literal> and test it: To install it in <literal>$(pwd)/inst</literal> and test it:
<screen> <screen>
[nix-shell]$ make install [nix-shell]$ make install
[nix-shell]$ make installcheck [nix-shell]$ make installcheck
[nix-shell]$ ./inst/bin/nix --version
nix (Nix) 2.4
</screen>
If you have a flakes-enabled nix you can replace:
<screen>
$ nix-shell
</screen>
by:
<screen>
$ nix develop
</screen> </screen>
</para> </para>

4
mk/precompiled-headers.mk
View file

@ -21,13 +21,13 @@ clean-files += $(GCH) $(PCH)
ifeq ($(PRECOMPILE_HEADERS), 1) ifeq ($(PRECOMPILE_HEADERS), 1)
ifeq ($(CXX), g++) ifeq ($(findstring g++,$(CXX)), g++)
GLOBAL_CXXFLAGS_PCH += -include $(buildprefix)precompiled-headers.h -Winvalid-pch GLOBAL_CXXFLAGS_PCH += -include $(buildprefix)precompiled-headers.h -Winvalid-pch
GLOBAL_ORDER_AFTER += $(GCH) GLOBAL_ORDER_AFTER += $(GCH)
else ifeq ($(CXX), clang++) else ifeq ($(findstring clang++,$(CXX)), clang++)
GLOBAL_CXXFLAGS_PCH += -include-pch $(PCH) -Winvalid-pch GLOBAL_CXXFLAGS_PCH += -include-pch $(PCH) -Winvalid-pch

4
scripts/install-nix-from-closure.sh
View file

@ -207,7 +207,7 @@ if [ -z "$NIX_INSTALLER_NO_MODIFY_PROFILE" ]; then
if [ -w "$fn" ]; then if [ -w "$fn" ]; then
if ! grep -q "$p" "$fn"; then if ! grep -q "$p" "$fn"; then
echo "modifying $fn..." >&2 echo "modifying $fn..." >&2
echo "if [ -e $p ]; then . $p; fi # added by Nix installer" >> "$fn" echo -e "\nif [ -e $p ]; then . $p; fi # added by Nix installer" >> "$fn"
fi fi
added=1 added=1
break break
@ -218,7 +218,7 @@ if [ -z "$NIX_INSTALLER_NO_MODIFY_PROFILE" ]; then
if [ -w "$fn" ]; then if [ -w "$fn" ]; then
if ! grep -q "$p" "$fn"; then if ! grep -q "$p" "$fn"; then
echo "modifying $fn..." >&2 echo "modifying $fn..." >&2
echo "if [ -e $p ]; then . $p; fi # added by Nix installer" >> "$fn" echo -e "\nif [ -e $p ]; then . $p; fi # added by Nix installer" >> "$fn"
fi fi
added=1 added=1
break break

3
shell.nix Normal file
View file

@ -0,0 +1,3 @@
(import (fetchTarball https://github.com/edolstra/flake-compat/archive/master.tar.gz) {
src = ./.;
}).shellNix

79
src/libexpr/flake/flakeref.cc
View file

@ -102,56 +102,61 @@ std::pair<FlakeRef, std::string> parseFlakeRefWithFragment(
percentDecode(std::string(match[6]))); percentDecode(std::string(match[6])));
} }
/* Check if 'url' is a path (either absolute or relative to
'baseDir'). If so, search upward to the root of the repo
(i.e. the directory containing .git). */
else if (std::regex_match(url, match, pathUrlRegex)) { else if (std::regex_match(url, match, pathUrlRegex)) {
std::string path = match[1]; std::string path = match[1];
if (!baseDir && !hasPrefix(path, "/")) std::string fragment = percentDecode(std::string(match[3]));
throw BadURL("flake reference '%s' is not an absolute path", url);
path = absPath(path, baseDir, true);
if (!S_ISDIR(lstat(path).st_mode)) if (baseDir) {
throw BadURL("path '%s' is not a flake (because it's not a directory)", path); /* Check if 'url' is a path (either absolute or relative
to 'baseDir'). If so, search upward to the root of the
repo (i.e. the directory containing .git). */
if (!allowMissing && !pathExists(path + "/flake.nix")) path = absPath(path, baseDir, true);
throw BadURL("path '%s' is not a flake (because it doesn't contain a 'flake.nix' file)", path);
auto fragment = percentDecode(std::string(match[3])); if (!S_ISDIR(lstat(path).st_mode))
throw BadURL("path '%s' is not a flake (because it's not a directory)", path);
auto flakeRoot = path; if (!allowMissing && !pathExists(path + "/flake.nix"))
std::string subdir; throw BadURL("path '%s' is not a flake (because it doesn't contain a 'flake.nix' file)", path);
while (flakeRoot != "/") { auto flakeRoot = path;
if (pathExists(flakeRoot + "/.git")) { std::string subdir;
auto base = std::string("git+file://") + flakeRoot;
auto parsedURL = ParsedURL{ while (flakeRoot != "/") {
.url = base, // FIXME if (pathExists(flakeRoot + "/.git")) {
.base = base, auto base = std::string("git+file://") + flakeRoot;
.scheme = "git+file",
.authority = "",
.path = flakeRoot,
.query = decodeQuery(match[2]),
};
if (subdir != "") { auto parsedURL = ParsedURL{
if (parsedURL.query.count("dir")) .url = base, // FIXME
throw Error("flake URL '%s' has an inconsistent 'dir' parameter", url); .base = base,
parsedURL.query.insert_or_assign("dir", subdir); .scheme = "git+file",
.authority = "",
.path = flakeRoot,
.query = decodeQuery(match[2]),
};
if (subdir != "") {
if (parsedURL.query.count("dir"))
throw Error("flake URL '%s' has an inconsistent 'dir' parameter", url);
parsedURL.query.insert_or_assign("dir", subdir);
}
if (pathExists(flakeRoot + "/.git/shallow"))
parsedURL.query.insert_or_assign("shallow", "1");
return std::make_pair(
FlakeRef(Input::fromURL(parsedURL), get(parsedURL.query, "dir").value_or("")),
fragment);
} }
if (pathExists(flakeRoot + "/.git/shallow")) subdir = std::string(baseNameOf(flakeRoot)) + (subdir.empty() ? "" : "/" + subdir);
parsedURL.query.insert_or_assign("shallow", "1"); flakeRoot = dirOf(flakeRoot);
return std::make_pair(
FlakeRef(Input::fromURL(parsedURL), get(parsedURL.query, "dir").value_or("")),
fragment);
} }
subdir = std::string(baseNameOf(flakeRoot)) + (subdir.empty() ? "" : "/" + subdir); } else {
flakeRoot = dirOf(flakeRoot); if (!hasPrefix(path, "/"))
throw BadURL("flake reference '%s' is not an absolute path", url);
path = canonPath(path);
} }
fetchers::Attrs attrs; fetchers::Attrs attrs;

2
src/libstore/build.cc
View file

@ -2774,7 +2774,7 @@ struct RestrictedStore : public LocalFSStore
goal.addDependency(info.path); goal.addDependency(info.path);
} }
StorePath addToStoreFromDump(const string & dump, const string & name, StorePath addToStoreFromDump(Source & dump, const string & name,
FileIngestionMethod method = FileIngestionMethod::Recursive, HashType hashAlgo = htSHA256, RepairFlag repair = NoRepair) override FileIngestionMethod method = FileIngestionMethod::Recursive, HashType hashAlgo = htSHA256, RepairFlag repair = NoRepair) override
{ {
auto path = next->addToStoreFromDump(dump, name, method, hashAlgo, repair); auto path = next->addToStoreFromDump(dump, name, method, hashAlgo, repair);

62
src/libstore/daemon.cc
View file

@ -173,31 +173,6 @@ struct TunnelSource : BufferedSource
} }
}; };
/* If the NAR archive contains a single file at top-level, then save
the contents of the file to `s'. Otherwise barf. */
struct RetrieveRegularNARSink : ParseSink
{
bool regular;
string s;
RetrieveRegularNARSink() : regular(true) { }
void createDirectory(const Path & path)
{
regular = false;
}
void receiveContents(unsigned char * data, unsigned int len)
{
s.append((const char *) data, len);
}
void createSymlink(const Path & path, const string & target)
{
regular = false;
}
};
struct ClientSettings struct ClientSettings
{ {
bool keepFailed; bool keepFailed;
@ -375,25 +350,28 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
} }
case wopAddToStore: { case wopAddToStore: {
std::string s, baseName; HashType hashAlgo;
std::string baseName;
FileIngestionMethod method; FileIngestionMethod method;
{ {
bool fixed; uint8_t recursive; bool fixed;
from >> baseName >> fixed /* obsolete */ >> recursive >> s; uint8_t recursive;
std::string hashAlgoRaw;
from >> baseName >> fixed /* obsolete */ >> recursive >> hashAlgoRaw;
if (recursive > (uint8_t) FileIngestionMethod::Recursive) if (recursive > (uint8_t) FileIngestionMethod::Recursive)
throw Error("unsupported FileIngestionMethod with value of %i; you may need to upgrade nix-daemon", recursive); throw Error("unsupported FileIngestionMethod with value of %i; you may need to upgrade nix-daemon", recursive);
method = FileIngestionMethod { recursive }; method = FileIngestionMethod { recursive };
/* Compatibility hack. */ /* Compatibility hack. */
if (!fixed) { if (!fixed) {
s = "sha256"; hashAlgoRaw = "sha256";
method = FileIngestionMethod::Recursive; method = FileIngestionMethod::Recursive;
} }
hashAlgo = parseHashType(hashAlgoRaw);
} }
HashType hashAlgo = parseHashType(s);
StringSink savedNAR; StringSink saved;
TeeSource savedNARSource(from, savedNAR); TeeSource savedNARSource(from, saved);
RetrieveRegularNARSink savedRegular; RetrieveRegularNARSink savedRegular { saved };
if (method == FileIngestionMethod::Recursive) { if (method == FileIngestionMethod::Recursive) {
/* Get the entire NAR dump from the client and save it to /* Get the entire NAR dump from the client and save it to
@ -407,11 +385,9 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
logger->startWork(); logger->startWork();
if (!savedRegular.regular) throw Error("regular file expected"); if (!savedRegular.regular) throw Error("regular file expected");
auto path = store->addToStoreFromDump( // FIXME: try to stream directly from `from`.
method == FileIngestionMethod::Recursive ? *savedNAR.s : savedRegular.s, StringSource dumpSource { *saved.s };
baseName, auto path = store->addToStoreFromDump(dumpSource, baseName, method, hashAlgo);
method,
hashAlgo);
logger->stopWork(); logger->stopWork();
to << store->printStorePath(path); to << store->printStorePath(path);
@ -727,15 +703,15 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
if (!trusted) if (!trusted)
info.ultimate = false; info.ultimate = false;
std::string saved;
std::unique_ptr<Source> source; std::unique_ptr<Source> source;
if (GET_PROTOCOL_MINOR(clientVersion) >= 21) if (GET_PROTOCOL_MINOR(clientVersion) >= 21)
source = std::make_unique<TunnelSource>(from, to); source = std::make_unique<TunnelSource>(from, to);
else { else {
TeeParseSink tee(from); StringSink saved;
parseDump(tee, tee.source); TeeSource tee { from, saved };
saved = std::move(*tee.saved.s); ParseSink ether;
source = std::make_unique<StringSource>(saved); parseDump(ether, tee);
source = std::make_unique<StringSource>(std::move(*saved.s));
} }
logger->startWork(); logger->startWork();

8
src/libstore/derivations.cc
View file

@ -7,14 +7,6 @@
namespace nix { namespace nix {
const StorePath & BasicDerivation::findOutput(const string & id) const
{
auto i = outputs.find(id);
if (i == outputs.end())
throw Error("derivation has no output '%s'", id);
return i->second.path;
}
bool BasicDerivation::isBuiltin() const bool BasicDerivation::isBuiltin() const
{ {

4
src/libstore/derivations.hh
View file

@ -39,10 +39,6 @@ struct BasicDerivation
BasicDerivation() { } BasicDerivation() { }
virtual ~BasicDerivation() { }; virtual ~BasicDerivation() { };
/* Return the path corresponding to the output identifier `id' in
the given derivation. */
const StorePath & findOutput(const std::string & id) const;
bool isBuiltin() const; bool isBuiltin() const;
/* Return true iff this is a fixed-output derivation. */ /* Return true iff this is a fixed-output derivation. */

12
src/libstore/export-import.cc
View file

@ -60,8 +60,10 @@ StorePaths Store::importPaths(Source & source, CheckSigsFlag checkSigs)
if (n != 1) throw Error("input doesn't look like something created by 'nix-store --export'"); if (n != 1) throw Error("input doesn't look like something created by 'nix-store --export'");
/* Extract the NAR from the source. */ /* Extract the NAR from the source. */
TeeParseSink tee(source); StringSink saved;
parseDump(tee, tee.source); TeeSource tee { source, saved };
ParseSink ether;
parseDump(ether, tee);
uint32_t magic = readInt(source); uint32_t magic = readInt(source);
if (magic != exportMagic) if (magic != exportMagic)
@ -77,15 +79,15 @@ StorePaths Store::importPaths(Source & source, CheckSigsFlag checkSigs)
if (deriver != "") if (deriver != "")
info.deriver = parseStorePath(deriver); info.deriver = parseStorePath(deriver);
info.narHash = hashString(htSHA256, *tee.saved.s); info.narHash = hashString(htSHA256, *saved.s);
info.narSize = tee.saved.s->size(); info.narSize = saved.s->size();
// Ignore optional legacy signature. // Ignore optional legacy signature.
if (readInt(source) == 1) if (readInt(source) == 1)
readString(source); readString(source);
// Can't use underlying source, which would have been exhausted // Can't use underlying source, which would have been exhausted
auto source = StringSource { *tee.saved.s }; auto source = StringSource { *saved.s };
addToStore(info, source, NoRepair, checkSigs); addToStore(info, source, NoRepair, checkSigs);
res.push_back(info.path); res.push_back(info.path);

39
src/libstore/filetransfer.cc
View file

@ -124,7 +124,7 @@ struct curlFileTransfer : public FileTransfer
if (requestHeaders) curl_slist_free_all(requestHeaders); if (requestHeaders) curl_slist_free_all(requestHeaders);
try { try {
if (!done) if (!done)
fail(FileTransferError(Interrupted, "download of '%s' was interrupted", request.uri)); fail(FileTransferError(Interrupted, nullptr, "download of '%s' was interrupted", request.uri));
} catch (...) { } catch (...) {
ignoreException(); ignoreException();
} }
@ -145,6 +145,7 @@ struct curlFileTransfer : public FileTransfer
LambdaSink finalSink; LambdaSink finalSink;
std::shared_ptr<CompressionSink> decompressionSink; std::shared_ptr<CompressionSink> decompressionSink;
std::optional<StringSink> errorSink;
std::exception_ptr writeException; std::exception_ptr writeException;
@ -154,9 +155,19 @@ struct curlFileTransfer : public FileTransfer
size_t realSize = size * nmemb; size_t realSize = size * nmemb;
result.bodySize += realSize; result.bodySize += realSize;
if (!decompressionSink) if (!decompressionSink) {
decompressionSink = makeDecompressionSink(encoding, finalSink); decompressionSink = makeDecompressionSink(encoding, finalSink);
if (! successfulStatuses.count(getHTTPStatus())) {
// In this case we want to construct a TeeSink, to keep
// the response around (which we figure won't be big
// like an actual download should be) to improve error
// messages.
errorSink = StringSink { };
}
}
if (errorSink)
(*errorSink)((unsigned char *) contents, realSize);
(*decompressionSink)((unsigned char *) contents, realSize); (*decompressionSink)((unsigned char *) contents, realSize);
return realSize; return realSize;
@ -412,16 +423,21 @@ struct curlFileTransfer : public FileTransfer
attempt++; attempt++;
std::shared_ptr<std::string> response;
if (errorSink)
response = errorSink->s;
auto exc = auto exc =
code == CURLE_ABORTED_BY_CALLBACK && _isInterrupted code == CURLE_ABORTED_BY_CALLBACK && _isInterrupted
? FileTransferError(Interrupted, fmt("%s of '%s' was interrupted", request.verb(), request.uri)) ? FileTransferError(Interrupted, response, "%s of '%s' was interrupted", request.verb(), request.uri)
: httpStatus != 0 : httpStatus != 0
? FileTransferError(err, ? FileTransferError(err,
response,
fmt("unable to %s '%s': HTTP error %d ('%s')", fmt("unable to %s '%s': HTTP error %d ('%s')",
request.verb(), request.uri, httpStatus, statusMsg) request.verb(), request.uri, httpStatus, statusMsg)
+ (code == CURLE_OK ? "" : fmt(" (curl error: %s)", curl_easy_strerror(code))) + (code == CURLE_OK ? "" : fmt(" (curl error: %s)", curl_easy_strerror(code)))
) )
: FileTransferError(err, : FileTransferError(err,
response,
fmt("unable to %s '%s': %s (%d)", fmt("unable to %s '%s': %s (%d)",
request.verb(), request.uri, curl_easy_strerror(code), code)); request.verb(), request.uri, curl_easy_strerror(code), code));
@ -679,7 +695,7 @@ struct curlFileTransfer : public FileTransfer
auto s3Res = s3Helper.getObject(bucketName, key); auto s3Res = s3Helper.getObject(bucketName, key);
FileTransferResult res; FileTransferResult res;
if (!s3Res.data) if (!s3Res.data)
throw FileTransferError(NotFound, fmt("S3 object '%s' does not exist", request.uri)); throw FileTransferError(NotFound, nullptr, "S3 object '%s' does not exist", request.uri);
res.data = s3Res.data; res.data = s3Res.data;
callback(std::move(res)); callback(std::move(res));
#else #else
@ -824,6 +840,21 @@ void FileTransfer::download(FileTransferRequest && request, Sink & sink)
} }
} }
template<typename... Args>
FileTransferError::FileTransferError(FileTransfer::Error error, std::shared_ptr<string> response, const Args & ... args)
: Error(args...), error(error), response(response)
{
const auto hf = hintfmt(args...);
// FIXME: Due to https://github.com/NixOS/nix/issues/3841 we don't know how
// to print different messages for different verbosity levels. For now
// we add some heuristics for detecting when we want to show the response.
if (response && (response->size() < 1024 || response->find("<html>") != string::npos)) {
err.hint = hintfmt("%1%\n\nresponse body:\n\n%2%", normaltxt(hf.str()), *response);
} else {
err.hint = hf;
}
}
bool isUri(const string & s) bool isUri(const string & s)
{ {
if (s.compare(0, 8, "channel:") == 0) return true; if (s.compare(0, 8, "channel:") == 0) return true;

8
src/libstore/filetransfer.hh
View file

@ -103,10 +103,12 @@ class FileTransferError : public Error
{ {
public: public:
FileTransfer::Error error; FileTransfer::Error error;
std::shared_ptr<string> response; // intentionally optional
template<typename... Args> template<typename... Args>
FileTransferError(FileTransfer::Error error, const Args & ... args) FileTransferError(FileTransfer::Error error, std::shared_ptr<string> response, const Args & ... args);
: Error(args...), error(error)
{ } virtual const char* sname() const override { return "FileTransferError"; }
}; };
bool isUri(const string & s); bool isUri(const string & s);

168
src/libstore/local-store.cc
View file

@ -1033,82 +1033,26 @@ void LocalStore::addToStore(const ValidPathInfo & info, Source & source,
} }
StorePath LocalStore::addToStoreFromDump(const string & dump, const string & name,
FileIngestionMethod method, HashType hashAlgo, RepairFlag repair)
{
Hash h = hashString(hashAlgo, dump);
auto dstPath = makeFixedOutputPath(method, h, name);
addTempRoot(dstPath);
if (repair || !isValidPath(dstPath)) {
/* The first check above is an optimisation to prevent
unnecessary lock acquisition. */
auto realPath = Store::toRealPath(dstPath);
PathLocks outputLock({realPath});
if (repair || !isValidPath(dstPath)) {
deletePath(realPath);
autoGC();
if (method == FileIngestionMethod::Recursive) {
StringSource source(dump);
restorePath(realPath, source);
} else
writeFile(realPath, dump);
canonicalisePathMetaData(realPath, -1);
/* Register the SHA-256 hash of the NAR serialisation of
the path in the database. We may just have computed it
above (if called with recursive == true and hashAlgo ==
sha256); otherwise, compute it here. */
HashResult hash = method == FileIngestionMethod::Recursive
? HashResult {
hashAlgo == htSHA256 ? h : hashString(htSHA256, dump),
dump.size(),
}
: hashPath(htSHA256, realPath);
optimisePath(realPath); // FIXME: combine with hashPath()
ValidPathInfo info(dstPath);
info.narHash = hash.first;
info.narSize = hash.second;
info.ca = FixedOutputHash { .method = method, .hash = h };
registerValidPath(info);
}
outputLock.setDeletion(true);
}
return dstPath;
}
StorePath LocalStore::addToStore(const string & name, const Path & _srcPath, StorePath LocalStore::addToStore(const string & name, const Path & _srcPath,
FileIngestionMethod method, HashType hashAlgo, PathFilter & filter, RepairFlag repair) FileIngestionMethod method, HashType hashAlgo, PathFilter & filter, RepairFlag repair)
{ {
Path srcPath(absPath(_srcPath)); Path srcPath(absPath(_srcPath));
auto source = sinkToSource([&](Sink & sink) {
if (method == FileIngestionMethod::Recursive)
dumpPath(srcPath, sink, filter);
else
readFile(srcPath, sink);
});
return addToStoreFromDump(*source, name, method, hashAlgo, repair);
}
if (method != FileIngestionMethod::Recursive)
return addToStoreFromDump(readFile(srcPath), name, method, hashAlgo, repair);
/* For computing the NAR hash. */ StorePath LocalStore::addToStoreFromDump(Source & source0, const string & name,
auto sha256Sink = std::make_unique<HashSink>(htSHA256); FileIngestionMethod method, HashType hashAlgo, RepairFlag repair)
{
/* For computing the store path. In recursive SHA-256 mode, this /* For computing the store path. */
is the same as the NAR hash, so no need to do it again. */ auto hashSink = std::make_unique<HashSink>(hashAlgo);
std::unique_ptr<HashSink> hashSink = TeeSource source { source0, *hashSink };
hashAlgo == htSHA256
? nullptr
: std::make_unique<HashSink>(hashAlgo);
/* Read the source path into memory, but only if it's up to /* Read the source path into memory, but only if it's up to
narBufferSize bytes. If it's larger, write it to a temporary narBufferSize bytes. If it's larger, write it to a temporary
@ -1116,55 +1060,49 @@ StorePath LocalStore::addToStore(const string & name, const Path & _srcPath,
destination store path is already valid, we just delete the destination store path is already valid, we just delete the
temporary path. Otherwise, we move it to the destination store temporary path. Otherwise, we move it to the destination store
path. */ path. */
bool inMemory = true; bool inMemory = false;
std::string nar;
auto source = sinkToSource([&](Sink & sink) { std::string dump;
LambdaSink sink2([&](const unsigned char * buf, size_t len) { /* Fill out buffer, and decide whether we are working strictly in
(*sha256Sink)(buf, len); memory based on whether we break out because the buffer is full
if (hashSink) (*hashSink)(buf, len); or the original source is empty */
while (dump.size() < settings.narBufferSize) {
if (inMemory) { auto oldSize = dump.size();
if (nar.size() + len > settings.narBufferSize) { constexpr size_t chunkSize = 65536;
inMemory = false; auto want = std::min(chunkSize, settings.narBufferSize - oldSize);
sink << 1; dump.resize(oldSize + want);
sink((const unsigned char *) nar.data(), nar.size()); auto got = 0;
nar.clear(); try {
} else { got = source.read((uint8_t *) dump.data() + oldSize, want);
nar.append((const char *) buf, len); } catch (EndOfFile &) {
} inMemory = true;
} break;
}
if (!inMemory) sink(buf, len); dump.resize(oldSize + got);
}); }
dumpPath(srcPath, sink2, filter);
});
std::unique_ptr<AutoDelete> delTempDir; std::unique_ptr<AutoDelete> delTempDir;
Path tempPath; Path tempPath;
try { if (!inMemory) {
/* Wait for the source coroutine to give us some dummy /* Drain what we pulled so far, and then keep on pulling */
data. This is so that we don't create the temporary StringSource dumpSource { dump };
directory if the NAR fits in memory. */ ChainSource bothSource { dumpSource, source };
readInt(*source);
auto tempDir = createTempDir(realStoreDir, "add"); auto tempDir = createTempDir(realStoreDir, "add");
delTempDir = std::make_unique<AutoDelete>(tempDir); delTempDir = std::make_unique<AutoDelete>(tempDir);
tempPath = tempDir + "/x"; tempPath = tempDir + "/x";
restorePath(tempPath, *source); if (method == FileIngestionMethod::Recursive)
restorePath(tempPath, bothSource);
else
writeFile(tempPath, bothSource);
} catch (EndOfFile &) { dump.clear();
if (!inMemory) throw;
/* The NAR fits in memory, so we didn't do restorePath(). */
} }
auto sha256 = sha256Sink->finish(); auto [hash, size] = hashSink->finish();
Hash hash = hashSink ? hashSink->finish().first : sha256.first;
auto dstPath = makeFixedOutputPath(method, hash, name); auto dstPath = makeFixedOutputPath(method, hash, name);
@ -1186,22 +1124,34 @@ StorePath LocalStore::addToStore(const string & name, const Path & _srcPath,
autoGC(); autoGC();
if (inMemory) { if (inMemory) {
StringSource dumpSource { dump };
/* Restore from the NAR in memory. */ /* Restore from the NAR in memory. */
StringSource source(nar); if (method == FileIngestionMethod::Recursive)
restorePath(realPath, source); restorePath(realPath, dumpSource);
else
writeFile(realPath, dumpSource);
} else { } else {
/* Move the temporary path we restored above. */ /* Move the temporary path we restored above. */
if (rename(tempPath.c_str(), realPath.c_str())) if (rename(tempPath.c_str(), realPath.c_str()))
throw Error("renaming '%s' to '%s'", tempPath, realPath); throw Error("renaming '%s' to '%s'", tempPath, realPath);
} }
/* For computing the nar hash. In recursive SHA-256 mode, this
is the same as the store hash, so no need to do it again. */
auto narHash = std::pair { hash, size };
if (method != FileIngestionMethod::Recursive || hashAlgo != htSHA256) {
HashSink narSink { htSHA256 };
dumpPath(realPath, narSink);
narHash = narSink.finish();
}
canonicalisePathMetaData(realPath, -1); // FIXME: merge into restorePath canonicalisePathMetaData(realPath, -1); // FIXME: merge into restorePath
optimisePath(realPath); optimisePath(realPath);
ValidPathInfo info(dstPath); ValidPathInfo info(dstPath);
info.narHash = sha256.first; info.narHash = narHash.first;
info.narSize = sha256.second; info.narSize = narHash.second;
info.ca = FixedOutputHash { .method = method, .hash = hash }; info.ca = FixedOutputHash { .method = method, .hash = hash };
registerValidPath(info); registerValidPath(info);
} }

2
src/libstore/local-store.hh
View file

@ -153,7 +153,7 @@ public:
in `dump', which is either a NAR serialisation (if recursive == in `dump', which is either a NAR serialisation (if recursive ==
true) or simply the contents of a regular file (if recursive == true) or simply the contents of a regular file (if recursive ==
false). */ false). */
StorePath addToStoreFromDump(const string & dump, const string & name, StorePath addToStoreFromDump(Source & dump, const string & name,
FileIngestionMethod method = FileIngestionMethod::Recursive, HashType hashAlgo = htSHA256, RepairFlag repair = NoRepair) override; FileIngestionMethod method = FileIngestionMethod::Recursive, HashType hashAlgo = htSHA256, RepairFlag repair = NoRepair) override;
StorePath addTextToStore(const string & name, const string & s, StorePath addTextToStore(const string & name, const string & s,

81
src/libstore/profiles.cc
View file

@ -12,30 +12,24 @@
namespace nix { namespace nix {
static bool cmpGensByNumber(const Generation & a, const Generation & b)
{
return a.number < b.number;
}
/* Parse a generation name of the format /* Parse a generation name of the format
`<profilename>-<number>-link'. */ `<profilename>-<number>-link'. */
static int parseName(const string & profileName, const string & name) static std::optional<GenerationNumber> parseName(const string & profileName, const string & name)
{ {
if (string(name, 0, profileName.size() + 1) != profileName + "-") return -1; if (string(name, 0, profileName.size() + 1) != profileName + "-") return {};
string s = string(name, profileName.size() + 1); string s = string(name, profileName.size() + 1);
string::size_type p = s.find("-link"); string::size_type p = s.find("-link");
if (p == string::npos) return -1; if (p == string::npos) return {};
int n; unsigned int n;
if (string2Int(string(s, 0, p), n) && n >= 0) if (string2Int(string(s, 0, p), n) && n >= 0)
return n; return n;
else else
return -1; return {};
} }
Generations findGenerations(Path profile, int & curGen) std::pair<Generations, std::optional<GenerationNumber>> findGenerations(Path profile)
{ {
Generations gens; Generations gens;
@ -43,30 +37,34 @@ Generations findGenerations(Path profile, int & curGen)
auto profileName = std::string(baseNameOf(profile)); auto profileName = std::string(baseNameOf(profile));
for (auto & i : readDirectory(profileDir)) { for (auto & i : readDirectory(profileDir)) {
int n; if (auto n = parseName(profileName, i.name)) {
if ((n = parseName(profileName, i.name)) != -1) { auto path = profileDir + "/" + i.name;
Generation gen;
gen.path = profileDir + "/" + i.name;
gen.number = n;
struct stat st; struct stat st;
if (lstat(gen.path.c_str(), &st) != 0) if (lstat(path.c_str(), &st) != 0)
throw SysError("statting '%1%'", gen.path); throw SysError("statting '%1%'", path);
gen.creationTime = st.st_mtime; gens.push_back({
gens.push_back(gen); .number = *n,
.path = path,
.creationTime = st.st_mtime
});
} }
} }
gens.sort(cmpGensByNumber); gens.sort([](const Generation & a, const Generation & b)
{
return a.number < b.number;
});
curGen = pathExists(profile) return {
gens,
pathExists(profile)
? parseName(profileName, readLink(profile)) ? parseName(profileName, readLink(profile))
: -1; : std::nullopt
};
return gens;
} }
static void makeName(const Path & profile, unsigned int num, static void makeName(const Path & profile, GenerationNumber num,
Path & outLink) Path & outLink)
{ {
Path prefix = (format("%1%-%2%") % profile % num).str(); Path prefix = (format("%1%-%2%") % profile % num).str();
@ -78,10 +76,9 @@ Path createGeneration(ref<LocalFSStore> store, Path profile, Path outPath)
{ {
/* The new generation number should be higher than old the /* The new generation number should be higher than old the
previous ones. */ previous ones. */
int dummy; auto [gens, dummy] = findGenerations(profile);
Generations gens = findGenerations(profile, dummy);
unsigned int num; GenerationNumber num;
if (gens.size() > 0) { if (gens.size() > 0) {
Generation last = gens.back(); Generation last = gens.back();
@ -121,7 +118,7 @@ static void removeFile(const Path & path)
} }
void deleteGeneration(const Path & profile, unsigned int gen) void deleteGeneration(const Path & profile, GenerationNumber gen)
{ {
Path generation; Path generation;
makeName(profile, gen, generation); makeName(profile, gen, generation);
@ -129,7 +126,7 @@ void deleteGeneration(const Path & profile, unsigned int gen)
} }
static void deleteGeneration2(const Path & profile, unsigned int gen, bool dryRun) static void deleteGeneration2(const Path & profile, GenerationNumber gen, bool dryRun)
{ {
if (dryRun) if (dryRun)
printInfo(format("would remove generation %1%") % gen); printInfo(format("would remove generation %1%") % gen);
@ -140,31 +137,29 @@ static void deleteGeneration2(const Path & profile, unsigned int gen, bool dryRu
} }
void deleteGenerations(const Path & profile, const std::set<unsigned int> & gensToDelete, bool dryRun) void deleteGenerations(const Path & profile, const std::set<GenerationNumber> & gensToDelete, bool dryRun)
{ {
PathLocks lock; PathLocks lock;
lockProfile(lock, profile); lockProfile(lock, profile);
int curGen; auto [gens, curGen] = findGenerations(profile);
Generations gens = findGenerations(profile, curGen);
if (gensToDelete.find(curGen) != gensToDelete.end()) if (gensToDelete.count(*curGen))
throw Error("cannot delete current generation of profile %1%'", profile); throw Error("cannot delete current generation of profile %1%'", profile);
for (auto & i : gens) { for (auto & i : gens) {
if (gensToDelete.find(i.number) == gensToDelete.end()) continue; if (!gensToDelete.count(i.number)) continue;
deleteGeneration2(profile, i.number, dryRun); deleteGeneration2(profile, i.number, dryRun);
} }
} }
void deleteGenerationsGreaterThan(const Path & profile, int max, bool dryRun) void deleteGenerationsGreaterThan(const Path & profile, GenerationNumber max, bool dryRun)
{ {
PathLocks lock; PathLocks lock;
lockProfile(lock, profile); lockProfile(lock, profile);
int curGen;
bool fromCurGen = false; bool fromCurGen = false;
Generations gens = findGenerations(profile, curGen); auto [gens, curGen] = findGenerations(profile);
for (auto i = gens.rbegin(); i != gens.rend(); ++i) { for (auto i = gens.rbegin(); i != gens.rend(); ++i) {
if (i->number == curGen) { if (i->number == curGen) {
fromCurGen = true; fromCurGen = true;
@ -186,8 +181,7 @@ void deleteOldGenerations(const Path & profile, bool dryRun)
PathLocks lock; PathLocks lock;
lockProfile(lock, profile); lockProfile(lock, profile);
int curGen; auto [gens, curGen] = findGenerations(profile);
Generations gens = findGenerations(profile, curGen);
for (auto & i : gens) for (auto & i : gens)
if (i.number != curGen) if (i.number != curGen)
@ -200,8 +194,7 @@ void deleteGenerationsOlderThan(const Path & profile, time_t t, bool dryRun)
PathLocks lock; PathLocks lock;
lockProfile(lock, profile); lockProfile(lock, profile);
int curGen; auto [gens, curGen] = findGenerations(profile);
Generations gens = findGenerations(profile, curGen);
bool canDelete = false; bool canDelete = false;
for (auto i = gens.rbegin(); i != gens.rend(); ++i) for (auto i = gens.rbegin(); i != gens.rend(); ++i)

25
src/libstore/profiles.hh
View file

@ -9,37 +9,32 @@
namespace nix { namespace nix {
typedef unsigned int GenerationNumber;
struct Generation struct Generation
{ {
int number; GenerationNumber number;
Path path; Path path;
time_t creationTime; time_t creationTime;
Generation()
{
number = -1;
}
operator bool() const
{
return number != -1;
}
}; };
typedef list<Generation> Generations; typedef std::list<Generation> Generations;
/* Returns the list of currently present generations for the specified /* Returns the list of currently present generations for the specified
profile, sorted by generation number. */ profile, sorted by generation number. Also returns the number of
Generations findGenerations(Path profile, int & curGen); the current generation. */
std::pair<Generations, std::optional<GenerationNumber>> findGenerations(Path profile);
class LocalFSStore; class LocalFSStore;
Path createGeneration(ref<LocalFSStore> store, Path profile, Path outPath); Path createGeneration(ref<LocalFSStore> store, Path profile, Path outPath);
void deleteGeneration(const Path & profile, unsigned int gen); void deleteGeneration(const Path & profile, GenerationNumber gen);
void deleteGenerations(const Path & profile, const std::set<unsigned int> & gensToDelete, bool dryRun); void deleteGenerations(const Path & profile, const std::set<GenerationNumber> & gensToDelete, bool dryRun);
void deleteGenerationsGreaterThan(const Path & profile, const int max, bool dryRun); void deleteGenerationsGreaterThan(const Path & profile, GenerationNumber max, bool dryRun);
void deleteOldGenerations(const Path & profile, bool dryRun); void deleteOldGenerations(const Path & profile, bool dryRun);

15
src/libstore/references.cc
View file

@ -48,13 +48,12 @@ static void search(const unsigned char * s, size_t len,
struct RefScanSink : Sink struct RefScanSink : Sink
{ {
HashSink hashSink;
StringSet hashes; StringSet hashes;
StringSet seen; StringSet seen;
string tail; string tail;
RefScanSink() : hashSink(htSHA256) { } RefScanSink() { }
void operator () (const unsigned char * data, size_t len); void operator () (const unsigned char * data, size_t len);
}; };
@ -62,8 +61,6 @@ struct RefScanSink : Sink
void RefScanSink::operator () (const unsigned char * data, size_t len) void RefScanSink::operator () (const unsigned char * data, size_t len)
{ {
hashSink(data, len);
/* It's possible that a reference spans the previous and current /* It's possible that a reference spans the previous and current
fragment, so search in the concatenation of the tail of the fragment, so search in the concatenation of the tail of the
previous fragment and the start of the current fragment. */ previous fragment and the start of the current fragment. */
@ -82,7 +79,9 @@ void RefScanSink::operator () (const unsigned char * data, size_t len)
std::pair<PathSet, HashResult> scanForReferences(const string & path, std::pair<PathSet, HashResult> scanForReferences(const string & path,
const PathSet & refs) const PathSet & refs)
{ {
RefScanSink sink; RefScanSink refsSink;
HashSink hashSink { htSHA256 };
TeeSink sink { refsSink, hashSink };
std::map<string, Path> backMap; std::map<string, Path> backMap;
/* For efficiency (and a higher hit rate), just search for the /* For efficiency (and a higher hit rate), just search for the
@ -97,7 +96,7 @@ std::pair<PathSet, HashResult> scanForReferences(const string & path,
assert(s.size() == refLength); assert(s.size() == refLength);
assert(backMap.find(s) == backMap.end()); assert(backMap.find(s) == backMap.end());
// parseHash(htSHA256, s); // parseHash(htSHA256, s);
sink.hashes.insert(s); refsSink.hashes.insert(s);
backMap[s] = i; backMap[s] = i;
} }
@ -106,13 +105,13 @@ std::pair<PathSet, HashResult> scanForReferences(const string & path,
/* Map the hashes found back to their store paths. */ /* Map the hashes found back to their store paths. */
PathSet found; PathSet found;
for (auto & i : sink.seen) { for (auto & i : refsSink.seen) {
std::map<string, Path>::iterator j; std::map<string, Path>::iterator j;
if ((j = backMap.find(i)) == backMap.end()) abort(); if ((j = backMap.find(i)) == backMap.end()) abort();
found.insert(j->second); found.insert(j->second);
} }
auto hash = sink.hashSink.finish(); auto hash = hashSink.finish();
return std::pair<PathSet, HashResult>(found, hash); return std::pair<PathSet, HashResult>(found, hash);
} }

88
src/libstore/store-api.cc
View file

@ -222,20 +222,73 @@ StorePath Store::computeStorePathForText(const string & name, const string & s,
} }
/*
The aim of this function is to compute in one pass the correct ValidPathInfo for
the files that we are trying to add to the store. To accomplish that in one
pass, given the different kind of inputs that we can take (normal nar archives,
nar archives with non SHA-256 hashes, and flat files), we set up a net of sinks
and aliases. Also, since the dataflow is obfuscated by this, we include here a
graphviz diagram:
digraph graphname {
node [shape=box]
fileSource -> narSink
narSink [style=dashed]
narSink -> unsualHashTee [style = dashed, label = "Recursive && !SHA-256"]
narSink -> narHashSink [style = dashed, label = "else"]
unsualHashTee -> narHashSink
unsualHashTee -> caHashSink
fileSource -> parseSink
parseSink [style=dashed]
parseSink-> fileSink [style = dashed, label = "Flat"]
parseSink -> blank [style = dashed, label = "Recursive"]
fileSink -> caHashSink
}
*/
ValidPathInfo Store::addToStoreSlow(std::string_view name, const Path & srcPath, ValidPathInfo Store::addToStoreSlow(std::string_view name, const Path & srcPath,
FileIngestionMethod method, HashType hashAlgo, FileIngestionMethod method, HashType hashAlgo,
std::optional<Hash> expectedCAHash) std::optional<Hash> expectedCAHash)
{ {
/* FIXME: inefficient: we're reading/hashing 'tmpFile' three HashSink narHashSink { htSHA256 };
times. */ HashSink caHashSink { hashAlgo };
auto [narHash, narSize] = hashPath(htSHA256, srcPath); /* Note that fileSink and unusualHashTee must be mutually exclusive, since
they both write to caHashSink. Note that that requisite is currently true
because the former is only used in the flat case. */
RetrieveRegularNARSink fileSink { caHashSink };
TeeSink unusualHashTee { narHashSink, caHashSink };
auto hash = method == FileIngestionMethod::Recursive auto & narSink = method == FileIngestionMethod::Recursive && hashAlgo != htSHA256
? hashAlgo == htSHA256 ? static_cast<Sink &>(unusualHashTee)
? narHash : narHashSink;
: hashPath(hashAlgo, srcPath).first
: hashFile(hashAlgo, srcPath); /* Functionally, this means that fileSource will yield the content of
srcPath. The fact that we use scratchpadSink as a temporary buffer here
is an implementation detail. */
auto fileSource = sinkToSource([&](Sink & scratchpadSink) {
dumpPath(srcPath, scratchpadSink);
});
/* tapped provides the same data as fileSource, but we also write all the
information to narSink. */
TeeSource tapped { *fileSource, narSink };
ParseSink blank;
auto & parseSink = method == FileIngestionMethod::Flat
? fileSink
: blank;
/* The information that flows from tapped (besides being replicated in
narSink), is now put in parseSink. */
parseDump(parseSink, tapped);
/* We extract the result of the computation from the sink by calling
finish. */
auto [narHash, narSize] = narHashSink.finish();
auto hash = method == FileIngestionMethod::Recursive && hashAlgo == htSHA256
? narHash
: caHashSink.finish().first;
if (expectedCAHash && expectedCAHash != hash) if (expectedCAHash && expectedCAHash != hash)
throw Error("hash mismatch for '%s'", srcPath); throw Error("hash mismatch for '%s'", srcPath);
@ -246,8 +299,8 @@ ValidPathInfo Store::addToStoreSlow(std::string_view name, const Path & srcPath,
info.ca = FixedOutputHash { .method = method, .hash = hash }; info.ca = FixedOutputHash { .method = method, .hash = hash };
if (!isValidPath(info.path)) { if (!isValidPath(info.path)) {
auto source = sinkToSource([&](Sink & sink) { auto source = sinkToSource([&](Sink & scratchpadSink) {
dumpPath(srcPath, sink); dumpPath(srcPath, scratchpadSink);
}); });
addToStore(info, *source); addToStore(info, *source);
} }
@ -914,12 +967,20 @@ ref<Store> openStore(const std::string & uri_,
throw Error("don't know how to open Nix store '%s'", uri); throw Error("don't know how to open Nix store '%s'", uri);
} }
static bool isNonUriPath(const std::string & spec) {
return
// is not a URL
spec.find("://") == std::string::npos
// Has at least one path separator, and so isn't a single word that
// might be special like "auto"
&& spec.find("/") != std::string::npos;
}
StoreType getStoreType(const std::string & uri, const std::string & stateDir) StoreType getStoreType(const std::string & uri, const std::string & stateDir)
{ {
if (uri == "daemon") { if (uri == "daemon") {
return tDaemon; return tDaemon;
} else if (uri == "local" || hasPrefix(uri, "/")) { } else if (uri == "local" || isNonUriPath(uri)) {
return tLocal; return tLocal;
} else if (uri == "" || uri == "auto") { } else if (uri == "" || uri == "auto") {
if (access(stateDir.c_str(), R_OK | W_OK) == 0) if (access(stateDir.c_str(), R_OK | W_OK) == 0)
@ -943,8 +1004,9 @@ static RegisterStoreImplementation regStore([](
return std::shared_ptr<Store>(std::make_shared<UDSRemoteStore>(params)); return std::shared_ptr<Store>(std::make_shared<UDSRemoteStore>(params));
case tLocal: { case tLocal: {
Store::Params params2 = params; Store::Params params2 = params;
if (hasPrefix(uri, "/")) if (isNonUriPath(uri)) {
params2["root"] = uri; params2["root"] = absPath(uri);
}
return std::shared_ptr<Store>(std::make_shared<LocalStore>(params2)); return std::shared_ptr<Store>(std::make_shared<LocalStore>(params2));
} }
default: default:

2
src/libstore/store-api.hh
View file

@ -461,7 +461,7 @@ public:
std::optional<Hash> expectedCAHash = {}); std::optional<Hash> expectedCAHash = {});
// FIXME: remove? // FIXME: remove?
virtual StorePath addToStoreFromDump(const string & dump, const string & name, virtual StorePath addToStoreFromDump(Source & dump, const string & name,
FileIngestionMethod method = FileIngestionMethod::Recursive, HashType hashAlgo = htSHA256, RepairFlag repair = NoRepair) FileIngestionMethod method = FileIngestionMethod::Recursive, HashType hashAlgo = htSHA256, RepairFlag repair = NoRepair)
{ {
throw Error("addToStoreFromDump() is not supported by this store"); throw Error("addToStoreFromDump() is not supported by this store");

25
src/libutil/archive.hh
View file

@ -63,12 +63,29 @@ struct ParseSink
virtual void createSymlink(const Path & path, const string & target) { }; virtual void createSymlink(const Path & path, const string & target) { };
}; };
struct TeeParseSink : ParseSink /* If the NAR archive contains a single file at top-level, then save
the contents of the file to `s'. Otherwise barf. */
struct RetrieveRegularNARSink : ParseSink
{ {
StringSink saved; bool regular = true;
TeeSource source; Sink & sink;
TeeParseSink(Source & source) : source(source, saved) { } RetrieveRegularNARSink(Sink & sink) : sink(sink) { }
void createDirectory(const Path & path)
{
regular = false;
}
void receiveContents(unsigned char * data, unsigned int len)
{
sink(data, len);
}
void createSymlink(const Path & path, const string & target)
{
regular = false;
}
}; };
void parseDump(ParseSink & sink, Source & source); void parseDump(ParseSink & sink, Source & source);

13
src/libutil/serialise.cc
View file

@ -322,5 +322,18 @@ void StringSink::operator () (const unsigned char * data, size_t len)
s->append((const char *) data, len); s->append((const char *) data, len);
} }
size_t ChainSource::read(unsigned char * data, size_t len)
{
if (useSecond) {
return source2.read(data, len);
} else {
try {
return source1.read(data, len);
} catch (EndOfFile &) {
useSecond = true;
return this->read(data, len);
}
}
}
} }

15
src/libutil/serialise.hh
View file

@ -189,7 +189,7 @@ struct TeeSource : Source
size_t read(unsigned char * data, size_t len) size_t read(unsigned char * data, size_t len)
{ {
size_t n = orig.read(data, len); size_t n = orig.read(data, len);
sink(data, len); sink(data, n);
return n; return n;
} }
}; };
@ -256,6 +256,19 @@ struct LambdaSource : Source
} }
}; };
/* Chain two sources together so after the first is exhausted, the second is
used */
struct ChainSource : Source
{
Source & source1, & source2;
bool useSecond = false;
ChainSource(Source & s1, Source & s2)
: source1(s1), source2(s2)
{ }
size_t read(unsigned char * data, size_t len) override;
};
/* Convert a function that feeds data into a Sink into a Source. The /* Convert a function that feeds data into a Sink into a Source. The
Source executes the function as a coroutine. */ Source executes the function as a coroutine. */

2
src/libutil/util.cc
View file

@ -1581,7 +1581,7 @@ AutoCloseFD createUnixDomainSocket(const Path & path, mode_t mode)
struct sockaddr_un addr; struct sockaddr_un addr;
addr.sun_family = AF_UNIX; addr.sun_family = AF_UNIX;
if (path.size() >= sizeof(addr.sun_path)) if (path.size() + 1 >= sizeof(addr.sun_path))
throw Error("socket path '%1%' is too long", path); throw Error("socket path '%1%' is too long", path);
strcpy(addr.sun_path, path.c_str()); strcpy(addr.sun_path, path.c_str());

31
src/nix-env/nix-env.cc
View file

@ -381,7 +381,8 @@ static void queryInstSources(EvalState & state,
if (path.isDerivation()) { if (path.isDerivation()) {
elem.setDrvPath(state.store->printStorePath(path)); elem.setDrvPath(state.store->printStorePath(path));
elem.setOutPath(state.store->printStorePath(state.store->derivationFromPath(path).findOutput("out"))); auto outputs = state.store->queryDerivationOutputMap(path);
elem.setOutPath(state.store->printStorePath(outputs.at("out")));
if (name.size() >= drvExtension.size() && if (name.size() >= drvExtension.size() &&
string(name, name.size() - drvExtension.size()) == drvExtension) string(name, name.size() - drvExtension.size()) == drvExtension)
name = string(name, 0, name.size() - drvExtension.size()); name = string(name, 0, name.size() - drvExtension.size());
@ -1208,18 +1209,17 @@ static void opSwitchProfile(Globals & globals, Strings opFlags, Strings opArgs)
} }
static const int prevGen = -2; static constexpr GenerationNumber prevGen = std::numeric_limits<GenerationNumber>::max();
static void switchGeneration(Globals & globals, int dstGen) static void switchGeneration(Globals & globals, GenerationNumber dstGen)
{ {
PathLocks lock; PathLocks lock;
lockProfile(lock, globals.profile); lockProfile(lock, globals.profile);
int curGen; auto [gens, curGen] = findGenerations(globals.profile);
Generations gens = findGenerations(globals.profile, curGen);
Generation dst; std::optional<Generation> dst;
for (auto & i : gens) for (auto & i : gens)
if ((dstGen == prevGen && i.number < curGen) || if ((dstGen == prevGen && i.number < curGen) ||
(dstGen >= 0 && i.number == dstGen)) (dstGen >= 0 && i.number == dstGen))
@ -1227,18 +1227,16 @@ static void switchGeneration(Globals & globals, int dstGen)
if (!dst) { if (!dst) {
if (dstGen == prevGen) if (dstGen == prevGen)
throw Error("no generation older than the current (%1%) exists", throw Error("no generation older than the current (%1%) exists", curGen.value_or(0));
curGen);
else else
throw Error("generation %1% does not exist", dstGen); throw Error("generation %1% does not exist", dstGen);
} }
printInfo(format("switching from generation %1% to %2%") printInfo("switching from generation %1% to %2%", curGen.value_or(0), dst->number);
% curGen % dst.number);
if (globals.dryRun) return; if (globals.dryRun) return;
switchLink(globals.profile, dst.path); switchLink(globals.profile, dst->path);
} }
@ -1249,7 +1247,7 @@ static void opSwitchGeneration(Globals & globals, Strings opFlags, Strings opArg
if (opArgs.size() != 1) if (opArgs.size() != 1)
throw UsageError("exactly one argument expected"); throw UsageError("exactly one argument expected");
int dstGen; GenerationNumber dstGen;
if (!string2Int(opArgs.front(), dstGen)) if (!string2Int(opArgs.front(), dstGen))
throw UsageError("expected a generation number"); throw UsageError("expected a generation number");
@ -1278,8 +1276,7 @@ static void opListGenerations(Globals & globals, Strings opFlags, Strings opArgs
PathLocks lock; PathLocks lock;
lockProfile(lock, globals.profile); lockProfile(lock, globals.profile);
int curGen; auto [gens, curGen] = findGenerations(globals.profile);
Generations gens = findGenerations(globals.profile, curGen);
RunPager pager; RunPager pager;
@ -1308,14 +1305,14 @@ static void opDeleteGenerations(Globals & globals, Strings opFlags, Strings opAr
if(opArgs.front().size() < 2) if(opArgs.front().size() < 2)
throw Error("invalid number of generations ‘%1%’", opArgs.front()); throw Error("invalid number of generations ‘%1%’", opArgs.front());
string str_max = string(opArgs.front(), 1, opArgs.front().size()); string str_max = string(opArgs.front(), 1, opArgs.front().size());
int max; GenerationNumber max;
if (!string2Int(str_max, max) || max == 0) if (!string2Int(str_max, max) || max == 0)
throw Error("invalid number of generations to keep ‘%1%’", opArgs.front()); throw Error("invalid number of generations to keep ‘%1%’", opArgs.front());
deleteGenerationsGreaterThan(globals.profile, max, globals.dryRun); deleteGenerationsGreaterThan(globals.profile, max, globals.dryRun);
} else { } else {
std::set<unsigned int> gens; std::set<GenerationNumber> gens;
for (auto & i : opArgs) { for (auto & i : opArgs) {
unsigned int n; GenerationNumber n;
if (!string2Int(i, n)) if (!string2Int(i, n))
throw UsageError("invalid generation number '%1%'", i); throw UsageError("invalid generation number '%1%'", i);
gens.insert(n); gens.insert(n);

6
src/nix/command.hh
View file

@ -244,4 +244,10 @@ void completeFlakeRefWithFragment(
const Strings & defaultFlakeAttrPaths, const Strings & defaultFlakeAttrPaths,
std::string_view prefix); std::string_view prefix);
void printClosureDiff(
ref<Store> store,
const StorePath & beforePath,
const StorePath & afterPath,
std::string_view indent);
} }

100
src/nix/diff-closures.cc
View file

@ -6,7 +6,7 @@
#include <regex> #include <regex>
using namespace nix; namespace nix {
struct Info struct Info
{ {
@ -52,6 +52,60 @@ std::string showVersions(const std::set<std::string> & versions)
return concatStringsSep(", ", versions2); return concatStringsSep(", ", versions2);
} }
void printClosureDiff(
ref<Store> store,
const StorePath & beforePath,
const StorePath & afterPath,
std::string_view indent)
{
auto beforeClosure = getClosureInfo(store, beforePath);
auto afterClosure = getClosureInfo(store, afterPath);
std::set<std::string> allNames;
for (auto & [name, _] : beforeClosure) allNames.insert(name);
for (auto & [name, _] : afterClosure) allNames.insert(name);
for (auto & name : allNames) {
auto & beforeVersions = beforeClosure[name];
auto & afterVersions = afterClosure[name];
auto totalSize = [&](const std::map<std::string, std::map<StorePath, Info>> & versions)
{
uint64_t sum = 0;
for (auto & [_, paths] : versions)
for (auto & [path, _] : paths)
sum += store->queryPathInfo(path)->narSize;
return sum;
};
auto beforeSize = totalSize(beforeVersions);
auto afterSize = totalSize(afterVersions);
auto sizeDelta = (int64_t) afterSize - (int64_t) beforeSize;
auto showDelta = abs(sizeDelta) >= 8 * 1024;
std::set<std::string> removed, unchanged;
for (auto & [version, _] : beforeVersions)
if (!afterVersions.count(version)) removed.insert(version); else unchanged.insert(version);
std::set<std::string> added;
for (auto & [version, _] : afterVersions)
if (!beforeVersions.count(version)) added.insert(version);
if (showDelta || !removed.empty() || !added.empty()) {
std::vector<std::string> items;
if (!removed.empty() || !added.empty())
items.push_back(fmt("%s → %s", showVersions(removed), showVersions(added)));
if (showDelta)
items.push_back(fmt("%s%+.1f KiB" ANSI_NORMAL, sizeDelta > 0 ? ANSI_RED : ANSI_GREEN, sizeDelta / 1024.0));
std::cout << fmt("%s%s: %s\n", indent, name, concatStringsSep(", ", items));
}
}
}
}
using namespace nix;
struct CmdDiffClosures : SourceExprCommand struct CmdDiffClosures : SourceExprCommand
{ {
std::string _before, _after; std::string _before, _after;
@ -85,49 +139,7 @@ struct CmdDiffClosures : SourceExprCommand
auto beforePath = toStorePath(store, Realise::Outputs, operateOn, before); auto beforePath = toStorePath(store, Realise::Outputs, operateOn, before);
auto after = parseInstallable(store, _after); auto after = parseInstallable(store, _after);
auto afterPath = toStorePath(store, Realise::Outputs, operateOn, after); auto afterPath = toStorePath(store, Realise::Outputs, operateOn, after);
printClosureDiff(store, beforePath, afterPath, "");
auto beforeClosure = getClosureInfo(store, beforePath);
auto afterClosure = getClosureInfo(store, afterPath);
std::set<std::string> allNames;
for (auto & [name, _] : beforeClosure) allNames.insert(name);
for (auto & [name, _] : afterClosure) allNames.insert(name);
for (auto & name : allNames) {
auto & beforeVersions = beforeClosure[name];
auto & afterVersions = afterClosure[name];
auto totalSize = [&](const std::map<std::string, std::map<StorePath, Info>> & versions)
{
uint64_t sum = 0;
for (auto & [_, paths] : versions)
for (auto & [path, _] : paths)
sum += store->queryPathInfo(path)->narSize;
return sum;
};
auto beforeSize = totalSize(beforeVersions);
auto afterSize = totalSize(afterVersions);
auto sizeDelta = (int64_t) afterSize - (int64_t) beforeSize;
auto showDelta = abs(sizeDelta) >= 8 * 1024;
std::set<std::string> removed, unchanged;
for (auto & [version, _] : beforeVersions)
if (!afterVersions.count(version)) removed.insert(version); else unchanged.insert(version);
std::set<std::string> added;
for (auto & [version, _] : afterVersions)
if (!beforeVersions.count(version)) added.insert(version);
if (showDelta || !removed.empty() || !added.empty()) {
std::vector<std::string> items;
if (!removed.empty() || !added.empty())
items.push_back(fmt("%s → %s", showVersions(removed), showVersions(added)));
if (showDelta)
items.push_back(fmt("%s%+.1f KiB" ANSI_NORMAL, sizeDelta > 0 ? ANSI_RED : ANSI_GREEN, sizeDelta / 1024.0));
std::cout << fmt("%s: %s\n", name, concatStringsSep(", ", items));
}
}
} }
}; };

1
src/nix/edit.cc
View file

@ -45,6 +45,7 @@ struct CmdEdit : InstallableCommand
auto args = editorFor(pos); auto args = editorFor(pos);
restoreSignals();
execvp(args.front().c_str(), stringsToCharPtrs(args).data()); execvp(args.front().c_str(), stringsToCharPtrs(args).data());
std::string command; std::string command;

43
src/nix/profile.cc
View file

@ -7,6 +7,7 @@
#include "builtins/buildenv.hh" #include "builtins/buildenv.hh"
#include "flake/flakeref.hh" #include "flake/flakeref.hh"
#include "../nix-env/user-env.hh" #include "../nix-env/user-env.hh"
#include "profiles.hh"
#include <nlohmann/json.hpp> #include <nlohmann/json.hpp>
#include <regex> #include <regex>
@ -394,6 +395,46 @@ struct CmdProfileInfo : virtual EvalCommand, virtual StoreCommand, MixDefaultPro
} }
}; };
struct CmdProfileDiffClosures : virtual StoreCommand, MixDefaultProfile
{
std::string description() override
{
return "show the closure difference between each generation of a profile";
}
Examples examples() override
{
return {
Example{
"To show what changed between each generation of the NixOS system profile:",
"nix profile diff-closure --profile /nix/var/nix/profiles/system"
},
};
}
void run(ref<Store> store) override
{
auto [gens, curGen] = findGenerations(*profile);
std::optional<Generation> prevGen;
bool first = true;
for (auto & gen : gens) {
if (prevGen) {
if (!first) std::cout << "\n";
first = false;
std::cout << fmt("Generation %d -> %d:\n", prevGen->number, gen.number);
printClosureDiff(store,
store->followLinksToStorePath(prevGen->path),
store->followLinksToStorePath(gen.path),
" ");
}
prevGen = gen;
}
}
};
struct CmdProfile : virtual MultiCommand, virtual Command struct CmdProfile : virtual MultiCommand, virtual Command
{ {
CmdProfile() CmdProfile()
@ -402,6 +443,7 @@ struct CmdProfile : virtual MultiCommand, virtual Command
{"remove", []() { return make_ref<CmdProfileRemove>(); }}, {"remove", []() { return make_ref<CmdProfileRemove>(); }},
{"upgrade", []() { return make_ref<CmdProfileUpgrade>(); }}, {"upgrade", []() { return make_ref<CmdProfileUpgrade>(); }},
{"info", []() { return make_ref<CmdProfileInfo>(); }}, {"info", []() { return make_ref<CmdProfileInfo>(); }},
{"diff-closures", []() { return make_ref<CmdProfileDiffClosures>(); }},
}) })
{ } { }
@ -425,4 +467,3 @@ struct CmdProfile : virtual MultiCommand, virtual Command
}; };
static auto r1 = registerCommand<CmdProfile>("profile"); static auto r1 = registerCommand<CmdProfile>("profile");

1
src/nix/registry.cc
View file

@ -111,6 +111,7 @@ struct CmdRegistryPin : virtual Args, EvalCommand
fetchers::Attrs extraAttrs; fetchers::Attrs extraAttrs;
if (ref.subdir != "") extraAttrs["dir"] = ref.subdir; if (ref.subdir != "") extraAttrs["dir"] = ref.subdir;
userRegistry->add(ref.input, resolved, extraAttrs); userRegistry->add(ref.input, resolved, extraAttrs);
userRegistry->write(fetchers::getUserRegistryPath());
} }
}; };

9
tests/flakes.sh
View file

@ -18,7 +18,6 @@ registry=$TEST_ROOT/registry.json
flake1Dir=$TEST_ROOT/flake1 flake1Dir=$TEST_ROOT/flake1
flake2Dir=$TEST_ROOT/flake2 flake2Dir=$TEST_ROOT/flake2
flake3Dir=$TEST_ROOT/flake3 flake3Dir=$TEST_ROOT/flake3
flake4Dir=$TEST_ROOT/flake4
flake5Dir=$TEST_ROOT/flake5 flake5Dir=$TEST_ROOT/flake5
flake6Dir=$TEST_ROOT/flake6 flake6Dir=$TEST_ROOT/flake6
flake7Dir=$TEST_ROOT/flake7 flake7Dir=$TEST_ROOT/flake7
@ -390,14 +389,12 @@ cat > $flake3Dir/flake.nix <<EOF
}; };
} }
EOF EOF
git -C $flake3Dir add flake.nix nix flake update $flake3Dir
git -C $flake3Dir add flake.nix flake.lock
git -C $flake3Dir commit -m 'Remove packages.xyzzy' git -C $flake3Dir commit -m 'Remove packages.xyzzy'
git -C $flake3Dir checkout master git -C $flake3Dir checkout master
# Test whether fuzzy-matching works for IsAlias # Test whether fuzzy-matching works for registry entries.
(! nix build -o $TEST_ROOT/result flake4/removeXyzzy#xyzzy)
# Test whether fuzzy-matching works for IsGit
(! nix build -o $TEST_ROOT/result flake4/removeXyzzy#xyzzy) (! nix build -o $TEST_ROOT/result flake4/removeXyzzy#xyzzy)
nix build -o $TEST_ROOT/result flake4/removeXyzzy#sth nix build -o $TEST_ROOT/result flake4/removeXyzzy#sth

20
tests/local-store.sh Normal file
View file

@ -0,0 +1,20 @@
source common.sh
cd $TEST_ROOT
echo example > example.txt
mkdir -p ./x
NIX_STORE_DIR=$TEST_ROOT/x
CORRECT_PATH=$(nix-store --store ./x --add example.txt)
PATH1=$(nix path-info --store ./x $CORRECT_PATH)
[ $CORRECT_PATH == $PATH1 ]
PATH2=$(nix path-info --store "$PWD/x" $CORRECT_PATH)
[ $CORRECT_PATH == $PATH2 ]
# FIXME we could also test the query parameter version:
# PATH3=$(nix path-info --store "local?store=$PWD/x" $CORRECT_PATH)
# [ $CORRECT_PATH == $PATH3 ]

2
tests/local.mk
View file

@ -6,7 +6,7 @@ nix_tests = \
gc-auto.sh \ gc-auto.sh \
referrers.sh user-envs.sh logging.sh nix-build.sh misc.sh fixed.sh \ referrers.sh user-envs.sh logging.sh nix-build.sh misc.sh fixed.sh \
gc-runtime.sh check-refs.sh filter-source.sh \ gc-runtime.sh check-refs.sh filter-source.sh \
remote-store.sh export.sh export-graph.sh \ local-store.sh remote-store.sh export.sh export-graph.sh \
timeout.sh secure-drv-outputs.sh nix-channel.sh \ timeout.sh secure-drv-outputs.sh nix-channel.sh \
multiple-outputs.sh import-derivation.sh fetchurl.sh optimise-store.sh \ multiple-outputs.sh import-derivation.sh fetchurl.sh optimise-store.sh \
binary-cache.sh nix-profile.sh repair.sh dump-db.sh case-hack.sh \ binary-cache.sh nix-profile.sh repair.sh dump-db.sh case-hack.sh \