Commit graph

131 commits

Author SHA1 Message Date
Guillaume Maudoux 5c56570726 Also obfuscate the number of memory roots 2019-03-10 00:56:09 +01:00
Guillaume Maudoux ebc86550f9 Make roots a map of store paths to pinning links
This new structure makes more sense as there may be many sources rooting
the same store path. Many profiles can reference the same path but this
is even more true with /proc/<pid>/maps where distinct pids can and
often do map the same store path.
This implementation is also more efficient as the `Roots` map contains
only one entry per rooted store path.
2019-03-10 00:56:09 +01:00
Guillaume Maudoux a17f86ce3a Obfuscate memory roots for non-root users 2019-03-10 00:56:09 +01:00
Eelco Dolstra f6a3dfe4e0
Merge all nix-* binaries into nix
These are all symlinks to 'nix' now, reducing the installed size by
about ~1.7 MiB.
2018-10-26 12:54:00 +02:00
Eelco Dolstra 0fda9b22c7
Remove unnecessary typecast 2018-10-04 13:16:30 +02:00
Will Dietz 8df367a92d nix-daemon: allow setting builders to "" by any user (untrusted)
(cherry picked from commit a94a2eb1cb1c81e90a7529be5fecac27899a3442)
2018-09-25 03:22:08 -05:00
Eelco Dolstra 2825e05d21
Make adding paths via nix-store --serve run in constant memory
It adds a new operation, cmdAddToStoreNar, that does the same thing as
the corresponding nix-daemon operation, i.e. call addToStore(). This
replaces cmdImportPaths, which has the major issue that it sends the
NAR first and the store path second, thus requiring us to store the
incoming NAR either in memory or on disk until we decide what to do
with it.

For example, this reduces the memory usage of

  $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79

from 267 MiB to 12 MiB.

Probably fixes #1988.
2018-08-03 21:21:16 +02:00
Eelco Dolstra e388739098
nix-daemon: Respect --store
For example, this allows you to do run nix-daemon as a non-privileged
user:

  eelco$ NIX_STATE_DIR=~/my-nix/nix/var nix-daemon --store ~/my-nix/

The NIX_STATE_DIR is still needed because settings.nixDaemonSocketFile
is not derived from settings.storeUri (and we can't derive it from the
store's state directory because we don't want to open the store in the
parent process).
2018-07-03 19:52:32 +02:00
Eelco Dolstra c1d445ecec
Move some Download-specific settings to download.cc 2018-05-30 13:29:50 +02:00
Eelco Dolstra 6185d25e52
Make 'nix copy --to daemon' run in constant memory (daemon side)
Continuation of 97002b684c. This makes
the daemon use constant memory. For example, it reduces the daemon's
maximum RSS on

  $ nix copy --from ~/my-nix --to daemon /nix/store/1n7x0yv8vq6zi90hfmian84vdhd04bgp-blender-2.79a

from 264 MiB to 7 MiB.

We now use a TunnelSource to prevent the connection from ending up in
an undefined state if an exception is thrown while the NAR is being
sent.

Issue https://github.com/NixOS/nix/issues/1681.
2018-05-30 13:26:48 +02:00
Will Dietz 14c464b6c8 nix-daemon: remove unused "pendingMsgs" variable 2018-05-26 13:09:55 -05:00
Will Dietz b08923b4a0 serialise: fix buffer size used, hide method for internal use only
Fixes #2169.
2018-05-21 17:30:38 -05:00
Shea Levy 346c0ac361
nix-daemon: Exit successfully when interrupted.
Fixes #2058.
2018-04-09 08:05:54 -04:00
Eelco Dolstra f87e286e82
Merge branch 'fix/avoid-large-stack-buffers' of https://github.com/dtzWill/nix 2018-03-22 13:19:25 +01:00
Eelco Dolstra 48662d151b
Reduce substitution memory consumption
copyStorePath() now pipes the output of srcStore->narFromPath()
directly into dstStore->addToStore(). The sink used by the former is
converted into a source usable by the latter using
boost::coroutine2. This is based on [1].

This reduces the maximum resident size of

  $ nix build --store ~/my-nix/ /nix/store/b0zlxla7dmy1iwc3g459rjznx59797xy-binutils-2.28.1 --substituters file:///tmp/binary-cache-xz/ --no-require-sigs

from 418592 KiB to 53416 KiB. (The previous commit also reduced the
runtime from ~4.2s to ~3.4s, not sure why.) A further improvement will
be to download files into a Sink.

[1] https://github.com/NixOS/nix/compare/master...Mathnerd314:dump-fix-coroutine#diff-dcbcac55a634031f9cc73707da6e4b18

Issue #1969.
2018-03-16 20:35:59 +01:00
Will Dietz d98755b0c2 nix-daemon: preserve errno in signal handler (thanks tsan) 2018-03-14 21:39:46 -05:00
Will Dietz c89a3d5368 don't allocate large buffers on the stack 2018-03-02 10:52:04 -06:00
Shea Levy 88cd2d41ac
Add plugins to make Nix more extensible.
All plugins in plugin-files will be dlopened, allowing them to
statically construct instances of the various Register* types Nix
supports.
2018-02-08 12:44:37 -05:00
Will Dietz f201b7733e More completely fix recursive nix, unbreak tests
See:
88b5d0c8e8 (commitcomment-27406365)
2018-02-07 15:19:28 -06:00
Eelco Dolstra 88b5d0c8e8
Prevent accidental recursive Nix 2018-02-07 19:07:38 +01:00
Eelco Dolstra 7b9583680e
Improve error message with --repair for untrusted users 2018-01-08 19:13:48 +01:00
Eelco Dolstra 25f32625e2
Remove the remote-builds option
This is superfluous since you can now just set "builders" to empty,
e.g. "--builders ''".
2017-10-24 11:00:16 +02:00
Eelco Dolstra 2cc345b95f
Give activities a verbosity level again
And print them (separately from the progress bar) given sufficient -v
flags.
2017-08-28 19:13:24 +02:00
Eelco Dolstra cfc8132391
Don't send progress messages to older clients 2017-08-28 18:54:23 +02:00
Eelco Dolstra fe34b91289
Tunnel progress messages from the daemon to the client
This makes the progress bar work for non-root users.
2017-08-28 18:49:42 +02:00
Eelco Dolstra 8fff3e7bb5
Make TunnelLogger thread-safe
Now that we use threads in lots of places, it's possible for
TunnelLogger::log() to be called asynchronously from other threads
than the main loop. So we need to ensure that STDERR_NEXT messages
don't clobber other messages.
2017-08-28 14:17:07 +02:00
Eelco Dolstra 40bffe0a43
Progress indicator: Cleanup 2017-08-16 20:56:03 +02:00
Jörg Thalheim 2fd8f8bb99 Replace Unicode quotes in user-facing strings by ASCII
Relevant RFC: NixOS/rfcs#4

$ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
2017-07-30 12:32:45 +01:00
Eelco Dolstra c0015e87af
Support base-64 hashes
Also simplify the Hash API.

Fixes #1437.
2017-07-04 15:07:41 +02:00
Eelco Dolstra fcca702a96
Replace a few bool flags with enums
Functions like copyClosure() had 3 bool arguments, which creates a
severe risk of mixing up arguments.

Also, implement copyClosure() using copyPaths().
2017-07-03 11:38:08 +02:00
Eelco Dolstra b01d62285c
Improve progress indicator 2017-05-16 16:09:57 +02:00
Eelco Dolstra 1a8e15053a
Don't allow untrusted users to set info.ultimate
Note that a trusted signature was still required in this case so it
was not a huge deal.
2017-05-11 13:58:09 +02:00
Eelco Dolstra 895f00c372
Suppress warning about ssh-auth-sock 2017-04-28 16:55:52 +02:00
Eelco Dolstra 4410e9d995
Setting: Remove "Tag" template argument 2017-04-20 16:52:53 +02:00
Eelco Dolstra 76cb3c702c
Reimplement trusted-substituters (aka trusted-binary-caches) 2017-04-20 13:41:29 +02:00
Eelco Dolstra 3872371f25
Minor cleanup 2017-04-14 13:42:22 +02:00
Eelco Dolstra 01dcdfcf33
nix-daemon: Don't set untrusted-* settings
These are no longer used anywhere.
2017-04-14 11:59:51 +02:00
Eelco Dolstra 1673c373c9
nix-daemon: Don't die if the user sends an unknown setting 2017-04-14 11:57:02 +02:00
Eelco Dolstra ba9ad29fdb
Convert Settings to the new config system
This makes all config options self-documenting.

Unknown or unparseable config settings and --option flags now cause a
warning.
2017-04-13 20:53:23 +02:00
Eelco Dolstra 1860070548
Merge branch 'rework-options' of https://github.com/copumpkin/nix 2017-04-13 16:15:51 +02:00
Eelco Dolstra ba20730b3f
Implement RemoteStore::queryMissing()
This provides a significant speedup, e.g. 64 s -> 12 s for

  nix-build --dry-run -I nixpkgs=channel:nixos-16.03 '<nixpkgs/nixos/tests/misc.nix>' -A test

on a cold local and CloudFront cache.

The alternative is to use lots of concurrent daemon connections but
that seems wasteful.
2017-04-06 18:40:19 +02:00
Eelco Dolstra 256940fc48
nix-daemon: Disable path info cache
This is useless because the client also caches path info, and can
cause problems for long-running clients like hydra-queue-runner
(i.e. it may return cached info about paths that have been
garbage-collected).
2017-04-06 14:30:31 +02:00
Eelco Dolstra fa125b9b28
TeeSink: Pre-reserve string space
When receiving a very large file, this can prevent the string from
having tobe copied, which temporarily doubles memory consumption.
2017-03-01 16:16:04 +01:00
Eelco Dolstra f61f67ddee
RemoteStore::addToStore(): Send NAR rather than string containing NAR
This allows the NAR to be streamed in the future (though we're not
doing that yet).
2017-03-01 16:07:15 +01:00
Eelco Dolstra c4a40949d9
Handle importing NARs containing files greater than 4 GiB
Also templatize readInt() to work for various integer types.
2017-03-01 13:52:54 +01:00
Dan Peebles e7cb2847ab Explicitly model all settings and fail on unrecognized ones
Previously, the Settings class allowed other code to query for string
properties, which led to a proliferation of code all over the place making
up new options without any sort of central registry of valid options. This
commit pulls all those options back into the central Settings class and
removes the public get() methods, to discourage future abuses like that.

Furthermore, because we know the full set of options ahead of time, we
now fail loudly if someone enters an unrecognized option, thus preventing
subtle typos. With some template fun, we could probably also dump the full
set of options (with documentation, defaults, etc.) to the command line,
but I'm not doing that yet here.
2017-02-22 20:19:25 -05:00
Eelco Dolstra f023f64f40
RemoteStore::addToStore(): Pass content-addressability assertion
... and use this in Downloader::downloadCached(). This fixes

  $ nix-build https://nixos.org/channels/nixos-16.09-small/nixexprs.tar.xz -A hello
  error: cannot import path ‘/nix/store/csfbp1s60dkgmk9f8g0zk0mwb7hzgabd-nixexprs.tar.xz’ because it lacks a valid signature
2017-02-22 16:58:00 +01:00
Tuomas Tynkkynen 649a81bcd6 nix-daemon: Don't splice with len=SIZE_MAX
Currently, 'nix-daemon --stdio' is always failing for me, due to the
splice call always failing with (on a 32-bit host):

splice(0, NULL, 3, NULL, 4294967295, SPLICE_F_MOVE) = -1 EINVAL (Invalid argument)

With a bit of ftracing (and luck) the problem seems to be that splice()
always fails with EINVAL if the len cast as ssize_t is negative:
http://lxr.free-electrons.com/source/fs/read_write.c?v=4.4#L384

So use SSIZE_MAX instead of SIZE_MAX.
2017-02-13 15:14:44 +02:00
Eelco Dolstra ddb5577f2e
Move SavingSourceAdapter to serialise.hh 2017-02-07 19:29:20 +01:00
Eelco Dolstra 215b70f51e
Revert "Get rid of unicode quotes (#1140)"
This reverts commit f78126bfd6. There
really is no need for such a massive change...
2016-11-26 00:38:01 +01:00