Commit graph

75 commits

Author SHA1 Message Date
Eelco Dolstra ed6c646f44
Doh 2018-08-06 17:27:08 +02:00
Eelco Dolstra 2825e05d21
Make adding paths via nix-store --serve run in constant memory
It adds a new operation, cmdAddToStoreNar, that does the same thing as
the corresponding nix-daemon operation, i.e. call addToStore(). This
replaces cmdImportPaths, which has the major issue that it sends the
NAR first and the store path second, thus requiring us to store the
incoming NAR either in memory or on disk until we decide what to do
with it.

For example, this reduces the memory usage of

  $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79

from 267 MiB to 12 MiB.

Probably fixes #1988.
2018-08-03 21:21:16 +02:00
Eelco Dolstra e268bbc054
LegacySSHStore: Add remote-store option
This is primarily useful for testing, e.g.

  $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' ...
2018-08-03 21:20:38 +02:00
Eelco Dolstra 81ea8bd5ce
Simplify the callback mechanism 2018-05-30 13:34:37 +02:00
Eelco Dolstra 11898d6a37 Make 'nix copy --to ssh://...' run in constant memory 2018-03-21 23:13:06 +01:00
Eelco Dolstra 47f7e5585b Make 'nix copy --from ssh://...' run in constant memory
For instance, this reduced the memory consumption of

  $ nix copy --from ssh://localhost --to ~/my-nix /nix/store/1n7x0yv8vq6zi90hfmian84vdhd04bgp-blender-2.79a

from 632 MiB to 16 MiB.
2018-03-21 23:10:40 +01:00
Eelco Dolstra 8615cfb130
LegacySSHStore: Allow overriding the path to nix-store 2018-03-21 15:10:46 +01:00
Eelco Dolstra 3ed8290e53
LegacySSHStore: Include signatures etc. 2017-09-08 16:55:27 +02:00
Eelco Dolstra c5e4404580
nix copy: Revive progress bar 2017-08-16 20:56:02 +02:00
Jörg Thalheim 2fd8f8bb99 Replace Unicode quotes in user-facing strings by ASCII
Relevant RFC: NixOS/rfcs#4

$ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
2017-07-30 12:32:45 +01:00
Eelco Dolstra fcca702a96
Replace a few bool flags with enums
Functions like copyClosure() had 3 bool arguments, which creates a
severe risk of mixing up arguments.

Also, implement copyClosure() using copyPaths().
2017-07-03 11:38:08 +02:00
Eelco Dolstra cd4d2705ec
build-remote: Fix fallback to other machines when connecting fails
Opening an SSHStore or LegacySSHStore does not actually establish a
connection, so the try/catch block here did nothing. Added a
Store::connect() method to test whether a connection can be
established.
2017-05-02 15:46:09 +02:00
Eelco Dolstra feefcb3a98
build-remote: Ugly hackery to get build logs to work
The build hook mechanism expects build log output to go to file
descriptor 4, so do that.
2017-05-02 12:02:23 +02:00
Eelco Dolstra 3a5f04f48c
build-remote: Don't require signatures
This restores the old behaviour.
2017-05-01 20:03:25 +02:00
Eelco Dolstra deac171925
Implement LegacySSHStore::buildDerivation()
This makes LegacySSHStore usable by build-remote and
hydra-queue-runner.
2017-05-01 17:30:16 +02:00
Eelco Dolstra 0dddcf867a
Add a dummy Store::buildPaths() method
This default implementation of buildPaths() does nothing if all
requested paths are already valid, and throws an "unsupported
operation" error otherwise. This fixes a regression introduced by
c30330df6f in binary cache and legacy
SSH stores.
2017-05-01 13:43:34 +02:00
Eelco Dolstra 2040240e23
Add a Config class to simplify adding configuration settings
The typical use is to inherit Config and add Setting<T> members:

  class MyClass : private Config
  {
    Setting<int> foo{this, 123, "foo", "the number of foos to use"};
    Setting<std::string> bar{this, "blabla", "bar", "the name of the bar"};

    MyClass() : Config(readConfigFile("/etc/my-app.conf"))
    {
      std::cout << foo << "\n"; // will print 123 unless overriden
    }
  };

Currently, this is used by Store and its subclasses for store
parameters. You now get a warning if you specify a non-existant store
parameter in a store URI.
2017-04-13 16:03:31 +02:00
Eelco Dolstra 287084d688
ssh:// -> ssh-ng://, legacy-ssh:// -> ssh:// 2017-03-16 14:19:32 +01:00
Eelco Dolstra c5b83d8913
copyPaths(): Use queryValidPaths() to reduce SSH latency 2017-03-16 13:50:01 +01:00
Eelco Dolstra ea7fa88131
LegacySSHStore: Provide a faster implementation of computeFSClosure()
This avoids the latency of the standard implementation, which can make
a huge difference (e.g. 16.5s -> 0.5s on a NixOS system closure).
2017-03-16 11:44:52 +01:00
Eelco Dolstra 7a716ef2a5
Fix nix-copy-closure --to 2017-03-16 10:58:48 +01:00
Eelco Dolstra 577ebeaefb
Improve SSH handling
* Unify SSH code in SSHStore and LegacySSHStore.

* Fix a race starting the SSH master. We now wait synchronously for
  the SSH master to finish starting. This prevents the SSH clients
  from starting their own connections.

* Don't use a master if max-connections == 1.

* Add a "max-connections" store parameter.

* Add a "compress" store parameter.
2017-03-03 19:05:50 +01:00
Eelco Dolstra f61f67ddee
RemoteStore::addToStore(): Send NAR rather than string containing NAR
This allows the NAR to be streamed in the future (though we're not
doing that yet).
2017-03-01 16:07:15 +01:00
Eelco Dolstra 99bbddedb1
Fix building without S3 support
http://hydra.nixos.org/build/49031196/nixlog/2/raw
2017-02-21 13:15:07 +01:00
Eelco Dolstra caa5793b4a
Add a LegacySSHStore that uses nix-store --serve
This is useful for nix-copy-closure.
2017-02-07 19:29:21 +01:00