Commit graph

23 commits

Author SHA1 Message Date
John Ericson 68dfb8c6ae Optimize addToStoreSlow and remove TeeParseSink 2020-07-16 05:09:41 +00:00
Eelco Dolstra 0a9da00a10 NarAccessor: Run in constant memory 2020-07-13 17:30:42 +02:00
Eelco Dolstra fc84c358d9 Make 'nix copy' to file:// binary caches run in constant memory 2020-07-13 16:28:45 +02:00
Eelco Dolstra f1c0b2c0e1
Add O(1)-memory copyPath() function 2019-10-27 18:18:58 +01:00
Eelco Dolstra 737ed88f35
Modularize config settings
Allow global config settings to be defined in multiple Config
classes. For example, this means that libutil can have settings and
evaluator settings can be moved out of libstore. The Config classes
are registered in a new GlobalConfig class to which config files
etc. are applied.

Relevant to https://github.com/NixOS/nix/issues/2009 in that it
removes the need for ad hoc handling of useCaseHack, which was the
underlying cause of that issue.
2018-05-30 13:28:01 +02:00
Eelco Dolstra 47f7e5585b Make 'nix copy --from ssh://...' run in constant memory
For instance, this reduced the memory consumption of

  $ nix copy --from ssh://localhost --to ~/my-nix /nix/store/1n7x0yv8vq6zi90hfmian84vdhd04bgp-blender-2.79a

from 632 MiB to 16 MiB.
2018-03-21 23:10:40 +01:00
Eelco Dolstra 72cd52c3cd
builtins.fetchgit: Support importing a working tree
For example, you can write

  src = fetchgit ./.;

and if ./. refers to an unclean working tree, that tree will be copied
to the Nix store. This removes the need for "cleanSource".
2017-10-30 19:59:25 +01:00
Eelco Dolstra 1cf4801108
Revert fa125b9b28
This causes quadratic performance.
2017-03-06 13:56:20 +01:00
Eelco Dolstra fa125b9b28
TeeSink: Pre-reserve string space
When receiving a very large file, this can prevent the string from
having tobe copied, which temporarily doubles memory consumption.
2017-03-01 16:16:04 +01:00
Eelco Dolstra 1042c10fd0 Add NAR / Store accessor abstraction
This is primary to allow hydra-queue-runner to extract files like
"nix-support/hydra-build-products" from NARs in binary caches.
2016-02-25 17:43:19 +01:00
Eelco Dolstra 9ccbd55c5b BinaryCacheStore: Implement addToStore()
So now you can do

  $ NIX_REMOTE=file:///tmp/binary-cache nix-instantiate '<nixpkgs>' -A hello

and lots of other operations.
2016-02-24 16:52:28 +01:00
Eelco Dolstra 276a40b31f Handle case collisions on case-insensitive systems
When running NixOps under Mac OS X, we need to be able to import store
paths built on Linux into the local Nix store. However, HFS+ is
usually case-insensitive, so if there are directories with file names
that differ only in case, then importing will fail.

The solution is to add a suffix ("~nix~case~hack~<integer>") to
colliding files. For instance, if we have a directory containing
xt_CONNMARK.h and xt_connmark.h, then the latter will be renamed to
"xt_connmark.h~nix~case~hack~1". If a store path is dumped as a NAR,
the suffixes are removed. Thus, importing and exporting via a
case-insensitive Nix store is round-tripping. So when NixOps calls
nix-copy-closure to copy the path to a Linux machine, you get the
original file names back.

Closes #119.
2014-07-16 16:02:05 +02:00
Eelco Dolstra b7fd2c2822 Use "#pragma once" to prevent repeated header file inclusion 2012-07-18 14:59:03 -04:00
Eelco Dolstra a0e3b84fac * Revert r15436. This was a workaround for a bug in btrfs which seems
to have been fixed now.
2010-05-04 12:42:58 +00:00
Michael Raskin 098cb9d233 Add an ftruncate call paired with fallocate to play safe with some FSes (namely, BtrFS fallocate sets file size to allocated size, i.e. multiple of block size) 2009-05-04 08:10:24 +00:00
Eelco Dolstra 77d272623f * NAR archives: handle files larger than 2^32 bytes. Previously it
would just silently store only (fileSize % 2^32) bytes.
* Use posix_fallocate if available when unpacking archives.
* Provide a better error message when trying to unpack something that
  isn't a NAR archive.
2009-03-22 17:36:43 +00:00
Eelco Dolstra 5eaf644c99 * A simple API for parsing NAR archives. 2008-12-03 17:30:32 +00:00
Eelco Dolstra a3e6415ba8 * New primop builtins.filterSource, which can be used to filter files
from a source directory.  All files for which a predicate function
  returns true are copied to the store.  Typical example is to leave
  out the .svn directory:

    stdenv.mkDerivation {
      ...
      src = builtins.filterSource
        (path: baseNameOf (toString path) != ".svn")
        ./source-dir;
      # as opposed to
      #   src = ./source-dir;
    }

  This is important because the .svn directory influences the hash in
  a rather unpredictable and variable way.
2006-12-12 23:05:01 +00:00
Eelco Dolstra b438d37558 * In dumpPath(): pass a function object that allows files to be
selectively in/excluded from the dump.
2006-12-12 21:51:02 +00:00
Eelco Dolstra 40b3f64b55 * Skeleton of the privileged worker program.
* Some refactoring: put the NAR archive integer/string serialisation
  code in a separate file so it can be reused by the worker protocol
  implementation.
2006-11-30 19:19:59 +00:00
Eelco Dolstra 75068e7d75 * Use a proper namespace.
* Optimise header file usage a bit.
* Compile the parser as C++.
2006-09-04 21:06:23 +00:00
Eelco Dolstra 4bbdcfbb45 * Don't use fork() in copyPath(), but a string buffer. 2005-03-03 13:58:02 +00:00
Eelco Dolstra 8798fae304 * Source tree refactoring. 2003-11-18 10:47:59 +00:00
Renamed from src/libnix/archive.hh (Browse further)