philiptaron/lix
0
0
Fork 0
forked from lix-project/lix
Code Pull requests Activity
6338 commits 26 branches 4 tags 76 MiB
Commit graph

3710 commits

Author SHA1 Message Date
Will Dietz 54b1c59643 download.cc: fix readCallback return type (int -> size_t) 
I'm not sure if curl ever asks for enough data at once
for truncation to occur but better safe than sorry.
 2018-06-01 10:59:26 -05:00
Eelco Dolstra 73357500ac
Improve binary cache upload messages 
Don't say "download" when we mean "upload".
 2018-06-01 14:14:22 +02:00
Peter Simons 93aa3bea2e
Merge pull request #767 from mogorman/garbage_collect_keep_last_few 
Implement --delete-generations + flag for keeping last N number of gens
 2018-05-31 10:00:21 +02:00
Eelco Dolstra 5d4a9d5677
Fix static assertion failure on 32-bit systems 2018-05-30 17:47:30 +02:00
Eelco Dolstra 3c2de9830d
RemoteStore::addToStore(): Ensure capacity is decreased on exceptions 2018-05-30 13:48:31 +02:00
Eelco Dolstra a4c1618876
Fix deadlock in RemoteStore::queryPathInfoUncached() 2018-05-30 13:46:14 +02:00
Eelco Dolstra 7d21863bb3
Make <nix/fetchurl.nix> run in constant memory 
E.g.

  nix-build --store ~/my-nix/ -E 'import <nix/fetchurl.nix> { url = https://cache.nixos.org/nar/0nwi996rgq4b914qyx0mv2wq4k80hjac7xilikavagw7kxmn2iiv.nar.xz; sha256 = "0nwi996rgq4b914qyx0mv2wq4k80hjac7xilikavagw7kxmn2iiv"; }'

now runs in 17 MiB (was 70 MiB), while

  nix-build --store ~/my-nix/ -E 'import <nix/fetchurl.nix> { url = https://cache.nixos.org/nar/0nwi996rgq4b914qyx0mv2wq4k80hjac7xilikavagw7kxmn2iiv.nar.xz; sha256 = "0d2fxljdih3nc5dqx41hjzic3141ajil94m8kdbpryq569dpsbvb"; unpack = true; }'

runs in 17 MiB (was 346 MiB).
 2018-05-30 13:42:51 +02:00
Eelco Dolstra 5a654fd7dd
getDownloader(): Simplify 2018-05-30 13:42:30 +02:00
Eelco Dolstra e87e4a60d6
Make HttpBinaryCacheStore::narFromPath() run in constant memory 
This reduces memory consumption of

  nix copy --from https://cache.nixos.org --to ~/my-nix /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79

from 176 MiB to 82 MiB. (The remaining memory is probably due to xz
decompression overhead.)

Issue https://github.com/NixOS/nix/issues/1681.
Issue https://github.com/NixOS/nix/issues/1969.
 2018-05-30 13:42:29 +02:00
Eelco Dolstra 08ec757726
Make LocalBinaryCacheStore::narFromPath() run in constant memory 
This reduces memory consumption of

  nix copy --from file://... --to ~/my-nix /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79

from 514 MiB to 18 MiB for an uncompressed binary cache, and from 192
MiB to 53 MiB for a bzipped binary cache. It may also be faster
because fetching can happen concurrently with decompression/writing.

Continuation of 48662d151b.

Issue https://github.com/NixOS/nix/issues/1681.
 2018-05-30 13:34:57 +02:00
Eelco Dolstra 81ea8bd5ce
Simplify the callback mechanism 2018-05-30 13:34:37 +02:00
Eelco Dolstra 1672bcd230
Move evaluator-specific settings out of libstore 2018-05-30 13:29:50 +02:00
Eelco Dolstra c1d445ecec
Move some Download-specific settings to download.cc 2018-05-30 13:29:50 +02:00
Eelco Dolstra 737ed88f35
Modularize config settings 
Allow global config settings to be defined in multiple Config
classes. For example, this means that libutil can have settings and
evaluator settings can be moved out of libstore. The Config classes
are registered in a new GlobalConfig class to which config files
etc. are applied.

Relevant to https://github.com/NixOS/nix/issues/2009 in that it
removes the need for ad hoc handling of useCaseHack, which was the
underlying cause of that issue.
 2018-05-30 13:28:01 +02:00
Eelco Dolstra e606cd412f
Fix assertion failure in storePathToHash() 
Fixes https://github.com/NixOS/nix/issues/2015.
 2018-05-30 13:27:19 +02:00
Eelco Dolstra 6185d25e52
Make 'nix copy --to daemon' run in constant memory (daemon side) 
Continuation of 97002b684c. This makes
the daemon use constant memory. For example, it reduces the daemon's
maximum RSS on

  $ nix copy --from ~/my-nix --to daemon /nix/store/1n7x0yv8vq6zi90hfmian84vdhd04bgp-blender-2.79a

from 264 MiB to 7 MiB.

We now use a TunnelSource to prevent the connection from ending up in
an undefined state if an exception is thrown while the NAR is being
sent.

Issue https://github.com/NixOS/nix/issues/1681.
 2018-05-30 13:26:48 +02:00
Eelco Dolstra 23d6bb583a
LocalStore::addToStore: Throw an exception if no hash is given 2018-05-30 13:25:40 +02:00
Eelco Dolstra 3cab639e21
Use extended SQLite error codes 2018-05-30 13:25:03 +02:00
Eelco Dolstra bd56b5fe3f
Merge pull request #2187 from bgamari/stoi-exceptions 
json-to-value: Use strtol instead of strtoi
 2018-05-28 14:54:49 +02:00
Ben Gamari 7f560b81ea json-to-value: Use strtol instead of strtoi 
NixInt is long, so strtoi is too restrictive.
 2018-05-26 18:43:46 -04:00
Will Dietz 93ae90de0f eval.cc: add message to static_assert, message can be omitted w/c++17 2018-05-26 14:04:41 -05:00
Will Dietz 14c464b6c8 nix-daemon: remove unused "pendingMsgs" variable 2018-05-26 13:09:55 -05:00
Will Dietz ca9c6cb95d libstore/download: remove unused function "getTime()" 2018-05-25 19:49:45 -05:00
Eelco Dolstra 743359bc8a
Merge pull request #2157 from volth/bitwise 
add builtins: __bitAnd,  __bitOr,  __bitXor
 2018-05-24 15:00:39 +02:00
Eelco Dolstra 4bb8741b98
Make Env self-describing 
If the Env denotes a 'with', then values[0] may be an Expr* cast to a
Value*. For code that generically traverses Values/Envs, it's useful
to know this.
 2018-05-22 16:02:32 +02:00
Eelco Dolstra 9fd7cf98db
Memoise checkSourcePath() 
This prevents hydra-eval-jobs from statting the same files over and
over again.
 2018-05-22 13:02:14 +02:00
Will Dietz b08923b4a0 serialise: fix buffer size used, hide method for internal use only 
Fixes #2169.
 2018-05-21 17:30:38 -05:00
volth 6cc28c0589
add `mod' and bitwise builtins: camel-case function names 2018-05-16 10:52:19 +00:00
volth f3c090f91c
add mod' and bitwise builtins: remove mod' and shifts 2018-05-16 06:57:11 +00:00
volth 49b7cf1813
add `mod' and bitwise builtins: remove infix functions 2018-05-16 06:55:24 +00:00
Eelco Dolstra 966407bcf1
Merge pull request #2145 from serokell/uncached-channel 
make sure not to use cached channels for nix-channel --update
 2018-05-15 14:57:18 +02:00
volth 8a6a14e1f5 add `mod' and bitwise builtins 2018-05-12 16:50:39 +00:00
Eelco Dolstra 1ad19232c4
Don't return negative numbers from the flex tokenizer 
Fixes #1374.
Closes #2129.
 2018-05-11 12:05:12 +02:00
Eelco Dolstra f3c85f9eb3
Revert "Throw a specific error for incomplete parse errors." 
This reverts commit 6498adb002. We don't
actually use IncompleteParseError in 'nix repl'.
 2018-05-11 11:40:50 +02:00
Eelco Dolstra 2b3c1b3a88
Remove mention of default since we already know the default 2018-05-10 12:06:28 +02:00
Eelco Dolstra 38def17627
Merge branch 's3-multipart-uploads' of https://github.com/AmineChikhaoui/nix 2018-05-10 12:06:02 +02:00
Yorick b9289e4855
make sure not to use cached channels for nix-channel --update 
fixes #1964
 2018-05-09 16:18:20 +02:00
Eelco Dolstra a91c4ca01f
In restricted eval mode, allow access to the closure of store paths 
E.g. this makes

  nix eval --restrict-eval -I /nix/store/foo '(builtins.readFile "/nix/store/foo/symlink/bla")'

(where /nix/store/foo/symlink is a symlink to another path in the
closure of /nix/store/foo) succeed.

This fixes a regression in Hydra compared to Nix 1.x (where there were
no restrictions at all on access to the Nix store).
 2018-05-09 15:45:05 +02:00
AmineChikhaoui 854c0860f4
share the executor between multiple copy threads. 2018-05-07 15:07:00 +01:00
AmineChikhaoui 591e75cd01
add a FIXME note to find a way to abort the multipart uploads in 
case the nix command is interrupted.
 2018-05-07 14:27:53 +01:00
AmineChikhaoui 129394fb95
Support multi-part uploads for large NARs that exceed the size of 
5Gb.
 2018-05-07 14:23:51 +01:00
Eelco Dolstra a1adcdf087
nix-shell: Only download requested outputs 
Fixes #2116.
 2018-05-07 14:58:25 +02:00
Eelco Dolstra c717d8e3bd
Don't set a CA assertion for paths with references 
Really fixes #2133.
 2018-05-03 14:49:56 +02:00
Eelco Dolstra 80a7b16593
LocalStore::addValidPath(): Throw error when the CA assertion is wrong 
Closes #2133.
 2018-05-03 11:02:40 +02:00
Eelco Dolstra 53ec5ac69f
Fix some random -Wconversion warnings 2018-05-02 13:56:34 +02:00
Eelco Dolstra 4a2c948943
Fix bzip2 compression of files > 4 GiB 
Bzip2's 'avail_in' parameter is declared as an unsigned int, so
assigning a size_t length to it led to silent truncation.

Fixes #2111.
 2018-05-02 12:54:30 +02:00
Will Dietz 8e7d77d494 globals.hh: don't use '==' to compare string literals 
Saw this in logs, also reported here:

8e6108ff71 (r28707288)
 2018-04-23 20:57:05 -05:00
Shea Levy 8e6108ff71
Merge branch 'aarch64-armv7' of git://github.com/lheckemann/nix 
Support extra compatible architectures (#1916)
 2018-04-23 08:48:22 -04:00
Shea Levy e2b028353b
Merge branch 'pos-crash-fix' of git://github.com/dezgeg/nix 2018-04-23 08:43:54 -04:00
Shea Levy 58f9ae0dcd
Merge branch 'hash-mismatch' of git://github.com/LnL7/nix 2018-04-23 08:40:34 -04:00
Andrew Dunham 3a918014b2 Fix library ordering in Makefiles 
The existing ordering linked `libutil` before `libstore`, which causes
link failures when building statically. This is due to `libstore` using
functions from `libutil`, and the fact that symbol resolution works
"forward" - i.e. if you pass `-lfoo -lbar -lbaz`, any symbols that
`libbar` uses from `libbaz` will be resolved, but symbols from `libfoo`
will not since it comes first in the command line.

All this to say: this commit reorders the libraries which fixes the link
errors.
 2018-04-21 21:10:52 -07:00
Peter Simons 040acdcee2
Merge pull request #1707 from dtzWill/fix/issue-1692 
ignore "interrupted" exception in progress callback
 2018-04-21 16:51:19 +02:00
Daiderd Jordan 13d4d1c0a2
download: make hash mismatch error consistent with fetchurl 2018-04-20 21:07:32 +02:00
Will Dietz 6d9129014d build.cc: fix bind-mount of /dev/{pts,ptmx} fallback 
Don't bind-mount these to themselves,
mount them into the chroot directory.

Fixes pty issues when using sandbox on CentOS 7.4.
(build of perlPackages.IOTty fails before this change)
 2018-04-20 12:24:23 -05:00
Daniel Poelzleithner b712d4674b Allow multiple search experssions in nix search 
The common use case is to search for packages containing multiple words
like a "git" "frontend". Having only one expressions makes this simple regular
use case very complicated. Instead, search accepts multiple regular epressions
which all need to match.

nix search git 'gui|frontend'

returns a list of all git uis for example
 2018-04-18 21:08:35 +02:00
Shea Levy b37f5ae31d
isFunction: True on primops. 
Fixes #2073
 2018-04-17 14:33:12 -04:00
Eelco Dolstra a4aac7f88c
Handle arguments in $EDITOR 
Fixes #2079.
 2018-04-17 12:16:04 +02:00
Eelco Dolstra c74f838620
exportReferences: Check whether the path is in the input closure 
Fixes #2082.
 2018-04-17 12:03:27 +02:00
Félix Baylac-Jacqué a91fb422fe
nix repl: Fix multiline SIGINT handling. 
Fixes #2076
 2018-04-16 16:09:30 +02:00
Eelco Dolstra d34fa2bcc3
Fix #1921 2018-04-13 15:42:35 +02:00
Shea Levy dc0a542c9f
initPlugins: Fix dlopen error message. 2018-04-11 21:02:50 -04:00
Eelco Dolstra 1839a5542a
Fix #2057 2018-04-11 11:42:17 +02:00
Eelco Dolstra d6aaa35478
Fix perl-bindings 
https://hydra.nixos.org/build/72636781
 2018-04-10 13:11:43 +02:00
Shea Levy a38fe5c1a8
Export required C++ version in pkgconfig. 2018-04-09 11:32:43 -04:00
Shea Levy 7459388448
Merge branch 'fix/ruby-shebang' of git://github.com/samueldr/nix 2018-04-09 10:54:39 -04:00
Shea Levy fd98fca7bb
Make prim_exec and prim_importNative available to plugins 2018-04-09 10:26:50 -04:00
Shea Levy 346c0ac361
nix-daemon: Exit successfully when interrupted. 
Fixes #2058.
 2018-04-09 08:05:54 -04:00
Samuel Dionne-Riel 399f43c3d5 nix-shell: Fixes use with ruby shebangs. 
The ported code in 80ebc553ec was incorrectly ported.

```
-            $envCommand = "exec $execArgs $interpreter -e 'load(\"$script\")' -- ${\(join ' ', (map shellEscape, @savedArgs))}";
...
+                    envCommand = (format("exec %1% %2% -e 'load(\"%3%\") -- %4%") % execArgs % interpreter % script % joined.str()).str();
```

The single-quote finishing the small ruby snippet was lost in
translation.
 2018-04-07 22:38:45 -04:00
AmineChikhaoui 62d75ad3e1
rename the options to mention it's a narinfo TTL as disk cache is used all over the place for other operations 2018-04-06 11:05:15 +01:00
AmineChikhaoui 86930ed414
add documentation for the local disk cache TTL config 2018-04-06 10:51:18 +01:00
AmineChikhaoui 2855c3d965
Make the TTL for disk cache configurable, we can now completely disable 
disk cache lookup for example by doing:
  nix copy --from <binary-cahe> <store-path> --option \
  positive-disk-cache-ttl 0

Issues: #1885 #2035
 2018-04-06 10:51:18 +01:00
Eelco Dolstra 1d9742b95f
<nix/buildenv.nix>: Ignore dangling symlinks 
Fixes

  error: getting status of '/nix/store/j8p0vv89k1pf0cn7kmfsdcs7bshwga1i-firefox-52.7.2esr/share/icons/hicolor/48x48/apps/firefox.png': No such file or directory

https://github.com/NixOS/nix/issues/1934

Also improve error message on directory/non-directory collisions.
 2018-04-05 20:04:42 +02:00
Eelco Dolstra 5c904e10eb
<nix/buildenv.nix>: Skip non-directory top-level paths 
Fixes https://github.com/NixOS/nix/issues/1934 (at least the "error:
opening directory '/nix/store/...-linux-config-4.4.14': Not a
directory" issue).
 2018-04-05 19:34:56 +02:00
Tuomas Tynkkynen af86132e1a libexpr: Make unsafeGetAttrPos not crash on noPos 
Currently e.g. `builtins.unsafeGetAttrPos "abort" builtins` will
eventually segfault because pos->file is an unset Symbol.

Found by afl-fuzz.
 2018-04-03 15:54:42 +03:00
Jonas Chevalier 865ca2402f nix copy: add an example with S3 
I couldn't find a good example how to use it with non-us-east-1 buckets.
 2018-03-30 02:31:14 +01:00
Eelco Dolstra 446bb88f13
Remove reference to non-existent manpages 2018-03-29 12:29:10 +02:00
Eelco Dolstra 2b61c74922
Process --option use-case-hack properly 
Fixes https://github.com/NixOS/nix/issues/2009.
 2018-03-29 12:28:29 +02:00
Eelco Dolstra 9d1220a01d
ValidPathInfo::isContentAddressed(): Ensure there are no references 2018-03-29 12:27:42 +02:00
Will Dietz 74da813912 download: improve error for hash mismatch ("store mismatch") 
Fixes #1905
 2018-03-22 15:22:06 -05:00
Will Dietz 59b32403f2 download.cc: fix path for cached files 2018-03-22 15:05:46 -05:00
Eelco Dolstra 28eb9621cc
Fix build 2018-03-22 13:46:19 +01:00
Eelco Dolstra dce8af59a6
Tweak 2018-03-22 13:32:18 +01:00
Eelco Dolstra f87e286e82
Merge branch 'fix/avoid-large-stack-buffers' of https://github.com/dtzWill/nix 2018-03-22 13:19:25 +01:00
Shea Levy c24e0439b2
Merge branch 'fix/execl-null-sentinel' of git://github.com/dtzWill/nix 2018-03-21 23:25:53 -04:00
Eelco Dolstra ec91840e70 Also make the backwards compatible case of RemoteStore::addToStore run in constant memory 
Just because we can.
 2018-03-21 23:46:03 +01:00
Eelco Dolstra 97002b684c Make 'nix copy --to daemon' run in constant memory 2018-03-21 23:42:52 +01:00
Will Dietz f66ae22288 shared.cc: style: prefer "nullptr" over '(char *)NULL' 2018-03-21 17:17:37 -05:00
Eelco Dolstra 11898d6a37 Make 'nix copy --to ssh://...' run in constant memory 2018-03-21 23:13:06 +01:00
Eelco Dolstra 47f7e5585b Make 'nix copy --from ssh://...' run in constant memory 
For instance, this reduced the memory consumption of

  $ nix copy --from ssh://localhost --to ~/my-nix /nix/store/1n7x0yv8vq6zi90hfmian84vdhd04bgp-blender-2.79a

from 632 MiB to 16 MiB.
 2018-03-21 23:10:40 +01:00
Eelco Dolstra 8615cfb130
LegacySSHStore: Allow overriding the path to nix-store 2018-03-21 15:10:46 +01:00
Will Dietz 02d36fdab7 execl: cast NULL sentinel to (char *), per man page and compiler warning 
From exec(3):

> The list of arguments must be terminated by a null pointer, and, since these
> are variadic functions, this pointer must be cast (char *) NULL
 2018-03-20 13:31:58 -05:00
Eelco Dolstra 03d8136b02
Remove unused channel-cache directory 2018-03-20 18:22:49 +01:00
Eelco Dolstra e0c1597910
Style fix 2018-03-20 17:33:45 +01:00
Eelco Dolstra 668ac3ea2c
Make <nix/buildenv.nix> a builtin builder 
This avoids sandbox annoyances.
 2018-03-20 17:28:09 +01:00
Eelco Dolstra 9d40787938
Move builtinFetchurl to its own file 2018-03-20 17:01:36 +01:00
Eelco Dolstra 5df427f35b
Slight simplification 2018-03-20 16:32:59 +01:00
Eelco Dolstra 21ebf6a160
SSHMaster: Optionally pass -v to ssh 2018-03-20 15:30:55 +01:00
Eelco Dolstra 1aca195e52
Hack to get SSH error messages from build-remote 
E.g.

  cannot build on 'ssh://mac1': cannot connect to 'mac1': bash: nix-store: command not found
  cannot build on 'ssh://mac2': cannot connect to 'mac2': Host key verification failed.
  cannot build on 'ssh://mac3': cannot connect to 'mac3': Received disconnect from 213... port 6001:2: Too many authentication failures
  Authentication failed.
 2018-03-20 15:25:26 +01:00
Will Dietz f5a2136a40 serialise.cc: remove pessimising move 
from clang6:

src/libutil/serialise.cc:189:23: warning: moving a temporary object prevents copy elision [-Wpessimizing-move]
 2018-03-19 19:18:47 -05:00
Will Dietz 9b33201e72 util: rename stdout/stdin members to avoid conflicts w/standard macro 
(cherry picked from commit c389a7fb617ed7bcd617efa68c6a48c00405310d)
 2018-03-19 11:18:56 -05:00