Commit graph

749 commits

Author SHA1 Message Date
Eelco Dolstra 462421d345 Backport libfetchers from the flakes branch
This provides a pluggable mechanism for defining new fetchers. It adds
a builtin function 'fetchTree' that generalizes existing fetchers like
'fetchGit', 'fetchMercurial' and 'fetchTarball'. 'fetchTree' takes a
set of attributes, e.g.

  fetchTree {
    type = "git";
    url = "https://example.org/repo.git";
    ref = "some-branch";
    rev = "abcdef...";
  }

The existing fetchers are just wrappers around this. Note that the
input attributes to fetchTree are the same as flake input
specifications and flake lock file entries.

All fetchers share a common cache stored in
~/.cache/nix/fetcher-cache-v1.sqlite. This replaces the ad hoc caching
mechanisms in fetchGit and download.cc (e.g. ~/.cache/nix/{tarballs,git-revs*}).

This also adds support for Git worktrees (c169ea5904).
2020-04-07 09:03:14 +02:00
Eelco Dolstra e322a16523 Remove global -I flags
(cherry picked from commit 2c692a3b14)
2020-03-30 15:30:19 +02:00
Eelco Dolstra 0a10854f85 Misc changes from the flakes branch 2020-03-24 14:34:47 +01:00
Eelco Dolstra 7a8de57d3e Pretty-print 'nix why-depends' / 'nix-store -q --tree' output
Extracted from 678301072f.
2020-03-24 14:26:23 +01:00
Eelco Dolstra 9950cdec35 Move some corepkgs into the nix binary 2020-03-11 16:57:48 +01:00
Eelco Dolstra d700eecea9
Add test for foldl' 2020-03-04 11:43:48 +01:00
Eelco Dolstra aef635da78 Fix derivation computation with __structuredAttrs and multiple outputs
Fixes

  error: derivation '/nix/store/klivma7r7h5lndb99f7xxmlh5whyayvg-zlib-1.2.11.drv' has incorrect output '/nix/store/fv98nnx5ykgbq8sqabilkgkbc4169q05-zlib-1.2.11-dev', should be '/nix/store/adm7pilzlj3z5k249s8b4wv3scprhzi1-zlib-1.2.11-dev'
2020-01-21 21:14:13 +01:00
Nikola Knezevic 52a8f9295b Add support for \u escape in fromJSON
As fromTOML supports \u and \U escapes, bring fromJSON on par. As JSON defaults
to UTF-8 encoding (every JSON parser must support UTF-8), this change parses the
`\u hex hex hex hex` sequence (\u followed by 4 hexadecimal digits) into an
UTF-8 representation.

Add a test to verify correct parsing, using all escape sequences from json.org.
2020-01-07 00:09:58 +01:00
edef c65a6fa86a passAsFile: leave out the hash prefix
Having a colon in the path may cause issues, and having the hash
function indicated isn't actually necessary. We now verify the path 
format in the tests to prevent regressions.
2020-01-02 23:56:06 +00:00
Eelco Dolstra 54bf5ba422 nix-store -r: Handle symlinks to store paths
Fixes #3270.
2019-12-16 19:11:47 +01:00
Eelco Dolstra 5a6d6da7ae Validate tarball components 2019-12-13 19:05:26 +01:00
Eelco Dolstra 4581159e3f Simplify tarball test 2019-12-13 17:26:58 +01:00
Tom Bereknyei c6295a3afd Initial gzip support
Closes #3256
2019-12-13 03:34:15 -05:00
Eelco Dolstra cce218f950 Add base32 encoder/decoder 2019-12-10 13:37:23 +01:00
Eelco Dolstra c1d18050b4 Disable recursive Nix test on macOS
https://hydra.nixos.org/build/107724274
2019-12-03 19:19:14 +01:00
Eelco Dolstra ac2bc721d8 Merge remote-tracking branch 'origin/recursive-nix' 2019-12-02 12:34:46 +01:00
Eelco Dolstra 6a9c815734 Remove most of <nix/config.nix>
This is no longer needed.
2019-11-26 22:07:28 +01:00
Eelco Dolstra 872740cf60
Merge pull request #3238 from puckipedia/attrset-overrides-dynamic
Ensure enough space in attrset bindings
2019-11-26 20:14:55 +01:00
Eelco Dolstra c13193017f
Disallow empty store path names
Fixes #3239.
2019-11-26 20:12:15 +01:00
puck cdadbf7708 Add testcase for attrset using __overrides and dynamic attrs 2019-11-25 13:03:54 +00:00
Eric Culp 6c041e8413 Replace $TMPDIR with $TEST_ROOT in tests/fetchurl.sh
$TMPDIR isn't necessarily set and would cause this test to fail.
2019-11-08 12:08:10 -08:00
Eelco Dolstra c119ab9db0
Enable recursive Nix using a feature
Derivations that want to use recursion should now set

  requiredSystemFeatures = [ "recursive-nix" ];

to make the daemon socket appear.

Also, Nix should be configured with "experimental-features =
recursive-nix".
2019-11-06 00:55:03 +01:00
Eelco Dolstra 2af9561316
Add a test for recursive Nix 2019-11-06 00:55:03 +01:00
Eelco Dolstra e1725ba946
Fix VM tests 2019-11-05 11:12:25 +01:00
Eelco Dolstra 0e459d79a6
Merge branch 'issue-3147-inNixShell-arg' of https://github.com/hercules-ci/nix 2019-10-27 17:10:19 +01:00
Robert Hensing 9d612c393a Add inNixShell = true to nix-shell auto-call
This is an alternative to the IN_NIX_SHELL environment variable,
allowing the expression to adapt itself to nix-shell without
triggering those adaptations when used as a dependency of another
shell.

Closes #3147
2019-10-27 13:16:02 +01:00
Eelco Dolstra e012384fe9
Merge branch 'tojson-tostring-fix' of https://github.com/mayflower/nix 2019-10-27 12:18:35 +01:00
Robin Gloster e583df5280
builtins.toJSON: fix __toString usage 2019-10-27 10:15:51 +01:00
Eelco Dolstra aabf5c86c9
Add experimental-features setting
Experimental features are now opt-in. There is currently one
experimental feature: "nix-command" (which enables the "nix"
command. This will allow us to merge experimental features more
quickly, without committing to supporting them indefinitely.

Typical usage:

$ nix build --experimental-features 'nix-command flakes' nixpkgs#hello

(cherry picked from commit 8e478c2341,
without the "flakes" feature)
2019-10-21 13:34:44 +02:00
Eelco Dolstra 9348f9291e
nix-env: Create ~/.nix-profile automatically 2019-10-09 23:35:01 +02:00
Eelco Dolstra 26762ceb86
nix-profile.sh: Don't create .nix-channels
This is already done by the installer, so no need to do it again.
2019-10-09 23:35:01 +02:00
Eelco Dolstra 5a303093dc
Remove world-writability from per-user directories
'nix-daemon' now creates subdirectories for users when they first
connect.

Fixes #509 (CVE-2019-17365).
Should also fix #3127.
2019-10-09 23:34:48 +02:00
Jonas Chevalier 619cc4af85
function-trace: always show the trace
If the user invokes nix with --trace-function-calls it means that they
want to see the trace.
2019-09-18 23:23:21 +02:00
Eelco Dolstra 5fad9d01c2
gc-auto.sh: Increase sleep time 2019-09-04 12:52:54 +02:00
Eelco Dolstra 08ee364950
gc-auto.sh: More test fixes 2019-09-03 18:11:43 +02:00
Eelco Dolstra cec50290bf
gc-auto.sh: Add some more instrumentation 2019-09-03 15:45:32 +02:00
Eelco Dolstra a2c4fcd5e9 Don't rely on st_blocks
It doesn't seem very reliable on ZFS.
2019-08-29 14:49:58 +02:00
Eelco Dolstra 31f5ecfaa5 Maybe fix #3058 2019-08-29 12:35:15 +02:00
Eelco Dolstra ecb0a23d51 Add some more instrumentation 2019-08-29 12:10:01 +02:00
Eelco Dolstra b6120d26a8
gc-auto.sh: Increase verbosity 2019-08-28 22:19:31 +02:00
Eelco Dolstra 73728874ab Hopefully fix post-hook test on macOS
https://hydra.nixos.org/build/99262744
2019-08-27 17:01:54 +02:00
Eelco Dolstra 477f82e5a7
Merge pull request #2782 from grahamc/flames
Track function start and end
2019-08-15 14:20:42 +02:00
Graham Christensen ee9c988a1b
Track function start and ends for flame graphs
With this patch, and this file I called `log.py`:

    #!/usr/bin/env nix-shell
    #!nix-shell -i python3 -p python3 --pure

    import sys
    from pprint import pprint

    stack = []
    timestack = []

    for line in open(sys.argv[1]):
        components = line.strip().split(" ", 2)
        if components[0] != "function-trace":
            continue

        direction = components[1]
        components = components[2].rsplit(" ", 2)

        loc = components[0]
        _at = components[1]
        time = int(components[2])

        if direction == "entered":
            stack.append(loc)
            timestack.append(time)
        elif direction == "exited":
            dur = time - timestack.pop()
            vst = ";".join(stack)
            print(f"{vst} {dur}")
            stack.pop()

and:

    nix-instantiate --trace-function-calls -vvvv ../nixpkgs/pkgs/top-level/release.nix -A unstable > log.matthewbauer 2>&1
    ./log.py ./log.matthewbauer > log.matthewbauer.folded
    flamegraph.pl --title matthewbauer-post-pr log.matthewbauer.folded > log.matthewbauer.folded.svg

I can make flame graphs like: http://gsc.io/log.matthewbauer.folded.svg

---

Includes test cases around function call failures and tryEval. Uses
RAII so the finish is always called at the end of the function.
2019-08-14 16:09:35 -04:00
Eelco Dolstra 05a10dd835
tests/post-hook.sh: Don't put result link in cwd 2019-08-08 15:47:56 +02:00
Eelco Dolstra 2053ac7747
Rename file for consistency 2019-08-08 12:18:46 +02:00
Eelco Dolstra 56df30cd3f
Merge pull request #2995 from tweag/post-build-hook
Add a post build hook
2019-08-07 15:02:29 +02:00
Eelco Dolstra ec415d7166
Add a test for auto-GC
This currently fails because we're using POSIX file locks. So when the
garbage collector opens and closes its own temproots file, it causes
the lock to be released and then deleted by another GC instance.
2019-08-02 18:39:16 +02:00
regnat 7c5596734f
Add a post-build-hook
Passing `--post-build-hook /foo/bar` to a nix-* command will cause
`/foo/bar` to be executed after each build with the following
environment variables set:

    DRV_PATH=/nix/store/drv-that-has-been-built.drv
    OUT_PATHS=/nix/store/...build /nix/store/...build-bin /nix/store/...build-dev

This can be useful in particular to upload all the builded artifacts to
the cache (including the ones that don't appear in the runtime closure
of the final derivation or are built because of IFD).

This new feature prints the stderr/stdout output to the `nix-build`
and `nix build` client, and the output is printed in a Nix 2
compatible format:

    [nix]$ ./inst/bin/nix-build ./test.nix
    these derivations will be built:
      /nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv
    building '/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv'...
    hello!
    bye!
    running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'...
    post-build-hook: + sleep 1
    post-build-hook: + echo 'Signing paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: Signing paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: + sleep 1
    post-build-hook: + echo 'Uploading paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: Uploading paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: + sleep 1
    post-build-hook: + printf 'very important stuff'
    /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation

    [nix-shell:~/projects/github.com/NixOS/nix]$ ./inst/bin/nix build -L -f ./test.nix
    my-example-derivation> hello!
    my-example-derivation> bye!
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + echo 'Signing paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> Signing paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + echo 'Uploading paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> Uploading paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + printf 'very important stuff'
    [1 built, 0.0 MiB DL]

Co-authored-by: Graham Christensen <graham@grahamc.com>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2019-08-02 10:48:15 -04:00
Eelco Dolstra 219d645987
Merge pull request #3013 from basvandijk/disable-lsof-for-darwin-tests
Disable findRuntimeRoots on darwin when running tests because lsof is slow
2019-07-30 11:34:18 +02:00
Bas van Dijk ee1e3132ca Disable findRuntimeRoots on darwin when running tests because lsof is slow
See: https://github.com/NixOS/nix/issues/3011
2019-07-30 11:29:03 +02:00