Commit graph

714 commits

Author SHA1 Message Date
Eelco Dolstra cec50290bf
gc-auto.sh: Add some more instrumentation 2019-09-03 15:45:32 +02:00
Eelco Dolstra a2c4fcd5e9 Don't rely on st_blocks
It doesn't seem very reliable on ZFS.
2019-08-29 14:49:58 +02:00
Eelco Dolstra 31f5ecfaa5 Maybe fix #3058 2019-08-29 12:35:15 +02:00
Eelco Dolstra ecb0a23d51 Add some more instrumentation 2019-08-29 12:10:01 +02:00
Eelco Dolstra b6120d26a8
gc-auto.sh: Increase verbosity 2019-08-28 22:19:31 +02:00
Eelco Dolstra 73728874ab Hopefully fix post-hook test on macOS
https://hydra.nixos.org/build/99262744
2019-08-27 17:01:54 +02:00
Eelco Dolstra 477f82e5a7
Merge pull request #2782 from grahamc/flames
Track function start and end
2019-08-15 14:20:42 +02:00
Graham Christensen ee9c988a1b
Track function start and ends for flame graphs
With this patch, and this file I called `log.py`:

    #!/usr/bin/env nix-shell
    #!nix-shell -i python3 -p python3 --pure

    import sys
    from pprint import pprint

    stack = []
    timestack = []

    for line in open(sys.argv[1]):
        components = line.strip().split(" ", 2)
        if components[0] != "function-trace":
            continue

        direction = components[1]
        components = components[2].rsplit(" ", 2)

        loc = components[0]
        _at = components[1]
        time = int(components[2])

        if direction == "entered":
            stack.append(loc)
            timestack.append(time)
        elif direction == "exited":
            dur = time - timestack.pop()
            vst = ";".join(stack)
            print(f"{vst} {dur}")
            stack.pop()

and:

    nix-instantiate --trace-function-calls -vvvv ../nixpkgs/pkgs/top-level/release.nix -A unstable > log.matthewbauer 2>&1
    ./log.py ./log.matthewbauer > log.matthewbauer.folded
    flamegraph.pl --title matthewbauer-post-pr log.matthewbauer.folded > log.matthewbauer.folded.svg

I can make flame graphs like: http://gsc.io/log.matthewbauer.folded.svg

---

Includes test cases around function call failures and tryEval. Uses
RAII so the finish is always called at the end of the function.
2019-08-14 16:09:35 -04:00
Eelco Dolstra 05a10dd835
tests/post-hook.sh: Don't put result link in cwd 2019-08-08 15:47:56 +02:00
Eelco Dolstra 2053ac7747
Rename file for consistency 2019-08-08 12:18:46 +02:00
Eelco Dolstra 56df30cd3f
Merge pull request #2995 from tweag/post-build-hook
Add a post build hook
2019-08-07 15:02:29 +02:00
Eelco Dolstra ec415d7166
Add a test for auto-GC
This currently fails because we're using POSIX file locks. So when the
garbage collector opens and closes its own temproots file, it causes
the lock to be released and then deleted by another GC instance.
2019-08-02 18:39:16 +02:00
regnat 7c5596734f
Add a post-build-hook
Passing `--post-build-hook /foo/bar` to a nix-* command will cause
`/foo/bar` to be executed after each build with the following
environment variables set:

    DRV_PATH=/nix/store/drv-that-has-been-built.drv
    OUT_PATHS=/nix/store/...build /nix/store/...build-bin /nix/store/...build-dev

This can be useful in particular to upload all the builded artifacts to
the cache (including the ones that don't appear in the runtime closure
of the final derivation or are built because of IFD).

This new feature prints the stderr/stdout output to the `nix-build`
and `nix build` client, and the output is printed in a Nix 2
compatible format:

    [nix]$ ./inst/bin/nix-build ./test.nix
    these derivations will be built:
      /nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv
    building '/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv'...
    hello!
    bye!
    running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'...
    post-build-hook: + sleep 1
    post-build-hook: + echo 'Signing paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: Signing paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: + sleep 1
    post-build-hook: + echo 'Uploading paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: Uploading paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: + sleep 1
    post-build-hook: + printf 'very important stuff'
    /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation

    [nix-shell:~/projects/github.com/NixOS/nix]$ ./inst/bin/nix build -L -f ./test.nix
    my-example-derivation> hello!
    my-example-derivation> bye!
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + echo 'Signing paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> Signing paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + echo 'Uploading paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> Uploading paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + printf 'very important stuff'
    [1 built, 0.0 MiB DL]

Co-authored-by: Graham Christensen <graham@grahamc.com>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2019-08-02 10:48:15 -04:00
Eelco Dolstra 219d645987
Merge pull request #3013 from basvandijk/disable-lsof-for-darwin-tests
Disable findRuntimeRoots on darwin when running tests because lsof is slow
2019-07-30 11:34:18 +02:00
Bas van Dijk ee1e3132ca Disable findRuntimeRoots on darwin when running tests because lsof is slow
See: https://github.com/NixOS/nix/issues/3011
2019-07-30 11:29:03 +02:00
Bas van Dijk 89865144c3 Allow builtins.pathExists to check the existence of /nix/store paths
This makes it consitent with builtins.readDir.
2019-07-30 11:27:35 +02:00
Matthew Bauer 03addc3b0a Use $HOME instead of $USER
$USER/.nix-profile will not be a path. I think $HOME/.nix-profile was
the origininal intent.

/cc @Grahamc
2019-07-25 09:44:01 -04:00
Graham Christensen 68bdd83dc8
timeout: test for error code 2019-07-02 11:18:36 -04:00
Daiderd Jordan a52c331edb
build: replace 100 offset for build exit codes 2019-07-02 00:12:38 +02:00
Daiderd Jordan 99ee3755dd
build: add tests for --check status codes 2019-07-02 00:12:38 +02:00
Eelco Dolstra 94f11d0a61
Fix abort in fromTOML
Fixes #2969.
2019-06-24 17:09:09 +02:00
Eelco Dolstra 99cec651c9
Add more fromTOML tests 2019-06-24 16:52:11 +02:00
Eelco Dolstra 5064971ded
Fix test failures when $TMPDIR changes
(cherry picked from commit c38c726eb5)
2019-06-15 16:36:33 +02:00
Eelco Dolstra aec545c20b Fix segfault in builtin fetchurl with hashed mirrors + SRI hashes 2019-06-01 15:27:43 +02:00
Eelco Dolstra 17ef3e6f41
Enable more fromTOML tests
cpptoml now parses almost all examples from the spec.
2019-05-29 12:22:52 +02:00
Eelco Dolstra cfd74aef1e
Fix eval-okay-fromTOML test
Turns out we were mis-parsing single-quoted attributes, e.g. 'key2'.
2019-05-29 12:12:02 +02:00
Eelco Dolstra 9eaebbf575
Merge branch 'attrPaths' of https://github.com/NinjaTrappeur/nix 2019-05-28 22:59:05 +02:00
Daiderd Jordan d75bdb5793
build: add test for sandboxed --check 2019-05-12 16:46:21 +02:00
Will Dietz a834861876
fix hashfile test that wasn't failing due to eval laziness
See:
7becb1bf1c (r33450554)
2019-05-08 10:04:54 +02:00
Daniel Schaefer 3f192ac80c Add builtins.hashFile
For text files it is possible to do it like so:
`builtins.hashString "sha256" (builtins.readFile /tmp/a)`
but that doesn't work for binary files.

With builtins.hashFile any kind of file can be conveniently hashed.
2019-05-03 17:23:36 +02:00
Eelco Dolstra e489f5cabf Update eval-okay-types.exp to match #1828 2019-03-27 21:07:04 +01:00
Jonas Chevalier 514b3c7f83
Add isPath primop
this is added for completeness' sake since all the other possible
`builtins.typeOf` results have a corresponding `builtins.is<Type>`
2019-03-24 11:36:49 +01:00
Guillaume Maudoux 115e2c8c67 Update tests to the new --roots format 2019-03-10 01:50:33 +01:00
Eelco Dolstra b71e1fb342
Restore --init calls in tests 2019-03-04 18:27:45 +08:00
Jonas Chevalier b402148d8f
remove noop uses of nix-store --init
the nix-store --init command is a noop apparently
2019-02-22 21:07:53 +01:00
Shea Levy b30be6b450
Add builtins.appendContext.
A partner of builtins.getContext, useful for the same reasons.
2019-01-31 08:52:23 -05:00
Shea Levy 1d757292d0
Add builtins.getContext.
This can be very helpful when debugging, as well as enabling complex
black magic like surgically removing a single dependency from a
string's context.
2019-01-14 11:27:10 -05:00
Will Dietz aa7e52abff tests/fetchurl: fix after changing default hash from 512 to 256 2018-12-14 22:37:20 -06:00
Eelco Dolstra 6024dc1d97
Support SRI hashes
SRI hashes (https://www.w3.org/TR/SRI/) combine the hash algorithm and
a base-64 hash. This allows more concise and standard hash
specifications. For example, instead of

  import <nix/fetchurl.nl> {
    url = https://nixos.org/releases/nix/nix-2.1.3/nix-2.1.3.tar.xz;
    sha256 = "5d22dad058d5c800d65a115f919da22938c50dd6ba98c5e3a183172d149840a4";
  };

you can write

  import <nix/fetchurl.nl> {
    url = https://nixos.org/releases/nix/nix-2.1.3/nix-2.1.3.tar.xz;
    hash = "sha256-XSLa0FjVyADWWhFfkZ2iKTjFDda6mMXjoYMXLRSYQKQ=";
  };

In fixed-output derivations, the outputHashAlgo is no longer mandatory
if outputHash specifies the hash (either as an SRI or in the old
"<type>:<hash>" format).

'nix hash-{file,path}' now print hashes in SRI format by default. I
also reverted them to use SHA-256 by default because that's what we're
using most of the time in Nixpkgs.

Suggested by @zimbatm.
2018-12-13 14:30:52 +01:00
Eelco Dolstra 3f4de91d80
Merge branch 'better-git-cache' of https://github.com/graham-at-target/nix 2018-11-20 20:41:19 +01:00
Eelco Dolstra 812e39313c
Enable sandboxing by default
Closes #179.
2018-11-07 17:08:28 +01:00
Eelco Dolstra 18b4c53f71
Restore old (dis)allowedRequisites behaviour for self-references
stdenv relies on this. So ignore self-references (but only in legacy non-structured attributes mode).
2018-10-27 15:41:53 +02:00
Eelco Dolstra 3cd15c5b1f
Per-output reference and closure size checks
In structured-attributes derivations, you can now specify per-output
checks such as:

  outputChecks."out" = {
    # The closure of 'out' must not be larger than 256 MiB.
    maxClosureSize = 256 * 1024 * 1024;

    # It must not refer to C compiler or to the 'dev' output.
    disallowedRequisites = [ stdenv.cc "dev" ];
  };

  outputChecks."dev" = {
    # The 'dev' output must not be larger than 128 KiB.
    maxSize = 128 * 1024;
  };

Also fixed a bug in allowedRequisites that caused it to ignore
self-references.
2018-10-23 01:29:16 +02:00
Eelco Dolstra 01bd66bf83
Merge pull request #2452 from ElvishJerricco/verify-sigs-overflow
Fix overflow when verifying signatures of content addressable paths
2018-10-08 12:03:03 +02:00
Eelco Dolstra 1e7b8deea7
Check requiredSystemFeatures for local builds
For example, this prevents a "kvm" build on machines that don't have
KVM.

Fixes #2012.
2018-09-28 16:10:27 +02:00
Will Fancher b7091ce41e Add a test for signed content-addressed paths 2018-09-25 22:18:52 -04:00
Graham Christensen 0aca1ffb6e Prefer 'remote builder' over 'slave' 2018-09-10 18:57:46 +02:00
Eelco Dolstra 9b1bdf2db8
FIx floating point evaluation
Fixes #2361.
2018-08-19 11:59:49 +02:00
Graham Christensen 02098d2073 fetchGit: use a better caching scheme
The current usage technically works by putting multiple different
repos in to the same git directory. However, it is very slow as
Git tries very hard to find common commits between the two
repositories. If the two repositories are large (like Nixpkgs and
another long-running project,) it is maddeningly slow.

This change busts the cache for existing deployments, but users
will be promptly repaid in per-repository performance.
2018-08-17 11:27:34 -04:00
Eelco Dolstra fa4def3d46
Require libbrotli 2018-08-06 14:06:54 +02:00