Commit graph

7197 commits

Author SHA1 Message Date
Eelco Dolstra e349f2c0a3
Use BSD instead of POSIX file locks
POSIX file locks are essentially incompatible with multithreading. BSD
locks have much saner semantics. We need this now that there can be
multiple concurrent LocalStore::buildPaths() invocations.
2019-08-02 18:39:16 +02:00
Eelco Dolstra ec415d7166
Add a test for auto-GC
This currently fails because we're using POSIX file locks. So when the
garbage collector opens and closes its own temproots file, it causes
the lock to be released and then deleted by another GC instance.
2019-08-02 18:39:16 +02:00
regnat 7c5596734f
Add a post-build-hook
Passing `--post-build-hook /foo/bar` to a nix-* command will cause
`/foo/bar` to be executed after each build with the following
environment variables set:

    DRV_PATH=/nix/store/drv-that-has-been-built.drv
    OUT_PATHS=/nix/store/...build /nix/store/...build-bin /nix/store/...build-dev

This can be useful in particular to upload all the builded artifacts to
the cache (including the ones that don't appear in the runtime closure
of the final derivation or are built because of IFD).

This new feature prints the stderr/stdout output to the `nix-build`
and `nix build` client, and the output is printed in a Nix 2
compatible format:

    [nix]$ ./inst/bin/nix-build ./test.nix
    these derivations will be built:
      /nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv
    building '/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv'...
    hello!
    bye!
    running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'...
    post-build-hook: + sleep 1
    post-build-hook: + echo 'Signing paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: Signing paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: + sleep 1
    post-build-hook: + echo 'Uploading paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: Uploading paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: + sleep 1
    post-build-hook: + printf 'very important stuff'
    /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation

    [nix-shell:~/projects/github.com/NixOS/nix]$ ./inst/bin/nix build -L -f ./test.nix
    my-example-derivation> hello!
    my-example-derivation> bye!
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + echo 'Signing paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> Signing paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + echo 'Uploading paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> Uploading paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + printf 'very important stuff'
    [1 built, 0.0 MiB DL]

Co-authored-by: Graham Christensen <graham@grahamc.com>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2019-08-02 10:48:15 -04:00
Eelco Dolstra 320126aeeb
Tweak min-free/max-free descriptions 2019-08-02 14:04:09 +02:00
Matthew Bauer 9a0855bbb6 Don’t rely on EPERM
startProcess does not appear to send the exit code to the helper
correctly. Not sure why this is, but it is probably safe to just
fallback on all sandbox errors.
2019-07-30 17:53:37 -04:00
Eelco Dolstra 41d010fff6
Merge pull request #3009 from codedownio/add-pname-and-version-to-json
Add pname and version to nix-env -q --json
2019-07-30 11:43:45 +02:00
Eelco Dolstra 219d645987
Merge pull request #3013 from basvandijk/disable-lsof-for-darwin-tests
Disable findRuntimeRoots on darwin when running tests because lsof is slow
2019-07-30 11:34:18 +02:00
Eelco Dolstra 7680357ccc
Merge pull request #3012 from basvandijk/fix-pathExists
Allow builtins.pathExists to check the existence of /nix/store paths
2019-07-30 11:33:37 +02:00
Bas van Dijk ee1e3132ca Disable findRuntimeRoots on darwin when running tests because lsof is slow
See: https://github.com/NixOS/nix/issues/3011
2019-07-30 11:29:03 +02:00
Bas van Dijk 89865144c3 Allow builtins.pathExists to check the existence of /nix/store paths
This makes it consitent with builtins.readDir.
2019-07-30 11:27:35 +02:00
Tom McLaughlin cd933b22d2 Add pname and version to nix-env -q --json 2019-07-27 19:40:51 -07:00
Matthew Bauer 11d8534629 Use sandbox fallback when cloning fails in builder
When sandbox-fallback = true (the default), the Nix builder will fall
back to disabled sandbox mode when the kernel doesn’t allow users to
set it up. This prevents hard errors from occuring in tricky places,
especially the initial installer. To restore the previous behavior,
users can set:

  sandbox-fallback = false

in their /etc/nix/nix.conf configuration.
2019-07-25 14:42:30 -04:00
Matthew Bauer d171090530 Disable CLONE_NEWUSER when it’s unavailable
Some kernels disable "unpriveleged user namespaces". This is
unfortunate, but we can still use mount namespaces. Anyway, since each
builder has its own nixbld user, we already have most of the benefits
of user namespaces.
2019-07-25 14:42:25 -04:00
Eelco Dolstra 41a5246685
Merge pull request #3008 from matthewbauer/fix-typo
Use $HOME instead of $USER
2019-07-25 17:47:39 +02:00
Eelco Dolstra 1fb8e2605a
Merge pull request #3007 from matthewbauer/add-user-default
Add default for USER when unset
2019-07-25 17:46:05 +02:00
Matthew Bauer 03addc3b0a Use $HOME instead of $USER
$USER/.nix-profile will not be a path. I think $HOME/.nix-profile was
the origininal intent.

/cc @Grahamc
2019-07-25 09:44:01 -04:00
Matthew Bauer c82a856b36 Add default for USER when unset
uses $(id -u -n) when USER is unset, this is needed on some weird
setups in Docker. Fixes #971
2019-07-25 09:39:44 -04:00
Domen Kožar b640f69a4d
Merge pull request #3004 from zimbatm/shared-funding
Remove .github/FUNDING.yml
2019-07-23 15:22:32 +02:00
Jonas Chevalier 9031a6838c
Remove .github/FUNDING.yml
The configuration is now done through the shared configuration repo:

https://github.com/nixos/.github
2019-07-23 15:21:23 +02:00
Eelco Dolstra 1bace4022f
Merge pull request #2749 from grahamc/docs-cores-max-jobs
docs: document balancing cores and max-jobs
2019-07-19 14:40:16 +02:00
Graham Christensen cf6172f05e
docs: document balancing cores and max-jobs 2019-07-19 08:28:44 -04:00
Domen Kožar 5e0a64229b
Add Open Collective 2019-07-18 10:57:26 +02:00
Eelco Dolstra 2f853b20df
Merge pull request #2975 from matthewbauer/fix-nsswitch-issue
Don’t use entire /etc/nsswitch.conf file
2019-07-13 17:08:02 +02:00
Eelco Dolstra 53247d6b11
Resume NAR downloads
This is a much simpler fix to the 'error 9 while decompressing xz
file' problem than 78fa47a7f0. We just
do a ranged HTTP request starting after the data that we previously
wrote into the sink.

Fixes #2952, #379.
2019-07-10 23:12:17 +02:00
Eelco Dolstra 00f6fafad6
HttpBinaryCacheStore: Use default number of retries for NARs 2019-07-10 23:05:04 +02:00
Eelco Dolstra f76b2a7fdd
Downloader: Use warn() 2019-07-10 22:27:50 +02:00
Eelco Dolstra 03f09e1d18
Revert "Fix 'error 9 while decompressing xz file'"
This reverts commit 78fa47a7f0.
2019-07-10 19:46:15 +02:00
Eelco Dolstra aa739e7839
nix copy: Rename --substitute to --substitute-on-destination
'--substitute' was being shadowed by the regular '--substitute' (the
short-hand for '--option substitute true').

Fixes #2983.
2019-07-10 11:28:37 +02:00
Eelco Dolstra b5ae85f088
Merge pull request #2882 from grahamc/docs/1115-tarball-ttl
tarball-ttl: document
2019-07-06 00:15:27 +02:00
Graham Christensen 648bdf153d
tarball-ttl: document
Incorporates text from Niklas Hambüchen in #2978

Closes #1115
2019-07-05 15:55:28 -04:00
Eelco Dolstra e486d8d40e Revert 82b7f0e840, cd8bc06e87, c3db9e6f8f
This breaks the tarball job: https://hydra.nixos.org/build/95714570
2019-07-05 00:35:59 +02:00
Eelco Dolstra 7d6ba1dc90
Merge branch 'autoconf-ubuntu-16.04-fixes' of https://github.com/nh2/nix 2019-07-03 08:02:45 +02:00
Niklas Hambüchen 82b7f0e840 autoconf: Implement release tarball detection. Fixes #257.
This should finally allow us to address all cases of build errors due to
differences between release tarballs and building from git.

See also https://github.com/NixOS/nix/issues/506#issuecomment-507312587
2019-07-03 04:32:25 +02:00
Niklas Hambüchen cd8bc06e87 autoconf: Add comment on use of false.
This is to avoid confusion as in commit
a0d29040f7.
2019-07-03 04:32:25 +02:00
Niklas Hambüchen c3db9e6f8f autoconf: Check if --nonet works. Fixes #967 #506.
Also give a helpful error message on what package the user likely
has to install to make it work.
2019-07-03 04:32:25 +02:00
Niklas Hambüchen a96006d97f Get BOOST_LDFLAGS from autoconf, fix Ubuntu 16.04 build.
Our use of boost::coroutine2 depends on -lboost_context,
which in turn depends on `-lboost_thread`, which in turn depends
on `-lboost_system`.

I suspect that this builds on nix only because of low-level hacks
like NIX_LDFLAGS.

This commit passes the proper linker flags, thus fixing bootstrap
builds on non-nix distributions like Ubuntu 16.04.

With these changes, I can build Nix on Ubuntu 16.04 using:

    ./bootstrap.sh
    ./configure --prefix=$HOME/editline-prefix \
      --disable-doc-gen \
      CXX=g++-7 \
      --with-boost=$HOME/boost-prefix \
      EDITLINE_CFLAGS=-I$HOME/editline-prefix/include \
      EDITLINE_LIBS=-leditline \
      LDFLAGS=-L$HOME/editline-prefix/lib
    make

where

* g++-7 comes from gcc-7 from
  https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test,
* editline 1.14 from https://github.com/troglobit/editline/releases/tag/1.14.0
	was installed into `$HOME/editline-prefix`
  (because Ubuntu 16.04's `editline` is too old to have the function nix uses),
* boost 1.66 from
	https://www.boost.org/doc/libs/1_66_0/more/getting_started/unix-variants.html
	was installed into $HOME/boost-prefix (because Ubuntu 16.04 only has 1.58)
2019-07-03 04:32:25 +02:00
Niklas Hambüchen d203c554fa Fix C++ compatibility with older editline versions.
For example, Ubuntu 16.04 and many similar long-term-support distros
have older versions.
2019-07-03 04:32:25 +02:00
Niklas Hambüchen b49c3a9db5 Makefile.config.in: Remove HAVE_READLINE.
It was forgotten to be removed with
commit c5f23f10a8
and so it until now stayed unsubstituted as `HAVE_READLINE = @HAVE_READLINE@`
in Makefile.config.
2019-07-03 04:32:25 +02:00
Niklas Hambüchen 717e821b99 autoconf: Allow overriding CFLAGS/CXXFLAGS from outside.
As is normal for autoconf-based projects.

For example, it is a common use case to do

    ./configure CXXFLAGS=-O0

This did not work for nix until now, because the `CXXFLAGS=` declaration
would unconditionally erase what the user had specified.

The custom `OPTIMIZE` flag is removed, but the default `-O3` is retained;
autoconf would default to `-g -O2` by default otherwise as documented on:

https://www.gnu.org/software/autoconf/manual/autoconf-2.60/html_node/C-Compiler.html
https://www.gnu.org/software/autoconf/manual/autoconf-2.60/html_node/C_002b_002b-Compiler.html
2019-07-03 04:32:25 +02:00
Niklas Hambüchen 20129bd83d autoconf: Fix AC_STRUCT_DIRENT_D_TYPE being used before AC_PROG_CC.
That was incorrect, because checking the dirent type already requires
a working compiler.

It had the effect that setting e.g. `: ${CFLAGS=""}` before `AC_PROG_CC`
as per `AC_PROG_CC`'s documentation would have no effect, because
`AC_STRUCT_DIRENT_D_TYPE` would automatically set CFLASGS.

(In a followup commit `: ${CFLAGS=""}` will be used, so it's important
to get this working first.)
2019-07-03 04:32:25 +02:00
Sergei Trofimovich fe068eca00 mk: add support for passing LDFLAGS to libs and bins
autotools-based systems usually allow user to
append own LDFLAGS like
    LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu"
at ./configure stage

This change plumbs LDFLAGS through similar to existing CXXFLAGS variable.

Signed-off-by: Sergei Trofimovich <siarheit@google.com>
2019-07-03 04:32:25 +02:00
Niklas Hambüchen 57daa860e8 autoconf: Fix C++17 detection not working on Ubuntu 16.04.
And probably many other distributions.

Until now, ./configure would fail silently printing a warning

    ./configure: line 4621: AX_CXX_COMPILE_STDCXX_17: command not found

and then continuing, later failing with a C++ #error saying that some C++11
feature isn't supported (it didn't even get to the C++17 features).

This is because older distributions don't come with the
`AX_CXX_COMPILE_STDCXX_17` m4 macro.

This commit vendors that macro accordingly.

Now ./configure complains correctly:

    configure: error: *** A compiler with support for C++17 language features is required.

On Ubuntu 16.04, ./configure completes if a newer compiler is used, e.g. with
gcc-7 from https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test
using:

    ./bootstrap.sh
    ./configure CXX=g++-7 --disable-doc-gen --with-boost=$(nix-build --no-link '<nixpkgs>' -A boost.dev)
2019-07-03 04:32:25 +02:00
Niklas Hambüchen 1f97b16b1d autoconf: Work around editline not being found on Ubuntu 16.04.
And probably other Linux distributions with long-term support releases.

Also update manual stating what version is needed;
I checked that 1.14 is the oldest version with which current nix compiles,
and added autoconf feature checks for some functions added in that release
that nix uses.
2019-07-03 04:32:25 +02:00
Niklas Hambüchen 00a450026f autoconf: Detect boost, require version, set CXXFLAGS.
This turns previous compiler errors complaining about missing files
into proper ./configure time errors telling the user which version
of boost is required.
2019-07-03 04:32:25 +02:00
Niklas Hambüchen 96cd3d6073 autoconf: Change quotes in description.
The unbalanced single-quotes cause many editor syntax highlighters
to interpret the rest of the file as a string literal, making it easier
to make syntax mistakes in absence of proper highlighting.
2019-07-03 04:32:25 +02:00
Eelco Dolstra 7e1c85c5fb
Merge pull request #2779 from LnL7/build-exit-codes
build: add exit code for hash and check mismatches
2019-07-02 17:37:49 +02:00
Graham Christensen 68bdd83dc8
timeout: test for error code 2019-07-02 11:18:36 -04:00
Eelco Dolstra db700f730e
Merge pull request #2974 from grahamc/invalid-name
checkStoreName: give more precise/verbose error information
2019-07-02 16:12:01 +02:00
Eelco Dolstra 7c0b0dbec8
Merge pull request #2724 from LnL7/manpage-add-fixed
nix-store: document --add-fixed
2019-07-02 15:47:40 +02:00
Eelco Dolstra 33db1d35ae
Merge pull request #2582 from LnL7/fetchgit-refs
fetchGit: allow fetching explicit refs
2019-07-02 15:44:31 +02:00