Once upon a time, I wrote my bachelors thesis about functional
deployment mechanisms.
I had to evaluate several szenarios where package management and
deployment were relevant. One szenario was to do distributed builds
over several machines.
I told myself: Weee, nix can do this! And with nix, this is actually
save, as you do not have side effects when building!
So I started. I use a cloud to set up four virtual machines where I
wanted to do the build. A fifth machine was used as master to distribute
the builds. All was good.
I created the necessary SSH keys, made sure every machine was reachable
by the master and configured the build in my remotes.conf.
When I started to try to build weechat from source, the build failed. It
failed, telling me
error: unable to start any build; either increase ‘--max-jobs’ or enable distributed builds
And I started to dig around. I digged long and good. But I wasn't able
to find the issue.
I double and triple checked my environment variables, my settings, the
SSH key and everything.
I reached out to fellow Nixers by asking on the nixos IRC channel. And I
got help. But we weren't able to find the issue, either.
So I became frustrated. I re-did all the environment variables.
And suddenly,... it worked! What did I change? Well... I made the
environment variables which contained pathes contain absolute pathes
rather than relatives.
And because I like to share my knowledge, this should be put into the
documentation, so others do not bang their heads against the wall
because something is not documented somewhere.
Docbook XSL got updated to version 1.79.1 in NixOS/nixpkgs@fb893a8 and
we're still referring to the hardcoded previous version.
So instead of just updating this to 1.79.1 we're going to use "current"
in the hope that this won't happen again.
I have tested this by building the manual under Nix(OS) but I haven't
tested this in a non-Nix environment, so I'm not sure whether this could
have implications.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Cc: @edolstra
The existing "nix-build" examples were failing:
error: cannot auto-call a function that has an argument without a default value (‘system’)
Thanks to @groxxda on irc for pointing out the fix!
Updated to completely remove unneeded path argument, suggested by @joachifm
Updated to remove other occurences of `all-packages.nix` from files as
suggested by @domenkozar
Some benchmarking suggested this as a good value. Running
$ benchmark -f ... -t 25 -- sh -c 'rm -f /nix/var/nix/binary-cache*; nix-store -r /nix/store/x5z8a2yvz8h6ccmhwrwrp9igg03575jg-nixos-15.09.git.5fd87e1M.drv --dry-run --option binary-caches-parallel-connections <N>'
gave the following mean elapsed times for these values of N:
N=10: 3.3541
N=20: 2.9320
N=25: 2.6690
N=30: 2.9417
N=50: 3.2021
N=100: 3.5718
N=150: 4.2079
Memory usage is also reduced (N=150 used 186 MB, N=25 only 68 MB).
Closes#708.
- rename options but leav old names as lower-priority aliases,
also "-dirs" -> "-paths" to get closer to the meaning
- update docs to reflect the new names (old aliases are not documented),
including a new file with release notes
- tests need an update after corresponding changes to nixpkgs
- __noChroot is left as it is (after discussion on the PR)
Passing "--option build-repeat <N>" will cause every build to be
repeated N times. If the build output differs between any round, the
build is rejected, and the output paths are not registered as
valid. This is primarily useful to verify build determinism. (We
already had a --check option to repeat a previously succeeded
build. However, with --check, non-deterministic builds are registered
in the DB. Preventing that is useful for Hydra to ensure that
non-deterministic builds don't end up getting published at all.)
This allows overriding the name component of the resulting Nix store
path, which is necessary if the base name of the URI contains
"illegal" characters.
This allows nix-prefetch-url to prefetch the output of fetchzip and
its wrappers (like fetchFromGitHub). For example:
$ nix-prefetch-url --unpack https://github.com/NixOS/patchelf/archive/0.8.tar.gz
or from a Nix expression:
$ nix-prefetch-url -A nix-repl.src
In the latter case, --unpack can be omitted because nix-repl.src is a
fetchFromGitHub derivation and thus has "outputHashMode" set to
"recursive".
For example,
$ nix-prefetch-url -A hello.src
will prefetch the file specified by the fetchurl call in the attribute
‘hello.src’ from the Nix expression in the current directory. This
differs from ‘nix-build -A hello.src’ in that it doesn't verify the
hash.
You can also specify a path to the Nix expression:
$ nix-prefetch-url ~/Dev/nixpkgs -A hello.src
List elements (typically used in ‘patches’ attributes) also work:
$ nix-prefetch-url -A portmidi.patches.0
This hook can be used to set system-specific per-derivation build
settings that don't fit into the derivation model and are too complex or
volatile to be hard-coded into nix. Currently, the pre-build hook can
only add chroot dirs/files through the interface, but it also has full
access to the chroot root.
The specific use case for this is systems where the operating system ABI
is more complex than just the kernel-support system calls. For example,
on OS X there is a set of system-provided frameworks that can reliably
be accessed by any program linked to them, no matter the version the
program is running on. Unfortunately, those frameworks do not
necessarily live in the same locations on each version of OS X, nor do
their dependencies, and thus nix needs to know the specific version of
OS X currently running in order to make those frameworks available. The
pre-build hook is a perfect mechanism for doing just that.
This hook can be used to set system specific per-derivation build
settings that don't fit into the derivation model and are too complex or
volatile to be hard-coded into nix. Currently, the pre-build hook can
only add chroot dirs/files.
The specific use case for this is systems where the operating system ABI
is more complex than just the kernel-supported system calls. For
example, on OS X there is a set of system-provided frameworks that can
reliably be accessed by any program linked to them, no matter the
version the program is running on. Unfortunately, those frameworks do
not necessarily live in the same locations on each version of OS X, nor
do their dependencies, and thus nix needs to know the specific version
of OS X currently running in order to make those frameworks available.
The pre-build hook is a perfect mechanism for doing just that.
If ‘build-use-chroot’ is set to ‘true’, fixed-output derivations are
now also chrooted. However, unlike normal derivations, they don't get
a private network namespace, so they can still access the
network. Also, the use of the ‘__noChroot’ derivation attribute is
no longer allowed.
Setting ‘build-use-chroot’ to ‘relaxed’ gives the old behaviour.
If ‘--option restrict-eval true’ is given, the evaluator will throw an
exception if an attempt is made to access any file outside of the Nix
search path. This is primarily intended for Hydra, where we don't want
people doing ‘builtins.readFile ~/.ssh/id_dsa’ or stuff like that.
‘--run’ is like ‘--command’, except that it runs the command in a
non-interactive shell. This is important if you do things like:
$ nix-shell --command make
Hitting Ctrl-C while make is running drops you into the interactive
Nix shell, which is probably not what you want. So you can now do
$ nix-shell --run make
instead.
'... another level of indirection not shown in the figure above ...'
but in the 'user-environments.png' figure there is '~/.nix-profile'.
the figure was updated with the commit: f982df3 on Mar 16, 2005.
‘trusted-users’ is a list of users and groups that have elevated
rights, such as the ability to specify binary caches. It defaults to
‘root’. A typical value would be ‘@wheel’ to specify all users in the
wheel group.
‘allowed-users’ is a list of users and groups that are allowed to
connect to the daemon. It defaults to ‘*’. A typical value would be
‘@users’ to specify the ‘users’ group.
If a build log is not available locally, then ‘nix-store -l’ will now
try to download it from the servers listed in the ‘log-servers’ option
in nix.conf. For instance, if you have:
log-servers = http://hydra.nixos.org/log
then it will try to get logs from http://hydra.nixos.org/log/<base
name of the store path>. So you can do things like:
$ nix-store -l $(which xterm)
and get a log even if xterm wasn't built locally.
The option '--delete-generations Nd' deletes all generations older than N
days. However, most likely the user does not want to delete the
generation that was active N days ago.
For example, say that you have these 3 generations:
1: <30 days ago>
2: <15 days ago>
3: <1 hour ago>
If you do --delete-generations 7d (say, as part of a cron job), most
likely you still want to keep generation 2, i.e. the generation that was
active 7 days ago (and for most of the past 7 days, in fact).
This patch fixes this issue. Note that this also affects
'nix-collect-garbage --delete-older-than Nd'.
Thanks to @roconnor for noticing the issue!
This allows you to easily set up a build environment containing the
specified packages from Nixpkgs. For example:
$ nix-shell -p sqlite xorg.libX11 hello
will start a shell in which the given packages are present.
The tarball can now be unpacked anywhere. The installation script
uses "sudo" to create /nix if it doesn't exist. It also fetches the
nixpkgs-unstable channel.
This allows running nix-instantiate --eval-only without performing the
evaluation in readonly mode, letting features like import from
derivation and automatic substitution of builtins.storePath paths work.
Signed-off-by: Shea Levy <shea@shealevy.com>
Combined with the previous changes, stack traces involving derivations
are now much less verbose, since something like
while evaluating the builtin function `getAttr':
while evaluating the builtin function `derivationStrict':
while instantiating the derivation named `gtk+-2.24.20' at `/home/eelco/Dev/nixpkgs/pkgs/development/libraries/gtk+/2.x.nix:11:3':
while evaluating the derivation attribute `propagatedNativeBuildInputs' at `/home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/default.nix:78:17':
while evaluating the attribute `outPath' at `/nix/store/212ngf4ph63mp6p1np2bapkfikpakfv7-nix-1.6/share/nix/corepkgs/derivation.nix:18:9':
...
now reads
while evaluating the attribute `propagatedNativeBuildInputs' of the derivation `gtk+-2.24.20' at `/home/eelco/Dev/nixpkgs/pkgs/development/libraries/gtk+/2.x.nix:11:3':
...
This is equivalent to running ‘nix-env -e '*'’ first, except that it
happens in a single transaction. Thus, ‘nix-env -i pkgs...’ replaces
the profile with the specified set of packages.
The main motivation is to support declarative package management
(similar to environment.systemPackages in NixOS). That is, if you
have a specification ‘profile.nix’ like this:
with import <nixpkgs> {};
[ thunderbird
geeqie
...
]
then after any change to ‘profile.nix’, you can run:
$ nix-env -f profile.nix -ir
to update the profile to match the specification. (Without the ‘-r’
flag, if you remove a package from ‘profile.nix’, it won't be removed
from the actual profile.)
Suggested by @zefhemel.
This allows providing additional binary caches, useful in scripts like
Hydra's build reproduction scripts, in particular because untrusted
caches are ignored.
This should make live easier for single-user (non-daemon)
installations. Note that when the daemon is used, the "calling user"
is root so we're not using any untrusted caches.
This flag causes paths that do not have a known substitute to be
quietly ignored. This is mostly useful for Charon, allowing it to
speed up deployment by letting a machine use substitutes for all
substitutable paths, instead of uploading them. The latter is
frequently faster, e.g. if the target machine has a fast Internet
connection while the source machine is on a slow ADSL line.
Binary caches can now specify a priority in their nix-cache-info file.
The binary cache substituter checks caches in order of priority. This
is to ensure that fast, static caches like nixos.org/binary-cache are
processed before slow, dynamic caches like hydra.nixos.org.
This allows disabling the use of binary caches, e.g.
$ nix-build ... --option use-binary-caches false
Note that
$ nix-build ... --option binary-caches ''
does not disable all binary caches, since the caches defined by
channels will still be used.
This operation allows fixing corrupted or accidentally deleted store
paths by redownloading them using substituters, if available.
Since the corrupted path cannot be replaced atomically, there is a
very small time window (one system call) during which neither the old
(corrupted) nor the new (repaired) contents are available. So
repairing should be used with some care on critical packages like
Glibc.
In Nixpkgs, the attribute in all-packages.nix corresponding to a
package is usually equal to the package name. However, this doesn't
work if the package contains a dash, which is fairly common. The
convention is to replace the dash with an underscore (e.g. "dbus-lib"
becomes "dbus_glib"), but that's annoying. So now dashes are valid in
variable / attribute names, allowing you to write:
dbus-glib = callPackage ../development/libraries/dbus-glib { };
and
buildInputs = [ dbus-glib ];
Since we don't have a negation or subtraction operation in Nix, this
is unambiguous.
Channels can now advertise a binary cache by creating a file
<channel-url>/binary-cache-url. The channel unpacker puts these in
its "binary-caches" subdirectory. Thus, the URLS of the binary caches
for the channels added by root appear in
/nix/var/nix/profiles/per-user/eelco/channels/binary-caches/*. The
binary cache substituter reads these and adds them to the list of
binary caches.
Mandatory features are features that MUST be present in a derivation's
requiredSystemFeatures attribute. One application is performance
testing, where we have a dedicated machine to run performance tests
(and nothing else). Then we would add the label "perf" to the
machine's mandatory features and to the performance testing
derivations.
"nix-channel --add" now accepts a second argument: the channel name.
This allows channels to have a nicer name than (say) nixpkgs_unstable.
If no name is given, it defaults to the last component of the URL
(with "-unstable" or "-stable" removed).
Also, channels are now stored in a profile
(/nix/var/nix/profiles/per-user/$USER/channels). One advantage of
this is that it allows rollbacks (e.g. if "nix-channel --update" gives
an undesirable update).
Nix now requires SQLite and bzip2 to be pre-installed. SQLite is
detected using pkg-config. We required DBD::SQLite anyway, so
depending on SQLite is not a big problem.
The --with-bzip2, --with-openssl and --with-sqlite flags are gone.
environment of the given derivation in a format that can be sourced
by the shell, e.g.
$ eval "$(nix-store --print-env $(nix-instantiate /etc/nixos/nixpkgs -A pkg))"
$ NIX_BUILD_TOP=/tmp
$ source $stdenv/setup
This is especially useful to reproduce the environment used to build
a package outside of its builder for development purposes.
TODO: add a nix-build option to do the above and fetch the
dependencies of the derivation as well.
the contents of any of the given store paths have been modified.
E.g.
$ nix-store --verify-path $(nix-store -qR /var/run/current-system)
path `/nix/store/m2smyiwbxidlprfxfz4rjlvz2c3mg58y-etc' was modified! expected hash `fc87e271c5fdf179b47939b08ad13440493805584b35e3014109d04d8436e7b8', got `20f1a47281b3c0cbe299ce47ad5ca7340b20ab34246426915fce0ee9116483aa'
All paths are checked; the exit code is 1 if any path has been
modified, 0 otherwise.
brackets, e.g.
import <nixpkgs/pkgs/lib>
are resolved by looking them up relative to the elements listed in
the search path. This allows us to get rid of hacks like
import "${builtins.getEnv "NIXPKGS_ALL"}/pkgs/lib"
The search path can be specified through the ‘-I’ command-line flag
and through the colon-separated ‘NIX_PATH’ environment variable,
e.g.,
$ nix-build -I /etc/nixos ...
If a file is not found in the search path, an error message is
lazily thrown.
(Linux) machines no longer maintain the atime because it's too
expensive, and on the machines where --use-atime is useful (like the
buildfarm), reading the atimes on the entire Nix store takes way too
much time to make it practical.
NixOS evaluation errors in particular look intimidating and
generally aren't very useful. Ideally the builtins.throw messages
should be self-contained.
UTC) rather than 0 (00:00:00). 1 is a better choice because some
programs use 0 as a special value. For instance, the Template
Toolkit uses a timestamp of 0 to denote the non-existence of a file,
so it barfs on files in the Nix store (see
template-toolkit-nix-store.patch in Nixpkgs). Similarly, Maya 2008
fails to load script directories with a timestamp of 0 and can't be
patched because it's closed source.
This will also shut up those "implausibly old time stamp" GNU tar
warnings.
Scott Olson
Bob van der Linden