Commit graph

6484 commits

Author SHA1 Message Date
Matthew Bauer 3d90ab9345 Fix extra slash in canonPath output
When you have a symlink like:

  /tmp -> ./private/tmp

you need to resolve ./private/tmp relative to /tmp’s dir: ‘/’. Unlike
any other path output by dirOf, / ends with a slash. We don’t want
trailing slashes here since we will append another slash in the next
comoponent, so clear s like we would if it was a symlink to an absoute
path.

This should fix at least part of the issue in
https://github.com/NixOS/nix/issues/4822, will need confirmation that
it actually fixes the problem to close though.

Introduced in f3f228700a.
2021-05-18 16:38:55 -05:00
Eelco Dolstra bd6cf25952
Merge pull request #4819 from NixOS/ca/fix-nix-run
Resolve the program path in `nix run`
2021-05-17 16:28:14 +02:00
Eelco Dolstra 6849ae82de
Merge pull request #4818 from NixOS/ca/cli-use-builtpaths
Enforce the use of properly built paths in libcmd
2021-05-17 16:15:40 +02:00
regnat f0c0052d4b Resolve the program path in nix run
Fix #4768
2021-05-17 09:48:51 +02:00
regnat 2105084645 Enfore the use of properly built paths in libcmd
Replace `DerivedPathWithHints` by a new `BuiltPath` type that serves as
a proof that the corresponding path has been built.
2021-05-17 08:45:08 +02:00
Emily 559a504da7 sandbox: allow SystemVersionCompat.plist on Darwin
For whatever reason, many programs trying to access SystemVersion.plist
also open SystemVersionCompat.plist; this includes Python code and
coreutils’ `cat(1)` (but not the native macOS `/bin/cat`). Illustratory
`dtruss(1m)` output:

    open("/System/Library/CoreServices/SystemVersion.plist\0", 0x0, 0x0)		 = 3 0
    open("/System/Library/CoreServices/SystemVersionCompat.plist\0", 0x0, 0x0)		 = 4 0

I assume this is a Big Sur change relating to the 10.16.x/11.x
version compatibility divide and that it’s something along the lines of
a hook inside libSystem.

Fixes a lot of sandboxed package builds under Big Sur.
2021-05-17 01:24:31 +01:00
Eelco Dolstra 340f831ebe
Merge pull request #4798 from matthewbauer/relock-wait-for-build-slot-goals
Relock wait for build slot goals
2021-05-13 13:07:35 +02:00
Eelco Dolstra 89a4ede92b
Merge pull request #4801 from matthewbauer/fix-tokenize-output-names
Fix tokenize output names in DerivedPath
2021-05-13 13:06:29 +02:00
Matthew Bauer 8c7e043de2 Fix tokenize output names in drv
This should fix the issue described in
https://discourse.nixos.org/t/derivation-does-not-have-wanted-outputs-dev-out/12905.

Specifically, we get an error of

  error: derivation '/nix/store/_.drv' does not have wanted outputs 'dev,out'

when a path like /nix/store/_.drv!dev,out is sent to the daemon.
2021-05-12 21:40:28 -05:00
Matthew Bauer 9a14335845 Relock wait for build slot goals
When we don’t have enough free job slots to run a goal, we put it in
the waitForBuildSlot list & unlock its output locks. This will
continue from where we left off (tryLocalBuild). However, we need the
locks to get reacquired when/if the goal ever restarts. So, we need to
send it back through tryToBuild to get reqacquire those locks.

I think this bug was introduced in
https://github.com/NixOS/nix/pull/4570. It leads to some builds
starting without proper locks.
2021-05-12 12:15:32 -05:00
regnat ec613603ba DerivedPathWithHints -> BuiltPath
Just a renaming for now
2021-05-12 16:19:51 +02:00
Lorenzo Manacorda 4029f4b05b doc: mention Priority for substituters 2021-05-12 13:27:05 +02:00
Eelco Dolstra 7f9759b18d
Merge pull request #4781 from NixOS/locally_cache_the_remote_realisations
Add a realisations disk cache
2021-05-10 20:37:57 +02:00
Eelco Dolstra 8768398d5f
Merge pull request #4787 from Ma27/builtins-ceil-floor
Implement `builtins.floor` and `builtins.ceil` using the C library functions internally
2021-05-10 20:37:27 +02:00
regnat d5d19582ef Simplify the realisations disk cache 2021-05-10 17:47:14 +02:00
regnat ab96c1ee50 Remove useless parents
I never remember the exact syntax of the `switch` statement
2021-05-10 17:36:49 +02:00
Maximilian Bosch bec33858bb
primops: math.h -> cmath
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-05-10 16:41:10 +02:00
Eelco Dolstra b68a85df0a
Merge pull request #4776 from NixOS/fix-ca-normalization
Properly normalize the content-addressed paths
2021-05-10 16:11:38 +02:00
Maximilian Bosch 7f7f99f350
Implement builtins.floor and builtins.ceil using the C library functions internally
Closes #4782

Note: even though the type is internally called `NixFloat`, it's
actually a `double`.
2021-05-10 12:19:32 +02:00
Eelco Dolstra 2bb49fb6d0
Merge pull request #4786 from matthewbauer/mkdir-local-share-nix
Create parent for trusted list path before writing
2021-05-10 11:14:45 +02:00
Matthew Bauer 2f63cc02de Create parent trusted list path before writing
This makes sure that $HOME/.local/share/nix exists before we try to
write to it.
2021-05-08 21:31:28 -05:00
regnat b66234134f Add a realisations disk cache
Similar to the nar-info disk cache (and using the same db).
This makes rebuilds muuch faster.

- This works regardless of the ca-derivations experimental feature.
  I could modify the logic to not touch the db if the flag isn’t there,
  but given that this is a trash-able local cache, it doesn’t seem to be
  really worth it.
- We could unify the `NARs` and `Realisation` tables to only have one
  generic kv table. This is left as an exercise to the reader.
- I didn’t update the cache db version number as the new schema just
  adds a new table to the previous one, so the db will be transparently
  migrated and is backwards-compatible.

Fix #4746
2021-05-06 17:38:20 +02:00
Matthew Bauer 1e3a9033b7 Use derivation output name from toDerivation
This was previously done in https://github.com/NixOS/nix/pull/4515 but
got clobbered away in https://github.com/NixOS/nix/pull/4594.

--------------------------------------------------------------------------------

This fixes an issue where derivations with a primary output that is
not "out" would fail with:

$ nix profile install nixpkgs#sqlite
error: opening directory '/nix/store/2a2ydlgyydly5czcc8lg12n6qqkfz863-sqlite-3.34.1-bin': No such file or directory

This happens because while derivations produce every output when
built, you might not have them if you didn't build the derivation
yourself (for instance, the store path was fetch from a binary cache).
This uses outputName provided from DerivationInfo which appears to
match the first output of the derivation.
2021-05-05 16:33:05 -05:00
regnat bf485dcf46 Properly normalize the content-addressed paths
Make sure that their timestamp are always normalized.
Otherwise, strange − and non-deterministic − things might happen, like
https://github.com/NixOS/nixpkgs/issues/121813

Fix #4775
2021-05-05 21:00:08 +02:00
regnat dadfbce318 Fix the double-slash in the realisations path
Make sure that we always access the realisations under
`binaryCacheUrl/realisations` and not `binaryCacheUrl//realisations`

Fix #4766
2021-05-04 10:35:34 +02:00
Eelco Dolstra ef13c9c223
Merge pull request #4750 from NinjaTrappeur/nin-build-dry-run-json
nix build: make dry-run to print a json output if --json is enabled
2021-05-03 16:14:29 +02:00
Eelco Dolstra 40378fbcba
Merge pull request #4761 from emilazy/issue-4658-mark-impure-host-deps-as-optional
Mark `__impureHostDeps` paths as optional
2021-05-03 16:13:36 +02:00
Maximilian Bosch e5951a6b2f
Bump version number for DerivedPath changes
I guess I misunderstood John's initial explanation about why wildcards
for outputs are sent to older stores[1]. My `nix-daemon` from 2021-03-26
also has version 1.29, but misses the wildcard[2]. So bumping seems to
be the right call.

[1] https://github.com/NixOS/nix/pull/4759#issuecomment-830812464
[2] 255d145ba7
2021-05-03 01:12:23 +02:00
Emily c4355a52fa Mark __impureHostDeps paths as optional
Starting in macOS 11, the on-disk dylib bundles are no longer available,
but nixpkgs needs to be able to keep compatibility with older versions
that require `/usr/lib/libSystem.B.dylib` in `__impureHostDeps`. Allow
it to keep backwards compatibility with these versions by marking these
dependencies as optional.

Fixes #4658.
2021-05-02 05:30:50 +01:00
Domen Kožar e3e78ee2a2
Merge pull request #4751 from Ma27/storepath-pos
primops/storePath: add trace to pure mode error
2021-04-27 19:05:23 +02:00
Maximilian Bosch e6a360a4d9
primops/storePath: add trace to pure mode error
As described in #4745 it's otherwise fairly hard to understand where
this is coming from. Say you have an expression which uses e.g.
`types.package`:

``` nix
{ outputs = { self, nixpkgs }: {
    packages.x86_64-linux.hello = let
      foo = nixpkgs.lib.evalModules {
        modules = [
          {
            options.foo.bar = with nixpkgs.lib; mkOption { type = types.package; };
          }
          {
            foo.bar = ./.;
          }
        ];
      };
    in builtins.trace foo.config.foo.bar.outPath nixpkgs.legacyPackages.x86_64-linux.hello;
    defaultPackage.x86_64-linux = self.packages.x86_64-linux.hello;
  };
}
```

Then you'll get an error trace like this:

```
error: 'builtins.storePath' is not allowed in pure evaluation mode

       at /nix/store/p4h2x6r80njkb0j2rc1xjhhl99yri3zb-source/lib/attrsets.nix:328:15:

          327|     let
          328|       path' = builtins.storePath path;
             |               ^
          329|       res =

       … while evaluating the attribute 'config.foo.bar.outPath'

       at /nix/store/p4h2x6r80njkb0j2rc1xjhhl99yri3zb-source/lib/attrsets.nix:332:11:

          331|           name = sanitizeDerivationName (builtins.substring 33 (-1) (baseNameOf path'));
          332|           outPath = path';
             |           ^
          333|           outputs = [ "out" ];

       … while evaluating the attribute 'packages.x86_64-linux.hello'

       at /nix/store/6c1rfsqzrhjw1235palzjmf5vihcpci7-source/flake.nix:3:5:

            2| { outputs = { self, nixpkgs }: {
            3|     packages.x86_64-linux.hello = let
             |     ^
            4|       foo = nixpkgs.lib.evalModules {
```

Fixes #4745
2021-04-27 17:24:38 +02:00
Félix Baylac-Jacqué 7dca9c24c4
nix build: make dry-run to print a json output if --json is enabled 2021-04-27 11:28:47 +02:00
Eelco Dolstra fe2bf464cf Merge branch 'remove-trailing-spaces' of github.com:NixOS/nix 2021-04-23 15:09:44 +02:00
regnat 31313d1401 Replace the trailing markdown spaces by a backslash
They are equivalent according to
<https://spec.commonmark.org/0.29/#hard-line-breaks>,
and the trailing spaces tend to be a pain (because the make git
complain, editors tend to want to remove them − the `.editorconfig`
actually specifies that − etc..).
2021-04-23 14:37:21 +02:00
Eelco Dolstra 4549d65b4f
Merge pull request #4732 from NixOS/4725-always-register-the-realisations
Aways register the realisations
2021-04-23 14:22:03 +02:00
Eelco Dolstra 293220bed5
Merge pull request #4440 from Ma27/misc-pos-fixes
Miscellaneous improvements for positioning in eval-errors
2021-04-23 11:10:59 +02:00
regnat d3df747cb6 Revert "Make nix shell fallback to static outputs when needed"
This reverts commit 8d66f5f110.

This fix isn't needed anymore now that the realisations are always
properly registered
2021-04-23 09:34:43 +02:00
regnat 9161e02039 Always register the realisations of input-addressed drvs
Fix #4725
2021-04-22 20:07:02 +02:00
Eelco Dolstra d9864be4b7 Merge branch 'master' of github.com:NixOS/nix 2021-04-22 10:33:00 +02:00
Eelco Dolstra 78a2303b3d Merge branch 'nix-derivation-query-store-path' of https://github.com/matthewbauer/nix 2021-04-22 10:29:18 +02:00
Eelco Dolstra dd83b29b90
Merge pull request #4724 from NixOS/nix-shell-missing-outputs
Make `nix shell` fallback to static outputs when needed
2021-04-22 10:25:55 +02:00
Eelco Dolstra 5385755227 Remove makeDecompressionSource()
This function doesn't support all compression methods (i.e. 'none' and
'br') so it shouldn't be exposed.

Also restore the original decompress() as a wrapper around
makeDecompressionSink().
2021-04-22 10:23:20 +02:00
Eelco Dolstra a6eebcff36 Merge branch 's3-decompress' of https://github.com/lukegb/nix 2021-04-22 10:20:40 +02:00
Luke Granger-Brown 97dde3cdd9 libutil: allow decompression with none/empty method
The S3 store relies on the ability to be able to decompress things with
an empty method, because it just passes the value of the Content-Encoding
directly to decompress.

If the file is not compressed, then this will cause the compression
routine to get confused.

This caused NixOS/nixpkgs#120120.
2021-04-22 02:33:59 +00:00
Antoine Martin ac38c4cdc2 Fix typo in flake doc 2021-04-22 03:07:57 +02:00
Matthew Bauer 428062e578 Get deriver when passing --derivation
This makes Nix look up paths derivations when they are passed as a
store paths. So:

$ nix path-info --derivation /nix/store/0pisd259nldh8yfjvw663mspm60cn5v8-hello-2.10

now gives

/nix/store/qp724w90516n4bk5r9gfb37vzmqdh3z7-hello-2.10.drv

instead of "".

If no deriver is available, Nix now errors instead of silently
ignoring that argument.
2021-04-21 15:32:33 -05:00
regnat 8d66f5f110 Make nix shell fallback to static outputs when needed
In case of pure input-addressed derivations, the build loop doesn't
guaranty that the realisations are stored in the db (if the output paths
are already there or can be substituted, the realisations won't be
registered). This caused `nix shell` to fail in some cases because it
was assuming that the realisations were always existing.

A better (but more involved) fix would probably to ensure that we always
register the realisations, but in the meantime this patches the surface
issue.

Fix #4721
2021-04-21 17:46:27 +02:00
Eelco Dolstra 8d651a1f68 Fix brotli compression of files > 128 KiB
This has been broken since faa31f4084.
2021-04-20 22:54:49 +02:00
Eelco Dolstra b60b0d62d6
Merge pull request #4720 from alyssais/wait
Include sys/wait.h everywhere WIFEXITED etc is used
2021-04-20 16:09:30 +02:00
Alyssa Ross 9ac6534f7c
Include sys/wait.h everywhere WIFEXITED etc is used
This is required on NetBSD, and I think FreeBSD too.
2021-04-19 18:31:58 +00:00