Commit graph

7418 commits

Author SHA1 Message Date
Eelco Dolstra 670feb000a Add 'path' fetcher
This fetchers copies a plain directory (i.e. not a Git/Mercurial
repository) to the store (or does nothing if the path is already a
store path).

One use case is to pin the 'nixpkgs' flake used to build the current
NixOS system, and prevent it from being garbage-collected, via a
system registry entry like this:

  {
      "from": {
          "id": "nixpkgs",
          "type": "indirect"
      },
      "to": {
          "type": "path",
          "path": "/nix/store/rralhl3wj4rdwzjn16g7d93mibvlr521-source",
          "lastModified": 1585388205,
          "rev": "b0c285807d6a9f1b7562ec417c24fa1a30ecc31a"
      },
      "exact": true
  }

Note the fake "lastModified" and "rev" attributes that ensure that the
flake gives the same evaluation results as the corresponding
Git/GitHub inputs.

(cherry picked from commit 12f9379123)
2020-04-07 09:08:51 +02:00
Eelco Dolstra 462421d345 Backport libfetchers from the flakes branch
This provides a pluggable mechanism for defining new fetchers. It adds
a builtin function 'fetchTree' that generalizes existing fetchers like
'fetchGit', 'fetchMercurial' and 'fetchTarball'. 'fetchTree' takes a
set of attributes, e.g.

  fetchTree {
    type = "git";
    url = "https://example.org/repo.git";
    ref = "some-branch";
    rev = "abcdef...";
  }

The existing fetchers are just wrappers around this. Note that the
input attributes to fetchTree are the same as flake input
specifications and flake lock file entries.

All fetchers share a common cache stored in
~/.cache/nix/fetcher-cache-v1.sqlite. This replaces the ad hoc caching
mechanisms in fetchGit and download.cc (e.g. ~/.cache/nix/{tarballs,git-revs*}).

This also adds support for Git worktrees (c169ea5904).
2020-04-07 09:03:14 +02:00
Eelco Dolstra ebb20a5356
Merge pull request #3474 from cole-h/error-on-unsupported-protocol
Don't retry on "unsupported protocol" error
2020-04-06 09:07:13 +02:00
Cole Helbling c976cb0b8a
Don't retry on "unsupported protocol" error
When encountering an unsupported protocol, there's no need to retry.
Chances are, it won't suddenly be supported between retry attempts;
error instead. Otherwise, you see something like the following:

    $ nix-env -i -f git://git@github.com/foo/bar
    warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 335 ms
    warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 604 ms
    warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 1340 ms
    warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 2685 ms

With this change, you now see:

    $ nix-env -i -f git://git@github.com/foo/bar
    error: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1)
2020-04-05 09:00:34 -07:00
Eelco Dolstra 5e7ccdc9e3 Publish a tarball containing the crates we depend on
This is needed since we no longer produce a source tarball.

(cherry picked from commit bf70a047a0)
2020-04-03 20:14:34 +02:00
Eelco Dolstra 63fa92605b nix-env: Refuse to operate on a new-style profile
This prevents users from accidentally nuking their profile via
nix-env.

(cherry picked from commit 021634e3e3)
2020-04-03 20:14:34 +02:00
Eelco Dolstra 47c568ee32
Merge pull request #3470 from andir/remove-unusued-attrError
libexpr: remove unused attrError
2020-04-03 09:36:13 +02:00
Andreas Rammhold 4fc4eb6c93 libexpr: remove unused attrError
The attrError variable is no longer used but still allocated on every
call to the findAlongAttrPath function.
2020-04-02 17:04:00 +02:00
Silvan Mosberger c34e96f7e0
Make function arguments retain position info
This allows querying the location of function arguments. E.g.

  builtins.unsafeGetAttrPos "x" (builtins.functionArgs ({ x }: null))

  => { column = 57; file = "/home/infinisil/src/nix/inst/test.nix"; line = 1; }
2020-04-02 05:52:52 +02:00
Eelco Dolstra a7540294cf
Merge pull request #3460 from NixOS/dev-shell
Backport 'nix dev-shell' from the flakes branch
2020-03-31 14:46:15 +02:00
Eelco Dolstra d4d456c6b1
Merge pull request #3463 from Ninlives/placeholder-passAsFile
fix placeholder not substituted in passAsFile
2020-03-31 13:50:39 +02:00
Eelco Dolstra 3166b97174 nix shell -> nix dev-shell 2020-03-31 13:45:28 +02:00
mlatus 12556e5709 fix placeholder not substituted in passAsFile 2020-03-31 19:40:16 +08:00
Eelco Dolstra e1a94ad852 Backport 'nix dev-shell' from the flakes branch
This also adds a '--profile' option to 'nix build' (replacing 'nix-env
--set').
2020-03-30 19:16:45 +02:00
Eelco Dolstra 367577d9a6 Fix macOS build 2020-03-30 17:00:40 +02:00
Eelco Dolstra e322a16523 Remove global -I flags
(cherry picked from commit 2c692a3b14)
2020-03-30 15:30:19 +02:00
Julian Stecklina 40c023ecfe fetchGit: don't use std::filesystem to filter git repos
Using std::filesystem means also having to link with -lstdc++fs on
some platforms and it's hard to discover for what platforms this is
needed. As all the functionality is already implemented as utilities,
use those instead.
2020-03-30 00:32:42 +02:00
Bjørn Forsman f686efeed4 fetchGit: fix submodule corner case by fetching all refs from cacheDir
Due to fetchGit not checking if rev is an ancestor of ref (there is even
a FIXME comment about it in the code), the cache repo might not have the
ref even though it has the rev. This doesn't matter when submodule =
false, but the submodule = true code blows up because it tries to fetch
the (missing) ref from the cache repo.

Fix this in the simplest way possible: fetch all refs from the local
cache repo when submodules = true.

TODO: Add tests.
2020-03-29 22:29:58 +02:00
Bjørn Forsman cc522d0d23 fetchGit: fix submodules = true for dirty trees 2020-03-29 22:29:58 +02:00
Bjørn Forsman b306b7039e fetchGit: checkout rev instead of latest ref
Major bugfix for the submodules = true code path.

TODO: Add tests.
2020-03-29 22:29:58 +02:00
Bjørn Forsman be84049baf tests/fetchGitSubmodules.sh: more checks 2020-03-29 22:29:58 +02:00
Bjørn Forsman 369fffd6f1 fetchGit: add submodules attribute to the .link file
The .link file is used as a lock, so I think we should put the
"submodule" attribute in there since turning on submodules creates a new
.link file path.
2020-03-29 22:29:58 +02:00
Bjørn Forsman 6864ad7cf5 fetchGit: fix submodule output attribute
Before this change it would be false for all evaluations but the first.
Now it follows the input argument (as it should).
2020-03-29 22:29:58 +02:00
Bjørn Forsman 587e259bfd tests/fetchGitSubmodules: add more tests 2020-03-29 22:29:58 +02:00
Bjørn Forsman 002a3a95dc fetchGit: fix "fatal: couldn't find remote ref refs/heads/master" issue with submodules 2020-03-29 22:29:58 +02:00
Julian Stecklina cc4fe977e5 Link to stdc++fs
Some platforms seem to still require linking with stdc++fs to enable
STL std::filesystem support.
2020-03-29 22:29:58 +02:00
Julian Stecklina 6c00a9545f Fix typo in submodule test 2020-03-29 22:29:58 +02:00
Julian Stecklina 435366ed3c Rename fetchGit fetchSubmodules to just submodules 2020-03-29 22:29:58 +02:00
Julian Stecklina c846abb5cc Add more test for git submodule functionality 2020-03-29 22:29:58 +02:00
Julian Stecklina c8d33de777 Add git submodule fixes from @bjornfor
This fixes fetching repositories with no submodules and also cleans up
.git files in checkouts.
2020-03-29 22:29:57 +02:00
Julian Stecklina ea861be292 Add documentation for submodule support in fetchGit 2020-03-29 22:29:57 +02:00
Julian Stecklina c2a24c2b88 Add test for fetchGit submodule support 2020-03-29 22:29:57 +02:00
Julian Stecklina f58604ac32 Add fetchSubmodules to builtins.fetchGit
There are some downsides to this features:

 - Submodules are not cached (unlike the root repo),
 - Full checkouts are created in a temporary directory.
2020-03-29 22:29:57 +02:00
Eelco Dolstra eb1911e277
Merge pull request #3445 from gnprice/pr-installer-colors
installer: Fix terminal colors.
2020-03-25 09:04:24 +01:00
Greg Price 7313aa267b installer: Fix terminal colors.
The install-multi-user script uses blue, green, and red colors, as
well as bold and underline, to add helpful formatting that helps
structure its rather voluminous output.

Unfortunately, the terminal escape sequences it uses are not quite
well-formed.  The relevant information is all there, just obscured
by some extra noise, a leading parameter `38`.  Empirically, the
result is:

 * On macOS, in both Terminal.app and iTerm2, the spurious `38` is
   ignored, the rest of the escape sequence is applied, and the colors
   show up as intended.

 * On Linux, in at least gnome-terminal and xterm, the spurious `38`
   and the next parameter after it are ignored, and what's left is
   applied.  So in the sequence `38;4;32`, the 4 (underline) is
   ignored but the 32 (green) takes effect; in a more typical sequence
   like `38;34`, the 34 (blue) is ignored and nothing happens.

These codes are all unchanged since this script's origins as a
Darwin-only script -- so the fact that they work fine in common macOS
terminals goes some way to explain how the bug arose.

Happily, we can make the colors work as intended by just deleting the
extra `38;`.  Tested in all four terminals mentioned above; the new
codes work correctly on all of them, and on the two macOS terminals
they work exactly the same as before.

---

In a bit more technical detail -- perhaps more than anyone, me
included, ever wanted to know, but now that I've gone and learned it
I'll write it down anyway :) -- here's what's happening in these codes:

An ECMA-48 "control sequence" begins with `\033[` aka "CSI", contains
any number of parameters as semicolon-separated decimal numbers (plus
sometimes other wrinkles), and ends with a byte from 0x40..0x7e.  In
our case, with `m` aka "SGR", "Select Graphic Rendition".

An SGR control sequence `\033[...m` sets colors, fonts, text styles,
etc.  In particular a parameter `31` means red, `32` green, `34` blue,
`4` underline, and `0` means reset to normal.  Those are all we use.

There is also a `38`.  This is used for setting colors too... but it
needs arguments.  `38;5;nn` is color nn from a 256-color palette, and
`38;2;rr;gg;bb` has the given RGB values.

There is no meaning defined for `38;1` or `38;34` etc.  On seeing a
parameter `38` followed by an unrecognized argument for it, apparently
some implementations (as seen on macOS) discard only the `38` and
others (as seen on Linux) discard the argument too before resuming.
2020-03-24 21:15:01 -07:00
Eelco Dolstra 0a10854f85 Misc changes from the flakes branch 2020-03-24 14:34:47 +01:00
Eelco Dolstra c85097da7c Fix --refresh with --no-net
https://hydra.nixos.org/build/110879699
(cherry picked from commit 5bbe793abf)
2020-03-24 14:26:23 +01:00
Eelco Dolstra 6b824c78f1 nix: Add --refresh as an alias for --tarball-ttl 0
(cherry picked from commit e721f99817)
2020-03-24 14:26:23 +01:00
Eelco Dolstra 777e21e596 nix path-info --json: Print hash in SRI format
(cherry picked from commit 442e665d6d)
2020-03-24 14:26:23 +01:00
Eelco Dolstra 7a8de57d3e Pretty-print 'nix why-depends' / 'nix-store -q --tree' output
Extracted from 678301072f.
2020-03-24 14:26:23 +01:00
Eelco Dolstra 4260a22a55 absPath(): Use std::optional
(cherry picked from commit 1bf9eb21b7)
2020-03-24 14:25:28 +01:00
Eelco Dolstra f9611c7ae4 buildenv: Eliminate global variables, other cleanup
(cherry picked from commit b82f75464d)
2020-03-24 14:06:47 +01:00
Eelco Dolstra 76e7d958ed Fix coverage build
https://hydra.nixos.org/build/110757285
(cherry picked from commit b430a81a1f)
2020-03-24 14:06:47 +01:00
Eelco Dolstra 231a8aa2c2 nix edit: Support non-derivation attributes
E.g.

  $ nix edit .#nixosConfigurations.bla

now works.

(cherry picked from commit d2032edb2f)
2020-03-24 14:06:47 +01:00
Eelco Dolstra c1ca4f0acc findAlongAttrPath(): Return position
(cherry picked from commit 0b013a54dc)
2020-03-24 14:06:47 +01:00
Eelco Dolstra 1eb952d27a findAlongAttrPath(): Throw AttrPathNotFound
(cherry picked from commit 6b0ca8e803)
2020-03-24 14:06:47 +01:00
Eelco Dolstra edc34cc1a2 Add function for quoting strings
(cherry picked from commit 7dcf5b011a)
2020-03-24 13:44:04 +01:00
Eelco Dolstra 5a7e7fc35f Use std::string_view
(cherry picked from commit 6529490cc1)
2020-03-24 13:26:37 +01:00
Eelco Dolstra c34a20e1f6 EvalState::allocAttr(): Add convenience method
(cherry picked from commit c02da99757)
2020-03-24 13:26:37 +01:00
Domen Kožar ddc6aaa8b2
Merge pull request #3441 from gnprice/pr-doc-store-ro
doc: Files in the store have modes 444/555, not 644/755
2020-03-24 10:26:50 +01:00