Commit graph

74 commits

Author SHA1 Message Date
Eelco Dolstra 55cefd41d6 Merge branch 'fetchgit-recursive' of https://github.com/blitz/nix 2020-04-07 13:45:17 +02:00
Eelco Dolstra 462421d345 Backport libfetchers from the flakes branch
This provides a pluggable mechanism for defining new fetchers. It adds
a builtin function 'fetchTree' that generalizes existing fetchers like
'fetchGit', 'fetchMercurial' and 'fetchTarball'. 'fetchTree' takes a
set of attributes, e.g.

  fetchTree {
    type = "git";
    url = "https://example.org/repo.git";
    ref = "some-branch";
    rev = "abcdef...";
  }

The existing fetchers are just wrappers around this. Note that the
input attributes to fetchTree are the same as flake input
specifications and flake lock file entries.

All fetchers share a common cache stored in
~/.cache/nix/fetcher-cache-v1.sqlite. This replaces the ad hoc caching
mechanisms in fetchGit and download.cc (e.g. ~/.cache/nix/{tarballs,git-revs*}).

This also adds support for Git worktrees (c169ea5904).
2020-04-07 09:03:14 +02:00
Eelco Dolstra e322a16523 Remove global -I flags
(cherry picked from commit 2c692a3b14)
2020-03-30 15:30:19 +02:00
Julian Stecklina 40c023ecfe fetchGit: don't use std::filesystem to filter git repos
Using std::filesystem means also having to link with -lstdc++fs on
some platforms and it's hard to discover for what platforms this is
needed. As all the functionality is already implemented as utilities,
use those instead.
2020-03-30 00:32:42 +02:00
Bjørn Forsman f686efeed4 fetchGit: fix submodule corner case by fetching all refs from cacheDir
Due to fetchGit not checking if rev is an ancestor of ref (there is even
a FIXME comment about it in the code), the cache repo might not have the
ref even though it has the rev. This doesn't matter when submodule =
false, but the submodule = true code blows up because it tries to fetch
the (missing) ref from the cache repo.

Fix this in the simplest way possible: fetch all refs from the local
cache repo when submodules = true.

TODO: Add tests.
2020-03-29 22:29:58 +02:00
Bjørn Forsman cc522d0d23 fetchGit: fix submodules = true for dirty trees 2020-03-29 22:29:58 +02:00
Bjørn Forsman b306b7039e fetchGit: checkout rev instead of latest ref
Major bugfix for the submodules = true code path.

TODO: Add tests.
2020-03-29 22:29:58 +02:00
Bjørn Forsman 369fffd6f1 fetchGit: add submodules attribute to the .link file
The .link file is used as a lock, so I think we should put the
"submodule" attribute in there since turning on submodules creates a new
.link file path.
2020-03-29 22:29:58 +02:00
Bjørn Forsman 6864ad7cf5 fetchGit: fix submodule output attribute
Before this change it would be false for all evaluations but the first.
Now it follows the input argument (as it should).
2020-03-29 22:29:58 +02:00
Bjørn Forsman 002a3a95dc fetchGit: fix "fatal: couldn't find remote ref refs/heads/master" issue with submodules 2020-03-29 22:29:58 +02:00
Julian Stecklina 435366ed3c Rename fetchGit fetchSubmodules to just submodules 2020-03-29 22:29:58 +02:00
Julian Stecklina c8d33de777 Add git submodule fixes from @bjornfor
This fixes fetching repositories with no submodules and also cleans up
.git files in checkouts.
2020-03-29 22:29:57 +02:00
Julian Stecklina f58604ac32 Add fetchSubmodules to builtins.fetchGit
There are some downsides to this features:

 - Submodules are not cached (unlike the root repo),
 - Full checkouts are created in a temporary directory.
2020-03-29 22:29:57 +02:00
Eelco Dolstra bbe97dff8b Make the Store API more type-safe
Most functions now take a StorePath argument rather than a Path (which
is just an alias for std::string). The StorePath constructor ensures
that the path is syntactically correct (i.e. it looks like
<store-dir>/<base32-hash>-<name>). Similarly, functions like
buildPaths() now take a StorePathWithOutputs, rather than abusing Path
by adding a '!<outputs>' suffix.

Note that the StorePath type is implemented in Rust. This involves
some hackery to allow Rust values to be used directly in C++, via a
helper type whose destructor calls the Rust type's drop()
function. The main issue is the dynamic nature of C++ move semantics:
after we have moved a Rust value, we should not call the drop function
on the original value. So when we move a value, we set the original
value to bitwise zero, and the destructor only calls drop() if the
value is not bitwise zero. This should be sufficient for most types.

Also lots of minor cleanups to the C++ API to make it more modern
(e.g. using std::optional and std::string_view in some places).
2019-12-10 22:06:05 +01:00
Eelco Dolstra 8918bae098 Drop remaining uses of external "tar"
Also, fetchGit now runs in O(1) memory since we pipe the output of
'git archive' directly into unpackTarball() (rather than first reading
it all into memory).
2019-11-26 22:07:28 +01:00
Benjamin Hipple 80d5ec6ff4 Minor updates to inline comments
Add missing docstring on InstallableCommand. Also, some of these were wrapped
when they're right next to a line longer than the unwrapped line, so we can just
unwrap them to save vertical space.
2019-10-31 05:56:37 -04:00
Eelco Dolstra bd79c1f6f6 Don't catch exceptions by value
(cherry picked from commit 893be6f5e3)
2019-09-22 21:56:56 +02:00
Eelco Dolstra 33db1d35ae
Merge pull request #2582 from LnL7/fetchgit-refs
fetchGit: allow fetching explicit refs
2019-07-02 15:44:31 +02:00
Eelco Dolstra 64ec087f58
Fix 32-bit overflow with --no-net
--no-net causes tarballTtl to be set to the largest 32-bit integer,
which causes comparison like 'time + tarballTtl < other_time' to
fail on 32-bit systems. So cast them to 64-bit first.

https://hydra.nixos.org/build/95076624
(cherry picked from commit 29ccb2e969)
2019-06-24 22:16:43 +02:00
Eelco Dolstra b43e1e186e
CachedDownloadResult: Include store path
Also, make fetchGit and fetchMercurial update allowedPaths properly.

(Maybe the evaluator, rather than the caller of the evaluator, should
apply toRealPath(), but that's a bigger change.)

(cherry picked from commit 5c34d66538)
2019-06-24 21:59:27 +02:00
Eelco Dolstra 94f11d0a61
Fix abort in fromTOML
Fixes #2969.
2019-06-24 17:09:09 +02:00
Eelco Dolstra ef52ccf035
experimental/optional -> optional 2019-03-14 14:10:52 +01:00
Shea Levy b30be6b450
Add builtins.appendContext.
A partner of builtins.getContext, useful for the same reasons.
2019-01-31 08:52:23 -05:00
Shea Levy 1d757292d0
Add builtins.getContext.
This can be very helpful when debugging, as well as enabling complex
black magic like surgically removing a single dependency from a
string's context.
2019-01-14 11:27:10 -05:00
Daiderd Jordan 7e35e914c1
fetchGit: allow fetching explicit refs
Trying to fetch refs that are not in refs/heads currently fails because
it looks for refs/heads/refs/foo instead of refs/foo.

eg.

	builtins.fetchGit {
	  url = https://github.com/NixOS/nixpkgs.git;
	  ref = "refs/pull/1024/head;
	}
2018-12-14 20:12:22 +01:00
CHEIKH Chawki fa5143c722 Solve hg "abandoned transaction" issue 2018-12-06 13:57:59 +01:00
Eelco Dolstra 4aee93d5ce
fetchGit: Drop unnecessary localRef 2018-11-20 20:59:44 +01:00
Eelco Dolstra 3f4de91d80
Merge branch 'better-git-cache' of https://github.com/graham-at-target/nix 2018-11-20 20:41:19 +01:00
Eelco Dolstra 475a0a54a9
fetchGit/fetchMercurial: Don't absolutize paths
This is already done by coerceToString(), provided that the argument
is a path (e.g. 'fetchGit ./bla'). It fixes the handling of URLs like
git@github.com:owner/repo.git. It breaks 'fetchGit "./bla"', but that
was never intended to work anyway and is inconsistent with other
builtin functions (e.g. 'readFile "./bla"' fails).
2018-09-01 00:19:49 +02:00
Graham Christensen 02098d2073 fetchGit: use a better caching scheme
The current usage technically works by putting multiple different
repos in to the same git directory. However, it is very slow as
Git tries very hard to find common commits between the two
repositories. If the two repositories are large (like Nixpkgs and
another long-running project,) it is maddeningly slow.

This change busts the cache for existing deployments, but users
will be promptly repaid in per-repository performance.
2018-08-17 11:27:34 -04:00
Eelco Dolstra c2de2ff385
Fix build on 32-bit systems and macOS
Apparently, on macOS, 'long' != 'int64_t'.

https://hydra.nixos.org/build/77100756
2018-07-11 21:12:18 +02:00
Eelco Dolstra efd04888ca
Shup up a warning 2018-07-11 21:05:09 +02:00
Eelco Dolstra ea3c9dab5f
Include cpptoml for build simplicity 2018-07-03 18:39:36 +02:00
Eelco Dolstra 3b1f54cf06
Add a fromTOML primop
This is primarily useful for processing Cargo.lock files.
2018-07-03 18:39:36 +02:00
Eelco Dolstra 1672bcd230
Move evaluator-specific settings out of libstore 2018-05-30 13:29:50 +02:00
Eelco Dolstra 138af2e554
Shut up signedness warning 2018-03-19 11:57:15 +01:00
Guillaume Maudoux 80735c4cc9
fetchGit: Fix debug message 2018-03-13 10:28:23 +01:00
Will Dietz e89d02bf03 fetchGit: use "HEAD" as default ref 2018-02-28 16:34:34 -06:00
Eelco Dolstra d4dcffd643
Add pure evaluation mode
In this mode, the following restrictions apply:

* The builtins currentTime, currentSystem and storePath throw an
  error.

* $NIX_PATH and -I are ignored.

* fetchGit and fetchMercurial require a revision hash.

* fetchurl and fetchTarball require a sha256 attribute.

* No file system access is allowed outside of the paths returned by
  fetch{Git,Mercurial,url,Tarball}. Thus 'nix build -f ./foo.nix' is
  not allowed.

Thus, the evaluation result is completely reproducible from the
command line arguments. E.g.

  nix build --pure-eval '(
    let
      nix = fetchGit { url = https://github.com/NixOS/nixpkgs.git; rev = "9c927de4b179a6dd210dd88d34bda8af4b575680"; };
      nixpkgs = fetchGit { url = https://github.com/NixOS/nixpkgs.git; ref = "release-17.09"; rev = "66b4de79e3841530e6d9c6baf98702aa1f7124e4"; };
    in (import (nix + "/release.nix") { inherit nix nixpkgs; }).build.x86_64-linux
  )'

The goal is to enable completely reproducible and traceable
evaluation. For example, a NixOS configuration could be fully
described by a single Git commit hash. 'nixos-rebuild' would do
something like

  nix build --pure-eval '(
    (import (fetchGit { url = file:///my-nixos-config; rev = "..."; })).system
  ')

where the Git repository /my-nixos-config would use further fetchGit
calls or Git externals to fetch Nixpkgs and whatever other
dependencies it has. Either way, the commit hash would uniquely
identify the NixOS configuration and allow it to reproduced.
2018-01-16 19:23:18 +01:00
Will Dietz 428680b307 fetchGit: fix creation of uninitialized cache dir, let git create it
fetchGit test (as modified in previous commit) now passes.
2018-01-09 09:05:18 -06:00
Will Dietz 2e6f06c37e fetchGit: Fix handling of local repo when not using 'master' branch
Add tests checking this behavior.
2017-12-22 15:29:52 -06:00
Shea Levy eedbc4e06c
fetchGit: Ignore tarballTtl if rev is set and not in the repo.
Fixes #1697.
2017-11-24 06:09:24 -05:00
Eelco Dolstra d7da6c9ea9
fetchGit/fetchMercurial: Fix directory inclusion check
E.g. the existence of .gitignore would cause .git to be included.
2017-11-21 19:34:46 +01:00
Eelco Dolstra 7a4d9574d9
fetchgit.cc -> fetchGit.cc 2017-11-03 13:55:31 +01:00
Eelco Dolstra 6cf7c6a6b0
Remove git:// support in NIX_PATH
This didn't support specifying a revision/branch, and was restricted
to git:// URIs (since https:// or ssh:// would be ambiguous).
2017-11-03 13:55:31 +01:00
Eelco Dolstra ee6ac38848
fetchGit/fetchMercurial: Filter out directories with untracked files 2017-11-03 13:55:31 +01:00
Eelco Dolstra 4dee01da7c
fetchGit: Add a test 2017-11-03 13:55:30 +01:00
Eelco Dolstra 0e77aa3982
fetchGit: Don't do a remote fetch if we already have the rev 2017-11-03 13:55:30 +01:00
tv 5ab37f0e44
fetchMercurial: fix error message 2017-11-02 23:37:42 +01:00
Eelco Dolstra 212e72c609 Fix build
https://hydra.nixos.org/build/63172338
2017-11-01 21:32:30 +01:00