puck/lix
0
0
Fork 0
forked from lix-project/lix
Code Pull requests Activity
lix/src/libexpr/primops/fetchGit.cc

86 lines
3.1 KiB
C++
Raw Normal View History

Add fetchgit builtin The function builtins.fetchgit fetches Git repositories at evaluation time, similar to builtins.fetchTarball. (Perhaps the name should be changed, being confusing with respect to Nixpkgs's fetchgit function, with works at build time.) Example: (import (builtins.fetchgit git://github.com/NixOS/nixpkgs) {}).hello or (import (builtins.fetchgit { url = git://github.com/NixOS/nixpkgs-channels; rev = "nixos-16.03"; }) {}).hello Note that the result does not contain a .git directory.
2016-04-29 18:14:44 +00:00
#include "primops.hh"
#include "eval-inline.hh"
#include "store-api.hh"
fetchGit: use a better caching scheme The current usage technically works by putting multiple different repos in to the same git directory. However, it is very slow as Git tries very hard to find common commits between the two repositories. If the two repositories are large (like Nixpkgs and another long-running project,) it is maddeningly slow. This change busts the cache for existing deployments, but users will be promptly repaid in per-repository performance.
2018-08-17 14:29:32 +00:00
#include "hash.hh"
Backport libfetchers from the flakes branch This provides a pluggable mechanism for defining new fetchers. It adds a builtin function 'fetchTree' that generalizes existing fetchers like 'fetchGit', 'fetchMercurial' and 'fetchTarball'. 'fetchTree' takes a set of attributes, e.g. fetchTree { type = "git"; url = "https://example.org/repo.git"; ref = "some-branch"; rev = "abcdef..."; } The existing fetchers are just wrappers around this. Note that the input attributes to fetchTree are the same as flake input specifications and flake lock file entries. All fetchers share a common cache stored in ~/.cache/nix/fetcher-cache-v1.sqlite. This replaces the ad hoc caching mechanisms in fetchGit and download.cc (e.g. ~/.cache/nix/{tarballs,git-revs*}). This also adds support for Git worktrees (c169ea59049f861aaba429f48b828d0820b74d1d).
2020-03-30 14:04:18 +00:00
#include "fetchers.hh"
#include "url.hh"
builtins.fetchgit: Support a "name" attribute The "name" attribute defaults to "source", which we should use for all similar functions (e.g. fetchTarball and in Hydra) to ensure that we get a consistent store path regardless of how the tree is fetched. "source" is not necessarily a correct label, but using an empty name is problematic: you get an ugly store path ending in a dash, and it's impossible to have a fixed-output derivation that produces that path because ".drv" is not a valid store name. Fixes #904.
2017-10-30 08:57:38 +00:00
Add fetchgit builtin The function builtins.fetchgit fetches Git repositories at evaluation time, similar to builtins.fetchTarball. (Perhaps the name should be changed, being confusing with respect to Nixpkgs's fetchgit function, with works at build time.) Example: (import (builtins.fetchgit git://github.com/NixOS/nixpkgs) {}).hello or (import (builtins.fetchgit { url = git://github.com/NixOS/nixpkgs-channels; rev = "nixos-16.03"; }) {}).hello Note that the result does not contain a .git directory.
2016-04-29 18:14:44 +00:00
namespace nix {
fetchgit -> fetchGit Almost all other primops are camelCase so no reason not to use that here.
2017-10-30 09:25:08 +00:00
static void prim_fetchGit(EvalState & state, const Pos & pos, Value * * args, Value & v)
Add fetchgit builtin The function builtins.fetchgit fetches Git repositories at evaluation time, similar to builtins.fetchTarball. (Perhaps the name should be changed, being confusing with respect to Nixpkgs's fetchgit function, with works at build time.) Example: (import (builtins.fetchgit git://github.com/NixOS/nixpkgs) {}).hello or (import (builtins.fetchgit { url = git://github.com/NixOS/nixpkgs-channels; rev = "nixos-16.03"; }) {}).hello Note that the result does not contain a .git directory.
2016-04-29 18:14:44 +00:00
{
std::string url;
experimental/optional -> optional
2019-02-12 12:43:32 +00:00
std::optional<std::string> ref;
Backport libfetchers from the flakes branch This provides a pluggable mechanism for defining new fetchers. It adds a builtin function 'fetchTree' that generalizes existing fetchers like 'fetchGit', 'fetchMercurial' and 'fetchTarball'. 'fetchTree' takes a set of attributes, e.g. fetchTree { type = "git"; url = "https://example.org/repo.git"; ref = "some-branch"; rev = "abcdef..."; } The existing fetchers are just wrappers around this. Note that the input attributes to fetchTree are the same as flake input specifications and flake lock file entries. All fetchers share a common cache stored in ~/.cache/nix/fetcher-cache-v1.sqlite. This replaces the ad hoc caching mechanisms in fetchGit and download.cc (e.g. ~/.cache/nix/{tarballs,git-revs*}). This also adds support for Git worktrees (c169ea59049f861aaba429f48b828d0820b74d1d).
2020-03-30 14:04:18 +00:00
std::optional<Hash> rev;
builtins.fetchgit: Support a "name" attribute The "name" attribute defaults to "source", which we should use for all similar functions (e.g. fetchTarball and in Hydra) to ensure that we get a consistent store path regardless of how the tree is fetched. "source" is not necessarily a correct label, but using an empty name is problematic: you get an ugly store path ending in a dash, and it's impossible to have a fixed-output derivation that produces that path because ".drv" is not a valid store name. Fixes #904.
2017-10-30 08:57:38 +00:00
std::string name = "source";
Add fetchSubmodules to builtins.fetchGit There are some downsides to this features: - Submodules are not cached (unlike the root repo), - Full checkouts are created in a temporary directory.
2019-10-25 15:45:14 +00:00
bool fetchSubmodules = false;
Make "fetchGit /path" work
2017-10-30 12:18:28 +00:00
PathSet context;
Add fetchgit builtin The function builtins.fetchgit fetches Git repositories at evaluation time, similar to builtins.fetchTarball. (Perhaps the name should be changed, being confusing with respect to Nixpkgs's fetchgit function, with works at build time.) Example: (import (builtins.fetchgit git://github.com/NixOS/nixpkgs) {}).hello or (import (builtins.fetchgit { url = git://github.com/NixOS/nixpkgs-channels; rev = "nixos-16.03"; }) {}).hello Note that the result does not contain a .git directory.
2016-04-29 18:14:44 +00:00
state.forceValue(*args[0]);
if (args[0]->type == tAttrs) {
state.forceAttrs(*args[0], pos);
for (auto & attr : *args[0]->attrs) {
builtins.fetchgit: Support a "name" attribute The "name" attribute defaults to "source", which we should use for all similar functions (e.g. fetchTarball and in Hydra) to ensure that we get a consistent store path regardless of how the tree is fetched. "source" is not necessarily a correct label, but using an empty name is problematic: you get an ugly store path ending in a dash, and it's impossible to have a fixed-output derivation that produces that path because ".drv" is not a valid store name. Fixes #904.
2017-10-30 08:57:38 +00:00
string n(attr.name);
Make "fetchGit /path" work
2017-10-30 12:18:28 +00:00
if (n == "url")
builtins.fetchgit: Support paths
2017-03-02 10:46:28 +00:00
url = state.coerceToString(*attr.pos, *attr.value, context, false, false);
builtins.fetchgit: Support a "name" attribute The "name" attribute defaults to "source", which we should use for all similar functions (e.g. fetchTarball and in Hydra) to ensure that we get a consistent store path regardless of how the tree is fetched. "source" is not necessarily a correct label, but using an empty name is problematic: you get an ugly store path ending in a dash, and it's impossible to have a fixed-output derivation that produces that path because ".drv" is not a valid store name. Fixes #904.
2017-10-30 08:57:38 +00:00
else if (n == "ref")
builtins.fetchgit: Support specifying commit hashes This adds an argument "rev" specififying the Git commit hash. The existing argument "rev" is renamed to "ref". The default value for "ref" is "master". When specifying a hash, it's necessary to specify a ref since we're not cloning the entire repository but only fetching a specific ref. Example usage: builtins.fetchgit { url = https://github.com/NixOS/nixpkgs.git; ref = "release-16.03"; rev = "c1c0484041ab6f9c6858c8ade80a8477c9ae4442"; };
2017-07-27 16:08:23 +00:00
ref = state.forceStringNoCtx(*attr.value, *attr.pos);
builtins.fetchgit: Support a "name" attribute The "name" attribute defaults to "source", which we should use for all similar functions (e.g. fetchTarball and in Hydra) to ensure that we get a consistent store path regardless of how the tree is fetched. "source" is not necessarily a correct label, but using an empty name is problematic: you get an ugly store path ending in a dash, and it's impossible to have a fixed-output derivation that produces that path because ".drv" is not a valid store name. Fixes #904.
2017-10-30 08:57:38 +00:00
else if (n == "rev")
Backport libfetchers from the flakes branch This provides a pluggable mechanism for defining new fetchers. It adds a builtin function 'fetchTree' that generalizes existing fetchers like 'fetchGit', 'fetchMercurial' and 'fetchTarball'. 'fetchTree' takes a set of attributes, e.g. fetchTree { type = "git"; url = "https://example.org/repo.git"; ref = "some-branch"; rev = "abcdef..."; } The existing fetchers are just wrappers around this. Note that the input attributes to fetchTree are the same as flake input specifications and flake lock file entries. All fetchers share a common cache stored in ~/.cache/nix/fetcher-cache-v1.sqlite. This replaces the ad hoc caching mechanisms in fetchGit and download.cc (e.g. ~/.cache/nix/{tarballs,git-revs*}). This also adds support for Git worktrees (c169ea59049f861aaba429f48b828d0820b74d1d).
2020-03-30 14:04:18 +00:00
rev = Hash(state.forceStringNoCtx(*attr.value, *attr.pos), htSHA1);
builtins.fetchgit: Support a "name" attribute The "name" attribute defaults to "source", which we should use for all similar functions (e.g. fetchTarball and in Hydra) to ensure that we get a consistent store path regardless of how the tree is fetched. "source" is not necessarily a correct label, but using an empty name is problematic: you get an ugly store path ending in a dash, and it's impossible to have a fixed-output derivation that produces that path because ".drv" is not a valid store name. Fixes #904.
2017-10-30 08:57:38 +00:00
else if (n == "name")
name = state.forceStringNoCtx(*attr.value, *attr.pos);
Rename fetchGit fetchSubmodules to just submodules
2020-03-02 13:50:13 +00:00
else if (n == "submodules")
Add fetchSubmodules to builtins.fetchGit There are some downsides to this features: - Submodules are not cached (unlike the root repo), - Full checkouts are created in a temporary directory.
2019-10-25 15:45:14 +00:00
fetchSubmodules = state.forceBool(*attr.value, *attr.pos);
Add fetchgit builtin The function builtins.fetchgit fetches Git repositories at evaluation time, similar to builtins.fetchTarball. (Perhaps the name should be changed, being confusing with respect to Nixpkgs's fetchgit function, with works at build time.) Example: (import (builtins.fetchgit git://github.com/NixOS/nixpkgs) {}).hello or (import (builtins.fetchgit { url = git://github.com/NixOS/nixpkgs-channels; rev = "nixos-16.03"; }) {}).hello Note that the result does not contain a .git directory.
2016-04-29 18:14:44 +00:00
else
fetchgit -> fetchGit Almost all other primops are camelCase so no reason not to use that here.
2017-10-30 09:25:08 +00:00
throw EvalError("unsupported argument '%s' to 'fetchGit', at %s", attr.name, *attr.pos);
Add fetchgit builtin The function builtins.fetchgit fetches Git repositories at evaluation time, similar to builtins.fetchTarball. (Perhaps the name should be changed, being confusing with respect to Nixpkgs's fetchgit function, with works at build time.) Example: (import (builtins.fetchgit git://github.com/NixOS/nixpkgs) {}).hello or (import (builtins.fetchgit { url = git://github.com/NixOS/nixpkgs-channels; rev = "nixos-16.03"; }) {}).hello Note that the result does not contain a .git directory.
2016-04-29 18:14:44 +00:00
}
if (url.empty())
Replace Unicode quotes in user-facing strings by ASCII Relevant RFC: NixOS/rfcs#4 $ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
2017-07-30 11:27:57 +00:00
throw EvalError(format("'url' argument required, at %1%") % pos);
Add fetchgit builtin The function builtins.fetchgit fetches Git repositories at evaluation time, similar to builtins.fetchTarball. (Perhaps the name should be changed, being confusing with respect to Nixpkgs's fetchgit function, with works at build time.) Example: (import (builtins.fetchgit git://github.com/NixOS/nixpkgs) {}).hello or (import (builtins.fetchgit { url = git://github.com/NixOS/nixpkgs-channels; rev = "nixos-16.03"; }) {}).hello Note that the result does not contain a .git directory.
2016-04-29 18:14:44 +00:00
} else
Make "fetchGit /path" work
2017-10-30 12:18:28 +00:00
url = state.coerceToString(pos, *args[0], context, false, false);
Add option allowed-uris This allows network access in restricted eval mode.
2017-10-30 11:39:59 +00:00
// FIXME: git externals probably can be used to bypass the URI
// whitelist. Ah well.
state.checkURI(url);
Backport libfetchers from the flakes branch This provides a pluggable mechanism for defining new fetchers. It adds a builtin function 'fetchTree' that generalizes existing fetchers like 'fetchGit', 'fetchMercurial' and 'fetchTarball'. 'fetchTree' takes a set of attributes, e.g. fetchTree { type = "git"; url = "https://example.org/repo.git"; ref = "some-branch"; rev = "abcdef..."; } The existing fetchers are just wrappers around this. Note that the input attributes to fetchTree are the same as flake input specifications and flake lock file entries. All fetchers share a common cache stored in ~/.cache/nix/fetcher-cache-v1.sqlite. This replaces the ad hoc caching mechanisms in fetchGit and download.cc (e.g. ~/.cache/nix/{tarballs,git-revs*}). This also adds support for Git worktrees (c169ea59049f861aaba429f48b828d0820b74d1d).
2020-03-30 14:04:18 +00:00
if (evalSettings.pureEval && !rev)
throw Error("in pure evaluation mode, 'fetchGit' requires a Git revision");
Merge branch 'fetchgit-recursive' of https://github.com/blitz/nix
2020-04-07 11:45:17 +00:00
fetchers::Attrs attrs;
attrs.insert_or_assign("type", "git");
attrs.insert_or_assign("url", url.find("://") != std::string::npos ? url : "file://" + url);
if (ref) attrs.insert_or_assign("ref", *ref);
if (rev) attrs.insert_or_assign("rev", rev->gitRev());
if (fetchSubmodules) attrs.insert_or_assign("submodules", true);
auto input = fetchers::inputFromAttrs(attrs);
Backport libfetchers from the flakes branch This provides a pluggable mechanism for defining new fetchers. It adds a builtin function 'fetchTree' that generalizes existing fetchers like 'fetchGit', 'fetchMercurial' and 'fetchTarball'. 'fetchTree' takes a set of attributes, e.g. fetchTree { type = "git"; url = "https://example.org/repo.git"; ref = "some-branch"; rev = "abcdef..."; } The existing fetchers are just wrappers around this. Note that the input attributes to fetchTree are the same as flake input specifications and flake lock file entries. All fetchers share a common cache stored in ~/.cache/nix/fetcher-cache-v1.sqlite. This replaces the ad hoc caching mechanisms in fetchGit and download.cc (e.g. ~/.cache/nix/{tarballs,git-revs*}). This also adds support for Git worktrees (c169ea59049f861aaba429f48b828d0820b74d1d).
2020-03-30 14:04:18 +00:00
Merge branch 'fetchgit-recursive' of https://github.com/blitz/nix
2020-04-07 11:45:17 +00:00
// FIXME: use name?
Backport libfetchers from the flakes branch This provides a pluggable mechanism for defining new fetchers. It adds a builtin function 'fetchTree' that generalizes existing fetchers like 'fetchGit', 'fetchMercurial' and 'fetchTarball'. 'fetchTree' takes a set of attributes, e.g. fetchTree { type = "git"; url = "https://example.org/repo.git"; ref = "some-branch"; rev = "abcdef..."; } The existing fetchers are just wrappers around this. Note that the input attributes to fetchTree are the same as flake input specifications and flake lock file entries. All fetchers share a common cache stored in ~/.cache/nix/fetcher-cache-v1.sqlite. This replaces the ad hoc caching mechanisms in fetchGit and download.cc (e.g. ~/.cache/nix/{tarballs,git-revs*}). This also adds support for Git worktrees (c169ea59049f861aaba429f48b828d0820b74d1d).
2020-03-30 14:04:18 +00:00
auto [tree, input2] = input->fetchTree(state.store);
Add fetchgit builtin The function builtins.fetchgit fetches Git repositories at evaluation time, similar to builtins.fetchTarball. (Perhaps the name should be changed, being confusing with respect to Nixpkgs's fetchgit function, with works at build time.) Example: (import (builtins.fetchgit git://github.com/NixOS/nixpkgs) {}).hello or (import (builtins.fetchgit { url = git://github.com/NixOS/nixpkgs-channels; rev = "nixos-16.03"; }) {}).hello Note that the result does not contain a .git directory.
2016-04-29 18:14:44 +00:00
builtins.fetchGit: Return an attrset with revision info This adds rev, shortRev and revCount attributes, equal to what Hydra provides. E.g. $ nix eval '(fetchGit https://github.com/NixOS/patchelf.git)' { outPath = "/nix/store/ghigrkw02l440g8vfxa9wj4c3zpfmw99-source"; rev = "29c085fd9d3fc972f75b3961905d6b4ecce7eb2b"; revCount = 303; shortRev = "29c085f"; }
2017-10-30 10:49:03 +00:00
state.mkAttrs(v, 8);
Backport libfetchers from the flakes branch This provides a pluggable mechanism for defining new fetchers. It adds a builtin function 'fetchTree' that generalizes existing fetchers like 'fetchGit', 'fetchMercurial' and 'fetchTarball'. 'fetchTree' takes a set of attributes, e.g. fetchTree { type = "git"; url = "https://example.org/repo.git"; ref = "some-branch"; rev = "abcdef..."; } The existing fetchers are just wrappers around this. Note that the input attributes to fetchTree are the same as flake input specifications and flake lock file entries. All fetchers share a common cache stored in ~/.cache/nix/fetcher-cache-v1.sqlite. This replaces the ad hoc caching mechanisms in fetchGit and download.cc (e.g. ~/.cache/nix/{tarballs,git-revs*}). This also adds support for Git worktrees (c169ea59049f861aaba429f48b828d0820b74d1d).
2020-03-30 14:04:18 +00:00
auto storePath = state.store->printStorePath(tree.storePath);
mkString(*state.allocAttr(v, state.sOutPath), storePath, PathSet({storePath}));
// Backward compatibility: set 'rev' to
// 0000000000000000000000000000000000000000 for a dirty tree.
auto rev2 = input2->getRev().value_or(Hash(htSHA1));
mkString(*state.allocAttr(v, state.symbols.create("rev")), rev2.gitRev());
mkString(*state.allocAttr(v, state.symbols.create("shortRev")), rev2.gitShortRev());
// Backward compatibility: set 'revCount' to 0 for a dirty tree.
mkInt(*state.allocAttr(v, state.symbols.create("revCount")),
tree.info.revCount.value_or(0));
Merge branch 'fetchgit-recursive' of https://github.com/blitz/nix
2020-04-07 11:45:17 +00:00
mkBool(*state.allocAttr(v, state.symbols.create("submodules")), fetchSubmodules);
builtins.fetchGit: Return an attrset with revision info This adds rev, shortRev and revCount attributes, equal to what Hydra provides. E.g. $ nix eval '(fetchGit https://github.com/NixOS/patchelf.git)' { outPath = "/nix/store/ghigrkw02l440g8vfxa9wj4c3zpfmw99-source"; rev = "29c085fd9d3fc972f75b3961905d6b4ecce7eb2b"; revCount = 303; shortRev = "29c085f"; }
2017-10-30 10:49:03 +00:00
v.attrs->sort();
Add pure evaluation mode In this mode, the following restrictions apply: * The builtins currentTime, currentSystem and storePath throw an error. * $NIX_PATH and -I are ignored. * fetchGit and fetchMercurial require a revision hash. * fetchurl and fetchTarball require a sha256 attribute. * No file system access is allowed outside of the paths returned by fetch{Git,Mercurial,url,Tarball}. Thus 'nix build -f ./foo.nix' is not allowed. Thus, the evaluation result is completely reproducible from the command line arguments. E.g. nix build --pure-eval '( let nix = fetchGit { url = https://github.com/NixOS/nixpkgs.git; rev = "9c927de4b179a6dd210dd88d34bda8af4b575680"; }; nixpkgs = fetchGit { url = https://github.com/NixOS/nixpkgs.git; ref = "release-17.09"; rev = "66b4de79e3841530e6d9c6baf98702aa1f7124e4"; }; in (import (nix + "/release.nix") { inherit nix nixpkgs; }).build.x86_64-linux )' The goal is to enable completely reproducible and traceable evaluation. For example, a NixOS configuration could be fully described by a single Git commit hash. 'nixos-rebuild' would do something like nix build --pure-eval '( (import (fetchGit { url = file:///my-nixos-config; rev = "..."; })).system ') where the Git repository /my-nixos-config would use further fetchGit calls or Git externals to fetch Nixpkgs and whatever other dependencies it has. Either way, the commit hash would uniquely identify the NixOS configuration and allow it to reproduced.
2018-01-16 17:50:38 +00:00
if (state.allowedPaths)
Backport libfetchers from the flakes branch This provides a pluggable mechanism for defining new fetchers. It adds a builtin function 'fetchTree' that generalizes existing fetchers like 'fetchGit', 'fetchMercurial' and 'fetchTarball'. 'fetchTree' takes a set of attributes, e.g. fetchTree { type = "git"; url = "https://example.org/repo.git"; ref = "some-branch"; rev = "abcdef..."; } The existing fetchers are just wrappers around this. Note that the input attributes to fetchTree are the same as flake input specifications and flake lock file entries. All fetchers share a common cache stored in ~/.cache/nix/fetcher-cache-v1.sqlite. This replaces the ad hoc caching mechanisms in fetchGit and download.cc (e.g. ~/.cache/nix/{tarballs,git-revs*}). This also adds support for Git worktrees (c169ea59049f861aaba429f48b828d0820b74d1d).
2020-03-30 14:04:18 +00:00
state.allowedPaths->insert(tree.actualPath);
Add fetchgit builtin The function builtins.fetchgit fetches Git repositories at evaluation time, similar to builtins.fetchTarball. (Perhaps the name should be changed, being confusing with respect to Nixpkgs's fetchgit function, with works at build time.) Example: (import (builtins.fetchgit git://github.com/NixOS/nixpkgs) {}).hello or (import (builtins.fetchgit { url = git://github.com/NixOS/nixpkgs-channels; rev = "nixos-16.03"; }) {}).hello Note that the result does not contain a .git directory.
2016-04-29 18:14:44 +00:00
}
fetchgit -> fetchGit Almost all other primops are camelCase so no reason not to use that here.
2017-10-30 09:25:08 +00:00
static RegisterPrimOp r("fetchGit", 1, prim_fetchGit);
Add fetchgit builtin The function builtins.fetchgit fetches Git repositories at evaluation time, similar to builtins.fetchTarball. (Perhaps the name should be changed, being confusing with respect to Nixpkgs's fetchgit function, with works at build time.) Example: (import (builtins.fetchgit git://github.com/NixOS/nixpkgs) {}).hello or (import (builtins.fetchgit { url = git://github.com/NixOS/nixpkgs-channels; rev = "nixos-16.03"; }) {}).hello Note that the result does not contain a .git directory.
2016-04-29 18:14:44 +00:00
}
Copy permalink