Commit graph

227 commits

Author SHA1 Message Date
Eelco Dolstra 703d863a48 Trivial changes from the lazy-trees branch 2022-12-07 14:06:34 +01:00
Théophane Hufschmitt 90ed041677
Merge pull request #7039 from Mic92/libfetchers-variant
libfetchers: avoid api.github.com ratelimit if no github token is set
2022-10-31 11:48:51 +01:00
Andrew Brooks a259084c50 Fix #7146
When fetching a non-local git repo by ref (and no rev), don't consider unrelated
cached revs for the same repository.
2022-10-14 18:04:47 -05:00
Jörg Thalheim d9abce4ad4 libfetchers: avoid api.github.com ratelimit if no github token is set
If we don't have any github token, we won't be able to fetch private
repos, but we are also more likely to run into API limits since
we don't have a token. To mitigate this only ever use the github api
if we actually have a token.
2022-10-01 10:38:09 +02:00
Eelco Dolstra c80a74b7d5 Don't pass --force to 'git add'
Fixes #5810.
2022-08-17 16:59:02 +02:00
Guillaume Girol e8109cf405 fetchGit: document shallow argument 2022-06-26 12:00:00 +00:00
Eelco Dolstra 4b6cc3da62 Fetch flake-registry.json from channels.nixos.org
Using fastly is slightly faster, provides some resilience due to a
high stale TTL, and allows some usage metrics.
2022-06-23 23:56:26 +02:00
Gabriel Fontes 9f6b4639c2
fix sourcehut brach/tag resolving regression
nixos/nix#6290 introduced a regex pattern to account for tags when
resolving sourcehut refs. nixos/nix#4638 reafactored the code,
accidentally treating the pattern as a regular string, causing all
non-HEAD ref resolving to break.

This fixes the regression and adds more test cases to avoid future
breakage.
2022-06-11 16:52:20 -03:00
Naïm Favier da8f8668ca
libfetchers/git: add missing --git-dir flags 2022-06-10 12:57:13 +02:00
Théophane Hufschmitt 027fd45230 Fix a segfault in the git fetcher
The git fetcher code used to dereference the (potentially empty) `ref`
input attribute. This was magically working, probably because the
compiler somehow outsmarted us, but is now blowing up with newer nixpkgs
versions.

Fix that by not trying to access this field while we don't know for sure
that it has been defined.

Fix #6554
2022-05-27 16:15:28 +02:00
Eelco Dolstra d8398d33c9
Typo 2022-05-25 15:29:27 +02:00
Eelco Dolstra 89a8955e79
Merge remote-tracking branch 'origin/file-fetcher' 2022-05-25 15:15:16 +02:00
Maximilian Bosch b916c08feb
libfetchers: drop getGitDir and hardcode .git
As discussed[1] this is most likely not desirable.

[1] https://github.com/NixOS/nix/pull/6440#issuecomment-1120876248
2022-05-24 14:20:48 +02:00
Tony Olagbaiye 5b8c1deb18 fetchTree: Allow fetching plain files
Add a new `file` fetcher type, which will fetch a plain file over
http(s), or from the local file.

Because plain `http(s)://` or `file://` urls can already correspond to
`tarball` inputs (if the path ends-up with a know archive extension),
the URL parsing logic is a bit convuluted in that:

- {http,https,file}:// urls will be interpreted as either a tarball or a
  file input, depending on the extensions of the path part (so
  `https://foo.com/bar` will be a `file` input and
  `https://foo.com/bar.tar.gz` as a `tarball` input)
- `file+{something}://` urls will be interpreted as `file` urls (with
  the `file+` part removed)
- `tarball+{something}://` urls will be interpreted as `tarball` urls (with
  the `tarball+` part removed)

Fix #3785

Co-Authored-By: Tony Olagbaiye <me@fron.io>
2022-05-19 18:24:49 +02:00
Norbert Melzer 831e2743ea fix GitHub URL template 2022-05-12 00:56:39 +02:00
Eelco Dolstra eb957ad6d8
Merge pull request #6497 from danielfullmer/ghe-fetcher-url
Use correct URL for GitHub Enterprise
2022-05-10 19:23:22 +02:00
Daniel Fullmer 7a3d5b2ff0 Use correct URL for GitHub Enterprise
For GitHub Enterprise, the API is accessed through a slightly different
URL. See [1], where it says:

> Use http(s)://[hostname]/api/v3 to access the API for GitHub
> Enterprise Server.

Also tested working on a GHE instance.

[1] https://docs.github.com/en/enterprise-server@3.3/rest/guides/getting-started-with-the-rest-api
2022-05-06 13:13:11 -07:00
Théophane Hufschmitt e68676e6c8 Fix the parsing of the sourcehut refs file
Since a26be9f3b8, the same parser is used
to parse the result of sourcehut’s `HEAD` endpoint (coming from [git
dumb protocol]) and the output of `git ls-remote`. However, they are very
slightly different (the former doesn’t specify the current reference
since it’s implied to be `HEAD`).

Unify both, and make the parser a bit more robust and understandable (by
making it more typed and adding tests for it)

[git dumb protocol]: https://git-scm.com/book/en/v2/Git-Internals-Transfer-Protocols#_the_dumb_protocol
2022-05-04 14:38:59 +02:00
Eelco Dolstra 61289ceee3 Style fixes 2022-05-02 13:37:53 +02:00
Eelco Dolstra 564faa6b4e
Merge pull request #6470 from Ma27/git-followup
libfetchers/git: fix every occasion of a permission error
2022-05-02 13:33:08 +02:00
Maximilian Bosch 1849e6a1f6
libfetchers/git: fix every occasion of a permission error
I'm afraid I missed a few problematic `git(1)`-calls while implementing
PR #6440, sorry for that! Upon investigating what went wrong, I realized
that I only tested against the "cached"-case by accident because my
git-checkout with my system's flake was apparently cached during my
debugging.

I managed to trigger the original issue again by running:

    $ git commit --allow-empty -m "tmp"
    $ sudo nixos-rebuild switch --flake .# -L --builders ''

Since `repoDir` points to the checkout that's potentially owned by
another user, I decided to add `--git-dir` to each call affecting
`repoDir`.

Since the `tmpDir` for the temporary submodule-checkout is created by
Nix itself, it doesn't seem to be an issue.

Sorry for that, it should be fine now.
2022-04-30 15:56:12 +02:00
Kjetil Orbekk 9bf296c970 Extract git reference parsing to a shared library
These utility functions can be shared between the git and github fetchers.
2022-04-29 18:46:21 -04:00
Kjetil Orbekk 1203e48926 Store cached head in cached git repo
The previous head caching implementation stored two paths in the local
cache; one for the cached git repo and another textfile containing the
resolved HEAD ref. This commit instead stores the resolved HEAD by
setting the HEAD ref in the local cache appropriately.
2022-04-29 18:46:21 -04:00
Kjetil Orbekk de54e1cd3f Refactor fetching of dirty workdir
Extract the handling of a local dirty workdir to a helper function.
2022-04-29 18:46:17 -04:00
Kjetil Orbekk 401e60f289 Resolve reference for remote repository
Resolves the HEAD reference from the remote repository instead
of assuming "master".
2022-04-29 18:42:28 -04:00
Maximilian Bosch d1f5356311
libfetchers/git: fix for nixos-rebuild
The `--git-dir=` must be `.` in some cases (for cached repos that are
"bare" repos in `~/.cache/nix/gitv3`). With this fix we can add
`--git-dir` to each `git`-invokation needed for `nixos-rebuild`.
2022-04-24 18:14:24 +02:00
Maximilian Bosch 0256e5578e
libfetchers/git: hardcode --git-dir
To demonstrate the problem:

* You need a `git` at 2.33.3 in your $PATH
* An expression like this in a git repository:

  ``` nix
  {
    outputs = { self, nixpkgs }: {
      packages.foo.x86_64-linux = with nixpkgs.legacyPackages.x86_64-linux;
        runCommand "snens" { } ''
          echo ${(builtins.fetchGit ./.).lastModifiedDate} > $out
        '';
    };
  }
  ```

Now, when instantiating the package via `builtins.getFlake`, it fails on
Nix 2.7 like this:

    $ nix-instantiate -E '(builtins.getFlake "'"$(pwd)"'").packages.foo.x86_64-linux'
    fatal: unsafe repository ('/nix/store/a7j3125km4h8l0p71q6ssfkxamfh5d61-source' is owned by someone else)
    To add an exception for this directory, call:

    	git config --global --add safe.directory /nix/store/a7j3125km4h8l0p71q6ssfkxamfh5d61-source
    error: program 'git' failed with exit code 128
    (use '--show-trace' to show detailed location information)

This breaks e.g. `nixops`-deployments using flakes with similar
expressions as shown above.

The cause for this is that `git(1)` tries to find the highest
`.git`-directory in the directory tree and if it finds a such a
directory, but with another owning user (root vs. the user who evaluates
the expression), it fails as above. This was changed recently to fix
CVE-2022-24765[1].

By explicitly specifying `--git-dir`, Git assumes to be in the top-level
directory and doesn't attempt to look for a `.git`-directory in the
parent directories and thus the code-path leading to said error is never
reached.

[1] https://lore.kernel.org/git/xmqqv8veb5i6.fsf@gitster.g/
2022-04-23 23:20:17 +02:00
John Ericson 75b62e5260 Avoid fmt when constructor already does it
There is a correctnes issue here, but #3724 will fix that. This is just
a cleanup for brevity's sake.
2022-04-19 01:44:11 +00:00
Eelco Dolstra d6effddd3b
Merge pull request #6387 from Uthar/fix
assert hash types for Git and Mercurial
2022-04-14 14:55:27 +02:00
Kasper Gałkowski 2769e43f61 assert hash types for Git and Mercurial 2022-04-12 21:13:14 +02:00
Eelco Dolstra f7276bc948
Merge pull request #6392 from danpls/fix-actualUrl-mercurial
libfetchers: Fix assertion (Mercurial)
2022-04-11 11:16:04 +02:00
Daniel Pauls d6b7529579 libfetchers: Fix assertion (Mercurial)
See commit 1e1cd6e7a for more information.
2022-04-09 19:10:23 +02:00
Daniel Pauls 770f7371f3 libfetchers: Replace regex to clarify intent 2022-04-09 17:00:14 +02:00
Théophane Hufschmitt 646af7325d
Merge pull request #6376 from Uthar/master
don't assume that rev is a SHA1 hash
2022-04-08 17:56:27 +02:00
Kasper Gałkowski 2c2fd4946f don't assume that rev is a SHA1 hash
This was a problem when writing a fetcher that uses e.g. sha256 hashes
for revisions. This doesn't actually do anything new, but allows for
creating such fetchers in the future (perhaps when support for Git's
SHA256 object format gains more popularity).
2022-04-07 19:49:47 +02:00
Daniel Pauls 1e1cd6e7a9 libfetchers: Fix assertion
The filter expects all paths to have a prefix of the raw `actualUrl`, but
`Store::addToStore(...)` provides absolute canonicalized paths.
To fix this create an absolute and canonicalized path from the `actualUrl` and
use it instead.

Fixes #6195.
2022-04-06 17:33:23 +02:00
Gabriel Fontes 9720797f69
Don't partial match sourcehut refs 2022-03-19 11:04:04 -03:00
Gabriel Fontes 345a8ee0cb
Fix sourcehut tag ref resolving 2022-03-19 10:56:13 -03:00
Maximilian Bosch 244baff2c7
libfetchers: remove obsolete filesystem #include 2022-03-15 12:32:11 +01:00
Maximilian Bosch 34e20c164c
libfetchers/path: set lastModified to path's mtime
When importing e.g. a local `nixpkgs` in a flake to test a change like

    {
      inputs.nixpkgs.url = path:/home/ma27/Projects/nixpkgs;
      outputs = /* ... */
    }

then the input is missing a `lastModified`-field that's e.g. used in
`nixpkgs.lib.nixosSystem`. Due to the missing `lastMoified`-field, the
mtime is set to 19700101:

    result -> /nix/store/b7dg1lmmsill2rsgyv2w7b6cnmixkvc1-nixos-system-nixos-22.05.19700101.dirty

With this change, the `path`-fetcher now sets a `lastModified` attribute
to the `mtime` just like it's the case in the `tarball`-fetcher already.
When building NixOS systems with `nixpkgs` being a `path`-input and this
patch, the output-path now looks like this:

    result -> /nix/store/ld2qf9c1s98dxmiwcaq5vn9k5ylzrm1s-nixos-system-nixos-22.05.20220217.dirty
2022-03-15 12:32:11 +01:00
Eelco Dolstra 167766b65c Style 2022-03-11 11:19:21 +01:00
Théophane Hufschmitt 7ebd6f1093
Merge pull request #5758 from mschwaig/fix-git-workspace-dirty-detection
git fetcher: fix detection of dirty git workspaces
2022-03-11 09:35:01 +01:00
Nicholas Sielicki 314852a10e
Point to new github oauth docs url
Previous URL was 404'ing.
2022-03-06 17:01:14 -06:00
John Ericson ea71da395f Move some stuff from Settings to a new FetchSettings.
Starting work on #5638

The exact boundary between `FetchSettings` and `EvalSettings` is not
clear to me, but that's fine. First lets clean out `libstore`, and then
worry about what, if anything, should be the separation between those
two.
2022-03-01 01:39:25 +00:00
Eelco Dolstra df552ff53e Remove std::string alias (for real this time)
Also use std::string_view in a few more places.
2022-02-25 16:13:02 +01:00
Eelco Dolstra 8518cebfce libfetchers: Rename immutable -> locked
This is more consistent with flake terminology.
2022-02-24 18:09:00 +01:00
Théophane Hufschmitt 6524eb4b77
Merge pull request #5342 from Misterio77/add-sourcehut
Add support for sourcehut input scheme
2022-02-24 09:09:21 +01:00
Martin Schwaighofer 53523c0ab8 git fetcher: set locale for rev-parse 2022-02-19 22:58:23 +01:00
Martin Schwaighofer 9504445cab git fetcher: distinguish errors more precisely 2022-02-19 22:58:23 +01:00
Martin Schwaighofer c7e527b82b git fetcher: invoke diff instead of diff-index
diff-index operates on the view that git has of the working tree,
which might be outdated. The higher-level diff command does this
automatically. This change also adds handling for submodules.

fixes #4140

Alternative fixes would be invoking update-index before diff-index or
matching more closely what require_clean_work_tree from git-sh-setup.sh
does, but both those options make it more difficult to reason about
correctness.
2022-02-19 22:58:23 +01:00