Commit graph

178 commits

Author SHA1 Message Date
Eelco Dolstra 950b46821f Remove TreeInfo
The attributes previously stored in TreeInfo (narHash, revCount,
lastModified) are now stored in Input. This makes it less arbitrary
what attributes are stored where.

As a result, the lock file format has changed. An entry like

    "info": {
      "lastModified": 1585405475,
      "narHash": "sha256-bESW0n4KgPmZ0luxvwJ+UyATrC6iIltVCsGdLiphVeE="
    },
    "locked": {
      "owner": "NixOS",
      "repo": "nixpkgs",
      "rev": "b88ff468e9850410070d4e0ccd68c7011f15b2be",
      "type": "github"
    },

is now stored as

    "locked": {
      "owner": "NixOS",
      "repo": "nixpkgs",
      "rev": "b88ff468e9850410070d4e0ccd68c7011f15b2be",
      "type": "github",
      "lastModified": 1585405475,
      "narHash": "sha256-bESW0n4KgPmZ0luxvwJ+UyATrC6iIltVCsGdLiphVeE="
    },

The 'Input' class is now a dumb set of attributes. All the fetcher
implementations subclass InputScheme, not Input. This simplifies the
API.

Also, fix substitution of flake inputs. This was broken since lazy
flake fetching started using fetchTree internally.
2020-05-30 00:44:11 +02:00
Matthew Kenigsberg 8d67794da1 handle circular flake dependencies in list-inputs 2020-05-21 17:06:11 -06:00
Eelco Dolstra 5f64655ff4 Move registry-related commands from 'nix flake' to 'nix registry'
This makes 'nix flake' less cluttered and more consistent (it's only
subcommands that operator on a flake). Also, the registry is not
inherently flake-related (e.g. fetchTree could also use it to remap
inputs).
2020-05-15 14:38:10 +02:00
Eelco Dolstra 649c2db308 nix flake: Add completion support 2020-05-11 22:10:33 +02:00
Eelco Dolstra 6f3244ce45 Merge remote-tracking branch 'origin/master' into flakes 2020-05-05 18:59:33 +02:00
Eelco Dolstra 6521c92ce8 Improve path:// handling
In particular, doing 'nix build /path/to/dir' now works if
/path/to/dir is not a Git tree (it only has to contain a flake.nix
file).

Also, 'nix flake init' no longer requires a Git tree (but it will do a
'git add flake.nix' if it's a Git tree)
2020-04-27 22:53:11 +02:00
Eelco Dolstra b69323f8c9 Revive 'nix search'
It uses the evaluation cache now rather than the ad hoc JSON cache.
2020-04-20 15:27:09 +02:00
Eelco Dolstra 42a12f9232 Move eval-cache.{cc,hh} 2020-04-20 13:14:59 +02:00
Eelco Dolstra 539a9c1c5f Get rid of the old eval cache 2020-04-20 13:13:52 +02:00
Eelco Dolstra 0725ab2fd7 Store more stuff in the evaluation cache
In particular, we store whether an attribute failed to evaluate (threw
an exception) or was an unsupported type. This is to ensure that a
repeated 'nix flake show' never has to evaluate anything, so it can
execute without fetching the flake.

With this, 'nix flake show nixpkgs/nixos-20.03 --legacy' executes in
0.6s (was 3.4s).
2020-04-19 23:07:06 +02:00
Eelco Dolstra 3738bcb05e Eval cache: Don't replace real attributes with placeholders 2020-04-18 15:12:31 +02:00
Eelco Dolstra 69cb9f7eee Wrap eval cache creation in a giant transaction
This speeds up the creation of the cache for the nixpkgs flake from
21.2s to 10.2s. Oddly, it also speeds up querying the cache
(i.e. running 'nix flake show nixpkgs/nixos-20.03 --legacy') from 4.2s
to 3.4s.

(For comparison, running with --no-eval-cache takes 9.5s, so the
overhead of building the SQLite cache is only 0.7s.)
2020-04-17 23:17:21 +02:00
Eelco Dolstra aaa109565e Use a more space/time-efficient representation for the eval cache 2020-04-17 23:04:21 +02:00
Eelco Dolstra bdb3226607 Add flag to disable the eval cache 2020-04-17 14:30:04 +02:00
Eelco Dolstra aa34c0ef51 nix flake show: Speed up eval cache bigly
In the fully cached case for the 'nixpkgs' flake, it went from 101s to
4.6s. Populating the cache went from 132s to 17.4s (which could
probably be improved further by combining INSERTs).
2020-04-17 13:57:02 +02:00
Eelco Dolstra 9ea4f93f88 nix flake show: Support apps 2020-04-17 01:21:24 +02:00
Eelco Dolstra 12b7eefbc5 nix flake show: Use evaluation cache 2020-04-17 01:02:29 +02:00
Eelco Dolstra 3b489e8843 Add 'nix flake show' command 2020-04-16 19:52:39 +02:00
Eelco Dolstra 29043e7e9e Fix 2020-04-16 19:01:49 +02:00
Eelco Dolstra 8f41847394 Use Logger::stdout() 2020-04-16 13:47:59 +02:00
Eelco Dolstra e5ea01c1a8
Remove flake 'edition' field
Future editions of flakes or the Nix language can be supported by
renaming flake.nix (e.g. flake-v2.nix). This avoids a bootstrap
problem where we don't know which grammar to use to parse
flake*.nix. It also allows a project to support multiple flake
editions, in theory.
2020-04-10 10:24:09 +02:00
Eelco Dolstra 68b43e01dd
nix flake info: Show resolved URL
This is useful for finding out what a registry lookup resolves to, e.g

  $ nix flake info patchelf
  Resolved URL:  github:NixOS/patchelf
  Locked URL:    github:NixOS/patchelf/cd7955af31698c571c30b7a0f78e59fd624d0229
2020-04-06 14:56:13 +02:00
Eelco Dolstra ce3173edc1
nix flake info --json: Don't evaluate
This makes its behaviour consistent with the non-json
variant. Querying the outputs should be done by another command
(e.g. 'nix search')
2020-04-06 14:39:47 +02:00
Eelco Dolstra ed13457dbf nix flake info --json: Show TreeInfo 2020-04-02 11:51:34 +02:00
Eelco Dolstra bd10a07d17 Registry: Use a struct instead of a tuple for entries 2020-04-01 23:03:27 +02:00
Eelco Dolstra 77ffaea4fa Add a system-wide flake registry /etc/nix/registry.json
One application for this is pinning the 'nixpkgs' flake to the exact
revision used to build the NixOS system, e.g.

  {
      "flakes": [
          {
              "from": {
                  "id": "nixpkgs",
                  "type": "indirect"
              },
              "to": {
                  "owner": "NixOS",
                  "repo": "nixpkgs",
                  "type": "github",
                  "rev": "b0c285807d6a9f1b7562ec417c24fa1a30ecc31a"
              }
          }
      ],
      "version": 2
  }
2020-04-01 22:56:50 +02:00
Eelco Dolstra e0a0ae0467 Move fetchers from libstore to libfetchers 2020-03-30 14:04:53 +02:00
Eelco Dolstra 4989c04dd2 nix flake info --json: Dump attr-style flakeref 2020-03-28 22:59:38 +01:00
Eelco Dolstra 2287e2f279 nix flake info: Show flake subdirectory 2020-03-28 18:05:50 +01:00
Eelco Dolstra 2a4e4f6a6e
Unified fetcher caching system 2020-03-17 22:35:29 +01:00
Eelco Dolstra ae9119167e Change the lock file to a graph
This enables support for cycles between flakes.
2020-03-12 22:06:57 +01:00
Eelco Dolstra 73c9840569 Restore subdir -> dir
Got this mixed up somewhere.
2020-02-20 23:44:06 +01:00
Eelco Dolstra d068f9ffff Restore subdir support in registries
Hacky...
2020-02-20 22:14:44 +01:00
Eelco Dolstra 46a284263f Fix build 2020-02-14 22:45:33 +01:00
Eelco Dolstra 0b013a54dc findAlongAttrPath(): Return position 2020-02-07 14:08:24 +01:00
Eelco Dolstra b270869466
Renamed ref / resolvedRef -> lockedRef 2020-02-02 00:06:59 +01:00
Eelco Dolstra 887730aab3
Remove superfluous TreeInfo::rev field 2020-02-01 23:54:20 +01:00
Eelco Dolstra 8451298b35
Factor out TreeInfo 2020-02-01 22:06:55 +01:00
Eelco Dolstra 5d70b454be
nix flake update: Imply --refresh 2020-02-01 12:30:49 +01:00
Eelco Dolstra 678301072f nix flake list-inputs: Pretty-print the tree 2020-01-31 14:09:27 +01:00
Eelco Dolstra a6e2b6b360 nix flake deps -> nix flake list-inputs
Also add a --json flag.
2020-01-31 13:00:50 +01:00
Eelco Dolstra ebfbfe9515 Use std::nullopt 2020-01-30 01:10:26 +01:00
Eelco Dolstra 3c54e9ba01 Add 'nix flake archive' command
This copies a flake and all its inputs recursively to a store (e.g. a
binary cache). This is intended to enable long-term reproducibility
for flakes. However this will also require #3253.

Example:

  $ nix flake archive --json --to file:///tmp/my-cache nixops
  {"path":"/nix/store/272igzkgl1gdzmabsjvb2kb2zqbphb3p-source","inputs":{"nixops-aws":{"path":"/nix/store/ybcykw13gr7iq1pzg18iyibbcv8k9q1v-source","inputs":{}},"nixops-hetzner":{"path":"/nix/store/6yn0205x3nz55w8ms3335p2841javz2d-source","inputs":{}},"nixpkgs":{"path":"/nix/store/li3lkr2ajrzphqqz3jj2avndnyd3i5lc-source","inputs":{}}}}

  $ ll /tmp/my-cache
  total 16
  -rw-r--r-- 1 eelco users 403 Jan 30 01:01 272igzkgl1gdzmabsjvb2kb2zqbphb3p.narinfo
  -rw-r--r-- 1 eelco users 403 Jan 30 01:01 6yn0205x3nz55w8ms3335p2841javz2d.narinfo
  -rw-r--r-- 1 eelco users 408 Jan 30 01:01 li3lkr2ajrzphqqz3jj2avndnyd3i5lc.narinfo
  drwxr-xr-x 2 eelco users   6 Jan 30 01:01 nar
  -rw-r--r-- 1 eelco users  21 Jan 30 01:01 nix-cache-info
  -rw-r--r-- 1 eelco users 404 Jan 30 01:01 ybcykw13gr7iq1pzg18iyibbcv8k9q1v.narinfo

Fixes #3336.
2020-01-30 01:00:34 +01:00
Eelco Dolstra 26f895a26d Clean up the lock file handling flags
Added a flag --no-update-lock-file to barf if the lock file needs any
changes. This is useful for CI systems if you're building a
checkout. Fixes #2947.

Renamed --no-save-lock-file to --no-write-lock-file. It is now a fatal
error if the lock file needs changes but --no-write-lock-file is not
given.
2020-01-29 21:04:28 +01:00
Eelco Dolstra f68bed7f67 Add flag --override-input to override specific lock file entries
E.g.

  $ nix flake update ~/Misc/eelco-configurations/hagbard \
    --override-input 'dwarffs/nixpkgs' ../my-nixpkgs

overrides the 'nixpkgs' input of the 'dwarffs' input of the top-level
flake.

Fixes #2837.
2020-01-29 18:41:25 +01:00
Eelco Dolstra 872a22fa23 resolveFlake -> lockFlake
"resolve" is ambiguous (also used for registry resolution).
2020-01-22 20:59:59 +01:00
Eelco Dolstra 90d55ed275 Fix 'nix flake update' 2020-01-22 19:28:27 +01:00
Eelco Dolstra 32f31a8c63 nix flake info: Don't show empty descriptions 2020-01-22 17:20:21 +01:00
Eelco Dolstra 9f4d8c6170 Pluggable fetchers
Flakes are now fetched using an extensible mechanism. Also lots of
other flake cleanups.
2020-01-21 22:56:04 +01:00
Eelco Dolstra ecb3a1afa2 Merge remote-tracking branch 'origin/master' into flakes 2019-12-11 14:53:30 +01:00
Eelco Dolstra ce27920936 Add start of 'nix profile' command 2019-10-22 00:22:15 +02:00
Eelco Dolstra 7d38060a0d Support non-x86_64-linux system types in flakes
A command like

  $ nix run nixpkgs#hello

will now build the attribute 'packages.${system}.hello' rather than
'packages.hello'. Note that this does mean that the flake needs to
export an attribute for every system type it supports, and you can't
build on unsupported systems. So 'packages' typically looks like this:

  packages = nixpkgs.lib.genAttrs ["x86_64-linux" "i686-linux"] (system: {
    hello = ...;
  });

The 'checks', 'defaultPackage', 'devShell', 'apps' and 'defaultApp'
outputs similarly are now attrsets that map system types to
derivations/apps. 'nix flake check' checks that the derivations for
all platforms evaluate correctly, but only builds the derivations in
'checks.${system}'.

Fixes #2861. (That issue also talks about access to ~/.config/nixpkgs
and --arg, but I think it's reasonable to say that flakes shouldn't
support those.)

The alternative to attribute selection is to pass the system type as
an argument to the flake's 'outputs' function, e.g. 'outputs = { self,
nixpkgs, system }: ...'. However, that approach would be at odds with
hermetic evaluation and make it impossible to enumerate the packages
provided by a flake.
2019-10-15 18:16:29 +02:00
Eelco Dolstra a0bd088d84
Move addRegistrOverrides 2019-10-08 17:00:55 +02:00
Eelco Dolstra 21304c11f9
uri -> url for consistency 2019-10-08 17:00:55 +02:00
Eelco Dolstra 382aa05ff7 nix flake info --json: Get rid of duplicate getFlake() call
Also fix some gcc warnings.
2019-09-22 21:53:01 +02:00
Eelco Dolstra 5573365dff
nix flake check: Validate nixosConfigurations outputs 2019-09-19 20:15:42 +02:00
Eelco Dolstra c67407172d
Record original flakerefs in the lock file again
If 'input.<name>.uri' changes, then the entry in the lockfile for
input <name> should be considered stale.

Also print some messages when lock file entries are added/updated.
2019-09-18 21:57:57 +02:00
Eelco Dolstra 55e55b34e6
nix flake check: Check hydraJobs 2019-09-10 17:39:55 +02:00
Eelco Dolstra 4b9dee6bcc
nix flake check: Do some basic checks on NixOS modules
Also show more position info.
2019-09-10 15:25:32 +02:00
Eelco Dolstra dc3f52a144
nix flake check: Check overlays 2019-09-10 14:52:22 +02:00
Eelco Dolstra 80c36d4562 Remove 'name' attribute from flakes
This is no longer needed since flakes are given an identity in the
'inputs' attribute.
2019-08-30 16:38:27 +02:00
Eelco Dolstra ad42a78469
Rename 'epoch' -> 'edition' 2019-07-11 13:54:53 +02:00
Eelco Dolstra d4fe9daed6
Simplify getFlake() / fetchFlake() logic 2019-06-21 19:04:58 +02:00
Eelco Dolstra a0de58f471
Make subcommand construction in MultiCommand lazy 2019-06-18 16:25:31 +02:00
Eelco Dolstra eb18aedccb
Merge pull request #2951 from NixOS/app-improvements
App / check improvements
2019-06-18 16:15:40 +02:00
Eelco Dolstra d4a48b12fa
Merge pull request #2917 from CSVdB/docs
Updated flake documentation
2019-06-18 16:05:55 +02:00
Eelco Dolstra 8a6704d826 Updated documentation 2019-06-18 11:08:04 +02:00
Eelco Dolstra 556f33422d
nix flake check: Ignore legacyPackages 2019-06-18 09:45:14 +02:00
Eelco Dolstra f2fcc163fa
nix flake check: Warn about unknown flake outputs 2019-06-17 18:05:32 +02:00
Eelco Dolstra 9d1207c02c
nix flake check: Check apps 2019-06-17 17:59:57 +02:00
Eelco Dolstra 3b2ebd029c
nix flake info --json: Revive enumerating the outputs 2019-06-17 17:31:34 +02:00
Eelco Dolstra 54aff8430c
Move flake-related stuff to src/libexpr/flake 2019-06-05 16:51:54 +02:00
Eelco Dolstra 1e53a07712
Make non-flake inputs lazy
Also add a proper test for non-flake inputs.
2019-06-04 20:56:42 +02:00
Eelco Dolstra 6dbd5c26e6
Make flake input fetching lazy
As long as the flake input is locked, it is now only fetched when it
is evaluated (e.g. "nixpkgs" is fetched when
"inputs.nixpkgs.<something>" is evaluated).

This required adding an "id" attribute to the members of "inputs" in
lockfiles, e.g.

  "inputs": {
    "nixpkgs/release-19.03": {
      "id": "nixpkgs",
      "inputs": {},
      "narHash": "sha256-eYtxncIMFVmOHaHBtTdPGcs/AnJqKqA6tHCm0UmPYQU=",
      "nonFlakeInputs": {},
      "uri": "github:edolstra/nixpkgs/e9d5882bb861dc48f8d46960e7c820efdbe8f9c1"
    }
  }

because the flake ID needs to be known beforehand to construct the
"inputs" attrset.

Fixes #2913.
2019-06-04 19:17:03 +02:00
Eelco Dolstra 094539ef4a
Rename requires -> inputs, provides -> outputs
Issue #2828.
2019-05-31 09:59:48 +02:00
Eelco Dolstra 49436bdbb7
nix flake info --json: List the "provides"
It also lists the contents of "checks" and "packages".

For example:

  $ nix flake info --json | jq
  {
    "branch": "HEAD",
    "description": "The purely functional package manager",
    "epoch": 2019,
    "id": "nix",
    "lastModified": 1559161142,
    "path": "/nix/store/2w2qla8735dbxah8gai8r1nsbf5x4f5d-source",
    "provides": {
      "checks": {
        "binaryTarball": {},
        "nix-copy-closure": {},
        "perlBindings": {},
        "remoteBuilds": {},
        "setuid": {}
      },
      "defaultPackage": {},
      "devShell": {},
      "hydraJobs": {},
      "packages": {
        "nix": {},
        "nix-perl-bindings": {}
      }
    },
    "revCount": 6955,
    "revision": "8cb24e04e8b6cc60e2504733afe78e0eadafcd98",
    "uri": "/home/eelco/Dev/nix"
  }

Fixes #2820.
2019-05-29 22:22:28 +02:00
Eelco Dolstra a9d3524e1f
nix flake check: Use read-only mode if we're not building 2019-05-29 21:00:44 +02:00
Eelco Dolstra 0e32b32fa3
nix flake check: Check defaultPackage, devShell and packages 2019-05-29 20:57:08 +02:00
Eelco Dolstra e0aaf05f4f
Add 'nix flake check' command
This evaluates all the 'provides' of a flake and builds the 'check'
attributes.
2019-05-29 17:25:41 +02:00
Eelco Dolstra 6e4a8c47f4
Put flake-related stuff in its own namespace 2019-05-29 15:31:07 +02:00
Eelco Dolstra 0f840483c7
Add date of last commit to SourceInfo
This is primarily useful for version string generation, where we need
a monotonically increasing number. The revcount is the preferred thing
to use, but isn't available for GitHub flakes (since it requires
fetching the entire history). The last commit timestamp OTOH can be
extracted from GitHub tarballs.
2019-05-29 10:10:36 +02:00
Eelco Dolstra 444786e6d3
nix flake info: Add missing newline 2019-05-28 14:11:19 +02:00
Eelco Dolstra 46294d60cd
printFlakeInfo: Separate JSON output 2019-05-28 14:01:57 +02:00
Eelco Dolstra e7e7a03baf
nix flake deps: Remove --json flag for now
It doesn't produce valid JSON at the moment (but a concatenation of
JSON objects). Anyway we probably should merge this command info 'nix
flake info'.
2019-05-28 13:22:11 +02:00
Eelco Dolstra fdf06ce72f
printFlakeInfo/printNonFlakeInfo: Factor out commonality 2019-05-28 13:21:06 +02:00
Eelco Dolstra 25e497bf9c
nix flake info/deps: Stop progress bar before printing output 2019-05-28 13:14:27 +02:00
Eelco Dolstra dda4f7167b
Remove redundant resolvedRef fields since they're already in SourceInfo 2019-05-28 13:12:43 +02:00
Eelco Dolstra 6d7efcfaeb
Store SourceInfo in Flake and NonFlake
This deduplicates some shared fields. Factoring out the commonality is
useful in places like makeFlakeValue().
2019-05-28 13:06:43 +02:00
Eelco Dolstra 66f1d7ee95 Fetch the flake registry from the NixOS/flake-registry repo 2019-05-22 22:56:46 +02:00
Eelco Dolstra e414bde6f9 Check the flake epoch
Closes #2883.
2019-05-22 14:31:40 +02:00
Eelco Dolstra 3e8ef9eb22 nix flake deps: Print flake dependencies 2019-05-22 14:04:18 +02:00
Eelco Dolstra 70136a9bf4 Move flake-related flags into a separate class
Also, rename --dont-save-lock-file to --no-save-lock-file and change
noRegistries to useRegistries.
2019-05-22 14:04:18 +02:00
Eelco Dolstra 2468672e30 Improve FlakeCommand
It now handles commonality like calling getFlake() and resolving
relative local flake refs.

Fixes #2822.
2019-05-22 14:03:28 +02:00
Nick Van den Broeck ef6ae61503 Lockfile handling in resolveFlake is fixed 2019-05-17 14:50:10 +02:00
Nick Van den Broeck 98f20dee41 Give errors in resolveFlake
If DontUpdate but the lockfile isn't correct
2019-05-16 15:02:25 +02:00
Nick Van den Broeck d9ad3723d5 Fixed issue 65
lockfile updating
2019-05-16 15:02:05 +02:00
Nick Van den Broeck 201f92e02c Fixed Flake data type and flake fetching 2019-05-07 05:57:08 +02:00
Eelco Dolstra 4588a6ff3c Merge remote-tracking branch 'tweag/flakeFlags' into flakes 2019-04-30 12:46:54 +02:00
Nick Van den Broeck 24b35bf9e7 Fixed issue #13 2019-04-30 11:26:45 +02:00
Nick Van den Broeck 35d1c95f7f Fix flag registry order 2019-04-30 11:04:31 +02:00