Commit graph

11532 commits

Author SHA1 Message Date
Eelco Dolstra 52a3b2ee63
Merge pull request #5524 from abathur/improve_existing_backup_profile_msg
installer: improve existing rc backup nag
2021-11-10 20:50:09 +01:00
Eelco Dolstra a218cfd6c7
Merge pull request #5528 from matthewbauer/recognize-nixosModule
Recognize singular "nixosModule" in nix flake show
2021-11-10 12:17:11 +01:00
Sergei Trofimovich edfc5b2f12 ca-specific-schema.sql: add index on RealisationsRefs(referrer) and (outputPath)
For a typical desktop system (~2K packages) we can easily get 100K
entries in RealisationsRefs. Without indices query for RealisationsRefs
requires linear scan.

RealisationsRefs(referrer)
--------------------------

Inefficiency is seen as a 100% CPU load of nix-daemon for the following
scenario:

    $ nix edit -f . bash # add unused environment variable, like FOO="1"
    # populate RealisationsRefs, build fresh system
    $ nix build -f nixos system --arg config '{ contentAddressedByDefault = true; }'
    $ nix edit -f . bash # add unused environment variable, like FOO="2"
    $ time nix build -f nixos system --arg config '{ contentAddressedByDefault = true; }'

In this case `bash `will be rebuilt a few times and then rest of CPU
time is spent on scanning RealisationsRefs table (about 5 CPU-minutes
on my machine).

Before the change:

    $ time nix build -f nixos system ... # step 4 above
    real    34m3,613s
    user    0m5,232s
    sys     0m0,758s

Of all this time about 29.5 minutes are taken by nix-daemon's CPU time.

After the change:

    $ time nix build -f nixos system ... # step 4 above
    real    4m50,061s
    user    0m5,038s
    sys     0m0,677s

Of all this time about 1 minute is taken by nix-daemon's CPU time.
Most of the time is spent polling for non-existent realisations on
cache-nixos.org.

Realisations(outputPath)
------------------------

After running CA system for two weeks I got ~1M entries in Realisations
table. `nix-collect-garbage` became very slow (seemingly 100 path deletions
per second). It happens due to a slow cascading delete from Realisations
triggered by deletion from ValidPaths.

The fix is to add an index on primary key from ValidPaths(id) that
triggers cascading deletions.

Before the change:
    $ time nix-collect-garbage -d --max-freed 100G
    <interrupted before finish, took too long>
    real    23m32.411s
    user    17m49.679s
    sys     4m50.609s

Most of time was spent in re-scanning Realisations table on each path deletion.

After the change:
    $ time nix-collect-garbage -d --max-freed 100G

    real    8m43.226s
    user    6m16.317s
    sys     1m40.188s

Time is spent scanning sqlite indices and in kernel when unlinking directories.
2021-11-10 08:32:05 +00:00
Matthew Bauer ceeecf2f9e Recognize singular "nixosModule" in nix flake show
This makes nixosModule appears as a "NixOS Module" like nixosModules
does.
2021-11-09 11:14:15 -06:00
Eelco Dolstra 67179472df
Merge pull request #5494 from tweag/balsoft/allow-references-in-addPath
Allow references in addPath
2021-11-09 15:57:39 +01:00
Travis A. Everett f7859eef49 installer: improve existing rc backup nag 2021-11-09 08:14:51 -06:00
Eelco Dolstra 884ef336c4
Merge pull request #5519 from edolstra/move-unshare
Unshare mount namespace in main()
2021-11-09 12:33:25 +01:00
Alexander Bantyev 0b005bc9d6
addToStore, addToStoreFromDump: refactor: pass refs by const reference
Co-Authored-By: Eelco Dolstra <edolstra@gmail.com>
2021-11-09 12:24:49 +03:00
Eelco Dolstra 25d2316e8f
Merge pull request #5520 from NixOS/dependabot/github_actions/actions/checkout-2.4.0
Bump actions/checkout from 2.3.5 to 2.4.0
2021-11-09 09:16:42 +01:00
Travis A. Everett 69431edbc1 installer: clarify starting assumption task
We had a macOS user present in Matrix with some confusion because the
lack of a clear task statement here made them think the error meant
that a problem had occurred during the preceding task in a macOS
install: "Fixing any leftover Nix volume state"
2021-11-08 17:03:09 -06:00
dependabot[bot] 732dd90428
Bump actions/checkout from 2.3.5 to 2.4.0
Bumps [actions/checkout](https://github.com/actions/checkout) from 2.3.5 to 2.4.0.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v2.3.5...v2.4.0)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-11-08 22:00:58 +00:00
Eelco Dolstra ff2af4d64e Unshare mount namespace in main()
Doing it as a side-effect of calling LocalStore::makeStoreWritable()
is very ugly.

Also, make sure that stopping the progress bar joins the update
thread, otherwise that thread should be unshared as well.
2021-11-08 22:00:45 +01:00
Eelco Dolstra 7a71621b7c Merge branch 'fix-writable-shell' of https://github.com/yorickvP/nix 2021-11-08 21:12:51 +01:00
Eelco Dolstra 6c2af1f201
Merge pull request #5434 from timothyklim/git-url-submodules
Support building flakes from a Git repo url with submodules
2021-11-08 14:13:42 +01:00
Eelco Dolstra d9c9d0e0eb
Merge pull request #5500 from abathur/fix_darwin_existing_mounted_volume
darwin-install: fix already-mounted store volumes
2021-11-08 13:55:19 +01:00
Eelco Dolstra 736d6ab721
Merge pull request #5504 from NixOS/flake-options-and-daemon
Make the flake options work when using the daemon
2021-11-08 13:54:55 +01:00
Eelco Dolstra c9ecc0948b
Merge pull request #5506 from NixOS/fix-post-hook-test-with-different-daemon
Make the post-build-hook use the daemon Nix package
2021-11-08 13:53:30 +01:00
Eelco Dolstra 0fd96eeb09
Merge pull request #5517 from Zimmi48/fix-cli-guideline-typo
Fix some typos in CLI guideline.
2021-11-08 13:52:57 +01:00
Eelco Dolstra b5cb31e032
Merge pull request #5514 from andir/let-body-unused
Remove unused "<let-body>" symbol
2021-11-08 13:52:32 +01:00
Théo Zimmermann d589782fb0
Fix some typos in CLI guideline. 2021-11-08 08:24:15 +01:00
Andreas Rammhold 8e7359db64
Remove unused "<let-body>" symbol
The requirement for the symbol has been removed since at least 7d47498.
2021-11-07 18:26:43 +01:00
Alexander Bantyev 9d4dcff37a
addPath: allow paths with references
Since 4806f2f6b0, we can't have paths with
references passed to builtins.{path,filterSource}. This prevents many cases
of those functions called on IFD outputs from working. Resolve this by
passing the references found in the original path to the added path.
2021-11-05 22:41:30 +03:00
Théophane Hufschmitt 3d6ee223d6
Merge pull request #5476 from NixOS/ca/fix-build-in-nix-repl
Fix `nix repl`’s building of CA derivations
2021-11-05 17:35:18 +01:00
regnat 1f3c3a3785 Make the flake options work when using the daemon
When setting flake-local options (with the `nixConfig` field), forward
these options to the daemon in case we’re using one.

This is necessary in particular for options like `binary-caches` or
`post-build-hook` to make sense.

Fix <343239fc8a (r44356843)>
2021-11-05 16:19:16 +01:00
Eelco Dolstra a1c1b0e553
Merge pull request #5501 from edolstra/optimize-calls
Optimize primop calls
2021-11-05 12:57:19 +01:00
regnat 93eadd5803 Make the post-build-hook use the daemon Nix package
Having the `post-build-hook` use `nix` from the client package can lead
to a deadlock in case there’s a db migration to do between both, as a
`nix` command running inside the hook will run as root (and as such will
bypass the daemon), so might trigger a db migration, which will get
stuck trying to get a global lock on the DB (as the daemon that ran the
hook already has a lock on it).
2021-11-05 11:11:33 +01:00
regnat 1a4c9ba50b Fix nix repl’s building of CA derivations
When running a `:b` command in the repl, after building the derivations
query the store for its outputs rather than just assuming that they are
known in the derivation itself (which isn’t true for CA derivations)

Fix #5328
2021-11-05 11:04:14 +01:00
Eelco Dolstra 7d6017b7a9
Merge pull request #5493 from jtojnar/patch-1
docs: Correct fallback user config path
2021-11-04 20:31:47 +01:00
Travis A. Everett abdf9f2a6e darwin-install: fix already-mounted store volumes
This adds an explicit unmount of the store volume to avoid cases
where the installer can hang in await_volume when:
- the user already has a store volume
- that volume is already mounted somewhere other than /nix
- they do not take a path through the installer that results in an
  explicit unmount (as both removing and encrypting the volume
  would do)
2021-11-04 14:09:40 -05:00
Eelco Dolstra 40925337a9 Remove maxPrimOpArity 2021-11-04 15:04:07 +01:00
Eelco Dolstra 05560f6350 Fix function-trace test case 2021-11-04 15:04:04 +01:00
Eelco Dolstra acd6bddec7 Fix derivation primop 2021-11-04 15:04:00 +01:00
Eelco Dolstra cbfbf71e08 Use callFunction() with an array for some calls with arity > 1 2021-11-04 15:03:57 +01:00
Eelco Dolstra bcf4780006 Add level / displacement types 2021-11-04 15:03:45 +01:00
Eelco Dolstra 81e7c40264 Optimize primop calls
We now parse function applications as a vector of arguments rather
than as a chain of binary applications, e.g. 'substring 1 2 "foo"' is
parsed as

  ExprCall { .fun = <substring>, .args = [ <1>, <2>, <"foo"> ] }

rather than

  ExprApp (ExprApp (ExprApp <substring> <1>) <2>) <"foo">

This allows primops to be called immediately (if enough arguments are
supplied) without having to allocate intermediate tPrimOpApp values.

On

  $ nix-instantiate --dry-run '<nixpkgs/nixos/release-combined.nix>' -A nixos.tests.simple.x86_64-linux

this gives a substantial performance improvement:

  user CPU time:      median =      0.9209  mean =      0.9218  stddev =      0.0073  min =      0.9086  max =      0.9340  [rejected, p=0.00000, Δ=-0.21433±0.00677]
  elapsed time:       median =      1.0585  mean =      1.0584  stddev =      0.0024  min =      1.0523  max =      1.0623  [rejected, p=0.00000, Δ=-0.20594±0.00236]

because it reduces the number of tPrimOpApp allocations from 551990 to
42534 (i.e. only small minority of primop calls are partially
applied) which in turn reduces time spent in the garbage collector.
2021-11-04 15:03:40 +01:00
Eelco Dolstra ab35cbd675 StaticEnv: Use std::vector instead of std::map 2021-11-04 15:03:34 +01:00
Eelco Dolstra c4bd6a15c2 Add helper function to check whether a function arg is 'X' or '_X'
Also allow '_'.
2021-11-04 14:52:35 +01:00
Jan Tojnar e5d4c2235f
docs: Correct fallback user config path
This is in line with XDG Base Directory Specification, where ~/.config is supposed to be used when XDG_CONFIG_HOME is unset.

It also better matches the reality, where ~/.config/nix.conf does not seem to be used.
2021-11-04 11:41:29 +01:00
Eelco Dolstra c5fd0b46ae
Merge pull request #5486 from mohe2015/bugfixes2
Fix leaking pthread_attr_t
2021-11-04 10:44:33 +01:00
Eelco Dolstra 3f447bcd5f
Merge pull request #5488 from JanCVanB/patch-1
Reword "we"s to "I"s for consistency
2021-11-04 10:41:35 +01:00
Jan Van Bruggen 7d56174c1e
Reword "we"s to "I"s for consistency
This script uses multiple forms of the first-person POV:
1. "We" to refer to the Nix team (1e7c796e66/scripts/install-multi-user.sh (L72))
2. "We" to refer to the combination of the installation script & the user/executor (1e7c796e66/scripts/install-multi-user.sh (L710))
3. "We" to refer to the installation script alone (1e7c796e66/scripts/install-multi-user.sh (L602))
4. "I" to refer to the installation script alone (1e7c796e66/scripts/install-multi-user.sh (L200))

Since I prefer POV 4 to POV 3, this changes all instances of POV 3 to POV 4.
2021-11-03 18:42:32 -06:00
Moritz Hedtke 6f291ed718
Fix leaking pthread_attr_t
pthread_attr_destroy was not called.
2021-11-03 22:54:16 +01:00
Eelco Dolstra 1e7c796e66
Merge pull request #5475 from doronbehar/SQLiteWAL-vfs
libstore: Use unix-dotfile vfs if useSQLiteWAL is false
2021-11-03 21:20:27 +01:00
Eelco Dolstra ae14113969
Merge pull request #5477 from league/check-overlay-args
In checkOverlay, accept underscored names for final/prev args.
2021-11-03 21:18:58 +01:00
Domen Kožar f1c9ee0364
Merge pull request #5480 from rex4539/typos
Fix typos
2021-11-03 11:07:19 -06:00
Dimitris Apostolou c34cc5e488
Fix typos 2021-11-03 18:11:20 +02:00
Christopher League 3f070cc417 In checkOverlay, accept underscored names for final/prev args.
Resolves #4416.
2021-11-03 09:25:27 -04:00
Doron Behar 14fcf17277 libstore: Use unix-dotfile vfs if useSQLiteWAL is false 2021-11-03 14:19:11 +02:00
Eelco Dolstra 133905b309
Merge pull request #5471 from simon04/patch-1
command-ref/nix-shell: fix --pure, --keep
2021-11-03 11:15:13 +01:00
regnat f4c869977c Make the DrvOutputSubstitutionGoal more async 2021-11-03 06:51:34 +01:00