John Ericson
1b8ebe92dc
Merge remote-tracking branch 'obsidian/split_build_cc' into typed-goal-maps
2020-10-12 20:47:22 +00:00
John Ericson
a73959e6be
Merge remote-tracking branch 'upstream/master' into split_build_cc
2020-10-12 17:20:46 +00:00
John Ericson
d334fd4882
Merge branches 'split_build_hh-0', 'split_build_hh-1', 'split_build_hh-2', 'split_build_hh-3' and 'split_build_hh-4' into split_build_cc
2020-10-12 17:20:12 +00:00
John Ericson
542972f029
Trim worker.hh
2020-10-12 17:16:48 +00:00
John Ericson
0e2306204a
Rename to hand-hold git (worker.hh)
2020-10-12 17:16:48 +00:00
John Ericson
3ffa3546bd
Trim substitution-goal.hh
2020-10-12 17:16:25 +00:00
John Ericson
e77a2344d5
Rename to hand-hold git (substitution-goal.hh)
2020-10-12 17:16:25 +00:00
John Ericson
d585b4c54f
Trim hook-instance.hh
2020-10-12 17:16:13 +00:00
John Ericson
10b749a156
Rename to hand-hold git (hook-instance.hh)
2020-10-12 17:16:13 +00:00
John Ericson
8067d32f2a
Trim goal.hh
2020-10-12 17:16:00 +00:00
John Ericson
0d0e345cdc
Rename to hand-hold git (goal.hh)
2020-10-12 17:16:00 +00:00
John Ericson
2ce726947a
Trim derivation-goal.hh
2020-10-12 17:15:32 +00:00
John Ericson
4eb8c69853
Rename to hand-hold git (derivation-goal.hh)
2020-10-12 17:15:32 +00:00
John Ericson
f7099965bf
Change .cc files to use split build headers
2020-10-12 17:08:52 +00:00
John Ericson
5a97621d6d
Prepare for build/*.hh headers
2020-10-12 17:07:51 +00:00
volth
eee18f88dd
Handle amount of disk space saved by hard linking being negative
...
Fixes bogus messages like "currently hard linking saves 17592186044416.00 MiB".
2020-10-12 16:06:38 +00:00
John Ericson
15fdb7cc6b
Split out uds-remote-store.{cc.hh}
2020-10-11 17:37:05 +00:00
John Ericson
38e3897162
Copy {uds-,}remote-store.{cc,hh}
...
This prepares for the splitting that happens in the next commit.
2020-10-11 17:18:26 +00:00
John Ericson
5c74a6147b
Properly type the derivation and substitution goal maps
...
As a bonus, Worker::removeGoal is less inefficient.
2020-10-11 17:07:14 +00:00
John Ericson
8cc510fb79
Merge branches 'build-1', 'build-2', 'build-3', 'build-4', 'build-5', 'build-6', 'build-7' and 'build-8' into split_build_cc
2020-10-11 16:44:34 +00:00
John Ericson
bcb67e1ed8
Trim lock.cc
2020-10-11 16:44:19 +00:00
John Ericson
e0be04129b
Rename to hand-hold git (lock.cc)
2020-10-11 16:44:14 +00:00
John Ericson
dbc588651c
Trim lock.hh
2020-10-11 16:44:07 +00:00
John Ericson
d0004bfcab
Rename to hand-hold git (lock.hh)
2020-10-11 16:43:59 +00:00
John Ericson
eed53ed87a
Trim build/worker.cc
2020-10-11 16:43:51 +00:00
John Ericson
904e315dae
Rename to hand-hold git (build/worker.cc)
2020-10-11 16:43:42 +00:00
John Ericson
d24ffe0eb1
Trim build/substitution-goal.cc
2020-10-11 16:43:24 +00:00
John Ericson
3633b3572b
Rename to hand-hold git (build/substitution-goal.cc)
2020-10-11 16:43:18 +00:00
John Ericson
4bdff7d1b0
Trim build/local-store-build.cc
2020-10-11 16:43:12 +00:00
John Ericson
dc5225cde5
Rename to hand-hold git (build/local-store-build.cc)
2020-10-11 16:43:05 +00:00
John Ericson
159054f730
Trim build/hook-instance.cc
2020-10-11 16:42:35 +00:00
John Ericson
f0b8987299
Rename to hand-hold git (build/hook-instance.cc)
2020-10-11 16:42:08 +00:00
John Ericson
819fe848ac
Trim build/goal.cc
2020-10-11 16:41:58 +00:00
John Ericson
184bfc301e
Rename to hand-hold git (build/goal.cc)
2020-10-11 16:41:18 +00:00
John Ericson
3bab1c5bb0
Trim build/derivation-goal.cc
2020-10-11 16:41:11 +00:00
John Ericson
9629290eda
Rename to hand-hold git (build/derivation-goal.cc)
2020-10-11 16:40:52 +00:00
John Ericson
a4f0fecb03
Trim build.hh
2020-10-11 16:40:34 +00:00
John Ericson
fc72cb0760
Rename to hand-hold git (build.hh)
2020-10-11 16:40:14 +00:00
John Ericson
428536fd75
Prepare for build/* files
2020-10-11 16:39:08 +00:00
John Ericson
aef44cbaa9
Split out commonChildInit
2020-10-11 16:38:46 +00:00
John Ericson
6cc1541782
Split out local-fs-store.hh
...
This matches the already-existing `local-fs-store.cc`.
2020-10-09 20:18:08 +00:00
Eelco Dolstra
636ec17139
Remove stray DerivationOutputsAndPaths type
2020-10-09 15:41:24 +02:00
Eelco Dolstra
97ffc1e013
Dynamically disable user namespaces if CLONE_NEWUSER fails
...
This makes builds work inside nixos-enter.
Fixes #3145 .
2020-10-07 22:46:01 +02:00
Eelco Dolstra
6aa64627c8
Support user namespaces being disabled
...
If max_user_namespaces is set to 0, then don't run the build in a user
namespace.
Fixes #4092 .
2020-10-07 22:02:36 +02:00
Eelco Dolstra
f66bbd8c7b
Doh
2020-10-07 21:25:06 +02:00
Eelco Dolstra
e705c24294
Tweak error messages
2020-10-07 17:28:43 +02:00
Eelco Dolstra
c43e882f54
Serialize exceptions from the daemon to the client
2020-10-07 17:13:54 +02:00
Eelco Dolstra
be149acfda
Serialize exceptions from the sandbox process to the parent
...
Fixes #4118 .
2020-10-07 16:34:03 +02:00
John Ericson
57d960dcd1
Remove generic std::optional<T> suppport from worker proto
...
See comment for rational; I think it's good to leave a comment lest
anyone is tempted to add such a sum-type instance again.
Fixes #4113
2020-10-07 12:50:37 +00:00
Eelco Dolstra
d761485010
Prevent a deadlock when user namespace setup fails
...
Observed on Centos 7 when user namespaces are disabled:
DerivationGoal::startBuilder() throws an exception, ~DerivationGoal()
waits for the child process to exit, but the child process hangs
forever in drainFD(userNamespaceSync.readSide.get()) in
DerivationGoal::runChild(). Not sure why the SIGKILL doesn't get
through.
Issue #4092 .
2020-10-06 18:57:23 +02:00
Eelco Dolstra
ad143c5b3b
Shut up some clang warnings
2020-10-06 14:52:30 +02:00
Eelco Dolstra
85c8be6286
Remove static variable name clashes
...
This was useful for an experiment with building Nix as a single
compilation unit. It's not very useful otherwise but also doesn't
hurt...
2020-10-06 13:49:20 +02:00
Eelco Dolstra
6691256e79
Factor out common showBytes()
2020-10-06 10:40:49 +02:00
Eelco Dolstra
d0bb544128
Add missing #pragma once
2020-10-06 10:40:07 +02:00
Eelco Dolstra
88a667e49e
Fix s3:// store
...
Fixes https://github.com/NixOS/nixos-org-configurations/issues/123 .
2020-10-05 17:53:30 +02:00
Eelco Dolstra
51c299213b
Merge pull request #3935 from obsidiansystems/binary-cache-addToStoreFromDump
...
Get rid of Hash::dummy from BinaryCacheStore
2020-10-05 14:41:28 +02:00
Eelco Dolstra
f3aba88737
Merge pull request #3895 from obsidiansystems/templated-daemon-protocol
...
More templated STL support for the daemon protocol
2020-10-05 14:40:27 +02:00
Eelco Dolstra
20a1e20d91
Style
2020-09-30 13:35:56 +02:00
John Ericson
69afaeace3
Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol
2020-09-30 00:42:28 +00:00
John Ericson
b759701652
nix::worker_proto -> worker_proto
2020-09-30 00:41:18 +00:00
John Ericson
45a0ed82f0
Revert "Use template structs instead of phantoms"
...
This reverts commit 9ab07e99f5
.
2020-09-30 00:39:06 +00:00
Kevin Quick
5e7838512e
Remove github-access-token in favor of access-token.
2020-09-29 16:26:34 -07:00
Kevin Quick
66c3959e8c
Merge branch 'master' into access-tokens
2020-09-29 08:32:06 -07:00
Eelco Dolstra
5999978a05
Make Headers an optional argument
2020-09-29 13:05:19 +02:00
Eelco Dolstra
cebd2fc35d
Merge branch 'github-api-token' of https://github.com/imalsogreg/nix
2020-09-29 12:17:00 +02:00
John Ericson
00135e13f4
Clarify comment a bit
2020-09-28 18:19:10 +00:00
John Ericson
80e335bb58
Use drvPath2
and give it a better name
2020-09-28 15:43:56 +00:00
John Ericson
10202bbf29
Merge remote-tracking branch 'upstream/master' into ca-floating-upstream
2020-09-28 15:39:11 +00:00
John Ericson
6c31297d80
Update src/libstore/binary-cache-store.cc
...
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2020-09-28 11:32:58 -04:00
Eelco Dolstra
649c465873
Merge pull request #4064 from serokell/balsoft/fix-max-jobs
...
Fix max-jobs option
2020-09-28 10:42:02 +02:00
John Ericson
25fffdda86
Remove redundant nar hash and size setting
...
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2020-09-26 10:17:30 -04:00
John Ericson
1832436526
Fix up BinaryCacheStore::addToStore taking a path
2020-09-26 04:56:29 +00:00
John Ericson
5db83dd771
BinaryCacheStore::addTextToStore include CA field
2020-09-26 03:21:36 +00:00
John Ericson
c40c832f19
Merge remote-tracking branch 'obsidian/master' into binary-cache-addToStoreFromDump
2020-09-26 00:21:45 +00:00
Gregory Hale
faa5607f54
Merge remote-tracking branch 'origin/master' into github-api-token
2020-09-25 12:10:58 -04:00
Eelco Dolstra
7d81582488
Merge pull request #3953 from obsidiansystems/basic-derivation-goal-outputs
...
Deduplicate basic derivation goals too
2020-09-25 17:21:19 +02:00
Kevin Quick
5a35cc29bf
Re-add support for github-access-token, but mark as deprecated.
2020-09-25 08:09:56 -07:00
Kevin Quick
ef2a14be19
Fix reference to older name for access-tokens config value.
2020-09-25 08:08:27 -07:00
Kevin Quick
8fba2a8b54
Update to use access-tokens configuration for github/gitlab access.
...
This change provides support for using access tokens with other
instances of GitHub and GitLab beyond just github.com and
gitlab.com (especially company-specific or foundation-specific
instances).
This change also provides the ability to specify the type of access
token being used, where different types may have different handling,
based on the forge type.
2020-09-24 22:49:44 -07:00
Alexander Bantyev
ed218e1d6c
Fix max-jobs option
...
After 0ed946aa61
, max-jobs setting (-j/--max-jobs)
stopped working.
The reason was that nrLocalBuilds (which compared to maxBuildJobs to figure
out whether the limit is reached or not) is not incremented yet when tryBuild
is started; So, the solution is to move the check to tryLocalBuild.
Closes https://github.com/nixos/nix/issues/3763
2020-09-25 00:07:42 +03:00
Eelco Dolstra
4ce8a3ed45
Hopefully fix EPERM on macOS
2020-09-23 21:29:10 +02:00
Eelco Dolstra
9a24ece122
Fix exception
2020-09-23 20:21:08 +02:00
Eelco Dolstra
236d9ee7f7
lstat() cleanup
2020-09-23 19:17:28 +02:00
Eelco Dolstra
688bd4fb50
After rewriting a path, make it read-only
2020-09-23 19:10:16 +02:00
Eelco Dolstra
cec9473871
DerivationGoal::registerOutputs(): Don't canonicalize twice
...
Fixes #4021 .
2020-09-23 18:21:22 +02:00
Eelco Dolstra
d4f8163d10
canonicalisePathMetaData_(): Change assertion to error message
2020-09-23 18:21:22 +02:00
Eelco Dolstra
e8f0b1e996
DerivationGoal::registerOutputs(): Fix bad format string
2020-09-23 18:21:22 +02:00
John Ericson
3f226f71c1
Return more info from BinaryCacheStore::addToStoreCommon
...
We don't need it yet, but we could/should in the future, and it's a
cost-free change since we already have the reference. I like it.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2020-09-23 14:40:41 +00:00
John Ericson
412b3a54fb
Clarify FIXME in BinaryCacheStore::addToStoreCommon
...
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2020-09-23 10:36:55 -04:00
John Ericson
9fbc31a65b
Get rid of Hash::dummy from BinaryCacheStore
2020-09-23 04:56:04 +00:00
John Ericson
993229cdaf
Deduplicate basic derivation goals too
...
See comments for security concerns.
Also optimize goal creation by not traversing map twice.
2020-09-22 17:13:59 +00:00
John Ericson
e9fc2031f0
Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol
2020-09-22 14:18:31 +00:00
regnat
67cc356bb3
Merge remote-tracking branch 'origin/master' into non-ca-depending-on-ca
2020-09-22 16:11:25 +02:00
Eelco Dolstra
980edd1f3a
RemoteStore::addCAToStore(): Don't hold connection while calling queryPathInfo()
...
This leads to a deadlock if we're at the connection limit.
2020-09-22 15:28:20 +02:00
Eelco Dolstra
5b107f2c5f
Merge pull request #4038 from maljub01/master
...
Add a nix.conf option for allowing a symlinked store
2020-09-22 13:19:22 +02:00
Eelco Dolstra
35a0ac1838
Style fixes
2020-09-22 11:40:19 +02:00
Eelco Dolstra
92ac8df0ec
Merge branch 'add-ca-to-store' of https://github.com/hercules-ci/nix
2020-09-22 11:31:33 +02:00
John Ericson
3786a801c3
Merge remote-tracking branch 'upstream/master' into ca-floating-upstream
2020-09-22 04:15:55 +00:00
John Ericson
b92d3b2edd
Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol
2020-09-22 00:45:55 +00:00
Cole Helbling
ba37299a03
Serialize SandboxMode enum to string for JSON
...
Rather than showing an integer as the default, instead show the boolean
referenced in the description.
The nix.conf.5 manpage used to show "default: 0", which is unnecessarily
opaque and confusing (doesn't 0 mean false, even though the default is
true?); now it properly shows that the default is true.
2020-09-21 10:36:45 -07:00
Eelco Dolstra
0716adaa8b
abstractsettingtojson.hh -> abstract-setting-to-json.hh
2020-09-21 18:49:43 +02:00
Eelco Dolstra
340ca382c4
Don't include nlohmann/json.hpp in globals.hh
...
This reduces compilation time by 207s.
Issue #4045 .
2020-09-21 18:47:18 +02:00
Eelco Dolstra
d51ba43047
Move Callback into its own header
...
This gets rid of the inclusion of <future> in util.hh, cutting
compilation time by ~20s (CPU time).
Issue #4045 .
2020-09-21 18:42:21 +02:00
Marwan Aljubeh
f80ffeb8c9
Update the variable name accordingly
2020-09-21 17:29:08 +01:00
Eelco Dolstra
e8e1d420f3
Don't include <regex> in header files
...
This reduces compilation time by ~15 seconds (CPU time).
Issue #4045 .
2020-09-21 18:22:45 +02:00
Marwan Aljubeh
4e1a04733d
Use a better name for the config option
2020-09-21 16:32:22 +01:00
Robert Hensing
ca30abb3fb
Document addCAToStore/addToStoreFromDump source drainage
...
Also checked that all usages satisfy the requirement and
removed dead code.
2020-09-21 07:55:47 +02:00
Robert Hensing
fa08db5c4c
wopAddToStore: return ValidPathInfo
...
A ValidPathInfo is created anyway. By returning it we can save a
roundtrip and we have a nicer interface.
2020-09-21 07:55:47 +02:00
Robert Hensing
7c68264085
wopAddToStore: add RepairFlag
2020-09-21 07:55:47 +02:00
Robert Hensing
fbf509c113
parseContentAddressMethodPrefix: use string_view
...
Co-authored-by: John Ericson <git@JohnEricson.me>
2020-09-21 07:55:47 +02:00
Robert Hensing
8279178b07
Move FramedSink next to FramedSource
2020-09-21 07:55:47 +02:00
Robert Hensing
ecc8088cb7
wopAddToStore: Throw to clarify unused refs
...
Co-authored-by: John Ericson <git@JohnEricson.me>
2020-09-21 07:55:47 +02:00
Robert Hensing
c602ebfb34
Refactor wopAddToStore to make wopAddTextToStore obsolete
2020-09-21 07:55:45 +02:00
Robert Hensing
e34fe47d0c
Overhaul wopAddToStore
2020-09-21 07:54:05 +02:00
Marwan Aljubeh
e40772cd35
Lint issue: replacing tabs with spaces
2020-09-18 17:18:45 +01:00
Marwan Aljubeh
c00e078343
Add a nix.conf option for allowing a symlinked store
2020-09-18 17:10:39 +01:00
Robert Hensing
14b30b3f3d
Move FramedSource and FramedSink, extract withFramedSink
2020-09-17 20:21:04 +02:00
Robert Hensing
dfa547c6a8
Add ContentAddressMethod and parse/render it
2020-09-17 20:21:04 +02:00
Robert Hensing
9ee3122ec7
Remove redundant import
2020-09-17 20:21:04 +02:00
John Ericson
b7df353f27
Merge remote-tracking branch 'upstream/master' into ca-floating-upstream
2020-09-17 16:33:10 +00:00
Eelco Dolstra
649d3aaf24
Merge pull request #3829 from obsidiansystems/remove-storetype-delegate-regStore
...
Remove storetype delegate reg store -- contains #3736
2020-09-17 13:55:01 +02:00
Eelco Dolstra
b94a35ef40
Merge pull request #4027 from tweag/fix-gc-of-ca-derivations
...
Fix garbage collection of CA derivations
2020-09-17 13:46:26 +02:00
regnat
520895b1da
Fix garbage collection of CA derivations
...
Fix #4026
2020-09-17 13:36:58 +02:00
John Ericson
f60b380a7f
Merge remote-tracking branch 'upstream/master' into remove-storetype-delegate-regStore
2020-09-16 22:35:24 +00:00
John Ericson
2741fffa35
Ensure resolved CA derivations are written
...
so we can link outputs to deriver and thus properly cache.
2020-09-16 17:57:32 +00:00
John Ericson
7fdbb377ba
Start to fix floating CA + remote building
2020-09-16 17:50:48 +00:00
John Ericson
c5ccebae00
Merge remote-tracking branch 'upstream/master' into ca-floating-upstream
2020-09-16 17:50:40 +00:00
Greg Hale
a303c0b6dc
Fetch commits from github/gitlab using Auth header
...
`nix flake info` calls the github 'commits' API, which requires
authorization when the repository is private. Currently this request
fails with a 404.
This commit adds an authorization header when calling the 'commits' API.
It also changes the way that the 'tarball' API authenticates, moving the
user's token from a query parameter into the Authorization header.
The query parameter method is recently deprecated and will be disallowed
in November 2020. Using them today triggers a warning email.
2020-09-16 13:46:48 -04:00
Théophane Hufschmitt
77a0e2c5be
Remove useless exception copy
...
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2020-09-16 14:00:21 +02:00
regnat
d72927aa7a
Fix the s3 store
...
Add some necessary casts in the initialisation of the store's config
2020-09-16 13:53:09 +02:00
regnat
e0817cbcdc
Don't include nlohmann/json.hpp in config.hh
...
Instead make a separate header with the template implementation of
`BaseSetting<T>::toJSONObj` that can be included where needed
2020-09-16 13:53:09 +02:00
regnat
fc2d31c423
Add (StoreConfig*)
casts to work around a GCC bug
...
Work around https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80431 that was
already there in the code but was accidentally removed in the last
commits
2020-09-16 13:53:09 +02:00
regnat
888f7afe9f
Fix build issues with gcc
2020-09-16 13:53:09 +02:00
regnat
b73adacc1e
Add a name to the stores
...
So that it can be printed by `nix describe-stores`
2020-09-16 13:53:09 +02:00
regnat
f24f0888f9
Document the new store hierarchy
2020-09-16 13:53:09 +02:00
regnat
29a632386e
fixup! Make the store plugins more introspectable
2020-09-16 13:53:09 +02:00
regnat
1129913c4e
fixup! Correctly call all the parent contructors of the stores
2020-09-16 13:53:09 +02:00
regnat
7f103dcddd
Properly filter the stores according to their declared uriSchemes
...
When opening a store, only try the stores whose `uriSchemes()` include
the current one
2020-09-16 13:53:09 +02:00
regnat
5895184df4
Correctly call all the parent contructors of the stores
...
Using virtual inheritance means that only the default constructors of
the parent classes will be called, which isn't what we want
2020-09-16 13:53:09 +02:00
regnat
22afa8fb4d
Separate store configs from the implems
...
Rework the `Store` hierarchy so that there's now one hierarchy for the
store configs and one for the implementations (where each implementation
extends the corresponding config). So a class hierarchy like
```
StoreConfig-------->Store
| |
v v
SubStoreConfig----->SubStore
| |
v v
SubSubStoreConfig-->SubSubStore
```
(with virtual inheritance to prevent DDD).
The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
2020-09-16 13:53:08 +02:00
regnat
aa4eac3788
fixup! Separate the instantiation and initialisation of the stores
2020-09-16 13:53:08 +02:00
regnat
3c525d1590
Complete the toJSON
instance for Setting<T>
...
Don't let it just contain the value, but also the other fields of the
setting (description, aliases, etc..)
2020-09-16 13:53:08 +02:00
regnat
3b57181f8e
Separate the instantiation and initialisation of the stores
...
Add a new `init()` method to the `Store` class that is supposed to
handle all the effectful initialisation needed to set-up the store.
The constructor should remain side-effect free and just initialize the
c++ data structure.
The goal behind that is that we can create “dummy” instances of each
store to query static properties about it (the parameters it accepts for
example)
2020-09-16 13:53:08 +02:00
regnat
fa32560169
Fix the registration of stores
2020-09-16 13:53:08 +02:00
regnat
7d5bdf8b56
Make the store plugins more introspectable
...
Directly register the store classes rather than a function to build an
instance of them.
This gives the possibility to introspect static members of the class or
choose different ways of instantiating them.
2020-09-16 13:53:08 +02:00
John Ericson
3a5cdd737c
Rename Derivation::pathOpt
to Derivation::path
...
We no longer need the `*Opt` to disambiguate.
2020-09-15 15:21:39 +00:00
John Ericson
6387550d58
Get rid of confusing std::optional<bool>
for validity
2020-09-15 15:19:45 +00:00
John Ericson
c4bf219b55
Don't link deriver until after any delayed exception is thrown
...
Otherwise, we will associate fixed-output derivations with outputs that
they did indeed produce, but which had the wrong hash. That's no good.
2020-09-15 14:28:06 +00:00
John Ericson
3ba552b245
Merge remote-tracking branch 'upstream/master' into single-ca-drv-build
2020-09-15 14:17:06 +00:00
John Ericson
c08c9f08c7
Merge remote-tracking branch 'upstream/master' into remove-storetype-delegate-regStore
2020-09-15 14:08:35 +00:00
regnat
057c6203b5
gracefully handle old daemon versions
...
Add a fallback path in `queryPartialDerivationOutputMap` for daemons
that don't support it.
Also upstreams a couple methods from `SSHStore` to `RemoteStore` as this
is needed to handle the fallback path.
2020-09-15 09:58:01 +02:00
Eelco Dolstra
a59e77d9e5
nix-daemon: Lower verbosity of restricted setting warning
...
Fixes #3992 .
2020-09-14 13:48:51 +02:00
aszlig
525b38eee8
Fix unspecified behaviour in readStorePathCAMap
...
When deploying a Hydra instance with current Nix master, most builds
would not run because of errors like this:
queue monitor: error: --- Error --- hydra-queue-runner
error: --- UsageError --- nix-daemon
not a content address because it is not in the form '<prefix>:<rest>': /nix/store/...-somedrv
The last error message is from parseContentAddress, which expects a
colon-separated string, however what we got here is a store path.
Looking at the worker protocol, the following message sent to the Nix
daemon caused the error above:
0x1E -> wopQuerySubstitutablePathInfos
0x01 -> Number of paths
0x16 -> Length of string
"/nix/store/...-somedrv"
0x00 -> Length of string
""
Looking at writeStorePathCAMap, the store path is indeed the first field
that's transmitted. However, readStorePathCAMap expects it to be the
*second* field *on my machine*, since expression evaluation order is a
classic form of unspecified behaviour[1] in C++.
This has been introduced in https://github.com/NixOS/nix/pull/3689 ,
specifically in commit 66a62b3189
.
[1]: https://en.wikipedia.org/wiki/Unspecified_behavior#Order_of_evaluation_of_subexpressions
Signed-off-by: aszlig <aszlig@nix.build>
2020-09-13 02:40:51 +02:00
John Ericson
98dfd7531d
Fix querying outputs for CA derivations some more
...
If we resolve using the known path of a derivation whose output we
didn't have, we previously blew up. Now we just fail gracefully,
returning the map of all outputs unknown.
2020-09-04 18:33:58 +00:00
John Ericson
075d399e3f
Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream
2020-09-04 16:04:35 +00:00
John Ericson
e9fad3006b
Fix some of the issues raised by @edolstra
...
- More and better comments
- The easier renames
2020-09-04 15:18:25 +00:00
John Ericson
e86dd59dcc
Apply suggestions from code review
...
Thanks!
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2020-09-04 10:48:50 -04:00
John Ericson
c9f1ed912c
Don't chmod symlink before moving outputs around
...
Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
2020-09-04 14:41:53 +00:00
John Ericson
aad4abcc9c
Fix floating CA tests
...
We will sometimes try to query the outputs of derivations we can't
resolve. That's fine; it just means we don't know what those outputs are
yet.
2020-09-04 01:17:38 +00:00
John Ericson
975a47f7fe
Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream
2020-09-03 22:09:04 +00:00
John Ericson
e7d93e7ece
Merge remote-tracking branch 'upstream/master' into single-ca-drv-build
2020-09-03 15:43:17 +00:00
Eelco Dolstra
8a945d6ddb
Remove gc-check-reachability
2020-09-03 11:30:15 +02:00
Eelco Dolstra
82b77a7726
addPermRoot(): Remove indirect flag
2020-09-03 11:29:34 +02:00
Eelco Dolstra
b07167be5a
createGeneration(): Always create an indirect root
...
This means profiles outside of /nix/var/nix/profiles don't get
garbage-collected. It also means we don't need to scan
/nix/var/nix/profiles for GC roots anymore, except for compatibility
with previously existing generations.
2020-09-03 11:13:53 +02:00
Eelco Dolstra
b74f5cdd23
createGeneration(): Take a StorePath
2020-09-03 11:06:56 +02:00
Domen Kožar
dd4b56c87f
Allow HTTP binary cache to request absolute uris
2020-09-01 21:35:48 +02:00
John Ericson
ef278d00f9
Merge remote-tracking branch 'upstream/master' into single-ca-drv-build
2020-09-01 18:01:48 +00:00
Eelco Dolstra
84f5cabbea
Merge remote-tracking branch 'origin/master' into markdown
2020-08-31 14:24:26 +02:00
John Ericson
02e0001fc0
Merge branch 'ca-floating-upstream' of github.com:obsidiansystems/nix into ca-floating-upstream
2020-08-28 21:23:09 +00:00
John Ericson
421ed527c7
Update src/libstore/build.cc
...
Thanks for catching, @regnat.
2020-08-28 17:22:57 -04:00
John Ericson
48e8828050
Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream
2020-08-28 20:38:25 +00:00
John Ericson
8017fe7487
Merge remote-tracking branch 'upstream/master' into single-ca-drv-build
2020-08-28 19:59:14 +00:00
Eelco Dolstra
eb75282b8d
Merge pull request #3434 from Ericson2314/derivation-header-include-order
...
Revise division of labor in deserialization of derivations
2020-08-27 16:39:28 +02:00
Eelco Dolstra
a0f19d9f3a
RemoteStore::addToStore(): Fix race between stderrThread and NAR writer
...
As pointed out by @B4dM4n, the call to to.flush() on stderrThread is
unsafe because the NAR writer thread is also writing to 'to'.
Fixes #3943 .
2020-08-27 14:50:51 +02:00
Eelco Dolstra
4bf5faf416
Merge remote-tracking branch 'origin/master' into markdown
2020-08-25 19:47:34 +02:00
John Ericson
8eb73a8724
CA derivations that depend on other CA derivations
...
Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
2020-08-24 19:01:15 +00:00
John Ericson
e0b0e18905
Add constructor for BasicDerivation -> Derivation
2020-08-24 19:01:07 +00:00
John Ericson
59979e7053
Fix bad debug format string
2020-08-24 19:01:07 +00:00
Eelco Dolstra
6a67e57019
Add DummyStore (dummy://)
...
DummyStore does not allow building or adding paths. This is useful for
evaluation tests when you don't want to initialize a "proper" store.
2020-08-24 18:54:16 +02:00
Eelco Dolstra
2530cc6827
Merge pull request #3950 from obsidiansystems/tabs-spaces
...
tabs -> spaces
2020-08-24 10:20:28 +02:00
John Ericson
b0b59fd05a
Merge remote-tracking branch 'obsidian/write-derivation-borrow' into HEAD
2020-08-23 15:11:10 +00:00
John Ericson
35e6288be1
writeDerivation
just needs a plain store reference
2020-08-23 15:01:11 +00:00
John Ericson
3a7b330b64
"Downstream placeholders" should not be store paths
...
Insead they should be opaque `/<hash>` like the placeholders we already
have.
2020-08-21 19:35:35 +00:00
John Ericson
422affe102
tabs -> spaces
...
Sorry I let the tab sneak in there in the first place.
2020-08-21 19:19:24 +00:00
John Ericson
c265e0ea82
Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol
2020-08-20 20:24:22 +00:00
John Ericson
27a3f82c0b
Merge remote-tracking branch 'upstream/master' into single-ca-drv-build
2020-08-20 18:28:17 +00:00
John Ericson
45a2f1baab
Rename drv output querying functions, like master
...
- `queryDerivationOutputMapAssumeTotal` -> `queryPartialDerivationOutputMap`
- `queryDerivationOutputMapAssumeTotal` -> `queryDerivationOutputMap
2020-08-20 18:14:12 +00:00
Eelco Dolstra
4d77513d97
Merge pull request #3859 from obsidiansystems/drv-outputs-map-allow-missing
...
`queryDerivationOutputMap` no longer assumes all outputs have a mapping
2020-08-20 16:49:23 +02:00
John Ericson
9a9d834dc7
Rename drv output querying functions
...
- `queryDerivationOutputMapAssumeTotal` -> `queryPartialDerivationOutputMap`
- `queryDerivationOutputMapAssumeTotal` -> `queryDerivationOutputMap`
2020-08-20 14:12:51 +00:00
Eelco Dolstra
acb99f03f9
Config: Use nlohmann/json
2020-08-20 11:02:16 +02:00
John Ericson
a83694c7a1
Use RemoteStore
to open connection for proxying daemon
...
Removes duplicate websocket opening code, and also means we should be
able to to ssh-ssh-... daemon relays, not just uds-uds-... ones.
2020-08-19 19:34:47 +00:00
Eelco Dolstra
c8fa39324a
Generate the nix.conf docs from the source code
...
This means we don't have two (divergent) sets of option descriptions
anymore.
2020-08-19 18:28:04 +02:00
John Ericson
d5af5763cf
Merge branch 'master' of github.com:NixOS/nix into remove-storetype-delegate-regStore
2020-08-19 15:54:50 +00:00
John Ericson
be0d429b95
Merge branch 'master' of github.com:NixOS/nix into templated-daemon-protocol
2020-08-19 03:17:41 +00:00
John Ericson
950ddfdb82
Merge remote-tracking branch 'upstream/master' into derivation-header-include-order
2020-08-18 14:36:44 +00:00
Eelco Dolstra
dfeb76dbf9
Merge pull request #3930 from obsidiansystems/legacy-ssh-build-paths
...
Define `LegacySSHStore::buildPaths` using `cmdBuildPaths`
2020-08-18 16:07:40 +02:00
Eelco Dolstra
7cdc739ece
Merge remote-tracking branch 'origin/master' into markdown
2020-08-17 13:43:39 +02:00
Eelco Dolstra
847a5392f4
Merge branch 'ca-no-need-trust' of https://github.com/obsidiansystems/nix
2020-08-17 10:57:02 +02:00
Eelco Dolstra
3c619f6290
Merge branch 'test-RemoteStore-buildDerivation' of https://github.com/obsidiansystems/nix
2020-08-17 10:53:18 +02:00
John Ericson
6f7ac5e865
Remove extra closing paren
2020-08-14 21:59:31 +00:00
John Ericson
3c8b5b6219
Merge remote-tracking branch 'upstream/master' into single-ca-drv-build
2020-08-14 17:00:13 +00:00
John Ericson
4b571ea321
Update src/libstore/daemon.cc
...
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2020-08-14 11:52:37 -04:00
Eelco Dolstra
13e49be660
Merge pull request #3875 from obsidiansystems/new-interface-for-path-pathOpt
...
Offer a safer interface for path and pathOpt
2020-08-14 17:19:19 +02:00
Eelco Dolstra
7714d9a943
Merge pull request #3924 from obsidiansystems/features-per-store
...
Make `system-features` a store setting
2020-08-14 17:13:07 +02:00
Eelco Dolstra
9b9d5297ba
Merge pull request #3909 from matthewbauer/readd-hashed-mirrors
...
Add hashed-mirrors back
2020-08-14 17:05:47 +02:00
Eelco Dolstra
d81f13f7cb
Merge pull request #3899 from obsidiansystems/make-narHash-not-optional
...
Make narHash in ValidPathInfo not optional
2020-08-14 17:00:18 +02:00
John Ericson
e1308b1211
Define LegacySSHStore::buildPaths
using cmdBuildPaths
...
Evidentally this was never implemented because Nix switched to using
`buildDerivation` exclusively before `build-remote.pl` was rewritten.
The `nix-copy-ssh` test (already) tests this.
2020-08-13 21:27:55 +00:00
John Ericson
5ccd94501d
Allow trustless building of CA derivations
...
Include a long comment explaining the policy. Perhaps this can be moved
to the manual at some point in the future.
Also bump the daemon protocol minor version, so clients can tell whether
`wopBuildDerivation` supports trustless CA derivation building. I hope
to take advantage of this in a follow-up PR to support trustless remote
building with the minimal sending of derivation closures.
2020-08-13 18:15:57 +00:00
John Ericson
85aacbee64
Use TeeSink
and TeeSouce
in a few more places
2020-08-13 14:51:17 +00:00
John Ericson
d2f2be0f70
Test RemoteStore::buildDerivation
...
Fix `wopNarFromPath` which needed a `toRealPath`.
2020-08-13 04:07:14 +00:00
John Ericson
5d67f18c86
Merge branch 'daemon-auth-cleanup' of github.com:obsidiansystems/nix into HEAD
2020-08-12 18:22:31 +00:00
John Ericson
4720853129
Make system-features
a store setting
...
This seems more correct. It also means one can specify the features a
store should support with --store and remote-store=..., which is useful.
I use this to clean up the build remotes test.
2020-08-12 18:13:00 +00:00
John Ericson
8d4162ff9e
Separate auth and logic for the daemon
...
Before, processConnection wanted to know a user name and user id, and
`nix-daemon --stdio`, when it isn't proxying to an underlying daemon,
would just assume "root" and 0. But `nix-daemon --stdio` (no proxying)
shouldn't make guesses about who holds the other end of its standard
streams.
Now processConnection takes an "auth hook", so `nix-daemon` can provide
the appropriate policy and daemon.cc doesn't need to know or care what
it is.
2020-08-12 15:22:33 +00:00
John Ericson
5f80aea795
Break out lambda so output can be matched just once
...
This is much better.
2020-08-12 02:23:31 +00:00
John Ericson
18834f7764
Recheck path validity after acquiring lock
...
It might have changed, and in any event this is how the cod used to work
so let's just keep it.
2020-08-11 23:44:02 +00:00
John Ericson
2de201254e
Don't assume a total output map in two places in build.cc
...
Thanks @regnat for catching one of them. The other follows for many of
the same reasons. I'm find fixing others on a need-to-fix basis,
provided their are no regressions.
2020-08-11 23:07:50 +00:00
John Ericson
4c6aac8fdf
Clarify comment on sandbox and temp fresh paths
2020-08-11 22:46:05 +00:00
John Ericson
6d57139050
Clarify outputReferences
variable with self-describing type
...
Thanks for the idea, @Regnat!
2020-08-11 22:34:09 +00:00
John Ericson
8a068bd025
Remove redundant equality check
2020-08-11 21:25:40 +00:00
John Ericson
07e3466eb4
Float comment to out describe gaveUpOnSubstitution
in general
2020-08-11 21:16:14 +00:00
John Ericson
d0f6e338dd
Apply suggestions from code review
...
Thanks!!
Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
2020-08-11 16:49:10 -04:00
John Ericson
2a0902634e
Fix error in merge breaking floating CA drvs
...
Forgot to add this hunk!
2020-08-11 00:13:19 +00:00
John Ericson
1b5c24662b
Merge branch 'small-drv-serialize-cleanup' of github.com:obsidiansystems/nix into single-ca-drv-build
2020-08-10 01:57:54 +00:00
John Ericson
581183d4d5
Deduplicate parsing and reading derivations
2020-08-10 01:40:50 +00:00
John Ericson
bcd0629c2e
Remove name parameter from writeDerivation
...
The name is now stored with the derivation itself.
2020-08-10 01:35:59 +00:00
John Ericson
e913a2989f
Squashed get CA derivations building
2020-08-07 19:51:55 +00:00
Eelco Dolstra
edfd676e05
Fix .ls file names in binary caches
...
These are not supposed to include the 'name' part of the store
path. This was broken by 759947bf72
.
2020-08-07 21:18:29 +02:00
John Ericson
8f92bb5ad9
Merge branch 'drv-outputs-map-allow-missing' of github.com:obsidiansystems/nix into templated-daemon-protocol
2020-08-07 18:51:01 +00:00
John Ericson
f7ba16f9cb
Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing
2020-08-07 17:46:39 +00:00
John Ericson
47644e49ca
Specialize std::optional<StorePath>
so this is backwards compatible
...
While I am cautious to break parametricity, I think it's OK in this
cases---we're not about to try to do some crazy polymorphic protocol
anytime soon.
2020-08-07 17:05:14 +00:00
Matthew Bauer
96c158d6e1
Fix build
2020-08-06 21:04:31 -05:00
Matthew Bauer
8abc577cc2
Merge remote-tracking branch 'origin/master' into readd-hashed-mirrors
2020-08-06 18:58:59 -05:00
Carlo Nucera
46f9dd56da
Fix bug due to non-deterministic arg eval order
2020-08-06 19:30:05 -04:00
Matthew Bauer
641c950701
Add hashed-mirrors back
...
Some users have their own hashed-mirrors setup, that is used to mirror
things in addition to what’s available on tarballs.nixos.org. Although
this should be feasable to do with a Binary Cache, it’s not always
easy, since you have to remember what "name" each of the tarballs has.
Continuing to support hashed-mirrors is cheap, so it’s best to leave
support in Nix. Note that NIX_HASHED_MIRRORS is also supported in
Nixpkgs through fetchurl.nix.
Note that this excludes tarballs.nixos.org from the default, as in
\#3689. All of these are available on cache.nixos.org.
2020-08-06 18:19:09 -05:00
Carlo Nucera
9ab07e99f5
Use template structs instead of phantoms
2020-08-06 18:04:13 -04:00
Carlo Nucera
3d8240c32e
Remove leftover commented code
2020-08-06 16:04:18 -04:00
Carlo Nucera
f795f0fabc
Merge branch 'drv-outputs-map-allow-missing-namespace' of github.com:obsidiansystems/nix into templated-daemon-protocol
2020-08-06 15:53:09 -04:00
John Ericson
e89b5bd0bf
Minimize the usage of Hash::dummy
2020-08-06 18:31:48 +00:00
John Ericson
5e59b25a23
Merge remote-tracking branch 'upstream/master' into make-narHash-not-optional
2020-08-06 15:34:15 +00:00
Carlo Nucera
8b175f58d1
Simplify the namespace
2020-08-05 17:57:07 -04:00
Carlo Nucera
0739d428e0
Solve template deduction problem
...
We had to predeclare our template functions
2020-08-05 17:49:45 -04:00
John Ericson
6c66331d5c
WIP: Put the worker protocol read
and write
in a namespace to disambig
2020-08-05 20:37:48 +00:00
Carlo Nucera
1d2e80ddd6
Merge branch 'master' of github.com:NixOS/nix into new-interface-for-path-pathOpt
2020-08-05 15:45:33 -04:00
John Ericson
ed96e603e1
Proxy -> Phantom to match Rust
...
Sorry, Haskell.
2020-08-05 19:44:08 +00:00
Carlo Nucera
8241e660ba
Remove Hash::operator bool ()
...
Since the hash is not optional anymore
2020-08-05 15:30:38 -04:00
Carlo Nucera
be6e1c6457
Merge branch 'master' of github.com:NixOS/nix into make-narHash-not-optional
2020-08-05 15:14:47 -04:00
Carlo Nucera
1ad6394b33
Add Hash::dummy to signal default value
...
We did this in the same spirit of the dummy value that's present in
libstore/path.hh
2020-08-05 15:11:49 -04:00
Carlo Nucera
1d71028f4d
Remove optionality in ValidPathInfo::narInfo
2020-08-05 14:42:48 -04:00
John Ericson
7302761f64
Merge remote-tracking branch 'obsidian/drv-outputs-map-allow-missing' into templated-daemon-protocol
2020-08-05 17:53:24 +00:00
Ben Burdette
f1a47a96b6
error messages for issue 2238
2020-08-05 10:58:00 -06:00
John Ericson
cf939055c8
Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing
2020-08-05 16:43:30 +00:00
John Ericson
0559ff3d8b
Merge branch 'master' of github.com:NixOS/nix into derivation-primop-floating-output
2020-08-05 14:56:40 +00:00
John Ericson
92ad550e96
Merge remote-tracking branch 'obsidian/misc-ca' into derivation-primop-floating-output
2020-08-05 14:51:41 +00:00
Eelco Dolstra
790b694be7
Style fix
2020-08-05 16:51:06 +02:00
Eelco Dolstra
e48f944e9d
Merge branch 'misc-ca' of https://github.com/obsidiansystems/nix
2020-08-05 16:50:05 +02:00
John Ericson
b9ebe373bb
Sed some names to perhaps avoid conflicts
2020-08-05 14:49:25 +00:00
Eelco Dolstra
25f7912156
Style fix
2020-08-05 16:47:48 +02:00
John Ericson
e561a13a58
Reanme DerivationType::Regular
defintion too
...
This is the one non-prefixed occurence
2020-08-05 14:45:56 +00:00
John Ericson
e7b0847f2d
Make names more consistent
2020-08-05 14:44:39 +00:00
John Ericson
839f0fe095
Merge remote-tracking branch 'upstream/master' into misc-ca
2020-08-05 14:40:01 +00:00
John Ericson
03f4fafc27
Merge remote-tracking branch 'upstream/master' into misc-ca
2020-08-05 14:36:25 +00:00
Eelco Dolstra
b91dc7ebad
Merge pull request #3730 from obsidiansystems/better-ca-parse-errors
...
Improve hash parsing and errors
2020-08-05 16:33:07 +02:00
Eelco Dolstra
75f220a595
Merge pull request #3864 from obsidiansystems/more-topo-sort
...
Abstract out topo sorting logic
2020-08-05 16:07:29 +02:00
John Ericson
16c98bf57c
Get rid of some unneeded temporaries
2020-08-04 22:36:31 +00:00
John Ericson
1dfcbebc95
Organize and format code a bit
2020-08-04 22:28:10 +00:00
John Ericson
45b6fdb22b
Remove unused functions
2020-08-04 22:10:13 +00:00
John Ericson
1bab8a321f
Remove unneeded definition
...
Template instantiations will cover this case fine.
2020-08-04 21:56:42 +00:00
John Ericson
2f2ae993dc
WIP systematize more of the worker protocol
...
This refactor should *not* change the wire protocol.
2020-08-04 19:02:05 +00:00
John Ericson
fbeb8695fb
Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing
2020-08-04 18:35:33 +00:00
John Ericson
d3452a5ed6
Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors
2020-08-04 16:13:20 +00:00
Eelco Dolstra
4e7f1c7f11
S3BinaryCacheStore: Fix size determination
2020-08-04 16:01:13 +02:00
Eelco Dolstra
dfe66420e7
Revert "Remove putBytes"
...
This reverts commit b8eea7e81a
.
2020-08-04 15:56:10 +02:00
Eelco Dolstra
327b1bf378
BinaryCacheStore: Explicitly flush file sink
...
The file sink is also flushed in its destructor, but we ignore any
exceptions in the destructor.
Issue #3886 .
2020-08-04 14:54:07 +02:00
Eelco Dolstra
ed52cf632b
Merge pull request #3892 from obsidiansystems/path-info-header
...
Move ValidPathInfo to its own header
2020-08-04 10:44:47 +02:00
John Ericson
062533f7cd
Merge remote-tracking branch 'upstream/master' into path-info-header
2020-08-03 21:02:28 +00:00
Eelco Dolstra
24e07c428f
Delete compressed NARs
...
Fixes #3891 .
2020-08-03 18:34:52 +02:00
John Ericson
54281f3ac1
addToStore
in terms of addToStoreFromDump
is not local-store-specific
2020-08-03 04:13:45 +00:00
John Ericson
9357512d73
Merge remote-tracking branch 'upstream/master' into derivation-header-include-order
2020-08-01 19:38:35 +00:00
Carlo Nucera
b6d97fdbf4
Merge branch 'master' of github.com:NixOS/nix into drv-outputs-map-allow-missing
2020-07-31 13:12:51 -04:00
Eelco Dolstra
bf290c2306
Merge remote-tracking branch 'origin/master' into markdown
2020-07-31 16:07:04 +02:00
Eelco Dolstra
1d0a7b54fa
Enable syntax highlighting
2020-07-31 15:43:25 +02:00
Carlo Nucera
eee6ef86cd
Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors
2020-07-30 18:27:25 -04:00
Matthew Bauer
05ac4db39a
Merge remote-tracking branch 'origin/master' into substitute-other-storedir
2020-07-30 12:38:24 -05:00
Eelco Dolstra
3f6e88a552
unsigned long long -> uint64_t
2020-07-30 13:34:04 +02:00
Eelco Dolstra
ebee2b7852
receiveContents(): unsigned int -> size_t
2020-07-30 13:00:30 +02:00
Eelco Dolstra
f63839bfa4
Cleanup
2020-07-29 18:04:51 +02:00
Eelco Dolstra
4c0077a07d
Fix RemoteStore::addToStore() latency
...
Since 6185d25e52
, this was very
latency-bound since it required a round-trip for every 32 KiB. So for
example copying a 514 MiB closure over a virtual ethernet device with
a articial delay of just 1 ms took 343s. Now it takes 2.7s.
Fixes #3372 .
2020-07-29 00:48:39 +02:00
Eelco Dolstra
c159f48a39
Cleanup
2020-07-29 00:24:55 +02:00
John Ericson
df707d05d1
Merge branch 'ca-drv' of github.com:Ericson2314/nix into misc-ca
2020-07-28 21:12:36 +00:00
Carlo Nucera
2980b244b7
Use assert(false) instead of abort()
2020-07-28 15:39:45 -04:00
Carlo Nucera
cfe96bd7ce
Merge branch 'master' of github.com:NixOS/nix into ca-drv
2020-07-28 15:36:45 -04:00
Carlo Nucera
c318d398f3
Merge branch 'misc-ca' of github.com:obsidiansystems/nix into new-interface-for-path-pathOpt
2020-07-28 14:22:24 -04:00
Carlo Nucera
7ef1e3cd14
Use the new interface
2020-07-28 13:59:24 -04:00
Carlo Nucera
d564ac1c50
Offer a safer interface for pathOpt
...
The new interface we offer provides a way of getting all the
DerivationOutputs with the storePaths directly, based on the observation
that it's the most common usecase.
2020-07-28 13:55:15 -04:00
Eelco Dolstra
b8eea7e81a
Remove putBytes
...
istream->tellg() returns -1 so we can't get the number of bytes
written.
Fixes 'uploaded 's3://nix-cache/nar/00819r9lp5kajr6baxfw5dhhc0cx8ndxaz43qmd2f0gn1hk1ynlp.nar.xz' (-1 bytes) in 11620 ms' messages.
2020-07-28 13:27:56 +02:00
John Ericson
8065c6d160
Abstract out topo sorting logic
2020-07-27 20:45:34 +00:00
John Ericson
951415b568
Require ca-derivations
everywhere we create a CA derivation
...
"create" as in read one in from a serialized form, or build one from
scratch in memory.
2020-07-27 17:56:36 +00:00
John Ericson
e32a9e124b
Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output
2020-07-27 17:50:06 +00:00
John Ericson
7cf978440c
Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca
2020-07-27 17:47:40 +00:00
John Ericson
e6115ca004
Merge remote-tracking branch 'upstream/master' into ca-derivation-data-types
2020-07-27 17:41:43 +00:00
John Ericson
1d7d94ceea
Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors
2020-07-27 16:17:50 +00:00
John Ericson
43f2bd8dc5
Merge remote-tracking branch 'upstream/master' into hash-always-has-type
2020-07-27 16:13:57 +00:00
John Ericson
387699fe7f
Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca
2020-07-27 15:00:28 +00:00
John Ericson
78466bcb2f
Merge branch 'optional-derivation-output-storepath' into ca-derivation-data-types
2020-07-27 14:40:08 +00:00
John Ericson
d5bb67cfa4
Merge remote-tracking branch 'upstream/master' into optional-derivation-output-storepath
2020-07-27 14:29:32 +00:00
John Ericson
699fc89b39
Merge remote-tracking branch 'upstream/master' into hash-always-has-type
2020-07-27 14:22:45 +00:00
John Ericson
2c7557481b
queryDerivationOutputMap
no longer assumes all outputs have a mapping
...
This assumption is broken by CA derivations. Making a PR now to do the
breaking daemon change as soon as possible (if it is already too late,
we can bump protocol intead).
2020-07-24 21:14:06 +00:00
John Ericson
58e55c0923
Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output
2020-07-23 19:17:52 +00:00
John Ericson
a2af25c55a
Merge remote-tracking branch 'upstream/master' into misc-ca
2020-07-23 19:14:17 +00:00
John Ericson
c36916eca0
Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output
2020-07-23 15:06:56 +00:00
John Ericson
6fcd9966bf
Merge remote-tracking branch 'upstream/master' into misc-ca
2020-07-23 15:02:29 +00:00
John Ericson
66a2067288
Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors
2020-07-23 14:56:35 +00:00
John Ericson
2274f63453
Get rid of basicDerivation::findOutput
...
It's a tiny function which is:
- hardly worth abstrating over, and also only used once.
- doesn't work once we get CA drvs
I rewrote the one callsite to be forwards compatable with CA
derivations, and also potentially more performant: instead of reading in
the derivation it can ust consult the SQLite DB in the common case.
2020-07-23 14:39:02 +00:00
John Ericson
9423f64ee2
Parse CA derivations using new output variants
...
We no longer need `ParsedDerivation` because everything libstore needs
to know about is in the `BasicDerivation` proper.
2020-07-22 23:59:25 +00:00
John Ericson
c56356bacc
Separate concerns in scanForReferences
with TeeSink
...
This also will make it easier to use a `HashModuloSink` instead for CA
derivations.
2020-07-22 22:13:44 +00:00
Eelco Dolstra
ff314f186e
Merge pull request #3736 from obsidiansystems/allow-relative-paths-in-store-option
...
Allow relative paths in --store option
2020-07-21 19:43:07 +02:00
Eelco Dolstra
0951330680
Merge pull request #3714 from obsidiansystems/add-body-to-network-errors
...
Add response body to network errors
2020-07-21 18:09:42 +02:00
John Ericson
02639716ea
Merge branch 'allow-relative-paths-in-store-option' into remove-storetype-delegate-regStore
2020-07-21 15:55:27 +00:00
John Ericson
6cce32c8e8
Change logic for deciding what is a relative path for the local store
...
The was Eelco's prefered logic, and it looks good to me!
2020-07-21 15:39:47 +00:00
Carlo Nucera
922a845ffc
Update chunkSize to the suggested value
...
This was a suggested course of action in a review in one of our earlier
commits, https://github.com/NixOS/nix/pull/3801#discussion_r457557079
2020-07-21 10:24:19 -04:00
John Ericson
54e507a7aa
Merge remote-tracking branch 'upstream/master' into add-body-to-network-errors
2020-07-21 14:17:59 +00:00
John Ericson
5ce95b9529
Update src/libstore/build.cc
2020-07-21 09:47:40 -04:00
John Ericson
cdb3f39b64
Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors
2020-07-21 13:43:20 +00:00
John Ericson
5055c595bd
Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into ca-derivation-data-types
2020-07-21 01:20:53 +00:00
John Ericson
5a37413796
Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into from-dump-stream
2020-07-21 01:20:14 +00:00
John Ericson
bf0dca7790
Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into misc-ca
2020-07-21 01:18:33 +00:00
Carlo Nucera
9aae179f34
Correct bug, thoroughly document addToStoreSlow
2020-07-20 20:18:12 -04:00
Carlo Nucera
0ca9744694
Use heuristics to decide when to show the response
...
Due to https://github.com/NixOS/nix/issues/3841 we don't know how print
different messages for different verbosity levels.
2020-07-20 15:57:58 -04:00
Carlo Nucera
6357b1b0fb
Add another Unimplemented case
2020-07-20 14:17:25 -04:00
Carlo Nucera
007bf6d17a
Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca
2020-07-20 14:14:39 -04:00
Carlo Nucera
362ae93851
Add UnimplementedError to ease grepping for these
2020-07-20 14:13:37 -04:00
John Ericson
bf61871271
parser.hh -> split.hh
2020-07-20 17:42:34 +00:00
John Ericson
eab945f591
Merge remote-tracking branch 'upstream/master' into from-dump-stream
2020-07-20 17:29:23 +00:00
Eelco Dolstra
1c5f8bbfb5
Merge pull request #3822 from obsidiansystems/dump-thrice-fixme
...
Optimize `addToStoreSlow` and remove `TeeParseSink`
2020-07-20 18:55:05 +02:00
John Ericson
ac2fc7ba1f
Apply suggestions from code review
...
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2020-07-20 11:29:46 -04:00
Carlo Nucera
0aa79dcc6f
Remove StoreType abstraction and delegate regStore
...
to each Store implementation. The generic regStore implementation will
only be for the ambiguous shorthands, like "" and "auto".
This also could get us close to simplifying the daemon command.
2020-07-17 17:24:02 -04:00
John Ericson
6756cecfcf
Add DerivationType::CAFloating
2020-07-17 19:55:41 +00:00
Carlo Nucera
fefd6c9e5f
Merge branch 'master' of github.com:NixOS/nix into allow-relative-paths-in-store-option
2020-07-17 14:34:20 -04:00
Carlo Nucera
bbc633c98c
Revert "Don't anticipate CA but not fixed outputs for now"
...
This reverts commit 3a9e4c3262
.
2020-07-17 13:10:32 -04:00
Carlo Nucera
205dcd140d
Revert "Don't anticipate multiple CA outputs for now"
...
This reverts commit 74b251b2f3
.
2020-07-17 12:43:46 -04:00
Carlo Nucera
1feb8981df
Revert "Don't anticipate hash algo without hash in derivation for now"
...
This reverts commit 3804e3df9b
.
2020-07-17 12:05:54 -04:00
Carlo Nucera
487c5751c6
Merge branch 'ca-drv' of github.com:Ericson2314/nix into misc-ca
2020-07-17 11:32:59 -04:00
Carlo Nucera
5cb840541b
Merge branch 'multi-output-hashDerivationModulo' of github.com:Ericson2314/nix into misc-ca
2020-07-17 10:28:33 -04:00
Carlo Nucera
745a03cef5
Merge branch 'optional-derivation-output-storepath' of github.com:obsidiansystems/nix into ca-derivation-data-types
2020-07-16 13:36:01 -04:00
Carlo Nucera
048e916f64
Merge branch 'master' of github.com:NixOS/nix into optional-derivation-output-storepath
2020-07-16 13:32:28 -04:00
John Ericson
cc0d77f8c9
Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors
2020-07-16 17:28:52 +00:00
Carlo Nucera
e3a3406db8
Merge branch 'master' of github.com:NixOS/nix into add-body-to-network-errors
2020-07-16 12:04:10 -04:00
John Ericson
5ea817dace
Merge remote-tracking branch 'upstream/master' into hash-always-has-type
2020-07-16 14:58:53 +00:00
John Ericson
3dcca18c30
Fix bug in TeeSource
...
We use this to simplify `LocalStore::addToStoreFromDump`.
Also, hope I fixed build error with old clang (used in Darwin CI).
2020-07-16 13:39:27 +00:00
Carlo Nucera
e3b394b6e8
Small namespace fix
2020-07-16 09:36:02 -04:00
Eelco Dolstra
5517eee17e
Generations API cleanup
2020-07-16 15:18:21 +02:00
Eelco Dolstra
3f264916db
Merge remote-tracking branch 'origin/flakes'
2020-07-16 14:07:32 +02:00
John Ericson
68dfb8c6ae
Optimize addToStoreSlow
and remove TeeParseSink
2020-07-16 05:09:41 +00:00
John Ericson
5602637d9e
Revert "LocalStore::addToStoreFromDump copy in chunks"
...
This reverts commit 592851fb67
. We don't
need this extra feature anymore
2020-07-15 23:37:49 +00:00
John Ericson
bc109648c4
Get rid of LocalStore::addToStoreCommon
...
I got it to just become `LocalStore::addToStoreFromDump`, cleanly taking
a store and then doing nothing too fancy with it.
`LocalStore::addToStore(...Path...)` is now just a simple wrapper with a
bare-bones sinkToSource of the right dump command.
2020-07-15 23:14:30 +00:00
Carlo Nucera
455bdee205
Merge branch 'master' of github.com:NixOS/nix into derivation-header-include-order
2020-07-15 17:58:30 -04:00
Carlo Nucera
d090562348
Merge branch 'master' of github.com:NixOS/nix into hash-always-has-type
2020-07-15 17:21:01 -04:00
John Ericson
aff69e229c
Merge remote-tracking branch 'obsidian/fix-add-to-star-narhash' into from-dump-stream
2020-07-15 21:05:29 +00:00
Carlo Nucera
2d2a10e79a
Merge branch 'master' of github.com:NixOS/nix into add-body-to-network-errors
2020-07-15 13:06:08 -04:00
Eelco Dolstra
cff2157185
Revert "LocalStore::addToStore(srcPath): Handle the flat case"
...
This reverts commit a2c27022e9
. See
addToStoreSlow(), we don't need to handle this case efficiently
anymore. In fact, we can almost remove the method/hashAlgo arguments
since the non-recursive and/or non-SHA256 are almost not used anymore.
2020-07-15 12:49:03 +02:00
John Ericson
650c2c6558
Rename variable nar
-> dump
according to TODO
2020-07-14 21:28:50 +00:00
John Ericson
8173e7bfef
Fix localhost::addToStore(...Path...)
...
We were calculating the nar hash wrong when the file ingestion method
was flat. I don't think there's anything we can do in that case but dump
the file again, so that's what I do.
As an optomization, we again could reuse the original dump for just the
recursive and non-sha256 case, but I rather do that after this fix, and
after my other PRs which deduplicate this code.
2020-07-14 21:12:11 +00:00
John Ericson
592851fb67
LocalStore::addToStoreFromDump copy in chunks
...
Rather than copying byte-by-byte, we let the coroutine know how much
data we would like it to send back to us.
2020-07-14 13:56:36 +00:00
John Ericson
9de96ef7d4
Dedup LocalStore::addToStore*
...
The downsides is that the coroutine has byte-by-byte loop transfer. Will
fix that next.
2020-07-14 13:56:36 +00:00
John Ericson
c86fc3a965
Crudely make addToStoreFromDump
take Source
not string
...
I just as little beyond the type as possible, so the implementation
changes this enables can be reviewed separately.
2020-07-14 13:56:36 +00:00
John Ericson
9ec10046e0
Narrow scope of temporary value
2020-07-14 13:54:29 +00:00
Eelco Dolstra
832e111494
Merge remote-tracking branch 'origin/master' into flakes
2020-07-14 13:56:18 +02:00
Eelco Dolstra
926c3a6664
Doh
2020-07-14 11:55:54 +02:00
Eelco Dolstra
43b8e96d30
Fix 'nix verify --all' on a binary cache (cached case)
2020-07-13 20:17:00 +02:00
Eelco Dolstra
7c2fef0a81
Make 'nix copy' to s3:// binary caches run in constant memory
2020-07-13 20:07:19 +02:00
Eelco Dolstra
493961b689
Remove istringstream_nocopy
2020-07-13 18:31:19 +02:00
Eelco Dolstra
545bb2ed03
Remove 'accessor' from addToStore()
...
This is only used by hydra-queue-runner and it's better to implement
it there.
2020-07-13 18:31:19 +02:00
Eelco Dolstra
0a9da00a10
NarAccessor: Run in constant memory
2020-07-13 17:30:42 +02:00
Eelco Dolstra
fc84c358d9
Make 'nix copy' to file:// binary caches run in constant memory
2020-07-13 16:28:45 +02:00
Eelco Dolstra
400f1a9b59
Store::pathInfoToJSON(): Use consistent format for downloadHash
2020-07-13 16:26:09 +02:00
Eelco Dolstra
c0dd05131e
toStorePath(): Return a StorePath and the suffix
2020-07-13 16:25:48 +02:00
Eelco Dolstra
1d01ae816b
Fix 'nix verify --all' on a binary cache and add a test
2020-07-13 14:35:01 +02:00
John Ericson
c466cb2091
Merge branch 'hash-always-has-type' into better-ca-parse-errors
2020-07-13 03:02:09 +00:00
John Ericson
4415765385
Merge remote-tracking branch 'upstream/master' into hash-always-has-type
2020-07-13 03:01:44 +00:00
John Ericson
230c9b4329
Change types to prepare the way for CA derivations
...
We've added the variant to `DerivationOutput` to support them, but made
`DerivationOutput::path` partial to avoid actually implementing them.
With this chage, we can all collaborate on "just" removing
`DerivationOutput::path` calls to implement CA derivations.
2020-07-12 23:53:33 +00:00
John Ericson
fedfc913ad
Use more std::visit to prepare for new variant
...
N.B. not using `std::visit` for fetchurl because there is no attempt to
handle all the cases (e.g. no `else`) and lambda complicates early
return.
2020-07-12 23:51:40 +00:00
John Ericson
ffc18583b1
Move C++17 "pattern matching" boilerplat to utils.hh
2020-07-12 22:15:14 +00:00
John Ericson
503b425690
DerivationOutputExtensional -> DerivationOutputInputAddressed
...
Thanks @regnat for the great name.
2020-07-12 15:56:20 +00:00
John Ericson
a8d4707107
Undo erroneous indentation change
2020-07-12 15:54:12 +00:00
John Ericson
18152406ce
String .drv suffix to create derivation name
2020-07-12 15:40:14 +00:00
John Ericson
5d0b75e5b6
std::string_view for new derivation name parameters
2020-07-12 15:02:36 +00:00
John Ericson
abea26a968
BasicDerivation::findOutput cannot return reference anymore
2020-07-12 01:57:06 +00:00
John Ericson
442d43178b
Merge remote-tracking branch 'upstream/master' into optional-derivation-output-storepath
2020-07-11 02:27:45 +00:00
Matthew Bauer
acb74d4d94
Merge remote-tracking branch 'origin/master' into substitute-other-storedir
2020-07-10 18:13:45 -04:00
Eelco Dolstra
8efa23bb99
Avoid a redundant hash
2020-07-10 15:56:24 +02:00
Eelco Dolstra
5dff49f661
Factor out commonality between nix-prefetch-url and nix-store --add-fixed
2020-07-10 13:21:37 +02:00
John Ericson
323d51a7a0
Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors
2020-07-09 19:07:06 +00:00
Matthew Bauer
a7884970c5
Fix DerivationOutputExtensional name
2020-07-09 11:37:18 -04:00
Eelco Dolstra
a2c27022e9
LocalStore::addToStore(srcPath): Handle the flat case
...
This helps nix-prefetch-url when using a local store.
2020-07-09 15:54:32 +02:00
Matthew Bauer
06a4e15478
Fix build.cc on linux
2020-07-08 19:27:51 -04:00
Matthew Bauer
8e0d0689be
Only store hash of fixed derivation output
...
we don’t need a full storepath for a fixedoutput derivation. So just
putting the ingestion method + the hash is sufficient.
2020-07-08 19:11:39 -04:00
Eelco Dolstra
b981e5aacf
Cleanup
2020-07-08 22:07:21 +02:00
Eelco Dolstra
34f25124ba
Make LocalStore::addToStore(srcPath) run in constant memory
...
This reduces memory consumption of
nix-instantiate \
-E 'with import <nixpkgs> {}; runCommand "foo" { src = ./blender; } "echo foo"' \
--option nar-buffer-size 10000
(where ./blender is a 1.1 GiB tree) from 1716 to 36 MiB, while still
ensuring that we don't do any write I/O for small source paths (up to
'nar-buffer-size' bytes). The downside is that large paths are now
always written to a temporary location in the store, even if they
produce an already valid store path. Thus, adding large paths might be
slower and run out of disk space. ¯\_(ツ)_/¯ Of course, you can always
restore the old behaviour by setting 'nar-buffer-size' to a very high
value.
2020-07-08 22:07:21 +02:00
Matthew Bauer
af95a7c16b
Add name to BasicDerivation
...
We always have a name for BasicDerivation, since we have a derivation
store path that has a name.
2020-07-08 15:38:01 -04:00
Eelco Dolstra
9223603908
Merge remote-tracking branch 'origin/master' into flakes
2020-07-08 15:55:19 +02:00