3277 commits

Author	SHA1	Message	Date
Robert Hensing	bd9eb5c743	DerivationGoal: only retry if output closure incomplete is only problem	2020-10-18 14:26:37 +02:00
Robert Hensing	94f1e4a441	Typo	2020-10-18 14:26:37 +02:00
John Ericson	7ed46c1574	Explain that upcast_goal is still a static cast	2020-10-17 21:50:12 +00:00
John Ericson	57d0432b39	Just use auto in two places.	2020-10-17 21:47:52 +00:00
John Ericson	e6f8ae56d8	tab -> space	2020-10-17 21:45:31 +00:00
John Ericson	0fefc2a439	Merge remote-tracking branch 'upstream/master' into typed-goal-maps	2020-10-14 20:49:01 +00:00
Eelco Dolstra	11882d7c7c	Create /etc/passwd *after* figuring out the sandbox uid/gid ... Fixes build failures like # nix log /nix/store/gjaa0psfcmqvw7ivggsncx9w364p3s8s-sshd.conf-validated.drv No user exists for uid 30012	2020-10-14 12:20:58 +02:00
John Ericson	55592b253f	Add some more docs	2020-10-13 18:04:24 +00:00
John Ericson	13804f126e	Merge remote-tracking branch 'upstream/master' into typed-goal-maps	2020-10-13 18:02:32 +00:00
Eelco Dolstra	2653801939	Merge branch 'split_build_cc' of https://github.com/obsidiansystems/nix	2020-10-13 15:36:55 +02:00
John Ericson	1b8ebe92dc	Merge remote-tracking branch 'obsidian/split_build_cc' into typed-goal-maps	2020-10-12 20:47:22 +00:00
John Ericson	a73959e6be	Merge remote-tracking branch 'upstream/master' into split_build_cc	2020-10-12 17:20:46 +00:00
John Ericson	d334fd4882	Merge branches 'split_build_hh-0', 'split_build_hh-1', 'split_build_hh-2', 'split_build_hh-3' and 'split_build_hh-4' into split_build_cc	2020-10-12 17:20:12 +00:00
John Ericson	542972f029	Trim worker.hh	2020-10-12 17:16:48 +00:00
John Ericson	0e2306204a	Rename to hand-hold git (worker.hh)	2020-10-12 17:16:48 +00:00
John Ericson	3ffa3546bd	Trim substitution-goal.hh	2020-10-12 17:16:25 +00:00
John Ericson	e77a2344d5	Rename to hand-hold git (substitution-goal.hh)	2020-10-12 17:16:25 +00:00
John Ericson	d585b4c54f	Trim hook-instance.hh	2020-10-12 17:16:13 +00:00
John Ericson	10b749a156	Rename to hand-hold git (hook-instance.hh)	2020-10-12 17:16:13 +00:00
John Ericson	8067d32f2a	Trim goal.hh	2020-10-12 17:16:00 +00:00
John Ericson	0d0e345cdc	Rename to hand-hold git (goal.hh)	2020-10-12 17:16:00 +00:00
John Ericson	2ce726947a	Trim derivation-goal.hh	2020-10-12 17:15:32 +00:00
John Ericson	4eb8c69853	Rename to hand-hold git (derivation-goal.hh)	2020-10-12 17:15:32 +00:00
John Ericson	f7099965bf	Change .cc files to use split build headers	2020-10-12 17:08:52 +00:00
John Ericson	5a97621d6d	Prepare for build/*.hh headers	2020-10-12 17:07:51 +00:00
volth	eee18f88dd	Handle amount of disk space saved by hard linking being negative ... Fixes bogus messages like "currently hard linking saves 17592186044416.00 MiB".	2020-10-12 16:06:38 +00:00
John Ericson	15fdb7cc6b	Split out uds-remote-store.{cc.hh}	2020-10-11 17:37:05 +00:00
John Ericson	38e3897162	Copy {uds-,}remote-store.{cc,hh} ... This prepares for the splitting that happens in the next commit.	2020-10-11 17:18:26 +00:00
John Ericson	5c74a6147b	Properly type the derivation and substitution goal maps ... As a bonus, Worker::removeGoal is less inefficient.	2020-10-11 17:07:14 +00:00
John Ericson	8cc510fb79	Merge branches 'build-1', 'build-2', 'build-3', 'build-4', 'build-5', 'build-6', 'build-7' and 'build-8' into split_build_cc	2020-10-11 16:44:34 +00:00
John Ericson	bcb67e1ed8	Trim lock.cc	2020-10-11 16:44:19 +00:00
John Ericson	e0be04129b	Rename to hand-hold git (lock.cc)	2020-10-11 16:44:14 +00:00
John Ericson	dbc588651c	Trim lock.hh	2020-10-11 16:44:07 +00:00
John Ericson	d0004bfcab	Rename to hand-hold git (lock.hh)	2020-10-11 16:43:59 +00:00
John Ericson	eed53ed87a	Trim build/worker.cc	2020-10-11 16:43:51 +00:00
John Ericson	904e315dae	Rename to hand-hold git (build/worker.cc)	2020-10-11 16:43:42 +00:00
John Ericson	d24ffe0eb1	Trim build/substitution-goal.cc	2020-10-11 16:43:24 +00:00
John Ericson	3633b3572b	Rename to hand-hold git (build/substitution-goal.cc)	2020-10-11 16:43:18 +00:00
John Ericson	4bdff7d1b0	Trim build/local-store-build.cc	2020-10-11 16:43:12 +00:00
John Ericson	dc5225cde5	Rename to hand-hold git (build/local-store-build.cc)	2020-10-11 16:43:05 +00:00
John Ericson	159054f730	Trim build/hook-instance.cc	2020-10-11 16:42:35 +00:00
John Ericson	f0b8987299	Rename to hand-hold git (build/hook-instance.cc)	2020-10-11 16:42:08 +00:00
John Ericson	819fe848ac	Trim build/goal.cc	2020-10-11 16:41:58 +00:00
John Ericson	184bfc301e	Rename to hand-hold git (build/goal.cc)	2020-10-11 16:41:18 +00:00
John Ericson	3bab1c5bb0	Trim build/derivation-goal.cc	2020-10-11 16:41:11 +00:00
John Ericson	9629290eda	Rename to hand-hold git (build/derivation-goal.cc)	2020-10-11 16:40:52 +00:00
John Ericson	a4f0fecb03	Trim build.hh	2020-10-11 16:40:34 +00:00
John Ericson	fc72cb0760	Rename to hand-hold git (build.hh)	2020-10-11 16:40:14 +00:00
John Ericson	428536fd75	Prepare for build/* files	2020-10-11 16:39:08 +00:00
John Ericson	aef44cbaa9	Split out commonChildInit	2020-10-11 16:38:46 +00:00
John Ericson	6cc1541782	Split out local-fs-store.hh ... This matches the already-existing `local-fs-store.cc`.	2020-10-09 20:18:08 +00:00
Eelco Dolstra	636ec17139	Remove stray DerivationOutputsAndPaths type	2020-10-09 15:41:24 +02:00
Eelco Dolstra	97ffc1e013	Dynamically disable user namespaces if CLONE_NEWUSER fails ... This makes builds work inside nixos-enter. Fixes #3145.	2020-10-07 22:46:01 +02:00
Eelco Dolstra	6aa64627c8	Support user namespaces being disabled ... If max_user_namespaces is set to 0, then don't run the build in a user namespace. Fixes #4092.	2020-10-07 22:02:36 +02:00
Eelco Dolstra	f66bbd8c7b	Doh	2020-10-07 21:25:06 +02:00
Eelco Dolstra	e705c24294	Tweak error messages	2020-10-07 17:28:43 +02:00
Eelco Dolstra	c43e882f54	Serialize exceptions from the daemon to the client	2020-10-07 17:13:54 +02:00
Eelco Dolstra	be149acfda	Serialize exceptions from the sandbox process to the parent ... Fixes #4118.	2020-10-07 16:34:03 +02:00
John Ericson	57d960dcd1	Remove generic std::optional<T> suppport from worker proto ... See comment for rational; I think it's good to leave a comment lest anyone is tempted to add such a sum-type instance again. Fixes #4113	2020-10-07 12:50:37 +00:00
Eelco Dolstra	d761485010	Prevent a deadlock when user namespace setup fails ... Observed on Centos 7 when user namespaces are disabled: DerivationGoal::startBuilder() throws an exception, ~DerivationGoal() waits for the child process to exit, but the child process hangs forever in drainFD(userNamespaceSync.readSide.get()) in DerivationGoal::runChild(). Not sure why the SIGKILL doesn't get through. Issue #4092.	2020-10-06 18:57:23 +02:00
Eelco Dolstra	ad143c5b3b	Shut up some clang warnings	2020-10-06 14:52:30 +02:00
Eelco Dolstra	85c8be6286	Remove static variable name clashes ... This was useful for an experiment with building Nix as a single compilation unit. It's not very useful otherwise but also doesn't hurt...	2020-10-06 13:49:20 +02:00
Eelco Dolstra	6691256e79	Factor out common showBytes()	2020-10-06 10:40:49 +02:00
Eelco Dolstra	d0bb544128	Add missing #pragma once	2020-10-06 10:40:07 +02:00
Eelco Dolstra	88a667e49e	Fix s3:// store ... Fixes https://github.com/NixOS/nixos-org-configurations/issues/123.	2020-10-05 17:53:30 +02:00
Eelco Dolstra	51c299213b	Merge pull request #3935 from obsidiansystems/binary-cache-addToStoreFromDump ... Get rid of Hash::dummy from BinaryCacheStore	2020-10-05 14:41:28 +02:00
Eelco Dolstra	f3aba88737	Merge pull request #3895 from obsidiansystems/templated-daemon-protocol ... More templated STL support for the daemon protocol	2020-10-05 14:40:27 +02:00
Eelco Dolstra	20a1e20d91	Style	2020-09-30 13:35:56 +02:00
John Ericson	69afaeace3	Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol	2020-09-30 00:42:28 +00:00
John Ericson	b759701652	nix::worker_proto -> worker_proto	2020-09-30 00:41:18 +00:00
John Ericson	45a0ed82f0	Revert "Use template structs instead of phantoms" ... This reverts commit 9ab07e99f5.	2020-09-30 00:39:06 +00:00
Kevin Quick	5e7838512e	Remove github-access-token in favor of access-token.	2020-09-29 16:26:34 -07:00
Kevin Quick	66c3959e8c	Merge branch 'master' into access-tokens	2020-09-29 08:32:06 -07:00
Eelco Dolstra	5999978a05	Make Headers an optional argument	2020-09-29 13:05:19 +02:00
Eelco Dolstra	cebd2fc35d	Merge branch 'github-api-token' of https://github.com/imalsogreg/nix	2020-09-29 12:17:00 +02:00
John Ericson	00135e13f4	Clarify comment a bit	2020-09-28 18:19:10 +00:00
John Ericson	80e335bb58	Use drvPath2 and give it a better name	2020-09-28 15:43:56 +00:00
John Ericson	10202bbf29	Merge remote-tracking branch 'upstream/master' into ca-floating-upstream	2020-09-28 15:39:11 +00:00
John Ericson	6c31297d80	Update src/libstore/binary-cache-store.cc ... Co-authored-by: Eelco Dolstra <edolstra@gmail.com>	2020-09-28 11:32:58 -04:00
Eelco Dolstra	649c465873	Merge pull request #4064 from serokell/balsoft/fix-max-jobs ... Fix max-jobs option	2020-09-28 10:42:02 +02:00
John Ericson	25fffdda86	Remove redundant nar hash and size setting ... Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>	2020-09-26 10:17:30 -04:00
John Ericson	1832436526	Fix up BinaryCacheStore::addToStore taking a path	2020-09-26 04:56:29 +00:00
John Ericson	5db83dd771	BinaryCacheStore::addTextToStore include CA field	2020-09-26 03:21:36 +00:00
John Ericson	c40c832f19	Merge remote-tracking branch 'obsidian/master' into binary-cache-addToStoreFromDump	2020-09-26 00:21:45 +00:00
Gregory Hale	faa5607f54	Merge remote-tracking branch 'origin/master' into github-api-token	2020-09-25 12:10:58 -04:00
Eelco Dolstra	7d81582488	Merge pull request #3953 from obsidiansystems/basic-derivation-goal-outputs ... Deduplicate basic derivation goals too	2020-09-25 17:21:19 +02:00
Kevin Quick	5a35cc29bf	Re-add support for github-access-token, but mark as deprecated.	2020-09-25 08:09:56 -07:00
Kevin Quick	ef2a14be19	Fix reference to older name for access-tokens config value.	2020-09-25 08:08:27 -07:00
Kevin Quick	8fba2a8b54	Update to use access-tokens configuration for github/gitlab access. ... This change provides support for using access tokens with other instances of GitHub and GitLab beyond just github.com and gitlab.com (especially company-specific or foundation-specific instances). This change also provides the ability to specify the type of access token being used, where different types may have different handling, based on the forge type.	2020-09-24 22:49:44 -07:00
Alexander Bantyev	ed218e1d6c	Fix max-jobs option ... After 0ed946aa61, max-jobs setting (-j/--max-jobs) stopped working. The reason was that nrLocalBuilds (which compared to maxBuildJobs to figure out whether the limit is reached or not) is not incremented yet when tryBuild is started; So, the solution is to move the check to tryLocalBuild. Closes https://github.com/nixos/nix/issues/3763	2020-09-25 00:07:42 +03:00
Eelco Dolstra	4ce8a3ed45	Hopefully fix EPERM on macOS	2020-09-23 21:29:10 +02:00
Eelco Dolstra	9a24ece122	Fix exception	2020-09-23 20:21:08 +02:00
Eelco Dolstra	236d9ee7f7	lstat() cleanup	2020-09-23 19:17:28 +02:00
Eelco Dolstra	688bd4fb50	After rewriting a path, make it read-only	2020-09-23 19:10:16 +02:00
Eelco Dolstra	cec9473871	DerivationGoal::registerOutputs(): Don't canonicalize twice ... Fixes #4021.	2020-09-23 18:21:22 +02:00
Eelco Dolstra	d4f8163d10	canonicalisePathMetaData_(): Change assertion to error message	2020-09-23 18:21:22 +02:00
Eelco Dolstra	e8f0b1e996	DerivationGoal::registerOutputs(): Fix bad format string	2020-09-23 18:21:22 +02:00
John Ericson	3f226f71c1	Return more info from BinaryCacheStore::addToStoreCommon ... We don't need it yet, but we could/should in the future, and it's a cost-free change since we already have the reference. I like it. Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>	2020-09-23 14:40:41 +00:00
John Ericson	412b3a54fb	Clarify FIXME in BinaryCacheStore::addToStoreCommon ... Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>	2020-09-23 10:36:55 -04:00
John Ericson	9fbc31a65b	Get rid of Hash::dummy from BinaryCacheStore	2020-09-23 04:56:04 +00:00
John Ericson	993229cdaf	Deduplicate basic derivation goals too ... See comments for security concerns. Also optimize goal creation by not traversing map twice.	2020-09-22 17:13:59 +00:00
John Ericson	e9fc2031f0	Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol	2020-09-22 14:18:31 +00:00
regnat	67cc356bb3	Merge remote-tracking branch 'origin/master' into non-ca-depending-on-ca	2020-09-22 16:11:25 +02:00
Eelco Dolstra	980edd1f3a	RemoteStore::addCAToStore(): Don't hold connection while calling queryPathInfo() ... This leads to a deadlock if we're at the connection limit.	2020-09-22 15:28:20 +02:00
Eelco Dolstra	5b107f2c5f	Merge pull request #4038 from maljub01/master ... Add a nix.conf option for allowing a symlinked store	2020-09-22 13:19:22 +02:00
Eelco Dolstra	35a0ac1838	Style fixes	2020-09-22 11:40:19 +02:00
Eelco Dolstra	92ac8df0ec	Merge branch 'add-ca-to-store' of https://github.com/hercules-ci/nix	2020-09-22 11:31:33 +02:00
John Ericson	3786a801c3	Merge remote-tracking branch 'upstream/master' into ca-floating-upstream	2020-09-22 04:15:55 +00:00
John Ericson	b92d3b2edd	Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol	2020-09-22 00:45:55 +00:00
Cole Helbling	ba37299a03	Serialize SandboxMode enum to string for JSON ... Rather than showing an integer as the default, instead show the boolean referenced in the description. The nix.conf.5 manpage used to show "default: 0", which is unnecessarily opaque and confusing (doesn't 0 mean false, even though the default is true?); now it properly shows that the default is true.	2020-09-21 10:36:45 -07:00
Eelco Dolstra	0716adaa8b	abstractsettingtojson.hh -> abstract-setting-to-json.hh	2020-09-21 18:49:43 +02:00
Eelco Dolstra	340ca382c4	Don't include nlohmann/json.hpp in globals.hh ... This reduces compilation time by 207s. Issue #4045.	2020-09-21 18:47:18 +02:00
Eelco Dolstra	d51ba43047	Move Callback into its own header ... This gets rid of the inclusion of <future> in util.hh, cutting compilation time by ~20s (CPU time). Issue #4045.	2020-09-21 18:42:21 +02:00
Marwan Aljubeh	f80ffeb8c9	Update the variable name accordingly	2020-09-21 17:29:08 +01:00
Eelco Dolstra	e8e1d420f3	Don't include <regex> in header files ... This reduces compilation time by ~15 seconds (CPU time). Issue #4045.	2020-09-21 18:22:45 +02:00
Marwan Aljubeh	4e1a04733d	Use a better name for the config option	2020-09-21 16:32:22 +01:00
Robert Hensing	ca30abb3fb	Document addCAToStore/addToStoreFromDump source drainage ... Also checked that all usages satisfy the requirement and removed dead code.	2020-09-21 07:55:47 +02:00
Robert Hensing	fa08db5c4c	wopAddToStore: return ValidPathInfo ... A ValidPathInfo is created anyway. By returning it we can save a roundtrip and we have a nicer interface.	2020-09-21 07:55:47 +02:00
Robert Hensing	7c68264085	wopAddToStore: add RepairFlag	2020-09-21 07:55:47 +02:00
Robert Hensing	fbf509c113	parseContentAddressMethodPrefix: use string_view ... Co-authored-by: John Ericson <git@JohnEricson.me>	2020-09-21 07:55:47 +02:00
Robert Hensing	8279178b07	Move FramedSink next to FramedSource	2020-09-21 07:55:47 +02:00
Robert Hensing	ecc8088cb7	wopAddToStore: Throw to clarify unused refs ... Co-authored-by: John Ericson <git@JohnEricson.me>	2020-09-21 07:55:47 +02:00
Robert Hensing	c602ebfb34	Refactor wopAddToStore to make wopAddTextToStore obsolete	2020-09-21 07:55:45 +02:00
Robert Hensing	e34fe47d0c	Overhaul wopAddToStore	2020-09-21 07:54:05 +02:00
Marwan Aljubeh	e40772cd35	Lint issue: replacing tabs with spaces	2020-09-18 17:18:45 +01:00
Marwan Aljubeh	c00e078343	Add a nix.conf option for allowing a symlinked store	2020-09-18 17:10:39 +01:00
Robert Hensing	14b30b3f3d	Move FramedSource and FramedSink, extract withFramedSink	2020-09-17 20:21:04 +02:00
Robert Hensing	dfa547c6a8	Add ContentAddressMethod and parse/render it	2020-09-17 20:21:04 +02:00
Robert Hensing	9ee3122ec7	Remove redundant import	2020-09-17 20:21:04 +02:00
John Ericson	b7df353f27	Merge remote-tracking branch 'upstream/master' into ca-floating-upstream	2020-09-17 16:33:10 +00:00
Eelco Dolstra	649d3aaf24	Merge pull request #3829 from obsidiansystems/remove-storetype-delegate-regStore ... Remove storetype delegate reg store -- contains #3736	2020-09-17 13:55:01 +02:00
Eelco Dolstra	b94a35ef40	Merge pull request #4027 from tweag/fix-gc-of-ca-derivations ... Fix garbage collection of CA derivations	2020-09-17 13:46:26 +02:00
regnat	520895b1da	Fix garbage collection of CA derivations ... Fix #4026	2020-09-17 13:36:58 +02:00
John Ericson	f60b380a7f	Merge remote-tracking branch 'upstream/master' into remove-storetype-delegate-regStore	2020-09-16 22:35:24 +00:00
John Ericson	2741fffa35	Ensure resolved CA derivations are written ... so we can link outputs to deriver and thus properly cache.	2020-09-16 17:57:32 +00:00
John Ericson	7fdbb377ba	Start to fix floating CA + remote building	2020-09-16 17:50:48 +00:00
John Ericson	c5ccebae00	Merge remote-tracking branch 'upstream/master' into ca-floating-upstream	2020-09-16 17:50:40 +00:00
Greg Hale	a303c0b6dc	Fetch commits from github/gitlab using Auth header ... `nix flake info` calls the github 'commits' API, which requires authorization when the repository is private. Currently this request fails with a 404. This commit adds an authorization header when calling the 'commits' API. It also changes the way that the 'tarball' API authenticates, moving the user's token from a query parameter into the Authorization header. The query parameter method is recently deprecated and will be disallowed in November 2020. Using them today triggers a warning email.	2020-09-16 13:46:48 -04:00
Théophane Hufschmitt	77a0e2c5be	Remove useless exception copy ... Co-authored-by: Eelco Dolstra <edolstra@gmail.com>	2020-09-16 14:00:21 +02:00
regnat	d72927aa7a	Fix the s3 store ... Add some necessary casts in the initialisation of the store's config	2020-09-16 13:53:09 +02:00
regnat	e0817cbcdc	Don't include nlohmann/json.hpp in config.hh ... Instead make a separate header with the template implementation of `BaseSetting<T>::toJSONObj` that can be included where needed	2020-09-16 13:53:09 +02:00
regnat	fc2d31c423	Add (StoreConfig*) casts to work around a GCC bug ... Work around https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80431 that was already there in the code but was accidentally removed in the last commits	2020-09-16 13:53:09 +02:00
regnat	888f7afe9f	Fix build issues with gcc	2020-09-16 13:53:09 +02:00
regnat	b73adacc1e	Add a name to the stores ... So that it can be printed by `nix describe-stores`	2020-09-16 13:53:09 +02:00
regnat	f24f0888f9	Document the new store hierarchy	2020-09-16 13:53:09 +02:00
regnat	29a632386e	fixup! Make the store plugins more introspectable	2020-09-16 13:53:09 +02:00
regnat	1129913c4e	fixup! Correctly call all the parent contructors of the stores	2020-09-16 13:53:09 +02:00
regnat	7f103dcddd	Properly filter the stores according to their declared uriSchemes ... When opening a store, only try the stores whose `uriSchemes()` include the current one	2020-09-16 13:53:09 +02:00
regnat	5895184df4	Correctly call all the parent contructors of the stores ... Using virtual inheritance means that only the default constructors of the parent classes will be called, which isn't what we want	2020-09-16 13:53:09 +02:00
regnat	22afa8fb4d	Separate store configs from the implems ... Rework the `Store` hierarchy so that there's now one hierarchy for the store configs and one for the implementations (where each implementation extends the corresponding config). So a class hierarchy like ``` StoreConfig-------->Store | | v v SubStoreConfig----->SubStore | | v v SubSubStoreConfig-->SubSubStore ``` (with virtual inheritance to prevent DDD). The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself	2020-09-16 13:53:08 +02:00
regnat	aa4eac3788	fixup! Separate the instantiation and initialisation of the stores	2020-09-16 13:53:08 +02:00
regnat	3c525d1590	Complete the toJSON instance for Setting<T> ... Don't let it just contain the value, but also the other fields of the setting (description, aliases, etc..)	2020-09-16 13:53:08 +02:00
regnat	3b57181f8e	Separate the instantiation and initialisation of the stores ... Add a new `init()` method to the `Store` class that is supposed to handle all the effectful initialisation needed to set-up the store. The constructor should remain side-effect free and just initialize the c++ data structure. The goal behind that is that we can create “dummy” instances of each store to query static properties about it (the parameters it accepts for example)	2020-09-16 13:53:08 +02:00
regnat	fa32560169	Fix the registration of stores	2020-09-16 13:53:08 +02:00
regnat	7d5bdf8b56	Make the store plugins more introspectable ... Directly register the store classes rather than a function to build an instance of them. This gives the possibility to introspect static members of the class or choose different ways of instantiating them.	2020-09-16 13:53:08 +02:00
John Ericson	3a5cdd737c	Rename Derivation::pathOpt to Derivation::path ... We no longer need the `*Opt` to disambiguate.	2020-09-15 15:21:39 +00:00
John Ericson	6387550d58	Get rid of confusing std::optional<bool> for validity	2020-09-15 15:19:45 +00:00
John Ericson	c4bf219b55	Don't link deriver until after any delayed exception is thrown ... Otherwise, we will associate fixed-output derivations with outputs that they did indeed produce, but which had the wrong hash. That's no good.	2020-09-15 14:28:06 +00:00
John Ericson	3ba552b245	Merge remote-tracking branch 'upstream/master' into single-ca-drv-build	2020-09-15 14:17:06 +00:00
John Ericson	c08c9f08c7	Merge remote-tracking branch 'upstream/master' into remove-storetype-delegate-regStore	2020-09-15 14:08:35 +00:00
regnat	057c6203b5	gracefully handle old daemon versions ... Add a fallback path in `queryPartialDerivationOutputMap` for daemons that don't support it. Also upstreams a couple methods from `SSHStore` to `RemoteStore` as this is needed to handle the fallback path.	2020-09-15 09:58:01 +02:00
Eelco Dolstra	a59e77d9e5	nix-daemon: Lower verbosity of restricted setting warning ... Fixes #3992.	2020-09-14 13:48:51 +02:00
aszlig	525b38eee8	Fix unspecified behaviour in readStorePathCAMap ... When deploying a Hydra instance with current Nix master, most builds would not run because of errors like this: queue monitor: error: --- Error --- hydra-queue-runner error: --- UsageError --- nix-daemon not a content address because it is not in the form '<prefix>:<rest>': /nix/store/...-somedrv The last error message is from parseContentAddress, which expects a colon-separated string, however what we got here is a store path. Looking at the worker protocol, the following message sent to the Nix daemon caused the error above: 0x1E -> wopQuerySubstitutablePathInfos 0x01 -> Number of paths 0x16 -> Length of string "/nix/store/...-somedrv" 0x00 -> Length of string "" Looking at writeStorePathCAMap, the store path is indeed the first field that's transmitted. However, readStorePathCAMap expects it to be the *second* field *on my machine*, since expression evaluation order is a classic form of unspecified behaviour[1] in C++. This has been introduced in https://github.com/NixOS/nix/pull/3689, specifically in commit 66a62b3189. [1]: https://en.wikipedia.org/wiki/Unspecified_behavior#Order_of_evaluation_of_subexpressions Signed-off-by: aszlig <aszlig@nix.build>	2020-09-13 02:40:51 +02:00
John Ericson	98dfd7531d	Fix querying outputs for CA derivations some more ... If we resolve using the known path of a derivation whose output we didn't have, we previously blew up. Now we just fail gracefully, returning the map of all outputs unknown.	2020-09-04 18:33:58 +00:00
John Ericson	075d399e3f	Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream	2020-09-04 16:04:35 +00:00
John Ericson	e9fad3006b	Fix some of the issues raised by @edolstra ... - More and better comments - The easier renames	2020-09-04 15:18:25 +00:00
John Ericson	e86dd59dcc	Apply suggestions from code review ... Thanks! Co-authored-by: Eelco Dolstra <edolstra@gmail.com>	2020-09-04 10:48:50 -04:00
John Ericson	c9f1ed912c	Don't chmod symlink before moving outputs around ... Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>	2020-09-04 14:41:53 +00:00
John Ericson	aad4abcc9c	Fix floating CA tests ... We will sometimes try to query the outputs of derivations we can't resolve. That's fine; it just means we don't know what those outputs are yet.	2020-09-04 01:17:38 +00:00
John Ericson	975a47f7fe	Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream	2020-09-03 22:09:04 +00:00
John Ericson	e7d93e7ece	Merge remote-tracking branch 'upstream/master' into single-ca-drv-build	2020-09-03 15:43:17 +00:00
Eelco Dolstra	8a945d6ddb	Remove gc-check-reachability	2020-09-03 11:30:15 +02:00
Eelco Dolstra	82b77a7726	addPermRoot(): Remove indirect flag	2020-09-03 11:29:34 +02:00
Eelco Dolstra	b07167be5a	createGeneration(): Always create an indirect root ... This means profiles outside of /nix/var/nix/profiles don't get garbage-collected. It also means we don't need to scan /nix/var/nix/profiles for GC roots anymore, except for compatibility with previously existing generations.	2020-09-03 11:13:53 +02:00
Eelco Dolstra	b74f5cdd23	createGeneration(): Take a StorePath	2020-09-03 11:06:56 +02:00
Domen Kožar	dd4b56c87f	Allow HTTP binary cache to request absolute uris	2020-09-01 21:35:48 +02:00
John Ericson	ef278d00f9	Merge remote-tracking branch 'upstream/master' into single-ca-drv-build	2020-09-01 18:01:48 +00:00
Eelco Dolstra	84f5cabbea	Merge remote-tracking branch 'origin/master' into markdown	2020-08-31 14:24:26 +02:00
John Ericson	02e0001fc0	Merge branch 'ca-floating-upstream' of github.com:obsidiansystems/nix into ca-floating-upstream	2020-08-28 21:23:09 +00:00
John Ericson	421ed527c7	Update src/libstore/build.cc ... Thanks for catching, @regnat.	2020-08-28 17:22:57 -04:00
John Ericson	48e8828050	Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream	2020-08-28 20:38:25 +00:00
John Ericson	8017fe7487	Merge remote-tracking branch 'upstream/master' into single-ca-drv-build	2020-08-28 19:59:14 +00:00
Eelco Dolstra	eb75282b8d	Merge pull request #3434 from Ericson2314/derivation-header-include-order ... Revise division of labor in deserialization of derivations	2020-08-27 16:39:28 +02:00
Eelco Dolstra	a0f19d9f3a	RemoteStore::addToStore(): Fix race between stderrThread and NAR writer ... As pointed out by @B4dM4n, the call to to.flush() on stderrThread is unsafe because the NAR writer thread is also writing to 'to'. Fixes #3943.	2020-08-27 14:50:51 +02:00
Eelco Dolstra	4bf5faf416	Merge remote-tracking branch 'origin/master' into markdown	2020-08-25 19:47:34 +02:00
John Ericson	8eb73a8724	CA derivations that depend on other CA derivations ... Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>	2020-08-24 19:01:15 +00:00
John Ericson	e0b0e18905	Add constructor for BasicDerivation -> Derivation	2020-08-24 19:01:07 +00:00
John Ericson	59979e7053	Fix bad debug format string	2020-08-24 19:01:07 +00:00
Eelco Dolstra	6a67e57019	Add DummyStore (dummy://) ... DummyStore does not allow building or adding paths. This is useful for evaluation tests when you don't want to initialize a "proper" store.	2020-08-24 18:54:16 +02:00
Eelco Dolstra	2530cc6827	Merge pull request #3950 from obsidiansystems/tabs-spaces ... tabs -> spaces	2020-08-24 10:20:28 +02:00
John Ericson	b0b59fd05a	Merge remote-tracking branch 'obsidian/write-derivation-borrow' into HEAD	2020-08-23 15:11:10 +00:00
John Ericson	35e6288be1	writeDerivation just needs a plain store reference	2020-08-23 15:01:11 +00:00
John Ericson	3a7b330b64	"Downstream placeholders" should not be store paths ... Insead they should be opaque `/<hash>` like the placeholders we already have.	2020-08-21 19:35:35 +00:00
John Ericson	422affe102	tabs -> spaces ... Sorry I let the tab sneak in there in the first place.	2020-08-21 19:19:24 +00:00
John Ericson	c265e0ea82	Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol	2020-08-20 20:24:22 +00:00
John Ericson	27a3f82c0b	Merge remote-tracking branch 'upstream/master' into single-ca-drv-build	2020-08-20 18:28:17 +00:00
John Ericson	45a2f1baab	Rename drv output querying functions, like master ... - `queryDerivationOutputMapAssumeTotal` -> `queryPartialDerivationOutputMap` - `queryDerivationOutputMapAssumeTotal` -> `queryDerivationOutputMap	2020-08-20 18:14:12 +00:00
Eelco Dolstra	4d77513d97	Merge pull request #3859 from obsidiansystems/drv-outputs-map-allow-missing ... `queryDerivationOutputMap` no longer assumes all outputs have a mapping	2020-08-20 16:49:23 +02:00
John Ericson	9a9d834dc7	Rename drv output querying functions ... - `queryDerivationOutputMapAssumeTotal` -> `queryPartialDerivationOutputMap` - `queryDerivationOutputMapAssumeTotal` -> `queryDerivationOutputMap`	2020-08-20 14:12:51 +00:00
Eelco Dolstra	acb99f03f9	Config: Use nlohmann/json	2020-08-20 11:02:16 +02:00
John Ericson	a83694c7a1	Use RemoteStore to open connection for proxying daemon ... Removes duplicate websocket opening code, and also means we should be able to to ssh-ssh-... daemon relays, not just uds-uds-... ones.	2020-08-19 19:34:47 +00:00
Eelco Dolstra	c8fa39324a	Generate the nix.conf docs from the source code ... This means we don't have two (divergent) sets of option descriptions anymore.	2020-08-19 18:28:04 +02:00
John Ericson	d5af5763cf	Merge branch 'master' of github.com:NixOS/nix into remove-storetype-delegate-regStore	2020-08-19 15:54:50 +00:00
John Ericson	be0d429b95	Merge branch 'master' of github.com:NixOS/nix into templated-daemon-protocol	2020-08-19 03:17:41 +00:00
John Ericson	950ddfdb82	Merge remote-tracking branch 'upstream/master' into derivation-header-include-order	2020-08-18 14:36:44 +00:00
Eelco Dolstra	dfeb76dbf9	Merge pull request #3930 from obsidiansystems/legacy-ssh-build-paths ... Define `LegacySSHStore::buildPaths` using `cmdBuildPaths`	2020-08-18 16:07:40 +02:00
Eelco Dolstra	7cdc739ece	Merge remote-tracking branch 'origin/master' into markdown	2020-08-17 13:43:39 +02:00
Eelco Dolstra	847a5392f4	Merge branch 'ca-no-need-trust' of https://github.com/obsidiansystems/nix	2020-08-17 10:57:02 +02:00
Eelco Dolstra	3c619f6290	Merge branch 'test-RemoteStore-buildDerivation' of https://github.com/obsidiansystems/nix	2020-08-17 10:53:18 +02:00
John Ericson	6f7ac5e865	Remove extra closing paren	2020-08-14 21:59:31 +00:00
John Ericson	3c8b5b6219	Merge remote-tracking branch 'upstream/master' into single-ca-drv-build	2020-08-14 17:00:13 +00:00
John Ericson	4b571ea321	Update src/libstore/daemon.cc ... Co-authored-by: Eelco Dolstra <edolstra@gmail.com>	2020-08-14 11:52:37 -04:00
Eelco Dolstra	13e49be660	Merge pull request #3875 from obsidiansystems/new-interface-for-path-pathOpt ... Offer a safer interface for path and pathOpt	2020-08-14 17:19:19 +02:00
Eelco Dolstra	7714d9a943	Merge pull request #3924 from obsidiansystems/features-per-store ... Make `system-features` a store setting	2020-08-14 17:13:07 +02:00
Eelco Dolstra	9b9d5297ba	Merge pull request #3909 from matthewbauer/readd-hashed-mirrors ... Add hashed-mirrors back	2020-08-14 17:05:47 +02:00
Eelco Dolstra	d81f13f7cb	Merge pull request #3899 from obsidiansystems/make-narHash-not-optional ... Make narHash in ValidPathInfo not optional	2020-08-14 17:00:18 +02:00
John Ericson	e1308b1211	Define LegacySSHStore::buildPaths using cmdBuildPaths ... Evidentally this was never implemented because Nix switched to using `buildDerivation` exclusively before `build-remote.pl` was rewritten. The `nix-copy-ssh` test (already) tests this.	2020-08-13 21:27:55 +00:00
John Ericson	5ccd94501d	Allow trustless building of CA derivations ... Include a long comment explaining the policy. Perhaps this can be moved to the manual at some point in the future. Also bump the daemon protocol minor version, so clients can tell whether `wopBuildDerivation` supports trustless CA derivation building. I hope to take advantage of this in a follow-up PR to support trustless remote building with the minimal sending of derivation closures.	2020-08-13 18:15:57 +00:00
John Ericson	85aacbee64	Use TeeSink and TeeSouce in a few more places	2020-08-13 14:51:17 +00:00
John Ericson	d2f2be0f70	Test RemoteStore::buildDerivation ... Fix `wopNarFromPath` which needed a `toRealPath`.	2020-08-13 04:07:14 +00:00
John Ericson	5d67f18c86	Merge branch 'daemon-auth-cleanup' of github.com:obsidiansystems/nix into HEAD	2020-08-12 18:22:31 +00:00
John Ericson	4720853129	Make system-features a store setting ... This seems more correct. It also means one can specify the features a store should support with --store and remote-store=..., which is useful. I use this to clean up the build remotes test.	2020-08-12 18:13:00 +00:00
John Ericson	8d4162ff9e	Separate auth and logic for the daemon ... Before, processConnection wanted to know a user name and user id, and `nix-daemon --stdio`, when it isn't proxying to an underlying daemon, would just assume "root" and 0. But `nix-daemon --stdio` (no proxying) shouldn't make guesses about who holds the other end of its standard streams. Now processConnection takes an "auth hook", so `nix-daemon` can provide the appropriate policy and daemon.cc doesn't need to know or care what it is.	2020-08-12 15:22:33 +00:00
John Ericson	5f80aea795	Break out lambda so output can be matched just once ... This is much better.	2020-08-12 02:23:31 +00:00
John Ericson	18834f7764	Recheck path validity after acquiring lock ... It might have changed, and in any event this is how the cod used to work so let's just keep it.	2020-08-11 23:44:02 +00:00
John Ericson	2de201254e	Don't assume a total output map in two places in build.cc ... Thanks @regnat for catching one of them. The other follows for many of the same reasons. I'm find fixing others on a need-to-fix basis, provided their are no regressions.	2020-08-11 23:07:50 +00:00
John Ericson	4c6aac8fdf	Clarify comment on sandbox and temp fresh paths	2020-08-11 22:46:05 +00:00
John Ericson	6d57139050	Clarify outputReferences variable with self-describing type ... Thanks for the idea, @Regnat!	2020-08-11 22:34:09 +00:00
John Ericson	8a068bd025	Remove redundant equality check	2020-08-11 21:25:40 +00:00
John Ericson	07e3466eb4	Float comment to out describe gaveUpOnSubstitution in general	2020-08-11 21:16:14 +00:00
John Ericson	d0f6e338dd	Apply suggestions from code review ... Thanks!! Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>	2020-08-11 16:49:10 -04:00
John Ericson	2a0902634e	Fix error in merge breaking floating CA drvs ... Forgot to add this hunk!	2020-08-11 00:13:19 +00:00
John Ericson	1b5c24662b	Merge branch 'small-drv-serialize-cleanup' of github.com:obsidiansystems/nix into single-ca-drv-build	2020-08-10 01:57:54 +00:00
John Ericson	581183d4d5	Deduplicate parsing and reading derivations	2020-08-10 01:40:50 +00:00
John Ericson	bcd0629c2e	Remove name parameter from writeDerivation ... The name is now stored with the derivation itself.	2020-08-10 01:35:59 +00:00
John Ericson	e913a2989f	Squashed get CA derivations building	2020-08-07 19:51:55 +00:00
Eelco Dolstra	edfd676e05	Fix .ls file names in binary caches ... These are not supposed to include the 'name' part of the store path. This was broken by 759947bf72.	2020-08-07 21:18:29 +02:00
John Ericson	8f92bb5ad9	Merge branch 'drv-outputs-map-allow-missing' of github.com:obsidiansystems/nix into templated-daemon-protocol	2020-08-07 18:51:01 +00:00
John Ericson	f7ba16f9cb	Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing	2020-08-07 17:46:39 +00:00
John Ericson	47644e49ca	Specialize std::optional<StorePath> so this is backwards compatible ... While I am cautious to break parametricity, I think it's OK in this cases---we're not about to try to do some crazy polymorphic protocol anytime soon.	2020-08-07 17:05:14 +00:00
Matthew Bauer	96c158d6e1	Fix build	2020-08-06 21:04:31 -05:00
Matthew Bauer	8abc577cc2	Merge remote-tracking branch 'origin/master' into readd-hashed-mirrors	2020-08-06 18:58:59 -05:00
Carlo Nucera	46f9dd56da	Fix bug due to non-deterministic arg eval order	2020-08-06 19:30:05 -04:00
Matthew Bauer	641c950701	Add hashed-mirrors back ... Some users have their own hashed-mirrors setup, that is used to mirror things in addition to what’s available on tarballs.nixos.org. Although this should be feasable to do with a Binary Cache, it’s not always easy, since you have to remember what "name" each of the tarballs has. Continuing to support hashed-mirrors is cheap, so it’s best to leave support in Nix. Note that NIX_HASHED_MIRRORS is also supported in Nixpkgs through fetchurl.nix. Note that this excludes tarballs.nixos.org from the default, as in \#3689. All of these are available on cache.nixos.org.	2020-08-06 18:19:09 -05:00
Carlo Nucera	9ab07e99f5	Use template structs instead of phantoms	2020-08-06 18:04:13 -04:00
Carlo Nucera	3d8240c32e	Remove leftover commented code	2020-08-06 16:04:18 -04:00
Carlo Nucera	f795f0fabc	Merge branch 'drv-outputs-map-allow-missing-namespace' of github.com:obsidiansystems/nix into templated-daemon-protocol	2020-08-06 15:53:09 -04:00
John Ericson	e89b5bd0bf	Minimize the usage of Hash::dummy	2020-08-06 18:31:48 +00:00
John Ericson	5e59b25a23	Merge remote-tracking branch 'upstream/master' into make-narHash-not-optional	2020-08-06 15:34:15 +00:00
Carlo Nucera	8b175f58d1	Simplify the namespace	2020-08-05 17:57:07 -04:00
Carlo Nucera	0739d428e0	Solve template deduction problem ... We had to predeclare our template functions	2020-08-05 17:49:45 -04:00
John Ericson	6c66331d5c	WIP: Put the worker protocol read and write in a namespace to disambig	2020-08-05 20:37:48 +00:00
Carlo Nucera	1d2e80ddd6	Merge branch 'master' of github.com:NixOS/nix into new-interface-for-path-pathOpt	2020-08-05 15:45:33 -04:00
John Ericson	ed96e603e1	Proxy -> Phantom to match Rust ... Sorry, Haskell.	2020-08-05 19:44:08 +00:00
Carlo Nucera	8241e660ba	Remove Hash::operator bool () ... Since the hash is not optional anymore	2020-08-05 15:30:38 -04:00
Carlo Nucera	be6e1c6457	Merge branch 'master' of github.com:NixOS/nix into make-narHash-not-optional	2020-08-05 15:14:47 -04:00
Carlo Nucera	1ad6394b33	Add Hash::dummy to signal default value ... We did this in the same spirit of the dummy value that's present in libstore/path.hh	2020-08-05 15:11:49 -04:00
Carlo Nucera	1d71028f4d	Remove optionality in ValidPathInfo::narInfo	2020-08-05 14:42:48 -04:00
John Ericson	7302761f64	Merge remote-tracking branch 'obsidian/drv-outputs-map-allow-missing' into templated-daemon-protocol	2020-08-05 17:53:24 +00:00
Ben Burdette	f1a47a96b6	error messages for issue 2238	2020-08-05 10:58:00 -06:00
John Ericson	cf939055c8	Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing	2020-08-05 16:43:30 +00:00
John Ericson	0559ff3d8b	Merge branch 'master' of github.com:NixOS/nix into derivation-primop-floating-output	2020-08-05 14:56:40 +00:00
John Ericson	92ad550e96	Merge remote-tracking branch 'obsidian/misc-ca' into derivation-primop-floating-output	2020-08-05 14:51:41 +00:00
Eelco Dolstra	790b694be7	Style fix	2020-08-05 16:51:06 +02:00
Eelco Dolstra	e48f944e9d	Merge branch 'misc-ca' of https://github.com/obsidiansystems/nix	2020-08-05 16:50:05 +02:00
John Ericson	b9ebe373bb	Sed some names to perhaps avoid conflicts	2020-08-05 14:49:25 +00:00
Eelco Dolstra	25f7912156	Style fix	2020-08-05 16:47:48 +02:00
John Ericson	e561a13a58	Reanme DerivationType::Regular defintion too ... This is the one non-prefixed occurence	2020-08-05 14:45:56 +00:00
John Ericson	e7b0847f2d	Make names more consistent	2020-08-05 14:44:39 +00:00
John Ericson	839f0fe095	Merge remote-tracking branch 'upstream/master' into misc-ca	2020-08-05 14:40:01 +00:00
John Ericson	03f4fafc27	Merge remote-tracking branch 'upstream/master' into misc-ca	2020-08-05 14:36:25 +00:00
Eelco Dolstra	b91dc7ebad	Merge pull request #3730 from obsidiansystems/better-ca-parse-errors ... Improve hash parsing and errors	2020-08-05 16:33:07 +02:00
Eelco Dolstra	75f220a595	Merge pull request #3864 from obsidiansystems/more-topo-sort ... Abstract out topo sorting logic	2020-08-05 16:07:29 +02:00
John Ericson	16c98bf57c	Get rid of some unneeded temporaries	2020-08-04 22:36:31 +00:00
John Ericson	1dfcbebc95	Organize and format code a bit	2020-08-04 22:28:10 +00:00
John Ericson	45b6fdb22b	Remove unused functions	2020-08-04 22:10:13 +00:00
John Ericson	1bab8a321f	Remove unneeded definition ... Template instantiations will cover this case fine.	2020-08-04 21:56:42 +00:00
John Ericson	2f2ae993dc	WIP systematize more of the worker protocol ... This refactor should *not* change the wire protocol.	2020-08-04 19:02:05 +00:00
John Ericson	fbeb8695fb	Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing	2020-08-04 18:35:33 +00:00
John Ericson	d3452a5ed6	Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors	2020-08-04 16:13:20 +00:00
Eelco Dolstra	4e7f1c7f11	S3BinaryCacheStore: Fix size determination	2020-08-04 16:01:13 +02:00
Eelco Dolstra	dfe66420e7	Revert "Remove putBytes" ... This reverts commit b8eea7e81a.	2020-08-04 15:56:10 +02:00
Eelco Dolstra	327b1bf378	BinaryCacheStore: Explicitly flush file sink ... The file sink is also flushed in its destructor, but we ignore any exceptions in the destructor. Issue #3886.	2020-08-04 14:54:07 +02:00
Eelco Dolstra	ed52cf632b	Merge pull request #3892 from obsidiansystems/path-info-header ... Move ValidPathInfo to its own header	2020-08-04 10:44:47 +02:00
John Ericson	062533f7cd	Merge remote-tracking branch 'upstream/master' into path-info-header	2020-08-03 21:02:28 +00:00
Eelco Dolstra	24e07c428f	Delete compressed NARs ... Fixes #3891.	2020-08-03 18:34:52 +02:00
John Ericson	54281f3ac1	addToStore in terms of addToStoreFromDump is not local-store-specific	2020-08-03 04:13:45 +00:00
John Ericson	9357512d73	Merge remote-tracking branch 'upstream/master' into derivation-header-include-order	2020-08-01 19:38:35 +00:00
Carlo Nucera	b6d97fdbf4	Merge branch 'master' of github.com:NixOS/nix into drv-outputs-map-allow-missing	2020-07-31 13:12:51 -04:00
Eelco Dolstra	bf290c2306	Merge remote-tracking branch 'origin/master' into markdown	2020-07-31 16:07:04 +02:00
Eelco Dolstra	1d0a7b54fa	Enable syntax highlighting	2020-07-31 15:43:25 +02:00
Carlo Nucera	eee6ef86cd	Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors	2020-07-30 18:27:25 -04:00
Matthew Bauer	05ac4db39a	Merge remote-tracking branch 'origin/master' into substitute-other-storedir	2020-07-30 12:38:24 -05:00
Eelco Dolstra	3f6e88a552	unsigned long long -> uint64_t	2020-07-30 13:34:04 +02:00
Eelco Dolstra	ebee2b7852	receiveContents(): unsigned int -> size_t	2020-07-30 13:00:30 +02:00
Eelco Dolstra	f63839bfa4	Cleanup	2020-07-29 18:04:51 +02:00
Eelco Dolstra	4c0077a07d	Fix RemoteStore::addToStore() latency ... Since 6185d25e52, this was very latency-bound since it required a round-trip for every 32 KiB. So for example copying a 514 MiB closure over a virtual ethernet device with a articial delay of just 1 ms took 343s. Now it takes 2.7s. Fixes #3372.	2020-07-29 00:48:39 +02:00
Eelco Dolstra	c159f48a39	Cleanup	2020-07-29 00:24:55 +02:00
John Ericson	df707d05d1	Merge branch 'ca-drv' of github.com:Ericson2314/nix into misc-ca	2020-07-28 21:12:36 +00:00
Carlo Nucera	2980b244b7	Use assert(false) instead of abort()	2020-07-28 15:39:45 -04:00
Carlo Nucera	cfe96bd7ce	Merge branch 'master' of github.com:NixOS/nix into ca-drv	2020-07-28 15:36:45 -04:00
Carlo Nucera	c318d398f3	Merge branch 'misc-ca' of github.com:obsidiansystems/nix into new-interface-for-path-pathOpt	2020-07-28 14:22:24 -04:00
Carlo Nucera	7ef1e3cd14	Use the new interface	2020-07-28 13:59:24 -04:00
Carlo Nucera	d564ac1c50	Offer a safer interface for pathOpt ... The new interface we offer provides a way of getting all the DerivationOutputs with the storePaths directly, based on the observation that it's the most common usecase.	2020-07-28 13:55:15 -04:00
Eelco Dolstra	b8eea7e81a	Remove putBytes ... istream->tellg() returns -1 so we can't get the number of bytes written. Fixes 'uploaded 's3://nix-cache/nar/00819r9lp5kajr6baxfw5dhhc0cx8ndxaz43qmd2f0gn1hk1ynlp.nar.xz' (-1 bytes) in 11620 ms' messages.	2020-07-28 13:27:56 +02:00
John Ericson	8065c6d160	Abstract out topo sorting logic	2020-07-27 20:45:34 +00:00
John Ericson	951415b568	Require ca-derivations everywhere we create a CA derivation ... "create" as in read one in from a serialized form, or build one from scratch in memory.	2020-07-27 17:56:36 +00:00
John Ericson	e32a9e124b	Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output	2020-07-27 17:50:06 +00:00
John Ericson	7cf978440c	Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca	2020-07-27 17:47:40 +00:00
John Ericson	e6115ca004	Merge remote-tracking branch 'upstream/master' into ca-derivation-data-types	2020-07-27 17:41:43 +00:00
John Ericson	1d7d94ceea	Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors	2020-07-27 16:17:50 +00:00
John Ericson	43f2bd8dc5	Merge remote-tracking branch 'upstream/master' into hash-always-has-type	2020-07-27 16:13:57 +00:00
John Ericson	387699fe7f	Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca	2020-07-27 15:00:28 +00:00
John Ericson	78466bcb2f	Merge branch 'optional-derivation-output-storepath' into ca-derivation-data-types	2020-07-27 14:40:08 +00:00
John Ericson	d5bb67cfa4	Merge remote-tracking branch 'upstream/master' into optional-derivation-output-storepath	2020-07-27 14:29:32 +00:00
John Ericson	699fc89b39	Merge remote-tracking branch 'upstream/master' into hash-always-has-type	2020-07-27 14:22:45 +00:00
John Ericson	2c7557481b	queryDerivationOutputMap no longer assumes all outputs have a mapping ... This assumption is broken by CA derivations. Making a PR now to do the breaking daemon change as soon as possible (if it is already too late, we can bump protocol intead).	2020-07-24 21:14:06 +00:00
John Ericson	58e55c0923	Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output	2020-07-23 19:17:52 +00:00
John Ericson	a2af25c55a	Merge remote-tracking branch 'upstream/master' into misc-ca	2020-07-23 19:14:17 +00:00
John Ericson	c36916eca0	Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output	2020-07-23 15:06:56 +00:00
John Ericson	6fcd9966bf	Merge remote-tracking branch 'upstream/master' into misc-ca	2020-07-23 15:02:29 +00:00
John Ericson	66a2067288	Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors	2020-07-23 14:56:35 +00:00
John Ericson	2274f63453	Get rid of basicDerivation::findOutput ... It's a tiny function which is: - hardly worth abstrating over, and also only used once. - doesn't work once we get CA drvs I rewrote the one callsite to be forwards compatable with CA derivations, and also potentially more performant: instead of reading in the derivation it can ust consult the SQLite DB in the common case.	2020-07-23 14:39:02 +00:00
John Ericson	9423f64ee2	Parse CA derivations using new output variants ... We no longer need `ParsedDerivation` because everything libstore needs to know about is in the `BasicDerivation` proper.	2020-07-22 23:59:25 +00:00
John Ericson	c56356bacc	Separate concerns in scanForReferences with TeeSink ... This also will make it easier to use a `HashModuloSink` instead for CA derivations.	2020-07-22 22:13:44 +00:00
Eelco Dolstra	ff314f186e	Merge pull request #3736 from obsidiansystems/allow-relative-paths-in-store-option ... Allow relative paths in --store option	2020-07-21 19:43:07 +02:00
Eelco Dolstra	0951330680	Merge pull request #3714 from obsidiansystems/add-body-to-network-errors ... Add response body to network errors	2020-07-21 18:09:42 +02:00
John Ericson	02639716ea	Merge branch 'allow-relative-paths-in-store-option' into remove-storetype-delegate-regStore	2020-07-21 15:55:27 +00:00
John Ericson	6cce32c8e8	Change logic for deciding what is a relative path for the local store ... The was Eelco's prefered logic, and it looks good to me!	2020-07-21 15:39:47 +00:00
Carlo Nucera	922a845ffc	Update chunkSize to the suggested value ... This was a suggested course of action in a review in one of our earlier commits, https://github.com/NixOS/nix/pull/3801#discussion_r457557079	2020-07-21 10:24:19 -04:00
John Ericson	54e507a7aa	Merge remote-tracking branch 'upstream/master' into add-body-to-network-errors	2020-07-21 14:17:59 +00:00
John Ericson	5ce95b9529	Update src/libstore/build.cc	2020-07-21 09:47:40 -04:00
John Ericson	cdb3f39b64	Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors	2020-07-21 13:43:20 +00:00
John Ericson	5055c595bd	Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into ca-derivation-data-types	2020-07-21 01:20:53 +00:00
John Ericson	5a37413796	Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into from-dump-stream	2020-07-21 01:20:14 +00:00
John Ericson	bf0dca7790	Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into misc-ca	2020-07-21 01:18:33 +00:00
Carlo Nucera	9aae179f34	Correct bug, thoroughly document addToStoreSlow	2020-07-20 20:18:12 -04:00
Carlo Nucera	0ca9744694	Use heuristics to decide when to show the response ... Due to https://github.com/NixOS/nix/issues/3841 we don't know how print different messages for different verbosity levels.	2020-07-20 15:57:58 -04:00
Carlo Nucera	6357b1b0fb	Add another Unimplemented case	2020-07-20 14:17:25 -04:00
Carlo Nucera	007bf6d17a	Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca	2020-07-20 14:14:39 -04:00
Carlo Nucera	362ae93851	Add UnimplementedError to ease grepping for these	2020-07-20 14:13:37 -04:00
John Ericson	bf61871271	parser.hh -> split.hh	2020-07-20 17:42:34 +00:00
John Ericson	eab945f591	Merge remote-tracking branch 'upstream/master' into from-dump-stream	2020-07-20 17:29:23 +00:00
Eelco Dolstra	1c5f8bbfb5	Merge pull request #3822 from obsidiansystems/dump-thrice-fixme ... Optimize `addToStoreSlow` and remove `TeeParseSink`	2020-07-20 18:55:05 +02:00
John Ericson	ac2fc7ba1f	Apply suggestions from code review ... Co-authored-by: Eelco Dolstra <edolstra@gmail.com>	2020-07-20 11:29:46 -04:00
Carlo Nucera	0aa79dcc6f	Remove StoreType abstraction and delegate regStore ... to each Store implementation. The generic regStore implementation will only be for the ambiguous shorthands, like "" and "auto". This also could get us close to simplifying the daemon command.	2020-07-17 17:24:02 -04:00
John Ericson	6756cecfcf	Add DerivationType::CAFloating	2020-07-17 19:55:41 +00:00
Carlo Nucera	fefd6c9e5f	Merge branch 'master' of github.com:NixOS/nix into allow-relative-paths-in-store-option	2020-07-17 14:34:20 -04:00
Carlo Nucera	bbc633c98c	Revert "Don't anticipate CA but not fixed outputs for now" ... This reverts commit 3a9e4c3262.	2020-07-17 13:10:32 -04:00
Carlo Nucera	205dcd140d	Revert "Don't anticipate multiple CA outputs for now" ... This reverts commit 74b251b2f3.	2020-07-17 12:43:46 -04:00
Carlo Nucera	1feb8981df	Revert "Don't anticipate hash algo without hash in derivation for now" ... This reverts commit 3804e3df9b.	2020-07-17 12:05:54 -04:00
Carlo Nucera	487c5751c6	Merge branch 'ca-drv' of github.com:Ericson2314/nix into misc-ca	2020-07-17 11:32:59 -04:00
Carlo Nucera	5cb840541b	Merge branch 'multi-output-hashDerivationModulo' of github.com:Ericson2314/nix into misc-ca	2020-07-17 10:28:33 -04:00
Carlo Nucera	745a03cef5	Merge branch 'optional-derivation-output-storepath' of github.com:obsidiansystems/nix into ca-derivation-data-types	2020-07-16 13:36:01 -04:00
Carlo Nucera	048e916f64	Merge branch 'master' of github.com:NixOS/nix into optional-derivation-output-storepath	2020-07-16 13:32:28 -04:00
John Ericson	cc0d77f8c9	Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors	2020-07-16 17:28:52 +00:00
Carlo Nucera	e3a3406db8	Merge branch 'master' of github.com:NixOS/nix into add-body-to-network-errors	2020-07-16 12:04:10 -04:00
John Ericson	5ea817dace	Merge remote-tracking branch 'upstream/master' into hash-always-has-type	2020-07-16 14:58:53 +00:00
John Ericson	3dcca18c30	Fix bug in TeeSource ... We use this to simplify `LocalStore::addToStoreFromDump`. Also, hope I fixed build error with old clang (used in Darwin CI).	2020-07-16 13:39:27 +00:00
Carlo Nucera	e3b394b6e8	Small namespace fix	2020-07-16 09:36:02 -04:00
Eelco Dolstra	5517eee17e	Generations API cleanup	2020-07-16 15:18:21 +02:00
Eelco Dolstra	3f264916db	Merge remote-tracking branch 'origin/flakes'	2020-07-16 14:07:32 +02:00
John Ericson	68dfb8c6ae	Optimize addToStoreSlow and remove TeeParseSink	2020-07-16 05:09:41 +00:00
John Ericson	5602637d9e	Revert "LocalStore::addToStoreFromDump copy in chunks" ... This reverts commit 592851fb67. We don't need this extra feature anymore	2020-07-15 23:37:49 +00:00
John Ericson	bc109648c4	Get rid of LocalStore::addToStoreCommon ... I got it to just become `LocalStore::addToStoreFromDump`, cleanly taking a store and then doing nothing too fancy with it. `LocalStore::addToStore(...Path...)` is now just a simple wrapper with a bare-bones sinkToSource of the right dump command.	2020-07-15 23:14:30 +00:00
Carlo Nucera	455bdee205	Merge branch 'master' of github.com:NixOS/nix into derivation-header-include-order	2020-07-15 17:58:30 -04:00
Carlo Nucera	d090562348	Merge branch 'master' of github.com:NixOS/nix into hash-always-has-type	2020-07-15 17:21:01 -04:00
John Ericson	aff69e229c	Merge remote-tracking branch 'obsidian/fix-add-to-star-narhash' into from-dump-stream	2020-07-15 21:05:29 +00:00
Carlo Nucera	2d2a10e79a	Merge branch 'master' of github.com:NixOS/nix into add-body-to-network-errors	2020-07-15 13:06:08 -04:00
Eelco Dolstra	cff2157185	Revert "LocalStore::addToStore(srcPath): Handle the flat case" ... This reverts commit a2c27022e9. See addToStoreSlow(), we don't need to handle this case efficiently anymore. In fact, we can almost remove the method/hashAlgo arguments since the non-recursive and/or non-SHA256 are almost not used anymore.	2020-07-15 12:49:03 +02:00
John Ericson	650c2c6558	Rename variable nar -> dump according to TODO	2020-07-14 21:28:50 +00:00
John Ericson	8173e7bfef	Fix localhost::addToStore(...Path...) ... We were calculating the nar hash wrong when the file ingestion method was flat. I don't think there's anything we can do in that case but dump the file again, so that's what I do. As an optomization, we again could reuse the original dump for just the recursive and non-sha256 case, but I rather do that after this fix, and after my other PRs which deduplicate this code.	2020-07-14 21:12:11 +00:00
John Ericson	592851fb67	LocalStore::addToStoreFromDump copy in chunks ... Rather than copying byte-by-byte, we let the coroutine know how much data we would like it to send back to us.	2020-07-14 13:56:36 +00:00
John Ericson	9de96ef7d4	Dedup LocalStore::addToStore* ... The downsides is that the coroutine has byte-by-byte loop transfer. Will fix that next.	2020-07-14 13:56:36 +00:00
John Ericson	c86fc3a965	Crudely make addToStoreFromDump take Source not string ... I just as little beyond the type as possible, so the implementation changes this enables can be reviewed separately.	2020-07-14 13:56:36 +00:00
John Ericson	9ec10046e0	Narrow scope of temporary value	2020-07-14 13:54:29 +00:00
Eelco Dolstra	832e111494	Merge remote-tracking branch 'origin/master' into flakes	2020-07-14 13:56:18 +02:00
Eelco Dolstra	926c3a6664	Doh	2020-07-14 11:55:54 +02:00
Eelco Dolstra	43b8e96d30	Fix 'nix verify --all' on a binary cache (cached case)	2020-07-13 20:17:00 +02:00
Eelco Dolstra	7c2fef0a81	Make 'nix copy' to s3:// binary caches run in constant memory	2020-07-13 20:07:19 +02:00
Eelco Dolstra	493961b689	Remove istringstream_nocopy	2020-07-13 18:31:19 +02:00
Eelco Dolstra	545bb2ed03	Remove 'accessor' from addToStore() ... This is only used by hydra-queue-runner and it's better to implement it there.	2020-07-13 18:31:19 +02:00
Eelco Dolstra	0a9da00a10	NarAccessor: Run in constant memory	2020-07-13 17:30:42 +02:00
Eelco Dolstra	fc84c358d9	Make 'nix copy' to file:// binary caches run in constant memory	2020-07-13 16:28:45 +02:00
Eelco Dolstra	400f1a9b59	Store::pathInfoToJSON(): Use consistent format for downloadHash	2020-07-13 16:26:09 +02:00
Eelco Dolstra	c0dd05131e	toStorePath(): Return a StorePath and the suffix	2020-07-13 16:25:48 +02:00
Eelco Dolstra	1d01ae816b	Fix 'nix verify --all' on a binary cache and add a test	2020-07-13 14:35:01 +02:00
John Ericson	c466cb2091	Merge branch 'hash-always-has-type' into better-ca-parse-errors	2020-07-13 03:02:09 +00:00
John Ericson	4415765385	Merge remote-tracking branch 'upstream/master' into hash-always-has-type	2020-07-13 03:01:44 +00:00
John Ericson	230c9b4329	Change types to prepare the way for CA derivations ... We've added the variant to `DerivationOutput` to support them, but made `DerivationOutput::path` partial to avoid actually implementing them. With this chage, we can all collaborate on "just" removing `DerivationOutput::path` calls to implement CA derivations.	2020-07-12 23:53:33 +00:00
John Ericson	fedfc913ad	Use more std::visit to prepare for new variant ... N.B. not using `std::visit` for fetchurl because there is no attempt to handle all the cases (e.g. no `else`) and lambda complicates early return.	2020-07-12 23:51:40 +00:00
John Ericson	ffc18583b1	Move C++17 "pattern matching" boilerplat to utils.hh	2020-07-12 22:15:14 +00:00
John Ericson	503b425690	DerivationOutputExtensional -> DerivationOutputInputAddressed ... Thanks @regnat for the great name.	2020-07-12 15:56:20 +00:00
John Ericson	a8d4707107	Undo erroneous indentation change	2020-07-12 15:54:12 +00:00
John Ericson	18152406ce	String .drv suffix to create derivation name	2020-07-12 15:40:14 +00:00
John Ericson	5d0b75e5b6	std::string_view for new derivation name parameters	2020-07-12 15:02:36 +00:00
John Ericson	abea26a968	BasicDerivation::findOutput cannot return reference anymore	2020-07-12 01:57:06 +00:00
John Ericson	442d43178b	Merge remote-tracking branch 'upstream/master' into optional-derivation-output-storepath	2020-07-11 02:27:45 +00:00
Matthew Bauer	acb74d4d94	Merge remote-tracking branch 'origin/master' into substitute-other-storedir	2020-07-10 18:13:45 -04:00
Eelco Dolstra	8efa23bb99	Avoid a redundant hash	2020-07-10 15:56:24 +02:00
Eelco Dolstra	5dff49f661	Factor out commonality between nix-prefetch-url and nix-store --add-fixed	2020-07-10 13:21:37 +02:00
John Ericson	323d51a7a0	Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors	2020-07-09 19:07:06 +00:00
Matthew Bauer	a7884970c5	Fix DerivationOutputExtensional name	2020-07-09 11:37:18 -04:00
Eelco Dolstra	a2c27022e9	LocalStore::addToStore(srcPath): Handle the flat case ... This helps nix-prefetch-url when using a local store.	2020-07-09 15:54:32 +02:00
Matthew Bauer	06a4e15478	Fix build.cc on linux	2020-07-08 19:27:51 -04:00
Matthew Bauer	8e0d0689be	Only store hash of fixed derivation output ... we don’t need a full storepath for a fixedoutput derivation. So just putting the ingestion method + the hash is sufficient.	2020-07-08 19:11:39 -04:00
Eelco Dolstra	b981e5aacf	Cleanup	2020-07-08 22:07:21 +02:00
Eelco Dolstra	34f25124ba	Make LocalStore::addToStore(srcPath) run in constant memory ... This reduces memory consumption of nix-instantiate \ -E 'with import <nixpkgs> {}; runCommand "foo" { src = ./blender; } "echo foo"' \ --option nar-buffer-size 10000 (where ./blender is a 1.1 GiB tree) from 1716 to 36 MiB, while still ensuring that we don't do any write I/O for small source paths (up to 'nar-buffer-size' bytes). The downside is that large paths are now always written to a temporary location in the store, even if they produce an already valid store path. Thus, adding large paths might be slower and run out of disk space. ¯\_(ツ)_/¯ Of course, you can always restore the old behaviour by setting 'nar-buffer-size' to a very high value.	2020-07-08 22:07:21 +02:00
Matthew Bauer	af95a7c16b	Add name to BasicDerivation ... We always have a name for BasicDerivation, since we have a derivation store path that has a name.	2020-07-08 15:38:01 -04:00
Eelco Dolstra	9223603908	Merge remote-tracking branch 'origin/master' into flakes	2020-07-08 15:55:19 +02:00
Eelco Dolstra	16ec7785ca	Fix 'got unknown message type 1 from Nix daemon' ... Example: $ nix-build -E 'with import <nixpkgs> {}; runCommand "foo" { x = runCommand "bar" {} "exit 1"; } "echo foo; exit 1"' warning: unknown setting 'auto-allocate-uids' these 2 derivations will be built: /nix/store/v4fbdbhcdi949929a67g8farwf72zgam-bar.drv /nix/store/k4fsvrjl7cp2xpz7927iv7g0dqj1zyhs-foo.drv warning: unknown setting 'auto-allocate-uids' building '/nix/store/v4fbdbhcdi949929a67g8farwf72zgam-bar.drv'... error: --- Error ----------------------------------------------------------------------------------------------------------------------------------------------------------------- nix-daemon builder for '/nix/store/v4fbdbhcdi949929a67g8farwf72zgam-bar.drv' failed with exit code 1 error: --- Error ------------------------------------------------------------------------------------------------------------------------------------------------------------------ nix-build got unknown message type 1 from Nix daemon	2020-07-08 15:53:14 +02:00
Eelco Dolstra	1ab9da9154	Merge remote-tracking branch 'origin/master' into flakes	2020-07-07 14:38:57 +02:00
Eelco Dolstra	7c9ece5dca	exportReferencesGraph: Fix support for non-top-level store paths ... Fixes #3471.	2020-07-07 14:25:43 +02:00
Carlo Nucera	0f05a36e20	Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors	2020-07-06 16:57:09 -04:00
Eelco Dolstra	54712aaf8a	Merge remote-tracking branch 'origin/master' into flakes	2020-07-06 16:40:10 +02:00
John Ericson	a38ab99d57	Merge remote-tracking branch 'upstream/master' into derivation-header-include-order	2020-07-05 21:49:01 +00:00
John Ericson	465daa9396	Merge remote-tracking branch 'upstream/master' into add-body-to-network-errors	2020-07-03 17:08:39 +00:00
John Ericson	3134db1a83	Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors	2020-07-03 14:12:38 +00:00
John Ericson	dbffd309fe	Merge branch 'master' of github.com:NixOS/nix into hash-always-has-type	2020-07-03 14:11:38 +00:00
Ben Burdette	b29a4ea1dc	Merge branch 'master' into add-trace	2020-07-03 07:57:36 -06:00
regnat	223fbe644a	Shorten the path to the test root ... Fix a socket length failure on the OSX builders	2020-07-03 09:20:01 +02:00
John Ericson	13796be78d	Have splitPrefix and splitPrefixTo parser helpers	2020-07-02 23:18:22 +00:00
John Ericson	2f93d9f2ba	Merge branch 'hash-always-has-type' into HEAD	2020-07-02 21:47:51 +00:00
John Ericson	1be279af26	Fix Narinfo corruption detection bug ... The aim of this check was just to ensure each key occurs once.	2020-07-02 21:46:10 +00:00
Carlo Nucera	1fc835aa22	Tighten parsing for drv files and pathinfo	2020-07-02 11:57:21 -04:00
Matthew Bauer	fc2ab42e86	Merge remote-tracking branch 'origin/master' into substitute-other-storedir	2020-07-02 11:14:04 -04:00
Matthew Bauer	d2e8b9ff0e	Store subPath in SubstitutionGoal	2020-07-02 11:12:05 -04:00
Ben Burdette	bf2788e4c1	move showTrace to new loggerSettings	2020-07-02 09:04:31 -06:00
Matthew Bauer	1f9cb06db2	Try next when no ca exists and have different store dirs	2020-07-02 10:59:24 -04:00
Carlo Nucera	263ccdd489	Rename two hash constructors to proper functions	2020-07-01 18:34:18 -04:00
Eelco Dolstra	38ccf2e241	Cleanup	2020-07-01 15:31:34 +02:00
Eelco Dolstra	86a4aba6c4	Merge branch 'remote-query-outputs' of https://github.com/tweag/nix	2020-07-01 15:10:29 +02:00
Ben Burdette	9159dfe3d8	comments and cleanup	2020-06-30 16:31:55 -06:00
Matthew Bauer	09833126f3	Merge remote-tracking branch 'origin/master' into substitute-other-storedir	2020-06-30 16:35:08 -04:00
Ben Burdette	ddb81ca126	Merge branch 'master' into add-trace	2020-06-30 12:21:45 -06:00
Carlo Nucera	77b51f4598	Factor the prefix splitting in content-address	2020-06-30 11:57:46 -04:00
John Ericson	a83566e5bc	Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors	2020-06-29 18:40:34 +00:00
John Ericson	987a4a0be9	Merge remote-tracking branch 'upstream/master' into hash-always-has-type	2020-06-29 18:03:21 +00:00
Ben Burdette	8f81fae116	showTrace flag in loggers	2020-06-29 10:20:51 -06:00
Eelco Dolstra	58bc3b6578	Merge pull request #3729 from obsidiansystems/simpler-hased-mirror ... hashed-mirrors: Use parsed derivation output rather than reconstructing it	2020-06-29 14:04:12 +02:00
Eelco Dolstra	64232f3ea6	Merge pull request #3749 from rodarima/master ... Fall back to copyPath if link fails with EPERM	2020-06-29 13:31:24 +02:00
Carlo Nucera	278a679782	Merge branch 'master' of github.com:NixOS/nix into allow-relative-paths-in-store-option	2020-06-26 10:40:13 -04:00
Eelco Dolstra	adf2fbbdc2	Merge remote-tracking branch 'origin/master' into flakes	2020-06-26 08:46:46 +02:00
John Ericson	d92d4f85a5	Move ValidPathInfo to its own header	2020-06-25 17:48:49 +00:00
Eelco Dolstra	de2641ae99	Fix empty std::optional dereference in writeDerivation() ... https://hydra.nixos.org/build/123017579	2020-06-25 15:50:30 +02:00
Rodrigo	3a642187c3	Fall back to copyPath if link fails with EPERM ... BeeGFS doesn't allow hard-links and returns EPERM, so we fall back to copyPath. See https://github.com/NixOS/nix/issues/3748	2020-06-25 12:03:26 +02:00
regnat	d38f860c3e	Add a way to get all the outputs of a derivation with their label ... Generalize `queryDerivationOutputNames` and `queryDerivationOutputs` by adding a `queryDerivationOutputMap` that returns the map `outputName=>outputPath` (not that this is not equivalent to merging the results of `queryDerivationOutputs` and `queryDerivationOutputNames` as sets don't preserve the order, so we would end up with an incorrect mapping). squash! Add a way to get all the outputs of a derivation with their label Rename StorePathMap to OutputPathMap	2020-06-24 20:38:40 +02:00
John Ericson	d9fb9e9f26	Merge remote-tracking branch 'upstream/master' into ca-drv	2020-06-24 15:41:29 +00:00
John Ericson	98e5d1af03	Merge remote-tracking branch 'upstream/master' into hash-always-has-type	2020-06-23 17:03:37 +00:00
Carlo Nucera	e197bc6229	Enable the --store option to take relative paths ... In nix commands which accept --store options, we can now specify a relative path, which will be canonicalized.	2020-06-23 11:13:18 -04:00
John Ericson	55d4bd6e0e	Improve content address parsing ... - Ensure hash is in form <algo>-<prefix> and not SRI. - Better errors if something goes wrong - string_view for no coppying	2020-06-22 18:08:27 +00:00
Matthew Bauer	66a62b3189	Merge remote-tracking branch 'origin/master' into substitute-other-storedir	2020-06-22 13:08:11 -04:00
John Ericson	f4a5913125	hashed-mirrors: Use parsed derivation output rather than reconstructing it ... Now the derivation outputs are parsed up front, we can avoid a reparse by doing it. Also, this just feels a bit better as the `output*` env vars are more of a `libnixexpr` interface than `libnixstore` interface: ultimately, it's the derivation outputs that decide whether the derivation is fixed-output. Yes, hashed mirrors might go away with #3689, but this bit of code would be moved rather than deleted, so it's worth doing a cleanup anyways I think.	2020-06-22 15:17:20 +00:00
John Ericson	93bbe6e8ab	Merge remote-tracking branch 'upstream/master' into ca-drv	2020-06-21 21:11:58 +00:00
John Ericson	3804e3df9b	Don't anticipate hash algo without hash in derivation for now ... When we merge with master, the new lack of string types make this case impossible (after parsing). Later, when we actually implemenent CA-derivations, we'll change the types to allow that.	2020-06-21 21:05:37 +00:00
John Ericson	8313f0e939	Merge remote-tracking branch 'upstream/master' into derivation-header-include-order	2020-06-21 20:39:10 +00:00
John Ericson	bf9f040112	Tweak declaration ... I think this is clearer	2020-06-21 16:51:39 +00:00
John Ericson	fdeabf7160	Merge remote-tracking branch 'upstream/master' into multi-output-hashDerivationModulo	2020-06-21 16:43:17 +00:00
Matthew Bauer	f2a6cee334	Update worker protocol to support sending storepath maps ... We need to also send the ca to daemon in addition to the path.	2020-06-19 18:06:19 -04:00
John Ericson	507aa48739	WIP: Make Hash always store a valid hash type	2020-06-19 21:48:57 +00:00
Matthew Bauer	e8e1f5282f	Replace error message in getDerivationCA	2020-06-19 15:19:09 -04:00
Matthew Bauer	0c9c1b8826	Return map of StorePaths in copyPaths ... This allows the caller to know what values were actually added to the store.	2020-06-19 14:48:57 -04:00
Matthew Bauer	7e11cf3399	Merge remote-tracking branch 'origin/master' into substitute-other-storedir	2020-06-19 14:40:49 -04:00
Matthew Bauer	2796b01e86	Merge remote-tracking branch 'origin/master' into substitute-other-storedir	2020-06-19 14:22:06 -04:00
John Ericson	68294746ae	Merge remote-tracking branch 'upstream/master' into no-hash-type-unknown	2020-06-19 17:53:34 +00:00
John Ericson	911fc88bcb	More designated initializers	2020-06-19 17:42:56 +00:00
John Ericson	2f0e395c99	Merge remote-tracking branch 'me/no-stringly-typed-derivation-output' into validPathInfo-ca-proper-datatype	2020-06-19 15:26:59 +00:00
John Ericson	fb39a5e00c	Remove unneeded constructor for DerivationOutputHash	2020-06-19 15:11:11 +00:00
John Ericson	01dc8b0bab	Merge remote-tracking branch 'upstream/master' into no-stringly-typed-derivation-output	2020-06-19 14:59:05 +00:00
John Ericson	145d88cb2a	Use designated initializers for DerivationOutputHash	2020-06-19 14:58:30 +00:00
John Ericson	237d88c97e	FileSystemHash -> DerivationOutputHash	2020-06-19 14:47:10 +00:00
John Ericson	3fc58a9638	Remove some Base:: that crept in	2020-06-19 00:24:47 +00:00
John Ericson	3f8dcfe3fd	Merge branch 'validPathInfo-temp' into validPathInfo-ca-proper-datatype	2020-06-18 23:01:58 +00:00
John Ericson	669c3992e8	Merge branch 'no-hash-type-unknown' into validPathInfo-temp	2020-06-18 22:33:07 +00:00
John Ericson	15abb2aa2b	Revert the enum struct change ... Not a regular git revert as there have been many merges and things.	2020-06-18 22:11:26 +00:00
John Ericson	bbbf3602a3	Merge branch 'enum-class' into no-hash-type-unknown	2020-06-18 22:11:19 +00:00
John Ericson	40526fbea5	Merge remote-tracking branch 'upstream/master' into enum-class	2020-06-18 21:38:15 +00:00
John Ericson	1b23fe4afb	Fix bugs ... - Bad dynamic cast target ...classic - std::shared_ptr need explicit deref	2020-06-18 19:03:10 +00:00
John Ericson	fb432b0bd2	Merge remote-tracking branch 'obsidian/fix-url-format' into add-body-to-network-errors	2020-06-18 18:08:02 +00:00
John Ericson	639e20dc3e	Prevent '%' in URL from causing crashes ... We have a larger problem that passsing computed strings to the first variable argument of many exception constructors is unsafe because that first variable argument is interpreted not as a plain string, but format string, and if it contains '%' boost::format will abort, since there are no arguments to the format string. In this particular instance '%' was used as part of an escape code in a URL, which, when the download failed, caused Nix to abort displaying the `FileTransferError`.	2020-06-18 17:54:16 +00:00
John Ericson	74b219ef6e	Adjust FileTransferError message to use opt response	2020-06-18 14:48:45 +00:00
regnat	4fef2ba7e4	Rename content-addressed-paths into ca-derivations ... See <https://github.com/NixOS/nix/pull/3710#issuecomment-645480333>	2020-06-18 09:25:55 +02:00
Carlo Nucera	004570a377	Add HTTP responses to FileTransferErrors	2020-06-17 22:34:31 +00:00
Carlo Nucera	a835c740ca	Replace TransferItem::status with a local variable ... Everywhere seems to use `getHTTPStatus` now.	2020-06-17 22:16:24 +00:00
Carlo Nucera	fb05a6adcf	Eliminate old TeeSink abstraction ... This was introduced in fa125b9b28, and then "reverted" in 1cf4801108, except that revert left the struct around doing nothing useful. We're removing it all the way now because we want to make a new `TeeSink` complementing the already-exiting `TeeSource`, that is actually a completely different concept as far as the class hierarchy is concerned.	2020-06-17 22:07:22 +00:00
Eelco Dolstra	2b8f33bf5f	Merge pull request #3713 from matthewbauer/cleanup-warnings ... Cleanup class StorePath warning	2020-06-17 21:19:30 +02:00
Matthew Bauer	5e631e3304	Add StorePathCAMap for querySubstitutablePathInfos ... I’m not 100% sure this is wanted since it kind of makes everything have to know about ca even if they don’t really want to. But it also make things easier in dealing with looking up ca.	2020-06-17 15:03:05 -04:00
Matthew Bauer	be50de1142	Make sure references are empty for store path replacing ... also copy info2 instead of casting	2020-06-17 14:15:08 -04:00
Matthew Bauer	8974755d19	Add assert for replaced storePath	2020-06-17 14:04:46 -04:00
Eelco Dolstra	3078404e35	Merge pull request #3712 from obsidiansystems/make-http-successful-states-coherent ... Make successful states coherent	2020-06-17 19:37:46 +02:00
Matthew Bauer	7bd88cc1dc	Merge remote-tracking branch 'origin/master' into substitute-other-storedir	2020-06-17 13:32:29 -04:00
Matthew Bauer	22d7d36703	Remove unused narInfoFile in binary-cache-store	2020-06-17 13:27:19 -04:00
Matthew Bauer	f767bedfac	Replace struct StorePath with class StorePath ... also a similar case with struct Goal	2020-06-17 13:26:37 -04:00
Carlo Nucera	4930cb48a2	Include review comments	2020-06-17 12:58:59 -04:00
Matthew Bauer	6438ba1e99	Update strings from review comment	2020-06-17 12:31:01 -04:00
Eelco Dolstra	2f51cd8dc9	Merge pull request #3710 from tweag/reserve_ca_derivations ... Reserve the `__contentAddressed` derivation parameter	2020-06-17 18:28:26 +02:00
Eelco Dolstra	4d5169bdd5	Merge pull request #3707 from p01arst0rm/outdated-function-fix ... replaced uncaught_exception with uncaught_exceptions	2020-06-17 18:26:01 +02:00
regnat	480b54e1c6	fixup! Reserve the __contentAddressed derivation parameter	2020-06-17 17:37:04 +02:00
Carlo Nucera	079c6e87de	Make successful states coherent ... The successful states used in these two places in the code were slightly different. Should they be the same list?	2020-06-17 11:16:16 -04:00
regnat	56d75bf4fc	Reserve the __contentAddressed derivation parameter ... Not implementing anything here, just throwing an error if a derivation sets `__contentAddressed = true` without `--experimental-features content-addressed-paths` (and also with it as there's nothing implemented yet)	2020-06-17 15:41:17 +02:00
Eelco Dolstra	fdff09e57c	Fix coverage build	2020-06-17 15:18:10 +02:00
Eelco Dolstra	ccbea8255c	Merge pull request #3657 from obsidiansystems/sligthly-improve-store-path-documentation ... Clarify the description of StorePath inputs	2020-06-17 14:54:37 +02:00
Eelco Dolstra	1524752c17	Merge remote-tracking branch 'origin/master' into flakes	2020-06-17 10:26:52 +02:00
John Ericson	517f5980e2	Merge remote-tracking branch 'upstream/master' into no-stringly-typed-derivation-output	2020-06-17 04:58:43 +00:00
John Ericson	18493fd9c4	Move some Store functions from derivations.cc to store-api.cc ... This further continues with the dependency inverstion. Also I just went ahead and exposed `parseDerivation`: it seems like the more proper building block, and not a bad thing to expose if we are trying to be less wedded to drv files on disk anywas.	2020-06-17 04:30:16 +00:00
John Ericson	7130f0a3a6	Don't need abstract struct Derivation in local-store	2020-06-17 04:05:38 +00:00
John Ericson	21ef342172	Merge remote-tracking branch 'upstream/master' into derivation-header-include-order	2020-06-17 03:52:01 +00:00
John Ericson	02928f76fd	Merge remote-tracking branch 'upstream/master' into multi-output-hashDerivationModulo	2020-06-17 03:51:02 +00:00
John Ericson	49308ef039	Merge remote-tracking branch 'upstream/master' into ca-drv	2020-06-17 03:39:39 +00:00
p01arst0rm	c9d06558b6	replaced uncaught_exception with uncaught_exceptions	2020-06-17 03:15:47 +01:00
Eelco Dolstra	29542865ce	Remove StorePath::clone() and related functions	2020-06-16 22:20:18 +02:00
Eelco Dolstra	759947bf72	StorePath: Rewrite in C++ ... On nix-env -qa -f '<nixpkgs>', this reduces maximum RSS by 20970 KiB and runtime by 0.8%. This is mostly because we're not parsing the hash part as a hash anymore (just validating that it consists of base-32 characters). Also, replace storePathToHash() by StorePath::hashPart().	2020-06-16 14:28:41 +02:00
Eelco Dolstra	72e17290d4	Fix FTP support ... Fixes #3618.	2020-06-16 11:53:04 +02:00
Eelco Dolstra	a588b6b19d	Print only one error message if a build fails ... E.g. instead of error: --- BuildError ----------------------------------------------- nix builder for '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed with exit code 1 error: --- Error ---------------------------------------------------- nix build of '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed we now get error: --- Error ---------------------------------------------------- nix builder for '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed with exit code 1	2020-06-15 19:35:31 +02:00
Eelco Dolstra	31707735b6	Remove unnecessary amDone() overrides	2020-06-15 16:47:21 +02:00
Eelco Dolstra	5ed5d7acbd	Improve "waiting for locks" messages ... These are now shown in the progress bar. Closes #3577.	2020-06-15 16:03:29 +02:00
Eelco Dolstra	e14e62fddd	Remove trailing whitespace	2020-06-15 14:12:39 +02:00
Eelco Dolstra	1fb762d11f	Get rid of explicit ErrorInfo constructors	2020-06-15 14:06:58 +02:00
Eelco Dolstra	7a77762961	Merge branch 'errors-phase-2' of https://github.com/bburdette/nix	2020-06-15 11:46:31 +02:00
Matthew Bauer	c214cda940	Correctly substitute from different storeDir ... Originally, the test was only checking for different “real” storeDir. That’s an easy case to handle, but the much harder one is if different virtual store dirs are used. To do this, we need the SubstitutionGoal to know about the ca, so it can recalculate the path to copy it over. An important note here is that the store path passed to copyStorePath needs to be one for srcStore - so that queryPathInfo works properly. This also adds an error message when the store path from queryPathInfo is different from the one we requested.	2020-06-13 00:07:42 -05:00
Matthew Bauer	b2cb288cdd	Add makeFixedOutputPathFromCA function ... This puts what we are already doing into a shared method. It just needs a path name and a ca and produces a store path.	2020-06-12 16:36:35 -05:00
John Ericson	f6f01416b7	Use std::string_view in a few more places	2020-06-12 21:32:30 +00:00
Matthew Bauer	88120442d2	Debug when storePath changes ... these rewrites should be transparent, but they are important to know about when debugging	2020-06-12 15:32:52 -05:00
Matthew Bauer	5b386b05f5	Recompute storePath based on isContentAddressed	2020-06-12 14:57:45 -05:00
Matthew Bauer	85d01e1f65	Don’t use makeStorePath	2020-06-12 14:53:30 -05:00
Matthew Bauer	9077b9e6b2	Separate dstStore path from srcStore path	2020-06-12 14:27:28 -05:00
Matthew Bauer	2f2ac850b5	Compute new store path correctly	2020-06-12 14:03:17 -05:00
Matthew Bauer	11c97070f3	Fix storeDir != storeDir condition ... this needs to only continue if the path replacement fails.	2020-06-12 10:14:03 -05:00
Matthew Bauer	3e3eaa90dd	Remove hashed-mirrors	2020-06-12 09:51:44 -05:00
Matthew Bauer	79c169d1c6	Allow substituting from different storeDir ... Substituters can substitute from one store dir to another with a little bit of help. The store api just needs to have a CA so it can recompute the store path based on the new store dir. We can only do this for fixed output derivations with no references, though.	2020-06-12 09:49:09 -05:00
Eelco Dolstra	9f736dd89d	Add Store::readDerivation() convenience function	2020-06-12 13:04:52 +02:00
Eelco Dolstra	045b07200c	Remove Store::queryDerivationOutputNames() ... This function was used in only one place, where it could easily be replaced by readDerivation() since it's not performance-critical. (This function appears to have been modelled after queryDerivationOutputs(), which exists only to make the garbage collector faster.)	2020-06-12 12:46:33 +02:00
Ben Burdette	ef1b3f21b6	Merge remote-tracking branch 'upstream/master' into errors-phase-2	2020-06-11 14:06:35 -06:00
Eelco Dolstra	ac4d43a31b	Merge pull request #3073 from tweag/machine-logs ... Add an option to print the logs in a machine-readable format	2020-06-11 15:45:18 +02:00
Eelco Dolstra	dd9bb11d0d	Move names.{cc,hh} to libstore	2020-06-11 15:42:18 +02:00
Eelco Dolstra	b9ae1bdd7a	Merge pull request #3655 from zimbatm/hash-encoding-prepare ... libutils/hash: remove default encoding	2020-06-10 11:48:38 +02:00
Matthew Bauer	b2c8061b44	Disable extra-platforms = i686-linux on wsl1 (#3676) ... WSL1 doesn’t support i686-linux emulation, see https://github.com/microsoft/wsl/issues/2468	2020-06-09 21:53:53 +00:00
Eelco Dolstra	29e0748847	Show HTTP status message ... For example: warning: unable to download 'https://api.github.com/repos/edolstra/dwarffs/commits/master': HTTP error 403 ('rate limit exceeded'); using cached version	2020-06-09 14:20:22 +02:00
Eelco Dolstra	447ea52b07	FileTransfer: Don't store status since curl already does that	2020-06-09 14:05:15 +02:00
regnat	4983401440	Unify the printing of the logs between bar-with-logs and raw ... Make the printing of the build logs systematically go through the logger, and replicate the behavior of `no-build-output` by having two different loggers (one that prints the build logs and one that doesn't)	2020-06-08 09:31:15 +02:00
Carlo Nucera	d614166cb6	Fix condition error and make test suite pass	2020-06-04 17:21:21 -04:00
John Ericson	e5cc1ebc5d	Merge remote-tracking branch 'upstream/master' into no-stringly-typed-derivation-output	2020-06-04 21:04:35 +00:00
John Ericson	94ddea9e2f	Use readString rather than >> temporary ... Fixed the rest of these before, but this one slipped through.	2020-06-04 20:55:08 +00:00
John Ericson	2041499b5e	Flip boolean ... Thanks Matt!	2020-06-04 20:42:25 +00:00
John Ericson	ed86acf02a	Use some std::optional::has_value for clarity	2020-06-04 20:42:02 +00:00
Matthew Bauer	2299ef705c	Add error message when FileIngestionMethod is out of bounds ... bool coerces anything >0 to true, but in the future we may have other file ingestion methods. This shows a better error message when the “recursive” byte isn’t 1.	2020-06-04 11:32:39 -05:00
John Ericson	53bc8ff152	No C++ designated initializers yet with Clang 7	2020-06-03 20:45:14 -04:00
Ben Burdette	4335ba999b	Merge remote-tracking branch 'upstream/master' into errors-phase-2	2020-06-03 17:00:00 -06:00
Carlo Nucera	132d6f2c24	Clarify the description of StorePath construction	2020-06-03 16:08:32 -04:00
John Ericson	74b251b2f3	Don't anticipate multiple CA outputs for now	2020-06-03 18:53:04 +00:00