3565 commits

Author	SHA1	Message	Date
Eelco Dolstra	3b7e00ce22	Move primeCache() to Worker::run() ... We need the missing path info to communicate the worker's remaining goals to the progress bar.	2020-12-01 13:44:48 +01:00
regnat	9bd8184f1f	Allow fixed-output derivations to depend on (floating) content-addressed ones ... Fix an overlook of https://github.com/NixOS/nix/pull/4056	2020-11-27 15:39:24 +01:00
regnat	13c557fe82	fix the hash rewriting for ca-derivations	2020-11-25 11:33:00 +01:00
John Ericson	2113ae2d85	Make drv hash modulo memo table thread-safe ... Let's get one step closer to the daemon not needing to fork.	2020-11-19 16:50:06 +00:00
Eelco Dolstra	df5c69a94e	Merge pull request #4180 from Ma27/ssh-ng-substitute ... Allow substituting paths when building remotely using `ssh-ng://`	2020-11-17 14:01:04 +01:00
Eelco Dolstra	e6b7c7b79c	Cleanup	2020-11-17 13:58:55 +01:00
Eelco Dolstra	bccff827dc	Fix deadlock in IFD through the daemon ... Fixes #4235.	2020-11-17 13:50:36 +01:00
regnat	7de21f6664	Make the sql debug statements more useful ... Print the expanded sql query (with the variables bound to their value) rather than the original one in case of error	2020-11-17 10:05:45 +01:00
Jake Waksbaum	01db455733	Fix deadlock in nix-store when max-connections=1 ... This fixes a bug I encountered where `nix-store -qR` will deadlock when the `--include-outputs` flag is passed and `max-connections=1`. The deadlock occurs because `RemoteStore::queryDerivationOutputs` takes the only connection from the connection pool and uses it to check the daemon version. If the version is new enough, it calls `Store::queryDerivationOutputs`, which eventually calls `RemoteStore::queryPartialDerivationOutputMap`, where we take another connection from the connection pool to check the version again. Because we still haven't released the connection from the caller, this waits for a connection to be available, causing a deadlock. This diff solves the issue by using `getProtocol` to check the protocol version in the caller `RemoteStore::queryDerivationOutputs`, which immediately frees the connection back to the pool before returning the protocol version. That way we've already freed the connection by the time we call `RemoteStore::queryPartialDerivationOutputMap`.	2020-11-16 02:35:50 -05:00
Maximilian Bosch	3a63fc6cd5	Allow substituting paths when building remotely using ssh-ng:// ... Until now, it was not possible to substitute missing paths from e.g. `https://cache.nixos.org` on a remote server when building on it using the new `ssh-ng` protocol. This is because every store implementation except legacy `ssh://` ignores the substitution flag passed to `Store::queryValidPaths` while the `legacy-ssh-store` substitutes the remote store using `cmdQueryValidPaths` when the remote store is opened with `nix-store --serve`. This patch slightly modifies the daemon protocol to allow passing an integer value suggesting whether to substitute missing paths during `wopQueryValidPaths`. To implement this on the daemon-side, the substitution logic from `nix-store --serve` has been moved into a protected method named `Store::substitutePaths` which gets currently called from `LocalStore::queryValidPaths` and `Store::queryValidPaths` if `maybeSubstitute` is `true`. Fixes #2770	2020-11-05 20:12:37 +01:00
Eelco Dolstra	e8c379555f	LocalStore: Get rid of recursive_mutex	2020-11-03 14:45:24 +01:00
Eelco Dolstra	797a52e31d	Add FIXME	2020-11-03 14:45:24 +01:00
Eelco Dolstra	7cf874c17d	Don't use readDerivation() in addValidPath() ... readDerivation() requires a valid path. Fixes #4210.	2020-11-02 18:46:44 +01:00
Eelco Dolstra	ab2ef851b6	Merge pull request #4207 from hercules-ci/fix-RemoteStore-filterSource-deadlock ... Fix RemoteStore pool deadlock in filterSource etc	2020-11-02 14:46:10 +01:00
Eelco Dolstra	db5424bf09	Don't send eval-related settings to the daemon	2020-11-02 13:57:58 +01:00
Robert Hensing	e8a45d07bc	Restore RestrictedStore.addToStoreFromDump implementation ... It was accidentally removed in commit ca30abb3fb	2020-10-31 23:56:03 +01:00
Robert Hensing	2192cac634	Fix RemoteStore pool deadlock in filterSource etc	2020-10-30 21:47:34 +01:00
Eelco Dolstra	ff4dea63c9	Generalize extra-* settings ... This removes the extra-substituters and extra-sandbox-paths settings and instead makes every array setting extensible by setting "extra-<name> = <value>" in the configuration file or passing "--<name> <value>" on the command line.	2020-10-29 18:17:39 +01:00
stev	869c0321ff	Alter "wanted:" to "specified:" in hash mismatch output ... This makes it even clearer which of the two hashes was specified in the nix files. Some may think that "wanted" and "got" is obvious, but: "got" could mean "got in nix file" and "wanted" could mean "want to see in nix file".	2020-10-29 00:33:14 +01:00
Eelco Dolstra	02a1facbdc	Merge pull request #4056 from tweag/non-ca-depending-on-ca ... Allow non-CA derivations to depend on CA ones	2020-10-27 17:38:29 +01:00
regnat	bc081bcd81	Inline unkownHashes ... See https://github.com/NixOS/nix/pull/4056#discussion_r493661632	2020-10-27 07:29:25 +01:00
regnat	c092fa4702	Allow non-CA derivations to depend on CA derivations	2020-10-27 07:29:23 +01:00
Christian Kampka	461cf2b856	Add NIX_CONFIG env var for applying nix.conf overrides	2020-10-21 13:41:26 +02:00
Eelco Dolstra	fda835b231	Merge pull request #4143 from obsidiansystems/typed-goal-maps ... Properly type the derivation and substitution goal maps	2020-10-18 18:12:21 +02:00
Robert Hensing	bd9eb5c743	DerivationGoal: only retry if output closure incomplete is only problem	2020-10-18 14:26:37 +02:00
Robert Hensing	94f1e4a441	Typo	2020-10-18 14:26:37 +02:00
John Ericson	7ed46c1574	Explain that upcast_goal is still a static cast	2020-10-17 21:50:12 +00:00
John Ericson	57d0432b39	Just use auto in two places.	2020-10-17 21:47:52 +00:00
John Ericson	e6f8ae56d8	tab -> space	2020-10-17 21:45:31 +00:00
John Ericson	d3cfc14e3a	Merge remote-tracking branch 'upstream/master' into auto-uid-allocation	2020-10-17 19:31:13 +00:00
John Ericson	801e6d96d8	Merge commit 'd334fd48824b41b57e267cd2926fa9619b7718e3' into auto-uid-allocation	2020-10-17 19:31:06 +00:00
John Ericson	2c9a8e7421	Merge commit '97ffc1e0139e124b7e36b5d1a62b90300f231118' into auto-uid-allocation	2020-10-17 19:20:18 +00:00
John Ericson	81c5f754a7	Merge commit '6aa64627c8e431c3b187f7bb44c943d06e39b929' into auto-uid-allocation	2020-10-17 19:12:59 +00:00
John Ericson	2546c63373	Merge commit 'f66bbd8c7bb1472facf8917e58e3cd4f6ddfa1b5' into auto-uid-allocation	2020-10-17 17:25:17 +00:00
John Ericson	0fefc2a439	Merge remote-tracking branch 'upstream/master' into typed-goal-maps	2020-10-14 20:49:01 +00:00
Eelco Dolstra	11882d7c7c	Create /etc/passwd *after* figuring out the sandbox uid/gid ... Fixes build failures like # nix log /nix/store/gjaa0psfcmqvw7ivggsncx9w364p3s8s-sshd.conf-validated.drv No user exists for uid 30012	2020-10-14 12:20:58 +02:00
John Ericson	55592b253f	Add some more docs	2020-10-13 18:04:24 +00:00
John Ericson	13804f126e	Merge remote-tracking branch 'upstream/master' into typed-goal-maps	2020-10-13 18:02:32 +00:00
Eelco Dolstra	2653801939	Merge branch 'split_build_cc' of https://github.com/obsidiansystems/nix	2020-10-13 15:36:55 +02:00
John Ericson	1b8ebe92dc	Merge remote-tracking branch 'obsidian/split_build_cc' into typed-goal-maps	2020-10-12 20:47:22 +00:00
John Ericson	a73959e6be	Merge remote-tracking branch 'upstream/master' into split_build_cc	2020-10-12 17:20:46 +00:00
John Ericson	d334fd4882	Merge branches 'split_build_hh-0', 'split_build_hh-1', 'split_build_hh-2', 'split_build_hh-3' and 'split_build_hh-4' into split_build_cc	2020-10-12 17:20:12 +00:00
John Ericson	542972f029	Trim worker.hh	2020-10-12 17:16:48 +00:00
John Ericson	0e2306204a	Rename to hand-hold git (worker.hh)	2020-10-12 17:16:48 +00:00
John Ericson	3ffa3546bd	Trim substitution-goal.hh	2020-10-12 17:16:25 +00:00
John Ericson	e77a2344d5	Rename to hand-hold git (substitution-goal.hh)	2020-10-12 17:16:25 +00:00
John Ericson	d585b4c54f	Trim hook-instance.hh	2020-10-12 17:16:13 +00:00
John Ericson	10b749a156	Rename to hand-hold git (hook-instance.hh)	2020-10-12 17:16:13 +00:00
John Ericson	8067d32f2a	Trim goal.hh	2020-10-12 17:16:00 +00:00
John Ericson	0d0e345cdc	Rename to hand-hold git (goal.hh)	2020-10-12 17:16:00 +00:00
John Ericson	2ce726947a	Trim derivation-goal.hh	2020-10-12 17:15:32 +00:00
John Ericson	4eb8c69853	Rename to hand-hold git (derivation-goal.hh)	2020-10-12 17:15:32 +00:00
John Ericson	f7099965bf	Change .cc files to use split build headers	2020-10-12 17:08:52 +00:00
John Ericson	5a97621d6d	Prepare for build/*.hh headers	2020-10-12 17:07:51 +00:00
volth	eee18f88dd	Handle amount of disk space saved by hard linking being negative ... Fixes bogus messages like "currently hard linking saves 17592186044416.00 MiB".	2020-10-12 16:06:38 +00:00
John Ericson	15fdb7cc6b	Split out uds-remote-store.{cc.hh}	2020-10-11 17:37:05 +00:00
John Ericson	38e3897162	Copy {uds-,}remote-store.{cc,hh} ... This prepares for the splitting that happens in the next commit.	2020-10-11 17:18:26 +00:00
John Ericson	5c74a6147b	Properly type the derivation and substitution goal maps ... As a bonus, Worker::removeGoal is less inefficient.	2020-10-11 17:07:14 +00:00
John Ericson	8cc510fb79	Merge branches 'build-1', 'build-2', 'build-3', 'build-4', 'build-5', 'build-6', 'build-7' and 'build-8' into split_build_cc	2020-10-11 16:44:34 +00:00
John Ericson	bcb67e1ed8	Trim lock.cc	2020-10-11 16:44:19 +00:00
John Ericson	e0be04129b	Rename to hand-hold git (lock.cc)	2020-10-11 16:44:14 +00:00
John Ericson	dbc588651c	Trim lock.hh	2020-10-11 16:44:07 +00:00
John Ericson	d0004bfcab	Rename to hand-hold git (lock.hh)	2020-10-11 16:43:59 +00:00
John Ericson	eed53ed87a	Trim build/worker.cc	2020-10-11 16:43:51 +00:00
John Ericson	904e315dae	Rename to hand-hold git (build/worker.cc)	2020-10-11 16:43:42 +00:00
John Ericson	d24ffe0eb1	Trim build/substitution-goal.cc	2020-10-11 16:43:24 +00:00
John Ericson	3633b3572b	Rename to hand-hold git (build/substitution-goal.cc)	2020-10-11 16:43:18 +00:00
John Ericson	4bdff7d1b0	Trim build/local-store-build.cc	2020-10-11 16:43:12 +00:00
John Ericson	dc5225cde5	Rename to hand-hold git (build/local-store-build.cc)	2020-10-11 16:43:05 +00:00
John Ericson	159054f730	Trim build/hook-instance.cc	2020-10-11 16:42:35 +00:00
John Ericson	f0b8987299	Rename to hand-hold git (build/hook-instance.cc)	2020-10-11 16:42:08 +00:00
John Ericson	819fe848ac	Trim build/goal.cc	2020-10-11 16:41:58 +00:00
John Ericson	184bfc301e	Rename to hand-hold git (build/goal.cc)	2020-10-11 16:41:18 +00:00
John Ericson	3bab1c5bb0	Trim build/derivation-goal.cc	2020-10-11 16:41:11 +00:00
John Ericson	9629290eda	Rename to hand-hold git (build/derivation-goal.cc)	2020-10-11 16:40:52 +00:00
John Ericson	a4f0fecb03	Trim build.hh	2020-10-11 16:40:34 +00:00
John Ericson	fc72cb0760	Rename to hand-hold git (build.hh)	2020-10-11 16:40:14 +00:00
John Ericson	428536fd75	Prepare for build/* files	2020-10-11 16:39:08 +00:00
John Ericson	aef44cbaa9	Split out commonChildInit	2020-10-11 16:38:46 +00:00
John Ericson	6cc1541782	Split out local-fs-store.hh ... This matches the already-existing `local-fs-store.cc`.	2020-10-09 20:18:08 +00:00
Eelco Dolstra	636ec17139	Remove stray DerivationOutputsAndPaths type	2020-10-09 15:41:24 +02:00
Eelco Dolstra	97ffc1e013	Dynamically disable user namespaces if CLONE_NEWUSER fails ... This makes builds work inside nixos-enter. Fixes #3145.	2020-10-07 22:46:01 +02:00
Eelco Dolstra	6aa64627c8	Support user namespaces being disabled ... If max_user_namespaces is set to 0, then don't run the build in a user namespace. Fixes #4092.	2020-10-07 22:02:36 +02:00
Eelco Dolstra	f66bbd8c7b	Doh	2020-10-07 21:25:06 +02:00
Eelco Dolstra	e705c24294	Tweak error messages	2020-10-07 17:28:43 +02:00
Eelco Dolstra	c43e882f54	Serialize exceptions from the daemon to the client	2020-10-07 17:13:54 +02:00
Eelco Dolstra	be149acfda	Serialize exceptions from the sandbox process to the parent ... Fixes #4118.	2020-10-07 16:34:03 +02:00
John Ericson	57d960dcd1	Remove generic std::optional<T> suppport from worker proto ... See comment for rational; I think it's good to leave a comment lest anyone is tempted to add such a sum-type instance again. Fixes #4113	2020-10-07 12:50:37 +00:00
Eelco Dolstra	d761485010	Prevent a deadlock when user namespace setup fails ... Observed on Centos 7 when user namespaces are disabled: DerivationGoal::startBuilder() throws an exception, ~DerivationGoal() waits for the child process to exit, but the child process hangs forever in drainFD(userNamespaceSync.readSide.get()) in DerivationGoal::runChild(). Not sure why the SIGKILL doesn't get through. Issue #4092.	2020-10-06 18:57:23 +02:00
Eelco Dolstra	ad143c5b3b	Shut up some clang warnings	2020-10-06 14:52:30 +02:00
Eelco Dolstra	85c8be6286	Remove static variable name clashes ... This was useful for an experiment with building Nix as a single compilation unit. It's not very useful otherwise but also doesn't hurt...	2020-10-06 13:49:20 +02:00
Eelco Dolstra	6691256e79	Factor out common showBytes()	2020-10-06 10:40:49 +02:00
Eelco Dolstra	d0bb544128	Add missing #pragma once	2020-10-06 10:40:07 +02:00
Eelco Dolstra	88a667e49e	Fix s3:// store ... Fixes https://github.com/NixOS/nixos-org-configurations/issues/123.	2020-10-05 17:53:30 +02:00
Eelco Dolstra	51c299213b	Merge pull request #3935 from obsidiansystems/binary-cache-addToStoreFromDump ... Get rid of Hash::dummy from BinaryCacheStore	2020-10-05 14:41:28 +02:00
Eelco Dolstra	f3aba88737	Merge pull request #3895 from obsidiansystems/templated-daemon-protocol ... More templated STL support for the daemon protocol	2020-10-05 14:40:27 +02:00
Eelco Dolstra	20a1e20d91	Style	2020-09-30 13:35:56 +02:00
John Ericson	69afaeace3	Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol	2020-09-30 00:42:28 +00:00
John Ericson	b759701652	nix::worker_proto -> worker_proto	2020-09-30 00:41:18 +00:00
John Ericson	45a0ed82f0	Revert "Use template structs instead of phantoms" ... This reverts commit 9ab07e99f5.	2020-09-30 00:39:06 +00:00
Kevin Quick	5e7838512e	Remove github-access-token in favor of access-token.	2020-09-29 16:26:34 -07:00
Kevin Quick	66c3959e8c	Merge branch 'master' into access-tokens	2020-09-29 08:32:06 -07:00
Eelco Dolstra	5999978a05	Make Headers an optional argument	2020-09-29 13:05:19 +02:00
Eelco Dolstra	cebd2fc35d	Merge branch 'github-api-token' of https://github.com/imalsogreg/nix	2020-09-29 12:17:00 +02:00
John Ericson	00135e13f4	Clarify comment a bit	2020-09-28 18:19:10 +00:00
John Ericson	80e335bb58	Use drvPath2 and give it a better name	2020-09-28 15:43:56 +00:00
John Ericson	10202bbf29	Merge remote-tracking branch 'upstream/master' into ca-floating-upstream	2020-09-28 15:39:11 +00:00
John Ericson	6c31297d80	Update src/libstore/binary-cache-store.cc ... Co-authored-by: Eelco Dolstra <edolstra@gmail.com>	2020-09-28 11:32:58 -04:00
Eelco Dolstra	649c465873	Merge pull request #4064 from serokell/balsoft/fix-max-jobs ... Fix max-jobs option	2020-09-28 10:42:02 +02:00
John Ericson	25fffdda86	Remove redundant nar hash and size setting ... Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>	2020-09-26 10:17:30 -04:00
John Ericson	1832436526	Fix up BinaryCacheStore::addToStore taking a path	2020-09-26 04:56:29 +00:00
John Ericson	5db83dd771	BinaryCacheStore::addTextToStore include CA field	2020-09-26 03:21:36 +00:00
John Ericson	c40c832f19	Merge remote-tracking branch 'obsidian/master' into binary-cache-addToStoreFromDump	2020-09-26 00:21:45 +00:00
Gregory Hale	faa5607f54	Merge remote-tracking branch 'origin/master' into github-api-token	2020-09-25 12:10:58 -04:00
Eelco Dolstra	7d81582488	Merge pull request #3953 from obsidiansystems/basic-derivation-goal-outputs ... Deduplicate basic derivation goals too	2020-09-25 17:21:19 +02:00
Kevin Quick	5a35cc29bf	Re-add support for github-access-token, but mark as deprecated.	2020-09-25 08:09:56 -07:00
Kevin Quick	ef2a14be19	Fix reference to older name for access-tokens config value.	2020-09-25 08:08:27 -07:00
Kevin Quick	8fba2a8b54	Update to use access-tokens configuration for github/gitlab access. ... This change provides support for using access tokens with other instances of GitHub and GitLab beyond just github.com and gitlab.com (especially company-specific or foundation-specific instances). This change also provides the ability to specify the type of access token being used, where different types may have different handling, based on the forge type.	2020-09-24 22:49:44 -07:00
Alexander Bantyev	ed218e1d6c	Fix max-jobs option ... After 0ed946aa61, max-jobs setting (-j/--max-jobs) stopped working. The reason was that nrLocalBuilds (which compared to maxBuildJobs to figure out whether the limit is reached or not) is not incremented yet when tryBuild is started; So, the solution is to move the check to tryLocalBuild. Closes https://github.com/nixos/nix/issues/3763	2020-09-25 00:07:42 +03:00
Eelco Dolstra	4ce8a3ed45	Hopefully fix EPERM on macOS	2020-09-23 21:29:10 +02:00
Eelco Dolstra	9a24ece122	Fix exception	2020-09-23 20:21:08 +02:00
Eelco Dolstra	236d9ee7f7	lstat() cleanup	2020-09-23 19:17:28 +02:00
Eelco Dolstra	688bd4fb50	After rewriting a path, make it read-only	2020-09-23 19:10:16 +02:00
Eelco Dolstra	cec9473871	DerivationGoal::registerOutputs(): Don't canonicalize twice ... Fixes #4021.	2020-09-23 18:21:22 +02:00
Eelco Dolstra	d4f8163d10	canonicalisePathMetaData_(): Change assertion to error message	2020-09-23 18:21:22 +02:00
Eelco Dolstra	e8f0b1e996	DerivationGoal::registerOutputs(): Fix bad format string	2020-09-23 18:21:22 +02:00
John Ericson	3f226f71c1	Return more info from BinaryCacheStore::addToStoreCommon ... We don't need it yet, but we could/should in the future, and it's a cost-free change since we already have the reference. I like it. Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>	2020-09-23 14:40:41 +00:00
John Ericson	412b3a54fb	Clarify FIXME in BinaryCacheStore::addToStoreCommon ... Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>	2020-09-23 10:36:55 -04:00
John Ericson	9fbc31a65b	Get rid of Hash::dummy from BinaryCacheStore	2020-09-23 04:56:04 +00:00
John Ericson	993229cdaf	Deduplicate basic derivation goals too ... See comments for security concerns. Also optimize goal creation by not traversing map twice.	2020-09-22 17:13:59 +00:00
John Ericson	e9fc2031f0	Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol	2020-09-22 14:18:31 +00:00
regnat	67cc356bb3	Merge remote-tracking branch 'origin/master' into non-ca-depending-on-ca	2020-09-22 16:11:25 +02:00
Eelco Dolstra	980edd1f3a	RemoteStore::addCAToStore(): Don't hold connection while calling queryPathInfo() ... This leads to a deadlock if we're at the connection limit.	2020-09-22 15:28:20 +02:00
Eelco Dolstra	5b107f2c5f	Merge pull request #4038 from maljub01/master ... Add a nix.conf option for allowing a symlinked store	2020-09-22 13:19:22 +02:00
Eelco Dolstra	35a0ac1838	Style fixes	2020-09-22 11:40:19 +02:00
Eelco Dolstra	92ac8df0ec	Merge branch 'add-ca-to-store' of https://github.com/hercules-ci/nix	2020-09-22 11:31:33 +02:00
John Ericson	3786a801c3	Merge remote-tracking branch 'upstream/master' into ca-floating-upstream	2020-09-22 04:15:55 +00:00
John Ericson	b92d3b2edd	Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol	2020-09-22 00:45:55 +00:00
Cole Helbling	ba37299a03	Serialize SandboxMode enum to string for JSON ... Rather than showing an integer as the default, instead show the boolean referenced in the description. The nix.conf.5 manpage used to show "default: 0", which is unnecessarily opaque and confusing (doesn't 0 mean false, even though the default is true?); now it properly shows that the default is true.	2020-09-21 10:36:45 -07:00
Eelco Dolstra	0716adaa8b	abstractsettingtojson.hh -> abstract-setting-to-json.hh	2020-09-21 18:49:43 +02:00
Eelco Dolstra	340ca382c4	Don't include nlohmann/json.hpp in globals.hh ... This reduces compilation time by 207s. Issue #4045.	2020-09-21 18:47:18 +02:00
Eelco Dolstra	d51ba43047	Move Callback into its own header ... This gets rid of the inclusion of <future> in util.hh, cutting compilation time by ~20s (CPU time). Issue #4045.	2020-09-21 18:42:21 +02:00
Marwan Aljubeh	f80ffeb8c9	Update the variable name accordingly	2020-09-21 17:29:08 +01:00
Eelco Dolstra	e8e1d420f3	Don't include <regex> in header files ... This reduces compilation time by ~15 seconds (CPU time). Issue #4045.	2020-09-21 18:22:45 +02:00
Marwan Aljubeh	4e1a04733d	Use a better name for the config option	2020-09-21 16:32:22 +01:00
Robert Hensing	ca30abb3fb	Document addCAToStore/addToStoreFromDump source drainage ... Also checked that all usages satisfy the requirement and removed dead code.	2020-09-21 07:55:47 +02:00
Robert Hensing	fa08db5c4c	wopAddToStore: return ValidPathInfo ... A ValidPathInfo is created anyway. By returning it we can save a roundtrip and we have a nicer interface.	2020-09-21 07:55:47 +02:00
Robert Hensing	7c68264085	wopAddToStore: add RepairFlag	2020-09-21 07:55:47 +02:00
Robert Hensing	fbf509c113	parseContentAddressMethodPrefix: use string_view ... Co-authored-by: John Ericson <git@JohnEricson.me>	2020-09-21 07:55:47 +02:00
Robert Hensing	8279178b07	Move FramedSink next to FramedSource	2020-09-21 07:55:47 +02:00
Robert Hensing	ecc8088cb7	wopAddToStore: Throw to clarify unused refs ... Co-authored-by: John Ericson <git@JohnEricson.me>	2020-09-21 07:55:47 +02:00
Robert Hensing	c602ebfb34	Refactor wopAddToStore to make wopAddTextToStore obsolete	2020-09-21 07:55:45 +02:00
Robert Hensing	e34fe47d0c	Overhaul wopAddToStore	2020-09-21 07:54:05 +02:00
Marwan Aljubeh	e40772cd35	Lint issue: replacing tabs with spaces	2020-09-18 17:18:45 +01:00
Marwan Aljubeh	c00e078343	Add a nix.conf option for allowing a symlinked store	2020-09-18 17:10:39 +01:00
Robert Hensing	14b30b3f3d	Move FramedSource and FramedSink, extract withFramedSink	2020-09-17 20:21:04 +02:00
Robert Hensing	dfa547c6a8	Add ContentAddressMethod and parse/render it	2020-09-17 20:21:04 +02:00
Robert Hensing	9ee3122ec7	Remove redundant import	2020-09-17 20:21:04 +02:00
John Ericson	b7df353f27	Merge remote-tracking branch 'upstream/master' into ca-floating-upstream	2020-09-17 16:33:10 +00:00
Eelco Dolstra	649d3aaf24	Merge pull request #3829 from obsidiansystems/remove-storetype-delegate-regStore ... Remove storetype delegate reg store -- contains #3736	2020-09-17 13:55:01 +02:00
Eelco Dolstra	b94a35ef40	Merge pull request #4027 from tweag/fix-gc-of-ca-derivations ... Fix garbage collection of CA derivations	2020-09-17 13:46:26 +02:00
regnat	520895b1da	Fix garbage collection of CA derivations ... Fix #4026	2020-09-17 13:36:58 +02:00
John Ericson	f60b380a7f	Merge remote-tracking branch 'upstream/master' into remove-storetype-delegate-regStore	2020-09-16 22:35:24 +00:00
John Ericson	2741fffa35	Ensure resolved CA derivations are written ... so we can link outputs to deriver and thus properly cache.	2020-09-16 17:57:32 +00:00
John Ericson	7fdbb377ba	Start to fix floating CA + remote building	2020-09-16 17:50:48 +00:00
John Ericson	c5ccebae00	Merge remote-tracking branch 'upstream/master' into ca-floating-upstream	2020-09-16 17:50:40 +00:00
Greg Hale	a303c0b6dc	Fetch commits from github/gitlab using Auth header ... `nix flake info` calls the github 'commits' API, which requires authorization when the repository is private. Currently this request fails with a 404. This commit adds an authorization header when calling the 'commits' API. It also changes the way that the 'tarball' API authenticates, moving the user's token from a query parameter into the Authorization header. The query parameter method is recently deprecated and will be disallowed in November 2020. Using them today triggers a warning email.	2020-09-16 13:46:48 -04:00
Théophane Hufschmitt	77a0e2c5be	Remove useless exception copy ... Co-authored-by: Eelco Dolstra <edolstra@gmail.com>	2020-09-16 14:00:21 +02:00
regnat	d72927aa7a	Fix the s3 store ... Add some necessary casts in the initialisation of the store's config	2020-09-16 13:53:09 +02:00
regnat	e0817cbcdc	Don't include nlohmann/json.hpp in config.hh ... Instead make a separate header with the template implementation of `BaseSetting<T>::toJSONObj` that can be included where needed	2020-09-16 13:53:09 +02:00
regnat	fc2d31c423	Add (StoreConfig*) casts to work around a GCC bug ... Work around https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80431 that was already there in the code but was accidentally removed in the last commits	2020-09-16 13:53:09 +02:00
regnat	888f7afe9f	Fix build issues with gcc	2020-09-16 13:53:09 +02:00
regnat	b73adacc1e	Add a name to the stores ... So that it can be printed by `nix describe-stores`	2020-09-16 13:53:09 +02:00
regnat	f24f0888f9	Document the new store hierarchy	2020-09-16 13:53:09 +02:00
regnat	29a632386e	fixup! Make the store plugins more introspectable	2020-09-16 13:53:09 +02:00
regnat	1129913c4e	fixup! Correctly call all the parent contructors of the stores	2020-09-16 13:53:09 +02:00
regnat	7f103dcddd	Properly filter the stores according to their declared uriSchemes ... When opening a store, only try the stores whose `uriSchemes()` include the current one	2020-09-16 13:53:09 +02:00
regnat	5895184df4	Correctly call all the parent contructors of the stores ... Using virtual inheritance means that only the default constructors of the parent classes will be called, which isn't what we want	2020-09-16 13:53:09 +02:00
regnat	22afa8fb4d	Separate store configs from the implems ... Rework the `Store` hierarchy so that there's now one hierarchy for the store configs and one for the implementations (where each implementation extends the corresponding config). So a class hierarchy like ``` StoreConfig-------->Store | | v v SubStoreConfig----->SubStore | | v v SubSubStoreConfig-->SubSubStore ``` (with virtual inheritance to prevent DDD). The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself	2020-09-16 13:53:08 +02:00
regnat	aa4eac3788	fixup! Separate the instantiation and initialisation of the stores	2020-09-16 13:53:08 +02:00
regnat	3c525d1590	Complete the toJSON instance for Setting<T> ... Don't let it just contain the value, but also the other fields of the setting (description, aliases, etc..)	2020-09-16 13:53:08 +02:00
regnat	3b57181f8e	Separate the instantiation and initialisation of the stores ... Add a new `init()` method to the `Store` class that is supposed to handle all the effectful initialisation needed to set-up the store. The constructor should remain side-effect free and just initialize the c++ data structure. The goal behind that is that we can create “dummy” instances of each store to query static properties about it (the parameters it accepts for example)	2020-09-16 13:53:08 +02:00
regnat	fa32560169	Fix the registration of stores	2020-09-16 13:53:08 +02:00
regnat	7d5bdf8b56	Make the store plugins more introspectable ... Directly register the store classes rather than a function to build an instance of them. This gives the possibility to introspect static members of the class or choose different ways of instantiating them.	2020-09-16 13:53:08 +02:00
John Ericson	3a5cdd737c	Rename Derivation::pathOpt to Derivation::path ... We no longer need the `*Opt` to disambiguate.	2020-09-15 15:21:39 +00:00
John Ericson	6387550d58	Get rid of confusing std::optional<bool> for validity	2020-09-15 15:19:45 +00:00
John Ericson	c4bf219b55	Don't link deriver until after any delayed exception is thrown ... Otherwise, we will associate fixed-output derivations with outputs that they did indeed produce, but which had the wrong hash. That's no good.	2020-09-15 14:28:06 +00:00
John Ericson	3ba552b245	Merge remote-tracking branch 'upstream/master' into single-ca-drv-build	2020-09-15 14:17:06 +00:00
John Ericson	c08c9f08c7	Merge remote-tracking branch 'upstream/master' into remove-storetype-delegate-regStore	2020-09-15 14:08:35 +00:00
regnat	057c6203b5	gracefully handle old daemon versions ... Add a fallback path in `queryPartialDerivationOutputMap` for daemons that don't support it. Also upstreams a couple methods from `SSHStore` to `RemoteStore` as this is needed to handle the fallback path.	2020-09-15 09:58:01 +02:00
Eelco Dolstra	a59e77d9e5	nix-daemon: Lower verbosity of restricted setting warning ... Fixes #3992.	2020-09-14 13:48:51 +02:00
aszlig	525b38eee8	Fix unspecified behaviour in readStorePathCAMap ... When deploying a Hydra instance with current Nix master, most builds would not run because of errors like this: queue monitor: error: --- Error --- hydra-queue-runner error: --- UsageError --- nix-daemon not a content address because it is not in the form '<prefix>:<rest>': /nix/store/...-somedrv The last error message is from parseContentAddress, which expects a colon-separated string, however what we got here is a store path. Looking at the worker protocol, the following message sent to the Nix daemon caused the error above: 0x1E -> wopQuerySubstitutablePathInfos 0x01 -> Number of paths 0x16 -> Length of string "/nix/store/...-somedrv" 0x00 -> Length of string "" Looking at writeStorePathCAMap, the store path is indeed the first field that's transmitted. However, readStorePathCAMap expects it to be the *second* field *on my machine*, since expression evaluation order is a classic form of unspecified behaviour[1] in C++. This has been introduced in https://github.com/NixOS/nix/pull/3689, specifically in commit 66a62b3189. [1]: https://en.wikipedia.org/wiki/Unspecified_behavior#Order_of_evaluation_of_subexpressions Signed-off-by: aszlig <aszlig@nix.build>	2020-09-13 02:40:51 +02:00
John Ericson	98dfd7531d	Fix querying outputs for CA derivations some more ... If we resolve using the known path of a derivation whose output we didn't have, we previously blew up. Now we just fail gracefully, returning the map of all outputs unknown.	2020-09-04 18:33:58 +00:00
John Ericson	075d399e3f	Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream	2020-09-04 16:04:35 +00:00
John Ericson	e9fad3006b	Fix some of the issues raised by @edolstra ... - More and better comments - The easier renames	2020-09-04 15:18:25 +00:00
John Ericson	e86dd59dcc	Apply suggestions from code review ... Thanks! Co-authored-by: Eelco Dolstra <edolstra@gmail.com>	2020-09-04 10:48:50 -04:00
John Ericson	c9f1ed912c	Don't chmod symlink before moving outputs around ... Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>	2020-09-04 14:41:53 +00:00
John Ericson	aad4abcc9c	Fix floating CA tests ... We will sometimes try to query the outputs of derivations we can't resolve. That's fine; it just means we don't know what those outputs are yet.	2020-09-04 01:17:38 +00:00
John Ericson	975a47f7fe	Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream	2020-09-03 22:09:04 +00:00
John Ericson	e7d93e7ece	Merge remote-tracking branch 'upstream/master' into single-ca-drv-build	2020-09-03 15:43:17 +00:00
Eelco Dolstra	8a945d6ddb	Remove gc-check-reachability	2020-09-03 11:30:15 +02:00
Eelco Dolstra	82b77a7726	addPermRoot(): Remove indirect flag	2020-09-03 11:29:34 +02:00
Eelco Dolstra	b07167be5a	createGeneration(): Always create an indirect root ... This means profiles outside of /nix/var/nix/profiles don't get garbage-collected. It also means we don't need to scan /nix/var/nix/profiles for GC roots anymore, except for compatibility with previously existing generations.	2020-09-03 11:13:53 +02:00
Eelco Dolstra	b74f5cdd23	createGeneration(): Take a StorePath	2020-09-03 11:06:56 +02:00
Domen Kožar	dd4b56c87f	Allow HTTP binary cache to request absolute uris	2020-09-01 21:35:48 +02:00
John Ericson	ef278d00f9	Merge remote-tracking branch 'upstream/master' into single-ca-drv-build	2020-09-01 18:01:48 +00:00
Eelco Dolstra	84f5cabbea	Merge remote-tracking branch 'origin/master' into markdown	2020-08-31 14:24:26 +02:00
John Ericson	02e0001fc0	Merge branch 'ca-floating-upstream' of github.com:obsidiansystems/nix into ca-floating-upstream	2020-08-28 21:23:09 +00:00
John Ericson	421ed527c7	Update src/libstore/build.cc ... Thanks for catching, @regnat.	2020-08-28 17:22:57 -04:00
John Ericson	48e8828050	Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream	2020-08-28 20:38:25 +00:00
John Ericson	8017fe7487	Merge remote-tracking branch 'upstream/master' into single-ca-drv-build	2020-08-28 19:59:14 +00:00
Eelco Dolstra	eb75282b8d	Merge pull request #3434 from Ericson2314/derivation-header-include-order ... Revise division of labor in deserialization of derivations	2020-08-27 16:39:28 +02:00
Eelco Dolstra	a0f19d9f3a	RemoteStore::addToStore(): Fix race between stderrThread and NAR writer ... As pointed out by @B4dM4n, the call to to.flush() on stderrThread is unsafe because the NAR writer thread is also writing to 'to'. Fixes #3943.	2020-08-27 14:50:51 +02:00
Eelco Dolstra	4bf5faf416	Merge remote-tracking branch 'origin/master' into markdown	2020-08-25 19:47:34 +02:00
John Ericson	8eb73a8724	CA derivations that depend on other CA derivations ... Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>	2020-08-24 19:01:15 +00:00
John Ericson	e0b0e18905	Add constructor for BasicDerivation -> Derivation	2020-08-24 19:01:07 +00:00
John Ericson	59979e7053	Fix bad debug format string	2020-08-24 19:01:07 +00:00
Eelco Dolstra	6a67e57019	Add DummyStore (dummy://) ... DummyStore does not allow building or adding paths. This is useful for evaluation tests when you don't want to initialize a "proper" store.	2020-08-24 18:54:16 +02:00
Eelco Dolstra	2530cc6827	Merge pull request #3950 from obsidiansystems/tabs-spaces ... tabs -> spaces	2020-08-24 10:20:28 +02:00
John Ericson	b0b59fd05a	Merge remote-tracking branch 'obsidian/write-derivation-borrow' into HEAD	2020-08-23 15:11:10 +00:00
John Ericson	35e6288be1	writeDerivation just needs a plain store reference	2020-08-23 15:01:11 +00:00
John Ericson	3a7b330b64	"Downstream placeholders" should not be store paths ... Insead they should be opaque `/<hash>` like the placeholders we already have.	2020-08-21 19:35:35 +00:00
John Ericson	422affe102	tabs -> spaces ... Sorry I let the tab sneak in there in the first place.	2020-08-21 19:19:24 +00:00
John Ericson	c265e0ea82	Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol	2020-08-20 20:24:22 +00:00
John Ericson	27a3f82c0b	Merge remote-tracking branch 'upstream/master' into single-ca-drv-build	2020-08-20 18:28:17 +00:00
John Ericson	45a2f1baab	Rename drv output querying functions, like master ... - `queryDerivationOutputMapAssumeTotal` -> `queryPartialDerivationOutputMap` - `queryDerivationOutputMapAssumeTotal` -> `queryDerivationOutputMap	2020-08-20 18:14:12 +00:00
Eelco Dolstra	4d77513d97	Merge pull request #3859 from obsidiansystems/drv-outputs-map-allow-missing ... `queryDerivationOutputMap` no longer assumes all outputs have a mapping	2020-08-20 16:49:23 +02:00
John Ericson	9a9d834dc7	Rename drv output querying functions ... - `queryDerivationOutputMapAssumeTotal` -> `queryPartialDerivationOutputMap` - `queryDerivationOutputMapAssumeTotal` -> `queryDerivationOutputMap`	2020-08-20 14:12:51 +00:00
Eelco Dolstra	acb99f03f9	Config: Use nlohmann/json	2020-08-20 11:02:16 +02:00
John Ericson	a83694c7a1	Use RemoteStore to open connection for proxying daemon ... Removes duplicate websocket opening code, and also means we should be able to to ssh-ssh-... daemon relays, not just uds-uds-... ones.	2020-08-19 19:34:47 +00:00
Eelco Dolstra	c8fa39324a	Generate the nix.conf docs from the source code ... This means we don't have two (divergent) sets of option descriptions anymore.	2020-08-19 18:28:04 +02:00
John Ericson	d5af5763cf	Merge branch 'master' of github.com:NixOS/nix into remove-storetype-delegate-regStore	2020-08-19 15:54:50 +00:00
John Ericson	be0d429b95	Merge branch 'master' of github.com:NixOS/nix into templated-daemon-protocol	2020-08-19 03:17:41 +00:00
John Ericson	950ddfdb82	Merge remote-tracking branch 'upstream/master' into derivation-header-include-order	2020-08-18 14:36:44 +00:00
Eelco Dolstra	dfeb76dbf9	Merge pull request #3930 from obsidiansystems/legacy-ssh-build-paths ... Define `LegacySSHStore::buildPaths` using `cmdBuildPaths`	2020-08-18 16:07:40 +02:00
Eelco Dolstra	7cdc739ece	Merge remote-tracking branch 'origin/master' into markdown	2020-08-17 13:43:39 +02:00
Eelco Dolstra	847a5392f4	Merge branch 'ca-no-need-trust' of https://github.com/obsidiansystems/nix	2020-08-17 10:57:02 +02:00
Eelco Dolstra	3c619f6290	Merge branch 'test-RemoteStore-buildDerivation' of https://github.com/obsidiansystems/nix	2020-08-17 10:53:18 +02:00
John Ericson	6f7ac5e865	Remove extra closing paren	2020-08-14 21:59:31 +00:00
John Ericson	3c8b5b6219	Merge remote-tracking branch 'upstream/master' into single-ca-drv-build	2020-08-14 17:00:13 +00:00
John Ericson	4b571ea321	Update src/libstore/daemon.cc ... Co-authored-by: Eelco Dolstra <edolstra@gmail.com>	2020-08-14 11:52:37 -04:00
Eelco Dolstra	13e49be660	Merge pull request #3875 from obsidiansystems/new-interface-for-path-pathOpt ... Offer a safer interface for path and pathOpt	2020-08-14 17:19:19 +02:00
Eelco Dolstra	7714d9a943	Merge pull request #3924 from obsidiansystems/features-per-store ... Make `system-features` a store setting	2020-08-14 17:13:07 +02:00
Eelco Dolstra	9b9d5297ba	Merge pull request #3909 from matthewbauer/readd-hashed-mirrors ... Add hashed-mirrors back	2020-08-14 17:05:47 +02:00
Eelco Dolstra	d81f13f7cb	Merge pull request #3899 from obsidiansystems/make-narHash-not-optional ... Make narHash in ValidPathInfo not optional	2020-08-14 17:00:18 +02:00
John Ericson	e1308b1211	Define LegacySSHStore::buildPaths using cmdBuildPaths ... Evidentally this was never implemented because Nix switched to using `buildDerivation` exclusively before `build-remote.pl` was rewritten. The `nix-copy-ssh` test (already) tests this.	2020-08-13 21:27:55 +00:00
John Ericson	5ccd94501d	Allow trustless building of CA derivations ... Include a long comment explaining the policy. Perhaps this can be moved to the manual at some point in the future. Also bump the daemon protocol minor version, so clients can tell whether `wopBuildDerivation` supports trustless CA derivation building. I hope to take advantage of this in a follow-up PR to support trustless remote building with the minimal sending of derivation closures.	2020-08-13 18:15:57 +00:00
John Ericson	85aacbee64	Use TeeSink and TeeSouce in a few more places	2020-08-13 14:51:17 +00:00
John Ericson	d2f2be0f70	Test RemoteStore::buildDerivation ... Fix `wopNarFromPath` which needed a `toRealPath`.	2020-08-13 04:07:14 +00:00
John Ericson	5d67f18c86	Merge branch 'daemon-auth-cleanup' of github.com:obsidiansystems/nix into HEAD	2020-08-12 18:22:31 +00:00
John Ericson	4720853129	Make system-features a store setting ... This seems more correct. It also means one can specify the features a store should support with --store and remote-store=..., which is useful. I use this to clean up the build remotes test.	2020-08-12 18:13:00 +00:00
John Ericson	8d4162ff9e	Separate auth and logic for the daemon ... Before, processConnection wanted to know a user name and user id, and `nix-daemon --stdio`, when it isn't proxying to an underlying daemon, would just assume "root" and 0. But `nix-daemon --stdio` (no proxying) shouldn't make guesses about who holds the other end of its standard streams. Now processConnection takes an "auth hook", so `nix-daemon` can provide the appropriate policy and daemon.cc doesn't need to know or care what it is.	2020-08-12 15:22:33 +00:00
John Ericson	5f80aea795	Break out lambda so output can be matched just once ... This is much better.	2020-08-12 02:23:31 +00:00
John Ericson	18834f7764	Recheck path validity after acquiring lock ... It might have changed, and in any event this is how the cod used to work so let's just keep it.	2020-08-11 23:44:02 +00:00
John Ericson	2de201254e	Don't assume a total output map in two places in build.cc ... Thanks @regnat for catching one of them. The other follows for many of the same reasons. I'm find fixing others on a need-to-fix basis, provided their are no regressions.	2020-08-11 23:07:50 +00:00
John Ericson	4c6aac8fdf	Clarify comment on sandbox and temp fresh paths	2020-08-11 22:46:05 +00:00
John Ericson	6d57139050	Clarify outputReferences variable with self-describing type ... Thanks for the idea, @Regnat!	2020-08-11 22:34:09 +00:00
John Ericson	8a068bd025	Remove redundant equality check	2020-08-11 21:25:40 +00:00
John Ericson	07e3466eb4	Float comment to out describe gaveUpOnSubstitution in general	2020-08-11 21:16:14 +00:00
John Ericson	d0f6e338dd	Apply suggestions from code review ... Thanks!! Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>	2020-08-11 16:49:10 -04:00
John Ericson	2a0902634e	Fix error in merge breaking floating CA drvs ... Forgot to add this hunk!	2020-08-11 00:13:19 +00:00
John Ericson	1b5c24662b	Merge branch 'small-drv-serialize-cleanup' of github.com:obsidiansystems/nix into single-ca-drv-build	2020-08-10 01:57:54 +00:00
John Ericson	581183d4d5	Deduplicate parsing and reading derivations	2020-08-10 01:40:50 +00:00
John Ericson	bcd0629c2e	Remove name parameter from writeDerivation ... The name is now stored with the derivation itself.	2020-08-10 01:35:59 +00:00
John Ericson	e913a2989f	Squashed get CA derivations building	2020-08-07 19:51:55 +00:00
Eelco Dolstra	edfd676e05	Fix .ls file names in binary caches ... These are not supposed to include the 'name' part of the store path. This was broken by 759947bf72.	2020-08-07 21:18:29 +02:00
John Ericson	8f92bb5ad9	Merge branch 'drv-outputs-map-allow-missing' of github.com:obsidiansystems/nix into templated-daemon-protocol	2020-08-07 18:51:01 +00:00
John Ericson	f7ba16f9cb	Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing	2020-08-07 17:46:39 +00:00
John Ericson	47644e49ca	Specialize std::optional<StorePath> so this is backwards compatible ... While I am cautious to break parametricity, I think it's OK in this cases---we're not about to try to do some crazy polymorphic protocol anytime soon.	2020-08-07 17:05:14 +00:00
Matthew Bauer	96c158d6e1	Fix build	2020-08-06 21:04:31 -05:00
Matthew Bauer	8abc577cc2	Merge remote-tracking branch 'origin/master' into readd-hashed-mirrors	2020-08-06 18:58:59 -05:00
Carlo Nucera	46f9dd56da	Fix bug due to non-deterministic arg eval order	2020-08-06 19:30:05 -04:00
Matthew Bauer	641c950701	Add hashed-mirrors back ... Some users have their own hashed-mirrors setup, that is used to mirror things in addition to what’s available on tarballs.nixos.org. Although this should be feasable to do with a Binary Cache, it’s not always easy, since you have to remember what "name" each of the tarballs has. Continuing to support hashed-mirrors is cheap, so it’s best to leave support in Nix. Note that NIX_HASHED_MIRRORS is also supported in Nixpkgs through fetchurl.nix. Note that this excludes tarballs.nixos.org from the default, as in \#3689. All of these are available on cache.nixos.org.	2020-08-06 18:19:09 -05:00
Carlo Nucera	9ab07e99f5	Use template structs instead of phantoms	2020-08-06 18:04:13 -04:00
Carlo Nucera	3d8240c32e	Remove leftover commented code	2020-08-06 16:04:18 -04:00
Carlo Nucera	f795f0fabc	Merge branch 'drv-outputs-map-allow-missing-namespace' of github.com:obsidiansystems/nix into templated-daemon-protocol	2020-08-06 15:53:09 -04:00
John Ericson	e89b5bd0bf	Minimize the usage of Hash::dummy	2020-08-06 18:31:48 +00:00
John Ericson	5e59b25a23	Merge remote-tracking branch 'upstream/master' into make-narHash-not-optional	2020-08-06 15:34:15 +00:00
Carlo Nucera	8b175f58d1	Simplify the namespace	2020-08-05 17:57:07 -04:00
Carlo Nucera	0739d428e0	Solve template deduction problem ... We had to predeclare our template functions	2020-08-05 17:49:45 -04:00
John Ericson	6c66331d5c	WIP: Put the worker protocol read and write in a namespace to disambig	2020-08-05 20:37:48 +00:00
Carlo Nucera	1d2e80ddd6	Merge branch 'master' of github.com:NixOS/nix into new-interface-for-path-pathOpt	2020-08-05 15:45:33 -04:00
John Ericson	ed96e603e1	Proxy -> Phantom to match Rust ... Sorry, Haskell.	2020-08-05 19:44:08 +00:00
Carlo Nucera	8241e660ba	Remove Hash::operator bool () ... Since the hash is not optional anymore	2020-08-05 15:30:38 -04:00
Carlo Nucera	be6e1c6457	Merge branch 'master' of github.com:NixOS/nix into make-narHash-not-optional	2020-08-05 15:14:47 -04:00
Carlo Nucera	1ad6394b33	Add Hash::dummy to signal default value ... We did this in the same spirit of the dummy value that's present in libstore/path.hh	2020-08-05 15:11:49 -04:00
Carlo Nucera	1d71028f4d	Remove optionality in ValidPathInfo::narInfo	2020-08-05 14:42:48 -04:00
John Ericson	7302761f64	Merge remote-tracking branch 'obsidian/drv-outputs-map-allow-missing' into templated-daemon-protocol	2020-08-05 17:53:24 +00:00
Ben Burdette	f1a47a96b6	error messages for issue 2238	2020-08-05 10:58:00 -06:00
John Ericson	cf939055c8	Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing	2020-08-05 16:43:30 +00:00
John Ericson	0559ff3d8b	Merge branch 'master' of github.com:NixOS/nix into derivation-primop-floating-output	2020-08-05 14:56:40 +00:00
John Ericson	92ad550e96	Merge remote-tracking branch 'obsidian/misc-ca' into derivation-primop-floating-output	2020-08-05 14:51:41 +00:00
Eelco Dolstra	790b694be7	Style fix	2020-08-05 16:51:06 +02:00
Eelco Dolstra	e48f944e9d	Merge branch 'misc-ca' of https://github.com/obsidiansystems/nix	2020-08-05 16:50:05 +02:00
John Ericson	b9ebe373bb	Sed some names to perhaps avoid conflicts	2020-08-05 14:49:25 +00:00
Eelco Dolstra	25f7912156	Style fix	2020-08-05 16:47:48 +02:00
John Ericson	e561a13a58	Reanme DerivationType::Regular defintion too ... This is the one non-prefixed occurence	2020-08-05 14:45:56 +00:00
John Ericson	e7b0847f2d	Make names more consistent	2020-08-05 14:44:39 +00:00
John Ericson	839f0fe095	Merge remote-tracking branch 'upstream/master' into misc-ca	2020-08-05 14:40:01 +00:00
John Ericson	03f4fafc27	Merge remote-tracking branch 'upstream/master' into misc-ca	2020-08-05 14:36:25 +00:00
Eelco Dolstra	b91dc7ebad	Merge pull request #3730 from obsidiansystems/better-ca-parse-errors ... Improve hash parsing and errors	2020-08-05 16:33:07 +02:00
Eelco Dolstra	75f220a595	Merge pull request #3864 from obsidiansystems/more-topo-sort ... Abstract out topo sorting logic	2020-08-05 16:07:29 +02:00
John Ericson	16c98bf57c	Get rid of some unneeded temporaries	2020-08-04 22:36:31 +00:00
John Ericson	1dfcbebc95	Organize and format code a bit	2020-08-04 22:28:10 +00:00
John Ericson	45b6fdb22b	Remove unused functions	2020-08-04 22:10:13 +00:00
John Ericson	1bab8a321f	Remove unneeded definition ... Template instantiations will cover this case fine.	2020-08-04 21:56:42 +00:00
John Ericson	2f2ae993dc	WIP systematize more of the worker protocol ... This refactor should *not* change the wire protocol.	2020-08-04 19:02:05 +00:00
John Ericson	fbeb8695fb	Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing	2020-08-04 18:35:33 +00:00
John Ericson	d3452a5ed6	Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors	2020-08-04 16:13:20 +00:00
Eelco Dolstra	4e7f1c7f11	S3BinaryCacheStore: Fix size determination	2020-08-04 16:01:13 +02:00
Eelco Dolstra	dfe66420e7	Revert "Remove putBytes" ... This reverts commit b8eea7e81a.	2020-08-04 15:56:10 +02:00
Eelco Dolstra	327b1bf378	BinaryCacheStore: Explicitly flush file sink ... The file sink is also flushed in its destructor, but we ignore any exceptions in the destructor. Issue #3886.	2020-08-04 14:54:07 +02:00
Eelco Dolstra	ed52cf632b	Merge pull request #3892 from obsidiansystems/path-info-header ... Move ValidPathInfo to its own header	2020-08-04 10:44:47 +02:00
John Ericson	062533f7cd	Merge remote-tracking branch 'upstream/master' into path-info-header	2020-08-03 21:02:28 +00:00
Eelco Dolstra	24e07c428f	Delete compressed NARs ... Fixes #3891.	2020-08-03 18:34:52 +02:00
John Ericson	54281f3ac1	addToStore in terms of addToStoreFromDump is not local-store-specific	2020-08-03 04:13:45 +00:00
John Ericson	9357512d73	Merge remote-tracking branch 'upstream/master' into derivation-header-include-order	2020-08-01 19:38:35 +00:00
Carlo Nucera	b6d97fdbf4	Merge branch 'master' of github.com:NixOS/nix into drv-outputs-map-allow-missing	2020-07-31 13:12:51 -04:00
Eelco Dolstra	bf290c2306	Merge remote-tracking branch 'origin/master' into markdown	2020-07-31 16:07:04 +02:00
Eelco Dolstra	1d0a7b54fa	Enable syntax highlighting	2020-07-31 15:43:25 +02:00
Carlo Nucera	eee6ef86cd	Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors	2020-07-30 18:27:25 -04:00
Matthew Bauer	05ac4db39a	Merge remote-tracking branch 'origin/master' into substitute-other-storedir	2020-07-30 12:38:24 -05:00
Eelco Dolstra	3f6e88a552	unsigned long long -> uint64_t	2020-07-30 13:34:04 +02:00
Eelco Dolstra	ebee2b7852	receiveContents(): unsigned int -> size_t	2020-07-30 13:00:30 +02:00
Eelco Dolstra	f63839bfa4	Cleanup	2020-07-29 18:04:51 +02:00
Eelco Dolstra	4c0077a07d	Fix RemoteStore::addToStore() latency ... Since 6185d25e52, this was very latency-bound since it required a round-trip for every 32 KiB. So for example copying a 514 MiB closure over a virtual ethernet device with a articial delay of just 1 ms took 343s. Now it takes 2.7s. Fixes #3372.	2020-07-29 00:48:39 +02:00
Eelco Dolstra	c159f48a39	Cleanup	2020-07-29 00:24:55 +02:00
John Ericson	df707d05d1	Merge branch 'ca-drv' of github.com:Ericson2314/nix into misc-ca	2020-07-28 21:12:36 +00:00
Carlo Nucera	2980b244b7	Use assert(false) instead of abort()	2020-07-28 15:39:45 -04:00
Carlo Nucera	cfe96bd7ce	Merge branch 'master' of github.com:NixOS/nix into ca-drv	2020-07-28 15:36:45 -04:00
Carlo Nucera	c318d398f3	Merge branch 'misc-ca' of github.com:obsidiansystems/nix into new-interface-for-path-pathOpt	2020-07-28 14:22:24 -04:00
Carlo Nucera	7ef1e3cd14	Use the new interface	2020-07-28 13:59:24 -04:00
Carlo Nucera	d564ac1c50	Offer a safer interface for pathOpt ... The new interface we offer provides a way of getting all the DerivationOutputs with the storePaths directly, based on the observation that it's the most common usecase.	2020-07-28 13:55:15 -04:00
Eelco Dolstra	b8eea7e81a	Remove putBytes ... istream->tellg() returns -1 so we can't get the number of bytes written. Fixes 'uploaded 's3://nix-cache/nar/00819r9lp5kajr6baxfw5dhhc0cx8ndxaz43qmd2f0gn1hk1ynlp.nar.xz' (-1 bytes) in 11620 ms' messages.	2020-07-28 13:27:56 +02:00
John Ericson	8065c6d160	Abstract out topo sorting logic	2020-07-27 20:45:34 +00:00
John Ericson	951415b568	Require ca-derivations everywhere we create a CA derivation ... "create" as in read one in from a serialized form, or build one from scratch in memory.	2020-07-27 17:56:36 +00:00
John Ericson	e32a9e124b	Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output	2020-07-27 17:50:06 +00:00
John Ericson	7cf978440c	Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca	2020-07-27 17:47:40 +00:00
John Ericson	e6115ca004	Merge remote-tracking branch 'upstream/master' into ca-derivation-data-types	2020-07-27 17:41:43 +00:00
John Ericson	1d7d94ceea	Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors	2020-07-27 16:17:50 +00:00
John Ericson	43f2bd8dc5	Merge remote-tracking branch 'upstream/master' into hash-always-has-type	2020-07-27 16:13:57 +00:00
John Ericson	387699fe7f	Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca	2020-07-27 15:00:28 +00:00
John Ericson	78466bcb2f	Merge branch 'optional-derivation-output-storepath' into ca-derivation-data-types	2020-07-27 14:40:08 +00:00
John Ericson	d5bb67cfa4	Merge remote-tracking branch 'upstream/master' into optional-derivation-output-storepath	2020-07-27 14:29:32 +00:00
John Ericson	699fc89b39	Merge remote-tracking branch 'upstream/master' into hash-always-has-type	2020-07-27 14:22:45 +00:00
John Ericson	2c7557481b	queryDerivationOutputMap no longer assumes all outputs have a mapping ... This assumption is broken by CA derivations. Making a PR now to do the breaking daemon change as soon as possible (if it is already too late, we can bump protocol intead).	2020-07-24 21:14:06 +00:00
John Ericson	58e55c0923	Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output	2020-07-23 19:17:52 +00:00
John Ericson	a2af25c55a	Merge remote-tracking branch 'upstream/master' into misc-ca	2020-07-23 19:14:17 +00:00
John Ericson	c36916eca0	Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output	2020-07-23 15:06:56 +00:00
John Ericson	6fcd9966bf	Merge remote-tracking branch 'upstream/master' into misc-ca	2020-07-23 15:02:29 +00:00
John Ericson	66a2067288	Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors	2020-07-23 14:56:35 +00:00
John Ericson	2274f63453	Get rid of basicDerivation::findOutput ... It's a tiny function which is: - hardly worth abstrating over, and also only used once. - doesn't work once we get CA drvs I rewrote the one callsite to be forwards compatable with CA derivations, and also potentially more performant: instead of reading in the derivation it can ust consult the SQLite DB in the common case.	2020-07-23 14:39:02 +00:00
John Ericson	9423f64ee2	Parse CA derivations using new output variants ... We no longer need `ParsedDerivation` because everything libstore needs to know about is in the `BasicDerivation` proper.	2020-07-22 23:59:25 +00:00
John Ericson	c56356bacc	Separate concerns in scanForReferences with TeeSink ... This also will make it easier to use a `HashModuloSink` instead for CA derivations.	2020-07-22 22:13:44 +00:00
Eelco Dolstra	ff314f186e	Merge pull request #3736 from obsidiansystems/allow-relative-paths-in-store-option ... Allow relative paths in --store option	2020-07-21 19:43:07 +02:00
Eelco Dolstra	0951330680	Merge pull request #3714 from obsidiansystems/add-body-to-network-errors ... Add response body to network errors	2020-07-21 18:09:42 +02:00
John Ericson	02639716ea	Merge branch 'allow-relative-paths-in-store-option' into remove-storetype-delegate-regStore	2020-07-21 15:55:27 +00:00
John Ericson	6cce32c8e8	Change logic for deciding what is a relative path for the local store ... The was Eelco's prefered logic, and it looks good to me!	2020-07-21 15:39:47 +00:00
Carlo Nucera	922a845ffc	Update chunkSize to the suggested value ... This was a suggested course of action in a review in one of our earlier commits, https://github.com/NixOS/nix/pull/3801#discussion_r457557079	2020-07-21 10:24:19 -04:00
John Ericson	54e507a7aa	Merge remote-tracking branch 'upstream/master' into add-body-to-network-errors	2020-07-21 14:17:59 +00:00
John Ericson	5ce95b9529	Update src/libstore/build.cc	2020-07-21 09:47:40 -04:00
John Ericson	cdb3f39b64	Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors	2020-07-21 13:43:20 +00:00
John Ericson	5055c595bd	Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into ca-derivation-data-types	2020-07-21 01:20:53 +00:00
John Ericson	5a37413796	Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into from-dump-stream	2020-07-21 01:20:14 +00:00
John Ericson	bf0dca7790	Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into misc-ca	2020-07-21 01:18:33 +00:00
Carlo Nucera	9aae179f34	Correct bug, thoroughly document addToStoreSlow	2020-07-20 20:18:12 -04:00
Carlo Nucera	0ca9744694	Use heuristics to decide when to show the response ... Due to https://github.com/NixOS/nix/issues/3841 we don't know how print different messages for different verbosity levels.	2020-07-20 15:57:58 -04:00
Carlo Nucera	6357b1b0fb	Add another Unimplemented case	2020-07-20 14:17:25 -04:00
Carlo Nucera	007bf6d17a	Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca	2020-07-20 14:14:39 -04:00
Carlo Nucera	362ae93851	Add UnimplementedError to ease grepping for these	2020-07-20 14:13:37 -04:00
John Ericson	bf61871271	parser.hh -> split.hh	2020-07-20 17:42:34 +00:00
John Ericson	eab945f591	Merge remote-tracking branch 'upstream/master' into from-dump-stream	2020-07-20 17:29:23 +00:00
Eelco Dolstra	1c5f8bbfb5	Merge pull request #3822 from obsidiansystems/dump-thrice-fixme ... Optimize `addToStoreSlow` and remove `TeeParseSink`	2020-07-20 18:55:05 +02:00
John Ericson	ac2fc7ba1f	Apply suggestions from code review ... Co-authored-by: Eelco Dolstra <edolstra@gmail.com>	2020-07-20 11:29:46 -04:00
Carlo Nucera	0aa79dcc6f	Remove StoreType abstraction and delegate regStore ... to each Store implementation. The generic regStore implementation will only be for the ambiguous shorthands, like "" and "auto". This also could get us close to simplifying the daemon command.	2020-07-17 17:24:02 -04:00
John Ericson	6756cecfcf	Add DerivationType::CAFloating	2020-07-17 19:55:41 +00:00
Carlo Nucera	fefd6c9e5f	Merge branch 'master' of github.com:NixOS/nix into allow-relative-paths-in-store-option	2020-07-17 14:34:20 -04:00
Carlo Nucera	bbc633c98c	Revert "Don't anticipate CA but not fixed outputs for now" ... This reverts commit 3a9e4c3262.	2020-07-17 13:10:32 -04:00
Carlo Nucera	205dcd140d	Revert "Don't anticipate multiple CA outputs for now" ... This reverts commit 74b251b2f3.	2020-07-17 12:43:46 -04:00
Carlo Nucera	1feb8981df	Revert "Don't anticipate hash algo without hash in derivation for now" ... This reverts commit 3804e3df9b.	2020-07-17 12:05:54 -04:00
Carlo Nucera	487c5751c6	Merge branch 'ca-drv' of github.com:Ericson2314/nix into misc-ca	2020-07-17 11:32:59 -04:00
Carlo Nucera	5cb840541b	Merge branch 'multi-output-hashDerivationModulo' of github.com:Ericson2314/nix into misc-ca	2020-07-17 10:28:33 -04:00
Carlo Nucera	745a03cef5	Merge branch 'optional-derivation-output-storepath' of github.com:obsidiansystems/nix into ca-derivation-data-types	2020-07-16 13:36:01 -04:00
Carlo Nucera	048e916f64	Merge branch 'master' of github.com:NixOS/nix into optional-derivation-output-storepath	2020-07-16 13:32:28 -04:00
John Ericson	cc0d77f8c9	Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors	2020-07-16 17:28:52 +00:00
Carlo Nucera	e3a3406db8	Merge branch 'master' of github.com:NixOS/nix into add-body-to-network-errors	2020-07-16 12:04:10 -04:00
John Ericson	5ea817dace	Merge remote-tracking branch 'upstream/master' into hash-always-has-type	2020-07-16 14:58:53 +00:00
John Ericson	3dcca18c30	Fix bug in TeeSource ... We use this to simplify `LocalStore::addToStoreFromDump`. Also, hope I fixed build error with old clang (used in Darwin CI).	2020-07-16 13:39:27 +00:00
Carlo Nucera	e3b394b6e8	Small namespace fix	2020-07-16 09:36:02 -04:00
Eelco Dolstra	5517eee17e	Generations API cleanup	2020-07-16 15:18:21 +02:00
Eelco Dolstra	3f264916db	Merge remote-tracking branch 'origin/flakes'	2020-07-16 14:07:32 +02:00
John Ericson	68dfb8c6ae	Optimize addToStoreSlow and remove TeeParseSink	2020-07-16 05:09:41 +00:00
John Ericson	5602637d9e	Revert "LocalStore::addToStoreFromDump copy in chunks" ... This reverts commit 592851fb67. We don't need this extra feature anymore	2020-07-15 23:37:49 +00:00
John Ericson	bc109648c4	Get rid of LocalStore::addToStoreCommon ... I got it to just become `LocalStore::addToStoreFromDump`, cleanly taking a store and then doing nothing too fancy with it. `LocalStore::addToStore(...Path...)` is now just a simple wrapper with a bare-bones sinkToSource of the right dump command.	2020-07-15 23:14:30 +00:00
Carlo Nucera	455bdee205	Merge branch 'master' of github.com:NixOS/nix into derivation-header-include-order	2020-07-15 17:58:30 -04:00
Carlo Nucera	d090562348	Merge branch 'master' of github.com:NixOS/nix into hash-always-has-type	2020-07-15 17:21:01 -04:00
John Ericson	aff69e229c	Merge remote-tracking branch 'obsidian/fix-add-to-star-narhash' into from-dump-stream	2020-07-15 21:05:29 +00:00
Carlo Nucera	2d2a10e79a	Merge branch 'master' of github.com:NixOS/nix into add-body-to-network-errors	2020-07-15 13:06:08 -04:00
Eelco Dolstra	cff2157185	Revert "LocalStore::addToStore(srcPath): Handle the flat case" ... This reverts commit a2c27022e9. See addToStoreSlow(), we don't need to handle this case efficiently anymore. In fact, we can almost remove the method/hashAlgo arguments since the non-recursive and/or non-SHA256 are almost not used anymore.	2020-07-15 12:49:03 +02:00
John Ericson	650c2c6558	Rename variable nar -> dump according to TODO	2020-07-14 21:28:50 +00:00
John Ericson	8173e7bfef	Fix localhost::addToStore(...Path...) ... We were calculating the nar hash wrong when the file ingestion method was flat. I don't think there's anything we can do in that case but dump the file again, so that's what I do. As an optomization, we again could reuse the original dump for just the recursive and non-sha256 case, but I rather do that after this fix, and after my other PRs which deduplicate this code.	2020-07-14 21:12:11 +00:00
John Ericson	592851fb67	LocalStore::addToStoreFromDump copy in chunks ... Rather than copying byte-by-byte, we let the coroutine know how much data we would like it to send back to us.	2020-07-14 13:56:36 +00:00
John Ericson	9de96ef7d4	Dedup LocalStore::addToStore* ... The downsides is that the coroutine has byte-by-byte loop transfer. Will fix that next.	2020-07-14 13:56:36 +00:00
John Ericson	c86fc3a965	Crudely make addToStoreFromDump take Source not string ... I just as little beyond the type as possible, so the implementation changes this enables can be reviewed separately.	2020-07-14 13:56:36 +00:00
John Ericson	9ec10046e0	Narrow scope of temporary value	2020-07-14 13:54:29 +00:00
Eelco Dolstra	832e111494	Merge remote-tracking branch 'origin/master' into flakes	2020-07-14 13:56:18 +02:00
Eelco Dolstra	926c3a6664	Doh	2020-07-14 11:55:54 +02:00
Eelco Dolstra	43b8e96d30	Fix 'nix verify --all' on a binary cache (cached case)	2020-07-13 20:17:00 +02:00
Eelco Dolstra	7c2fef0a81	Make 'nix copy' to s3:// binary caches run in constant memory	2020-07-13 20:07:19 +02:00
Eelco Dolstra	493961b689	Remove istringstream_nocopy	2020-07-13 18:31:19 +02:00
Eelco Dolstra	545bb2ed03	Remove 'accessor' from addToStore() ... This is only used by hydra-queue-runner and it's better to implement it there.	2020-07-13 18:31:19 +02:00
Eelco Dolstra	0a9da00a10	NarAccessor: Run in constant memory	2020-07-13 17:30:42 +02:00
Eelco Dolstra	fc84c358d9	Make 'nix copy' to file:// binary caches run in constant memory	2020-07-13 16:28:45 +02:00
Eelco Dolstra	400f1a9b59	Store::pathInfoToJSON(): Use consistent format for downloadHash	2020-07-13 16:26:09 +02:00
Eelco Dolstra	c0dd05131e	toStorePath(): Return a StorePath and the suffix	2020-07-13 16:25:48 +02:00
Eelco Dolstra	1d01ae816b	Fix 'nix verify --all' on a binary cache and add a test	2020-07-13 14:35:01 +02:00
John Ericson	c466cb2091	Merge branch 'hash-always-has-type' into better-ca-parse-errors	2020-07-13 03:02:09 +00:00
John Ericson	4415765385	Merge remote-tracking branch 'upstream/master' into hash-always-has-type	2020-07-13 03:01:44 +00:00
John Ericson	230c9b4329	Change types to prepare the way for CA derivations ... We've added the variant to `DerivationOutput` to support them, but made `DerivationOutput::path` partial to avoid actually implementing them. With this chage, we can all collaborate on "just" removing `DerivationOutput::path` calls to implement CA derivations.	2020-07-12 23:53:33 +00:00
John Ericson	fedfc913ad	Use more std::visit to prepare for new variant ... N.B. not using `std::visit` for fetchurl because there is no attempt to handle all the cases (e.g. no `else`) and lambda complicates early return.	2020-07-12 23:51:40 +00:00
John Ericson	ffc18583b1	Move C++17 "pattern matching" boilerplat to utils.hh	2020-07-12 22:15:14 +00:00
John Ericson	503b425690	DerivationOutputExtensional -> DerivationOutputInputAddressed ... Thanks @regnat for the great name.	2020-07-12 15:56:20 +00:00
John Ericson	a8d4707107	Undo erroneous indentation change	2020-07-12 15:54:12 +00:00
John Ericson	18152406ce	String .drv suffix to create derivation name	2020-07-12 15:40:14 +00:00
John Ericson	5d0b75e5b6	std::string_view for new derivation name parameters	2020-07-12 15:02:36 +00:00
John Ericson	abea26a968	BasicDerivation::findOutput cannot return reference anymore	2020-07-12 01:57:06 +00:00
John Ericson	442d43178b	Merge remote-tracking branch 'upstream/master' into optional-derivation-output-storepath	2020-07-11 02:27:45 +00:00
Matthew Bauer	acb74d4d94	Merge remote-tracking branch 'origin/master' into substitute-other-storedir	2020-07-10 18:13:45 -04:00
Eelco Dolstra	8efa23bb99	Avoid a redundant hash	2020-07-10 15:56:24 +02:00
Eelco Dolstra	5dff49f661	Factor out commonality between nix-prefetch-url and nix-store --add-fixed	2020-07-10 13:21:37 +02:00
John Ericson	323d51a7a0	Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors	2020-07-09 19:07:06 +00:00
Matthew Bauer	a7884970c5	Fix DerivationOutputExtensional name	2020-07-09 11:37:18 -04:00
Eelco Dolstra	a2c27022e9	LocalStore::addToStore(srcPath): Handle the flat case ... This helps nix-prefetch-url when using a local store.	2020-07-09 15:54:32 +02:00
Matthew Bauer	06a4e15478	Fix build.cc on linux	2020-07-08 19:27:51 -04:00
Matthew Bauer	8e0d0689be	Only store hash of fixed derivation output ... we don’t need a full storepath for a fixedoutput derivation. So just putting the ingestion method + the hash is sufficient.	2020-07-08 19:11:39 -04:00
Eelco Dolstra	b981e5aacf	Cleanup	2020-07-08 22:07:21 +02:00
Eelco Dolstra	34f25124ba	Make LocalStore::addToStore(srcPath) run in constant memory ... This reduces memory consumption of nix-instantiate \ -E 'with import <nixpkgs> {}; runCommand "foo" { src = ./blender; } "echo foo"' \ --option nar-buffer-size 10000 (where ./blender is a 1.1 GiB tree) from 1716 to 36 MiB, while still ensuring that we don't do any write I/O for small source paths (up to 'nar-buffer-size' bytes). The downside is that large paths are now always written to a temporary location in the store, even if they produce an already valid store path. Thus, adding large paths might be slower and run out of disk space. ¯\_(ツ)_/¯ Of course, you can always restore the old behaviour by setting 'nar-buffer-size' to a very high value.	2020-07-08 22:07:21 +02:00
Matthew Bauer	af95a7c16b	Add name to BasicDerivation ... We always have a name for BasicDerivation, since we have a derivation store path that has a name.	2020-07-08 15:38:01 -04:00
Eelco Dolstra	9223603908	Merge remote-tracking branch 'origin/master' into flakes	2020-07-08 15:55:19 +02:00
Eelco Dolstra	16ec7785ca	Fix 'got unknown message type 1 from Nix daemon' ... Example: $ nix-build -E 'with import <nixpkgs> {}; runCommand "foo" { x = runCommand "bar" {} "exit 1"; } "echo foo; exit 1"' warning: unknown setting 'auto-allocate-uids' these 2 derivations will be built: /nix/store/v4fbdbhcdi949929a67g8farwf72zgam-bar.drv /nix/store/k4fsvrjl7cp2xpz7927iv7g0dqj1zyhs-foo.drv warning: unknown setting 'auto-allocate-uids' building '/nix/store/v4fbdbhcdi949929a67g8farwf72zgam-bar.drv'... error: --- Error ----------------------------------------------------------------------------------------------------------------------------------------------------------------- nix-daemon builder for '/nix/store/v4fbdbhcdi949929a67g8farwf72zgam-bar.drv' failed with exit code 1 error: --- Error ------------------------------------------------------------------------------------------------------------------------------------------------------------------ nix-build got unknown message type 1 from Nix daemon	2020-07-08 15:53:14 +02:00
Eelco Dolstra	1ab9da9154	Merge remote-tracking branch 'origin/master' into flakes	2020-07-07 14:38:57 +02:00
Eelco Dolstra	7c9ece5dca	exportReferencesGraph: Fix support for non-top-level store paths ... Fixes #3471.	2020-07-07 14:25:43 +02:00
Carlo Nucera	0f05a36e20	Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors	2020-07-06 16:57:09 -04:00
Eelco Dolstra	54712aaf8a	Merge remote-tracking branch 'origin/master' into flakes	2020-07-06 16:40:10 +02:00
Eelco Dolstra	7349f257da	Only mount /sys in uid-range builds ... Maybe this should be a separate system feature... /sys exposes a lot of impure info about the host system.	2020-07-06 13:50:33 +02:00
Eelco Dolstra	8c4cce553c	Fix macOS build	2020-07-06 13:50:33 +02:00
Eelco Dolstra	ba50c3efa3	Add "uid-range" and "systemd-cgroup" system features ... "uid-range" provides 65536 UIDs to a build and runs the build as root in its user namespace. "systemd-cgroup" allows the build to mount the systemd cgroup controller (needed for running systemd-nspawn and NixOS containers). Also, add a configuration option "auto-allocate-uids" which is needed to enable these features, and some experimental feature gates. So to enable support for containers you need the following in nix.conf: experimental-features = auto-allocate-uids systemd-cgroup auto-allocate-uids = true system-features = uid-range systemd-cgroup	2020-07-06 13:50:33 +02:00
Eelco Dolstra	570c443f56	Simplify cgroup creation	2020-07-06 13:50:33 +02:00
Eelco Dolstra	7bdcf43b40	Destroy the cgroup prior to building	2020-07-06 13:50:33 +02:00
Eelco Dolstra	ca2f64bcda	Reduce # of UIDs per build to 65536 ... 2^18 was overkill. The idea was to enable multiple containers to run inside a build. However, those containers can use the same UID range - we don't really care about perfect isolation between containers inside a build.	2020-07-06 13:50:33 +02:00
Eelco Dolstra	f5fa3de759	Run builds in their own cgroup ... Also, run builds in a cgroup namespace (ensuring /proc/self/cgroup doesn't leak information about the outside world) and mount /sys. This enables running systemd-nspawn and thus NixOS containers in a Nix build.	2020-07-06 13:50:33 +02:00
Eelco Dolstra	c3e0a68c7e	canonicalisePathMetaData(): Support a UID range	2020-07-06 13:50:33 +02:00
Eelco Dolstra	836573a9a2	Dynamically allocate UIDs ... Rather than rely on a nixbld group, we now allocate UIDs/GIDs dynamically starting at a configurable ID (872415232 by default). Also, we allocate 2^18 UIDs and GIDs per build, and run the build as root in its UID namespace. (This should not be the default since it breaks some builds. We probably should enable this conditional on a requiredSystemFeature.) The goal is to be able to run (NixOS) containers in a build. However, this will also require some cgroup initialisation. The 2^18 UIDs/GIDs is intended to provide enough ID space to run multiple containers per build, e.g. for distributed NixOS tests.	2020-07-06 13:50:33 +02:00
John Ericson	a38ab99d57	Merge remote-tracking branch 'upstream/master' into derivation-header-include-order	2020-07-05 21:49:01 +00:00
John Ericson	465daa9396	Merge remote-tracking branch 'upstream/master' into add-body-to-network-errors	2020-07-03 17:08:39 +00:00
John Ericson	3134db1a83	Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors	2020-07-03 14:12:38 +00:00
John Ericson	dbffd309fe	Merge branch 'master' of github.com:NixOS/nix into hash-always-has-type	2020-07-03 14:11:38 +00:00
Ben Burdette	b29a4ea1dc	Merge branch 'master' into add-trace	2020-07-03 07:57:36 -06:00
regnat	223fbe644a	Shorten the path to the test root ... Fix a socket length failure on the OSX builders	2020-07-03 09:20:01 +02:00
John Ericson	13796be78d	Have splitPrefix and splitPrefixTo parser helpers	2020-07-02 23:18:22 +00:00
John Ericson	2f93d9f2ba	Merge branch 'hash-always-has-type' into HEAD	2020-07-02 21:47:51 +00:00
John Ericson	1be279af26	Fix Narinfo corruption detection bug ... The aim of this check was just to ensure each key occurs once.	2020-07-02 21:46:10 +00:00
Carlo Nucera	1fc835aa22	Tighten parsing for drv files and pathinfo	2020-07-02 11:57:21 -04:00
Matthew Bauer	fc2ab42e86	Merge remote-tracking branch 'origin/master' into substitute-other-storedir	2020-07-02 11:14:04 -04:00
Matthew Bauer	d2e8b9ff0e	Store subPath in SubstitutionGoal	2020-07-02 11:12:05 -04:00
Ben Burdette	bf2788e4c1	move showTrace to new loggerSettings	2020-07-02 09:04:31 -06:00
Matthew Bauer	1f9cb06db2	Try next when no ca exists and have different store dirs	2020-07-02 10:59:24 -04:00
Carlo Nucera	263ccdd489	Rename two hash constructors to proper functions	2020-07-01 18:34:18 -04:00
Eelco Dolstra	38ccf2e241	Cleanup	2020-07-01 15:31:34 +02:00
Eelco Dolstra	86a4aba6c4	Merge branch 'remote-query-outputs' of https://github.com/tweag/nix	2020-07-01 15:10:29 +02:00
Ben Burdette	9159dfe3d8	comments and cleanup	2020-06-30 16:31:55 -06:00
Matthew Bauer	09833126f3	Merge remote-tracking branch 'origin/master' into substitute-other-storedir	2020-06-30 16:35:08 -04:00
Ben Burdette	ddb81ca126	Merge branch 'master' into add-trace	2020-06-30 12:21:45 -06:00
Carlo Nucera	77b51f4598	Factor the prefix splitting in content-address	2020-06-30 11:57:46 -04:00
John Ericson	a83566e5bc	Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors	2020-06-29 18:40:34 +00:00
John Ericson	987a4a0be9	Merge remote-tracking branch 'upstream/master' into hash-always-has-type	2020-06-29 18:03:21 +00:00
Ben Burdette	8f81fae116	showTrace flag in loggers	2020-06-29 10:20:51 -06:00
Eelco Dolstra	58bc3b6578	Merge pull request #3729 from obsidiansystems/simpler-hased-mirror ... hashed-mirrors: Use parsed derivation output rather than reconstructing it	2020-06-29 14:04:12 +02:00
Eelco Dolstra	64232f3ea6	Merge pull request #3749 from rodarima/master ... Fall back to copyPath if link fails with EPERM	2020-06-29 13:31:24 +02:00
Carlo Nucera	278a679782	Merge branch 'master' of github.com:NixOS/nix into allow-relative-paths-in-store-option	2020-06-26 10:40:13 -04:00
Eelco Dolstra	adf2fbbdc2	Merge remote-tracking branch 'origin/master' into flakes	2020-06-26 08:46:46 +02:00
John Ericson	d92d4f85a5	Move ValidPathInfo to its own header	2020-06-25 17:48:49 +00:00
Eelco Dolstra	de2641ae99	Fix empty std::optional dereference in writeDerivation() ... https://hydra.nixos.org/build/123017579	2020-06-25 15:50:30 +02:00
Rodrigo	3a642187c3	Fall back to copyPath if link fails with EPERM ... BeeGFS doesn't allow hard-links and returns EPERM, so we fall back to copyPath. See https://github.com/NixOS/nix/issues/3748	2020-06-25 12:03:26 +02:00
regnat	d38f860c3e	Add a way to get all the outputs of a derivation with their label ... Generalize `queryDerivationOutputNames` and `queryDerivationOutputs` by adding a `queryDerivationOutputMap` that returns the map `outputName=>outputPath` (not that this is not equivalent to merging the results of `queryDerivationOutputs` and `queryDerivationOutputNames` as sets don't preserve the order, so we would end up with an incorrect mapping). squash! Add a way to get all the outputs of a derivation with their label Rename StorePathMap to OutputPathMap	2020-06-24 20:38:40 +02:00
John Ericson	d9fb9e9f26	Merge remote-tracking branch 'upstream/master' into ca-drv	2020-06-24 15:41:29 +00:00
John Ericson	98e5d1af03	Merge remote-tracking branch 'upstream/master' into hash-always-has-type	2020-06-23 17:03:37 +00:00
Carlo Nucera	e197bc6229	Enable the --store option to take relative paths ... In nix commands which accept --store options, we can now specify a relative path, which will be canonicalized.	2020-06-23 11:13:18 -04:00
John Ericson	55d4bd6e0e	Improve content address parsing ... - Ensure hash is in form <algo>-<prefix> and not SRI. - Better errors if something goes wrong - string_view for no coppying	2020-06-22 18:08:27 +00:00
Matthew Bauer	66a62b3189	Merge remote-tracking branch 'origin/master' into substitute-other-storedir	2020-06-22 13:08:11 -04:00
John Ericson	f4a5913125	hashed-mirrors: Use parsed derivation output rather than reconstructing it ... Now the derivation outputs are parsed up front, we can avoid a reparse by doing it. Also, this just feels a bit better as the `output*` env vars are more of a `libnixexpr` interface than `libnixstore` interface: ultimately, it's the derivation outputs that decide whether the derivation is fixed-output. Yes, hashed mirrors might go away with #3689, but this bit of code would be moved rather than deleted, so it's worth doing a cleanup anyways I think.	2020-06-22 15:17:20 +00:00
John Ericson	93bbe6e8ab	Merge remote-tracking branch 'upstream/master' into ca-drv	2020-06-21 21:11:58 +00:00
John Ericson	3804e3df9b	Don't anticipate hash algo without hash in derivation for now ... When we merge with master, the new lack of string types make this case impossible (after parsing). Later, when we actually implemenent CA-derivations, we'll change the types to allow that.	2020-06-21 21:05:37 +00:00
John Ericson	8313f0e939	Merge remote-tracking branch 'upstream/master' into derivation-header-include-order	2020-06-21 20:39:10 +00:00
John Ericson	bf9f040112	Tweak declaration ... I think this is clearer	2020-06-21 16:51:39 +00:00
John Ericson	fdeabf7160	Merge remote-tracking branch 'upstream/master' into multi-output-hashDerivationModulo	2020-06-21 16:43:17 +00:00
Matthew Bauer	f2a6cee334	Update worker protocol to support sending storepath maps ... We need to also send the ca to daemon in addition to the path.	2020-06-19 18:06:19 -04:00
John Ericson	507aa48739	WIP: Make Hash always store a valid hash type	2020-06-19 21:48:57 +00:00
Matthew Bauer	e8e1f5282f	Replace error message in getDerivationCA	2020-06-19 15:19:09 -04:00
Matthew Bauer	0c9c1b8826	Return map of StorePaths in copyPaths ... This allows the caller to know what values were actually added to the store.	2020-06-19 14:48:57 -04:00
Matthew Bauer	7e11cf3399	Merge remote-tracking branch 'origin/master' into substitute-other-storedir	2020-06-19 14:40:49 -04:00
Matthew Bauer	2796b01e86	Merge remote-tracking branch 'origin/master' into substitute-other-storedir	2020-06-19 14:22:06 -04:00
John Ericson	68294746ae	Merge remote-tracking branch 'upstream/master' into no-hash-type-unknown	2020-06-19 17:53:34 +00:00
John Ericson	911fc88bcb	More designated initializers	2020-06-19 17:42:56 +00:00
John Ericson	2f0e395c99	Merge remote-tracking branch 'me/no-stringly-typed-derivation-output' into validPathInfo-ca-proper-datatype	2020-06-19 15:26:59 +00:00
John Ericson	fb39a5e00c	Remove unneeded constructor for DerivationOutputHash	2020-06-19 15:11:11 +00:00
John Ericson	01dc8b0bab	Merge remote-tracking branch 'upstream/master' into no-stringly-typed-derivation-output	2020-06-19 14:59:05 +00:00
John Ericson	145d88cb2a	Use designated initializers for DerivationOutputHash	2020-06-19 14:58:30 +00:00
John Ericson	237d88c97e	FileSystemHash -> DerivationOutputHash	2020-06-19 14:47:10 +00:00
John Ericson	3fc58a9638	Remove some Base:: that crept in	2020-06-19 00:24:47 +00:00
John Ericson	3f8dcfe3fd	Merge branch 'validPathInfo-temp' into validPathInfo-ca-proper-datatype	2020-06-18 23:01:58 +00:00
John Ericson	669c3992e8	Merge branch 'no-hash-type-unknown' into validPathInfo-temp	2020-06-18 22:33:07 +00:00
John Ericson	15abb2aa2b	Revert the enum struct change ... Not a regular git revert as there have been many merges and things.	2020-06-18 22:11:26 +00:00
John Ericson	bbbf3602a3	Merge branch 'enum-class' into no-hash-type-unknown	2020-06-18 22:11:19 +00:00
John Ericson	40526fbea5	Merge remote-tracking branch 'upstream/master' into enum-class	2020-06-18 21:38:15 +00:00
John Ericson	1b23fe4afb	Fix bugs ... - Bad dynamic cast target ...classic - std::shared_ptr need explicit deref	2020-06-18 19:03:10 +00:00
John Ericson	fb432b0bd2	Merge remote-tracking branch 'obsidian/fix-url-format' into add-body-to-network-errors	2020-06-18 18:08:02 +00:00
John Ericson	639e20dc3e	Prevent '%' in URL from causing crashes ... We have a larger problem that passsing computed strings to the first variable argument of many exception constructors is unsafe because that first variable argument is interpreted not as a plain string, but format string, and if it contains '%' boost::format will abort, since there are no arguments to the format string. In this particular instance '%' was used as part of an escape code in a URL, which, when the download failed, caused Nix to abort displaying the `FileTransferError`.	2020-06-18 17:54:16 +00:00
John Ericson	74b219ef6e	Adjust FileTransferError message to use opt response	2020-06-18 14:48:45 +00:00
regnat	4fef2ba7e4	Rename content-addressed-paths into ca-derivations ... See <https://github.com/NixOS/nix/pull/3710#issuecomment-645480333>	2020-06-18 09:25:55 +02:00
Carlo Nucera	004570a377	Add HTTP responses to FileTransferErrors	2020-06-17 22:34:31 +00:00
Carlo Nucera	a835c740ca	Replace TransferItem::status with a local variable ... Everywhere seems to use `getHTTPStatus` now.	2020-06-17 22:16:24 +00:00
Carlo Nucera	fb05a6adcf	Eliminate old TeeSink abstraction ... This was introduced in fa125b9b28, and then "reverted" in 1cf4801108, except that revert left the struct around doing nothing useful. We're removing it all the way now because we want to make a new `TeeSink` complementing the already-exiting `TeeSource`, that is actually a completely different concept as far as the class hierarchy is concerned.	2020-06-17 22:07:22 +00:00
Eelco Dolstra	2b8f33bf5f	Merge pull request #3713 from matthewbauer/cleanup-warnings ... Cleanup class StorePath warning	2020-06-17 21:19:30 +02:00
Matthew Bauer	5e631e3304	Add StorePathCAMap for querySubstitutablePathInfos ... I’m not 100% sure this is wanted since it kind of makes everything have to know about ca even if they don’t really want to. But it also make things easier in dealing with looking up ca.	2020-06-17 15:03:05 -04:00
Matthew Bauer	be50de1142	Make sure references are empty for store path replacing ... also copy info2 instead of casting	2020-06-17 14:15:08 -04:00
Matthew Bauer	8974755d19	Add assert for replaced storePath	2020-06-17 14:04:46 -04:00
Eelco Dolstra	3078404e35	Merge pull request #3712 from obsidiansystems/make-http-successful-states-coherent ... Make successful states coherent	2020-06-17 19:37:46 +02:00
Matthew Bauer	7bd88cc1dc	Merge remote-tracking branch 'origin/master' into substitute-other-storedir	2020-06-17 13:32:29 -04:00
Matthew Bauer	22d7d36703	Remove unused narInfoFile in binary-cache-store	2020-06-17 13:27:19 -04:00
Matthew Bauer	f767bedfac	Replace struct StorePath with class StorePath ... also a similar case with struct Goal	2020-06-17 13:26:37 -04:00
Carlo Nucera	4930cb48a2	Include review comments	2020-06-17 12:58:59 -04:00
Matthew Bauer	6438ba1e99	Update strings from review comment	2020-06-17 12:31:01 -04:00
Eelco Dolstra	2f51cd8dc9	Merge pull request #3710 from tweag/reserve_ca_derivations ... Reserve the `__contentAddressed` derivation parameter	2020-06-17 18:28:26 +02:00
Eelco Dolstra	4d5169bdd5	Merge pull request #3707 from p01arst0rm/outdated-function-fix ... replaced uncaught_exception with uncaught_exceptions	2020-06-17 18:26:01 +02:00
regnat	480b54e1c6	fixup! Reserve the __contentAddressed derivation parameter	2020-06-17 17:37:04 +02:00
Carlo Nucera	079c6e87de	Make successful states coherent ... The successful states used in these two places in the code were slightly different. Should they be the same list?	2020-06-17 11:16:16 -04:00
regnat	56d75bf4fc	Reserve the __contentAddressed derivation parameter ... Not implementing anything here, just throwing an error if a derivation sets `__contentAddressed = true` without `--experimental-features content-addressed-paths` (and also with it as there's nothing implemented yet)	2020-06-17 15:41:17 +02:00
Eelco Dolstra	fdff09e57c	Fix coverage build	2020-06-17 15:18:10 +02:00
Eelco Dolstra	ccbea8255c	Merge pull request #3657 from obsidiansystems/sligthly-improve-store-path-documentation ... Clarify the description of StorePath inputs	2020-06-17 14:54:37 +02:00
Eelco Dolstra	1524752c17	Merge remote-tracking branch 'origin/master' into flakes	2020-06-17 10:26:52 +02:00
John Ericson	517f5980e2	Merge remote-tracking branch 'upstream/master' into no-stringly-typed-derivation-output	2020-06-17 04:58:43 +00:00
John Ericson	18493fd9c4	Move some Store functions from derivations.cc to store-api.cc ... This further continues with the dependency inverstion. Also I just went ahead and exposed `parseDerivation`: it seems like the more proper building block, and not a bad thing to expose if we are trying to be less wedded to drv files on disk anywas.	2020-06-17 04:30:16 +00:00
John Ericson	7130f0a3a6	Don't need abstract struct Derivation in local-store	2020-06-17 04:05:38 +00:00
John Ericson	21ef342172	Merge remote-tracking branch 'upstream/master' into derivation-header-include-order	2020-06-17 03:52:01 +00:00
John Ericson	02928f76fd	Merge remote-tracking branch 'upstream/master' into multi-output-hashDerivationModulo	2020-06-17 03:51:02 +00:00
John Ericson	49308ef039	Merge remote-tracking branch 'upstream/master' into ca-drv	2020-06-17 03:39:39 +00:00
p01arst0rm	c9d06558b6	replaced uncaught_exception with uncaught_exceptions	2020-06-17 03:15:47 +01:00
Eelco Dolstra	29542865ce	Remove StorePath::clone() and related functions	2020-06-16 22:20:18 +02:00
Eelco Dolstra	759947bf72	StorePath: Rewrite in C++ ... On nix-env -qa -f '<nixpkgs>', this reduces maximum RSS by 20970 KiB and runtime by 0.8%. This is mostly because we're not parsing the hash part as a hash anymore (just validating that it consists of base-32 characters). Also, replace storePathToHash() by StorePath::hashPart().	2020-06-16 14:28:41 +02:00
Eelco Dolstra	72e17290d4	Fix FTP support ... Fixes #3618.	2020-06-16 11:53:04 +02:00
Eelco Dolstra	a588b6b19d	Print only one error message if a build fails ... E.g. instead of error: --- BuildError ----------------------------------------------- nix builder for '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed with exit code 1 error: --- Error ---------------------------------------------------- nix build of '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed we now get error: --- Error ---------------------------------------------------- nix builder for '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed with exit code 1	2020-06-15 19:35:31 +02:00
Eelco Dolstra	31707735b6	Remove unnecessary amDone() overrides	2020-06-15 16:47:21 +02:00
Eelco Dolstra	5ed5d7acbd	Improve "waiting for locks" messages ... These are now shown in the progress bar. Closes #3577.	2020-06-15 16:03:29 +02:00
Eelco Dolstra	e14e62fddd	Remove trailing whitespace	2020-06-15 14:12:39 +02:00
Eelco Dolstra	1fb762d11f	Get rid of explicit ErrorInfo constructors	2020-06-15 14:06:58 +02:00
Eelco Dolstra	7a77762961	Merge branch 'errors-phase-2' of https://github.com/bburdette/nix	2020-06-15 11:46:31 +02:00
Matthew Bauer	c214cda940	Correctly substitute from different storeDir ... Originally, the test was only checking for different “real” storeDir. That’s an easy case to handle, but the much harder one is if different virtual store dirs are used. To do this, we need the SubstitutionGoal to know about the ca, so it can recalculate the path to copy it over. An important note here is that the store path passed to copyStorePath needs to be one for srcStore - so that queryPathInfo works properly. This also adds an error message when the store path from queryPathInfo is different from the one we requested.	2020-06-13 00:07:42 -05:00
Matthew Bauer	b2cb288cdd	Add makeFixedOutputPathFromCA function ... This puts what we are already doing into a shared method. It just needs a path name and a ca and produces a store path.	2020-06-12 16:36:35 -05:00
John Ericson	f6f01416b7	Use std::string_view in a few more places	2020-06-12 21:32:30 +00:00
Matthew Bauer	88120442d2	Debug when storePath changes ... these rewrites should be transparent, but they are important to know about when debugging	2020-06-12 15:32:52 -05:00
Matthew Bauer	5b386b05f5	Recompute storePath based on isContentAddressed	2020-06-12 14:57:45 -05:00
Matthew Bauer	85d01e1f65	Don’t use makeStorePath	2020-06-12 14:53:30 -05:00
Matthew Bauer	9077b9e6b2	Separate dstStore path from srcStore path	2020-06-12 14:27:28 -05:00
Matthew Bauer	2f2ac850b5	Compute new store path correctly	2020-06-12 14:03:17 -05:00
Matthew Bauer	11c97070f3	Fix storeDir != storeDir condition ... this needs to only continue if the path replacement fails.	2020-06-12 10:14:03 -05:00
Matthew Bauer	3e3eaa90dd	Remove hashed-mirrors	2020-06-12 09:51:44 -05:00
Matthew Bauer	79c169d1c6	Allow substituting from different storeDir ... Substituters can substitute from one store dir to another with a little bit of help. The store api just needs to have a CA so it can recompute the store path based on the new store dir. We can only do this for fixed output derivations with no references, though.	2020-06-12 09:49:09 -05:00
Eelco Dolstra	9f736dd89d	Add Store::readDerivation() convenience function	2020-06-12 13:04:52 +02:00
Eelco Dolstra	045b07200c	Remove Store::queryDerivationOutputNames() ... This function was used in only one place, where it could easily be replaced by readDerivation() since it's not performance-critical. (This function appears to have been modelled after queryDerivationOutputs(), which exists only to make the garbage collector faster.)	2020-06-12 12:46:33 +02:00
Ben Burdette	ef1b3f21b6	Merge remote-tracking branch 'upstream/master' into errors-phase-2	2020-06-11 14:06:35 -06:00
Eelco Dolstra	ac4d43a31b	Merge pull request #3073 from tweag/machine-logs ... Add an option to print the logs in a machine-readable format	2020-06-11 15:45:18 +02:00
Eelco Dolstra	dd9bb11d0d	Move names.{cc,hh} to libstore	2020-06-11 15:42:18 +02:00
Eelco Dolstra	b9ae1bdd7a	Merge pull request #3655 from zimbatm/hash-encoding-prepare ... libutils/hash: remove default encoding	2020-06-10 11:48:38 +02:00
Matthew Bauer	b2c8061b44	Disable extra-platforms = i686-linux on wsl1 (#3676) ... WSL1 doesn’t support i686-linux emulation, see https://github.com/microsoft/wsl/issues/2468	2020-06-09 21:53:53 +00:00
Eelco Dolstra	29e0748847	Show HTTP status message ... For example: warning: unable to download 'https://api.github.com/repos/edolstra/dwarffs/commits/master': HTTP error 403 ('rate limit exceeded'); using cached version	2020-06-09 14:20:22 +02:00
Eelco Dolstra	447ea52b07	FileTransfer: Don't store status since curl already does that	2020-06-09 14:05:15 +02:00
regnat	4983401440	Unify the printing of the logs between bar-with-logs and raw ... Make the printing of the build logs systematically go through the logger, and replicate the behavior of `no-build-output` by having two different loggers (one that prints the build logs and one that doesn't)	2020-06-08 09:31:15 +02:00
Carlo Nucera	d614166cb6	Fix condition error and make test suite pass	2020-06-04 17:21:21 -04:00
John Ericson	e5cc1ebc5d	Merge remote-tracking branch 'upstream/master' into no-stringly-typed-derivation-output	2020-06-04 21:04:35 +00:00
John Ericson	94ddea9e2f	Use readString rather than >> temporary ... Fixed the rest of these before, but this one slipped through.	2020-06-04 20:55:08 +00:00
John Ericson	2041499b5e	Flip boolean ... Thanks Matt!	2020-06-04 20:42:25 +00:00
John Ericson	ed86acf02a	Use some std::optional::has_value for clarity	2020-06-04 20:42:02 +00:00
Matthew Bauer	2299ef705c	Add error message when FileIngestionMethod is out of bounds ... bool coerces anything >0 to true, but in the future we may have other file ingestion methods. This shows a better error message when the “recursive” byte isn’t 1.	2020-06-04 11:32:39 -05:00
John Ericson	53bc8ff152	No C++ designated initializers yet with Clang 7	2020-06-03 20:45:14 -04:00
Ben Burdette	4335ba999b	Merge remote-tracking branch 'upstream/master' into errors-phase-2	2020-06-03 17:00:00 -06:00
Carlo Nucera	132d6f2c24	Clarify the description of StorePath construction	2020-06-03 16:08:32 -04:00
John Ericson	74b251b2f3	Don't anticipate multiple CA outputs for now	2020-06-03 18:53:04 +00:00
John Ericson	3a9e4c3262	Don't anticipate CA but not fixed outputs for now	2020-06-03 18:50:45 +00:00
John Ericson	2500403059	Use enum and predicates rather than bitfile for derivation type	2020-06-03 17:38:54 +00:00
John Ericson	6b7f4ec4ab	Merge remote-tracking branch 'upstream/master' into ca-drv	2020-06-03 16:36:06 +00:00
Eelco Dolstra	c20591ddc3	Merge remote-tracking branch 'origin/master' into flakes	2020-06-03 16:15:22 +02:00
Jonas Chevalier	6ee03b8444	libutils/hash: remove default encoding ... This will make it easier to reason about the hash encoding and switch to SRI everywhere where possible.	2020-06-03 13:49:51 +02:00
John Ericson	3c78ac348c	Merge remote-tracking branch 'obsidian/no-hash-type-unknown' into validPathInfo-ca-proper-datatype	2020-06-03 04:44:24 +00:00
John Ericson	406dbb7fce	outputHashAlgo can be blank so parse accordingly ... It is blank for SRI hashes.	2020-06-02 21:09:15 +00:00
Carlo Nucera	75d2581390	Typo	2020-06-02 16:21:18 -04:00
Carlo Nucera	78f137e931	Validate text version instead, throw Errors	2020-06-02 16:20:22 -04:00
Carlo Nucera	a5cdf1867e	Add assertions for SHA256 in fixed case	2020-06-02 16:13:08 -04:00
Carlo Nucera	fd2eb41e64	Move file-hash to content-address	2020-06-02 15:44:58 -04:00
Carlo Nucera	343c20a404	WIP Completed implementation	2020-06-02 15:23:21 -04:00
Carlo Nucera	390bf64858	WIP	2020-06-02 14:15:58 -04:00
John Ericson	a33270ce1d	Clean up ValidPathInfo::isContentAddressed with std::visit	2020-06-02 17:04:21 +00:00
John Ericson	25e61812f3	Apply suggestions from code review ... Co-authored-by: Matthew Bauer <mjbauer95@gmail.com>	2020-06-02 12:47:18 -04:00
John Ericson	64cffb804a	Merge remote-tracking branch 'upstream/master' into no-hash-type-unknown	2020-06-02 16:07:25 +00:00
John Ericson	450dcf2c1b	Remove HashType::Unknown ... Instead, `Hash` uses `std::optional<HashType>`. In the future, we may also make `Hash` itself require a known hash type, encoraging people to use `std::optional<Hash>` instead.	2020-06-02 15:52:13 +00:00
Carlo Nucera	a5d820a0a3	Change parseCa(Opt) to parseContentAddress(Opt)	2020-06-02 11:00:10 -04:00
John Ericson	1b6461f671	Merge remote-tracking branch 'upstream/master' into validPathInfo-ca-proper-datatype	2020-06-02 14:31:18 +00:00
Ben Burdette	d82d230b40	elide the 'ErrorInfo' in logError and logWarning calls	2020-06-02 08:22:24 -06:00
John Ericson	efcd30da89	WIP	2020-06-02 00:37:43 +00:00
John Ericson	754c910953	WIP more progress	2020-06-01 19:26:40 -04:00
Carlo Nucera	da39092a39	WIP	2020-06-01 18:53:31 -04:00
Carlo Nucera	0e9438b6d3	Create new file-hash files	2020-06-01 17:32:40 -04:00
Carlo Nucera	0cb67ecbd3	Merge branch 'derivation-header-include-order' of github.com:Ericson2314/nix into validPathInfo-ca-proper-datatype	2020-06-01 17:13:11 -04:00
Carlo Nucera	f4b89e11a4	Merge branch 'no-stringly-typed-derivation-output' of github.com:Ericson2314/nix into validPathInfo-ca-proper-datatype	2020-06-01 17:12:50 -04:00
John Ericson	fac0c2d54a	Remove addToStore variant as requested by FIXME ... The idea is it's always more flexible to consumer a `Source` than a plain string, and it might even reduce memory consumption. I also looked at `addToStoreFromDump` with its `// FIXME: remove?`, but the worked needed for that is far more up for interpretation, so I punted for now.	2020-05-29 17:02:32 -04:00
Ben Burdette	734283d636	Merge remote-tracking branch 'upstream/master' into errors-phase-2	2020-05-29 09:51:37 -06:00
Carlo Nucera	6dd471ebf6	Fixing the result of merge	2020-05-28 12:14:36 -04:00
Carlo Nucera	4f597fb901	Merge branch 'master' of github.com:NixOS/nix into enum-class	2020-05-28 10:58:22 -04:00
John Ericson	5b4cd84bc2	Merge remote-tracking branch 'me/more-rust-ffi' into no-stringly-typed-derivation-output	2020-05-28 10:35:53 -04:00
John Ericson	ef71caba29	Merge remote-tracking branch 'upstream/master' into more-rust-ffi	2020-05-28 10:31:46 -04:00
Eelco Dolstra	17ca997fc6	Merge remote-tracking branch 'origin/master' into flakes	2020-05-28 12:55:24 +02:00
Matthew Bauer	c66441a646	Rename some variables named “recursive” to “method” ... This is much less confusing since recursive is no longer a boolean.	2020-05-27 13:21:26 -05:00
Carlo Nucera	d49e65ba9d	Merge remote-tracking branch 'john-ericson/enum-FileIngestionMethod' into no-stringly-typed-derivation-output	2020-05-26 12:30:48 -04:00
Carlo Nucera	0f3f901071	Merge remote-tracking branch 'origin/master' into more-rust-ffi	2020-05-26 11:46:42 -04:00
Carlo Nucera	6d73c10041	Merge remote-tracking branch 'origin/master' into enum-FileIngestionMethod	2020-05-26 11:14:08 -04:00
Eelco Dolstra	5ef64f05e6	Cleanup	2020-05-18 15:50:29 +02:00
Eelco Dolstra	0ed946aa61	Merge branch 'wait-for-builders' of https://github.com/serokell/nix	2020-05-18 13:48:45 +02:00
Ben Burdette	92123c6c79	Merge remote-tracking branch 'upstream/master' into errors-phase-2	2020-05-15 07:00:36 -06:00
Alexander Bantyev	183dd28266	Don't lock a user while doing remote builds	2020-05-14 17:00:54 +03:00
Ben Burdette	ef9dd9f9bc	formatting and a few minor changes	2020-05-13 15:56:39 -06:00
Ben Burdette	c79d4addab	consistent capitalization	2020-05-13 10:02:18 -06:00
Ben Burdette	bfca5fc395	change status messages to info level	2020-05-13 09:52:36 -06:00
Ben Burdette	19cffc29c9	remove unused extra json fields	2020-05-12 12:09:12 -06:00
Ben Burdette	ec870b9c85	new pos format for more errors	2020-05-12 10:52:26 -06:00
Eelco Dolstra	ebc024df22	Show hint how to enable experimental features	2020-05-12 15:47:09 +02:00
Ben Burdette	b93c1bf3d6	fixes to merged code	2020-05-11 15:52:15 -06:00
Ben Burdette	59b1f5c701	Merge branch 'master' into errors-phase-2	2020-05-11 14:35:30 -06:00
Ben Burdette	958e81987b	switch from printError warnings to logWarnings	2020-05-11 13:02:16 -06:00
Domen Kožar	5bdb67c843	Merge pull request #3568 from kolloch/outputHashModeError ... libstore/build.cc: more explicit error about form of output	2020-05-11 18:14:32 +02:00
Domen Kožar	1d8144e36b	Update src/libstore/build.cc	2020-05-11 18:14:23 +02:00
Benjamin Hipple	146f9c114f	doc: consistently refer to 'fixed-output' with a dash ... General cleanup that makes it easier to search for the term.	2020-05-09 10:58:43 -04:00
Alexander Bantyev	772e5db828	Mention build users in the 'waiting for' message	2020-05-08 12:29:00 +03:00
Alexander Bantyev	14073fb76b	Don't block while waiting for build users	2020-05-08 12:22:39 +03:00
Ben Burdette	e3901638b5	todo removal	2020-05-06 15:01:13 -06:00
Eelco Dolstra	2f8ee4578f	Merge remote-tracking branch 'origin/master' into flakes	2020-05-06 12:01:40 +02:00
Eelco Dolstra	74a1bfdcab	Merge pull request #3546 from guibou/nix_readfile_on_0_sized_files ... builtins.readFile: do not truncate content	2020-05-06 11:33:55 +02:00
Peter Kolloch	9be46859a9	libstore/build.cc: more explicit about form of output ... Be more explicit about why we expect a regular file as output when outputHashMode=flat for a fixed output derivation.	2020-05-06 11:21:12 +02:00
Eelco Dolstra	02c5914ea4	Merge pull request #3562 from pikajude/master ... Use fragment size for autoGC capacity calculation	2020-05-06 11:19:30 +02:00
Eelco Dolstra	fd4911269f	Revert "Merge pull request #3558 from LnL7/ssh-ng-stderr" ... This reverts commit 3ebfbecdd1, reversing changes made to c089c52d5f. https://github.com/NixOS/nix/pull/3558	2020-05-06 10:54:18 +02:00
Eelco Dolstra	6f3244ce45	Merge remote-tracking branch 'origin/master' into flakes	2020-05-05 18:59:33 +02:00
Alexander Bantyev	04967dee9d	Wait for build users when none are available	2020-05-05 13:04:36 +03:00
Ben Burdette	7ffb5efdbc	appending to hints; remove _printError	2020-05-04 16:19:57 -06:00
Jude Taylor	e2fc575c61	nix auto-gc: use fragment size	2020-05-04 14:42:06 -07:00
Eelco Dolstra	a721a0b114	Flag: Use designated initializers	2020-05-04 22:40:19 +02:00
Ben Burdette	ab6f0b9641	convert some printError calls to logError	2020-05-03 08:01:25 -06:00
Daiderd Jordan	f16e24f95e	remote-store: don't log raw stderr by default ... For remote stores the log messages are already forwarded as structured STDERR_RESULT messages so the old format is duplicate information. But still included with -vvv since it could be useful for debugging problems. $ nix build -L /nix/store/nl71b2niws857ffiaggyrkjwgx9jjzc0-foo.drv --store ssh-ng://localhost Hello World! foo> Hello World! [1/0/1 built] building foo Fixes #3556	2020-05-02 23:40:50 +02:00
Ben Burdette	4b99c09f5c	convert some errors	2020-05-01 14:32:06 -06:00
Eelco Dolstra	941f95284a	Merge remote-tracking branch 'origin/master' into flakes	2020-05-01 11:59:56 +02:00
Ben Burdette	39ff80d031	errorinfo constructor test	2020-04-29 18:57:05 -06:00
Guillaume Bouchard	7afcb5af98	Remove the drain argument from readFile ... Now it is always `drain` (see previous commit).	2020-04-29 18:43:45 +02:00
Antoine Eiche	ca93b26db6	Only call grantpt on MacOS systems ... The commit 3cc1125595 adds a `grantpt` call on the builder pseudo terminal fd. This call is actually only required for MacOS, but it however requires a RW access to /dev/pts which is only RO bindmounted in the Bazel Linux sandbox. So, Nix can not be actually run in the Bazel Linux sandbox for unneeded reasons.	2020-04-29 15:43:20 +02:00
Ben Burdette	22e6490311	Error classname as name	2020-04-28 21:06:08 -06:00
Eelco Dolstra	ef4d3fc111	Merge remote-tracking branch 'origin/master' into flakes	2020-04-23 15:16:18 +02:00
Dustin DeWeese	c0d940978a	Replace select() with poll() to allow waiting on more than FD_SETSIZE fds	2020-04-21 16:21:28 -07:00
Ben Burdette	e4fb9a3849	remove 'format' from Error constructor calls	2020-04-21 17:07:07 -06:00
Ben Burdette	15e9564fd1	logEI for tunnelLogger and progressbar	2020-04-19 17:16:51 -06:00
Ben Burdette	3d5b1032a1	logError, logWarning; Logger functions; switch to Verbosity enum	2020-04-17 15:07:44 -06:00
Eelco Dolstra	aaa109565e	Use a more space/time-efficient representation for the eval cache	2020-04-17 23:04:21 +02:00
Eelco Dolstra	7a9687ba30	SQLiteStmt: Use std::string_view	2020-04-17 01:00:56 +02:00
Eelco Dolstra	0858738355	Merge remote-tracking branch 'origin/master' into flakes	2020-04-16 18:27:37 +02:00
Eelco Dolstra	a118293bd0	Merge pull request #3458 from zimbatm/nix-user-conf-dir ... NIX_USER_CONF_FILES	2020-04-15 13:00:28 +02:00
Jonas Chevalier	895516cadf	add NIX_USER_CONF_FILES ... Motivation: maintain project-level configuration files. Document the whole situation a bit better so that it corresponds to the implementation, and add NIX_USER_CONF_FILES that allows overriding which user files Nix will load during startup.	2020-04-14 18:45:06 +02:00
Eelco Dolstra	c0c2cb871d	Merge remote-tracking branch 'origin/master' into flakes	2020-04-14 13:02:55 +02:00
Bruce Toll	8132d0a12e	Fix nix-build --check -K in sandbox w/o root ... Temporarily add user-write permission to build directory so that it can be moved out of the sandbox to the store with a .check suffix. This is necessary because the build directory has already had its permissions set read-only, but write permission is required to update the directory's parent link to move it out of the sandbox. Updated the related --check "derivation may not be deterministic" messages to consistently use the real store paths. Added test for non-root sandbox nix-build --check -K to demonstrate issue and help prevent regressions.	2020-04-10 16:23:10 -04:00
Domen Kožar	db25a6d7bb	Merge pull request #2689 from tollb/fix/delete_tmp_dir_when_build_check_ok ... Delete temporary directory on successful build	2020-04-10 09:47:57 +02:00
Bruce Toll	16a4864759	Delete temporary directory on successful build ... With --check and the --keep-failed (-K) flag, the temporary directory was being retained regardless of whether the build was successful and reproducible. This removes the temporary directory, as expected, on a reproducible check build. Added tests to verify that temporary build directories are not retained unnecessarily, particularly when using --check with --keep-failed.	2020-04-09 16:37:41 -04:00
Philipp Middendorf	04bedda0b6	gc.cc: Ignore hidden files in temproots	2020-04-09 17:05:29 +02:00
Eelco Dolstra	d103c79144	Merge remote-tracking branch 'origin/master' into flakes	2020-04-09 15:54:21 +02:00
Nikola Knezevic	f5095594e7	datatransfer.{cc,hh} -> filetransfer.{cc,hh}	2020-04-08 22:26:57 +02:00
Nikola Knezevic	c330109bfa	DataTransfer -> FileTransfer	2020-04-08 22:26:57 +02:00
Nikola Knezevic	7848372b0f	Add upload method	2020-04-08 22:26:57 +02:00
Nikola Knezevic	a0c5931208	actDownload -> actDataTransfer	2020-04-08 22:26:57 +02:00
Nikola Knezevic	c4c1ae0a00	DownloadError -> DataTransferError	2020-04-08 22:26:57 +02:00
Nikola Knezevic	213d124277	DownloadItem -> TransferItem	2020-04-08 22:26:57 +02:00
Nikola Knezevic	cd391206e6	{get,make,new}Downloader -> DataTransfer	2020-04-08 22:26:57 +02:00
Nikola Knezevic	2df2741ec6	enqueueDownload -> enqueueDataTransfer	2020-04-08 22:26:57 +02:00
Nikola Knezevic	142ed7fe45	DownloadResult -> DataTransferResult	2020-04-08 22:26:57 +02:00
Nikola Knezevic	e5cc53beec	DownloadSettings -> DataTransferSettings	2020-04-08 22:26:57 +02:00
Nikola Knezevic	741e9012d3	Rename src/lib/download.* to src/lib/datatransfer.*	2020-04-08 22:26:57 +02:00
Nikola Knezevic	65ef57e0cb	DownloadRequest -> DataTransferRequest	2020-04-08 22:26:57 +02:00
Domen Kožar	bf81b31559	build.cc: improve message if home directory exists	2020-04-08 18:27:10 +02:00
Eelco Dolstra	1ab8d6ac18	Downloader: Only write data to the sink on a 200 response ... Hopefully fixes #3278.	2020-04-08 15:27:09 +02:00
Eelco Dolstra	54955867a6	Merge remote-tracking branch 'origin/master' into flakes	2020-04-07 14:29:45 +02:00
Eelco Dolstra	cd39709003	Cleanup	2020-04-07 09:27:17 +02:00
Eelco Dolstra	462421d345	Backport libfetchers from the flakes branch ... This provides a pluggable mechanism for defining new fetchers. It adds a builtin function 'fetchTree' that generalizes existing fetchers like 'fetchGit', 'fetchMercurial' and 'fetchTarball'. 'fetchTree' takes a set of attributes, e.g. fetchTree { type = "git"; url = "https://example.org/repo.git"; ref = "some-branch"; rev = "abcdef..."; } The existing fetchers are just wrappers around this. Note that the input attributes to fetchTree are the same as flake input specifications and flake lock file entries. All fetchers share a common cache stored in ~/.cache/nix/fetcher-cache-v1.sqlite. This replaces the ad hoc caching mechanisms in fetchGit and download.cc (e.g. ~/.cache/nix/{tarballs,git-revs*}). This also adds support for Git worktrees (c169ea5904).	2020-04-07 09:03:14 +02:00
Cole Helbling	c976cb0b8a	Don't retry on "unsupported protocol" error ... When encountering an unsupported protocol, there's no need to retry. Chances are, it won't suddenly be supported between retry attempts; error instead. Otherwise, you see something like the following: $ nix-env -i -f git://git@github.com/foo/bar warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 335 ms warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 604 ms warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 1340 ms warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 2685 ms With this change, you now see: $ nix-env -i -f git://git@github.com/foo/bar error: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1)	2020-04-05 09:00:34 -07:00
Eelco Dolstra	4fdec5f61d	Merge remote-tracking branch 'origin/master' into flakes	2020-04-01 00:03:50 +02:00
mlatus	12556e5709	fix placeholder not substituted in passAsFile	2020-03-31 19:40:16 +08:00
John Ericson	51afea3af2	Never cast FileIngestionMethod to or from boolean	2020-03-30 22:31:51 +00:00
John Ericson	c251b011cd	Merge remote-tracking branch 'upstream/master' into enum-FileIngestionMethod	2020-03-30 18:16:44 -04:00
John Ericson	832bd534dc	Store parsed hashes in DerivationOutput ... It's best to detect invalid data as soon as possible, with data types that make storing it impossible.	2020-03-30 11:33:35 -04:00
John Ericson	f5494d9442	Merge remote-tracking branch 'me/enum-FileIngestionMethod' into HEAD	2020-03-30 11:08:13 -04:00
Eelco Dolstra	d15d91cad1	Makefile cleanup	2020-03-30 16:50:13 +02:00
Eelco Dolstra	e322a16523	Remove global -I flags ... (cherry picked from commit 2c692a3b14)	2020-03-30 15:30:19 +02:00
Eelco Dolstra	2c692a3b14	Remove global -I flags	2020-03-30 14:39:33 +02:00
Eelco Dolstra	e0a0ae0467	Move fetchers from libstore to libfetchers	2020-03-30 14:04:53 +02:00
Eelco Dolstra	2287cc6486	Fix segfault	2020-03-30 13:31:55 +02:00
John Ericson	225e62a56a	Replace some bool recursive with a new FileIngestionMethod enum	2020-03-29 15:16:20 -04:00
John Ericson	87b32bab05	Use enum struct and drop prefixes ... This does a few enums; the rest will be gotten in subsequent commits.	2020-03-29 11:23:15 -04:00
Eelco Dolstra	2287e2f279	nix flake info: Show flake subdirectory	2020-03-28 18:05:50 +01:00
John Ericson	e433d4af4c	Extend Rust FFI ... Do idiomatic C++ copy and move constructors for a few things, so wrapping structs' defaults can work.	2020-03-25 16:12:14 -04:00
John Ericson	bcde5456cc	Flip dependency so store-api.hh includes derivations.hh ... I think it makes more sense to define the data model (derivations), before the operations (store api).	2020-03-24 20:39:45 +00:00
Eelco Dolstra	0a10854f85	Misc changes from the flakes branch	2020-03-24 14:34:47 +01:00
Eelco Dolstra	777e21e596	nix path-info --json: Print hash in SRI format ... (cherry picked from commit 442e665d6d)	2020-03-24 14:26:23 +01:00
Eelco Dolstra	f9611c7ae4	buildenv: Eliminate global variables, other cleanup ... (cherry picked from commit b82f75464d)	2020-03-24 14:06:47 +01:00
Eelco Dolstra	edc34cc1a2	Add function for quoting strings ... (cherry picked from commit 7dcf5b011a)	2020-03-24 13:44:04 +01:00
Eelco Dolstra	5a7e7fc35f	Use std::string_view ... (cherry picked from commit 6529490cc1)	2020-03-24 13:26:37 +01:00
Eelco Dolstra	7abe3bde8a	Merge remote-tracking branch 'origin/master' into flakes	2020-03-23 13:27:54 +01:00
Eelco Dolstra	5885e20404	Merge pull request #3429 from LnL7/darwin-sandbox ... darwin sandbox	2020-03-23 09:13:36 +01:00
Daiderd Jordan	2e9bc1245c	sandbox: fix /bin/sh on catalina ... Sadly 10.15 changed /bin/sh to a shim which executes bash, this means it can't be used anymore without also opening up the sandbox to allow bash. Failed to exec /bin/bash as variant for /bin/sh (1: Operation not permitted).	2020-03-20 22:12:30 +01:00
Daiderd Jordan	f6c122aaeb	sandbox: allow pty devices ... Nix now runs builds with a pseudo-terminal to enable colored build output.	2020-03-20 21:58:45 +01:00
Daiderd Jordan	7f2df903d9	libstore: relax default sandbox-paths on darwin	2020-03-20 21:31:20 +01:00
Daiderd Jordan	afb78ebd34	libstore: disable resolve-system-dependencies hook ... This is used to determine the dependency tree of impure libraries so nix knows what paths to open in the sandbox. With the less restrictive defaults it isn't needed anymore.	2020-03-20 21:21:56 +01:00
John Ericson	e317324236	Apply suggestions from code review	2020-03-19 23:38:51 -04:00
John Ericson	d5b3328dd1	Apply suggestions from code review ... Co-Authored-By: Cole Helbling <cole.e.helbling@outlook.com>	2020-03-19 23:37:52 -04:00
John Ericson	f1cf3ab870	hashDerivationModulo: Generalize for multiple fixed ouputs per drv ... See documentattion in header and comments in implementation for details. This is actually done in preparation for floating ca derivations, not multi-output fixed ca derivations, but the distinction doesn't yet mattter. Thanks @cole-h for finding and fixing a bunch of typos.	2020-03-19 10:30:49 -04:00
Eelco Dolstra	8aa354fdfd	Register flake-registry.json as a GC root again	2020-03-19 11:45:34 +01:00
Eelco Dolstra	1c127e6a82	downloadFile(): Use expired file if the download fails	2020-03-19 11:42:50 +01:00
John Ericson	049179ba07	Fix typos ... Thanks @asymmetric I failed to do them all in one batch Co-Authored-By: asymmetric <lorenzo@mailbox.org>	2020-03-18 19:07:05 -04:00
Eelco Dolstra	f6ddf48882	Get rid of downloadCached() ... Everything uses the generic caching system now.	2020-03-18 17:24:19 +01:00
Eelco Dolstra	c5ec95e2c7	tarball.cc: Use ETags	2020-03-18 15:14:23 +01:00
Eelco Dolstra	1e7ce1d6da	tarball / github fetchers: Use generic caching system	2020-03-18 14:08:25 +01:00
Eelco Dolstra	38e360154d	Git: Use unified caching system	2020-03-17 22:35:29 +01:00
Eelco Dolstra	d1165d8791	Require shallow clones to be requested explicitly ... If you do a fetchTree on a Git repository, whether the result contains a revCount attribute should not depend on whether that repository happens to be a shallow clone or not. That would complicate caching a lot and would be semantically messy. So applying fetchTree/fetchGit to a shallow repository is now an error unless you pass the attribute 'shallow = true'. If 'shallow = true', we don't return revCount, even if the repository is not actually shallow. Note that Nix itself is not doing shallow clones at the moment. But it could do so as an optimisation if the user specifies 'shallow = true'. Issue #2988.	2020-03-17 22:35:29 +01:00
Eelco Dolstra	2a4e4f6a6e	Unified fetcher caching system	2020-03-17 22:35:29 +01:00
John Ericson	e5178fd22d	Fix typos ... Thanks @asymmetric! Co-Authored-By: asymmetric <lorenzo@mailbox.org>	2020-03-16 16:40:13 -04:00
Eelco Dolstra	fbcb897e21	Add a test for shallow Git clones ... Also, don't return a revCount anymore for shallow or dirty Git trees, since it's incorrect. Closes #2988.	2020-03-16 13:20:32 +01:00
John Ericson	2be64efb02	Generalize isFixedOutput in preparation for CA drvs ... Today's fixed output derivations and regular derivations differ in a few ways which are largely orthogonal. This replaces `isFixedOutput` with a `type` that returns an enum of possible combinations.	2020-03-15 11:05:37 -04:00
Eelco Dolstra	0c2088d438	Merge remote-tracking branch 'origin/master' into flakes	2020-03-13 17:03:30 +01:00
Eelco Dolstra	b816515f61	Fix ca-references feature check ... Fixes #3406.	2020-03-13 13:15:51 +01:00
Robert Hensing	3f55f8a8fb	pathInfoCache: Respect disk cache TTLs #3398	2020-03-12 10:30:28 +01:00
Eelco Dolstra	e188fe7c6d	Move call-flake.nix into libexpr	2020-03-11 17:04:51 +01:00
Eelco Dolstra	73b6d87e17	Merge remote-tracking branch 'origin/master' into flakes	2020-03-04 13:58:42 +01:00
Eelco Dolstra	d37dc71e3c	nix-build: Fix !<output> handling ... This was broken by 22a754c091. https://hydra.nixos.org/eval/1573669	2020-03-04 13:56:17 +01:00
Eelco Dolstra	75db069f92	Optimise Derivation::unparse() ... In nix-instantiate --dry-run '<nixpkgs/nixos/release-combined.nix>' -A nixos.tests.simple.x86_64-linux this reduces time spent in unparse() from 9.15% to 4.31%. The main culprit was appending characters one at a time to the destination string. Even though the string has enough capacity, push_back() still needs to check this on every call.	2020-03-04 11:44:45 +01:00
Eelco Dolstra	22a754c091	Fix GC failures on bad store path names ... It failed on names like '/nix/store/9ip48nkc9rfy0a4yaw98lp6gipqlib1a-'.	2020-02-28 18:07:10 +01:00
Eelco Dolstra	4a4521f462	Fix nlohmann::json exception	2020-02-20 23:47:02 +01:00
Eelco Dolstra	73c9840569	Restore subdir -> dir ... Got this mixed up somewhere.	2020-02-20 23:44:06 +01:00
Eelco Dolstra	d068f9ffff	Restore subdir support in registries ... Hacky...	2020-02-20 22:14:44 +01:00
Eelco Dolstra	1d99c4ab25	Merge pull request #3229 from Ma27/flakes-fetchgit-worktree-support ... builtins.fetchGit: Fix build when fetching a git worktree	2020-02-19 14:11:18 +01:00
Maximilian Bosch	c169ea5904	builtins.fetchGit: Fix build when fetching a git worktree ... Worktrees[1] are a feature of git which allow you to check out a ref in a different directory. While playing around with flakes I realized that git repositories in a worktree checkout break when trying to build a flake: ``` $ git worktree add ../nixpkgs-flakes nixpkgs-flakes $ cd ../nixpkgs-flakes $ nix build .#hello error: opening directory '/home/ma27/Projects/nixpkgs-flakes/.git/refs/heads': Not a directory ``` This issue has been fixed by determining with `git rev-parse --git-common-dir` where the actual `.git` directory is. Please note that this issue only exists on the `flakes` branch, fetching worktree checkouts with Nix master seems to work fine. [1] https://git-scm.com/docs/git-worktree	2020-02-19 14:00:36 +01:00
Eelco Dolstra	f3505a7899	Merge remote-tracking branch 'origin/master' into flakes	2020-02-19 12:57:45 +01:00
Eelco Dolstra	c4d3674de6	Merge pull request #3353 from tbsmoest/priv_tobias_pr_set_deathsig-1.4 ... Fix PR_SET_PDEATHSIG results in Broken pipe (#2395)	2020-02-19 12:29:12 +01:00
Eelco Dolstra	553e584f92	LocalStore::checkDerivationOutputs(): Improve error message	2020-02-18 17:51:48 +01:00
Eelco Dolstra	b5e3c04c03	Fix URL parser ... Fixes #3361.	2020-02-18 12:51:26 +01:00
Eelco Dolstra	6529490cc1	nix eval-hydra-jobs: Support job names as aggregate constituents ... Fixes https://github.com/NixOS/hydra/issues/715.	2020-02-17 15:53:59 +01:00
Eelco Dolstra	8a78bcf6a2	LocalStore::checkDerivationOutputs(): Improve error message	2020-02-17 15:46:07 +01:00
Eelco Dolstra	6208d24c38	Merge remote-tracking branch 'origin/master' into flakes	2020-02-14 22:42:35 +01:00
Tobias Möst	3e347220c8	Fix PR_SET_PDEATHSIG results in Broken pipe (#2395) ... The ssh client is lazily started by the first worker thread, that requires a ssh connection. To avoid the ssh client to be killed, when the worker process is stopped, do not set PR_SET_PDEATHSIG.	2020-02-14 07:51:44 +01:00
Eelco Dolstra	d8972317fc	Prevent uninitialized StorePath creation	2020-02-13 16:12:16 +01:00
Eelco Dolstra	d4df99a334	Parse narHash attribute for all input types	2020-02-11 23:53:24 +01:00
Eelco Dolstra	442e665d6d	nix path-info --json: Print hash in SRI format	2020-02-11 23:50:16 +01:00
Eelco Dolstra	84a3a5c3cd	Merge remote-tracking branch 'origin/master' into flakes	2020-02-07 13:57:31 +01:00
Eelco Dolstra	d82b78bf51	Fix segfault in gcc on i686-linux ... src/libstore/ssh-store.cc: In constructor 'nix::SSHStore::SSHStore(const string&, const Params&)': src/libstore/ssh-store.cc:31:21: internal compiler error: Segmentation fault compress) ^ Please submit a full bug report, with preprocessed source if appropriate. https://hydra.nixos.org/build/111545609	2020-02-07 13:01:48 +01:00
Eelco Dolstra	379852a152	Registry: Use attr notation instead of URLs	2020-02-06 14:27:31 +01:00
Eelco Dolstra	022287060b	Merge remote-tracking branch 'origin/master' into flakes	2020-02-05 15:41:51 +01:00
Eelco Dolstra	9d7fb62db6	Add option --commit-lock-file	2020-02-05 14:48:49 +01:00
Eelco Dolstra	e2213d77a2	Set 'ref' properly for local trees	2020-02-04 21:55:57 +01:00
Eelco Dolstra	db88cb401b	Merge pull request #3344 from LnL7/ssh-ng-remote-params ... ssh-store: add remote-store and remote-program query params	2020-02-04 10:10:08 +01:00
Daiderd Jordan	8745c63d3c	ssh-store: add remote-store and remote-program query params ... Brings the functionality of ssh-ng:// in sync with the legacy ssh:// implementation. Specifying the remote store uri enables various useful things. eg. $ nix copy --to ssh-ng://cache?remote-store=file://mnt/cache --all	2020-02-03 23:22:28 +01:00
Eelco Dolstra	0a4e911cf4	Install headers in the correct location	2020-02-03 18:04:09 +01:00
Eelco Dolstra	a2628b43bb	Fix URL parser ... Fixes #3062.	2020-02-03 15:27:26 +01:00
Eelco Dolstra	d070e1c532	Fix parsing of '#nixosConfigurations."hostname".config...' ... This is not strictly speaking valid but who cares.	2020-02-03 14:29:34 +01:00
Eelco Dolstra	d5334c466b	Automatically do git/hg add on flake.lock	2020-02-02 16:33:22 +01:00
Eelco Dolstra	a9ebc3ea5d	Remove the git+ and hg+ prefixes from structured input refs	2020-02-02 13:06:00 +01:00
Eelco Dolstra	fad9faf354	Add TreeInfo::computeStorePath()	2020-02-02 12:29:53 +01:00
Eelco Dolstra	7bcc9f2aaf	Add convenience overload for toRealPath()	2020-02-02 12:28:56 +01:00
Eelco Dolstra	958ec5de56	Cleanup	2020-02-02 11:31:58 +01:00
Eelco Dolstra	887730aab3	Remove superfluous TreeInfo::rev field	2020-02-01 23:54:20 +01:00
Eelco Dolstra	b9d64f9318	Record TreeInfo in the lock file ... Necessary for #3253.	2020-02-01 23:33:44 +01:00
Eelco Dolstra	8451298b35	Factor out TreeInfo	2020-02-01 22:06:55 +01:00
Eelco Dolstra	185c3c8240	Cleanup	2020-01-31 19:35:28 +01:00
Eelco Dolstra	8414685c0f	Change lock file format to use an attribute representation of flake refs rather than URLs	2020-01-31 19:16:40 +01:00
Eelco Dolstra	88b44b1e94	Fix flake update check	2020-01-29 23:14:23 +01:00
Eelco Dolstra	7be1a07a45	Merge pull request #3335 from domenkozar/retry-429 ... retry on HTTP status code 429	2020-01-29 16:22:46 +01:00
Domen Kožar	48ddb8e481	retry on HTTP status code 429	2020-01-29 11:47:39 +01:00
Eelco Dolstra	ab41e9d543	Merge remote-tracking branch 'origin/master' into flakes	2020-01-28 13:19:30 +01:00
Eelco Dolstra	1af7b94c1d	Add support for tarball flake inputs ... For example, $ nix flake info https://github.com/edolstra/dwarffs/archive/master.tar.gz Fixes #2929.	2020-01-28 13:11:02 +01:00
Eelco Dolstra	5046233b5a	Add Mercurial tests	2020-01-27 13:45:49 +01:00
Eelco Dolstra	cc22cf662b	Respect lock files of inputs + fine-grained lock file control ... When computing a lock file, we now respect the lock files of flake inputs. This is important for usability / reproducibility. For example, the 'nixops' flake depends on the 'nixops-aws' and 'nixops-hetzner' repositories. So when the 'nixops' flake is used in another flake, we want the versions of 'nixops-aws' and 'nixops-hetzner' locked by the the 'nixops' flake because those presumably have been tested. This can lead to a proliferation of versions of flakes like 'nixpkgs' (since every flake's lock file could depend on a different version of 'nixpkgs'). This is not a major issue when using Nixpkgs overlays or NixOS modules, since then the top-level flake composes those overlays/modules into *its* version of Nixpkgs and all other versions are ignored. Lock file computation has been made a bit more lazy so it won't try to fetch all those versions of 'nixpkgs'. However, in case it's necessary to minimize flake versions, there now are two input attributes that allow this. First, you can copy an input from another flake, as follows: inputs.nixpkgs.follows = "dwarffs/nixpkgs"; This states that the calling flake's 'nixpkgs' input shall be the same as the 'nixpkgs' input of the 'dwarffs' input. Second, you can override inputs of inputs: inputs.nixpkgs.url = github:edolstra/nixpkgs/<hash>; inputs.nixops.inputs.nixpkgs.url = github:edolstra/nixpkgs/<hash>; or equivalently, using 'follows': inputs.nixpkgs.url = github:edolstra/nixpkgs/<hash>; inputs.nixops.inputs.nixpkgs.follows = "nixpkgs"; This states that the 'nixpkgs' input of the 'nixops' input shall be the same as the calling flake's 'nixpkgs' input. Finally, at '-v' Nix now prints the changes to the lock file, e.g. $ nix flake update ~/Misc/eelco-configurations/hagbard inputs of flake 'git+file:///home/eelco/Misc/eelco-configurations?subdir=hagbard' changed: updated 'nixpkgs': 'github:edolstra/nixpkgs/7845bf5f4b3013df1cf036e9c9c3a55a30331db9' -> 'github:edolstra/nixpkgs/03f3def66a104a221aac8b751eeb7075374848fd' removed 'nixops' removed 'nixops/nixops-aws' removed 'nixops/nixops-hetzner' removed 'nixops/nixpkgs'	2020-01-24 22:05:11 +01:00
Robin Gloster	f8dbde0813	structured-attrs: chown .attrs.* files to builder ... Otherwise `chmod .`'ing the build directory doesn't work anymore, which is done in nixpkgs if sourceRoot is set to '.'.	2020-01-23 17:38:07 +01:00
Eelco Dolstra	9a5ca802c7	clang fixes ... https://hydra.nixos.org/build/110757171	2020-01-22 21:26:19 +01:00
Eelco Dolstra	d506bd587a	Fix clang warning	2020-01-22 21:20:01 +01:00
Eelco Dolstra	b5c9dbc84f	Fix --override-flake and add a test	2020-01-22 20:00:58 +01:00
Eelco Dolstra	b33b94748c	Convert fetchMercurial to a input type ... This enables Mercurial flakes. It also fixes a bug in pure mode where you could use a branch/tag name rather than a revision.	2020-01-21 23:49:32 +01:00
Eelco Dolstra	9f4d8c6170	Pluggable fetchers ... Flakes are now fetched using an extensible mechanism. Also lots of other flake cleanups.	2020-01-21 22:56:04 +01:00
Eelco Dolstra	6fadb3fc03	Merge remote-tracking branch 'origin/master' into flakes	2020-01-21 21:18:52 +01:00
Eelco Dolstra	aef635da78	Fix derivation computation with __structuredAttrs and multiple outputs ... Fixes error: derivation '/nix/store/klivma7r7h5lndb99f7xxmlh5whyayvg-zlib-1.2.11.drv' has incorrect output '/nix/store/fv98nnx5ykgbq8sqabilkgkbc4169q05-zlib-1.2.11-dev', should be '/nix/store/adm7pilzlj3z5k249s8b4wv3scprhzi1-zlib-1.2.11-dev'	2020-01-21 21:14:13 +01:00
Daiderd Jordan	8b3217f832	build: remove warning when in sandboxing test mode ... Introduced in 66fccd5832, but somehow breaks the secure-drv-outputs test.	2020-01-13 22:09:18 +01:00
John Ericson	d64ab5131c	unbreak build without pch	2020-01-13 21:45:33 +01:00