Commit graph

4042 commits

Author SHA1 Message Date
Valentin Gagarin 24b781773f fix random docs errors
remove link to the contributing guide from user documentation.
it doesn't help here, and the target at first glance shows redundant
information.
2023-12-02 03:02:59 +01:00
John Ericson 91b6833686 Move tests to separate directories, and document
Today, with the tests inside a `tests` intermingled with the
corresponding library's source code, we have a few problems:

- We have to be careful that wildcards don't end up with tests being
  built as part of Nix proper, or test headers being installed as part
  of Nix proper.

- Tests in libraries but not executables is not right:

  - It means each executable runs the previous unit tests again, because
    it needs the libraries.

  - It doesn't work right on Windows, which doesn't want you to load a
    DLL just for the side global variable . It could be made to work
    with the dlopen equivalent, but that's gross!

This reorg solves these problems.

There is a remaining problem which is that sibbling headers (like
`hash.hh` the test header vs `hash.hh` the main `libnixutil` header) end
up shadowing each other. This PR doesn't solve that. That is left as
future work for a future PR.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-12-01 10:48:58 -05:00
John Ericson d59bdbe4fd Add two missing #include "nar-info.hh"
GitHub's racy CI caused this oversight to sneak through.
2023-12-01 10:20:19 -05:00
Robert Hensing fcf09813c6
Merge pull request #6236 from obsidiansystems/store-dir-config
Factor out `StoreDirConfig`
2023-12-01 15:38:14 +01:00
John Ericson 82359eba6b
Merge pull request #9233 from bouk/bouk/apply-config-inner
config: add included files into parsedContents before applying
2023-12-01 08:23:32 -05:00
Valentin Gagarin 4781e7fa70 Document each store type on its own page
This makes for more useful manual table of contents, that displays the
information at a glance.

The `nix help-stores` command is kept as-is, even though it will show up
in the manual with the same information as these pages due to the way it
is written as a "`--help`-style" command. Deciding what to do with that
command is left for a later PR.

This change also lists all store types at the top of the respective overview page.

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems
2023-12-01 01:27:52 +01:00
Eelco Dolstra b6a3fde6b7
Merge pull request #9465 from obsidiansystems/build-dir
Use `buildprefix` in a few more places
2023-11-30 15:28:37 +01:00
John Ericson a7115a47ef Improve ACL clearing support (fixing FreeBSD build)
The problem was that f880469173 forgot
that the `#include <sys/xattr.h>` was guarded by an `#ifdef __linux__`.

However, the build failure was only on FreeBSD --- turns out other
platforms have this header too!

The fix therefore uses a new configure check so we properly clear ACLs
on more platforms.
2023-11-29 21:09:19 -05:00
John Ericson 52e0911302 Use buildprefix in a few more places
`installcheck` doesn't yet work, but the rest of the build can now
happen mostly inside a separate build directory.

Progress on #9342

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-11-29 19:49:07 -05:00
ivan770 54b6847655 doc: fix machine-specific capabilities leaking 2023-11-24 11:17:35 -05:00
Robert Hensing 030fb86dd1
Merge pull request #9394 from NixOS/separate-file-canonicalise
Put `canonicaliseTimestampAndPermissions` in its own header/file
2023-11-22 18:27:52 +01:00
Eelco Dolstra 38844943d0
Merge pull request #9413 from edolstra/fetchToStore-cache
Persistently cache `InputAccessor::fetchToStore()`
2023-11-22 12:19:25 +01:00
Matej Urbas 06b8902562 MountedSSHStore: stores on shared filesystems 2023-11-21 13:34:01 -05:00
Matej Urbas 226b0f3956 Extend the worker protocol with wopAddPermRoot 2023-11-21 13:26:55 -05:00
John Ericson f880469173 Put canonicaliseTimestampAndPermissions in its own header/file
It is not inherently tied to `LocalStore`, it could probably even go in
`libnixutil`. Functions not attached to `LocalStore` should not be
declared in `local-store.hh`.

I am moving it to facilitate experimenting for #9344. If
canonicalisation should be done client-side in client-side builds, there
wouldn't be a `LocalStore` at all so having to include that header to
get this freestanding function is cumbersome and wrong.

Perhaps canonicalisation should still be done server-side for security
reasons --- I don't mean to make that judgement call now --- but even if
so, this freestanding function still isn't connected to `LocalStore` so
while less urgent it is still better to move out of this header.
2023-11-21 12:57:59 -05:00
Eelco Dolstra 99d5204baa Persistently cache InputAccessor::fetchToStore()
This avoids repeated copying of the same source tree between Nix
invocations. It requires the accessor to have a "fingerprint" (e.g. a
Git revision) that uniquely determines its contents.
2023-11-20 20:04:37 +01:00
Robert Hensing a5e51a9e02 refactor Worker::childStarted/Terminated: use switch
Preparation for RFC 92 dynamic derivations.
2023-11-20 13:32:48 +01:00
Robert Hensing 7ac39ff05c refactor Store::buildPaths: convert to string earlier
Preparation for RFC 92 dynamic derivations.
2023-11-20 13:11:58 +01:00
Théophane Hufschmitt b1e7d7cad6
Merge pull request #9280 from R-VdP/rvdp/fix_remote_logging_phase_reporting
Include phase reporting in log file for ssh-ng builds
2023-11-17 14:37:09 +01:00
Théophane Hufschmitt 7ba4e073e8
Merge pull request #6469 from gbpdt/fix/skipped_build_locking
Unlock output paths when a derivation is already built
2023-11-16 21:59:25 +01:00
Théophane Hufschmitt fabae98ab4
Merge pull request #9360 from obsidiansystems/install-unit-tests
Allow installing unit tests
2023-11-16 21:25:21 +01:00
John Ericson 6c8f4ef350 Allow installing unit tests
Closes #9343

See that issue for motivation.

Installing these is disabled by default, but we enable it (and the
additional output we want isntall these too so as not to clutter the
existing ones) to use in cross builds and dev shells.
2023-11-16 09:55:42 -05:00
Robert Hensing 9fa133dde5 readProcLink: Replace unnecessary value judgement by actual info 2023-11-16 15:38:21 +01:00
Théophane Hufschmitt ba3cb4a049 Remove all the occurences of VLAs
There's generally no strict reason for using them, and they are somewhat
fishy, so let's avoid them.
2023-11-16 12:27:30 +01:00
Andreas Stührk ad99c8950b Update comment to reflect bind mounts are now used for store in chroot 2023-11-13 23:29:48 +01:00
Bouke van der Bijl d6898cd58b Move applyConfigFile to lambda inside libstore 2023-11-13 17:14:05 +01:00
Graham Christensen fd5a4a8467 nix upgrade-nix: make the source URL an option
This new option enables organizations to more easily manage their Nix
fleet's deployment, and ensure a consistent and planned rollout of Nix
upgrades.
2023-11-10 12:12:28 -05:00
Jacek Galowicz df8bfe84cc Fix consts and casts 2023-11-08 17:29:55 +01:00
Jacek Galowicz c581143e0c Use structured binding for json iteration 2023-11-08 17:29:55 +01:00
Jacek Galowicz 77dceb2844 Drop obsolete assert and cast 2023-11-08 17:29:55 +01:00
Jacek Galowicz 6a47629530 Fix initialization of struct members (wrong order) 2023-11-08 17:29:55 +01:00
John Ericson dcc49744ed
Merge pull request #9293 from ThinkChaos/ssh-misc-improvments
SSH small improvments
2023-11-06 13:33:51 -05:00
John Ericson cc46ea1630 Make nix path-info --json return an object not array
Before it returned a list of JSON objects with store object information,
including the path in each object. Now, it maps the paths to JSON
objects with the metadata sans path.

This matches how `nix derivation show` works.

Quite hillariously, none of our existing functional tests caught this
change to `path-info --json` though they did use it. So just new
functional tests need to be added.
2023-11-06 11:06:31 -05:00
John Ericson a7212e169b Include compression in the NarInfo JSON format
It was forgotten before.
2023-11-06 11:06:31 -05:00
John Ericson 937e02e7b9 Shuffle ValidPathInfo JSON rendering
`Store::pathInfoToJSON` was a rather baroque functions, being full of
parameters to support both parsed derivations and `nix path-info`. The
common core of each, a simple `dValidPathInfo::toJSON` function, is
factored out, but the rest of the logic is just duplicated and then
specialized to its use-case (at which point it is no longer that
duplicated).

This keeps the human oriented CLI logic (which is currently unstable)
and the core domain logic (export reference graphs with structured
attrs, which is stable), separate, which I think is better.
2023-11-06 11:06:31 -05:00
John Ericson 0b0d1b5214 Add comparison functions for NarInfo
We will need these for tests.
2023-11-06 09:51:21 -05:00
ThinkChaos 2fb49759b8
fix(ssh): log first line of stdout
Spent a while debugging why `nix-copy-closure` wasn't working anymore
and it was my shell RC printing something I added for debug.
Hopefully this can save someone else some time.
2023-11-06 08:46:19 -05:00
ThinkChaos 6472c3bf0d
fix(ssh): extraneous master processes 2023-11-06 08:45:14 -05:00
John Ericson ac89bb064a Split up util.{hh,cc}
All OS and IO operations should be moved out, leaving only some misc
portable pure functions.

This is useful to avoid copious CPP when doing things like Windows and
Emscripten ports.

Newly exposed functions to break cycles:

 - `restoreSignals`
 - `updateWindowSize`
2023-11-05 12:20:02 -05:00
John Ericson 2678b51b31 Narrower scope for nativeSystem
I don't think we need a CPP defininition and a header entry, and this
way allows constant expression elimination.
2023-11-05 12:18:28 -05:00
John Ericson dde1d86338 Restrict some code to StoreDirConfig
- part of eval cache
 - part of derivations
 - derived path
 - store path with outputs
 - serializers
2023-11-04 19:05:36 -04:00
John Ericson e97ac09abe Factor out StoreDirConfig
More progress on #5729.
2023-11-04 19:05:36 -04:00
Ramses 60b363936d
libstore/ssh-ng: Fix phase reporting in log files.
When doing local builds, we get phase reporting lines in the log file,
they look like '@nix {"action":"setPhase","phase":"unpackPhase"}'.
With the ssh-ng protocol, we do have access to these messages, but since we
are only including messages of type resBuildLogLine in the logs, the phase
information does not end up in the log file.

The phase reporting could probably be improved altoghether (it looks like it
is kind of accidental that these JSON messages for phase reporting show up
but others don't, just because they are actually emitted by nixpkgs' stdenv),
but as a first step I propose to make ssh-ng behave in the same way as local builds do.
2023-11-03 12:30:25 +01:00
Eelco Dolstra b0455e9931 Fix uninitialized variable warnings on i686-linux
https://hydra.nixos.org/build/239849607
2023-11-03 12:04:20 +01:00
Eelco Dolstra 55dd1244d2 parseDerivation(): Fix warning about uninitialized 'version' variable 2023-11-03 12:04:20 +01:00
John Ericson d15c3a33e6 Don't use std::invocable C++ concept yet
It s not supported on all platforms yet. Can revert this once it is.
2023-11-02 12:06:29 -04:00
John Ericson b107431816 Systematize characterization tests a bit more
Deduplicating code moreover enforcing the pattern means:

 - It is easier to write new characterization tests because less boilerplate

 - It is harder to mess up new tests because there are fewer places to
   make mistakes.

Co-authored-by: Jacek Galowicz <jacek@galowicz.de>
2023-11-02 12:06:08 -04:00
John Ericson 90de958637
Merge pull request #9265 from obsidiansystems/better-parse-sink
Make `ParseSink` a bit better
2023-11-02 09:28:29 -04:00
Eelco Dolstra 5223114c93
Merge pull request #9269 from edolstra/unify-accessor
Unify `FSAccessor` and `SourceAccessor`
2023-11-02 14:23:10 +01:00
Eelco Dolstra e47984ce0b
Fix whitespace
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-11-01 20:19:08 +01:00