lix/src/libstore/download.hh
Eelco Dolstra 75989bdca7 Make computeFSClosure() single-threaded again
The fact that queryPathInfo() is synchronous meant that we needed a
thread for every concurrent binary cache lookup, even though they end
up being handled by the same download thread. Requiring hundreds of
threads is not a good idea. So now there is an asynchronous version of
queryPathInfo() that takes a callback function to process the
result. Similarly, enqueueDownload() now takes a callback rather than
returning a future.

Thus, a command like

  nix path-info --store https://cache.nixos.org/ -r /nix/store/slljrzwmpygy1daay14kjszsr9xix063-nixos-16.09beta231.dccf8c5

that returns 4941 paths now takes 1.87s using only 2 threads (the main
thread and the downloader thread). (This is with a prewarmed
CloudFront.)
2016-09-16 18:54:14 +02:00

77 lines
2 KiB
C++
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

#pragma once
#include "types.hh"
#include "hash.hh"
#include <string>
#include <future>
namespace nix {
struct DownloadRequest
{
std::string uri;
std::string expectedETag;
bool verifyTLS = true;
enum { yes, no, automatic } showProgress = yes;
bool head = false;
size_t tries = 1;
unsigned int baseRetryTimeMs = 250;
DownloadRequest(const std::string & uri) : uri(uri) { }
};
struct DownloadResult
{
bool cached;
std::string etag;
std::string effectiveUrl;
std::shared_ptr<std::string> data;
};
class Store;
struct Downloader
{
/* Enqueue a download request, returning a future to the result of
the download. The future may throw a DownloadError
exception. */
virtual void enqueueDownload(const DownloadRequest & request,
std::function<void(const DownloadResult &)> success,
std::function<void(std::exception_ptr exc)> failure) = 0;
std::future<DownloadResult> enqueueDownload(const DownloadRequest & request);
/* Synchronously download a file. */
DownloadResult download(const DownloadRequest & request);
/* Check if the specified file is already in ~/.cache/nix/tarballs
and is more recent than tarball-ttl seconds. Otherwise,
use the recorded ETag to verify if the server has a more
recent version, and if so, download it to the Nix store. */
Path downloadCached(ref<Store> store, const string & uri, bool unpack, string name = "",
const Hash & expectedHash = Hash(), string * effectiveUri = nullptr);
enum Error { NotFound, Forbidden, Misc, Transient, Interrupted };
};
/* Return a shared Downloader object. Using this object is preferred
because it enables connection reuse and HTTP/2 multiplexing. */
ref<Downloader> getDownloader();
/* Return a new Downloader object. */
ref<Downloader> makeDownloader();
class DownloadError : public Error
{
public:
Downloader::Error error;
DownloadError(Downloader::Error error, const FormatOrString & fs)
: Error(fs), error(error)
{ }
};
bool isUri(const string & s);
}