summaryrefslogtreecommitdiff
path: root/guix/scripts/substitute.scm
AgeCommit message (Collapse)Author
2023-12-04substitute: Do not exit when failing to find a nar.Ludovic Courtès
Fixes <https://issues.guix.gnu.org/67575>. * guix/scripts/substitute.scm (process-substitution/fallback): Use ‘report-error’ instead of ‘leave’. Write status line to PORT. * tests/substitute.scm ("substitute, narinfo is available but nar is missing"): Adjust accordingly. Change-Id: Ic7297dbd563c007111ec2167c8d52505a07d4822
2023-10-17scripts: substitute: Handle EPIPE errors when displaying help.Simon Tournier
* guix/scripts/substitute.scm (%options): Handle EPIPE errors when displaying help.
2023-06-08substitute: Gracefully retry after failed partial downloads.Ludovic Courtès
Fixes <https://issues.guix.gnu.org/63443>. Reported by Attila Lendvai <attila@lendvai.name>. * guix/scripts/substitute.scm (catch-system-error): New macro. (download-nar): Add call to 'delete-file-recursively'. * tests/substitute.scm ("substitute, previous partial download around"): New test.
2023-06-08substitute: Delete cached narinfos more than two-month old.Ludovic Courtès
This allows 'guix substitute' to shrink the cache a bit more, which saves space and improves performance of cache-cleanup phases since fewer entries need to be traversed. * guix/scripts/substitute.scm (cached-narinfo-expiration-time): Define 'max-ttl' and use it as an upper bound.
2023-05-30substitute: If a server's nar URL is 404, try the next one(s).Ludovic Courtès
If a substitute server advertises in its narinfo, for example, both a /zstd and a /lzip URL but the /zstd URL is unreachable, try the /lzip URL. Fixes <https://issues.guix.gnu.org/63634>. * guix/narinfo.scm (narinfo-preferred-uris): New procedure. (narinfo-best-uri): Rebase on top of it. * guix/scripts/substitute.scm (download-nar)[try-fetch]: New procedure. Use 'narinfo-preferred-uris' and 'try-fetch' to attempt all the URLs of NARINFO. * tests/substitute.scm (request-substitution): Remove 'parameterize'. Delete DESTINATION. ("substitute, preferred nar URL is 404, other is 200"): New test.
2023-05-22substitute: Rethrow with 'raise-exception', not 'throw'.Ludovic Courtès
Rethrowing with 'throw' doesn't work as intended when the exception being rethrown is a SRFI-34 exception. Fixes <https://issues.guix.gnu.org/55820>. * guix/scripts/substitute.scm (kind-and-args-exception?): New variable. (call-with-cached-connection): Rewrite using 'guard' instead of 'catch' and 'raise' instead of 'throw'. (system-error?): Use 'kind-and-args-exception?' instead of local definition.
2023-04-19substitute: Download nar from another server upon ETIMEDOUT.Ludovic Courtès
Previously, 'guix substitute' would fail abruptly with something like: guix substitute: warning: while fetching https://ci.guix.gnu.org/nar/lzip/…-example: server is somewhat slow guix substitute: warning: try `--no-substitutes' if the problem persists guix substitute: error: connect*: Connection timed out substitution of /gnu/store/…-example failed * guix/scripts/substitute.scm (network-error?): Add ETIMEDOUT.
2023-03-13guix: Strip #:use-module lists.Ludovic Courtès
This was obtained by setting up this environment: guix shell -D guix --with-input=guile@3.0.9=guile-next \ --with-commit=guile-next=e2ed33ef0445c867fe56c247054aa67e834861f2 -- make -j5 then adding 'unused-module' to (@@ (guix build compiler) %warnings), building, and checking all the "unused module" warnings and removing those that were definitely unused.
2023-01-09substitute: Parse '_NIX_OPTIONS' once.Ludovic Courtès
* guix/scripts/substitute.scm (find-daemon-option): Arrange to call 'daemon-options' only once.
2022-12-05substitute: Fix typo in 'open-connection-for-uri/cached' docstring.Ludovic Courtès
* guix/scripts/substitute.scm (open-connection-for-uri/cached): Fix docstring.
2022-09-28substitute: Retry downloading when a nar is unavailable.Ludovic Courtès
Fixes <https://issues.guix.gnu.org/57978> Reported by Attila Lendvai <attila@lendvai.name>. Previously, if a narinfo was available but its corresponding nar was missing (for instance because the narinfo was cached and the server became unreachable in the meantime), 'guix substitute --substitute' would try to download the nar from its preferred location and abort when that fails. This change forces one retry with each of the URLs. * guix/scripts/substitute.scm (download-nar): Do not catch 'http-get-error?' exceptions. (system-error?, network-error?, process-substitution/fallback): New procedures. (process-substitution): Call 'process-substitution/fallback' upon 'network-error?'. * tests/substitute.scm ("substitute, first URL has narinfo but lacks nar, second URL unauthorized") ("substitute, first URL has narinfo but nar is 404, both URLs authorized") ("substitute, first URL has narinfo but nar is 404, one URL authorized") ("substitute, narinfo is available but nar is missing"): New tests.
2022-09-28substitute: Split nar download.Ludovic Courtès
* guix/scripts/substitute.scm (download-nar): New procedure, with most of the code moved from... (process-substitution): ... here. Call it.
2022-06-26substitute: Use SRFI-71 instead of SRFI-11.Ludovic Courtès
* guix/scripts/substitute.scm (display-narinfo-data) (open-connection-for-uri/cached) (process-substitution): Use SRFI-71 instead of SRFI-11.
2022-05-27substitute: Check for --help or --version options early.yarl-baudig@mailoo.org
Running `guix archive --authorize` sets /etc/guix/acl to 600 via with-atomic-file-output via mkstemp!. Then running `guix substitute --help/--version` fails on "permission denied". While "guix substitute" is an internal tool, the options --help and --version exist and you should be able to run those from the command line. * guix/scripts/substitute.scm (guix-substitute): Earlier check for --help or --version. Signed-off-by: Ludovic Courtès <ludo@gnu.org>
2021-12-29substitute: Document the limited UI.Tobias Geerinckx-Rice
* guix/scripts/substitute.scm (show-help): Accept exactly one OPTION.
2021-07-16substitute: Fix handling of short option "-h".Hartmut Goebel
The short option was listed in the help-text, but not recognized.
2021-06-25guix substitute: Adjust comment about GnuTLS bug.Ludovic Courtès
* guix/scripts/substitute.scm (call-with-cached-connection): Adjust comment.
2021-06-18Start enabling substitutes from bordeaux.guix.gnu.org.Christopher Baines
In addition to substitutes from ci.guix.gnu.org. There are more changes that can be made in the future, but these changes seem like a good start. * config-daemon.ac (guix_substitute_urls): Add https://bordeaux.guix.gnu.org. * guix/scripts/substitute.scm (%default-substitute-urls): Add http://bordeaux.guix.gnu.org. * guix/store.scm (%default-substitute-urls): Add bordeaux.guix.gnu.org. * doc/guix.texi: Adjust accordingly. * doc/contributing.texi: Adjust accordingly.
2021-06-06scripts: substitute: Cache connection when looking for narinfos.Mathieu Othacehe
The process-substitution procedure is opening two distinct connections. The first one when looking for narinfo by calling lookup-narinfo and the other one when fetching nar files. Cache the connection when looking for narinfos so that process-substitution only opens one connection. * guix/scripts/substitute.scm (lookup-narinfo): Cache connection by using open-connection-for-uri/cached.
2021-06-01publish: Add keep-alive support when sending nar.Mathieu Othacehe
The default Guile web server implementation supports the keep alive mechanism. However, in our custom http-write implementation, the connection is unconditionally close after sending nar files. To prevent that, when supported, add the client port to the server poll set so that further requests can be handled without closing the connection. * guix/scripts/publish.scm (nar-response-port): Rename it into ... (nar-compressed-port): ... this procedure. Operate directly on a given PORT. (http-write): Add keep-alive support when sending nar files. * guix/scripts/substitute.scm (process-substitution): Pass the download size to the progress-report-port procedure so that it doesn't block reading from the input port when keep-alive is supported.
2021-04-24http-client, substitute: Gracefully handle GnuTLS EAGAIN/EINTR.Ludovic Courtès
Partly fixes <https://bugs.gnu.org/47867>. Reported by Florian Pelz <pelzflorian@pelzflorian.de>. In GnuTLS up to 3.7.1 included, GNUTLS_E_AGAIN and GNUTLS_E_INTERRUPTED are not handled by 'write_to_session_record_port' and could be thrown at the caller. This patch works around that by dropping connections altogether and restarting when this happens. * guix/http-client.scm (false-if-networking-error): Swallow ERROR/AGAIN and ERROR/INTERRUPTED. * guix/scripts/substitute.scm (call-with-cached-connection): Likewise.
2021-04-09daemon: 'guix substitute' replies on FD 4.Ludovic Courtès
This avoids the situation where error messages would unintentionally go to stderr and be wrongfully interpreted as a reply by the daemon. Fixes <https://bugs.gnu.org/46362>. This is a followup to ee3226e9d54891c7e696912245e4904435be191c. * guix/scripts/substitute.scm (display-narinfo-data): Add 'port' parameter and honor it. (process-query): Likewise. (process-substitution): Likewise. (%error-to-file-descriptor-4?, with-redirected-error-port): Remove. (%reply-file-descriptor): New variable. (guix-substitute): Remove use of 'with-redirected-error-port'. Define 'reply-port' and pass it to 'process-query' and 'process-substitution'. * nix/libstore/build.cc (SubstitutionGoal::handleChildOutput): Swap 'builderOut' and 'fromAgent'. * nix/libstore/local-store.cc (LocalStore::getLineFromSubstituter): Likewise. * tests/substitute.scm <top level>: Set '%reply-file-descriptor' rather than '%error-to-file-descriptor-4?'.
2021-03-31substitute: Emit a single newline upon completion.Ludovic Courtès
The immediate effect is that, with '--verbosity=1', only two lines are displayed for each substitute, instead of two lines followed by an empty line. * guix/scripts/substitute.scm (process-substitution): Emit a single newline upon completion when PRINT-BUILD-TRACE? is true.
2021-03-29Fix substitutes server discovery.Mathieu Othacehe
This is a follow-up of bc3896db25c788c181c7bcd65754e7cd378e9d9f. * guix/scripts/substitute.scm (%local-substitute-urls): Test for "true" instead of "yes".
2021-03-21substitute: Choose compression method based on past CPU usage.Ludovic Courtès
This stems from the observation that substitute download can be CPU-bound when high-speed networks are in use: https://lists.gnu.org/archive/html/guix-devel/2020-12/msg00177.html * guix/narinfo.scm (decompresses-faster?): New procedure. (narinfo-best-uri): Add #:fast-decompression?. * guix/scripts/substitute.scm (%prefer-fast-decompression?): New variable. (call-with-cpu-usage-monitoring): New procedure. (with-cpu-usage-monitoring): New macro. (display-narinfo-data, process-substitution): Pass #:fast-decompression? to 'narinfo-best-uri'. (process-substitution): Wrap 'restore-file' call in 'with-cpu-usage-monitoring'. Set '%prefer-fast-decompression?'.
2021-03-17scripts: substitute: Tweak error reporting in process-substitution.Christopher Baines
The call-with-connection-error-handling was added in 20c08a8a45d0f137ead7c05e720456b2aea44402, but that error handling was previously inside of open-connection-for-uri/maybe, which is related to (call-)with-cached-connection which was used in process-substitution, but only actually used with call-with-cached-connection when used in fetch-narinfos. There's some handling for similar errors within with-networking, which is used within process-substitution. * guix/scripts/substitute.scm (process-substitution): Remove call-with-connection-error-handling call.
2021-03-17scripts: substitute: Add back some error handling.Christopher Baines
In f50f5751fff4cfc6d5abba9681054569694b7a5c, the way fetch was called within process-substitution was changed. As call-with-cached-connection actually includes important error handling for the opening of a HTTP request, this change removed some error handling. This commit adds that back. Fixes <https://bugs.gnu.org/47157>. * guix/scripts/substitute.scm (call-with-cached-connection): New procedure. (with-cached-connection): New syntax rule. (process-substitution): Retry once for some errors when making HTTP requests to fetch substitutes.
2021-03-09guix: substitutes: Make progress reporting configurable.Christopher Baines
Rather than always outputting to (current-error-port) in lookup-narinfos (which is called from within lookup-narinfos/diverse), take a procedure which should return a progress reporter, and defer any output to that. As this is now general purpose code, make the default behaviour to output nothing. Maintain the current behaviour of the substitute script by moving the progress reporter implementation there, and passing it in when calling lookup-narinfos/diverse. These changes should be generally useful, but I'm particularly looking at getting guix weather to do progress reporting differently, with this new flexibility. * guix/substitutes.scm (fetch-narinfos): Take a procedure to make a progress-reporter, and use that rather than the hardcoded behaviour. (lookup-narinfos): Add #:make-progress-reporter keyword argument, and pass this through to fetch-narinfos. (lookup-narinfos/diverse): Add a #:make-progress-reporter keyword argument, and pass this through to lookup-narinfos. * guix/scripts/substitute.scm (process-query): Pass a progress-reporter to lookup-narinfos/diverse.
2021-03-05guix: Split (guix substitutes) from (guix scripts substitute).Christopher Baines
This means there's a module for working with substitutes, rather than all the code sitting in the script. The need for this can be seen with the weather and challenge scripts, that now don't have to use code from the substitute script, but can instead use the substitute module. The separation here between the actual functionality of the substitute script and the underlying functionality used both there and elsewhere should make maintenance easier moving forward. This commit just moves code, none of the code should have been changed significantly. * guix/scripts/substitute.scm (%narinfo-cache-directory, %narinfo-ttl, %narinfo-negative-ttl, %narinfo-transient-error-ttl, %unreachable-hosts): Move variables to guix/substitutes.scm. (narinfo-cache-file, cached-narinfo, cache-narinfo!, narinfo-request, read-to-eof, call-with-connection-error-handling, fetch-narinfos, lookup-narinfos, lookup-narinfos/diverse): Move procedures to guix/substitutes.scm. * guix/substitutes.scm: New file. * Makefile.am: Add it. * guix/narinfo.scm: Remove redundant module. * guix/scripts/challenge.scm: Change (guix scripts substitute) to (guix substitutes). * guix/scripts/weather.scm: Change (guix scripts substitute) to (guix substitutes).
2021-02-24substitute: Ensure backtraces go to file descriptor 4.Ludovic Courtès
Fixes <https://bugs.gnu.org/46362>. Reported by Zhu Zihao <all_but_last@163.com>. Previously, backtraces due to uncaught exceptions would always to go file descriptor 2; the daemon would read it and error out with something like: error: got unexpected path `Backtrace:' from substituter This patch fixes that by ensuring backtraces are properly displayed on file descriptor 4. * guix/scripts/substitute.scm (with-redirected-error-port): New macro. (guix-substitute): Use 'with-redirected-error-port' instead of 'parameterize'.
2021-02-24substitute: Do not unwind stack traces upon uncaught exceptions.Ludovic Courtès
This is similar to commit a168c3e4f8d580f70e1c26bcdfc5b8378b2fa42d. * guix/scripts/substitute.scm (with-networking): Use 'with-throw-handler' instead of 'catch'.
2021-02-22substitute: Rework connection error handling.Christopher Baines
This is part of trying to reduce the interdependency of code within the substitute module. This commit addresses some of the error handling that was performed through open-connection-for-uri/maybe. The new approach is to use call-with-connection-error-handling, and wrap calls to http-multiple-get and http-fetch with that procedure, which takes care of handling connection errors. I think this is even slightly more rigerous than the previous setup, because this approach handles connection errors that occur when http-multiple-get reconnects to a host. * guix/scripts/substitute.scm (open-connection-for-uri/maybe): Transform in to call-with-connection-error-handling. (fetch-narinfos): Use call-with-connection-error-handling. (process-query): Replace open-connection-for-uri/maybe with open-connection-for-uri/cached. (open-connection-for-uri/cached): Set a default timeout, matching the behaviour in open-connection-for-uri/maybe. (process-substitution): Use call-with-connection-error-handling.
2021-02-22substitute: Remove fetch-narinfos use open-connection-for-uri/maybe.Christopher Baines
At least by default. Instead, make the open-connection procedure a parameter, and make the default guix:open-connection-for-uri. Do so similarly for lookup-narinfos and lookup-narinfos/diverse which work towards calling fetch-narinfos. This means this code can be moved to a different module, without having use/move the connection caching code. * guix/scripts/substitute.scm (fetch-narinfos): Add #:open-connection argument, and call http-multiple-get with it. (lookup-narinfos) Add #:open-connection argument, and call fetch-narinfos with it. (lookup-narinfos/diverse): Add #:open-connection argument, and call lookup-narinfos with it. (process-query): Call lookup-narinfos/diverse with #:open-connection open-connection-for-uri/maybe.
2021-02-22substitute: Inline fetch in to process-substitutes.Christopher Baines
As it's only called in one place, and this should make the code easier to read. * guix/scripts/substitute.scm (fetch): Move procedure inside… (process-substitution): …here.
2021-02-22substitute: Remove redundant fetch arguments.Christopher Baines
It's just called in one place, with hardcoded argument values, so just inline them. * guix/scripts/substitute.scm (fetch): Remove arguments that don't vary, copy the values from the call site in process-substitution. (process-substitution): Remove unnecessary argument values from fetch call.
2021-02-22substitute: Remove now redundant connection caching helpers.Christopher Baines
Failures now should be handled where they occur, and if there's a problem that's symptomatic of an issue with the connection, the port should be closed. * guix/scripts/substitute.scm (call-with-cached-connection): Remove procedure. (with-cached-connection): Remove syntax rule.
2021-02-22substitute: Change connection cache handling in process-substitution.Christopher Baines
Just pass open-connection-for-uri/maybe to http-fetch, this removes the need for with-cached-connection and passing the port in. * guix/scripts/substitute.scm (fetch): Don't take a port as an argument, and pass open-connection-for-uri/maybe to http-fetch. (process-substitution): Don't call fetch with with-cached-connection.
2021-02-22substitute: Stop using call-with-cached-connection in fetch-narinfos.Christopher Baines
Instead, just pass open-connection-for-uri/maybe to http-multiple-get. This code should be functionaly similar to the previous code. The eventual aim of this is to make the connection caching not mandatory in fetch-narinfos. * guix/scripts/substitute.scm (fetch-narinfos): Remove use of call-with-cached-connection.
2021-02-22substitute: open-connection-for-uri/maybe add #:verify-certificate?.Christopher Baines
As this is used by http-fetch and http-multiple-get when they call the specified open connection procedure. * guix/scripts/substitute.scm (open-connection-for-uri/maybe): Support #:verify-certificate?.
2021-02-22guix: Move http-multiple-get to (guix http-client).Christopher Baines
From (guix scripts substitute). This will make it easier to reuse this code. * guix/scripts/substitute.scm (http-multiple-get): Remove, and move to… * guix/http-client.scm (http-multiple-get): …here.
2021-02-22substitute: Remove redundant let block from fetch.Christopher Baines
* guix/scripts/substitute.scm (fetch): Remove redundant let block.
2021-02-22substitute: Remove connection handling from fetch.Christopher Baines
http-fetch does this, so just use that code instead. * guix/scripts/substitute.scm (fetch): Remove connection handling when the port is closed.
2021-02-22substitute: Remove buffer handling from fetch.Christopher Baines
http-fetch does this, so just set the right option. * guix/scripts/substitute.scm (fetch): Remove buffering code, and pass #:buffered? to http-fetch.
2021-01-21Revert "substitute: Remove extra 'newline' call."Ludovic Courtès
This reverts commit d5a1b0e86f248edd53c785b910b5d51ee708b0fc. This change turned out to be misguided; it would lead to "substitute:" lines being printed on the client side, without anything beyond the colon sign.
2021-01-16guix: Move narinfo code from substitute script to module.Christopher Baines
This separation between the code for dealing with narinfos from the code doing that for a purpose should make things clearer, and better support components other that the substitute script in using this code. This is just moving the code around, no code should have been significantly changed. * guix/scripts/substitute.scm (<narinfo>): Move record type to (guix narinfo). (fields->alist, narinfo-hash-algorithm+value, narinfo-hash->sha256, narinfo-signature->canonical-sexp, narinfo-maker, read-narinfo, narinfo-sha256, valid-narinfo?, write-narinfo, narinfo->string, string->narinfo, equivalent-narinfo?, supported-compression?, compresses-better?, narinfo-best-uri): Move procedures to (guix narinfo). (%compression-methods): Move variable to (guix narinfo). * guix/narinfo.scm: New file. * Makefile.am (MODULES): Add it. * po/guix/POTFILES.in: Add 'guix/narinfo.scm'.
2021-01-16substitute: Untangle skipping authentication from valid-narinfo?.Christopher Baines
Rather than having valid-narinfo? evaluate to #t if %allow-unauthenticated-substitutes? is set to #t, just use (const #t) for valid-narinfo? when %allow-unauthenticated-substitutes? is set to #t. This will allow moving valid-narinfo? in to a (guix substitutes) module. * guix/scripts/substitute.scm (process-query, process-substitution): Change the authorized? argument to lookup-narinfo and lookup-narinfos/diverse based on %allow-unauthenticated-substitutes?. (valid-narinfo?): Remove use of %allow-unauthenticated-substitutes?.
2021-01-13substitute: Add zstd support.Ludovic Courtès
* guix/scripts/substitute.scm (%compression-methods): Add zstd. (compresses-better?): "lzip" always wins.
2021-01-13substitute: Remove extra 'newline' call.Ludovic Courtès
This led to extra "substitute:" lines since commit 79c6614f58a57b985daf8940766319e440311db0 or thereabouts. * guix/scripts/substitute.scm (fetch-narinfos): Remove extra 'newline' call, which can be traced back to d3a652037ef879f9279bc056c43d15ba7afcbb25 (2015).
2021-01-04substitute: Handle "invalid session" GnuTLS errors on reused connections.Ludovic Courtès
Reported by Christopher Baines <mail@cbaines.net> at <https://issues.guix.gnu.org/45323#2>. * guix/scripts/substitute.scm (call-with-cached-connection): Handle 'gnutls-error and ERROR/INVALID-SESSION.
2020-12-23substitute: Reuse connections for '--query'.Ludovic Courtès
This significantly speeds up things like substituting the closure of a .drv. This is a followup to 5ff521452b9ec2aae9ed8e4bb7bdc250a581f203. * guix/scripts/substitute.scm (http-multiple-get): Add #:open-connection and #:keep-alive? and honor them. (open-connection-for-uri/maybe): Use 'open-connection-for-uri/cached' instead of 'guix:open-connection-for-uri'. Call 'http-multiple-get' within 'call-with-cached-connection'. (open-connection-for-uri/cached): Add #:timeout and #:verify-certificate? and honor them. (call-with-cached-connection): Add 'open-connection' parameter and honor it.
generated by cgit v1.2.3 (git 2.46.2) at 2024-11-10 06:42:07 +0000