summaryrefslogtreecommitdiff
path: root/guix/scripts/publish.scm
AgeCommit message (Expand)Author
2017-10-05publish: Cache uncompressed nars as well....Fixes <https://bugs.gnu.org/28664>. Reported by Maxim Cournoyer <maxim.cournoyer@gmail.com>. * guix/scripts/publish.scm (bake-narinfo+nar): When COMPRESSION is 'none, write NAR instead of doing nothing. (make-request-handler): Use 'render-nar/cached' for /nar URLs with no compression. * tests/publish.scm ("with cache", "with cache, uncompressed"): Adjust expected result accordingly. Ludovic Courtès
2017-07-21publish: Make the cache eviction policy less aggressive....Suggested by Mark H Weaver <mhw@netris.org>. * guix/scripts/publish.scm (nar-expiration-time): New procedure. (render-narinfo/cached): Use it as the #:entry-expiration passed to 'maybe-remove-expired-cache-entries'. Ludovic Courtès
2017-07-21publish: Avoid 'valid-path?' RPC for non-existent items....* guix/scripts/publish.scm (render-narinfo/cached): Call 'file-exists?' before calling 'valid-path?'. This makes the 404 path slightly faster. Ludovic Courtès
2017-07-21publish: Remove 'regexp-exec' call from the hot path....* guix/scripts/publish.scm (extract-narinfo-hash): Rewrite without resorting to regexps. Ludovic Courtès
2017-06-12derivations: Introduce 'read-derivation-from-file'....This avoids the open/fstat/close syscalls upon a cache hit that we had with the previous idiom: (call-with-input-file file read-derivation) where caching happened in 'read-derivation' itself. * guix/derivations.scm (%read-derivation): Rename to... (read-derivation): ... this. (read-derivation-from-file): New procedure. (derivation-prerequisites, substitution-oracle) (derivation-prerequisites-to-build): (derivation-path->output-path, derivation-path->output-paths): (derivation-path->base16-hash, map-derivation): Use 'read-derivation-from-file' instead of (call-with-input-file … read-derivation). * guix/grafts.scm (item->deriver): Likewise. * guix/scripts/build.scm (log-url, options->things-to-build): Likewise. * guix/scripts/graph.scm (file->derivation): Remove. (derivation-dependencies, %derivation-node-type): Use 'read-derivation-from-file' instead. * guix/scripts/offload.scm (guix-offload): Likewise. * guix/scripts/perform-download.scm (guix-perform-download): Likewise. * guix/scripts/publish.scm (load-derivation): Remove. (narinfo-string): Use 'read-derivation-from-file'. Ludovic Courtès
2017-05-28scripts: Set thread names....This allows 'guix publish' threads as well as 'guix substitute' and 'guix offload' processes to be properly labeled in 'top', 'pstree', etc. * guix/workers.scm (worker-thunk): Add #:thread-name parameter and honor it. (make-pool): Likewise. * guix/scripts/publish.scm (http-write): Add calls to 'set-thread-name' in bodies of 'call-with-new-thread'. (guix-publish): Call 'set-thread-name'. Pass #:thread-name to 'make-pool'. * guix/scripts/offload.scm (guix-offload): Call 'set-thread-name'. * guix/scripts/substitute.scm (guix-substitute): Likewise. Ludovic Courtès
2017-05-24publish: Do not bake the same item several times....Before this patch, there was a possibility that we'd queue several baking tasks for the same item. The first one would bake the item but subsequent tasks wouldn't check whether the item had been baked in the meantime and would re-bake it. * guix/scripts/publish.scm (render-narinfo/cached): Don't call 'bake-narinfo+nar' when (file-exists? cached). Ludovic Courtès
2017-05-18publish: Fix narinfo rendering for already-compressed items....Fixes <http://bugs.gnu.org/26975>. Reported by Mark H Weaver <mhw@netris.org>. * guix/scripts/publish.scm (bake-narinfo+nar): Pass #f as the 2nd argument to 'stat' and properly handle #f. * tests/publish.scm (wait-for-file): New procedure. ("with cache"): Remove 'wait-for-file' procedure. ("with cache, uncompressed"): New test. Ludovic Courtès
2017-05-11publish: Advertise a short TTL for "baking" 404s....* guix/scripts/publish.scm (not-found): Add #:phrase and #:ttl parameters and honor them. * tests/publish.scm ("with cache"): Check the 'cache-control' header on of the 404 response. Ludovic Courtès
2017-05-03ui: Rename '_' to 'G_'....This avoids collisions with '_' when the latter is used as a 'match' pattern for instance. See <https://lists.gnu.org/archive/html/guix-devel/2017-04/msg00464.html>. * guix/ui.scm: Rename '_' to 'G_'. * po/guix/Makevars (XGETTEXT_OPTIONS): Adjust accordingly. * build-aux/compile-all.scm (warnings): Remove 'format'. * gnu/packages.scm, gnu/services.scm, gnu/services/shepherd.scm, gnu/system.scm, gnu/system/shadow.scm, guix/gnupg.scm, guix/http-client.scm, guix/import/cpan.scm, guix/import/elpa.scm, guix/import/pypi.scm, guix/nar.scm, guix/scripts.scm, guix/scripts/archive.scm, guix/scripts/authenticate.scm, guix/scripts/build.scm, guix/scripts/challenge.scm, guix/scripts/container.scm, guix/scripts/container/exec.scm, guix/scripts/copy.scm, guix/scripts/download.scm, guix/scripts/edit.scm, guix/scripts/environment.scm, guix/scripts/gc.scm, guix/scripts/graph.scm, guix/scripts/hash.scm, guix/scripts/import.scm, guix/scripts/import/cpan.scm, guix/scripts/import/cran.scm, guix/scripts/import/crate.scm, guix/scripts/import/elpa.scm, guix/scripts/import/gem.scm, guix/scripts/import/gnu.scm, guix/scripts/import/hackage.scm, guix/scripts/import/nix.scm, guix/scripts/import/pypi.scm, guix/scripts/import/stackage.scm, guix/scripts/lint.scm, guix/scripts/offload.scm, guix/scripts/pack.scm, guix/scripts/package.scm, guix/scripts/perform-download.scm, guix/scripts/publish.scm, guix/scripts/pull.scm, guix/scripts/refresh.scm, guix/scripts/size.scm, guix/scripts/substitute.scm, guix/scripts/system.scm, guix/ssh.scm, guix/upstream.scm: Use 'G_' instead of '_'. Most of this change was obtained by running: "sed -i -e's/(_ "/(G_ "/g' `find -name \*.scm`". Ludovic Courtès
2017-05-02publish: Use a larger zlib buffer for compression with '--cache'....* guix/scripts/publish.scm (bake-narinfo+nar): Pass #:buffer-size to 'call-with-gzip-output-port'. Ludovic Courtès
2017-05-01publish: Produce a "FileSize" narinfo field when possible....* guix/scripts/publish.scm (narinfo-string): Add #:file-size parameter. Produce a "FileSize" field when COMPRESSION is eq? to '%no-compression' or when FILE-SIZE is true. (bake-narinfo+nar): Pass #:file-size. * tests/publish.scm ("/*.narinfo") ("/*.narinfo with properly encoded '+' sign") ("with cache"): Check for "FileSize". Ludovic Courtès
2017-04-19publish: Add a handler for / and /index.html....Suggested by Quiliro <quiliro@riseup.net> in <https://bugs.gnu.org/26567>. * guix/scripts/publish.scm (render-home-page): New procedure. (make-request-handler): Handle it. Ludovic Courtès
2017-04-18publish: Remove expired cache entries when '--ttl' is used....* guix/scripts/publish.scm (narinfo-files): New procedure. (render-narinfo/cached)[delete-file]: New procedure. Add call to 'maybe-remove-expired-cache-entries'. * doc/guix.texi (Invoking guix publish): Document the interation between --cache and --ttl. Ludovic Courtès
2017-04-18publish: Add '--cache' and '--workers'....Fixes <http://bugs.gnu.org/26201>. Reported by <dian_cecht@zoho.com>. These options allow nars to be "baked" off-line and cached instead of being compressed on the fly. As a side-effect, this allows us to provide a 'Content-Length' header for nars. * guix/scripts/publish.scm (show-help, %options): Add '--cache' and '--workers'. (%default-options): Add 'workers'. (nar-cache-file, narinfo-cache-file, run-single-baker): New procedures. (single-baker): New macro. (render-narinfo/cached, bake-narinfo+nar) (render-nar/cached): New procedures. (make-request-handler): Add #:cache and #:pool parameters and honor them. (run-publish-server): Likewise. (guix-publish): Honor '--cache' and '--workers'. * tests/publish.scm ("with cache"): New test. * doc/guix.texi (Invoking guix publish): Document it. Ludovic Courtès
2017-04-18publish: Use 'sendfile' when possible....* guix/scripts/publish.scm (http-write): In the 'application/octet-stream' case, use 'sendfile' when OUTPUT is a file port. Ludovic Courtès
2017-04-18publish: Introduce 'actual-compression'....* guix/scripts/publish.scm (actual-compression): New procedure. (narinfo-string): Use it. Ludovic Courtès
2017-03-22publish: Add '--nar-path'....* guix/scripts/publish.scm (show-help, %options): Add '--nar-path'. (%default-options): Add 'nar-path'. (guix-publish): Honor it. Ludovic Courtès
2017-03-22publish: Make the nar URL prefix a parameter....* guix/scripts/publish.scm (narinfo-string): Add #:nar-path and honor it. (render-narinfo): Likewise. (make-request-handler): Likewise. (run-publish-server): Likewise. * tests/publish.scm ("custom nar path"): New test. Ludovic Courtès
2017-03-22publish: Add '--public-key' and '--private-key'....* guix/scripts/publish.scm (show-help, %options): Add --public-key and --private-key. * doc/guix.texi (Invoking guix publish): Document it. Ludovic Courtès
2017-03-22publish: The public and private keys are now SRFI-39 parameters....* guix/scripts/publish.scm (%default-options): Add 'public-key-file' and 'private-key-file'. (lazy-read-file-sexp): Remove. (%private-key, %public-key): Turn into SRFI-39 parameters. (signed-string, render-narinfo): Adjust accordingly. (guix-publish): Honor 'public-key-file' and 'private-key-file' from OPTS. Use 'parameterize'. * guix/pk-crypto.scm (read-file-sexp): New procedure. * tests/publish.scm: Initialize '%public-key' and '%private-key'. Ludovic Courtès
2016-12-04publish: Factorize 'content-length' addition....* guix/scripts/publish.scm (with-content-length): New procedure. (http-write) <application/octet-stream>: Use it. Ludovic Courtès
2016-10-19Use (ice-9 binary-ports) instead of (rnrs io ports)....This reduces the closure of (guix ui) from 123 to 106 modules. * guix/derivations.scm: Use (ice-9 binary-ports) instead of (rnrs io ports). (map-derivation)[substitute-file]: Use 'read-string' instead of 'get-string-all'. * guix/ftp-client.scm: Likewise. * guix/hash.scm: Likewise. * guix/http-client.scm: Likewise. * guix/pki.scm (ensure-acl, current-acl): Likewise. * guix/scripts/archive.scm (authorize-key)[read-key]: Likewise. * guix/scripts/authenticate.scm (read-canonical-sexp) (read-hash-data): Likewise. * guix/scripts/download.scm: Likewise. * guix/scripts/offload.scm (register-gc-root, remove-gc-roots) (send-files): Likewise. * guix/scripts/publish.scm (lazy-read-file-sexp): Likewise. * guix/scripts/refresh.scm: Likewise. * guix/scripts/substitute.scm (check-acl-initialized): Likewise. * guix/serialization.scm (read-maybe-utf8-string): Likewise. * guix/scripts/hash.scm (guix-hash): Use 'force-output' instead of 'flush-output-port'. * guix/store.scm (process-stderr): Likewise. * guix/tests.scm: Likewise. * guix/utils.scm: Use (ice-9 binary-ports) and autoload (rnrs io ports) for 'make-custom-binary-input-port'. Ludovic Courtès
2016-08-01publish: Do not compress already-compressed files....* guix/scripts/publish.scm (narinfo-string): Force %NO-COMPRESSION when STORE-PATH matches 'compressed-file?'. * guix/utils.scm (compressed-file?): New procedure. * tests/publish.scm ("/*.narinfo for a compressed file"): New test. Ludovic Courtès
2016-07-20publish: Handle '/file' URLs, for content-addressed files....* guix/scripts/publish.scm (render-content-addressed-file): New procedure. (http-write): Add 'application/octet-stream' case. (make-request-handler): Add /file/NAME/sha256/HASH URLs. * tests/publish.scm ("/file/NAME/sha256/HASH") ("/file/NAME/sha256/INVALID-NIX-BASE32-STRING") ("/file/NAME/sha256/INVALID-HASH"): New tests. * doc/guix.texi (Invoking guix publish): Mention the /file URLs. Ludovic Courtès
2016-07-19publish: Keep compression disabled when zlib is missing....Reported by David Thompson. * guix/scripts/publish.scm (%options)[--compression]: Warn if LEVEL > 0 and zlib is missing, and return RESULT. Ludovic Courtès
2016-07-19publish: Add '--compression'....* guix/scripts/publish.scm (show-help, %options): Add '--compression'. (<compression>): New record type. (%no-compression, %default-gzip-compression): New variables. (%default-options): Add 'compression' key. (narinfo-string): Add #:compression parameter and honor it. (render-narinfo): Likewise. (render-nar): Likewise. <top level>: Add call to 'declare-header!'. (swallow-zlib-error): New macro. (nar-response-port): New procedure. (http-write): Add call to 'force-output'. Use 'nar-response-port' instead of 'response-port'. Use 'swallow-zlib-error'. (make-request-handler): Add #:compression parameter and honor it. Add "nar/gzip" URL handler. (run-publish-server): Add #:compression parameter and honor it. (guix-publish): Honor --compression. * tests/publish.scm (http-get-port, wait-until-ready): New procedures. <top level>: Run main server with "-C0". Call 'wait-until-ready'. ("/nar/gzip/*", "/*.narinfo with compression"): New tests. * doc/guix.texi (Invoking guix publish): Document it. Ludovic Courtès
2016-06-09publish: Add '--ttl'....* guix/scripts/publish.scm (show-help, %options): Add --ttl. (render-narinfo): Add #:ttl and honor it. (make-request-handler): Add #:narinfo-ttl and honor it. (run-publish-server): Likewise. (guix-publish): Honor --ttl, pass it to 'run-publish-server'. Ludovic Courtès
2016-06-08publish: Encore URIs that appear in narinfos....Fixes <http://bugs.gnu.org/21888>. Reported by iyzsong@member.fsf.org (宋文武). * guix/scripts/publish.scm (narinfo-string): Use 'encode-and-join-uri-path' instead of 'string-append' to compute URL. * tests/publish.scm ("/*.narinfo with properly encoded '+' sign"): ("/nar/ with properly encoded '+' sign"): New tests. Ludovic Courtès
2016-02-26store: 'path-info-deriver' is #f when there is no deriver....* guix/store.scm (read-path-info): Use #f when we get the empty string for DERIVER. * guix/scripts/publish.scm (narinfo-string): Adjust accordingly. * tests/store.scm ("path-info-deriver"): New test. Ludovic Courtès
2016-02-25publish: Do not publish nars for invalid store items....Before that, /nar requests could succeed if the requested store item exists but is invalid (although such requests were unlikely because the corresponding narinfo request would have failed.) * guix/scripts/publish.scm (render-nar): Add 'store' parameter. Use 'valid-path?' instead of 'file-exists?'. (make-request-handler): Adjust 'render-nar' call accordingly. * tests/publish.scm ("/nar/invalid"): New test. Ludovic Courtès
2016-02-24publish: Move 'query-path-info' call where it belongs....* guix/scripts/publish.scm (narinfo-string): Remove 'path-info' parameter and add 'store' parameter. Call 'query-path-info'. (render-narinfo): Adjust accordingly. Ludovic Courtès
2015-10-06publish: Fix file descriptor leak....A client closing the connection while reading from a /nar URL would leave an open file descriptor in the server. This patch fixes it. * guix/scripts/publish.scm (swallow-EPIPE): New macro. (http-write): Use it around 'write-file' call. Ludovic Courtès
2015-09-18Add (guix scripts)....* guix/ui.scm: Add missing copyright lines. (args-fold*, environment-build-options, %default-argument-handler, parse-command-line): Move to ... * guix/scripts.scm: ...here. New file. * guix/scripts/archive.scm: Use it. * guix/scripts/build.scm: Likewise. * guix/scripts/download.scm: Likewise. * guix/scripts/edit.scm: Likewise. * guix/scripts/environment.scm: Likewise. * guix/scripts/gc.scm: Likewise. * guix/scripts/graph.scm: Likewise. * guix/scripts/hash.scm: Likewise. * guix/scripts/import/cpan.scm: Likewise. * guix/scripts/import/cran.scm: Likewise. * guix/scripts/import/elpa.scm: Likewise. * guix/scripts/import/gem.scm: Likewise. * guix/scripts/import/gnu.scm: Likewise. * guix/scripts/import/hackage.scm: Likewise. * guix/scripts/import/nix.scm: Likewise. * guix/scripts/import/pypi.scm: Likewise. * guix/scripts/lint.scm: Likewise. * guix/scripts/package.scm: Likewise. * guix/scripts/publish.scm: Likewise. * guix/scripts/pull.scm: Likewise. * guix/scripts/refresh.scm: Likewise. * guix/scripts/size.scm: Likewise. * guix/scripts/system.scm: Likewise. * tests/ui.scm (with-environment-variable, "parse-command-line", "parse-command-line and --no options"): Move to ... * tests/scripts.scm: ...here. New file. * Makefile.am (MODULES): Add guix/scripts.scm. (SCM_TESTS): Add tests/scripts.scm. * po/guix/POTFILES.in: Add guix/scripts.scm. Alex Kost
2015-09-04publish: Gracefully handle the lack of a deriver....* guix/scripts/publish.scm (narinfo-string): Catch 'system-error' around 'load-derivation' call; return BASE-INFO upon ENOENT. This allows us to return the narinfo even if DERIVER is missing. Before that, the exception would be uncaught, leading to 500 Internal Error on the client side. Ludovic Courtès
2015-07-23Fix typos in translatable strings....Reported by Anders Jonsson <anders.jonsson@norsjovallen.se>. * gnu/packages/backup.scm, gnu/packages/databases.scm, gnu/packages/linux.scm, gnu/packages/perl.scm, gnu/packages/web.scm, guix/scripts/lint.scm, guix/scripts/publish.scm: Fix typos in translatable strings. Ludovic Courtès
2015-07-20publish: Do not load archive content in memory....Previously, before replying to a /nar/* request, 'guix publish' would first build up the whole nar into memory (as a consequence of <http://bugs.gnu.org/21093>), which obviously doesn't scale. * guix/scripts/publish.scm (render-nar): Return STORE-PATH instead of a procedure that calls 'write-file'. (sans-content-length): New procedure. (http-write): For 'x-nix-archive', don't call '%http-write'. Instead, call 'write-file' right from here, using BODY as the file name. Ludovic Courtès
2015-07-19publish: Serve /nar requests in a separate thread....* guix/scripts/publish.scm (%http-write): New variable. (http-write): New procedure. (concurrent-http-server): New variable. (run-publish-server): Use it. Ludovic Courtès
2015-07-18publish: Write hashes in nix-base32 format....* guix/scripts/publish.scm (narinfo-string): Use 'bytevector->nix-base32-string', not 'bytevector->base32-string'. Ludovic Courtès
2015-05-12publish: Add '--listen'....* guix/scripts/publish.scm (show-help, %options): Add --listen. (getaddrinfo*): New procedure. (%default-options): Add 'address'. (open-server-socket): Replace 'addr' and 'port' with 'address', a sockaddr. (guix-publish): Adjust accordingly. Augment "publishing" message with the actual address. * doc/guix.texi (Invoking guix publish): Document it. Ludovic Courtès
2015-05-12publish: Add '--user' option....* guix/scripts/publish.scm (show-help): Add --user. (%options): Likewise. (run-publish-server): Change 'port' parameter to 'socket'. Pass #:socket instead of #:addr and #:port to 'run-server'. Update caller accordingly. (open-server-socket, gather-user-privileges): New procedures. (guix-publish): Use them. Force %PRIVATE-KEY and %PUBLIC-KEY early on. Warn when running as root. * doc/guix.texi (Invoking guix publish): Document --user. Ludovic Courtès
2015-04-04scripts: Add 'publish' command....* guix/scripts/publish.scm: New file. * po/guix/POTFILES.in: Add it. * tests/publish.scm: New file. * Makefile.am (MODULES): Add script module. (SCM_TESTS): Add test module. * doc/guix.texi ("Invoking guix publish"): New node. David Thompson