summaryrefslogtreecommitdiff
path: root/guix/serialization.scm
AgeCommit message (Expand)Author
2023-03-13guix: Strip #:use-module lists....This was obtained by setting up this environment: guix shell -D guix --with-input=guile@3.0.9=guile-next \ --with-commit=guile-next=e2ed33ef0445c867fe56c247054aa67e834861f2 -- make -j5 then adding 'unused-module' to (@@ (guix build compiler) %warnings), building, and checking all the "unused module" warnings and removing those that were definitely unused. Ludovic Courtès
2021-03-01serialization: Micro-optimize string literal output in 'write-file-tree'....This reduces allocations and bit twiddling in the loop. * guix/serialization.scm (write-literal-strings): New macro. (write-file-tree): Use it in lieu of 'write-string' calls where applicable. Ludovic Courtès
2021-01-22store: Add 'find-roots' RPC....* guix/serialization.scm (read-string-pairs): New procedure. * guix/store.scm (read-arg): Add support for 'string-pairs'. (find-roots): New procedure. * tests/store.scm ("add-indirect-root and find-roots"): New test. Ludovic Courtès
2020-12-19daemon: Delegate deduplication to 'guix substitute'....This removes the main source of latency between subsequent downloads. * nix/libstore/build.cc (SubstitutionGoal::tryToRun): Add a "deduplicate" key to ENV. (SubstitutionGoal::finished): Remove call to 'optimisePath'. * guix/scripts/substitute.scm (process-substitution)[destination-in-store?] [dump-file/deduplicate*]: New variables. Pass #:dump-file to 'restore-file'. * guix/scripts/substitute.scm (guix-substitute)[deduplicate?]: New variable. Pass #:deduplicate? to 'process-substitution'. * guix/serialization.scm (dump-file): Export and augment 'dump-file'. Ludovic Courtès
2020-12-15nar: Deduplicate files right as they are restored....This avoids having to traverse and re-read the files that we have just restored, thereby reducing I/O. * guix/serialization.scm (dump-file): New procedure. (restore-file): Add #:dump-file parameter and honor it. * guix/store/deduplication.scm (tee, dump-file/deduplicate): New procedures. * guix/nar.scm (restore-one-item): Pass #:dump-file to 'restore-file'. (finalize-store-file): Pass #:deduplicate? #f to 'register-items'. * tests/nar.scm <top level>: Call 'setenv' to set "NIX_STORE". Ludovic Courtès
2020-12-15serialization: 'restore-file' sets canonical timestamp and permissions....* guix/serialization.scm (restore-file): Set the permissions and mtime of FILE. * guix/nar.scm (finalize-store-file): Pass #:reset-timestamps? #f to 'register-items'. * tests/nar.scm (rm-rf): Add 'chmod' calls to ensure files are writable. ("write-file + restore-file with symlinks"): Ensure every file in OUTPUT passes 'canonical-file?'. * tests/guix-archive.sh: Run "chmod -R +w" before "rm -rf". Ludovic Courtès
2020-12-15serialization: 'fold-archive' notifies about directory processing completion....* guix/serialization.scm (fold-archive): Call PROC with a 'directory-complete tag when done with a directory. (restore-file): Handle it. * guix/scripts/archive.scm (list-contents): Likewise. * guix/scripts/challenge.scm (archive-contents): Likewise. * tests/nar.scm ("write-file-tree + fold-archive"): Adjust accordingly. Ludovic Courtès
2020-01-24serialization: 'read-byte-string' makes a single read(2) call....On "guix build libreoffice -nd", this reduces the number of read(2) system calls from 10,434 to 8092. * guix/serialization.scm (sub-bytevector): New procedure. (read-byte-string): Make a single 'get-bytevector-n*' call and use 'sub-bytevector'. Ludovic Courtès
2020-01-15serialize: Export 'dump-port*'....* guix/serialization.scm (dump): Export as 'dump-port*'. * guix/scripts/challenge.scm (dump-port*): Remove. Ludovic Courtès
2019-12-12serialization: Remove unused procedure....* guix/serialization.scm (write-contents): Remove. Ludovic Courtès
2019-12-12serialization: Add 'fold-archive'....* guix/serialization.scm (read-contents): Remove. (read-file-type, fold-archive): New procedures. (restore-file): Rewrite in terms of 'fold-archive'. * tests/nar.scm ("write-file-tree + fold-archive") ("write-file-tree + fold-archive, flat file"): New tests. Ludovic Courtès
2019-01-21store: Rename '&nix-error' to '&store-error'....* guix/store.scm (&nix-error): Rename to... (&store-error): ... this, and adjust users. (&nix-connection-error): Rename to... (&store-connection-error): ... this, and adjust users. (&nix-protocol-error): Rename to... (&store-protocol-error): ... this, adjust users. (&nix-error, &nix-connection-error, &nix-protocol-error): Define these condition types and their getters as deprecrated aliases. * build-aux/run-system-tests.scm, guix/derivations.scm, guix/grafts.scm, guix/scripts/challenge.scm, guix/scripts/graph.scm, guix/scripts/lint.scm, guix/scripts/offload.scm, guix/serialization.scm, guix/ssh.scm, guix/tests.scm, guix/ui.scm, tests/derivations.scm, tests/gexp.scm, tests/guix-daemon.sh, tests/packages.scm, tests/store.scm, doc/guix.texi: Adjust to use the new names. Ludovic Courtès
2019-01-18serialization: 'restore-file' errors out upon non-convertible file names....Fixes <https://bugs.gnu.org/33603>. Reported by Maxim Cournoyer <maxim.cournoyer@gmail.com>. * guix/serialization.scm (port-conversion-strategy): New variable. (restore-file): Parameterize it. * tests/nar.scm ("restore-file with non-UTF8 locale"): New test. Ludovic Courtès
2018-09-23serialization: Remove redundancy in 'write-file'....* guix/serialization.scm (write-file): Remove redundant 'member' call. Ludovic Courtès
2018-07-19serialization: Add 'write-file-tree'....* guix/serialization.scm (write-contents-from-port): New procedure. (write-contents): Write in terms of 'write-contents-from-port'. (filter/sort-directory-entries, write-file-tree): New procedures. (write-file): Rewrite in terms of 'write-file-tree'. * tests/nar.scm ("write-file-tree + restore-file"): New test. Ludovic Courtès
2018-01-11store: Fix potential over-reads in 'import-paths'....Previously 'process-stderr' would always pass a bytevector of MAX-LEN to then daemon in the %stderr-read case (i.e., 'import-paths'), instead of LEN (where LEN <= MAX-LEN). In practice the extra bytes didn't cause a protocol violation or anything because they happen at the end of the stream, which typically contains the canonical sexp of the signature, and the extra zeros were just ignored. * guix/serialization.scm (write-bytevector): Add optional 'l' parameter and honor it. * guix/store.scm (process-stderr): Pass LEN to 'write-bytevector'. Ludovic Courtès
2017-04-17serialization: Remove Guile < 2.0.9 workaround....* guix/serialization.scm (write-contents): Assume 'sendfile' is always defined. Ludovic Courtès
2017-03-18build: Require Guile >= 2.0.9....* configure.ac: Bump requirement to 2.0.9. * doc/guix.texi (Requirements): Adjust accordingly. * README (Requirements): Likewise. * build-aux/download.scm: Remove workaround for <http://bugs.gnu.org/13095>. * guix/build/download.scm: Likewise. (http-fetch)[post-2.0.7?]: Remove. Remove conditional code for not POST-2.0.7?. * guix/http-client.scm: Remove workaround for <http://bugs.gnu.org/13095>. (http-fetch)[post-2.0.7?]: Remove. Remove conditional code for not POST-2.0.7?. * guix/serialization.scm (read-latin1-string): Remove mention of 2.0.9. * tests/nar.scm: Use (ice-9 control). (let/ec): Remove. Ludovic Courtès
2017-01-30store: Add 'add-data-to-store'....* guix/serialization.scm (write-bytevector): New procedure. (write-string): Rewrite in terms of 'write-bytevector'. * guix/store.scm (write-arg): Add 'bytevector' case. (add-data-to-store): New procedure, from former 'add-text-to-store'. (add-text-to-store): Rewrite in terms of 'add-data-to-store'. * tests/store.scm ("add-data-to-store"): New test. Ludovic Courtès
2016-10-19Use (ice-9 binary-ports) instead of (rnrs io ports)....This reduces the closure of (guix ui) from 123 to 106 modules. * guix/derivations.scm: Use (ice-9 binary-ports) instead of (rnrs io ports). (map-derivation)[substitute-file]: Use 'read-string' instead of 'get-string-all'. * guix/ftp-client.scm: Likewise. * guix/hash.scm: Likewise. * guix/http-client.scm: Likewise. * guix/pki.scm (ensure-acl, current-acl): Likewise. * guix/scripts/archive.scm (authorize-key)[read-key]: Likewise. * guix/scripts/authenticate.scm (read-canonical-sexp) (read-hash-data): Likewise. * guix/scripts/download.scm: Likewise. * guix/scripts/offload.scm (register-gc-root, remove-gc-roots) (send-files): Likewise. * guix/scripts/publish.scm (lazy-read-file-sexp): Likewise. * guix/scripts/refresh.scm: Likewise. * guix/scripts/substitute.scm (check-acl-initialized): Likewise. * guix/serialization.scm (read-maybe-utf8-string): Likewise. * guix/scripts/hash.scm (guix-hash): Use 'force-output' instead of 'flush-output-port'. * guix/store.scm (process-stderr): Likewise. * guix/tests.scm: Likewise. * guix/utils.scm: Use (ice-9 binary-ports) and autoload (rnrs io ports) for 'make-custom-binary-input-port'. Ludovic Courtès
2016-06-12serialization: Add #:select? parameter to 'write-file'....* guix/serialization.scm (write-file): Add #:select? parameter and honor it. * tests/nar.scm ("write-file #:select? + restore-file"): New test. Ludovic Courtès
2016-05-04utils: Move combinators to (guix combinators)....* guix/utils.scm (compile-time-value, memoize, fold2) (fold-tree, fold-tree-leaves): Move to... * guix/combinators: ... here. New file. * tests/utils.scm ("fold2, 1 list", "fold2, 2 lists") (fold-tree tests): Move to... * tests/combinators.scm: ... here. New file. * Makefile.am (MODULES, SCM_TESTS): Add them. * gnu/packages.scm, gnu/packages/bootstrap.scm, gnu/services/herd.scm, guix/build-system/gnu.scm, guix/build-system/python.scm, guix/derivations.scm, guix/gnu-maintenance.scm, guix/import/elpa.scm, guix/scripts/archive.scm, guix/scripts/build.scm, guix/scripts/graph.scm, guix/scripts/lint.scm, guix/scripts/size.scm, guix/scripts/substitute.scm, guix/serialization.scm, guix/store.scm, guix/ui.scm: Adjust imports accordingly. Ludovic Courtès
2015-05-10Remove assorted Guile 2.0.5 workarounds....* guix/scripts/authenticate.scm (%default-port-conversion-strategy): Remove. * guix/scripts/substitute.scm (fetch): Remove 2.0.5 special cases. * guix/serialization.scm (write-file): Remove 'scandir' workaround. * guix/ui.scm (command-files): Likewise. Ludovic Courtès
2015-05-01serialization: Adjust the permissive UTF-8 decoder to Guile 2.0.12ish....* guix/serialization.scm (read-maybe-utf8-string): Use 'set-port-encoding!' and 'set-port-conversion-strategy!' instead of setting '%default-port-encoding' and '%default-port-conversion-strategy'. This accounts for Guile commit d574d96, which changes bytevector input ports to use ISO-8859-1. Ludovic Courtès
2015-03-05store: Attempt to decode build logs as UTF-8....* guix/serialization.scm (read-maybe-utf8-string): New procedure. * guix/store.scm (process-stderr): Use it for the build log and errors. * tests/store.scm ("current-build-output-port, UTF-8", "current-build-output-port, UTF-8 + garbage"): New tests. Ludovic Courtès
2015-03-05serialization: Factorize 'read-byte-string'....* guix/serialization.scm (read-byte-string): New procedure. (read-string, read-latin1-string): Use it. Ludovic Courtès
2015-02-07serialization: Check for EOF and incomplete input conditions....Fixes <http://bugs.gnu.org/19756>. Reported by <sleep_walker@suse.cz>. * guix/serialization.scm (currently-restored-file): New variable. (get-bytevector-n*): New procedure. (read-int, read-long-long, read-string, read-latin1-string, read-contents): Use it instead of 'get-bytevector-n'. (restore-file): Parameterize 'currently-restored-file' and set it. * tests/nar.scm ("restore-file with incomplete input"): New test. Ludovic Courtès
2015-01-18serialization: Read Latin-1 strings with 'get-bytevector-n'....* guix/serialization.scm (read-latin1-string): Use 'get-bytevector-n' instead of 'get-string-n'. Use 'list->string' etc. to convert the bytevector to a string. Ludovic Courtès
2014-10-09Break module cycle involving (guix store) and (guix ui)....Before, there was a cycle along the lines of: (guix store) -> (guix nar) -> (guix ui) -> (guix store) This caused problems, as discussed at: http://lists.gnu.org/archive/html/guix-devel/2014-10/msg00109.html This patch removes cycles in the (guix ...) modules. * guix/nar.scm (&nar-error, &nar-read-error, dump, write-contents, read-contents, %archive-version-1, write-file, restore-file): Move to... * guix/serialization.scm: ... here. * guix/store.scm: Remove dependency on (guix nar). * guix/scripts/hash.scm, guix/scripts/offload.scm, guix/scripts/substitute-binary.scm, tests/nar.scm, tests/store.scm, tests/substitute-binary.scm: Adjust accordingly. Ludovic Courtès
2014-03-09store: Add #:timeout build option....* guix/serialization.scm (write-string-pairs): New procedure. * guix/store.scm (write-arg): Add 'string-pairs' case. (set-build-options): Add 'timeout' keyword parameter. Honor it. * tests/derivations.scm ("build-expression->derivation and timeout"): New test. Ludovic Courtès
2013-04-04Add (guix nar) and (guix serialization)....* guix/store.scm (write-int, read-int, write-long-long, read-long-long, write-padding, write-string, read-string, read-latin1-string, write-string-list, read-string-list, write-store-path, read-store-path, write-store-path-list, read-store-path-list): Move to serialization.scm. (write-contents, write-file): Move to nar.scm. * guix/nar.scm, guix/serialization.scm: New files. * Makefile.am (MODULES): Add them. Ludovic Courtès