summaryrefslogtreecommitdiff
path: root/guix/derivations.scm
AgeCommit message (Collapse)Author
2023-01-09derivations: 'read-derivation' correctly handles case with empty hash.Ludovic Courtès
Reported by Stephen Paul Weber <singpolyma@singpolyma.net> at <https://lists.gnu.org/archive/html/guix-devel/2023-01/msg00035.html>. * guix/derivations.scm (read-derivation)[outputs->alist]: Treat the empty hash case as non-fixed-output whether or not the hash algorithm is the empty string, and preserve the hash algorithm in <derivation-output>. * tests/derivations.scm ("'download' built-in builder, no fixed-output hash") ("fixed-output-derivation?, no hash", "read-derivation with hash = #f"): New tests.
2022-03-07derivations: Coalesce inputs that have the same output path.Ludovic Courtès
Fixes <https://issues.guix.gnu.org/54209>. * guix/derivations.scm (coalesce-duplicate-inputs): Use the output paths of DRV as a hash table key. * tests/derivations.scm ("derivation with duplicate fixed-output inputs"): Expect a single input for FINAL. ("derivation with equivalent fixed-output inputs"): New test.
2021-11-18derivations: Officially deprecate 'build-expression->derivation'.Ludovic Courtès
This procedure had been semi-deprecated since 2014, starting with commit 21b679f6944f4e1f09f949322f5242b761dc22a7! * guix/derivations.scm (build-expression->derivation): Define using 'define-deprecated'.
2021-07-27derivations: Make 'coalesce-duplicate-inputs' linear in the number of inputs.Ludovic Courtès
Partly fixes <https://issues.guix.gnu.org/49439>. Reported by Ricardo Wurmus <rekado@elephly.net>. When running the command: guix environment pigx-scrnaseq --search-paths --no-grafts this change reduces total heap allocations from 1.4GiB to 717MiB (49%) and wall-clock time from 7.5s to 5.7s (24%). Without '--no-grafts', heap allocations go from 2.1GiB to 1.4GiB (33%) and wall-clock time from 12.1s to 10.9s (10%). * guix/derivations.scm (coalesce-duplicate-inputs): Rewrite using a hash table to make it O(N) rather than O(N²).
2020-08-28derivations: Avoid uses of 'write' in 'write-derivation'.Ludovic Courtès
This leads a 4% improvement on the wall-clock time of: guix build -e '(@@ (gnu packages libreoffice) libreoffice)' --no-grafts -d * guix/derivations.scm (escaped-string): New procedure. (write-derivation)[write-escaped-string]: New procedure. [write-string-list, write-output, write-env-var]: Use it.
2020-08-28derivations: Avoid uses of 'display' in 'write-derivation'.Ludovic Courtès
This yields a 4% improvement on the wall-clock time of: guix build -e '(@@ (gnu packages libreoffice) libreoffice)' --no-grafts -d * guix/derivations.scm (write-sequence, write-list, write-tuple): Use 'put-char' instead of 'display'. (write-derivation): Use 'put-string' and 'put-char', and remove unused 'format' binding.
2020-02-03Merge branch 'staging' into core-updatesMarius Bakke
2020-01-30derivations: Import modules with 'add-file-tree-to-store'.Ludovic Courtès
This reduces the number of RPCs on "guix build libreoffice -nd" from 2,589 to 2,558 (1%). * guix/derivations.scm (imported-files): Rewrite to call to 'add-file-tree-to-store'. Remove #:system and #:guile parameters. (%imported-modules): Adjust docstring to say that result is a store item, and adjust 'imported-files' call. (%compiled-modules): Adjust accordingly. (imported+compiled-modules): Likewise. (build-expression->derivation): Likewise.
2020-01-23derivations: Inline 'find' in 'coalesce-duplicate-inputs'.Ludovic Courtès
Previously the first argument to 'find' would show up high in profiles of 'package-derivation'. This speeds things up a big, especially on Guile 3. * guix/derivations.scm (coalesce-duplicate-inputs)[find]: New procedure.
2019-12-07derivations: Add 'derivation-input-fold'.Ludovic Courtès
* guix/derivations.scm (derivation-input-fold): New procedure. (substitution-oracle)[closure]: Rewrite in terms of 'derivation-input-fold'. * tests/derivations.scm ("derivation-input-fold"): New test.
2019-11-07derivations: Use a regular hash table for the module cache.Ludovic Courtès
The hit rate of the 'add-data-to-store' cache goes from 10% to 4% on: guix build -e '(@@ (gnu packages libreoffice) libreoffice)' -nd * guix/derivations.scm (%module-cache): Turn into a regular hash table. It didn't make sense to use a weak-value hash table given that values are pairs.
2019-10-28derivation: Remove memoization invalidation for 'derivation->bytevector'.Ludovic Courtès
This is a followup to d727a9343d861cf775645df8be5bfefd43d6c6f0, which broke 'hydra-jobs' from (gnu ci). * guix/derivations.scm (invalidate-derivation-caches!): Remove call to 'invalidate-memoization!' for 'derivation->bytevector'.
2019-10-27derivations: Don't memoize 'derivation->bytevector'.Ludovic Courtès
Its hit rate was only 8%. Removing it reduces heap size of "guix build libreoffice -nd" from 69MiB to 61MiB and the wall-clock time is unchanged. * guix/derivations.scm (derivation->bytevector): Change from 'mlambda' to 'lambda'.
2019-10-27derivations: 'build-expression->derivation' caches its module derivations.Ludovic Courtès
This reduces the number of lookups in the 'add-data-to-store' cache from 7505 to 3329 (hit rate from 68% to 27%) when running: GUIX_PROFILING=add-data-to-store-cache guix build libreoffice -nd The execution time of "guix build libreoffice -nd" goes from 2.12s to 1.87s. * guix/derivations.scm (%module-cache): New variable. (imported+compiled-modules)[key]: New variable. Lookup KEY in %MODULE-CACHE and populate %MODULE-CACHE upon cache miss.
2019-10-27derivations: Introduce 'imported+compiled-modules'.Ludovic Courtès
* guix/derivations.scm (imported+compiled-modules): New procedure. (build-expression->derivation): Use it instead of separate calls to '%imported-modules' and '%compiled-modules'.
2019-08-17derivations: Fix typo in docstring.Ludovic Courtès
* guix/derivations.scm (derivation-build-plan): Fix typo in the docstring.
2019-08-16derivations: Delete duplicate inputs when computing derivation hash.Ludovic Courtès
Fixes <https://bugs.gnu.org/36777>. Reported by Carl Dong <contact@carldong.me>. * guix/derivations.scm (derivation/masked-inputs): Call 'delete-duplicates' on INPUTS. * tests/derivations.scm ("derivation with duplicate fixed-output inputs"): New test.
2019-07-15gexp: 'lowered-gexp-guile' now returns a <derivation-input>.Ludovic Courtès
* guix/derivations.scm (derivation-input-output-path): New procedure. * guix/gexp.scm (lower-gexp): Wrap GUILE in a <derivation-input>. (gexp->derivation): Adjust accordingly. * guix/remote.scm (remote-pipe-for-gexp, remote-eval): Adjust accordingly. * tests/gexp.scm ("lower-gexp"): Adjust accordingly.
2019-07-15derivations: Deprecate the previous calling convention.Ludovic Courtès
We will eventually require #:inputs to be a list of <derivation-input>; store items will have to be passed as #:sources, already interned. * guix/derivations.scm (warn-about-derivation-deprecation): New procedure. (derivation): Add #:%deprecation-warning? parameter. [warn-deprecation]: New macro. [input->derivation-input, input->source]: Use it. (build-expression->derivation): Pass #:%deprecation-warning?. * po/guix/POTFILES.in: Add guix/derivations.scm.
2019-07-15derivations: 'map-derivation' uses the new 'derivation' calling convention.Ludovic Courtès
* guix/derivations.scm (map-derivation)[input->output-paths]: Adjust to deal with an argument that's either 'derivation-input?' or a string. [rewritten-input]: Return a <derivation-input> or a string. Pass #:inputs and #:sources to 'derivation'.
2019-07-15derivations: 'derivation' primitive accepts <derivation> and #:sources.Ludovic Courtès
This brings us closer to the <derivation> data type. * guix/derivations.scm (derivation): Add #:sources parameter. [input->derivation-input]: Add clause for 'derivation-input?'. Honor SOURCES.
2019-07-05derivations: Simplify 'substitution-oracle'.Ludovic Courtès
* guix/derivations.scm (substitution-oracle)[valid?, dependencies]: Remove. [closure]: New procedure. Rename parameter from 'drv' to 'inputs-or-drv' and adjust accordingly. (derivation-build-plan): Pass INPUTS directly to 'substitution-oracle'. * guix/ui.scm (show-what-to-build)[substitutable-info]: Likewise.
2019-07-05derivations: 'derivation-build-plan' recurses on substituables.Ludovic Courtès
This fixes a bug whereby "guix build texlive -n" would report: 0.0 MB would be downloaded: /gnu/store/…-texlive-20180414 instead of: The following derivation would be built: /gnu/store/…-texlive-texmf-20180414.drv 2,595.2 MB would be downloaded: /gnu/store/…-texlive-20180414-texmf.tar.xz /gnu/store/…-texlive-20180414 where 'texlive-texmf' is a non-substitutable dependency of 'texlive'. * guix/derivations.scm (dependencies-of-substitutables): New procedure. (derivation-build-plan): When 'input-substitutable-info' returns true, append the subset of DEPS that corresponds to SUBSTITUABLES to the first argument of 'loop'. * guix/ui.scm (show-what-to-build): Remove half-baked traversal of DOWNLOAD. * tests/derivations.scm ("derivation-build-plan and substitutes, non-substitutable dep"): New test.
2019-07-03derivations: 'substitution-oracle' no longer calls 'read-derivation-from-file'.Ludovic Courtès
This is a followup to 5cf4b26d52bcea382d98fb4becce89be9ee37b55. * guix/derivations.scm (substitution-oracle): Use 'derivation-input-derivation' instead of calling 'read-derivation-from-file'.
2019-07-03derivations: Avoid another call to 'read-derivation-from-file'.Ludovic Courtès
This is a followup to 5cf4b26d52bcea382d98fb4becce89be9ee37b55. * guix/derivations.scm (derivation-path->base16-hash): Rename to... (derivation-base16-hash): ... this, and take a <derivation> rather than a file name. Use 'mlambdaq' instead of 'mlambda'. (derivation/masked-inputs): Adjust accordingly. (invalidate-derivation-caches!): Likewise.
2019-07-03derivations: 'map-derivation' no longer calls 'read-derivation-from-file'.Ludovic Courtès
This is a followup to 5cf4b26d52bcea382d98fb4becce89be9ee37b55. * guix/derivations.scm (map-derivation)[rewritten-input]: Avoid call to 'read-derivation-from-file'.
2019-07-03derivations: Work around pathological memoization behavior.Ludovic Courtès
This works around <https://bugs.gnu.org/36487>. * guix/derivations.scm (invalidate-derivation-caches!): Comment out 'hash-clear!' call.
2019-07-02derivations: Update 'build-derivations' docstring.Ludovic Courtès
This is a followup to 7c690a47381f645ec5ec0a1fd6ffc34dba1b69c2. * guix/derivations.scm (build-derivations): Update docstring.
2019-06-27derivations: 'build-derivations' can be passed derivation inputs.Ludovic Courtès
* guix/derivations.scm (build-derivations): Accept <derivation-input> records among DERIVATIONS. * tests/derivations.scm ("build-derivations with specific output"): Test it.
2019-06-27derivations: 'derivation' preserves pointer equality.Ludovic Courtès
* guix/derivations.scm (derivation): Check if FILE is already in %DERIVATION-CACHE and return it if it is.
2019-06-27derivations: <derivation-input> now aggregates a <derivation>.Ludovic Courtès
Consequently, the whole graph of <derivation> object is readily available without having to go through 'read-derivation-from-file', which could have cache misses if the requested <derivation> object had been GC'd in the meantime. This is an important property for the performance of things like 'derivation-build-plan' that traverse the derivation graph. * guix/derivations.scm (<derivation-input>): Replace 'path' field by 'derivation'. (derivation-input-path): Adjust accordingly. (derivation-input-key): New procedure. (derivation-input-output-paths): Adjust accordingly. (coalesce-duplicate-inputs): Likewise. (derivation-prerequisites): Use 'derivation-input-key' to compute keys for INPUT-SET. (derivation-build-plan): Likewise. (read-derivation): Add optional 'read-derivation-from-file' parameter. [make-input-drvs]: Call it. (write-derivation)[write-input]: Adjust to new <derivation-input>. (derivation/masked-inputs): Likewise, and remove redundant 'coalesce-duplicate-inputs' call. (derivation)[input->derivation-input]: Change to consider only the derivation case. Update call to 'make-derivation-input'. [input->source]: New procedure. Separate sources from inputs. (map-derivation): Adjust to new <derivation-input>. * tests/derivations.scm ("parse & export"): Pass a second argument to 'read-derivation'. ("build-expression->derivation and derivation-prerequisites") ("derivation-prerequisites and valid-derivation-input?"): Adjust to new <derivation-input>.
2019-06-27derivations: Rewrite and replace 'derivations-prerequisites-to-build'.Ludovic Courtès
The new 'derivation-build-plan' procedure has a more appropriate signature: it takes a list of <derivation-inputs> instead of taking one <derivation>. Its body is also much simpler. * guix/derivations.scm (derivation-build-plan): New procedure. (derivation-prerequisites-to-build): Express in terms of 'derivation-build-plan' and mark as deprecated. * tests/derivations.scm: Change 'derivation-prerequisites-to-build' tests to 'derivation-build-plan' and adjust accordingly.
2019-06-27derivations: Add 'derivation-input'.Ludovic Courtès
* guix/derivations.scm (derivation-input): New procedure. * tests/grafts.scm (make-derivation-input): Remove. ("graft-derivation, unused outputs not depended on"): Use 'derivation-input'.
2019-06-19derivations: Add 'derivation-input-derivation'.Ludovic Courtès
* guix/derivations.scm (derivation-input-derivation): New procedure. (derivation-prerequisites): Use it.
2019-06-10store: 'build-things' accepts derivation/output pairs.Ludovic Courtès
This allows callers to request the substitution of a single derivation output. * guix/store.scm (build-things): Accept derivation/output pairs among THINGS. * guix/derivations.scm (build-derivations): Likewise. * tests/store.scm ("substitute + build-things with specific output"): New test. * tests/derivations.scm ("build-derivations with specific output"): New test. * doc/guix.texi (The Store): Adjust accordingly.
2019-04-16derivations: Reduce 'valid-path?' RPCs in 'derivation-prerequisites-to-build'.Ludovic Courtès
On a profile with 280 packages, this reduces the number of 'valid-paths?' RPCs made by 'guix package -nu' from 6K to 500. * guix/derivations.scm (derivation-prerequisites-to-build)[built?]: Memoize 'valid-path?' calls.
2019-01-21store: Rename '&nix-error' to '&store-error'.Ludovic Courtès
* guix/store.scm (&nix-error): Rename to... (&store-error): ... this, and adjust users. (&nix-connection-error): Rename to... (&store-connection-error): ... this, and adjust users. (&nix-protocol-error): Rename to... (&store-protocol-error): ... this, adjust users. (&nix-error, &nix-connection-error, &nix-protocol-error): Define these condition types and their getters as deprecrated aliases. * build-aux/run-system-tests.scm, guix/derivations.scm, guix/grafts.scm, guix/scripts/challenge.scm, guix/scripts/graph.scm, guix/scripts/lint.scm, guix/scripts/offload.scm, guix/serialization.scm, guix/ssh.scm, guix/tests.scm, guix/ui.scm, tests/derivations.scm, tests/gexp.scm, tests/guix-daemon.sh, tests/packages.scm, tests/store.scm, doc/guix.texi: Adjust to use the new names.
2018-11-28derivations: Add properties.Ludovic Courtès
* guix/derivations.scm (derivation): Add #:properties parameter. [user+system-env-vars]: Honor it. (derivation-properties): New procedure. (build-expression->derivation): Add #:properties and pass it to 'derivation'. * guix/gexp.scm (gexp->derivation): Likewise. * tests/derivations.scm ("derivation-properties"): New test. * tests/gexp.scm ("gexp->derivation properties"): New test. * doc/guix.texi (Derivations, G-Expressions): Adjust accordingly.
2018-09-04Switch to Guile-Gcrypt.Ludovic Courtès
This removes (guix hash) and (guix pk-crypto), which now live as part of Guile-Gcrypt (version 0.1.0.) * guix/gcrypt.scm, guix/hash.scm, guix/pk-crypto.scm, tests/hash.scm, tests/pk-crypto.scm: Remove. * configure.ac: Test for Guile-Gcrypt. Remove LIBGCRYPT and LIBGCRYPT_LIBDIR assignments. * m4/guix.m4 (GUIX_ASSERT_LIBGCRYPT_USABLE): Remove. * README: Add Guile-Gcrypt to the dependencies; move libgcrypt as "required unless --disable-daemon". * doc/guix.texi (Requirements): Likewise. * gnu/packages/bash.scm, guix/derivations.scm, guix/docker.scm, guix/git.scm, guix/http-client.scm, guix/import/cpan.scm, guix/import/cran.scm, guix/import/crate.scm, guix/import/elpa.scm, guix/import/gnu.scm, guix/import/hackage.scm, guix/import/texlive.scm, guix/import/utils.scm, guix/nar.scm, guix/pki.scm, guix/scripts/archive.scm, guix/scripts/authenticate.scm, guix/scripts/download.scm, guix/scripts/hash.scm, guix/scripts/pack.scm, guix/scripts/publish.scm, guix/scripts/refresh.scm, guix/scripts/substitute.scm, guix/store.scm, guix/store/deduplication.scm, guix/tests.scm, tests/base32.scm, tests/builders.scm, tests/challenge.scm, tests/cpan.scm, tests/crate.scm, tests/derivations.scm, tests/gem.scm, tests/nar.scm, tests/opam.scm, tests/pki.scm, tests/publish.scm, tests/pypi.scm, tests/store-deduplication.scm, tests/store.scm, tests/substitute.scm: Adjust imports. * gnu/system/vm.scm: Likewise. (guile-sqlite3&co): Rename to... (gcrypt-sqlite3&co): ... this. Add GUILE-GCRYPT. (expression->derivation-in-linux-vm)[config]: Remove. (iso9660-image)[config]: Remove. (qemu-image)[config]: Remove. (system-docker-image)[config]: Remove. * guix/scripts/pack.scm: Adjust imports. (guile-sqlite3&co): Rename to... (gcrypt-sqlite3&co): ... this. Add GUILE-GCRYPT. (self-contained-tarball)[build]: Call 'make-config.scm' without #:libgcrypt argument. (squashfs-image)[libgcrypt]: Remove. [build]: Call 'make-config.scm' without #:libgcrypt. (docker-image)[config, json]: Remove. [build]: Add GUILE-GCRYPT to the extensions Remove (guix config) from the imported modules. * guix/self.scm (specification->package): Remove "libgcrypt", add "guile-gcrypt". (compiled-guix): Remove #:libgcrypt. [guile-gcrypt]: New variable. [dependencies]: Add it. [*core-modules*]: Remove #:libgcrypt from 'make-config.scm' call. Add #:extensions. [*config*]: Remove #:libgcrypt from 'make-config.scm' call. (%dependency-variables): Remove %libgcrypt. (make-config.scm): Remove #:libgcrypt. * build-aux/build-self.scm (guile-gcrypt): New variable. (make-config.scm): Remove #:libgcrypt. (build-program)[fake-gcrypt-hash]: New variable. Add (gcrypt hash) to the imported modules. Adjust load path assignments. * gnu/packages/package-management.scm (guix)[propagated-inputs]: Add GUILE-GCRYPT. [arguments]: In 'wrap-program' phase, add GUILE-GCRYPT to the search path.
2018-01-11derivations: Fix typo in docstring.Ludovic Courtès
* guix/derivations.scm (derivation-prerequisites): Fix typo in docstring.
2017-12-13hydra: Invalidate derivation caches after each architecture evaluation.Ludovic Courtès
This reduces max RSS from 1.3G to 1.0G. * guix/derivations.scm (invalidate-derivation-caches!): New procedure. * build-aux/hydra/gnu-system.scm (hydra-jobs): Use it. Add 'format' call.
2017-12-13derivations: Don't memoize 'derivation-hash'.Ludovic Courtès
This has little or no run-time impact and slightly reduces the memory footprint. * guix/derivations.scm (derivation-hash): Replace 'mlambda' with 'lambda'.
2017-12-13derivations: Split 'derivation-hash' in two procedures.Ludovic Courtès
* guix/derivations.scm (derivation/masked-inputs): New procedure. (derivation-hash): Use it instead of the inline code.
2017-12-13Revert "derivations: 'derivation-hash' assumes inputs are coalesced."Ludovic Courtès
This reverts commit 1d008d9f8c44dfdb808235d451b72f255e72f103. Reported by Rutger Helling <rhelling@mykolab.com>. Fixes <https://bugs.gnu.org/29689>.
2017-12-12derivations: 'derivation-hash' assumes inputs are coalesced.Ludovic Courtès
* guix/derivations.scm (derivation-hash): Remove redundant 'coalesce-duplicate-inputs' call.
2017-12-12derivations: Use 'define-immutable-record-type' as appropriate.Ludovic Courtès
This is a followup to dc673fa1131fb5d1e5ca29acb4a693cfb906986f. * guix/derivations.scm (<derivation-output>, <derivation-input>): Use 'define-immutable-record-type'.
2017-06-12derivations: Introduce 'read-derivation-from-file'.Ludovic Courtès
This avoids the open/fstat/close syscalls upon a cache hit that we had with the previous idiom: (call-with-input-file file read-derivation) where caching happened in 'read-derivation' itself. * guix/derivations.scm (%read-derivation): Rename to... (read-derivation): ... this. (read-derivation-from-file): New procedure. (derivation-prerequisites, substitution-oracle) (derivation-prerequisites-to-build): (derivation-path->output-path, derivation-path->output-paths): (derivation-path->base16-hash, map-derivation): Use 'read-derivation-from-file' instead of (call-with-input-file … read-derivation). * guix/grafts.scm (item->deriver): Likewise. * guix/scripts/build.scm (log-url, options->things-to-build): Likewise. * guix/scripts/graph.scm (file->derivation): Remove. (derivation-dependencies, %derivation-node-type): Use 'read-derivation-from-file' instead. * guix/scripts/offload.scm (guix-offload): Likewise. * guix/scripts/perform-download.scm (guix-perform-download): Likewise. * guix/scripts/publish.scm (load-derivation): Remove. (narinfo-string): Use 'read-derivation-from-file'.
2017-06-02derivations: 'derivation-prerequisites-to-build' returns <substitutable>.Ludovic Courtès
* guix/derivations.scm (derivation-prerequisites-to-build): Rename #:substitutable? to #:substitutable-info. [derivation-substitutable?]: Rename to... [derivation-substitutable-info]: ... this. Return a list of <substitutable>. Second return value is now a list of <substitutable> instead of a list of strings. * guix/ui.scm (show-what-to-build)[substitutable?]: Rename to... [substitutable-info]: ... this. Adjust to new 'derivation-prerequisites-to-build' return value type. * tests/derivations.scm ("derivation-prerequisites-to-build and substitutes"): Adjust. ("derivation-prerequisites-to-build and substitutes, local build"): Likewise.
2017-06-02derivations: 'substitution-oracle' returns a <substitutable>.Ludovic Courtès
* guix/derivations.scm (substitution-oracle): Use 'substitution-path-info' instead of 'substitution-paths'. Turn SUBST into a vhash from path to <substitutable>. Change the returned procedure to provide a <substitutable> instead of a Boolean. * tests/derivations.scm ("substitution-oracle and #:substitute? #f"): Mock 'substitutable-path-info' instead of 'substitutable-paths'.
2017-05-04store: Add store path computation procedures.Ludovic Courtès
* guix/derivations.scm (compressed-hash, store-path) (output-path, fixed-output-path): Move to... * guix/store.scm: ... here.
generated by cgit v1.2.3 (git 2.46.2) at 2024-11-17 13:24:13 +0000