summaryrefslogtreecommitdiff
path: root/guix/derivations.scm
AgeCommit message (Collapse)Author
2020-01-23derivations: Inline 'find' in 'coalesce-duplicate-inputs'.Ludovic Courtès
Previously the first argument to 'find' would show up high in profiles of 'package-derivation'. This speeds things up a big, especially on Guile 3. * guix/derivations.scm (coalesce-duplicate-inputs)[find]: New procedure.
2019-12-07derivations: Add 'derivation-input-fold'.Ludovic Courtès
* guix/derivations.scm (derivation-input-fold): New procedure. (substitution-oracle)[closure]: Rewrite in terms of 'derivation-input-fold'. * tests/derivations.scm ("derivation-input-fold"): New test.
2019-11-07derivations: Use a regular hash table for the module cache.Ludovic Courtès
The hit rate of the 'add-data-to-store' cache goes from 10% to 4% on: guix build -e '(@@ (gnu packages libreoffice) libreoffice)' -nd * guix/derivations.scm (%module-cache): Turn into a regular hash table. It didn't make sense to use a weak-value hash table given that values are pairs.
2019-10-28derivation: Remove memoization invalidation for 'derivation->bytevector'.Ludovic Courtès
This is a followup to d727a9343d861cf775645df8be5bfefd43d6c6f0, which broke 'hydra-jobs' from (gnu ci). * guix/derivations.scm (invalidate-derivation-caches!): Remove call to 'invalidate-memoization!' for 'derivation->bytevector'.
2019-10-27derivations: Don't memoize 'derivation->bytevector'.Ludovic Courtès
Its hit rate was only 8%. Removing it reduces heap size of "guix build libreoffice -nd" from 69MiB to 61MiB and the wall-clock time is unchanged. * guix/derivations.scm (derivation->bytevector): Change from 'mlambda' to 'lambda'.
2019-10-27derivations: 'build-expression->derivation' caches its module derivations.Ludovic Courtès
This reduces the number of lookups in the 'add-data-to-store' cache from 7505 to 3329 (hit rate from 68% to 27%) when running: GUIX_PROFILING=add-data-to-store-cache guix build libreoffice -nd The execution time of "guix build libreoffice -nd" goes from 2.12s to 1.87s. * guix/derivations.scm (%module-cache): New variable. (imported+compiled-modules)[key]: New variable. Lookup KEY in %MODULE-CACHE and populate %MODULE-CACHE upon cache miss.
2019-10-27derivations: Introduce 'imported+compiled-modules'.Ludovic Courtès
* guix/derivations.scm (imported+compiled-modules): New procedure. (build-expression->derivation): Use it instead of separate calls to '%imported-modules' and '%compiled-modules'.
2019-08-17derivations: Fix typo in docstring.Ludovic Courtès
* guix/derivations.scm (derivation-build-plan): Fix typo in the docstring.
2019-08-16derivations: Delete duplicate inputs when computing derivation hash.Ludovic Courtès
Fixes <https://bugs.gnu.org/36777>. Reported by Carl Dong <contact@carldong.me>. * guix/derivations.scm (derivation/masked-inputs): Call 'delete-duplicates' on INPUTS. * tests/derivations.scm ("derivation with duplicate fixed-output inputs"): New test.
2019-07-15gexp: 'lowered-gexp-guile' now returns a <derivation-input>.Ludovic Courtès
* guix/derivations.scm (derivation-input-output-path): New procedure. * guix/gexp.scm (lower-gexp): Wrap GUILE in a <derivation-input>. (gexp->derivation): Adjust accordingly. * guix/remote.scm (remote-pipe-for-gexp, remote-eval): Adjust accordingly. * tests/gexp.scm ("lower-gexp"): Adjust accordingly.
2019-07-15derivations: Deprecate the previous calling convention.Ludovic Courtès
We will eventually require #:inputs to be a list of <derivation-input>; store items will have to be passed as #:sources, already interned. * guix/derivations.scm (warn-about-derivation-deprecation): New procedure. (derivation): Add #:%deprecation-warning? parameter. [warn-deprecation]: New macro. [input->derivation-input, input->source]: Use it. (build-expression->derivation): Pass #:%deprecation-warning?. * po/guix/POTFILES.in: Add guix/derivations.scm.
2019-07-15derivations: 'map-derivation' uses the new 'derivation' calling convention.Ludovic Courtès
* guix/derivations.scm (map-derivation)[input->output-paths]: Adjust to deal with an argument that's either 'derivation-input?' or a string. [rewritten-input]: Return a <derivation-input> or a string. Pass #:inputs and #:sources to 'derivation'.
2019-07-15derivations: 'derivation' primitive accepts <derivation> and #:sources.Ludovic Courtès
This brings us closer to the <derivation> data type. * guix/derivations.scm (derivation): Add #:sources parameter. [input->derivation-input]: Add clause for 'derivation-input?'. Honor SOURCES.
2019-07-05derivations: Simplify 'substitution-oracle'.Ludovic Courtès
* guix/derivations.scm (substitution-oracle)[valid?, dependencies]: Remove. [closure]: New procedure. Rename parameter from 'drv' to 'inputs-or-drv' and adjust accordingly. (derivation-build-plan): Pass INPUTS directly to 'substitution-oracle'. * guix/ui.scm (show-what-to-build)[substitutable-info]: Likewise.
2019-07-05derivations: 'derivation-build-plan' recurses on substituables.Ludovic Courtès
This fixes a bug whereby "guix build texlive -n" would report: 0.0 MB would be downloaded: /gnu/store/…-texlive-20180414 instead of: The following derivation would be built: /gnu/store/…-texlive-texmf-20180414.drv 2,595.2 MB would be downloaded: /gnu/store/…-texlive-20180414-texmf.tar.xz /gnu/store/…-texlive-20180414 where 'texlive-texmf' is a non-substitutable dependency of 'texlive'. * guix/derivations.scm (dependencies-of-substitutables): New procedure. (derivation-build-plan): When 'input-substitutable-info' returns true, append the subset of DEPS that corresponds to SUBSTITUABLES to the first argument of 'loop'. * guix/ui.scm (show-what-to-build): Remove half-baked traversal of DOWNLOAD. * tests/derivations.scm ("derivation-build-plan and substitutes, non-substitutable dep"): New test.
2019-07-03derivations: 'substitution-oracle' no longer calls 'read-derivation-from-file'.Ludovic Courtès
This is a followup to 5cf4b26d52bcea382d98fb4becce89be9ee37b55. * guix/derivations.scm (substitution-oracle): Use 'derivation-input-derivation' instead of calling 'read-derivation-from-file'.
2019-07-03derivations: Avoid another call to 'read-derivation-from-file'.Ludovic Courtès
This is a followup to 5cf4b26d52bcea382d98fb4becce89be9ee37b55. * guix/derivations.scm (derivation-path->base16-hash): Rename to... (derivation-base16-hash): ... this, and take a <derivation> rather than a file name. Use 'mlambdaq' instead of 'mlambda'. (derivation/masked-inputs): Adjust accordingly. (invalidate-derivation-caches!): Likewise.
2019-07-03derivations: 'map-derivation' no longer calls 'read-derivation-from-file'.Ludovic Courtès
This is a followup to 5cf4b26d52bcea382d98fb4becce89be9ee37b55. * guix/derivations.scm (map-derivation)[rewritten-input]: Avoid call to 'read-derivation-from-file'.
2019-07-03derivations: Work around pathological memoization behavior.Ludovic Courtès
This works around <https://bugs.gnu.org/36487>. * guix/derivations.scm (invalidate-derivation-caches!): Comment out 'hash-clear!' call.
2019-07-02derivations: Update 'build-derivations' docstring.Ludovic Courtès
This is a followup to 7c690a47381f645ec5ec0a1fd6ffc34dba1b69c2. * guix/derivations.scm (build-derivations): Update docstring.
2019-06-27derivations: 'build-derivations' can be passed derivation inputs.Ludovic Courtès
* guix/derivations.scm (build-derivations): Accept <derivation-input> records among DERIVATIONS. * tests/derivations.scm ("build-derivations with specific output"): Test it.
2019-06-27derivations: 'derivation' preserves pointer equality.Ludovic Courtès
* guix/derivations.scm (derivation): Check if FILE is already in %DERIVATION-CACHE and return it if it is.
2019-06-27derivations: <derivation-input> now aggregates a <derivation>.Ludovic Courtès
Consequently, the whole graph of <derivation> object is readily available without having to go through 'read-derivation-from-file', which could have cache misses if the requested <derivation> object had been GC'd in the meantime. This is an important property for the performance of things like 'derivation-build-plan' that traverse the derivation graph. * guix/derivations.scm (<derivation-input>): Replace 'path' field by 'derivation'. (derivation-input-path): Adjust accordingly. (derivation-input-key): New procedure. (derivation-input-output-paths): Adjust accordingly. (coalesce-duplicate-inputs): Likewise. (derivation-prerequisites): Use 'derivation-input-key' to compute keys for INPUT-SET. (derivation-build-plan): Likewise. (read-derivation): Add optional 'read-derivation-from-file' parameter. [make-input-drvs]: Call it. (write-derivation)[write-input]: Adjust to new <derivation-input>. (derivation/masked-inputs): Likewise, and remove redundant 'coalesce-duplicate-inputs' call. (derivation)[input->derivation-input]: Change to consider only the derivation case. Update call to 'make-derivation-input'. [input->source]: New procedure. Separate sources from inputs. (map-derivation): Adjust to new <derivation-input>. * tests/derivations.scm ("parse & export"): Pass a second argument to 'read-derivation'. ("build-expression->derivation and derivation-prerequisites") ("derivation-prerequisites and valid-derivation-input?"): Adjust to new <derivation-input>.
2019-06-27derivations: Rewrite and replace 'derivations-prerequisites-to-build'.Ludovic Courtès
The new 'derivation-build-plan' procedure has a more appropriate signature: it takes a list of <derivation-inputs> instead of taking one <derivation>. Its body is also much simpler. * guix/derivations.scm (derivation-build-plan): New procedure. (derivation-prerequisites-to-build): Express in terms of 'derivation-build-plan' and mark as deprecated. * tests/derivations.scm: Change 'derivation-prerequisites-to-build' tests to 'derivation-build-plan' and adjust accordingly.
2019-06-27derivations: Add 'derivation-input'.Ludovic Courtès
* guix/derivations.scm (derivation-input): New procedure. * tests/grafts.scm (make-derivation-input): Remove. ("graft-derivation, unused outputs not depended on"): Use 'derivation-input'.
2019-06-19derivations: Add 'derivation-input-derivation'.Ludovic Courtès
* guix/derivations.scm (derivation-input-derivation): New procedure. (derivation-prerequisites): Use it.
2019-06-10store: 'build-things' accepts derivation/output pairs.Ludovic Courtès
This allows callers to request the substitution of a single derivation output. * guix/store.scm (build-things): Accept derivation/output pairs among THINGS. * guix/derivations.scm (build-derivations): Likewise. * tests/store.scm ("substitute + build-things with specific output"): New test. * tests/derivations.scm ("build-derivations with specific output"): New test. * doc/guix.texi (The Store): Adjust accordingly.
2019-04-16derivations: Reduce 'valid-path?' RPCs in 'derivation-prerequisites-to-build'.Ludovic Courtès
On a profile with 280 packages, this reduces the number of 'valid-paths?' RPCs made by 'guix package -nu' from 6K to 500. * guix/derivations.scm (derivation-prerequisites-to-build)[built?]: Memoize 'valid-path?' calls.
2019-01-21store: Rename '&nix-error' to '&store-error'.Ludovic Courtès
* guix/store.scm (&nix-error): Rename to... (&store-error): ... this, and adjust users. (&nix-connection-error): Rename to... (&store-connection-error): ... this, and adjust users. (&nix-protocol-error): Rename to... (&store-protocol-error): ... this, adjust users. (&nix-error, &nix-connection-error, &nix-protocol-error): Define these condition types and their getters as deprecrated aliases. * build-aux/run-system-tests.scm, guix/derivations.scm, guix/grafts.scm, guix/scripts/challenge.scm, guix/scripts/graph.scm, guix/scripts/lint.scm, guix/scripts/offload.scm, guix/serialization.scm, guix/ssh.scm, guix/tests.scm, guix/ui.scm, tests/derivations.scm, tests/gexp.scm, tests/guix-daemon.sh, tests/packages.scm, tests/store.scm, doc/guix.texi: Adjust to use the new names.
2018-11-28derivations: Add properties.Ludovic Courtès
* guix/derivations.scm (derivation): Add #:properties parameter. [user+system-env-vars]: Honor it. (derivation-properties): New procedure. (build-expression->derivation): Add #:properties and pass it to 'derivation'. * guix/gexp.scm (gexp->derivation): Likewise. * tests/derivations.scm ("derivation-properties"): New test. * tests/gexp.scm ("gexp->derivation properties"): New test. * doc/guix.texi (Derivations, G-Expressions): Adjust accordingly.
2018-09-04Switch to Guile-Gcrypt.Ludovic Courtès
This removes (guix hash) and (guix pk-crypto), which now live as part of Guile-Gcrypt (version 0.1.0.) * guix/gcrypt.scm, guix/hash.scm, guix/pk-crypto.scm, tests/hash.scm, tests/pk-crypto.scm: Remove. * configure.ac: Test for Guile-Gcrypt. Remove LIBGCRYPT and LIBGCRYPT_LIBDIR assignments. * m4/guix.m4 (GUIX_ASSERT_LIBGCRYPT_USABLE): Remove. * README: Add Guile-Gcrypt to the dependencies; move libgcrypt as "required unless --disable-daemon". * doc/guix.texi (Requirements): Likewise. * gnu/packages/bash.scm, guix/derivations.scm, guix/docker.scm, guix/git.scm, guix/http-client.scm, guix/import/cpan.scm, guix/import/cran.scm, guix/import/crate.scm, guix/import/elpa.scm, guix/import/gnu.scm, guix/import/hackage.scm, guix/import/texlive.scm, guix/import/utils.scm, guix/nar.scm, guix/pki.scm, guix/scripts/archive.scm, guix/scripts/authenticate.scm, guix/scripts/download.scm, guix/scripts/hash.scm, guix/scripts/pack.scm, guix/scripts/publish.scm, guix/scripts/refresh.scm, guix/scripts/substitute.scm, guix/store.scm, guix/store/deduplication.scm, guix/tests.scm, tests/base32.scm, tests/builders.scm, tests/challenge.scm, tests/cpan.scm, tests/crate.scm, tests/derivations.scm, tests/gem.scm, tests/nar.scm, tests/opam.scm, tests/pki.scm, tests/publish.scm, tests/pypi.scm, tests/store-deduplication.scm, tests/store.scm, tests/substitute.scm: Adjust imports. * gnu/system/vm.scm: Likewise. (guile-sqlite3&co): Rename to... (gcrypt-sqlite3&co): ... this. Add GUILE-GCRYPT. (expression->derivation-in-linux-vm)[config]: Remove. (iso9660-image)[config]: Remove. (qemu-image)[config]: Remove. (system-docker-image)[config]: Remove. * guix/scripts/pack.scm: Adjust imports. (guile-sqlite3&co): Rename to... (gcrypt-sqlite3&co): ... this. Add GUILE-GCRYPT. (self-contained-tarball)[build]: Call 'make-config.scm' without #:libgcrypt argument. (squashfs-image)[libgcrypt]: Remove. [build]: Call 'make-config.scm' without #:libgcrypt. (docker-image)[config, json]: Remove. [build]: Add GUILE-GCRYPT to the extensions Remove (guix config) from the imported modules. * guix/self.scm (specification->package): Remove "libgcrypt", add "guile-gcrypt". (compiled-guix): Remove #:libgcrypt. [guile-gcrypt]: New variable. [dependencies]: Add it. [*core-modules*]: Remove #:libgcrypt from 'make-config.scm' call. Add #:extensions. [*config*]: Remove #:libgcrypt from 'make-config.scm' call. (%dependency-variables): Remove %libgcrypt. (make-config.scm): Remove #:libgcrypt. * build-aux/build-self.scm (guile-gcrypt): New variable. (make-config.scm): Remove #:libgcrypt. (build-program)[fake-gcrypt-hash]: New variable. Add (gcrypt hash) to the imported modules. Adjust load path assignments. * gnu/packages/package-management.scm (guix)[propagated-inputs]: Add GUILE-GCRYPT. [arguments]: In 'wrap-program' phase, add GUILE-GCRYPT to the search path.
2018-01-11derivations: Fix typo in docstring.Ludovic Courtès
* guix/derivations.scm (derivation-prerequisites): Fix typo in docstring.
2017-12-13hydra: Invalidate derivation caches after each architecture evaluation.Ludovic Courtès
This reduces max RSS from 1.3G to 1.0G. * guix/derivations.scm (invalidate-derivation-caches!): New procedure. * build-aux/hydra/gnu-system.scm (hydra-jobs): Use it. Add 'format' call.
2017-12-13derivations: Don't memoize 'derivation-hash'.Ludovic Courtès
This has little or no run-time impact and slightly reduces the memory footprint. * guix/derivations.scm (derivation-hash): Replace 'mlambda' with 'lambda'.
2017-12-13derivations: Split 'derivation-hash' in two procedures.Ludovic Courtès
* guix/derivations.scm (derivation/masked-inputs): New procedure. (derivation-hash): Use it instead of the inline code.
2017-12-13Revert "derivations: 'derivation-hash' assumes inputs are coalesced."Ludovic Courtès
This reverts commit 1d008d9f8c44dfdb808235d451b72f255e72f103. Reported by Rutger Helling <rhelling@mykolab.com>. Fixes <https://bugs.gnu.org/29689>.
2017-12-12derivations: 'derivation-hash' assumes inputs are coalesced.Ludovic Courtès
* guix/derivations.scm (derivation-hash): Remove redundant 'coalesce-duplicate-inputs' call.
2017-12-12derivations: Use 'define-immutable-record-type' as appropriate.Ludovic Courtès
This is a followup to dc673fa1131fb5d1e5ca29acb4a693cfb906986f. * guix/derivations.scm (<derivation-output>, <derivation-input>): Use 'define-immutable-record-type'.
2017-06-12derivations: Introduce 'read-derivation-from-file'.Ludovic Courtès
This avoids the open/fstat/close syscalls upon a cache hit that we had with the previous idiom: (call-with-input-file file read-derivation) where caching happened in 'read-derivation' itself. * guix/derivations.scm (%read-derivation): Rename to... (read-derivation): ... this. (read-derivation-from-file): New procedure. (derivation-prerequisites, substitution-oracle) (derivation-prerequisites-to-build): (derivation-path->output-path, derivation-path->output-paths): (derivation-path->base16-hash, map-derivation): Use 'read-derivation-from-file' instead of (call-with-input-file … read-derivation). * guix/grafts.scm (item->deriver): Likewise. * guix/scripts/build.scm (log-url, options->things-to-build): Likewise. * guix/scripts/graph.scm (file->derivation): Remove. (derivation-dependencies, %derivation-node-type): Use 'read-derivation-from-file' instead. * guix/scripts/offload.scm (guix-offload): Likewise. * guix/scripts/perform-download.scm (guix-perform-download): Likewise. * guix/scripts/publish.scm (load-derivation): Remove. (narinfo-string): Use 'read-derivation-from-file'.
2017-06-02derivations: 'derivation-prerequisites-to-build' returns <substitutable>.Ludovic Courtès
* guix/derivations.scm (derivation-prerequisites-to-build): Rename #:substitutable? to #:substitutable-info. [derivation-substitutable?]: Rename to... [derivation-substitutable-info]: ... this. Return a list of <substitutable>. Second return value is now a list of <substitutable> instead of a list of strings. * guix/ui.scm (show-what-to-build)[substitutable?]: Rename to... [substitutable-info]: ... this. Adjust to new 'derivation-prerequisites-to-build' return value type. * tests/derivations.scm ("derivation-prerequisites-to-build and substitutes"): Adjust. ("derivation-prerequisites-to-build and substitutes, local build"): Likewise.
2017-06-02derivations: 'substitution-oracle' returns a <substitutable>.Ludovic Courtès
* guix/derivations.scm (substitution-oracle): Use 'substitution-path-info' instead of 'substitution-paths'. Turn SUBST into a vhash from path to <substitutable>. Change the returned procedure to provide a <substitutable> instead of a Boolean. * tests/derivations.scm ("substitution-oracle and #:substitute? #f"): Mock 'substitutable-path-info' instead of 'substitutable-paths'.
2017-05-04store: Add store path computation procedures.Ludovic Courtès
* guix/derivations.scm (compressed-hash, store-path) (output-path, fixed-output-path): Move to... * guix/store.scm: ... here.
2017-04-21derivations: Restore UTF-8 encoding of build scripts.Ludovic Courtès
Reported by Mark H Weaver <mhw@netris.org> at <https://lists.gnu.org/archive/html/guix-devel/2017-04/msg00457.html>. * guix/derivations.scm (build-expression->derivation): Use a string output port for the expression. This reverts part of 2dce88d5bbe7a65e101c0734d1c6db44ecc8c299. * tests/derivations.scm ("build-expression->derivation and builder encoding"): New test.
2017-04-20derivations: Avoid string-to-bytevector conversions.Ludovic Courtès
On Guile 2.2.1, this yields a 5% speedup on: guix build libreoffice xmonad certbot -n --no-substitutes --no-build-hook * guix/derivations.scm (derivation->string): Rename to... (derivation->bytevector): ... this. Use 'open-bytevector-output-port' instead of 'call-with-output-string'. (derivation-hash): Remove string-to-bytevector conversion before 'sha256' call. (build-expression->derivation): Use 'add-data-to-store' and an bytevector port instead of a string port for the expression.
2017-03-28derivations: Do not fetch narinfos for non-substitutable items.Ludovic Courtès
This avoids connections to substitute servers for derivations that are not substitutable anyway, such as profiles. Reported by Andy Wingo. * guix/derivations.scm (substitution-oracle): Skip derivations that do not pass 'substitutable-derivation?'. * tests/derivations.scm ("substitution-oracle and #:substitute? #f"): New test.
2017-03-16utils: Move base16 procedures to (guix base16).Ludovic Courtès
* guix/utils.scm (bytevector->base16-string, base16-string->bytevector): Move to... * guix/base16.scm: ... here. New file. * tests/utils.scm ("bytevector->base16-string->bytevector"): Move to... * tests/base16.scm: ... here. New file. * Makefile.am (MODULES): Add guix/base16.scm. (SCM_TESTS): Add tests/base16.scm. * build-aux/download.scm, guix/derivations.scm, guix/docker.scm, guix/import/snix.scm, guix/pk-crypto.scm, guix/scripts/authenticate.scm, guix/scripts/download.scm, guix/scripts/hash.scm, guix/store.scm, tests/hash.scm, tests/pk-crypto.scm: Adjust imports accordingly.
2017-01-28Use 'mlambda' instead of 'memoize'.Ludovic Courtès
* gnu/packages.scm (find-newest-available-packages): Use 'mlambda' instead of (memoize (lambda ...) ...). * gnu/packages/bootstrap.scm (package-with-bootstrap-guile): Likewise. * guix/build-system/gnu.scm (package-with-explicit-inputs)[rewritten-input]: Likewise. * guix/build-system/python.scm (package-with-explicit-python)[transform]: Likewise. * guix/derivations.scm (derivation->string): Likewise. * guix/gnu-maintenance.scm (gnu-package?): Likewise. * guix/modules.scm (module-file-dependencies): Likewise. * guix/scripts/graph.scm (standard-package-set): Likewise. * guix/scripts/lint.scm (official-gnu-packages*): Likewise. * guix/store.scm (store-regexp*): Likewise. * guix/utils.scm (location): Likewise.
2017-01-28Add (guix memoization).Ludovic Courtès
* guix/combinators.scm (memoize): Remove. * guix/memoization.scm: New file. * Makefile.am (MODULES): Add it. * gnu/packages.scm, gnu/packages/bootstrap.scm, guix/build-system/gnu.scm, guix/build-system/python.scm, guix/derivations.scm, guix/gnu-maintenance.scm, guix/import/cran.scm, guix/import/elpa.scm, guix/modules.scm, guix/scripts/build.scm, guix/scripts/graph.scm, guix/scripts/lint.scm, guix/store.scm, guix/utils.scm: Adjust imports accordingly.
2017-01-11derivations: Make <derivation> record datatype immutable.Mathieu Lirzin
* guix/derivations.scm (<derivation>): Make it immutable. (derivation): Use generic 'set-field' instead of ad-hoc functional setter.
2017-01-05derivations: Share a cache between 'derivation' and 'read-derivation'.Ludovic Courtès
This leads a 13% speedup on 'guix build libreoffice -d' and 18% on 'guix build gnome -d'. * guix/derivations.scm (%derivation-cache): New variable. (read-derivation): Use it instead of the private 'cache' variable. (derivation): Populate %DERIVATION-CACHE before returning.