summaryrefslogtreecommitdiff
path: root/guix/grafts.scm
AgeCommit message (Collapse)Author
2021-06-08grafts: Cache the derivation/graft mapping for the whole session.Ludovic Courtès
Partly fixes <https://bugs.gnu.org/41702>. Reported by Lars-Dominik Braun <ldb@leibniz-psychology.org>. Previously, 'graft-derivation' would start anew at every call. When creating a profile with lots of packages, it would potentially do the same work multiple times. The per-session cache addresses this. It increases the derivation-graft-cache hit rate from 77.9% to 80.1% on: GUIX_PROFILING="derivation-graft-cache" ./pre-inst-env \ guix environment --ad-hoc libreoffice inkscape krita darktable -n The effect is more visible on the pathological case below, where cache hit rate goes from 75% to 87% and wall-clock time from 5.0s to 3.5s: GUIX_PROFILING="derivation-graft-cache" ./pre-inst-env \ guix environment --ad-hoc r-learnr --search-paths * guix/grafts.scm (%graft-cache): New variable. (graft-derivation): Add calls to 'store-connection-cache' and 'set-store-connection-cache!'.
2021-06-08grafts: Use SRFI-71 instead of SRFI-11.Ludovic Courtès
* guix/grafts.scm (reference-origins): Use SRFI-71 'let*'.
2021-06-08grafts: Record cache lookups for profiling.Ludovic Courtès
* guix/grafts.scm (record-cache-lookup!): New procedure. (with-cache): Use it.
2021-02-23grafts: Inline 'grafting?' and 'set-grafting'.Ludovic Courtès
As for 'current-target-system' & co., this makes sure we don't needlessly allocate closures. * guix/grafts.scm (grafting?, set-grafting): Inline.
2020-06-06grafts: Improve performance for derivations with many inputs.Ludovic Courtès
Partly fixes <https://bugs.gnu.org/41702>. Reported by Lars-Dominik Braun <ldb@leibniz-psychology.org>. Previously we'd potentially traverse the same sub-graph of DEPS several times. With this patch, command: guix environment --ad-hoc r-learnr --search-paths goes from 11.3s to 4.6s. * guix/grafts.scm (reference-origin): Rename to... (reference-origins): ... this. Change 'item' parameter to 'items'. [lookup-derivers]: New procedure. (cumulative-grafts)[dependency-grafts]: Change 'item' to 'items' and use 'reference-origins'. Remove 'mapm' around 'dependency-grafts' call.
2020-04-02grafts: Simplify access to store item references.Ludovic Courtès
This is a followup to 710854304b1ab29332edcb76f3de532e0724c197. This also slightly reduces the number of 'query-references' RPCs, for instance from 176 to 166 from "guix build emacs -d". * guix/grafts.scm (references-oracle): Remove. (non-self-references): Remove 'references' parameter and add 'store'. Add 'references*' procedure and use it instead of 'references'. Adjust caller accordingly. (cumulative-grafts): Remove 'references' parameter and adjust caller accordingly.
2020-03-29grafts: Don't rely on substitute info for missing store items.Ludovic Courtès
Fixes <https://bugs.gnu.org/22990>. * guix/grafts.scm (references-oracle)[references*]: Remove call to 'substitution-oracle' and to 'references/substitutes'. Use 'references/cached' and 'build-derivations' right away instead.
2019-07-02grafts: 'references-oracle' now takes a derivation input.Ludovic Courtès
That way, if we end up calling 'build-derivations', we'll only build the outputs that we really need. * guix/grafts.scm (references-oracle): Rename 'drv' to 'input'. [output-paths]: Remove. Adjust accordingly. (graft-derivation): Adjust call to 'references-oracle'.
2019-06-19grafts: Avoid 'query-valid-derivers' RPC.Ludovic Courtès
Previously we'd make 502 'query-valid-derivers' RPCs for "guix build vim -d", and after this patch, we don't do any. Furthermore, the previous strategy was "stateful" in the sense that 'item->deriver' could return a derivation that is not the one that was actually computed by this process, but an "equivalent" one (due to fixed-output derivations); which one is chosen would depend on the state of the store. This in turn means that we'd have to call 'read-derivation-from-file' to actually read .drv files (as opposed to getting them from %DERIVATION-CACHE). This is costly and doesn't work with GUIX_DAEMON_SOCKET=ssh://…. * guix/grafts.scm (item->deriver): Remove. (reference-origin): New procedure. (cumulative-grafts): Use it instead of 'item->deriver'.
2019-01-21store: Rename '&nix-error' to '&store-error'.Ludovic Courtès
* guix/store.scm (&nix-error): Rename to... (&store-error): ... this, and adjust users. (&nix-connection-error): Rename to... (&store-connection-error): ... this, and adjust users. (&nix-protocol-error): Rename to... (&store-protocol-error): ... this, adjust users. (&nix-error, &nix-connection-error, &nix-protocol-error): Define these condition types and their getters as deprecrated aliases. * build-aux/run-system-tests.scm, guix/derivations.scm, guix/grafts.scm, guix/scripts/challenge.scm, guix/scripts/graph.scm, guix/scripts/lint.scm, guix/scripts/offload.scm, guix/serialization.scm, guix/ssh.scm, guix/tests.scm, guix/ui.scm, tests/derivations.scm, tests/gexp.scm, tests/guix-daemon.sh, tests/packages.scm, tests/store.scm, doc/guix.texi: Adjust to use the new names.
2018-12-04grafts: Mark as non substitutable.Ludovic Courtès
* guix/grafts.scm (graft-derivation/shallow): Pass #:substitutable? to 'build-expression->derivation'.
2018-11-28grafts: Record metadata as derivation properties.Ludovic Courtès
* guix/grafts.scm (graft-derivation/shallow): Pass #:properties to 'build-expression->derivation'. * tests/grafts.scm ("graft-derivation, grafted item is a direct dependency"): Check the value returned by 'derivation-properties'.
2018-11-12store: Add a functional object cache and use it in 'lower-object'.Ludovic Courtès
This leads to ~25% improvements on things like: guix system build desktop.tmpl --no-grafts -d * guix/store.scm (<nix-server>)[object-cache]: New field. * guix/store.scm (open-connection): Initialize it. (cache-object-mapping, lookup-cached-object, %mcached): New procedures. (mcached): New macro. * guix/gexp.scm (lower-object): Use it. * guix/grafts.scm (grafting?): New procedure.
2018-08-24grafts: Add (guix build debug-link) and use it.Ludovic Courtès
Fixes <https://bugs.gnu.org/19973>. Reported by Mark H Weaver <mhw@netris.org>. * guix/build/debug-link.scm: New file. * guix/build/graft.scm (%graft-hooks): New variable. (graft): Add #:hooks and honor it. * guix/grafts.scm (graft-derivation/shallow): Add (guix build debug-link) and (guix elf) to #:modules. * tests/debug-link.scm: New file. * Makefile.am (MODULES): Add guix/build/debug-link.scm. (SCM_TESTS): Add tests/debug-link.scm.
2018-08-24grafts: Add high-level 'graft' procedure on the build side.Ludovic Courtès
* guix/build/graft.scm (graft): New procedure. * guix/grafts.scm (graft-derivation/shallow)[build]: Use it instead of inline code.
2017-06-12derivations: Introduce 'read-derivation-from-file'.Ludovic Courtès
This avoids the open/fstat/close syscalls upon a cache hit that we had with the previous idiom: (call-with-input-file file read-derivation) where caching happened in 'read-derivation' itself. * guix/derivations.scm (%read-derivation): Rename to... (read-derivation): ... this. (read-derivation-from-file): New procedure. (derivation-prerequisites, substitution-oracle) (derivation-prerequisites-to-build): (derivation-path->output-path, derivation-path->output-paths): (derivation-path->base16-hash, map-derivation): Use 'read-derivation-from-file' instead of (call-with-input-file … read-derivation). * guix/grafts.scm (item->deriver): Likewise. * guix/scripts/build.scm (log-url, options->things-to-build): Likewise. * guix/scripts/graph.scm (file->derivation): Remove. (derivation-dependencies, %derivation-node-type): Use 'read-derivation-from-file' instead. * guix/scripts/offload.scm (guix-offload): Likewise. * guix/scripts/perform-download.scm (guix-perform-download): Likewise. * guix/scripts/publish.scm (load-derivation): Remove. (narinfo-string): Use 'read-derivation-from-file'.
2017-01-25grafts: Do not pull derivation outputs not depended on.Ludovic Courtès
Fixes <http://bugs.gnu.org/24886>. Previously, the grafting derivation of, say, brdf-explorer would pull in qt:doc even though brdf-explorer depends only on qt:out, not qt:doc. * guix/grafts.scm (with-cache): Use 'vhash-assoc' and 'vhash-cons' instead of 'vhash-assq' and 'vhash-consq'. (cumulative-grafts): Pass #:outputs to 'graft-derivation/shallow'. Use OUTPUTS instead of (derivation-output-names drv). (graft-derivation): Add #:outputs parameter; pass it to 'cumulative-grafts'. * tests/grafts.scm (make-derivation-input): New variable. ("graft-derivation, replaced derivation has multiple outputs"): Make sure P2:zzz is not part of the outputs of P3D. ("graft-derivation with #:outputs") ("graft-derivation, unused outputs not depended on"): New tests.
2017-01-24grafts: Shallow grafting can be performed on a subset of the outputs.Ludovic Courtès
* guix/grafts.scm (graft-derivation/shallow): Add #:outputs parameter. [outputs]: Rename to... [output-pairs]: ... this. Adjust 'build-expression->derivation' call accordingly.
2017-01-16grafts: Preserve the cache across recursive calls.Ludovic Courtès
Before this commit, we'd lose the cache across recursive calls to 'cumulative-grafts', which isn't great performance-wise. This bug was already present before d38bc9a9f6feefc465964531520fee5663a12f48. * guix/grafts.scm (with-cache): In the miss case, call 'current-state' after EXP has been evaluated.
2017-01-16grafts: Actually cache grafts during the derivation DAG traversal.Ludovic Courtès
This fixes a regression introduced in d38bc9a9f6feefc465964531520fee5663a12f48 whereby the cache was effectively disabled. Reported by Thomas Danckaert <thomas.danckaert@gmail.com>. * guix/grafts.scm (with-cache): In the cache miss case, wrap body in 'mbegin'.
2017-01-04grafts: Move caching to a new 'with-cache' macro.Ludovic Courtès
* guix/grafts.scm (with-cache): New macro. (cumulative-grafts)[return/cache]: Remove. Use 'with-cache' instead.
2016-10-17grafts: Apply the right grafts in the presence of multiple outputs.Ludovic Courtès
Fixes <http://bugs.gnu.org/24712>. * guix/grafts.scm (cumulative-grafts): Add grafts for all the outputs of DRV. * tests/grafts.scm ("graft-derivation, replaced derivation has multiple outputs"): New test.
2016-10-15grafts: Remove unused variables and confusing monad use.Ludovic Courtès
* guix/grafts.scm (cumulative-grafts)[return/cache]: Use %STATE-MONAD, not %STORE-MONAD. Remove unused 'origins' variable and unnecessary inner 'cache' variable.
2016-10-14grafts: 'graft-derivation' does now introduce grafts that shadow other grafts.Ludovic Courtès
Partly fixes <http://bugs.gnu.org/24418>. * guix/grafts.scm (cumulative-grafts)[graft-origin?]: New procedure. [dependency-grafts]: Use it in new 'if' around recursive call. * tests/grafts.scm ("graft-derivation, grafts are not shadowed"): New test.
2016-05-25grafts: Create only one grafted variant of each derivation.Ludovic Courtès
Currently, with several grafts applicable to Inkscape, this makes: guix gc -R $(guix build inkscape -d) | wc -l go from 2376 to 2266 (4.6%). * guix/grafts.scm (cumulative-grafts): Pass 'graft-derivation/shallow' the subset of GRAFTS that applies to DRV.
2016-03-14grafts: Update the narinfo cache before building a derivation.Ludovic Courtès
* guix/grafts.scm (references-oracle)[references*]: Add call to 'substitution-oracle'.
2016-03-05grafts: Memoize intermediate results in 'cumulative-grafts'.Ludovic Courtès
The time for: guix build inkscape -n --no-substitutes goes down by 30% (in the presence of 3 replacements among all the packages.) * guix/grafts.scm (cumulative-grafts): Turn into a monadic procedure in %STATE-MONAD. Use the current state as a derivation-to-graft cache. (graft-derivation): Call 'cumulative-grafts' within 'run-with-state'.
2016-03-05grafts: Use dependency information from substitutes when possible.Ludovic Courtès
This avoids starting derivation builds just for the sake of knowing the references of their outputs, thereby restoring the expected behavior of --dry-run when substitutes are available. * guix/grafts.scm (non-self-references): Remove 'store' parameter, and add 'references'. Use it. Update caller. (references-oracle): New variable. (cumulative-grafts): Add 'references' parameter and use it. Update callers. (graft-derivation): Remove 'build-derivations' call. Add call to 'references-oracle'.
2016-03-01grafts: Graft recursively.Ludovic Courtès
Fixes <http://bugs.gnu.org/22139>. * guix/grafts.scm (graft-derivation): Rename to... (graft-derivation/shallow): ... this. (graft-origin-file-name, item->deriver, non-self-references) (cumulative-grafts, graft-derivation): New procedures * tests/grafts.scm ("graft-derivation, grafted item is a direct dependency"): Clarify title. Use 'grafted' instead of 'graft' to refer to the grafted derivation. ("graft-derivation, grafted item is an indirect dependency") ("graft-derivation, no dependencies on grafted output"): New tests. * guix/packages.scm (input-graft): Change to take a package instead of an input. (input-cross-graft): Likewise. (fold-bag-dependencies): New procedure. (bag-grafts): Rewrite in terms of 'fold-bag-dependencies'. * tests/packages.scm ("package-derivation, indirect grafts"): Comment out. * doc/guix.texi (Security Updates): Mention run-time dependencies and recursive grafting.
2016-02-27grafts: Consider all the outputs in the graft mapping.Ludovic Courtès
Before that, outputs of a derivation could be left referring to the ungrafted version of the derivation. * guix/grafts.scm (graft-derivation)[outputs]: Change to a list of name/file pairs. * guix/grafts.scm (graft-derivation)[build]: Add 'old-outputs' variable and use it when computing 'mapping'. Use 'mapping' directly. * tests/grafts.scm ("graft-derivation, multiple outputs"): New test.
2016-02-27grafts: Slight simplification.Ludovic Courtès
* guix/grafts.scm (graft-derivation)[output-names]: Use 'derivation-output-names'.
2016-02-26grafts: Add record type printer.Ludovic Courtès
* guix/grafts.scm (write-graft): New procedure. Register it as a printer for <graft>.
2016-02-22grafts: 'name' parameter of 'graft-derivation' is now optional.Ludovic Courtès
* guix/grafts.scm (graft-derivation): Name 'name' a keyword parameter. * guix/packages.scm (package-derivation, package-cross-derivation): Adjust accordingly. * tests/grafts.scm ("graft-derivation"): Likewise. * tests/packages.scm ("package-derivation, indirect grafts"): Likewise.
2016-02-22derivations: Move grafts to (guix grafts).Ludovic Courtès
* guix/derivations.scm (<graft>, graft-derivation, %graft?) (set-grafting): Move to... * guix/grafts.scm: ... here. New file. * guix/gexp.scm, guix/packages.scm, tests/packages.scm, guix/scripts/build.scm: Use it. * Makefile.am (MODULES): Add it. (SCM_TESTS): Add tests/grafts.scm. * tests/derivations.scm ("graft-derivation"): Move to... * tests/grafts.scm: ... here. New file.
generated by cgit v1.2.3 (git 2.46.2) at 2024-11-06 09:33:26 +0000