| Age | Commit message (Expand) | Author |
|---|
| 2017-10-12 | offload: Reduce the number of calls to 'machine-load'....Previously we would call 'machine-load' once per machine, which was very
costly when there were many machines. Now we arrange to call it only
once on average (when all the machines have the same 'speed' value).
* guix/scripts/offload.scm (random-seed, shuffle): New procedures.
(choose-build-machine)[machines+slots+loads]: Rename to...
[machines+slots]: ... this. Remove load from the tuples therein.
[undecorate]: Adjust accordingly.
[machine-less-loaded-or-faster?]: Remove.
[machine-faster?]: New procedure.
Sort MACHINES+SLOTS according to 'machine-faster?'. Call
'machine-load?' as the last thing.
| Ludovic Courtès |
| 2017-07-25 | offload: Fix potential file descriptor and memory leak....The '%slots' list could grow indefinitely; in practice though,
guix-daemon is likely to restart 'guix offload' often enough.
* guix/scripts/offload.scm (%slots): Remove.
(choose-build-machine): Don't 'set!' %SLOTS. Return the acquired slot
as a second value.
(process-request): Adjust accordingly. Release the returned slot after
'transfer-and-offload'.
| Ludovic Courtès |
| 2017-07-25 | offload: Disconnect sessions created by 'machine-load'....This fixes a memory leak that can be seen by running:
(map (lambda _ (machine-load m)) (iota 1000))
* guix/scripts/offload.scm (machine-load): Add call to 'disconnect!'.
| Ludovic Courtès |
| 2017-06-12 | derivations: Introduce 'read-derivation-from-file'....This avoids the open/fstat/close syscalls upon a cache hit that we had
with the previous idiom:
(call-with-input-file file read-derivation)
where caching happened in 'read-derivation' itself.
* guix/derivations.scm (%read-derivation): Rename to...
(read-derivation): ... this.
(read-derivation-from-file): New procedure.
(derivation-prerequisites, substitution-oracle)
(derivation-prerequisites-to-build):
(derivation-path->output-path, derivation-path->output-paths):
(derivation-path->base16-hash, map-derivation): Use
'read-derivation-from-file' instead of (call-with-input-file …
read-derivation).
* guix/grafts.scm (item->deriver): Likewise.
* guix/scripts/build.scm (log-url, options->things-to-build): Likewise.
* guix/scripts/graph.scm (file->derivation): Remove.
(derivation-dependencies, %derivation-node-type): Use
'read-derivation-from-file' instead.
* guix/scripts/offload.scm (guix-offload): Likewise.
* guix/scripts/perform-download.scm (guix-perform-download): Likewise.
* guix/scripts/publish.scm (load-derivation): Remove.
(narinfo-string): Use 'read-derivation-from-file'.
| Ludovic Courtès |
| 2017-05-28 | scripts: Set thread names....This allows 'guix publish' threads as well as 'guix substitute' and
'guix offload' processes to be properly labeled in 'top', 'pstree', etc.
* guix/workers.scm (worker-thunk): Add #:thread-name parameter and honor it.
(make-pool): Likewise.
* guix/scripts/publish.scm (http-write): Add calls to 'set-thread-name'
in bodies of 'call-with-new-thread'.
(guix-publish): Call 'set-thread-name'. Pass #:thread-name to 'make-pool'.
* guix/scripts/offload.scm (guix-offload): Call 'set-thread-name'.
* guix/scripts/substitute.scm (guix-substitute): Likewise.
| Ludovic Courtès |
| 2017-05-03 | ui: Rename '_' to 'G_'....This avoids collisions with '_' when the latter is used as a 'match'
pattern for instance. See
<https://lists.gnu.org/archive/html/guix-devel/2017-04/msg00464.html>.
* guix/ui.scm: Rename '_' to 'G_'.
* po/guix/Makevars (XGETTEXT_OPTIONS): Adjust accordingly.
* build-aux/compile-all.scm (warnings): Remove 'format'.
* gnu/packages.scm,
gnu/services.scm,
gnu/services/shepherd.scm,
gnu/system.scm,
gnu/system/shadow.scm,
guix/gnupg.scm,
guix/http-client.scm,
guix/import/cpan.scm,
guix/import/elpa.scm,
guix/import/pypi.scm,
guix/nar.scm,
guix/scripts.scm,
guix/scripts/archive.scm,
guix/scripts/authenticate.scm,
guix/scripts/build.scm,
guix/scripts/challenge.scm,
guix/scripts/container.scm,
guix/scripts/container/exec.scm,
guix/scripts/copy.scm,
guix/scripts/download.scm,
guix/scripts/edit.scm,
guix/scripts/environment.scm,
guix/scripts/gc.scm,
guix/scripts/graph.scm,
guix/scripts/hash.scm,
guix/scripts/import.scm,
guix/scripts/import/cpan.scm,
guix/scripts/import/cran.scm,
guix/scripts/import/crate.scm,
guix/scripts/import/elpa.scm,
guix/scripts/import/gem.scm,
guix/scripts/import/gnu.scm,
guix/scripts/import/hackage.scm,
guix/scripts/import/nix.scm,
guix/scripts/import/pypi.scm,
guix/scripts/import/stackage.scm,
guix/scripts/lint.scm,
guix/scripts/offload.scm,
guix/scripts/pack.scm,
guix/scripts/package.scm,
guix/scripts/perform-download.scm,
guix/scripts/publish.scm,
guix/scripts/pull.scm,
guix/scripts/refresh.scm,
guix/scripts/size.scm,
guix/scripts/substitute.scm,
guix/scripts/system.scm,
guix/ssh.scm,
guix/upstream.scm: Use 'G_' instead of '_'. Most of this change was
obtained by running: "sed -i -e's/(_ "/(G_ "/g' `find -name \*.scm`".
| Ludovic Courtès |
| 2017-04-21 | offload: Avoid using '_' as a 'match' pattern....* guix/scripts/offload.scm (host-key->type+key, machine-load)
(process-request, guix-offload): Do not use '_' as a 'match' pattern.
| Ludovic Courtès |
| 2016-12-31 | Add (guix ssh) module....* guix/scripts/offload.scm (connect-to-remote-daemon)
(store-import-channel, store-export-channel, send-files)
(retrieve-files): Move to (guix ssh).
(nonce): Add optional 'name' parameter and use it.
(retrieve-files*): New procedure.
(transfer-and-offload): Use it instead of 'retrieve-files', and add
first parameter to 'send-files'.
(assert-node-can-import): Likewise.
(assert-node-can-export): Use 'retrieve-files' instead of
'store-export-channel'.
* guix/ssh.scm: New file.
* configure.ac: Use 'GUIX_CHECK_GUILE_SSH' and define 'HAVE_GUILE_SSH'
Automake conditional.
* Makefile.am (MODULES) [HAVE_GUILE_SSH]: Add guix/ssh.scm.
| Ludovic Courtès |
| 2016-12-09 | offload: Allow testing machines that match a regexp....* guix/scripts/offload.scm (check-machine-availability): Add 'pred'
parameter and honor it.
(guix-offload): for the "test" sub-command, accept an extra 'regexp'
parameter. Pass a second argument to 'check-machine-availability'.
| Ludovic Courtès |
| 2016-12-09 | offload: Test each machine only once....* guix/scripts/offload.scm (check-machine-availability)[build-machine=?]:
New procedure.
Add call to 'delete-duplicates'.
| Ludovic Courtès |
| 2016-12-09 | offload: Do not read ~/.ssh/known_hosts....* guix/scripts/offload.scm (open-ssh-session): Pass #:knownhosts to
'make-session'.
| Ludovic Courtès |
| 2016-12-06 | offload: Increase the connection timeout....* guix/scripts/offload.scm (open-ssh-session): Set #:timeout to 10.
| Ludovic Courtès |
| 2016-12-06 | offload: Send the build log to the right file descriptor....This fixes a regression introduced in
21531add3205e400707c8fbfd841845f9a71863a whereby the build log would no
longer be sent to FD 4, thereby leading the daemon to not see the build
log.
* guix/scripts/offload.scm (transfer-and-offload): Parameterize
CURRENT-BUILD-OUTPUT-PORT.
| Ludovic Courtès |
| 2016-12-05 | offload: Fix plural of some messages....* guix/scripts/offload.scm (send-files): Use 'N_' for possibly plural
message. Write "store item" instead of "store file".
(retrieve-files): Likewise.
| Ludovic Courtès |
| 2016-12-05 | offload: Make the compression methods configurable....* guix/scripts/offload.scm (<build-machine>)[compression]
[compression-level]: New fields.
(open-ssh-session): Honor them.
* doc/guix.texi (Daemon Offload Setup): Document them.
| Ludovic Courtès |
| 2016-12-05 | offload: Add "test" sub-command....* guix/scripts/offload.scm (assert-node-repl, assert-node-has-guix)
(nonce, assert-node-can-import, assert-node-can-export)
(check-machine-availability): New procedures.
(%random-state): New variable.
(guix-offload): Add case for "test".
* doc/guix.texi (Daemon Offload Setup): Document it. Remove obsolete
bit about remote invocation of 'guix build'.
| Ludovic Courtès |
| 2016-12-01 | offload: Do not abort when a machine is unreachable....* guix/scripts/offload.scm (machine-load): Wrap 'open-ssh-session' call
in 'false-if-exception'; return +inf.0 if it returns #f.
| Ludovic Courtès |
| 2016-12-01 | offload: Gracefully report connection failures....* guix/scripts/offload.scm (open-ssh-session): Check the return value of
'connect!'. Call 'leave' when it's not 'ok.
| Ludovic Courtès |
| 2016-12-01 | offload: Warn about the lack of zlib support....* guix/scripts/offload.scm (guix-offload): Print a warning when
'zlib-support?' returns false.
| Ludovic Courtès |
| 2016-12-01 | offload: Remove redundant call to 'topologically-sorted' in 'send-file'....* guix/scripts/offload.scm (send-files): Remove call to
'topologically-sorted'.
| Ludovic Courtès |
| 2016-11-26 | offload: Call 'machine-load' only once per machine....This fixes a longstanding issue where 'choose-build-machine' would make
on average O(N log(N)) calls to 'machine-load', plus an extra call for
the selected machine, instead of N calls.
* guix/scripts/offload.scm (machine-load): Add comment.
(machine-power-factor, machine-less-loaded-or-faster?): Remove.
(choose-build-machine)[machines+slots]: Rename to...
[machines+slots+loads]: ... this.
[undecorate]: Adjust accordingly.
[machine-less-loaded-or-faster?]: New procedure.
Remove extra 'machine-load' call in body.
| Ludovic Courtès |
| 2016-11-25 | offload: Drop 'remote-pipe'....* guix/scripts/offload.scm (remote-pipe): Remove.
(machine-load): Use 'open-remote-pipe*' instead of 'remote-pipe'.
| Ludovic Courtès |
| 2016-11-25 | offload: Rewrite to make direct RPCs to the remote daemon....* guix/scripts/offload.scm (<build-machine>)[daemon-socket]: New field.
(connect-to-remote-daemon): New procedure.
(%gc-root-file, register-gc-root, remove-gc-roots, offload): Remove.
(transfer-and-offload): Rewrite using 'connect-to-remote-daemon' and
RPCs over SSH.
(store-import-channel, store-export-channel): New procedures.
(send-files, retrieve-files): Rewrite using these.
| Ludovic Courtès |
| 2016-11-25 | offload: Remove 'with-nar-error-handling' macro....* guix/scripts/offload.scm (with-nar-error-handling): Remove.
(guix-offload): Use 'with-error-handling' instead.
| Ludovic Courtès |
| 2016-11-25 | offload: Reuse SSH session during 'transfer-and-offload'....* guix/scripts/offload.scm (remote-pipe): Replace 'machine' parameter
with 'session'. Remove 'open-ssh-session' call.
(register-gc-root): Replace 'machine' with 'session'. Use '
session-get' instead of 'build-machine-name'.
(remove-gc-roots, offload, send-files, retrieve-files): Likewise.
(transfer-and-offload): Add 'open-ssh-session' call. Handle 'offload'
errors here.
(machine-load): Add call to 'open-ssh-session'.
| Ludovic Courtès |
| 2016-11-25 | offload: Use Guile-SSH instead of GNU lsh....* guix/scripts/offload.scm (<build-machine>)[ssh-options]: Remove.
[host-key, host-key-type]: New fields.
(%lsh-command, %lshg-command, user-lsh-private-key): Remove.
(user-openssh-private-key, private-key-from-file*): New procedures.
(host-key->type+key, open-ssh-session): New procedures.
(remote-pipe): Remove 'mode' parameter. Rewrite in terms of
'open-ssh-session' etc. Update users.
(send-files)[missing-files]: Rewrite using the bidirectional channel
port.
Remove call to 'call-with-compressed-output-port'.
(retrieve-files): Remove call to 'call-with-decompressed-port'.
(machine-load): Remove exit status logic.
* doc/guix.texi (Requirements): Mention Guile-SSH.
(Daemon Offload Setup): Document 'host-key' and 'private-key'. Show the
default value on each @item line.
* m4/guix.m4 (GUIX_CHECK_GUILE_SSH): New macro.
* config-daemon.ac: Use 'GUIX_CHECK_GUILE_SSH'. Set
'HAVE_DAEMON_OFFLOAD_HOOK' as a function of that.
| Ludovic Courtès |
| 2016-10-19 | Use (ice-9 binary-ports) instead of (rnrs io ports)....This reduces the closure of (guix ui) from 123 to 106 modules.
* guix/derivations.scm: Use (ice-9 binary-ports) instead of (rnrs io
ports).
(map-derivation)[substitute-file]: Use 'read-string' instead of
'get-string-all'.
* guix/ftp-client.scm: Likewise.
* guix/hash.scm: Likewise.
* guix/http-client.scm: Likewise.
* guix/pki.scm (ensure-acl, current-acl): Likewise.
* guix/scripts/archive.scm (authorize-key)[read-key]: Likewise.
* guix/scripts/authenticate.scm (read-canonical-sexp)
(read-hash-data): Likewise.
* guix/scripts/download.scm: Likewise.
* guix/scripts/offload.scm (register-gc-root, remove-gc-roots)
(send-files): Likewise.
* guix/scripts/publish.scm (lazy-read-file-sexp): Likewise.
* guix/scripts/refresh.scm: Likewise.
* guix/scripts/substitute.scm (check-acl-initialized): Likewise.
* guix/serialization.scm (read-maybe-utf8-string): Likewise.
* guix/scripts/hash.scm (guix-hash): Use 'force-output' instead of
'flush-output-port'.
* guix/store.scm (process-stderr): Likewise.
* guix/tests.scm: Likewise.
* guix/utils.scm: Use (ice-9 binary-ports) and autoload (rnrs io ports)
for 'make-custom-binary-input-port'.
| Ludovic Courtès |
| 2016-09-20 | ui: Do not shadow '_' where it's used as a literal syntax match....Fixes compilation with Guile 2.1.
Reported by Mu Lei.
* guix/ui.scm (report-load-error)
(warn-about-load-error, read/eval-package-expression): Use 'rest'
instead of '_' as the pattern variable name.
* gnu/packages.scm (%find-package): Likewise.
* guix/scripts/build.scm (transform-package-inputs): Likewise.
* guix/scripts/hash.scm (guix-hash): Likewise.
* guix/scripts/import/gnu.scm (%options, guix-import-gnu): Likewise.
* guix/scripts/import/nix.scm (guix-import-nix): Likewise.
* guix/scripts/offload.scm (build-machines): Likewise.
* guix/scripts/refresh.scm (%options): Likewise.
* guix/scripts/substitute.scm (narinfo-signature->canonical-sexp):
Likewise.
| Ludovic Courtès |
| 2016-05-31 | offload: Use (guix build syscalls)....This is a followup to 4e0ea3eb288c2143b44bf324c64047762c72d3b3.
* guix/scripts/offload.scm: Use (guix build syscalls).
| Ludovic Courtès |
| 2015-09-25 | offload: Use gzip instead of xz for compression on the master....* guix/scripts/offload.scm (send-files): Use gzip --fast instead of xz.
| Ludovic Courtès |
| 2015-07-09 | offload: Add 'ssh-options' field to <build-machine>....* guix/scripts/offload.scm (<build-machine>)[ssh-options]: New field.
(remote-pipe): Use it.
(send-files): Likewise.
| Ludovic Courtès |
| 2015-06-17 | offload: Fix sorting bug in 'choose-build-machine'....* guix/scripts/offload.scm (choose-build-machine)[undecorate]: Return the
boolean result of pred instead of the best machine+slot.
| Mark H Weaver |
| 2015-02-05 | offload: Better report failure to create the GC root directory....Suggested by Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>.
* guix/scripts/offload.scm (register-gc-root)[script]: Replace
'false-if-exception' with a finer-grain 'system-error handler.
Provide the name of MACHINE in 'leave' error message.
| Ludovic Courtès |
| 2015-02-05 | offload: Warn about SSH client issues....Suggested by Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>.
* guix/scripts/offload.scm (remote-pipe): Remove unneeded 'catch'.
(machine-load): Check the exit value upon (close-pipe pipe). Call
'warning' when it is non-zero.
| Ludovic Courtès |
| 2014-10-29 | offload: Remove mutual exclusion on transfers....Suggested by Mark H. Weaver <mhw@netris.org>
at <http://lists.gnu.org/archive/html/guix-devel/2014-10/msg00352.html>.
* guix/scripts/offload.scm (transfer-and-offload): Remove uses of
'with-machine-lock'.
| Ludovic Courtès |
| 2014-10-09 | Break module cycle involving (guix store) and (guix ui)....Before, there was a cycle along the lines of:
(guix store) -> (guix nar) -> (guix ui) -> (guix store)
This caused problems, as discussed at:
http://lists.gnu.org/archive/html/guix-devel/2014-10/msg00109.html
This patch removes cycles in the (guix ...) modules.
* guix/nar.scm (&nar-error, &nar-read-error, dump, write-contents,
read-contents, %archive-version-1, write-file, restore-file): Move to...
* guix/serialization.scm: ... here.
* guix/store.scm: Remove dependency on (guix nar).
* guix/scripts/hash.scm, guix/scripts/offload.scm,
guix/scripts/substitute-binary.scm, tests/nar.scm, tests/store.scm,
tests/substitute-binary.scm: Adjust accordingly.
| Ludovic Courtès |
| 2014-09-20 | offload: Use a total order when sorting available machines....* guix/scripts/offload.scm (machine-less-loaded?, machine-faster?):
Remove.
(machine-power-factor): New procedure.
(machine-less-loaded-or-faster?): Use it.
| Ludovic Courtès |
| 2014-09-20 | offload: Try another machine when the "best" machine is overloaded....* guix/scripts/offload.scm (choose-build-machine): When BEST is
overloaded, try the other machines.
| Ludovic Courtès |
| 2014-08-29 | offload: Ignore EEXIST when registering a .drv as a GC root....Fixes <http://bugs.gnu.org/18115>.
Reported by Mark H Weaver <mhw@netris.org>.
* guix/scripts/offload.scm (register-gc-root)[script]: Wrap 'symlink'
call in "catch 'system-error", and ignore EEXIST errors.
| Ludovic Courtès |
| 2014-08-29 | offload: Ignore unreachable machines....Fixes <http://bugs.gnu.org/18070>.
Reported by Andreas Enge <andreas@enge.fr>.
* guix/scripts/offload.scm (remote-pipe): Augment docstring.
(machine-load): Return +inf.0 instead of 1 if MACHINE does not respond
or responds badly.
| Ludovic Courtès |
| 2014-06-27 | guix {system,offload}: Improve reporting of syntax errors....* guix/scripts/system.scm (read-operating-system) <catch handler>: Add
case for 'syntax-error'. Correct message for default case.
* guix/scripts/offload.scm (build-machines) <catch handler>: Add case
for 'syntax-error'.
* tests/guix-system.sh: New file.
* Makefile.am (SH_TESTS): Add it.
| Ludovic Courtès |
| 2014-04-14 | offload: '{send,receive}-files' wait for completion of the transfer....Fixes situations where the remote 'guix build' is invoked before the
.drv has been completely copied, as reported at
<https://lists.gnu.org/archive/html/guix-devel/2014-04/msg00174.html>.
In some cases 'send-files' would return before the other end is done
importing the files, and so the subsequent 'guix build' invocation would
just miss the .drv file it refers to.
* guix/utils.scm (call-with-decompressed-port): Don't close PORT.
(call-with-compressed-output-port): Likewise.
* tests/utils.scm ("compressed-output-port + decompressed-port"): Adjust
accordingly.
* guix/scripts/offload.scm (send-files): Add explicit (close-pipe pipe)
call.
(retrieve-files): Likewise.
| Ludovic Courtès |
| 2014-04-14 | offload: Better synchronize with remote invocation of 'guix archive --missing'....* guix/scripts/offload.scm (send-files)[missing-files]: Call 'waitpid'
after reading all of MISSING.
| Ludovic Courtès |
| 2014-04-08 | offload: Remove all the GC roots in case of multiple-output derivations....* guix/scripts/offload.scm (remove-gc-root): Rename to...
(remove-gc-roots): ... this.
[builder]: Use 'scandir' and remove all the files starting with
%GC-ROOT-FILE.
(transfer-and-offload): Adjust to renaming; remove
'false-if-exception' wraps.
| Ludovic Courtès |
| 2014-04-08 | offload: Bail out when failing to register a GC root on the build machine....* guix/scripts/offload.scm (register-gc-root): Call 'leave' when
'close-pipe' returns non-zero.
| Ludovic Courtès |
| 2014-04-03 | offload: Prevent the '.drv' and build result from being GC'd....Before that, there was a small time window during which the GC could
wipe the .drv (before 'guix build' has been called), or the build
result (before 'retrieve-files' has started.)
* guix/scripts/offload.scm (remote-pipe): Add #:quote? parameter and
honor it.
(%gc-root-file): New variable.
(register-gc-root, remove-gc-root): New procedures.
(offload): Adjust comment. Run 'guix build' with '-r %GC-ROOT-FILE'.
(transfer-and-offload): Call 'register-gc-root' before
sending (derivation-file-name DRV). Call 'remove-gc-root' after the
call to 'offload' or 'retrieve-files'.
(send-files): Call 'remote-pipe' with #:quote? #f.
(retrieve-files): Likewise.
| Ludovic Courtès |
| 2014-03-31 | offload: Exit with code 100 upon build failures....* guix/scripts/offload.scm (transfer-and-offload): Exit with code 100
upon build failure.
| Ludovic Courtès |
| 2014-03-26 | offload: Remove erroneous 'close-pipe' call....* guix/scripts/offload.scm (send-files): Remove 'close-pipe' call from
'guard' handler ('pipe' here referred to Guile's 'pipe' procedure.)
| Ludovic Courtès |
| 2014-03-26 | offload: Wait for the processes involved in 'guix archive --missing'....* guix/scripts/offload.scm (send-files): Keep the second return value of
'filtered-port'. Call 'waitpid' on it.
| Ludovic Courtès |
| 2014-03-26 | offload: Allow one transfer in each direction simultaneously....* guix/scripts/offload.scm (transfer-and-offload): Use 'upload' lock
instead of 'bandwidth' around 'send-files' calls, and 'download' lock
around 'retrieve-files' call.
| Ludovic Courtès |
