| Age | Commit message (Expand) | Author |
|---|
| 2017-07-25 | offload: Fix potential file descriptor and memory leak....The '%slots' list could grow indefinitely; in practice though,
guix-daemon is likely to restart 'guix offload' often enough.
* guix/scripts/offload.scm (%slots): Remove.
(choose-build-machine): Don't 'set!' %SLOTS. Return the acquired slot
as a second value.
(process-request): Adjust accordingly. Release the returned slot after
'transfer-and-offload'.
| Ludovic Courtès |
| 2017-07-25 | offload: Disconnect sessions created by 'machine-load'....This fixes a memory leak that can be seen by running:
(map (lambda _ (machine-load m)) (iota 1000))
* guix/scripts/offload.scm (machine-load): Add call to 'disconnect!'.
| Ludovic Courtès |
| 2017-06-12 | derivations: Introduce 'read-derivation-from-file'....This avoids the open/fstat/close syscalls upon a cache hit that we had
with the previous idiom:
(call-with-input-file file read-derivation)
where caching happened in 'read-derivation' itself.
* guix/derivations.scm (%read-derivation): Rename to...
(read-derivation): ... this.
(read-derivation-from-file): New procedure.
(derivation-prerequisites, substitution-oracle)
(derivation-prerequisites-to-build):
(derivation-path->output-path, derivation-path->output-paths):
(derivation-path->base16-hash, map-derivation): Use
'read-derivation-from-file' instead of (call-with-input-file …
read-derivation).
* guix/grafts.scm (item->deriver): Likewise.
* guix/scripts/build.scm (log-url, options->things-to-build): Likewise.
* guix/scripts/graph.scm (file->derivation): Remove.
(derivation-dependencies, %derivation-node-type): Use
'read-derivation-from-file' instead.
* guix/scripts/offload.scm (guix-offload): Likewise.
* guix/scripts/perform-download.scm (guix-perform-download): Likewise.
* guix/scripts/publish.scm (load-derivation): Remove.
(narinfo-string): Use 'read-derivation-from-file'.
| Ludovic Courtès |
| 2017-05-28 | scripts: Set thread names....This allows 'guix publish' threads as well as 'guix substitute' and
'guix offload' processes to be properly labeled in 'top', 'pstree', etc.
* guix/workers.scm (worker-thunk): Add #:thread-name parameter and honor it.
(make-pool): Likewise.
* guix/scripts/publish.scm (http-write): Add calls to 'set-thread-name'
in bodies of 'call-with-new-thread'.
(guix-publish): Call 'set-thread-name'. Pass #:thread-name to 'make-pool'.
* guix/scripts/offload.scm (guix-offload): Call 'set-thread-name'.
* guix/scripts/substitute.scm (guix-substitute): Likewise.
| Ludovic Courtès |
| 2017-05-03 | ui: Rename '_' to 'G_'....This avoids collisions with '_' when the latter is used as a 'match'
pattern for instance. See
<https://lists.gnu.org/archive/html/guix-devel/2017-04/msg00464.html>.
* guix/ui.scm: Rename '_' to 'G_'.
* po/guix/Makevars (XGETTEXT_OPTIONS): Adjust accordingly.
* build-aux/compile-all.scm (warnings): Remove 'format'.
* gnu/packages.scm,
gnu/services.scm,
gnu/services/shepherd.scm,
gnu/system.scm,
gnu/system/shadow.scm,
guix/gnupg.scm,
guix/http-client.scm,
guix/import/cpan.scm,
guix/import/elpa.scm,
guix/import/pypi.scm,
guix/nar.scm,
guix/scripts.scm,
guix/scripts/archive.scm,
guix/scripts/authenticate.scm,
guix/scripts/build.scm,
guix/scripts/challenge.scm,
guix/scripts/container.scm,
guix/scripts/container/exec.scm,
guix/scripts/copy.scm,
guix/scripts/download.scm,
guix/scripts/edit.scm,
guix/scripts/environment.scm,
guix/scripts/gc.scm,
guix/scripts/graph.scm,
guix/scripts/hash.scm,
guix/scripts/import.scm,
guix/scripts/import/cpan.scm,
guix/scripts/import/cran.scm,
guix/scripts/import/crate.scm,
guix/scripts/import/elpa.scm,
guix/scripts/import/gem.scm,
guix/scripts/import/gnu.scm,
guix/scripts/import/hackage.scm,
guix/scripts/import/nix.scm,
guix/scripts/import/pypi.scm,
guix/scripts/import/stackage.scm,
guix/scripts/lint.scm,
guix/scripts/offload.scm,
guix/scripts/pack.scm,
guix/scripts/package.scm,
guix/scripts/perform-download.scm,
guix/scripts/publish.scm,
guix/scripts/pull.scm,
guix/scripts/refresh.scm,
guix/scripts/size.scm,
guix/scripts/substitute.scm,
guix/scripts/system.scm,
guix/ssh.scm,
guix/upstream.scm: Use 'G_' instead of '_'. Most of this change was
obtained by running: "sed -i -e's/(_ "/(G_ "/g' `find -name \*.scm`".
| Ludovic Courtès |
| 2017-04-21 | offload: Avoid using '_' as a 'match' pattern....* guix/scripts/offload.scm (host-key->type+key, machine-load)
(process-request, guix-offload): Do not use '_' as a 'match' pattern.
| Ludovic Courtès |
| 2016-12-31 | Add (guix ssh) module....* guix/scripts/offload.scm (connect-to-remote-daemon)
(store-import-channel, store-export-channel, send-files)
(retrieve-files): Move to (guix ssh).
(nonce): Add optional 'name' parameter and use it.
(retrieve-files*): New procedure.
(transfer-and-offload): Use it instead of 'retrieve-files', and add
first parameter to 'send-files'.
(assert-node-can-import): Likewise.
(assert-node-can-export): Use 'retrieve-files' instead of
'store-export-channel'.
* guix/ssh.scm: New file.
* configure.ac: Use 'GUIX_CHECK_GUILE_SSH' and define 'HAVE_GUILE_SSH'
Automake conditional.
* Makefile.am (MODULES) [HAVE_GUILE_SSH]: Add guix/ssh.scm.
| Ludovic Courtès |
| 2016-12-09 | offload: Allow testing machines that match a regexp....* guix/scripts/offload.scm (check-machine-availability): Add 'pred'
parameter and honor it.
(guix-offload): for the "test" sub-command, accept an extra 'regexp'
parameter. Pass a second argument to 'check-machine-availability'.
| Ludovic Courtès |
| 2016-12-09 | offload: Test each machine only once....* guix/scripts/offload.scm (check-machine-availability)[build-machine=?]:
New procedure.
Add call to 'delete-duplicates'.
| Ludovic Courtès |
| 2016-12-09 | offload: Do not read ~/.ssh/known_hosts....* guix/scripts/offload.scm (open-ssh-session): Pass #:knownhosts to
'make-session'.
| Ludovic Courtès |
| 2016-12-06 | offload: Increase the connection timeout....* guix/scripts/offload.scm (open-ssh-session): Set #:timeout to 10.
| Ludovic Courtès |
| 2016-12-06 | offload: Send the build log to the right file descriptor....This fixes a regression introduced in
21531add3205e400707c8fbfd841845f9a71863a whereby the build log would no
longer be sent to FD 4, thereby leading the daemon to not see the build
log.
* guix/scripts/offload.scm (transfer-and-offload): Parameterize
CURRENT-BUILD-OUTPUT-PORT.
| Ludovic Courtès |
| 2016-12-05 | offload: Fix plural of some messages....* guix/scripts/offload.scm (send-files): Use 'N_' for possibly plural
message. Write "store item" instead of "store file".
(retrieve-files): Likewise.
| Ludovic Courtès |
| 2016-12-05 | offload: Make the compression methods configurable....* guix/scripts/offload.scm (<build-machine>)[compression]
[compression-level]: New fields.
(open-ssh-session): Honor them.
* doc/guix.texi (Daemon Offload Setup): Document them.
| Ludovic Courtès |
| 2016-12-05 | offload: Add "test" sub-command....* guix/scripts/offload.scm (assert-node-repl, assert-node-has-guix)
(nonce, assert-node-can-import, assert-node-can-export)
(check-machine-availability): New procedures.
(%random-state): New variable.
(guix-offload): Add case for "test".
* doc/guix.texi (Daemon Offload Setup): Document it. Remove obsolete
bit about remote invocation of 'guix build'.
| Ludovic Courtès |
| 2016-12-01 | offload: Do not abort when a machine is unreachable....* guix/scripts/offload.scm (machine-load): Wrap 'open-ssh-session' call
in 'false-if-exception'; return +inf.0 if it returns #f.
| Ludovic Courtès |
| 2016-12-01 | offload: Gracefully report connection failures....* guix/scripts/offload.scm (open-ssh-session): Check the return value of
'connect!'. Call 'leave' when it's not 'ok.
| Ludovic Courtès |
| 2016-12-01 | offload: Warn about the lack of zlib support....* guix/scripts/offload.scm (guix-offload): Print a warning when
'zlib-support?' returns false.
| Ludovic Courtès |
| 2016-12-01 | offload: Remove redundant call to 'topologically-sorted' in 'send-file'....* guix/scripts/offload.scm (send-files): Remove call to
'topologically-sorted'.
| Ludovic Courtès |
| 2016-11-26 | offload: Call 'machine-load' only once per machine....This fixes a longstanding issue where 'choose-build-machine' would make
on average O(N log(N)) calls to 'machine-load', plus an extra call for
the selected machine, instead of N calls.
* guix/scripts/offload.scm (machine-load): Add comment.
(machine-power-factor, machine-less-loaded-or-faster?): Remove.
(choose-build-machine)[machines+slots]: Rename to...
[machines+slots+loads]: ... this.
[undecorate]: Adjust accordingly.
[machine-less-loaded-or-faster?]: New procedure.
Remove extra 'machine-load' call in body.
| Ludovic Courtès |
| 2016-11-25 | offload: Drop 'remote-pipe'....* guix/scripts/offload.scm (remote-pipe): Remove.
(machine-load): Use 'open-remote-pipe*' instead of 'remote-pipe'.
| Ludovic Courtès |
| 2016-11-25 | offload: Rewrite to make direct RPCs to the remote daemon....* guix/scripts/offload.scm (<build-machine>)[daemon-socket]: New field.
(connect-to-remote-daemon): New procedure.
(%gc-root-file, register-gc-root, remove-gc-roots, offload): Remove.
(transfer-and-offload): Rewrite using 'connect-to-remote-daemon' and
RPCs over SSH.
(store-import-channel, store-export-channel): New procedures.
(send-files, retrieve-files): Rewrite using these.
| Ludovic Courtès |
| 2016-11-25 | offload: Remove 'with-nar-error-handling' macro....* guix/scripts/offload.scm (with-nar-error-handling): Remove.
(guix-offload): Use 'with-error-handling' instead.
| Ludovic Courtès |
| 2016-11-25 | offload: Reuse SSH session during 'transfer-and-offload'....* guix/scripts/offload.scm (remote-pipe): Replace 'machine' parameter
with 'session'. Remove 'open-ssh-session' call.
(register-gc-root): Replace 'machine' with 'session'. Use '
session-get' instead of 'build-machine-name'.
(remove-gc-roots, offload, send-files, retrieve-files): Likewise.
(transfer-and-offload): Add 'open-ssh-session' call. Handle 'offload'
errors here.
(machine-load): Add call to 'open-ssh-session'.
| Ludovic Courtès |
| 2016-11-25 | offload: Use Guile-SSH instead of GNU lsh....* guix/scripts/offload.scm (<build-machine>)[ssh-options]: Remove.
[host-key, host-key-type]: New fields.
(%lsh-command, %lshg-command, user-lsh-private-key): Remove.
(user-openssh-private-key, private-key-from-file*): New procedures.
(host-key->type+key, open-ssh-session): New procedures.
(remote-pipe): Remove 'mode' parameter. Rewrite in terms of
'open-ssh-session' etc. Update users.
(send-files)[missing-files]: Rewrite using the bidirectional channel
port.
Remove call to 'call-with-compressed-output-port'.
(retrieve-files): Remove call to 'call-with-decompressed-port'.
(machine-load): Remove exit status logic.
* doc/guix.texi (Requirements): Mention Guile-SSH.
(Daemon Offload Setup): Document 'host-key' and 'private-key'. Show the
default value on each @item line.
* m4/guix.m4 (GUIX_CHECK_GUILE_SSH): New macro.
* config-daemon.ac: Use 'GUIX_CHECK_GUILE_SSH'. Set
'HAVE_DAEMON_OFFLOAD_HOOK' as a function of that.
| Ludovic Courtès |
| 2016-10-19 | Use (ice-9 binary-ports) instead of (rnrs io ports)....This reduces the closure of (guix ui) from 123 to 106 modules.
* guix/derivations.scm: Use (ice-9 binary-ports) instead of (rnrs io
ports).
(map-derivation)[substitute-file]: Use 'read-string' instead of
'get-string-all'.
* guix/ftp-client.scm: Likewise.
* guix/hash.scm: Likewise.
* guix/http-client.scm: Likewise.
* guix/pki.scm (ensure-acl, current-acl): Likewise.
* guix/scripts/archive.scm (authorize-key)[read-key]: Likewise.
* guix/scripts/authenticate.scm (read-canonical-sexp)
(read-hash-data): Likewise.
* guix/scripts/download.scm: Likewise.
* guix/scripts/offload.scm (register-gc-root, remove-gc-roots)
(send-files): Likewise.
* guix/scripts/publish.scm (lazy-read-file-sexp): Likewise.
* guix/scripts/refresh.scm: Likewise.
* guix/scripts/substitute.scm (check-acl-initialized): Likewise.
* guix/serialization.scm (read-maybe-utf8-string): Likewise.
* guix/scripts/hash.scm (guix-hash): Use 'force-output' instead of
'flush-output-port'.
* guix/store.scm (process-stderr): Likewise.
* guix/tests.scm: Likewise.
* guix/utils.scm: Use (ice-9 binary-ports) and autoload (rnrs io ports)
for 'make-custom-binary-input-port'.
| Ludovic Courtès |
| 2016-09-20 | ui: Do not shadow '_' where it's used as a literal syntax match....Fixes compilation with Guile 2.1.
Reported by Mu Lei.
* guix/ui.scm (report-load-error)
(warn-about-load-error, read/eval-package-expression): Use 'rest'
instead of '_' as the pattern variable name.
* gnu/packages.scm (%find-package): Likewise.
* guix/scripts/build.scm (transform-package-inputs): Likewise.
* guix/scripts/hash.scm (guix-hash): Likewise.
* guix/scripts/import/gnu.scm (%options, guix-import-gnu): Likewise.
* guix/scripts/import/nix.scm (guix-import-nix): Likewise.
* guix/scripts/offload.scm (build-machines): Likewise.
* guix/scripts/refresh.scm (%options): Likewise.
* guix/scripts/substitute.scm (narinfo-signature->canonical-sexp):
Likewise.
| Ludovic Courtès |
| 2016-05-31 | offload: Use (guix build syscalls)....This is a followup to 4e0ea3eb288c2143b44bf324c64047762c72d3b3.
* guix/scripts/offload.scm: Use (guix build syscalls).
| Ludovic Courtès |
| 2015-09-25 | offload: Use gzip instead of xz for compression on the master....* guix/scripts/offload.scm (send-files): Use gzip --fast instead of xz.
| Ludovic Courtès |
| 2015-07-09 | offload: Add 'ssh-options' field to <build-machine>....* guix/scripts/offload.scm (<build-machine>)[ssh-options]: New field.
(remote-pipe): Use it.
(send-files): Likewise.
| Ludovic Courtès |
| 2015-06-17 | offload: Fix sorting bug in 'choose-build-machine'....* guix/scripts/offload.scm (choose-build-machine)[undecorate]: Return the
boolean result of pred instead of the best machine+slot.
| Mark H Weaver |
| 2015-02-05 | offload: Better report failure to create the GC root directory....Suggested by Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>.
* guix/scripts/offload.scm (register-gc-root)[script]: Replace
'false-if-exception' with a finer-grain 'system-error handler.
Provide the name of MACHINE in 'leave' error message.
| Ludovic Courtès |
| 2015-02-05 | offload: Warn about SSH client issues....Suggested by Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>.
* guix/scripts/offload.scm (remote-pipe): Remove unneeded 'catch'.
(machine-load): Check the exit value upon (close-pipe pipe). Call
'warning' when it is non-zero.
| Ludovic Courtès |
| 2014-10-29 | offload: Remove mutual exclusion on transfers....Suggested by Mark H. Weaver <mhw@netris.org>
at <http://lists.gnu.org/archive/html/guix-devel/2014-10/msg00352.html>.
* guix/scripts/offload.scm (transfer-and-offload): Remove uses of
'with-machine-lock'.
| Ludovic Courtès |
| 2014-10-09 | Break module cycle involving (guix store) and (guix ui)....Before, there was a cycle along the lines of:
(guix store) -> (guix nar) -> (guix ui) -> (guix store)
This caused problems, as discussed at:
http://lists.gnu.org/archive/html/guix-devel/2014-10/msg00109.html
This patch removes cycles in the (guix ...) modules.
* guix/nar.scm (&nar-error, &nar-read-error, dump, write-contents,
read-contents, %archive-version-1, write-file, restore-file): Move to...
* guix/serialization.scm: ... here.
* guix/store.scm: Remove dependency on (guix nar).
* guix/scripts/hash.scm, guix/scripts/offload.scm,
guix/scripts/substitute-binary.scm, tests/nar.scm, tests/store.scm,
tests/substitute-binary.scm: Adjust accordingly.
| Ludovic Courtès |
| 2014-09-20 | offload: Use a total order when sorting available machines....* guix/scripts/offload.scm (machine-less-loaded?, machine-faster?):
Remove.
(machine-power-factor): New procedure.
(machine-less-loaded-or-faster?): Use it.
| Ludovic Courtès |
| 2014-09-20 | offload: Try another machine when the "best" machine is overloaded....* guix/scripts/offload.scm (choose-build-machine): When BEST is
overloaded, try the other machines.
| Ludovic Courtès |
| 2014-08-29 | offload: Ignore EEXIST when registering a .drv as a GC root....Fixes <http://bugs.gnu.org/18115>.
Reported by Mark H Weaver <mhw@netris.org>.
* guix/scripts/offload.scm (register-gc-root)[script]: Wrap 'symlink'
call in "catch 'system-error", and ignore EEXIST errors.
| Ludovic Courtès |
| 2014-08-29 | offload: Ignore unreachable machines....Fixes <http://bugs.gnu.org/18070>.
Reported by Andreas Enge <andreas@enge.fr>.
* guix/scripts/offload.scm (remote-pipe): Augment docstring.
(machine-load): Return +inf.0 instead of 1 if MACHINE does not respond
or responds badly.
| Ludovic Courtès |
| 2014-06-27 | guix {system,offload}: Improve reporting of syntax errors....* guix/scripts/system.scm (read-operating-system) <catch handler>: Add
case for 'syntax-error'. Correct message for default case.
* guix/scripts/offload.scm (build-machines) <catch handler>: Add case
for 'syntax-error'.
* tests/guix-system.sh: New file.
* Makefile.am (SH_TESTS): Add it.
| Ludovic Courtès |
| 2014-04-14 | offload: '{send,receive}-files' wait for completion of the transfer....Fixes situations where the remote 'guix build' is invoked before the
.drv has been completely copied, as reported at
<https://lists.gnu.org/archive/html/guix-devel/2014-04/msg00174.html>.
In some cases 'send-files' would return before the other end is done
importing the files, and so the subsequent 'guix build' invocation would
just miss the .drv file it refers to.
* guix/utils.scm (call-with-decompressed-port): Don't close PORT.
(call-with-compressed-output-port): Likewise.
* tests/utils.scm ("compressed-output-port + decompressed-port"): Adjust
accordingly.
* guix/scripts/offload.scm (send-files): Add explicit (close-pipe pipe)
call.
(retrieve-files): Likewise.
| Ludovic Courtès |
| 2014-04-14 | offload: Better synchronize with remote invocation of 'guix archive --missing'....* guix/scripts/offload.scm (send-files)[missing-files]: Call 'waitpid'
after reading all of MISSING.
| Ludovic Courtès |
| 2014-04-08 | offload: Remove all the GC roots in case of multiple-output derivations....* guix/scripts/offload.scm (remove-gc-root): Rename to...
(remove-gc-roots): ... this.
[builder]: Use 'scandir' and remove all the files starting with
%GC-ROOT-FILE.
(transfer-and-offload): Adjust to renaming; remove
'false-if-exception' wraps.
| Ludovic Courtès |
| 2014-04-08 | offload: Bail out when failing to register a GC root on the build machine....* guix/scripts/offload.scm (register-gc-root): Call 'leave' when
'close-pipe' returns non-zero.
| Ludovic Courtès |
| 2014-04-03 | offload: Prevent the '.drv' and build result from being GC'd....Before that, there was a small time window during which the GC could
wipe the .drv (before 'guix build' has been called), or the build
result (before 'retrieve-files' has started.)
* guix/scripts/offload.scm (remote-pipe): Add #:quote? parameter and
honor it.
(%gc-root-file): New variable.
(register-gc-root, remove-gc-root): New procedures.
(offload): Adjust comment. Run 'guix build' with '-r %GC-ROOT-FILE'.
(transfer-and-offload): Call 'register-gc-root' before
sending (derivation-file-name DRV). Call 'remove-gc-root' after the
call to 'offload' or 'retrieve-files'.
(send-files): Call 'remote-pipe' with #:quote? #f.
(retrieve-files): Likewise.
| Ludovic Courtès |
| 2014-03-31 | offload: Exit with code 100 upon build failures....* guix/scripts/offload.scm (transfer-and-offload): Exit with code 100
upon build failure.
| Ludovic Courtès |
| 2014-03-26 | offload: Remove erroneous 'close-pipe' call....* guix/scripts/offload.scm (send-files): Remove 'close-pipe' call from
'guard' handler ('pipe' here referred to Guile's 'pipe' procedure.)
| Ludovic Courtès |
| 2014-03-26 | offload: Wait for the processes involved in 'guix archive --missing'....* guix/scripts/offload.scm (send-files): Keep the second return value of
'filtered-port'. Call 'waitpid' on it.
| Ludovic Courtès |
| 2014-03-26 | offload: Allow one transfer in each direction simultaneously....* guix/scripts/offload.scm (transfer-and-offload): Use 'upload' lock
instead of 'bandwidth' around 'send-files' calls, and 'download' lock
around 'retrieve-files' call.
| Ludovic Courtès |
| 2014-03-26 | offload: Disable SSH-level compression....* guix/scripts/offload.scm (remote-pipe): Remove '-z' lsh command line
argument. This makes transfers almost an order of magnitude slower.
OpenSSH's ssh(1) man page notes: "Compression is desirable on modem lines
and other slow connections, but will only slow down things on fast
networks." See also
<http://www.spikelab.org/blog/transfer-largedata-scp-tarssh-tarnc-compared.html>.
| Ludovic Courtès |
