| Age | Commit message (Expand) | Author |
|---|
| 2014-04-14 | offload: '{send,receive}-files' wait for completion of the transfer....Fixes situations where the remote 'guix build' is invoked before the
.drv has been completely copied, as reported at
<https://lists.gnu.org/archive/html/guix-devel/2014-04/msg00174.html>.
In some cases 'send-files' would return before the other end is done
importing the files, and so the subsequent 'guix build' invocation would
just miss the .drv file it refers to.
* guix/utils.scm (call-with-decompressed-port): Don't close PORT.
(call-with-compressed-output-port): Likewise.
* tests/utils.scm ("compressed-output-port + decompressed-port"): Adjust
accordingly.
* guix/scripts/offload.scm (send-files): Add explicit (close-pipe pipe)
call.
(retrieve-files): Likewise.
| Ludovic Courtès |
| 2014-04-14 | offload: Better synchronize with remote invocation of 'guix archive --missing'....* guix/scripts/offload.scm (send-files)[missing-files]: Call 'waitpid'
after reading all of MISSING.
| Ludovic Courtès |
| 2014-04-08 | offload: Remove all the GC roots in case of multiple-output derivations....* guix/scripts/offload.scm (remove-gc-root): Rename to...
(remove-gc-roots): ... this.
[builder]: Use 'scandir' and remove all the files starting with
%GC-ROOT-FILE.
(transfer-and-offload): Adjust to renaming; remove
'false-if-exception' wraps.
| Ludovic Courtès |
| 2014-04-08 | offload: Bail out when failing to register a GC root on the build machine....* guix/scripts/offload.scm (register-gc-root): Call 'leave' when
'close-pipe' returns non-zero.
| Ludovic Courtès |
| 2014-04-03 | offload: Prevent the '.drv' and build result from being GC'd....Before that, there was a small time window during which the GC could
wipe the .drv (before 'guix build' has been called), or the build
result (before 'retrieve-files' has started.)
* guix/scripts/offload.scm (remote-pipe): Add #:quote? parameter and
honor it.
(%gc-root-file): New variable.
(register-gc-root, remove-gc-root): New procedures.
(offload): Adjust comment. Run 'guix build' with '-r %GC-ROOT-FILE'.
(transfer-and-offload): Call 'register-gc-root' before
sending (derivation-file-name DRV). Call 'remove-gc-root' after the
call to 'offload' or 'retrieve-files'.
(send-files): Call 'remote-pipe' with #:quote? #f.
(retrieve-files): Likewise.
| Ludovic Courtès |
| 2014-03-31 | offload: Exit with code 100 upon build failures....* guix/scripts/offload.scm (transfer-and-offload): Exit with code 100
upon build failure.
| Ludovic Courtès |
| 2014-03-26 | offload: Remove erroneous 'close-pipe' call....* guix/scripts/offload.scm (send-files): Remove 'close-pipe' call from
'guard' handler ('pipe' here referred to Guile's 'pipe' procedure.)
| Ludovic Courtès |
| 2014-03-26 | offload: Wait for the processes involved in 'guix archive --missing'....* guix/scripts/offload.scm (send-files): Keep the second return value of
'filtered-port'. Call 'waitpid' on it.
| Ludovic Courtès |
| 2014-03-26 | offload: Allow one transfer in each direction simultaneously....* guix/scripts/offload.scm (transfer-and-offload): Use 'upload' lock
instead of 'bandwidth' around 'send-files' calls, and 'download' lock
around 'retrieve-files' call.
| Ludovic Courtès |
| 2014-03-26 | offload: Disable SSH-level compression....* guix/scripts/offload.scm (remote-pipe): Remove '-z' lsh command line
argument. This makes transfers almost an order of magnitude slower.
OpenSSH's ssh(1) man page notes: "Compression is desirable on modem lines
and other slow connections, but will only slow down things on fast
networks." See also
<http://www.spikelab.org/blog/transfer-largedata-scp-tarssh-tarnc-compared.html>.
| Ludovic Courtès |
| 2014-03-24 | offload: Compress files being sent/retrieved....* guix/scripts/offload.scm (send-files): Add "xz -dc |" to the remote
pipe command. Pass PIPE through 'call-with-compressed-output-port'.
Remove 'close-pipe' call.
(retrieve-files): Add "| xz -c" to the remote pipe command. Pass PIPE
through 'call-with-decompressed-port'. Remove 'close-pipe' call.
| Ludovic Courtès |
| 2014-03-19 | offload: Send build logs to file descriptor 4....* guix/scripts/offload.scm (with-error-to-port): New macro.
(remote-pipe): Add #:error-port parameter. Use 'with-error-to-port'
around 'open-pipe*' call.
(build-log-port): New procedure.
(offload): Change #:log-port to default to (build-log-port). Call
'remote-pipe' with #:error-port LOG-PORT.
| Ludovic Courtès |
| 2014-03-13 | offload: Fix 'choose-build-machine' for several machines....* guix/scripts/offload.scm (choose-build-machine)[undecorate]: Turn into
a two-argument procedure.
| Ludovic Courtès |
| 2014-03-13 | offload: Convert the port number to a string when invoking lsh....* guix/scripts/offload.scm (remote-pipe, send-files): Pass the result of
'build-machine-port' to 'number->string'.
| Ludovic Courtès |
| 2014-03-13 | offload: Allow build machines to specify a port number....* guix/scripts/offload.scm (<build-machine>): Add 'port' field.
(remote-pipe, send-files): Use lsh's '-p' option when invoking it.
| Ludovic Courtès |
| 2014-03-09 | offload: Honor absolute build timeouts....* guix/scripts/offload.scm (offload): Remove default value for
'build-timeout'. Pass '--timeout' to the remote 'guix build'
process.
(transfer-and-offload, process-request): Remove default value for
'build-timeout'.
| Ludovic Courtès |
| 2014-03-09 | offload: Move macro definitions before use....* guix/scripts/offload.scm (lock-file, unlock-file, with-file-lock,
with-machine-lock, machine-slot-file, acquire-build-slot,
release-build-slot): Move definitions above their first use.
| Ludovic Courtès |
| 2014-03-09 | offload: Distinguish between 'decline' and 'postpone'....* guix/scripts/offload.scm (transfer-and-offload): New procedure, with
core formerly in 'process-request'.
(choose-build-machine): Remove 'requirements' parameter.
(process-request): Reply 'decline' when none of MACHINES matches the
requirements, and 'postpone' when MACHINES are busy.
| Ludovic Courtès |
| 2014-03-08 | offload: Prevent locked files from being GC'd....* guix/scripts/offload.scm (%slots): New variable.
(choose-build-machine): Add SLOT to '%slots'.
| Ludovic Courtès |
| 2014-03-08 | offload: Fix thinko....* guix/scripts/offload.scm (choose-build-machine)[machine+slots]: Use
'filter-map', not 'filter'.
| Ludovic Courtès |
| 2014-03-08 | offload: Make 'parallel-builds' a hard limit....* guix/scripts/offload.scm (machine-choice-lock-file,
machine-slot-file, acquire-build-slot, release-build-slot): New
procedures.
(choose-build-machine): Operate with (machine-choice-lock-file)
taken. Acquire a build slot for each of MACHINES. Release those not
used.
| Ludovic Courtès |
| 2014-03-08 | offload: Further generalize lock files....* guix/scripts/offload.scm (lock-machine, unlock-machine): Remove.
(lock-file, unlock-file): New procedures.
(with-file-lock): New macro.
(with-machine-lock): Rewrite in terms of 'with-file-lock'.
| Ludovic Courtès |
| 2014-03-08 | offload: Generalize the machine lock mechanism....* guix/scripts/offload.scm (lock-machine): Add 'hint' parameter.
(unlock-machine): Remove 'machine' parameter.
(with-machine-lock): Add 'hint' parameter, and pass it down.
(process-request): Adjust uses of 'with-machine-lock' to pass the
'bandwidth hint.
| Ludovic Courtès |
| 2014-03-06 | offload: Serialize file transfers to build machines....* guix/scripts/offload.scm (machine-lock-file, lock-machine,
unlock-machine): New procedures.
(with-machine-lock): New macro.
(process-request): Wrap 'send-files' and 'retrieve-files' calls in
'with-machine-lock'.
| Ludovic Courtès |
| 2014-03-01 | offload: Comment out attempt to set up an lsh gateway....* guix/scripts/offload.scm (open-ssh-gateway): Comment out.
(process-request): Remove call to 'open-ssh-gateway' and to 'kill'.
| Ludovic Courtès |
| 2014-03-01 | offload: 'remote-pipe' uses the right SSH key....* guix/scripts/offload.scm (remote-pipe): Pass -i when invoking
%LSHG-COMMAND.
| Ludovic Courtès |
| 2014-03-01 | offload: Take the target machine load into account....* guix/scripts/offload.scm (machine-load, machine-less-loaded?,
machine-less-loaded-or-faster?): New procedures.
(choose-build-machine): Use 'machine-less-loaded-or-faster?' when
sorting. Return the head of MACHINES unless it's loaded is >= 2.
| Ludovic Courtès |
| 2014-02-02 | offload: Have 'build-machines' honor its argument....* guix/scripts/offload.scm (build-machines): Honor FILE.
| Ludovic Courtès |
| 2014-01-27 | offload: Do not try to retrieve anything upon build failure....* guix/scripts/offload.scm (offload): Add 'log-port' keyword parameter.
Handle log display here. Return the result of (close-pipe pipe).
(process-request): Adjust 'offload' call site accordingly. Call
'retrieve-files' only when 'offload' returns zero; exit when 'offload'
returns non-zero.
| Ludovic Courtès |
| 2014-01-24 | Add 'guix offload' as a daemon build hook....* nix/nix-daemon/guix-daemon.cc (GUIX_OPT_NO_BUILD_HOOK): New macro.
(options): Add '--no-build-hook'.
(parse_opt): Handle it.
(main)[HAVE_DAEMON_OFFLOAD_HOOK]: Set 'useBuildHook' by default.
Set $NIX_BUILD_HOOK to our offload hook unless otherwise specified.
[!HAVE_DAEMON_OFFLOAD_HOOK]: Clear 'useBuildHook'.
* pre-inst-env.in: Set and export NIX_BUILD_HOOK.
* nix/scripts/offload.in, guix/scripts/offload.scm: New files.
* guix/ui.scm (show-guix-help)[internal?]: Add "offload".
* config-daemon.ac: Call 'GUIX_CHECK_UNBUFFERED_CBIP'.
Instantiate 'nix/scripts/offload'. Set 'BUILD_DAEMON_OFFLOAD'
conditional, and optionally define 'HAVE_DEAMON_OFFLOAD_HOOK' cpp
macro.
* daemon.am (nodist_pkglibexec_SCRIPTS)[BUILD_DAEMON_OFFLOAD]: Add it.
* Makefile.am (MODULES)[BUILD_DAEMON_OFFLOAD]: Add
'guix/scripts/offload.scm'.
(EXTRA_DIST)[!BUILD_DAEMON_OFFLOAD]: Likewise.
* m4/guix.m4 (GUIX_CHECK_UNBUFFERED_CBIP): New macro.
* doc/guix.texi (Setting Up the Daemon): Move most of the body to...
(Build Environment Setup): ... this. New subsection.
(Daemon Offload Setup): New subsection.
| Ludovic Courtès |
