Age | Commit message (Collapse) | Author |
|
* guix/scripts/offload.scm (open-ssh-session): Have 'max-silent-time'
default to #f rather than -1, which is not a valid timeout value.
Adjust body accordingly.
|
|
* guix/scripts/offload.scm: Export 'build-machine' accessors.
|
|
Fixes <https://issues.guix.gnu.org/43773>.
The computed normalized load was previously obtained by dividing the load
average as found in /proc/loadavg by the number of parallel builds defined for
a build machine.
This normalized load didn't allow to compare machines with different number of
cores, as the load average reported by /proc/loadavg can be as high as the
number of cores; thus comparing that value to a fixed threshold of 2.0 would
mean machines with multiple cores were more likely to be flagged as overloaded
compared to single core machines.
This can be fixed by normalizing using the available number of cores instead
of the number of parallel jobs.
* guix/scripts/offload.scm (<build-machine>)[overload-threshold]: New field.
(node-load): Modify to return a normalized load value between 0 and 1, taking
into account the number of cores available.
(normalized-load): Remove procedure.
(report-load): New procedure.
(choose-build-machine): Adjust to use the modified 'node-load' and the new
'report-load' and 'build-machine-overload-threshold' procedures.
(check-machine-status): Adjust.
* doc/guix.texi (Daemon Offload Setup): Document the offload scheduler and the
new 'overload-threshold' field.
|
|
This changes 'guix help' to print a short synopsis for each command and
to group commands by category.
* guix/scripts.scm (synopsis, category): New variables.
(define-command-categories, define-command): New macros.
(%command-categories): New variable.
* guix/ui.scm (<command>): New record type.
(source-file-command): New procedure.
(command-files): Return absolute file names.
(commands): Return a list of <command> records.
(show-guix-help)[display-commands, category-predicate]: New procedures.
Display commands grouped in three categories.
* guix/scripts/archive.scm (guix-archive): Use 'define-command'.
* guix/scripts/authenticate.scm (guix-authenticate): Likewise.
* guix/scripts/build.scm (guix-build): Likewise.
* guix/scripts/challenge.scm (guix-challenge): Likewise.
* guix/scripts/container.scm (guix-container): Likewise.
* guix/scripts/copy.scm (guix-copy): Likewise.
* guix/scripts/deploy.scm (guix-deploy): Likewise.
* guix/scripts/describe.scm (guix-describe): Likewise.
* guix/scripts/download.scm (guix-download): Likewise.
* guix/scripts/edit.scm (guix-edit): Likewise.
* guix/scripts/environment.scm (guix-environment): Likewise.
* guix/scripts/gc.scm (guix-gc): Likewise.
* guix/scripts/git.scm (guix-git): Likewise.
* guix/scripts/graph.scm (guix-graph): Likewise.
* guix/scripts/hash.scm (guix-hash): Likewise.
* guix/scripts/import.scm (guix-import): Likewise.
* guix/scripts/install.scm (guix-install): Likewise.
* guix/scripts/lint.scm (guix-lint): Likewise.
* guix/scripts/offload.scm (guix-offload): Likewise.
* guix/scripts/pack.scm (guix-pack): Likewise.
* guix/scripts/package.scm (guix-package): Likewise.
* guix/scripts/perform-download.scm (guix-perform-download): Likewise.
* guix/scripts/processes.scm (guix-processes): Likewise.
* guix/scripts/publish.scm (guix-publish): Likewise.
* guix/scripts/pull.scm (guix-pull): Likewise.
* guix/scripts/refresh.scm (guix-refresh): Likewise.
* guix/scripts/remove.scm (guix-remove): Likewise.
* guix/scripts/repl.scm (guix-repl): Likewise.
* guix/scripts/search.scm (guix-search): Likewise.
* guix/scripts/show.scm (guix-show): Likewise.
* guix/scripts/size.scm (guix-size): Likewise.
* guix/scripts/substitute.scm (guix-substitute): Likewise.
* guix/scripts/system.scm (guix-system): Likewise.
* guix/scripts/time-machine.scm (guix-time-machine): Likewise.
* guix/scripts/upgrade.scm (guix-upgrade): Likewise.
* guix/scripts/weather.scm (guix-weather): Likewise.
|
|
Fixes <https://bugs.gnu.org/42740>.
* guix/scripts/copy.scm (send-to-remote-host): Keep the result of
'connect-to-remote-daemon' in scope, and explicitly close it after the
call to 'send-files'.
(retrieve-from-remote-host): Explicitly close REMOTE and disconnect
SESSION.
* guix/scripts/offload.scm (transfer-and-offload): Explicitly close
STORE and disconnect SESSION upon completion.
|
|
* guix/scripts/offload.scm (<build-machine>)[systems]: New field.
[system]: Accessor changed to %build-machine-system. Default to #f.
* guix/scripts/offload.scm (build-machine-system): Wrap %build-machine-system
with a deprecation warning.
(build-machine-systems): Access the new systems field or fallback to use
build-machine-system, for backward compatibility.
(machine-matches?): Adjust.
* tests/offload.scm: Add tests...
* Makefile.am (SCM_TESTS): ...and register them.
* doc/guix.texi (Daemon Offload Setup): Update doc.
|
|
* guix/scripts/offload.scm (guix-offload): Update help string.
|
|
* gnu.scm (%try-use-modules): Use 'formatted-message' instead of
'&message'.
* gnu/machine/digital-ocean.scm (maybe-raise-unsupported-configuration-error):
Likewise.
* gnu/machine/ssh.scm (machine-check-file-system-availability): Likewise.
(machine-check-building-for-appropriate-system): Likewise.
(deploy-managed-host): Likewise.
(maybe-raise-unsupported-configuration-error): Likewise.
* gnu/packages.scm (search-patch): Likewise.
* gnu/services.scm (%service-with-default-value): Likewise.
(files->etc-directory): Likewise.
(fold-services): Likewise.
* gnu/system.scm (locale-name->definition*): Likewise.
* gnu/system/mapped-devices.scm (check-device-initrd-modules): Likewise.
(check-luks-device): Likewise.
* guix/channels.scm (latest-channel-instance): Likewise.
* guix/cve.scm (json->cve-items): Likewise.
* guix/git-authenticate.scm (commit-signing-key): Likewise.
(commit-authorized-keys): Likewise.
(authenticate-commit): Likewise.
(verify-introductory-commit): Likewise.
* guix/remote.scm (remote-pipe-for-gexp): Likewise.
* guix/scripts/graph.scm (assert-package): Likewise.
* guix/scripts/offload.scm (private-key-from-file*): Likewise.
* guix/ssh.scm (authenticate-server*): Likewise.
(open-ssh-session): Likewise.
(remote-inferior): Likewise.
* guix/ui.scm (matching-generations): Likewise.
* guix/upstream.scm (package-update): Likewise.
* tests/channels.scm ("latest-channel-instances, missing introduction for 'guix'"):
Catch 'formatted-message?'.
("authenticate-channel, wrong first commit signer"): Likewise.
* tests/lint.scm ("patches: not found"): Adjust message string.
* tests/packages.scm ("patch not found yields a run-time error"): Catch
'formatted-message?'.
* guix/lint.scm (check-patch-file-names): Handle 'formatted-message?'.
(check-derivation): Ditto.
|
|
* guix/scripts/offload.scm (host-key->type+key): Remove.
(open-ssh-session): Replace server authentication code with a call to
'authenticate-server*'.
* guix/ssh.scm (host-key->type+key, authenticate-server*): New
procedures.
|
|
This is a followup to bc69ea2d605810cc32e13ed03d5848b8dc358b61.
* guix/scripts/build.scm (show-build-options-help): Rename
"--no-build-hook" to "--no-offload".
(%standard-build-options): Likewise, and warn when "--no-build-hook" is
passed.
* nix/nix-daemon/guix-daemon.cc (options): Add "--no-offload" and mark
"--no-build-hook" as hidden.
* guix/scripts/offload.scm: Adjust comment.
* doc/guix.texi (Invoking guix-daemon, Common Build Options): Replace
"--no-build-hook" with "--no-offload".
* etc/completion/fish/guix.fish, etc/completion/zsh/_guix: Adjust
accordingly.
|
|
Fixes <https://bugs.gnu.org/37762>.
* guix/scripts/offload.scm (open-ssh-session): Add 'max-silent-time'
parameter. Add call to 'session-set!' before returning SESSION.
(transfer-and-offload): Pass MAX-SILENT-TIME to 'open-ssh-session'.
(%short-timeout): New variable.
(choose-build-machine): Pass %SHORT-TIMEOUT to 'open-ssh-session'.
(check-machine-availability): Likewise.
(check-machine-status): Likewise.
|
|
This is useful when a single machine appears several time, with
different port numbers.
* guix/scripts/offload.scm (machine-slot-file): Add MACHINE's port to
the file name.
|
|
* guix/scripts/offload.scm (lock-file, unlock-file, with-file-lock):
Move to...
* guix/build/syscalls.scm: ... here.
|
|
* guix/scripts/offload.scm (check-machine-status): Report the time
difference for each MACHINE.
|
|
* guix/store.scm (&nix-error): Rename to...
(&store-error): ... this, and adjust users.
(&nix-connection-error): Rename to...
(&store-connection-error): ... this, and adjust users.
(&nix-protocol-error): Rename to...
(&store-protocol-error): ... this, adjust users.
(&nix-error, &nix-connection-error, &nix-protocol-error): Define these
condition types and their getters as deprecrated aliases.
* build-aux/run-system-tests.scm, guix/derivations.scm,
guix/grafts.scm, guix/scripts/challenge.scm,
guix/scripts/graph.scm, guix/scripts/lint.scm,
guix/scripts/offload.scm, guix/serialization.scm,
guix/ssh.scm, guix/tests.scm, guix/ui.scm,
tests/derivations.scm, tests/gexp.scm, tests/guix-daemon.sh,
tests/packages.scm, tests/store.scm, doc/guix.texi: Adjust to use the
new names.
|
|
This extra level of locking turned out to be unnecessary.
* guix/scripts/offload.scm (with-machine-lock): Remove.
(machine-lock-file): Remove.
(acquire-build-slot): Remove surrounding 'with-machine-lock'.
|
|
This lock was unnecessary and it led to a contention when many 'guix
offload' processes are polling for available machines.
* guix/scripts/offload.scm (machine-choice-lock-file): Remove.
(choose-build-machine): Remove surrounding 'with-file-lock (machine-lock-file)'.
|
|
This is a followup to ed7b44370f71126087eb953f36aad8dc4c44109f;
following that commit, 'guix offload test' and 'guix offload status'
would abort with a backtrace instead of clearly diagnosing a missing
'guix' command on the build machine.
* guix/scripts/offload.scm (assert-node-has-guix): Call 'leave' when
NODE is not an inferior. Remove 'catch' blocks for 'node-repl-error'.
(check-machine-availability): Invoke 'assert-node-has-guix' first.
(check-machine-status): Print a warning when 'remote-inferior' returns #f.
|
|
Using inferiors and thus 'guix repl' simplifies setup on build
machines (no need to worry about GUILE_LOAD_PATH etc.)
Furthermore, the 'guix repl -t machine' protocol running in a remote
pipe addresses several issues with the current implementation of nodes
and RREPLs in Guile-SSH: fewer round trips, doesn't leave a 'guile
--listen' process behind it, stateless (since a new process is started
each time), more efficient (the SSH channel can be reused), more
reliable (no 'pgrep', 'pkill', and shellology; see
<https://github.com/artyom-poptsov/guile-ssh/issues/11> as an example.)
* guix/ssh.scm (inferior-remote-eval): New procedure.
(send-files): Use it instead of 'make-node' and 'node-eval'.
* guix/scripts/offload.scm (node-guile-version): New procedure.
(node-free-disk-space, transfer-and-offload, node-load)
(choose-build-machine, assert-node-has-guix): Use 'remote-inferior'
instead of 'make-node' and 'inferior-eval' instead of 'node-eval'.
(assert-node-can-import, assert-node-can-export): Likewise, and add
'session' parameter.
(check-machine-availability): Likewise, and add calls to
'close-inferior' and 'disconnect!'.
(check-machine-status): Likewise.
* doc/guix.texi (Daemon Offload Setup): Remove bit related to 'guile' in
$PATH and $GUILE_LOAD_PATH; mention 'guix' alone.
|
|
Fixes a regression introduced in
bbe66a530a014e8146d63002a5294941e935f863 whereby the actual
load (non-normalized) would be displayed.
* guix/scripts/offload.scm (check-machine-status): Add call to
'normalized-load'.
|
|
Previously, if a remote build would fail due to lack of disk space, this
would be considered a permanent failure and thus cached as a build
failure if the local daemon runs with '--cache-failures'.
* guix/scripts/offload.scm (transfer-and-offload): Upon
'nix-protocol-error?' call 'node-free-disk-space' and return 1 instead
of 100 if the result if lower than 10 MiB.
|
|
Fixes <https://bugs.gnu.org/33378>.
* guix/scripts/offload.scm (node-free-disk-space): New procedure.
(%minimum-disk-space): New variable.
(choose-build-machine): Call 'node-free-disk-space' and take it into
account in addition to LOAD.
(check-machine-status): Display the free disk space.
|
|
* guix/scripts/offload.scm (machine-load): Remove.
(node-load, normalized-load): New procedures.
(choose-build-machine): Call 'open-ssh-session' and 'make-node' from
here; pass the node to 'node-load'.
(check-machine-status): Use 'node-load' instead of 'machine-load'. Call
'disconnect!' on SESSION.
|
|
Reported by Maxim Cournoyer <maxim.cournoyer@gmail.com>
in <https://bugs.gnu.org/31824>.
* guix/scripts/offload.scm (assert-node-has-guix): Fix typo in failure
message; add missing argument.
|
|
* guix/scripts/offload.scm (build-machines): Check the result of FILE.
Ignore it if it's not a list of <build-machine>.
|
|
* guix/scripts/offload.scm (call-with-timeout): New procedure.
(with-timeout): New macro.
(process-request): Use it around 'transfer-and-offload' call.
|
|
Previously we were looking at the load of the past 5 minutes, which
means that, after a build, we could end up waiting for 5 minutes for
that metric to be low enough.
* guix/scripts/offload.scm (machine-load): Compute RAW based on ONE, not
FIVE.
|
|
Fixes <https://bugs.gnu.org/26008>.
Reported by Myles English <mylesenglish@gmail.com>.
* guix/ssh.scm (retrieve-files*): Move error reporting to...
(report-guile-error, report-module-error): ... here. New procedures.
* guix/scripts/offload.scm (assert-node-repl): Use 'report-guile-error'.
(assert-node-has-guix): Explicitly check for 'use-modules' first. Use
'report-module-error'.
|
|
Fixes <https://bugs.gnu.org/28057>.
Reported by Ricardo Wurmus <rekado@elephly.net>.
* guix/scripts/offload.scm (assert-node-has-guix): Catch
'node-repl-error' and call 'leave'.
|
|
This fixes a regression in 'retrieve-files*' introduced in
896fec476f728183b331cbb6e2afb891207b4205, whereby (guix scripts offload)
would not read the initial sexp now sent by the remote host via
'store-export-channel'. This would effectively prevent file retrieval
entirely when offloading.
* guix/ssh.scm (retrieve-files*): New procedure, like former
'retrieve-files' but with an extra #:import parameter.
(retrieve-files): Rewrite in terms of 'retrieve-files*'.
(file-retrieval-port): Make private.
* guix/scripts/offload.scm (transfer-and-offload): Pass #:import to
'retrieve-files*'.
(retrieve-files*): Remove.
|
|
* guix/scripts/offload.scm (check-machine-status): New procedure.
(guix-offload): Call it when the argument is "status".
* doc/guix.texi (Daemon Offload Setup): Document it.
|
|
* guix/scripts/offload.scm (build-machines): Comment out
'(set! %fresh-auto-compile #t)' since with Guile 2.2.3 it could lead to
an actual rebuild of everything that gets loaded from there on. See
<https://bugs.gnu.org/29226>.
* guix/ui.scm (load*): Likewise.
|
|
Previously we would call 'machine-load' once per machine, which was very
costly when there were many machines. Now we arrange to call it only
once on average (when all the machines have the same 'speed' value).
* guix/scripts/offload.scm (random-seed, shuffle): New procedures.
(choose-build-machine)[machines+slots+loads]: Rename to...
[machines+slots]: ... this. Remove load from the tuples therein.
[undecorate]: Adjust accordingly.
[machine-less-loaded-or-faster?]: Remove.
[machine-faster?]: New procedure.
Sort MACHINES+SLOTS according to 'machine-faster?'. Call
'machine-load?' as the last thing.
|
|
The '%slots' list could grow indefinitely; in practice though,
guix-daemon is likely to restart 'guix offload' often enough.
* guix/scripts/offload.scm (%slots): Remove.
(choose-build-machine): Don't 'set!' %SLOTS. Return the acquired slot
as a second value.
(process-request): Adjust accordingly. Release the returned slot after
'transfer-and-offload'.
|
|
This fixes a memory leak that can be seen by running:
(map (lambda _ (machine-load m)) (iota 1000))
* guix/scripts/offload.scm (machine-load): Add call to 'disconnect!'.
|
|
This avoids the open/fstat/close syscalls upon a cache hit that we had
with the previous idiom:
(call-with-input-file file read-derivation)
where caching happened in 'read-derivation' itself.
* guix/derivations.scm (%read-derivation): Rename to...
(read-derivation): ... this.
(read-derivation-from-file): New procedure.
(derivation-prerequisites, substitution-oracle)
(derivation-prerequisites-to-build):
(derivation-path->output-path, derivation-path->output-paths):
(derivation-path->base16-hash, map-derivation): Use
'read-derivation-from-file' instead of (call-with-input-file …
read-derivation).
* guix/grafts.scm (item->deriver): Likewise.
* guix/scripts/build.scm (log-url, options->things-to-build): Likewise.
* guix/scripts/graph.scm (file->derivation): Remove.
(derivation-dependencies, %derivation-node-type): Use
'read-derivation-from-file' instead.
* guix/scripts/offload.scm (guix-offload): Likewise.
* guix/scripts/perform-download.scm (guix-perform-download): Likewise.
* guix/scripts/publish.scm (load-derivation): Remove.
(narinfo-string): Use 'read-derivation-from-file'.
|
|
This allows 'guix publish' threads as well as 'guix substitute' and
'guix offload' processes to be properly labeled in 'top', 'pstree', etc.
* guix/workers.scm (worker-thunk): Add #:thread-name parameter and honor it.
(make-pool): Likewise.
* guix/scripts/publish.scm (http-write): Add calls to 'set-thread-name'
in bodies of 'call-with-new-thread'.
(guix-publish): Call 'set-thread-name'. Pass #:thread-name to 'make-pool'.
* guix/scripts/offload.scm (guix-offload): Call 'set-thread-name'.
* guix/scripts/substitute.scm (guix-substitute): Likewise.
|
|
This avoids collisions with '_' when the latter is used as a 'match'
pattern for instance. See
<https://lists.gnu.org/archive/html/guix-devel/2017-04/msg00464.html>.
* guix/ui.scm: Rename '_' to 'G_'.
* po/guix/Makevars (XGETTEXT_OPTIONS): Adjust accordingly.
* build-aux/compile-all.scm (warnings): Remove 'format'.
* gnu/packages.scm,
gnu/services.scm,
gnu/services/shepherd.scm,
gnu/system.scm,
gnu/system/shadow.scm,
guix/gnupg.scm,
guix/http-client.scm,
guix/import/cpan.scm,
guix/import/elpa.scm,
guix/import/pypi.scm,
guix/nar.scm,
guix/scripts.scm,
guix/scripts/archive.scm,
guix/scripts/authenticate.scm,
guix/scripts/build.scm,
guix/scripts/challenge.scm,
guix/scripts/container.scm,
guix/scripts/container/exec.scm,
guix/scripts/copy.scm,
guix/scripts/download.scm,
guix/scripts/edit.scm,
guix/scripts/environment.scm,
guix/scripts/gc.scm,
guix/scripts/graph.scm,
guix/scripts/hash.scm,
guix/scripts/import.scm,
guix/scripts/import/cpan.scm,
guix/scripts/import/cran.scm,
guix/scripts/import/crate.scm,
guix/scripts/import/elpa.scm,
guix/scripts/import/gem.scm,
guix/scripts/import/gnu.scm,
guix/scripts/import/hackage.scm,
guix/scripts/import/nix.scm,
guix/scripts/import/pypi.scm,
guix/scripts/import/stackage.scm,
guix/scripts/lint.scm,
guix/scripts/offload.scm,
guix/scripts/pack.scm,
guix/scripts/package.scm,
guix/scripts/perform-download.scm,
guix/scripts/publish.scm,
guix/scripts/pull.scm,
guix/scripts/refresh.scm,
guix/scripts/size.scm,
guix/scripts/substitute.scm,
guix/scripts/system.scm,
guix/ssh.scm,
guix/upstream.scm: Use 'G_' instead of '_'. Most of this change was
obtained by running: "sed -i -e's/(_ "/(G_ "/g' `find -name \*.scm`".
|
|
* guix/scripts/offload.scm (host-key->type+key, machine-load)
(process-request, guix-offload): Do not use '_' as a 'match' pattern.
|
|
* guix/scripts/offload.scm (connect-to-remote-daemon)
(store-import-channel, store-export-channel, send-files)
(retrieve-files): Move to (guix ssh).
(nonce): Add optional 'name' parameter and use it.
(retrieve-files*): New procedure.
(transfer-and-offload): Use it instead of 'retrieve-files', and add
first parameter to 'send-files'.
(assert-node-can-import): Likewise.
(assert-node-can-export): Use 'retrieve-files' instead of
'store-export-channel'.
* guix/ssh.scm: New file.
* configure.ac: Use 'GUIX_CHECK_GUILE_SSH' and define 'HAVE_GUILE_SSH'
Automake conditional.
* Makefile.am (MODULES) [HAVE_GUILE_SSH]: Add guix/ssh.scm.
|
|
* guix/scripts/offload.scm (check-machine-availability): Add 'pred'
parameter and honor it.
(guix-offload): for the "test" sub-command, accept an extra 'regexp'
parameter. Pass a second argument to 'check-machine-availability'.
|
|
* guix/scripts/offload.scm (check-machine-availability)[build-machine=?]:
New procedure.
Add call to 'delete-duplicates'.
|
|
* guix/scripts/offload.scm (open-ssh-session): Pass #:knownhosts to
'make-session'.
|
|
* guix/scripts/offload.scm (open-ssh-session): Set #:timeout to 10.
|
|
This fixes a regression introduced in
21531add3205e400707c8fbfd841845f9a71863a whereby the build log would no
longer be sent to FD 4, thereby leading the daemon to not see the build
log.
* guix/scripts/offload.scm (transfer-and-offload): Parameterize
CURRENT-BUILD-OUTPUT-PORT.
|
|
* guix/scripts/offload.scm (send-files): Use 'N_' for possibly plural
message. Write "store item" instead of "store file".
(retrieve-files): Likewise.
|
|
* guix/scripts/offload.scm (<build-machine>)[compression]
[compression-level]: New fields.
(open-ssh-session): Honor them.
* doc/guix.texi (Daemon Offload Setup): Document them.
|
|
* guix/scripts/offload.scm (assert-node-repl, assert-node-has-guix)
(nonce, assert-node-can-import, assert-node-can-export)
(check-machine-availability): New procedures.
(%random-state): New variable.
(guix-offload): Add case for "test".
* doc/guix.texi (Daemon Offload Setup): Document it. Remove obsolete
bit about remote invocation of 'guix build'.
|
|
* guix/scripts/offload.scm (machine-load): Wrap 'open-ssh-session' call
in 'false-if-exception'; return +inf.0 if it returns #f.
|
|
* guix/scripts/offload.scm (open-ssh-session): Check the return value of
'connect!'. Call 'leave' when it's not 'ok.
|