services: Add ganeti.

* gnu/services/ganeti.scm, gnu/tests/ganeti.scm: New files.
* doc/guix.texi (Virtualization Services): Document the Ganeti services.

1 files changed, 646 insertions, 0 deletions

diff --git a/doc/guix.texi b/doc/guix.texi
index 17338ed764..a6ee679b11 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -24975,6 +24975,652 @@ the @code{--snapshot} flag using something along these lines:
           (options '("--hda"))))
 @end lisp
 
+@subsubheading Ganeti
+
+@cindex ganeti
+
+@quotation Note
+This service is considered experimental.  Configuration options may be changed
+in a backwards-incompatible manner, and not all features have been thorougly
+tested. Users of this service are encouraged to share their experience at
+@email{guix-devel@@gnu.org}.
+@end quotation
+
+Ganeti is a virtual machine management system.  It is designed to keep virtual
+machines running on a cluster of servers even in the event of hardware failures,
+and to make maintenance and recovery tasks easy.  It consists of multiple
+services which are described later in this section.  In addition to the Ganeti
+service, you will need the OpenSSH service (@pxref{Networking Services,
+@code{openssh-service-type}}), and update the @file{/etc/hosts} file
+(@pxref{operating-system Reference, @code{hosts-file}}) with the cluster name
+and address (or use a DNS server).
+
+All nodes participating in a Ganeti cluster should have the same Ganeti and
+@file{/etc/hosts} configuration.  Here is an example configuration for a Ganeti
+cluster node that supports multiple storage backends, and installs the
+@code{debootstrap} and @code{guix} @dfn{OS providers}:
+
+@lisp
+(use-package-modules virtualization)
+(use-service-modules base ganeti networking ssh)
+(operating-system
+  ;; @dots{}
+  (host-name "node1")
+  (hosts-file (plain-file "hosts" (format #f "
+127.0.0.1       localhost
+::1             localhost
+
+192.168.1.200   ganeti.example.com
+192.168.1.201   node1.example.com node1
+192.168.1.202   node2.example.com node2
+")))
+
+  ;; Install QEMU so we can use KVM-based instances, and LVM, DRBD and Ceph
+  ;; in order to use the "plain", "drbd" and "rbd" storage backends.
+  (packages (append (map specification->package
+                         '("qemu" "lvm2" "drbd-utils" "ceph"
+                           ;; Add the debootstrap and guix OS providers.
+                           "ganeti-instance-guix" "ganeti-instance-debootstrap"))
+                    %base-packages))
+  (services
+   (append (list (static-networking-service "eth0" "192.168.1.201"
+                                            #:netmask "255.255.255.0"
+                                            #:gateway "192.168.1.254"
+                                            #:name-servers '("192.168.1.252"
+                                                             "192.168.1.253"))
+
+                 ;; Ganeti uses SSH to communicate between nodes.
+                 (service openssh-service-type
+                          (openssh-configuration
+                           (permit-root-login 'without-password)))
+
+                 (service ganeti-service-type
+                          (ganeti-configuration
+                           ;; This list specifies allowed file system paths
+                           ;; for storing virtual machine images.
+                           (file-storage-paths '("/srv/ganeti/file-storage"))
+                           ;; This variable configures a single "variant" for
+                           ;; both Debootstrap and Guix that works with KVM.
+                           (os %default-ganeti-os))))
+           %base-services)))
+@end lisp
+
+Users are advised to read the
+@url{http://docs.ganeti.org/ganeti/master/html/admin.html,Ganeti
+administrators guide} to learn about the various cluster options and
+day-to-day operations.  There is also a
+@url{https://guix.gnu.org/blog/2020/ganeti-on-guix/,blog post}
+describing how to configure a small cluster.
+
+@defvr {Scheme Variable} ganeti-service-type
+This is a service type that includes all the various services that Ganeti
+nodes should run.
+
+Its value is a @code{ganeti-configuration} object that defines the package
+to use for CLI operations, as well as configuration for the various daemons.
+@end defvr
+
+@deftp {Data Type} ganeti-configuration
+The @code{ganeti} service takes the following configuration options:
+
+@table @asis
+@item @code{ganeti} (default: @code{ganeti})
+The @code{ganeti} package to use.  It will be installed to the system profile
+and make @command{gnt-cluster}, @command{gnt-instance}, etc available.  Note
+that the value specified here does not affect the other services as each refer
+to a specific @code{ganeti} package (see below).
+
+@item @code{noded-configuration} (default: @code{(ganeti-noded-configuration)})
+@itemx @code{confd-configuration} (default: @code{(ganeti-confd-configuration)})
+@itemx @code{wconfd-configuration} (default: @code{(ganeti-wconfd-configuration)})
+@itemx @code{luxid-configuration} (default: @code{(ganeti-luxid-configuration)})
+@itemx @code{rapi-configuration} (default: @code{(ganeti-rapi-configuration)})
+@itemx @code{kvmd-configuration} (default: @code{(ganeti-kvmd-configuration)})
+@itemx @code{mond-configuration} (default: @code{(ganeti-mond-configuration)})
+@itemx @code{metad-configuration} (default: @code{(ganeti-metad-configuration)})
+@itemx @code{watcher-configuration} (default: @code{(ganeti-watcher-configuration)})
+@itemx @code{cleaner-configuration} (default: @code{(ganeti-cleaner-configuration)})
+
+These options control the various daemons and cron jobs that are distributed
+with Ganeti.  The possible values for these are described in detail below.
+To override a setting, you must use the configuration type for that service:
+
+@lisp
+(service ganeti-service-type
+         (ganeti-configuration
+          (rapi-configuration
+           (ganeti-rapi-configuration
+            (interface "eth1"))))
+          (watcher-configuration
+           (ganeti-watcher-configuration
+            (rapi-ip "10.0.0.1"))))
+@end lisp
+
+@item @code{file-storage-paths} (default: @code{'()})
+List of allowed directories for file storage backend.
+
+@item @code{os} (default: @code{%default-ganeti-os})
+List of @code{<ganeti-os>} records.
+@end table
+
+In essence @code{ganeti-service-type} is shorthand for declaring each service
+individually:
+
+@lisp
+(service ganeti-noded-service-type)
+(service ganeti-confd-service-type)
+(service ganeti-wconfd-service-type)
+(service ganeti-luxid-service-type)
+(service ganeti-kvmd-service-type)
+(service ganeti-mond-service-type)
+(service ganeti-metad-service-type)
+(service ganeti-watcher-service-type)
+(service ganeti-cleaner-service-type)
+@end lisp
+
+Plus a service extension for @code{etc-service-type} that configures the file
+storage backend and OS variants.
+
+@end deftp
+
+@deftp {Data Type} ganeti-os
+This data type is suitable for passing to the @code{os} configuration of
+Ganeti.  It takes the following parameters:
+
+@table @asis
+@item @code{name}
+The name for this OS provider.  It is only used to specify where the
+configuration ends up.  Setting it to ``debootstrap'' will create
+@file{/etc/ganeti/instance-debootstrap}.
+
+@item @code{extension}
+The file extension for variants of this OS type.  For example
+@file{.conf} or @file{.scm}.
+
+@item @code{variants} (default: @code{'()})
+List of @code{ganeti-os-variant} objects for this OS.
+
+@end table
+@end deftp
+
+@deftp {Data Type} ganeti-os-variant
+This is the data type for a Ganeti OS variant.  It takes the following
+parameters:
+
+@table @asis
+@item @code{name}
+The name of this variant.
+
+@item @code{configuration}
+A configuration file for this variant.
+@end table
+@end deftp
+
+@defvr {Scheme Variable} %default-debootstrap-hooks
+This variable contains hooks to configure networking and the GRUB bootloader.
+@end defvr
+
+@defvr {Scheme Variable} %default-debootstrap-extra-pkgs
+This variable contains a list of packages suitable for a fully-virtualized guest.
+@end defvr
+
+@deftp {Data Type} debootstrap-configuration
+
+This data type creates configuration files suitable for the debootstrap OS provider.
+
+@table @asis
+@item @code{hooks} (default: @code{%default-debootstrap-hooks})
+When not @code{#f}, this must be a G-expression that specifies a directory with
+scripts that will run when the OS is installed.  It can also be a list of
+@code{(name . file-like)} pairs.  For example:
+
+@lisp
+
+`((99-hello-world . ,(plain-file "#!/bin/sh\necho Hello, World")))
+
+@end lisp
+
+That will create a directory with one executable named @code{99-hello-world}
+and run it every time this variant is installed.  If set to @code{#f}, hooks
+in @file{/etc/ganeti/instance-debootstrap/hooks} will be used, if any.
+@item @code{proxy} (default: @code{#f})
+HTTP proxy to use, if any.
+@item @code{mirror} (default: @code{#f})
+The Debian mirror.  Typically something like @code{http://ftp.no.debian.org/debian}.
+The default varies depending on the distribution.
+@item @code{arch} (default: @code{#f})
+The dpkg architecture.  Set to @code{armhf} to debootstrap an ARMv7 instance
+on an AArch64 host.  Default is to use the current system architecture.
+@item @code{suite} (default: @code{"stable"})
+When set, this must be a Debian distribution ``suite'' such as @code{buster}
+or @code{focal}.  If set to @code{#f}, the default for the OS provider is used.
+@item @code{extra-pkgs} (default: @code{%default-debootstrap-extra-pkgs})
+List of extra packages that will get installed by dpkg in addition
+to the minimal system.
+@item @code{components} (default: @code{#f})
+When set, must be a list of Debian repository ``components''.  For example
+@code{'("main" "contrib")}.
+@item @code{generate-cache?} (default: @code{#t})
+Whether to automatically cache the generated debootstrap archive.
+@item @code{clean-cache} (default: @code{14})
+Discard the cache after this amount of days.  Use @code{#f} to never
+clear the cache.
+@item @code{partition-style} (default: @code{'msdos})
+The type of partition to create.  When set, it must be one of
+@code{'msdos}, @code{'none} or a string.
+@item @code{partition-alignment} (default: @code{2048})
+Alignment of the partition in sectors.
+@end table
+@end deftp
+
+@deffn {Scheme Procedure} debootstrap-variant
+This is a helper procedure that creates a @code{ganeti-os-variant} record.  It
+takes two parameters: a name and a @code{debootstrap-configuration} object.
+@end deffn
+
+@deffn {Scheme Procedure} debootstrap-os
+This is a helper procedure that creates a @code{ganeti-os} record.  It takes
+a list of variants created with @code{debootstrap-variant}.
+@end deffn
+
+@deffn {Scheme Procedure} guix-variant
+This is a helper procedure that creates a @code{ganeti-os-variant} record for
+use with the Guix OS provider.  It takes a name and a G-expression that returns
+a ``file-like'' (@pxref{G-Expressions, file-like objects}) object containing a
+Guix System configuration.
+@end deffn
+
+@deffn {Scheme Procedure} guix-os
+This is a helper procedure that creates a @code{ganeti-os} record.  It
+takes a list of variants produced by @code{guix-variant}.
+@end deffn
+
+@defvr {Scheme Variable} %default-debootstrap-variants
+This is a convenience variable to make the debootstrap provider work
+``out of the box'' without users having to declare variants manually.  It
+contains a single debootstrap variant with the default configuration:
+
+@lisp
+(list (debootstrap-variant
+       "default"
+       (debootstrap-configuration))))
+@end lisp
+@end defvr
+
+@defvr {Scheme Variable} %default-guix-variants
+This is a convenience variable to make the Guix OS provider work without
+additional configuration.  It creates a virtual machine that has an SSH
+server, a serial console, and authorizes the Ganeti hosts SSH keys.
+
+@lisp
+(list (guix-variant
+       "default"
+       (file-append ganeti-instance-guix
+                    "/share/doc/ganeti-instance-guix/examples/dynamic.scm"))))
+@end lisp
+@end defvr
+
+Users can implement support for OS providers unbeknownst to Guix by extending
+the @code{ganeti-os} and @code{ganeti-os-variant} records appropriately.
+For example:
+
+@lisp
+(ganeti-os
+ (name "custom")
+ (extension ".conf")
+ (variants
+  (list (ganeti-os-variant
+         (name "foo")
+         (configuration (plain-file "bar" "this is fine"))))))
+@end lisp
+
+That creates @file{/etc/ganeti/instance-custom/variants/foo.conf} which points
+to a file in the store with contents @code{this is fine}.  It also creates
+@file{/etc/ganeti/instance-custom/variants/variants.list} with contents @code{foo}.
+
+Obviously this may not work for all OS providers out there.  If you find the
+interface limiting, please reach out to @email{guix-devel@@gnu.org}.
+
+The rest of this section documents the various services that are included by
+@code{ganeti-service-type}.
+
+@defvr {Scheme Variable} ganeti-noded-service-type
+@command{ganeti-noded} is the daemon responsible for node-specific functions
+within the Ganeti system.  The value of this service must be a
+@code{ganeti-noded-configuration} object.
+@end defvr
+
+@deftp {Data Type} ganeti-noded-configuration
+This is the configuration for the @code{ganeti-noded} service.
+
+@table @asis
+@item @code{ganeti} (default: @code{ganeti})
+The @code{ganeti} package to use for this service.
+
+@item @code{port} (default: @code{1811})
+The TCP port on which the node daemon listens for network requests.
+
+@item @code{address} (default: @code{"0.0.0.0"})
+The network address that the daemon will bind to.  The default address means
+bind to all available addresses.
+
+@item @code{interface} (default: @code{#f})
+When this is set, it must be a specific network interface (e.g.@: @code{eth0})
+that the daemon will bind to.
+
+@item @code{max-clients} (default: @code{20})
+This sets a limit on the maximum number of simultaneous client connections
+that the daemon will handle.  Connections above this count are accepted, but
+no responses will be sent until enough connections have closed.
+
+@item @code{ssl?} (default: @code{#t})
+Whether to use SSL/TLS to encrypt network communications.  The certificate
+is automatically provisioned by the cluster and can be rotated with
+@command{gnt-cluster renew-crypto}.
+
+@item @code{ssl-key} (default: @file{"/var/lib/ganeti/server.pem"})
+This can be used to provide a specific encryption key for TLS communications.
+
+@item @code{ssl-cert} (default: @file{"/var/lib/ganeti/server.pem"})
+This can be used to provide a specific certificate for TLS communications.
+
+@item @code{debug?} (default: @code{#f})
+When true, the daemon performs additional logging for debugging purposes.
+Note that this will leak encryption details to the log files, use with caution.
+
+@end table
+@end deftp
+
+@defvr {Scheme Variable} ganeti-confd-service-type
+@command{ganeti-confd} answers queries related to the configuration of a
+Ganeti cluster.  The purpose of this daemon is to have a highly available
+and fast way to query cluster configuration values.  It is automatically
+active on all @dfn{master candidates}.  The value of this service must be a
+@code{ganeti-confd-configuration} object.
+
+@end defvr
+
+@deftp {Data Type} ganeti-confd-configuration
+This is the configuration for the @code{ganeti-confd} service.
+
+@table @asis
+@item @code{ganeti} (default: @code{ganeti})
+The @code{ganeti} package to use for this service.
+
+@item @code{port} (default: @code{1814})
+The UDP port on which to listen for network requests.
+
+@item @code{address} (default: @code{"0.0.0.0"})
+Network address that the daemon will bind to.
+
+@item @code{debug?} (default: @code{#f})
+When true, the daemon performs additional logging for debugging purposes.
+
+@end table
+@end deftp
+
+@defvr {Scheme Variable} ganeti-wconfd-service-type
+@command{ganeti-wconfd} is the daemon that has authoritative knowledge
+about the cluster configuration and is the only entity that can accept
+changes to it.  All jobs that need to modify the configuration will do so
+by sending appropriate requests to this daemon.  It only runs on the
+@dfn{master node} and will automatically disable itself on other nodes.
+
+The value of this service must be a
+@code{ganeti-wconfd-configuration} object.
+@end defvr
+
+@deftp {Data Type} ganeti-wconfd-configuration
+This is the configuration for the @code{ganeti-wconfd} service.
+
+@table @asis
+@item @code{ganeti} (default: @code{ganeti})
+The @code{ganeti} package to use for this service.
+
+@item @code{no-voting?} (default: @code{#f})
+The daemon will refuse to start if the majority of cluster nodes does not
+agree that it is running on the master node.  Set to @code{#t} to start
+even if a quorum can not be reached (dangerous, use with caution).
+
+@item @code{debug?} (default: @code{#f})
+When true, the daemon performs additional logging for debugging purposes.
+
+@end table
+@end deftp
+
+@defvr {Scheme Variable} ganeti-luxid-service-type
+@command{ganeti-luxid} is a daemon used to answer queries related to the
+configuration and the current live state of a Ganeti cluster.  Additionally,
+it is the authorative daemon for the Ganeti job queue.   Jobs can be
+submitted via this daemon and it schedules and starts them.
+
+It takes a @code{ganeti-luxid-configuration} object.
+@end defvr
+
+@deftp {Data Type} ganeti-luxid-configuration
+This is the configuration for the @code{ganeti-wconfd} service.
+
+@table @asis
+@item @code{ganeti} (default: @code{ganeti})
+The @code{ganeti} package to use for this service.
+
+@item @code{no-voting?} (default: @code{#f})
+The daemon will refuse to start if it cannot verify that the majority of
+cluster nodes believes that it is running on the master node.  Set to
+@code{#t} to ignore such checks and start anyway (this can be dangerous).
+
+@item @code{debug?} (default: @code{#f})
+When true, the daemon performs additional logging for debugging purposes.
+
+@end table
+@end deftp
+
+@defvr {Scheme Variable} ganeti-rapi-service-type
+@command{ganeti-rapi} provides a remote API for Ganeti clusters.  It runs on
+the master node and can be used to perform cluster actions programmatically
+via a JSON-based RPC protocol.
+
+Most query operations are allowed without authentication (unless
+@var{require-authentication?} is set), whereas write operations require
+explicit authorization via the @file{/var/lib/ganeti/rapi/users} file.  See
+the @url{http://docs.ganeti.org/ganeti/master/html/rapi.html, Ganeti Remote
+API documentation} for more information.
+
+The value of this service must be a @code{ganeti-rapi-configuration} object.
+@end defvr
+
+@deftp {Data Type} ganeti-rapi-configuration
+This is the configuration for the @code{ganeti-rapi} service.
+
+@table @asis
+@item @code{ganeti} (default: @code{ganeti})
+The @code{ganeti} package to use for this service.
+
+@item @code{require-authentication?} (default: @code{#f})
+Whether to require authentication even for read-only operations.
+
+@item @code{port} (default: @code{5080})
+The TCP port on which to listen to API requests.
+
+@item @code{address} (default: @code{"0.0.0.0"})
+The network address that the service will bind to.  By default it listens
+on all configured addresses.
+
+@item @code{interface} (default: @code{#f})
+When set, it must specify a specific network interface such as @code{eth0}
+that the daemon will bind to.
+
+@item @code{max-clients} (default: @code{20})
+The maximum number of simultaneous client requests to handle.  Further
+connections are allowed, but no responses are sent until enough connections
+have closed.
+
+@item @code{ssl?} (default: @code{#f})
+Whether to use SSL/TLS encryption on the RAPI port.
+
+@item @code{ssl-key} (default: @file{"/var/lib/ganeti/server.pem"})
+This can be used to provide a specific encryption key for TLS communications.
+
+@item @code{ssl-cert} (default: @file{"/var/lib/ganeti/server.pem"})
+This can be used to provide a specific certificate for TLS communications.
+
+@item @code{debug?} (default: @code{#f})
+When true, the daemon performs additional logging for debugging purposes.
+Note that this will leak encryption details to the log files, use with caution.
+
+@end table
+@end deftp
+
+@defvr {Scheme Variable} ganeti-kvmd-service-type
+@command{ganeti-kvmd} is responsible for determining whether a given KVM
+instance was shut down by an administrator or a user.  Normally Ganeti will
+restart an instance that was not stopped through Ganeti itself.  If the
+cluster option @code{user_shutdown} is true, this daemon monitors the
+@code{QMP} socket provided by QEMU and listens for shutdown events, and
+marks the instance as @dfn{USER_down} instead of @dfn{ERROR_down} when
+it shuts down gracefully by itself.
+
+It takes a @code{ganeti-kvmd-configuration} object.
+@end defvr
+
+@deftp {Data Type} ganeti-kvmd-configuration
+
+@table @asis
+@item @code{ganeti} (default: @code{ganeti})
+The @code{ganeti} package to use for this service.
+
+@item @code{debug?} (default: @code{#f})
+When true, the daemon performs additional logging for debugging purposes.
+
+@end table
+@end deftp
+
+@defvr {Scheme Variable} ganeti-mond-service-type
+@command{ganeti-mond} is an optional daemon that provides Ganeti monitoring
+functionality.  It is responsible for running data collectors and publish the
+collected information through a HTTP interface.
+
+It takes a @code{ganeti-mond-configuration} object.
+@end defvr
+
+@deftp {Data Type} ganeti-mond-configuration
+
+@table @asis
+@item @code{ganeti} (default: @code{ganeti})
+The @code{ganeti} package to use for this service.
+
+@item @code{port} (default: @code{1815})
+The port on which the daemon will listen.
+
+@item @code{address} (default: @code{"0.0.0.0"})
+The network address that the daemon will bind to.  By default it binds to all
+available interfaces.
+
+@item @code{debug?} (default: @code{#f})
+When true, the daemon performs additional logging for debugging purposes.
+
+@end table
+@end deftp
+
+@defvr {Scheme Variable} ganeti-metad-service-type
+@command{ganeti-metad} is an optional daemon that can be used to provide
+information about the cluster to instances or OS install scripts.  It is
+not included in @code{ganeti-service-type} because using it requires
+additional configuration and support in OS providers.
+
+It takes a @code{ganeti-metad-configuration} object.
+@end defvr
+
+@deftp {Data Type} ganeti-metad-configuration
+
+@table @asis
+@item @code{ganeti} (default: @code{ganeti})
+The @code{ganeti} package to use for this service.
+
+@item @code{port} (default: @code{80})
+The port on which the daemon will listen.
+
+@item @code{address} (default: @code{#f})
+If set, the daemon will bind to this address only.  If left unset, the behavior
+depends on the cluster configuration.
+
+@item @code{debug?} (default: @code{#f})
+When true, the daemon performs additional logging for debugging purposes.
+
+@end table
+@end deftp
+
+@defvr {Scheme Variable} ganeti-watcher-service-type
+@command{ganeti-watcher} is a script designed to run periodically and ensure
+the health of a cluster.  It will automatically restart instances that have
+stopped without Ganetis consent, and repairs DRBD links in case a node has
+rebooted.  It also archives old cluster jobs and restarts Ganeti daemons
+that are not running.  If the cluster parameter @code{ensure_node_health}
+is set, the watcher will also shutdown instances and DRBD devices if the
+node it is running on is declared offline by known master candidates.
+
+It can be paused on all nodes with @command{gnt-cluster watcher pause}.
+
+The service takes a @code{ganeti-watcher-configuration} object.
+@end defvr
+
+@deftp {Data Type} ganeti-watcher-configuration
+
+@table @asis
+@item @code{ganeti} (default: @code{ganeti})
+The @code{ganeti} package to use for this service.
+
+@item @code{schedule} (default: @code{'(next-second-from (next-minute (range 0 60 5)))})
+How often to run the script.  The default is every five minutes.
+
+@item @code{rapi-ip} (default: @code{#f})
+This option needs to be specified only if the RAPI daemon is configured to use
+a particular interface or address.  By default the cluster address is used.
+
+@item @code{job-age} (default: @code{(* 6 3600)})
+Archive cluster jobs older than this age, specified in seconds.  The default
+is 6 hours.  This keeps @command{gnt-job list} manageable.
+
+@item @code{verify-disks?} (default: @code{#t})
+If this is @code{#f}, the watcher will not try to repair broken DRBD links
+automatically.  Administrators will need to use @command{gnt-cluster verify-disks}
+manually instead.
+
+@item @code{debug?} (default: @code{#f})
+When @code{#t}, the script performs additional logging for debugging purposes.
+
+@end table
+@end deftp
+
+@defvr {Scheme Variable} ganeti-cleaner-service-type
+@command{ganeti-cleaner} is a script designed to run periodically and remove
+old files from the cluster.  This service type controls two @dfn{cron jobs}:
+one intended for the master node that permanently purges old cluster jobs,
+and one intended for every node that removes expired X509 certificates, keys,
+and outdated @command{ganeti-watcher} information.  Like all Ganeti services,
+it is safe to include even on non-master nodes as it will disable itself as
+necessary.
+
+It takes a @code{ganeti-cleaner-configuration} object.
+@end defvr
+
+@deftp {Data Type} ganeti-cleaner-configuration
+
+@table @asis
+@item @code{ganeti} (default: @code{ganeti})
+The @code{ganeti} package to use for the @command{gnt-cleaner} command.
+
+@item @code{master-schedule} (default: @code{"45 1 * * *"})
+How often to run the master cleaning job.  The default is once per day, at
+01:45:00.
+
+@item @code{node-schedule} (default: @code{"45 2 * * *"})
+How often to run the node cleaning job.  The default is once per day, at
+02:45:00.
+
+@end table
+@end deftp
+
 @node Version Control Services
 @subsection Version Control Services