From d423d6892a187c7b365e89964de77879748e4e40 Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Wed, 13 Feb 2019 16:14:40 +0100 Subject: doc: Add a "Development" chapter. * doc/guix.texi (Managing Software the Guix Way): Add reference to the "Development" section. (Invoking guix environment, Invoking guix pack): Move to... (Development): ... here. New node. --- doc/guix.texi | 7962 +++++++++++++++++++++++++++++---------------------------- 1 file changed, 3998 insertions(+), 3964 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index 6980672094..1ac077d98a 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -119,6 +119,7 @@ Project}. * Installation:: Installing Guix. * System Installation:: Installing the whole operating system. * Package Management:: Package installation, upgrade, etc. +* Development:: Guix-aided software development. * Programming Interface:: Using Guix in Scheme. * Utilities:: Package management commands. * System Configuration:: Configuring the operating system. @@ -137,6 +138,11 @@ Project}. @detailmenu --- The Detailed Node Listing --- +Introduction + +* Managing Software the Guix Way:: What's special. +* GNU Distribution:: The packages and tools. + Installation * Binary Installation:: Getting Guix running in no time! @@ -159,7 +165,7 @@ System Installation * USB Stick and DVD Installation:: Preparing the installation medium. * Preparing for Installation:: Networking, partitioning, etc. * Proceeding with the Installation:: The real thing. -* Installing Guix in a VM:: Guix System playground. +* Installing Guix in a VM:: Guix System playground. * Building the Installation Image:: How this comes to be. Package Management @@ -173,7 +179,6 @@ Package Management * Channels:: Customizing the package collection. * Inferiors:: Interacting with another revision of Guix. * Invoking guix describe:: Display information about your Guix revision. -* Invoking guix pack:: Creating software bundles. * Invoking guix archive:: Exporting and importing store files. Substitutes @@ -185,6 +190,11 @@ Substitutes * Substitution Failure:: What happens when substitution fails. * On Trusting Binaries:: How can you trust that binary blob? +Development + +* Invoking guix environment:: Setting up development environments. +* Invoking guix pack:: Creating software bundles. + Programming Interface * Package Modules:: Packages from the programmer's viewpoint. @@ -212,7 +222,6 @@ Utilities * Invoking guix lint:: Finding errors in package definitions. * Invoking guix size:: Profiling disk usage. * Invoking guix graph:: Visualizing the graph of packages. -* Invoking guix environment:: Setting up development environments. * Invoking guix publish:: Sharing substitutes. * Invoking guix challenge:: Challenging substitute servers. * Invoking guix copy:: Copying to and from a remote store. @@ -242,7 +251,7 @@ System Configuration * Initial RAM Disk:: Linux-Libre bootstrapping. * Bootloader Configuration:: Configuring the boot loader. * Invoking guix system:: Instantiating a system configuration. -* Running Guix in a VM:: How to run Guix System in a virtual machine. +* Running Guix in a VM:: How to run Guix System in a virtual machine. * Defining Services:: Adding new service definitions. Services @@ -309,12 +318,18 @@ group everything under the ``Guix'' banner since, after all, Guix System is readily available through the @command{guix system} command, even if you're using a different distro underneath!}. @xref{GNU Distribution}. +@menu +* Managing Software the Guix Way:: What's special. +* GNU Distribution:: The packages and tools. +@end menu + @node Managing Software the Guix Way @section Managing Software the Guix Way @cindex user interfaces Guix provides a command-line package management interface -(@pxref{Invoking guix package}), a set of command-line utilities +(@pxref{Package Management}), tools to help with software development +(@pxref{Development}), command-line utilities for more advanced usage, (@pxref{Utilities}), as well as Scheme programming interfaces (@pxref{Programming Interface}). @cindex build daemon @@ -1762,7 +1777,7 @@ available. * USB Stick and DVD Installation:: Preparing the installation medium. * Preparing for Installation:: Networking, partitioning, etc. * Proceeding with the Installation:: The real thing. -* Installing Guix in a VM:: Guix System playground. +* Installing Guix in a VM:: Guix System playground. * Building the Installation Image:: How this comes to be. @end menu @@ -2431,7 +2446,6 @@ guix package -i emacs-guix * Channels:: Customizing the package collection. * Inferiors:: Interacting with another revision of Guix. * Invoking guix describe:: Display information about your Guix revision. -* Invoking guix pack:: Creating software bundles. * Invoking guix archive:: Exporting and importing store files. @end menu @@ -4040,233 +4054,6 @@ produce a list of channel specifications in Recutils format. Display information about @var{profile}. @end table -@node Invoking guix pack -@section Invoking @command{guix pack} - -Occasionally you want to pass software to people who are not (yet!) -lucky enough to be using Guix. You'd tell them to run @command{guix -package -i @var{something}}, but that's not possible in this case. This -is where @command{guix pack} comes in. - -@quotation Note -If you are looking for ways to exchange binaries among machines that -already run Guix, @pxref{Invoking guix copy}, @ref{Invoking guix -publish}, and @ref{Invoking guix archive}. -@end quotation - -@cindex pack -@cindex bundle -@cindex application bundle -@cindex software bundle -The @command{guix pack} command creates a shrink-wrapped @dfn{pack} or -@dfn{software bundle}: it creates a tarball or some other archive -containing the binaries of the software you're interested in, and all -its dependencies. The resulting archive can be used on any machine that -does not have Guix, and people can run the exact same binaries as those -you have with Guix. The pack itself is created in a bit-reproducible -fashion, so anyone can verify that it really contains the build results -that you pretend to be shipping. - -For example, to create a bundle containing Guile, Emacs, Geiser, and all -their dependencies, you can run: - -@example -$ guix pack guile emacs geiser -@dots{} -/gnu/store/@dots{}-pack.tar.gz -@end example - -The result here is a tarball containing a @file{/gnu/store} directory -with all the relevant packages. The resulting tarball contains a -@dfn{profile} with the three packages of interest; the profile is the -same as would be created by @command{guix package -i}. It is this -mechanism that is used to create Guix's own standalone binary tarball -(@pxref{Binary Installation}). - -Users of this pack would have to run -@file{/gnu/store/@dots{}-profile/bin/guile} to run Guile, which you may -find inconvenient. To work around it, you can create, say, a -@file{/opt/gnu/bin} symlink to the profile: - -@example -guix pack -S /opt/gnu/bin=bin guile emacs geiser -@end example - -@noindent -That way, users can happily type @file{/opt/gnu/bin/guile} and enjoy. - -@cindex relocatable binaries, with @command{guix pack} -What if the recipient of your pack does not have root privileges on -their machine, and thus cannot unpack it in the root file system? In -that case, you will want to use the @code{--relocatable} option (see -below). This option produces @dfn{relocatable binaries}, meaning they -they can be placed anywhere in the file system hierarchy: in the example -above, users can unpack your tarball in their home directory and -directly run @file{./opt/gnu/bin/guile}. - -@cindex Docker, build an image with guix pack -Alternatively, you can produce a pack in the Docker image format using -the following command: - -@example -guix pack -f docker guile emacs geiser -@end example - -@noindent -The result is a tarball that can be passed to the @command{docker load} -command. See the -@uref{https://docs.docker.com/engine/reference/commandline/load/, Docker -documentation} for more information. - -@cindex Singularity, build an image with guix pack -@cindex SquashFS, build an image with guix pack -Yet another option is to produce a SquashFS image with the following -command: - -@example -guix pack -f squashfs guile emacs geiser -@end example - -@noindent -The result is a SquashFS file system image that can either be mounted or -directly be used as a file system container image with the -@uref{http://singularity.lbl.gov, Singularity container execution -environment}, using commands like @command{singularity shell} or -@command{singularity exec}. - -Several command-line options allow you to customize your pack: - -@table @code -@item --format=@var{format} -@itemx -f @var{format} -Produce a pack in the given @var{format}. - -The available formats are: - -@table @code -@item tarball -This is the default format. It produces a tarball containing all the -specified binaries and symlinks. - -@item docker -This produces a tarball that follows the -@uref{https://github.com/docker/docker/blob/master/image/spec/v1.2.md, -Docker Image Specification}. - -@item squashfs -This produces a SquashFS image containing all the specified binaries and -symlinks, as well as empty mount points for virtual file systems like -procfs. -@end table - -@item --relocatable -@itemx -R -Produce @dfn{relocatable binaries}---i.e., binaries that can be placed -anywhere in the file system hierarchy and run from there. For example, -if you create a pack containing Bash with: - -@example -guix pack -R -S /mybin=bin bash -@end example - -@noindent -...@: you can copy that pack to a machine that lacks Guix, and from your -home directory as a normal user, run: - -@example -tar xf pack.tar.gz -./mybin/sh -@end example - -@noindent -In that shell, if you type @code{ls /gnu/store}, you'll notice that -@file{/gnu/store} shows up and contains all the dependencies of -@code{bash}, even though the machine actually lacks @file{/gnu/store} -altogether! That is probably the simplest way to deploy Guix-built -software on a non-Guix machine. - -There's a gotcha though: this technique relies on the @dfn{user -namespace} feature of the kernel Linux, which allows unprivileged users -to mount or change root. Old versions of Linux did not support it, and -some GNU/Linux distributions turn it off; on these systems, programs -from the pack @emph{will fail to run}, unless they are unpacked in the -root file system. - -@item --expression=@var{expr} -@itemx -e @var{expr} -Consider the package @var{expr} evaluates to. - -This has the same purpose as the same-named option in @command{guix -build} (@pxref{Additional Build Options, @code{--expression} in -@command{guix build}}). - -@item --manifest=@var{file} -@itemx -m @var{file} -Use the packages contained in the manifest object returned by the Scheme -code in @var{file}. - -This has a similar purpose as the same-named option in @command{guix -package} (@pxref{profile-manifest, @option{--manifest}}) and uses the -same manifest files. It allows you to define a collection of packages -once and use it both for creating profiles and for creating archives -for use on machines that do not have Guix installed. Note that you can -specify @emph{either} a manifest file @emph{or} a list of packages, -but not both. - -@item --system=@var{system} -@itemx -s @var{system} -Attempt to build for @var{system}---e.g., @code{i686-linux}---instead of -the system type of the build host. - -@item --target=@var{triplet} -@cindex cross-compilation -Cross-build for @var{triplet}, which must be a valid GNU triplet, such -as @code{"mips64el-linux-gnu"} (@pxref{Specifying target triplets, GNU -configuration triplets,, autoconf, Autoconf}). - -@item --compression=@var{tool} -@itemx -C @var{tool} -Compress the resulting tarball using @var{tool}---one of @code{gzip}, -@code{bzip2}, @code{xz}, @code{lzip}, or @code{none} for no compression. - -@item --symlink=@var{spec} -@itemx -S @var{spec} -Add the symlinks specified by @var{spec} to the pack. This option can -appear several times. - -@var{spec} has the form @code{@var{source}=@var{target}}, where -@var{source} is the symlink that will be created and @var{target} is the -symlink target. - -For instance, @code{-S /opt/gnu/bin=bin} creates a @file{/opt/gnu/bin} -symlink pointing to the @file{bin} sub-directory of the profile. - -@item --localstatedir -@itemx --profile-name=@var{name} -Include the ``local state directory'', @file{/var/guix}, in the resulting -pack, and notably the @file{/var/guix/profiles/per-user/root/@var{name}} -profile---by default @var{name} is @code{guix-profile}, which corresponds to -@file{~root/.guix-profile}. - -@file{/var/guix} contains the store database (@pxref{The Store}) as well -as garbage-collector roots (@pxref{Invoking guix gc}). Providing it in -the pack means that the store is ``complete'' and manageable by Guix; -not providing it pack means that the store is ``dead'': items cannot be -added to it or removed from it after extraction of the pack. - -One use case for this is the Guix self-contained binary tarball -(@pxref{Binary Installation}). - -@item --bootstrap -Use the bootstrap binaries to build the pack. This option is only -useful to Guix developers. -@end table - -In addition, @command{guix pack} supports all the common build options -(@pxref{Common Build Options}) and all the package transformation -options (@pxref{Package Transformation Options}). - - @node Invoking guix archive @section Invoking @command{guix archive} @@ -4431,4785 +4218,5032 @@ archive contents coming from possibly untrusted substitute servers. @end table + @c ********************************************************************* -@node Programming Interface -@chapter Programming Interface +@node Development +@chapter Development -GNU Guix provides several Scheme programming interfaces (APIs) to -define, build, and query packages. The first interface allows users to -write high-level package definitions. These definitions refer to -familiar packaging concepts, such as the name and version of a package, -its build system, and its dependencies. These definitions can then be -turned into concrete build actions. +@cindex software development +If you are a software developer, Guix provides tools that you should find +helpful---independently of the language you're developing in. This is what +this chapter is about. -Build actions are performed by the Guix daemon, on behalf of users. In a -standard setup, the daemon has write access to the store---the -@file{/gnu/store} directory---whereas users do not. The recommended -setup also has the daemon perform builds in chroots, under a specific -build users, to minimize interference with the rest of the system. - -@cindex derivation -Lower-level APIs are available to interact with the daemon and the -store. To instruct the daemon to perform a build action, users actually -provide it with a @dfn{derivation}. A derivation is a low-level -representation of the build actions to be taken, and the environment in -which they should occur---derivations are to package definitions what -assembly is to C programs. The term ``derivation'' comes from the fact -that build results @emph{derive} from them. - -This chapter describes all these APIs in turn, starting from high-level -package definitions. +The @command{guix environment} command provides a convenient way to set up +@dfn{development environments} containing all the dependencies and tools +necessary to work on the software package of your choice. The @command{guix +pack} command allows you to create @dfn{application bundles} that can be +easily distributed to users who do not run Guix. @menu -* Package Modules:: Packages from the programmer's viewpoint. -* Defining Packages:: Defining new packages. -* Build Systems:: Specifying how packages are built. -* The Store:: Manipulating the package store. -* Derivations:: Low-level interface to package derivations. -* The Store Monad:: Purely functional interface to the store. -* G-Expressions:: Manipulating build expressions. -* Invoking guix repl:: Fiddling with Guix interactively. +* Invoking guix environment:: Setting up development environments. +* Invoking guix pack:: Creating software bundles. @end menu -@node Package Modules -@section Package Modules +@node Invoking guix environment +@section Invoking @command{guix environment} -From a programming viewpoint, the package definitions of the -GNU distribution are provided by Guile modules in the @code{(gnu packages -@dots{})} name space@footnote{Note that packages under the @code{(gnu -packages @dots{})} module name space are not necessarily ``GNU -packages''. This module naming scheme follows the usual Guile module -naming convention: @code{gnu} means that these modules are distributed -as part of the GNU system, and @code{packages} identifies modules that -define packages.} (@pxref{Modules, Guile modules,, guile, GNU Guile -Reference Manual}). For instance, the @code{(gnu packages emacs)} -module exports a variable named @code{emacs}, which is bound to a -@code{} object (@pxref{Defining Packages}). +@cindex reproducible build environments +@cindex development environments +@cindex @command{guix environment} +@cindex environment, package build environment +The purpose of @command{guix environment} is to assist hackers in +creating reproducible development environments without polluting their +package profile. The @command{guix environment} tool takes one or more +packages, builds all of their inputs, and creates a shell +environment to use them. -The @code{(gnu packages @dots{})} module name space is -automatically scanned for packages by the command-line tools. For -instance, when running @code{guix package -i emacs}, all the @code{(gnu -packages @dots{})} modules are scanned until one that exports a package -object whose name is @code{emacs} is found. This package search -facility is implemented in the @code{(gnu packages)} module. +The general syntax is: -@cindex customization, of packages -@cindex package module search path -Users can store package definitions in modules with different -names---e.g., @code{(my-packages emacs)}@footnote{Note that the file -name and module name must match. For instance, the @code{(my-packages -emacs)} module must be stored in a @file{my-packages/emacs.scm} file -relative to the load path specified with @option{--load-path} or -@code{GUIX_PACKAGE_PATH}. @xref{Modules and the File System,,, -guile, GNU Guile Reference Manual}, for details.}. There are two ways to make -these package definitions visible to the user interfaces: +@example +guix environment @var{options} @var{package}@dots{} +@end example -@enumerate -@item -By adding the directory containing your package modules to the search path -with the @code{-L} flag of @command{guix package} and other commands -(@pxref{Common Build Options}), or by setting the @code{GUIX_PACKAGE_PATH} -environment variable described below. +The following example spawns a new shell set up for the development of +GNU@tie{}Guile: -@item -By defining a @dfn{channel} and configuring @command{guix pull} so that it -pulls from it. A channel is essentially a Git repository containing package -modules. @xref{Channels}, for more information on how to define and use -channels. -@end enumerate +@example +guix environment guile +@end example -@code{GUIX_PACKAGE_PATH} works similarly to other search path variables: +If the needed dependencies are not built yet, @command{guix environment} +automatically builds them. The environment of the new shell is an augmented +version of the environment that @command{guix environment} was run in. +It contains the necessary search paths for building the given package +added to the existing environment variables. To create a ``pure'' +environment, in which the original environment variables have been unset, +use the @code{--pure} option@footnote{Users sometimes wrongfully augment +environment variables such as @code{PATH} in their @file{~/.bashrc} +file. As a consequence, when @code{guix environment} launches it, Bash +may read @file{~/.bashrc}, thereby introducing ``impurities'' in these +environment variables. It is an error to define such environment +variables in @file{.bashrc}; instead, they should be defined in +@file{.bash_profile}, which is sourced only by log-in shells. +@xref{Bash Startup Files,,, bash, The GNU Bash Reference Manual}, for +details on Bash start-up files.}. -@defvr {Environment Variable} GUIX_PACKAGE_PATH -This is a colon-separated list of directories to search for additional -package modules. Directories listed in this variable take precedence -over the own modules of the distribution. -@end defvr +@vindex GUIX_ENVIRONMENT +@command{guix environment} defines the @code{GUIX_ENVIRONMENT} +variable in the shell it spawns; its value is the file name of the +profile of this environment. This allows users to, say, define a +specific prompt for development environments in their @file{.bashrc} +(@pxref{Bash Startup Files,,, bash, The GNU Bash Reference Manual}): -The distribution is fully @dfn{bootstrapped} and @dfn{self-contained}: -each package is built based solely on other packages in the -distribution. The root of this dependency graph is a small set of -@dfn{bootstrap binaries}, provided by the @code{(gnu packages -bootstrap)} module. For more information on bootstrapping, -@pxref{Bootstrapping}. +@example +if [ -n "$GUIX_ENVIRONMENT" ] +then + export PS1="\u@@\h \w [dev]\$ " +fi +@end example -@node Defining Packages -@section Defining Packages +@noindent +...@: or to browse the profile: -The high-level interface to package definitions is implemented in the -@code{(guix packages)} and @code{(guix build-system)} modules. As an -example, the package definition, or @dfn{recipe}, for the GNU Hello -package looks like this: +@example +$ ls "$GUIX_ENVIRONMENT/bin" +@end example + +Additionally, more than one package may be specified, in which case the +union of the inputs for the given packages are used. For example, the +command below spawns a shell where all of the dependencies of both Guile +and Emacs are available: @example -(define-module (gnu packages hello) - #:use-module (guix packages) - #:use-module (guix download) - #:use-module (guix build-system gnu) - #:use-module (guix licenses) - #:use-module (gnu packages gawk)) +guix environment guile emacs +@end example -(define-public hello - (package - (name "hello") - (version "2.10") - (source (origin - (method url-fetch) - (uri (string-append "mirror://gnu/hello/hello-" version - ".tar.gz")) - (sha256 - (base32 - "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i")))) - (build-system gnu-build-system) - (arguments '(#:configure-flags '("--enable-silent-rules"))) - (inputs `(("gawk" ,gawk))) - (synopsis "Hello, GNU world: An example GNU package") - (description "Guess what GNU Hello prints!") - (home-page "http://www.gnu.org/software/hello/") - (license gpl3+))) +Sometimes an interactive shell session is not desired. An arbitrary +command may be invoked by placing the @code{--} token to separate the +command from the rest of the arguments: + +@example +guix environment guile -- make -j4 @end example -@noindent -Without being a Scheme expert, the reader may have guessed the meaning -of the various fields here. This expression binds the variable -@code{hello} to a @code{} object, which is essentially a record -(@pxref{SRFI-9, Scheme records,, guile, GNU Guile Reference Manual}). -This package object can be inspected using procedures found in the -@code{(guix packages)} module; for instance, @code{(package-name hello)} -returns---surprise!---@code{"hello"}. +In other situations, it is more convenient to specify the list of +packages needed in the environment. For example, the following command +runs @command{python} from an environment containing Python@tie{}2.7 and +NumPy: -With luck, you may be able to import part or all of the definition of -the package you are interested in from another repository, using the -@code{guix import} command (@pxref{Invoking guix import}). +@example +guix environment --ad-hoc python2-numpy python-2.7 -- python +@end example -In the example above, @var{hello} is defined in a module of its own, -@code{(gnu packages hello)}. Technically, this is not strictly -necessary, but it is convenient to do so: all the packages defined in -modules under @code{(gnu packages @dots{})} are automatically known to -the command-line tools (@pxref{Package Modules}). +Furthermore, one might want the dependencies of a package and also some +additional packages that are not build-time or runtime dependencies, but +are useful when developing nonetheless. Because of this, the +@code{--ad-hoc} flag is positional. Packages appearing before +@code{--ad-hoc} are interpreted as packages whose dependencies will be +added to the environment. Packages appearing after are interpreted as +packages that will be added to the environment directly. For example, +the following command creates a Guix development environment that +additionally includes Git and strace: -There are a few points worth noting in the above package definition: +@example +guix environment guix --ad-hoc git strace +@end example -@itemize -@item -The @code{source} field of the package is an @code{} object -(@pxref{origin Reference}, for the complete reference). -Here, the @code{url-fetch} method from @code{(guix download)} is used, -meaning that the source is a file to be downloaded over FTP or HTTP. +Sometimes it is desirable to isolate the environment as much as +possible, for maximal purity and reproducibility. In particular, when +using Guix on a host distro that is not Guix System, it is desirable to +prevent access to @file{/usr/bin} and other system-wide resources from +the development environment. For example, the following command spawns +a Guile REPL in a ``container'' where only the store and the current +working directory are mounted: -The @code{mirror://gnu} prefix instructs @code{url-fetch} to use one of -the GNU mirrors defined in @code{(guix download)}. +@example +guix environment --ad-hoc --container guile -- guile +@end example -The @code{sha256} field specifies the expected SHA256 hash of the file -being downloaded. It is mandatory, and allows Guix to check the -integrity of the file. The @code{(base32 @dots{})} form introduces the -base32 representation of the hash. You can obtain this information with -@code{guix download} (@pxref{Invoking guix download}) and @code{guix -hash} (@pxref{Invoking guix hash}). +@quotation Note +The @code{--container} option requires Linux-libre 3.19 or newer. +@end quotation -@cindex patches -When needed, the @code{origin} form can also have a @code{patches} field -listing patches to be applied, and a @code{snippet} field giving a -Scheme expression to modify the source code. +The available options are summarized below. -@item -@cindex GNU Build System -The @code{build-system} field specifies the procedure to build the -package (@pxref{Build Systems}). Here, @var{gnu-build-system} -represents the familiar GNU Build System, where packages may be -configured, built, and installed with the usual @code{./configure && -make && make check && make install} command sequence. +@table @code +@item --root=@var{file} +@itemx -r @var{file} +@cindex persistent environment +@cindex garbage collector root, for environments +Make @var{file} a symlink to the profile for this environment, and +register it as a garbage collector root. -@item -The @code{arguments} field specifies options for the build system -(@pxref{Build Systems}). Here it is interpreted by -@var{gnu-build-system} as a request run @file{configure} with the -@code{--enable-silent-rules} flag. +This is useful if you want to protect your environment from garbage +collection, to make it ``persistent''. -@cindex quote -@cindex quoting -@findex ' -@findex quote -What about these quote (@code{'}) characters? They are Scheme syntax to -introduce a literal list; @code{'} is synonymous with @code{quote}. -@xref{Expression Syntax, quoting,, guile, GNU Guile Reference Manual}, -for details. Here the value of the @code{arguments} field is a list of -arguments passed to the build system down the road, as with @code{apply} -(@pxref{Fly Evaluation, @code{apply},, guile, GNU Guile Reference -Manual}). - -The hash-colon (@code{#:}) sequence defines a Scheme @dfn{keyword} -(@pxref{Keywords,,, guile, GNU Guile Reference Manual}), and -@code{#:configure-flags} is a keyword used to pass a keyword argument -to the build system (@pxref{Coding With Keywords,,, guile, GNU Guile -Reference Manual}). - -@item -The @code{inputs} field specifies inputs to the build process---i.e., -build-time or run-time dependencies of the package. Here, we define an -input called @code{"gawk"} whose value is that of the @var{gawk} -variable; @var{gawk} is itself bound to a @code{} object. +When this option is omitted, the environment is protected from garbage +collection only for the duration of the @command{guix environment} +session. This means that next time you recreate the same environment, +you could have to rebuild or re-download packages. @xref{Invoking guix +gc}, for more on GC roots. -@cindex backquote (quasiquote) -@findex ` -@findex quasiquote -@cindex comma (unquote) -@findex , -@findex unquote -@findex ,@@ -@findex unquote-splicing -Again, @code{`} (a backquote, synonymous with @code{quasiquote}) allows -us to introduce a literal list in the @code{inputs} field, while -@code{,} (a comma, synonymous with @code{unquote}) allows us to insert a -value in that list (@pxref{Expression Syntax, unquote,, guile, GNU Guile -Reference Manual}). +@item --expression=@var{expr} +@itemx -e @var{expr} +Create an environment for the package or list of packages that +@var{expr} evaluates to. -Note that GCC, Coreutils, Bash, and other essential tools do not need to -be specified as inputs here. Instead, @var{gnu-build-system} takes care -of ensuring that they are present (@pxref{Build Systems}). +For example, running: -However, any other dependencies need to be specified in the -@code{inputs} field. Any dependency not specified here will simply be -unavailable to the build process, possibly leading to a build failure. -@end itemize +@example +guix environment -e '(@@ (gnu packages maths) petsc-openmpi)' +@end example -@xref{package Reference}, for a full description of possible fields. +starts a shell with the environment for this specific variant of the +PETSc package. -Once a package definition is in place, the -package may actually be built using the @code{guix build} command-line -tool (@pxref{Invoking guix build}), troubleshooting any build failures -you encounter (@pxref{Debugging Build Failures}). You can easily jump back to the -package definition using the @command{guix edit} command -(@pxref{Invoking guix edit}). -@xref{Packaging Guidelines}, for -more information on how to test package definitions, and -@ref{Invoking guix lint}, for information on how to check a definition -for style conformance. -@vindex GUIX_PACKAGE_PATH -Lastly, @pxref{Channels}, for information -on how to extend the distribution by adding your own package definitions -in a ``channel''. +Running: -Finally, updating the package definition to a new upstream version -can be partly automated by the @command{guix refresh} command -(@pxref{Invoking guix refresh}). +@example +guix environment --ad-hoc -e '(@@ (gnu) %base-packages)' +@end example -Behind the scenes, a derivation corresponding to the @code{} -object is first computed by the @code{package-derivation} procedure. -That derivation is stored in a @code{.drv} file under @file{/gnu/store}. -The build actions it prescribes may then be realized by using the -@code{build-derivations} procedure (@pxref{The Store}). +starts a shell with all the base system packages available. -@deffn {Scheme Procedure} package-derivation @var{store} @var{package} [@var{system}] -Return the @code{} object of @var{package} for @var{system} -(@pxref{Derivations}). +The above commands only use the default output of the given packages. +To select other outputs, two element tuples can be specified: -@var{package} must be a valid @code{} object, and @var{system} -must be a string denoting the target system type---e.g., -@code{"x86_64-linux"} for an x86_64 Linux-based GNU system. @var{store} -must be a connection to the daemon, which operates on the store -(@pxref{The Store}). -@end deffn +@example +guix environment --ad-hoc -e '(list (@@ (gnu packages bash) bash) "include")' +@end example -@noindent -@cindex cross-compilation -Similarly, it is possible to compute a derivation that cross-builds a -package for some other system: +@item --load=@var{file} +@itemx -l @var{file} +Create an environment for the package or list of packages that the code +within @var{file} evaluates to. -@deffn {Scheme Procedure} package-cross-derivation @var{store} @ - @var{package} @var{target} [@var{system}] -Return the @code{} object of @var{package} cross-built from -@var{system} to @var{target}. +As an example, @var{file} might contain a definition like this +(@pxref{Defining Packages}): -@var{target} must be a valid GNU triplet denoting the target hardware -and operating system, such as @code{"mips64el-linux-gnu"} -(@pxref{Configuration Names, GNU configuration triplets,, configure, GNU -Configure and Build System}). -@end deffn +@example +@verbatiminclude environment-gdb.scm +@end example -@cindex package transformations -@cindex input rewriting -@cindex dependency tree rewriting -Packages can be manipulated in arbitrary ways. An example of a useful -transformation is @dfn{input rewriting}, whereby the dependency tree of -a package is rewritten by replacing specific inputs by others: +@item --manifest=@var{file} +@itemx -m @var{file} +Create an environment for the packages contained in the manifest object +returned by the Scheme code in @var{file}. -@deffn {Scheme Procedure} package-input-rewriting @var{replacements} @ - [@var{rewrite-name}] -Return a procedure that, when passed a package, replaces its direct and -indirect dependencies (but not its implicit inputs) according to -@var{replacements}. @var{replacements} is a list of package pairs; the -first element of each pair is the package to replace, and the second one -is the replacement. +This is similar to the same-named option in @command{guix package} +(@pxref{profile-manifest, @option{--manifest}}) and uses the same +manifest files. -Optionally, @var{rewrite-name} is a one-argument procedure that takes -the name of a package and returns its new name after rewrite. -@end deffn +@item --ad-hoc +Include all specified packages in the resulting environment, as if an +@i{ad hoc} package were defined with them as inputs. This option is +useful for quickly creating an environment without having to write a +package expression to contain the desired inputs. -@noindent -Consider this example: +For instance, the command: @example -(define libressl-instead-of-openssl - ;; This is a procedure to replace OPENSSL by LIBRESSL, - ;; recursively. - (package-input-rewriting `((,openssl . ,libressl)))) - -(define git-with-libressl - (libressl-instead-of-openssl git)) +guix environment --ad-hoc guile guile-sdl -- guile @end example -@noindent -Here we first define a rewriting procedure that replaces @var{openssl} -with @var{libressl}. Then we use it to define a @dfn{variant} of the -@var{git} package that uses @var{libressl} instead of @var{openssl}. -This is exactly what the @option{--with-input} command-line option does -(@pxref{Package Transformation Options, @option{--with-input}}). +runs @command{guile} in an environment where Guile and Guile-SDL are +available. -A more generic procedure to rewrite a package dependency graph is -@code{package-mapping}: it supports arbitrary changes to nodes in the -graph. +Note that this example implicitly asks for the default output of +@code{guile} and @code{guile-sdl}, but it is possible to ask for a +specific output---e.g., @code{glib:bin} asks for the @code{bin} output +of @code{glib} (@pxref{Packages with Multiple Outputs}). -@deffn {Scheme Procedure} package-mapping @var{proc} [@var{cut?}] -Return a procedure that, given a package, applies @var{proc} to all the packages -depended on and returns the resulting package. The procedure stops recursion -when @var{cut?} returns true for a given package. -@end deffn +This option may be composed with the default behavior of @command{guix +environment}. Packages appearing before @code{--ad-hoc} are interpreted +as packages whose dependencies will be added to the environment, the +default behavior. Packages appearing after are interpreted as packages +that will be added to the environment directly. -@menu -* package Reference:: The package data type. -* origin Reference:: The origin data type. -@end menu +@item --pure +Unset existing environment variables when building the new environment. +This has the effect of creating an environment in which search paths +only contain package inputs. +@item --search-paths +Display the environment variable definitions that make up the +environment. -@node package Reference -@subsection @code{package} Reference +@item --system=@var{system} +@itemx -s @var{system} +Attempt to build for @var{system}---e.g., @code{i686-linux}. -This section summarizes all the options available in @code{package} -declarations (@pxref{Defining Packages}). +@item --container +@itemx -C +@cindex container +Run @var{command} within an isolated container. The current working +directory outside the container is mapped inside the container. +Additionally, unless overridden with @code{--user}, a dummy home +directory is created that matches the current user's home directory, and +@file{/etc/passwd} is configured accordingly. The spawned process runs +as the current user outside the container, but has root privileges in +the context of the container. -@deftp {Data Type} package -This is the data type representing a package recipe. +@item --network +@itemx -N +For containers, share the network namespace with the host system. +Containers created without this flag only have access to the loopback +device. -@table @asis -@item @code{name} -The name of the package, as a string. +@item --link-profile +@itemx -P +For containers, link the environment profile to +@file{~/.guix-profile} within the container. This is equivalent to +running the command @command{ln -s $GUIX_ENVIRONMENT ~/.guix-profile} +within the container. Linking will fail and abort the environment if +the directory already exists, which will certainly be the case if +@command{guix environment} was invoked in the user's home directory. -@item @code{version} -The version of the package, as a string. +Certain packages are configured to look in +@code{~/.guix-profile} for configuration files and data;@footnote{For +example, the @code{fontconfig} package inspects +@file{~/.guix-profile/share/fonts} for additional fonts.} +@code{--link-profile} allows these programs to behave as expected within +the environment. -@item @code{source} -An object telling how the source code for the package should be -acquired. Most of the time, this is an @code{origin} object, which -denotes a file fetched from the Internet (@pxref{origin Reference}). It -can also be any other ``file-like'' object such as a @code{local-file}, -which denotes a file from the local file system (@pxref{G-Expressions, -@code{local-file}}). +@item --user=@var{user} +@itemx -u @var{user} +For containers, use the username @var{user} in place of the current +user. The generated @file{/etc/passwd} entry within the container will +contain the name @var{user}; the home directory will be +@file{/home/USER}; and no user GECOS data will be copied. @var{user} +need not exist on the system. -@item @code{build-system} -The build system that should be used to build the package (@pxref{Build -Systems}). - -@item @code{arguments} (default: @code{'()}) -The arguments that should be passed to the build system. This is a -list, typically containing sequential keyword-value pairs. - -@item @code{inputs} (default: @code{'()}) -@itemx @code{native-inputs} (default: @code{'()}) -@itemx @code{propagated-inputs} (default: @code{'()}) -@cindex inputs, of packages -These fields list dependencies of the package. Each one is a list of -tuples, where each tuple has a label for the input (a string) as its -first element, a package, origin, or derivation as its second element, -and optionally the name of the output thereof that should be used, which -defaults to @code{"out"} (@pxref{Packages with Multiple Outputs}, for -more on package outputs). For example, the list below specifies three -inputs: +Additionally, any shared or exposed path (see @code{--share} and +@code{--expose} respectively) whose target is within the current user's +home directory will be remapped relative to @file{/home/USER}; this +includes the automatic mapping of the current working directory. @example -`(("libffi" ,libffi) - ("libunistring" ,libunistring) - ("glib:bin" ,glib "bin")) ;the "bin" output of Glib +# will expose paths as /home/foo/wd, /home/foo/test, and /home/foo/target +cd $HOME/wd +guix environment --container --user=foo \ + --expose=$HOME/test \ + --expose=/tmp/target=$HOME/target @end example -@cindex cross compilation, package dependencies -The distinction between @code{native-inputs} and @code{inputs} is -necessary when considering cross-compilation. When cross-compiling, -dependencies listed in @code{inputs} are built for the @emph{target} -architecture; conversely, dependencies listed in @code{native-inputs} -are built for the architecture of the @emph{build} machine. - -@code{native-inputs} is typically used to list tools needed at -build time, but not at run time, such as Autoconf, Automake, pkg-config, -Gettext, or Bison. @command{guix lint} can report likely mistakes in -this area (@pxref{Invoking guix lint}). - -@anchor{package-propagated-inputs} -Lastly, @code{propagated-inputs} is similar to @code{inputs}, but the -specified packages will be automatically installed alongside the package -they belong to (@pxref{package-cmd-propagated-inputs, @command{guix -package}}, for information on how @command{guix package} deals with -propagated inputs.) +While this will limit the leaking of user identity through home paths +and each of the user fields, this is only one useful component of a +broader privacy/anonymity solution---not one in and of itself. -For example this is necessary when a C/C++ library needs headers of -another library to compile, or when a pkg-config file refers to another -one @i{via} its @code{Requires} field. +@item --expose=@var{source}[=@var{target}] +For containers, expose the file system @var{source} from the host system +as the read-only file system @var{target} within the container. If +@var{target} is not specified, @var{source} is used as the target mount +point in the container. -Another example where @code{propagated-inputs} is useful is for languages -that lack a facility to record the run-time search path akin to the -@code{RUNPATH} of ELF files; this includes Guile, Python, Perl, and -more. To ensure that libraries written in those languages can find -library code they depend on at run time, run-time dependencies must be -listed in @code{propagated-inputs} rather than @code{inputs}. +The example below spawns a Guile REPL in a container in which the user's +home directory is accessible read-only via the @file{/exchange} +directory: -@item @code{self-native-input?} (default: @code{#f}) -This is a Boolean field telling whether the package should use itself as -a native input when cross-compiling. +@example +guix environment --container --expose=$HOME=/exchange --ad-hoc guile -- guile +@end example -@item @code{outputs} (default: @code{'("out")}) -The list of output names of the package. @xref{Packages with Multiple -Outputs}, for typical uses of additional outputs. +@item --share=@var{source}[=@var{target}] +For containers, share the file system @var{source} from the host system +as the writable file system @var{target} within the container. If +@var{target} is not specified, @var{source} is used as the target mount +point in the container. -@item @code{native-search-paths} (default: @code{'()}) -@itemx @code{search-paths} (default: @code{'()}) -A list of @code{search-path-specification} objects describing -search-path environment variables honored by the package. +The example below spawns a Guile REPL in a container in which the user's +home directory is accessible for both reading and writing via the +@file{/exchange} directory: -@item @code{replacement} (default: @code{#f}) -This must be either @code{#f} or a package object that will be used as a -@dfn{replacement} for this package. @xref{Security Updates, grafts}, -for details. +@example +guix environment --container --share=$HOME=/exchange --ad-hoc guile -- guile +@end example +@end table -@item @code{synopsis} -A one-line description of the package. +@command{guix environment} +also supports all of the common build options that @command{guix +build} supports (@pxref{Common Build Options}) as well as package +transformation options (@pxref{Package Transformation Options}). -@item @code{description} -A more elaborate description of the package. +@node Invoking guix pack +@section Invoking @command{guix pack} -@item @code{license} -@cindex license, of packages -The license of the package; a value from @code{(guix licenses)}, -or a list of such values. +Occasionally you want to pass software to people who are not (yet!) +lucky enough to be using Guix. You'd tell them to run @command{guix +package -i @var{something}}, but that's not possible in this case. This +is where @command{guix pack} comes in. -@item @code{home-page} -The URL to the home-page of the package, as a string. +@quotation Note +If you are looking for ways to exchange binaries among machines that +already run Guix, @pxref{Invoking guix copy}, @ref{Invoking guix +publish}, and @ref{Invoking guix archive}. +@end quotation -@item @code{supported-systems} (default: @var{%supported-systems}) -The list of systems supported by the package, as strings of the form -@code{architecture-kernel}, for example @code{"x86_64-linux"}. +@cindex pack +@cindex bundle +@cindex application bundle +@cindex software bundle +The @command{guix pack} command creates a shrink-wrapped @dfn{pack} or +@dfn{software bundle}: it creates a tarball or some other archive +containing the binaries of the software you're interested in, and all +its dependencies. The resulting archive can be used on any machine that +does not have Guix, and people can run the exact same binaries as those +you have with Guix. The pack itself is created in a bit-reproducible +fashion, so anyone can verify that it really contains the build results +that you pretend to be shipping. -@item @code{maintainers} (default: @code{'()}) -The list of maintainers of the package, as @code{maintainer} objects. +For example, to create a bundle containing Guile, Emacs, Geiser, and all +their dependencies, you can run: -@item @code{location} (default: source location of the @code{package} form) -The source location of the package. It is useful to override this when -inheriting from another package, in which case this field is not -automatically corrected. -@end table -@end deftp +@example +$ guix pack guile emacs geiser +@dots{} +/gnu/store/@dots{}-pack.tar.gz +@end example +The result here is a tarball containing a @file{/gnu/store} directory +with all the relevant packages. The resulting tarball contains a +@dfn{profile} with the three packages of interest; the profile is the +same as would be created by @command{guix package -i}. It is this +mechanism that is used to create Guix's own standalone binary tarball +(@pxref{Binary Installation}). -@node origin Reference -@subsection @code{origin} Reference +Users of this pack would have to run +@file{/gnu/store/@dots{}-profile/bin/guile} to run Guile, which you may +find inconvenient. To work around it, you can create, say, a +@file{/opt/gnu/bin} symlink to the profile: -This section summarizes all the options available in @code{origin} -declarations (@pxref{Defining Packages}). +@example +guix pack -S /opt/gnu/bin=bin guile emacs geiser +@end example -@deftp {Data Type} origin -This is the data type representing a source code origin. +@noindent +That way, users can happily type @file{/opt/gnu/bin/guile} and enjoy. -@table @asis -@item @code{uri} -An object containing the URI of the source. The object type depends on -the @code{method} (see below). For example, when using the -@var{url-fetch} method of @code{(guix download)}, the valid @code{uri} -values are: a URL represented as a string, or a list thereof. +@cindex relocatable binaries, with @command{guix pack} +What if the recipient of your pack does not have root privileges on +their machine, and thus cannot unpack it in the root file system? In +that case, you will want to use the @code{--relocatable} option (see +below). This option produces @dfn{relocatable binaries}, meaning they +they can be placed anywhere in the file system hierarchy: in the example +above, users can unpack your tarball in their home directory and +directly run @file{./opt/gnu/bin/guile}. -@item @code{method} -A procedure that handles the URI. +@cindex Docker, build an image with guix pack +Alternatively, you can produce a pack in the Docker image format using +the following command: -Examples include: +@example +guix pack -f docker guile emacs geiser +@end example -@table @asis -@item @var{url-fetch} from @code{(guix download)} -download a file from the HTTP, HTTPS, or FTP URL specified in the -@code{uri} field; +@noindent +The result is a tarball that can be passed to the @command{docker load} +command. See the +@uref{https://docs.docker.com/engine/reference/commandline/load/, Docker +documentation} for more information. -@vindex git-fetch -@item @var{git-fetch} from @code{(guix git-download)} -clone the Git version control repository, and check out the revision -specified in the @code{uri} field as a @code{git-reference} object; a -@code{git-reference} looks like this: +@cindex Singularity, build an image with guix pack +@cindex SquashFS, build an image with guix pack +Yet another option is to produce a SquashFS image with the following +command: @example -(git-reference - (url "git://git.debian.org/git/pkg-shadow/shadow") - (commit "v4.1.5.1")) +guix pack -f squashfs guile emacs geiser @end example -@end table -@item @code{sha256} -A bytevector containing the SHA-256 hash of the source. Typically the -@code{base32} form is used here to generate the bytevector from a -base-32 string. +@noindent +The result is a SquashFS file system image that can either be mounted or +directly be used as a file system container image with the +@uref{http://singularity.lbl.gov, Singularity container execution +environment}, using commands like @command{singularity shell} or +@command{singularity exec}. -You can obtain this information using @code{guix download} -(@pxref{Invoking guix download}) or @code{guix hash} (@pxref{Invoking -guix hash}). +Several command-line options allow you to customize your pack: -@item @code{file-name} (default: @code{#f}) -The file name under which the source code should be saved. When this is -@code{#f}, a sensible default value will be used in most cases. In case -the source is fetched from a URL, the file name from the URL will be -used. For version control checkouts, it is recommended to provide the -file name explicitly because the default is not very descriptive. +@table @code +@item --format=@var{format} +@itemx -f @var{format} +Produce a pack in the given @var{format}. -@item @code{patches} (default: @code{'()}) -A list of file names, origins, or file-like objects (@pxref{G-Expressions, -file-like objects}) pointing to patches to be applied to the source. +The available formats are: -This list of patches must be unconditional. In particular, it cannot -depend on the value of @code{%current-system} or -@code{%current-target-system}. +@table @code +@item tarball +This is the default format. It produces a tarball containing all the +specified binaries and symlinks. -@item @code{snippet} (default: @code{#f}) -A G-expression (@pxref{G-Expressions}) or S-expression that will be run -in the source directory. This is a convenient way to modify the source, -sometimes more convenient than a patch. +@item docker +This produces a tarball that follows the +@uref{https://github.com/docker/docker/blob/master/image/spec/v1.2.md, +Docker Image Specification}. -@item @code{patch-flags} (default: @code{'("-p1")}) -A list of command-line flags that should be passed to the @code{patch} -command. +@item squashfs +This produces a SquashFS image containing all the specified binaries and +symlinks, as well as empty mount points for virtual file systems like +procfs. +@end table -@item @code{patch-inputs} (default: @code{#f}) -Input packages or derivations to the patching process. When this is -@code{#f}, the usual set of inputs necessary for patching are provided, -such as GNU@tie{}Patch. +@item --relocatable +@itemx -R +Produce @dfn{relocatable binaries}---i.e., binaries that can be placed +anywhere in the file system hierarchy and run from there. For example, +if you create a pack containing Bash with: -@item @code{modules} (default: @code{'()}) -A list of Guile modules that should be loaded during the patching -process and while running the code in the @code{snippet} field. +@example +guix pack -R -S /mybin=bin bash +@end example -@item @code{patch-guile} (default: @code{#f}) -The Guile package that should be used in the patching process. When -this is @code{#f}, a sensible default is used. -@end table -@end deftp +@noindent +...@: you can copy that pack to a machine that lacks Guix, and from your +home directory as a normal user, run: +@example +tar xf pack.tar.gz +./mybin/sh +@end example -@node Build Systems -@section Build Systems +@noindent +In that shell, if you type @code{ls /gnu/store}, you'll notice that +@file{/gnu/store} shows up and contains all the dependencies of +@code{bash}, even though the machine actually lacks @file{/gnu/store} +altogether! That is probably the simplest way to deploy Guix-built +software on a non-Guix machine. -@cindex build system -Each package definition specifies a @dfn{build system} and arguments for -that build system (@pxref{Defining Packages}). This @code{build-system} -field represents the build procedure of the package, as well as implicit -dependencies of that build procedure. +There's a gotcha though: this technique relies on the @dfn{user +namespace} feature of the kernel Linux, which allows unprivileged users +to mount or change root. Old versions of Linux did not support it, and +some GNU/Linux distributions turn it off; on these systems, programs +from the pack @emph{will fail to run}, unless they are unpacked in the +root file system. -Build systems are @code{} objects. The interface to -create and manipulate them is provided by the @code{(guix build-system)} -module, and actual build systems are exported by specific modules. +@item --expression=@var{expr} +@itemx -e @var{expr} +Consider the package @var{expr} evaluates to. -@cindex bag (low-level package representation) -Under the hood, build systems first compile package objects to -@dfn{bags}. A @dfn{bag} is like a package, but with less -ornamentation---in other words, a bag is a lower-level representation of -a package, which includes all the inputs of that package, including some -that were implicitly added by the build system. This intermediate -representation is then compiled to a derivation (@pxref{Derivations}). +This has the same purpose as the same-named option in @command{guix +build} (@pxref{Additional Build Options, @code{--expression} in +@command{guix build}}). -Build systems accept an optional list of @dfn{arguments}. In package -definitions, these are passed @i{via} the @code{arguments} field -(@pxref{Defining Packages}). They are typically keyword arguments -(@pxref{Optional Arguments, keyword arguments in Guile,, guile, GNU -Guile Reference Manual}). The value of these arguments is usually -evaluated in the @dfn{build stratum}---i.e., by a Guile process launched -by the daemon (@pxref{Derivations}). +@item --manifest=@var{file} +@itemx -m @var{file} +Use the packages contained in the manifest object returned by the Scheme +code in @var{file}. -The main build system is @var{gnu-build-system}, which implements the -standard build procedure for GNU and many other packages. It -is provided by the @code{(guix build-system gnu)} module. +This has a similar purpose as the same-named option in @command{guix +package} (@pxref{profile-manifest, @option{--manifest}}) and uses the +same manifest files. It allows you to define a collection of packages +once and use it both for creating profiles and for creating archives +for use on machines that do not have Guix installed. Note that you can +specify @emph{either} a manifest file @emph{or} a list of packages, +but not both. -@defvr {Scheme Variable} gnu-build-system -@var{gnu-build-system} represents the GNU Build System, and variants -thereof (@pxref{Configuration, configuration and makefile conventions,, -standards, GNU Coding Standards}). +@item --system=@var{system} +@itemx -s @var{system} +Attempt to build for @var{system}---e.g., @code{i686-linux}---instead of +the system type of the build host. -@cindex build phases -In a nutshell, packages using it are configured, built, and installed with -the usual @code{./configure && make && make check && make install} -command sequence. In practice, a few additional steps are often needed. -All these steps are split up in separate @dfn{phases}, -notably@footnote{Please see the @code{(guix build gnu-build-system)} -modules for more details about the build phases.}: +@item --target=@var{triplet} +@cindex cross-compilation +Cross-build for @var{triplet}, which must be a valid GNU triplet, such +as @code{"mips64el-linux-gnu"} (@pxref{Specifying target triplets, GNU +configuration triplets,, autoconf, Autoconf}). -@table @code -@item unpack -Unpack the source tarball, and change the current directory to the -extracted source tree. If the source is actually a directory, copy it -to the build tree, and enter that directory. +@item --compression=@var{tool} +@itemx -C @var{tool} +Compress the resulting tarball using @var{tool}---one of @code{gzip}, +@code{bzip2}, @code{xz}, @code{lzip}, or @code{none} for no compression. -@item patch-source-shebangs -Patch shebangs encountered in source files so they refer to the right -store file names. For instance, this changes @code{#!/bin/sh} to -@code{#!/gnu/store/@dots{}-bash-4.3/bin/sh}. +@item --symlink=@var{spec} +@itemx -S @var{spec} +Add the symlinks specified by @var{spec} to the pack. This option can +appear several times. -@item configure -Run the @file{configure} script with a number of default options, such -as @code{--prefix=/gnu/store/@dots{}}, as well as the options specified -by the @code{#:configure-flags} argument. +@var{spec} has the form @code{@var{source}=@var{target}}, where +@var{source} is the symlink that will be created and @var{target} is the +symlink target. -@item build -Run @code{make} with the list of flags specified with -@code{#:make-flags}. If the @code{#:parallel-build?} argument is true -(the default), build with @code{make -j}. +For instance, @code{-S /opt/gnu/bin=bin} creates a @file{/opt/gnu/bin} +symlink pointing to the @file{bin} sub-directory of the profile. -@item check -Run @code{make check}, or some other target specified with -@code{#:test-target}, unless @code{#:tests? #f} is passed. If the -@code{#:parallel-tests?} argument is true (the default), run @code{make -check -j}. +@item --localstatedir +@itemx --profile-name=@var{name} +Include the ``local state directory'', @file{/var/guix}, in the resulting +pack, and notably the @file{/var/guix/profiles/per-user/root/@var{name}} +profile---by default @var{name} is @code{guix-profile}, which corresponds to +@file{~root/.guix-profile}. -@item install -Run @code{make install} with the flags listed in @code{#:make-flags}. +@file{/var/guix} contains the store database (@pxref{The Store}) as well +as garbage-collector roots (@pxref{Invoking guix gc}). Providing it in +the pack means that the store is ``complete'' and manageable by Guix; +not providing it pack means that the store is ``dead'': items cannot be +added to it or removed from it after extraction of the pack. -@item patch-shebangs -Patch shebangs on the installed executable files. +One use case for this is the Guix self-contained binary tarball +(@pxref{Binary Installation}). -@item strip -Strip debugging symbols from ELF files (unless @code{#:strip-binaries?} -is false), copying them to the @code{debug} output when available -(@pxref{Installing Debugging Files}). +@item --bootstrap +Use the bootstrap binaries to build the pack. This option is only +useful to Guix developers. @end table -@vindex %standard-phases -The build-side module @code{(guix build gnu-build-system)} defines -@var{%standard-phases} as the default list of build phases. -@var{%standard-phases} is a list of symbol/procedure pairs, where the -procedure implements the actual phase. +In addition, @command{guix pack} supports all the common build options +(@pxref{Common Build Options}) and all the package transformation +options (@pxref{Package Transformation Options}). -The list of phases used for a particular package can be changed with the -@code{#:phases} parameter. For instance, passing: -@example -#:phases (modify-phases %standard-phases (delete 'configure)) -@end example +@c ********************************************************************* +@node Programming Interface +@chapter Programming Interface -means that all the phases described above will be used, except the -@code{configure} phase. +GNU Guix provides several Scheme programming interfaces (APIs) to +define, build, and query packages. The first interface allows users to +write high-level package definitions. These definitions refer to +familiar packaging concepts, such as the name and version of a package, +its build system, and its dependencies. These definitions can then be +turned into concrete build actions. -In addition, this build system ensures that the ``standard'' environment -for GNU packages is available. This includes tools such as GCC, libc, -Coreutils, Bash, Make, Diffutils, grep, and sed (see the @code{(guix -build-system gnu)} module for a complete list). We call these the -@dfn{implicit inputs} of a package, because package definitions do not -have to mention them. -@end defvr +Build actions are performed by the Guix daemon, on behalf of users. In a +standard setup, the daemon has write access to the store---the +@file{/gnu/store} directory---whereas users do not. The recommended +setup also has the daemon perform builds in chroots, under a specific +build users, to minimize interference with the rest of the system. -Other @code{} objects are defined to support other -conventions and tools used by free software packages. They inherit most -of @var{gnu-build-system}, and differ mainly in the set of inputs -implicitly added to the build process, and in the list of phases -executed. Some of these build systems are listed below. +@cindex derivation +Lower-level APIs are available to interact with the daemon and the +store. To instruct the daemon to perform a build action, users actually +provide it with a @dfn{derivation}. A derivation is a low-level +representation of the build actions to be taken, and the environment in +which they should occur---derivations are to package definitions what +assembly is to C programs. The term ``derivation'' comes from the fact +that build results @emph{derive} from them. -@defvr {Scheme Variable} ant-build-system -This variable is exported by @code{(guix build-system ant)}. It -implements the build procedure for Java packages that can be built with -@url{http://ant.apache.org/, Ant build tool}. +This chapter describes all these APIs in turn, starting from high-level +package definitions. -It adds both @code{ant} and the @dfn{Java Development Kit} (JDK) as -provided by the @code{icedtea} package to the set of inputs. Different -packages can be specified with the @code{#:ant} and @code{#:jdk} -parameters, respectively. +@menu +* Package Modules:: Packages from the programmer's viewpoint. +* Defining Packages:: Defining new packages. +* Build Systems:: Specifying how packages are built. +* The Store:: Manipulating the package store. +* Derivations:: Low-level interface to package derivations. +* The Store Monad:: Purely functional interface to the store. +* G-Expressions:: Manipulating build expressions. +* Invoking guix repl:: Fiddling with Guix interactively. +@end menu -When the original package does not provide a suitable Ant build file, -the parameter @code{#:jar-name} can be used to generate a minimal Ant -build file @file{build.xml} with tasks to build the specified jar -archive. In this case the parameter @code{#:source-dir} can be used to -specify the source sub-directory, defaulting to ``src''. +@node Package Modules +@section Package Modules -The @code{#:main-class} parameter can be used with the minimal ant -buildfile to specify the main class of the resulting jar. This makes the -jar file executable. The @code{#:test-include} parameter can be used to -specify the list of junit tests to run. It defaults to -@code{(list "**/*Test.java")}. The @code{#:test-exclude} can be used to -disable some tests. It defaults to @code{(list "**/Abstract*.java")}, -because abstract classes cannot be run as tests. +From a programming viewpoint, the package definitions of the +GNU distribution are provided by Guile modules in the @code{(gnu packages +@dots{})} name space@footnote{Note that packages under the @code{(gnu +packages @dots{})} module name space are not necessarily ``GNU +packages''. This module naming scheme follows the usual Guile module +naming convention: @code{gnu} means that these modules are distributed +as part of the GNU system, and @code{packages} identifies modules that +define packages.} (@pxref{Modules, Guile modules,, guile, GNU Guile +Reference Manual}). For instance, the @code{(gnu packages emacs)} +module exports a variable named @code{emacs}, which is bound to a +@code{} object (@pxref{Defining Packages}). -The parameter @code{#:build-target} can be used to specify the Ant task -that should be run during the @code{build} phase. By default the -``jar'' task will be run. +The @code{(gnu packages @dots{})} module name space is +automatically scanned for packages by the command-line tools. For +instance, when running @code{guix package -i emacs}, all the @code{(gnu +packages @dots{})} modules are scanned until one that exports a package +object whose name is @code{emacs} is found. This package search +facility is implemented in the @code{(gnu packages)} module. + +@cindex customization, of packages +@cindex package module search path +Users can store package definitions in modules with different +names---e.g., @code{(my-packages emacs)}@footnote{Note that the file +name and module name must match. For instance, the @code{(my-packages +emacs)} module must be stored in a @file{my-packages/emacs.scm} file +relative to the load path specified with @option{--load-path} or +@code{GUIX_PACKAGE_PATH}. @xref{Modules and the File System,,, +guile, GNU Guile Reference Manual}, for details.}. There are two ways to make +these package definitions visible to the user interfaces: + +@enumerate +@item +By adding the directory containing your package modules to the search path +with the @code{-L} flag of @command{guix package} and other commands +(@pxref{Common Build Options}), or by setting the @code{GUIX_PACKAGE_PATH} +environment variable described below. + +@item +By defining a @dfn{channel} and configuring @command{guix pull} so that it +pulls from it. A channel is essentially a Git repository containing package +modules. @xref{Channels}, for more information on how to define and use +channels. +@end enumerate + +@code{GUIX_PACKAGE_PATH} works similarly to other search path variables: +@defvr {Environment Variable} GUIX_PACKAGE_PATH +This is a colon-separated list of directories to search for additional +package modules. Directories listed in this variable take precedence +over the own modules of the distribution. @end defvr -@defvr {Scheme Variable} android-ndk-build-system -@cindex Android distribution -@cindex Android NDK build system -This variable is exported by @code{(guix build-system android-ndk)}. It -implements a build procedure for Android NDK (native development kit) -packages using a Guix-specific build process. +The distribution is fully @dfn{bootstrapped} and @dfn{self-contained}: +each package is built based solely on other packages in the +distribution. The root of this dependency graph is a small set of +@dfn{bootstrap binaries}, provided by the @code{(gnu packages +bootstrap)} module. For more information on bootstrapping, +@pxref{Bootstrapping}. -The build system assumes that packages install their public interface -(header) files to the subdirectory "include" of the "out" output and -their libraries to the subdirectory "lib" of the "out" output. +@node Defining Packages +@section Defining Packages -It's also assumed that the union of all the dependencies of a package -has no conflicting files. +The high-level interface to package definitions is implemented in the +@code{(guix packages)} and @code{(guix build-system)} modules. As an +example, the package definition, or @dfn{recipe}, for the GNU Hello +package looks like this: -For the time being, cross-compilation is not supported - so right now -the libraries and header files are assumed to be host tools. +@example +(define-module (gnu packages hello) + #:use-module (guix packages) + #:use-module (guix download) + #:use-module (guix build-system gnu) + #:use-module (guix licenses) + #:use-module (gnu packages gawk)) -@end defvr +(define-public hello + (package + (name "hello") + (version "2.10") + (source (origin + (method url-fetch) + (uri (string-append "mirror://gnu/hello/hello-" version + ".tar.gz")) + (sha256 + (base32 + "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i")))) + (build-system gnu-build-system) + (arguments '(#:configure-flags '("--enable-silent-rules"))) + (inputs `(("gawk" ,gawk))) + (synopsis "Hello, GNU world: An example GNU package") + (description "Guess what GNU Hello prints!") + (home-page "http://www.gnu.org/software/hello/") + (license gpl3+))) +@end example -@defvr {Scheme Variable} asdf-build-system/source -@defvrx {Scheme Variable} asdf-build-system/sbcl -@defvrx {Scheme Variable} asdf-build-system/ecl +@noindent +Without being a Scheme expert, the reader may have guessed the meaning +of the various fields here. This expression binds the variable +@code{hello} to a @code{} object, which is essentially a record +(@pxref{SRFI-9, Scheme records,, guile, GNU Guile Reference Manual}). +This package object can be inspected using procedures found in the +@code{(guix packages)} module; for instance, @code{(package-name hello)} +returns---surprise!---@code{"hello"}. -These variables, exported by @code{(guix build-system asdf)}, implement -build procedures for Common Lisp packages using -@url{https://common-lisp.net/project/asdf/, ``ASDF''}. ASDF is a system -definition facility for Common Lisp programs and libraries. +With luck, you may be able to import part or all of the definition of +the package you are interested in from another repository, using the +@code{guix import} command (@pxref{Invoking guix import}). -The @code{asdf-build-system/source} system installs the packages in -source form, and can be loaded using any common lisp implementation, via -ASDF. The others, such as @code{asdf-build-system/sbcl}, install binary -systems in the format which a particular implementation understands. -These build systems can also be used to produce executable programs, or -lisp images which contain a set of packages pre-loaded. +In the example above, @var{hello} is defined in a module of its own, +@code{(gnu packages hello)}. Technically, this is not strictly +necessary, but it is convenient to do so: all the packages defined in +modules under @code{(gnu packages @dots{})} are automatically known to +the command-line tools (@pxref{Package Modules}). -The build system uses naming conventions. For binary packages, the -package name should be prefixed with the lisp implementation, such as -@code{sbcl-} for @code{asdf-build-system/sbcl}. +There are a few points worth noting in the above package definition: -Additionally, the corresponding source package should be labeled using -the same convention as python packages (see @ref{Python Modules}), using -the @code{cl-} prefix. +@itemize +@item +The @code{source} field of the package is an @code{} object +(@pxref{origin Reference}, for the complete reference). +Here, the @code{url-fetch} method from @code{(guix download)} is used, +meaning that the source is a file to be downloaded over FTP or HTTP. -For binary packages, each system should be defined as a Guix package. -If one package @code{origin} contains several systems, package variants -can be created in order to build all the systems. Source packages, -which use @code{asdf-build-system/source}, may contain several systems. +The @code{mirror://gnu} prefix instructs @code{url-fetch} to use one of +the GNU mirrors defined in @code{(guix download)}. -In order to create executable programs and images, the build-side -procedures @code{build-program} and @code{build-image} can be used. -They should be called in a build phase after the @code{create-symlinks} -phase, so that the system which was just built can be used within the -resulting image. @code{build-program} requires a list of Common Lisp -expressions to be passed as the @code{#:entry-program} argument. +The @code{sha256} field specifies the expected SHA256 hash of the file +being downloaded. It is mandatory, and allows Guix to check the +integrity of the file. The @code{(base32 @dots{})} form introduces the +base32 representation of the hash. You can obtain this information with +@code{guix download} (@pxref{Invoking guix download}) and @code{guix +hash} (@pxref{Invoking guix hash}). -If the system is not defined within its own @code{.asd} file of the same -name, then the @code{#:asd-file} parameter should be used to specify -which file the system is defined in. Furthermore, if the package -defines a system for its tests in a separate file, it will be loaded -before the tests are run if it is specified by the -@code{#:test-asd-file} parameter. If it is not set, the files -@code{-tests.asd}, @code{-test.asd}, @code{tests.asd}, -and @code{test.asd} will be tried if they exist. +@cindex patches +When needed, the @code{origin} form can also have a @code{patches} field +listing patches to be applied, and a @code{snippet} field giving a +Scheme expression to modify the source code. -If for some reason the package must be named in a different way than the -naming conventions suggest, the @code{#:asd-system-name} parameter can -be used to specify the name of the system. +@item +@cindex GNU Build System +The @code{build-system} field specifies the procedure to build the +package (@pxref{Build Systems}). Here, @var{gnu-build-system} +represents the familiar GNU Build System, where packages may be +configured, built, and installed with the usual @code{./configure && +make && make check && make install} command sequence. -@end defvr +@item +The @code{arguments} field specifies options for the build system +(@pxref{Build Systems}). Here it is interpreted by +@var{gnu-build-system} as a request run @file{configure} with the +@code{--enable-silent-rules} flag. -@defvr {Scheme Variable} cargo-build-system -@cindex Rust programming language -@cindex Cargo (Rust build system) -This variable is exported by @code{(guix build-system cargo)}. It -supports builds of packages using Cargo, the build tool of the -@uref{https://www.rust-lang.org, Rust programming language}. +@cindex quote +@cindex quoting +@findex ' +@findex quote +What about these quote (@code{'}) characters? They are Scheme syntax to +introduce a literal list; @code{'} is synonymous with @code{quote}. +@xref{Expression Syntax, quoting,, guile, GNU Guile Reference Manual}, +for details. Here the value of the @code{arguments} field is a list of +arguments passed to the build system down the road, as with @code{apply} +(@pxref{Fly Evaluation, @code{apply},, guile, GNU Guile Reference +Manual}). -In its @code{configure} phase, this build system replaces dependencies -specified in the @file{Carto.toml} file with inputs to the Guix package. -The @code{install} phase installs the binaries, and it also installs the -source code and @file{Cargo.toml} file. -@end defvr +The hash-colon (@code{#:}) sequence defines a Scheme @dfn{keyword} +(@pxref{Keywords,,, guile, GNU Guile Reference Manual}), and +@code{#:configure-flags} is a keyword used to pass a keyword argument +to the build system (@pxref{Coding With Keywords,,, guile, GNU Guile +Reference Manual}). -@cindex Clojure (programming language) -@cindex simple Clojure build system -@defvr {Scheme Variable} clojure-build-system -This variable is exported by @code{(guix build-system clojure)}. It implements -a simple build procedure for @uref{https://clojure.org/, Clojure} packages -using plain old @code{compile} in Clojure. Cross-compilation is not supported -yet. +@item +The @code{inputs} field specifies inputs to the build process---i.e., +build-time or run-time dependencies of the package. Here, we define an +input called @code{"gawk"} whose value is that of the @var{gawk} +variable; @var{gawk} is itself bound to a @code{} object. -It adds @code{clojure}, @code{icedtea} and @code{zip} to the set of inputs. -Different packages can be specified with the @code{#:clojure}, @code{#:jdk} and -@code{#:zip} parameters, respectively. +@cindex backquote (quasiquote) +@findex ` +@findex quasiquote +@cindex comma (unquote) +@findex , +@findex unquote +@findex ,@@ +@findex unquote-splicing +Again, @code{`} (a backquote, synonymous with @code{quasiquote}) allows +us to introduce a literal list in the @code{inputs} field, while +@code{,} (a comma, synonymous with @code{unquote}) allows us to insert a +value in that list (@pxref{Expression Syntax, unquote,, guile, GNU Guile +Reference Manual}). -A list of source directories, test directories and jar names can be specified -with the @code{#:source-dirs}, @code{#:test-dirs} and @code{#:jar-names} -parameters, respectively. Compile directory and main class can be specified -with the @code{#:compile-dir} and @code{#:main-class} parameters, respectively. -Other parameters are documented below. +Note that GCC, Coreutils, Bash, and other essential tools do not need to +be specified as inputs here. Instead, @var{gnu-build-system} takes care +of ensuring that they are present (@pxref{Build Systems}). -This build system is an extension of @var{ant-build-system}, but with the -following phases changed: +However, any other dependencies need to be specified in the +@code{inputs} field. Any dependency not specified here will simply be +unavailable to the build process, possibly leading to a build failure. +@end itemize -@table @code +@xref{package Reference}, for a full description of possible fields. -@item build -This phase calls @code{compile} in Clojure to compile source files and runs -@command{jar} to create jars from both source files and compiled files -according to the include list and exclude list specified in -@code{#:aot-include} and @code{#:aot-exclude}, respectively. The exclude list -has priority over the include list. These lists consist of symbols -representing Clojure libraries or the special keyword @code{#:all} representing -all Clojure libraries found under the source directories. The parameter -@code{#:omit-source?} decides if source should be included into the jars. +Once a package definition is in place, the +package may actually be built using the @code{guix build} command-line +tool (@pxref{Invoking guix build}), troubleshooting any build failures +you encounter (@pxref{Debugging Build Failures}). You can easily jump back to the +package definition using the @command{guix edit} command +(@pxref{Invoking guix edit}). +@xref{Packaging Guidelines}, for +more information on how to test package definitions, and +@ref{Invoking guix lint}, for information on how to check a definition +for style conformance. +@vindex GUIX_PACKAGE_PATH +Lastly, @pxref{Channels}, for information +on how to extend the distribution by adding your own package definitions +in a ``channel''. -@item check -This phase runs tests according to the include list and exclude list specified -in @code{#:test-include} and @code{#:test-exclude}, respectively. Their -meanings are analogous to that of @code{#:aot-include} and -@code{#:aot-exclude}, except that the special keyword @code{#:all} now -stands for all Clojure libraries found under the test directories. The -parameter @code{#:tests?} decides if tests should be run. +Finally, updating the package definition to a new upstream version +can be partly automated by the @command{guix refresh} command +(@pxref{Invoking guix refresh}). -@item install -This phase installs all jars built previously. +Behind the scenes, a derivation corresponding to the @code{} +object is first computed by the @code{package-derivation} procedure. +That derivation is stored in a @code{.drv} file under @file{/gnu/store}. +The build actions it prescribes may then be realized by using the +@code{build-derivations} procedure (@pxref{The Store}). + +@deffn {Scheme Procedure} package-derivation @var{store} @var{package} [@var{system}] +Return the @code{} object of @var{package} for @var{system} +(@pxref{Derivations}). + +@var{package} must be a valid @code{} object, and @var{system} +must be a string denoting the target system type---e.g., +@code{"x86_64-linux"} for an x86_64 Linux-based GNU system. @var{store} +must be a connection to the daemon, which operates on the store +(@pxref{The Store}). +@end deffn + +@noindent +@cindex cross-compilation +Similarly, it is possible to compute a derivation that cross-builds a +package for some other system: + +@deffn {Scheme Procedure} package-cross-derivation @var{store} @ + @var{package} @var{target} [@var{system}] +Return the @code{} object of @var{package} cross-built from +@var{system} to @var{target}. + +@var{target} must be a valid GNU triplet denoting the target hardware +and operating system, such as @code{"mips64el-linux-gnu"} +(@pxref{Configuration Names, GNU configuration triplets,, configure, GNU +Configure and Build System}). +@end deffn + +@cindex package transformations +@cindex input rewriting +@cindex dependency tree rewriting +Packages can be manipulated in arbitrary ways. An example of a useful +transformation is @dfn{input rewriting}, whereby the dependency tree of +a package is rewritten by replacing specific inputs by others: + +@deffn {Scheme Procedure} package-input-rewriting @var{replacements} @ + [@var{rewrite-name}] +Return a procedure that, when passed a package, replaces its direct and +indirect dependencies (but not its implicit inputs) according to +@var{replacements}. @var{replacements} is a list of package pairs; the +first element of each pair is the package to replace, and the second one +is the replacement. + +Optionally, @var{rewrite-name} is a one-argument procedure that takes +the name of a package and returns its new name after rewrite. +@end deffn + +@noindent +Consider this example: + +@example +(define libressl-instead-of-openssl + ;; This is a procedure to replace OPENSSL by LIBRESSL, + ;; recursively. + (package-input-rewriting `((,openssl . ,libressl)))) + +(define git-with-libressl + (libressl-instead-of-openssl git)) +@end example + +@noindent +Here we first define a rewriting procedure that replaces @var{openssl} +with @var{libressl}. Then we use it to define a @dfn{variant} of the +@var{git} package that uses @var{libressl} instead of @var{openssl}. +This is exactly what the @option{--with-input} command-line option does +(@pxref{Package Transformation Options, @option{--with-input}}). + +A more generic procedure to rewrite a package dependency graph is +@code{package-mapping}: it supports arbitrary changes to nodes in the +graph. + +@deffn {Scheme Procedure} package-mapping @var{proc} [@var{cut?}] +Return a procedure that, given a package, applies @var{proc} to all the packages +depended on and returns the resulting package. The procedure stops recursion +when @var{cut?} returns true for a given package. +@end deffn + +@menu +* package Reference:: The package data type. +* origin Reference:: The origin data type. +@end menu + + +@node package Reference +@subsection @code{package} Reference + +This section summarizes all the options available in @code{package} +declarations (@pxref{Defining Packages}). + +@deftp {Data Type} package +This is the data type representing a package recipe. + +@table @asis +@item @code{name} +The name of the package, as a string. + +@item @code{version} +The version of the package, as a string. + +@item @code{source} +An object telling how the source code for the package should be +acquired. Most of the time, this is an @code{origin} object, which +denotes a file fetched from the Internet (@pxref{origin Reference}). It +can also be any other ``file-like'' object such as a @code{local-file}, +which denotes a file from the local file system (@pxref{G-Expressions, +@code{local-file}}). + +@item @code{build-system} +The build system that should be used to build the package (@pxref{Build +Systems}). + +@item @code{arguments} (default: @code{'()}) +The arguments that should be passed to the build system. This is a +list, typically containing sequential keyword-value pairs. + +@item @code{inputs} (default: @code{'()}) +@itemx @code{native-inputs} (default: @code{'()}) +@itemx @code{propagated-inputs} (default: @code{'()}) +@cindex inputs, of packages +These fields list dependencies of the package. Each one is a list of +tuples, where each tuple has a label for the input (a string) as its +first element, a package, origin, or derivation as its second element, +and optionally the name of the output thereof that should be used, which +defaults to @code{"out"} (@pxref{Packages with Multiple Outputs}, for +more on package outputs). For example, the list below specifies three +inputs: + +@example +`(("libffi" ,libffi) + ("libunistring" ,libunistring) + ("glib:bin" ,glib "bin")) ;the "bin" output of Glib +@end example + +@cindex cross compilation, package dependencies +The distinction between @code{native-inputs} and @code{inputs} is +necessary when considering cross-compilation. When cross-compiling, +dependencies listed in @code{inputs} are built for the @emph{target} +architecture; conversely, dependencies listed in @code{native-inputs} +are built for the architecture of the @emph{build} machine. + +@code{native-inputs} is typically used to list tools needed at +build time, but not at run time, such as Autoconf, Automake, pkg-config, +Gettext, or Bison. @command{guix lint} can report likely mistakes in +this area (@pxref{Invoking guix lint}). + +@anchor{package-propagated-inputs} +Lastly, @code{propagated-inputs} is similar to @code{inputs}, but the +specified packages will be automatically installed alongside the package +they belong to (@pxref{package-cmd-propagated-inputs, @command{guix +package}}, for information on how @command{guix package} deals with +propagated inputs.) + +For example this is necessary when a C/C++ library needs headers of +another library to compile, or when a pkg-config file refers to another +one @i{via} its @code{Requires} field. + +Another example where @code{propagated-inputs} is useful is for languages +that lack a facility to record the run-time search path akin to the +@code{RUNPATH} of ELF files; this includes Guile, Python, Perl, and +more. To ensure that libraries written in those languages can find +library code they depend on at run time, run-time dependencies must be +listed in @code{propagated-inputs} rather than @code{inputs}. + +@item @code{self-native-input?} (default: @code{#f}) +This is a Boolean field telling whether the package should use itself as +a native input when cross-compiling. + +@item @code{outputs} (default: @code{'("out")}) +The list of output names of the package. @xref{Packages with Multiple +Outputs}, for typical uses of additional outputs. + +@item @code{native-search-paths} (default: @code{'()}) +@itemx @code{search-paths} (default: @code{'()}) +A list of @code{search-path-specification} objects describing +search-path environment variables honored by the package. + +@item @code{replacement} (default: @code{#f}) +This must be either @code{#f} or a package object that will be used as a +@dfn{replacement} for this package. @xref{Security Updates, grafts}, +for details. + +@item @code{synopsis} +A one-line description of the package. + +@item @code{description} +A more elaborate description of the package. + +@item @code{license} +@cindex license, of packages +The license of the package; a value from @code{(guix licenses)}, +or a list of such values. + +@item @code{home-page} +The URL to the home-page of the package, as a string. + +@item @code{supported-systems} (default: @var{%supported-systems}) +The list of systems supported by the package, as strings of the form +@code{architecture-kernel}, for example @code{"x86_64-linux"}. + +@item @code{maintainers} (default: @code{'()}) +The list of maintainers of the package, as @code{maintainer} objects. + +@item @code{location} (default: source location of the @code{package} form) +The source location of the package. It is useful to override this when +inheriting from another package, in which case this field is not +automatically corrected. @end table +@end deftp -Apart from the above, this build system also contains an additional phase: -@table @code +@node origin Reference +@subsection @code{origin} Reference -@item install-doc -This phase installs all top-level files with base name matching -@var{%doc-regex}. A different regex can be specified with the -@code{#:doc-regex} parameter. All files (recursively) inside the documentation -directories specified in @code{#:doc-dirs} are installed as well. -@end table -@end defvr +This section summarizes all the options available in @code{origin} +declarations (@pxref{Defining Packages}). -@defvr {Scheme Variable} cmake-build-system -This variable is exported by @code{(guix build-system cmake)}. It -implements the build procedure for packages using the -@url{http://www.cmake.org, CMake build tool}. +@deftp {Data Type} origin +This is the data type representing a source code origin. -It automatically adds the @code{cmake} package to the set of inputs. -Which package is used can be specified with the @code{#:cmake} -parameter. +@table @asis +@item @code{uri} +An object containing the URI of the source. The object type depends on +the @code{method} (see below). For example, when using the +@var{url-fetch} method of @code{(guix download)}, the valid @code{uri} +values are: a URL represented as a string, or a list thereof. -The @code{#:configure-flags} parameter is taken as a list of flags -passed to the @command{cmake} command. The @code{#:build-type} -parameter specifies in abstract terms the flags passed to the compiler; -it defaults to @code{"RelWithDebInfo"} (short for ``release mode with -debugging information''), which roughly means that code is compiled with -@code{-O2 -g}, as is the case for Autoconf-based packages by default. -@end defvr +@item @code{method} +A procedure that handles the URI. -@defvr {Scheme Variable} dune-build-system -This variable is exported by @code{(guix build-system dune)}. It -supports builds of packages using @uref{https://dune.build/, Dune}, a build -tool for the OCaml programming language. It is implemented as an extension -of the @code{ocaml-build-system} which is described below. As such, the -@code{#:ocaml} and @code{#:findlib} parameters can be passed to this build -system. +Examples include: -It automatically adds the @code{dune} package to the set of inputs. -Which package is used can be specified with the @code{#:dune} -parameter. +@table @asis +@item @var{url-fetch} from @code{(guix download)} +download a file from the HTTP, HTTPS, or FTP URL specified in the +@code{uri} field; -There is no @code{configure} phase because dune packages typically don't -need to be configured. The @code{#:build-flags} parameter is taken as a -list of flags passed to the @code{dune} command during the build. +@vindex git-fetch +@item @var{git-fetch} from @code{(guix git-download)} +clone the Git version control repository, and check out the revision +specified in the @code{uri} field as a @code{git-reference} object; a +@code{git-reference} looks like this: -The @code{#:jbuild?} parameter can be passed to use the @code{jbuild} -command instead of the more recent @code{dune} command while building -a package. Its default value is @code{#f}. -@end defvr +@example +(git-reference + (url "git://git.debian.org/git/pkg-shadow/shadow") + (commit "v4.1.5.1")) +@end example +@end table -@defvr {Scheme Variable} go-build-system -This variable is exported by @code{(guix build-system go)}. It -implements a build procedure for Go packages using the standard -@url{https://golang.org/cmd/go/#hdr-Compile_packages_and_dependencies, -Go build mechanisms}. +@item @code{sha256} +A bytevector containing the SHA-256 hash of the source. Typically the +@code{base32} form is used here to generate the bytevector from a +base-32 string. -The user is expected to provide a value for the key @code{#:import-path} -and, in some cases, @code{#:unpack-path}. The -@url{https://golang.org/doc/code.html#ImportPaths, import path} -corresponds to the file system path expected by the package's build -scripts and any referring packages, and provides a unique way to -refer to a Go package. It is typically based on a combination of the -package source code's remote URI and file system hierarchy structure. In -some cases, you will need to unpack the package's source code to a -different directory structure than the one indicated by the import path, -and @code{#:unpack-path} should be used in such cases. +You can obtain this information using @code{guix download} +(@pxref{Invoking guix download}) or @code{guix hash} (@pxref{Invoking +guix hash}). -Packages that provide Go libraries should be installed along with their -source code. The key @code{#:install-source?}, which defaults to -@code{#t}, controls whether or not the source code is installed. It can -be set to @code{#f} for packages that only provide executable files. -@end defvr +@item @code{file-name} (default: @code{#f}) +The file name under which the source code should be saved. When this is +@code{#f}, a sensible default value will be used in most cases. In case +the source is fetched from a URL, the file name from the URL will be +used. For version control checkouts, it is recommended to provide the +file name explicitly because the default is not very descriptive. -@defvr {Scheme Variable} glib-or-gtk-build-system -This variable is exported by @code{(guix build-system glib-or-gtk)}. It -is intended for use with packages making use of GLib or GTK+. +@item @code{patches} (default: @code{'()}) +A list of file names, origins, or file-like objects (@pxref{G-Expressions, +file-like objects}) pointing to patches to be applied to the source. -This build system adds the following two phases to the ones defined by -@var{gnu-build-system}: +This list of patches must be unconditional. In particular, it cannot +depend on the value of @code{%current-system} or +@code{%current-target-system}. -@table @code -@item glib-or-gtk-wrap -The phase @code{glib-or-gtk-wrap} ensures that programs in -@file{bin/} are able to find GLib ``schemas'' and -@uref{https://developer.gnome.org/gtk3/stable/gtk-running.html, GTK+ -modules}. This is achieved by wrapping the programs in launch scripts -that appropriately set the @code{XDG_DATA_DIRS} and @code{GTK_PATH} -environment variables. +@item @code{snippet} (default: @code{#f}) +A G-expression (@pxref{G-Expressions}) or S-expression that will be run +in the source directory. This is a convenient way to modify the source, +sometimes more convenient than a patch. -It is possible to exclude specific package outputs from that wrapping -process by listing their names in the -@code{#:glib-or-gtk-wrap-excluded-outputs} parameter. This is useful -when an output is known not to contain any GLib or GTK+ binaries, and -where wrapping would gratuitously add a dependency of that output on -GLib and GTK+. +@item @code{patch-flags} (default: @code{'("-p1")}) +A list of command-line flags that should be passed to the @code{patch} +command. -@item glib-or-gtk-compile-schemas -The phase @code{glib-or-gtk-compile-schemas} makes sure that all -@uref{https://developer.gnome.org/gio/stable/glib-compile-schemas.html, -GSettings schemas} of GLib are compiled. Compilation is performed by the -@command{glib-compile-schemas} program. It is provided by the package -@code{glib:bin} which is automatically imported by the build system. -The @code{glib} package providing @command{glib-compile-schemas} can be -specified with the @code{#:glib} parameter. +@item @code{patch-inputs} (default: @code{#f}) +Input packages or derivations to the patching process. When this is +@code{#f}, the usual set of inputs necessary for patching are provided, +such as GNU@tie{}Patch. + +@item @code{modules} (default: @code{'()}) +A list of Guile modules that should be loaded during the patching +process and while running the code in the @code{snippet} field. + +@item @code{patch-guile} (default: @code{#f}) +The Guile package that should be used in the patching process. When +this is @code{#f}, a sensible default is used. @end table +@end deftp -Both phases are executed after the @code{install} phase. -@end defvr -@defvr {Scheme Variable} guile-build-system -This build system is for Guile packages that consist exclusively of Scheme -code and that are so lean that they don't even have a makefile, let alone a -@file{configure} script. It compiles Scheme code using @command{guild -compile} (@pxref{Compilation,,, guile, GNU Guile Reference Manual}) and -installs the @file{.scm} and @file{.go} files in the right place. It also -installs documentation. +@node Build Systems +@section Build Systems -This build system supports cross-compilation by using the @code{--target} -option of @command{guild compile}. +@cindex build system +Each package definition specifies a @dfn{build system} and arguments for +that build system (@pxref{Defining Packages}). This @code{build-system} +field represents the build procedure of the package, as well as implicit +dependencies of that build procedure. -Packages built with @code{guile-build-system} must provide a Guile package in -their @code{native-inputs} field. -@end defvr +Build systems are @code{} objects. The interface to +create and manipulate them is provided by the @code{(guix build-system)} +module, and actual build systems are exported by specific modules. -@defvr {Scheme Variable} minify-build-system -This variable is exported by @code{(guix build-system minify)}. It -implements a minification procedure for simple JavaScript packages. +@cindex bag (low-level package representation) +Under the hood, build systems first compile package objects to +@dfn{bags}. A @dfn{bag} is like a package, but with less +ornamentation---in other words, a bag is a lower-level representation of +a package, which includes all the inputs of that package, including some +that were implicitly added by the build system. This intermediate +representation is then compiled to a derivation (@pxref{Derivations}). -It adds @code{uglify-js} to the set of inputs and uses it to compress -all JavaScript files in the @file{src} directory. A different minifier -package can be specified with the @code{#:uglify-js} parameter, but it -is expected that the package writes the minified code to the standard -output. +Build systems accept an optional list of @dfn{arguments}. In package +definitions, these are passed @i{via} the @code{arguments} field +(@pxref{Defining Packages}). They are typically keyword arguments +(@pxref{Optional Arguments, keyword arguments in Guile,, guile, GNU +Guile Reference Manual}). The value of these arguments is usually +evaluated in the @dfn{build stratum}---i.e., by a Guile process launched +by the daemon (@pxref{Derivations}). -When the input JavaScript files are not all located in the @file{src} -directory, the parameter @code{#:javascript-files} can be used to -specify a list of file names to feed to the minifier. -@end defvr +The main build system is @var{gnu-build-system}, which implements the +standard build procedure for GNU and many other packages. It +is provided by the @code{(guix build-system gnu)} module. -@defvr {Scheme Variable} ocaml-build-system -This variable is exported by @code{(guix build-system ocaml)}. It implements -a build procedure for @uref{https://ocaml.org, OCaml} packages, which consists -of choosing the correct set of commands to run for each package. OCaml -packages can expect many different commands to be run. This build system will -try some of them. +@defvr {Scheme Variable} gnu-build-system +@var{gnu-build-system} represents the GNU Build System, and variants +thereof (@pxref{Configuration, configuration and makefile conventions,, +standards, GNU Coding Standards}). -When the package has a @file{setup.ml} file present at the top-level, it will -run @code{ocaml setup.ml -configure}, @code{ocaml setup.ml -build} and -@code{ocaml setup.ml -install}. The build system will assume that this file -was generated by @uref{http://oasis.forge.ocamlcore.org/, OASIS} and will take -care of setting the prefix and enabling tests if they are not disabled. You -can pass configure and build flags with the @code{#:configure-flags} and -@code{#:build-flags}. The @code{#:test-flags} key can be passed to change the -set of flags used to enable tests. The @code{#:use-make?} key can be used to -bypass this system in the build and install phases. +@cindex build phases +In a nutshell, packages using it are configured, built, and installed with +the usual @code{./configure && make && make check && make install} +command sequence. In practice, a few additional steps are often needed. +All these steps are split up in separate @dfn{phases}, +notably@footnote{Please see the @code{(guix build gnu-build-system)} +modules for more details about the build phases.}: + +@table @code +@item unpack +Unpack the source tarball, and change the current directory to the +extracted source tree. If the source is actually a directory, copy it +to the build tree, and enter that directory. + +@item patch-source-shebangs +Patch shebangs encountered in source files so they refer to the right +store file names. For instance, this changes @code{#!/bin/sh} to +@code{#!/gnu/store/@dots{}-bash-4.3/bin/sh}. + +@item configure +Run the @file{configure} script with a number of default options, such +as @code{--prefix=/gnu/store/@dots{}}, as well as the options specified +by the @code{#:configure-flags} argument. + +@item build +Run @code{make} with the list of flags specified with +@code{#:make-flags}. If the @code{#:parallel-build?} argument is true +(the default), build with @code{make -j}. -When the package has a @file{configure} file, it is assumed that it is a -hand-made configure script that requires a different argument format than -in the @code{gnu-build-system}. You can add more flags with the -@code{#:configure-flags} key. +@item check +Run @code{make check}, or some other target specified with +@code{#:test-target}, unless @code{#:tests? #f} is passed. If the +@code{#:parallel-tests?} argument is true (the default), run @code{make +check -j}. -When the package has a @file{Makefile} file (or @code{#:use-make?} is -@code{#t}), it will be used and more flags can be passed to the build and -install phases with the @code{#:make-flags} key. +@item install +Run @code{make install} with the flags listed in @code{#:make-flags}. -Finally, some packages do not have these files and use a somewhat standard -location for its build system. In that case, the build system will run -@code{ocaml pkg/pkg.ml} or @code{ocaml pkg/build.ml} and take care of -providing the path to the required findlib module. Additional flags can -be passed via the @code{#:build-flags} key. Install is taken care of by -@command{opam-installer}. In this case, the @code{opam} package must -be added to the @code{native-inputs} field of the package definition. +@item patch-shebangs +Patch shebangs on the installed executable files. -Note that most OCaml packages assume they will be installed in the same -directory as OCaml, which is not what we want in guix. In particular, they -will install @file{.so} files in their module's directory, which is usually -fine because it is in the OCaml compiler directory. In guix though, these -libraries cannot be found and we use @code{CAML_LD_LIBRARY_PATH}. This -variable points to @file{lib/ocaml/site-lib/stubslibs} and this is where -@file{.so} libraries should be installed. -@end defvr +@item strip +Strip debugging symbols from ELF files (unless @code{#:strip-binaries?} +is false), copying them to the @code{debug} output when available +(@pxref{Installing Debugging Files}). +@end table -@defvr {Scheme Variable} python-build-system -This variable is exported by @code{(guix build-system python)}. It -implements the more or less standard build procedure used by Python -packages, which consists in running @code{python setup.py build} and -then @code{python setup.py install --prefix=/gnu/store/@dots{}}. +@vindex %standard-phases +The build-side module @code{(guix build gnu-build-system)} defines +@var{%standard-phases} as the default list of build phases. +@var{%standard-phases} is a list of symbol/procedure pairs, where the +procedure implements the actual phase. -For packages that install stand-alone Python programs under @code{bin/}, -it takes care of wrapping these programs so that their @code{PYTHONPATH} -environment variable points to all the Python libraries they depend on. +The list of phases used for a particular package can be changed with the +@code{#:phases} parameter. For instance, passing: -Which Python package is used to perform the build can be specified with -the @code{#:python} parameter. This is a useful way to force a package -to be built for a specific version of the Python interpreter, which -might be necessary if the package is only compatible with a single -interpreter version. +@example +#:phases (modify-phases %standard-phases (delete 'configure)) +@end example -By default guix calls @code{setup.py} under control of -@code{setuptools}, much like @command{pip} does. Some packages are not -compatible with setuptools (and pip), thus you can disable this by -setting the @code{#:use-setuptools} parameter to @code{#f}. +means that all the phases described above will be used, except the +@code{configure} phase. + +In addition, this build system ensures that the ``standard'' environment +for GNU packages is available. This includes tools such as GCC, libc, +Coreutils, Bash, Make, Diffutils, grep, and sed (see the @code{(guix +build-system gnu)} module for a complete list). We call these the +@dfn{implicit inputs} of a package, because package definitions do not +have to mention them. @end defvr -@defvr {Scheme Variable} perl-build-system -This variable is exported by @code{(guix build-system perl)}. It -implements the standard build procedure for Perl packages, which either -consists in running @code{perl Build.PL --prefix=/gnu/store/@dots{}}, -followed by @code{Build} and @code{Build install}; or in running -@code{perl Makefile.PL PREFIX=/gnu/store/@dots{}}, followed by -@code{make} and @code{make install}, depending on which of -@code{Build.PL} or @code{Makefile.PL} is present in the package -distribution. Preference is given to the former if both @code{Build.PL} -and @code{Makefile.PL} exist in the package distribution. This -preference can be reversed by specifying @code{#t} for the -@code{#:make-maker?} parameter. +Other @code{} objects are defined to support other +conventions and tools used by free software packages. They inherit most +of @var{gnu-build-system}, and differ mainly in the set of inputs +implicitly added to the build process, and in the list of phases +executed. Some of these build systems are listed below. -The initial @code{perl Makefile.PL} or @code{perl Build.PL} invocation -passes flags specified by the @code{#:make-maker-flags} or -@code{#:module-build-flags} parameter, respectively. +@defvr {Scheme Variable} ant-build-system +This variable is exported by @code{(guix build-system ant)}. It +implements the build procedure for Java packages that can be built with +@url{http://ant.apache.org/, Ant build tool}. -Which Perl package is used can be specified with @code{#:perl}. -@end defvr +It adds both @code{ant} and the @dfn{Java Development Kit} (JDK) as +provided by the @code{icedtea} package to the set of inputs. Different +packages can be specified with the @code{#:ant} and @code{#:jdk} +parameters, respectively. -@defvr {Scheme Variable} r-build-system -This variable is exported by @code{(guix build-system r)}. It -implements the build procedure used by @uref{http://r-project.org, R} -packages, which essentially is little more than running @code{R CMD -INSTALL --library=/gnu/store/@dots{}} in an environment where -@code{R_LIBS_SITE} contains the paths to all R package inputs. Tests -are run after installation using the R function -@code{tools::testInstalledPackage}. -@end defvr +When the original package does not provide a suitable Ant build file, +the parameter @code{#:jar-name} can be used to generate a minimal Ant +build file @file{build.xml} with tasks to build the specified jar +archive. In this case the parameter @code{#:source-dir} can be used to +specify the source sub-directory, defaulting to ``src''. -@defvr {Scheme Variable} texlive-build-system -This variable is exported by @code{(guix build-system texlive)}. It is -used to build TeX packages in batch mode with a specified engine. The -build system sets the @code{TEXINPUTS} variable to find all TeX source -files in the inputs. +The @code{#:main-class} parameter can be used with the minimal ant +buildfile to specify the main class of the resulting jar. This makes the +jar file executable. The @code{#:test-include} parameter can be used to +specify the list of junit tests to run. It defaults to +@code{(list "**/*Test.java")}. The @code{#:test-exclude} can be used to +disable some tests. It defaults to @code{(list "**/Abstract*.java")}, +because abstract classes cannot be run as tests. -By default it runs @code{luatex} on all files ending on @code{ins}. A -different engine and format can be specified with the -@code{#:tex-format} argument. Different build targets can be specified -with the @code{#:build-targets} argument, which expects a list of file -names. The build system adds only @code{texlive-bin} and -@code{texlive-latex-base} (both from @code{(gnu packages tex}) to the -inputs. Both can be overridden with the arguments @code{#:texlive-bin} -and @code{#:texlive-latex-base}, respectively. +The parameter @code{#:build-target} can be used to specify the Ant task +that should be run during the @code{build} phase. By default the +``jar'' task will be run. -The @code{#:tex-directory} parameter tells the build system where to -install the built files under the texmf tree. @end defvr -@defvr {Scheme Variable} ruby-build-system -This variable is exported by @code{(guix build-system ruby)}. It -implements the RubyGems build procedure used by Ruby packages, which -involves running @code{gem build} followed by @code{gem install}. +@defvr {Scheme Variable} android-ndk-build-system +@cindex Android distribution +@cindex Android NDK build system +This variable is exported by @code{(guix build-system android-ndk)}. It +implements a build procedure for Android NDK (native development kit) +packages using a Guix-specific build process. -The @code{source} field of a package that uses this build system -typically references a gem archive, since this is the format that Ruby -developers use when releasing their software. The build system unpacks -the gem archive, potentially patches the source, runs the test suite, -repackages the gem, and installs it. Additionally, directories and -tarballs may be referenced to allow building unreleased gems from Git or -a traditional source release tarball. +The build system assumes that packages install their public interface +(header) files to the subdirectory "include" of the "out" output and +their libraries to the subdirectory "lib" of the "out" output. -Which Ruby package is used can be specified with the @code{#:ruby} -parameter. A list of additional flags to be passed to the @command{gem} -command can be specified with the @code{#:gem-flags} parameter. -@end defvr +It's also assumed that the union of all the dependencies of a package +has no conflicting files. -@defvr {Scheme Variable} waf-build-system -This variable is exported by @code{(guix build-system waf)}. It -implements a build procedure around the @code{waf} script. The common -phases---@code{configure}, @code{build}, and @code{install}---are -implemented by passing their names as arguments to the @code{waf} -script. +For the time being, cross-compilation is not supported - so right now +the libraries and header files are assumed to be host tools. -The @code{waf} script is executed by the Python interpreter. Which -Python package is used to run the script can be specified with the -@code{#:python} parameter. @end defvr -@defvr {Scheme Variable} scons-build-system -This variable is exported by @code{(guix build-system scons)}. It -implements the build procedure used by the SCons software construction -tool. This build system runs @code{scons} to build the package, -@code{scons test} to run tests, and then @code{scons install} to install -the package. +@defvr {Scheme Variable} asdf-build-system/source +@defvrx {Scheme Variable} asdf-build-system/sbcl +@defvrx {Scheme Variable} asdf-build-system/ecl -Additional flags to be passed to @code{scons} can be specified with the -@code{#:scons-flags} parameter. The version of Python used to run SCons -can be specified by selecting the appropriate SCons package with the -@code{#:scons} parameter. -@end defvr +These variables, exported by @code{(guix build-system asdf)}, implement +build procedures for Common Lisp packages using +@url{https://common-lisp.net/project/asdf/, ``ASDF''}. ASDF is a system +definition facility for Common Lisp programs and libraries. -@defvr {Scheme Variable} haskell-build-system -This variable is exported by @code{(guix build-system haskell)}. It -implements the Cabal build procedure used by Haskell packages, which -involves running @code{runhaskell Setup.hs configure ---prefix=/gnu/store/@dots{}} and @code{runhaskell Setup.hs build}. -Instead of installing the package by running @code{runhaskell Setup.hs -install}, to avoid trying to register libraries in the read-only -compiler store directory, the build system uses @code{runhaskell -Setup.hs copy}, followed by @code{runhaskell Setup.hs register}. In -addition, the build system generates the package documentation by -running @code{runhaskell Setup.hs haddock}, unless @code{#:haddock? #f} -is passed. Optional Haddock parameters can be passed with the help of -the @code{#:haddock-flags} parameter. If the file @code{Setup.hs} is -not found, the build system looks for @code{Setup.lhs} instead. +The @code{asdf-build-system/source} system installs the packages in +source form, and can be loaded using any common lisp implementation, via +ASDF. The others, such as @code{asdf-build-system/sbcl}, install binary +systems in the format which a particular implementation understands. +These build systems can also be used to produce executable programs, or +lisp images which contain a set of packages pre-loaded. -Which Haskell compiler is used can be specified with the @code{#:haskell} -parameter which defaults to @code{ghc}. -@end defvr +The build system uses naming conventions. For binary packages, the +package name should be prefixed with the lisp implementation, such as +@code{sbcl-} for @code{asdf-build-system/sbcl}. -@defvr {Scheme Variable} dub-build-system -This variable is exported by @code{(guix build-system dub)}. It -implements the Dub build procedure used by D packages, which -involves running @code{dub build} and @code{dub run}. -Installation is done by copying the files manually. +Additionally, the corresponding source package should be labeled using +the same convention as python packages (see @ref{Python Modules}), using +the @code{cl-} prefix. + +For binary packages, each system should be defined as a Guix package. +If one package @code{origin} contains several systems, package variants +can be created in order to build all the systems. Source packages, +which use @code{asdf-build-system/source}, may contain several systems. + +In order to create executable programs and images, the build-side +procedures @code{build-program} and @code{build-image} can be used. +They should be called in a build phase after the @code{create-symlinks} +phase, so that the system which was just built can be used within the +resulting image. @code{build-program} requires a list of Common Lisp +expressions to be passed as the @code{#:entry-program} argument. + +If the system is not defined within its own @code{.asd} file of the same +name, then the @code{#:asd-file} parameter should be used to specify +which file the system is defined in. Furthermore, if the package +defines a system for its tests in a separate file, it will be loaded +before the tests are run if it is specified by the +@code{#:test-asd-file} parameter. If it is not set, the files +@code{-tests.asd}, @code{-test.asd}, @code{tests.asd}, +and @code{test.asd} will be tried if they exist. + +If for some reason the package must be named in a different way than the +naming conventions suggest, the @code{#:asd-system-name} parameter can +be used to specify the name of the system. -Which D compiler is used can be specified with the @code{#:ldc} -parameter which defaults to @code{ldc}. @end defvr -@defvr {Scheme Variable} emacs-build-system -This variable is exported by @code{(guix build-system emacs)}. It -implements an installation procedure similar to the packaging system -of Emacs itself (@pxref{Packages,,, emacs, The GNU Emacs Manual}). +@defvr {Scheme Variable} cargo-build-system +@cindex Rust programming language +@cindex Cargo (Rust build system) +This variable is exported by @code{(guix build-system cargo)}. It +supports builds of packages using Cargo, the build tool of the +@uref{https://www.rust-lang.org, Rust programming language}. -It first creates the @code{@var{package}-autoloads.el} file, then it -byte compiles all Emacs Lisp files. Differently from the Emacs -packaging system, the Info documentation files are moved to the standard -documentation directory and the @file{dir} file is deleted. Each -package is installed in its own directory under -@file{share/emacs/site-lisp/guix.d}. +In its @code{configure} phase, this build system replaces dependencies +specified in the @file{Carto.toml} file with inputs to the Guix package. +The @code{install} phase installs the binaries, and it also installs the +source code and @file{Cargo.toml} file. @end defvr -@defvr {Scheme Variable} font-build-system -This variable is exported by @code{(guix build-system font)}. It -implements an installation procedure for font packages where upstream -provides pre-compiled TrueType, OpenType, etc.@: font files that merely -need to be copied into place. It copies font files to standard -locations in the output directory. -@end defvr +@cindex Clojure (programming language) +@cindex simple Clojure build system +@defvr {Scheme Variable} clojure-build-system +This variable is exported by @code{(guix build-system clojure)}. It implements +a simple build procedure for @uref{https://clojure.org/, Clojure} packages +using plain old @code{compile} in Clojure. Cross-compilation is not supported +yet. -@defvr {Scheme Variable} meson-build-system -This variable is exported by @code{(guix build-system meson)}. It -implements the build procedure for packages that use -@url{http://mesonbuild.com, Meson} as their build system. +It adds @code{clojure}, @code{icedtea} and @code{zip} to the set of inputs. +Different packages can be specified with the @code{#:clojure}, @code{#:jdk} and +@code{#:zip} parameters, respectively. -It adds both Meson and @uref{https://ninja-build.org/, Ninja} to the set -of inputs, and they can be changed with the parameters @code{#:meson} -and @code{#:ninja} if needed. The default Meson is -@code{meson-for-build}, which is special because it doesn't clear the -@code{RUNPATH} of binaries and libraries when they are installed. +A list of source directories, test directories and jar names can be specified +with the @code{#:source-dirs}, @code{#:test-dirs} and @code{#:jar-names} +parameters, respectively. Compile directory and main class can be specified +with the @code{#:compile-dir} and @code{#:main-class} parameters, respectively. +Other parameters are documented below. -This build system is an extension of @var{gnu-build-system}, but with the -following phases changed to some specific for Meson: +This build system is an extension of @var{ant-build-system}, but with the +following phases changed: @table @code -@item configure -The phase runs @code{meson} with the flags specified in -@code{#:configure-flags}. The flag @code{--build-type} is always set to -@code{plain} unless something else is specified in @code{#:build-type}. - @item build -The phase runs @code{ninja} to build the package in parallel by default, but -this can be changed with @code{#:parallel-build?}. +This phase calls @code{compile} in Clojure to compile source files and runs +@command{jar} to create jars from both source files and compiled files +according to the include list and exclude list specified in +@code{#:aot-include} and @code{#:aot-exclude}, respectively. The exclude list +has priority over the include list. These lists consist of symbols +representing Clojure libraries or the special keyword @code{#:all} representing +all Clojure libraries found under the source directories. The parameter +@code{#:omit-source?} decides if source should be included into the jars. @item check -The phase runs @code{ninja} with the target specified in @code{#:test-target}, -which is @code{"test"} by default. +This phase runs tests according to the include list and exclude list specified +in @code{#:test-include} and @code{#:test-exclude}, respectively. Their +meanings are analogous to that of @code{#:aot-include} and +@code{#:aot-exclude}, except that the special keyword @code{#:all} now +stands for all Clojure libraries found under the test directories. The +parameter @code{#:tests?} decides if tests should be run. @item install -The phase runs @code{ninja install} and can not be changed. +This phase installs all jars built previously. @end table -Apart from that, the build system also adds the following phases: +Apart from the above, this build system also contains an additional phase: @table @code -@item fix-runpath -This phase ensures that all binaries can find the libraries they need. -It searches for required libraries in subdirectories of the package being -built, and adds those to @code{RUNPATH} where needed. It also removes -references to libraries left over from the build phase by -@code{meson-for-build}, such as test dependencies, that aren't actually -required for the program to run. - -@item glib-or-gtk-wrap -This phase is the phase provided by @code{glib-or-gtk-build-system}, and it -is not enabled by default. It can be enabled with @code{#:glib-or-gtk?}. - -@item glib-or-gtk-compile-schemas -This phase is the phase provided by @code{glib-or-gtk-build-system}, and it -is not enabled by default. It can be enabled with @code{#:glib-or-gtk?}. +@item install-doc +This phase installs all top-level files with base name matching +@var{%doc-regex}. A different regex can be specified with the +@code{#:doc-regex} parameter. All files (recursively) inside the documentation +directories specified in @code{#:doc-dirs} are installed as well. @end table @end defvr -Lastly, for packages that do not need anything as sophisticated, a -``trivial'' build system is provided. It is trivial in the sense that -it provides basically no support: it does not pull any implicit inputs, -and does not have a notion of build phases. +@defvr {Scheme Variable} cmake-build-system +This variable is exported by @code{(guix build-system cmake)}. It +implements the build procedure for packages using the +@url{http://www.cmake.org, CMake build tool}. -@defvr {Scheme Variable} trivial-build-system -This variable is exported by @code{(guix build-system trivial)}. +It automatically adds the @code{cmake} package to the set of inputs. +Which package is used can be specified with the @code{#:cmake} +parameter. -This build system requires a @code{#:builder} argument. This argument -must be a Scheme expression that builds the package output(s)---as -with @code{build-expression->derivation} (@pxref{Derivations, -@code{build-expression->derivation}}). +The @code{#:configure-flags} parameter is taken as a list of flags +passed to the @command{cmake} command. The @code{#:build-type} +parameter specifies in abstract terms the flags passed to the compiler; +it defaults to @code{"RelWithDebInfo"} (short for ``release mode with +debugging information''), which roughly means that code is compiled with +@code{-O2 -g}, as is the case for Autoconf-based packages by default. @end defvr -@node The Store -@section The Store - -@cindex store -@cindex store items -@cindex store paths - -Conceptually, the @dfn{store} is the place where derivations that have -been built successfully are stored---by default, @file{/gnu/store}. -Sub-directories in the store are referred to as @dfn{store items} or -sometimes @dfn{store paths}. The store has an associated database that -contains information such as the store paths referred to by each store -path, and the list of @emph{valid} store items---results of successful -builds. This database resides in @file{@var{localstatedir}/guix/db}, -where @var{localstatedir} is the state directory specified @i{via} -@option{--localstatedir} at configure time, usually @file{/var}. +@defvr {Scheme Variable} dune-build-system +This variable is exported by @code{(guix build-system dune)}. It +supports builds of packages using @uref{https://dune.build/, Dune}, a build +tool for the OCaml programming language. It is implemented as an extension +of the @code{ocaml-build-system} which is described below. As such, the +@code{#:ocaml} and @code{#:findlib} parameters can be passed to this build +system. -The store is @emph{always} accessed by the daemon on behalf of its clients -(@pxref{Invoking guix-daemon}). To manipulate the store, clients -connect to the daemon over a Unix-domain socket, send requests to it, -and read the result---these are remote procedure calls, or RPCs. +It automatically adds the @code{dune} package to the set of inputs. +Which package is used can be specified with the @code{#:dune} +parameter. -@quotation Note -Users must @emph{never} modify files under @file{/gnu/store} directly. -This would lead to inconsistencies and break the immutability -assumptions of Guix's functional model (@pxref{Introduction}). +There is no @code{configure} phase because dune packages typically don't +need to be configured. The @code{#:build-flags} parameter is taken as a +list of flags passed to the @code{dune} command during the build. -@xref{Invoking guix gc, @command{guix gc --verify}}, for information on -how to check the integrity of the store and attempt recovery from -accidental modifications. -@end quotation +The @code{#:jbuild?} parameter can be passed to use the @code{jbuild} +command instead of the more recent @code{dune} command while building +a package. Its default value is @code{#f}. +@end defvr -The @code{(guix store)} module provides procedures to connect to the -daemon, and to perform RPCs. These are described below. By default, -@code{open-connection}, and thus all the @command{guix} commands, -connect to the local daemon or to the URI specified by the -@code{GUIX_DAEMON_SOCKET} environment variable. +@defvr {Scheme Variable} go-build-system +This variable is exported by @code{(guix build-system go)}. It +implements a build procedure for Go packages using the standard +@url{https://golang.org/cmd/go/#hdr-Compile_packages_and_dependencies, +Go build mechanisms}. -@defvr {Environment Variable} GUIX_DAEMON_SOCKET -When set, the value of this variable should be a file name or a URI -designating the daemon endpoint. When it is a file name, it denotes a -Unix-domain socket to connect to. In addition to file names, the -supported URI schemes are: +The user is expected to provide a value for the key @code{#:import-path} +and, in some cases, @code{#:unpack-path}. The +@url{https://golang.org/doc/code.html#ImportPaths, import path} +corresponds to the file system path expected by the package's build +scripts and any referring packages, and provides a unique way to +refer to a Go package. It is typically based on a combination of the +package source code's remote URI and file system hierarchy structure. In +some cases, you will need to unpack the package's source code to a +different directory structure than the one indicated by the import path, +and @code{#:unpack-path} should be used in such cases. -@table @code -@item file -@itemx unix -These are for Unix-domain sockets. -@code{file:///var/guix/daemon-socket/socket} is equivalent to -@file{/var/guix/daemon-socket/socket}. +Packages that provide Go libraries should be installed along with their +source code. The key @code{#:install-source?}, which defaults to +@code{#t}, controls whether or not the source code is installed. It can +be set to @code{#f} for packages that only provide executable files. +@end defvr -@item guix -@cindex daemon, remote access -@cindex remote access to the daemon -@cindex daemon, cluster setup -@cindex clusters, daemon setup -These URIs denote connections over TCP/IP, without encryption nor -authentication of the remote host. The URI must specify the host name -and optionally a port number (by default port 44146 is used): +@defvr {Scheme Variable} glib-or-gtk-build-system +This variable is exported by @code{(guix build-system glib-or-gtk)}. It +is intended for use with packages making use of GLib or GTK+. -@example -guix://master.guix.example.org:1234 -@end example +This build system adds the following two phases to the ones defined by +@var{gnu-build-system}: -This setup is suitable on local networks, such as clusters, where only -trusted nodes may connect to the build daemon at -@code{master.guix.example.org}. +@table @code +@item glib-or-gtk-wrap +The phase @code{glib-or-gtk-wrap} ensures that programs in +@file{bin/} are able to find GLib ``schemas'' and +@uref{https://developer.gnome.org/gtk3/stable/gtk-running.html, GTK+ +modules}. This is achieved by wrapping the programs in launch scripts +that appropriately set the @code{XDG_DATA_DIRS} and @code{GTK_PATH} +environment variables. -The @code{--listen} option of @command{guix-daemon} can be used to -instruct it to listen for TCP connections (@pxref{Invoking guix-daemon, -@code{--listen}}). +It is possible to exclude specific package outputs from that wrapping +process by listing their names in the +@code{#:glib-or-gtk-wrap-excluded-outputs} parameter. This is useful +when an output is known not to contain any GLib or GTK+ binaries, and +where wrapping would gratuitously add a dependency of that output on +GLib and GTK+. -@item ssh -@cindex SSH access to build daemons -These URIs allow you to connect to a remote daemon over -SSH@footnote{This feature requires Guile-SSH (@pxref{Requirements}).}. -A typical URL might look like this: +@item glib-or-gtk-compile-schemas +The phase @code{glib-or-gtk-compile-schemas} makes sure that all +@uref{https://developer.gnome.org/gio/stable/glib-compile-schemas.html, +GSettings schemas} of GLib are compiled. Compilation is performed by the +@command{glib-compile-schemas} program. It is provided by the package +@code{glib:bin} which is automatically imported by the build system. +The @code{glib} package providing @command{glib-compile-schemas} can be +specified with the @code{#:glib} parameter. +@end table -@example -ssh://charlie@@guix.example.org:22 -@end example +Both phases are executed after the @code{install} phase. +@end defvr -As for @command{guix copy}, the usual OpenSSH client configuration files -are honored (@pxref{Invoking guix copy}). -@end table +@defvr {Scheme Variable} guile-build-system +This build system is for Guile packages that consist exclusively of Scheme +code and that are so lean that they don't even have a makefile, let alone a +@file{configure} script. It compiles Scheme code using @command{guild +compile} (@pxref{Compilation,,, guile, GNU Guile Reference Manual}) and +installs the @file{.scm} and @file{.go} files in the right place. It also +installs documentation. -Additional URI schemes may be supported in the future. +This build system supports cross-compilation by using the @code{--target} +option of @command{guild compile}. -@c XXX: Remove this note when the protocol incurs fewer round trips -@c and when (guix derivations) no longer relies on file system access. -@quotation Note -The ability to connect to remote build daemons is considered -experimental as of @value{VERSION}. Please get in touch with us to -share any problems or suggestions you may have (@pxref{Contributing}). -@end quotation +Packages built with @code{guile-build-system} must provide a Guile package in +their @code{native-inputs} field. @end defvr -@deffn {Scheme Procedure} open-connection [@var{uri}] [#:reserve-space? #t] -Connect to the daemon over the Unix-domain socket at @var{uri} (a string). When -@var{reserve-space?} is true, instruct it to reserve a little bit of -extra space on the file system so that the garbage collector can still -operate should the disk become full. Return a server object. - -@var{file} defaults to @var{%default-socket-path}, which is the normal -location given the options that were passed to @command{configure}. -@end deffn +@defvr {Scheme Variable} minify-build-system +This variable is exported by @code{(guix build-system minify)}. It +implements a minification procedure for simple JavaScript packages. -@deffn {Scheme Procedure} close-connection @var{server} -Close the connection to @var{server}. -@end deffn +It adds @code{uglify-js} to the set of inputs and uses it to compress +all JavaScript files in the @file{src} directory. A different minifier +package can be specified with the @code{#:uglify-js} parameter, but it +is expected that the package writes the minified code to the standard +output. -@defvr {Scheme Variable} current-build-output-port -This variable is bound to a SRFI-39 parameter, which refers to the port -where build and error logs sent by the daemon should be written. +When the input JavaScript files are not all located in the @file{src} +directory, the parameter @code{#:javascript-files} can be used to +specify a list of file names to feed to the minifier. @end defvr -Procedures that make RPCs all take a server object as their first -argument. +@defvr {Scheme Variable} ocaml-build-system +This variable is exported by @code{(guix build-system ocaml)}. It implements +a build procedure for @uref{https://ocaml.org, OCaml} packages, which consists +of choosing the correct set of commands to run for each package. OCaml +packages can expect many different commands to be run. This build system will +try some of them. -@deffn {Scheme Procedure} valid-path? @var{server} @var{path} -@cindex invalid store items -Return @code{#t} when @var{path} designates a valid store item and -@code{#f} otherwise (an invalid item may exist on disk but still be -invalid, for instance because it is the result of an aborted or failed -build.) +When the package has a @file{setup.ml} file present at the top-level, it will +run @code{ocaml setup.ml -configure}, @code{ocaml setup.ml -build} and +@code{ocaml setup.ml -install}. The build system will assume that this file +was generated by @uref{http://oasis.forge.ocamlcore.org/, OASIS} and will take +care of setting the prefix and enabling tests if they are not disabled. You +can pass configure and build flags with the @code{#:configure-flags} and +@code{#:build-flags}. The @code{#:test-flags} key can be passed to change the +set of flags used to enable tests. The @code{#:use-make?} key can be used to +bypass this system in the build and install phases. -A @code{&store-protocol-error} condition is raised if @var{path} is not -prefixed by the store directory (@file{/gnu/store}). -@end deffn +When the package has a @file{configure} file, it is assumed that it is a +hand-made configure script that requires a different argument format than +in the @code{gnu-build-system}. You can add more flags with the +@code{#:configure-flags} key. -@deffn {Scheme Procedure} add-text-to-store @var{server} @var{name} @var{text} [@var{references}] -Add @var{text} under file @var{name} in the store, and return its store -path. @var{references} is the list of store paths referred to by the -resulting store path. -@end deffn +When the package has a @file{Makefile} file (or @code{#:use-make?} is +@code{#t}), it will be used and more flags can be passed to the build and +install phases with the @code{#:make-flags} key. -@deffn {Scheme Procedure} build-derivations @var{server} @var{derivations} -Build @var{derivations} (a list of @code{} objects or -derivation paths), and return when the worker is done building them. -Return @code{#t} on success. -@end deffn +Finally, some packages do not have these files and use a somewhat standard +location for its build system. In that case, the build system will run +@code{ocaml pkg/pkg.ml} or @code{ocaml pkg/build.ml} and take care of +providing the path to the required findlib module. Additional flags can +be passed via the @code{#:build-flags} key. Install is taken care of by +@command{opam-installer}. In this case, the @code{opam} package must +be added to the @code{native-inputs} field of the package definition. -Note that the @code{(guix monads)} module provides a monad as well as -monadic versions of the above procedures, with the goal of making it -more convenient to work with code that accesses the store (@pxref{The -Store Monad}). +Note that most OCaml packages assume they will be installed in the same +directory as OCaml, which is not what we want in guix. In particular, they +will install @file{.so} files in their module's directory, which is usually +fine because it is in the OCaml compiler directory. In guix though, these +libraries cannot be found and we use @code{CAML_LD_LIBRARY_PATH}. This +variable points to @file{lib/ocaml/site-lib/stubslibs} and this is where +@file{.so} libraries should be installed. +@end defvr -@c FIXME -@i{This section is currently incomplete.} +@defvr {Scheme Variable} python-build-system +This variable is exported by @code{(guix build-system python)}. It +implements the more or less standard build procedure used by Python +packages, which consists in running @code{python setup.py build} and +then @code{python setup.py install --prefix=/gnu/store/@dots{}}. -@node Derivations -@section Derivations +For packages that install stand-alone Python programs under @code{bin/}, +it takes care of wrapping these programs so that their @code{PYTHONPATH} +environment variable points to all the Python libraries they depend on. -@cindex derivations -Low-level build actions and the environment in which they are performed -are represented by @dfn{derivations}. A derivation contains the -following pieces of information: +Which Python package is used to perform the build can be specified with +the @code{#:python} parameter. This is a useful way to force a package +to be built for a specific version of the Python interpreter, which +might be necessary if the package is only compatible with a single +interpreter version. -@itemize -@item -The outputs of the derivation---derivations produce at least one file or -directory in the store, but may produce more. +By default guix calls @code{setup.py} under control of +@code{setuptools}, much like @command{pip} does. Some packages are not +compatible with setuptools (and pip), thus you can disable this by +setting the @code{#:use-setuptools} parameter to @code{#f}. +@end defvr -@item -The inputs of the derivations, which may be other derivations or plain -files in the store (patches, build scripts, etc.) +@defvr {Scheme Variable} perl-build-system +This variable is exported by @code{(guix build-system perl)}. It +implements the standard build procedure for Perl packages, which either +consists in running @code{perl Build.PL --prefix=/gnu/store/@dots{}}, +followed by @code{Build} and @code{Build install}; or in running +@code{perl Makefile.PL PREFIX=/gnu/store/@dots{}}, followed by +@code{make} and @code{make install}, depending on which of +@code{Build.PL} or @code{Makefile.PL} is present in the package +distribution. Preference is given to the former if both @code{Build.PL} +and @code{Makefile.PL} exist in the package distribution. This +preference can be reversed by specifying @code{#t} for the +@code{#:make-maker?} parameter. -@item -The system type targeted by the derivation---e.g., @code{x86_64-linux}. +The initial @code{perl Makefile.PL} or @code{perl Build.PL} invocation +passes flags specified by the @code{#:make-maker-flags} or +@code{#:module-build-flags} parameter, respectively. -@item -The file name of a build script in the store, along with the arguments -to be passed. +Which Perl package is used can be specified with @code{#:perl}. +@end defvr -@item -A list of environment variables to be defined. +@defvr {Scheme Variable} r-build-system +This variable is exported by @code{(guix build-system r)}. It +implements the build procedure used by @uref{http://r-project.org, R} +packages, which essentially is little more than running @code{R CMD +INSTALL --library=/gnu/store/@dots{}} in an environment where +@code{R_LIBS_SITE} contains the paths to all R package inputs. Tests +are run after installation using the R function +@code{tools::testInstalledPackage}. +@end defvr -@end itemize +@defvr {Scheme Variable} texlive-build-system +This variable is exported by @code{(guix build-system texlive)}. It is +used to build TeX packages in batch mode with a specified engine. The +build system sets the @code{TEXINPUTS} variable to find all TeX source +files in the inputs. -@cindex derivation path -Derivations allow clients of the daemon to communicate build actions to -the store. They exist in two forms: as an in-memory representation, -both on the client- and daemon-side, and as files in the store whose -name end in @code{.drv}---these files are referred to as @dfn{derivation -paths}. Derivations paths can be passed to the @code{build-derivations} -procedure to perform the build actions they prescribe (@pxref{The -Store}). +By default it runs @code{luatex} on all files ending on @code{ins}. A +different engine and format can be specified with the +@code{#:tex-format} argument. Different build targets can be specified +with the @code{#:build-targets} argument, which expects a list of file +names. The build system adds only @code{texlive-bin} and +@code{texlive-latex-base} (both from @code{(gnu packages tex}) to the +inputs. Both can be overridden with the arguments @code{#:texlive-bin} +and @code{#:texlive-latex-base}, respectively. -@cindex fixed-output derivations -Operations such as file downloads and version-control checkouts for -which the expected content hash is known in advance are modeled as -@dfn{fixed-output derivations}. Unlike regular derivations, the outputs -of a fixed-output derivation are independent of its inputs---e.g., a -source code download produces the same result regardless of the download -method and tools being used. +The @code{#:tex-directory} parameter tells the build system where to +install the built files under the texmf tree. +@end defvr -The @code{(guix derivations)} module provides a representation of -derivations as Scheme objects, along with procedures to create and -otherwise manipulate derivations. The lowest-level primitive to create -a derivation is the @code{derivation} procedure: +@defvr {Scheme Variable} ruby-build-system +This variable is exported by @code{(guix build-system ruby)}. It +implements the RubyGems build procedure used by Ruby packages, which +involves running @code{gem build} followed by @code{gem install}. -@deffn {Scheme Procedure} derivation @var{store} @var{name} @var{builder} @ - @var{args} [#:outputs '("out")] [#:hash #f] [#:hash-algo #f] @ - [#:recursive? #f] [#:inputs '()] [#:env-vars '()] @ - [#:system (%current-system)] [#:references-graphs #f] @ - [#:allowed-references #f] [#:disallowed-references #f] @ - [#:leaked-env-vars #f] [#:local-build? #f] @ - [#:substitutable? #t] [#:properties '()] -Build a derivation with the given arguments, and return the resulting -@code{} object. +The @code{source} field of a package that uses this build system +typically references a gem archive, since this is the format that Ruby +developers use when releasing their software. The build system unpacks +the gem archive, potentially patches the source, runs the test suite, +repackages the gem, and installs it. Additionally, directories and +tarballs may be referenced to allow building unreleased gems from Git or +a traditional source release tarball. + +Which Ruby package is used can be specified with the @code{#:ruby} +parameter. A list of additional flags to be passed to the @command{gem} +command can be specified with the @code{#:gem-flags} parameter. +@end defvr + +@defvr {Scheme Variable} waf-build-system +This variable is exported by @code{(guix build-system waf)}. It +implements a build procedure around the @code{waf} script. The common +phases---@code{configure}, @code{build}, and @code{install}---are +implemented by passing their names as arguments to the @code{waf} +script. + +The @code{waf} script is executed by the Python interpreter. Which +Python package is used to run the script can be specified with the +@code{#:python} parameter. +@end defvr -When @var{hash} and @var{hash-algo} are given, a -@dfn{fixed-output derivation} is created---i.e., one whose result is -known in advance, such as a file download. If, in addition, -@var{recursive?} is true, then that fixed output may be an executable -file or a directory and @var{hash} must be the hash of an archive -containing this output. +@defvr {Scheme Variable} scons-build-system +This variable is exported by @code{(guix build-system scons)}. It +implements the build procedure used by the SCons software construction +tool. This build system runs @code{scons} to build the package, +@code{scons test} to run tests, and then @code{scons install} to install +the package. -When @var{references-graphs} is true, it must be a list of file -name/store path pairs. In that case, the reference graph of each store -path is exported in the build environment in the corresponding file, in -a simple text format. +Additional flags to be passed to @code{scons} can be specified with the +@code{#:scons-flags} parameter. The version of Python used to run SCons +can be specified by selecting the appropriate SCons package with the +@code{#:scons} parameter. +@end defvr -When @var{allowed-references} is true, it must be a list of store items -or outputs that the derivation's output may refer to. Likewise, -@var{disallowed-references}, if true, must be a list of things the -outputs may @emph{not} refer to. +@defvr {Scheme Variable} haskell-build-system +This variable is exported by @code{(guix build-system haskell)}. It +implements the Cabal build procedure used by Haskell packages, which +involves running @code{runhaskell Setup.hs configure +--prefix=/gnu/store/@dots{}} and @code{runhaskell Setup.hs build}. +Instead of installing the package by running @code{runhaskell Setup.hs +install}, to avoid trying to register libraries in the read-only +compiler store directory, the build system uses @code{runhaskell +Setup.hs copy}, followed by @code{runhaskell Setup.hs register}. In +addition, the build system generates the package documentation by +running @code{runhaskell Setup.hs haddock}, unless @code{#:haddock? #f} +is passed. Optional Haddock parameters can be passed with the help of +the @code{#:haddock-flags} parameter. If the file @code{Setup.hs} is +not found, the build system looks for @code{Setup.lhs} instead. -When @var{leaked-env-vars} is true, it must be a list of strings -denoting environment variables that are allowed to ``leak'' from the -daemon's environment to the build environment. This is only applicable -to fixed-output derivations---i.e., when @var{hash} is true. The main -use is to allow variables such as @code{http_proxy} to be passed to -derivations that download files. +Which Haskell compiler is used can be specified with the @code{#:haskell} +parameter which defaults to @code{ghc}. +@end defvr -When @var{local-build?} is true, declare that the derivation is not a -good candidate for offloading and should rather be built locally -(@pxref{Daemon Offload Setup}). This is the case for small derivations -where the costs of data transfers would outweigh the benefits. +@defvr {Scheme Variable} dub-build-system +This variable is exported by @code{(guix build-system dub)}. It +implements the Dub build procedure used by D packages, which +involves running @code{dub build} and @code{dub run}. +Installation is done by copying the files manually. -When @var{substitutable?} is false, declare that substitutes of the -derivation's output should not be used (@pxref{Substitutes}). This is -useful, for instance, when building packages that capture details of the -host CPU instruction set. +Which D compiler is used can be specified with the @code{#:ldc} +parameter which defaults to @code{ldc}. +@end defvr -@var{properties} must be an association list describing ``properties'' of the -derivation. It is kept as-is, uninterpreted, in the derivation. -@end deffn +@defvr {Scheme Variable} emacs-build-system +This variable is exported by @code{(guix build-system emacs)}. It +implements an installation procedure similar to the packaging system +of Emacs itself (@pxref{Packages,,, emacs, The GNU Emacs Manual}). -@noindent -Here's an example with a shell script as its builder, assuming -@var{store} is an open connection to the daemon, and @var{bash} points -to a Bash executable in the store: +It first creates the @code{@var{package}-autoloads.el} file, then it +byte compiles all Emacs Lisp files. Differently from the Emacs +packaging system, the Info documentation files are moved to the standard +documentation directory and the @file{dir} file is deleted. Each +package is installed in its own directory under +@file{share/emacs/site-lisp/guix.d}. +@end defvr -@lisp -(use-modules (guix utils) - (guix store) - (guix derivations)) +@defvr {Scheme Variable} font-build-system +This variable is exported by @code{(guix build-system font)}. It +implements an installation procedure for font packages where upstream +provides pre-compiled TrueType, OpenType, etc.@: font files that merely +need to be copied into place. It copies font files to standard +locations in the output directory. +@end defvr -(let ((builder ; add the Bash script to the store - (add-text-to-store store "my-builder.sh" - "echo hello world > $out\n" '()))) - (derivation store "foo" - bash `("-e" ,builder) - #:inputs `((,bash) (,builder)) - #:env-vars '(("HOME" . "/homeless")))) -@result{} # /gnu/store/@dots{}-foo> -@end lisp +@defvr {Scheme Variable} meson-build-system +This variable is exported by @code{(guix build-system meson)}. It +implements the build procedure for packages that use +@url{http://mesonbuild.com, Meson} as their build system. -As can be guessed, this primitive is cumbersome to use directly. A -better approach is to write build scripts in Scheme, of course! The -best course of action for that is to write the build code as a -``G-expression'', and to pass it to @code{gexp->derivation}. For more -information, @pxref{G-Expressions}. +It adds both Meson and @uref{https://ninja-build.org/, Ninja} to the set +of inputs, and they can be changed with the parameters @code{#:meson} +and @code{#:ninja} if needed. The default Meson is +@code{meson-for-build}, which is special because it doesn't clear the +@code{RUNPATH} of binaries and libraries when they are installed. -Once upon a time, @code{gexp->derivation} did not exist and constructing -derivations with build code written in Scheme was achieved with -@code{build-expression->derivation}, documented below. This procedure -is now deprecated in favor of the much nicer @code{gexp->derivation}. +This build system is an extension of @var{gnu-build-system}, but with the +following phases changed to some specific for Meson: -@deffn {Scheme Procedure} build-expression->derivation @var{store} @ - @var{name} @var{exp} @ - [#:system (%current-system)] [#:inputs '()] @ - [#:outputs '("out")] [#:hash #f] [#:hash-algo #f] @ - [#:recursive? #f] [#:env-vars '()] [#:modules '()] @ - [#:references-graphs #f] [#:allowed-references #f] @ - [#:disallowed-references #f] @ - [#:local-build? #f] [#:substitutable? #t] [#:guile-for-build #f] -Return a derivation that executes Scheme expression @var{exp} as a -builder for derivation @var{name}. @var{inputs} must be a list of -@code{(name drv-path sub-drv)} tuples; when @var{sub-drv} is omitted, -@code{"out"} is assumed. @var{modules} is a list of names of Guile -modules from the current search path to be copied in the store, -compiled, and made available in the load path during the execution of -@var{exp}---e.g., @code{((guix build utils) (guix build -gnu-build-system))}. +@table @code -@var{exp} is evaluated in an environment where @code{%outputs} is bound -to a list of output/path pairs, and where @code{%build-inputs} is bound -to a list of string/output-path pairs made from @var{inputs}. -Optionally, @var{env-vars} is a list of string pairs specifying the name -and value of environment variables visible to the builder. The builder -terminates by passing the result of @var{exp} to @code{exit}; thus, when -@var{exp} returns @code{#f}, the build is considered to have failed. +@item configure +The phase runs @code{meson} with the flags specified in +@code{#:configure-flags}. The flag @code{--build-type} is always set to +@code{plain} unless something else is specified in @code{#:build-type}. -@var{exp} is built using @var{guile-for-build} (a derivation). When -@var{guile-for-build} is omitted or is @code{#f}, the value of the -@code{%guile-for-build} fluid is used instead. +@item build +The phase runs @code{ninja} to build the package in parallel by default, but +this can be changed with @code{#:parallel-build?}. -See the @code{derivation} procedure for the meaning of -@var{references-graphs}, @var{allowed-references}, -@var{disallowed-references}, @var{local-build?}, and -@var{substitutable?}. -@end deffn +@item check +The phase runs @code{ninja} with the target specified in @code{#:test-target}, +which is @code{"test"} by default. -@noindent -Here's an example of a single-output derivation that creates a directory -containing one file: +@item install +The phase runs @code{ninja install} and can not be changed. +@end table -@lisp -(let ((builder '(let ((out (assoc-ref %outputs "out"))) - (mkdir out) ; create /gnu/store/@dots{}-goo - (call-with-output-file (string-append out "/test") - (lambda (p) - (display '(hello guix) p)))))) - (build-expression->derivation store "goo" builder)) +Apart from that, the build system also adds the following phases: -@result{} # @dots{}> -@end lisp +@table @code +@item fix-runpath +This phase ensures that all binaries can find the libraries they need. +It searches for required libraries in subdirectories of the package being +built, and adds those to @code{RUNPATH} where needed. It also removes +references to libraries left over from the build phase by +@code{meson-for-build}, such as test dependencies, that aren't actually +required for the program to run. -@node The Store Monad -@section The Store Monad +@item glib-or-gtk-wrap +This phase is the phase provided by @code{glib-or-gtk-build-system}, and it +is not enabled by default. It can be enabled with @code{#:glib-or-gtk?}. -@cindex monad +@item glib-or-gtk-compile-schemas +This phase is the phase provided by @code{glib-or-gtk-build-system}, and it +is not enabled by default. It can be enabled with @code{#:glib-or-gtk?}. +@end table +@end defvr -The procedures that operate on the store described in the previous -sections all take an open connection to the build daemon as their first -argument. Although the underlying model is functional, they either have -side effects or depend on the current state of the store. +Lastly, for packages that do not need anything as sophisticated, a +``trivial'' build system is provided. It is trivial in the sense that +it provides basically no support: it does not pull any implicit inputs, +and does not have a notion of build phases. -The former is inconvenient: the connection to the build daemon has to be -carried around in all those functions, making it impossible to compose -functions that do not take that parameter with functions that do. The -latter can be problematic: since store operations have side effects -and/or depend on external state, they have to be properly sequenced. +@defvr {Scheme Variable} trivial-build-system +This variable is exported by @code{(guix build-system trivial)}. -@cindex monadic values -@cindex monadic functions -This is where the @code{(guix monads)} module comes in. This module -provides a framework for working with @dfn{monads}, and a particularly -useful monad for our uses, the @dfn{store monad}. Monads are a -construct that allows two things: associating ``context'' with values -(in our case, the context is the store), and building sequences of -computations (here computations include accesses to the store). Values -in a monad---values that carry this additional context---are called -@dfn{monadic values}; procedures that return such values are called -@dfn{monadic procedures}. +This build system requires a @code{#:builder} argument. This argument +must be a Scheme expression that builds the package output(s)---as +with @code{build-expression->derivation} (@pxref{Derivations, +@code{build-expression->derivation}}). +@end defvr -Consider this ``normal'' procedure: +@node The Store +@section The Store -@example -(define (sh-symlink store) - ;; Return a derivation that symlinks the 'bash' executable. - (let* ((drv (package-derivation store bash)) - (out (derivation->output-path drv)) - (sh (string-append out "/bin/bash"))) - (build-expression->derivation store "sh" - `(symlink ,sh %output)))) -@end example +@cindex store +@cindex store items +@cindex store paths + +Conceptually, the @dfn{store} is the place where derivations that have +been built successfully are stored---by default, @file{/gnu/store}. +Sub-directories in the store are referred to as @dfn{store items} or +sometimes @dfn{store paths}. The store has an associated database that +contains information such as the store paths referred to by each store +path, and the list of @emph{valid} store items---results of successful +builds. This database resides in @file{@var{localstatedir}/guix/db}, +where @var{localstatedir} is the state directory specified @i{via} +@option{--localstatedir} at configure time, usually @file{/var}. -Using @code{(guix monads)} and @code{(guix gexp)}, it may be rewritten -as a monadic function: +The store is @emph{always} accessed by the daemon on behalf of its clients +(@pxref{Invoking guix-daemon}). To manipulate the store, clients +connect to the daemon over a Unix-domain socket, send requests to it, +and read the result---these are remote procedure calls, or RPCs. -@example -(define (sh-symlink) - ;; Same, but return a monadic value. - (mlet %store-monad ((drv (package->derivation bash))) - (gexp->derivation "sh" - #~(symlink (string-append #$drv "/bin/bash") - #$output)))) -@end example +@quotation Note +Users must @emph{never} modify files under @file{/gnu/store} directly. +This would lead to inconsistencies and break the immutability +assumptions of Guix's functional model (@pxref{Introduction}). -There are several things to note in the second version: the @code{store} -parameter is now implicit and is ``threaded'' in the calls to the -@code{package->derivation} and @code{gexp->derivation} monadic -procedures, and the monadic value returned by @code{package->derivation} -is @dfn{bound} using @code{mlet} instead of plain @code{let}. +@xref{Invoking guix gc, @command{guix gc --verify}}, for information on +how to check the integrity of the store and attempt recovery from +accidental modifications. +@end quotation -As it turns out, the call to @code{package->derivation} can even be -omitted since it will take place implicitly, as we will see later -(@pxref{G-Expressions}): +The @code{(guix store)} module provides procedures to connect to the +daemon, and to perform RPCs. These are described below. By default, +@code{open-connection}, and thus all the @command{guix} commands, +connect to the local daemon or to the URI specified by the +@code{GUIX_DAEMON_SOCKET} environment variable. -@example -(define (sh-symlink) - (gexp->derivation "sh" - #~(symlink (string-append #$bash "/bin/bash") - #$output))) -@end example +@defvr {Environment Variable} GUIX_DAEMON_SOCKET +When set, the value of this variable should be a file name or a URI +designating the daemon endpoint. When it is a file name, it denotes a +Unix-domain socket to connect to. In addition to file names, the +supported URI schemes are: -@c See -@c -@c for the funny quote. -Calling the monadic @code{sh-symlink} has no effect. As someone once -said, ``you exit a monad like you exit a building on fire: by running''. -So, to exit the monad and get the desired effect, one must use -@code{run-with-store}: +@table @code +@item file +@itemx unix +These are for Unix-domain sockets. +@code{file:///var/guix/daemon-socket/socket} is equivalent to +@file{/var/guix/daemon-socket/socket}. + +@item guix +@cindex daemon, remote access +@cindex remote access to the daemon +@cindex daemon, cluster setup +@cindex clusters, daemon setup +These URIs denote connections over TCP/IP, without encryption nor +authentication of the remote host. The URI must specify the host name +and optionally a port number (by default port 44146 is used): @example -(run-with-store (open-connection) (sh-symlink)) -@result{} /gnu/store/...-sh-symlink +guix://master.guix.example.org:1234 @end example -Note that the @code{(guix monad-repl)} module extends the Guile REPL with -new ``meta-commands'' to make it easier to deal with monadic procedures: -@code{run-in-store}, and @code{enter-store-monad}. The former is used -to ``run'' a single monadic value through the store: +This setup is suitable on local networks, such as clusters, where only +trusted nodes may connect to the build daemon at +@code{master.guix.example.org}. -@example -scheme@@(guile-user)> ,run-in-store (package->derivation hello) -$1 = # @dots{}> -@end example +The @code{--listen} option of @command{guix-daemon} can be used to +instruct it to listen for TCP connections (@pxref{Invoking guix-daemon, +@code{--listen}}). -The latter enters a recursive REPL, where all the return values are -automatically run through the store: +@item ssh +@cindex SSH access to build daemons +These URIs allow you to connect to a remote daemon over +SSH@footnote{This feature requires Guile-SSH (@pxref{Requirements}).}. +A typical URL might look like this: @example -scheme@@(guile-user)> ,enter-store-monad -store-monad@@(guile-user) [1]> (package->derivation hello) -$2 = # @dots{}> -store-monad@@(guile-user) [1]> (text-file "foo" "Hello!") -$3 = "/gnu/store/@dots{}-foo" -store-monad@@(guile-user) [1]> ,q -scheme@@(guile-user)> +ssh://charlie@@guix.example.org:22 @end example -@noindent -Note that non-monadic values cannot be returned in the -@code{store-monad} REPL. - -The main syntactic forms to deal with monads in general are provided by -the @code{(guix monads)} module and are described below. +As for @command{guix copy}, the usual OpenSSH client configuration files +are honored (@pxref{Invoking guix copy}). +@end table -@deffn {Scheme Syntax} with-monad @var{monad} @var{body} ... -Evaluate any @code{>>=} or @code{return} forms in @var{body} as being -in @var{monad}. -@end deffn +Additional URI schemes may be supported in the future. -@deffn {Scheme Syntax} return @var{val} -Return a monadic value that encapsulates @var{val}. -@end deffn +@c XXX: Remove this note when the protocol incurs fewer round trips +@c and when (guix derivations) no longer relies on file system access. +@quotation Note +The ability to connect to remote build daemons is considered +experimental as of @value{VERSION}. Please get in touch with us to +share any problems or suggestions you may have (@pxref{Contributing}). +@end quotation +@end defvr -@deffn {Scheme Syntax} >>= @var{mval} @var{mproc} ... -@dfn{Bind} monadic value @var{mval}, passing its ``contents'' to monadic -procedures @var{mproc}@dots{}@footnote{This operation is commonly -referred to as ``bind'', but that name denotes an unrelated procedure in -Guile. Thus we use this somewhat cryptic symbol inherited from the -Haskell language.}. There can be one @var{mproc} or several of them, as -in this example: +@deffn {Scheme Procedure} open-connection [@var{uri}] [#:reserve-space? #t] +Connect to the daemon over the Unix-domain socket at @var{uri} (a string). When +@var{reserve-space?} is true, instruct it to reserve a little bit of +extra space on the file system so that the garbage collector can still +operate should the disk become full. Return a server object. -@example -(run-with-state - (with-monad %state-monad - (>>= (return 1) - (lambda (x) (return (+ 1 x))) - (lambda (x) (return (* 2 x))))) - 'some-state) +@var{file} defaults to @var{%default-socket-path}, which is the normal +location given the options that were passed to @command{configure}. +@end deffn -@result{} 4 -@result{} some-state -@end example +@deffn {Scheme Procedure} close-connection @var{server} +Close the connection to @var{server}. @end deffn -@deffn {Scheme Syntax} mlet @var{monad} ((@var{var} @var{mval}) ...) @ - @var{body} ... -@deffnx {Scheme Syntax} mlet* @var{monad} ((@var{var} @var{mval}) ...) @ - @var{body} ... -Bind the variables @var{var} to the monadic values @var{mval} in -@var{body}, which is a sequence of expressions. As with the bind -operator, this can be thought of as ``unpacking'' the raw, non-monadic -value ``contained'' in @var{mval} and making @var{var} refer to that -raw, non-monadic value within the scope of the @var{body}. The form -(@var{var} -> @var{val}) binds @var{var} to the ``normal'' value -@var{val}, as per @code{let}. The binding operations occur in sequence -from left to right. The last expression of @var{body} must be a monadic -expression, and its result will become the result of the @code{mlet} or -@code{mlet*} when run in the @var{monad}. +@defvr {Scheme Variable} current-build-output-port +This variable is bound to a SRFI-39 parameter, which refers to the port +where build and error logs sent by the daemon should be written. +@end defvr -@code{mlet*} is to @code{mlet} what @code{let*} is to @code{let} -(@pxref{Local Bindings,,, guile, GNU Guile Reference Manual}). -@end deffn +Procedures that make RPCs all take a server object as their first +argument. -@deffn {Scheme System} mbegin @var{monad} @var{mexp} ... -Bind @var{mexp} and the following monadic expressions in sequence, -returning the result of the last expression. Every expression in the -sequence must be a monadic expression. +@deffn {Scheme Procedure} valid-path? @var{server} @var{path} +@cindex invalid store items +Return @code{#t} when @var{path} designates a valid store item and +@code{#f} otherwise (an invalid item may exist on disk but still be +invalid, for instance because it is the result of an aborted or failed +build.) -This is akin to @code{mlet}, except that the return values of the -monadic expressions are ignored. In that sense, it is analogous to -@code{begin}, but applied to monadic expressions. +A @code{&store-protocol-error} condition is raised if @var{path} is not +prefixed by the store directory (@file{/gnu/store}). @end deffn -@deffn {Scheme System} mwhen @var{condition} @var{mexp0} @var{mexp*} ... -When @var{condition} is true, evaluate the sequence of monadic -expressions @var{mexp0}..@var{mexp*} as in an @code{mbegin}. When -@var{condition} is false, return @code{*unspecified*} in the current -monad. Every expression in the sequence must be a monadic expression. +@deffn {Scheme Procedure} add-text-to-store @var{server} @var{name} @var{text} [@var{references}] +Add @var{text} under file @var{name} in the store, and return its store +path. @var{references} is the list of store paths referred to by the +resulting store path. @end deffn -@deffn {Scheme System} munless @var{condition} @var{mexp0} @var{mexp*} ... -When @var{condition} is false, evaluate the sequence of monadic -expressions @var{mexp0}..@var{mexp*} as in an @code{mbegin}. When -@var{condition} is true, return @code{*unspecified*} in the current -monad. Every expression in the sequence must be a monadic expression. +@deffn {Scheme Procedure} build-derivations @var{server} @var{derivations} +Build @var{derivations} (a list of @code{} objects or +derivation paths), and return when the worker is done building them. +Return @code{#t} on success. @end deffn -@cindex state monad -The @code{(guix monads)} module provides the @dfn{state monad}, which -allows an additional value---the state---to be @emph{threaded} through -monadic procedure calls. - -@defvr {Scheme Variable} %state-monad -The state monad. Procedures in the state monad can access and change -the state that is threaded. - -Consider the example below. The @code{square} procedure returns a value -in the state monad. It returns the square of its argument, but also -increments the current state value: - -@example -(define (square x) - (mlet %state-monad ((count (current-state))) - (mbegin %state-monad - (set-current-state (+ 1 count)) - (return (* x x))))) +Note that the @code{(guix monads)} module provides a monad as well as +monadic versions of the above procedures, with the goal of making it +more convenient to work with code that accesses the store (@pxref{The +Store Monad}). -(run-with-state (sequence %state-monad (map square (iota 3))) 0) -@result{} (0 1 4) -@result{} 3 -@end example +@c FIXME +@i{This section is currently incomplete.} -When ``run'' through @var{%state-monad}, we obtain that additional state -value, which is the number of @code{square} calls. -@end defvr +@node Derivations +@section Derivations -@deffn {Monadic Procedure} current-state -Return the current state as a monadic value. -@end deffn +@cindex derivations +Low-level build actions and the environment in which they are performed +are represented by @dfn{derivations}. A derivation contains the +following pieces of information: -@deffn {Monadic Procedure} set-current-state @var{value} -Set the current state to @var{value} and return the previous state as a -monadic value. -@end deffn +@itemize +@item +The outputs of the derivation---derivations produce at least one file or +directory in the store, but may produce more. -@deffn {Monadic Procedure} state-push @var{value} -Push @var{value} to the current state, which is assumed to be a list, -and return the previous state as a monadic value. -@end deffn +@item +The inputs of the derivations, which may be other derivations or plain +files in the store (patches, build scripts, etc.) -@deffn {Monadic Procedure} state-pop -Pop a value from the current state and return it as a monadic value. -The state is assumed to be a list. -@end deffn +@item +The system type targeted by the derivation---e.g., @code{x86_64-linux}. -@deffn {Scheme Procedure} run-with-state @var{mval} [@var{state}] -Run monadic value @var{mval} starting with @var{state} as the initial -state. Return two values: the resulting value, and the resulting state. -@end deffn +@item +The file name of a build script in the store, along with the arguments +to be passed. -The main interface to the store monad, provided by the @code{(guix -store)} module, is as follows. +@item +A list of environment variables to be defined. -@defvr {Scheme Variable} %store-monad -The store monad---an alias for @var{%state-monad}. +@end itemize -Values in the store monad encapsulate accesses to the store. When its -effect is needed, a value of the store monad must be ``evaluated'' by -passing it to the @code{run-with-store} procedure (see below.) -@end defvr +@cindex derivation path +Derivations allow clients of the daemon to communicate build actions to +the store. They exist in two forms: as an in-memory representation, +both on the client- and daemon-side, and as files in the store whose +name end in @code{.drv}---these files are referred to as @dfn{derivation +paths}. Derivations paths can be passed to the @code{build-derivations} +procedure to perform the build actions they prescribe (@pxref{The +Store}). -@deffn {Scheme Procedure} run-with-store @var{store} @var{mval} [#:guile-for-build] [#:system (%current-system)] -Run @var{mval}, a monadic value in the store monad, in @var{store}, an -open store connection. -@end deffn +@cindex fixed-output derivations +Operations such as file downloads and version-control checkouts for +which the expected content hash is known in advance are modeled as +@dfn{fixed-output derivations}. Unlike regular derivations, the outputs +of a fixed-output derivation are independent of its inputs---e.g., a +source code download produces the same result regardless of the download +method and tools being used. -@deffn {Monadic Procedure} text-file @var{name} @var{text} [@var{references}] -Return as a monadic value the absolute file name in the store of the file -containing @var{text}, a string. @var{references} is a list of store items that the -resulting text file refers to; it defaults to the empty list. -@end deffn +The @code{(guix derivations)} module provides a representation of +derivations as Scheme objects, along with procedures to create and +otherwise manipulate derivations. The lowest-level primitive to create +a derivation is the @code{derivation} procedure: -@deffn {Monadic Procedure} binary-file @var{name} @var{data} [@var{references}] -Return as a monadic value the absolute file name in the store of the file -containing @var{data}, a bytevector. @var{references} is a list of store -items that the resulting binary file refers to; it defaults to the empty list. -@end deffn +@deffn {Scheme Procedure} derivation @var{store} @var{name} @var{builder} @ + @var{args} [#:outputs '("out")] [#:hash #f] [#:hash-algo #f] @ + [#:recursive? #f] [#:inputs '()] [#:env-vars '()] @ + [#:system (%current-system)] [#:references-graphs #f] @ + [#:allowed-references #f] [#:disallowed-references #f] @ + [#:leaked-env-vars #f] [#:local-build? #f] @ + [#:substitutable? #t] [#:properties '()] +Build a derivation with the given arguments, and return the resulting +@code{} object. -@deffn {Monadic Procedure} interned-file @var{file} [@var{name}] @ - [#:recursive? #t] [#:select? (const #t)] -Return the name of @var{file} once interned in the store. Use -@var{name} as its store name, or the basename of @var{file} if -@var{name} is omitted. +When @var{hash} and @var{hash-algo} are given, a +@dfn{fixed-output derivation} is created---i.e., one whose result is +known in advance, such as a file download. If, in addition, +@var{recursive?} is true, then that fixed output may be an executable +file or a directory and @var{hash} must be the hash of an archive +containing this output. -When @var{recursive?} is true, the contents of @var{file} are added -recursively; if @var{file} designates a flat file and @var{recursive?} -is true, its contents are added, and its permission bits are kept. +When @var{references-graphs} is true, it must be a list of file +name/store path pairs. In that case, the reference graph of each store +path is exported in the build environment in the corresponding file, in +a simple text format. -When @var{recursive?} is true, call @code{(@var{select?} @var{file} -@var{stat})} for each directory entry, where @var{file} is the entry's -absolute file name and @var{stat} is the result of @code{lstat}; exclude -entries for which @var{select?} does not return true. +When @var{allowed-references} is true, it must be a list of store items +or outputs that the derivation's output may refer to. Likewise, +@var{disallowed-references}, if true, must be a list of things the +outputs may @emph{not} refer to. -The example below adds a file to the store, under two different names: +When @var{leaked-env-vars} is true, it must be a list of strings +denoting environment variables that are allowed to ``leak'' from the +daemon's environment to the build environment. This is only applicable +to fixed-output derivations---i.e., when @var{hash} is true. The main +use is to allow variables such as @code{http_proxy} to be passed to +derivations that download files. -@example -(run-with-store (open-connection) - (mlet %store-monad ((a (interned-file "README")) - (b (interned-file "README" "LEGU-MIN"))) - (return (list a b)))) +When @var{local-build?} is true, declare that the derivation is not a +good candidate for offloading and should rather be built locally +(@pxref{Daemon Offload Setup}). This is the case for small derivations +where the costs of data transfers would outweigh the benefits. -@result{} ("/gnu/store/rwm@dots{}-README" "/gnu/store/44i@dots{}-LEGU-MIN") -@end example +When @var{substitutable?} is false, declare that substitutes of the +derivation's output should not be used (@pxref{Substitutes}). This is +useful, for instance, when building packages that capture details of the +host CPU instruction set. +@var{properties} must be an association list describing ``properties'' of the +derivation. It is kept as-is, uninterpreted, in the derivation. @end deffn -The @code{(guix packages)} module exports the following package-related -monadic procedures: +@noindent +Here's an example with a shell script as its builder, assuming +@var{store} is an open connection to the daemon, and @var{bash} points +to a Bash executable in the store: -@deffn {Monadic Procedure} package-file @var{package} [@var{file}] @ - [#:system (%current-system)] [#:target #f] @ - [#:output "out"] -Return as a monadic -value in the absolute file name of @var{file} within the @var{output} -directory of @var{package}. When @var{file} is omitted, return the name -of the @var{output} directory of @var{package}. When @var{target} is -true, use it as a cross-compilation target triplet. -@end deffn +@lisp +(use-modules (guix utils) + (guix store) + (guix derivations)) -@deffn {Monadic Procedure} package->derivation @var{package} [@var{system}] -@deffnx {Monadic Procedure} package->cross-derivation @var{package} @ - @var{target} [@var{system}] -Monadic version of @code{package-derivation} and -@code{package-cross-derivation} (@pxref{Defining Packages}). -@end deffn +(let ((builder ; add the Bash script to the store + (add-text-to-store store "my-builder.sh" + "echo hello world > $out\n" '()))) + (derivation store "foo" + bash `("-e" ,builder) + #:inputs `((,bash) (,builder)) + #:env-vars '(("HOME" . "/homeless")))) +@result{} # /gnu/store/@dots{}-foo> +@end lisp +As can be guessed, this primitive is cumbersome to use directly. A +better approach is to write build scripts in Scheme, of course! The +best course of action for that is to write the build code as a +``G-expression'', and to pass it to @code{gexp->derivation}. For more +information, @pxref{G-Expressions}. -@node G-Expressions -@section G-Expressions +Once upon a time, @code{gexp->derivation} did not exist and constructing +derivations with build code written in Scheme was achieved with +@code{build-expression->derivation}, documented below. This procedure +is now deprecated in favor of the much nicer @code{gexp->derivation}. -@cindex G-expression -@cindex build code quoting -So we have ``derivations'', which represent a sequence of build actions -to be performed to produce an item in the store (@pxref{Derivations}). -These build actions are performed when asking the daemon to actually -build the derivations; they are run by the daemon in a container -(@pxref{Invoking guix-daemon}). +@deffn {Scheme Procedure} build-expression->derivation @var{store} @ + @var{name} @var{exp} @ + [#:system (%current-system)] [#:inputs '()] @ + [#:outputs '("out")] [#:hash #f] [#:hash-algo #f] @ + [#:recursive? #f] [#:env-vars '()] [#:modules '()] @ + [#:references-graphs #f] [#:allowed-references #f] @ + [#:disallowed-references #f] @ + [#:local-build? #f] [#:substitutable? #t] [#:guile-for-build #f] +Return a derivation that executes Scheme expression @var{exp} as a +builder for derivation @var{name}. @var{inputs} must be a list of +@code{(name drv-path sub-drv)} tuples; when @var{sub-drv} is omitted, +@code{"out"} is assumed. @var{modules} is a list of names of Guile +modules from the current search path to be copied in the store, +compiled, and made available in the load path during the execution of +@var{exp}---e.g., @code{((guix build utils) (guix build +gnu-build-system))}. -@cindex strata of code -It should come as no surprise that we like to write these build actions -in Scheme. When we do that, we end up with two @dfn{strata} of Scheme -code@footnote{The term @dfn{stratum} in this context was coined by -Manuel Serrano et al.@: in the context of their work on Hop. Oleg -Kiselyov, who has written insightful -@url{http://okmij.org/ftp/meta-programming/#meta-scheme, essays and code -on this topic}, refers to this kind of code generation as -@dfn{staging}.}: the ``host code''---code that defines packages, talks -to the daemon, etc.---and the ``build code''---code that actually -performs build actions, such as making directories, invoking -@command{make}, etc. +@var{exp} is evaluated in an environment where @code{%outputs} is bound +to a list of output/path pairs, and where @code{%build-inputs} is bound +to a list of string/output-path pairs made from @var{inputs}. +Optionally, @var{env-vars} is a list of string pairs specifying the name +and value of environment variables visible to the builder. The builder +terminates by passing the result of @var{exp} to @code{exit}; thus, when +@var{exp} returns @code{#f}, the build is considered to have failed. -To describe a derivation and its build actions, one typically needs to -embed build code inside host code. It boils down to manipulating build -code as data, and the homoiconicity of Scheme---code has a direct -representation as data---comes in handy for that. But we need more than -the normal @code{quasiquote} mechanism in Scheme to construct build -expressions. +@var{exp} is built using @var{guile-for-build} (a derivation). When +@var{guile-for-build} is omitted or is @code{#f}, the value of the +@code{%guile-for-build} fluid is used instead. -The @code{(guix gexp)} module implements @dfn{G-expressions}, a form of -S-expressions adapted to build expressions. G-expressions, or -@dfn{gexps}, consist essentially of three syntactic forms: @code{gexp}, -@code{ungexp}, and @code{ungexp-splicing} (or simply: @code{#~}, -@code{#$}, and @code{#$@@}), which are comparable to -@code{quasiquote}, @code{unquote}, and @code{unquote-splicing}, -respectively (@pxref{Expression Syntax, @code{quasiquote},, guile, -GNU Guile Reference Manual}). However, there are major differences: +See the @code{derivation} procedure for the meaning of +@var{references-graphs}, @var{allowed-references}, +@var{disallowed-references}, @var{local-build?}, and +@var{substitutable?}. +@end deffn -@itemize -@item -Gexps are meant to be written to a file and run or manipulated by other -processes. +@noindent +Here's an example of a single-output derivation that creates a directory +containing one file: -@item -When a high-level object such as a package or derivation is unquoted -inside a gexp, the result is as if its output file name had been -introduced. +@lisp +(let ((builder '(let ((out (assoc-ref %outputs "out"))) + (mkdir out) ; create /gnu/store/@dots{}-goo + (call-with-output-file (string-append out "/test") + (lambda (p) + (display '(hello guix) p)))))) + (build-expression->derivation store "goo" builder)) -@item -Gexps carry information about the packages or derivations they refer to, -and these dependencies are automatically added as inputs to the build -processes that use them. -@end itemize +@result{} # @dots{}> +@end lisp -@cindex lowering, of high-level objects in gexps -This mechanism is not limited to package and derivation -objects: @dfn{compilers} able to ``lower'' other high-level objects to -derivations or files in the store can be defined, -such that these objects can also be inserted -into gexps. For example, a useful type of high-level objects that can be -inserted in a gexp is ``file-like objects'', which make it easy to -add files to the store and to refer to them in -derivations and such (see @code{local-file} and @code{plain-file} -below.) -To illustrate the idea, here is an example of a gexp: +@node The Store Monad +@section The Store Monad -@example -(define build-exp - #~(begin - (mkdir #$output) - (chdir #$output) - (symlink (string-append #$coreutils "/bin/ls") - "list-files"))) -@end example +@cindex monad -This gexp can be passed to @code{gexp->derivation}; we obtain a -derivation that builds a directory containing exactly one symlink to -@file{/gnu/store/@dots{}-coreutils-8.22/bin/ls}: +The procedures that operate on the store described in the previous +sections all take an open connection to the build daemon as their first +argument. Although the underlying model is functional, they either have +side effects or depend on the current state of the store. -@example -(gexp->derivation "the-thing" build-exp) -@end example +The former is inconvenient: the connection to the build daemon has to be +carried around in all those functions, making it impossible to compose +functions that do not take that parameter with functions that do. The +latter can be problematic: since store operations have side effects +and/or depend on external state, they have to be properly sequenced. -As one would expect, the @code{"/gnu/store/@dots{}-coreutils-8.22"} string is -substituted to the reference to the @var{coreutils} package in the -actual build code, and @var{coreutils} is automatically made an input to -the derivation. Likewise, @code{#$output} (equivalent to @code{(ungexp -output)}) is replaced by a string containing the directory name of the -output of the derivation. +@cindex monadic values +@cindex monadic functions +This is where the @code{(guix monads)} module comes in. This module +provides a framework for working with @dfn{monads}, and a particularly +useful monad for our uses, the @dfn{store monad}. Monads are a +construct that allows two things: associating ``context'' with values +(in our case, the context is the store), and building sequences of +computations (here computations include accesses to the store). Values +in a monad---values that carry this additional context---are called +@dfn{monadic values}; procedures that return such values are called +@dfn{monadic procedures}. -@cindex cross compilation -In a cross-compilation context, it is useful to distinguish between -references to the @emph{native} build of a package---that can run on the -host---versus references to cross builds of a package. To that end, the -@code{#+} plays the same role as @code{#$}, but is a reference to a -native package build: +Consider this ``normal'' procedure: @example -(gexp->derivation "vi" - #~(begin - (mkdir #$output) - (system* (string-append #+coreutils "/bin/ln") - "-s" - (string-append #$emacs "/bin/emacs") - (string-append #$output "/bin/vi"))) - #:target "mips64el-linux-gnu") +(define (sh-symlink store) + ;; Return a derivation that symlinks the 'bash' executable. + (let* ((drv (package-derivation store bash)) + (out (derivation->output-path drv)) + (sh (string-append out "/bin/bash"))) + (build-expression->derivation store "sh" + `(symlink ,sh %output)))) @end example -@noindent -In the example above, the native build of @var{coreutils} is used, so -that @command{ln} can actually run on the host; but then the -cross-compiled build of @var{emacs} is referenced. - -@cindex imported modules, for gexps -@findex with-imported-modules -Another gexp feature is @dfn{imported modules}: sometimes you want to be -able to use certain Guile modules from the ``host environment'' in the -gexp, so those modules should be imported in the ``build environment''. -The @code{with-imported-modules} form allows you to express that: +Using @code{(guix monads)} and @code{(guix gexp)}, it may be rewritten +as a monadic function: @example -(let ((build (with-imported-modules '((guix build utils)) - #~(begin - (use-modules (guix build utils)) - (mkdir-p (string-append #$output "/bin")))))) - (gexp->derivation "empty-dir" - #~(begin - #$build - (display "success!\n") - #t))) +(define (sh-symlink) + ;; Same, but return a monadic value. + (mlet %store-monad ((drv (package->derivation bash))) + (gexp->derivation "sh" + #~(symlink (string-append #$drv "/bin/bash") + #$output)))) @end example -@noindent -In this example, the @code{(guix build utils)} module is automatically -pulled into the isolated build environment of our gexp, such that -@code{(use-modules (guix build utils))} works as expected. +There are several things to note in the second version: the @code{store} +parameter is now implicit and is ``threaded'' in the calls to the +@code{package->derivation} and @code{gexp->derivation} monadic +procedures, and the monadic value returned by @code{package->derivation} +is @dfn{bound} using @code{mlet} instead of plain @code{let}. -@cindex module closure -@findex source-module-closure -Usually you want the @emph{closure} of the module to be imported---i.e., -the module itself and all the modules it depends on---rather than just -the module; failing to do that, attempts to use the module will fail -because of missing dependent modules. The @code{source-module-closure} -procedure computes the closure of a module by looking at its source file -headers, which comes in handy in this case: +As it turns out, the call to @code{package->derivation} can even be +omitted since it will take place implicitly, as we will see later +(@pxref{G-Expressions}): @example -(use-modules (guix modules)) ;for 'source-module-closure' - -(with-imported-modules (source-module-closure - '((guix build utils) - (gnu build vm))) - (gexp->derivation "something-with-vms" - #~(begin - (use-modules (guix build utils) - (gnu build vm)) - @dots{}))) +(define (sh-symlink) + (gexp->derivation "sh" + #~(symlink (string-append #$bash "/bin/bash") + #$output))) @end example -@cindex extensions, for gexps -@findex with-extensions -In the same vein, sometimes you want to import not just pure-Scheme -modules, but also ``extensions'' such as Guile bindings to C libraries -or other ``full-blown'' packages. Say you need the @code{guile-json} -package available on the build side, here's how you would do it: +@c See +@c +@c for the funny quote. +Calling the monadic @code{sh-symlink} has no effect. As someone once +said, ``you exit a monad like you exit a building on fire: by running''. +So, to exit the monad and get the desired effect, one must use +@code{run-with-store}: @example -(use-modules (gnu packages guile)) ;for 'guile-json' - -(with-extensions (list guile-json) - (gexp->derivation "something-with-json" - #~(begin - (use-modules (json)) - @dots{}))) +(run-with-store (open-connection) (sh-symlink)) +@result{} /gnu/store/...-sh-symlink @end example -The syntactic form to construct gexps is summarized below. - -@deffn {Scheme Syntax} #~@var{exp} -@deffnx {Scheme Syntax} (gexp @var{exp}) -Return a G-expression containing @var{exp}. @var{exp} may contain one -or more of the following forms: - -@table @code -@item #$@var{obj} -@itemx (ungexp @var{obj}) -Introduce a reference to @var{obj}. @var{obj} may have one of the -supported types, for example a package or a -derivation, in which case the @code{ungexp} form is replaced by its -output file name---e.g., @code{"/gnu/store/@dots{}-coreutils-8.22}. - -If @var{obj} is a list, it is traversed and references to supported -objects are substituted similarly. - -If @var{obj} is another gexp, its contents are inserted and its -dependencies are added to those of the containing gexp. - -If @var{obj} is another kind of object, it is inserted as is. - -@item #$@var{obj}:@var{output} -@itemx (ungexp @var{obj} @var{output}) -This is like the form above, but referring explicitly to the -@var{output} of @var{obj}---this is useful when @var{obj} produces -multiple outputs (@pxref{Packages with Multiple Outputs}). - -@item #+@var{obj} -@itemx #+@var{obj}:output -@itemx (ungexp-native @var{obj}) -@itemx (ungexp-native @var{obj} @var{output}) -Same as @code{ungexp}, but produces a reference to the @emph{native} -build of @var{obj} when used in a cross compilation context. +Note that the @code{(guix monad-repl)} module extends the Guile REPL with +new ``meta-commands'' to make it easier to deal with monadic procedures: +@code{run-in-store}, and @code{enter-store-monad}. The former is used +to ``run'' a single monadic value through the store: -@item #$output[:@var{output}] -@itemx (ungexp output [@var{output}]) -Insert a reference to derivation output @var{output}, or to the main -output when @var{output} is omitted. +@example +scheme@@(guile-user)> ,run-in-store (package->derivation hello) +$1 = # @dots{}> +@end example -This only makes sense for gexps passed to @code{gexp->derivation}. +The latter enters a recursive REPL, where all the return values are +automatically run through the store: -@item #$@@@var{lst} -@itemx (ungexp-splicing @var{lst}) -Like the above, but splices the contents of @var{lst} inside the -containing list. +@example +scheme@@(guile-user)> ,enter-store-monad +store-monad@@(guile-user) [1]> (package->derivation hello) +$2 = # @dots{}> +store-monad@@(guile-user) [1]> (text-file "foo" "Hello!") +$3 = "/gnu/store/@dots{}-foo" +store-monad@@(guile-user) [1]> ,q +scheme@@(guile-user)> +@end example -@item #+@@@var{lst} -@itemx (ungexp-native-splicing @var{lst}) -Like the above, but refers to native builds of the objects listed in -@var{lst}. +@noindent +Note that non-monadic values cannot be returned in the +@code{store-monad} REPL. -@end table +The main syntactic forms to deal with monads in general are provided by +the @code{(guix monads)} module and are described below. -G-expressions created by @code{gexp} or @code{#~} are run-time objects -of the @code{gexp?} type (see below.) +@deffn {Scheme Syntax} with-monad @var{monad} @var{body} ... +Evaluate any @code{>>=} or @code{return} forms in @var{body} as being +in @var{monad}. @end deffn -@deffn {Scheme Syntax} with-imported-modules @var{modules} @var{body}@dots{} -Mark the gexps defined in @var{body}@dots{} as requiring @var{modules} -in their execution environment. +@deffn {Scheme Syntax} return @var{val} +Return a monadic value that encapsulates @var{val}. +@end deffn -Each item in @var{modules} can be the name of a module, such as -@code{(guix build utils)}, or it can be a module name, followed by an -arrow, followed by a file-like object: +@deffn {Scheme Syntax} >>= @var{mval} @var{mproc} ... +@dfn{Bind} monadic value @var{mval}, passing its ``contents'' to monadic +procedures @var{mproc}@dots{}@footnote{This operation is commonly +referred to as ``bind'', but that name denotes an unrelated procedure in +Guile. Thus we use this somewhat cryptic symbol inherited from the +Haskell language.}. There can be one @var{mproc} or several of them, as +in this example: @example -`((guix build utils) - (guix gcrypt) - ((guix config) => ,(scheme-file "config.scm" - #~(define-module @dots{})))) +(run-with-state + (with-monad %state-monad + (>>= (return 1) + (lambda (x) (return (+ 1 x))) + (lambda (x) (return (* 2 x))))) + 'some-state) + +@result{} 4 +@result{} some-state @end example +@end deffn -@noindent -In the example above, the first two modules are taken from the search -path, and the last one is created from the given file-like object. +@deffn {Scheme Syntax} mlet @var{monad} ((@var{var} @var{mval}) ...) @ + @var{body} ... +@deffnx {Scheme Syntax} mlet* @var{monad} ((@var{var} @var{mval}) ...) @ + @var{body} ... +Bind the variables @var{var} to the monadic values @var{mval} in +@var{body}, which is a sequence of expressions. As with the bind +operator, this can be thought of as ``unpacking'' the raw, non-monadic +value ``contained'' in @var{mval} and making @var{var} refer to that +raw, non-monadic value within the scope of the @var{body}. The form +(@var{var} -> @var{val}) binds @var{var} to the ``normal'' value +@var{val}, as per @code{let}. The binding operations occur in sequence +from left to right. The last expression of @var{body} must be a monadic +expression, and its result will become the result of the @code{mlet} or +@code{mlet*} when run in the @var{monad}. -This form has @emph{lexical} scope: it has an effect on the gexps -directly defined in @var{body}@dots{}, but not on those defined, say, in -procedures called from @var{body}@dots{}. +@code{mlet*} is to @code{mlet} what @code{let*} is to @code{let} +(@pxref{Local Bindings,,, guile, GNU Guile Reference Manual}). @end deffn -@deffn {Scheme Syntax} with-extensions @var{extensions} @var{body}@dots{} -Mark the gexps defined in @var{body}@dots{} as requiring -@var{extensions} in their build and execution environment. -@var{extensions} is typically a list of package objects such as those -defined in the @code{(gnu packages guile)} module. +@deffn {Scheme System} mbegin @var{monad} @var{mexp} ... +Bind @var{mexp} and the following monadic expressions in sequence, +returning the result of the last expression. Every expression in the +sequence must be a monadic expression. -Concretely, the packages listed in @var{extensions} are added to the -load path while compiling imported modules in @var{body}@dots{}; they -are also added to the load path of the gexp returned by -@var{body}@dots{}. +This is akin to @code{mlet}, except that the return values of the +monadic expressions are ignored. In that sense, it is analogous to +@code{begin}, but applied to monadic expressions. @end deffn -@deffn {Scheme Procedure} gexp? @var{obj} -Return @code{#t} if @var{obj} is a G-expression. +@deffn {Scheme System} mwhen @var{condition} @var{mexp0} @var{mexp*} ... +When @var{condition} is true, evaluate the sequence of monadic +expressions @var{mexp0}..@var{mexp*} as in an @code{mbegin}. When +@var{condition} is false, return @code{*unspecified*} in the current +monad. Every expression in the sequence must be a monadic expression. @end deffn -G-expressions are meant to be written to disk, either as code building -some derivation, or as plain files in the store. The monadic procedures -below allow you to do that (@pxref{The Store Monad}, for more -information about monads.) - -@deffn {Monadic Procedure} gexp->derivation @var{name} @var{exp} @ - [#:system (%current-system)] [#:target #f] [#:graft? #t] @ - [#:hash #f] [#:hash-algo #f] @ - [#:recursive? #f] [#:env-vars '()] [#:modules '()] @ - [#:module-path @var{%load-path}] @ - [#:effective-version "2.2"] @ - [#:references-graphs #f] [#:allowed-references #f] @ - [#:disallowed-references #f] @ - [#:leaked-env-vars #f] @ - [#:script-name (string-append @var{name} "-builder")] @ - [#:deprecation-warnings #f] @ - [#:local-build? #f] [#:substitutable? #t] @ - [#:properties '()] [#:guile-for-build #f] -Return a derivation @var{name} that runs @var{exp} (a gexp) with -@var{guile-for-build} (a derivation) on @var{system}; @var{exp} is -stored in a file called @var{script-name}. When @var{target} is true, -it is used as the cross-compilation target triplet for packages referred -to by @var{exp}. - -@var{modules} is deprecated in favor of @code{with-imported-modules}. -Its meaning is to -make @var{modules} available in the evaluation context of @var{exp}; -@var{modules} is a list of names of Guile modules searched in -@var{module-path} to be copied in the store, compiled, and made available in -the load path during the execution of @var{exp}---e.g., @code{((guix -build utils) (guix build gnu-build-system))}. +@deffn {Scheme System} munless @var{condition} @var{mexp0} @var{mexp*} ... +When @var{condition} is false, evaluate the sequence of monadic +expressions @var{mexp0}..@var{mexp*} as in an @code{mbegin}. When +@var{condition} is true, return @code{*unspecified*} in the current +monad. Every expression in the sequence must be a monadic expression. +@end deffn -@var{effective-version} determines the string to use when adding extensions of -@var{exp} (see @code{with-extensions}) to the search path---e.g., @code{"2.2"}. +@cindex state monad +The @code{(guix monads)} module provides the @dfn{state monad}, which +allows an additional value---the state---to be @emph{threaded} through +monadic procedure calls. -@var{graft?} determines whether packages referred to by @var{exp} should be grafted when -applicable. +@defvr {Scheme Variable} %state-monad +The state monad. Procedures in the state monad can access and change +the state that is threaded. -When @var{references-graphs} is true, it must be a list of tuples of one of the -following forms: +Consider the example below. The @code{square} procedure returns a value +in the state monad. It returns the square of its argument, but also +increments the current state value: @example -(@var{file-name} @var{package}) -(@var{file-name} @var{package} @var{output}) -(@var{file-name} @var{derivation}) -(@var{file-name} @var{derivation} @var{output}) -(@var{file-name} @var{store-item}) -@end example +(define (square x) + (mlet %state-monad ((count (current-state))) + (mbegin %state-monad + (set-current-state (+ 1 count)) + (return (* x x))))) -The right-hand-side of each element of @var{references-graphs} is automatically made -an input of the build process of @var{exp}. In the build environment, each -@var{file-name} contains the reference graph of the corresponding item, in a simple -text format. +(run-with-state (sequence %state-monad (map square (iota 3))) 0) +@result{} (0 1 4) +@result{} 3 +@end example -@var{allowed-references} must be either @code{#f} or a list of output names and packages. -In the latter case, the list denotes store items that the result is allowed to -refer to. Any reference to another store item will lead to a build error. -Similarly for @var{disallowed-references}, which can list items that must not be -referenced by the outputs. +When ``run'' through @var{%state-monad}, we obtain that additional state +value, which is the number of @code{square} calls. +@end defvr -@var{deprecation-warnings} determines whether to show deprecation warnings while -compiling modules. It can be @code{#f}, @code{#t}, or @code{'detailed}. +@deffn {Monadic Procedure} current-state +Return the current state as a monadic value. +@end deffn -The other arguments are as for @code{derivation} (@pxref{Derivations}). +@deffn {Monadic Procedure} set-current-state @var{value} +Set the current state to @var{value} and return the previous state as a +monadic value. @end deffn -@cindex file-like objects -The @code{local-file}, @code{plain-file}, @code{computed-file}, -@code{program-file}, and @code{scheme-file} procedures below return -@dfn{file-like objects}. That is, when unquoted in a G-expression, -these objects lead to a file in the store. Consider this G-expression: +@deffn {Monadic Procedure} state-push @var{value} +Push @var{value} to the current state, which is assumed to be a list, +and return the previous state as a monadic value. +@end deffn -@example -#~(system* #$(file-append glibc "/sbin/nscd") "-f" - #$(local-file "/tmp/my-nscd.conf")) -@end example +@deffn {Monadic Procedure} state-pop +Pop a value from the current state and return it as a monadic value. +The state is assumed to be a list. +@end deffn -The effect here is to ``intern'' @file{/tmp/my-nscd.conf} by copying it -to the store. Once expanded, for instance @i{via} -@code{gexp->derivation}, the G-expression refers to that copy under -@file{/gnu/store}; thus, modifying or removing the file in @file{/tmp} -does not have any effect on what the G-expression does. -@code{plain-file} can be used similarly; it differs in that the file -content is directly passed as a string. +@deffn {Scheme Procedure} run-with-state @var{mval} [@var{state}] +Run monadic value @var{mval} starting with @var{state} as the initial +state. Return two values: the resulting value, and the resulting state. +@end deffn -@deffn {Scheme Procedure} local-file @var{file} [@var{name}] @ - [#:recursive? #f] [#:select? (const #t)] -Return an object representing local file @var{file} to add to the store; this -object can be used in a gexp. If @var{file} is a relative file name, it is looked -up relative to the source file where this form appears. @var{file} will be added to -the store under @var{name}--by default the base name of @var{file}. +The main interface to the store monad, provided by the @code{(guix +store)} module, is as follows. -When @var{recursive?} is true, the contents of @var{file} are added recursively; if @var{file} -designates a flat file and @var{recursive?} is true, its contents are added, and its -permission bits are kept. +@defvr {Scheme Variable} %store-monad +The store monad---an alias for @var{%state-monad}. -When @var{recursive?} is true, call @code{(@var{select?} @var{file} -@var{stat})} for each directory entry, where @var{file} is the entry's -absolute file name and @var{stat} is the result of @code{lstat}; exclude -entries for which @var{select?} does not return true. +Values in the store monad encapsulate accesses to the store. When its +effect is needed, a value of the store monad must be ``evaluated'' by +passing it to the @code{run-with-store} procedure (see below.) +@end defvr -This is the declarative counterpart of the @code{interned-file} monadic -procedure (@pxref{The Store Monad, @code{interned-file}}). +@deffn {Scheme Procedure} run-with-store @var{store} @var{mval} [#:guile-for-build] [#:system (%current-system)] +Run @var{mval}, a monadic value in the store monad, in @var{store}, an +open store connection. @end deffn -@deffn {Scheme Procedure} plain-file @var{name} @var{content} -Return an object representing a text file called @var{name} with the given -@var{content} (a string or a bytevector) to be added to the store. - -This is the declarative counterpart of @code{text-file}. +@deffn {Monadic Procedure} text-file @var{name} @var{text} [@var{references}] +Return as a monadic value the absolute file name in the store of the file +containing @var{text}, a string. @var{references} is a list of store items that the +resulting text file refers to; it defaults to the empty list. @end deffn -@deffn {Scheme Procedure} computed-file @var{name} @var{gexp} @ - [#:options '(#:local-build? #t)] -Return an object representing the store item @var{name}, a file or -directory computed by @var{gexp}. @var{options} -is a list of additional arguments to pass to @code{gexp->derivation}. - -This is the declarative counterpart of @code{gexp->derivation}. +@deffn {Monadic Procedure} binary-file @var{name} @var{data} [@var{references}] +Return as a monadic value the absolute file name in the store of the file +containing @var{data}, a bytevector. @var{references} is a list of store +items that the resulting binary file refers to; it defaults to the empty list. @end deffn -@deffn {Monadic Procedure} gexp->script @var{name} @var{exp} @ - [#:guile (default-guile)] [#:module-path %load-path] -Return an executable script @var{name} that runs @var{exp} using -@var{guile}, with @var{exp}'s imported modules in its search path. -Look up @var{exp}'s modules in @var{module-path}. +@deffn {Monadic Procedure} interned-file @var{file} [@var{name}] @ + [#:recursive? #t] [#:select? (const #t)] +Return the name of @var{file} once interned in the store. Use +@var{name} as its store name, or the basename of @var{file} if +@var{name} is omitted. -The example below builds a script that simply invokes the @command{ls} -command: +When @var{recursive?} is true, the contents of @var{file} are added +recursively; if @var{file} designates a flat file and @var{recursive?} +is true, its contents are added, and its permission bits are kept. -@example -(use-modules (guix gexp) (gnu packages base)) +When @var{recursive?} is true, call @code{(@var{select?} @var{file} +@var{stat})} for each directory entry, where @var{file} is the entry's +absolute file name and @var{stat} is the result of @code{lstat}; exclude +entries for which @var{select?} does not return true. -(gexp->script "list-files" - #~(execl #$(file-append coreutils "/bin/ls") - "ls")) -@end example +The example below adds a file to the store, under two different names: -When ``running'' it through the store (@pxref{The Store Monad, -@code{run-with-store}}), we obtain a derivation that produces an -executable file @file{/gnu/store/@dots{}-list-files} along these lines: +@example +(run-with-store (open-connection) + (mlet %store-monad ((a (interned-file "README")) + (b (interned-file "README" "LEGU-MIN"))) + (return (list a b)))) -@example -#!/gnu/store/@dots{}-guile-2.0.11/bin/guile -ds -!# -(execl "/gnu/store/@dots{}-coreutils-8.22"/bin/ls" "ls") +@result{} ("/gnu/store/rwm@dots{}-README" "/gnu/store/44i@dots{}-LEGU-MIN") @end example -@end deffn - -@deffn {Scheme Procedure} program-file @var{name} @var{exp} @ - [#:guile #f] [#:module-path %load-path] -Return an object representing the executable store item @var{name} that -runs @var{gexp}. @var{guile} is the Guile package used to execute that -script. Imported modules of @var{gexp} are looked up in @var{module-path}. -This is the declarative counterpart of @code{gexp->script}. @end deffn -@deffn {Monadic Procedure} gexp->file @var{name} @var{exp} @ - [#:set-load-path? #t] [#:module-path %load-path] @ - [#:splice? #f] @ - [#:guile (default-guile)] -Return a derivation that builds a file @var{name} containing @var{exp}. -When @var{splice?} is true, @var{exp} is considered to be a list of -expressions that will be spliced in the resulting file. +The @code{(guix packages)} module exports the following package-related +monadic procedures: -When @var{set-load-path?} is true, emit code in the resulting file to -set @code{%load-path} and @code{%load-compiled-path} to honor -@var{exp}'s imported modules. Look up @var{exp}'s modules in -@var{module-path}. +@deffn {Monadic Procedure} package-file @var{package} [@var{file}] @ + [#:system (%current-system)] [#:target #f] @ + [#:output "out"] +Return as a monadic +value in the absolute file name of @var{file} within the @var{output} +directory of @var{package}. When @var{file} is omitted, return the name +of the @var{output} directory of @var{package}. When @var{target} is +true, use it as a cross-compilation target triplet. +@end deffn -The resulting file holds references to all the dependencies of @var{exp} -or a subset thereof. +@deffn {Monadic Procedure} package->derivation @var{package} [@var{system}] +@deffnx {Monadic Procedure} package->cross-derivation @var{package} @ + @var{target} [@var{system}] +Monadic version of @code{package-derivation} and +@code{package-cross-derivation} (@pxref{Defining Packages}). @end deffn -@deffn {Scheme Procedure} scheme-file @var{name} @var{exp} [#:splice? #f] -Return an object representing the Scheme file @var{name} that contains -@var{exp}. -This is the declarative counterpart of @code{gexp->file}. -@end deffn +@node G-Expressions +@section G-Expressions -@deffn {Monadic Procedure} text-file* @var{name} @var{text} @dots{} -Return as a monadic value a derivation that builds a text file -containing all of @var{text}. @var{text} may list, in addition to -strings, objects of any type that can be used in a gexp: packages, -derivations, local file objects, etc. The resulting store file holds -references to all these. +@cindex G-expression +@cindex build code quoting +So we have ``derivations'', which represent a sequence of build actions +to be performed to produce an item in the store (@pxref{Derivations}). +These build actions are performed when asking the daemon to actually +build the derivations; they are run by the daemon in a container +(@pxref{Invoking guix-daemon}). -This variant should be preferred over @code{text-file} anytime the file -to create will reference items from the store. This is typically the -case when building a configuration file that embeds store file names, -like this: +@cindex strata of code +It should come as no surprise that we like to write these build actions +in Scheme. When we do that, we end up with two @dfn{strata} of Scheme +code@footnote{The term @dfn{stratum} in this context was coined by +Manuel Serrano et al.@: in the context of their work on Hop. Oleg +Kiselyov, who has written insightful +@url{http://okmij.org/ftp/meta-programming/#meta-scheme, essays and code +on this topic}, refers to this kind of code generation as +@dfn{staging}.}: the ``host code''---code that defines packages, talks +to the daemon, etc.---and the ``build code''---code that actually +performs build actions, such as making directories, invoking +@command{make}, etc. -@example -(define (profile.sh) - ;; Return the name of a shell script in the store that - ;; initializes the 'PATH' environment variable. - (text-file* "profile.sh" - "export PATH=" coreutils "/bin:" - grep "/bin:" sed "/bin\n")) -@end example +To describe a derivation and its build actions, one typically needs to +embed build code inside host code. It boils down to manipulating build +code as data, and the homoiconicity of Scheme---code has a direct +representation as data---comes in handy for that. But we need more than +the normal @code{quasiquote} mechanism in Scheme to construct build +expressions. -In this example, the resulting @file{/gnu/store/@dots{}-profile.sh} file -will reference @var{coreutils}, @var{grep}, and @var{sed}, thereby -preventing them from being garbage-collected during its lifetime. -@end deffn +The @code{(guix gexp)} module implements @dfn{G-expressions}, a form of +S-expressions adapted to build expressions. G-expressions, or +@dfn{gexps}, consist essentially of three syntactic forms: @code{gexp}, +@code{ungexp}, and @code{ungexp-splicing} (or simply: @code{#~}, +@code{#$}, and @code{#$@@}), which are comparable to +@code{quasiquote}, @code{unquote}, and @code{unquote-splicing}, +respectively (@pxref{Expression Syntax, @code{quasiquote},, guile, +GNU Guile Reference Manual}). However, there are major differences: -@deffn {Scheme Procedure} mixed-text-file @var{name} @var{text} @dots{} -Return an object representing store file @var{name} containing -@var{text}. @var{text} is a sequence of strings and file-like objects, -as in: +@itemize +@item +Gexps are meant to be written to a file and run or manipulated by other +processes. -@example -(mixed-text-file "profile" - "export PATH=" coreutils "/bin:" grep "/bin") -@end example +@item +When a high-level object such as a package or derivation is unquoted +inside a gexp, the result is as if its output file name had been +introduced. -This is the declarative counterpart of @code{text-file*}. -@end deffn +@item +Gexps carry information about the packages or derivations they refer to, +and these dependencies are automatically added as inputs to the build +processes that use them. +@end itemize -@deffn {Scheme Procedure} file-union @var{name} @var{files} -Return a @code{} that builds a directory containing all of @var{files}. -Each item in @var{files} must be a two-element list where the first element is the -file name to use in the new directory, and the second element is a gexp -denoting the target file. Here's an example: +@cindex lowering, of high-level objects in gexps +This mechanism is not limited to package and derivation +objects: @dfn{compilers} able to ``lower'' other high-level objects to +derivations or files in the store can be defined, +such that these objects can also be inserted +into gexps. For example, a useful type of high-level objects that can be +inserted in a gexp is ``file-like objects'', which make it easy to +add files to the store and to refer to them in +derivations and such (see @code{local-file} and @code{plain-file} +below.) + +To illustrate the idea, here is an example of a gexp: @example -(file-union "etc" - `(("hosts" ,(plain-file "hosts" - "127.0.0.1 localhost")) - ("bashrc" ,(plain-file "bashrc" - "alias ls='ls --color=auto'")))) +(define build-exp + #~(begin + (mkdir #$output) + (chdir #$output) + (symlink (string-append #$coreutils "/bin/ls") + "list-files"))) @end example -This yields an @code{etc} directory containing these two files. -@end deffn - -@deffn {Scheme Procedure} directory-union @var{name} @var{things} -Return a directory that is the union of @var{things}, where @var{things} is a list of -file-like objects denoting directories. For example: +This gexp can be passed to @code{gexp->derivation}; we obtain a +derivation that builds a directory containing exactly one symlink to +@file{/gnu/store/@dots{}-coreutils-8.22/bin/ls}: @example -(directory-union "guile+emacs" (list guile emacs)) +(gexp->derivation "the-thing" build-exp) @end example -yields a directory that is the union of the @code{guile} and @code{emacs} packages. -@end deffn - -@deffn {Scheme Procedure} file-append @var{obj} @var{suffix} @dots{} -Return a file-like object that expands to the concatenation of @var{obj} -and @var{suffix}, where @var{obj} is a lowerable object and each -@var{suffix} is a string. +As one would expect, the @code{"/gnu/store/@dots{}-coreutils-8.22"} string is +substituted to the reference to the @var{coreutils} package in the +actual build code, and @var{coreutils} is automatically made an input to +the derivation. Likewise, @code{#$output} (equivalent to @code{(ungexp +output)}) is replaced by a string containing the directory name of the +output of the derivation. -As an example, consider this gexp: +@cindex cross compilation +In a cross-compilation context, it is useful to distinguish between +references to the @emph{native} build of a package---that can run on the +host---versus references to cross builds of a package. To that end, the +@code{#+} plays the same role as @code{#$}, but is a reference to a +native package build: @example -(gexp->script "run-uname" - #~(system* #$(file-append coreutils - "/bin/uname"))) +(gexp->derivation "vi" + #~(begin + (mkdir #$output) + (system* (string-append #+coreutils "/bin/ln") + "-s" + (string-append #$emacs "/bin/emacs") + (string-append #$output "/bin/vi"))) + #:target "mips64el-linux-gnu") @end example -The same effect could be achieved with: +@noindent +In the example above, the native build of @var{coreutils} is used, so +that @command{ln} can actually run on the host; but then the +cross-compiled build of @var{emacs} is referenced. + +@cindex imported modules, for gexps +@findex with-imported-modules +Another gexp feature is @dfn{imported modules}: sometimes you want to be +able to use certain Guile modules from the ``host environment'' in the +gexp, so those modules should be imported in the ``build environment''. +The @code{with-imported-modules} form allows you to express that: @example -(gexp->script "run-uname" - #~(system* (string-append #$coreutils - "/bin/uname"))) +(let ((build (with-imported-modules '((guix build utils)) + #~(begin + (use-modules (guix build utils)) + (mkdir-p (string-append #$output "/bin")))))) + (gexp->derivation "empty-dir" + #~(begin + #$build + (display "success!\n") + #t))) @end example -There is one difference though: in the @code{file-append} case, the -resulting script contains the absolute file name as a string, whereas in -the second case, the resulting script contains a @code{(string-append -@dots{})} expression to construct the file name @emph{at run time}. -@end deffn - - -Of course, in addition to gexps embedded in ``host'' code, there are -also modules containing build tools. To make it clear that they are -meant to be used in the build stratum, these modules are kept in the -@code{(guix build @dots{})} name space. +@noindent +In this example, the @code{(guix build utils)} module is automatically +pulled into the isolated build environment of our gexp, such that +@code{(use-modules (guix build utils))} works as expected. -@cindex lowering, of high-level objects in gexps -Internally, high-level objects are @dfn{lowered}, using their compiler, -to either derivations or store items. For instance, lowering a package -yields a derivation, and lowering a @code{plain-file} yields a store -item. This is achieved using the @code{lower-object} monadic procedure. +@cindex module closure +@findex source-module-closure +Usually you want the @emph{closure} of the module to be imported---i.e., +the module itself and all the modules it depends on---rather than just +the module; failing to do that, attempts to use the module will fail +because of missing dependent modules. The @code{source-module-closure} +procedure computes the closure of a module by looking at its source file +headers, which comes in handy in this case: -@deffn {Monadic Procedure} lower-object @var{obj} [@var{system}] @ - [#:target #f] -Return as a value in @var{%store-monad} the derivation or store item -corresponding to @var{obj} for @var{system}, cross-compiling for -@var{target} if @var{target} is true. @var{obj} must be an object that -has an associated gexp compiler, such as a @code{}. -@end deffn +@example +(use-modules (guix modules)) ;for 'source-module-closure' -@node Invoking guix repl -@section Invoking @command{guix repl} +(with-imported-modules (source-module-closure + '((guix build utils) + (gnu build vm))) + (gexp->derivation "something-with-vms" + #~(begin + (use-modules (guix build utils) + (gnu build vm)) + @dots{}))) +@end example -@cindex REPL, read-eval-print loop -The @command{guix repl} command spawns a Guile @dfn{read-eval-print loop} -(REPL) for interactive programming (@pxref{Using Guile Interactively,,, guile, -GNU Guile Reference Manual}). Compared to just launching the @command{guile} -command, @command{guix repl} guarantees that all the Guix modules and all its -dependencies are available in the search path. You can use it this way: +@cindex extensions, for gexps +@findex with-extensions +In the same vein, sometimes you want to import not just pure-Scheme +modules, but also ``extensions'' such as Guile bindings to C libraries +or other ``full-blown'' packages. Say you need the @code{guile-json} +package available on the build side, here's how you would do it: @example -$ guix repl -scheme@@(guile-user)> ,use (gnu packages base) -scheme@@(guile-user)> coreutils -$1 = # -@end example +(use-modules (gnu packages guile)) ;for 'guile-json' -@cindex inferiors -In addition, @command{guix repl} implements a simple machine-readable REPL -protocol for use by @code{(guix inferior)}, a facility to interact with -@dfn{inferiors}, separate processes running a potentially different revision -of Guix. +(with-extensions (list guile-json) + (gexp->derivation "something-with-json" + #~(begin + (use-modules (json)) + @dots{}))) +@end example -The available options are as follows: +The syntactic form to construct gexps is summarized below. -@table @code -@item --type=@var{type} -@itemx -t @var{type} -Start a REPL of the given @var{TYPE}, which can be one of the following: +@deffn {Scheme Syntax} #~@var{exp} +@deffnx {Scheme Syntax} (gexp @var{exp}) +Return a G-expression containing @var{exp}. @var{exp} may contain one +or more of the following forms: @table @code -@item guile -This is default, and it spawns a standard full-featured Guile REPL. -@item machine -Spawn a REPL that uses the machine-readable protocol. This is the protocol -that the @code{(guix inferior)} module speaks. -@end table +@item #$@var{obj} +@itemx (ungexp @var{obj}) +Introduce a reference to @var{obj}. @var{obj} may have one of the +supported types, for example a package or a +derivation, in which case the @code{ungexp} form is replaced by its +output file name---e.g., @code{"/gnu/store/@dots{}-coreutils-8.22}. -@item --listen=@var{endpoint} -By default, @command{guix repl} reads from standard input and writes to -standard output. When this option is passed, it will instead listen for -connections on @var{endpoint}. Here are examples of valid options: +If @var{obj} is a list, it is traversed and references to supported +objects are substituted similarly. -@table @code -@item --listen=tcp:37146 -Accept connections on localhost on port 37146. +If @var{obj} is another gexp, its contents are inserted and its +dependencies are added to those of the containing gexp. -@item --listen=unix:/tmp/socket -Accept connections on the Unix-domain socket @file{/tmp/socket}. -@end table -@end table +If @var{obj} is another kind of object, it is inserted as is. -@c ********************************************************************* -@node Utilities -@chapter Utilities +@item #$@var{obj}:@var{output} +@itemx (ungexp @var{obj} @var{output}) +This is like the form above, but referring explicitly to the +@var{output} of @var{obj}---this is useful when @var{obj} produces +multiple outputs (@pxref{Packages with Multiple Outputs}). -This section describes Guix command-line utilities. Some of them are -primarily targeted at developers and users who write new package -definitions, while others are more generally useful. They complement -the Scheme programming interface of Guix in a convenient way. +@item #+@var{obj} +@itemx #+@var{obj}:output +@itemx (ungexp-native @var{obj}) +@itemx (ungexp-native @var{obj} @var{output}) +Same as @code{ungexp}, but produces a reference to the @emph{native} +build of @var{obj} when used in a cross compilation context. -@menu -* Invoking guix build:: Building packages from the command line. -* Invoking guix edit:: Editing package definitions. -* Invoking guix download:: Downloading a file and printing its hash. -* Invoking guix hash:: Computing the cryptographic hash of a file. -* Invoking guix import:: Importing package definitions. -* Invoking guix refresh:: Updating package definitions. -* Invoking guix lint:: Finding errors in package definitions. -* Invoking guix size:: Profiling disk usage. -* Invoking guix graph:: Visualizing the graph of packages. -* Invoking guix environment:: Setting up development environments. -* Invoking guix publish:: Sharing substitutes. -* Invoking guix challenge:: Challenging substitute servers. -* Invoking guix copy:: Copying to and from a remote store. -* Invoking guix container:: Process isolation. -* Invoking guix weather:: Assessing substitute availability. -* Invoking guix processes:: Listing client processes. -@end menu +@item #$output[:@var{output}] +@itemx (ungexp output [@var{output}]) +Insert a reference to derivation output @var{output}, or to the main +output when @var{output} is omitted. -@node Invoking guix build -@section Invoking @command{guix build} +This only makes sense for gexps passed to @code{gexp->derivation}. -@cindex package building -@cindex @command{guix build} -The @command{guix build} command builds packages or derivations and -their dependencies, and prints the resulting store paths. Note that it -does not modify the user's profile---this is the job of the -@command{guix package} command (@pxref{Invoking guix package}). Thus, -it is mainly useful for distribution developers. +@item #$@@@var{lst} +@itemx (ungexp-splicing @var{lst}) +Like the above, but splices the contents of @var{lst} inside the +containing list. -The general syntax is: +@item #+@@@var{lst} +@itemx (ungexp-native-splicing @var{lst}) +Like the above, but refers to native builds of the objects listed in +@var{lst}. -@example -guix build @var{options} @var{package-or-derivation}@dots{} -@end example +@end table -As an example, the following command builds the latest versions of Emacs -and of Guile, displays their build logs, and finally displays the -resulting directories: +G-expressions created by @code{gexp} or @code{#~} are run-time objects +of the @code{gexp?} type (see below.) +@end deffn -@example -guix build emacs guile -@end example +@deffn {Scheme Syntax} with-imported-modules @var{modules} @var{body}@dots{} +Mark the gexps defined in @var{body}@dots{} as requiring @var{modules} +in their execution environment. -Similarly, the following command builds all the available packages: +Each item in @var{modules} can be the name of a module, such as +@code{(guix build utils)}, or it can be a module name, followed by an +arrow, followed by a file-like object: @example -guix build --quiet --keep-going \ - `guix package -A | cut -f1,2 --output-delimiter=@@` +`((guix build utils) + (guix gcrypt) + ((guix config) => ,(scheme-file "config.scm" + #~(define-module @dots{})))) @end example -@var{package-or-derivation} may be either the name of a package found in -the software distribution such as @code{coreutils} or -@code{coreutils@@8.20}, or a derivation such as -@file{/gnu/store/@dots{}-coreutils-8.19.drv}. In the former case, a -package with the corresponding name (and optionally version) is searched -for among the GNU distribution modules (@pxref{Package Modules}). - -Alternatively, the @code{--expression} option may be used to specify a -Scheme expression that evaluates to a package; this is useful when -disambiguating among several same-named packages or package variants is -needed. - -There may be zero or more @var{options}. The available options are -described in the subsections below. - -@menu -* Common Build Options:: Build options for most commands. -* Package Transformation Options:: Creating variants of packages. -* Additional Build Options:: Options specific to 'guix build'. -* Debugging Build Failures:: Real life packaging experience. -@end menu - -@node Common Build Options -@subsection Common Build Options - -A number of options that control the build process are common to -@command{guix build} and other commands that can spawn builds, such as -@command{guix package} or @command{guix archive}. These are the -following: +@noindent +In the example above, the first two modules are taken from the search +path, and the last one is created from the given file-like object. -@table @code +This form has @emph{lexical} scope: it has an effect on the gexps +directly defined in @var{body}@dots{}, but not on those defined, say, in +procedures called from @var{body}@dots{}. +@end deffn -@item --load-path=@var{directory} -@itemx -L @var{directory} -Add @var{directory} to the front of the package module search path -(@pxref{Package Modules}). +@deffn {Scheme Syntax} with-extensions @var{extensions} @var{body}@dots{} +Mark the gexps defined in @var{body}@dots{} as requiring +@var{extensions} in their build and execution environment. +@var{extensions} is typically a list of package objects such as those +defined in the @code{(gnu packages guile)} module. -This allows users to define their own packages and make them visible to -the command-line tools. +Concretely, the packages listed in @var{extensions} are added to the +load path while compiling imported modules in @var{body}@dots{}; they +are also added to the load path of the gexp returned by +@var{body}@dots{}. +@end deffn -@item --keep-failed -@itemx -K -Keep the build tree of failed builds. Thus, if a build fails, its build -tree is kept under @file{/tmp}, in a directory whose name is shown at -the end of the build log. This is useful when debugging build issues. -@xref{Debugging Build Failures}, for tips and tricks on how to debug -build issues. +@deffn {Scheme Procedure} gexp? @var{obj} +Return @code{#t} if @var{obj} is a G-expression. +@end deffn -This option has no effect when connecting to a remote daemon with a -@code{guix://} URI (@pxref{The Store, the @code{GUIX_DAEMON_SOCKET} -variable}). +G-expressions are meant to be written to disk, either as code building +some derivation, or as plain files in the store. The monadic procedures +below allow you to do that (@pxref{The Store Monad}, for more +information about monads.) -@item --keep-going -@itemx -k -Keep going when some of the derivations fail to build; return only once -all the builds have either completed or failed. +@deffn {Monadic Procedure} gexp->derivation @var{name} @var{exp} @ + [#:system (%current-system)] [#:target #f] [#:graft? #t] @ + [#:hash #f] [#:hash-algo #f] @ + [#:recursive? #f] [#:env-vars '()] [#:modules '()] @ + [#:module-path @var{%load-path}] @ + [#:effective-version "2.2"] @ + [#:references-graphs #f] [#:allowed-references #f] @ + [#:disallowed-references #f] @ + [#:leaked-env-vars #f] @ + [#:script-name (string-append @var{name} "-builder")] @ + [#:deprecation-warnings #f] @ + [#:local-build? #f] [#:substitutable? #t] @ + [#:properties '()] [#:guile-for-build #f] +Return a derivation @var{name} that runs @var{exp} (a gexp) with +@var{guile-for-build} (a derivation) on @var{system}; @var{exp} is +stored in a file called @var{script-name}. When @var{target} is true, +it is used as the cross-compilation target triplet for packages referred +to by @var{exp}. -The default behavior is to stop as soon as one of the specified -derivations has failed. +@var{modules} is deprecated in favor of @code{with-imported-modules}. +Its meaning is to +make @var{modules} available in the evaluation context of @var{exp}; +@var{modules} is a list of names of Guile modules searched in +@var{module-path} to be copied in the store, compiled, and made available in +the load path during the execution of @var{exp}---e.g., @code{((guix +build utils) (guix build gnu-build-system))}. -@item --dry-run -@itemx -n -Do not build the derivations. +@var{effective-version} determines the string to use when adding extensions of +@var{exp} (see @code{with-extensions}) to the search path---e.g., @code{"2.2"}. -@anchor{fallback-option} -@item --fallback -When substituting a pre-built binary fails, fall back to building -packages locally (@pxref{Substitution Failure}). +@var{graft?} determines whether packages referred to by @var{exp} should be grafted when +applicable. -@item --substitute-urls=@var{urls} -@anchor{client-substitute-urls} -Consider @var{urls} the whitespace-separated list of substitute source -URLs, overriding the default list of URLs of @command{guix-daemon} -(@pxref{daemon-substitute-urls,, @command{guix-daemon} URLs}). +When @var{references-graphs} is true, it must be a list of tuples of one of the +following forms: -This means that substitutes may be downloaded from @var{urls}, provided -they are signed by a key authorized by the system administrator -(@pxref{Substitutes}). +@example +(@var{file-name} @var{package}) +(@var{file-name} @var{package} @var{output}) +(@var{file-name} @var{derivation}) +(@var{file-name} @var{derivation} @var{output}) +(@var{file-name} @var{store-item}) +@end example -When @var{urls} is the empty string, substitutes are effectively -disabled. +The right-hand-side of each element of @var{references-graphs} is automatically made +an input of the build process of @var{exp}. In the build environment, each +@var{file-name} contains the reference graph of the corresponding item, in a simple +text format. -@item --no-substitutes -Do not use substitutes for build products. That is, always build things -locally instead of allowing downloads of pre-built binaries -(@pxref{Substitutes}). +@var{allowed-references} must be either @code{#f} or a list of output names and packages. +In the latter case, the list denotes store items that the result is allowed to +refer to. Any reference to another store item will lead to a build error. +Similarly for @var{disallowed-references}, which can list items that must not be +referenced by the outputs. -@item --no-grafts -Do not ``graft'' packages. In practice, this means that package updates -available as grafts are not applied. @xref{Security Updates}, for more -information on grafts. +@var{deprecation-warnings} determines whether to show deprecation warnings while +compiling modules. It can be @code{#f}, @code{#t}, or @code{'detailed}. -@item --rounds=@var{n} -Build each derivation @var{n} times in a row, and raise an error if -consecutive build results are not bit-for-bit identical. +The other arguments are as for @code{derivation} (@pxref{Derivations}). +@end deffn -This is a useful way to detect non-deterministic builds processes. -Non-deterministic build processes are a problem because they make it -practically impossible for users to @emph{verify} whether third-party -binaries are genuine. @xref{Invoking guix challenge}, for more. +@cindex file-like objects +The @code{local-file}, @code{plain-file}, @code{computed-file}, +@code{program-file}, and @code{scheme-file} procedures below return +@dfn{file-like objects}. That is, when unquoted in a G-expression, +these objects lead to a file in the store. Consider this G-expression: -Note that, currently, the differing build results are not kept around, -so you will have to manually investigate in case of an error---e.g., by -stashing one of the build results with @code{guix archive --export} -(@pxref{Invoking guix archive}), then rebuilding, and finally comparing -the two results. +@example +#~(system* #$(file-append glibc "/sbin/nscd") "-f" + #$(local-file "/tmp/my-nscd.conf")) +@end example -@item --no-build-hook -Do not attempt to offload builds @i{via} the ``build hook'' of the daemon -(@pxref{Daemon Offload Setup}). That is, always build things locally -instead of offloading builds to remote machines. +The effect here is to ``intern'' @file{/tmp/my-nscd.conf} by copying it +to the store. Once expanded, for instance @i{via} +@code{gexp->derivation}, the G-expression refers to that copy under +@file{/gnu/store}; thus, modifying or removing the file in @file{/tmp} +does not have any effect on what the G-expression does. +@code{plain-file} can be used similarly; it differs in that the file +content is directly passed as a string. -@item --max-silent-time=@var{seconds} -When the build or substitution process remains silent for more than -@var{seconds}, terminate it and report a build failure. +@deffn {Scheme Procedure} local-file @var{file} [@var{name}] @ + [#:recursive? #f] [#:select? (const #t)] +Return an object representing local file @var{file} to add to the store; this +object can be used in a gexp. If @var{file} is a relative file name, it is looked +up relative to the source file where this form appears. @var{file} will be added to +the store under @var{name}--by default the base name of @var{file}. -By default, the daemon's setting is honored (@pxref{Invoking -guix-daemon, @code{--max-silent-time}}). +When @var{recursive?} is true, the contents of @var{file} are added recursively; if @var{file} +designates a flat file and @var{recursive?} is true, its contents are added, and its +permission bits are kept. -@item --timeout=@var{seconds} -Likewise, when the build or substitution process lasts for more than -@var{seconds}, terminate it and report a build failure. +When @var{recursive?} is true, call @code{(@var{select?} @var{file} +@var{stat})} for each directory entry, where @var{file} is the entry's +absolute file name and @var{stat} is the result of @code{lstat}; exclude +entries for which @var{select?} does not return true. -By default, the daemon's setting is honored (@pxref{Invoking -guix-daemon, @code{--timeout}}). +This is the declarative counterpart of the @code{interned-file} monadic +procedure (@pxref{The Store Monad, @code{interned-file}}). +@end deffn -@c Note: This option is actually not part of %standard-build-options but -@c most programs honor it. -@cindex verbosity, of the command-line tools -@cindex build logs, verbosity -@item -v @var{level} -@itemx --verbosity=@var{level} -Use the given verbosity @var{level}, an integer. Choosing 0 means that no -output is produced, 1 is for quiet output, and 2 shows all the build log -output on standard error. +@deffn {Scheme Procedure} plain-file @var{name} @var{content} +Return an object representing a text file called @var{name} with the given +@var{content} (a string or a bytevector) to be added to the store. -@item --cores=@var{n} -@itemx -c @var{n} -Allow the use of up to @var{n} CPU cores for the build. The special -value @code{0} means to use as many CPU cores as available. +This is the declarative counterpart of @code{text-file}. +@end deffn -@item --max-jobs=@var{n} -@itemx -M @var{n} -Allow at most @var{n} build jobs in parallel. @xref{Invoking -guix-daemon, @code{--max-jobs}}, for details about this option and the -equivalent @command{guix-daemon} option. +@deffn {Scheme Procedure} computed-file @var{name} @var{gexp} @ + [#:options '(#:local-build? #t)] +Return an object representing the store item @var{name}, a file or +directory computed by @var{gexp}. @var{options} +is a list of additional arguments to pass to @code{gexp->derivation}. -@item --debug=@var{level} -Produce debugging output coming from the build daemon. @var{level} must be an -integer between 0 and 5; higher means more verbose output. Setting a level of -4 or more may be helpful when debugging setup issues with the build daemon. +This is the declarative counterpart of @code{gexp->derivation}. +@end deffn -@end table +@deffn {Monadic Procedure} gexp->script @var{name} @var{exp} @ + [#:guile (default-guile)] [#:module-path %load-path] +Return an executable script @var{name} that runs @var{exp} using +@var{guile}, with @var{exp}'s imported modules in its search path. +Look up @var{exp}'s modules in @var{module-path}. -Behind the scenes, @command{guix build} is essentially an interface to -the @code{package-derivation} procedure of the @code{(guix packages)} -module, and to the @code{build-derivations} procedure of the @code{(guix -derivations)} module. +The example below builds a script that simply invokes the @command{ls} +command: -In addition to options explicitly passed on the command line, -@command{guix build} and other @command{guix} commands that support -building honor the @code{GUIX_BUILD_OPTIONS} environment variable. +@example +(use-modules (guix gexp) (gnu packages base)) -@defvr {Environment Variable} GUIX_BUILD_OPTIONS -Users can define this variable to a list of command line options that -will automatically be used by @command{guix build} and other -@command{guix} commands that can perform builds, as in the example -below: +(gexp->script "list-files" + #~(execl #$(file-append coreutils "/bin/ls") + "ls")) +@end example + +When ``running'' it through the store (@pxref{The Store Monad, +@code{run-with-store}}), we obtain a derivation that produces an +executable file @file{/gnu/store/@dots{}-list-files} along these lines: @example -$ export GUIX_BUILD_OPTIONS="--no-substitutes -c 2 -L /foo/bar" +#!/gnu/store/@dots{}-guile-2.0.11/bin/guile -ds +!# +(execl "/gnu/store/@dots{}-coreutils-8.22"/bin/ls" "ls") @end example +@end deffn -These options are parsed independently, and the result is appended to -the parsed command-line options. -@end defvr +@deffn {Scheme Procedure} program-file @var{name} @var{exp} @ + [#:guile #f] [#:module-path %load-path] +Return an object representing the executable store item @var{name} that +runs @var{gexp}. @var{guile} is the Guile package used to execute that +script. Imported modules of @var{gexp} are looked up in @var{module-path}. +This is the declarative counterpart of @code{gexp->script}. +@end deffn -@node Package Transformation Options -@subsection Package Transformation Options +@deffn {Monadic Procedure} gexp->file @var{name} @var{exp} @ + [#:set-load-path? #t] [#:module-path %load-path] @ + [#:splice? #f] @ + [#:guile (default-guile)] +Return a derivation that builds a file @var{name} containing @var{exp}. +When @var{splice?} is true, @var{exp} is considered to be a list of +expressions that will be spliced in the resulting file. -@cindex package variants -Another set of command-line options supported by @command{guix build} -and also @command{guix package} are @dfn{package transformation -options}. These are options that make it possible to define @dfn{package -variants}---for instance, packages built from different source code. -This is a convenient way to create customized packages on the fly -without having to type in the definitions of package variants -(@pxref{Defining Packages}). +When @var{set-load-path?} is true, emit code in the resulting file to +set @code{%load-path} and @code{%load-compiled-path} to honor +@var{exp}'s imported modules. Look up @var{exp}'s modules in +@var{module-path}. -@table @code +The resulting file holds references to all the dependencies of @var{exp} +or a subset thereof. +@end deffn -@item --with-source=@var{source} -@itemx --with-source=@var{package}=@var{source} -@itemx --with-source=@var{package}@@@var{version}=@var{source} -Use @var{source} as the source of @var{package}, and @var{version} as -its version number. -@var{source} must be a file name or a URL, as for @command{guix -download} (@pxref{Invoking guix download}). +@deffn {Scheme Procedure} scheme-file @var{name} @var{exp} [#:splice? #f] +Return an object representing the Scheme file @var{name} that contains +@var{exp}. -When @var{package} is omitted, -it is taken to be the package name specified on the -command line that matches the base of @var{source}---e.g., -if @var{source} is @code{/src/guile-2.0.10.tar.gz}, the corresponding -package is @code{guile}. +This is the declarative counterpart of @code{gexp->file}. +@end deffn -Likewise, when @var{version} is omitted, the version string is inferred from -@var{source}; in the previous example, it is @code{2.0.10}. +@deffn {Monadic Procedure} text-file* @var{name} @var{text} @dots{} +Return as a monadic value a derivation that builds a text file +containing all of @var{text}. @var{text} may list, in addition to +strings, objects of any type that can be used in a gexp: packages, +derivations, local file objects, etc. The resulting store file holds +references to all these. -This option allows users to try out versions of packages other than the -one provided by the distribution. The example below downloads -@file{ed-1.7.tar.gz} from a GNU mirror and uses that as the source for -the @code{ed} package: +This variant should be preferred over @code{text-file} anytime the file +to create will reference items from the store. This is typically the +case when building a configuration file that embeds store file names, +like this: @example -guix build ed --with-source=mirror://gnu/ed/ed-1.7.tar.gz +(define (profile.sh) + ;; Return the name of a shell script in the store that + ;; initializes the 'PATH' environment variable. + (text-file* "profile.sh" + "export PATH=" coreutils "/bin:" + grep "/bin:" sed "/bin\n")) @end example -As a developer, @code{--with-source} makes it easy to test release -candidates: - -@example -guix build guile --with-source=../guile-2.0.9.219-e1bb7.tar.xz -@end example +In this example, the resulting @file{/gnu/store/@dots{}-profile.sh} file +will reference @var{coreutils}, @var{grep}, and @var{sed}, thereby +preventing them from being garbage-collected during its lifetime. +@end deffn -@dots{} or to build from a checkout in a pristine environment: +@deffn {Scheme Procedure} mixed-text-file @var{name} @var{text} @dots{} +Return an object representing store file @var{name} containing +@var{text}. @var{text} is a sequence of strings and file-like objects, +as in: @example -$ git clone git://git.sv.gnu.org/guix.git -$ guix build guix --with-source=guix@@1.0=./guix +(mixed-text-file "profile" + "export PATH=" coreutils "/bin:" grep "/bin") @end example -@item --with-input=@var{package}=@var{replacement} -Replace dependency on @var{package} by a dependency on -@var{replacement}. @var{package} must be a package name, and -@var{replacement} must be a package specification such as @code{guile} -or @code{guile@@1.8}. +This is the declarative counterpart of @code{text-file*}. +@end deffn -For instance, the following command builds Guix, but replaces its -dependency on the current stable version of Guile with a dependency on -the legacy version of Guile, @code{guile@@2.0}: +@deffn {Scheme Procedure} file-union @var{name} @var{files} +Return a @code{} that builds a directory containing all of @var{files}. +Each item in @var{files} must be a two-element list where the first element is the +file name to use in the new directory, and the second element is a gexp +denoting the target file. Here's an example: @example -guix build --with-input=guile=guile@@2.0 guix +(file-union "etc" + `(("hosts" ,(plain-file "hosts" + "127.0.0.1 localhost")) + ("bashrc" ,(plain-file "bashrc" + "alias ls='ls --color=auto'")))) @end example -This is a recursive, deep replacement. So in this example, both -@code{guix} and its dependency @code{guile-json} (which also depends on -@code{guile}) get rebuilt against @code{guile@@2.0}. - -This is implemented using the @code{package-input-rewriting} Scheme -procedure (@pxref{Defining Packages, @code{package-input-rewriting}}). - -@item --with-graft=@var{package}=@var{replacement} -This is similar to @code{--with-input} but with an important difference: -instead of rebuilding the whole dependency chain, @var{replacement} is -built and then @dfn{grafted} onto the binaries that were initially -referring to @var{package}. @xref{Security Updates}, for more -information on grafts. +This yields an @code{etc} directory containing these two files. +@end deffn -For example, the command below grafts version 3.5.4 of GnuTLS onto Wget -and all its dependencies, replacing references to the version of GnuTLS -they currently refer to: +@deffn {Scheme Procedure} directory-union @var{name} @var{things} +Return a directory that is the union of @var{things}, where @var{things} is a list of +file-like objects denoting directories. For example: @example -guix build --with-graft=gnutls=gnutls@@3.5.4 wget +(directory-union "guile+emacs" (list guile emacs)) @end example -This has the advantage of being much faster than rebuilding everything. -But there is a caveat: it works if and only if @var{package} and -@var{replacement} are strictly compatible---for example, if they provide -a library, the application binary interface (ABI) of those libraries -must be compatible. If @var{replacement} is somehow incompatible with -@var{package}, then the resulting package may be unusable. Use with -care! +yields a directory that is the union of the @code{guile} and @code{emacs} packages. +@end deffn -@item --with-branch=@var{package}=@var{branch} -@cindex Git, using the latest commit -@cindex latest commit, building -Build @var{package} from the latest commit of @var{branch}. The @code{source} -field of @var{package} must be an origin with the @code{git-fetch} method -(@pxref{origin Reference}) or a @code{git-checkout} object; the repository URL -is taken from that @code{source}. Git sub-modules of the repository are -fetched, recursively. +@deffn {Scheme Procedure} file-append @var{obj} @var{suffix} @dots{} +Return a file-like object that expands to the concatenation of @var{obj} +and @var{suffix}, where @var{obj} is a lowerable object and each +@var{suffix} is a string. -For instance, the following command builds @code{guile-sqlite3} from the -latest commit of its @code{master} branch, and then builds @code{guix} (which -depends on it) and @code{cuirass} (which depends on @code{guix}) against this -specific @code{guile-sqlite3} build: +As an example, consider this gexp: @example -guix build --with-branch=guile-sqlite3=master cuirass +(gexp->script "run-uname" + #~(system* #$(file-append coreutils + "/bin/uname"))) @end example -@cindex continuous integration -Obviously, since it uses the latest commit of the given branch, the result of -such a command varies over time. Nevertheless it is a convenient way to -rebuild entire software stacks against the latest commit of one or more -packages. This is particularly useful in the context of continuous -integration (CI). +The same effect could be achieved with: -Checkouts are kept in a cache under @file{~/.cache/guix/checkouts} to speed up -consecutive accesses to the same repository. You may want to clean it up once -in a while to save disk space. +@example +(gexp->script "run-uname" + #~(system* (string-append #$coreutils + "/bin/uname"))) +@end example -@item --with-commit=@var{package}=@var{commit} -This is similar to @code{--with-branch}, except that it builds from -@var{commit} rather than the tip of a branch. @var{commit} must be a valid -Git commit SHA1 identifier. -@end table +There is one difference though: in the @code{file-append} case, the +resulting script contains the absolute file name as a string, whereas in +the second case, the resulting script contains a @code{(string-append +@dots{})} expression to construct the file name @emph{at run time}. +@end deffn -@node Additional Build Options -@subsection Additional Build Options -The command-line options presented below are specific to @command{guix -build}. +Of course, in addition to gexps embedded in ``host'' code, there are +also modules containing build tools. To make it clear that they are +meant to be used in the build stratum, these modules are kept in the +@code{(guix build @dots{})} name space. -@table @code +@cindex lowering, of high-level objects in gexps +Internally, high-level objects are @dfn{lowered}, using their compiler, +to either derivations or store items. For instance, lowering a package +yields a derivation, and lowering a @code{plain-file} yields a store +item. This is achieved using the @code{lower-object} monadic procedure. -@item --quiet -@itemx -q -Build quietly, without displaying the build log; this is equivalent to -@code{--verbosity=0}. Upon completion, the build log is kept in @file{/var} -(or similar) and can always be retrieved using the @option{--log-file} option. +@deffn {Monadic Procedure} lower-object @var{obj} [@var{system}] @ + [#:target #f] +Return as a value in @var{%store-monad} the derivation or store item +corresponding to @var{obj} for @var{system}, cross-compiling for +@var{target} if @var{target} is true. @var{obj} must be an object that +has an associated gexp compiler, such as a @code{}. +@end deffn -@item --file=@var{file} -@itemx -f @var{file} -Build the package, derivation, or other file-like object that the code within -@var{file} evaluates to (@pxref{G-Expressions, file-like objects}). +@node Invoking guix repl +@section Invoking @command{guix repl} -As an example, @var{file} might contain a package definition like this -(@pxref{Defining Packages}): +@cindex REPL, read-eval-print loop +The @command{guix repl} command spawns a Guile @dfn{read-eval-print loop} +(REPL) for interactive programming (@pxref{Using Guile Interactively,,, guile, +GNU Guile Reference Manual}). Compared to just launching the @command{guile} +command, @command{guix repl} guarantees that all the Guix modules and all its +dependencies are available in the search path. You can use it this way: @example -@verbatiminclude package-hello.scm +$ guix repl +scheme@@(guile-user)> ,use (gnu packages base) +scheme@@(guile-user)> coreutils +$1 = # @end example -@item --expression=@var{expr} -@itemx -e @var{expr} -Build the package or derivation @var{expr} evaluates to. +@cindex inferiors +In addition, @command{guix repl} implements a simple machine-readable REPL +protocol for use by @code{(guix inferior)}, a facility to interact with +@dfn{inferiors}, separate processes running a potentially different revision +of Guix. -For example, @var{expr} may be @code{(@@ (gnu packages guile) -guile-1.8)}, which unambiguously designates this specific variant of -version 1.8 of Guile. +The available options are as follows: -Alternatively, @var{expr} may be a G-expression, in which case it is used -as a build program passed to @code{gexp->derivation} -(@pxref{G-Expressions}). +@table @code +@item --type=@var{type} +@itemx -t @var{type} +Start a REPL of the given @var{TYPE}, which can be one of the following: -Lastly, @var{expr} may refer to a zero-argument monadic procedure -(@pxref{The Store Monad}). The procedure must return a derivation as a -monadic value, which is then passed through @code{run-with-store}. +@table @code +@item guile +This is default, and it spawns a standard full-featured Guile REPL. +@item machine +Spawn a REPL that uses the machine-readable protocol. This is the protocol +that the @code{(guix inferior)} module speaks. +@end table -@item --source -@itemx -S -Build the source derivations of the packages, rather than the packages -themselves. +@item --listen=@var{endpoint} +By default, @command{guix repl} reads from standard input and writes to +standard output. When this option is passed, it will instead listen for +connections on @var{endpoint}. Here are examples of valid options: -For instance, @code{guix build -S gcc} returns something like -@file{/gnu/store/@dots{}-gcc-4.7.2.tar.bz2}, which is the GCC -source tarball. +@table @code +@item --listen=tcp:37146 +Accept connections on localhost on port 37146. -The returned source tarball is the result of applying any patches and -code snippets specified in the package @code{origin} (@pxref{Defining -Packages}). +@item --listen=unix:/tmp/socket +Accept connections on the Unix-domain socket @file{/tmp/socket}. +@end table +@end table -@item --sources -Fetch and return the source of @var{package-or-derivation} and all their -dependencies, recursively. This is a handy way to obtain a local copy -of all the source code needed to build @var{packages}, allowing you to -eventually build them even without network access. It is an extension -of the @code{--source} option and can accept one of the following -optional argument values: +@c ********************************************************************* +@node Utilities +@chapter Utilities + +This section describes Guix command-line utilities. Some of them are +primarily targeted at developers and users who write new package +definitions, while others are more generally useful. They complement +the Scheme programming interface of Guix in a convenient way. + +@menu +* Invoking guix build:: Building packages from the command line. +* Invoking guix edit:: Editing package definitions. +* Invoking guix download:: Downloading a file and printing its hash. +* Invoking guix hash:: Computing the cryptographic hash of a file. +* Invoking guix import:: Importing package definitions. +* Invoking guix refresh:: Updating package definitions. +* Invoking guix lint:: Finding errors in package definitions. +* Invoking guix size:: Profiling disk usage. +* Invoking guix graph:: Visualizing the graph of packages. +* Invoking guix publish:: Sharing substitutes. +* Invoking guix challenge:: Challenging substitute servers. +* Invoking guix copy:: Copying to and from a remote store. +* Invoking guix container:: Process isolation. +* Invoking guix weather:: Assessing substitute availability. +* Invoking guix processes:: Listing client processes. +@end menu + +@node Invoking guix build +@section Invoking @command{guix build} -@table @code -@item package -This value causes the @code{--sources} option to behave in the same way -as the @code{--source} option. +@cindex package building +@cindex @command{guix build} +The @command{guix build} command builds packages or derivations and +their dependencies, and prints the resulting store paths. Note that it +does not modify the user's profile---this is the job of the +@command{guix package} command (@pxref{Invoking guix package}). Thus, +it is mainly useful for distribution developers. -@item all -Build the source derivations of all packages, including any source that -might be listed as @code{inputs}. This is the default value. +The general syntax is: @example -$ guix build --sources tzdata -The following derivations will be built: - /gnu/store/@dots{}-tzdata2015b.tar.gz.drv - /gnu/store/@dots{}-tzcode2015b.tar.gz.drv +guix build @var{options} @var{package-or-derivation}@dots{} @end example -@item transitive -Build the source derivations of all packages, as well of all transitive -inputs to the packages. This can be used e.g.@: to -prefetch package source for later offline building. +As an example, the following command builds the latest versions of Emacs +and of Guile, displays their build logs, and finally displays the +resulting directories: @example -$ guix build --sources=transitive tzdata -The following derivations will be built: - /gnu/store/@dots{}-tzcode2015b.tar.gz.drv - /gnu/store/@dots{}-findutils-4.4.2.tar.xz.drv - /gnu/store/@dots{}-grep-2.21.tar.xz.drv - /gnu/store/@dots{}-coreutils-8.23.tar.xz.drv - /gnu/store/@dots{}-make-4.1.tar.xz.drv - /gnu/store/@dots{}-bash-4.3.tar.xz.drv -@dots{} +guix build emacs guile @end example -@end table - -@item --system=@var{system} -@itemx -s @var{system} -Attempt to build for @var{system}---e.g., @code{i686-linux}---instead of -the system type of the build host. - -@quotation Note -The @code{--system} flag is for @emph{native} compilation and must not -be confused with cross-compilation. See @code{--target} below for -information on cross-compilation. -@end quotation - -An example use of this is on Linux-based systems, which can emulate -different personalities. For instance, passing -@code{--system=i686-linux} on an @code{x86_64-linux} system or -@code{--system=armhf-linux} on an @code{aarch64-linux} system allows you -to build packages in a complete 32-bit environment. - -@quotation Note -Building for an @code{armhf-linux} system is unconditionally enabled on -@code{aarch64-linux} machines, although certain aarch64 chipsets do not -allow for this functionality, notably the ThunderX. -@end quotation +Similarly, the following command builds all the available packages: -Similarly, when transparent emulation with QEMU and @code{binfmt_misc} -is enabled (@pxref{Virtualization Services, -@code{qemu-binfmt-service-type}}), you can build for any system for -which a QEMU @code{binfmt_misc} handler is installed. +@example +guix build --quiet --keep-going \ + `guix package -A | cut -f1,2 --output-delimiter=@@` +@end example -Builds for a system other than that of the machine you are using can -also be offloaded to a remote machine of the right architecture. -@xref{Daemon Offload Setup}, for more information on offloading. +@var{package-or-derivation} may be either the name of a package found in +the software distribution such as @code{coreutils} or +@code{coreutils@@8.20}, or a derivation such as +@file{/gnu/store/@dots{}-coreutils-8.19.drv}. In the former case, a +package with the corresponding name (and optionally version) is searched +for among the GNU distribution modules (@pxref{Package Modules}). -@item --target=@var{triplet} -@cindex cross-compilation -Cross-build for @var{triplet}, which must be a valid GNU triplet, such -as @code{"mips64el-linux-gnu"} (@pxref{Specifying target triplets, GNU -configuration triplets,, autoconf, Autoconf}). +Alternatively, the @code{--expression} option may be used to specify a +Scheme expression that evaluates to a package; this is useful when +disambiguating among several same-named packages or package variants is +needed. -@anchor{build-check} -@item --check -@cindex determinism, checking -@cindex reproducibility, checking -Rebuild @var{package-or-derivation}, which are already available in the -store, and raise an error if the build results are not bit-for-bit -identical. +There may be zero or more @var{options}. The available options are +described in the subsections below. -This mechanism allows you to check whether previously installed -substitutes are genuine (@pxref{Substitutes}), or whether the build result -of a package is deterministic. @xref{Invoking guix challenge}, for more -background information and tools. +@menu +* Common Build Options:: Build options for most commands. +* Package Transformation Options:: Creating variants of packages. +* Additional Build Options:: Options specific to 'guix build'. +* Debugging Build Failures:: Real life packaging experience. +@end menu -When used in conjunction with @option{--keep-failed}, the differing -output is kept in the store, under @file{/gnu/store/@dots{}-check}. -This makes it easy to look for differences between the two results. +@node Common Build Options +@subsection Common Build Options -@item --repair -@cindex repairing store items -@cindex corruption, recovering from -Attempt to repair the specified store items, if they are corrupt, by -re-downloading or rebuilding them. +A number of options that control the build process are common to +@command{guix build} and other commands that can spawn builds, such as +@command{guix package} or @command{guix archive}. These are the +following: -This operation is not atomic and thus restricted to @code{root}. +@table @code -@item --derivations -@itemx -d -Return the derivation paths, not the output paths, of the given -packages. +@item --load-path=@var{directory} +@itemx -L @var{directory} +Add @var{directory} to the front of the package module search path +(@pxref{Package Modules}). -@item --root=@var{file} -@itemx -r @var{file} -@cindex GC roots, adding -@cindex garbage collector roots, adding -Make @var{file} a symlink to the result, and register it as a garbage -collector root. +This allows users to define their own packages and make them visible to +the command-line tools. -Consequently, the results of this @command{guix build} invocation are -protected from garbage collection until @var{file} is removed. When -that option is omitted, build results are eligible for garbage -collection as soon as the build completes. @xref{Invoking guix gc}, for -more on GC roots. +@item --keep-failed +@itemx -K +Keep the build tree of failed builds. Thus, if a build fails, its build +tree is kept under @file{/tmp}, in a directory whose name is shown at +the end of the build log. This is useful when debugging build issues. +@xref{Debugging Build Failures}, for tips and tricks on how to debug +build issues. -@item --log-file -@cindex build logs, access -Return the build log file names or URLs for the given -@var{package-or-derivation}, or raise an error if build logs are -missing. +This option has no effect when connecting to a remote daemon with a +@code{guix://} URI (@pxref{The Store, the @code{GUIX_DAEMON_SOCKET} +variable}). -This works regardless of how packages or derivations are specified. For -instance, the following invocations are equivalent: +@item --keep-going +@itemx -k +Keep going when some of the derivations fail to build; return only once +all the builds have either completed or failed. -@example -guix build --log-file `guix build -d guile` -guix build --log-file `guix build guile` -guix build --log-file guile -guix build --log-file -e '(@@ (gnu packages guile) guile-2.0)' -@end example +The default behavior is to stop as soon as one of the specified +derivations has failed. -If a log is unavailable locally, and unless @code{--no-substitutes} is -passed, the command looks for a corresponding log on one of the -substitute servers (as specified with @code{--substitute-urls}.) +@item --dry-run +@itemx -n +Do not build the derivations. -So for instance, imagine you want to see the build log of GDB on MIPS, -but you are actually on an @code{x86_64} machine: +@anchor{fallback-option} +@item --fallback +When substituting a pre-built binary fails, fall back to building +packages locally (@pxref{Substitution Failure}). -@example -$ guix build --log-file gdb -s mips64el-linux -https://@value{SUBSTITUTE-SERVER}/log/@dots{}-gdb-7.10 -@end example +@item --substitute-urls=@var{urls} +@anchor{client-substitute-urls} +Consider @var{urls} the whitespace-separated list of substitute source +URLs, overriding the default list of URLs of @command{guix-daemon} +(@pxref{daemon-substitute-urls,, @command{guix-daemon} URLs}). -You can freely access a huge library of build logs! -@end table +This means that substitutes may be downloaded from @var{urls}, provided +they are signed by a key authorized by the system administrator +(@pxref{Substitutes}). -@node Debugging Build Failures -@subsection Debugging Build Failures +When @var{urls} is the empty string, substitutes are effectively +disabled. -@cindex build failures, debugging -When defining a new package (@pxref{Defining Packages}), you will -probably find yourself spending some time debugging and tweaking the -build until it succeeds. To do that, you need to operate the build -commands yourself in an environment as close as possible to the one the -build daemon uses. +@item --no-substitutes +Do not use substitutes for build products. That is, always build things +locally instead of allowing downloads of pre-built binaries +(@pxref{Substitutes}). -To that end, the first thing to do is to use the @option{--keep-failed} -or @option{-K} option of @command{guix build}, which will keep the -failed build tree in @file{/tmp} or whatever directory you specified as -@code{TMPDIR} (@pxref{Invoking guix build, @code{--keep-failed}}). +@item --no-grafts +Do not ``graft'' packages. In practice, this means that package updates +available as grafts are not applied. @xref{Security Updates}, for more +information on grafts. -From there on, you can @command{cd} to the failed build tree and source -the @file{environment-variables} file, which contains all the -environment variable definitions that were in place when the build -failed. So let's say you're debugging a build failure in package -@code{foo}; a typical session would look like this: +@item --rounds=@var{n} +Build each derivation @var{n} times in a row, and raise an error if +consecutive build results are not bit-for-bit identical. -@example -$ guix build foo -K -@dots{} @i{build fails} -$ cd /tmp/guix-build-foo.drv-0 -$ source ./environment-variables -$ cd foo-1.2 -@end example +This is a useful way to detect non-deterministic builds processes. +Non-deterministic build processes are a problem because they make it +practically impossible for users to @emph{verify} whether third-party +binaries are genuine. @xref{Invoking guix challenge}, for more. -Now, you can invoke commands as if you were the daemon (almost) and -troubleshoot your build process. +Note that, currently, the differing build results are not kept around, +so you will have to manually investigate in case of an error---e.g., by +stashing one of the build results with @code{guix archive --export} +(@pxref{Invoking guix archive}), then rebuilding, and finally comparing +the two results. -Sometimes it happens that, for example, a package's tests pass when you -run them manually but they fail when the daemon runs them. This can -happen because the daemon runs builds in containers where, unlike in our -environment above, network access is missing, @file{/bin/sh} does not -exist, etc. (@pxref{Build Environment Setup}). +@item --no-build-hook +Do not attempt to offload builds @i{via} the ``build hook'' of the daemon +(@pxref{Daemon Offload Setup}). That is, always build things locally +instead of offloading builds to remote machines. -In such cases, you may need to run inspect the build process from within -a container similar to the one the build daemon creates: +@item --max-silent-time=@var{seconds} +When the build or substitution process remains silent for more than +@var{seconds}, terminate it and report a build failure. -@example -$ guix build -K foo -@dots{} -$ cd /tmp/guix-build-foo.drv-0 -$ guix environment --no-grafts -C foo --ad-hoc strace gdb -[env]# source ./environment-variables -[env]# cd foo-1.2 -@end example +By default, the daemon's setting is honored (@pxref{Invoking +guix-daemon, @code{--max-silent-time}}). -Here, @command{guix environment -C} creates a container and spawns a new -shell in it (@pxref{Invoking guix environment}). The @command{--ad-hoc -strace gdb} part adds the @command{strace} and @command{gdb} commands to -the container, which would may find handy while debugging. The -@option{--no-grafts} option makes sure we get the exact same -environment, with ungrafted packages (@pxref{Security Updates}, for more -info on grafts). +@item --timeout=@var{seconds} +Likewise, when the build or substitution process lasts for more than +@var{seconds}, terminate it and report a build failure. -To get closer to a container like that used by the build daemon, we can -remove @file{/bin/sh}: +By default, the daemon's setting is honored (@pxref{Invoking +guix-daemon, @code{--timeout}}). -@example -[env]# rm /bin/sh -@end example +@c Note: This option is actually not part of %standard-build-options but +@c most programs honor it. +@cindex verbosity, of the command-line tools +@cindex build logs, verbosity +@item -v @var{level} +@itemx --verbosity=@var{level} +Use the given verbosity @var{level}, an integer. Choosing 0 means that no +output is produced, 1 is for quiet output, and 2 shows all the build log +output on standard error. -(Don't worry, this is harmless: this is all happening in the throw-away -container created by @command{guix environment}.) +@item --cores=@var{n} +@itemx -c @var{n} +Allow the use of up to @var{n} CPU cores for the build. The special +value @code{0} means to use as many CPU cores as available. -The @command{strace} command is probably not in the search path, but we -can run: +@item --max-jobs=@var{n} +@itemx -M @var{n} +Allow at most @var{n} build jobs in parallel. @xref{Invoking +guix-daemon, @code{--max-jobs}}, for details about this option and the +equivalent @command{guix-daemon} option. -@example -[env]# $GUIX_ENVIRONMENT/bin/strace -f -o log make check -@end example +@item --debug=@var{level} +Produce debugging output coming from the build daemon. @var{level} must be an +integer between 0 and 5; higher means more verbose output. Setting a level of +4 or more may be helpful when debugging setup issues with the build daemon. -In this way, not only you will have reproduced the environment variables -the daemon uses, you will also be running the build process in a container -similar to the one the daemon uses. +@end table +Behind the scenes, @command{guix build} is essentially an interface to +the @code{package-derivation} procedure of the @code{(guix packages)} +module, and to the @code{build-derivations} procedure of the @code{(guix +derivations)} module. -@node Invoking guix edit -@section Invoking @command{guix edit} +In addition to options explicitly passed on the command line, +@command{guix build} and other @command{guix} commands that support +building honor the @code{GUIX_BUILD_OPTIONS} environment variable. -@cindex @command{guix edit} -@cindex package definition, editing -So many packages, so many source files! The @command{guix edit} command -facilitates the life of users and packagers by pointing their editor at -the source file containing the definition of the specified packages. -For instance: +@defvr {Environment Variable} GUIX_BUILD_OPTIONS +Users can define this variable to a list of command line options that +will automatically be used by @command{guix build} and other +@command{guix} commands that can perform builds, as in the example +below: @example -guix edit gcc@@4.9 vim +$ export GUIX_BUILD_OPTIONS="--no-substitutes -c 2 -L /foo/bar" @end example -@noindent -launches the program specified in the @code{VISUAL} or in the -@code{EDITOR} environment variable to view the recipe of GCC@tie{}4.9.3 -and that of Vim. +These options are parsed independently, and the result is appended to +the parsed command-line options. +@end defvr -If you are using a Guix Git checkout (@pxref{Building from Git}), or -have created your own packages on @code{GUIX_PACKAGE_PATH} -(@pxref{Package Modules}), you will be able to edit the package -recipes. In other cases, you will be able to examine the read-only recipes -for packages currently in the store. +@node Package Transformation Options +@subsection Package Transformation Options -@node Invoking guix download -@section Invoking @command{guix download} +@cindex package variants +Another set of command-line options supported by @command{guix build} +and also @command{guix package} are @dfn{package transformation +options}. These are options that make it possible to define @dfn{package +variants}---for instance, packages built from different source code. +This is a convenient way to create customized packages on the fly +without having to type in the definitions of package variants +(@pxref{Defining Packages}). -@cindex @command{guix download} -@cindex downloading package sources -When writing a package definition, developers typically need to download -a source tarball, compute its SHA256 hash, and write that -hash in the package definition (@pxref{Defining Packages}). The -@command{guix download} tool helps with this task: it downloads a file -from the given URI, adds it to the store, and prints both its file name -in the store and its SHA256 hash. +@table @code -The fact that the downloaded file is added to the store saves bandwidth: -when the developer eventually tries to build the newly defined package -with @command{guix build}, the source tarball will not have to be -downloaded again because it is already in the store. It is also a -convenient way to temporarily stash files, which may be deleted -eventually (@pxref{Invoking guix gc}). +@item --with-source=@var{source} +@itemx --with-source=@var{package}=@var{source} +@itemx --with-source=@var{package}@@@var{version}=@var{source} +Use @var{source} as the source of @var{package}, and @var{version} as +its version number. +@var{source} must be a file name or a URL, as for @command{guix +download} (@pxref{Invoking guix download}). -The @command{guix download} command supports the same URIs as used in -package definitions. In particular, it supports @code{mirror://} URIs. -@code{https} URIs (HTTP over TLS) are supported @emph{provided} the -Guile bindings for GnuTLS are available in the user's environment; when -they are not available, an error is raised. @xref{Guile Preparations, -how to install the GnuTLS bindings for Guile,, gnutls-guile, -GnuTLS-Guile}, for more information. +When @var{package} is omitted, +it is taken to be the package name specified on the +command line that matches the base of @var{source}---e.g., +if @var{source} is @code{/src/guile-2.0.10.tar.gz}, the corresponding +package is @code{guile}. -@command{guix download} verifies HTTPS server certificates by loading -the certificates of X.509 authorities from the directory pointed to by -the @code{SSL_CERT_DIR} environment variable (@pxref{X.509 -Certificates}), unless @option{--no-check-certificate} is used. +Likewise, when @var{version} is omitted, the version string is inferred from +@var{source}; in the previous example, it is @code{2.0.10}. -The following options are available: +This option allows users to try out versions of packages other than the +one provided by the distribution. The example below downloads +@file{ed-1.7.tar.gz} from a GNU mirror and uses that as the source for +the @code{ed} package: -@table @code -@item --format=@var{fmt} -@itemx -f @var{fmt} -Write the hash in the format specified by @var{fmt}. For more -information on the valid values for @var{fmt}, @pxref{Invoking guix hash}. +@example +guix build ed --with-source=mirror://gnu/ed/ed-1.7.tar.gz +@end example -@item --no-check-certificate -Do not validate the X.509 certificates of HTTPS servers. +As a developer, @code{--with-source} makes it easy to test release +candidates: -When using this option, you have @emph{absolutely no guarantee} that you -are communicating with the authentic server responsible for the given -URL, which makes you vulnerable to ``man-in-the-middle'' attacks. +@example +guix build guile --with-source=../guile-2.0.9.219-e1bb7.tar.xz +@end example -@item --output=@var{file} -@itemx -o @var{file} -Save the downloaded file to @var{file} instead of adding it to the -store. -@end table +@dots{} or to build from a checkout in a pristine environment: -@node Invoking guix hash -@section Invoking @command{guix hash} +@example +$ git clone git://git.sv.gnu.org/guix.git +$ guix build guix --with-source=guix@@1.0=./guix +@end example -@cindex @command{guix hash} -The @command{guix hash} command computes the SHA256 hash of a file. -It is primarily a convenience tool for anyone contributing to the -distribution: it computes the cryptographic hash of a file, which can be -used in the definition of a package (@pxref{Defining Packages}). +@item --with-input=@var{package}=@var{replacement} +Replace dependency on @var{package} by a dependency on +@var{replacement}. @var{package} must be a package name, and +@var{replacement} must be a package specification such as @code{guile} +or @code{guile@@1.8}. -The general syntax is: +For instance, the following command builds Guix, but replaces its +dependency on the current stable version of Guile with a dependency on +the legacy version of Guile, @code{guile@@2.0}: @example -guix hash @var{option} @var{file} +guix build --with-input=guile=guile@@2.0 guix @end example -When @var{file} is @code{-} (a hyphen), @command{guix hash} computes the -hash of data read from standard input. @command{guix hash} has the -following options: - -@table @code +This is a recursive, deep replacement. So in this example, both +@code{guix} and its dependency @code{guile-json} (which also depends on +@code{guile}) get rebuilt against @code{guile@@2.0}. -@item --format=@var{fmt} -@itemx -f @var{fmt} -Write the hash in the format specified by @var{fmt}. +This is implemented using the @code{package-input-rewriting} Scheme +procedure (@pxref{Defining Packages, @code{package-input-rewriting}}). -Supported formats: @code{nix-base32}, @code{base32}, @code{base16} -(@code{hex} and @code{hexadecimal} can be used as well). +@item --with-graft=@var{package}=@var{replacement} +This is similar to @code{--with-input} but with an important difference: +instead of rebuilding the whole dependency chain, @var{replacement} is +built and then @dfn{grafted} onto the binaries that were initially +referring to @var{package}. @xref{Security Updates}, for more +information on grafts. -If the @option{--format} option is not specified, @command{guix hash} -will output the hash in @code{nix-base32}. This representation is used -in the definitions of packages. +For example, the command below grafts version 3.5.4 of GnuTLS onto Wget +and all its dependencies, replacing references to the version of GnuTLS +they currently refer to: -@item --recursive -@itemx -r -Compute the hash on @var{file} recursively. +@example +guix build --with-graft=gnutls=gnutls@@3.5.4 wget +@end example -In this case, the hash is computed on an archive containing @var{file}, -including its children if it is a directory. Some of the metadata of -@var{file} is part of the archive; for instance, when @var{file} is a -regular file, the hash is different depending on whether @var{file} is -executable or not. Metadata such as time stamps has no impact on the -hash (@pxref{Invoking guix archive}). -@c FIXME: Replace xref above with xref to an ``Archive'' section when -@c it exists. +This has the advantage of being much faster than rebuilding everything. +But there is a caveat: it works if and only if @var{package} and +@var{replacement} are strictly compatible---for example, if they provide +a library, the application binary interface (ABI) of those libraries +must be compatible. If @var{replacement} is somehow incompatible with +@var{package}, then the resulting package may be unusable. Use with +care! -@item --exclude-vcs -@itemx -x -When combined with @option{--recursive}, exclude version control system -directories (@file{.bzr}, @file{.git}, @file{.hg}, etc.) +@item --with-branch=@var{package}=@var{branch} +@cindex Git, using the latest commit +@cindex latest commit, building +Build @var{package} from the latest commit of @var{branch}. The @code{source} +field of @var{package} must be an origin with the @code{git-fetch} method +(@pxref{origin Reference}) or a @code{git-checkout} object; the repository URL +is taken from that @code{source}. Git sub-modules of the repository are +fetched, recursively. -@vindex git-fetch -As an example, here is how you would compute the hash of a Git checkout, -which is useful when using the @code{git-fetch} method (@pxref{origin -Reference}): +For instance, the following command builds @code{guile-sqlite3} from the +latest commit of its @code{master} branch, and then builds @code{guix} (which +depends on it) and @code{cuirass} (which depends on @code{guix}) against this +specific @code{guile-sqlite3} build: @example -$ git clone http://example.org/foo.git -$ cd foo -$ guix hash -rx . +guix build --with-branch=guile-sqlite3=master cuirass @end example -@end table -@node Invoking guix import -@section Invoking @command{guix import} +@cindex continuous integration +Obviously, since it uses the latest commit of the given branch, the result of +such a command varies over time. Nevertheless it is a convenient way to +rebuild entire software stacks against the latest commit of one or more +packages. This is particularly useful in the context of continuous +integration (CI). -@cindex importing packages -@cindex package import -@cindex package conversion -@cindex Invoking @command{guix import} -The @command{guix import} command is useful for people who would like to -add a package to the distribution with as little work as -possible---a legitimate demand. The command knows of a few -repositories from which it can ``import'' package metadata. The result -is a package definition, or a template thereof, in the format we know -(@pxref{Defining Packages}). +Checkouts are kept in a cache under @file{~/.cache/guix/checkouts} to speed up +consecutive accesses to the same repository. You may want to clean it up once +in a while to save disk space. -The general syntax is: +@item --with-commit=@var{package}=@var{commit} +This is similar to @code{--with-branch}, except that it builds from +@var{commit} rather than the tip of a branch. @var{commit} must be a valid +Git commit SHA1 identifier. +@end table -@example -guix import @var{importer} @var{options}@dots{} -@end example +@node Additional Build Options +@subsection Additional Build Options -@var{importer} specifies the source from which to import package -metadata, and @var{options} specifies a package identifier and other -options specific to @var{importer}. Currently, the available -``importers'' are: +The command-line options presented below are specific to @command{guix +build}. @table @code -@item gnu -Import metadata for the given GNU package. This provides a template -for the latest version of that GNU package, including the hash of its -source tarball, and its canonical synopsis and description. -Additional information such as the package dependencies and its -license needs to be figured out manually. +@item --quiet +@itemx -q +Build quietly, without displaying the build log; this is equivalent to +@code{--verbosity=0}. Upon completion, the build log is kept in @file{/var} +(or similar) and can always be retrieved using the @option{--log-file} option. -For example, the following command returns a package definition for -GNU@tie{}Hello: +@item --file=@var{file} +@itemx -f @var{file} +Build the package, derivation, or other file-like object that the code within +@var{file} evaluates to (@pxref{G-Expressions, file-like objects}). + +As an example, @var{file} might contain a package definition like this +(@pxref{Defining Packages}): @example -guix import gnu hello +@verbatiminclude package-hello.scm @end example -Specific command-line options are: - -@table @code -@item --key-download=@var{policy} -As for @code{guix refresh}, specify the policy to handle missing OpenPGP -keys when verifying the package signature. @xref{Invoking guix -refresh, @code{--key-download}}. -@end table +@item --expression=@var{expr} +@itemx -e @var{expr} +Build the package or derivation @var{expr} evaluates to. -@item pypi -@cindex pypi -Import metadata from the @uref{https://pypi.python.org/, Python Package -Index}. Information is taken from the JSON-formatted description -available at @code{pypi.python.org} and usually includes all the relevant -information, including package dependencies. For maximum efficiency, it -is recommended to install the @command{unzip} utility, so that the -importer can unzip Python wheels and gather data from them. +For example, @var{expr} may be @code{(@@ (gnu packages guile) +guile-1.8)}, which unambiguously designates this specific variant of +version 1.8 of Guile. -The command below imports metadata for the @code{itsdangerous} Python -package: +Alternatively, @var{expr} may be a G-expression, in which case it is used +as a build program passed to @code{gexp->derivation} +(@pxref{G-Expressions}). -@example -guix import pypi itsdangerous -@end example +Lastly, @var{expr} may refer to a zero-argument monadic procedure +(@pxref{The Store Monad}). The procedure must return a derivation as a +monadic value, which is then passed through @code{run-with-store}. -@table @code -@item --recursive -@itemx -r -Traverse the dependency graph of the given upstream package recursively -and generate package expressions for all those packages that are not yet -in Guix. -@end table +@item --source +@itemx -S +Build the source derivations of the packages, rather than the packages +themselves. -@item gem -@cindex gem -Import metadata from @uref{https://rubygems.org/, RubyGems}. Information -is taken from the JSON-formatted description available at -@code{rubygems.org} and includes most relevant information, including -runtime dependencies. There are some caveats, however. The metadata -doesn't distinguish between synopses and descriptions, so the same string -is used for both fields. Additionally, the details of non-Ruby -dependencies required to build native extensions is unavailable and left -as an exercise to the packager. +For instance, @code{guix build -S gcc} returns something like +@file{/gnu/store/@dots{}-gcc-4.7.2.tar.bz2}, which is the GCC +source tarball. -The command below imports metadata for the @code{rails} Ruby package: +The returned source tarball is the result of applying any patches and +code snippets specified in the package @code{origin} (@pxref{Defining +Packages}). -@example -guix import gem rails -@end example +@item --sources +Fetch and return the source of @var{package-or-derivation} and all their +dependencies, recursively. This is a handy way to obtain a local copy +of all the source code needed to build @var{packages}, allowing you to +eventually build them even without network access. It is an extension +of the @code{--source} option and can accept one of the following +optional argument values: @table @code -@item --recursive -@itemx -r -Traverse the dependency graph of the given upstream package recursively -and generate package expressions for all those packages that are not yet -in Guix. -@end table +@item package +This value causes the @code{--sources} option to behave in the same way +as the @code{--source} option. -@item cpan -@cindex CPAN -Import metadata from @uref{https://www.metacpan.org/, MetaCPAN}. -Information is taken from the JSON-formatted metadata provided through -@uref{https://fastapi.metacpan.org/, MetaCPAN's API} and includes most -relevant information, such as module dependencies. License information -should be checked closely. If Perl is available in the store, then the -@code{corelist} utility will be used to filter core modules out of the -list of dependencies. +@item all +Build the source derivations of all packages, including any source that +might be listed as @code{inputs}. This is the default value. -The command command below imports metadata for the @code{Acme::Boolean} -Perl module: +@example +$ guix build --sources tzdata +The following derivations will be built: + /gnu/store/@dots{}-tzdata2015b.tar.gz.drv + /gnu/store/@dots{}-tzcode2015b.tar.gz.drv +@end example + +@item transitive +Build the source derivations of all packages, as well of all transitive +inputs to the packages. This can be used e.g.@: to +prefetch package source for later offline building. @example -guix import cpan Acme::Boolean +$ guix build --sources=transitive tzdata +The following derivations will be built: + /gnu/store/@dots{}-tzcode2015b.tar.gz.drv + /gnu/store/@dots{}-findutils-4.4.2.tar.xz.drv + /gnu/store/@dots{}-grep-2.21.tar.xz.drv + /gnu/store/@dots{}-coreutils-8.23.tar.xz.drv + /gnu/store/@dots{}-make-4.1.tar.xz.drv + /gnu/store/@dots{}-bash-4.3.tar.xz.drv +@dots{} @end example -@item cran -@cindex CRAN -@cindex Bioconductor -Import metadata from @uref{https://cran.r-project.org/, CRAN}, the -central repository for the @uref{http://r-project.org, GNU@tie{}R -statistical and graphical environment}. +@end table -Information is extracted from the @code{DESCRIPTION} file of the package. +@item --system=@var{system} +@itemx -s @var{system} +Attempt to build for @var{system}---e.g., @code{i686-linux}---instead of +the system type of the build host. -The command command below imports metadata for the @code{Cairo} -R package: +@quotation Note +The @code{--system} flag is for @emph{native} compilation and must not +be confused with cross-compilation. See @code{--target} below for +information on cross-compilation. +@end quotation -@example -guix import cran Cairo -@end example +An example use of this is on Linux-based systems, which can emulate +different personalities. For instance, passing +@code{--system=i686-linux} on an @code{x86_64-linux} system or +@code{--system=armhf-linux} on an @code{aarch64-linux} system allows you +to build packages in a complete 32-bit environment. -When @code{--recursive} is added, the importer will traverse the -dependency graph of the given upstream package recursively and generate -package expressions for all those packages that are not yet in Guix. +@quotation Note +Building for an @code{armhf-linux} system is unconditionally enabled on +@code{aarch64-linux} machines, although certain aarch64 chipsets do not +allow for this functionality, notably the ThunderX. +@end quotation -When @code{--archive=bioconductor} is added, metadata is imported from -@uref{https://www.bioconductor.org/, Bioconductor}, a repository of R -packages for for the analysis and comprehension of high-throughput -genomic data in bioinformatics. +Similarly, when transparent emulation with QEMU and @code{binfmt_misc} +is enabled (@pxref{Virtualization Services, +@code{qemu-binfmt-service-type}}), you can build for any system for +which a QEMU @code{binfmt_misc} handler is installed. -Information is extracted from the @code{DESCRIPTION} file of a package -published on the web interface of the Bioconductor SVN repository. +Builds for a system other than that of the machine you are using can +also be offloaded to a remote machine of the right architecture. +@xref{Daemon Offload Setup}, for more information on offloading. -The command below imports metadata for the @code{GenomicRanges} -R package: +@item --target=@var{triplet} +@cindex cross-compilation +Cross-build for @var{triplet}, which must be a valid GNU triplet, such +as @code{"mips64el-linux-gnu"} (@pxref{Specifying target triplets, GNU +configuration triplets,, autoconf, Autoconf}). + +@anchor{build-check} +@item --check +@cindex determinism, checking +@cindex reproducibility, checking +Rebuild @var{package-or-derivation}, which are already available in the +store, and raise an error if the build results are not bit-for-bit +identical. -@example -guix import cran --archive=bioconductor GenomicRanges -@end example +This mechanism allows you to check whether previously installed +substitutes are genuine (@pxref{Substitutes}), or whether the build result +of a package is deterministic. @xref{Invoking guix challenge}, for more +background information and tools. -@item texlive -@cindex TeX Live -@cindex CTAN -Import metadata from @uref{http://www.ctan.org/, CTAN}, the -comprehensive TeX archive network for TeX packages that are part of the -@uref{https://www.tug.org/texlive/, TeX Live distribution}. +When used in conjunction with @option{--keep-failed}, the differing +output is kept in the store, under @file{/gnu/store/@dots{}-check}. +This makes it easy to look for differences between the two results. -Information about the package is obtained through the XML API provided -by CTAN, while the source code is downloaded from the SVN repository of -the Tex Live project. This is done because the CTAN does not keep -versioned archives. +@item --repair +@cindex repairing store items +@cindex corruption, recovering from +Attempt to repair the specified store items, if they are corrupt, by +re-downloading or rebuilding them. -The command command below imports metadata for the @code{fontspec} -TeX package: +This operation is not atomic and thus restricted to @code{root}. -@example -guix import texlive fontspec -@end example +@item --derivations +@itemx -d +Return the derivation paths, not the output paths, of the given +packages. -When @code{--archive=DIRECTORY} is added, the source code is downloaded -not from the @file{latex} sub-directory of the @file{texmf-dist/source} -tree in the TeX Live SVN repository, but from the specified sibling -directory under the same root. +@item --root=@var{file} +@itemx -r @var{file} +@cindex GC roots, adding +@cindex garbage collector roots, adding +Make @var{file} a symlink to the result, and register it as a garbage +collector root. -The command below imports metadata for the @code{ifxetex} package from -CTAN while fetching the sources from the directory -@file{texmf/source/generic}: +Consequently, the results of this @command{guix build} invocation are +protected from garbage collection until @var{file} is removed. When +that option is omitted, build results are eligible for garbage +collection as soon as the build completes. @xref{Invoking guix gc}, for +more on GC roots. -@example -guix import texlive --archive=generic ifxetex -@end example +@item --log-file +@cindex build logs, access +Return the build log file names or URLs for the given +@var{package-or-derivation}, or raise an error if build logs are +missing. -@item json -@cindex JSON, import -Import package metadata from a local JSON file. Consider the following -example package definition in JSON format: +This works regardless of how packages or derivations are specified. For +instance, the following invocations are equivalent: @example -@{ - "name": "hello", - "version": "2.10", - "source": "mirror://gnu/hello/hello-2.10.tar.gz", - "build-system": "gnu", - "home-page": "https://www.gnu.org/software/hello/", - "synopsis": "Hello, GNU world: An example GNU package", - "description": "GNU Hello prints a greeting.", - "license": "GPL-3.0+", - "native-inputs": ["gcc@@6"] -@} +guix build --log-file `guix build -d guile` +guix build --log-file `guix build guile` +guix build --log-file guile +guix build --log-file -e '(@@ (gnu packages guile) guile-2.0)' @end example -The field names are the same as for the @code{} record -(@xref{Defining Packages}). References to other packages are provided -as JSON lists of quoted package specification strings such as -@code{guile} or @code{guile@@2.0}. +If a log is unavailable locally, and unless @code{--no-substitutes} is +passed, the command looks for a corresponding log on one of the +substitute servers (as specified with @code{--substitute-urls}.) -The importer also supports a more explicit source definition using the -common fields for @code{} records: +So for instance, imagine you want to see the build log of GDB on MIPS, +but you are actually on an @code{x86_64} machine: @example -@{ - @dots{} - "source": @{ - "method": "url-fetch", - "uri": "mirror://gnu/hello/hello-2.10.tar.gz", - "sha256": @{ - "base32": "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i" - @} - @} - @dots{} -@} +$ guix build --log-file gdb -s mips64el-linux +https://@value{SUBSTITUTE-SERVER}/log/@dots{}-gdb-7.10 @end example -The command below reads metadata from the JSON file @code{hello.json} -and outputs a package expression: +You can freely access a huge library of build logs! +@end table -@example -guix import json hello.json -@end example +@node Debugging Build Failures +@subsection Debugging Build Failures -@item nix -Import metadata from a local copy of the source of the -@uref{http://nixos.org/nixpkgs/, Nixpkgs distribution}@footnote{This -relies on the @command{nix-instantiate} command of -@uref{http://nixos.org/nix/, Nix}.}. Package definitions in Nixpkgs are -typically written in a mixture of Nix-language and Bash code. This -command only imports the high-level package structure that is written in -the Nix language. It normally includes all the basic fields of a -package definition. +@cindex build failures, debugging +When defining a new package (@pxref{Defining Packages}), you will +probably find yourself spending some time debugging and tweaking the +build until it succeeds. To do that, you need to operate the build +commands yourself in an environment as close as possible to the one the +build daemon uses. -When importing a GNU package, the synopsis and descriptions are replaced -by their canonical upstream variant. +To that end, the first thing to do is to use the @option{--keep-failed} +or @option{-K} option of @command{guix build}, which will keep the +failed build tree in @file{/tmp} or whatever directory you specified as +@code{TMPDIR} (@pxref{Invoking guix build, @code{--keep-failed}}). -Usually, you will first need to do: +From there on, you can @command{cd} to the failed build tree and source +the @file{environment-variables} file, which contains all the +environment variable definitions that were in place when the build +failed. So let's say you're debugging a build failure in package +@code{foo}; a typical session would look like this: @example -export NIX_REMOTE=daemon +$ guix build foo -K +@dots{} @i{build fails} +$ cd /tmp/guix-build-foo.drv-0 +$ source ./environment-variables +$ cd foo-1.2 @end example -@noindent -so that @command{nix-instantiate} does not try to open the Nix database. +Now, you can invoke commands as if you were the daemon (almost) and +troubleshoot your build process. -As an example, the command below imports the package definition of -LibreOffice (more precisely, it imports the definition of the package -bound to the @code{libreoffice} top-level attribute): +Sometimes it happens that, for example, a package's tests pass when you +run them manually but they fail when the daemon runs them. This can +happen because the daemon runs builds in containers where, unlike in our +environment above, network access is missing, @file{/bin/sh} does not +exist, etc. (@pxref{Build Environment Setup}). + +In such cases, you may need to run inspect the build process from within +a container similar to the one the build daemon creates: @example -guix import nix ~/path/to/nixpkgs libreoffice +$ guix build -K foo +@dots{} +$ cd /tmp/guix-build-foo.drv-0 +$ guix environment --no-grafts -C foo --ad-hoc strace gdb +[env]# source ./environment-variables +[env]# cd foo-1.2 @end example -@item hackage -@cindex hackage -Import metadata from the Haskell community's central package archive -@uref{https://hackage.haskell.org/, Hackage}. Information is taken from -Cabal files and includes all the relevant information, including package -dependencies. - -Specific command-line options are: - -@table @code -@item --stdin -@itemx -s -Read a Cabal file from standard input. -@item --no-test-dependencies -@itemx -t -Do not include dependencies required only by the test suites. -@item --cabal-environment=@var{alist} -@itemx -e @var{alist} -@var{alist} is a Scheme alist defining the environment in which the -Cabal conditionals are evaluated. The accepted keys are: @code{os}, -@code{arch}, @code{impl} and a string representing the name of a flag. -The value associated with a flag has to be either the symbol -@code{true} or @code{false}. The value associated with other keys -has to conform to the Cabal file format definition. The default value -associated with the keys @code{os}, @code{arch} and @code{impl} is -@samp{linux}, @samp{x86_64} and @samp{ghc}, respectively. -@item --recursive -@itemx -r -Traverse the dependency graph of the given upstream package recursively -and generate package expressions for all those packages that are not yet -in Guix. -@end table +Here, @command{guix environment -C} creates a container and spawns a new +shell in it (@pxref{Invoking guix environment}). The @command{--ad-hoc +strace gdb} part adds the @command{strace} and @command{gdb} commands to +the container, which would may find handy while debugging. The +@option{--no-grafts} option makes sure we get the exact same +environment, with ungrafted packages (@pxref{Security Updates}, for more +info on grafts). -The command below imports metadata for the latest version of the -@code{HTTP} Haskell package without including test dependencies and -specifying the value of the flag @samp{network-uri} as @code{false}: +To get closer to a container like that used by the build daemon, we can +remove @file{/bin/sh}: @example -guix import hackage -t -e "'((\"network-uri\" . false))" HTTP +[env]# rm /bin/sh @end example -A specific package version may optionally be specified by following the -package name by an at-sign and a version number as in the following example: +(Don't worry, this is harmless: this is all happening in the throw-away +container created by @command{guix environment}.) + +The @command{strace} command is probably not in the search path, but we +can run: @example -guix import hackage mtl@@2.1.3.1 +[env]# $GUIX_ENVIRONMENT/bin/strace -f -o log make check @end example -@item stackage -@cindex stackage -The @code{stackage} importer is a wrapper around the @code{hackage} one. -It takes a package name, looks up the package version included in a -long-term support (LTS) @uref{https://www.stackage.org, Stackage} -release and uses the @code{hackage} importer to retrieve its metadata. -Note that it is up to you to select an LTS release compatible with the -GHC compiler used by Guix. +In this way, not only you will have reproduced the environment variables +the daemon uses, you will also be running the build process in a container +similar to the one the daemon uses. -Specific command-line options are: -@table @code -@item --no-test-dependencies -@itemx -t -Do not include dependencies required only by the test suites. -@item --lts-version=@var{version} -@itemx -l @var{version} -@var{version} is the desired LTS release version. If omitted the latest -release is used. -@item --recursive -@itemx -r -Traverse the dependency graph of the given upstream package recursively -and generate package expressions for all those packages that are not yet -in Guix. -@end table +@node Invoking guix edit +@section Invoking @command{guix edit} -The command below imports metadata for the @code{HTTP} Haskell package -included in the LTS Stackage release version 7.18: +@cindex @command{guix edit} +@cindex package definition, editing +So many packages, so many source files! The @command{guix edit} command +facilitates the life of users and packagers by pointing their editor at +the source file containing the definition of the specified packages. +For instance: @example -guix import stackage --lts-version=7.18 HTTP +guix edit gcc@@4.9 vim @end example -@item elpa -@cindex elpa -Import metadata from an Emacs Lisp Package Archive (ELPA) package -repository (@pxref{Packages,,, emacs, The GNU Emacs Manual}). +@noindent +launches the program specified in the @code{VISUAL} or in the +@code{EDITOR} environment variable to view the recipe of GCC@tie{}4.9.3 +and that of Vim. -Specific command-line options are: +If you are using a Guix Git checkout (@pxref{Building from Git}), or +have created your own packages on @code{GUIX_PACKAGE_PATH} +(@pxref{Package Modules}), you will be able to edit the package +recipes. In other cases, you will be able to examine the read-only recipes +for packages currently in the store. -@table @code -@item --archive=@var{repo} -@itemx -a @var{repo} -@var{repo} identifies the archive repository from which to retrieve the -information. Currently the supported repositories and their identifiers -are: -@itemize - -@item -@uref{http://elpa.gnu.org/packages, GNU}, selected by the @code{gnu} -identifier. This is the default. -Packages from @code{elpa.gnu.org} are signed with one of the keys -contained in the GnuPG keyring at -@file{share/emacs/25.1/etc/package-keyring.gpg} (or similar) in the -@code{emacs} package (@pxref{Package Installation, ELPA package -signatures,, emacs, The GNU Emacs Manual}). +@node Invoking guix download +@section Invoking @command{guix download} -@item -@uref{http://stable.melpa.org/packages, MELPA-Stable}, selected by the -@code{melpa-stable} identifier. +@cindex @command{guix download} +@cindex downloading package sources +When writing a package definition, developers typically need to download +a source tarball, compute its SHA256 hash, and write that +hash in the package definition (@pxref{Defining Packages}). The +@command{guix download} tool helps with this task: it downloads a file +from the given URI, adds it to the store, and prints both its file name +in the store and its SHA256 hash. -@item -@uref{http://melpa.org/packages, MELPA}, selected by the @code{melpa} -identifier. -@end itemize +The fact that the downloaded file is added to the store saves bandwidth: +when the developer eventually tries to build the newly defined package +with @command{guix build}, the source tarball will not have to be +downloaded again because it is already in the store. It is also a +convenient way to temporarily stash files, which may be deleted +eventually (@pxref{Invoking guix gc}). -@item --recursive -@itemx -r -Traverse the dependency graph of the given upstream package recursively -and generate package expressions for all those packages that are not yet -in Guix. -@end table +The @command{guix download} command supports the same URIs as used in +package definitions. In particular, it supports @code{mirror://} URIs. +@code{https} URIs (HTTP over TLS) are supported @emph{provided} the +Guile bindings for GnuTLS are available in the user's environment; when +they are not available, an error is raised. @xref{Guile Preparations, +how to install the GnuTLS bindings for Guile,, gnutls-guile, +GnuTLS-Guile}, for more information. -@item crate -@cindex crate -Import metadata from the crates.io Rust package repository -@uref{https://crates.io, crates.io}. +@command{guix download} verifies HTTPS server certificates by loading +the certificates of X.509 authorities from the directory pointed to by +the @code{SSL_CERT_DIR} environment variable (@pxref{X.509 +Certificates}), unless @option{--no-check-certificate} is used. -@item opam -@cindex OPAM -@cindex OCaml -Import metadata from the @uref{https://opam.ocaml.org/, OPAM} package -repository used by the OCaml community. -@end table +The following options are available: -The structure of the @command{guix import} code is modular. It would be -useful to have more importers for other package formats, and your help -is welcome here (@pxref{Contributing}). +@table @code +@item --format=@var{fmt} +@itemx -f @var{fmt} +Write the hash in the format specified by @var{fmt}. For more +information on the valid values for @var{fmt}, @pxref{Invoking guix hash}. -@node Invoking guix refresh -@section Invoking @command{guix refresh} +@item --no-check-certificate +Do not validate the X.509 certificates of HTTPS servers. -@cindex @command {guix refresh} -The primary audience of the @command{guix refresh} command is developers -of the GNU software distribution. By default, it reports any packages -provided by the distribution that are outdated compared to the latest -upstream version, like this: +When using this option, you have @emph{absolutely no guarantee} that you +are communicating with the authentic server responsible for the given +URL, which makes you vulnerable to ``man-in-the-middle'' attacks. -@example -$ guix refresh -gnu/packages/gettext.scm:29:13: gettext would be upgraded from 0.18.1.1 to 0.18.2.1 -gnu/packages/glib.scm:77:12: glib would be upgraded from 2.34.3 to 2.37.0 -@end example +@item --output=@var{file} +@itemx -o @var{file} +Save the downloaded file to @var{file} instead of adding it to the +store. +@end table -Alternately, one can specify packages to consider, in which case a -warning is emitted for packages that lack an updater: +@node Invoking guix hash +@section Invoking @command{guix hash} + +@cindex @command{guix hash} +The @command{guix hash} command computes the SHA256 hash of a file. +It is primarily a convenience tool for anyone contributing to the +distribution: it computes the cryptographic hash of a file, which can be +used in the definition of a package (@pxref{Defining Packages}). + +The general syntax is: @example -$ guix refresh coreutils guile guile-ssh -gnu/packages/ssh.scm:205:2: warning: no updater for guile-ssh -gnu/packages/guile.scm:136:12: guile would be upgraded from 2.0.12 to 2.0.13 +guix hash @var{option} @var{file} @end example -@command{guix refresh} browses the upstream repository of each package and determines -the highest version number of the releases therein. The command -knows how to update specific types of packages: GNU packages, ELPA -packages, etc.---see the documentation for @option{--type} below. There -are many packages, though, for which it lacks a method to determine -whether a new upstream release is available. However, the mechanism is -extensible, so feel free to get in touch with us to add a new method! +When @var{file} is @code{-} (a hyphen), @command{guix hash} computes the +hash of data read from standard input. @command{guix hash} has the +following options: @table @code +@item --format=@var{fmt} +@itemx -f @var{fmt} +Write the hash in the format specified by @var{fmt}. + +Supported formats: @code{nix-base32}, @code{base32}, @code{base16} +(@code{hex} and @code{hexadecimal} can be used as well). + +If the @option{--format} option is not specified, @command{guix hash} +will output the hash in @code{nix-base32}. This representation is used +in the definitions of packages. + @item --recursive -Consider the packages specified, and all the packages upon which they depend. +@itemx -r +Compute the hash on @var{file} recursively. + +In this case, the hash is computed on an archive containing @var{file}, +including its children if it is a directory. Some of the metadata of +@var{file} is part of the archive; for instance, when @var{file} is a +regular file, the hash is different depending on whether @var{file} is +executable or not. Metadata such as time stamps has no impact on the +hash (@pxref{Invoking guix archive}). +@c FIXME: Replace xref above with xref to an ``Archive'' section when +@c it exists. + +@item --exclude-vcs +@itemx -x +When combined with @option{--recursive}, exclude version control system +directories (@file{.bzr}, @file{.git}, @file{.hg}, etc.) + +@vindex git-fetch +As an example, here is how you would compute the hash of a Git checkout, +which is useful when using the @code{git-fetch} method (@pxref{origin +Reference}): @example -$ guix refresh --recursive coreutils -gnu/packages/acl.scm:35:2: warning: no updater for acl -gnu/packages/m4.scm:30:12: info: 1.4.18 is already the latest version of m4 -gnu/packages/xml.scm:68:2: warning: no updater for expat -gnu/packages/multiprecision.scm:40:12: info: 6.1.2 is already the latest version of gmp -@dots{} +$ git clone http://example.org/foo.git +$ cd foo +$ guix hash -rx . @end example - @end table -Sometimes the upstream name differs from the package name used in Guix, -and @command{guix refresh} needs a little help. Most updaters honor the -@code{upstream-name} property in package definitions, which can be used -to that effect: +@node Invoking guix import +@section Invoking @command{guix import} + +@cindex importing packages +@cindex package import +@cindex package conversion +@cindex Invoking @command{guix import} +The @command{guix import} command is useful for people who would like to +add a package to the distribution with as little work as +possible---a legitimate demand. The command knows of a few +repositories from which it can ``import'' package metadata. The result +is a package definition, or a template thereof, in the format we know +(@pxref{Defining Packages}). + +The general syntax is: @example -(define-public network-manager - (package - (name "network-manager") - ;; @dots{} - (properties '((upstream-name . "NetworkManager"))))) +guix import @var{importer} @var{options}@dots{} @end example -When passed @code{--update}, it modifies distribution source files to -update the version numbers and source tarball hashes of those package -recipes (@pxref{Defining Packages}). This is achieved by downloading -each package's latest source tarball and its associated OpenPGP -signature, authenticating the downloaded tarball against its signature -using @command{gpg}, and finally computing its hash. When the public -key used to sign the tarball is missing from the user's keyring, an -attempt is made to automatically retrieve it from a public key server; -when this is successful, the key is added to the user's keyring; otherwise, -@command{guix refresh} reports an error. - -The following options are supported: +@var{importer} specifies the source from which to import package +metadata, and @var{options} specifies a package identifier and other +options specific to @var{importer}. Currently, the available +``importers'' are: @table @code +@item gnu +Import metadata for the given GNU package. This provides a template +for the latest version of that GNU package, including the hash of its +source tarball, and its canonical synopsis and description. -@item --expression=@var{expr} -@itemx -e @var{expr} -Consider the package @var{expr} evaluates to. +Additional information such as the package dependencies and its +license needs to be figured out manually. -This is useful to precisely refer to a package, as in this example: +For example, the following command returns a package definition for +GNU@tie{}Hello: @example -guix refresh -l -e '(@@@@ (gnu packages commencement) glibc-final)' +guix import gnu hello @end example -This command lists the dependents of the ``final'' libc (essentially all -the packages.) +Specific command-line options are: + +@table @code +@item --key-download=@var{policy} +As for @code{guix refresh}, specify the policy to handle missing OpenPGP +keys when verifying the package signature. @xref{Invoking guix +refresh, @code{--key-download}}. +@end table + +@item pypi +@cindex pypi +Import metadata from the @uref{https://pypi.python.org/, Python Package +Index}. Information is taken from the JSON-formatted description +available at @code{pypi.python.org} and usually includes all the relevant +information, including package dependencies. For maximum efficiency, it +is recommended to install the @command{unzip} utility, so that the +importer can unzip Python wheels and gather data from them. -@item --update -@itemx -u -Update distribution source files (package recipes) in place. This is -usually run from a checkout of the Guix source tree (@pxref{Running -Guix Before It Is Installed}): +The command below imports metadata for the @code{itsdangerous} Python +package: @example -$ ./pre-inst-env guix refresh -s non-core -u +guix import pypi itsdangerous @end example -@xref{Defining Packages}, for more information on package definitions. - -@item --select=[@var{subset}] -@itemx -s @var{subset} -Select all the packages in @var{subset}, one of @code{core} or -@code{non-core}. - -The @code{core} subset refers to all the packages at the core of the -distribution---i.e., packages that are used to build ``everything -else''. This includes GCC, libc, Binutils, Bash, etc. Usually, -changing one of these packages in the distribution entails a rebuild of -all the others. Thus, such updates are an inconvenience to users in -terms of build time or bandwidth used to achieve the upgrade. +@table @code +@item --recursive +@itemx -r +Traverse the dependency graph of the given upstream package recursively +and generate package expressions for all those packages that are not yet +in Guix. +@end table -The @code{non-core} subset refers to the remaining packages. It is -typically useful in cases where an update of the core packages would be -inconvenient. +@item gem +@cindex gem +Import metadata from @uref{https://rubygems.org/, RubyGems}. Information +is taken from the JSON-formatted description available at +@code{rubygems.org} and includes most relevant information, including +runtime dependencies. There are some caveats, however. The metadata +doesn't distinguish between synopses and descriptions, so the same string +is used for both fields. Additionally, the details of non-Ruby +dependencies required to build native extensions is unavailable and left +as an exercise to the packager. -@item --manifest=@var{file} -@itemx -m @var{file} -Select all the packages from the manifest in @var{file}. This is useful to -check if any packages of the user manifest can be updated. +The command below imports metadata for the @code{rails} Ruby package: -@item --type=@var{updater} -@itemx -t @var{updater} -Select only packages handled by @var{updater} (may be a comma-separated -list of updaters). Currently, @var{updater} may be one of: +@example +guix import gem rails +@end example @table @code -@item gnu -the updater for GNU packages; -@item gnome -the updater for GNOME packages; -@item kde -the updater for KDE packages; -@item xorg -the updater for X.org packages; -@item kernel.org -the updater for packages hosted on kernel.org; -@item elpa -the updater for @uref{http://elpa.gnu.org/, ELPA} packages; -@item cran -the updater for @uref{https://cran.r-project.org/, CRAN} packages; -@item bioconductor -the updater for @uref{https://www.bioconductor.org/, Bioconductor} R packages; -@item cpan -the updater for @uref{http://www.cpan.org/, CPAN} packages; -@item pypi -the updater for @uref{https://pypi.python.org, PyPI} packages. -@item gem -the updater for @uref{https://rubygems.org, RubyGems} packages. -@item github -the updater for @uref{https://github.com, GitHub} packages. -@item hackage -the updater for @uref{https://hackage.haskell.org, Hackage} packages. -@item stackage -the updater for @uref{https://www.stackage.org, Stackage} packages. -@item crate -the updater for @uref{https://crates.io, Crates} packages. +@item --recursive +@itemx -r +Traverse the dependency graph of the given upstream package recursively +and generate package expressions for all those packages that are not yet +in Guix. @end table -For instance, the following command only checks for updates of Emacs -packages hosted at @code{elpa.gnu.org} and for updates of CRAN packages: +@item cpan +@cindex CPAN +Import metadata from @uref{https://www.metacpan.org/, MetaCPAN}. +Information is taken from the JSON-formatted metadata provided through +@uref{https://fastapi.metacpan.org/, MetaCPAN's API} and includes most +relevant information, such as module dependencies. License information +should be checked closely. If Perl is available in the store, then the +@code{corelist} utility will be used to filter core modules out of the +list of dependencies. + +The command command below imports metadata for the @code{Acme::Boolean} +Perl module: @example -$ guix refresh --type=elpa,cran -gnu/packages/statistics.scm:819:13: r-testthat would be upgraded from 0.10.0 to 0.11.0 -gnu/packages/emacs.scm:856:13: emacs-auctex would be upgraded from 11.88.6 to 11.88.9 +guix import cpan Acme::Boolean @end example -@end table +@item cran +@cindex CRAN +@cindex Bioconductor +Import metadata from @uref{https://cran.r-project.org/, CRAN}, the +central repository for the @uref{http://r-project.org, GNU@tie{}R +statistical and graphical environment}. -In addition, @command{guix refresh} can be passed one or more package -names, as in this example: +Information is extracted from the @code{DESCRIPTION} file of the package. + +The command command below imports metadata for the @code{Cairo} +R package: @example -$ ./pre-inst-env guix refresh -u emacs idutils gcc@@4.8 +guix import cran Cairo @end example -@noindent -The command above specifically updates the @code{emacs} and -@code{idutils} packages. The @code{--select} option would have no -effect in this case. - -When considering whether to upgrade a package, it is sometimes -convenient to know which packages would be affected by the upgrade and -should be checked for compatibility. For this the following option may -be used when passing @command{guix refresh} one or more package names: +When @code{--recursive} is added, the importer will traverse the +dependency graph of the given upstream package recursively and generate +package expressions for all those packages that are not yet in Guix. -@table @code +When @code{--archive=bioconductor} is added, metadata is imported from +@uref{https://www.bioconductor.org/, Bioconductor}, a repository of R +packages for for the analysis and comprehension of high-throughput +genomic data in bioinformatics. -@item --list-updaters -@itemx -L -List available updaters and exit (see @option{--type} above.) +Information is extracted from the @code{DESCRIPTION} file of a package +published on the web interface of the Bioconductor SVN repository. -For each updater, display the fraction of packages it covers; at the -end, display the fraction of packages covered by all these updaters. +The command below imports metadata for the @code{GenomicRanges} +R package: -@item --list-dependent -@itemx -l -List top-level dependent packages that would need to be rebuilt as a -result of upgrading one or more packages. +@example +guix import cran --archive=bioconductor GenomicRanges +@end example -@xref{Invoking guix graph, the @code{reverse-package} type of -@command{guix graph}}, for information on how to visualize the list of -dependents of a package. +@item texlive +@cindex TeX Live +@cindex CTAN +Import metadata from @uref{http://www.ctan.org/, CTAN}, the +comprehensive TeX archive network for TeX packages that are part of the +@uref{https://www.tug.org/texlive/, TeX Live distribution}. -@end table +Information about the package is obtained through the XML API provided +by CTAN, while the source code is downloaded from the SVN repository of +the Tex Live project. This is done because the CTAN does not keep +versioned archives. -Be aware that the @code{--list-dependent} option only -@emph{approximates} the rebuilds that would be required as a result of -an upgrade. More rebuilds might be required under some circumstances. +The command command below imports metadata for the @code{fontspec} +TeX package: @example -$ guix refresh --list-dependent flex -Building the following 120 packages would ensure 213 dependent packages are rebuilt: -hop@@2.4.0 geiser@@0.4 notmuch@@0.18 mu@@0.9.9.5 cflow@@1.4 idutils@@4.6 @dots{} +guix import texlive fontspec @end example -The command above lists a set of packages that could be built to check -for compatibility with an upgraded @code{flex} package. - -@table @code +When @code{--archive=DIRECTORY} is added, the source code is downloaded +not from the @file{latex} sub-directory of the @file{texmf-dist/source} +tree in the TeX Live SVN repository, but from the specified sibling +directory under the same root. -@item --list-transitive -List all the packages which one or more packages depend upon. +The command below imports metadata for the @code{ifxetex} package from +CTAN while fetching the sources from the directory +@file{texmf/source/generic}: @example -$ guix refresh --list-transitive flex -flex@@2.6.4 depends on the following 25 packages: perl@@5.28.0 help2man@@1.47.6 -bison@@3.0.5 indent@@2.2.10 tar@@1.30 gzip@@1.9 bzip2@@1.0.6 xz@@5.2.4 file@@5.33 @dots{} +guix import texlive --archive=generic ifxetex @end example -@end table +@item json +@cindex JSON, import +Import package metadata from a local JSON file. Consider the following +example package definition in JSON format: -The command above lists a set of packages which, when changed, would cause -@code{flex} to be rebuilt. +@example +@{ + "name": "hello", + "version": "2.10", + "source": "mirror://gnu/hello/hello-2.10.tar.gz", + "build-system": "gnu", + "home-page": "https://www.gnu.org/software/hello/", + "synopsis": "Hello, GNU world: An example GNU package", + "description": "GNU Hello prints a greeting.", + "license": "GPL-3.0+", + "native-inputs": ["gcc@@6"] +@} +@end example -The following options can be used to customize GnuPG operation: +The field names are the same as for the @code{} record +(@xref{Defining Packages}). References to other packages are provided +as JSON lists of quoted package specification strings such as +@code{guile} or @code{guile@@2.0}. -@table @code +The importer also supports a more explicit source definition using the +common fields for @code{} records: -@item --gpg=@var{command} -Use @var{command} as the GnuPG 2.x command. @var{command} is searched -for in @code{$PATH}. +@example +@{ + @dots{} + "source": @{ + "method": "url-fetch", + "uri": "mirror://gnu/hello/hello-2.10.tar.gz", + "sha256": @{ + "base32": "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i" + @} + @} + @dots{} +@} +@end example -@item --keyring=@var{file} -Use @var{file} as the keyring for upstream keys. @var{file} must be in the -@dfn{keybox format}. Keybox files usually have a name ending in @file{.kbx} -and the GNU@tie{}Privacy Guard (GPG) can manipulate these files -(@pxref{kbxutil, @command{kbxutil},, gnupg, Using the GNU Privacy Guard}, for -information on a tool to manipulate keybox files). +The command below reads metadata from the JSON file @code{hello.json} +and outputs a package expression: + +@example +guix import json hello.json +@end example -When this option is omitted, @command{guix refresh} uses -@file{~/.config/guix/upstream/trustedkeys.kbx} as the keyring for upstream -signing keys. OpenPGP signatures are checked against keys from this keyring; -missing keys are downloaded to this keyring as well (see -@option{--key-download} below.) +@item nix +Import metadata from a local copy of the source of the +@uref{http://nixos.org/nixpkgs/, Nixpkgs distribution}@footnote{This +relies on the @command{nix-instantiate} command of +@uref{http://nixos.org/nix/, Nix}.}. Package definitions in Nixpkgs are +typically written in a mixture of Nix-language and Bash code. This +command only imports the high-level package structure that is written in +the Nix language. It normally includes all the basic fields of a +package definition. -You can export keys from your default GPG keyring into a keybox file using -commands like this one: +When importing a GNU package, the synopsis and descriptions are replaced +by their canonical upstream variant. + +Usually, you will first need to do: @example -gpg --export rms@@gnu.org | kbxutil --import-openpgp >> mykeyring.kbx +export NIX_REMOTE=daemon @end example -Likewise, you can fetch keys to a specific keybox file like this: +@noindent +so that @command{nix-instantiate} does not try to open the Nix database. + +As an example, the command below imports the package definition of +LibreOffice (more precisely, it imports the definition of the package +bound to the @code{libreoffice} top-level attribute): @example -gpg --no-default-keyring --keyring mykeyring.kbx \ - --recv-keys @value{OPENPGP-SIGNING-KEY-ID} +guix import nix ~/path/to/nixpkgs libreoffice @end example -@ref{GPG Configuration Options, @option{--keyring},, gnupg, Using the GNU -Privacy Guard}, for more information on GPG's @option{--keyring} option. +@item hackage +@cindex hackage +Import metadata from the Haskell community's central package archive +@uref{https://hackage.haskell.org/, Hackage}. Information is taken from +Cabal files and includes all the relevant information, including package +dependencies. -@item --key-download=@var{policy} -Handle missing OpenPGP keys according to @var{policy}, which may be one -of: +Specific command-line options are: @table @code -@item always -Always download missing OpenPGP keys from the key server, and add them -to the user's GnuPG keyring. - -@item never -Never try to download missing OpenPGP keys. Instead just bail out. - -@item interactive -When a package signed with an unknown OpenPGP key is encountered, ask -the user whether to download it or not. This is the default behavior. +@item --stdin +@itemx -s +Read a Cabal file from standard input. +@item --no-test-dependencies +@itemx -t +Do not include dependencies required only by the test suites. +@item --cabal-environment=@var{alist} +@itemx -e @var{alist} +@var{alist} is a Scheme alist defining the environment in which the +Cabal conditionals are evaluated. The accepted keys are: @code{os}, +@code{arch}, @code{impl} and a string representing the name of a flag. +The value associated with a flag has to be either the symbol +@code{true} or @code{false}. The value associated with other keys +has to conform to the Cabal file format definition. The default value +associated with the keys @code{os}, @code{arch} and @code{impl} is +@samp{linux}, @samp{x86_64} and @samp{ghc}, respectively. +@item --recursive +@itemx -r +Traverse the dependency graph of the given upstream package recursively +and generate package expressions for all those packages that are not yet +in Guix. @end table -@item --key-server=@var{host} -Use @var{host} as the OpenPGP key server when importing a public key. +The command below imports metadata for the latest version of the +@code{HTTP} Haskell package without including test dependencies and +specifying the value of the flag @samp{network-uri} as @code{false}: -@end table +@example +guix import hackage -t -e "'((\"network-uri\" . false))" HTTP +@end example -The @code{github} updater uses the -@uref{https://developer.github.com/v3/, GitHub API} to query for new -releases. When used repeatedly e.g.@: when refreshing all packages, -GitHub will eventually refuse to answer any further API requests. By -default 60 API requests per hour are allowed, and a full refresh on all -GitHub packages in Guix requires more than this. Authentication with -GitHub through the use of an API token alleviates these limits. To use -an API token, set the environment variable @code{GUIX_GITHUB_TOKEN} to a -token procured from @uref{https://github.com/settings/tokens} or -otherwise. +A specific package version may optionally be specified by following the +package name by an at-sign and a version number as in the following example: +@example +guix import hackage mtl@@2.1.3.1 +@end example -@node Invoking guix lint -@section Invoking @command{guix lint} +@item stackage +@cindex stackage +The @code{stackage} importer is a wrapper around the @code{hackage} one. +It takes a package name, looks up the package version included in a +long-term support (LTS) @uref{https://www.stackage.org, Stackage} +release and uses the @code{hackage} importer to retrieve its metadata. +Note that it is up to you to select an LTS release compatible with the +GHC compiler used by Guix. -@cindex @command{guix lint} -@cindex package, checking for errors -The @command{guix lint} command is meant to help package developers avoid -common errors and use a consistent style. It runs a number of checks on -a given set of packages in order to find common mistakes in their -definitions. Available @dfn{checkers} include (see -@code{--list-checkers} for a complete list): +Specific command-line options are: @table @code -@item synopsis -@itemx description -Validate certain typographical and stylistic rules about package -descriptions and synopses. +@item --no-test-dependencies +@itemx -t +Do not include dependencies required only by the test suites. +@item --lts-version=@var{version} +@itemx -l @var{version} +@var{version} is the desired LTS release version. If omitted the latest +release is used. +@item --recursive +@itemx -r +Traverse the dependency graph of the given upstream package recursively +and generate package expressions for all those packages that are not yet +in Guix. +@end table -@item inputs-should-be-native -Identify inputs that should most likely be native inputs. +The command below imports metadata for the @code{HTTP} Haskell package +included in the LTS Stackage release version 7.18: -@item source -@itemx home-page -@itemx mirror-url -@itemx github-url -@itemx source-file-name -Probe @code{home-page} and @code{source} URLs and report those that are -invalid. Suggest a @code{mirror://} URL when applicable. If the -@code{source} URL redirects to a GitHub URL, recommend usage of the GitHub -URL. Check that the source file name is meaningful, e.g.@: is not just a -version number or ``git-checkout'', without a declared @code{file-name} -(@pxref{origin Reference}). +@example +guix import stackage --lts-version=7.18 HTTP +@end example -@item source-unstable-tarball -Parse the @code{source} URL to determine if a tarball from GitHub is -autogenerated or if it is a release tarball. Unfortunately GitHub's -autogenerated tarballs are sometimes regenerated. +@item elpa +@cindex elpa +Import metadata from an Emacs Lisp Package Archive (ELPA) package +repository (@pxref{Packages,,, emacs, The GNU Emacs Manual}). -@item cve -@cindex security vulnerabilities -@cindex CVE, Common Vulnerabilities and Exposures -Report known vulnerabilities found in the Common Vulnerabilities and -Exposures (CVE) databases of the current and past year -@uref{https://nvd.nist.gov/download.cfm#CVE_FEED, published by the US -NIST}. +Specific command-line options are: -To view information about a particular vulnerability, visit pages such as: +@table @code +@item --archive=@var{repo} +@itemx -a @var{repo} +@var{repo} identifies the archive repository from which to retrieve the +information. Currently the supported repositories and their identifiers +are: +@itemize - +@item +@uref{http://elpa.gnu.org/packages, GNU}, selected by the @code{gnu} +identifier. This is the default. + +Packages from @code{elpa.gnu.org} are signed with one of the keys +contained in the GnuPG keyring at +@file{share/emacs/25.1/etc/package-keyring.gpg} (or similar) in the +@code{emacs} package (@pxref{Package Installation, ELPA package +signatures,, emacs, The GNU Emacs Manual}). -@itemize @item -@indicateurl{https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-YYYY-ABCD} +@uref{http://stable.melpa.org/packages, MELPA-Stable}, selected by the +@code{melpa-stable} identifier. + @item -@indicateurl{https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-YYYY-ABCD} +@uref{http://melpa.org/packages, MELPA}, selected by the @code{melpa} +identifier. @end itemize -@noindent -where @code{CVE-YYYY-ABCD} is the CVE identifier---e.g., -@code{CVE-2015-7554}. +@item --recursive +@itemx -r +Traverse the dependency graph of the given upstream package recursively +and generate package expressions for all those packages that are not yet +in Guix. +@end table -Package developers can specify in package recipes the -@uref{https://nvd.nist.gov/cpe.cfm,Common Platform Enumeration (CPE)} -name and version of the package when they differ from the name or version -that Guix uses, as in this example: +@item crate +@cindex crate +Import metadata from the crates.io Rust package repository +@uref{https://crates.io, crates.io}. -@example -(package - (name "grub") - ;; @dots{} - ;; CPE calls this package "grub2". - (properties '((cpe-name . "grub2") - (cpe-version . "2.3"))) -@end example +@item opam +@cindex OPAM +@cindex OCaml +Import metadata from the @uref{https://opam.ocaml.org/, OPAM} package +repository used by the OCaml community. +@end table -@c See . -Some entries in the CVE database do not specify which version of a -package they apply to, and would thus ``stick around'' forever. Package -developers who found CVE alerts and verified they can be ignored can -declare them as in this example: +The structure of the @command{guix import} code is modular. It would be +useful to have more importers for other package formats, and your help +is welcome here (@pxref{Contributing}). -@example -(package - (name "t1lib") - ;; @dots{} - ;; These CVEs no longer apply and can be safely ignored. - (properties `((lint-hidden-cve . ("CVE-2011-0433" - "CVE-2011-1553" - "CVE-2011-1554" - "CVE-2011-5244"))))) -@end example +@node Invoking guix refresh +@section Invoking @command{guix refresh} -@item formatting -Warn about obvious source code formatting issues: trailing white space, -use of tabulations, etc. -@end table +@cindex @command {guix refresh} +The primary audience of the @command{guix refresh} command is developers +of the GNU software distribution. By default, it reports any packages +provided by the distribution that are outdated compared to the latest +upstream version, like this: + +@example +$ guix refresh +gnu/packages/gettext.scm:29:13: gettext would be upgraded from 0.18.1.1 to 0.18.2.1 +gnu/packages/glib.scm:77:12: glib would be upgraded from 2.34.3 to 2.37.0 +@end example -The general syntax is: +Alternately, one can specify packages to consider, in which case a +warning is emitted for packages that lack an updater: @example -guix lint @var{options} @var{package}@dots{} +$ guix refresh coreutils guile guile-ssh +gnu/packages/ssh.scm:205:2: warning: no updater for guile-ssh +gnu/packages/guile.scm:136:12: guile would be upgraded from 2.0.12 to 2.0.13 @end example -If no package is given on the command line, then all packages are checked. -The @var{options} may be zero or more of the following: +@command{guix refresh} browses the upstream repository of each package and determines +the highest version number of the releases therein. The command +knows how to update specific types of packages: GNU packages, ELPA +packages, etc.---see the documentation for @option{--type} below. There +are many packages, though, for which it lacks a method to determine +whether a new upstream release is available. However, the mechanism is +extensible, so feel free to get in touch with us to add a new method! @table @code -@item --list-checkers -@itemx -l -List and describe all the available checkers that will be run on packages -and exit. - -@item --checkers -@itemx -c -Only enable the checkers specified in a comma-separated list using the -names returned by @code{--list-checkers}. - -@end table - -@node Invoking guix size -@section Invoking @command{guix size} - -@cindex size -@cindex package size -@cindex closure -@cindex @command{guix size} -The @command{guix size} command helps package developers profile the -disk usage of packages. It is easy to overlook the impact of an -additional dependency added to a package, or the impact of using a -single output for a package that could easily be split (@pxref{Packages -with Multiple Outputs}). Such are the typical issues that -@command{guix size} can highlight. -The command can be passed one or more package specifications -such as @code{gcc@@4.8} -or @code{guile:debug}, or a file name in the store. Consider this -example: +@item --recursive +Consider the packages specified, and all the packages upon which they depend. @example -$ guix size coreutils -store item total self -/gnu/store/@dots{}-gcc-5.5.0-lib 60.4 30.1 38.1% -/gnu/store/@dots{}-glibc-2.27 30.3 28.8 36.6% -/gnu/store/@dots{}-coreutils-8.28 78.9 15.0 19.0% -/gnu/store/@dots{}-gmp-6.1.2 63.1 2.7 3.4% -/gnu/store/@dots{}-bash-static-4.4.12 1.5 1.5 1.9% -/gnu/store/@dots{}-acl-2.2.52 61.1 0.4 0.5% -/gnu/store/@dots{}-attr-2.4.47 60.6 0.2 0.3% -/gnu/store/@dots{}-libcap-2.25 60.5 0.2 0.2% -total: 78.9 MiB +$ guix refresh --recursive coreutils +gnu/packages/acl.scm:35:2: warning: no updater for acl +gnu/packages/m4.scm:30:12: info: 1.4.18 is already the latest version of m4 +gnu/packages/xml.scm:68:2: warning: no updater for expat +gnu/packages/multiprecision.scm:40:12: info: 6.1.2 is already the latest version of gmp +@dots{} @end example -@cindex closure -The store items listed here constitute the @dfn{transitive closure} of -Coreutils---i.e., Coreutils and all its dependencies, recursively---as -would be returned by: +@end table + +Sometimes the upstream name differs from the package name used in Guix, +and @command{guix refresh} needs a little help. Most updaters honor the +@code{upstream-name} property in package definitions, which can be used +to that effect: @example -$ guix gc -R /gnu/store/@dots{}-coreutils-8.23 +(define-public network-manager + (package + (name "network-manager") + ;; @dots{} + (properties '((upstream-name . "NetworkManager"))))) @end example -Here the output shows three columns next to store items. The first column, -labeled ``total'', shows the size in mebibytes (MiB) of the closure of -the store item---that is, its own size plus the size of all its -dependencies. The next column, labeled ``self'', shows the size of the -item itself. The last column shows the ratio of the size of the item -itself to the space occupied by all the items listed here. +When passed @code{--update}, it modifies distribution source files to +update the version numbers and source tarball hashes of those package +recipes (@pxref{Defining Packages}). This is achieved by downloading +each package's latest source tarball and its associated OpenPGP +signature, authenticating the downloaded tarball against its signature +using @command{gpg}, and finally computing its hash. When the public +key used to sign the tarball is missing from the user's keyring, an +attempt is made to automatically retrieve it from a public key server; +when this is successful, the key is added to the user's keyring; otherwise, +@command{guix refresh} reports an error. -In this example, we see that the closure of Coreutils weighs in at -79@tie{}MiB, most of which is taken by libc and GCC's run-time support -libraries. (That libc and GCC's libraries represent a large fraction of -the closure is not a problem @i{per se} because they are always available -on the system anyway.) +The following options are supported: -When the package(s) passed to @command{guix size} are available in the -store@footnote{More precisely, @command{guix size} looks for the -@emph{ungrafted} variant of the given package(s), as returned by -@code{guix build @var{package} --no-grafts}. @xref{Security Updates}, -for information on grafts.}, @command{guix size} queries the daemon to determine its -dependencies, and measures its size in the store, similar to @command{du --ms --apparent-size} (@pxref{du invocation,,, coreutils, GNU -Coreutils}). +@table @code -When the given packages are @emph{not} in the store, @command{guix size} -reports information based on the available substitutes -(@pxref{Substitutes}). This makes it possible it to profile disk usage of -store items that are not even on disk, only available remotely. +@item --expression=@var{expr} +@itemx -e @var{expr} +Consider the package @var{expr} evaluates to. -You can also specify several package names: +This is useful to precisely refer to a package, as in this example: @example -$ guix size coreutils grep sed bash -store item total self -/gnu/store/@dots{}-coreutils-8.24 77.8 13.8 13.4% -/gnu/store/@dots{}-grep-2.22 73.1 0.8 0.8% -/gnu/store/@dots{}-bash-4.3.42 72.3 4.7 4.6% -/gnu/store/@dots{}-readline-6.3 67.6 1.2 1.2% -@dots{} -total: 102.3 MiB +guix refresh -l -e '(@@@@ (gnu packages commencement) glibc-final)' @end example -@noindent -In this example we see that the combination of the four packages takes -102.3@tie{}MiB in total, which is much less than the sum of each closure -since they have a lot of dependencies in common. - -The available options are: - -@table @option +This command lists the dependents of the ``final'' libc (essentially all +the packages.) -@item --substitute-urls=@var{urls} -Use substitute information from @var{urls}. -@xref{client-substitute-urls, the same option for @code{guix build}}. +@item --update +@itemx -u +Update distribution source files (package recipes) in place. This is +usually run from a checkout of the Guix source tree (@pxref{Running +Guix Before It Is Installed}): -@item --sort=@var{key} -Sort lines according to @var{key}, one of the following options: +@example +$ ./pre-inst-env guix refresh -s non-core -u +@end example -@table @code -@item self -the size of each item (the default); -@item closure -the total size of the item's closure. -@end table +@xref{Defining Packages}, for more information on package definitions. -@item --map-file=@var{file} -Write a graphical map of disk usage in PNG format to @var{file}. +@item --select=[@var{subset}] +@itemx -s @var{subset} +Select all the packages in @var{subset}, one of @code{core} or +@code{non-core}. -For the example above, the map looks like this: +The @code{core} subset refers to all the packages at the core of the +distribution---i.e., packages that are used to build ``everything +else''. This includes GCC, libc, Binutils, Bash, etc. Usually, +changing one of these packages in the distribution entails a rebuild of +all the others. Thus, such updates are an inconvenience to users in +terms of build time or bandwidth used to achieve the upgrade. -@image{images/coreutils-size-map,5in,, map of Coreutils disk usage -produced by @command{guix size}} +The @code{non-core} subset refers to the remaining packages. It is +typically useful in cases where an update of the core packages would be +inconvenient. -This option requires that -@uref{http://wingolog.org/software/guile-charting/, Guile-Charting} be -installed and visible in Guile's module search path. When that is not -the case, @command{guix size} fails as it tries to load it. +@item --manifest=@var{file} +@itemx -m @var{file} +Select all the packages from the manifest in @var{file}. This is useful to +check if any packages of the user manifest can be updated. -@item --system=@var{system} -@itemx -s @var{system} -Consider packages for @var{system}---e.g., @code{x86_64-linux}. +@item --type=@var{updater} +@itemx -t @var{updater} +Select only packages handled by @var{updater} (may be a comma-separated +list of updaters). Currently, @var{updater} may be one of: +@table @code +@item gnu +the updater for GNU packages; +@item gnome +the updater for GNOME packages; +@item kde +the updater for KDE packages; +@item xorg +the updater for X.org packages; +@item kernel.org +the updater for packages hosted on kernel.org; +@item elpa +the updater for @uref{http://elpa.gnu.org/, ELPA} packages; +@item cran +the updater for @uref{https://cran.r-project.org/, CRAN} packages; +@item bioconductor +the updater for @uref{https://www.bioconductor.org/, Bioconductor} R packages; +@item cpan +the updater for @uref{http://www.cpan.org/, CPAN} packages; +@item pypi +the updater for @uref{https://pypi.python.org, PyPI} packages. +@item gem +the updater for @uref{https://rubygems.org, RubyGems} packages. +@item github +the updater for @uref{https://github.com, GitHub} packages. +@item hackage +the updater for @uref{https://hackage.haskell.org, Hackage} packages. +@item stackage +the updater for @uref{https://www.stackage.org, Stackage} packages. +@item crate +the updater for @uref{https://crates.io, Crates} packages. @end table -@node Invoking guix graph -@section Invoking @command{guix graph} - -@cindex DAG -@cindex @command{guix graph} -@cindex package dependencies -Packages and their dependencies form a @dfn{graph}, specifically a -directed acyclic graph (DAG). It can quickly become difficult to have a -mental model of the package DAG, so the @command{guix graph} command -provides a visual representation of the DAG. By default, -@command{guix graph} emits a DAG representation in the input format of -@uref{http://www.graphviz.org/, Graphviz}, so its output can be passed -directly to the @command{dot} command of Graphviz. It can also emit an -HTML page with embedded JavaScript code to display a ``chord diagram'' -in a Web browser, using the @uref{https://d3js.org/, d3.js} library, or -emit Cypher queries to construct a graph in a graph database supporting -the @uref{http://www.opencypher.org/, openCypher} query language. -The general syntax is: +For instance, the following command only checks for updates of Emacs +packages hosted at @code{elpa.gnu.org} and for updates of CRAN packages: @example -guix graph @var{options} @var{package}@dots{} +$ guix refresh --type=elpa,cran +gnu/packages/statistics.scm:819:13: r-testthat would be upgraded from 0.10.0 to 0.11.0 +gnu/packages/emacs.scm:856:13: emacs-auctex would be upgraded from 11.88.6 to 11.88.9 @end example -For example, the following command generates a PDF file representing the -package DAG for the GNU@tie{}Core Utilities, showing its build-time -dependencies: +@end table + +In addition, @command{guix refresh} can be passed one or more package +names, as in this example: @example -guix graph coreutils | dot -Tpdf > dag.pdf +$ ./pre-inst-env guix refresh -u emacs idutils gcc@@4.8 @end example -The output looks like this: +@noindent +The command above specifically updates the @code{emacs} and +@code{idutils} packages. The @code{--select} option would have no +effect in this case. -@image{images/coreutils-graph,2in,,Dependency graph of the GNU Coreutils} +When considering whether to upgrade a package, it is sometimes +convenient to know which packages would be affected by the upgrade and +should be checked for compatibility. For this the following option may +be used when passing @command{guix refresh} one or more package names: -Nice little graph, no? +@table @code -But there is more than one graph! The one above is concise: it is the -graph of package objects, omitting implicit inputs such as GCC, libc, -grep, etc. It is often useful to have such a concise graph, but -sometimes one may want to see more details. @command{guix graph} supports -several types of graphs, allowing you to choose the level of detail: +@item --list-updaters +@itemx -L +List available updaters and exit (see @option{--type} above.) -@table @code -@item package -This is the default type used in the example above. It shows the DAG of -package objects, excluding implicit dependencies. It is concise, but -filters out many details. +For each updater, display the fraction of packages it covers; at the +end, display the fraction of packages covered by all these updaters. -@item reverse-package -This shows the @emph{reverse} DAG of packages. For example: +@item --list-dependent +@itemx -l +List top-level dependent packages that would need to be rebuilt as a +result of upgrading one or more packages. + +@xref{Invoking guix graph, the @code{reverse-package} type of +@command{guix graph}}, for information on how to visualize the list of +dependents of a package. + +@end table + +Be aware that the @code{--list-dependent} option only +@emph{approximates} the rebuilds that would be required as a result of +an upgrade. More rebuilds might be required under some circumstances. @example -guix graph --type=reverse-package ocaml +$ guix refresh --list-dependent flex +Building the following 120 packages would ensure 213 dependent packages are rebuilt: +hop@@2.4.0 geiser@@0.4 notmuch@@0.18 mu@@0.9.9.5 cflow@@1.4 idutils@@4.6 @dots{} @end example -...@: yields the graph of packages that depend on OCaml. - -Note that for core packages this can yield huge graphs. If all you want -is to know the number of packages that depend on a given package, use -@command{guix refresh --list-dependent} (@pxref{Invoking guix refresh, -@option{--list-dependent}}). +The command above lists a set of packages that could be built to check +for compatibility with an upgraded @code{flex} package. -@item bag-emerged -This is the package DAG, @emph{including} implicit inputs. +@table @code -For instance, the following command: +@item --list-transitive +List all the packages which one or more packages depend upon. @example -guix graph --type=bag-emerged coreutils | dot -Tpdf > dag.pdf +$ guix refresh --list-transitive flex +flex@@2.6.4 depends on the following 25 packages: perl@@5.28.0 help2man@@1.47.6 +bison@@3.0.5 indent@@2.2.10 tar@@1.30 gzip@@1.9 bzip2@@1.0.6 xz@@5.2.4 file@@5.33 @dots{} @end example -...@: yields this bigger graph: +@end table -@image{images/coreutils-bag-graph,,5in,Detailed dependency graph of the GNU Coreutils} +The command above lists a set of packages which, when changed, would cause +@code{flex} to be rebuilt. -At the bottom of the graph, we see all the implicit inputs of -@var{gnu-build-system} (@pxref{Build Systems, @code{gnu-build-system}}). +The following options can be used to customize GnuPG operation: -Now, note that the dependencies of these implicit inputs---that is, the -@dfn{bootstrap dependencies} (@pxref{Bootstrapping})---are not shown -here, for conciseness. +@table @code -@item bag -Similar to @code{bag-emerged}, but this time including all the bootstrap -dependencies. +@item --gpg=@var{command} +Use @var{command} as the GnuPG 2.x command. @var{command} is searched +for in @code{$PATH}. -@item bag-with-origins -Similar to @code{bag}, but also showing origins and their dependencies. +@item --keyring=@var{file} +Use @var{file} as the keyring for upstream keys. @var{file} must be in the +@dfn{keybox format}. Keybox files usually have a name ending in @file{.kbx} +and the GNU@tie{}Privacy Guard (GPG) can manipulate these files +(@pxref{kbxutil, @command{kbxutil},, gnupg, Using the GNU Privacy Guard}, for +information on a tool to manipulate keybox files). -@item derivation -This is the most detailed representation: It shows the DAG of -derivations (@pxref{Derivations}) and plain store items. Compared to -the above representation, many additional nodes are visible, including -build scripts, patches, Guile modules, etc. +When this option is omitted, @command{guix refresh} uses +@file{~/.config/guix/upstream/trustedkeys.kbx} as the keyring for upstream +signing keys. OpenPGP signatures are checked against keys from this keyring; +missing keys are downloaded to this keyring as well (see +@option{--key-download} below.) -For this type of graph, it is also possible to pass a @file{.drv} file -name instead of a package name, as in: +You can export keys from your default GPG keyring into a keybox file using +commands like this one: @example -guix graph -t derivation `guix system build -d my-config.scm` +gpg --export rms@@gnu.org | kbxutil --import-openpgp >> mykeyring.kbx @end example -@item module -This is the graph of @dfn{package modules} (@pxref{Package Modules}). -For example, the following command shows the graph for the package -module that defines the @code{guile} package: +Likewise, you can fetch keys to a specific keybox file like this: @example -guix graph -t module guile | dot -Tpdf > module-graph.pdf +gpg --no-default-keyring --keyring mykeyring.kbx \ + --recv-keys @value{OPENPGP-SIGNING-KEY-ID} @end example + +@ref{GPG Configuration Options, @option{--keyring},, gnupg, Using the GNU +Privacy Guard}, for more information on GPG's @option{--keyring} option. + +@item --key-download=@var{policy} +Handle missing OpenPGP keys according to @var{policy}, which may be one +of: + +@table @code +@item always +Always download missing OpenPGP keys from the key server, and add them +to the user's GnuPG keyring. + +@item never +Never try to download missing OpenPGP keys. Instead just bail out. + +@item interactive +When a package signed with an unknown OpenPGP key is encountered, ask +the user whether to download it or not. This is the default behavior. @end table -All the types above correspond to @emph{build-time dependencies}. The -following graph type represents the @emph{run-time dependencies}: +@item --key-server=@var{host} +Use @var{host} as the OpenPGP key server when importing a public key. + +@end table + +The @code{github} updater uses the +@uref{https://developer.github.com/v3/, GitHub API} to query for new +releases. When used repeatedly e.g.@: when refreshing all packages, +GitHub will eventually refuse to answer any further API requests. By +default 60 API requests per hour are allowed, and a full refresh on all +GitHub packages in Guix requires more than this. Authentication with +GitHub through the use of an API token alleviates these limits. To use +an API token, set the environment variable @code{GUIX_GITHUB_TOKEN} to a +token procured from @uref{https://github.com/settings/tokens} or +otherwise. + + +@node Invoking guix lint +@section Invoking @command{guix lint} + +@cindex @command{guix lint} +@cindex package, checking for errors +The @command{guix lint} command is meant to help package developers avoid +common errors and use a consistent style. It runs a number of checks on +a given set of packages in order to find common mistakes in their +definitions. Available @dfn{checkers} include (see +@code{--list-checkers} for a complete list): @table @code -@item references -This is the graph of @dfn{references} of a package output, as returned -by @command{guix gc --references} (@pxref{Invoking guix gc}). +@item synopsis +@itemx description +Validate certain typographical and stylistic rules about package +descriptions and synopses. -If the given package output is not available in the store, @command{guix -graph} attempts to obtain dependency information from substitutes. +@item inputs-should-be-native +Identify inputs that should most likely be native inputs. -Here you can also pass a store file name instead of a package name. For -example, the command below produces the reference graph of your profile -(which can be big!): +@item source +@itemx home-page +@itemx mirror-url +@itemx github-url +@itemx source-file-name +Probe @code{home-page} and @code{source} URLs and report those that are +invalid. Suggest a @code{mirror://} URL when applicable. If the +@code{source} URL redirects to a GitHub URL, recommend usage of the GitHub +URL. Check that the source file name is meaningful, e.g.@: is not just a +version number or ``git-checkout'', without a declared @code{file-name} +(@pxref{origin Reference}). + +@item source-unstable-tarball +Parse the @code{source} URL to determine if a tarball from GitHub is +autogenerated or if it is a release tarball. Unfortunately GitHub's +autogenerated tarballs are sometimes regenerated. + +@item cve +@cindex security vulnerabilities +@cindex CVE, Common Vulnerabilities and Exposures +Report known vulnerabilities found in the Common Vulnerabilities and +Exposures (CVE) databases of the current and past year +@uref{https://nvd.nist.gov/download.cfm#CVE_FEED, published by the US +NIST}. + +To view information about a particular vulnerability, visit pages such as: + +@itemize +@item +@indicateurl{https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-YYYY-ABCD} +@item +@indicateurl{https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-YYYY-ABCD} +@end itemize + +@noindent +where @code{CVE-YYYY-ABCD} is the CVE identifier---e.g., +@code{CVE-2015-7554}. + +Package developers can specify in package recipes the +@uref{https://nvd.nist.gov/cpe.cfm,Common Platform Enumeration (CPE)} +name and version of the package when they differ from the name or version +that Guix uses, as in this example: @example -guix graph -t references `readlink -f ~/.guix-profile` +(package + (name "grub") + ;; @dots{} + ;; CPE calls this package "grub2". + (properties '((cpe-name . "grub2") + (cpe-version . "2.3"))) @end example -@item referrers -This is the graph of the @dfn{referrers} of a store item, as returned by -@command{guix gc --referrers} (@pxref{Invoking guix gc}). - -This relies exclusively on local information from your store. For -instance, let us suppose that the current Inkscape is available in 10 -profiles on your machine; @command{guix graph -t referrers inkscape} -will show a graph rooted at Inkscape and with those 10 profiles linked -to it. +@c See . +Some entries in the CVE database do not specify which version of a +package they apply to, and would thus ``stick around'' forever. Package +developers who found CVE alerts and verified they can be ignored can +declare them as in this example: -It can help determine what is preventing a store item from being garbage -collected. +@example +(package + (name "t1lib") + ;; @dots{} + ;; These CVEs no longer apply and can be safely ignored. + (properties `((lint-hidden-cve . ("CVE-2011-0433" + "CVE-2011-1553" + "CVE-2011-1554" + "CVE-2011-5244"))))) +@end example +@item formatting +Warn about obvious source code formatting issues: trailing white space, +use of tabulations, etc. @end table -The available options are the following: +The general syntax is: -@table @option -@item --type=@var{type} -@itemx -t @var{type} -Produce a graph output of @var{type}, where @var{type} must be one of -the values listed above. +@example +guix lint @var{options} @var{package}@dots{} +@end example -@item --list-types -List the supported graph types. +If no package is given on the command line, then all packages are checked. +The @var{options} may be zero or more of the following: -@item --backend=@var{backend} -@itemx -b @var{backend} -Produce a graph using the selected @var{backend}. +@table @code +@item --list-checkers +@itemx -l +List and describe all the available checkers that will be run on packages +and exit. -@item --list-backends -List the supported graph backends. +@item --checkers +@itemx -c +Only enable the checkers specified in a comma-separated list using the +names returned by @code{--list-checkers}. -Currently, the available backends are Graphviz and d3.js. +@end table -@item --expression=@var{expr} -@itemx -e @var{expr} -Consider the package @var{expr} evaluates to. +@node Invoking guix size +@section Invoking @command{guix size} -This is useful to precisely refer to a package, as in this example: +@cindex size +@cindex package size +@cindex closure +@cindex @command{guix size} +The @command{guix size} command helps package developers profile the +disk usage of packages. It is easy to overlook the impact of an +additional dependency added to a package, or the impact of using a +single output for a package that could easily be split (@pxref{Packages +with Multiple Outputs}). Such are the typical issues that +@command{guix size} can highlight. + +The command can be passed one or more package specifications +such as @code{gcc@@4.8} +or @code{guile:debug}, or a file name in the store. Consider this +example: @example -guix graph -e '(@@@@ (gnu packages commencement) gnu-make-final)' +$ guix size coreutils +store item total self +/gnu/store/@dots{}-gcc-5.5.0-lib 60.4 30.1 38.1% +/gnu/store/@dots{}-glibc-2.27 30.3 28.8 36.6% +/gnu/store/@dots{}-coreutils-8.28 78.9 15.0 19.0% +/gnu/store/@dots{}-gmp-6.1.2 63.1 2.7 3.4% +/gnu/store/@dots{}-bash-static-4.4.12 1.5 1.5 1.9% +/gnu/store/@dots{}-acl-2.2.52 61.1 0.4 0.5% +/gnu/store/@dots{}-attr-2.4.47 60.6 0.2 0.3% +/gnu/store/@dots{}-libcap-2.25 60.5 0.2 0.2% +total: 78.9 MiB @end example -@item --system=@var{system} -@itemx -s @var{system} -Display the graph for @var{system}---e.g., @code{i686-linux}. +@cindex closure +The store items listed here constitute the @dfn{transitive closure} of +Coreutils---i.e., Coreutils and all its dependencies, recursively---as +would be returned by: -The package dependency graph is largely architecture-independent, but there -are some architecture-dependent bits that this option allows you to visualize. -@end table +@example +$ guix gc -R /gnu/store/@dots{}-coreutils-8.23 +@end example +Here the output shows three columns next to store items. The first column, +labeled ``total'', shows the size in mebibytes (MiB) of the closure of +the store item---that is, its own size plus the size of all its +dependencies. The next column, labeled ``self'', shows the size of the +item itself. The last column shows the ratio of the size of the item +itself to the space occupied by all the items listed here. -@node Invoking guix environment -@section Invoking @command{guix environment} +In this example, we see that the closure of Coreutils weighs in at +79@tie{}MiB, most of which is taken by libc and GCC's run-time support +libraries. (That libc and GCC's libraries represent a large fraction of +the closure is not a problem @i{per se} because they are always available +on the system anyway.) -@cindex reproducible build environments -@cindex development environments -@cindex @command{guix environment} -@cindex environment, package build environment -The purpose of @command{guix environment} is to assist hackers in -creating reproducible development environments without polluting their -package profile. The @command{guix environment} tool takes one or more -packages, builds all of their inputs, and creates a shell -environment to use them. +When the package(s) passed to @command{guix size} are available in the +store@footnote{More precisely, @command{guix size} looks for the +@emph{ungrafted} variant of the given package(s), as returned by +@code{guix build @var{package} --no-grafts}. @xref{Security Updates}, +for information on grafts.}, @command{guix size} queries the daemon to determine its +dependencies, and measures its size in the store, similar to @command{du +-ms --apparent-size} (@pxref{du invocation,,, coreutils, GNU +Coreutils}). -The general syntax is: +When the given packages are @emph{not} in the store, @command{guix size} +reports information based on the available substitutes +(@pxref{Substitutes}). This makes it possible it to profile disk usage of +store items that are not even on disk, only available remotely. + +You can also specify several package names: @example -guix environment @var{options} @var{package}@dots{} +$ guix size coreutils grep sed bash +store item total self +/gnu/store/@dots{}-coreutils-8.24 77.8 13.8 13.4% +/gnu/store/@dots{}-grep-2.22 73.1 0.8 0.8% +/gnu/store/@dots{}-bash-4.3.42 72.3 4.7 4.6% +/gnu/store/@dots{}-readline-6.3 67.6 1.2 1.2% +@dots{} +total: 102.3 MiB @end example -The following example spawns a new shell set up for the development of -GNU@tie{}Guile: +@noindent +In this example we see that the combination of the four packages takes +102.3@tie{}MiB in total, which is much less than the sum of each closure +since they have a lot of dependencies in common. -@example -guix environment guile -@end example +The available options are: -If the needed dependencies are not built yet, @command{guix environment} -automatically builds them. The environment of the new shell is an augmented -version of the environment that @command{guix environment} was run in. -It contains the necessary search paths for building the given package -added to the existing environment variables. To create a ``pure'' -environment, in which the original environment variables have been unset, -use the @code{--pure} option@footnote{Users sometimes wrongfully augment -environment variables such as @code{PATH} in their @file{~/.bashrc} -file. As a consequence, when @code{guix environment} launches it, Bash -may read @file{~/.bashrc}, thereby introducing ``impurities'' in these -environment variables. It is an error to define such environment -variables in @file{.bashrc}; instead, they should be defined in -@file{.bash_profile}, which is sourced only by log-in shells. -@xref{Bash Startup Files,,, bash, The GNU Bash Reference Manual}, for -details on Bash start-up files.}. +@table @option -@vindex GUIX_ENVIRONMENT -@command{guix environment} defines the @code{GUIX_ENVIRONMENT} -variable in the shell it spawns; its value is the file name of the -profile of this environment. This allows users to, say, define a -specific prompt for development environments in their @file{.bashrc} -(@pxref{Bash Startup Files,,, bash, The GNU Bash Reference Manual}): +@item --substitute-urls=@var{urls} +Use substitute information from @var{urls}. +@xref{client-substitute-urls, the same option for @code{guix build}}. -@example -if [ -n "$GUIX_ENVIRONMENT" ] -then - export PS1="\u@@\h \w [dev]\$ " -fi -@end example +@item --sort=@var{key} +Sort lines according to @var{key}, one of the following options: -@noindent -...@: or to browse the profile: +@table @code +@item self +the size of each item (the default); +@item closure +the total size of the item's closure. +@end table -@example -$ ls "$GUIX_ENVIRONMENT/bin" -@end example +@item --map-file=@var{file} +Write a graphical map of disk usage in PNG format to @var{file}. -Additionally, more than one package may be specified, in which case the -union of the inputs for the given packages are used. For example, the -command below spawns a shell where all of the dependencies of both Guile -and Emacs are available: +For the example above, the map looks like this: -@example -guix environment guile emacs -@end example +@image{images/coreutils-size-map,5in,, map of Coreutils disk usage +produced by @command{guix size}} -Sometimes an interactive shell session is not desired. An arbitrary -command may be invoked by placing the @code{--} token to separate the -command from the rest of the arguments: +This option requires that +@uref{http://wingolog.org/software/guile-charting/, Guile-Charting} be +installed and visible in Guile's module search path. When that is not +the case, @command{guix size} fails as it tries to load it. -@example -guix environment guile -- make -j4 -@end example +@item --system=@var{system} +@itemx -s @var{system} +Consider packages for @var{system}---e.g., @code{x86_64-linux}. -In other situations, it is more convenient to specify the list of -packages needed in the environment. For example, the following command -runs @command{python} from an environment containing Python@tie{}2.7 and -NumPy: +@end table -@example -guix environment --ad-hoc python2-numpy python-2.7 -- python -@end example +@node Invoking guix graph +@section Invoking @command{guix graph} -Furthermore, one might want the dependencies of a package and also some -additional packages that are not build-time or runtime dependencies, but -are useful when developing nonetheless. Because of this, the -@code{--ad-hoc} flag is positional. Packages appearing before -@code{--ad-hoc} are interpreted as packages whose dependencies will be -added to the environment. Packages appearing after are interpreted as -packages that will be added to the environment directly. For example, -the following command creates a Guix development environment that -additionally includes Git and strace: +@cindex DAG +@cindex @command{guix graph} +@cindex package dependencies +Packages and their dependencies form a @dfn{graph}, specifically a +directed acyclic graph (DAG). It can quickly become difficult to have a +mental model of the package DAG, so the @command{guix graph} command +provides a visual representation of the DAG. By default, +@command{guix graph} emits a DAG representation in the input format of +@uref{http://www.graphviz.org/, Graphviz}, so its output can be passed +directly to the @command{dot} command of Graphviz. It can also emit an +HTML page with embedded JavaScript code to display a ``chord diagram'' +in a Web browser, using the @uref{https://d3js.org/, d3.js} library, or +emit Cypher queries to construct a graph in a graph database supporting +the @uref{http://www.opencypher.org/, openCypher} query language. +The general syntax is: @example -guix environment guix --ad-hoc git strace +guix graph @var{options} @var{package}@dots{} @end example -Sometimes it is desirable to isolate the environment as much as -possible, for maximal purity and reproducibility. In particular, when -using Guix on a host distro that is not Guix System, it is desirable to -prevent access to @file{/usr/bin} and other system-wide resources from -the development environment. For example, the following command spawns -a Guile REPL in a ``container'' where only the store and the current -working directory are mounted: +For example, the following command generates a PDF file representing the +package DAG for the GNU@tie{}Core Utilities, showing its build-time +dependencies: @example -guix environment --ad-hoc --container guile -- guile -@end example - -@quotation Note -The @code{--container} option requires Linux-libre 3.19 or newer. -@end quotation +guix graph coreutils | dot -Tpdf > dag.pdf +@end example -The available options are summarized below. +The output looks like this: -@table @code -@item --root=@var{file} -@itemx -r @var{file} -@cindex persistent environment -@cindex garbage collector root, for environments -Make @var{file} a symlink to the profile for this environment, and -register it as a garbage collector root. +@image{images/coreutils-graph,2in,,Dependency graph of the GNU Coreutils} -This is useful if you want to protect your environment from garbage -collection, to make it ``persistent''. +Nice little graph, no? -When this option is omitted, the environment is protected from garbage -collection only for the duration of the @command{guix environment} -session. This means that next time you recreate the same environment, -you could have to rebuild or re-download packages. @xref{Invoking guix -gc}, for more on GC roots. +But there is more than one graph! The one above is concise: it is the +graph of package objects, omitting implicit inputs such as GCC, libc, +grep, etc. It is often useful to have such a concise graph, but +sometimes one may want to see more details. @command{guix graph} supports +several types of graphs, allowing you to choose the level of detail: -@item --expression=@var{expr} -@itemx -e @var{expr} -Create an environment for the package or list of packages that -@var{expr} evaluates to. +@table @code +@item package +This is the default type used in the example above. It shows the DAG of +package objects, excluding implicit dependencies. It is concise, but +filters out many details. -For example, running: +@item reverse-package +This shows the @emph{reverse} DAG of packages. For example: @example -guix environment -e '(@@ (gnu packages maths) petsc-openmpi)' +guix graph --type=reverse-package ocaml @end example -starts a shell with the environment for this specific variant of the -PETSc package. - -Running: +...@: yields the graph of packages that depend on OCaml. -@example -guix environment --ad-hoc -e '(@@ (gnu) %base-packages)' -@end example +Note that for core packages this can yield huge graphs. If all you want +is to know the number of packages that depend on a given package, use +@command{guix refresh --list-dependent} (@pxref{Invoking guix refresh, +@option{--list-dependent}}). -starts a shell with all the base system packages available. +@item bag-emerged +This is the package DAG, @emph{including} implicit inputs. -The above commands only use the default output of the given packages. -To select other outputs, two element tuples can be specified: +For instance, the following command: @example -guix environment --ad-hoc -e '(list (@@ (gnu packages bash) bash) "include")' +guix graph --type=bag-emerged coreutils | dot -Tpdf > dag.pdf @end example -@item --load=@var{file} -@itemx -l @var{file} -Create an environment for the package or list of packages that the code -within @var{file} evaluates to. +...@: yields this bigger graph: -As an example, @var{file} might contain a definition like this -(@pxref{Defining Packages}): +@image{images/coreutils-bag-graph,,5in,Detailed dependency graph of the GNU Coreutils} -@example -@verbatiminclude environment-gdb.scm -@end example +At the bottom of the graph, we see all the implicit inputs of +@var{gnu-build-system} (@pxref{Build Systems, @code{gnu-build-system}}). -@item --manifest=@var{file} -@itemx -m @var{file} -Create an environment for the packages contained in the manifest object -returned by the Scheme code in @var{file}. +Now, note that the dependencies of these implicit inputs---that is, the +@dfn{bootstrap dependencies} (@pxref{Bootstrapping})---are not shown +here, for conciseness. -This is similar to the same-named option in @command{guix package} -(@pxref{profile-manifest, @option{--manifest}}) and uses the same -manifest files. +@item bag +Similar to @code{bag-emerged}, but this time including all the bootstrap +dependencies. -@item --ad-hoc -Include all specified packages in the resulting environment, as if an -@i{ad hoc} package were defined with them as inputs. This option is -useful for quickly creating an environment without having to write a -package expression to contain the desired inputs. +@item bag-with-origins +Similar to @code{bag}, but also showing origins and their dependencies. -For instance, the command: +@item derivation +This is the most detailed representation: It shows the DAG of +derivations (@pxref{Derivations}) and plain store items. Compared to +the above representation, many additional nodes are visible, including +build scripts, patches, Guile modules, etc. + +For this type of graph, it is also possible to pass a @file{.drv} file +name instead of a package name, as in: @example -guix environment --ad-hoc guile guile-sdl -- guile +guix graph -t derivation `guix system build -d my-config.scm` @end example -runs @command{guile} in an environment where Guile and Guile-SDL are -available. +@item module +This is the graph of @dfn{package modules} (@pxref{Package Modules}). +For example, the following command shows the graph for the package +module that defines the @code{guile} package: -Note that this example implicitly asks for the default output of -@code{guile} and @code{guile-sdl}, but it is possible to ask for a -specific output---e.g., @code{glib:bin} asks for the @code{bin} output -of @code{glib} (@pxref{Packages with Multiple Outputs}). +@example +guix graph -t module guile | dot -Tpdf > module-graph.pdf +@end example +@end table -This option may be composed with the default behavior of @command{guix -environment}. Packages appearing before @code{--ad-hoc} are interpreted -as packages whose dependencies will be added to the environment, the -default behavior. Packages appearing after are interpreted as packages -that will be added to the environment directly. +All the types above correspond to @emph{build-time dependencies}. The +following graph type represents the @emph{run-time dependencies}: -@item --pure -Unset existing environment variables when building the new environment. -This has the effect of creating an environment in which search paths -only contain package inputs. +@table @code +@item references +This is the graph of @dfn{references} of a package output, as returned +by @command{guix gc --references} (@pxref{Invoking guix gc}). -@item --search-paths -Display the environment variable definitions that make up the -environment. +If the given package output is not available in the store, @command{guix +graph} attempts to obtain dependency information from substitutes. -@item --system=@var{system} -@itemx -s @var{system} -Attempt to build for @var{system}---e.g., @code{i686-linux}. +Here you can also pass a store file name instead of a package name. For +example, the command below produces the reference graph of your profile +(which can be big!): -@item --container -@itemx -C -@cindex container -Run @var{command} within an isolated container. The current working -directory outside the container is mapped inside the container. -Additionally, unless overridden with @code{--user}, a dummy home -directory is created that matches the current user's home directory, and -@file{/etc/passwd} is configured accordingly. The spawned process runs -as the current user outside the container, but has root privileges in -the context of the container. +@example +guix graph -t references `readlink -f ~/.guix-profile` +@end example -@item --network -@itemx -N -For containers, share the network namespace with the host system. -Containers created without this flag only have access to the loopback -device. +@item referrers +This is the graph of the @dfn{referrers} of a store item, as returned by +@command{guix gc --referrers} (@pxref{Invoking guix gc}). -@item --link-profile -@itemx -P -For containers, link the environment profile to -@file{~/.guix-profile} within the container. This is equivalent to -running the command @command{ln -s $GUIX_ENVIRONMENT ~/.guix-profile} -within the container. Linking will fail and abort the environment if -the directory already exists, which will certainly be the case if -@command{guix environment} was invoked in the user's home directory. +This relies exclusively on local information from your store. For +instance, let us suppose that the current Inkscape is available in 10 +profiles on your machine; @command{guix graph -t referrers inkscape} +will show a graph rooted at Inkscape and with those 10 profiles linked +to it. -Certain packages are configured to look in -@code{~/.guix-profile} for configuration files and data;@footnote{For -example, the @code{fontconfig} package inspects -@file{~/.guix-profile/share/fonts} for additional fonts.} -@code{--link-profile} allows these programs to behave as expected within -the environment. +It can help determine what is preventing a store item from being garbage +collected. -@item --user=@var{user} -@itemx -u @var{user} -For containers, use the username @var{user} in place of the current -user. The generated @file{/etc/passwd} entry within the container will -contain the name @var{user}; the home directory will be -@file{/home/USER}; and no user GECOS data will be copied. @var{user} -need not exist on the system. +@end table -Additionally, any shared or exposed path (see @code{--share} and -@code{--expose} respectively) whose target is within the current user's -home directory will be remapped relative to @file{/home/USER}; this -includes the automatic mapping of the current working directory. +The available options are the following: -@example -# will expose paths as /home/foo/wd, /home/foo/test, and /home/foo/target -cd $HOME/wd -guix environment --container --user=foo \ - --expose=$HOME/test \ - --expose=/tmp/target=$HOME/target -@end example +@table @option +@item --type=@var{type} +@itemx -t @var{type} +Produce a graph output of @var{type}, where @var{type} must be one of +the values listed above. -While this will limit the leaking of user identity through home paths -and each of the user fields, this is only one useful component of a -broader privacy/anonymity solution---not one in and of itself. +@item --list-types +List the supported graph types. -@item --expose=@var{source}[=@var{target}] -For containers, expose the file system @var{source} from the host system -as the read-only file system @var{target} within the container. If -@var{target} is not specified, @var{source} is used as the target mount -point in the container. +@item --backend=@var{backend} +@itemx -b @var{backend} +Produce a graph using the selected @var{backend}. -The example below spawns a Guile REPL in a container in which the user's -home directory is accessible read-only via the @file{/exchange} -directory: +@item --list-backends +List the supported graph backends. -@example -guix environment --container --expose=$HOME=/exchange --ad-hoc guile -- guile -@end example +Currently, the available backends are Graphviz and d3.js. -@item --share=@var{source}[=@var{target}] -For containers, share the file system @var{source} from the host system -as the writable file system @var{target} within the container. If -@var{target} is not specified, @var{source} is used as the target mount -point in the container. +@item --expression=@var{expr} +@itemx -e @var{expr} +Consider the package @var{expr} evaluates to. -The example below spawns a Guile REPL in a container in which the user's -home directory is accessible for both reading and writing via the -@file{/exchange} directory: +This is useful to precisely refer to a package, as in this example: @example -guix environment --container --share=$HOME=/exchange --ad-hoc guile -- guile +guix graph -e '(@@@@ (gnu packages commencement) gnu-make-final)' @end example + +@item --system=@var{system} +@itemx -s @var{system} +Display the graph for @var{system}---e.g., @code{i686-linux}. + +The package dependency graph is largely architecture-independent, but there +are some architecture-dependent bits that this option allows you to visualize. @end table -@command{guix environment} -also supports all of the common build options that @command{guix -build} supports (@pxref{Common Build Options}) as well as package -transformation options (@pxref{Package Transformation Options}). @node Invoking guix publish @@ -9892,7 +9926,7 @@ instance to support new system services. * Initial RAM Disk:: Linux-Libre bootstrapping. * Bootloader Configuration:: Configuring the boot loader. * Invoking guix system:: Instantiating a system configuration. -* Running Guix in a VM:: How to run Guix System in a virtual machine. +* Running Guix in a VM:: How to run Guix System in a virtual machine. * Defining Services:: Adding new service definitions. @end menu -- cgit v1.2.3