| Age | Commit message (Expand) | Author |
|---|
| 2020-12-08 | database: Remove unnecessary module imports....* guix/store/database.scm: Remove unnecessary imports added in
4b9eecd322e566783369795ebea63a479b51f486.
| Ludovic Courtès |
| 2020-09-14 | database: register-items: reduce transaction scope....It was made transactional in a4678c6ba18d8dbd79d931f80426eebf61be7ebe, with
the reasoning to prevent broken intermediate states from being visible. I
think this means something like an entry being in ValidPaths, but the Refs not
being inserted.
Using a transaction for this makes sense, but I think using one single
transaction for the whole register-items call is unnecessary to avoid broken
states from being visible, and could block other writes to the store database
while register-items is running. Because the deduplication and resetting
timestamps happens within the transaction as well, even though these things
don't involve the database, writes to the database will still be blocked while
this is happening.
To reduce the potential for register-items to block other writers to the
database for extended periods, this commit moves the transaction to just wrap
the call to sqlite-register. This is the one place where writes occur, so that
should prevent the broken intermediate states issue above. The one difference
this will make is some of the registered items will be visible to other
connections while others may be still being added. I think this is OK, as it's
equivalent to just registering different items.
* guix/store/database.scm (register-items): Reduce transaction scope.
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
| Christopher Baines |
| 2020-09-14 | database: document extra registration requirements....It's necessary that store items be locked and protected from garbage
collection while they are being registered. This documents that.
* guix/store/database.scm (register-path, register-items): document GC
protection and locking requirements.
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
| Caleb Ristvedt |
| 2020-09-14 | deduplication: pass store directory to replace-with-link....This causes with-writable-file to take into consideration the actual store
being used, as passed to 'deduplicate', rather than
whatever (%store-directory) may return.
* guix/store/deduplication.scm (replace-with-link): new keyword argument
'store'. Pass to with-writable-file.
(with-writable-file, call-with-writable-file): new store argument.
(deduplicate): pass store to replace-with-link.
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
| Caleb Ristvedt |
| 2020-07-28 | store: deduplication: Handle fs without d_type support....scandir* uses readdir, which means that the file type property can be 'unknown
if the underlying file-system does not support d_type. Make sure to fallback
to lstat in that case.
Fixes: https://issues.guix.gnu.org/issue/42579.
* guix/store/deduplication.scm (deduplicate): Handle the case where properties
is 'unknown because the underlying file-system does not support d_type.
| Mathieu Othacehe |
| 2020-07-11 | image: Do not set journal_model=WAL for the Hurd....This fixes <https://bugs.gnu.org/42151>.
* gnu/system/images/hurd.scm (hurd-initialize-root-partition): Use #:wal-mode #f
in call to ...
* gnu/build/image.scm (initialize-root-partition): ... this, add #:wal-mode?
parameter, pass it to ...
(register-closure): ... this, add #:wal-mode? parameter, pass it to ...
* guix/store/database.scm (with-database): ... this, add #:wal-mode?
parameter, pass it to ...
(call-with-database): ... this, add #:wal-mode? parameter; when
set to #f, do not set journal_model=WAL.
| Jan (janneke) Nieuwenhuizen |
| 2020-06-25 | deduplication: Leave the store permissions unchanged....Suggested by Caleb Ristvedt <caleb.ristvedt@cune.org>.
* guix/store/deduplication.scm (call-with-writable-file): Call THUNK
directly when FILE is (%store-directory).
| Ludovic Courtès |
| 2020-06-25 | deduplication: Fix default value of #:store in 'deduplicate'....* guix/store/deduplication.scm (deduplicate): Change #:store default
value to (%store-directory).
| Ludovic Courtès |
| 2020-06-25 | deduplication: Use 'dynamic-wind' when changing permissions of the parent....Suggested by Caleb Ristvedt <caleb.ristvedt@cune.org>.
* guix/store/deduplication.scm (call-with-writable-file): New procedure.
(with-writable-file): New macro.
(replace-with-link): Use it.
| Ludovic Courtès |
| 2020-06-22 | deduplicate: Avoid traversing directories twice....Until now, we'd call (nar-sha256 file) unconditionally. Thus, if FILE
was a directory, we would traverse it for no reason, and then call
'deduplicate' on FILE, which would again traverse it.
This change also removes redundant (mkdir-p store) calls from the loop,
and avoids 'lstat' calls by using 'scandir*'.
* guix/store/deduplication.scm (deduplicate): Add named loop. Move
'mkdir-p' outside the loop. Use 'scandir*' instead of 'scandir'. Do
not call 'nar-sha256' when FILE has type 'directory.
| Ludovic Courtès |
| 2020-06-18 | database: 'register-items' takes an open database....* guix/store/database.scm (store-database-directory)
(store-database-file): New procedures.
(call-with-database): Add call to 'mkdir-p'.
(register-items): Add 'db' parameter and remove #:state-directory and #:schema.
(register-path): Use 'store-database-file' and 'with-database', and
parameterize SQL-SCHEMA.
* gnu/build/image.scm (register-closure): Likewise.
* gnu/build/vm.scm (register-closure): Likewise.
* guix/scripts/pack.scm (store-database)[build]: Likewise.
| Ludovic Courtès |
| 2020-06-10 | database: separate transaction-handling and retry-handling....Previously call-with-transaction would both retry when SQLITE_BUSY errors were
thrown and do what its name suggested (start and rollback/commit a
transaction). This changes it to do only what its name implies, which
simplifies its implementation. Retrying is provided by the new
call-with-SQLITE_BUSY-retrying procedure.
* guix/store/database.scm (call-with-transaction): no longer restarts, new
#:restartable? argument controls whether "begin" or "begin immediate" is
used.
(call-with-SQLITE_BUSY-retrying, call-with-retrying-transaction,
call-with-retrying-savepoint): new procedures.
(register-items): use call-with-retrying-transaction to preserve old
behavior.
* .dir-locals.el (call-with-retrying-transaction,
call-with-retrying-savepoint): add indentation information.
| Caleb Ristvedt |
| 2020-06-10 | database: ensure update-or-insert is run within a transaction...update-or-insert can break if an insert occurs between when it decides whether
to update or insert and when it actually performs that operation. Putting the
check and the update/insert operation in the same transaction ensures that the
update/insert will only succeed if no other write has occurred in the middle.
* guix/store/database.scm (call-with-savepoint): new procedure.
(update-or-insert): use call-with-savepoint to ensure the read and the
insert/update occur within the same transaction.
| Caleb Ristvedt |
| 2020-06-10 | database: rewrite query procedures in terms of with-statement....Most of our queries would fail to finalize their statements properly if sqlite
returned an error during their execution. This resolves that, and also makes
them somewhat more concise as a side-effect.
This also makes some small changes to improve certain queries where behavior
was strange or overly verbose.
* guix/store/database.scm (call-with-statement): new procedure.
(with-statement): new macro.
(last-insert-row-id, path-id, update-or-insert, add-references): rewrite to
use with-statement.
(update-or-insert): factor last-insert-row-id out of the end of both
branches.
(add-references): remove pointless last-insert-row-id call.
* .dir-locals.el (with-statement): add indenting information.
| Caleb Ristvedt |
| 2020-06-10 | database: work around guile-sqlite3 bug preventing statement reset...guile-sqlite3 provides statement caching, making it unnecessary for sqlite to
keep re-preparing statements that are frequently used. Unfortunately it
doesn't quite emulate the semantics of sqlite_finalize properly, because it
doesn't cause a commit if the statement being finalized is the last "active"
statement (see https://notabug.org/guile-sqlite3/guile-sqlite3/issues/12). We
work around this by wrapping sqlite-finalize with our own version that ensures
sqlite-reset is called, which does The Right Thing™.
* guix/store/database.scm (sqlite-finalize): new procedure that shadows the
sqlite-finalize from (sqlite3).
| Caleb Ristvedt |
| 2020-04-11 | database: 'reset-timestamps' can optionally preserve permissions....* guix/store/database.scm (reset-timestamps): Add
#:preserve-permissions? and honor it.
| Ludovic Courtès |
| 2020-02-22 | deduplication: Use nix-base32 encoding for link names....Fixes <https://bugs.gnu.org/39725>.
* guix/store/deduplication.scm (deduplicate): Use
'bytevector->nix-base32-string' instead of 'bytevector->base16-string'.
| Ludovic Courtès |
| 2019-09-08 | daemon: Invoke 'guix gc --list-busy' instead of 'list-runtime-roots'....* nix/scripts/list-runtime-roots.in: Remove.
* guix/store/roots.scm (%proc-directory): New variable.
(proc-file-roots, proc-exe-roots, proc-cwd-roots)
(proc-fd-roots, proc-maps-roots, proc-environ-roots)
(referenced-files, canonicalize-store-item, busy-store-items): New
procedures, taken from 'list-runtime-roots.in'.
* nix/libstore/globals.hh (Settings)[guixProgram]: New field.
* nix/libstore/globals.cc (Settings::processEnvironment): Initialize
'guixProgram'.
* nix/libstore/gc.cc (addAdditionalRoots): Drop code related to
'NIX_ROOT_FINDER'. Run "guix gc --list-busy".
* nix/local.mk (nodist_pkglibexec_SCRIPTS): Remove
'scripts/list-runtime-roots'.
* config-daemon.ac: Don't output nix/scripts/list-runtime-roots.
* build-aux/pre-inst-env.in: Don't set 'NIX_ROOT_FINDER'.
Set 'GUIX'.
* doc/guix.texi (Invoking guix gc): Document '--list-busy'.
* guix/scripts/gc.scm (show-help, %options): Add "--list-busy".
(guix-gc)[list-busy]: New procedure.
Handle the 'list-busy' action.
| Ludovic Courtès |
| 2019-04-25 | gnu, guix: Yearly ritual purging of the filesystems....* gnu/packages/android.scm (android-ext4-utils)[synopsis]: Fix ‘file
system’ spelling.
* gnu/packages/disk.scm (rmlint)[synopsis, description]: Likewise.
* gnu/packages/golang.scm (go-github-com-kr-fs)[synopsis, description]:
Likewise & edit for grammar.
* gnu/packages/ipfs.scm (gx, go-ipfs)[description]: Likewise.
* /gnu/packages/java.scm (java-commons-vfs)[synopsis]: Likewise.
* gnu/packages/linux.scm (fuseiso)[description]: Likewise.
(genext2fs)[synopsis, description]: Likewise.
* gnu/packages/package-management.scm (libostree)[description]: Likewise.
* gnu/packages/python-xyz.scm (python-requests-file)[description]:
Likewise & mark up.
* gnu/packages/rails.scm (ruby-with-advisory-lock)[description]:
Likewise.
* gnu/packages/ruby.scm (ruby-rerun)[description]: Likewise.
* guix/build/go-build-system.scm (setup-go-environment)<docstring>:
Likewise.
* guix/store/deduplication.scm (get-temp-link)<docstring>: Likewise.
| Tobias Geerinckx-Rice |
| 2019-04-10 | Add (guix store roots)....* guix/store/roots.scm, tests/store-roots.scm: New files.
* Makefile.am (STORE_MODULES): Add guix/store/roots.scm.
(SCM_TESTS): Add tests/store-roots.scm.
| Ludovic Courtès |
| 2019-03-07 | database: Make 'register-items' transactional....* guix/store/database.scm (SQLITE_BUSY, register-output-sql): New variables.
(add-references): Don't try finalizing after each use, only after all the
uses (otherwise a finalized statement would be used if #:cache? was #f).
(call-with-transaction): New procedure.
(register-items): Use call-with-transaction to prevent broken intermediate
states from being visible.
* .dir-locals.el (call-with-transaction): indent it.
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
| Caleb Ristvedt |
| 2019-01-23 | deduplication: Ignore EMLINK....Until now 'guix offload' would fail (transient failure) upon EMLINK.
* guix/store/deduplication.scm (replace-with-link)
(deduplicate): Ignore EMLINK.
| Ludovic Courtès |
| 2018-12-21 | database: Use "write-ahead log" mode and set a long "busy timeout"....This should avoid "database is locked" errors when there's a lot of
concurrency, for instance when offloading simultaneously a lot of
builds.
* guix/store/database.scm (call-with-database): Add two 'sqlite-exec'
calls to set 'journal_mode' and 'busy_timeout'.
| Ludovic Courtès |
| 2018-12-14 | deduplication: Gracefully handle ENOSPC raised by 'link' calls....Reported by Andreas Enge <andreas@enge.fr>
in <https://bugs.gnu.org/33676>.
* guix/store/deduplication.scm (replace-with-link): Catch ENOSPC around
'get-temp-link'. Do nothing when 'get-temp-link' throws ENOSPC. Move
code to restore PARENT's permissions outside of 'catch'.
* tests/store-deduplication.scm ("deduplicate, ENOSPC"): New test.
| Ludovic Courtès |
| 2018-11-23 | Update Guile-SQLite3 URL everywhere....* README: Update Guile-SQLite3 URL.
* doc/guix.texi (Requirements): Likewise.
* guix/store/database.scm (sqlite-exec): Likewise.
* m4/guix.m4 (GUIX_CHECK_GUILE_SQLITE3): Likewise.
| Ludovic Courtès |
| 2018-11-13 | deduplication: Restore directory mtime and permissions after deduplication....Fixes <https://bugs.gnu.org/33361>.
* guix/store/deduplication.scm (replace-with-link): Call 'set-file-time'
and 'chmod' after 'rename-file'.
* tests/nar.scm ("restore-file-set with directories (signed, valid)"):
New test.
| Ludovic Courtès |
| 2018-11-13 | nar: Access the database instead of connecting to the daemon....* guix/store/database.scm (%default-database-file): New variable.
(path-id): Export.
* guix/nar.scm (finalize-store-file): Use 'with-database' instead of
'with-store', and use 'path-id' instead of 'valid-path?'.
| Ludovic Courtès |
| 2018-09-23 | database: Register each store item only once....Fixes <https://bugs.gnu.org/32600>.
Reported by Leo Famulari.
* guix/store/database.scm (register-items): Check whether TO-REGISTER is
in DB by calling 'path-id', and skip the reset-timestamps,
registration, and deduplication phases when it is.
| Ludovic Courtès |
| 2018-09-23 | database: 'register-items' shows a progress bar....* guix/store/database.scm (register-items): Add #:log-port. Use
'progress-reporter/bar' to show a progress report.
(register-path): Pass #:log-port to 'register-items'.
| Ludovic Courtès |
| 2018-09-04 | Switch to Guile-Gcrypt....This removes (guix hash) and (guix pk-crypto), which now live as part of
Guile-Gcrypt (version 0.1.0.)
* guix/gcrypt.scm, guix/hash.scm, guix/pk-crypto.scm,
tests/hash.scm, tests/pk-crypto.scm: Remove.
* configure.ac: Test for Guile-Gcrypt. Remove LIBGCRYPT and
LIBGCRYPT_LIBDIR assignments.
* m4/guix.m4 (GUIX_ASSERT_LIBGCRYPT_USABLE): Remove.
* README: Add Guile-Gcrypt to the dependencies; move libgcrypt as
"required unless --disable-daemon".
* doc/guix.texi (Requirements): Likewise.
* gnu/packages/bash.scm, guix/derivations.scm, guix/docker.scm,
guix/git.scm, guix/http-client.scm, guix/import/cpan.scm,
guix/import/cran.scm, guix/import/crate.scm, guix/import/elpa.scm,
guix/import/gnu.scm, guix/import/hackage.scm,
guix/import/texlive.scm, guix/import/utils.scm, guix/nar.scm,
guix/pki.scm, guix/scripts/archive.scm,
guix/scripts/authenticate.scm, guix/scripts/download.scm,
guix/scripts/hash.scm, guix/scripts/pack.scm,
guix/scripts/publish.scm, guix/scripts/refresh.scm,
guix/scripts/substitute.scm, guix/store.scm,
guix/store/deduplication.scm, guix/tests.scm, tests/base32.scm,
tests/builders.scm, tests/challenge.scm, tests/cpan.scm,
tests/crate.scm, tests/derivations.scm, tests/gem.scm,
tests/nar.scm, tests/opam.scm, tests/pki.scm,
tests/publish.scm, tests/pypi.scm, tests/store-deduplication.scm,
tests/store.scm, tests/substitute.scm: Adjust imports.
* gnu/system/vm.scm: Likewise.
(guile-sqlite3&co): Rename to...
(gcrypt-sqlite3&co): ... this. Add GUILE-GCRYPT.
(expression->derivation-in-linux-vm)[config]: Remove.
(iso9660-image)[config]: Remove.
(qemu-image)[config]: Remove.
(system-docker-image)[config]: Remove.
* guix/scripts/pack.scm: Adjust imports.
(guile-sqlite3&co): Rename to...
(gcrypt-sqlite3&co): ... this. Add GUILE-GCRYPT.
(self-contained-tarball)[build]: Call 'make-config.scm' without
#:libgcrypt argument.
(squashfs-image)[libgcrypt]: Remove.
[build]: Call 'make-config.scm' without #:libgcrypt.
(docker-image)[config, json]: Remove.
[build]: Add GUILE-GCRYPT to the extensions Remove (guix config) from
the imported modules.
* guix/self.scm (specification->package): Remove "libgcrypt", add
"guile-gcrypt".
(compiled-guix): Remove #:libgcrypt.
[guile-gcrypt]: New variable.
[dependencies]: Add it.
[*core-modules*]: Remove #:libgcrypt from 'make-config.scm' call.
Add #:extensions.
[*config*]: Remove #:libgcrypt from 'make-config.scm' call.
(%dependency-variables): Remove %libgcrypt.
(make-config.scm): Remove #:libgcrypt.
* build-aux/build-self.scm (guile-gcrypt): New variable.
(make-config.scm): Remove #:libgcrypt.
(build-program)[fake-gcrypt-hash]: New variable.
Add (gcrypt hash) to the imported modules. Adjust load path
assignments.
* gnu/packages/package-management.scm (guix)[propagated-inputs]: Add
GUILE-GCRYPT.
[arguments]: In 'wrap-program' phase, add GUILE-GCRYPT to the search
path.
| Ludovic Courtès |
| 2018-07-20 | database: Reset timestamps to one second after the Epoch....Previously, store items registered in the database by this code (for
instance, store items retrieved by 'guix offload' and passed to
'restore-file-set') would have an mtime of 0 instead of 1.
This would cause problems for things like .go files: Guile would
consider them to be older than the corresponding .scm file, and
consequently it would ignore them and possibly use another (incorrect)
.go file.
Reported by Ricardo Wurmus.
* guix/store/database.scm (reset-timestamps): Pass 1, not 0, to
'utime'.
* tests/store-database.scm ("register-path"): Check the mtime of FILE
and REF.
| Ludovic Courtès |
| 2018-07-20 | deduplication: Work around Guile bug in 'seek'....Fixes <https://bugs.gnu.org/32161>.
Reported by Ricardo Wurmus <rekado@elephly.net>.
This mostly reverts 83099892e0cf0d9c59f5e1a0774331026e48baa8.
* guix/store/deduplication.scm (counting-wrapper-port): New procedure.
(nar-sha256): Use it.
| Ludovic Courtès |
| 2018-07-19 | deduplication: Remove 'counting-wrapper-port'....* guix/store/deduplication.scm (counting-wrapper-port): Remove.
(nar-sha256): Call 'port-position' directly on PORT.
| Ludovic Courtès |
| 2018-07-03 | database: 'reset-timestamps' now correctly handles symlinks....* guix/store/database.scm (reset-timestamps): Use 'utime' with
AT_SYMLINK_NOFOLLOW for symlinks.
| Ludovic Courtès |
| 2018-07-03 | deduplication: Remove 'false-if-system-error', now unused....* guix/store/deduplication.scm (false-if-system-error): Remove.
| Ludovic Courtès |
| 2018-07-03 | deduplication: Place link files under /gnu/store/.links....Previously they'd always be placed next to TO-REPLACE, which would lead
to EPERM in some cases.
* guix/store/deduplication.scm (replace-with-link): Add #:swap-directory
parameter and honor it. Add call to 'make-file-writable'. Catch
'system-error' around 'rename-file'.
(deduplicate): Pass #:swap-directory and remove uses of
'false-if-system-error'.
* tests/store-deduplication.scm ("deduplicate"): Add 'chmod' call.
| Ludovic Courtès |
| 2018-07-03 | deduplication: Fix incorrect use of 'throw'....* guix/store/deduplication.scm (get-temp-link): In handler, fix call to
'throw'.
| Ludovic Courtès |
| 2018-06-14 | database: Allow for deterministic database construction....Fixes <https://bugs.gnu.org/21073>.
* guix/store/database.scm (sqlite-register): Add #:time.
(%epoch): New variable.
(register-items): Add #:registration-time. Pass #:time to
'sqlite-register'.
* gnu/build/install.scm (register-closure): Pass #:registration-time.
| Ludovic Courtès |
| 2018-06-14 | install: Use 'reset-timestamps' from (guix store database)....* gnu/build/install.scm (reset-timestamps): Remove.
* gnu/build/vm.scm: Use 'reset-timestamps' from (guix store database).
| Ludovic Courtès |
| 2018-06-14 | database: Add 'register-items'....* guix/build/store-copy.scm (store-info): Export.
* guix/store/database.scm (register-items): New procedure.
(register-path): Implement in terms of 'register-items'.
* gnu/build/install.scm (register-closure): Use 'register-items' instead
of 'for-each' and 'register-path'.
| Ludovic Courtès |
| 2018-06-14 | database: 'sqlite-register' takes a database, not a file name....* guix/store/database.scm (sqlite-register): Remove #:db-file and add
'db' parameter. Remove #:schema and 'parameterize'.
(register-path): Wrap 'sqlite-register' call in 'with-database' and in
'parameterize'.
* tests/store-database.scm ("new database")
("register-path with unregistered references"): Adjust accordingly.
| Ludovic Courtès |
| 2018-06-14 | database: 'reset-timestamps' sets file permissions as well....* guix/store/database.scm (reset-timestamps): Add 'chmod' calls.
| Ludovic Courtès |
| 2018-06-14 | database: Replace existing entries in Refs....* guix/store/database.scm (add-reference-sql): Add "OR REPLACE".
| Ludovic Courtès |
| 2018-06-14 | database: Add #:reset-timestamps? to 'register-path'....* guix/store/database.scm (register-path): Add #:reset-timestamps? and
honor it.
| Ludovic Courtès |
| 2018-06-14 | database: Remove extra SQL parameter in 'update-or-insert'....* guix/store/database.scm (update-or-insert): Remove extra #:path
parameter.
| Ludovic Courtès |
| 2018-06-14 | deduplicate: Fix a couple of thinkos....* guix/store/deduplication.scm (get-temp-link): Turn 'args' in the 'catch'
handler into a rest argument.
(deduplicate): Use 'lstat' instead of 'file-is-directory?' to properly
handle symlinks. When iterating over the result of 'scandir', exclude
the ".links" sub-directory.
* tests/store-deduplication.scm ("deduplicate"): Create sub-directories
and call 'deduplicate' directly on STORE.
| Ludovic Courtès |
| 2018-06-14 | database: 'register-path' creates the database directory if needed....* guix/store/database.scm (register-path): Call 'mkdir-p'.
| Ludovic Courtès |
| 2018-06-14 | database: Provide a way to specify the schema location....* guix/store/database.scm (sqlite-register): Add #:schema. Parameterize
'sql-schema' based on this.
(register-path): Add #:schema and pass it to 'sqlite-register'.
| Ludovic Courtès |
| 2018-06-14 | database: Fail registration when encountering unregistered references....* guix/store/database.scm (add-reference-sql): Remove nested SELECT.
(add-references): Expect REFERENCES to be a list of ids.
(sqlite-register): Call 'path-id' for each of REFERENCES and pass it to
'add-references'.
* tests/store-database.scm ("register-path with unregistered references"):
New test.
| Ludovic Courtès |
| 2018-06-14 | database: 'with-database' can now initialize new databases....* nix/libstore/schema.sql: Rename to...
* guix/store/schema.sql: ... this.
* Makefile.am (nobase_dist_guilemodule_DATA): Add it.
* nix/local.mk (%D%/libstore/schema.sql.hh): Adjust accordingly.
* guix/store/database.scm (sql-schema): New variable.
(sqlite-exec, initialize-database, call-with-database): New procedures.
(with-database): Rewrite in terms of 'call-with-database'.
* tests/store-database.scm ("new database"): New test.
* guix/self.scm (compiled-guix)[*core-modules*]: Add 'schema.sql' to
#:extra-files.
| Ludovic Courtès |
