summaryrefslogtreecommitdiff
path: root/guix/store
AgeCommit message (Collapse)Author
2020-09-14database: register-items: reduce transaction scope.Christopher Baines
It was made transactional in a4678c6ba18d8dbd79d931f80426eebf61be7ebe, with the reasoning to prevent broken intermediate states from being visible. I think this means something like an entry being in ValidPaths, but the Refs not being inserted. Using a transaction for this makes sense, but I think using one single transaction for the whole register-items call is unnecessary to avoid broken states from being visible, and could block other writes to the store database while register-items is running. Because the deduplication and resetting timestamps happens within the transaction as well, even though these things don't involve the database, writes to the database will still be blocked while this is happening. To reduce the potential for register-items to block other writers to the database for extended periods, this commit moves the transaction to just wrap the call to sqlite-register. This is the one place where writes occur, so that should prevent the broken intermediate states issue above. The one difference this will make is some of the registered items will be visible to other connections while others may be still being added. I think this is OK, as it's equivalent to just registering different items. * guix/store/database.scm (register-items): Reduce transaction scope. Signed-off-by: Ludovic Courtès <ludo@gnu.org>
2020-09-14database: document extra registration requirements.Caleb Ristvedt
It's necessary that store items be locked and protected from garbage collection while they are being registered. This documents that. * guix/store/database.scm (register-path, register-items): document GC protection and locking requirements. Signed-off-by: Ludovic Courtès <ludo@gnu.org>
2020-09-14deduplication: pass store directory to replace-with-link.Caleb Ristvedt
This causes with-writable-file to take into consideration the actual store being used, as passed to 'deduplicate', rather than whatever (%store-directory) may return. * guix/store/deduplication.scm (replace-with-link): new keyword argument 'store'. Pass to with-writable-file. (with-writable-file, call-with-writable-file): new store argument. (deduplicate): pass store to replace-with-link. Signed-off-by: Ludovic Courtès <ludo@gnu.org>
2020-07-28store: deduplication: Handle fs without d_type support.Mathieu Othacehe
scandir* uses readdir, which means that the file type property can be 'unknown if the underlying file-system does not support d_type. Make sure to fallback to lstat in that case. Fixes: https://issues.guix.gnu.org/issue/42579. * guix/store/deduplication.scm (deduplicate): Handle the case where properties is 'unknown because the underlying file-system does not support d_type.
2020-07-11image: Do not set journal_model=WAL for the Hurd.Jan (janneke) Nieuwenhuizen
This fixes <https://bugs.gnu.org/42151>. * gnu/system/images/hurd.scm (hurd-initialize-root-partition): Use #:wal-mode #f in call to ... * gnu/build/image.scm (initialize-root-partition): ... this, add #:wal-mode? parameter, pass it to ... (register-closure): ... this, add #:wal-mode? parameter, pass it to ... * guix/store/database.scm (with-database): ... this, add #:wal-mode? parameter, pass it to ... (call-with-database): ... this, add #:wal-mode? parameter; when set to #f, do not set journal_model=WAL.
2020-06-25deduplication: Leave the store permissions unchanged.Ludovic Courtès
Suggested by Caleb Ristvedt <caleb.ristvedt@cune.org>. * guix/store/deduplication.scm (call-with-writable-file): Call THUNK directly when FILE is (%store-directory).
2020-06-25deduplication: Fix default value of #:store in 'deduplicate'.Ludovic Courtès
* guix/store/deduplication.scm (deduplicate): Change #:store default value to (%store-directory).
2020-06-25deduplication: Use 'dynamic-wind' when changing permissions of the parent.Ludovic Courtès
Suggested by Caleb Ristvedt <caleb.ristvedt@cune.org>. * guix/store/deduplication.scm (call-with-writable-file): New procedure. (with-writable-file): New macro. (replace-with-link): Use it.
2020-06-22deduplicate: Avoid traversing directories twice.Ludovic Courtès
Until now, we'd call (nar-sha256 file) unconditionally. Thus, if FILE was a directory, we would traverse it for no reason, and then call 'deduplicate' on FILE, which would again traverse it. This change also removes redundant (mkdir-p store) calls from the loop, and avoids 'lstat' calls by using 'scandir*'. * guix/store/deduplication.scm (deduplicate): Add named loop. Move 'mkdir-p' outside the loop. Use 'scandir*' instead of 'scandir'. Do not call 'nar-sha256' when FILE has type 'directory.
2020-06-18database: 'register-items' takes an open database.Ludovic Courtès
* guix/store/database.scm (store-database-directory) (store-database-file): New procedures. (call-with-database): Add call to 'mkdir-p'. (register-items): Add 'db' parameter and remove #:state-directory and #:schema. (register-path): Use 'store-database-file' and 'with-database', and parameterize SQL-SCHEMA. * gnu/build/image.scm (register-closure): Likewise. * gnu/build/vm.scm (register-closure): Likewise. * guix/scripts/pack.scm (store-database)[build]: Likewise.
2020-06-10database: separate transaction-handling and retry-handling.Caleb Ristvedt
Previously call-with-transaction would both retry when SQLITE_BUSY errors were thrown and do what its name suggested (start and rollback/commit a transaction). This changes it to do only what its name implies, which simplifies its implementation. Retrying is provided by the new call-with-SQLITE_BUSY-retrying procedure. * guix/store/database.scm (call-with-transaction): no longer restarts, new #:restartable? argument controls whether "begin" or "begin immediate" is used. (call-with-SQLITE_BUSY-retrying, call-with-retrying-transaction, call-with-retrying-savepoint): new procedures. (register-items): use call-with-retrying-transaction to preserve old behavior. * .dir-locals.el (call-with-retrying-transaction, call-with-retrying-savepoint): add indentation information.
2020-06-10database: ensure update-or-insert is run within a transactionCaleb Ristvedt
update-or-insert can break if an insert occurs between when it decides whether to update or insert and when it actually performs that operation. Putting the check and the update/insert operation in the same transaction ensures that the update/insert will only succeed if no other write has occurred in the middle. * guix/store/database.scm (call-with-savepoint): new procedure. (update-or-insert): use call-with-savepoint to ensure the read and the insert/update occur within the same transaction.
2020-06-10database: rewrite query procedures in terms of with-statement.Caleb Ristvedt
Most of our queries would fail to finalize their statements properly if sqlite returned an error during their execution. This resolves that, and also makes them somewhat more concise as a side-effect. This also makes some small changes to improve certain queries where behavior was strange or overly verbose. * guix/store/database.scm (call-with-statement): new procedure. (with-statement): new macro. (last-insert-row-id, path-id, update-or-insert, add-references): rewrite to use with-statement. (update-or-insert): factor last-insert-row-id out of the end of both branches. (add-references): remove pointless last-insert-row-id call. * .dir-locals.el (with-statement): add indenting information.
2020-06-10database: work around guile-sqlite3 bug preventing statement resetCaleb Ristvedt
guile-sqlite3 provides statement caching, making it unnecessary for sqlite to keep re-preparing statements that are frequently used. Unfortunately it doesn't quite emulate the semantics of sqlite_finalize properly, because it doesn't cause a commit if the statement being finalized is the last "active" statement (see https://notabug.org/guile-sqlite3/guile-sqlite3/issues/12). We work around this by wrapping sqlite-finalize with our own version that ensures sqlite-reset is called, which does The Right Thing™. * guix/store/database.scm (sqlite-finalize): new procedure that shadows the sqlite-finalize from (sqlite3).
2020-04-11database: 'reset-timestamps' can optionally preserve permissions.Ludovic Courtès
* guix/store/database.scm (reset-timestamps): Add #:preserve-permissions? and honor it.
2020-02-22deduplication: Use nix-base32 encoding for link names.Ludovic Courtès
Fixes <https://bugs.gnu.org/39725>. * guix/store/deduplication.scm (deduplicate): Use 'bytevector->nix-base32-string' instead of 'bytevector->base16-string'.
2019-09-08daemon: Invoke 'guix gc --list-busy' instead of 'list-runtime-roots'.Ludovic Courtès
* nix/scripts/list-runtime-roots.in: Remove. * guix/store/roots.scm (%proc-directory): New variable. (proc-file-roots, proc-exe-roots, proc-cwd-roots) (proc-fd-roots, proc-maps-roots, proc-environ-roots) (referenced-files, canonicalize-store-item, busy-store-items): New procedures, taken from 'list-runtime-roots.in'. * nix/libstore/globals.hh (Settings)[guixProgram]: New field. * nix/libstore/globals.cc (Settings::processEnvironment): Initialize 'guixProgram'. * nix/libstore/gc.cc (addAdditionalRoots): Drop code related to 'NIX_ROOT_FINDER'. Run "guix gc --list-busy". * nix/local.mk (nodist_pkglibexec_SCRIPTS): Remove 'scripts/list-runtime-roots'. * config-daemon.ac: Don't output nix/scripts/list-runtime-roots. * build-aux/pre-inst-env.in: Don't set 'NIX_ROOT_FINDER'. Set 'GUIX'. * doc/guix.texi (Invoking guix gc): Document '--list-busy'. * guix/scripts/gc.scm (show-help, %options): Add "--list-busy". (guix-gc)[list-busy]: New procedure. Handle the 'list-busy' action.
2019-04-25gnu, guix: Yearly ritual purging of the filesystems.Tobias Geerinckx-Rice
* gnu/packages/android.scm (android-ext4-utils)[synopsis]: Fix ‘file system’ spelling. * gnu/packages/disk.scm (rmlint)[synopsis, description]: Likewise. * gnu/packages/golang.scm (go-github-com-kr-fs)[synopsis, description]: Likewise & edit for grammar. * gnu/packages/ipfs.scm (gx, go-ipfs)[description]: Likewise. * /gnu/packages/java.scm (java-commons-vfs)[synopsis]: Likewise. * gnu/packages/linux.scm (fuseiso)[description]: Likewise. (genext2fs)[synopsis, description]: Likewise. * gnu/packages/package-management.scm (libostree)[description]: Likewise. * gnu/packages/python-xyz.scm (python-requests-file)[description]: Likewise & mark up. * gnu/packages/rails.scm (ruby-with-advisory-lock)[description]: Likewise. * gnu/packages/ruby.scm (ruby-rerun)[description]: Likewise. * guix/build/go-build-system.scm (setup-go-environment)<docstring>: Likewise. * guix/store/deduplication.scm (get-temp-link)<docstring>: Likewise.
2019-04-10Add (guix store roots).Ludovic Courtès
* guix/store/roots.scm, tests/store-roots.scm: New files. * Makefile.am (STORE_MODULES): Add guix/store/roots.scm. (SCM_TESTS): Add tests/store-roots.scm.
2019-03-07database: Make 'register-items' transactional.Caleb Ristvedt
* guix/store/database.scm (SQLITE_BUSY, register-output-sql): New variables. (add-references): Don't try finalizing after each use, only after all the uses (otherwise a finalized statement would be used if #:cache? was #f). (call-with-transaction): New procedure. (register-items): Use call-with-transaction to prevent broken intermediate states from being visible. * .dir-locals.el (call-with-transaction): indent it. Signed-off-by: Ludovic Courtès <ludo@gnu.org>
2019-01-23deduplication: Ignore EMLINK.Ludovic Courtès
Until now 'guix offload' would fail (transient failure) upon EMLINK. * guix/store/deduplication.scm (replace-with-link) (deduplicate): Ignore EMLINK.
2018-12-21database: Use "write-ahead log" mode and set a long "busy timeout".Ludovic Courtès
This should avoid "database is locked" errors when there's a lot of concurrency, for instance when offloading simultaneously a lot of builds. * guix/store/database.scm (call-with-database): Add two 'sqlite-exec' calls to set 'journal_mode' and 'busy_timeout'.
2018-12-14deduplication: Gracefully handle ENOSPC raised by 'link' calls.Ludovic Courtès
Reported by Andreas Enge <andreas@enge.fr> in <https://bugs.gnu.org/33676>. * guix/store/deduplication.scm (replace-with-link): Catch ENOSPC around 'get-temp-link'. Do nothing when 'get-temp-link' throws ENOSPC. Move code to restore PARENT's permissions outside of 'catch'. * tests/store-deduplication.scm ("deduplicate, ENOSPC"): New test.
2018-11-23Update Guile-SQLite3 URL everywhere.Ludovic Courtès
* README: Update Guile-SQLite3 URL. * doc/guix.texi (Requirements): Likewise. * guix/store/database.scm (sqlite-exec): Likewise. * m4/guix.m4 (GUIX_CHECK_GUILE_SQLITE3): Likewise.
2018-11-13deduplication: Restore directory mtime and permissions after deduplication.Ludovic Courtès
Fixes <https://bugs.gnu.org/33361>. * guix/store/deduplication.scm (replace-with-link): Call 'set-file-time' and 'chmod' after 'rename-file'. * tests/nar.scm ("restore-file-set with directories (signed, valid)"): New test.
2018-11-13nar: Access the database instead of connecting to the daemon.Ludovic Courtès
* guix/store/database.scm (%default-database-file): New variable. (path-id): Export. * guix/nar.scm (finalize-store-file): Use 'with-database' instead of 'with-store', and use 'path-id' instead of 'valid-path?'.
2018-09-23database: Register each store item only once.Ludovic Courtès
Fixes <https://bugs.gnu.org/32600>. Reported by Leo Famulari. * guix/store/database.scm (register-items): Check whether TO-REGISTER is in DB by calling 'path-id', and skip the reset-timestamps, registration, and deduplication phases when it is.
2018-09-23database: 'register-items' shows a progress bar.Ludovic Courtès
* guix/store/database.scm (register-items): Add #:log-port. Use 'progress-reporter/bar' to show a progress report. (register-path): Pass #:log-port to 'register-items'.
2018-09-04Switch to Guile-Gcrypt.Ludovic Courtès
This removes (guix hash) and (guix pk-crypto), which now live as part of Guile-Gcrypt (version 0.1.0.) * guix/gcrypt.scm, guix/hash.scm, guix/pk-crypto.scm, tests/hash.scm, tests/pk-crypto.scm: Remove. * configure.ac: Test for Guile-Gcrypt. Remove LIBGCRYPT and LIBGCRYPT_LIBDIR assignments. * m4/guix.m4 (GUIX_ASSERT_LIBGCRYPT_USABLE): Remove. * README: Add Guile-Gcrypt to the dependencies; move libgcrypt as "required unless --disable-daemon". * doc/guix.texi (Requirements): Likewise. * gnu/packages/bash.scm, guix/derivations.scm, guix/docker.scm, guix/git.scm, guix/http-client.scm, guix/import/cpan.scm, guix/import/cran.scm, guix/import/crate.scm, guix/import/elpa.scm, guix/import/gnu.scm, guix/import/hackage.scm, guix/import/texlive.scm, guix/import/utils.scm, guix/nar.scm, guix/pki.scm, guix/scripts/archive.scm, guix/scripts/authenticate.scm, guix/scripts/download.scm, guix/scripts/hash.scm, guix/scripts/pack.scm, guix/scripts/publish.scm, guix/scripts/refresh.scm, guix/scripts/substitute.scm, guix/store.scm, guix/store/deduplication.scm, guix/tests.scm, tests/base32.scm, tests/builders.scm, tests/challenge.scm, tests/cpan.scm, tests/crate.scm, tests/derivations.scm, tests/gem.scm, tests/nar.scm, tests/opam.scm, tests/pki.scm, tests/publish.scm, tests/pypi.scm, tests/store-deduplication.scm, tests/store.scm, tests/substitute.scm: Adjust imports. * gnu/system/vm.scm: Likewise. (guile-sqlite3&co): Rename to... (gcrypt-sqlite3&co): ... this. Add GUILE-GCRYPT. (expression->derivation-in-linux-vm)[config]: Remove. (iso9660-image)[config]: Remove. (qemu-image)[config]: Remove. (system-docker-image)[config]: Remove. * guix/scripts/pack.scm: Adjust imports. (guile-sqlite3&co): Rename to... (gcrypt-sqlite3&co): ... this. Add GUILE-GCRYPT. (self-contained-tarball)[build]: Call 'make-config.scm' without #:libgcrypt argument. (squashfs-image)[libgcrypt]: Remove. [build]: Call 'make-config.scm' without #:libgcrypt. (docker-image)[config, json]: Remove. [build]: Add GUILE-GCRYPT to the extensions Remove (guix config) from the imported modules. * guix/self.scm (specification->package): Remove "libgcrypt", add "guile-gcrypt". (compiled-guix): Remove #:libgcrypt. [guile-gcrypt]: New variable. [dependencies]: Add it. [*core-modules*]: Remove #:libgcrypt from 'make-config.scm' call. Add #:extensions. [*config*]: Remove #:libgcrypt from 'make-config.scm' call. (%dependency-variables): Remove %libgcrypt. (make-config.scm): Remove #:libgcrypt. * build-aux/build-self.scm (guile-gcrypt): New variable. (make-config.scm): Remove #:libgcrypt. (build-program)[fake-gcrypt-hash]: New variable. Add (gcrypt hash) to the imported modules. Adjust load path assignments. * gnu/packages/package-management.scm (guix)[propagated-inputs]: Add GUILE-GCRYPT. [arguments]: In 'wrap-program' phase, add GUILE-GCRYPT to the search path.
2018-07-20database: Reset timestamps to one second after the Epoch.Ludovic Courtès
Previously, store items registered in the database by this code (for instance, store items retrieved by 'guix offload' and passed to 'restore-file-set') would have an mtime of 0 instead of 1. This would cause problems for things like .go files: Guile would consider them to be older than the corresponding .scm file, and consequently it would ignore them and possibly use another (incorrect) .go file. Reported by Ricardo Wurmus. * guix/store/database.scm (reset-timestamps): Pass 1, not 0, to 'utime'. * tests/store-database.scm ("register-path"): Check the mtime of FILE and REF.
2018-07-20deduplication: Work around Guile bug in 'seek'.Ludovic Courtès
Fixes <https://bugs.gnu.org/32161>. Reported by Ricardo Wurmus <rekado@elephly.net>. This mostly reverts 83099892e0cf0d9c59f5e1a0774331026e48baa8. * guix/store/deduplication.scm (counting-wrapper-port): New procedure. (nar-sha256): Use it.
2018-07-19deduplication: Remove 'counting-wrapper-port'.Ludovic Courtès
* guix/store/deduplication.scm (counting-wrapper-port): Remove. (nar-sha256): Call 'port-position' directly on PORT.
2018-07-03database: 'reset-timestamps' now correctly handles symlinks.Ludovic Courtès
* guix/store/database.scm (reset-timestamps): Use 'utime' with AT_SYMLINK_NOFOLLOW for symlinks.
2018-07-03deduplication: Remove 'false-if-system-error', now unused.Ludovic Courtès
* guix/store/deduplication.scm (false-if-system-error): Remove.
2018-07-03deduplication: Place link files under /gnu/store/.links.Ludovic Courtès
Previously they'd always be placed next to TO-REPLACE, which would lead to EPERM in some cases. * guix/store/deduplication.scm (replace-with-link): Add #:swap-directory parameter and honor it. Add call to 'make-file-writable'. Catch 'system-error' around 'rename-file'. (deduplicate): Pass #:swap-directory and remove uses of 'false-if-system-error'. * tests/store-deduplication.scm ("deduplicate"): Add 'chmod' call.
2018-07-03deduplication: Fix incorrect use of 'throw'.Ludovic Courtès
* guix/store/deduplication.scm (get-temp-link): In handler, fix call to 'throw'.
2018-06-14database: Allow for deterministic database construction.Ludovic Courtès
Fixes <https://bugs.gnu.org/21073>. * guix/store/database.scm (sqlite-register): Add #:time. (%epoch): New variable. (register-items): Add #:registration-time. Pass #:time to 'sqlite-register'. * gnu/build/install.scm (register-closure): Pass #:registration-time.
2018-06-14install: Use 'reset-timestamps' from (guix store database).Ludovic Courtès
* gnu/build/install.scm (reset-timestamps): Remove. * gnu/build/vm.scm: Use 'reset-timestamps' from (guix store database).
2018-06-14database: Add 'register-items'.Ludovic Courtès
* guix/build/store-copy.scm (store-info): Export. * guix/store/database.scm (register-items): New procedure. (register-path): Implement in terms of 'register-items'. * gnu/build/install.scm (register-closure): Use 'register-items' instead of 'for-each' and 'register-path'.
2018-06-14database: 'sqlite-register' takes a database, not a file name.Ludovic Courtès
* guix/store/database.scm (sqlite-register): Remove #:db-file and add 'db' parameter. Remove #:schema and 'parameterize'. (register-path): Wrap 'sqlite-register' call in 'with-database' and in 'parameterize'. * tests/store-database.scm ("new database") ("register-path with unregistered references"): Adjust accordingly.
2018-06-14database: 'reset-timestamps' sets file permissions as well.Ludovic Courtès
* guix/store/database.scm (reset-timestamps): Add 'chmod' calls.
2018-06-14database: Replace existing entries in Refs.Ludovic Courtès
* guix/store/database.scm (add-reference-sql): Add "OR REPLACE".
2018-06-14database: Add #:reset-timestamps? to 'register-path'.Ludovic Courtès
* guix/store/database.scm (register-path): Add #:reset-timestamps? and honor it.
2018-06-14database: Remove extra SQL parameter in 'update-or-insert'.Ludovic Courtès
* guix/store/database.scm (update-or-insert): Remove extra #:path parameter.
2018-06-14deduplicate: Fix a couple of thinkos.Ludovic Courtès
* guix/store/deduplication.scm (get-temp-link): Turn 'args' in the 'catch' handler into a rest argument. (deduplicate): Use 'lstat' instead of 'file-is-directory?' to properly handle symlinks. When iterating over the result of 'scandir', exclude the ".links" sub-directory. * tests/store-deduplication.scm ("deduplicate"): Create sub-directories and call 'deduplicate' directly on STORE.
2018-06-14database: 'register-path' creates the database directory if needed.Ludovic Courtès
* guix/store/database.scm (register-path): Call 'mkdir-p'.
2018-06-14database: Provide a way to specify the schema location.Ludovic Courtès
* guix/store/database.scm (sqlite-register): Add #:schema. Parameterize 'sql-schema' based on this. (register-path): Add #:schema and pass it to 'sqlite-register'.
2018-06-14database: Fail registration when encountering unregistered references.Ludovic Courtès
* guix/store/database.scm (add-reference-sql): Remove nested SELECT. (add-references): Expect REFERENCES to be a list of ids. (sqlite-register): Call 'path-id' for each of REFERENCES and pass it to 'add-references'. * tests/store-database.scm ("register-path with unregistered references"): New test.
2018-06-14database: 'with-database' can now initialize new databases.Ludovic Courtès
* nix/libstore/schema.sql: Rename to... * guix/store/schema.sql: ... this. * Makefile.am (nobase_dist_guilemodule_DATA): Add it. * nix/local.mk (%D%/libstore/schema.sql.hh): Adjust accordingly. * guix/store/database.scm (sql-schema): New variable. (sqlite-exec, initialize-database, call-with-database): New procedures. (with-database): Rewrite in terms of 'call-with-database'. * tests/store-database.scm ("new database"): New test. * guix/self.scm (compiled-guix)[*core-modules*]: Add 'schema.sql' to #:extra-files.
2018-06-01Add (guix store deduplication).Caleb Ristvedt
* guix/store/database.scm (register-path): Add #:deduplicate? and call 'deduplicate' when it's true. (counting-wrapper-port, nar-sha256): Move to... * guix/store/deduplication.scm: ... here. New file. * tests/store-deduplication.scm: New file. * Makefile.am (STORE_MODULES): Add deduplication.scm. (SCM_TESTS) [HAVE_GUILE_SQLITE3]: Add store-deduplication.scm. Co-authored-by: Ludovic Courtès <ludo@gnu.org>
generated by cgit v1.2.3 (git 2.46.2) at 2024-10-31 03:29:06 +0000