summaryrefslogtreecommitdiff
path: root/guix/store
AgeCommit message (Collapse)Author
2023-06-18store: Tolerate non-existent GC root directories.Ludovic Courtès
* guix/store/roots.scm (gc-roots): Wrap 'scandir*' call in 'catch'. * tests/store-roots.scm ("gc-roots, initial"): New test. Move 'open-connection' call below.
2023-03-13guix: Strip #:use-module lists.Ludovic Courtès
This was obtained by setting up this environment: guix shell -D guix --with-input=guile@3.0.9=guile-next \ --with-commit=guile-next=e2ed33ef0445c867fe56c247054aa67e834861f2 -- make -j5 then adding 'unused-module' to (@@ (guix build compiler) %warnings), building, and checking all the "unused module" warnings and removing those that were definitely unused.
2022-12-10deduplicate: Use 'sendfile' only with file ports.Ludovic Courtès
Fixes a regression introduced in b129026e2e242e9068158ae6e6fcd8d7c5ea092e. * guix/store/deduplication.scm (dump-file/deduplicate): Use 'sendfile' only when INPUT' is a file port.
2022-12-10deduplicate: Use 'sendfile' for small file copies.Ludovic Courtès
* guix/store/deduplication.scm (dump-file/deduplicate): Use 'sendfile' instead of 'dump-port'. * tests/store-deduplication.scm ("copy-file/deduplicate, below %deduplication-minimum-size"): New test.
2022-11-06guix gc: Add '--vacuum-database'.Efraim Flashner
* guix/scripts/gc.scm (show-help, %options): Add '--vacuum-database'. * guix/store/database.scm (vacuum-database): New procedure. * doc/guix.texi (Invoking guix gc): Document the option.
2022-05-20guix: Remove duplicate definitions of dump-port.Ricardo Wurmus
* guix/store.scm (dump-port): Remove procedure; autoload from (guix build utils) instead. * guix/store/deduplication.scm (dump-port): Remove procedure; unhide dump-port from (guix build utils).
2021-11-17Merge branch 'master' into core-updates-frozenLudovic Courtès
2021-11-16daemon: Do not deduplicate files smaller than 8 KiB.Ludovic Courtès
Files smaller than 8 KiB typically represent ~70% of the entries in /gnu/store/.links but only contribute to ~4% of the space savings afforded by deduplication. Not considering these files for deduplication speeds up file insertion in the store and, more importantly, leaves 'removeUnusedLinks' with fewer entries to traverse, thereby speeding it up proportionally. Partly fixes <https://issues.guix.gnu.org/24937>. * config-daemon.ac: Remove symlink hard link check and CAN_LINK_SYMLINK definition. * guix/store/deduplication.scm (%deduplication-minimum-size): New variable. (deduplicate)[loop]: Do not recurse when FILE's size is below %DEDUPLICATION-MINIMUM-SIZE. (dump-port): New procedure. (dump-file/deduplicate)[hash]: Turn into... [dump-and-compute-hash]: ... this thunk. Call 'deduplicate' only when SIZE is greater than %DEDUPLICATION-MINIMUM-SIZE; otherwise call 'dump-port'. * nix/libstore/gc.cc (LocalStore::removeUnusedLinks): Drop files where st.st_size < deduplicationMinSize. * nix/libstore/local-store.hh (deduplicationMinSize): New declaration. * nix/libstore/optimise-store.cc (deduplicationMinSize): New variable. (LocalStore::optimisePath_): Return when PATH is a symlink or smaller than 'deduplicationMinSize'. * tests/derivations.scm ("identical files are deduplicated"): Produce files bigger than %DEDUPLICATION-MINIMUM-SIZE. * tests/nar.scm ("restore-file-set with directories (signed, valid)"): Likewise. * tests/store-deduplication.scm ("deduplicate, below %deduplication-minimum-size"): New test. ("deduplicate", "deduplicate, ENOSPC"): Produce files bigger than %DEDUPLICATION-MINIMUM-SIZE. * tests/store.scm ("substitute, deduplication"): Likewise.
2021-02-03Merge branch 'master' into core-updatesChristopher Baines
2021-01-31database: Validate #:nar-size and #:time when registering store items.Ludovic Courtès
* guix/store/database.scm (assert-integer): New procedure. (update-or-insert): Use it to validate NAR-SIZE and TIME. * tests/store-database.scm ("sqlite-register with incorrect size"): New test.
2021-01-31database: Remove workarounds unnecessary with guile-sqlite3 0.1.2.Ludovic Courtès
* guix/store/database.scm (sqlite-exec, sqlite-finalize): Remove.
2021-01-13Merge branch 'staging' into 'core-updates'.Maxim Cournoyer
Conflicts: gnu/local.mk gnu/packages/cmake.scm gnu/packages/curl.scm gnu/packages/gl.scm gnu/packages/glib.scm gnu/packages/guile.scm gnu/packages/node.scm gnu/packages/openldap.scm gnu/packages/package-management.scm gnu/packages/python-xyz.scm gnu/packages/python.scm gnu/packages/tls.scm gnu/packages/vpn.scm gnu/packages/xorg.scm
2020-12-19maint: Require Guile >= 2.2.6.Ludovic Courtès
* configure.ac: For Guile 2.2, require 2.2.6 or later. * guix/gexp.scm (define-syntax-parameter-once): Remove. Use 'define-syntax-parameter' instead. * guix/mnoads.scm: Likewise. * guix/inferior.scm (proxy)[select*]: Remove. * guix/scripts/publish.scm <top level>: Remove replacement for (@@ (web http) read-header-line). * guix/store/deduplication.scm (counting-wrapper-port): Remove. (nar-sha256): Call 'port-position' on PORT to compute SIZE.
2020-12-15deduplicate: Create the '.links' directory lazily.Ludovic Courtès
This avoids repeated (mkdir-p "/gnu/store/.links") calls when deduplicating lots of files. * guix/store/deduplication.scm (deduplicate): Remove initial call to 'mkdir-p'. Add ENOENT case in 'link' exception handler. Reindent. * tests/store-deduplication.scm ("deduplicate, ENOSPC"): Check for (<= links 4) to account for the initial 'link' call.
2020-12-15database: Honor 'SOURCE_DATE_EPOCH'.Ludovic Courtès
* guix/store/database.scm (timestamp): New procedure. (sqlite-register): Use it as the default for #:time. (register-items): Likewise for #:registeration-time.
2020-12-15database: Remove 'register-path'.Ludovic Courtès
* guix/store/database.scm (register-path): Remove. * tests/store-database.scm ("register-path"): Rename to... ("register-items"): ... this, and use 'register-items' instead of 'register-path'. ("register-path, directory"): Rename to... ("register-items, directory"): ... this, and use 'register-items' instead of 'register-path'. ("register-path with unregistered references"): Rename to... ("sqlite-register with unregistered references"): ... this.
2020-12-15system: 'init' does not recompute the hash of each store item.Ludovic Courtès
Fixes <https://bugs.gnu.org/44760>. Previously, the 'register-path' call would re-traverse ITEM to compute its nar hash, even though that hash is already known in the initial store. This patch also avoids repeated opening/closing of the database. * guix/store/database.scm (call-with-database): Export. * guix/scripts/system.scm (copy-item): Add 'db' parameter. Call 'sqlite-register' instead of 'register-path'. (copy-closure): Remove redundant call to 'references*'. Call 'call-with-database' and pass the database to 'copy-item'.
2020-12-15database: Remove #:deduplicate? and #:reset-timestamps? from 'register-path'.Ludovic Courtès
* guix/store/database.scm (register-path): Remove #:deduplicate? and #:reset-timestamps?. * guix/scripts/system.scm (copy-item): Adjust accordingly. * tests/store-database.scm ("register-path") ("register-path, directory"): Call 'reset-timestamps'.
2020-12-15database: Remove #:deduplicate? from 'register-items'.Ludovic Courtès
It is now up to the caller to deduplicate store contents. * guix/store/database.scm (register-items): Remove #:deduplicate? parameter and call to 'deduplicate'. (register-path): Call 'deduplicate' when #:deduplicate? is true. * gnu/build/image.scm (register-closure): Adjust call accordingly. * gnu/build/vm.scm (register-closure): Likewise. * guix/nar.scm (finalize-store-file): Likewise. * guix/scripts/pack.scm (store-database): Likewise.
2020-12-15store-copy: 'populate-store' can optionally deduplicate files.Ludovic Courtès
Until now deduplication was performed as an additional pass after copying files, which involve re-traversing all the files that had just been copied. * guix/store/deduplication.scm (copy-file/deduplicate): New procedure. * tests/store-deduplication.scm ("copy-file/deduplicate"): New test. * guix/build/store-copy.scm (populate-store): Add #:deduplicate? parameter and honor it. * tests/gexp.scm ("gexp->derivation, store copy"): Pass #:deduplicate? #f to 'populate-store'. * gnu/build/image.scm (initialize-root-partition): Pass #:deduplicate? to 'populate-store'. Pass #:deduplicate? #f to 'register-closure'. * gnu/build/vm.scm (root-partition-initializer): Likewise. * gnu/build/install.scm (populate-single-profile-directory): Pass #:deduplicate? #f to 'populate-store'. * gnu/build/linux-initrd.scm (build-initrd): Likewise. * guix/scripts/pack.scm (self-contained-tarball)[import-module?]: New procedure. [build]: Pass it as an argument to 'source-module-closure'. * guix/scripts/pack.scm (squashfs-image)[build]: Wrap in 'with-extensions'. * gnu/system/linux-initrd.scm (expression->initrd)[import-module?]: New procedure. [builder]: Pass it to 'source-module-closure'. * gnu/system/install.scm (cow-store-service-type)[import-module?]: New procedure. Pass it to 'source-module-closure'.
2020-12-15database: Remove #:reset-timestamps? from 'register-items'.Ludovic Courtès
The assumption now is that the caller took care of resetting timestamps and permissions. * guix/store/database.scm (register-items): Remove #:reset-timestamps? parameter and the call to 'reset-timestamps'. (register-path): Adjust accordingly and add call to 'reset-timestamps'. * gnu/build/image.scm (register-closure): Remove #:reset-timestamps? parameter to 'register-items'. * gnu/build/vm.scm (register-closure): Likewise. * guix/nar.scm (finalize-store-file): Adjust accordingly. * guix/scripts/pack.scm (store-database)[build]: Likewise.
2020-12-15nar: Deduplicate files right as they are restored.Ludovic Courtès
This avoids having to traverse and re-read the files that we have just restored, thereby reducing I/O. * guix/serialization.scm (dump-file): New procedure. (restore-file): Add #:dump-file parameter and honor it. * guix/store/deduplication.scm (tee, dump-file/deduplicate): New procedures. * guix/nar.scm (restore-one-item): Pass #:dump-file to 'restore-file'. (finalize-store-file): Pass #:deduplicate? #f to 'register-items'. * tests/nar.scm <top level>: Call 'setenv' to set "NIX_STORE".
2020-12-08database: Remove unnecessary module imports.Ludovic Courtès
* guix/store/database.scm: Remove unnecessary imports added in 4b9eecd322e566783369795ebea63a479b51f486.
2020-10-16store: Fix docstring typo.Tobias Geerinckx-Rice
* guix/store/roots.scm (gc-roots): Fix typo in docstring.
2020-09-14database: register-items: reduce transaction scope.Christopher Baines
It was made transactional in a4678c6ba18d8dbd79d931f80426eebf61be7ebe, with the reasoning to prevent broken intermediate states from being visible. I think this means something like an entry being in ValidPaths, but the Refs not being inserted. Using a transaction for this makes sense, but I think using one single transaction for the whole register-items call is unnecessary to avoid broken states from being visible, and could block other writes to the store database while register-items is running. Because the deduplication and resetting timestamps happens within the transaction as well, even though these things don't involve the database, writes to the database will still be blocked while this is happening. To reduce the potential for register-items to block other writers to the database for extended periods, this commit moves the transaction to just wrap the call to sqlite-register. This is the one place where writes occur, so that should prevent the broken intermediate states issue above. The one difference this will make is some of the registered items will be visible to other connections while others may be still being added. I think this is OK, as it's equivalent to just registering different items. * guix/store/database.scm (register-items): Reduce transaction scope. Signed-off-by: Ludovic Courtès <ludo@gnu.org>
2020-09-14database: document extra registration requirements.Caleb Ristvedt
It's necessary that store items be locked and protected from garbage collection while they are being registered. This documents that. * guix/store/database.scm (register-path, register-items): document GC protection and locking requirements. Signed-off-by: Ludovic Courtès <ludo@gnu.org>
2020-09-14deduplication: pass store directory to replace-with-link.Caleb Ristvedt
This causes with-writable-file to take into consideration the actual store being used, as passed to 'deduplicate', rather than whatever (%store-directory) may return. * guix/store/deduplication.scm (replace-with-link): new keyword argument 'store'. Pass to with-writable-file. (with-writable-file, call-with-writable-file): new store argument. (deduplicate): pass store to replace-with-link. Signed-off-by: Ludovic Courtès <ludo@gnu.org>
2020-07-28store: deduplication: Handle fs without d_type support.Mathieu Othacehe
scandir* uses readdir, which means that the file type property can be 'unknown if the underlying file-system does not support d_type. Make sure to fallback to lstat in that case. Fixes: https://issues.guix.gnu.org/issue/42579. * guix/store/deduplication.scm (deduplicate): Handle the case where properties is 'unknown because the underlying file-system does not support d_type.
2020-07-11image: Do not set journal_model=WAL for the Hurd.Jan (janneke) Nieuwenhuizen
This fixes <https://bugs.gnu.org/42151>. * gnu/system/images/hurd.scm (hurd-initialize-root-partition): Use #:wal-mode #f in call to ... * gnu/build/image.scm (initialize-root-partition): ... this, add #:wal-mode? parameter, pass it to ... (register-closure): ... this, add #:wal-mode? parameter, pass it to ... * guix/store/database.scm (with-database): ... this, add #:wal-mode? parameter, pass it to ... (call-with-database): ... this, add #:wal-mode? parameter; when set to #f, do not set journal_model=WAL.
2020-06-25deduplication: Leave the store permissions unchanged.Ludovic Courtès
Suggested by Caleb Ristvedt <caleb.ristvedt@cune.org>. * guix/store/deduplication.scm (call-with-writable-file): Call THUNK directly when FILE is (%store-directory).
2020-06-25deduplication: Fix default value of #:store in 'deduplicate'.Ludovic Courtès
* guix/store/deduplication.scm (deduplicate): Change #:store default value to (%store-directory).
2020-06-25deduplication: Use 'dynamic-wind' when changing permissions of the parent.Ludovic Courtès
Suggested by Caleb Ristvedt <caleb.ristvedt@cune.org>. * guix/store/deduplication.scm (call-with-writable-file): New procedure. (with-writable-file): New macro. (replace-with-link): Use it.
2020-06-22deduplicate: Avoid traversing directories twice.Ludovic Courtès
Until now, we'd call (nar-sha256 file) unconditionally. Thus, if FILE was a directory, we would traverse it for no reason, and then call 'deduplicate' on FILE, which would again traverse it. This change also removes redundant (mkdir-p store) calls from the loop, and avoids 'lstat' calls by using 'scandir*'. * guix/store/deduplication.scm (deduplicate): Add named loop. Move 'mkdir-p' outside the loop. Use 'scandir*' instead of 'scandir'. Do not call 'nar-sha256' when FILE has type 'directory.
2020-06-18database: 'register-items' takes an open database.Ludovic Courtès
* guix/store/database.scm (store-database-directory) (store-database-file): New procedures. (call-with-database): Add call to 'mkdir-p'. (register-items): Add 'db' parameter and remove #:state-directory and #:schema. (register-path): Use 'store-database-file' and 'with-database', and parameterize SQL-SCHEMA. * gnu/build/image.scm (register-closure): Likewise. * gnu/build/vm.scm (register-closure): Likewise. * guix/scripts/pack.scm (store-database)[build]: Likewise.
2020-06-10database: separate transaction-handling and retry-handling.Caleb Ristvedt
Previously call-with-transaction would both retry when SQLITE_BUSY errors were thrown and do what its name suggested (start and rollback/commit a transaction). This changes it to do only what its name implies, which simplifies its implementation. Retrying is provided by the new call-with-SQLITE_BUSY-retrying procedure. * guix/store/database.scm (call-with-transaction): no longer restarts, new #:restartable? argument controls whether "begin" or "begin immediate" is used. (call-with-SQLITE_BUSY-retrying, call-with-retrying-transaction, call-with-retrying-savepoint): new procedures. (register-items): use call-with-retrying-transaction to preserve old behavior. * .dir-locals.el (call-with-retrying-transaction, call-with-retrying-savepoint): add indentation information.
2020-06-10database: ensure update-or-insert is run within a transactionCaleb Ristvedt
update-or-insert can break if an insert occurs between when it decides whether to update or insert and when it actually performs that operation. Putting the check and the update/insert operation in the same transaction ensures that the update/insert will only succeed if no other write has occurred in the middle. * guix/store/database.scm (call-with-savepoint): new procedure. (update-or-insert): use call-with-savepoint to ensure the read and the insert/update occur within the same transaction.
2020-06-10database: rewrite query procedures in terms of with-statement.Caleb Ristvedt
Most of our queries would fail to finalize their statements properly if sqlite returned an error during their execution. This resolves that, and also makes them somewhat more concise as a side-effect. This also makes some small changes to improve certain queries where behavior was strange or overly verbose. * guix/store/database.scm (call-with-statement): new procedure. (with-statement): new macro. (last-insert-row-id, path-id, update-or-insert, add-references): rewrite to use with-statement. (update-or-insert): factor last-insert-row-id out of the end of both branches. (add-references): remove pointless last-insert-row-id call. * .dir-locals.el (with-statement): add indenting information.
2020-06-10database: work around guile-sqlite3 bug preventing statement resetCaleb Ristvedt
guile-sqlite3 provides statement caching, making it unnecessary for sqlite to keep re-preparing statements that are frequently used. Unfortunately it doesn't quite emulate the semantics of sqlite_finalize properly, because it doesn't cause a commit if the statement being finalized is the last "active" statement (see https://notabug.org/guile-sqlite3/guile-sqlite3/issues/12). We work around this by wrapping sqlite-finalize with our own version that ensures sqlite-reset is called, which does The Right Thing™. * guix/store/database.scm (sqlite-finalize): new procedure that shadows the sqlite-finalize from (sqlite3).
2020-04-11database: 'reset-timestamps' can optionally preserve permissions.Ludovic Courtès
* guix/store/database.scm (reset-timestamps): Add #:preserve-permissions? and honor it.
2020-02-22deduplication: Use nix-base32 encoding for link names.Ludovic Courtès
Fixes <https://bugs.gnu.org/39725>. * guix/store/deduplication.scm (deduplicate): Use 'bytevector->nix-base32-string' instead of 'bytevector->base16-string'.
2019-09-08daemon: Invoke 'guix gc --list-busy' instead of 'list-runtime-roots'.Ludovic Courtès
* nix/scripts/list-runtime-roots.in: Remove. * guix/store/roots.scm (%proc-directory): New variable. (proc-file-roots, proc-exe-roots, proc-cwd-roots) (proc-fd-roots, proc-maps-roots, proc-environ-roots) (referenced-files, canonicalize-store-item, busy-store-items): New procedures, taken from 'list-runtime-roots.in'. * nix/libstore/globals.hh (Settings)[guixProgram]: New field. * nix/libstore/globals.cc (Settings::processEnvironment): Initialize 'guixProgram'. * nix/libstore/gc.cc (addAdditionalRoots): Drop code related to 'NIX_ROOT_FINDER'. Run "guix gc --list-busy". * nix/local.mk (nodist_pkglibexec_SCRIPTS): Remove 'scripts/list-runtime-roots'. * config-daemon.ac: Don't output nix/scripts/list-runtime-roots. * build-aux/pre-inst-env.in: Don't set 'NIX_ROOT_FINDER'. Set 'GUIX'. * doc/guix.texi (Invoking guix gc): Document '--list-busy'. * guix/scripts/gc.scm (show-help, %options): Add "--list-busy". (guix-gc)[list-busy]: New procedure. Handle the 'list-busy' action.
2019-04-25gnu, guix: Yearly ritual purging of the filesystems.Tobias Geerinckx-Rice
* gnu/packages/android.scm (android-ext4-utils)[synopsis]: Fix ‘file system’ spelling. * gnu/packages/disk.scm (rmlint)[synopsis, description]: Likewise. * gnu/packages/golang.scm (go-github-com-kr-fs)[synopsis, description]: Likewise & edit for grammar. * gnu/packages/ipfs.scm (gx, go-ipfs)[description]: Likewise. * /gnu/packages/java.scm (java-commons-vfs)[synopsis]: Likewise. * gnu/packages/linux.scm (fuseiso)[description]: Likewise. (genext2fs)[synopsis, description]: Likewise. * gnu/packages/package-management.scm (libostree)[description]: Likewise. * gnu/packages/python-xyz.scm (python-requests-file)[description]: Likewise & mark up. * gnu/packages/rails.scm (ruby-with-advisory-lock)[description]: Likewise. * gnu/packages/ruby.scm (ruby-rerun)[description]: Likewise. * guix/build/go-build-system.scm (setup-go-environment)<docstring>: Likewise. * guix/store/deduplication.scm (get-temp-link)<docstring>: Likewise.
2019-04-10Add (guix store roots).Ludovic Courtès
* guix/store/roots.scm, tests/store-roots.scm: New files. * Makefile.am (STORE_MODULES): Add guix/store/roots.scm. (SCM_TESTS): Add tests/store-roots.scm.
2019-03-07database: Make 'register-items' transactional.Caleb Ristvedt
* guix/store/database.scm (SQLITE_BUSY, register-output-sql): New variables. (add-references): Don't try finalizing after each use, only after all the uses (otherwise a finalized statement would be used if #:cache? was #f). (call-with-transaction): New procedure. (register-items): Use call-with-transaction to prevent broken intermediate states from being visible. * .dir-locals.el (call-with-transaction): indent it. Signed-off-by: Ludovic Courtès <ludo@gnu.org>
2019-01-23deduplication: Ignore EMLINK.Ludovic Courtès
Until now 'guix offload' would fail (transient failure) upon EMLINK. * guix/store/deduplication.scm (replace-with-link) (deduplicate): Ignore EMLINK.
2018-12-21database: Use "write-ahead log" mode and set a long "busy timeout".Ludovic Courtès
This should avoid "database is locked" errors when there's a lot of concurrency, for instance when offloading simultaneously a lot of builds. * guix/store/database.scm (call-with-database): Add two 'sqlite-exec' calls to set 'journal_mode' and 'busy_timeout'.
2018-12-14deduplication: Gracefully handle ENOSPC raised by 'link' calls.Ludovic Courtès
Reported by Andreas Enge <andreas@enge.fr> in <https://bugs.gnu.org/33676>. * guix/store/deduplication.scm (replace-with-link): Catch ENOSPC around 'get-temp-link'. Do nothing when 'get-temp-link' throws ENOSPC. Move code to restore PARENT's permissions outside of 'catch'. * tests/store-deduplication.scm ("deduplicate, ENOSPC"): New test.
2018-11-23Update Guile-SQLite3 URL everywhere.Ludovic Courtès
* README: Update Guile-SQLite3 URL. * doc/guix.texi (Requirements): Likewise. * guix/store/database.scm (sqlite-exec): Likewise. * m4/guix.m4 (GUIX_CHECK_GUILE_SQLITE3): Likewise.
2018-11-13deduplication: Restore directory mtime and permissions after deduplication.Ludovic Courtès
Fixes <https://bugs.gnu.org/33361>. * guix/store/deduplication.scm (replace-with-link): Call 'set-file-time' and 'chmod' after 'rename-file'. * tests/nar.scm ("restore-file-set with directories (signed, valid)"): New test.
2018-11-13nar: Access the database instead of connecting to the daemon.Ludovic Courtès
* guix/store/database.scm (%default-database-file): New variable. (path-id): Export. * guix/nar.scm (finalize-store-file): Use 'with-database' instead of 'with-store', and use 'path-id' instead of 'valid-path?'.
generated by cgit v1.2.3 (git 2.46.2) at 2024-11-10 06:43:11 +0000