guix gc, any way to delete only packages that aren't required to build anything?

guix gc, any way to delete only packages that aren't required to build anything?

[Matthew Brooks]

[-- Attachment #1: Type: text/plain, Size: 697 bytes --]

Is there any recommended way to have guix gc only delete packages that aren't required to build anything?
I've made a custom bash script that loops through the output of "guix gc --list-dead" and tells guix to delete any dead packages that have no --referers, but it's really slow (it can take several hours on my laptop).
My first attempt was just telling guix to delete everything in --list-dead, but that seemed to still delete some things that were required to build other things currently on the system somewhere.

Of course, running plain "guix gc" with no arguments is much faster, but requires redownloading loads of packages whenever I next go to pull, upgrade, or install anything.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: guix gc, any way to delete only packages that aren't required to build anything?

[Ricardo Wurmus]

Hi Matthew,

> Is there any recommended way to have guix gc only delete packages that
> aren't required to build anything?

This requirement is a little vague.  Do you mean to only delete leaf
packages?  Or do you just want to keep a version of GCC?

You could create an environment containing all dependencies for a
package of interest and register it as a GC root, so it won’t be
collected.  For example:

    guix environment --root=$HOME/.dev-env guile

Then you can run “guix gc” with reckless abandon and be sure that at
least the dependencies of the “guile” package (at that particular
version of Guix) will not be collected.

There is also a daemon option, which might be of interest to you, though
it may keep more (or less) than what you want:

    --gc-keep-derivations[=yes/no]
                tell whether the GC must keep derivations
                corresponding to live outputs

Hope this helps!

--
Ricardo

Re: guix gc, any way to delete only packages that aren't required to build anything?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 913 bytes --]

I've been bugged by this as well (see
https://lists.gnu.org/archive/html/help-guix/2018-09/msg00028.html).  There are
a few options beside Ricardo's suggestion:

- You can start the Guix daemon with --gc-keep-outputs (see "(guix) Invoking
  guix-daemon" in the manual).  This will never delete anything that is still
  required, but this leads to an ever-growing store.  If you've got limited
  storage space, you might not want this.

- The Emacs mode guix.el has a nice interface for listing store items, in
  particular dead items.  You can sort item by size, check the number of
  referrers, etc.  This is much, much faster than you bash script and should
  suit your needs most of the time.  If you don't use Emacs, you can have still
  run it in batch mode or simply re-use the Scheme code to write your own Guile
  script.

Hope that helps!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: guix gc, any way to delete only packages that aren't required to build anything?

[Mark H Weaver]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> I've been bugged by this as well (see
> https://lists.gnu.org/archive/html/help-guix/2018-09/msg00028.html).  There are
> a few options beside Ricardo's suggestion:
>
> - You can start the Guix daemon with --gc-keep-outputs (see "(guix) Invoking
>   guix-daemon" in the manual).  This will never delete anything that is still
>   required, but this leads to an ever-growing store.  If you've got limited
>   storage space, you might not want this.

On my GuixSD system where substitutes are completely disabled and I
build everything locally, I've been running my Guix daemon with both
--gc-keep-derivations=yes and --gc-keep-outputs=yes for years.  Here's
the relevant excerpt of my OS config:

  (services (cons* <... my custom services here ...>
                   (modify-services (remove (lambda (svc)
                                              (eq? (service-kind svc)
                                                   wicd-service-type))
                                            %base-services)
                     (guix-service-type
                      config =>
                      (guix-configuration
                        (inherit config)
                        (use-substitutes? #f)
                        (authorized-keys '())
                        (substitute-urls '())
                        (extra-options '("--gc-keep-derivations=yes"
                                         "--gc-keep-outputs=yes")))))))

This configuration allows me to run "guix gc" without losing the
dependencies that are needed to build the packages that are currently
protected by GC roots, e.g. the packages in my profiles and system
generations.  It's not perfect: I lose a few packages apparently related
to grafts, but it's not too much in practice, and I can manually add
more GC roots to handle those cases.

Contrary to what Pierre wrote above, I've found that the use of these
flags certainly does _not_ lead to an ever-growing store.  On my GNOME 3
GuixSD system, my store requires ~30 gigabytes, and as long as I trim my
older profiles and system generations regularly, the space requirements
are fairly stable over time.

      Regards,
        Mark

Re: guix gc, any way to delete only packages that aren't required to build anything?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 363 bytes --]


> Contrary to what Pierre wrote above, I've found that the use of these
> flags certainly does _not_ lead to an ever-growing store.

According to the discussion I linked above, The reason behind the ever-growing
store are substitutes.  So if you don't use them, then the store size should be
stable indeed.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: guix gc, any way to delete only packages that aren't required to build anything?

[Mark H Weaver]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

>> Contrary to what Pierre wrote above, I've found that the use of these
>> flags certainly does _not_ lead to an ever-growing store.
>
> According to the discussion I linked above, The reason behind the ever-growing
> store are substitutes.  So if you don't use them, then the store size should be
> stable indeed.

I re-read the thread that you linked to, and I cannot find anything to
suggest that "The reason behind the ever-growing store are substitutes".
Can you cite the specific message that says so?

     Thanks,
       Mark

Re: guix gc, any way to delete only packages that aren't required to build anything?

[Joshua Branson]

Mark H Weaver <mhw@netris.org> writes:

> Pierre Neidhardt <mail@ambrevar.xyz> writes:
>
> On my GuixSD system where substitutes are completely disabled and I
> build everything locally, I've been running my Guix daemon with both
> --gc-keep-derivations=yes and --gc-keep-outputs=yes for years.  Here's
> the relevant excerpt of my OS config:
>

May I ask why you choose not to use substitutes? 

>
>       Regards,
>         Mark

Re: guix gc, any way to delete only packages that aren't required to build anything?

[Ludovic Courtès]

Hello,

Matthew Brooks <matthewfbrooks@mailbox.org> skribis:

> Is there any recommended way to have guix gc only delete packages that aren't required to build anything?

To complement what others say, I use a low-tech but good enough
approach, which is to GC just as much as I need, e.g., by running
“guix gc -F80G” once in a while.

The store items to be deleted are chosen at random but overall it works
well for me.

Ludo’.

Re: guix gc, any way to delete only packages that aren't required to build anything?

[Mark H Weaver]

Hi Joshua,

Joshua Branson <jbranso@fastmail.com> writes:

> Mark H Weaver <mhw@netris.org> writes:
>
>> Pierre Neidhardt <mail@ambrevar.xyz> writes:
>>
>> On my GuixSD system where substitutes are completely disabled and I
>> build everything locally, I've been running my Guix daemon with both
>> --gc-keep-derivations=yes and --gc-keep-outputs=yes for years.  Here's
>> the relevant excerpt of my OS config:
>>
>
> May I ask why you choose not to use substitutes? 

It's part of a (likely futile) effort to protect the integrity of my
laptops from powerful adversaries, to prevent my private cryptographic
keys from being stolen, and to try to prevent my machine from being used
to insert vulnerabilities into the source code of projects that I
contribute to.  I also keep my laptops with me at all times.

Unfortunately, the US government claims the authority to secretly demand
physical access to servers, and to forbid those coerced from telling
anyone what happened.  See:

  https://en.wikipedia.org/wiki/National_security_letter

I'm not sure about the policies of other governments, but even without
such policies, there are probably windows of time where the physical
security of colocated servers could be breached by bribing employees at
the hosting site.  I doubt there is much restraint in the use of these
methods today, besides a desire to avoid detection.  As a result, any
traditional build farm based on colocated servers is vulnerable to
compromise by powerful adversaries.

There are some additional benefits to building everything locally and
passing using the 'guix-daemon' options above.  It means that my
/gnu/store always contains the complete source code of everything on my
system, including everything needed the bootstrap from the bootstrap
binaries.  It also means that I always have a complete set of build logs
for everything on my system.

      Mark

Re: guix gc, any way to delete only packages that aren't required to build anything?

[Mark H Weaver]

Mark H Weaver <mhw@netris.org> writes:

> Unfortunately, the US government claims the authority to secretly demand
> physical access to servers, and to forbid those coerced from telling
> anyone what happened.  See:
>
>   https://en.wikipedia.org/wiki/National_security_letter

Well, having read the link I provided, it appears that NSLs cannot
legally be used to gain physical access to servers.

Regardless, I have little confidence in law enforcement agencies to
limit their activities to those permitted by law, nor do I expect many
employees of hosting sites to have the will and strength to resist law
enforcement agents making demands of them in the name of national
security.

       Mark

Re: guix gc, any way to delete only packages that aren't required to build anything?

[Joshua Branson]

Mark H Weaver <mhw@netris.org> writes:

> Hi Joshua,
>
> Joshua Branson <jbranso@fastmail.com> writes:
>
>> Mark H Weaver <mhw@netris.org> writes:
>
> There are some additional benefits to building everything locally and
> passing using the 'guix-daemon' options above.  It means that my
> /gnu/store always contains the complete source code of everything on my
> system, including everything needed the bootstrap from the bootstrap
> binaries.  It also means that I always have a complete set of build logs
> for everything on my system.

I suppose that would make it easier to debug your system if something breaks.

>
>       Mark

Re: guix gc, any way to delete only packages that aren't required to build anything?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 406 bytes --]


Mark H Weaver <mhw@netris.org> writes:
> I re-read the thread that you linked to, and I cannot find anything to
> suggest that "The reason behind the ever-growing store are substitutes".
> Can you cite the specific message that says so?

I re-read again, and I cannot find it either.  Either it was another thread, or
it's just me being confused :p

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: guix gc, any way to delete only packages that aren't required to build anything?

[George Clemmer]

Hi Mark,

Mark H Weaver <mhw@netris.org> writes:

> On my GuixSD system where substitutes are completely disabled and I
> build everything locally, I've been running my Guix daemon with both
> --gc-keep-derivations=yes and --gc-keep-outputs=yes for years.  Here's
> the relevant excerpt of my OS config:
...

Thanks. Very cool to see.  IIUC there is a Guix "bootstrap" component.
Using this config, where does that come from?

TIA, George

Re: guix gc, any way to delete only packages that aren't required to build anything?

[Mark H Weaver]

George Clemmer <myglc2@gmail.com> writes:

> Mark H Weaver <mhw@netris.org> writes:
>
>> On my GuixSD system where substitutes are completely disabled and I
>> build everything locally, I've been running my Guix daemon with both
>> --gc-keep-derivations=yes and --gc-keep-outputs=yes for years.  Here's
>> the relevant excerpt of my OS config:
> ...
>
> Thanks. Very cool to see.  IIUC there is a Guix "bootstrap" component.
> Using this config, where does that come from?

The bootstrap binaries currently consist of statically-linked
executables for 'bash', 'tar', 'xz', and 'mkdir', which are included in
the Guix source tree in the gnu/packages/bootstrap/ directory:

  https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/bootstrap

and also a few additional tarballs of statically-linked binaries
downloaded from:

  https://alpha.gnu.org/gnu/guix/bootstrap/

which includes guile, gcc, binutils, glibc, and a "static-binaries"
tarball with coreutils, tar, gzip, bzip2, xz, patch, sed, grep, and
gawk.

These bootstrap binaries were generated by the code in
gnu/packages/make-bootstrap.scm in the past, and are rarely changed.
They are fetched/imported into the store by code in
gnu/packages/bootstrap.scm.

The current bootstrap binaries for x86_64-linux and i686-linux were
generated in November 2013, although there is ongoing work to replace
these with a reduced set of bootstrap binaries:

  https://alpha.gnu.org/gnu/guix/bootstrap/i686-linux/20181020/

It is likely that our 'master' branch will be based on these reduced
bootstrap tarballs, or similar ones, within a few months.

      Mark

Re: guix gc, any way to delete only packages that aren't required to build anything?

[George Clemmer]

Mark H Weaver <mhw@netris.org> writes:

> The bootstrap binaries currently consist of statically-linked
> executables for 'bash', 'tar', 'xz', and 'mkdir', which are included in
> the Guix source tree in the gnu/packages/bootstrap/ directory:
>
>   https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/bootstrap
>
> and also a few additional tarballs of statically-linked binaries
> downloaded from:
>
>   https://alpha.gnu.org/gnu/guix/bootstrap/
>
> which includes guile, gcc, binutils, glibc, and a "static-binaries"
> tarball with coreutils, tar, gzip, bzip2, xz, patch, sed, grep, and
> gawk.
>
> These bootstrap binaries were generated by the code in
> gnu/packages/make-bootstrap.scm in the past, and are rarely changed.
> They are fetched/imported into the store by code in
> gnu/packages/bootstrap.scm.
>
> The current bootstrap binaries for x86_64-linux and i686-linux were
> generated in November 2013, although there is ongoing work to replace
> these with a reduced set of bootstrap binaries:
>
>   https://alpha.gnu.org/gnu/guix/bootstrap/i686-linux/20181020/
>
> It is likely that our 'master' branch will be based on these reduced
> bootstrap tarballs, or similar ones, within a few months.

Thank you Mark, this is really helpful. My I suggest replacing the
current (guix) Bootstrapping sentence ...

"Note that this is a question of interest only to the curious hacker,
not to the regular user, so you can shamelessly skip this section if you
consider yourself a “regular user”."

... with this?

Thanks, George

Re: guix gc, any way to delete only packages that aren't required to build anything?

[George Clemmer]

Mark H Weaver <mhw@netris.org> writes:

...
> On my GuixSD system where substitutes are completely disabled and I
> build everything locally, I've been running my Guix daemon with both
> --gc-keep-derivations=yes and --gc-keep-outputs=yes for years.  Here's
> the relevant excerpt of my OS config:
>
>   (services (cons* <... my custom services here ...>
>                    (modify-services (remove (lambda (svc)
>                                               (eq? (service-kind svc)
>                                                    wicd-service-type))
>                                             %base-services)
>                      (guix-service-type
>                       config =>
>                       (guix-configuration
>                         (inherit config)
>                         (use-substitutes? #f)
>                         (authorized-keys '())
>                         (substitute-urls '())
>                         (extra-options '("--gc-keep-derivations=yes"
>                                          "--gc-keep-outputs=yes")))))))
>
> This configuration allows me to run "guix gc" without losing the
> dependencies that are needed to build the packages that are currently
> protected by GC roots, e.g. the packages in my profiles and system
> generations.  It's not perfect: I lose a few packages apparently related
> to grafts, but it's not too much in practice, and I can manually add
> more GC roots to handle those cases.
>
> Contrary to what Pierre wrote above, I've found that the use of these
> flags certainly does _not_ lead to an ever-growing store.  On my GNOME 3
> GuixSD system, my store requires ~30 gigabytes, and as long as I trim my
> older profiles and system generations regularly, the space requirements
> are fairly stable over time.
>
>       Regards,
>         Mark

Hi Mark,

I switched to this approach. It works great and I see similar
results. Thanks again for posting it.

- George