linux-module-builder leads to huge store items

linux-module-builder leads to huge store items

[Ludovic Courtès]

Hello Danny & Mathieu,

While looking for sources of contention on ci.guix.gnu.org, I found that
“linux-libre-module-builder” leads to huuuuge trees in the store;
performing deduplication on these trees takes ages, and more generally
these big trees look unreasaonble to me.

--8<---------------cut here---------------start------------->8---
$ find /gnu/store/s9md49vjambvprpp5ba3aqvhxilw0w7k-linux-libre-module-builder-5.8.8|wc -l
85604
$ du -hs /gnu/store/s9md49vjambvprpp5ba3aqvhxilw0w7k-linux-libre-module-builder-5.8.8
1.1G    /gnu/store/s9md49vjambvprpp5ba3aqvhxilw0w7k-linux-libre-module-builder-5.8.8
--8<---------------cut here---------------end--------------->8---

Surely we don’t need that much just to build a kernel module.  :-)

(gnu build-system linux-module) has this:

  ;; TODO: Only preserve the minimum, i.e. [Kbuild], Kconfig,
  ;; scripts, include, ".config".

Which files exactly are needed?  Can we go ahead and implement this TODO?

I understand the need to “cut corners” while trying things out, but I’d
argue that this specific corner should have been restored before getting
into master.

Thanks,
Ludo’.

Re: linux-module-builder leads to huge store items

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 2444 bytes --]

Hi Ludo,

On Thu, 10 Sep 2020 10:09:11 +0200
Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

> Surely we don’t need that much just to build a kernel module.  :-)

Since Linux is a monolithic kernel and since C has no modules: Yes, yes we
do--in the general case.

Linux kernel developers have no interest in supporting out-of-tree modules for
free (and no interesting in having the Linux kernel API frozen) so there's no
incentive to support a use case where you can compile a Linux kernel module
without having the full source of the Linux kernel (that would be just asking
for license abuse, too).

That would mean that we would have to both design and then support out-of-tree
build infrastructure ourselves in Guix.  And for what?  It's only used when
building the module--it will be thrown away afterwards anyway.

Or is it not garbage-collected afterwards?  That would be a serious bug.

>   ;; TODO: Only preserve the minimum, i.e. [Kbuild], Kconfig,
>   ;; scripts, include, ".config".
> 
> Which files exactly are needed?  Can we go ahead and implement this TODO?

Who knows?  Nobody--not even Linux kernel developers.  Given the design of
C, it's not possible in general to know what is the interface and what is
the implementation.  I guess for the Linux kernel we could kinda wing it
because of the social conventions the Linux kernel enforces where it kinda
almost does have a module separation into interface and implementation.

> I understand the need to “cut corners” while trying things out, but I’d
> argue that this specific corner should have been restored before getting
> into master.

If you want that, I think you need to argue that with the main Linux kernel
maintainers (Greg Kroah-Hartmann etc).  I don't think it is
practical for us to maintain a weird minimal module-building-only fork
of Linux.  I'd be glad to be proven wrong, though.

Even if we did, there are much worse build-time storage space wastes
(*cough* llvm *cough*).

That said, I did a quick size comparison and found those big items:

150 MB a2fs24bgghjvlzq5lzr6zki7mqxx8mpi-linux-libre-module-builder-5.8.7/lib/modules/build/drivers/gpu/drm/amd/include/asic_reg
30 MB a2fs24bgghjvlzq5lzr6zki7mqxx8mpi-linux-libre-module-builder-5.8.7/lib/modules/build/drivers/staging

I don't think we need these in any potential Linux kernel module--but who
knows?  Maybe somebody does need those.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: linux-module-builder leads to huge store items

[Ludovic Courtès]

Hello,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> On Thu, 10 Sep 2020 10:09:11 +0200
> Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
>
>> Surely we don’t need that much just to build a kernel module.  :-)
>
> Since Linux is a monolithic kernel and since C has no modules: Yes, yes we
> do--in the general case.
>
> Linux kernel developers have no interest in supporting out-of-tree modules for
> free (and no interesting in having the Linux kernel API frozen) so there's no
> incentive to support a use case where you can compile a Linux kernel module
> without having the full source of the Linux kernel (that would be just asking
> for license abuse, too).

Understood.

> That would mean that we would have to both design and then support out-of-tree
> build infrastructure ourselves in Guix.  And for what?  It's only used when
> building the module--it will be thrown away afterwards anyway.
>
> Or is it not garbage-collected afterwards?  That would be a serious bug.

It *is* thrown-away afterwards.  But until then, it must be deduplicated
(upon build completion, reception, etc.), and that’s a lot of work I/O
wise given the large number of files.

>>   ;; TODO: Only preserve the minimum, i.e. [Kbuild], Kconfig,
>>   ;; scripts, include, ".config".
>> 
>> Which files exactly are needed?  Can we go ahead and implement this TODO?
>
> Who knows?

[...]

> Even if we did, there are much worse build-time storage space wastes
> (*cough* llvm *cough*).

But it’s build-time.

> That said, I did a quick size comparison and found those big items:
>
> 150 MB a2fs24bgghjvlzq5lzr6zki7mqxx8mpi-linux-libre-module-builder-5.8.7/lib/modules/build/drivers/gpu/drm/amd/include/asic_reg
> 30 MB a2fs24bgghjvlzq5lzr6zki7mqxx8mpi-linux-libre-module-builder-5.8.7/lib/modules/build/drivers/staging
>
> I don't think we need these in any potential Linux kernel module--but who
> knows?  Maybe somebody does need those.

So, I can think of the following ways to address that:

  1. Remove for example *.c, assuming the out-of-tree modules we package
     only care about *.h + build machinery.

  2. Don’t add all of the Linux source to the store, or at least not
     under a new name each time.  One way to do that would be to unpack
     the Linux-libre tree within the build tree of the out-of-tree
     module.  We’d be reducing GC stress at the expense of a
     (reasonable?) increase in module build time.

Thoughts?

Ludo’.

Re: linux-module-builder leads to huge store items

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 2483 bytes --]

Hi Ludo,

On Thu, 10 Sep 2020 18:26:57 +0200
Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

> > 150 MB a2fs24bgghjvlzq5lzr6zki7mqxx8mpi-linux-libre-module-builder-5.8.7/lib/modules/build/drivers/gpu/drm/amd/include/asic_reg
> > 30 MB a2fs24bgghjvlzq5lzr6zki7mqxx8mpi-linux-libre-module-builder-5.8.7/lib/modules/build/drivers/staging
> >
> > I don't think we need these in any potential Linux kernel module--but who
> > knows?  Maybe somebody does need those.  
> 
> So, I can think of the following ways to address that:
> 
>   1. Remove for example *.c, assuming the out-of-tree modules we package
>      only care about *.h + build machinery.

The biggest source files, see above, are header files
(drivers/gpu/drm/amd/include/asic_reg).

>   2. Don’t add all of the Linux source to the store, or at least not
>      under a new name each time.  One way to do that would be to unpack
>      the Linux-libre tree within the build tree of the out-of-tree
>      module.  We’d be reducing GC stress at the expense of a
>      (reasonable?) increase in module build time.

And patch it correctly etc just like make-linux-libre does?  That doesn't
sound safe.  The current setup is basically inheriting from the package
make-linux-libre gave us and changing the 'build phase slightly.  That means
it will be exactly the same otherwise.

The only way to reduce effort safely is to delete the set of source
files (especially header files) which we don't need for out-of-tree
modules.

I checked--that would reduce the size of linux-libre-builder by about 800 MB
(to 255 MB; see [1][2]).
That would mean that some modules can't be out-of-tree.  But I guess
that's okay--and we could not-delete some files later if need be.

Btw, linux-libre is currently not reproducible--see bug 43301.
That could introduce a security problem if the Linux kernel and
an out-of-tree module are compiled differently (they are) and then loaded
into the same kernel (even now).

[1] Toplevel files in linux-libre-module-builder would be then (sizes are included if size > 1 MB):

arch/   (134 MB)
block/  (2 MB)
certs/
COPYING
CREDITS
crypto/ (4 MB)
include/ (59 MB)
init/
ipc/
Kbuild
Kconfig
kernel/ (11 MB)
lib/ (6 MB)
LICENSES/
MAINTAINERS
Makefile
mm/ (5 MB)
Module.symvers (2 MB)
README
scripts/ (6 MB)
security/ (4 MB)
System.map (5 MB)
tools/ (14 MB)
usr/
virt/

[2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=43332

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: linux-module-builder leads to huge store items

[Ludovic Courtès]

Hi Danny,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

>> So, I can think of the following ways to address that:
>> 
>>   1. Remove for example *.c, assuming the out-of-tree modules we package
>>      only care about *.h + build machinery.
>
> The biggest source files, see above, are header files
> (drivers/gpu/drm/amd/include/asic_reg).
>
>>   2. Don’t add all of the Linux source to the store, or at least not
>>      under a new name each time.  One way to do that would be to unpack
>>      the Linux-libre tree within the build tree of the out-of-tree
>>      module.  We’d be reducing GC stress at the expense of a
>>      (reasonable?) increase in module build time.

[...]

> I checked--that would reduce the size of linux-libre-builder by about 800 MB
> (to 255 MB; see [1][2]).
> That would mean that some modules can't be out-of-tree.  But I guess
> that's okay--and we could not-delete some files later if need be.

Yes, that’s OK.  I’d say let’s delete as much as possible, and then, if
it eventually turns out to be a problem for a new out-of-tree module
package, we can re-add files selectively.

Could you give it a try?

> Btw, linux-libre is currently not reproducible--see bug 43301.
> That could introduce a security problem if the Linux kernel and
> an out-of-tree module are compiled differently (they are) and then loaded
> into the same kernel (even now).

One thing at a time.  :-)

Thanks,
Ludo’.

Re: linux-module-builder leads to huge store items

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 117 bytes --]

>Could you give it a try?

... I did.  See footnote 2.

[2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=43332

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]