Git repos with large submodules

Git repos with large submodules

[raingloom]

Hey all!

So, I recently gave a go to packaging EDK2 on my channel, and found out
that it requires several submodules, including OpenSSL, which take up
quite a bit of space and take way longer to download than necessary.

Since I couldn't find a way to shallow-init the submodules, I added
them as separate origins and copied / symlinked them after the unpack
phase.

Is there a better way to do this? I don't see any equivalent of
--shallow-submodules in guile-git or libgit2.

I was thinking of writing a bit of Scheme to handle importing git repos
with their submodules as origins, but that feels like a rather ugly
hack.
Maybe libgit or guile-git would be the best layer to fix this at.

Re: Git repos with large submodules

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 694 bytes --]

raingloom <raingloom@riseup.net> writes:

> Hey all!
>
> So, I recently gave a go to packaging EDK2 on my channel, and found out
> that it requires several submodules, including OpenSSL, which take up
> quite a bit of space and take way longer to download than necessary.
>
> Since I couldn't find a way to shallow-init the submodules, I added
> them as separate origins and copied / symlinked them after the unpack
> phase.

That sounds like a decent workaround.  Even better to symlink
(package-source openssl) to use the OpenSSL sources from Guix.

The ideal solution would of course be to provide OpenSSL as a regular
input and make the build system use it, but that's not always feasible.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Git repos with large submodules

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 928 bytes --]

On Sat, May 30, 2020 at 11:28:08PM +0200, Marius Bakke wrote:
> raingloom <raingloom@riseup.net> writes:
> 
> > Hey all!
> >
> > So, I recently gave a go to packaging EDK2 on my channel, and found out
> > that it requires several submodules, including OpenSSL, which take up
> > quite a bit of space and take way longer to download than necessary.
> >
> > Since I couldn't find a way to shallow-init the submodules, I added
> > them as separate origins and copied / symlinked them after the unpack
> > phase.
> 
> That sounds like a decent workaround.  Even better to symlink
> (package-source openssl) to use the OpenSSL sources from Guix.

Yes, we should definitely make that work instead of letting EDK2 handle
it. Same for the Oniguruma dependency, as that is also a very
security-sensitive program:

https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=oniguruma

Let us know if you need help raingloom!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]