Is Guix suitable for large monorepos?

Is Guix suitable for large monorepos?

[jgart]

Is Guix suitable for large monorepos?

How does Guix compare to pants in the python arena?

https://yewtu.be/watch?v=p4stnR1gCR4
https://www.pantsbuild.org/

Re: Is Guix suitable for large monorepos?

[Ryan Sundberg]

[-- Attachment #1.1: Type: text/plain, Size: 727 bytes --]

Hi jgart, I find that Guix works well with a (source) pointing to local
directories in a monorepo configuration. (hash) values are ignored for
(local-files) in package sources, which has some benefits for a monorepo
setup. It also works just as well, if not better, to help break up
monorepos by using a separate git project for every code base, as one
typically would, while keeping the high level coordination of all the
packages in one big super-repo (using Guile+Guix).

--
Sincerely,
Ryan Sundberg

On 7/21/22 9:55 PM, jgart wrote:
> Is Guix suitable for large monorepos?
> 
> How does Guix compare to pants in the python arena?
> 
> https://yewtu.be/watch?v=p4stnR1gCR4
> https://www.pantsbuild.org/


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: Is Guix suitable for large monorepos?

[Liliana Marie Prikler]

Am Donnerstag, dem 21.07.2022 um 23:55 -0500 schrieb jgart:
> Is Guix suitable for large monorepos?
> 
> How does Guix compare to pants in the python arena?
> 
> https://yewtu.be/watch?v=p4stnR1gCR4
> https://www.pantsbuild.org/
In my humble opinion, Guix is somewhat antithetical to monorepos.  At
the very least, the repo containing all your guix recipes is probably
distinct from the (many) repo(s) you pull your code form.  That said,
Guix can handle monorepos in multiple ways.  

You can split the monorepo into its constituent parts and build the
individual packages.  Examples, where this is "natively" supported
include GStreamer and clang, which also ship as separate tarballs.  An
example, where we do this from hand would be TeXLive.

You can also build the entire monorepo if your monorepo supports that
(just because you have a monorepo doesn't mean that it actually
builds).  Again, TeXLive is an example where we can do that.

Some comments to the video you've posted.
10:33 "How do you know who all the consumers of repo A even are?"
Well, if you have a working package manager (like Guix), you can
recursively check dependencies, e.g. using `guix refresh -l A'.
10:53 "How do I test the change?"
`guix build' already does the entire build for you.  Now granted, that
takes longer than checking the repo out yourself and running configure,
make, etc., but it'll work.
11:34 "Do the whole thing over again."
Well, Guix can help that with CI, but more importantly, there is
nothing you can do to not do the whole thing over again, even in a
monorepo.  No matter what Benjy claims to the contrary, he's either
wrong or (intentionally or otherwise) misrepresenting the cascade.
12:03 "This isn't very nice."
That's why you write ChangeLogs and guidelines for changing stuff?
13:36 "Breakages are immediately visible."
Breakages in Guix are visible once you (or CI) builds all the dependant
packages.
13:43 "... if you are scrupulous about running tests ..."
Are you really going to run all the tests from hex0 to node whenever
you flip a bit?  Chances are no.
13:55 "Codebase architecture enforces good teamwork and responsibility"
Probably maybe.  But rather than having "good teamwork" (a social
problem) and "responsibility" (a social problem) forced upon my team,
I'd rather have the team arrive at good teamwork without a monorepo
overlord.
14:09 "Monorepos can often be more flexible"
Flexibility is not inherently good.
14:49 "I have found, I have seen from experience without naming names,
that the codebase architecture is often a reflection of the structure
and function of your organization"
Don't tell him about Conway's Law.
15:10 "An engineering organization is not a bottom-up kind of thing"
(X) Doubt.
15:18 "In a well functioning engineering team, priorities and decisions
and effort allocation flow top-down"
(X) Doubt.
15:24 "Some sort of top-down organization is required"
(X) Doubt.
15:34 "... and I think this is why many large companies [...] have
adopted this monorepo architecture"
No, it's not.  It's because they are large, hierarchically structured,
top-down companies.  See Conway's Law.
17:38 "One is do less of it, and the other is do more of it
concurrently."
Good thing that Guix does both.

For pants themselves, I don't think I can take them seriously.  Not
just because I prefer skirts, but because merely looking at their repo,
I see a corporate hellhole with at least 4 kinds of lockfiles.  Are
these the people who you would trust to have valuable takes on
dependency management?

Cheers

Re: Is Guix suitable for large monorepos?

[Maxime Devos]

[-- Attachment #1.1.1: Type: text/plain, Size: 484 bytes --]

On 22-07-2022 08:58, Liliana Marie Prikler wrote:

> 13:43 "... if you are scrupulous about running tests ..."
> Are you really going to run all the tests from hex0 to node whenever
> you flip a bit?  Chances are no.

Also, with some exceptions (e.g. emacs packages and texlive), tests are 
run by default in Guix -- you'll have to disable them explicitly for 
them not to be run, and if you try to submit it without tests, reviewers 
will ask why.

Greetings,
Maxime.


[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 929 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: Is Guix suitable for large monorepos?

[Maxime Devos]

[-- Attachment #1.1.1: Type: text/plain, Size: 817 bytes --]

On 22-07-2022 08:58, Liliana Marie Prikler wrote:

> 15:10 "An engineering organization is not a bottom-up kind of thing"
> (X) Doubt.
> 15:18 "In a well functioning engineering team, priorities and decisions
> and effort allocation flow top-down"
> (X) Doubt.
> 15:24 "Some sort of top-down organization is required"
> (X) Doubt.

It's a bit of a stretch, but in a sense Guix is a bit top down, if you 
count Guix itself (and the reviewers, committeres, etc. as a while) as 
'top' and individual patch submitters as 'down'.  OTOH with Guix doesn't 
make decisions on what individual people should work on, only on the 
rules that the results should follow and there is no effort allocation, 
so possibly this is not the kind of 'top-down' that the video was 
referring to.

Greetings,
Maxime.


[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 929 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: Is Guix suitable for large monorepos?

[Liliana Marie Prikler]

Am Freitag, dem 22.07.2022 um 11:14 +0200 schrieb Maxime Devos:
> On 22-07-2022 08:58, Liliana Marie Prikler wrote:
> 
> > 15:10 "An engineering organization is not a bottom-up kind of
> > thing"
> > (X) Doubt.
> > 15:18 "In a well functioning engineering team, priorities and
> > decisions
> > and effort allocation flow top-down"
> > (X) Doubt.
> > 15:24 "Some sort of top-down organization is required"
> > (X) Doubt.
> 
> It's a bit of a stretch, but in a sense Guix is a bit top down, if
> you count Guix itself (and the reviewers, committeres, etc. as a
> while) as 'top' and individual patch submitters as 'down'.  OTOH with
> Guix doesn't make decisions on what individual people should work on,
> only on the rules that the results should follow and there is no
> effort allocation, so possibly this is not the kind of 'top-down'
> that the video was referring to.
To expand on this a little, there are about two important layers here:
committers and non-committers.  Within the review process both are the
same, except for the fact that only committers can actually push
changes to the repo (signing with their name and key).  Also, thanks to
channels, packages can really flow from the bottom up – someone who
wants to submit a package can first check it locally, and even while
the review is ongoing already use it on their machine.  Now there are
maintainers, who are even more powerful than committers and recently
also teams, but for better or worse neither of those matter too much in
your day to day operations.

In the context of the monorepo debate, I think it is also important to
distinguish between Guix, the package manager, and Guix, the channel. 
Governance decisions regarding the latter can largely be ignored if
you're working for MonoRepoCorp™.

Cheers

Re: Is Guix suitable for large monorepos?

[Katherine Cox-Buday]

jgart <jgart@dismail.de> writes:

> Is Guix suitable for large monorepos?

I use Guix against a large mono-repo, and there are a few
difficulties.

I should mention that I am not using Guix in the typical fashion with
fixed commits (packages are pointed at HEAD and I don't use checksums).
For my purposes, it's useful for me to build HEAD, although sometimes I
use transformation options if I'm interested in a specific branch or
commit.

I've defined a package for the entire repo and then individual packages
for the constituent projects which inherit from the repo package.

For new commits:

- Updating the source take a long time.
- Indexing the entire repo takes a long time.
- Copying over the entire tree to the build directory is unnecessary and
  takes time.

Sometimes I want to build a local change I haven't committed anywhere,
so I use the --with-source option. The problems are the same, but
they're exacerbated.

A happy medium I've landed on is to build some binaries outside of Guix
and then use packages that are defined in terms of the binaries. These
packages use patchelf to get everything linked correctly. This allows me
to reify these packages into Guix while keeping the standard workflow
from a programming language.

I've also considered a phase that, after checkout, deletes all files not
relevant to the project to lessen the burden of copying things over for
building.

So, in my opinion, these issues are not really Guix's fault, and it does
a pretty good job all things considered. I think Guix used in a CI/CD
pipeline (which I guess means cuirass) would handle mono-repositories
wonderfully and do exactly what I want: reproducible, verifiable,
builds. Guix as a development tool in these situations is still useful,
but has some things that make it a little difficult to work with.

Maybe someone has some ideas to improve things?

-- 
Katherine

Re: Is Guix suitable for large monorepos?

[Blake Shaw]

[-- Attachment #1: Type: text/plain, Size: 5350 bytes --]

Hiya guix,

I tried to reply to this quickly while I was travelling, but what started
with "sorry to brief" quickly became quite long -- nonetheless I think
theres some important insight into using Guix effectively in production I'd
still like to share, so will finish it off now that I'm back:

==
I'm traveling at the moment so excuse me for being brief but I saw
Katherine's message about long source rebuilds and want to try to chime in
quickly.

I had a difficult experience of using Guix in production against a
medium-large C++ repo (openFrameworks) in June when substitute availability
was failing a good deal of the time, along with several broken packages
"infecting" a decent area of oF's ~140 package dependency graph, to add to
the mayhem.

Considering it was a moment I needed to move fast, I created a hotfix
branch where I could disable tests, do quick gnarly fixes and keep moving.
Its worth noting that working like this you can really knock out a lot of
package "fixes" quickly, I was able to do 10 in a day, but also this was
while deploying a video mapping project so security wasn't a huge concern.

This initially worked decently, in terms of maintaining project momentum
while racing towards a delivery. But when it came to installing on the
servers and the laptop onsite, which came with their own difficulties, the
build times started weighing in more than I was able to handle.

I had been considering starting a Cuirass instance but was worried it would
be too difficult to figure out in constrained time. At some point I thought
"you know what lets try this just not sleep and see if cuirass can be a
solution, and if its immediately confusing shelve it and move on". But the
truth was...

...Cuirass was one of the quickest parts of the Guix system I've hit the
ground running with.

I really wish I had deployed it from the outset, and I surely will on all
future projects. Within a day of fiddling with it I had my linode providing
substitutes of not only my dependency graph, but also generations of three
different systems, as well as VMs. After the first night of building i
suddenly had lightning fast substitutes available and problems upstream
faded away. In short, had I used Cuirass from the get-go I would have saved
myself from a massive amount of difficulty.

The difficulty with reproducibility in production is that it you're
constrained for time, building back to a month old generation can be out of
the question. I think there is a "reproducibility ideal" that we and the
Nix folks often allude to as our reality, where "you just roll back", but
the conditions of production can often make this unfeasible. Before my next
gig I plan to automate things so that I can easily tag commits in git that
will trigger system generations to be built and stored as QEMU images that
I can quickly access as need, which I think will bring me closer to this
ideal. I'll be sure to share when it comes time (but currently taking a
little break from being deep in the weeds).

Also, my apologies that I haven't been contributing much... an experiment
with self-hosting my email earlier this year ended with a borked email
server and I have yet to setup gnus & git with this email address. Will be
rolling up my sleeves in the coming weeks :)

Happy hacking!
b



On Fri, Jul 22, 2022, 21:26 Katherine Cox-Buday <cox.katherine.e@gmail.com>
wrote:

> jgart <jgart@dismail.de> writes:
>
> > Is Guix suitable for large monorepos?
>
> I use Guix against a large mono-repo, and there are a few
> difficulties.
>
> I should mention that I am not using Guix in the typical fashion with
> fixed commits (packages are pointed at HEAD and I don't use checksums).
> For my purposes, it's useful for me to build HEAD, although sometimes I
> use transformation options if I'm interested in a specific branch or
> commit.
>
> I've defined a package for the entire repo and then individual packages
> for the constituent projects which inherit from the repo package.
>
> For new commits:
>
> - Updating the source take a long time.
> - Indexing the entire repo takes a long time.
> - Copying over the entire tree to the build directory is unnecessary and
>   takes time.
>
> Sometimes I want to build a local change I haven't committed anywhere,
> so I use the --with-source option. The problems are the same, but
> they're exacerbated.
>
> A happy medium I've landed on is to build some binaries outside of Guix
> and then use packages that are defined in terms of the binaries. These
> packages use patchelf to get everything linked correctly. This allows me
> to reify these packages into Guix while keeping the standard workflow
> from a programming language.
>
> I've also considered a phase that, after checkout, deletes all files not
> relevant to the project to lessen the burden of copying things over for
> building.
>
> So, in my opinion, these issues are not really Guix's fault, and it does
> a pretty good job all things considered. I think Guix used in a CI/CD
> pipeline (which I guess means cuirass) would handle mono-repositories
> wonderfully and do exactly what I want: reproducible, verifiable,
> builds. Guix as a development tool in these situations is still useful,
> but has some things that make it a little difficult to work with.
>
> Maybe someone has some ideas to improve things?
>
> --
> Katherine
>
>

[-- Attachment #2: Type: text/html, Size: 6576 bytes --]