New “ungrafting” branch

New “ungrafting” branch

[Ludovic Courtès]

Hi Guix!

Following discussions on IRC, I’ve created a new ‘ungrafting’ branch
that does nothing but ungraft things.

The rationale is that grafts incur additional overhead when installing
things (the time to create those grafts), so it’s good to clean them up
once in a while.  Ungrafting in a dedicated branch means we know the
branch is “safe”, unlike more exploratory branches like ‘staging’ and
‘core-updates’.

The plan is to start building it later today, and to hopefully be done
in a week or so.

Thoughts?

Let’s ungraft!

Ludo’.

Re: New “ungrafting” branch

[zimoun]

Hi Ludo,

On Tue, 8 Dec 2020 at 12:29, Ludovic Courtès <ludo@gnu.org> wrote:

> Following discussions on IRC, I’ve created a new ‘ungrafting’ branch
> that does nothing but ungraft things.
>
> The rationale is that grafts incur additional overhead when installing
> things (the time to create those grafts), so it’s good to clean them up
> once in a while.  Ungrafting in a dedicated branch means we know the
> branch is “safe”, unlike more exploratory branches like ‘staging’ and
> ‘core-updates’.

I have missed the discussion on IRC and I do not understand the
rationale.  What's the point of "ungrafting"?

All the best,
simon

Re: New “ungrafting” branch

[Andreas Enge]

Hello,

On Tue, Dec 08, 2020 at 03:13:40PM +0100, zimoun wrote:
> I have missed the discussion on IRC and I do not understand the
> rationale.  What's the point of "ungrafting"?

I suppose it is just a branch with all the patches that are grafted applied,
so that it is not necessary to graft them anymore. An excellent initiative,
I just built inkscape with 36 grafts!

Andreas

Re: New “ungrafting” branch

[Leo Famulari]

On Tue, Dec 08, 2020 at 03:13:40PM +0100, zimoun wrote:
> I have missed the discussion on IRC and I do not understand the
> rationale.  What's the point of "ungrafting"?

Grafts cause profile operations like `guix install` and `guix
environment` to be slower than without grafts.

Re: New “ungrafting” branch

[zimoun]

Hi Leo,

On Tue, 08 Dec 2020 at 13:20, Leo Famulari <leo@famulari.name> wrote:
> On Tue, Dec 08, 2020 at 03:13:40PM +0100, zimoun wrote:
>> I have missed the discussion on IRC and I do not understand the
>> rationale.  What's the point of "ungrafting"?
>
> Grafts cause profile operations like `guix install` and `guix
> environment` to be slower than without grafts.

Yes, for sure.  The manual is bit lying. :-)

                           The cost of grafting is typically very low, and
    order of magnitudes lower than a full rebuild of the dependency chain.

But if grafts is here, it is because of something.  Therefore, I do not
understand what it does mean “ungraft”.  Avoid the grafting mechanism
when it is possible?


All the best,
simon

Re: New “ungrafting” branch

[Leo Famulari]

On Mon, Dec 14, 2020 at 11:32:05AM +0100, zimoun wrote:
> But if grafts is here, it is because of something.  Therefore, I do not
> understand what it does mean “ungraft”.  Avoid the grafting mechanism
> when it is possible?

Grafts are used to "hack" the package dependency graph. We try to make
some change in the dependency graph without having to recompile
everything as normal.

When we say "ungraft", we mean that we want to effect the same change in
the dependency graph, but without using a graft.

For example, if there was an urgent security update of OpenSSL to
version 42, we would create a new package 'openssl-42' and add
(replacement openssl-42) to the primary 'openssl' package. To ungraft,
we'd remove the (replacement openssl-42) field from the 'openssl'
package, and simply update the 'openssl' package to version 42.

Does that make sense?

Before Guix had a recursive grafting mechanism, we had to deploy urgent
security updates by rebuilding everything. We either had to wait for
everything to be rebuilt, which might take too long, or make users
rebuild everything themselves, which could also take too long, and might
not work at all on some computers.

When we added the recursive grafting feature, we intended to use it to
deploy urgent security updates quickly, but we also intended to
"ungraft" quickly, maybe in a week or two. We never planned to keep
packages grafted for several months as we do now — grafts were
considered a necessary but unfortunate violation of the functional
packaging model.

Re: New “ungrafting” branch

[Mark H Weaver]

Hi Leo,

Leo Famulari <leo@famulari.name> writes:

> When we added the recursive grafting feature, we intended to use it to
> deploy urgent security updates quickly, but we also intended to
> "ungraft" quickly, maybe in a week or two. We never planned to keep
> packages grafted for several months as we do now — grafts were
> considered a necessary but unfortunate violation of the functional
> packaging model.

I disagree that grafts are a violation of the functional packaging
model.  The only violations of the functional package model that I'm
aware of are cases where packages fail to build deterministically.  In
other words, if a set of packages build deterministically, i.e. if the
process to build them is a mathematical function, then it will still be
a mathematical function when grafts are added.

However, one potential issue with grafting is that packages are _built_
against the original (buggy) packages, and then store references to
those buggy packages are later replaced (via copying, not mutation) with
references to their replacements.  In some cases, that might not have
the same result as building against the replacements to begin with.

So, it's still a mathematical function (assuming all packages build
deterministically), but not necessarily the same function as if normal
upgrades had been done instead of grafts.  That's the main issue with
grafts that I'm aware of.

One particular case where there's a difference is if there are any
references to grafted packages in build outputs that the reference
scanner is unable to find, e.g. references within compressed files, or
references encoded in UTF-16.

Even without grafts, these hidden references can be a problem, because
if *all* references to a dependency are invisible to the reference
scanner, then the dependency could be deleted by the garbage collector,
leaving those references dangling.

With grafts, the problem of these hidden references becomes far more
severe, because the hidden references will not be rewritten by the
grafting process, and thus references to packages with security flaws
could remain and be used at run time.  This is not a violation of the
functional packaging model, though.

Does that make sense?

      Regards,
        Mark

Re: New “ungrafting” branch

[Leo Famulari]

On Mon, Dec 14, 2020 at 03:03:16PM -0500, Mark H Weaver wrote:
> So, it's still a mathematical function (assuming all packages build
> deterministically), but not necessarily the same function as if normal
> upgrades had been done instead of grafts.  That's the main issue with
> grafts that I'm aware of.

I agree, it's not a violation of the functional packaging model in a
strict sense.

Re: New “ungrafting” branch

[Andreas Enge]

Hello,

Am Mon, Dec 14, 2020 at 04:00:36PM -0500 schrieb Leo Famulari:
> I agree, it's not a violation of the functional packaging model in a
> strict sense.

if I understand correctly, it remains confusing in a non-strict sense:
grafting is functional, but it is not the same function. Precisely, the
function for grafted packages has more arguments, the original input and the
replacement one. And only one of these arguments is visible in the package
definition, the other one enters through the backdoor (it is declared in
the input itself).

Andreas

Re: New “ungrafting” branch

[zimoun]

Hi Mark and Leo,

Thanks for the explanations.  All is clear.

All the best,
simon

Re: New “ungrafting” branch

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1000 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Following discussions on IRC, I’ve created a new ‘ungrafting’ branch
> that does nothing but ungraft things.
>
> The rationale is that grafts incur additional overhead when installing
> things (the time to create those grafts), so it’s good to clean them up
> once in a while.  Ungrafting in a dedicated branch means we know the
> branch is “safe”, unlike more exploratory branches like ‘staging’ and
> ‘core-updates’.
>
> The plan is to start building it later today, and to hopefully be done
> in a week or so.
>
> Thoughts?

Sounds good. I'm not quite sure what you mean by safe though, it's
definitely likely that things will behave the same, but it's not
certain. Additionally, I guess there will be cases were things fail to
build when the replacement is used in the build process.

I thought the process used to be to push the update to
staging/core-updates when the replacement was introduced to master?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: New “ungrafting” branch

[Ludovic Courtès]

Hi,

Christopher Baines <mail@cbaines.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Following discussions on IRC, I’ve created a new ‘ungrafting’ branch
>> that does nothing but ungraft things.
>>
>> The rationale is that grafts incur additional overhead when installing
>> things (the time to create those grafts), so it’s good to clean them up
>> once in a while.  Ungrafting in a dedicated branch means we know the
>> branch is “safe”, unlike more exploratory branches like ‘staging’ and
>> ‘core-updates’.
>>
>> The plan is to start building it later today, and to hopefully be done
>> in a week or so.
>>
>> Thoughts?
>
> Sounds good. I'm not quite sure what you mean by safe though, it's
> definitely likely that things will behave the same, but it's not
> certain. Additionally, I guess there will be cases were things fail to
> build when the replacement is used in the build process.

My naïve thought was that ungrafting things would just work.  It’s not
quite the case: there were problems with Pango in particular that Marius
fixed.

> I thought the process used to be to push the update to
> staging/core-updates when the replacement was introduced to master?

In practice what happened is that we’d ungraft on these branches lazily.
However, these branches introduce other changes that require more work,
so the effect was that we’d end up carrying many grafts for a long time.

This very focused branch is an attempt to address that.  We’ll see if it
was a good idea once it’s finally merged.

Thanks,
Ludo’.

Re: New “ungrafting” branch

[Maxim Cournoyer]

Hello Ludo,

Ludovic Courtès <ludo@gnu.org> writes:

> Hi Guix!
>
> Following discussions on IRC, I’ve created a new ‘ungrafting’ branch
> that does nothing but ungraft things.
>
> The rationale is that grafts incur additional overhead when installing
> things (the time to create those grafts), so it’s good to clean them up
> once in a while.  Ungrafting in a dedicated branch means we know the
> branch is “safe”, unlike more exploratory branches like ‘staging’ and
> ‘core-updates’.
>
> The plan is to start building it later today, and to hopefully be done
> in a week or so.
>
> Thoughts?

To me, staging is not enough exploratory the point to warrant an
additional ungrafting branch (my preference would be to rather see
staging merged sooner), but then I haven't kept a close look of what's
gone to it recently.

Having more branches also translates to more load on the CI, unless I'm
missing something.

Thank you for the initiative,

Maxim

Re: New “ungrafting” branch

[Mark H Weaver]

Hi Ludovic,

Ludovic Courtès <ludo@gnu.org> writes:

> Following discussions on IRC, I’ve created a new ‘ungrafting’ branch
> that does nothing but ungraft things.
>
> The rationale is that grafts incur additional overhead when installing
> things (the time to create those grafts), so it’s good to clean them up
> once in a while.  Ungrafting in a dedicated branch means we know the
> branch is “safe”, unlike more exploratory branches like ‘staging’ and
> ‘core-updates’.
>
> The plan is to start building it later today, and to hopefully be done
> in a week or so.

This is a good initiative, but it seems to have stalled.  Is there a
reason that it has not yet been merged into 'master'?

On my own system, where I don't use substitutes, I rebased my private
branch onto the 'ungrafting' branch a few weeks ago, rebuilt, and have
been happily using it since.  However, since the 'ungrafting' branch has
fallen behind, I've resorted to adding all of the individual ungrafting
commits onto my private branch, which I now periodically rebase on top
of 'master' as I've been doing for years.

     Regards,
       Mark

Re: New “ungrafting” branch

[Leo Famulari]

On Fri, Jan 15, 2021 at 12:57:50AM -0500, Mark H Weaver wrote:
> Ludovic Courtès <ludo@gnu.org> writes:
> 
> > Following discussions on IRC, I’ve created a new ‘ungrafting’ branch
> > that does nothing but ungraft things.
> >
> > The rationale is that grafts incur additional overhead when installing
> > things (the time to create those grafts), so it’s good to clean them up
> > once in a while.  Ungrafting in a dedicated branch means we know the
> > branch is “safe”, unlike more exploratory branches like ‘staging’ and
> > ‘core-updates’.
> >
> > The plan is to start building it later today, and to hopefully be done
> > in a week or so.
> 
> This is a good initiative, but it seems to have stalled.  Is there a
> reason that it has not yet been merged into 'master'?

The ungrafting branch was merged into the staging branch. Unfortunately,
the staging branch is moving very slowly. I perceive a lack of
interest in working on it :/

I think that the ungrafting branch should have been kept separate and
merged into master quickly.

Re: New “ungrafting” branch

[Ludovic Courtès]

Hi,

Leo Famulari <leo@famulari.name> skribis:

> On Fri, Jan 15, 2021 at 12:57:50AM -0500, Mark H Weaver wrote:
>> Ludovic Courtès <ludo@gnu.org> writes:
>> 
>> > Following discussions on IRC, I’ve created a new ‘ungrafting’ branch
>> > that does nothing but ungraft things.
>> >
>> > The rationale is that grafts incur additional overhead when installing
>> > things (the time to create those grafts), so it’s good to clean them up
>> > once in a while.  Ungrafting in a dedicated branch means we know the
>> > branch is “safe”, unlike more exploratory branches like ‘staging’ and
>> > ‘core-updates’.
>> >
>> > The plan is to start building it later today, and to hopefully be done
>> > in a week or so.
>> 
>> This is a good initiative, but it seems to have stalled.  Is there a
>> reason that it has not yet been merged into 'master'?
>
> The ungrafting branch was merged into the staging branch. Unfortunately,
> the staging branch is moving very slowly. I perceive a lack of
> interest in working on it :/

Apologies for not contributing so far, this month is busy for me.  I
think the slow feedback loop with ci.guix is one of the main reasons
people don’t contribute much.  Perhaps the whole process is also unclear
to people who’re not seasoned contributors?

> I think that the ungrafting branch should have been kept separate and
> merged into master quickly.

Yes, that was the goal, but again, ci.guix got in the way by not
providing quick feedback.  Then there were holidays, and then I was busy
with other things, as I guess is the case with the other people who
previously looked into it.

My assumption was that an ungrafting branch would be easy to handle:
trivial changes, no breakage.  But it’s not that simple: we met actual
breakage (Marius fixed non-trivial things), and again, the feedback loop
was too slow.

Ludo’.

Re: New “ungrafting” branch

[Leo Famulari]

On Thu, Jan 21, 2021 at 12:15:28PM +0100, Ludovic Courtès wrote:
> Apologies for not contributing so far, this month is busy for me.  I
> think the slow feedback loop with ci.guix is one of the main reasons
> people don’t contribute much.  Perhaps the whole process is also unclear
> to people who’re not seasoned contributors?

I understand. It wasn't realistic for me to assume people would be
around at this time of year.

The issues with CI definitely dissuade people, including me. I decided
to do this staging cycle because I really want / need a working alsa-lib
for foreign distros.

The CI process is confusing, even to seasoned contributors. That's my
experience as someone who led core-updates on Hydra. Getting a shell
account on berlin helped immensely, because it allows me to retry
building derivations "by hand". But, it's not so good that that is the
level we are working at, in my opinion.

Mathieu Othacehe's work is improving the situation rapidly, and the Guix
Data Service has helped immensely, since I started learning about it.
The Guix Build Coordinator is also great.

I think that we will be in better shape for the next cycle, which I
suppose will be core-updates. I recommend waiting a couple months to
begin building it.