Policy to remove obsolete packages

Policy to remove obsolete packages

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 510 bytes --]

Hi Guix,

on Guix-Days we had no more time in the QA track to talk about obsolete
packages.

In the FOSDEM-talk by Frederic Crozat (SuSE) about Distributions, he
mentioned their approach which I find good to adapt:


If a package is broken for more than 6 months, we should just remove it
from Guix. Prior to removing, we should announce on the dev mailing
list, maybe someone will care about it then. If there is no response
within 2 weeks, we really remove it.

What do you think?

Björn


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Policy to remove obsolete packages

[Pjotr Prins]

On Mon, Feb 04, 2019 at 12:16:12PM +0100, Björn Höfling wrote:
> Hi Guix,
> 
> on Guix-Days we had no more time in the QA track to talk about obsolete
> packages.
> 
> In the FOSDEM-talk by Frederic Crozat (SuSE) about Distributions, he
> mentioned their approach which I find good to adapt:
> 
> 
> If a package is broken for more than 6 months, we should just remove it
> from Guix. Prior to removing, we should announce on the dev mailing
> list, maybe someone will care about it then. If there is no response
> within 2 weeks, we really remove it.
> 
> What do you think?

+1.

It would also be nice to add an OBSOLETE/DEPRECATE tag or comment
which can be removed when a package works again. This could be made
visible as metadata. Maybe even link it with debbugs.

Pj.

Re: Policy to remove obsolete packages

[Andreas Enge]

On Mon, Feb 04, 2019 at 12:16:12PM +0100, Björn Höfling wrote:
> If a package is broken for more than 6 months, we should just remove it
> from Guix. Prior to removing, we should announce on the dev mailing
> list, maybe someone will care about it then. If there is no response
> within 2 weeks, we really remove it.

We were both in the talk and had the same reaction, so indeed I agree.

Andreas

Re: Policy to remove obsolete packages

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 860 bytes --]

On Mon, Feb 04, 2019 at 07:06:35PM +0100, Andreas Enge wrote:
> On Mon, Feb 04, 2019 at 12:16:12PM +0100, Björn Höfling wrote:
> > If a package is broken for more than 6 months, we should just remove it
> > from Guix. Prior to removing, we should announce on the dev mailing
> > list, maybe someone will care about it then. If there is no response
> > within 2 weeks, we really remove it.
> 
> We were both in the talk and had the same reaction, so indeed I agree.

I agree with this as well.

It might be tricky to know when a package has failed to build for 6
months. Do we keep build farm evaluations for that long? Can we automate
failure notifications after a certain date?

But, it's not so important whether or not it has been exactly 6 months.
At a certain point we know it's been too long and that the package is
not useful anymore.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Policy to remove obsolete packages

[Ludovic Courtès]

Hi!

Björn Höfling <bjoern.hoefling@bjoernhoefling.de> skribis:

> In the FOSDEM-talk by Frederic Crozat (SuSE) about Distributions, he
> mentioned their approach which I find good to adapt:
>
>
> If a package is broken for more than 6 months, we should just remove it
> from Guix. Prior to removing, we should announce on the dev mailing
> list, maybe someone will care about it then. If there is no response
> within 2 weeks, we really remove it.
>
> What do you think?

I believe a policy along these lines would make sense.

(Note that, IIUC, in openSuSE a package can be broken and yet remain
installable by users, because the last binary that was produced is still
around.)

Ludo’.

Re: Policy to remove obsolete packages

[zimoun]

Dear,

I agree too on the policy.

On Mon, 4 Feb 2019 at 23:26, Leo Famulari <leo@famulari.name> wrote:
>
> It might be tricky to know when a package has failed to build for 6
> months. Do we keep build farm evaluations for that long? Can we automate
> failure notifications after a certain date?

I agree.
To me the question is more about how to automatically detect that a
patch breaks another package than a policy to remove a broken package.

I mean if we are able to automatically detect 6 months later that a
package is broken, why are we not able to report the failure earlier?
And if it is not automatic, this means that someone reports  the
failure by hand, so somehow this patch is important to them and then
why remove it?




All the best,
simon

Re: Policy to remove obsolete packages

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 382 bytes --]

On Mon, 04 Feb 2019 23:52:47 +0100
Ludovic Courtès <ludo@gnu.org> wrote:


> (Note that, IIUC, in openSuSE a package can be broken and yet remain
> installable by users, because the last binary that was produced is
> still around.)

We have guix pull --commit=..., inferiors, channels and time-travel, so
there are plenty opportunities to keep old states :-)

Björn


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Policy to remove obsolete packages

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 1881 bytes --]

On Tue, 5 Feb 2019 00:47:42 +0100
zimoun <zimon.toutoune@gmail.com> wrote:

> To me the question is more about how to automatically detect that a
> patch breaks another package than a policy to remove a broken package.
> 
> I mean if we are able to automatically detect 6 months later that a
> package is broken, why are we not able to report the failure earlier?
> And if it is not automatic, this means that someone reports  the
> failure by hand, so somehow this patch is important to them and then
> why remove it?

1. To check that A has dependency B and upgrading A breaks B:

We discussed on Guix-Days on how to automate this and there are some
ideas around, but nothing fully-automated is there yet.

Current state:

When updating A, developers can call "guix refresh -l" and see which
B's are affected and can build them too (if not too much).

Also, the build farms are evaluating them eventually and you can
manually look up if something went wrong (though this is tedious and
errors can slip through).

So, no, there currently is no automatic notification/report. Thus,
someone does this by hand. If it is a package of interest to that
person, they at least write a bug report or have ideas on how to fix it.

It could also be a person that has no direct interest or is not an
expert for that package. For example, I sometimes just go through the
errors of the latest evaluations (https://hydra.gnu.org/evals) and see
if anything can be done.

Then I notice that it probably broke because of some qt-update, nobody
cared, nobody still cares, I'm looking on how to fix it, upstream has
no patches since months/years, project was forked, totally restructured
etc.

If then after a post to the list still nobody shouts up, I find it
reasonable to remove the package. Sure the period of 6 months is just a
rough notion.

Björn

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Policy to remove obsolete packages

[ng0]

Bjrn Hfling transcribed 846 bytes:
> On Mon, 04 Feb 2019 23:52:47 +0100
> Ludovic Courtès <ludo@gnu.org> wrote:
> 
> 
> > (Note that, IIUC, in openSuSE a package can be broken and yet remain
> > installable by users, because the last binary that was produced is
> > still around.)
> 
> We have guix pull --commit=..., inferiors, channels and time-travel, so
> there are plenty opportunities to keep old states :-)

There are many ways to keep it, but they are really sometimes just jumping through too many hoops. 
Or depending on what your idea of keeping old packages is. it should be easy, but
it involves a good amount[1] of work to build a much older version
with the otherwise almost-only recent,updating,master.
To the point where you have to do the logical thing and look into
which versions upstream or guix build around that time as dependencies
and simply "freeze" all the dependencies in your package.

1: amount depending on what you are building

There are other ways to handle obsolete packages, but I think they
don't map to how guix works:

a year or 2 back i experimented with a complete resructure of Guix,
and packages got split up differently (one module per package mostly)
leading to different kinds of problems and fixes.
a separate repository with the prefix -wip holds all the unstable,
obsolete, unfinished, etc packagesi (remotely comparable to how
ports trees are handled, but not quiet like it[1]). That's the gist
of it. just have a repository instead of dropping it from a tree.
Once it's fixed up in the "wip" repository, move it back into the
main repository.
I can elaborate more on this if you want me to once I'm no longer sick.

> Björn
> 

Re: Policy to remove obsolete packages

[swedebugia]

[-- Attachment #1: Type: text/plain, Size: 2898 bytes --]

ng0@n0.is skrev: (5 februari 2019 22:31:53 CET)
>Bjrn Hfling transcribed 846 bytes:
>> On Mon, 04 Feb 2019 23:52:47 +0100
>> Ludovic Courtès <ludo@gnu.org> wrote:
>> 
>> 
>> > (Note that, IIUC, in openSuSE a package can be broken and yet
>remain
>> > installable by users, because the last binary that was produced is
>> > still around.)
>> 
>> We have guix pull --commit=..., inferiors, channels and time-travel,
>so
>> there are plenty opportunities to keep old states :-)
>
>There are many ways to keep it, but they are really sometimes just
>jumping through too many hoops. 
>Or depending on what your idea of keeping old packages is. it should be
>easy, but
>it involves a good amount[1] of work to build a much older version
>with the otherwise almost-only recent,updating,master.
>To the point where you have to do the logical thing and look into
>which versions upstream or guix build around that time as dependencies
>and simply "freeze" all the dependencies in your package.
>
>1: amount depending on what you are building
>
>There are other ways to handle obsolete packages, but I think they
>don't map to how guix works:
>
>a year or 2 back i experimented with a complete resructure of Guix,
>and packages got split up differently (one module per package mostly)
>leading to different kinds of problems and fixes.
>a separate repository with the prefix -wip holds all the unstable,
>obsolete, unfinished, etc packagesi (remotely comparable to how
>ports trees are handled, but not quiet like it[1]). That's the gist
>of it. just have a repository instead of dropping it from a tree.
>Once it's fixed up in the "wip" repository, move it back into the
>main repository.
>I can elaborate more on this if you want me to once I'm no longer sick.
>
>> Björn
>> 

Interesting!
I like the idea of keeping it simple and now we tried the lumped modules approach. I don't like it so much to be honest.

It comes with obvious drawbacks when the package per file grow and subcategorization have to be done.

But is it efficient in guile to load hundreds of modules where all pull in more or less the same dependencies? 

If yes I think your idea is worthwhile Nils. 

We might have 3 repos: wip, core, extra

But switching to one module per package might involve a lot of work. Can we automate it somehow? 
If yes we will probably end up with a couple thousand modules that import more modules than necessary. E.g. it would be no breakage if the split script simply includes all use-module-lines of the parent in the new child modules.

Could we use an AI to help find unneded use-modules afterwards? Maybe just a half intelligent one that tries removing them one by one and sees if the derivation is computed correctly and report back a pairs of modules . use-modules-lines that are superfluous.
-- 
Sent from my k-9 mail for Android.

[-- Attachment #2: Type: text/html, Size: 3595 bytes --]

Re: Policy to remove obsolete packages

[ng0]

swedebugia transcribed 6.9K bytes:
> ng0@n0.is skrev: (5 februari 2019 22:31:53 CET)
> >Bjrn Hfling transcribed 846 bytes:
> >> On Mon, 04 Feb 2019 23:52:47 +0100
> >> Ludovic Courtès <ludo@gnu.org> wrote:
> >> 
> >> 
> >> > (Note that, IIUC, in openSuSE a package can be broken and yet
> >remain
> >> > installable by users, because the last binary that was produced is
> >> > still around.)
> >> 
> >> We have guix pull --commit=..., inferiors, channels and time-travel,
> >so
> >> there are plenty opportunities to keep old states :-)
> >
> >There are many ways to keep it, but they are really sometimes just
> >jumping through too many hoops. 
> >Or depending on what your idea of keeping old packages is. it should be
> >easy, but
> >it involves a good amount[1] of work to build a much older version
> >with the otherwise almost-only recent,updating,master.
> >To the point where you have to do the logical thing and look into
> >which versions upstream or guix build around that time as dependencies
> >and simply "freeze" all the dependencies in your package.
> >
> >1: amount depending on what you are building
> >
> >There are other ways to handle obsolete packages, but I think they
> >don't map to how guix works:
> >
> >a year or 2 back i experimented with a complete resructure of Guix,
> >and packages got split up differently (one module per package mostly)
> >leading to different kinds of problems and fixes.
> >a separate repository with the prefix -wip holds all the unstable,
> >obsolete, unfinished, etc packagesi (remotely comparable to how
> >ports trees are handled, but not quiet like it[1]). That's the gist
> >of it. just have a repository instead of dropping it from a tree.
> >Once it's fixed up in the "wip" repository, move it back into the
> >main repository.
> >I can elaborate more on this if you want me to once I'm no longer sick.
> >
> >> Björn
> >> 
> 
> Interesting!
> I like the idea of keeping it simple and now we tried the lumped modules approach. I don't like it so much to be honest.
> 
> It comes with obvious drawbacks when the package per file grow and subcategorization have to be done.
> 
> But is it efficient in guile to load hundreds of modules where all pull in more or less the same dependencies? 
> 
> If yes I think your idea is worthwhile Nils. 

(just aside: I prefer ng0) 
 
> We might have 3 repos: wip, core, extra
> 
> But switching to one module per package might involve a lot of work. Can we automate it somehow? 
> If yes we will probably end up with a couple thousand modules that import more modules than necessary. E.g. it would be no breakage if the split script simply includes all use-module-lines of the parent in the new child modules.
> 
> Could we use an AI to help find unneded use-modules afterwards? Maybe just a half intelligent one that tries removing them one by one and sees if the derivation is computed correctly and report back a pairs of modules . use-modules-lines that are superfluous.

my experiments (with guix) predated the current discussion on
reorganzing parts of it, so take it with a grain of salt.

I can not link to my layout (eventually I will get to
write about it, ideas moved on, ideas developed from this)
here for reasons of material not allowed in GuixSD. 

The gist at the time before I stopped the work, was to
create a very small guix core, extendable through
plugins by users, a templating system for system
configurations and all sorts of things I don't want
to explain now.
packages were split into some essential core packages
(given a very minimal server or some other factors,
this can not be separated from core) inside "guix"
and a bigger collection of everything else in "ports".
This went through some itterations, got a bunch of
new packages I never managed to send back (and fixes
which should make their way into guix.. we had an
functional Nim for example, no idea if someone
fixed it in guix master in the last year), and ports
followed this abbreviated style: 

.
bin/
~doc
~etc
ports/
ports/$category
ports/patches
ports/$category/$name/$name.scm
AUTHORS
CHANGELOG
LEGAL
LICENSE
MOVED
TODO
UPDATING
config.mk
Makefile
ports.scm

etc..

the category/name part was mostly so that I could have deeper
nestings for languages, python variants, variants in general,
and so forth. it was just an itteration of ideas in the beginning.
categories were largely arbitrary when they weren't cross-compared
to pkgsrc, ports, portage and others.

I can't really say if this would help with the layout guix has because
I went through great length dissecting that. At the time I had some
ideas growing, and some established, and wanted to see how they
could scuplture/terraform guix.

I think the one-module thing was discussed before in guix but I
did not follow it at the time.

The 'freezing DAGs' (sort of) is still something I'm thinking
about, but with additional ideas came additional thoughts
about languages and requirements, leading away from Guix. 

if you do let's call it technically incorrect "collision detection"
for such a large collection manually you want automation
as much as possible, as you pointed out.

just take it as some rambling about very detailed notes I have made but
not in the well-furbished format that I'd publish them on
my website and 50% or so still on (physical) paper,..

> -- 
> Sent from my k-9 mail for Android.

Re: Policy to remove obsolete packages

[Ricardo Wurmus]

swedebugia <swedebugia@riseup.net> writes:

> I like the idea of keeping it simple and now we tried the lumped
> modules approach. I don't like it so much to be honest.
>
> It comes with obvious drawbacks when the package per file grow and
> subcategorization have to be done.
>
> But is it efficient in guile to load hundreds of modules where all
> pull in more or less the same dependencies?
>
> If yes I think your idea is worthwhile Nils.
>
> We might have 3 repos: wip, core, extra
[…]

This is a tangent.  This thread is about removing obsolete packages.
Let’s not discuss moving packages each to their own module here.  This
has been discussed elsewhere and it’s a can of worms (which is fine if
you’re a bird, but not so good if you want to close the can again).

If you’re interested in playing with this you can do this in a local
branch and see how it behaves and what drawbacks it has.  But I don’t
think it’s a good use of our time discussing it (again).

--
Ricardo

Re: Policy to remove obsolete packages

[zimoun]

Hi,

I understand but I am not sure to see the points and/or advantages
about a policy.

From my opinion, obsolete package is not well-defined and define
cleanly what an obsolete package is will be bikeshedding. :-)
And I think that deprecated should come from upstream.
However, a popcon of the downloaded substitutes should provide which
packages are "important" and which are less; to have a better
"priority list"---if needed.

To me, all the QA dance of the "classic" distros come from two key
points: missing the rollback and the dependency hell. Because it is
hard to rollback if the update/upgrade fails, the user must be sure
that nothing will break.
Since Guix fixes these two points by design, it does not need a strong
QA, I guess.

But, I do agree with you that it should not be possible that `guix
pull [options]' then `guix build <package>' fail. Never. :-)
And maybe the "CI" should have a mechanism such that: pull from
branch-unstable, refresh and eval then automatically push to
branch-stable if ok, otherwise blame the committer who will manually
fix and will push again to branch-unstable. The regular user can add
the both branches with the channel mechanism and they will be more
sure that `guix pull --commit=' will always work and obtain the last
half baked cutting edge stuffs too.

And I also do agree that it is hard to find the information what it
went wrong. For example, recently I was not able to find what breaks
clang@3.5.


Well, talk does not cook the rice. :-)
(I mean not sure my words are relevant)

All the best,
simon

Re: Policy to remove obsolete packages

[swedebugia]

On 2019-02-06 23:32, Ricardo Wurmus wrote:
> This is a tangent.  This thread is about removing obsolete packages.
> Let’s not discuss moving packages each to their own module here.  This
> has been discussed elsewhere and it’s a can of worms (which is fine if
> you’re a bird, but not so good if you want to close the can again).
> 
> If you’re interested in playing with this you can do this in a local
> branch and see how it behaves and what drawbacks it has.  But I don’t
> think it’s a good use of our time discussing it (again).

Thanks for the heads up. I missed the conversation. Its not really that 
important to me anyway.

-- 
Cheers
Swedebugia

Re: Policy to remove obsolete packages

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 1584 bytes --]

Hi simon,

On Thu, 7 Feb 2019 13:40:53 +0100
zimoun <zimon.toutoune@gmail.com> wrote:

> Hi,
> 
> I understand but I am not sure to see the points and/or advantages
> about a policy.
> 
> From my opinion, obsolete package is not well-defined and define  
> cleanly what an obsolete package is will be bikeshedding. :-)
> And I think that deprecated should come from upstream.
> However, a popcon of the downloaded substitutes should provide which
> packages are "important" and which are less; to have a better
> "priority list"---if needed.

I really wouldn't take that "Policy" too formal: It is more like that:
There are packages that are just broken and nobody cares. But nobody
removed them from Guix, because everybody was unsure about it.

Now we agreed on that when a package is broken for more than 6 months
we should announce it on dev-list and if then nobody takes action, we
remove it. It is more like "community consensus" that it is very OK to
remove these packages.

CI is another related topic we are working on. In theory, a commit
should break nothing. Or at least the commiter should get a message
back of what exactly they broke.

> And I also do agree that it is hard to find the information what it
> went wrong. For example, recently I was not able to find what breaks
> clang@3.5.

Sometimes it magically helps to just write a bug-report with the
correct error-message and/or link to hydra.gnu.org's failure. I noticed
in the past that a good, concise bug-report gets attention to the right
people to fix it :-)

Björn

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]