Reducing "You found a bug" reports

Reducing "You found a bug" reports

[Ian Eure]

There’s a steady number of bug reports generated by the "You found 
a bug" message which happens during `guix pull's.  The 
overwhelming majority of these reports are caused by networking 
problems or the Guix infrastructure being unreliable or 
overloaded.  Many of these were submitted during the recent 
guix.gnu.org downtime.

Some of these that I see:

55066
62023
62830
61520
58309

...I’m sure there are many more.

Is there some way for this code to be smarter about when it prints 
the "report a bug" message, so it doesn’t tell users to report 
bugs when none exist?  Is there a way for it to notice that the 
problem is related to networking, and tell the users to try again 
in a little while?  Is it worth removing the "report a bug" 
message entirely?

It doesn’t feel great to tell users to report a bug for things 
that aren’t bugs.  They’re either closed, or never followed up on; 
it’s a poor experience on both ends.

Thanks,

  — Ian

Re: Reducing "You found a bug" reports

[Ludovic Courtès]

Hi,

Ian Eure <ian@retrospec.tv> skribis:

> Is there some way for this code to be smarter about when it prints the
> "report a bug" message, so it doesn’t tell users to report bugs when
> none exist?  Is there a way for it to notice that the problem is
> related to networking, and tell the users to try again in a little
> while?  Is it worth removing the "report a bug" message entirely?
>
> It doesn’t feel great to tell users to report a bug for things that
> aren’t bugs.  They’re either closed, or never followed up on; it’s a
> poor experience on both ends.

I agree, it’s pretty bad.

I’m fine removing the “report a bug” message, maybe replacing it with
some clearer diagnostic and suggestion?  WDYT?

Thanks,
Ludo’.

Re: Reducing "You found a bug" reports

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi Ludo' & Ian,

On Mon, Jun 17 2024, Ludovic Courtès wrote:

> I’m fine removing the “report a bug” message [...] WDYT?

Just a quick side note that some members in our community (not I) are
offended by the word "bug" to describe software defects.  Perhaps here
is a chance to replace it?

Kind regards
Felix, a beekeeper

Re: Reducing "You found a bug" reports

[Ricardo Wurmus]

Felix Lechner via "Development of GNU Guix and the GNU System distribution." <guix-devel@gnu.org> writes:

> On Mon, Jun 17 2024, Ludovic Courtès wrote:
>
>> I’m fine removing the “report a bug” message [...] WDYT?
>
> Just a quick side note that some members in our community (not I) are
> offended by the word "bug" to describe software defects.  Perhaps here
> is a chance to replace it?

I'm a big fan of bugs (the crawling kind, and based on my track record
apparently also the other kind).  The traditional use is about *dead*
bugs gumming up the works.  There is no insecticidal connotation;
removing bugs from open (...free?) machines is a necessity because they
have an unfortunate propensity for getting themselves into crevices that
they can't get out of.  Poor things.

If I was to drop the term "bug" I'd only do it to spite Thomas Edison,
whose letter appears to be the first documented case of using the term
in association with engineering defects.

-- 
Ricardo, a gardener

Re: Reducing "You found a bug" reports

[Attila Lendvai]

> Just a quick side note that some members in our community (not I) are
> offended by the word "bug" to describe software defects.


welcome to the internet of billions of people, where the cost of expressing one's offence doesn't cost anything anymore, and there are dozens of offended people even for the most innocent of statements.

and with the advent of LLMs, it can soon become a kind of a security hole for intellectual DoS attacks -- if not already.

not sure how to keep the SNR high, but the communities will need to adapt.

--
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“Knowledge will forever govern ignorance: And a people who mean to be their own Governors, must arm themselves with the power which knowledge gives.”
	— James Madison (1751–1836), 'Epilogue: Securing the Republic' (1822)

Re: Reducing "You found a bug" reports

[Simon Tournier]

Hi,

On Mon, 17 Jun 2024 at 14:59, Ludovic Courtès <ludo@gnu.org> wrote:

>> It doesn’t feel great to tell users to report a bug for things that
>> aren’t bugs.  They’re either closed, or never followed up on; it’s a
>> poor experience on both ends.
>
> I agree, it’s pretty bad.
>
> I’m fine removing the “report a bug” message, maybe replacing it with
> some clearer diagnostic and suggestion?  WDYT?

Could we detect if it comes from networking?  Somehow catch the error
and run some code that checks stuff and so report more “accurate”
messages?

Well, easier to say… :-)

Cheers,
simon

Re: Reducing "You found a bug" reports

[Ludovic Courtès]

Hi,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> On Mon, 17 Jun 2024 at 14:59, Ludovic Courtès <ludo@gnu.org> wrote:
>
>>> It doesn’t feel great to tell users to report a bug for things that
>>> aren’t bugs.  They’re either closed, or never followed up on; it’s a
>>> poor experience on both ends.
>>
>> I agree, it’s pretty bad.
>>
>> I’m fine removing the “report a bug” message, maybe replacing it with
>> some clearer diagnostic and suggestion?  WDYT?
>
> Could we detect if it comes from networking?  Somehow catch the error
> and run some code that checks stuff and so report more “accurate”
> messages?

Someone™ would need to analyze some of the reports we have (some of them
have already been commented on) to determine what happened and whether
that is something we could diagnose differently.

Ludo’.

Re: Reducing "You found a bug" reports

[Simon Tournier]

Hi,

On Sun, 21 Jul 2024 at 15:16, Ludovic Courtès <ludo@gnu.org> wrote:

>>> I’m fine removing the “report a bug” message, maybe replacing it with
>>> some clearer diagnostic and suggestion?  WDYT?
>>
>> Could we detect if it comes from networking?  Somehow catch the error
>> and run some code that checks stuff and so report more “accurate”
>> messages?
>
> Someone™ would need to analyze some of the reports we have (some of them
> have already been commented on) to determine what happened and whether
> that is something we could diagnose differently.

Over the years, I have spent some time in answering to some of these bug
reports; see [1] for one instance of an attempt to be this Someone™ ;-).

More than often, the issue is transient, hard if not impossible to
reproduce and thus hard to analyze.  Even, when it happens for me,
sometimes I say ok let try to collect some information for helping in
debugging that but then just running again “guix pull” makes it pass.

Excluding the bug between the keyboard and the chair, most of the time
it seems coming from either user’s network, or either substitute
servers.  Most of the time the bang is transient.

Bah I do not know exactly what or how – otherwise I would have proposed
a patch or something ;-) – but I think we need to “instrument“ the
command “guix pull” in order to collect a trace for then diagnosing.

Somehow, maybe it could be a good application of Maxim’s proposal for
logging [2].

Well, instead of an “useless” backtrace, it would be more helpful to
have a kind of log or coredump; maybe something under /var/log/guix/.

I think the next step is not « Someone™ would need to analyze some of
the reports » because the reports often provide only the same “useless“
backtrace, so instead the next step seems: « Someone™ would need to
implement a better way for generating good reports ». ;-)

Cheers,
simon

1: collection of “guix pull“ bug reports
Simon Tournier <zimon.toutoune@gmail.com>
Wed, 23 Aug 2023 18:17:20 +0200
id:86jztl20of.fsf@gmail.com
https://lists.gnu.org/archive/html/guix-devel/2023-08
https://yhetil.org/guix/86jztl20of.fsf@gmail.com

2: [bug#68946] [RFC PATCH 0/1] Add logging capability to Guix
Maxim Cournoyer <maxim.cournoyer@gmail.com>
Mon, 05 Feb 2024 23:12:00 -0500
id:cover.1707192720.git.maxim.cournoyer@gmail.com
https://issues.guix.gnu.org/68946
https://issues.guix.gnu.org/msgid/cover.1707192720.git.maxim.cournoyer@gmail.com
https://yhetil.org/guix/cover.1707192720.git.maxim.cournoyer@gmail.com

Re: Reducing "You found a bug" reports

[Suhail Singh]

Simon Tournier <zimon.toutoune@gmail.com> writes:

> More than often, the issue is transient, hard if not impossible to
> reproduce and thus hard to analyze.  Even, when it happens for me,
> sometimes I say ok let try to collect some information for helping in
> debugging that but then just running again “guix pull” makes it pass.

Doesn't that suggest, that it may help to reword the below:

#+caption: <https://issues.guix.gnu.org/55066#0-lineno145>
#+begin_quote
  guix pull: error: You found a bug: the program '/gnu/store/mpfp9nrzifhp3r5s3bv05b8xal5aa44f-compute-guix-derivation'
  failed to compute the derivation for Guix (version: "e32cc011bbe899fda432906776702f74fa6b1450"; system: "x86_64-linux";
  host version: "eb34ff16cc9038880e87e1a58a93331fca37ad92"; pull-version: 1).
  Please report the COMPLETE output above by email to <bug-guix@gnu.org>.
#+end_quote

So that we encourage users to try a few times (perhaps possibly with
--fallback), and in case of repeat failures to then submit a bug report
to <bug-guix@gnu.org>.  What about something like:

#+begin_example
  guix pull: error: An error occurred: the program '/gnu/store/mpfp9nrzifhp3r5s3bv05b8xal5aa44f-compute-guix-derivation'
  failed to compute the derivation for Guix (version: "e32cc011bbe899fda432906776702f74fa6b1450"; system: "x86_64-linux";
  host version: "eb34ff16cc9038880e87e1a58a93331fca37ad92"; pull-version: 1).

  Please try running "guix pull" again.  In case of repeated failure,
  please try passing "--fallback" to "guix pull".  In case that still
  doesn't resolve the error, please report the COMPLETE output above by
  email to <bug-guix@gnu.org>.
#+end_example

I.e., perhaps the first step is simply to reword above so that we don't
recommend that users submit a bug report on their first (possibly
transient) failure.

-- 
Suhail

toward a plan? (was Re: Reducing "You found a bug" reports)

[Simon Tournier]

Hi,

As Ian pointed out earlier, here some “guix pull” bugs:

    55066
    58309  closed
    61520  closed
    62023  closed
    62830

And most of them are transient or hard to reproduce.  More had been
listed in [1], it reads:

    63451
    63830
    64489
    64659  v1.4.0
    64753
    64963

And it’s often the same: transient or hard to reproduce.  Here a larger
collection continuing…

    64753
    65495
    65549
    65560  v1.4.0
    66600  v1.4.0
    67035
    67179  v1.2.0
    67482
    67806  v1.4.0
    67956  v1.2.0
    67965  v1.4.0
    68397  v1.4.0
    69127  v1.4.0
    69334
    69726  v1.3.0
    70075  v1.3.0
    70192
    70200  v1.4.0
    70201  v1.4.0
    70297  v1.3.0
    70646  v1.4.0
    70649
    70650  v1.4.0
    70651
    70658  v1.4.0
    70667
    70681  v1.3.0
    70940
    71426
    71437  v1.4.0
    71691  v1.4.0
    71908
    71945
    72028  v1.4.0
    72100
    72135  upgrade from v1.2.0    
    72332  v1.4.0
    72353  v1.4.0
    72563  v1.4.0
    72639  v1.4.0

Please note v1.4.0 means the host revision was v1.4.0.


After looking at some of these, we have 3 classes of bugs:

 1. transient
 2. pulling from too old host revision
 3. mix of both


IMHO, the next actions are:

 a) Replace this message:

--8<---------------cut here---------------start------------->8---
                      (message (format #f "You found a bug: the program '~a'
failed to compute the derivation for Guix (version: ~s; system: ~s;
host version: ~s; pull-version: ~s).
Please report the COMPLETE output above by email to <~a>.~%"
--8<---------------cut here---------------end--------------->8---

    by something like: “sorry, could you try again guix pull --commit=~s
    and report if it fails again.”


 b) Put here and there some logging [2] information.  Patch#68946 [2]
    provides logging facilities but is missing concrete user.

    It could be worth to have it.  So then, once “guix pull” fails, we
    could ask to re-run “guix pull --commit=<target> --log-level=debug”.

    This would help in having some information at failure time instead
    of asking them after.

    Moreover, it would provide information in order to diagnose all
    these transient errors and see if they could be catched up instead
    of erroring.

WDYT?

In all cases, feel free to pick one or more bugs from the list above and
investigate.  Many do not have any answer – which is not welcoming and
friendly.

Cheers,
simon



1: collection of “guix pull“ bug reports
Simon Tournier <zimon.toutoune@gmail.com>
Wed, 23 Aug 2023 18:17:20 +0200
id:86jztl20of.fsf@gmail.com
https://lists.gnu.org/archive/html/guix-devel/2023-08
https://yhetil.org/guix/86jztl20of.fsf@gmail.com

2: [bug#68946] [RFC PATCH 0/1] Add logging capability to Guix
Maxim Cournoyer <maxim.cournoyer@gmail.com>
Mon, 05 Feb 2024 23:12:00 -0500
id:cover.1707192720.git.maxim.cournoyer@gmail.com
https://issues.guix.gnu.org/68946
https://issues.guix.gnu.org/msgid/cover.1707192720.git.maxim.cournoyer@gmail.com
https://yhetil.org/guix/cover.1707192720.git.maxim.cournoyer@gmail.com

Re: toward a plan? (was Re: Reducing "You found a bug" reports)

[Suhail Singh]

Simon Tournier <zimon.toutoune@gmail.com> writes:

> IMHO, the next actions are:
>
>  a) Replace this message:
>
>                       (message (format #f "You found a bug: the program '~a'
> failed to compute the derivation for Guix (version: ~s; system: ~s;
> host version: ~s; pull-version: ~s).
> Please report the COMPLETE output above by email to <~a>.~%"
>
>     by something like: “sorry, could you try again guix pull --commit=~s
>     and report if it fails again.”

That would be an improvement over current state, however ...

>  b) Put here and there some logging [2] information.  Patch#68946 [2]
>     provides logging facilities but is missing concrete user.
>
>     It could be worth to have it.  So then, once “guix pull” fails, we
>     could ask to re-run “guix pull --commit=<target> --log-level=debug”.
>
>     This would help in having some information at failure time instead
>     of asking them after.
>
>     Moreover, it would provide information in order to diagnose all
>     these transient errors and see if they could be catched up instead
>     of erroring.
>
> WDYT?

I agree that this would be even better and IMHO we should implement b.
I.e., I think upon failure the message should:

1. recommend retrying at least once before submitting a bug report, and
2. should ensure submitted bug reports are as informative as possible.

-- 
Suhail