Bug statistics

Bug statistics

[Eli Zaretskii]

Does debbugs*.el support compiling statistics about our bugs?  Like
how many new bugs were reported and how many were closed during a
given period of time, including distribution by severity?

I think it might make sense to have such a feature, and perhaps even
post the statistics from time to time, so that we know how we are
doing on that front.

Does it make sense?  Would someone like to work on this?

TIA

Re: Bug statistics

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Does debbugs*.el support compiling statistics about our bugs?  Like
> how many new bugs were reported and how many were closed during a
> given period of time, including distribution by severity?

It doesn't, but there's this:

https://debbugs.gnu.org/rrd/emacs.html

I thought those charts were a bit too coarse, so I've written some code
that downloads all the debbugs data and mangles the info a bit, and then
output to Javascript so that you can zoom a bit:

http://quimby.gnus.org/circus/stats-emacs/stats-emacs.html

I haven't made my code do the separation by severity, but I've been
meaning to do that...

> I think it might make sense to have such a feature, and perhaps even
> post the statistics from time to time, so that we know how we are
> doing on that front.
>
> Does it make sense?  Would someone like to work on this?

The amount of data you need to do the computations client side are a
bit...  on the large side.  I think my script takes, like, ten minutes
to download the data, so I only run it very occasionally.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Bug statistics

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 20 Oct 2019 22:15:42 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Does debbugs*.el support compiling statistics about our bugs?  Like
> > how many new bugs were reported and how many were closed during a
> > given period of time, including distribution by severity?
> 
> It doesn't, but there's this:
> 
> https://debbugs.gnu.org/rrd/emacs.html

Thanks.  Maybe this is enough.  A distribution of the time since
report to solution would be nice, though.

How and by whom are these charts produced?

> I thought those charts were a bit too coarse, so I've written some code
> that downloads all the debbugs data and mangles the info a bit, and then
> output to Javascript so that you can zoom a bit:

I actually thought about producing input for something like Gnuplot.

> The amount of data you need to do the computations client side are a
> bit...  on the large side.  I think my script takes, like, ten minutes
> to download the data, so I only run it very occasionally.

Sure, but this is hardly needed every day.  Once a month or 3 months
should be good enough, I think.

Re: Bug statistics

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> The amount of data you need to do the computations client side are a
>> bit...  on the large side.  I think my script takes, like, ten minutes
>> to download the data, so I only run it very occasionally.
>
> Sure, but this is hardly needed every day.  Once a month or 3 months
> should be good enough, I think.

Hmmm, we made already an extension to the Debbugs::SOAP interface,
offered on debbugs.gnu.org (function search_est for searching in the
bugs database). So we could add another Debbugs::SOAP function to
provide statistical data.

OTOH, the long term plan is to merge our changes to upstream
Debbugs. Further extensions would be in the way.

Best regards, Michael.

Re: Bug statistics

[Lars Ingebrigtsen]

I was inspired to tweak the scripts a bit and output more information:

https://quimby.gnus.org/circus/stats-emacs/stats-emacs.html

The potentially most interesting data set it the final chart (and it's
not on the rrd page).  It shows the percentage of bugs that are closed
within a week/month/year of reporting.  It's pretty noisy, but it shows
that by this metric, things haven't changed much over the years: Since
2011 about 50% of reported bugs have been fixed within a month.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Bug statistics

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Mon, 21 Oct 2019 13:05:13 +0200
> 
> I was inspired to tweak the scripts a bit and output more information:
> 
> https://quimby.gnus.org/circus/stats-emacs/stats-emacs.html

Thanks.

Re: Bug statistics

[Stefan Monnier]

> Since 2011 about 50% of reported bugs have been fixed within a month.

Thanks Lars.  So it's not as bad as I thought,


        Stefan

Re: Bug statistics

[Lars Ingebrigtsen]

Michael Albinus <michael.albinus@gmx.de> writes:

> Hmmm, we made already an extension to the Debbugs::SOAP interface,
> offered on debbugs.gnu.org (function search_est for searching in the
> bugs database). So we could add another Debbugs::SOAP function to
> provide statistical data.
>
> OTOH, the long term plan is to merge our changes to upstream
> Debbugs. Further extensions would be in the way.

One thing that's a bit annoying with the data the SOAP interface returns
today is that it doesn't really have a field that says when a bug was
closed.  Instead if has a status field, and a last_modified field, and
that's what I'm using in my charts.  However, a bug may be modified
after it's closed: The main reason is when it's moved to the archive one
month later, and that's easy enough to check for (just subtract a month
when it's in the archive).

But it does mean that the charts I've made are slightly inaccurate,
which is annoying.

Is there a date field in the bugs database that says when the bug was
closed?  It seems like it, because the web interface includes that data
point.  Would it be possible to return that in the SOAP response, too?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Bug statistics

[Michael Albinus]

Lars Ingebrigtsen <larsi@gnus.org> writes:

Hi Lars,

>> Hmmm, we made already an extension to the Debbugs::SOAP interface,
>> offered on debbugs.gnu.org (function search_est for searching in the
>> bugs database). So we could add another Debbugs::SOAP function to
>> provide statistical data.
>>
>> OTOH, the long term plan is to merge our changes to upstream
>> Debbugs. Further extensions would be in the way.
>
> One thing that's a bit annoying with the data the SOAP interface returns
> today is that it doesn't really have a field that says when a bug was
> closed.  Instead if has a status field, and a last_modified field, and
> that's what I'm using in my charts.  However, a bug may be modified
> after it's closed: The main reason is when it's moved to the archive one
> month later, and that's easy enough to check for (just subtract a month
> when it's in the archive).

There is fixed_date, which is intended for this. But it doesn't seem to
be set.

> Is there a date field in the bugs database that says when the bug was
> closed?  It seems like it, because the web interface includes that data
> point.  Would it be possible to return that in the SOAP response, too?

Could you please give me an example bug where the web interface tells
this? I didn't find any bug showing this information.

Btw, if you want to see all data fields of a bug in the Emacs debbugs
list, select a bug and type "d".

Best regards, Michael.

Re: Bug statistics

[Lars Ingebrigtsen]

Michael Albinus <michael.albinus@gmx.de> writes:

> Could you please give me an example bug where the web interface tells
> this? I didn't find any bug showing this information.

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=18691

It has log_modified set to the archive time, and the close time at "03
Aug 2019 17:20:03" is nowhere to be seen in the SOAP data.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Bug statistics

[Michael Albinus]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=18691
>
> It has log_modified set to the archive time, and the close time at "03
> Aug 2019 17:20:03" is nowhere to be seen in the SOAP data.

As you see in that web page, it just reports actions at "Sat, 03 Aug
2019 17:20:03 GMT". This doesn't mean, that respective data fields are
added to the bug.

In debbugs.el, both found_date and fixed_date are documented. Plus the
comment "(empty for now)". I don't remember why I have written that, but
it looks like Debbugs::SOAP does not return these field values.

Would need to dig into the Perl sources. Let's see, whether I find
something.

Additionally, I will ask upstream.

Best regards, Michael.

Re: Bug statistics

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

> In debbugs.el, both found_date and fixed_date are documented. Plus the
> comment "(empty for now)". I don't remember why I have written that, but
> it looks like Debbugs::SOAP does not return these field values.

PS: Could you pls write bug about?

Best regards, Michael.

Re: Bug statistics

[Lars Ingebrigtsen]

Michael Albinus <michael.albinus@gmx.de> writes:

> In debbugs.el, both found_date and fixed_date are documented. Plus the
> comment "(empty for now)". I don't remember why I have written that, but
> it looks like Debbugs::SOAP does not return these field values.
>
> Would need to dig into the Perl sources. Let's see, whether I find
> something.
>
> Additionally, I will ask upstream.

Great!

But I wonder...  fixed_date is one thing, but you can close a bug
without marking it as fixed.  Is there a separate date for the "close"
action somewhere in the database?

For instance,

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=18688

was closed without being marked as "fixed".

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#37854: 27.0.50; debbugs-gnu doesn't return date bugs were closed

[Lars Ingebrigtsen]

Apparently somewhere in the database, debbugs knows the date a bug was
closed:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=18688

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=18691

Unless those pages are statically generated and just appended to, as in
a log?  I assumed they were generated on-the-fly from the data in the
database.

There's two data points that would be interesting: The "closed" date, and
the "fixed" date, which may be different.

The problem is that bugs may be updated after they're closed (for
instance when they are archived, but also by other stuff), which makes
it difficult to do accurate statistics about response time and the
like.  I mean, you can do statistics that are probably "good enough",
but it's kinda not very satisfying when you know there's methodological
errors there.


In GNU Emacs 27.0.50 (build 91, x86_64-pc-linux-gnu, GTK+ Version 3.24.5)
 of 2019-10-20 built on marnie
Repository revision: 78cb3791fa11c95756ee3917c63cfea774f128a2
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12004000
System Description: Debian GNU/Linux 10 (buster)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Bug statistics

[Lars Ingebrigtsen]

Michael Albinus <michael.albinus@gmx.de> writes:

> Michael Albinus <michael.albinus@gmx.de> writes:
>
>> In debbugs.el, both found_date and fixed_date are documented. Plus the
>> comment "(empty for now)". I don't remember why I have written that, but
>> it looks like Debbugs::SOAP does not return these field values.
>
> PS: Could you pls write bug about?

OK; done.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#37854: 27.0.50; debbugs-gnu doesn't return date bugs were closed

[Michael Albinus]

Lars Ingebrigtsen <larsi@gnus.org> writes:

[continuing at <37854@debbugs.gnu.org>]

> But I wonder...  fixed_date is one thing, but you can close a bug
> without marking it as fixed.  Is there a separate date for the "close"
> action somewhere in the database?
>
> For instance,
>
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=18688
>
> was closed without being marked as "fixed".

Well, it looks like this close date is not an attribute of the bug, but
something the database keeps internal for countdown of the 28 days,
until the bug will be archived. Once archived, this information doesn't
seem to be available anymore.

Anyway, I've sent a request to <https://bugs.debian.org/942843>.

Best regards, Michael.

bug#37854: 27.0.50; debbugs-gnu doesn't return date bugs were closed

[Lars Ingebrigtsen]

Michael Albinus <michael.albinus@gmx.de> writes:

> Well, it looks like this close date is not an attribute of the bug, but
> something the database keeps internal for countdown of the 28 days,
> until the bug will be archived. Once archived, this information doesn't
> seem to be available anymore.

Right.

I was also wondering why the charts I made estimate the number of tags
(like "moreinfo"/"patch") too high, but of course I only have what the
data look like now.

So while my charts say that there were 200 open "moreinfo" in, say,
December 2018, there weren't that many.  A bug report that got that tag
in later, and was closed later, will be counted as having had that tag
back then.

Would it be possible to include the entire change history in the SOAP
data?  That is, the timestamps when tags/severities are added/removed?
That is, is there a "log" of changes applied to a bug that can be
output?  If there was, that'd also fix the "closed" date handling
problem.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#37854: 27.0.50; debbugs-gnu doesn't return date bugs were closed

[Michael Albinus]

Lars Ingebrigtsen <larsi@gnus.org> writes:

Hi Lars,

> Would it be possible to include the entire change history in the SOAP
> data?  That is, the timestamps when tags/severities are added/removed?
> That is, is there a "log" of changes applied to a bug that can be
> output?  If there was, that'd also fix the "closed" date handling
> problem.

Such history cannot be retrieved from the database I fear. So it cannot
be provided by the SOAP interface.

What you see in the bug reports logs web pages are all the messages,
received for a given bug. They are not downloaded by SOAP, but by their
URL like <https://debbugs.gnu.org/cgi/bugreport.cgi?mboxstat=yes;bug=37818>.
This is the counterpart to what is shown via
<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=37818>.

All messages "bug marked as fixed in version 27.1, ...", as seen in that
example at the bottom, are control messages. One could extend the
"...bugreport.cgi?mboxstat=yes..." URL by another attribute
"mboxcontrol=yes" to download only control messages. They can be
identified by a "X-Debbugs-Envelope-To: control" header in the
messages. But this would miss the requests embedded directly in
messages, as seen in
<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=37818#32>, for example.

Best regards, Michael.

bug#37854: 27.0.50; debbugs-gnu doesn't return date bugs were closed

[Lars Ingebrigtsen]

Michael Albinus <michael.albinus@gmx.de> writes:

> All messages "bug marked as fixed in version 27.1, ...", as seen in that
> example at the bottom, are control messages. One could extend the
> "...bugreport.cgi?mboxstat=yes..." URL by another attribute
> "mboxcontrol=yes" to download only control messages. They can be
> identified by a "X-Debbugs-Envelope-To: control" header in the
> messages. But this would miss the requests embedded directly in
> messages, as seen in
> <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=37818#32>, for example.

Ah, I see.  Then building accurate statistics require downloading all
those (about 21K) URLs.  I mean, that's totally doable, but...  is that
something that the debbugs.gnu.org infrastructure team would frown upon?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#37854: 27.0.50; debbugs-gnu doesn't return date bugs were closed

[Michael Albinus]

Lars Ingebrigtsen <larsi@gnus.org> writes:

Hi Lars,

> Ah, I see.  Then building accurate statistics require downloading all
> those (about 21K) URLs.  I mean, that's totally doable, but...  is that
> something that the debbugs.gnu.org infrastructure team would frown upon?

For sure. It would block the server, I fear.

One could think about creating a snapshot copy of the database, and use
the Debbugs Perl modules for further analysis on a separate machine. But
I don't know whether this is worth the effort.

Best regards, Michael.