bug#37854: 27.0.50; debbugs-gnu doesn't return date bugs were closed

bug#37854: 27.0.50; debbugs-gnu doesn't return date bugs were closed

[Lars Ingebrigtsen]

Apparently somewhere in the database, debbugs knows the date a bug was
closed:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=18688

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=18691

Unless those pages are statically generated and just appended to, as in
a log?  I assumed they were generated on-the-fly from the data in the
database.

There's two data points that would be interesting: The "closed" date, and
the "fixed" date, which may be different.

The problem is that bugs may be updated after they're closed (for
instance when they are archived, but also by other stuff), which makes
it difficult to do accurate statistics about response time and the
like.  I mean, you can do statistics that are probably "good enough",
but it's kinda not very satisfying when you know there's methodological
errors there.


In GNU Emacs 27.0.50 (build 91, x86_64-pc-linux-gnu, GTK+ Version 3.24.5)
 of 2019-10-20 built on marnie
Repository revision: 78cb3791fa11c95756ee3917c63cfea774f128a2
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12004000
System Description: Debian GNU/Linux 10 (buster)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#37854: 27.0.50; debbugs-gnu doesn't return date bugs were closed

[Michael Albinus]

Lars Ingebrigtsen <larsi@gnus.org> writes:

[continuing at <37854@debbugs.gnu.org>]

> But I wonder...  fixed_date is one thing, but you can close a bug
> without marking it as fixed.  Is there a separate date for the "close"
> action somewhere in the database?
>
> For instance,
>
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=18688
>
> was closed without being marked as "fixed".

Well, it looks like this close date is not an attribute of the bug, but
something the database keeps internal for countdown of the 28 days,
until the bug will be archived. Once archived, this information doesn't
seem to be available anymore.

Anyway, I've sent a request to <https://bugs.debian.org/942843>.

Best regards, Michael.

bug#37854: 27.0.50; debbugs-gnu doesn't return date bugs were closed

[Lars Ingebrigtsen]

Michael Albinus <michael.albinus@gmx.de> writes:

> Well, it looks like this close date is not an attribute of the bug, but
> something the database keeps internal for countdown of the 28 days,
> until the bug will be archived. Once archived, this information doesn't
> seem to be available anymore.

Right.

I was also wondering why the charts I made estimate the number of tags
(like "moreinfo"/"patch") too high, but of course I only have what the
data look like now.

So while my charts say that there were 200 open "moreinfo" in, say,
December 2018, there weren't that many.  A bug report that got that tag
in later, and was closed later, will be counted as having had that tag
back then.

Would it be possible to include the entire change history in the SOAP
data?  That is, the timestamps when tags/severities are added/removed?
That is, is there a "log" of changes applied to a bug that can be
output?  If there was, that'd also fix the "closed" date handling
problem.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#37854: 27.0.50; debbugs-gnu doesn't return date bugs were closed

[Michael Albinus]

Lars Ingebrigtsen <larsi@gnus.org> writes:

Hi Lars,

> Would it be possible to include the entire change history in the SOAP
> data?  That is, the timestamps when tags/severities are added/removed?
> That is, is there a "log" of changes applied to a bug that can be
> output?  If there was, that'd also fix the "closed" date handling
> problem.

Such history cannot be retrieved from the database I fear. So it cannot
be provided by the SOAP interface.

What you see in the bug reports logs web pages are all the messages,
received for a given bug. They are not downloaded by SOAP, but by their
URL like <https://debbugs.gnu.org/cgi/bugreport.cgi?mboxstat=yes;bug=37818>.
This is the counterpart to what is shown via
<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=37818>.

All messages "bug marked as fixed in version 27.1, ...", as seen in that
example at the bottom, are control messages. One could extend the
"...bugreport.cgi?mboxstat=yes..." URL by another attribute
"mboxcontrol=yes" to download only control messages. They can be
identified by a "X-Debbugs-Envelope-To: control" header in the
messages. But this would miss the requests embedded directly in
messages, as seen in
<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=37818#32>, for example.

Best regards, Michael.

bug#37854: 27.0.50; debbugs-gnu doesn't return date bugs were closed

[Lars Ingebrigtsen]

Michael Albinus <michael.albinus@gmx.de> writes:

> All messages "bug marked as fixed in version 27.1, ...", as seen in that
> example at the bottom, are control messages. One could extend the
> "...bugreport.cgi?mboxstat=yes..." URL by another attribute
> "mboxcontrol=yes" to download only control messages. They can be
> identified by a "X-Debbugs-Envelope-To: control" header in the
> messages. But this would miss the requests embedded directly in
> messages, as seen in
> <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=37818#32>, for example.

Ah, I see.  Then building accurate statistics require downloading all
those (about 21K) URLs.  I mean, that's totally doable, but...  is that
something that the debbugs.gnu.org infrastructure team would frown upon?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#37854: 27.0.50; debbugs-gnu doesn't return date bugs were closed

[Michael Albinus]

Lars Ingebrigtsen <larsi@gnus.org> writes:

Hi Lars,

> Ah, I see.  Then building accurate statistics require downloading all
> those (about 21K) URLs.  I mean, that's totally doable, but...  is that
> something that the debbugs.gnu.org infrastructure team would frown upon?

For sure. It would block the server, I fear.

One could think about creating a snapshot copy of the database, and use
the Debbugs Perl modules for further analysis on a separate machine. But
I don't know whether this is worth the effort.

Best regards, Michael.