bug#63831: fix bug with failed inflation of .el.gz files in src/decompress.c

bug#63831: fix bug with failed inflation of .el.gz files in src/decompress.c

[Amritpal Singh]

[-- Attachment #1: Type: text/plain, Size: 1923 bytes --]

Bug Report:
Compile emacs with system's gzip program set to `pigz`.
Run emacs and then `M-x eww RET`

Expected behavior:
Enter URL prompt in mini-buffer

Actual behavior:
hashing failed '/usr/share/emacs/30.0.50/lisp/gnus/gnus.el.gz'

Report:
The bug has been reproduced on emacs version 29.0.91 and HEAD which
seems to be at 30.0.50.
Later, a copy of the aforementioned file was saved somewhere else and
the program was uninstalled. Then emacs was recompiled with system's
gzip program set to GNU gzip and the initial steps were repeated and
the expected behavior was the result.
This lead to believing either that there's a bug with how zlib's
`inflate()` handles archives or emacs code was having an issue with
archives files.

The hashes for gz archives generated with different programs were as follows
> md5sum gnus-gzip.el.gz
edb3d0ffba7f19ff1d4ec3f889609e8a  gnus-gzip.el.gz
> md5sum gnus.el.gz
985deaaec6a5845ac8d6bd9648957b50  gnus.el.gz

And when uncompressing these archives, the resulting file was the same
and the hash for the files was the same (omitted for brevity).

Now after logging some code in $EMACS_REPO/src/decompress.c, it was
learned that in the pigz specific case, `inflate()` was returning
Z_BUF_ERROR(-5) which is an indicator for zstream's either `avail_in`
or `avail_out` fields are 0.

Observing the code in `$EMACS_REPO/src/decompress.c`
L154:
    } while (!stream.avail_out);
only checks stream.avail_out and not stream.avail_in which also might
have been set to 0. A special case here can be constructed where
`avail_in` is 0, and the code keeps looping even though our input
buffer is empty and thus causing a Z_BUF_ERROR. Placing a simple check
for it fixes the bug in pigz's gz archives case and does not cause any
issue with gzip archives.

A simple patch with a fix is attached below, I would also like to
thank a friend of mine cortexauth whom also helped during my debug
sessions.

[-- Attachment #2: 0001-check-stream.avail_in-as-well-when-looping-to-inflat.patch --]
[-- Type: text/x-diff, Size: 734 bytes --]

From ffa8e140ed8b093c60f1238bf76935e815e82a21 Mon Sep 17 00:00:00 2001
From: icebarf <sysgrammer@protonmail.com>
Date: Fri, 2 Jun 2023 10:51:21 +0530
Subject: [PATCH] check stream.avail_in as well when looping to inflate gz
 archive

---
 src/decompress.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/decompress.c b/src/decompress.c
index 6ef17db..162f616 100644
--- a/src/decompress.c
+++ b/src/decompress.c
@@ -151,7 +151,7 @@ md5_gz_stream (FILE *source, void *resblock)
 	return -1;
 
       accumulate_and_process_md5 (out, MD5_BLOCKSIZE - stream.avail_out, &ctx);
-    } while (!stream.avail_out);
+    } while (stream.avail_in && !stream.avail_out);
 
   } while (res != Z_STREAM_END);
 
-- 
2.40.1

bug#63831: Accidentally submit two bug reports for the same topic

[Amritpal Singh]

I'd like to apologise for submitting two bug reports, as I'd sent
another after ~1.5 hour of submitting this bug report. I hadn't
received any acknowledgement from the automatic bug tracking system
and thought I'd submitted the report in a wrong method.

bug#63831: Accidentally submit two bug reports for the same topic

[Eli Zaretskii]

> From: Amritpal Singh <icy.amrit@gmail.com>
> Date: Fri, 2 Jun 2023 13:35:38 +0530
> 
> I'd like to apologise for submitting two bug reports, as I'd sent
> another after ~1.5 hour of submitting this bug report. I hadn't
> received any acknowledgement from the automatic bug tracking system
> and thought I'd submitted the report in a wrong method.

Please be more patient.  Each bug report is approved by a human, to
avoid the catastrophic consequences of admitting spam to the bug list.
So it might take some time, especially if the person who approves the
bug reports and you live in different time zones.

bug#63831: Accidentally submit two bug reports for the same topic

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Please be more patient.  Each bug report is approved by a human, to
  > avoid the catastrophic consequences of admitting spam to the bug list.
  > So it might take some time, especially if the person who approves the
  > bug reports and you live in different time zones.

There are good reasons for this, but can we do something to help
bug submitters understand the situation?

For instance, we could set up an immeidate auto-reply to inform them
of the delay to expect.

If we had two people to do the moderation, maybe they could work
at different times of day -- that would shorten the usual wait time
without incresaing the work.



-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#63831: Accidentally submit two bug reports for the same topic

[Amritpal Singh]

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

>   > Please be more patient.  Each bug report is approved by a human, to
>   > avoid the catastrophic consequences of admitting spam to the bug list.
>   > So it might take some time, especially if the person who approves the
>   > bug reports and you live in different time zones.

> There are good reasons for this, but can we do something to help
> bug submitters understand the situation?

> For instance, we could set up an immeidate auto-reply to inform them
> of the delay to expect.

> If we had two people to do the moderation, maybe they could work
> at different times of day -- that would shorten the usual wait time
> without incresaing the work.

While we're on the topic of bug report moderation, is it possible that the
current human moderation might be a little ineffective? We've observed
that me accidentally submitting a duplicate report made it through human
filters, and I had notice another instance of a "spam" email, where no
subject or body was present.
See: https://lists.gnu.org/archive/html/bug-gnu-emacs/2023-06/msg00107.html

I'm not trying to impugn someone else's ability of moderating but the current
system seems a little ineffective. Also I'd like to apologize if my
language/wording
seems abrasive towards the moderation team as this is no way an insult to their
work but rather a lack of implementation of better systems by management.

I'd also like to draw attention towards the actual bug+patch report
#63831, it's been
close to two days now and such a small change could easily be merged into HEAD.
But the bug does seem low priority (even though breaking) to me as well so
I would not question the maintainers' decision to prioritize critical
bugs over this.

I wish you a good day/night and, again would like to apoligise for any
rude language.

Regards,
Amritpal Singh

bug#63831: Accidentally submit two bug reports for the same topic

[Eli Zaretskii]

> From: Amritpal Singh <icy.amrit@gmail.com>
> Date: Sun, 4 Jun 2023 09:16:49 +0530
> Cc: "eliz@gnu.org" <eliz@gnu.org>, 63831@debbugs.gnu.org
> 
> While we're on the topic of bug report moderation, is it possible that the
> current human moderation might be a little ineffective?

Feel free to volunteer to help, in order to improve the quality and
the timeliness of the moderation.

> We've observed that me accidentally submitting a duplicate report
> made it through human filters, and I had notice another instance of
> a "spam" email, where no subject or body was present.  See:
> https://lists.gnu.org/archive/html/bug-gnu-emacs/2023-06/msg00107.html

It isn't spam.  The sender is an Emacs contributor.

> I'm not trying to impugn someone else's ability of moderating but the current
> system seems a little ineffective. Also I'd like to apologize if my
> language/wording
> seems abrasive towards the moderation team as this is no way an insult to their
> work but rather a lack of implementation of better systems by management.

I'm not sure I understand what you are suggesting here.  What better
system did you have in mind?

(In any case, discussing this as part of a specific bug report is
wrong; this should be taken to emacs-devel instead.)

> I'd also like to draw attention towards the actual bug+patch report
> #63831, it's been
> close to two days now and such a small change could easily be merged into HEAD.
> But the bug does seem low priority (even though breaking) to me as well so
> I would not question the maintainers' decision to prioritize critical
> bugs over this.

That bug is indeed low priority, and the change will go to master,
which makes it even lower priority (since we are in the process of
pretesting Emacs 29.1).  The bug report is in my queue, and two days
is really way too short to ring the alarm bells about such a minor
issue.

bug#63831: Accidentally submit two bug reports for the same topic

[Amritpal Singh]

> Feel free to volunteer to help, in order to improve the quality and
> the timeliness of the moderation.
Thank you! Please let me know how I can contribute to this.

> It isn't spam.  The sender is an Emacs contributor.
Apologies, I'm new to emacs mailing lists and unfamiliar with the
contributors, and I considered the empty email spam.

> I'm not sure I understand what you are suggesting here.  What better
> system did you have in mind?
>
> (In any case, discussing this as part of a specific bug report is
> wrong; this should be taken to emacs-devel instead.)
Like RMS suggested, perhaps an automated immediate response
system for reports would indeed be a good choice to indicate to
contributors that their report indeed has been received.

I agree with the latter part about moving to emacs-devel.

> That bug is indeed low priority, and the change will go to master,
> which makes it even lower priority (since we are in the process of
> pretesting Emacs 29.1).  The bug report is in my queue, and two days
> is really way too short to ring the alarm bells about such a minor
> issue.

I'm sorry for that as well, like I mentioned before I'm a newbie contributor and
do not have the know-how with how contributors and maintainers interact.
This is no excuse for being impatient but I thought I'd explain myself.

Thank you!

bug#63831: fix bug with failed inflation of .el.gz files in src/decompress.c

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi,

>> We've observed that me accidentally submitting a duplicate report
>> made it through human filters,

FTR, there are several moderators, so you cannot expect that both
reports have been seen by the same moderator (honestly, I don't
remember the case)

Furthermore, we don't decide on the contents. If something looks like a
valid bug report we approve, whatever it is about. No further checks on
duplicates or alike, that would exceed our limited resources.

>> I'm not trying to impugn someone else's ability of moderating but the current
>> system seems a little ineffective.

> I'm not sure I understand what you are suggesting here.  What better
> system did you have in mind?

I don't feel offended :-)

Anyway, what we do is manual, human work. Do you recommend to hand over
moderation to ChatGPT? :-)

Best regards, Michael.

bug#63831: Accidentally submit two bug reports for the same topic

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: icy.amrit@gmail.com, 63831@debbugs.gnu.org
> Date: Sat, 03 Jun 2023 17:27:27 -0400
> 
>   > Please be more patient.  Each bug report is approved by a human, to
>   > avoid the catastrophic consequences of admitting spam to the bug list.
>   > So it might take some time, especially if the person who approves the
>   > bug reports and you live in different time zones.
> 
> There are good reasons for this, but can we do something to help
> bug submitters understand the situation?
> 
> For instance, we could set up an immeidate auto-reply to inform them
> of the delay to expect.
> 
> If we had two people to do the moderation, maybe they could work
> at different times of day -- that would shorten the usual wait time
> without incresaing the work.

Michael, any comments?

bug#63831: Accidentally submit two bug reports for the same topic

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli & Richard,

>>   > Please be more patient.  Each bug report is approved by a human, to
>>   > avoid the catastrophic consequences of admitting spam to the bug list.
>>   > So it might take some time, especially if the person who approves the
>>   > bug reports and you live in different time zones.
>>
>> There are good reasons for this, but can we do something to help
>> bug submitters understand the situation?
>>
>> For instance, we could set up an immeidate auto-reply to inform them
>> of the delay to expect.

No. This would send the information to spam factories that the given
email address on debbugs.gnu.org is valid. Meaning increasing the
amount of spam.

>> If we had two people to do the moderation, maybe they could work
>> at different times of day -- that would shorten the usual wait time
>> without incresaing the work.

AFAIK, Bob Proulx and me are moderating. Bob lives in the USA
(Colorado?), I live in Germany, so there are different time zones.

Usually, I check debbugs for hold messages several times a day. A delay
of 1.5 hours, as reported here, is completely normal. We cannot change
it unless we could recruit more moderators.

Best regrds, Michael.

bug#63831: Accidentally submit two bug reports for the same topic

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > No. This would send the information to spam factories that the given
  > email address on debbugs.gnu.org is valid. Meaning increasing the
  > amount of spam.

Is the idea that, under the current system, spam never gets any response?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#63831: Accidentally submit two bug reports for the same topic

[Michael Albinus]

Richard Stallman <rms@gnu.org> writes:

Hi Richard,

>   > No. This would send the information to spam factories that the given
>   > email address on debbugs.gnu.org is valid. Meaning increasing the
>   > amount of spam.
>
> Is the idea that, under the current system, spam never gets any response?

As moderator, I can decide to "Reject" or "Discard" a message. Usually I
decide to discard, which means "no response", yes. The Help describes
the options as

--8<---------------cut here---------------start------------->8---
Reject -- Reject the message, sending a rejection notice to the sender,
and discarding the original message. For membership requests, reject the
change in membership status. In either case, you should add a reason for
the rejection in the accompanying text box.

Discard -- Throw away the original message, without sending a rejection
notice. For membership requests, this simply discards the request
without notice to the person making the request. This is usually the
action you want to take for known spam.
--8<---------------cut here---------------end--------------->8---

Best regards, Michael.

bug#63831: [PATCH] Elaboration on the bugfix for failed inflation of gz archives

[Amritpal Singh]

> Now after logging some code in $EMACS_REPO/src/decompress.c, it was
> learned that in the pigz specific case, `inflate()` was returning
> Z_BUF_ERROR(-5) which is an indicator for zstream's either `avail_in`
> or `avail_out` fields are 0.
>
> Observing the code in `$EMACS_REPO/src/decompress.c`
> L154:
>     } while (!stream.avail_out);
> only checks stream.avail_out and not stream.avail_in which also might
> have been set to 0. A special case here can be constructed where
> `avail_in` is 0, and the code keeps looping even though our input
> buffer is empty and thus causing a Z_BUF_ERROR. Placing a simple check
> for it fixes the bug in pigz's gz archives case and does not cause any
> issue with gzip archives.

I'd like to elaborate a bit further on this part mentioned in the bug
report-cum-patch email. The code in question where the bug occurs is
as follows:

> do {
>    stream.avail_in = fread (in, 1, MD5_BLOCKSIZE, source);
>    if (ferror (source)) {
>      inflateEnd (&stream);
>      return -1;
>    }
>    if (stream.avail_in == 0)
>      break;
>    stream.next_in = in;
>
>    do {
>      stream.avail_out = MD5_BLOCKSIZE;
>      stream.next_out = out;
>      res = inflate (&stream, Z_NO_FLUSH);
>
>      if (res != Z_OK && res != Z_STREAM_END)
>    return -1;
>
>      accumulate_and_process_md5 (out, MD5_BLOCKSIZE - stream.avail_out, &ctx);
L154:
>    } while (!stream.avail_out);
L155:
>
>  } while (res != Z_STREAM_END);


From here, it is a bit more clear what the originally mentioned `L154`
was referring to. The archives compressed using pigz set as the systems'
gzip program either via symlink as `gzip` in $PATH or using $GZIP_PROG,
fail to hash in this specific instance which is quite strange,
nonetheless this patch (provided in the original bug report) fixes the
issue of calling inflate() repeatedly (in the inner do-while() loop)
when we do not have any more input to process.

This happens so, as mentioned before, because `infalte()` can set
`stream.avail_in` to 0 which is the available number of bytes to
process. And since we're only checking if stream.avail_out hasn't been
set to 0 in the inner loop, the hashing fails even though our archive is
correctly made and decompresses successfully externally.

ps. The patch is in the original bug report.