arc-mode.el bug or malformed ZIP file?

arc-mode.el bug or malformed ZIP file?

[Stephen Berman]

I have a ZIP file that, when I try to visit it in archive-mode, Emacs
throws a file mode specification error on, specifically
args-out-of-range.  The reason is that the function archive-l-e in
arc-mode.el, which is supposed to return a buffer position in this case,
returns a number which exceeds the buffer size.  And the reason for this
is that archive-l-e builds its return value by iteratively calling (+
(ash result 8) (aref str (- len i))), where `str' is a string extracted
from the end of the ZIP file, which in this case is "\377\377\377\377",
so that the return value increases from 255 (the decimal value of octal
377) to 4294967040 after four iterations, which far exceeds the file
(and hence buffer) size.

Perhaps "\377\377\377\377" is an invalid string at the end of a ZIP file
(I checked two other ZIP files I have, which I can visit in archive-mode
with no problem, and they have different strings at the end, consisting
of bytes with smaller decimal values, so that the return value of
archive-l-e does not exceed the file size).  Yet when I call `unzip -l'
on the file in the shell, the contents are displayed, and I also had no
problem unpacking the file with unzip (and AFAICT the content is
undamaged).  So at least unzip can deal with this file.  So is this a
bug in arc-mode.el?

Steve Berman

Re: arc-mode.el bug or malformed ZIP file?

[Stefan Monnier]

> undamaged).  So at least unzip can deal with this file.  So is this a
> bug in arc-mode.el?

Probably.  Could be a new file format, for instance (e.g. maybe the old
format understood by arc-mode.el is limited to 2GB sizes or something
like that?).


        Stefan

Re: arc-mode.el bug or malformed ZIP file?

[Stephen Berman]

On Wed, 23 Sep 2020 22:05:55 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote:

>> undamaged).  So at least unzip can deal with this file.  So is this a
>> bug in arc-mode.el?
>
> Probably.  Could be a new file format, for instance (e.g. maybe the old
> format understood by arc-mode.el is limited to 2GB sizes or something
> like that?).

The size can't be the reason for the failure: the ZIP file whose
contents are not displayed in archive-mode is about 95 MB, but I have
some much larger ZIP file whose contents are displayed in archive-mode
with no problem.  But maybe there are more recent format changes for
other reasons.  I'm using a patched version of unzip 6.0, which is the
most recent from 2009, but the patch is from 2019 (though I don't see
anything in it about format changes).  I did search the web briefly but
found nothing about valid byte sequences at the end of the file.  But
I'll go ahead and file a bug report, maybe someone who knows more about
the ZIP format will see how to fix or extend arc-mode.el.

Steve Berman

Re: arc-mode.el bug or malformed ZIP file?

[Stefan Monnier]

>>> undamaged).  So at least unzip can deal with this file.  So is this a
>>> bug in arc-mode.el?
>> Probably.  Could be a new file format, for instance (e.g. maybe the old
>> format understood by arc-mode.el is limited to 2GB sizes or something
>> like that?).
> The size can't be the reason for the failure: the ZIP file whose

I'm not saying it's because of the size, but because of a change of
format (and this new format was introduced because of size limits in
the old format).

Eli just installed a patch a few days ago which seems to confirm my
suspicion:

    commit f31c6792ab98c8be343838f59d96e35c05353521 (emacs-27)
    Author: Eli Zaretskii <eliz@gnu.org>
    Date:   Fri Sep 25 17:11:11 2020 +0300
    
        Fix support for Zip64 zip files
        
        * lisp/arc-mode.el (archive-zip-summarize): Fix detection of Zip64
        central directory.  Support 64-bit file size field used by Zip64.
        (Bug#43597)


-- Stefan

Re: arc-mode.el bug or malformed ZIP file?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 28 Sep 2020 13:08:31 -0400
> Cc: emacs-devel@gnu.org
> 
> > The size can't be the reason for the failure: the ZIP file whose
> 
> I'm not saying it's because of the size, but because of a change of
> format (and this new format was introduced because of size limits in
> the old format).
> 
> Eli just installed a patch a few days ago which seems to confirm my
> suspicion:

Yes, the code we had supported 64-bit offset of central directory, but
didn't support 64-bit file size.