bug#52477: 27.1; .crx and .crx3 are zip files

bug#52477: 27.1; .crx and .crx3 are zip files

[Trent W. Buck]

These are .zip files; "unzip -l" works on them:

    https://edgedl.me.gvt1.com/edgedl/chromewebstore/L2Nocm9tZV9leHRlbnNpb24vYmxvYnMvMjg0QUFYSnN4MFUtaEQwNDZqVGRkVkFmZw/1.0.6.0_aemomkdncapdnfajjbbcbdebjljbpmpj.crx
    https://edgedl.me.gvt1.com/edgedl/release2/chrome_component/acoof3hxrw6kobkazuyehcxdgxeq_2749/jflookgnkcckhobaglndicnbbgbonegd_2749_all_jujpn2jyxpu4ped7yujq342ypq.crx3

I want to be able to press RET on them in dired, and
see a zip view of their contents.

This already works for .jar and .ods, so I hoped it was just a matter of
adding to a list somewhere, but I cannot see where.

But when I tried just changing ".crx" to ".zip", I got an error:

    File mode specification error: (error Buffer format not recognized)

I guess this needs more magic than I can manage in a single cup of tea. :-(

bug#52477: 27.1; .crx and .crx3 are zip files

[Stefan Kangas]

"Trent W. Buck" <trentbuck@gmail.com> writes:

> These are .zip files; "unzip -l" works on them:
>
>     https://edgedl.me.gvt1.com/edgedl/chromewebstore/L2Nocm9tZV9leHRlbnNpb24vYmxvYnMvMjg0QUFYSnN4MFUtaEQwNDZqVGRkVkFmZw/1.0.6.0_aemomkdncapdnfajjbbcbdebjljbpmpj.crx
>     https://edgedl.me.gvt1.com/edgedl/release2/chrome_component/acoof3hxrw6kobkazuyehcxdgxeq_2749/jflookgnkcckhobaglndicnbbgbonegd_2749_all_jujpn2jyxpu4ped7yujq342ypq.crx3
>
> I want to be able to press RET on them in dired, and
> see a zip view of their contents.
>
> This already works for .jar and .ods, so I hoped it was just a matter of
> adding to a list somewhere, but I cannot see where.
>
> But when I tried just changing ".crx" to ".zip", I got an error:
>
>     File mode specification error: (error Buffer format not recognized)

Just to add some initial observations:

    $ file 1.0.6.0_aemomkdncapdnfajjbbcbdebjljbpmpj.crx
    1.0.6.0_aemomkdncapdnfajjbbcbdebjljbpmpj.crx: Google Chrome
extension, version 3

The first couple of bytes of that files is "Cr24", while a normal zip
file starts with "PK".

bug#52477: 27.1; .crx and .crx3 are zip files

[Michael Albinus]

Stefan Kangas <stefan@marxist.se> writes:

Hi,

> Just to add some initial observations:
>
>     $ file 1.0.6.0_aemomkdncapdnfajjbbcbdebjljbpmpj.crx
>     1.0.6.0_aemomkdncapdnfajjbbcbdebjljbpmpj.crx: Google Chrome
> extension, version 3
>
> The first couple of bytes of that files is "Cr24", while a normal zip
> file starts with "PK".

Yes, and that's why libarchive(3) doesn't handle it. Sadly, I cannot add
it to tramp-archive.el, therefore.

But there's no reason to prevent it from adding to arc-mode.el and/or
dired-aux.el.

Best regards, Michael.

bug#52477: 27.1; .crx and .crx3 are zip files

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Date: Thu, 16 Dec 2021 09:56:57 +0100
> Cc: "Trent W. Buck" <trentbuck@gmail.com>, 52477@debbugs.gnu.org
> 
> > The first couple of bytes of that files is "Cr24", while a normal zip
> > file starts with "PK".
> 
> Yes, and that's why libarchive(3) doesn't handle it. Sadly, I cannot add
> it to tramp-archive.el, therefore.
> 
> But there's no reason to prevent it from adding to arc-mode.el and/or
> dired-aux.el.

If the file doesn't follow the PKZip spec in the initial signature,
who knows what else in that spec doesn't it follow?  The ZIP spec has
a lot of special rules and formatting requirements, so if a file
doesn't follow them, how can we support it without knowing its exact
format specification?

bug#52477: 27.1; .crx and .crx3 are zip files

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

>> > The first couple of bytes of that files is "Cr24", while a normal zip
>> > file starts with "PK".
>>
>> Yes, and that's why libarchive(3) doesn't handle it. Sadly, I cannot add
>> it to tramp-archive.el, therefore.
>>
>> But there's no reason to prevent it from adding to arc-mode.el and/or
>> dired-aux.el.
>
> If the file doesn't follow the PKZip spec in the initial signature,
> who knows what else in that spec doesn't it follow?  The ZIP spec has
> a lot of special rules and formatting requirements, so if a file
> doesn't follow them, how can we support it without knowing its exact
> format specification?

".crx" files are Chrome Extensions, as shown by Stefan. They are
specified, see for example <https://developer.chrome.com/docs/extensions/reference/>.

Personally, I have no strong opinion whether we shall support them.

Best regards, Michael.

bug#52477: 27.1; .crx and .crx3 are zip files

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: stefan@marxist.se,  trentbuck@gmail.com,  52477@debbugs.gnu.org
> Date: Thu, 16 Dec 2021 12:59:01 +0100
> 
> > If the file doesn't follow the PKZip spec in the initial signature,
> > who knows what else in that spec doesn't it follow?  The ZIP spec has
> > a lot of special rules and formatting requirements, so if a file
> > doesn't follow them, how can we support it without knowing its exact
> > format specification?
> 
> ".crx" files are Chrome Extensions, as shown by Stefan. They are
> specified, see for example <https://developer.chrome.com/docs/extensions/reference/>.
> 
> Personally, I have no strong opinion whether we shall support them.

Me neither, but if someone wants to extend arc-mode.el to support
those extension, that would be welcome, I think.  My point is that we
cannot just add this file type before we implement those extensions.

bug#52477: 27.1; .crx and .crx3 are zip files

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Me neither, but if someone wants to extend arc-mode.el to support
> those extension, that would be welcome, I think.  My point is that we
> cannot just add this file type before we implement those extensions.

Yup.  arc-mode hard-code a lot of knowledge about zip files, including
the "PK" header and stuff.  It's possible that that can just be replaced
with Cr24, but it seems rather unlikely that Google would create a new
zip file format if that was the only difference, I'd have thought?

(But I have no knowledge here.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#52477: 27.1; .crx and .crx3 are zip files

[Trent W. Buck]

Eli Zaretskii wrote:
> > From: Michael Albinus <michael.albinus@gmx.de>
> > Cc: stefan@marxist.se,  trentbuck@gmail.com,  52477@debbugs.gnu.org
> > Date: Thu, 16 Dec 2021 12:59:01 +0100
> > 
> > > If the file doesn't follow the PKZip spec in the initial signature,
> > > who knows what else in that spec doesn't it follow?  The ZIP spec has
> > > a lot of special rules and formatting requirements, so if a file
> > > doesn't follow them, how can we support it without knowing its exact
> > > format specification?
> > 
> > ".crx" files are Chrome Extensions, as shown by Stefan. They are
> > specified, see for example <https://developer.chrome.com/docs/extensions/reference/>.
> > 
> > Personally, I have no strong opinion whether we shall support them.
> 
> Me neither, but if someone wants to extend arc-mode.el to support
> those extension, that would be welcome, I think.  My point is that we
> cannot just add this file type before we implement those extensions.

FTR, as the end user, I don't really care HOW it's implemented.

I also don't really care about editing, i.e. read-only support is fine.
I have a similar "eyeball the result" workflow for .deb using

    /usr/share/emacs/site-lisp/elpa-src/debian-el-37/deb-view.el

Another case of "it's a zip, but" is the .pyz, which
has an (optional) shebang before the PK:

    bash5$ mkdir MyCoolApp
    bash5$ touch MyCoolApp/__init__.py
    bash5$ echo 'print("Hello, World!")' >MyCoolApp/__main__.py
    bash5$ python3 -m zipapp --compress --python=/usr/bin/python3 MyCoolApp
    bash5$ ./MyCoolApp.pyz
    Hello, World!
    bash5$ head -n1 MyCoolApp.pyz
    #!/usr/bin/python3
    bash5$ file MyCoolApp.pyz
    MyCoolApp.pyz: Zip archive data, made by v2.0 UNIX, extract using at least v2.0, last modified Tue Jun 10 02:57:41 2014, uncompressed size 23, method=deflate

    bash5$ emacs MyCoolApp.pyz
    <Emacs does not understand it>

Someone mentioned libarchive, originally from BSD, which provides a
standard interface to EVERY(ish) archive and compression format.

Is it reasonable for emacs to just chuck out arc-mode (and tar-mode
and jka-compr) and instead just use libarchive?
It's already used by vlc and a lot of GNOME stuff.

libarchive already understands both .pyz and .crx3 files, at least for reading:

    bash5$ bsdtar -vtf MyCoolApp.pyz
    -rw-r--r--  0 0      0          23 Dec 22 14:33 __main__.py
    -rw-r--r--  0 0      0           0 Dec 22 14:33 __init__.py

    bash5$ bsdtar -vtf jamhcnnkihinmdlkakkaopbjbbcngflc_99.0.4764.2_all_nfh7crk5nxg3omjxfwx5cuhrz4.crx3
    -rw-rw-r--  0 0      0        3484 Dec 13 12:34 hyph-und-ethi.hyb
    -rw-rw-r--  0 0      0        2712 Dec 13 12:34 hyph-tk.hyb
    -rw-rw-r--  0 0      0         703 Dec 13 12:34 hyph-te.hyb
    -rw-rw-r--  0 0      0         554 Dec 13 12:34 hyph-ta.hyb
    -rw-rw-r--  0 0      0        6631 Dec 13 12:34 hyph-sl.hyb
    -rw-rw-r--  0 0      0        1414 Dec 13 12:34 hyph-pt.hyb
    -rw-rw-r--  0 0      0         607 Dec 13 12:34 hyph-pa.hyb
    -rw-rw-r--  0 0      0         647 Dec 13 12:34 hyph-or.hyb
    -rw-rw-r--  0 0      0      145263 Dec 13 12:34 hyph-nn.hyb
    -rw-rw-r--  0 0      0      145263 Dec 13 12:34 hyph-nb.hyb
    -rw-rw-r--  0 0      0         687 Dec 13 12:34 hyph-mr.hyb
    -rw-rw-r--  0 0      0        5142 Dec 13 12:34 hyph-mn-cyrl.hyb
    -rw-rw-r--  0 0      0         776 Dec 13 12:34 hyph-ml.hyb
    -rw-rw-r--  0 0      0        1839 Dec 13 12:34 hyph-la.hyb
    -rw-rw-r--  0 0      0         711 Dec 13 12:34 hyph-kn.hyb
    -rw-rw-r--  0 0      0         605 Dec 13 12:34 hyph-hy.hyb
    -rw-rw-r--  0 0      0      317251 Dec 13 12:34 hyph-hu.hyb
    -rw-rw-r--  0 0      0        3031 Dec 13 12:34 hyph-hr.hyb
    -rw-rw-r--  0 0      0         687 Dec 13 12:34 hyph-hi.hyb
    -rw-rw-r--  0 0      0         655 Dec 13 12:34 hyph-gu.hyb
    -rw-rw-r--  0 0      0       35824 Dec 13 12:34 hyph-ga.hyb
    -rw-rw-r--  0 0      0        8165 Dec 13 12:34 hyph-fr.hyb
    -rw-rw-r--  0 0      0         665 Dec 13 12:34 hyph-eu.hyb
    -rw-rw-r--  0 0      0       21421 Dec 13 12:34 hyph-et.hyb
    -rw-rw-r--  0 0      0       14995 Dec 13 12:34 hyph-es.hyb
    -rw-rw-r--  0 0      0       59802 Dec 13 12:34 hyph-en-us.hyb
    -rw-rw-r--  0 0      0       46607 Dec 13 12:34 hyph-en-gb.hyb
    -rw-rw-r--  0 0      0      120218 Dec 13 12:34 hyph-de-ch-1901.hyb
    -rw-rw-r--  0 0      0      120412 Dec 13 12:34 hyph-de-1996.hyb
    -rw-rw-r--  0 0      0      121393 Dec 13 12:34 hyph-de-1901.hyb
    -rw-rw-r--  0 0      0        6967 Dec 13 12:34 hyph-da.hyb
    -rw-rw-r--  0 0      0       35913 Dec 13 12:34 hyph-cy.hyb
    -rw-rw-r--  0 0      0       52842 Dec 13 12:34 hyph-cu.hyb
    -rw-rw-r--  0 0      0         703 Dec 13 12:34 hyph-bn.hyb
    -rw-rw-r--  0 0      0        3467 Dec 13 12:34 hyph-bg.hyb
    -rw-rw-r--  0 0      0        6098 Dec 13 12:34 hyph-be.hyb
    -rw-rw-r--  0 0      0         703 Dec 13 12:34 hyph-as.hyb
    -rw-r--r--  0 0      0         178 Dec 13 12:34 manifest.json
    drwxrwxr-x  0 0      0           0 Dec 13 12:34 _metadata/
    -rw-rw-r--  0 0      0        5698 Dec 13 12:34 _metadata/verified_contents.json

bug#52477: 27.1; .crx and .crx3 are zip files

[Michael Albinus]

"Trent W. Buck" <trentbuck@gmail.com> writes:

Hi Trent,

> FTR, as the end user, I don't really care HOW it's implemented.
>
> I also don't really care about editing, i.e. read-only support is fine.
> I have a similar "eyeball the result" workflow for .deb using
>
>     /usr/share/emacs/site-lisp/elpa-src/debian-el-37/deb-view.el

.deb files are handled by libarchive(3), and so they are supported in tramp-archive.el.

> Someone mentioned libarchive, originally from BSD, which provides a
> standard interface to EVERY(ish) archive and compression format.
>
> Is it reasonable for emacs to just chuck out arc-mode (and tar-mode
> and jka-compr) and instead just use libarchive?
> It's already used by vlc and a lot of GNOME stuff.
>
> libarchive already understands both .pyz and .crx3 files, at least for reading:
>
>     bash5$ bsdtar -vtf MyCoolApp.pyz
>     -rw-r--r--  0 0      0          23 Dec 22 14:33 __main__.py
>
>     bash5$ bsdtar -vtf jamhcnnkihinmdlkakkaopbjbbcngflc_99.0.4764.2_all_nfh7crk5nxg3omjxfwx5cuhrz4.crx3
>     -rw-rw-r--  0 0      0        3484 Dec 13 12:34 hyph-und-ethi.hyb

No. Neither .pyz nor .crx3? formats are supported by libarchive, I've just
tested it with Nautilus, the GNOME file manager. bsdtar uses several
archive libraries, I guess another archive library must be responsible
for these formats.

However, I'm in favor to add native libarchive support to Emacs. The
current integration via tramp-archive has its limitations:

- It is based on the GVFS archive backend and D-Bus integration, in
  practice it means it is available for GNU/Linux only.

- The GVFS guys did not implement writing into archives, which is
  supported by native libarchive.

But as said above, this wouldn't mean that *all* archive formats are
supported.

Best regards, Michael.

bug#52477: 27.1; .crx and .crx3 are zip files

[Trent W. Buck]

Michael Albinus wrote:
> "Trent W. Buck" <trentbuck@gmail.com> writes:
> > Is it reasonable for emacs to just chuck out arc-mode (and tar-mode
> > and jka-compr) and instead just use libarchive?
> > It's already used by vlc and a lot of GNOME stuff.
> >
> > libarchive already understands both .pyz and .crx3 files, at least for reading: …
>
> No. Neither .pyz nor .crx3? formats are supported by libarchive, I've just
> tested it with Nautilus, the GNOME file manager. bsdtar uses several
> archive libraries, I guess another archive library must be responsible
> for these formats.

Ah, sorry for the confusion.

On my system (Debian 11) bsdtar is provided by "Source Package: libarchive" and
only depends on libarchive13 and libc6, so
I assumed it was a reasonable test of "what can libarchive do?".

If it is using another library, I cannot see how:

    bash5$ ldd /usr/bin/bsdtar
            linux-vdso.so.1 (0x00007ffc6b1b3000)
            libarchive.so.13 => /lib/x86_64-linux-gnu/libarchive.so.13 (0x00007fdf3c997000)
            libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fdf3c7d2000)
            libnettle.so.8 => /lib/x86_64-linux-gnu/libnettle.so.8 (0x00007fdf3c78a000)
            libacl.so.1 => /lib/x86_64-linux-gnu/libacl.so.1 (0x00007fdf3c77f000)
            liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fdf3c757000)
            libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x00007fdf3c67c000)
            liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1 (0x00007fdf3c657000)
            libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007fdf3c644000)
            libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fdf3c627000)
            libxml2.so.2 => /lib/x86_64-linux-gnu/libxml2.so.2 (0x00007fdf3c479000)
            /lib64/ld-linux-x86-64.so.2 (0x00007fdf3ca8f000)
            libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fdf3c457000)
            libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fdf3c44f000)
            libicuuc.so.67 => /lib/x86_64-linux-gnu/libicuuc.so.67 (0x00007fdf3c266000)
            libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fdf3c122000)
            libicudata.so.67 => /lib/x86_64-linux-gnu/libicudata.so.67 (0x00007fdf3a609000)
            libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fdf3a43c000)
            libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fdf3a422000)

    bash5$ strace -f /usr/bin/bsdtar tf *.pyz |& grep -F -e exec -e fork
    execve("/usr/bin/bsdtar", ["/usr/bin/bsdtar", "tf", "MyCoolApp.pyz"], 0x7ffc169ca188 /* 70 vars */) = 0

Nor can I see embedded copies of libraries in
https://salsa.debian.org/debian/libarchive/-/tree/master/tar

My C-fu is not up to scratch, but Python has a thin FFI interface to libarchive, and
that also works fine:

    bash5$ python3
    Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
    [GCC 10.2.1 20210110] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import libarchive
    >>> with libarchive.file_reader('MyCoolApp.pyz') as z:
    ...   for i in z:
    ...     print(i.strmode, i.size, i.name, sep='\t')
    ... 
    b'-rw-r--r--'	23	__main__.py
    b'-rw-r--r--'	0	__init__.py

I had a quick look, but I can't immediately tell why nautilus would not work, as
I cannot even find where nautilus uses libarchive.

When I tested nautilus 3.38.2-1+deb11u1, it would not even open a .zip.
It only offered to run file-roller.

I found file-roller includes nautilus/nautilus-fileroller.c
which uses a hard-coded archive_mime_types[] that it will activate for.
application/x-chrome-extension (.crx3) is not in that list.

There is no MIME type for .pyz because it's just a .zip with an
optional shebang prepended ("#![^\n]*\n").
Nautilus and file-roller treat a .pyz WITHOUT a shebang
identically to a .zip.
Presumably this is due to libmagic-like MIME guessing.


It seems to me that:

  1. when given a file in pyz/crx/crx format,
     libarchive will auto-detect and use zip format.

     I think it just finds the zip archive at an offset, and
     leaves content before that offset alone.

     This would allow it to read/extract/edit existing pyz/crx/crx3,
     but not create a new one.

     This is also true for libarchive bsdtar and Info-ZIP unzip,
     although the latter emits a warning about the offset.

  2. when given a file in pyz/crx/crx format,
     emacs arc-mode will not auto-detect zip format.

  3. when given a file in pyz/crx/crx3 format,
     file-roller & nautilus will not auto-detect any format.

bug#52477: 27.1; .crx and .crx3 are zip files

[Michael Albinus]

"Trent W. Buck" <trentbuck@gmail.com> writes:

Hi Trent,

> It seems to me that:
>
>   1. when given a file in pyz/crx/crx format,
>      libarchive will auto-detect and use zip format.
>
>      I think it just finds the zip archive at an offset, and
>      leaves content before that offset alone.
>
>      This would allow it to read/extract/edit existing pyz/crx/crx3,
>      but not create a new one.
>
>      This is also true for libarchive bsdtar and Info-ZIP unzip,
>      although the latter emits a warning about the offset.
>
>   2. when given a file in pyz/crx/crx format,
>      emacs arc-mode will not auto-detect zip format.
>
>   3. when given a file in pyz/crx/crx3 format,
>      file-roller & nautilus will not auto-detect any format.

Well, I don't know how in detail nautilus/file-roller have integrated
libarchive. I believe they use the GVFS mount backend which calls
libarchive, but I don't know it for sure. So we agree that a native
libarchive integration in Emacs is preferred.

Best regards, Michael.

bug#52477: 27.1; .crx and .crx3 are zip files

[Eli Zaretskii]

> Date: Wed, 22 Dec 2021 14:50:43 +1100
> From: "Trent W. Buck" <trentbuck@gmail.com>
> Cc: Michael Albinus <michael.albinus@gmx.de>, stefan@marxist.se,
> 	52477@debbugs.gnu.org
> 
> Someone mentioned libarchive, originally from BSD, which provides a
> standard interface to EVERY(ish) archive and compression format.
> 
> Is it reasonable for emacs to just chuck out arc-mode (and tar-mode
> and jka-compr) and instead just use libarchive?

It's reasonable, but Someone™ should write the code to do it.