bug#19392: Emacs searches for dabbrevs in archive buffers

bug#19392: Emacs searches for dabbrevs in archive buffers

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 778 bytes --]

Reproducible with 'emacs -q':

* In a freshly started Emacs, open an archive, e.g. a .tar.gz file in a
buffer.
* Switch to another buffer, type a couple of letters and then start
pressing 'M-/' (dabbrev-expand).
* It is very likely you will hit a "very useful" dabbrev in binary form
taken from the archive. E.g. I opened a large tarball, then switched to
*scratch*, typed "tz", and the very first dabbrev I got was "tz\327r".

Request: Emacs should ignore archive and other binary buffers when
generating dabbrevs. There are some settings to ignore buffers by name, but
this would rather ignore buffers by mode. Also, this should be the default
because I cannot imagine binary dabbrevs being useful to more than maybe
0.01% of users.

Reproduced on a recent Emacs trunk.

Paul

[-- Attachment #2: Type: text/html, Size: 991 bytes --]

bug#19392: Emacs searches for dabbrevs in archive buffers

[Eli Zaretskii]

> Date: Tue, 16 Dec 2014 10:32:13 +0100
> From: Paul Pogonyshev <pogonyshev@gmail.com>
> 
> * In a freshly started Emacs, open an archive, e.g. a .tar.gz file in a buffer.
> * Switch to another buffer, type a couple of letters and then start pressing
> 'M-/' (dabbrev-expand).
> * It is very likely you will hit a "very useful" dabbrev in binary form taken
> from the archive. E.g. I opened a large tarball, then switched to *scratch*,
> typed "tz", and the very first dabbrev I got was "tz\327r".
> 
> Request: Emacs should ignore archive and other binary buffers when generating
> dabbrevs.

What is the definition of "archive" for this purpose?

Also, when you visit a .tar.gz file, isn't most of the buffer readable
text (because Emacs automatically decompresses the archive)?

bug#19392: Emacs searches for dabbrevs in archive buffers

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 550 bytes --]

> What is the definition of "archive" for this purpose?

I would say any buffer that has certain major mode, e.g. 'archive-mode' or
'tar-mode'.

> Also, when you visit a .tar.gz file, isn't most of the buffer readable
> text (because Emacs automatically decompresses the archive)?​

Yes, it appears readable, but dabbrev still comes with various "useful"
binary data. Perhaps another approach would be to not search for abbrevs in
hidden text, but I don't know how that is implemented internally and if it
would be easy enough.

Paul

[-- Attachment #2: Type: text/html, Size: 688 bytes --]

bug#19392: Emacs searches for dabbrevs in archive buffers

[Lars Ingebrigtsen]

Paul Pogonyshev <pogonyshev@gmail.com> writes:

> * In a freshly started Emacs, open an archive, e.g. a .tar.gz file in a buffer.
> * Switch to another buffer, type a couple of letters and then start pressing 'M-/'
> (dabbrev-expand).
> * It is very likely you will hit a "very useful" dabbrev in binary form taken from
> the archive. E.g. I opened a large tarball, then switched to *scratch*, typed "tz",
> and the very first dabbrev I got was "tz\327r".

(I'm going through old bug reports that unfortunately got little response at
the time.)

I tried reproducing this in Emacs 28, and I don't get any of these
binary expansions.  Are you still seeing this in more recent Emacs
versions?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#19392: Emacs searches for dabbrevs in archive buffers

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 1034 bytes --]

Apparently not from tarballs anymore, but still e.g. from a `.zip' archive.
Though I have Emacs 27.1 here, not 28.

Paul

On Tue, 13 Jul 2021 at 20:09, Lars Ingebrigtsen <larsi@gnus.org> wrote:

> Paul Pogonyshev <pogonyshev@gmail.com> writes:
>
> > * In a freshly started Emacs, open an archive, e.g. a .tar.gz file in a
> buffer.
> > * Switch to another buffer, type a couple of letters and then start
> pressing 'M-/'
> > (dabbrev-expand).
> > * It is very likely you will hit a "very useful" dabbrev in binary form
> taken from
> > the archive. E.g. I opened a large tarball, then switched to *scratch*,
> typed "tz",
> > and the very first dabbrev I got was "tz\327r".
>
> (I'm going through old bug reports that unfortunately got little response
> at
> the time.)
>
> I tried reproducing this in Emacs 28, and I don't get any of these
> binary expansions.  Are you still seeing this in more recent Emacs
> versions?
>
> --
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no
>

[-- Attachment #2: Type: text/html, Size: 1592 bytes --]

bug#19392: Emacs searches for dabbrevs in archive buffers

[Lars Ingebrigtsen]

Paul Pogonyshev <pogonyshev@gmail.com> writes:

> Apparently not from tarballs anymore, but still e.g. from a `.zip'
> archive. Though I have Emacs 27.1 here, not 28.

I tried visiting a zip file here now (in Emacs 27.1), but I was unable
to make dabbrev-expand expand to any of the strings in the raw zip file.

Do you have a recipe, starting from "emacs -Q", to reproduce the bug?
(Including a test zip file, I guess.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#19392: Emacs searches for dabbrevs in archive buffers

[Paul Pogonyshev]

[-- Attachment #1.1: Type: text/plain, Size: 846 bytes --]

$ emacs -Q random.zip

1. Switch to buffer "*scratch*"
2. Type "P" (without quotes)
3. Press M-/
4. Here it finds expansion "PK" from buffer "random.zip".  Cycling
   with M-/ finds a few more, which are more binary-like.

On Tue, 13 Jul 2021 at 21:14, Lars Ingebrigtsen <larsi@gnus.org> wrote:

> Paul Pogonyshev <pogonyshev@gmail.com> writes:
>
> > Apparently not from tarballs anymore, but still e.g. from a `.zip'
> > archive. Though I have Emacs 27.1 here, not 28.
>
> I tried visiting a zip file here now (in Emacs 27.1), but I was unable
> to make dabbrev-expand expand to any of the strings in the raw zip file.
>
> Do you have a recipe, starting from "emacs -Q", to reproduce the bug?
> (Including a test zip file, I guess.)
>
> --
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no
>

[-- Attachment #1.2: Type: text/html, Size: 1612 bytes --]

[-- Attachment #2: random.zip --]
[-- Type: application/zip, Size: 1170 bytes --]

bug#19392: Emacs searches for dabbrevs in archive buffers

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 1153 bytes --]

Zipping binary data was not a good idea, but at least string "PK" doesn't
come from the archived "random.bin". You can create another test `.zip'
file by archiving a text file, to be sure.

Paul

On Tue, 13 Jul 2021 at 23:12, Paul Pogonyshev <pogonyshev@gmail.com> wrote:

> $ emacs -Q random.zip
>
> 1. Switch to buffer "*scratch*"
> 2. Type "P" (without quotes)
> 3. Press M-/
> 4. Here it finds expansion "PK" from buffer "random.zip".  Cycling
>    with M-/ finds a few more, which are more binary-like.
>
> On Tue, 13 Jul 2021 at 21:14, Lars Ingebrigtsen <larsi@gnus.org> wrote:
>
>> Paul Pogonyshev <pogonyshev@gmail.com> writes:
>>
>> > Apparently not from tarballs anymore, but still e.g. from a `.zip'
>> > archive. Though I have Emacs 27.1 here, not 28.
>>
>> I tried visiting a zip file here now (in Emacs 27.1), but I was unable
>> to make dabbrev-expand expand to any of the strings in the raw zip file.
>>
>> Do you have a recipe, starting from "emacs -Q", to reproduce the bug?
>> (Including a test zip file, I guess.)
>>
>> --
>> (domestic pets only, the antidote for overdose, milk.)
>>    bloggy blog: http://lars.ingebrigtsen.no
>>
>

[-- Attachment #2: Type: text/html, Size: 2217 bytes --]

bug#19392: Emacs searches for dabbrevs in archive buffers

[Eli Zaretskii]

> From: Paul Pogonyshev <pogonyshev@gmail.com>
> Date: Tue, 13 Jul 2021 23:16:44 +0200
> Cc: 19392@debbugs.gnu.org
> 
> Zipping binary data was not a good idea, but at least string "PK" doesn't come from the archived
> "random.bin". You can create another test `.zip' file by archiving a text file, to be sure.

FWIW, I don't see a bug here.  Why should we assume that "binary"
buffers could never have anything useful for M-/ ?

Note that Emacs only searches other buffers if it cannot find matches
in the current one, or exhausts the ones from the current buffer.  And
we already support dabbrev-select-buffers-function,
dabbrev-ignored-buffer-names, and dabbrev-ignored-buffer-regexps,
which could fine-tune which other buffers are searched.  So it sounds
like users who want to control which other buffers are searched have
more than enough knobs to have whatever behavior they want.

bug#19392: Emacs searches for dabbrevs in archive buffers

[Lars Ingebrigtsen]

Paul Pogonyshev <pogonyshev@gmail.com> writes:

> $ emacs -Q random.zip
>
> 1. Switch to buffer "*scratch*"
> 2. Type "P" (without quotes)
> 3. Press M-/
> 4. Here it finds expansion "PK" from buffer "random.zip".  Cycling
>    with M-/ finds a few more, which are more binary-like.

Thanks; with that recipe I can reproduce the behaviour.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#19392: Emacs searches for dabbrevs in archive buffers

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Note that Emacs only searches other buffers if it cannot find matches
> in the current one, or exhausts the ones from the current buffer.  And
> we already support dabbrev-select-buffers-function,
> dabbrev-ignored-buffer-names, and dabbrev-ignored-buffer-regexps,
> which could fine-tune which other buffers are searched.

We do have all of that, but I think that it would be reasonable (by
default) for `dabbrev--select-buffers' to ignore these "binary"
buffers.  Because the data there is mostly noise.

I think that could be easily implemented by allowing modes to set a
buffer-local variable like `dabbrev-ignore-buffer'.  I can see arc/tar
mode doing that, and perhaps image-mode, too?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#19392: Emacs searches for dabbrevs in archive buffers

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Paul Pogonyshev <pogonyshev@gmail.com>,  19392@debbugs.gnu.org
> Date: Wed, 14 Jul 2021 18:19:01 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Note that Emacs only searches other buffers if it cannot find matches
> > in the current one, or exhausts the ones from the current buffer.  And
> > we already support dabbrev-select-buffers-function,
> > dabbrev-ignored-buffer-names, and dabbrev-ignored-buffer-regexps,
> > which could fine-tune which other buffers are searched.
> 
> We do have all of that, but I think that it would be reasonable (by
> default) for `dabbrev--select-buffers' to ignore these "binary"
> buffers.  Because the data there is mostly noise.

If you want to do that, why not just add the appropriate regexp(s) to
dabbrev-ignored-buffer-regexps?

bug#19392: Emacs searches for dabbrevs in archive buffers

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> We do have all of that, but I think that it would be reasonable (by
>> default) for `dabbrev--select-buffers' to ignore these "binary"
>> buffers.  Because the data there is mostly noise.
>
> If you want to do that, why not just add the appropriate regexp(s) to
> dabbrev-ignored-buffer-regexps?

It's a user option, so we shouldn't add things to it dynamically, but we
could add a default that excludes these files.  The problem is, though,
that that would be a very complex regexp matching everything that
arc/tar/image modes covers, which is quite a lot.

The user option thing also goes for `dabbrev-ignored-buffer-names'.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#19392: Emacs searches for dabbrevs in archive buffers

[Stefan Kangas]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> We do have all of that, but I think that it would be reasonable (by
>>> default) for `dabbrev--select-buffers' to ignore these "binary"
>>> buffers.  Because the data there is mostly noise.
>>
>> If you want to do that, why not just add the appropriate regexp(s) to
>> dabbrev-ignored-buffer-regexps?
>
> It's a user option, so we shouldn't add things to it dynamically, but we
> could add a default that excludes these files.  The problem is, though,
> that that would be a very complex regexp matching everything that
> arc/tar/image modes covers, which is quite a lot.
>
> The user option thing also goes for `dabbrev-ignored-buffer-names'.

Shouldn't we rather have a defcustom to ignore certain major modes?
The name could be `dabbrev-ignored-major-modes'.

bug#19392: Emacs searches for dabbrevs in archive buffers

[Eli Zaretskii]

> From: Stefan Kangas <stefan@marxist.se>
> Date: Fri, 22 Oct 2021 19:43:18 -0400
> Cc: Eli Zaretskii <eliz@gnu.org>, pogonyshev@gmail.com, 19392@debbugs.gnu.org
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > Eli Zaretskii <eliz@gnu.org> writes:
> >
> >>> We do have all of that, but I think that it would be reasonable (by
> >>> default) for `dabbrev--select-buffers' to ignore these "binary"
> >>> buffers.  Because the data there is mostly noise.
> >>
> >> If you want to do that, why not just add the appropriate regexp(s) to
> >> dabbrev-ignored-buffer-regexps?
> >
> > It's a user option, so we shouldn't add things to it dynamically, but we
> > could add a default that excludes these files.  The problem is, though,
> > that that would be a very complex regexp matching everything that
> > arc/tar/image modes covers, which is quite a lot.
> >
> > The user option thing also goes for `dabbrev-ignored-buffer-names'.
> 
> Shouldn't we rather have a defcustom to ignore certain major modes?
> The name could be `dabbrev-ignored-major-modes'.

I don't necessarily object, but don't the existing defcustom's already
cover that, via buffer names?  We shouldn't over-engineer this, I
think.

bug#19392: Emacs searches for dabbrevs in archive buffers

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

>> > It's a user option, so we shouldn't add things to it dynamically, but we
>> > could add a default that excludes these files.  The problem is, though,
>> > that that would be a very complex regexp matching everything that
>> > arc/tar/image modes covers, which is quite a lot.
>>
>> Shouldn't we rather have a defcustom to ignore certain major modes?
>> The name could be `dabbrev-ignored-major-modes'.
>
> I don't necessarily object, but don't the existing defcustom's already
> cover that, via buffer names?  We shouldn't over-engineer this, I
> think.

Yes, let's try to avoid over-engineering.  This proposal was mostly just
an alternative idea that might allow us to avoid adding some hairy
and/or brittle regexps.

bug#19392: Emacs searches for dabbrevs in archive buffers

[Lars Ingebrigtsen]

Stefan Kangas <stefan@marxist.se> writes:

> Shouldn't we rather have a defcustom to ignore certain major modes?
> The name could be `dabbrev-ignored-major-modes'.

Are there a lot of these modes?  I'm just wondering whether this would
make things easier (in practice) or not.  So (off the top of my head),
this would be archive-mode, tar-mode, image-mode...  Uhm...  perhaps
those are the major ones?

Then it does sound like a simpler way to customise this, because there's
a lot of archive/image file names you'd have to match.

So I think adding `dabbrev-ignored-major-modes' sounds like a good idea.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#19392: Emacs searches for dabbrevs in archive buffers

[Stefan Kangas]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Are there a lot of these modes?  I'm just wondering whether this would
> make things easier (in practice) or not.  So (off the top of my head),
> this would be archive-mode, tar-mode, image-mode...  Uhm...  perhaps
> those are the major ones?

Perhaps `hexl-mode' as well?

You could throw in `tetris-mode' and the other ones in play/, I guess.

bug#19392: Emacs searches for dabbrevs in archive buffers

[Lars Ingebrigtsen]

Stefan Kangas <stefan@marxist.se> writes:

> Perhaps `hexl-mode' as well?

Possibly -- does it load the binary data into a buffer?

> You could throw in `tetris-mode' and the other ones in play/, I guess.

I don't think those have binary data (that may accidentally look like an
abbreviation). 

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#19392: Emacs searches for dabbrevs in archive buffers

[Stefan Kangas]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Stefan Kangas <stefan@marxist.se> writes:
>
>> Perhaps `hexl-mode' as well?
>
> Possibly -- does it load the binary data into a buffer?

It is just unlikely to produce anything useful.  Here's what the start
of a PNG file might look like, for example:

00000000: 8950 4e47 0d0a 1a0a 0000 000d 4948 4452  .PNG........IHDR
00000010: 0000 0a00 0000 05a0 0802 0000 001d 628d  ..............b.
00000020: 8800 0000 0373 4249 5408 0808 dbe1 4fe0  .....sBIT.....O.
00000030: 0000 2000 4944 4154 785e ecdd 0760 14d5  .. .IDATx^...`..
00000040: bec7 f1f4 4aef 5d40 a4ab 0822 0a8a 7410  ....J.]@..."..t.
00000050: c586 ca55 5111 bc56 042b 0f15 7bbd 8280  ...UQ..V.+..{...

>> You could throw in `tetris-mode' and the other ones in play/, I guess.
>
> I don't think those have binary data (that may accidentally look like an
> abbreviation).

Right, it is another case of a buffer that will just contain nothing
useful for completion purposes.  But perhaps it doesn't contain anything
harmful either.

bug#19392: Emacs searches for dabbrevs in archive buffers

[Lars Ingebrigtsen]

Stefan Kangas <stefan@marxist.se> writes:

> It is just unlikely to produce anything useful.  Here's what the start
> of a PNG file might look like, for example:
>
> 00000000: 8950 4e47 0d0a 1a0a 0000 000d 4948 4452  .PNG........IHDR
> 00000010: 0000 0a00 0000 05a0 0802 0000 001d 628d  ..............b.

True, that's a good candidate for exclusion, too.

>>> You could throw in `tetris-mode' and the other ones in play/, I guess.
>>
>> I don't think those have binary data (that may accidentally look like an
>> abbreviation).
>
> Right, it is another case of a buffer that will just contain nothing
> useful for completion purposes.  But perhaps it doesn't contain anything
> harmful either.

Yeah, it seems unlikely that there'll be much in a Tetris buffers that
looks like text.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#19392: Emacs searches for dabbrevs in archive buffers

[Lars Ingebrigtsen]

Paul Pogonyshev <pogonyshev@gmail.com> writes:

> $ emacs -Q random.zip
>
> 1. Switch to buffer "*scratch*"
> 2. Type "P" (without quotes)
> 3. Press M-/
> 4. Here it finds expansion "PK" from buffer "random.zip".  Cycling
>    with M-/ finds a few more, which are more binary-like.

I've now fixed this in Emacs 29.  Instead of making some kind of
framework for letting modes specify this themselves, it seems more user
friendly to just introduce a new dabbrev-ignored-buffer-modes variable,
so I did that.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#19392: Emacs searches for dabbrevs in archive buffers

[Eli Zaretskii]

> Cc: 19392@debbugs.gnu.org
> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Sat, 07 May 2022 13:20:47 +0200
> 
> Paul Pogonyshev <pogonyshev@gmail.com> writes:
> 
> > $ emacs -Q random.zip
> >
> > 1. Switch to buffer "*scratch*"
> > 2. Type "P" (without quotes)
> > 3. Press M-/
> > 4. Here it finds expansion "PK" from buffer "random.zip".  Cycling
> >    with M-/ finds a few more, which are more binary-like.
> 
> I've now fixed this in Emacs 29.  Instead of making some kind of
> framework for letting modes specify this themselves, it seems more user
> friendly to just introduce a new dabbrev-ignored-buffer-modes variable,
> so I did that.

Does tar-mode really belong to the list of those modes?  Unlike
archive-mode, where each file is separately compressed into binary
garbage, tar-mode shows the original files, and the only "binary"
parts are the headers, most of which are also plain text.  So I'd
suggest to remove tar-mode for the default list of modes.

bug#19392: Emacs searches for dabbrevs in archive buffers

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Does tar-mode really belong to the list of those modes?  Unlike
> archive-mode, where each file is separately compressed into binary
> garbage, tar-mode shows the original files, and the only "binary"
> parts are the headers, most of which are also plain text.  So I'd
> suggest to remove tar-mode for the default list of modes.

Ah, yes, I'd forgotten that.  So I've now removed the mode from the
default list.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no