bug#31551: 27.0.50; emacs hangs in hexl-mode on modula2 files

[Eli Zaretskii <eliz@gnu.org>]

> From: Alexandre Adolphe <alexandre.adolphe@gmail.com>
> Date: Tue, 22 May 2018 15:24:09 +0200
> 
> I run into an issue lately while trying to use the hexl-mode with
> a modula2 file that I found and emacs starts to hang after the
> hexl-mode starts. I tested it on emacs 24.5, 25.2 and master.
> 
> To reproduce the issue you can use :
> echo "abcd" > test.MOD
> emacs -Q --eval '(show-paren-mode 1)' test.MOD
> M-x hexl-mode
> 
> --> emacs hangs and use 100% cpu. It only happen on the first character so
> we can run out the hang with multiple C-g + an arrow key.
> 
> I dig into the issue and it appears that the was because of the
> show-paren-mode and the fact that the ":" character have meaning in
> modula-2-mode. Emacs is stuck in the loop of the
> m2-smie-refine-colon function because after the "00000000" that start
> the hexl buffer there is no more sexp to parse so the tok variable is
> empty but the (forward-sexp -1) call does not fail.
> 
> I fixed that myself by stopping the loop if the point is at the beginning
> of the buffer. Not sure if it correct but here is the diff :

Hmm... I think there's a more serious issue at hand here.  Hexl mode
is not a normal major mode: it doesn't undo the effects of the
previous major mode.  So what happens here is that the display is
entirely inappropriate for Modula2 sources, but the syntactic setup
made by m2-mode is still in effect, and that setup attaches special
meaning to ':' when scanning for parentheses.

In addition, Hexl mode does nothing to prevent point from entering the
parts of the buffer that are "outside the Hexl region", such as the
address column (which ends in a colon ':').  Hexl mode only redefines
some interactive cursor movement commands to avoid entering the
non-Hexl portions, but try "C-u 1 M-x goto-char RET", and Emacs will
happily go to the "forbidden" area.  (Even some cursor motion commands
are not intercepted: e.g., try backward-word.)  So Lisp programs can
easily move point into portions of the buffer that have nothing to do
with the actual file contents.

On top of that, show-paren-mode will happily turn itself on in a
buffer under Hexl mode, although it makes no sense to me: the Hexl
display shows no parens to work with.

The fact that syntactical customizations of a mode stay in effect when
the display changes so radically sounds like a potential cause of many
trouble.  So perhaps the right solution would be to temporarily
disable mode syntax while in Hexl mode.

Stefan, can you take a look and comment?