bug#63363: 28.2; emacs freezes opening python file w/ unclosed quotes

bug#63363: 28.2; emacs freezes opening python file w/ unclosed quotes

[James Mazer]

[-- Attachment #1: Type: text/plain, Size: 5514 bytes --]

Emacs 28.2 (and I think 28.1 at least) locks up when opening file
containing unclosed triple quotes in a docstring (ie, docstring started,
but not yet closed) in vanilla python mode.

To replicate, save the following to a file, say `foo.py`
-------cut here-------
def Foo:
    # comment out next line and works ok, incomplete comment -> locks up
    """
    pass

class Bar:
    labelmasks: torch.Tensor
    """blah blah blah - single line common ok, >1 line locks up
    """

class Baz:
    def __init__(self):
"""apostrophe in this quote causes lock up. It's better to..."""
pass
-------end cut-------

And then just run `emacs -Q foo.py`. Under X11, window appears and is
completely frozen, must be killed from another window. Same happens
if you open file with `find-file` etc. Seems to be something related to
trying to fontify the buffer in python-mode.

This also happens periodically when typing into a buffer if you create an
unclosed docstring. I can't reliably replicate that conditions, but appears
to be the same problem. Can also see this issue on  github where it was
originally posted (thinking it was an issue with the linux snap
distribution): https://github.com/alexmurray/emacs-snap/issues/44. This
does not appear to be related to the snap distribution and github issue
indicates that this also occurs in emacs-29.

Sending SIGUSR2 generates the following backtrace:
> Debugger entered--entering a function:
> * #f(compiled-function () #<bytecode 0xb8840b003fb641a>)()
>   font-lock-fontify-syntactically-region(1 347 nil)
>   font-lock-default-fontify-region(1 347 nil)
>   font-lock-fontify-region(1 347)
>   #f(compiled-function (fun) #<bytecode
0x19ba0d15e0d0b9bd>)(font-lock-fontify-region)
>   jit-lock--run-functions(1 347)
>   jit-lock-fontify-now(1 1501)
>   jit-lock-function(1)
>   redisplay_internal\ \(C\ function\)()

In GNU Emacs 28.2 (build 2, x86_64-pc-linux-gnu, GTK+ Version 3.24.20,
cairo version 1.16.0)
 of 2023-04-26 built on lcy02-amd64-043
Repository revision: 34c90cc5f22e16b027a731d10c3f58f9e44101e9
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12013000
System Description: Linux Mint 20.2

Configured using:
 'configure --prefix=/snap/emacs/current/usr --with-x-toolkit=gtk3
 --without-xaw3d --with-modules --with-cairo --with-native-compilation
 'CFLAGS=-isystem/build/emacs/parts/emacs/install/usr/include
 -isystem/build/emacs/parts/emacs/install/usr/include/x86_64-linux-gnu
 -O2' 'CPPFLAGS=-isystem/build/emacs/parts/emacs/install/usr/include
 -isystem/build/emacs/parts/emacs/install/usr/include/x86_64-linux-gnu'
 'LDFLAGS=-L/build/emacs/parts/emacs/install/lib
 -L/build/emacs/parts/emacs/install/usr/lib
 -L/build/emacs/parts/emacs/install/lib/x86_64-linux-gnu
 -L/build/emacs/parts/emacs/install/usr/lib/x86_64-linux-gnu''

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES
NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF
TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
rfc822 mml mml-sec epa derived epg rfc6068 epg-config gnus-util rmail
rmail-loaddefs auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache json map text-property-search time-date
subr-x seq byte-opt gv bytecomp byte-compile cconv mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader cl-loaddefs
cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils
iso-transl tooltip eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel term/x-win x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock
font-lock syntax font-core term/tty-colors frame minibuffer cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice button loaddefs faces cus-face macroexp files
window text-properties overlay sha1 md5 base64 format env code-pages
mule custom widget hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
cairo move-toolbar gtk x-toolkit x multi-tty make-network-process
native-compile emacs)

Memory information:
((conses 16 68541 8782)
 (symbols 48 6639 0)
 (strings 32 19479 2438)
 (string-bytes 1 666720)
 (vectors 16 14703)
 (vector-slots 8 301741 11225)
 (floats 8 22 32)
 (intervals 56 266 0)
 (buffers 992 11))

-- 
James Mazer
mazerj@gmail.com

[-- Attachment #2: Type: text/html, Size: 6856 bytes --]

bug#63363: 28.2; emacs freezes opening python file w/ unclosed quotes

[Ruijie Yu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/html, Size: 7538 bytes --]

bug#63363: 28.2; emacs freezes opening python file w/ unclosed quotes

[Eli Zaretskii]

> Cc: 63363@debbugs.gnu.org
> Date: Mon, 8 May 2023 15:47:29 +0800
> From:  Ruijie Yu via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> This is probably a duplicate of bug#62325, which is fixed not in 28 but in 29. 

FWIW, I cannot reproduce this neither in Emacs 28 nor in Emacs 29.

bug#63363: 28.2; emacs freezes opening python file w/ unclosed quotes

[James Mazer]

[-- Attachment #1: Type: text/plain, Size: 767 bytes --]

Can you give me any info on how you tried to reproduce it? OS, installation
method etc?
I get it pretty reliably and the snap package maintainer reported the
recipe I provided locks up for him on both 28 and 29.
If doesn't happen for you, I can try to replicate your setup..
Thanks


On Mon, May 8, 2023 at 5:39 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > Cc: 63363@debbugs.gnu.org
> > Date: Mon, 8 May 2023 15:47:29 +0800
> > From:  Ruijie Yu via "Bug reports for GNU Emacs,
> >  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> >
> > This is probably a duplicate of bug#62325, which is fixed not in 28 but
> in 29.
>
> FWIW, I cannot reproduce this neither in Emacs 28 nor in Emacs 29.
>


-- 
James Mazer
mazerj@gmail.com

[-- Attachment #2: Type: text/html, Size: 1822 bytes --]

bug#63363: 28.2; emacs freezes opening python file w/ unclosed quotes

[Eli Zaretskii]

> From: James Mazer <mazerj2006@gmail.com>
> Date: Mon, 8 May 2023 10:28:59 -0600
> Cc: Ruijie Yu <ruijie@netyu.xyz>, 63363@debbugs.gnu.org
> 
> Can you give me any info on how you tried to reproduce it? OS, installation method etc?

I did exactly what you said I should do:

  emacs -Q foo.py

I also tried starting "emacs -Q" and "C-x C-f foo.py" from inside
Emacs.  Neither of these two reproduced the problem.

As for the OS, I tried this on GNU/Linux and on MS-Windows, with
negative results in both cases.

My Emacs is built by myself and installed on the system where it was
built.

bug#63363: 28.2; emacs freezes opening python file w/ unclosed quotes

[James Mazer]

[-- Attachment #1: Type: text/plain, Size: 1043 bytes --]

Don't have time right now to do a custom build, but as sanity check, I just
quickly pulled the 28.2 gnu.org.emacs flatpak and tried that and I get
exactly the same issue, so it doesn't appear to be specific to the snap
build. I can't find a 29 snap or flatpak to test, though.



On Mon, May 8, 2023 at 10:40 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: James Mazer <mazerj2006@gmail.com>
> > Date: Mon, 8 May 2023 10:28:59 -0600
> > Cc: Ruijie Yu <ruijie@netyu.xyz>, 63363@debbugs.gnu.org
> >
> > Can you give me any info on how you tried to reproduce it? OS,
> installation method etc?
>
> I did exactly what you said I should do:
>
>   emacs -Q foo.py
>
> I also tried starting "emacs -Q" and "C-x C-f foo.py" from inside
> Emacs.  Neither of these two reproduced the problem.
>
> As for the OS, I tried this on GNU/Linux and on MS-Windows, with
> negative results in both cases.
>
> My Emacs is built by myself and installed on the system where it was
> built.
>


-- 
James Mazer
mazerj@gmail.com

[-- Attachment #2: Type: text/html, Size: 2180 bytes --]

bug#63363: 28.2; emacs freezes opening python file w/ unclosed quotes

[Ruijie Yu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

James Mazer <mazerj2006@gmail.com> writes:

> Don't have time right now to do a custom build, but as sanity check, I just quickly pulled the 28.2 gnu.org.emacs flatpak and tried that and I get
> exactly the same issue, so it doesn't appear to be specific to the snap build. I can't find a 29 snap or flatpak to test, though.

There is also another bug report (closed as fixed on 29), #62794, that I
think is the same as this report and #62325.  In particular, see this
message within that bug report:

msgid:ZDdE/j6dDbhCw1QF@nicku.org
https://mail.gnu.org/archive/html/bug-gnu-emacs/2023-04/msg00798.html

What Nick described in #62794 sounds exactly the same as your issue and
Eli's observation: that only _some type(s) of_ python files trigger the
error.  I, as well as Eli, were able to reproduce the issue of #62794
via using the file Nick attached in that message.

James, can you confirm that Nick's issue on #62794 is exactly the same
as yours in this bug report, and that you can also reproduce the issue
using Nick's attachment from that message?

And also, if you want to get a version of 29 to test, you can either get
the pretest or clone the code and run "make".  The resultant emacs
binary is located at "src/emacs".  Let me know if this is unclear.

-- 
Best,


RY

bug#63363: 28.2; emacs freezes opening python file w/ unclosed quotes

[James Mazer]

[-- Attachment #1: Type: text/plain, Size: 2758 bytes --]

Yes, seems to be the same issue. I just tried replicating his trigger and
locked up an emacs window.. here's the backtrace after sigusr2:

> Debugger entered--entering a function:
> * #f(compiled-function () #<bytecode 0xb893e2c8c539dfa>)()
>   syntax-ppss()
>   python-nav-end-of-statement()
>   python-nav-end-of-block()
>   python-info-statement-ends-block-p()
>   python-nav--forward-sexp(-1 nil nil)
>   python-nav-forward-sexp(-1 nil nil)
>   python-nav-backward-sexp()
>   python-info-docstring-p((0 nil 390 34 nil nil 0 nil 427 nil nil))
>   python-font-lock-syntactic-face-function((0 nil 390 34 nil nil 0 nil 427
> nil nil))
>   font-lock-fontify-syntactically-region(180 1700 nil)
>   font-lock-default-fontify-region(184 1684 nil)
>   font-lock-fontify-region(184 1684)
>   #f(compiled-function (fun) #<bytecode
> 0x19ba0a4f10e54dbd>)(font-lock-fontify-region)
>   jit-lock--run-functions(184 1684)
>   jit-lock-fontify-now(184 1684)
>   jit-lock-function(184)
>   redisplay_internal\ \(C\ function\)()


I'll see if I can spin up a version of 29. Been a while since I compiled
emacs from scratch... I'm actually surprised that there aren't packages for
28 and 29 more readily available for ubuntu. Not sure what's up with that.


On Mon, May 8, 2023 at 9:09 PM Ruijie Yu <ruijie@netyu.xyz> wrote:

>
> James Mazer <mazerj2006@gmail.com> writes:
>
> > Don't have time right now to do a custom build, but as sanity check, I
> just quickly pulled the 28.2 gnu.org.emacs flatpak and tried that and I get
> > exactly the same issue, so it doesn't appear to be specific to the snap
> build. I can't find a 29 snap or flatpak to test, though.
>
> There is also another bug report (closed as fixed on 29), #62794, that I
> think is the same as this report and #62325.  In particular, see this
> message within that bug report:
>
> msgid:ZDdE/j6dDbhCw1QF@nicku.org
> https://mail.gnu.org/archive/html/bug-gnu-emacs/2023-04/msg00798.html
>
> What Nick described in #62794 sounds exactly the same as your issue and
> Eli's observation: that only _some type(s) of_ python files trigger the
> error.  I, as well as Eli, were able to reproduce the issue of #62794
> via using the file Nick attached in that message.
>
> James, can you confirm that Nick's issue on #62794 is exactly the same
> as yours in this bug report, and that you can also reproduce the issue
> using Nick's attachment from that message?
>
> And also, if you want to get a version of 29 to test, you can either get
> the pretest or clone the code and run "make".  The resultant emacs
> binary is located at "src/emacs".  Let me know if this is unclear.
>
> --
> Best,
>
>
> RY
>


-- 
James Mazer
mazerj@gmail.com

[-- Attachment #2: Type: text/html, Size: 4176 bytes --]

bug#63363: 28.2; emacs freezes opening python file w/ unclosed quotes

[James Mazer]

[-- Attachment #1: Type: text/plain, Size: 3163 bytes --]

Just confirmed that with an emacs 29.0.90 snap (beta) I'm not seeing the
lock up. So it appears to be fixed downstream, but would be nice to be able
to fix in 28 until 29 is actually released :-)


On Tue, May 9, 2023 at 9:23 AM James Mazer <mazerj2006@gmail.com> wrote:

> Yes, seems to be the same issue. I just tried replicating his trigger and
> locked up an emacs window.. here's the backtrace after sigusr2:
>
>> Debugger entered--entering a function:
>> * #f(compiled-function () #<bytecode 0xb893e2c8c539dfa>)()
>>   syntax-ppss()
>>   python-nav-end-of-statement()
>>   python-nav-end-of-block()
>>   python-info-statement-ends-block-p()
>>   python-nav--forward-sexp(-1 nil nil)
>>   python-nav-forward-sexp(-1 nil nil)
>>   python-nav-backward-sexp()
>>   python-info-docstring-p((0 nil 390 34 nil nil 0 nil 427 nil nil))
>>   python-font-lock-syntactic-face-function((0 nil 390 34 nil nil 0 nil
>> 427 nil nil))
>>   font-lock-fontify-syntactically-region(180 1700 nil)
>>   font-lock-default-fontify-region(184 1684 nil)
>>   font-lock-fontify-region(184 1684)
>>   #f(compiled-function (fun) #<bytecode
>> 0x19ba0a4f10e54dbd>)(font-lock-fontify-region)
>>   jit-lock--run-functions(184 1684)
>>   jit-lock-fontify-now(184 1684)
>>   jit-lock-function(184)
>>   redisplay_internal\ \(C\ function\)()
>
>
> I'll see if I can spin up a version of 29. Been a while since I compiled
> emacs from scratch... I'm actually surprised that there aren't packages for
> 28 and 29 more readily available for ubuntu. Not sure what's up with that.
>
>
> On Mon, May 8, 2023 at 9:09 PM Ruijie Yu <ruijie@netyu.xyz> wrote:
>
>>
>> James Mazer <mazerj2006@gmail.com> writes:
>>
>> > Don't have time right now to do a custom build, but as sanity check, I
>> just quickly pulled the 28.2 gnu.org.emacs flatpak and tried that and I get
>> > exactly the same issue, so it doesn't appear to be specific to the snap
>> build. I can't find a 29 snap or flatpak to test, though.
>>
>> There is also another bug report (closed as fixed on 29), #62794, that I
>> think is the same as this report and #62325.  In particular, see this
>> message within that bug report:
>>
>> msgid:ZDdE/j6dDbhCw1QF@nicku.org
>> https://mail.gnu.org/archive/html/bug-gnu-emacs/2023-04/msg00798.html
>>
>> What Nick described in #62794 sounds exactly the same as your issue and
>> Eli's observation: that only _some type(s) of_ python files trigger the
>> error.  I, as well as Eli, were able to reproduce the issue of #62794
>> via using the file Nick attached in that message.
>>
>> James, can you confirm that Nick's issue on #62794 is exactly the same
>> as yours in this bug report, and that you can also reproduce the issue
>> using Nick's attachment from that message?
>>
>> And also, if you want to get a version of 29 to test, you can either get
>> the pretest or clone the code and run "make".  The resultant emacs
>> binary is located at "src/emacs".  Let me know if this is unclear.
>>
>> --
>> Best,
>>
>>
>> RY
>>
>
>
> --
> James Mazer
> mazerj@gmail.com
>
>

-- 
James Mazer
mazerj@gmail.com

[-- Attachment #2: Type: text/html, Size: 5327 bytes --]

bug#63363: 28.2; emacs freezes opening python file w/ unclosed quotes

[James Mazer]

[-- Attachment #1: Type: text/plain, Size: 3771 bytes --]

Disregard that -- until 29 is out, I just added
>
> (defun python-nav-end-of-statement--bug-override (orig-fn &optional noend)
>   (forward-line 1))
> (advice-add 'python-nav-end-of-statement :around
>    #'python-nav-end-of-statement--bug-override)

to my init.el (from https://debbugs.gnu.org/cgi/bugreport.cgi?bug=58780).
Seems to do the trick for now -- no more lost buffers :-)

On Wed, May 10, 2023 at 5:01 PM James Mazer <mazerj2006@gmail.com> wrote:

> Just confirmed that with an emacs 29.0.90 snap (beta) I'm not seeing the
> lock up. So it appears to be fixed downstream, but would be nice to be able
> to fix in 28 until 29 is actually released :-)
>
>
> On Tue, May 9, 2023 at 9:23 AM James Mazer <mazerj2006@gmail.com> wrote:
>
>> Yes, seems to be the same issue. I just tried replicating his trigger and
>> locked up an emacs window.. here's the backtrace after sigusr2:
>>
>>> Debugger entered--entering a function:
>>> * #f(compiled-function () #<bytecode 0xb893e2c8c539dfa>)()
>>>   syntax-ppss()
>>>   python-nav-end-of-statement()
>>>   python-nav-end-of-block()
>>>   python-info-statement-ends-block-p()
>>>   python-nav--forward-sexp(-1 nil nil)
>>>   python-nav-forward-sexp(-1 nil nil)
>>>   python-nav-backward-sexp()
>>>   python-info-docstring-p((0 nil 390 34 nil nil 0 nil 427 nil nil))
>>>   python-font-lock-syntactic-face-function((0 nil 390 34 nil nil 0 nil
>>> 427 nil nil))
>>>   font-lock-fontify-syntactically-region(180 1700 nil)
>>>   font-lock-default-fontify-region(184 1684 nil)
>>>   font-lock-fontify-region(184 1684)
>>>   #f(compiled-function (fun) #<bytecode
>>> 0x19ba0a4f10e54dbd>)(font-lock-fontify-region)
>>>   jit-lock--run-functions(184 1684)
>>>   jit-lock-fontify-now(184 1684)
>>>   jit-lock-function(184)
>>>   redisplay_internal\ \(C\ function\)()
>>
>>
>> I'll see if I can spin up a version of 29. Been a while since I compiled
>> emacs from scratch... I'm actually surprised that there aren't packages for
>> 28 and 29 more readily available for ubuntu. Not sure what's up with that.
>>
>>
>> On Mon, May 8, 2023 at 9:09 PM Ruijie Yu <ruijie@netyu.xyz> wrote:
>>
>>>
>>> James Mazer <mazerj2006@gmail.com> writes:
>>>
>>> > Don't have time right now to do a custom build, but as sanity check, I
>>> just quickly pulled the 28.2 gnu.org.emacs flatpak and tried that and I get
>>> > exactly the same issue, so it doesn't appear to be specific to the
>>> snap build. I can't find a 29 snap or flatpak to test, though.
>>>
>>> There is also another bug report (closed as fixed on 29), #62794, that I
>>> think is the same as this report and #62325.  In particular, see this
>>> message within that bug report:
>>>
>>> msgid:ZDdE/j6dDbhCw1QF@nicku.org
>>> https://mail.gnu.org/archive/html/bug-gnu-emacs/2023-04/msg00798.html
>>>
>>> What Nick described in #62794 sounds exactly the same as your issue and
>>> Eli's observation: that only _some type(s) of_ python files trigger the
>>> error.  I, as well as Eli, were able to reproduce the issue of #62794
>>> via using the file Nick attached in that message.
>>>
>>> James, can you confirm that Nick's issue on #62794 is exactly the same
>>> as yours in this bug report, and that you can also reproduce the issue
>>> using Nick's attachment from that message?
>>>
>>> And also, if you want to get a version of 29 to test, you can either get
>>> the pretest or clone the code and run "make".  The resultant emacs
>>> binary is located at "src/emacs".  Let me know if this is unclear.
>>>
>>> --
>>> Best,
>>>
>>>
>>> RY
>>>
>>
>>
>> --
>> James Mazer
>> mazerj@gmail.com
>>
>>
>
> --
> James Mazer
> mazerj@gmail.com
>
>

-- 
James Mazer
mazerj@gmail.com

[-- Attachment #2: Type: text/html, Size: 6871 bytes --]

bug#63363: 28.2; emacs freezes opening python file w/ unclosed quotes

[Eli Zaretskii]

> From: James Mazer <mazerj2006@gmail.com>
> Date: Wed, 10 May 2023 17:01:02 -0600
> Cc: Eli Zaretskii <eliz@gnu.org>, 63363@debbugs.gnu.org
> 
> Just confirmed that with an emacs 29.0.90 snap (beta) I'm not seeing the lock up. So it appears to be
> fixed downstream, but would be nice to be able to fix in 28 until 29 is actually released :-)

Thanks for testing in Emacs 29.  I'm therefore closing this bug.