* bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling
@ 2024-12-08 13:05 rehan malak
2024-12-08 17:37 ` Eli Zaretskii
0 siblings, 1 reply; 4+ messages in thread
From: rehan malak @ 2024-12-08 13:05 UTC (permalink / raw)
To: 74738
[-- Attachment #1: Type: text/plain, Size: 8963 bytes --]
Hi,
I can reproduce systematically this freeze dealing with a 10000lines
Python file :
wget https://raw.githubusercontent.com/hugsy/gef/refs/heads/main/gef.py
emacs -Q gef.py
PageDown several times or scrolling with the mouse
Emacs freezes, Ctrl-g not working, CPU 100%
It works also by searching : Ctrl-s show RET
then Ctrl-s several times
With a minimal .emacs :
(set debug-on-error t)
(set debug-on-quit t)
and before scrolling
M-x profiler-start RET
then in an external terminal
pkill -SIGUSR2 emacs
I get the backtrace :
Debugger entered--entering a function:
* #f(compiled-function () #<bytecode -0x179dcd1db31182ae>)()
syntax-ppss()
python-syntax-context-type()
python-nav-forward-block(-1)
python-nav-backward-block()
python-nav-beginning-of-block()
python-nav-end-of-block()
python-info-statement-ends-block-p()
python-info-end-of-block-p()
python-nav--forward-sexp(1 nil nil)
python-nav-forward-sexp(1)
forward-sexp(1)
up-list(1)
python--font-lock-f-strings(30419)
font-lock-fontify-keywords-region(28914 30419 nil)
font-lock-default-fontify-region(28914 30414 nil)
font-lock-fontify-region(28914 30414)
#f(compiled-function (fun) #<bytecode
-0x8a96d65e1315e71>)(font-lock-fontify-region)
run-hook-wrapped(#f(compiled-function (fun) #<bytecode
-0x8a96d65e1315e71>) font-lock-fontify-region)
jit-lock--run-functions(28914 30414)
jit-lock-fontify-now(28914 30414)
jit-lock-function(28914)
and can look at profiler report
M-x profiler-stop
M-x profiler-report
showing time spent in syntax-ppss
17770 88% - jit-lock-function
17770 88% - jit-lock-fontify-now
17770 88% - jit-lock--run-functions
17770 88% - run-hook-wrapped
17770 88% - #<byte-code-function FA0>
17770 88% - font-lock-fontify-region
17770 88% - font-lock-default-fontify-region
17690 87% - font-lock-fontify-keywords-region
17678 87% - python--font-lock-f-strings
17674 87% - up-list
17674 87% - forward-sexp
17674 87% - python-nav-forward-sexp
17674 87% - python-nav--forward-sexp
9253 45% - python-info-end-of-block-p
9245 45% - python-info-statement-ends-block-p
9221 45% - python-nav-end-of-block
8609 42% - python-nav-beginning-of-block
8557 42% - python-nav-backward-block
8557 42% - python-nav-forward-block
8005 39% - python-syntax-context-type
7997 39% - syntax-ppss
7097 35% parse-partial-sexp
753 3% + #<byte-code-function A23>
376 1% + python-nav-beginning-of-statement
132 0% re-search-backward
24 0% looking-at
28 0% current-indentation
8 0% + python-nav-beginning-of-statement
4 0% python-info-current-line-empty-p
4 0% + python-info-current-line-comment-p
604 2% + python-nav-end-of-statement
4 0% + python-info-current-line-comment-p
4 0% current-indentation
24 0% + python-nav-end-of-statement
8 0% + python-info-end-of-statement-p
8178 40% - python-info-statement-ends-block-p
8150 40% - python-nav-end-of-block
7754 38% - python-nav-beginning-of-block
7738 38% - python-nav-backward-block
7734 38% - python-nav-forward-block
7178 35% - python-syntax-context-type
7174 35% - syntax-ppss
6978 34% parse-partial-sexp
8 0% make-closure
8 0% #<byte-code-function B1E>
4 0% syntax-table
4 0% syntax-ppss--update-stats
412 2% + python-nav-beginning-of-statement
92 0% re-search-backward
44 0% looking-at
12 0% current-indentation
4 0% looking-at
380 1% + python-nav-end-of-statement
16 0% + python-util-forward-comment
28 0% + python-nav-end-of-statement
151 0% + python-nav-end-of-block
28 0% + python-info-beginning-of-block-p
24 0% + python-info-end-of-statement-p
20 0% + python-info-beginning-of-statement-p
8 0% + python-syntax-context-type
8 0% + python-info-statement-starts-block-p
4 0% + syntax-ppss
8 0% + #<byte-code-function 7E0>
4 0% + #<byte-code-function 740>
80 0% + font-lock-fontify-syntactically-region
2279 11% Automatic GC
73 0% + redisplay_internal (C function)
40 0% + command-execute
8 0% + ...
4 0% + mouse--click-1-maybe-follows-link
other people talking about python/profiling/syntax-ppss :
https://www.reddit.com/r/emacs/comments/z0oye9/emacs_freezes_when_opening_a_python_file/
https://www.reddit.com/r/emacs/comments/9h7onq/large_python_files_hanging_with_an_unmatched_quote/
Please tell me how I can help ?
Please tell me if there is a workaround without losing python syntax color ?
Thanks
Rehan
In GNU Emacs 31.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
3.24.43, cairo version 1.18.2) of 2024-12-08 built on cndl
Repository revision: b953bcb17047998c9e41cede7c5e5ffec22209b2
Repository branch: master
System Description: Debian GNU/Linux trixie/sid
Configured using:
'configure --prefix=/home/jean/local/emacs --with-pgtk'
Configured features:
CAIRO DBUS FREETYPE GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG LIBSELINUX
LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PGTK PNG SECCOMP SOUND THREADS
TIFF TOOLKIT_SCROLL_BARS WEBP XIM GTK3 ZLIB
Important settings:
value of $LANG: en_US.UTF-8
value of $XMODIFIERS: @im=ibus
locale-coding-system: utf-8-unix
Major mode: Python
Minor modes in effect:
tooltip-mode: t
global-eldoc-mode: t
eldoc-mode: t
show-paren-mode: t
electric-indent-mode: t
mouse-wheel-mode: t
tool-bar-mode: t
menu-bar-mode: t
file-name-shadow-mode: t
global-font-lock-mode: t
font-lock-mode: t
blink-cursor-mode: t
minibuffer-regexp-mode: t
line-number-mode: t
transient-mark-mode: t
auto-composition-mode: t
auto-encryption-mode: t
auto-compression-mode: t
Load-path shadows:
None found.
Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils cl-seq python
rx project byte-opt gv bytecomp byte-compile compat pcase treesit comint
subr-x ansi-osc ring cl-loaddefs cl-lib ansi-color rmc iso-transl
tooltip cconv eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel term/pgtk-win pgtk-win term/common-win
touch-screen pgtk-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode register
page tab-bar menu-bar rfn-eshadow isearch easymenu timer select
scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer nadvice seq simple cl-generic indonesian philippine
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp
files window text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget keymap hashtable-print-readable backquote
threads dbusbind inotify dynamic-setting system-font-setting
font-render-setting cairo gtk pgtk multi-tty move-toolbar
make-network-process emacs)
Memory information:
((conses 16 60995 13527) (symbols 48 7517 0) (strings 32 19122 1493)
(string-bytes 1 515162) (vectors 16 13554)
(vector-slots 8 145472 8433) (floats 8 26 2) (intervals 56 436 0)
(buffers 984 12))
[-- Attachment #2: Type: text/html, Size: 10952 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling
2024-12-08 13:05 bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling rehan malak
@ 2024-12-08 17:37 ` Eli Zaretskii
2024-12-09 14:58 ` rehan malak
2024-12-09 15:04 ` kobarity
0 siblings, 2 replies; 4+ messages in thread
From: Eli Zaretskii @ 2024-12-08 17:37 UTC (permalink / raw)
To: rehan malak, Stefan Monnier, kobarity; +Cc: 74738
> From: rehan malak <rehan.malak@gmail.com>
> Date: Sun, 8 Dec 2024 14:05:36 +0100
>
> I can reproduce systematically this freeze dealing with a 10000lines
> Python file :
>
> wget https://raw.githubusercontent.com/hugsy/gef/refs/heads/main/gef.py
> emacs -Q gef.py
A much smaller reproducer is attached below. Just visiting it freezes
Emacs. But if I change the line marked below:
@classmethod
def is_valid(cls, _: pathlib.Path) -> bool:
raise NotImplementedError
def __str__(self) -> str:
>>>>> return f"{self.name}('{self.path.absolute()}', entry @ {self.entry_point:#x})"
to say this instead:
return self.name
the problem goes away. So it is something in that complex expression
that trips syntax-ppss.
Stefan and kobarity, any suggestions or ideas?
> PageDown several times or scrolling with the mouse
>
> Emacs freezes, Ctrl-g not working, CPU 100%
>
> It works also by searching : Ctrl-s show RET
> then Ctrl-s several times
>
> With a minimal .emacs :
>
> (set debug-on-error t)
> (set debug-on-quit t)
>
> and before scrolling
>
> M-x profiler-start RET
>
> then in an external terminal
>
> pkill -SIGUSR2 emacs
>
> I get the backtrace :
>
> Debugger entered--entering a function:
> * #f(compiled-function () #<bytecode -0x179dcd1db31182ae>)()
> syntax-ppss()
> python-syntax-context-type()
> python-nav-forward-block(-1)
> python-nav-backward-block()
> python-nav-beginning-of-block()
> python-nav-end-of-block()
> python-info-statement-ends-block-p()
> python-info-end-of-block-p()
> python-nav--forward-sexp(1 nil nil)
> python-nav-forward-sexp(1)
> forward-sexp(1)
> up-list(1)
> python--font-lock-f-strings(30419)
> font-lock-fontify-keywords-region(28914 30419 nil)
> font-lock-default-fontify-region(28914 30414 nil)
> font-lock-fontify-region(28914 30414)
> #f(compiled-function (fun) #<bytecode -0x8a96d65e1315e71>)(font-lock-fontify-region)
> run-hook-wrapped(#f(compiled-function (fun) #<bytecode -0x8a96d65e1315e71>) font-lock-fontify-region)
> jit-lock--run-functions(28914 30414)
> jit-lock-fontify-now(28914 30414)
> jit-lock-function(28914)
I see something different, in GDB:
Lisp Backtrace:
"parse-partial-sexp" (0x9f1a788)
"syntax-ppss" (0x9f1a710)
"python-info-line-ends-backslash-p" (0x9f1a6b8)
"python-nav-end-of-statement" (0x9f1a670)
"python-nav-end-of-block" (0x9f1a630)
"python-info-statement-ends-block-p" (0x9f1a610)
"python-info-end-of-block-p" (0x9f1a5c0)
"python-nav--forward-sexp" (0x9f1a548)
"python-nav-forward-sexp" (0x9f1a4f8)
"forward-sexp" (0x9f1a4a8)
"up-list" (0x9f1a438)
"python--font-lock-f-strings" (0x9f1a3a0)
"font-lock-fontify-keywords-region" (0x9f1a308)
"font-lock-default-fontify-region" (0x9f1a2a0)
"font-lock-fontify-region" (0x9f1a230)
0xb9117d0 PVEC_CLOSURE
"run-hook-wrapped" (0x9f1a1c0)
"jit-lock--run-functions" (0x9f1a0e8)
"jit-lock-fontify-now" (0x9f1a058)
"jit-lock-function" (0x5ffba98)
"redisplay_internal (C function)" (0x0)
Here's the file with which I can reproduce the problem?
class FileFormat:
name: str
path: pathlib.Path
entry_point: int
checksec: dict[str, bool]
sections: list[FileFormatSection]
def __init__(self, path: str | pathlib.Path) -> None:
raise NotImplementedError
def __init_subclass__(cls: Type["FileFormat"], **kwargs):
global __registered_file_formats__
super().__init_subclass__(**kwargs)
required_attributes = ("name", "entry_point", "is_valid", "checksec",)
for attr in required_attributes:
if not hasattr(cls, attr):
raise NotImplementedError(f"File format '{cls.__name__}' is invalid: missing attribute '{attr}'")
__registered_file_formats__.add(cls)
return
@classmethod
def is_valid(cls, _: pathlib.Path) -> bool:
raise NotImplementedError
def __str__(self) -> str:
return f"{self.name}('{self.path.absolute()}', entry @ {self.entry_point:#x})"
class Elf(FileFormat):
"""Basic ELF parsing.
Ref:
- http://www.skyfree.org/linux/references/ELF_Format.pdf
- https://refspecs.linuxfoundation.org/elf/elfspec_ppc.pdf
- https://refspecs.linuxfoundation.org/ELF/ppc64/PPC-elf64abi.html
"""
class Class(enum.Enum):
ELF_32_BITS = 0x01
ELF_64_BITS = 0x02
ELF_MAGIC = 0x7f454c46
class Abi(enum.Enum):
X86_64 = 0x3e
X86_32 = 0x03
ARM = 0x28
MIPS = 0x08
POWERPC = 0x14
POWERPC64 = 0x15
SPARC = 0x02
SPARC64 = 0x2b
AARCH64 = 0xb7
RISCV = 0xf3
IA64 = 0x32
M68K = 0x04
class Type(enum.Enum):
ET_RELOC = 1
ET_EXEC = 2
ET_DYN = 3
ET_CORE = 4
class OsAbi(enum.Enum):
SYSTEMV = 0x00
HPUX = 0x01
NETBSD = 0x02
LINUX = 0x03
SOLARIS = 0x06
AIX = 0x07
IRIX = 0x08
FREEBSD = 0x09
OPENBSD = 0x0C
e_magic: int = ELF_MAGIC
e_class: "Elf.Class" = Class.ELF_32_BITS
e_endianness: Endianness = Endianness.LITTLE_ENDIAN
e_eiversion: int
e_osabi: "Elf.OsAbi"
e_abiversion: int
e_pad: bytes
e_type: "Elf.Type" = Type.ET_EXEC
e_machine: Abi = Abi.X86_32
e_version: int
e_entry: int
e_phoff: int
e_shoff: int
e_flags: int
e_ehsize: int
e_phentsize: int
e_phnum: int
e_shentsize: int
e_shnum: int
e_shstrndx: int
path: pathlib.Path
phdrs : list["Phdr"]
shdrs : list["Shdr"]
name: str = "ELF"
__checksec : dict[str, bool]
def __init__(self, path: str | pathlib.Path) -> None:
"""Instantiate an ELF object. A valid ELF must be provided, or an exception will be thrown."""
if isinstance(path, str):
self.path = pathlib.Path(path).expanduser()
elif isinstance(path, pathlib.Path):
self.path = path
else:
raise TypeError
if not self.path.exists():
raise FileNotFoundError(f"'{self.path}' not found/readable, most gef features will not work")
self.__checksec = {}
with self.path.open("rb") as self.fd:
# off 0x0
self.e_magic, e_class, e_endianness, self.e_eiversion = self.read_and_unpack(">IBBB")
if self.e_magic != Elf.ELF_MAGIC:
# The ELF is corrupted, GDB won't handle it, no point going further
raise RuntimeError("Not a valid ELF file (magic)")
self.e_class, self.e_endianness = Elf.Class(e_class), Endianness(e_endianness)
if self.e_endianness != gef.arch.endianness:
warn("Unexpected endianness for architecture")
endian = self.e_endianness
# off 0x7
e_osabi, self.e_abiversion = self.read_and_unpack(f"{endian}BB")
self.e_osabi = Elf.OsAbi(e_osabi)
# off 0x9
self.e_pad = self.read(7)
# off 0x10
e_type, e_machine, self.e_version = self.read_and_unpack(f"{endian}HHI")
self.e_type, self.e_machine = Elf.Type(e_type), Elf.Abi(e_machine)
# off 0x18
if self.e_class == Elf.Class.ELF_64_BITS:
self.e_entry, self.e_phoff, self.e_shoff = self.read_and_unpack(f"{endian}QQQ")
else:
self.e_entry, self.e_phoff, self.e_shoff = self.read_and_unpack(f"{endian}III")
self.e_flags, self.e_ehsize, self.e_phentsize, self.e_phnum = self.read_and_unpack(f"{endian}IHHH")
self.e_shentsize, self.e_shnum, self.e_shstrndx = self.read_and_unpack(f"{endian}HHH")
self.phdrs = []
for i in range(self.e_phnum):
self.phdrs.append(Phdr(self, self.e_phoff + self.e_phentsize * i))
self.shdrs = []
for i in range(self.e_shnum):
self.shdrs.append(Shdr(self, self.e_shoff + self.e_shentsize * i))
return
def read(self, size: int) -> bytes:
return self.fd.read(size)
def read_and_unpack(self, fmt: str) -> tuple[Any, ...]:
size = struct.calcsize(fmt)
data = self.fd.read(size)
return struct.unpack(fmt, data)
def seek(self, off: int) -> None:
self.fd.seek(off, 0)
def __str__(self) -> str:
return f"ELF('{self.path.absolute()}', {self.e_class.name}, {self.e_machine.name})"
def __repr__(self) -> str:
return f"ELF('{self.path.absolute()}', {self.e_class.name}, {self.e_machine.name})"
@property
def entry_point(self) -> int:
return self.e_entry
@classmethod
def is_valid(cls, path: pathlib.Path) -> bool:
return u32(path.open("rb").read(4), e = Endianness.BIG_ENDIAN) == Elf.ELF_MAGIC
@property
def checksec(self) -> dict[str, bool]:
"""Check the security property of the ELF binary. The following properties are:
- Canary
- NX
- PIE
- Fortify
- Partial/Full RelRO.
Return a dict() with the different keys mentioned above, and the boolean
associated whether the protection was found."""
if not self.__checksec:
def __check_security_property(opt: str, filename: str, pattern: str) -> bool:
cmd = [readelf,]
cmd += opt.split()
cmd += [filename,]
lines = gef_execute_external(cmd, as_list=True)
for line in lines:
if re.search(pattern, line):
return True
return False
abspath = str(self.path.absolute())
readelf = gef.session.constants["readelf"]
self.__checksec["Canary"] = __check_security_property("-rs", abspath, r"__stack_chk_fail") is True
has_gnu_stack = __check_security_property("-W -l", abspath, r"GNU_STACK") is True
if has_gnu_stack:
self.__checksec["NX"] = __check_security_property("-W -l", abspath, r"GNU_STACK.*RWE") is False
else:
self.__checksec["NX"] = False
self.__checksec["PIE"] = __check_security_property("-h", abspath, r":.*EXEC") is False
self.__checksec["Fortify"] = __check_security_property("-s", abspath, r"_chk@GLIBC") is True
self.__checksec["Partial RelRO"] = __check_security_property("-l", abspath, r"GNU_RELRO") is True
self.__checksec["Full RelRO"] = self.__checksec["Partial RelRO"] and __check_security_property("-d", abspath, r"BIND_NOW") is True
return self.__checksec
@classproperty
@deprecated("use `Elf.Abi.X86_64`")
def X86_64(cls) -> int: return Elf.Abi.X86_64.value # pylint: disable=no-self-argument
@classproperty
@deprecated("use `Elf.Abi.X86_32`")
def X86_32(cls) -> int : return Elf.Abi.X86_32.value # pylint: disable=no-self-argument
@classproperty
@deprecated("use `Elf.Abi.ARM`")
def ARM(cls) -> int : return Elf.Abi.ARM.value # pylint: disable=no-self-argument
@classproperty
@deprecated("use `Elf.Abi.MIPS`")
def MIPS(cls) -> int : return Elf.Abi.MIPS.value # pylint: disable=no-self-argument
@classproperty
@deprecated("use `Elf.Abi.POWERPC`")
def POWERPC(cls) -> int : return Elf.Abi.POWERPC.value # pylint: disable=no-self-argument
@classproperty
@deprecated("use `Elf.Abi.POWERPC64`")
def POWERPC64(cls) -> int : return Elf.Abi.POWERPC64.value # pylint: disable=no-self-argument
@classproperty
@deprecated("use `Elf.Abi.SPARC`")
def SPARC(cls) -> int : return Elf.Abi.SPARC.value # pylint: disable=no-self-argument
@classproperty
@deprecated("use `Elf.Abi.SPARC64`")
def SPARC64(cls) -> int : return Elf.Abi.SPARC64.value # pylint: disable=no-self-argument
@classproperty
@deprecated("use `Elf.Abi.AARCH64`")
def AARCH64(cls) -> int : return Elf.Abi.AARCH64.value # pylint: disable=no-self-argument
@classproperty
@deprecated("use `Elf.Abi.RISCV`")
def RISCV(cls) -> int : return Elf.Abi.RISCV.value # pylint: disable=no-self-argument
^ permalink raw reply [flat|nested] 4+ messages in thread
* bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling
2024-12-08 17:37 ` Eli Zaretskii
@ 2024-12-09 14:58 ` rehan malak
2024-12-09 15:04 ` kobarity
1 sibling, 0 replies; 4+ messages in thread
From: rehan malak @ 2024-12-09 14:58 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: kobarity, Stefan Monnier, 74738
[-- Attachment #1: Type: text/plain, Size: 14703 bytes --]
Hi,
Deleting all the "#" in the f-string of the gef.py file removes the problem
on my side.
sed -i -e 's/#0{align/0{align/g' gef.py
sed -i -e 's/#0{width/0{width/g' gef.py
sed -i -e 's/#07x}/07x}/g' gef.py
sed -i -e 's/#06x}/06x}/g' gef.py
sed -i -e 's/#04x}/04x}/g' gef.py
sed -i -e 's/#4x}/4x}/g' gef.py
sed -i -e 's/#8x}/8x}/g' gef.py
sed -i -e 's/#10x}/10x}/g' gef.py
sed -i -e 's/#x}/x}/g' gef.py
Your smaller example Eli also contains a f-string with the "#".
This python f-string format specifier is described here:
https://docs.python.org/3/library/string.html#format-specification-mini-language
Is this format specifier supported from the beginning ?
For example :
value = 0xab
print(f"{value:x} is a value")
=> color syntax : *is a value* has the color defined by
font-lock-string-face
while
value = 0xab
print(f"{value:#x} is a value")
=> color are messed up... *is* has color defined by font-lock-keyword-face
and *a value* has color defined by default face
Best,
Rehan
On Sun, Dec 8, 2024 at 6:37 PM Eli Zaretskii <eliz@gnu.org> wrote:
> > From: rehan malak <rehan.malak@gmail.com>
> > Date: Sun, 8 Dec 2024 14:05:36 +0100
> >
> > I can reproduce systematically this freeze dealing with a 10000lines
> > Python file :
> >
> > wget https://raw.githubusercontent.com/hugsy/gef/refs/heads/main/gef.py
> > emacs -Q gef.py
>
> A much smaller reproducer is attached below. Just visiting it freezes
> Emacs. But if I change the line marked below:
>
> @classmethod
> def is_valid(cls, _: pathlib.Path) -> bool:
> raise NotImplementedError
>
> def __str__(self) -> str:
> >>>>> return f"{self.name}('{self.path.absolute()}', entry @
> {self.entry_point:#x})"
>
> to say this instead:
>
> return self.name
>
> the problem goes away. So it is something in that complex expression
> that trips syntax-ppss.
>
> Stefan and kobarity, any suggestions or ideas?
>
> > PageDown several times or scrolling with the mouse
> >
> > Emacs freezes, Ctrl-g not working, CPU 100%
> >
> > It works also by searching : Ctrl-s show RET
> > then Ctrl-s several times
> >
> > With a minimal .emacs :
> >
> > (set debug-on-error t)
> > (set debug-on-quit t)
> >
> > and before scrolling
> >
> > M-x profiler-start RET
> >
> > then in an external terminal
> >
> > pkill -SIGUSR2 emacs
> >
> > I get the backtrace :
> >
> > Debugger entered--entering a function:
> > * #f(compiled-function () #<bytecode -0x179dcd1db31182ae>)()
> > syntax-ppss()
> > python-syntax-context-type()
> > python-nav-forward-block(-1)
> > python-nav-backward-block()
> > python-nav-beginning-of-block()
> > python-nav-end-of-block()
> > python-info-statement-ends-block-p()
> > python-info-end-of-block-p()
> > python-nav--forward-sexp(1 nil nil)
> > python-nav-forward-sexp(1)
> > forward-sexp(1)
> > up-list(1)
> > python--font-lock-f-strings(30419)
> > font-lock-fontify-keywords-region(28914 30419 nil)
> > font-lock-default-fontify-region(28914 30414 nil)
> > font-lock-fontify-region(28914 30414)
> > #f(compiled-function (fun) #<bytecode
> -0x8a96d65e1315e71>)(font-lock-fontify-region)
> > run-hook-wrapped(#f(compiled-function (fun) #<bytecode
> -0x8a96d65e1315e71>) font-lock-fontify-region)
> > jit-lock--run-functions(28914 30414)
> > jit-lock-fontify-now(28914 30414)
> > jit-lock-function(28914)
>
> I see something different, in GDB:
>
> Lisp Backtrace:
> "parse-partial-sexp" (0x9f1a788)
> "syntax-ppss" (0x9f1a710)
> "python-info-line-ends-backslash-p" (0x9f1a6b8)
> "python-nav-end-of-statement" (0x9f1a670)
> "python-nav-end-of-block" (0x9f1a630)
> "python-info-statement-ends-block-p" (0x9f1a610)
> "python-info-end-of-block-p" (0x9f1a5c0)
> "python-nav--forward-sexp" (0x9f1a548)
> "python-nav-forward-sexp" (0x9f1a4f8)
> "forward-sexp" (0x9f1a4a8)
> "up-list" (0x9f1a438)
> "python--font-lock-f-strings" (0x9f1a3a0)
> "font-lock-fontify-keywords-region" (0x9f1a308)
> "font-lock-default-fontify-region" (0x9f1a2a0)
> "font-lock-fontify-region" (0x9f1a230)
> 0xb9117d0 PVEC_CLOSURE
> "run-hook-wrapped" (0x9f1a1c0)
> "jit-lock--run-functions" (0x9f1a0e8)
> "jit-lock-fontify-now" (0x9f1a058)
> "jit-lock-function" (0x5ffba98)
> "redisplay_internal (C function)" (0x0)
>
> Here's the file with which I can reproduce the problem?
>
>
> class FileFormat:
> name: str
> path: pathlib.Path
> entry_point: int
> checksec: dict[str, bool]
> sections: list[FileFormatSection]
>
> def __init__(self, path: str | pathlib.Path) -> None:
> raise NotImplementedError
>
> def __init_subclass__(cls: Type["FileFormat"], **kwargs):
> global __registered_file_formats__
> super().__init_subclass__(**kwargs)
> required_attributes = ("name", "entry_point", "is_valid",
> "checksec",)
> for attr in required_attributes:
> if not hasattr(cls, attr):
> raise NotImplementedError(f"File format '{cls.__name__}'
> is invalid: missing attribute '{attr}'")
> __registered_file_formats__.add(cls)
> return
>
> @classmethod
> def is_valid(cls, _: pathlib.Path) -> bool:
> raise NotImplementedError
>
> def __str__(self) -> str:
> return f"{self.name}('{self.path.absolute()}', entry @
> {self.entry_point:#x})"
>
>
> class Elf(FileFormat):
> """Basic ELF parsing.
> Ref:
> - http://www.skyfree.org/linux/references/ELF_Format.pdf
> - https://refspecs.linuxfoundation.org/elf/elfspec_ppc.pdf
> - https://refspecs.linuxfoundation.org/ELF/ppc64/PPC-elf64abi.html
> """
> class Class(enum.Enum):
> ELF_32_BITS = 0x01
> ELF_64_BITS = 0x02
>
> ELF_MAGIC = 0x7f454c46
>
> class Abi(enum.Enum):
> X86_64 = 0x3e
> X86_32 = 0x03
> ARM = 0x28
> MIPS = 0x08
> POWERPC = 0x14
> POWERPC64 = 0x15
> SPARC = 0x02
> SPARC64 = 0x2b
> AARCH64 = 0xb7
> RISCV = 0xf3
> IA64 = 0x32
> M68K = 0x04
>
> class Type(enum.Enum):
> ET_RELOC = 1
> ET_EXEC = 2
> ET_DYN = 3
> ET_CORE = 4
>
> class OsAbi(enum.Enum):
> SYSTEMV = 0x00
> HPUX = 0x01
> NETBSD = 0x02
> LINUX = 0x03
> SOLARIS = 0x06
> AIX = 0x07
> IRIX = 0x08
> FREEBSD = 0x09
> OPENBSD = 0x0C
>
> e_magic: int = ELF_MAGIC
> e_class: "Elf.Class" = Class.ELF_32_BITS
> e_endianness: Endianness = Endianness.LITTLE_ENDIAN
> e_eiversion: int
> e_osabi: "Elf.OsAbi"
> e_abiversion: int
> e_pad: bytes
> e_type: "Elf.Type" = Type.ET_EXEC
> e_machine: Abi = Abi.X86_32
> e_version: int
> e_entry: int
> e_phoff: int
> e_shoff: int
> e_flags: int
> e_ehsize: int
> e_phentsize: int
> e_phnum: int
> e_shentsize: int
> e_shnum: int
> e_shstrndx: int
>
> path: pathlib.Path
> phdrs : list["Phdr"]
> shdrs : list["Shdr"]
> name: str = "ELF"
>
> __checksec : dict[str, bool]
>
> def __init__(self, path: str | pathlib.Path) -> None:
> """Instantiate an ELF object. A valid ELF must be provided, or an
> exception will be thrown."""
>
> if isinstance(path, str):
> self.path = pathlib.Path(path).expanduser()
> elif isinstance(path, pathlib.Path):
> self.path = path
> else:
> raise TypeError
>
> if not self.path.exists():
> raise FileNotFoundError(f"'{self.path}' not found/readable,
> most gef features will not work")
>
> self.__checksec = {}
>
> with self.path.open("rb") as self.fd:
> # off 0x0
> self.e_magic, e_class, e_endianness, self.e_eiversion =
> self.read_and_unpack(">IBBB")
> if self.e_magic != Elf.ELF_MAGIC:
> # The ELF is corrupted, GDB won't handle it, no point
> going further
> raise RuntimeError("Not a valid ELF file (magic)")
>
> self.e_class, self.e_endianness = Elf.Class(e_class),
> Endianness(e_endianness)
>
> if self.e_endianness != gef.arch.endianness:
> warn("Unexpected endianness for architecture")
>
> endian = self.e_endianness
>
> # off 0x7
> e_osabi, self.e_abiversion =
> self.read_and_unpack(f"{endian}BB")
> self.e_osabi = Elf.OsAbi(e_osabi)
>
> # off 0x9
> self.e_pad = self.read(7)
>
> # off 0x10
> e_type, e_machine, self.e_version =
> self.read_and_unpack(f"{endian}HHI")
> self.e_type, self.e_machine = Elf.Type(e_type),
> Elf.Abi(e_machine)
>
> # off 0x18
> if self.e_class == Elf.Class.ELF_64_BITS:
> self.e_entry, self.e_phoff, self.e_shoff =
> self.read_and_unpack(f"{endian}QQQ")
> else:
> self.e_entry, self.e_phoff, self.e_shoff =
> self.read_and_unpack(f"{endian}III")
>
> self.e_flags, self.e_ehsize, self.e_phentsize, self.e_phnum =
> self.read_and_unpack(f"{endian}IHHH")
> self.e_shentsize, self.e_shnum, self.e_shstrndx =
> self.read_and_unpack(f"{endian}HHH")
>
> self.phdrs = []
> for i in range(self.e_phnum):
> self.phdrs.append(Phdr(self, self.e_phoff +
> self.e_phentsize * i))
>
> self.shdrs = []
> for i in range(self.e_shnum):
> self.shdrs.append(Shdr(self, self.e_shoff +
> self.e_shentsize * i))
> return
>
> def read(self, size: int) -> bytes:
> return self.fd.read(size)
>
> def read_and_unpack(self, fmt: str) -> tuple[Any, ...]:
> size = struct.calcsize(fmt)
> data = self.fd.read(size)
> return struct.unpack(fmt, data)
>
> def seek(self, off: int) -> None:
> self.fd.seek(off, 0)
>
> def __str__(self) -> str:
> return f"ELF('{self.path.absolute()}', {self.e_class.name}, {
> self.e_machine.name})"
>
> def __repr__(self) -> str:
> return f"ELF('{self.path.absolute()}', {self.e_class.name}, {
> self.e_machine.name})"
>
> @property
> def entry_point(self) -> int:
> return self.e_entry
>
> @classmethod
> def is_valid(cls, path: pathlib.Path) -> bool:
> return u32(path.open("rb").read(4), e = Endianness.BIG_ENDIAN) ==
> Elf.ELF_MAGIC
>
> @property
> def checksec(self) -> dict[str, bool]:
> """Check the security property of the ELF binary. The following
> properties are:
> - Canary
> - NX
> - PIE
> - Fortify
> - Partial/Full RelRO.
> Return a dict() with the different keys mentioned above, and the
> boolean
> associated whether the protection was found."""
> if not self.__checksec:
> def __check_security_property(opt: str, filename: str,
> pattern: str) -> bool:
> cmd = [readelf,]
> cmd += opt.split()
> cmd += [filename,]
> lines = gef_execute_external(cmd, as_list=True)
> for line in lines:
> if re.search(pattern, line):
> return True
> return False
>
> abspath = str(self.path.absolute())
> readelf = gef.session.constants["readelf"]
> self.__checksec["Canary"] = __check_security_property("-rs",
> abspath, r"__stack_chk_fail") is True
> has_gnu_stack = __check_security_property("-W -l", abspath,
> r"GNU_STACK") is True
> if has_gnu_stack:
> self.__checksec["NX"] = __check_security_property("-W -l",
> abspath, r"GNU_STACK.*RWE") is False
> else:
> self.__checksec["NX"] = False
> self.__checksec["PIE"] = __check_security_property("-h",
> abspath, r":.*EXEC") is False
> self.__checksec["Fortify"] = __check_security_property("-s",
> abspath, r"_chk@GLIBC") is True
> self.__checksec["Partial RelRO"] =
> __check_security_property("-l", abspath, r"GNU_RELRO") is True
> self.__checksec["Full RelRO"] = self.__checksec["Partial
> RelRO"] and __check_security_property("-d", abspath, r"BIND_NOW") is True
> return self.__checksec
>
> @classproperty
> @deprecated("use `Elf.Abi.X86_64`")
> def X86_64(cls) -> int: return Elf.Abi.X86_64.value # pylint:
> disable=no-self-argument
>
> @classproperty
> @deprecated("use `Elf.Abi.X86_32`")
> def X86_32(cls) -> int : return Elf.Abi.X86_32.value # pylint:
> disable=no-self-argument
>
> @classproperty
> @deprecated("use `Elf.Abi.ARM`")
> def ARM(cls) -> int : return Elf.Abi.ARM.value # pylint:
> disable=no-self-argument
>
> @classproperty
> @deprecated("use `Elf.Abi.MIPS`")
> def MIPS(cls) -> int : return Elf.Abi.MIPS.value # pylint:
> disable=no-self-argument
>
> @classproperty
> @deprecated("use `Elf.Abi.POWERPC`")
> def POWERPC(cls) -> int : return Elf.Abi.POWERPC.value # pylint:
> disable=no-self-argument
>
> @classproperty
> @deprecated("use `Elf.Abi.POWERPC64`")
> def POWERPC64(cls) -> int : return Elf.Abi.POWERPC64.value # pylint:
> disable=no-self-argument
>
> @classproperty
> @deprecated("use `Elf.Abi.SPARC`")
> def SPARC(cls) -> int : return Elf.Abi.SPARC.value # pylint:
> disable=no-self-argument
>
> @classproperty
> @deprecated("use `Elf.Abi.SPARC64`")
> def SPARC64(cls) -> int : return Elf.Abi.SPARC64.value # pylint:
> disable=no-self-argument
>
> @classproperty
> @deprecated("use `Elf.Abi.AARCH64`")
> def AARCH64(cls) -> int : return Elf.Abi.AARCH64.value # pylint:
> disable=no-self-argument
>
> @classproperty
> @deprecated("use `Elf.Abi.RISCV`")
> def RISCV(cls) -> int : return Elf.Abi.RISCV.value # pylint:
> disable=no-self-argument
>
>
[-- Attachment #2: Type: text/html, Size: 19416 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling
2024-12-08 17:37 ` Eli Zaretskii
2024-12-09 14:58 ` rehan malak
@ 2024-12-09 15:04 ` kobarity
1 sibling, 0 replies; 4+ messages in thread
From: kobarity @ 2024-12-09 15:04 UTC (permalink / raw)
To: rehan malak, Eli Zaretskii; +Cc: 74738, Stefan Monnier
rehan malak wrote:
> Please tell me if there is a workaround without losing python syntax color ?
If you can configure Emacs to use tree-sitter, python-ts-mode would
not be affected by this problem.
Eli Zaretskii wrote:
>
> > From: rehan malak <rehan.malak@gmail.com>
> > Date: Sun, 8 Dec 2024 14:05:36 +0100
> >
> > I can reproduce systematically this freeze dealing with a 10000lines
> > Python file :
> >
> > wget https://raw.githubusercontent.com/hugsy/gef/refs/heads/main/gef.py
> > emacs -Q gef.py
It seems to be reproducible after the following commit:
commit 3b3274a85c2f5df21d76d82e0d7740005aa84fdf
Author: Stefan Monnier <monnier@iro.umontreal.ca>
Date: Fri Oct 16 14:03:59 2020 -0400
* lisp/progmodes/python.el: Teach f-strings to `font-lock`
(python--f-string-p, python--font-lock-f-strings): New functions.
(python-font-lock-keywords-maximum-decoration): Use them.
> A much smaller reproducer is attached below. Just visiting it freezes
> Emacs. But if I change the line marked below:
>
> @classmethod
> def is_valid(cls, _: pathlib.Path) -> bool:
> raise NotImplementedError
>
> def __str__(self) -> str:
> >>>>> return f"{self.name}('{self.path.absolute()}', entry @ {self.entry_point:#x})"
>
> to say this instead:
>
> return self.name
>
> the problem goes away. So it is something in that complex expression
> that trips syntax-ppss.
>
> Stefan and kobarity, any suggestions or ideas?
I could not reproduce the freeze, but it took several seconds to visit
the reproducer. It appears to be possible to work around this by
simply removing the format specifier ":#x".
return f"{self.name}('{self.path.absolute()}', entry @ {self.entry_point})"
So there may be a problem with font-lock when there is an f-string
format specifier, but I have not found the root cause yet.
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2024-12-09 15:04 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-12-08 13:05 bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling rehan malak
2024-12-08 17:37 ` Eli Zaretskii
2024-12-09 14:58 ` rehan malak
2024-12-09 15:04 ` kobarity
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).