* bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling
@ 2024-12-08 13:05 rehan malak
2024-12-08 17:37 ` Eli Zaretskii
0 siblings, 1 reply; 5+ messages in thread
From: rehan malak @ 2024-12-08 13:05 UTC (permalink / raw)
To: 74738
[-- Attachment #1: Type: text/plain, Size: 8963 bytes --]
Hi,
I can reproduce systematically this freeze dealing with a 10000lines
Python file :
wget https://raw.githubusercontent.com/hugsy/gef/refs/heads/main/gef.py
emacs -Q gef.py
PageDown several times or scrolling with the mouse
Emacs freezes, Ctrl-g not working, CPU 100%
It works also by searching : Ctrl-s show RET
then Ctrl-s several times
With a minimal .emacs :
(set debug-on-error t)
(set debug-on-quit t)
and before scrolling
M-x profiler-start RET
then in an external terminal
pkill -SIGUSR2 emacs
I get the backtrace :
Debugger entered--entering a function:
* #f(compiled-function () #<bytecode -0x179dcd1db31182ae>)()
syntax-ppss()
python-syntax-context-type()
python-nav-forward-block(-1)
python-nav-backward-block()
python-nav-beginning-of-block()
python-nav-end-of-block()
python-info-statement-ends-block-p()
python-info-end-of-block-p()
python-nav--forward-sexp(1 nil nil)
python-nav-forward-sexp(1)
forward-sexp(1)
up-list(1)
python--font-lock-f-strings(30419)
font-lock-fontify-keywords-region(28914 30419 nil)
font-lock-default-fontify-region(28914 30414 nil)
font-lock-fontify-region(28914 30414)
#f(compiled-function (fun) #<bytecode
-0x8a96d65e1315e71>)(font-lock-fontify-region)
run-hook-wrapped(#f(compiled-function (fun) #<bytecode
-0x8a96d65e1315e71>) font-lock-fontify-region)
jit-lock--run-functions(28914 30414)
jit-lock-fontify-now(28914 30414)
jit-lock-function(28914)
and can look at profiler report
M-x profiler-stop
M-x profiler-report
showing time spent in syntax-ppss
17770 88% - jit-lock-function
17770 88% - jit-lock-fontify-now
17770 88% - jit-lock--run-functions
17770 88% - run-hook-wrapped
17770 88% - #<byte-code-function FA0>
17770 88% - font-lock-fontify-region
17770 88% - font-lock-default-fontify-region
17690 87% - font-lock-fontify-keywords-region
17678 87% - python--font-lock-f-strings
17674 87% - up-list
17674 87% - forward-sexp
17674 87% - python-nav-forward-sexp
17674 87% - python-nav--forward-sexp
9253 45% - python-info-end-of-block-p
9245 45% - python-info-statement-ends-block-p
9221 45% - python-nav-end-of-block
8609 42% - python-nav-beginning-of-block
8557 42% - python-nav-backward-block
8557 42% - python-nav-forward-block
8005 39% - python-syntax-context-type
7997 39% - syntax-ppss
7097 35% parse-partial-sexp
753 3% + #<byte-code-function A23>
376 1% + python-nav-beginning-of-statement
132 0% re-search-backward
24 0% looking-at
28 0% current-indentation
8 0% + python-nav-beginning-of-statement
4 0% python-info-current-line-empty-p
4 0% + python-info-current-line-comment-p
604 2% + python-nav-end-of-statement
4 0% + python-info-current-line-comment-p
4 0% current-indentation
24 0% + python-nav-end-of-statement
8 0% + python-info-end-of-statement-p
8178 40% - python-info-statement-ends-block-p
8150 40% - python-nav-end-of-block
7754 38% - python-nav-beginning-of-block
7738 38% - python-nav-backward-block
7734 38% - python-nav-forward-block
7178 35% - python-syntax-context-type
7174 35% - syntax-ppss
6978 34% parse-partial-sexp
8 0% make-closure
8 0% #<byte-code-function B1E>
4 0% syntax-table
4 0% syntax-ppss--update-stats
412 2% + python-nav-beginning-of-statement
92 0% re-search-backward
44 0% looking-at
12 0% current-indentation
4 0% looking-at
380 1% + python-nav-end-of-statement
16 0% + python-util-forward-comment
28 0% + python-nav-end-of-statement
151 0% + python-nav-end-of-block
28 0% + python-info-beginning-of-block-p
24 0% + python-info-end-of-statement-p
20 0% + python-info-beginning-of-statement-p
8 0% + python-syntax-context-type
8 0% + python-info-statement-starts-block-p
4 0% + syntax-ppss
8 0% + #<byte-code-function 7E0>
4 0% + #<byte-code-function 740>
80 0% + font-lock-fontify-syntactically-region
2279 11% Automatic GC
73 0% + redisplay_internal (C function)
40 0% + command-execute
8 0% + ...
4 0% + mouse--click-1-maybe-follows-link
other people talking about python/profiling/syntax-ppss :
https://www.reddit.com/r/emacs/comments/z0oye9/emacs_freezes_when_opening_a_python_file/
https://www.reddit.com/r/emacs/comments/9h7onq/large_python_files_hanging_with_an_unmatched_quote/
Please tell me how I can help ?
Please tell me if there is a workaround without losing python syntax color ?
Thanks
Rehan
In GNU Emacs 31.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
3.24.43, cairo version 1.18.2) of 2024-12-08 built on cndl
Repository revision: b953bcb17047998c9e41cede7c5e5ffec22209b2
Repository branch: master
System Description: Debian GNU/Linux trixie/sid
Configured using:
'configure --prefix=/home/jean/local/emacs --with-pgtk'
Configured features:
CAIRO DBUS FREETYPE GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG LIBSELINUX
LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PGTK PNG SECCOMP SOUND THREADS
TIFF TOOLKIT_SCROLL_BARS WEBP XIM GTK3 ZLIB
Important settings:
value of $LANG: en_US.UTF-8
value of $XMODIFIERS: @im=ibus
locale-coding-system: utf-8-unix
Major mode: Python
Minor modes in effect:
tooltip-mode: t
global-eldoc-mode: t
eldoc-mode: t
show-paren-mode: t
electric-indent-mode: t
mouse-wheel-mode: t
tool-bar-mode: t
menu-bar-mode: t
file-name-shadow-mode: t
global-font-lock-mode: t
font-lock-mode: t
blink-cursor-mode: t
minibuffer-regexp-mode: t
line-number-mode: t
transient-mark-mode: t
auto-composition-mode: t
auto-encryption-mode: t
auto-compression-mode: t
Load-path shadows:
None found.
Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils cl-seq python
rx project byte-opt gv bytecomp byte-compile compat pcase treesit comint
subr-x ansi-osc ring cl-loaddefs cl-lib ansi-color rmc iso-transl
tooltip cconv eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel term/pgtk-win pgtk-win term/common-win
touch-screen pgtk-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode register
page tab-bar menu-bar rfn-eshadow isearch easymenu timer select
scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer nadvice seq simple cl-generic indonesian philippine
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp
files window text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget keymap hashtable-print-readable backquote
threads dbusbind inotify dynamic-setting system-font-setting
font-render-setting cairo gtk pgtk multi-tty move-toolbar
make-network-process emacs)
Memory information:
((conses 16 60995 13527) (symbols 48 7517 0) (strings 32 19122 1493)
(string-bytes 1 515162) (vectors 16 13554)
(vector-slots 8 145472 8433) (floats 8 26 2) (intervals 56 436 0)
(buffers 984 12))
[-- Attachment #2: Type: text/html, Size: 10952 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread* bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling 2024-12-08 13:05 bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling rehan malak @ 2024-12-08 17:37 ` Eli Zaretskii 2024-12-09 14:58 ` rehan malak 2024-12-09 15:04 ` kobarity 0 siblings, 2 replies; 5+ messages in thread From: Eli Zaretskii @ 2024-12-08 17:37 UTC (permalink / raw) To: rehan malak, Stefan Monnier, kobarity; +Cc: 74738 > From: rehan malak <rehan.malak@gmail.com> > Date: Sun, 8 Dec 2024 14:05:36 +0100 > > I can reproduce systematically this freeze dealing with a 10000lines > Python file : > > wget https://raw.githubusercontent.com/hugsy/gef/refs/heads/main/gef.py > emacs -Q gef.py A much smaller reproducer is attached below. Just visiting it freezes Emacs. But if I change the line marked below: @classmethod def is_valid(cls, _: pathlib.Path) -> bool: raise NotImplementedError def __str__(self) -> str: >>>>> return f"{self.name}('{self.path.absolute()}', entry @ {self.entry_point:#x})" to say this instead: return self.name the problem goes away. So it is something in that complex expression that trips syntax-ppss. Stefan and kobarity, any suggestions or ideas? > PageDown several times or scrolling with the mouse > > Emacs freezes, Ctrl-g not working, CPU 100% > > It works also by searching : Ctrl-s show RET > then Ctrl-s several times > > With a minimal .emacs : > > (set debug-on-error t) > (set debug-on-quit t) > > and before scrolling > > M-x profiler-start RET > > then in an external terminal > > pkill -SIGUSR2 emacs > > I get the backtrace : > > Debugger entered--entering a function: > * #f(compiled-function () #<bytecode -0x179dcd1db31182ae>)() > syntax-ppss() > python-syntax-context-type() > python-nav-forward-block(-1) > python-nav-backward-block() > python-nav-beginning-of-block() > python-nav-end-of-block() > python-info-statement-ends-block-p() > python-info-end-of-block-p() > python-nav--forward-sexp(1 nil nil) > python-nav-forward-sexp(1) > forward-sexp(1) > up-list(1) > python--font-lock-f-strings(30419) > font-lock-fontify-keywords-region(28914 30419 nil) > font-lock-default-fontify-region(28914 30414 nil) > font-lock-fontify-region(28914 30414) > #f(compiled-function (fun) #<bytecode -0x8a96d65e1315e71>)(font-lock-fontify-region) > run-hook-wrapped(#f(compiled-function (fun) #<bytecode -0x8a96d65e1315e71>) font-lock-fontify-region) > jit-lock--run-functions(28914 30414) > jit-lock-fontify-now(28914 30414) > jit-lock-function(28914) I see something different, in GDB: Lisp Backtrace: "parse-partial-sexp" (0x9f1a788) "syntax-ppss" (0x9f1a710) "python-info-line-ends-backslash-p" (0x9f1a6b8) "python-nav-end-of-statement" (0x9f1a670) "python-nav-end-of-block" (0x9f1a630) "python-info-statement-ends-block-p" (0x9f1a610) "python-info-end-of-block-p" (0x9f1a5c0) "python-nav--forward-sexp" (0x9f1a548) "python-nav-forward-sexp" (0x9f1a4f8) "forward-sexp" (0x9f1a4a8) "up-list" (0x9f1a438) "python--font-lock-f-strings" (0x9f1a3a0) "font-lock-fontify-keywords-region" (0x9f1a308) "font-lock-default-fontify-region" (0x9f1a2a0) "font-lock-fontify-region" (0x9f1a230) 0xb9117d0 PVEC_CLOSURE "run-hook-wrapped" (0x9f1a1c0) "jit-lock--run-functions" (0x9f1a0e8) "jit-lock-fontify-now" (0x9f1a058) "jit-lock-function" (0x5ffba98) "redisplay_internal (C function)" (0x0) Here's the file with which I can reproduce the problem? class FileFormat: name: str path: pathlib.Path entry_point: int checksec: dict[str, bool] sections: list[FileFormatSection] def __init__(self, path: str | pathlib.Path) -> None: raise NotImplementedError def __init_subclass__(cls: Type["FileFormat"], **kwargs): global __registered_file_formats__ super().__init_subclass__(**kwargs) required_attributes = ("name", "entry_point", "is_valid", "checksec",) for attr in required_attributes: if not hasattr(cls, attr): raise NotImplementedError(f"File format '{cls.__name__}' is invalid: missing attribute '{attr}'") __registered_file_formats__.add(cls) return @classmethod def is_valid(cls, _: pathlib.Path) -> bool: raise NotImplementedError def __str__(self) -> str: return f"{self.name}('{self.path.absolute()}', entry @ {self.entry_point:#x})" class Elf(FileFormat): """Basic ELF parsing. Ref: - http://www.skyfree.org/linux/references/ELF_Format.pdf - https://refspecs.linuxfoundation.org/elf/elfspec_ppc.pdf - https://refspecs.linuxfoundation.org/ELF/ppc64/PPC-elf64abi.html """ class Class(enum.Enum): ELF_32_BITS = 0x01 ELF_64_BITS = 0x02 ELF_MAGIC = 0x7f454c46 class Abi(enum.Enum): X86_64 = 0x3e X86_32 = 0x03 ARM = 0x28 MIPS = 0x08 POWERPC = 0x14 POWERPC64 = 0x15 SPARC = 0x02 SPARC64 = 0x2b AARCH64 = 0xb7 RISCV = 0xf3 IA64 = 0x32 M68K = 0x04 class Type(enum.Enum): ET_RELOC = 1 ET_EXEC = 2 ET_DYN = 3 ET_CORE = 4 class OsAbi(enum.Enum): SYSTEMV = 0x00 HPUX = 0x01 NETBSD = 0x02 LINUX = 0x03 SOLARIS = 0x06 AIX = 0x07 IRIX = 0x08 FREEBSD = 0x09 OPENBSD = 0x0C e_magic: int = ELF_MAGIC e_class: "Elf.Class" = Class.ELF_32_BITS e_endianness: Endianness = Endianness.LITTLE_ENDIAN e_eiversion: int e_osabi: "Elf.OsAbi" e_abiversion: int e_pad: bytes e_type: "Elf.Type" = Type.ET_EXEC e_machine: Abi = Abi.X86_32 e_version: int e_entry: int e_phoff: int e_shoff: int e_flags: int e_ehsize: int e_phentsize: int e_phnum: int e_shentsize: int e_shnum: int e_shstrndx: int path: pathlib.Path phdrs : list["Phdr"] shdrs : list["Shdr"] name: str = "ELF" __checksec : dict[str, bool] def __init__(self, path: str | pathlib.Path) -> None: """Instantiate an ELF object. A valid ELF must be provided, or an exception will be thrown.""" if isinstance(path, str): self.path = pathlib.Path(path).expanduser() elif isinstance(path, pathlib.Path): self.path = path else: raise TypeError if not self.path.exists(): raise FileNotFoundError(f"'{self.path}' not found/readable, most gef features will not work") self.__checksec = {} with self.path.open("rb") as self.fd: # off 0x0 self.e_magic, e_class, e_endianness, self.e_eiversion = self.read_and_unpack(">IBBB") if self.e_magic != Elf.ELF_MAGIC: # The ELF is corrupted, GDB won't handle it, no point going further raise RuntimeError("Not a valid ELF file (magic)") self.e_class, self.e_endianness = Elf.Class(e_class), Endianness(e_endianness) if self.e_endianness != gef.arch.endianness: warn("Unexpected endianness for architecture") endian = self.e_endianness # off 0x7 e_osabi, self.e_abiversion = self.read_and_unpack(f"{endian}BB") self.e_osabi = Elf.OsAbi(e_osabi) # off 0x9 self.e_pad = self.read(7) # off 0x10 e_type, e_machine, self.e_version = self.read_and_unpack(f"{endian}HHI") self.e_type, self.e_machine = Elf.Type(e_type), Elf.Abi(e_machine) # off 0x18 if self.e_class == Elf.Class.ELF_64_BITS: self.e_entry, self.e_phoff, self.e_shoff = self.read_and_unpack(f"{endian}QQQ") else: self.e_entry, self.e_phoff, self.e_shoff = self.read_and_unpack(f"{endian}III") self.e_flags, self.e_ehsize, self.e_phentsize, self.e_phnum = self.read_and_unpack(f"{endian}IHHH") self.e_shentsize, self.e_shnum, self.e_shstrndx = self.read_and_unpack(f"{endian}HHH") self.phdrs = [] for i in range(self.e_phnum): self.phdrs.append(Phdr(self, self.e_phoff + self.e_phentsize * i)) self.shdrs = [] for i in range(self.e_shnum): self.shdrs.append(Shdr(self, self.e_shoff + self.e_shentsize * i)) return def read(self, size: int) -> bytes: return self.fd.read(size) def read_and_unpack(self, fmt: str) -> tuple[Any, ...]: size = struct.calcsize(fmt) data = self.fd.read(size) return struct.unpack(fmt, data) def seek(self, off: int) -> None: self.fd.seek(off, 0) def __str__(self) -> str: return f"ELF('{self.path.absolute()}', {self.e_class.name}, {self.e_machine.name})" def __repr__(self) -> str: return f"ELF('{self.path.absolute()}', {self.e_class.name}, {self.e_machine.name})" @property def entry_point(self) -> int: return self.e_entry @classmethod def is_valid(cls, path: pathlib.Path) -> bool: return u32(path.open("rb").read(4), e = Endianness.BIG_ENDIAN) == Elf.ELF_MAGIC @property def checksec(self) -> dict[str, bool]: """Check the security property of the ELF binary. The following properties are: - Canary - NX - PIE - Fortify - Partial/Full RelRO. Return a dict() with the different keys mentioned above, and the boolean associated whether the protection was found.""" if not self.__checksec: def __check_security_property(opt: str, filename: str, pattern: str) -> bool: cmd = [readelf,] cmd += opt.split() cmd += [filename,] lines = gef_execute_external(cmd, as_list=True) for line in lines: if re.search(pattern, line): return True return False abspath = str(self.path.absolute()) readelf = gef.session.constants["readelf"] self.__checksec["Canary"] = __check_security_property("-rs", abspath, r"__stack_chk_fail") is True has_gnu_stack = __check_security_property("-W -l", abspath, r"GNU_STACK") is True if has_gnu_stack: self.__checksec["NX"] = __check_security_property("-W -l", abspath, r"GNU_STACK.*RWE") is False else: self.__checksec["NX"] = False self.__checksec["PIE"] = __check_security_property("-h", abspath, r":.*EXEC") is False self.__checksec["Fortify"] = __check_security_property("-s", abspath, r"_chk@GLIBC") is True self.__checksec["Partial RelRO"] = __check_security_property("-l", abspath, r"GNU_RELRO") is True self.__checksec["Full RelRO"] = self.__checksec["Partial RelRO"] and __check_security_property("-d", abspath, r"BIND_NOW") is True return self.__checksec @classproperty @deprecated("use `Elf.Abi.X86_64`") def X86_64(cls) -> int: return Elf.Abi.X86_64.value # pylint: disable=no-self-argument @classproperty @deprecated("use `Elf.Abi.X86_32`") def X86_32(cls) -> int : return Elf.Abi.X86_32.value # pylint: disable=no-self-argument @classproperty @deprecated("use `Elf.Abi.ARM`") def ARM(cls) -> int : return Elf.Abi.ARM.value # pylint: disable=no-self-argument @classproperty @deprecated("use `Elf.Abi.MIPS`") def MIPS(cls) -> int : return Elf.Abi.MIPS.value # pylint: disable=no-self-argument @classproperty @deprecated("use `Elf.Abi.POWERPC`") def POWERPC(cls) -> int : return Elf.Abi.POWERPC.value # pylint: disable=no-self-argument @classproperty @deprecated("use `Elf.Abi.POWERPC64`") def POWERPC64(cls) -> int : return Elf.Abi.POWERPC64.value # pylint: disable=no-self-argument @classproperty @deprecated("use `Elf.Abi.SPARC`") def SPARC(cls) -> int : return Elf.Abi.SPARC.value # pylint: disable=no-self-argument @classproperty @deprecated("use `Elf.Abi.SPARC64`") def SPARC64(cls) -> int : return Elf.Abi.SPARC64.value # pylint: disable=no-self-argument @classproperty @deprecated("use `Elf.Abi.AARCH64`") def AARCH64(cls) -> int : return Elf.Abi.AARCH64.value # pylint: disable=no-self-argument @classproperty @deprecated("use `Elf.Abi.RISCV`") def RISCV(cls) -> int : return Elf.Abi.RISCV.value # pylint: disable=no-self-argument ^ permalink raw reply [flat|nested] 5+ messages in thread
* bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling 2024-12-08 17:37 ` Eli Zaretskii @ 2024-12-09 14:58 ` rehan malak 2024-12-11 14:24 ` kobarity 2024-12-09 15:04 ` kobarity 1 sibling, 1 reply; 5+ messages in thread From: rehan malak @ 2024-12-09 14:58 UTC (permalink / raw) To: Eli Zaretskii; +Cc: kobarity, Stefan Monnier, 74738 [-- Attachment #1: Type: text/plain, Size: 14703 bytes --] Hi, Deleting all the "#" in the f-string of the gef.py file removes the problem on my side. sed -i -e 's/#0{align/0{align/g' gef.py sed -i -e 's/#0{width/0{width/g' gef.py sed -i -e 's/#07x}/07x}/g' gef.py sed -i -e 's/#06x}/06x}/g' gef.py sed -i -e 's/#04x}/04x}/g' gef.py sed -i -e 's/#4x}/4x}/g' gef.py sed -i -e 's/#8x}/8x}/g' gef.py sed -i -e 's/#10x}/10x}/g' gef.py sed -i -e 's/#x}/x}/g' gef.py Your smaller example Eli also contains a f-string with the "#". This python f-string format specifier is described here: https://docs.python.org/3/library/string.html#format-specification-mini-language Is this format specifier supported from the beginning ? For example : value = 0xab print(f"{value:x} is a value") => color syntax : *is a value* has the color defined by font-lock-string-face while value = 0xab print(f"{value:#x} is a value") => color are messed up... *is* has color defined by font-lock-keyword-face and *a value* has color defined by default face Best, Rehan On Sun, Dec 8, 2024 at 6:37 PM Eli Zaretskii <eliz@gnu.org> wrote: > > From: rehan malak <rehan.malak@gmail.com> > > Date: Sun, 8 Dec 2024 14:05:36 +0100 > > > > I can reproduce systematically this freeze dealing with a 10000lines > > Python file : > > > > wget https://raw.githubusercontent.com/hugsy/gef/refs/heads/main/gef.py > > emacs -Q gef.py > > A much smaller reproducer is attached below. Just visiting it freezes > Emacs. But if I change the line marked below: > > @classmethod > def is_valid(cls, _: pathlib.Path) -> bool: > raise NotImplementedError > > def __str__(self) -> str: > >>>>> return f"{self.name}('{self.path.absolute()}', entry @ > {self.entry_point:#x})" > > to say this instead: > > return self.name > > the problem goes away. So it is something in that complex expression > that trips syntax-ppss. > > Stefan and kobarity, any suggestions or ideas? > > > PageDown several times or scrolling with the mouse > > > > Emacs freezes, Ctrl-g not working, CPU 100% > > > > It works also by searching : Ctrl-s show RET > > then Ctrl-s several times > > > > With a minimal .emacs : > > > > (set debug-on-error t) > > (set debug-on-quit t) > > > > and before scrolling > > > > M-x profiler-start RET > > > > then in an external terminal > > > > pkill -SIGUSR2 emacs > > > > I get the backtrace : > > > > Debugger entered--entering a function: > > * #f(compiled-function () #<bytecode -0x179dcd1db31182ae>)() > > syntax-ppss() > > python-syntax-context-type() > > python-nav-forward-block(-1) > > python-nav-backward-block() > > python-nav-beginning-of-block() > > python-nav-end-of-block() > > python-info-statement-ends-block-p() > > python-info-end-of-block-p() > > python-nav--forward-sexp(1 nil nil) > > python-nav-forward-sexp(1) > > forward-sexp(1) > > up-list(1) > > python--font-lock-f-strings(30419) > > font-lock-fontify-keywords-region(28914 30419 nil) > > font-lock-default-fontify-region(28914 30414 nil) > > font-lock-fontify-region(28914 30414) > > #f(compiled-function (fun) #<bytecode > -0x8a96d65e1315e71>)(font-lock-fontify-region) > > run-hook-wrapped(#f(compiled-function (fun) #<bytecode > -0x8a96d65e1315e71>) font-lock-fontify-region) > > jit-lock--run-functions(28914 30414) > > jit-lock-fontify-now(28914 30414) > > jit-lock-function(28914) > > I see something different, in GDB: > > Lisp Backtrace: > "parse-partial-sexp" (0x9f1a788) > "syntax-ppss" (0x9f1a710) > "python-info-line-ends-backslash-p" (0x9f1a6b8) > "python-nav-end-of-statement" (0x9f1a670) > "python-nav-end-of-block" (0x9f1a630) > "python-info-statement-ends-block-p" (0x9f1a610) > "python-info-end-of-block-p" (0x9f1a5c0) > "python-nav--forward-sexp" (0x9f1a548) > "python-nav-forward-sexp" (0x9f1a4f8) > "forward-sexp" (0x9f1a4a8) > "up-list" (0x9f1a438) > "python--font-lock-f-strings" (0x9f1a3a0) > "font-lock-fontify-keywords-region" (0x9f1a308) > "font-lock-default-fontify-region" (0x9f1a2a0) > "font-lock-fontify-region" (0x9f1a230) > 0xb9117d0 PVEC_CLOSURE > "run-hook-wrapped" (0x9f1a1c0) > "jit-lock--run-functions" (0x9f1a0e8) > "jit-lock-fontify-now" (0x9f1a058) > "jit-lock-function" (0x5ffba98) > "redisplay_internal (C function)" (0x0) > > Here's the file with which I can reproduce the problem? > > > class FileFormat: > name: str > path: pathlib.Path > entry_point: int > checksec: dict[str, bool] > sections: list[FileFormatSection] > > def __init__(self, path: str | pathlib.Path) -> None: > raise NotImplementedError > > def __init_subclass__(cls: Type["FileFormat"], **kwargs): > global __registered_file_formats__ > super().__init_subclass__(**kwargs) > required_attributes = ("name", "entry_point", "is_valid", > "checksec",) > for attr in required_attributes: > if not hasattr(cls, attr): > raise NotImplementedError(f"File format '{cls.__name__}' > is invalid: missing attribute '{attr}'") > __registered_file_formats__.add(cls) > return > > @classmethod > def is_valid(cls, _: pathlib.Path) -> bool: > raise NotImplementedError > > def __str__(self) -> str: > return f"{self.name}('{self.path.absolute()}', entry @ > {self.entry_point:#x})" > > > class Elf(FileFormat): > """Basic ELF parsing. > Ref: > - http://www.skyfree.org/linux/references/ELF_Format.pdf > - https://refspecs.linuxfoundation.org/elf/elfspec_ppc.pdf > - https://refspecs.linuxfoundation.org/ELF/ppc64/PPC-elf64abi.html > """ > class Class(enum.Enum): > ELF_32_BITS = 0x01 > ELF_64_BITS = 0x02 > > ELF_MAGIC = 0x7f454c46 > > class Abi(enum.Enum): > X86_64 = 0x3e > X86_32 = 0x03 > ARM = 0x28 > MIPS = 0x08 > POWERPC = 0x14 > POWERPC64 = 0x15 > SPARC = 0x02 > SPARC64 = 0x2b > AARCH64 = 0xb7 > RISCV = 0xf3 > IA64 = 0x32 > M68K = 0x04 > > class Type(enum.Enum): > ET_RELOC = 1 > ET_EXEC = 2 > ET_DYN = 3 > ET_CORE = 4 > > class OsAbi(enum.Enum): > SYSTEMV = 0x00 > HPUX = 0x01 > NETBSD = 0x02 > LINUX = 0x03 > SOLARIS = 0x06 > AIX = 0x07 > IRIX = 0x08 > FREEBSD = 0x09 > OPENBSD = 0x0C > > e_magic: int = ELF_MAGIC > e_class: "Elf.Class" = Class.ELF_32_BITS > e_endianness: Endianness = Endianness.LITTLE_ENDIAN > e_eiversion: int > e_osabi: "Elf.OsAbi" > e_abiversion: int > e_pad: bytes > e_type: "Elf.Type" = Type.ET_EXEC > e_machine: Abi = Abi.X86_32 > e_version: int > e_entry: int > e_phoff: int > e_shoff: int > e_flags: int > e_ehsize: int > e_phentsize: int > e_phnum: int > e_shentsize: int > e_shnum: int > e_shstrndx: int > > path: pathlib.Path > phdrs : list["Phdr"] > shdrs : list["Shdr"] > name: str = "ELF" > > __checksec : dict[str, bool] > > def __init__(self, path: str | pathlib.Path) -> None: > """Instantiate an ELF object. A valid ELF must be provided, or an > exception will be thrown.""" > > if isinstance(path, str): > self.path = pathlib.Path(path).expanduser() > elif isinstance(path, pathlib.Path): > self.path = path > else: > raise TypeError > > if not self.path.exists(): > raise FileNotFoundError(f"'{self.path}' not found/readable, > most gef features will not work") > > self.__checksec = {} > > with self.path.open("rb") as self.fd: > # off 0x0 > self.e_magic, e_class, e_endianness, self.e_eiversion = > self.read_and_unpack(">IBBB") > if self.e_magic != Elf.ELF_MAGIC: > # The ELF is corrupted, GDB won't handle it, no point > going further > raise RuntimeError("Not a valid ELF file (magic)") > > self.e_class, self.e_endianness = Elf.Class(e_class), > Endianness(e_endianness) > > if self.e_endianness != gef.arch.endianness: > warn("Unexpected endianness for architecture") > > endian = self.e_endianness > > # off 0x7 > e_osabi, self.e_abiversion = > self.read_and_unpack(f"{endian}BB") > self.e_osabi = Elf.OsAbi(e_osabi) > > # off 0x9 > self.e_pad = self.read(7) > > # off 0x10 > e_type, e_machine, self.e_version = > self.read_and_unpack(f"{endian}HHI") > self.e_type, self.e_machine = Elf.Type(e_type), > Elf.Abi(e_machine) > > # off 0x18 > if self.e_class == Elf.Class.ELF_64_BITS: > self.e_entry, self.e_phoff, self.e_shoff = > self.read_and_unpack(f"{endian}QQQ") > else: > self.e_entry, self.e_phoff, self.e_shoff = > self.read_and_unpack(f"{endian}III") > > self.e_flags, self.e_ehsize, self.e_phentsize, self.e_phnum = > self.read_and_unpack(f"{endian}IHHH") > self.e_shentsize, self.e_shnum, self.e_shstrndx = > self.read_and_unpack(f"{endian}HHH") > > self.phdrs = [] > for i in range(self.e_phnum): > self.phdrs.append(Phdr(self, self.e_phoff + > self.e_phentsize * i)) > > self.shdrs = [] > for i in range(self.e_shnum): > self.shdrs.append(Shdr(self, self.e_shoff + > self.e_shentsize * i)) > return > > def read(self, size: int) -> bytes: > return self.fd.read(size) > > def read_and_unpack(self, fmt: str) -> tuple[Any, ...]: > size = struct.calcsize(fmt) > data = self.fd.read(size) > return struct.unpack(fmt, data) > > def seek(self, off: int) -> None: > self.fd.seek(off, 0) > > def __str__(self) -> str: > return f"ELF('{self.path.absolute()}', {self.e_class.name}, { > self.e_machine.name})" > > def __repr__(self) -> str: > return f"ELF('{self.path.absolute()}', {self.e_class.name}, { > self.e_machine.name})" > > @property > def entry_point(self) -> int: > return self.e_entry > > @classmethod > def is_valid(cls, path: pathlib.Path) -> bool: > return u32(path.open("rb").read(4), e = Endianness.BIG_ENDIAN) == > Elf.ELF_MAGIC > > @property > def checksec(self) -> dict[str, bool]: > """Check the security property of the ELF binary. The following > properties are: > - Canary > - NX > - PIE > - Fortify > - Partial/Full RelRO. > Return a dict() with the different keys mentioned above, and the > boolean > associated whether the protection was found.""" > if not self.__checksec: > def __check_security_property(opt: str, filename: str, > pattern: str) -> bool: > cmd = [readelf,] > cmd += opt.split() > cmd += [filename,] > lines = gef_execute_external(cmd, as_list=True) > for line in lines: > if re.search(pattern, line): > return True > return False > > abspath = str(self.path.absolute()) > readelf = gef.session.constants["readelf"] > self.__checksec["Canary"] = __check_security_property("-rs", > abspath, r"__stack_chk_fail") is True > has_gnu_stack = __check_security_property("-W -l", abspath, > r"GNU_STACK") is True > if has_gnu_stack: > self.__checksec["NX"] = __check_security_property("-W -l", > abspath, r"GNU_STACK.*RWE") is False > else: > self.__checksec["NX"] = False > self.__checksec["PIE"] = __check_security_property("-h", > abspath, r":.*EXEC") is False > self.__checksec["Fortify"] = __check_security_property("-s", > abspath, r"_chk@GLIBC") is True > self.__checksec["Partial RelRO"] = > __check_security_property("-l", abspath, r"GNU_RELRO") is True > self.__checksec["Full RelRO"] = self.__checksec["Partial > RelRO"] and __check_security_property("-d", abspath, r"BIND_NOW") is True > return self.__checksec > > @classproperty > @deprecated("use `Elf.Abi.X86_64`") > def X86_64(cls) -> int: return Elf.Abi.X86_64.value # pylint: > disable=no-self-argument > > @classproperty > @deprecated("use `Elf.Abi.X86_32`") > def X86_32(cls) -> int : return Elf.Abi.X86_32.value # pylint: > disable=no-self-argument > > @classproperty > @deprecated("use `Elf.Abi.ARM`") > def ARM(cls) -> int : return Elf.Abi.ARM.value # pylint: > disable=no-self-argument > > @classproperty > @deprecated("use `Elf.Abi.MIPS`") > def MIPS(cls) -> int : return Elf.Abi.MIPS.value # pylint: > disable=no-self-argument > > @classproperty > @deprecated("use `Elf.Abi.POWERPC`") > def POWERPC(cls) -> int : return Elf.Abi.POWERPC.value # pylint: > disable=no-self-argument > > @classproperty > @deprecated("use `Elf.Abi.POWERPC64`") > def POWERPC64(cls) -> int : return Elf.Abi.POWERPC64.value # pylint: > disable=no-self-argument > > @classproperty > @deprecated("use `Elf.Abi.SPARC`") > def SPARC(cls) -> int : return Elf.Abi.SPARC.value # pylint: > disable=no-self-argument > > @classproperty > @deprecated("use `Elf.Abi.SPARC64`") > def SPARC64(cls) -> int : return Elf.Abi.SPARC64.value # pylint: > disable=no-self-argument > > @classproperty > @deprecated("use `Elf.Abi.AARCH64`") > def AARCH64(cls) -> int : return Elf.Abi.AARCH64.value # pylint: > disable=no-self-argument > > @classproperty > @deprecated("use `Elf.Abi.RISCV`") > def RISCV(cls) -> int : return Elf.Abi.RISCV.value # pylint: > disable=no-self-argument > > [-- Attachment #2: Type: text/html, Size: 19416 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
* bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling 2024-12-09 14:58 ` rehan malak @ 2024-12-11 14:24 ` kobarity 0 siblings, 0 replies; 5+ messages in thread From: kobarity @ 2024-12-11 14:24 UTC (permalink / raw) To: rehan malak, Eli Zaretskii; +Cc: 74738, Stefan Monnier [-- Attachment #1: Type: text/plain, Size: 1682 bytes --] rehan malak wrote: > Deleting all the "#" in the f-string of the gef.py file removes the problem on my side. > sed -i -e 's/#0{align/0{align/g' gef.py > sed -i -e 's/#0{width/0{width/g' gef.py > sed -i -e 's/#07x}/07x}/g' gef.py > sed -i -e 's/#06x}/06x}/g' gef.py > sed -i -e 's/#04x}/04x}/g' gef.py > sed -i -e 's/#4x}/4x}/g' gef.py > sed -i -e 's/#8x}/8x}/g' gef.py > sed -i -e 's/#10x}/10x}/g' gef.py > sed -i -e 's/#x}/x}/g' gef.py > > Your smaller example Eli also contains a f-string with the "#". > > This python f-string format specifier is described here: > https://docs.python.org/3/library/string.html#format-specification-mini-language > > Is this format specifier supported from the beginning ? > > For example : > > value = 0xab > print(f"{value:x} is a value") > > => color syntax : is a value has the color defined by font-lock-string-face > > while > > value = 0xab > print(f"{value:#x} is a value") > > => color are messed up... is has color defined by font-lock-keyword-face and a value has color defined by default face The mechanism of the hang is still unclear, but I found the cause of the font-lock malfunction: when using `up-list' to find the corresponding closing brace for an opening brace in an f-string, if the format specifier contains "#", the rest of the string is considered a comment and the search fails. Therefore, it can be worked around by temporarily binding `parse-sexp-ignore-comments' to nil, as in the attached patch. Hangs on gef.py can also be avoided with this patch, but it assumes that the braces are properly closed. If you remove the closing braces of expressions containing "#", it will eventually hang. [-- Attachment #2: 0001-Fix-font-lock-of-Python-f-strings.patch --] [-- Type: text/plain, Size: 1999 bytes --] From 697f942d70e40bcd9bd32596313a9a371fffe35c Mon Sep 17 00:00:00 2001 From: kobarity <kobarity@gmail.com> Date: Wed, 11 Dec 2024 23:21:04 +0900 Subject: [PATCH] Fix font-lock of Python f-strings * lisp/progmodes/python.el (python--font-lock-f-strings): Bind 'parse-sexp-ignore-comments' to nil so that we can look for closing braces even if a hash is used in the format specifier. * test/lisp/progmodes/python-tests.el (python-font-lock-f-string-1): New test. (Bug#74738) --- lisp/progmodes/python.el | 4 +++- test/lisp/progmodes/python-tests.el | 8 ++++++++ 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el index cfa3cc59568..827f437013f 100644 --- a/lisp/progmodes/python.el +++ b/lisp/progmodes/python.el @@ -631,7 +631,9 @@ python--font-lock-f-strings (forward-char 1) ;Just skip over {{ (let ((beg (match-beginning 0)) (end (condition-case nil - (progn (up-list 1) (min send (point))) + (let ((parse-sexp-ignore-comments)) + (up-list 1) + (min send (point))) (scan-error send)))) (goto-char end) (put-text-property beg end 'face nil)))) diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el index 4cc4040d0ff..ac17c5b1263 100644 --- a/test/lisp/progmodes/python-tests.el +++ b/test/lisp/progmodes/python-tests.el @@ -709,6 +709,14 @@ python-font-lock-string-literal-concatenation (17 . font-lock-operator-face) (18) (19 . font-lock-string-face)))) +(ert-deftest python-font-lock-f-string-1 () + "Test for bug#74738." + (python-tests-assert-faces + "print(f\"{value:#x} is a value\")" + '((1 . font-lock-builtin-face) (6) + (8 . font-lock-string-face) (9) + (19 . font-lock-string-face) (31)))) + \f ;;; Indentation -- 2.43.0 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling 2024-12-08 17:37 ` Eli Zaretskii 2024-12-09 14:58 ` rehan malak @ 2024-12-09 15:04 ` kobarity 1 sibling, 0 replies; 5+ messages in thread From: kobarity @ 2024-12-09 15:04 UTC (permalink / raw) To: rehan malak, Eli Zaretskii; +Cc: 74738, Stefan Monnier rehan malak wrote: > Please tell me if there is a workaround without losing python syntax color ? If you can configure Emacs to use tree-sitter, python-ts-mode would not be affected by this problem. Eli Zaretskii wrote: > > > From: rehan malak <rehan.malak@gmail.com> > > Date: Sun, 8 Dec 2024 14:05:36 +0100 > > > > I can reproduce systematically this freeze dealing with a 10000lines > > Python file : > > > > wget https://raw.githubusercontent.com/hugsy/gef/refs/heads/main/gef.py > > emacs -Q gef.py It seems to be reproducible after the following commit: commit 3b3274a85c2f5df21d76d82e0d7740005aa84fdf Author: Stefan Monnier <monnier@iro.umontreal.ca> Date: Fri Oct 16 14:03:59 2020 -0400 * lisp/progmodes/python.el: Teach f-strings to `font-lock` (python--f-string-p, python--font-lock-f-strings): New functions. (python-font-lock-keywords-maximum-decoration): Use them. > A much smaller reproducer is attached below. Just visiting it freezes > Emacs. But if I change the line marked below: > > @classmethod > def is_valid(cls, _: pathlib.Path) -> bool: > raise NotImplementedError > > def __str__(self) -> str: > >>>>> return f"{self.name}('{self.path.absolute()}', entry @ {self.entry_point:#x})" > > to say this instead: > > return self.name > > the problem goes away. So it is something in that complex expression > that trips syntax-ppss. > > Stefan and kobarity, any suggestions or ideas? I could not reproduce the freeze, but it took several seconds to visit the reproducer. It appears to be possible to work around this by simply removing the format specifier ":#x". return f"{self.name}('{self.path.absolute()}', entry @ {self.entry_point})" So there may be a problem with font-lock when there is an f-string format specifier, but I have not found the root cause yet. ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2024-12-11 14:24 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-12-08 13:05 bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling rehan malak 2024-12-08 17:37 ` Eli Zaretskii 2024-12-09 14:58 ` rehan malak 2024-12-11 14:24 ` kobarity 2024-12-09 15:04 ` kobarity
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).