bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling

bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling

[rehan malak]

[-- Attachment #1: Type: text/plain, Size: 8963 bytes --]

Hi,
I can reproduce systematically this freeze dealing with a 10000lines
Python file :

wget https://raw.githubusercontent.com/hugsy/gef/refs/heads/main/gef.py
emacs -Q gef.py

PageDown several times or scrolling with the mouse

Emacs freezes, Ctrl-g not working, CPU 100%

It works also by searching : Ctrl-s show RET
then Ctrl-s several times




With a minimal .emacs :

(set debug-on-error t)
(set debug-on-quit t)

and before scrolling

M-x profiler-start RET

then in an external terminal

pkill -SIGUSR2 emacs

I get the backtrace :

Debugger entered--entering a function:
* #f(compiled-function () #<bytecode -0x179dcd1db31182ae>)()
  syntax-ppss()
  python-syntax-context-type()
  python-nav-forward-block(-1)
  python-nav-backward-block()
  python-nav-beginning-of-block()
  python-nav-end-of-block()
  python-info-statement-ends-block-p()
  python-info-end-of-block-p()
  python-nav--forward-sexp(1 nil nil)
  python-nav-forward-sexp(1)
  forward-sexp(1)
  up-list(1)
  python--font-lock-f-strings(30419)
  font-lock-fontify-keywords-region(28914 30419 nil)
  font-lock-default-fontify-region(28914 30414 nil)
  font-lock-fontify-region(28914 30414)
  #f(compiled-function (fun) #<bytecode
-0x8a96d65e1315e71>)(font-lock-fontify-region)
  run-hook-wrapped(#f(compiled-function (fun) #<bytecode
-0x8a96d65e1315e71>) font-lock-fontify-region)
  jit-lock--run-functions(28914 30414)
  jit-lock-fontify-now(28914 30414)
  jit-lock-function(28914)

and can look at profiler report

M-x profiler-stop
M-x profiler-report

showing time spent in syntax-ppss


       17770  88% - jit-lock-function
       17770  88%  - jit-lock-fontify-now
       17770  88%   - jit-lock--run-functions
       17770  88%    - run-hook-wrapped
       17770  88%     - #<byte-code-function FA0>
       17770  88%      - font-lock-fontify-region
       17770  88%       - font-lock-default-fontify-region
       17690  87%        - font-lock-fontify-keywords-region
       17678  87%         - python--font-lock-f-strings
       17674  87%          - up-list
       17674  87%           - forward-sexp
       17674  87%            - python-nav-forward-sexp
       17674  87%             - python-nav--forward-sexp
        9253  45%              - python-info-end-of-block-p
        9245  45%               - python-info-statement-ends-block-p
        9221  45%                - python-nav-end-of-block
        8609  42%                 - python-nav-beginning-of-block
        8557  42%                  - python-nav-backward-block
        8557  42%                   - python-nav-forward-block
        8005  39%                    - python-syntax-context-type
        7997  39%                     - syntax-ppss
        7097  35%                        parse-partial-sexp
         753   3%                      + #<byte-code-function A23>
         376   1%                    + python-nav-beginning-of-statement
         132   0%                      re-search-backward
          24   0%                      looking-at
          28   0%                    current-indentation
           8   0%                  + python-nav-beginning-of-statement
           4   0%                    python-info-current-line-empty-p
           4   0%                  + python-info-current-line-comment-p
         604   2%                 + python-nav-end-of-statement
           4   0%                 + python-info-current-line-comment-p
           4   0%                   current-indentation
          24   0%                + python-nav-end-of-statement
           8   0%               + python-info-end-of-statement-p
        8178  40%              - python-info-statement-ends-block-p
        8150  40%               - python-nav-end-of-block
        7754  38%                - python-nav-beginning-of-block
        7738  38%                 - python-nav-backward-block
        7734  38%                  - python-nav-forward-block
        7178  35%                   - python-syntax-context-type
        7174  35%                    - syntax-ppss
        6978  34%                       parse-partial-sexp
           8   0%                       make-closure
           8   0%                       #<byte-code-function B1E>
           4   0%                       syntax-table
           4   0%                       syntax-ppss--update-stats
         412   2%                   + python-nav-beginning-of-statement
          92   0%                     re-search-backward
          44   0%                     looking-at
          12   0%                   current-indentation
           4   0%                   looking-at
         380   1%                + python-nav-end-of-statement
          16   0%                + python-util-forward-comment
          28   0%               + python-nav-end-of-statement
         151   0%              + python-nav-end-of-block
          28   0%              + python-info-beginning-of-block-p
          24   0%              + python-info-end-of-statement-p
          20   0%              + python-info-beginning-of-statement-p
           8   0%              + python-syntax-context-type
           8   0%              + python-info-statement-starts-block-p
           4   0%          + syntax-ppss
           8   0%         + #<byte-code-function 7E0>
           4   0%         + #<byte-code-function 740>
          80   0%        + font-lock-fontify-syntactically-region
        2279  11%   Automatic GC
          73   0% + redisplay_internal (C function)
          40   0% + command-execute
           8   0% + ...
           4   0% + mouse--click-1-maybe-follows-link

other people talking about python/profiling/syntax-ppss :

https://www.reddit.com/r/emacs/comments/z0oye9/emacs_freezes_when_opening_a_python_file/
https://www.reddit.com/r/emacs/comments/9h7onq/large_python_files_hanging_with_an_unmatched_quote/

Please tell me how I can help ?

Please tell me if there is a workaround without losing python syntax color ?

Thanks

Rehan



In GNU Emacs 31.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
 3.24.43, cairo version 1.18.2) of 2024-12-08 built on cndl
Repository revision: b953bcb17047998c9e41cede7c5e5ffec22209b2
Repository branch: master
System Description: Debian GNU/Linux trixie/sid

Configured using:
 'configure --prefix=/home/jean/local/emacs --with-pgtk'

Configured features:
CAIRO DBUS FREETYPE GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG LIBSELINUX
LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PGTK PNG SECCOMP SOUND THREADS
TIFF TOOLKIT_SCROLL_BARS WEBP XIM GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Python

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  minibuffer-regexp-mode: t
  line-number-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils cl-seq python
rx project byte-opt gv bytecomp byte-compile compat pcase treesit comint
subr-x ansi-osc ring cl-loaddefs cl-lib ansi-color rmc iso-transl
tooltip cconv eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel term/pgtk-win pgtk-win term/common-win
touch-screen pgtk-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode register
page tab-bar menu-bar rfn-eshadow isearch easymenu timer select
scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer nadvice seq simple cl-generic indonesian philippine
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp
files window text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget keymap hashtable-print-readable backquote
threads dbusbind inotify dynamic-setting system-font-setting
font-render-setting cairo gtk pgtk multi-tty move-toolbar
make-network-process emacs)

Memory information:
((conses 16 60995 13527) (symbols 48 7517 0) (strings 32 19122 1493)
 (string-bytes 1 515162) (vectors 16 13554)
 (vector-slots 8 145472 8433) (floats 8 26 2) (intervals 56 436 0)
 (buffers 984 12))

[-- Attachment #2: Type: text/html, Size: 10952 bytes --]

bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling

[Eli Zaretskii]

> From: rehan malak <rehan.malak@gmail.com>
> Date: Sun, 8 Dec 2024 14:05:36 +0100
> 
> I can reproduce systematically this freeze dealing with a 10000lines
> Python file :
> 
> wget https://raw.githubusercontent.com/hugsy/gef/refs/heads/main/gef.py
> emacs -Q gef.py

A much smaller reproducer is attached below.  Just visiting it freezes
Emacs.  But if I change the line marked below:

    @classmethod
    def is_valid(cls, _: pathlib.Path) -> bool:
        raise NotImplementedError

    def __str__(self) -> str:
  >>>>> return f"{self.name}('{self.path.absolute()}', entry @ {self.entry_point:#x})"

to say this instead:

        return self.name

the problem goes away.  So it is something in that complex expression
that trips syntax-ppss.

Stefan and kobarity, any suggestions or ideas?

> PageDown several times or scrolling with the mouse
> 
> Emacs freezes, Ctrl-g not working, CPU 100%
> 
> It works also by searching : Ctrl-s show RET
> then Ctrl-s several times
> 
> With a minimal .emacs :
> 
> (set debug-on-error t) 
> (set debug-on-quit t) 
> 
> and before scrolling
> 
> M-x profiler-start RET
> 
> then in an external terminal
> 
> pkill -SIGUSR2 emacs
> 
> I get the backtrace :
> 
> Debugger entered--entering a function:
> * #f(compiled-function () #<bytecode -0x179dcd1db31182ae>)()
>   syntax-ppss()
>   python-syntax-context-type()
>   python-nav-forward-block(-1)
>   python-nav-backward-block()
>   python-nav-beginning-of-block()
>   python-nav-end-of-block()
>   python-info-statement-ends-block-p()
>   python-info-end-of-block-p()
>   python-nav--forward-sexp(1 nil nil)
>   python-nav-forward-sexp(1)
>   forward-sexp(1)
>   up-list(1)
>   python--font-lock-f-strings(30419)
>   font-lock-fontify-keywords-region(28914 30419 nil)
>   font-lock-default-fontify-region(28914 30414 nil)
>   font-lock-fontify-region(28914 30414)
>   #f(compiled-function (fun) #<bytecode -0x8a96d65e1315e71>)(font-lock-fontify-region)
>   run-hook-wrapped(#f(compiled-function (fun) #<bytecode -0x8a96d65e1315e71>) font-lock-fontify-region)
>   jit-lock--run-functions(28914 30414)
>   jit-lock-fontify-now(28914 30414)
>   jit-lock-function(28914)

I see something different, in GDB:

  Lisp Backtrace:
  "parse-partial-sexp" (0x9f1a788)
  "syntax-ppss" (0x9f1a710)
  "python-info-line-ends-backslash-p" (0x9f1a6b8)
  "python-nav-end-of-statement" (0x9f1a670)
  "python-nav-end-of-block" (0x9f1a630)
  "python-info-statement-ends-block-p" (0x9f1a610)
  "python-info-end-of-block-p" (0x9f1a5c0)
  "python-nav--forward-sexp" (0x9f1a548)
  "python-nav-forward-sexp" (0x9f1a4f8)
  "forward-sexp" (0x9f1a4a8)
  "up-list" (0x9f1a438)
  "python--font-lock-f-strings" (0x9f1a3a0)
  "font-lock-fontify-keywords-region" (0x9f1a308)
  "font-lock-default-fontify-region" (0x9f1a2a0)
  "font-lock-fontify-region" (0x9f1a230)
  0xb9117d0 PVEC_CLOSURE
  "run-hook-wrapped" (0x9f1a1c0)
  "jit-lock--run-functions" (0x9f1a0e8)
  "jit-lock-fontify-now" (0x9f1a058)
  "jit-lock-function" (0x5ffba98)
  "redisplay_internal (C function)" (0x0)

Here's the file with which I can reproduce the problem?


class FileFormat:
    name: str
    path: pathlib.Path
    entry_point: int
    checksec: dict[str, bool]
    sections: list[FileFormatSection]

    def __init__(self, path: str | pathlib.Path) -> None:
        raise NotImplementedError

    def __init_subclass__(cls: Type["FileFormat"], **kwargs):
        global __registered_file_formats__
        super().__init_subclass__(**kwargs)
        required_attributes = ("name", "entry_point", "is_valid", "checksec",)
        for attr in required_attributes:
            if not hasattr(cls, attr):
                raise NotImplementedError(f"File format '{cls.__name__}' is invalid: missing attribute '{attr}'")
        __registered_file_formats__.add(cls)
        return

    @classmethod
    def is_valid(cls, _: pathlib.Path) -> bool:
        raise NotImplementedError

    def __str__(self) -> str:
        return f"{self.name}('{self.path.absolute()}', entry @ {self.entry_point:#x})"


class Elf(FileFormat):
    """Basic ELF parsing.
    Ref:
    - http://www.skyfree.org/linux/references/ELF_Format.pdf
    - https://refspecs.linuxfoundation.org/elf/elfspec_ppc.pdf
    - https://refspecs.linuxfoundation.org/ELF/ppc64/PPC-elf64abi.html
    """
    class Class(enum.Enum):
        ELF_32_BITS       = 0x01
        ELF_64_BITS       = 0x02

    ELF_MAGIC         = 0x7f454c46

    class Abi(enum.Enum):
        X86_64            = 0x3e
        X86_32            = 0x03
        ARM               = 0x28
        MIPS              = 0x08
        POWERPC           = 0x14
        POWERPC64         = 0x15
        SPARC             = 0x02
        SPARC64           = 0x2b
        AARCH64           = 0xb7
        RISCV             = 0xf3
        IA64              = 0x32
        M68K              = 0x04

    class Type(enum.Enum):
        ET_RELOC          = 1
        ET_EXEC           = 2
        ET_DYN            = 3
        ET_CORE           = 4

    class OsAbi(enum.Enum):
        SYSTEMV     = 0x00
        HPUX        = 0x01
        NETBSD      = 0x02
        LINUX       = 0x03
        SOLARIS     = 0x06
        AIX         = 0x07
        IRIX        = 0x08
        FREEBSD     = 0x09
        OPENBSD     = 0x0C

    e_magic: int                = ELF_MAGIC
    e_class: "Elf.Class"        = Class.ELF_32_BITS
    e_endianness: Endianness    = Endianness.LITTLE_ENDIAN
    e_eiversion: int
    e_osabi: "Elf.OsAbi"
    e_abiversion: int
    e_pad: bytes
    e_type: "Elf.Type"          = Type.ET_EXEC
    e_machine: Abi              = Abi.X86_32
    e_version: int
    e_entry: int
    e_phoff: int
    e_shoff: int
    e_flags: int
    e_ehsize: int
    e_phentsize: int
    e_phnum: int
    e_shentsize: int
    e_shnum: int
    e_shstrndx: int

    path: pathlib.Path
    phdrs : list["Phdr"]
    shdrs : list["Shdr"]
    name: str = "ELF"

    __checksec : dict[str, bool]

    def __init__(self, path: str | pathlib.Path) -> None:
        """Instantiate an ELF object. A valid ELF must be provided, or an exception will be thrown."""

        if isinstance(path, str):
            self.path = pathlib.Path(path).expanduser()
        elif isinstance(path, pathlib.Path):
            self.path = path
        else:
            raise TypeError

        if not self.path.exists():
            raise FileNotFoundError(f"'{self.path}' not found/readable, most gef features will not work")

        self.__checksec = {}

        with self.path.open("rb") as self.fd:
            # off 0x0
            self.e_magic, e_class, e_endianness, self.e_eiversion = self.read_and_unpack(">IBBB")
            if self.e_magic != Elf.ELF_MAGIC:
                # The ELF is corrupted, GDB won't handle it, no point going further
                raise RuntimeError("Not a valid ELF file (magic)")

            self.e_class, self.e_endianness = Elf.Class(e_class), Endianness(e_endianness)

            if self.e_endianness != gef.arch.endianness:
                warn("Unexpected endianness for architecture")

            endian = self.e_endianness

            # off 0x7
            e_osabi, self.e_abiversion = self.read_and_unpack(f"{endian}BB")
            self.e_osabi = Elf.OsAbi(e_osabi)

            # off 0x9
            self.e_pad = self.read(7)

            # off 0x10
            e_type, e_machine, self.e_version = self.read_and_unpack(f"{endian}HHI")
            self.e_type, self.e_machine = Elf.Type(e_type), Elf.Abi(e_machine)

            # off 0x18
            if self.e_class == Elf.Class.ELF_64_BITS:
                self.e_entry, self.e_phoff, self.e_shoff = self.read_and_unpack(f"{endian}QQQ")
            else:
                self.e_entry, self.e_phoff, self.e_shoff = self.read_and_unpack(f"{endian}III")

            self.e_flags, self.e_ehsize, self.e_phentsize, self.e_phnum = self.read_and_unpack(f"{endian}IHHH")
            self.e_shentsize, self.e_shnum, self.e_shstrndx = self.read_and_unpack(f"{endian}HHH")

            self.phdrs = []
            for i in range(self.e_phnum):
                self.phdrs.append(Phdr(self, self.e_phoff + self.e_phentsize * i))

            self.shdrs = []
            for i in range(self.e_shnum):
                self.shdrs.append(Shdr(self, self.e_shoff + self.e_shentsize * i))
        return

    def read(self, size: int) -> bytes:
        return self.fd.read(size)

    def read_and_unpack(self, fmt: str) -> tuple[Any, ...]:
        size = struct.calcsize(fmt)
        data = self.fd.read(size)
        return struct.unpack(fmt, data)

    def seek(self, off: int) -> None:
        self.fd.seek(off, 0)

    def __str__(self) -> str:
        return f"ELF('{self.path.absolute()}', {self.e_class.name}, {self.e_machine.name})"

    def __repr__(self) -> str:
        return f"ELF('{self.path.absolute()}', {self.e_class.name}, {self.e_machine.name})"

    @property
    def entry_point(self) -> int:
        return self.e_entry

    @classmethod
    def is_valid(cls, path: pathlib.Path) -> bool:
        return u32(path.open("rb").read(4), e = Endianness.BIG_ENDIAN) == Elf.ELF_MAGIC

    @property
    def checksec(self) -> dict[str, bool]:
        """Check the security property of the ELF binary. The following properties are:
        - Canary
        - NX
        - PIE
        - Fortify
        - Partial/Full RelRO.
        Return a dict() with the different keys mentioned above, and the boolean
        associated whether the protection was found."""
        if not self.__checksec:
            def __check_security_property(opt: str, filename: str, pattern: str) -> bool:
                cmd   = [readelf,]
                cmd  += opt.split()
                cmd  += [filename,]
                lines = gef_execute_external(cmd, as_list=True)
                for line in lines:
                    if re.search(pattern, line):
                        return True
                return False

            abspath = str(self.path.absolute())
            readelf = gef.session.constants["readelf"]
            self.__checksec["Canary"] = __check_security_property("-rs", abspath, r"__stack_chk_fail") is True
            has_gnu_stack = __check_security_property("-W -l", abspath, r"GNU_STACK") is True
            if has_gnu_stack:
                self.__checksec["NX"] = __check_security_property("-W -l", abspath, r"GNU_STACK.*RWE") is False
            else:
                self.__checksec["NX"] = False
            self.__checksec["PIE"] = __check_security_property("-h", abspath, r":.*EXEC") is False
            self.__checksec["Fortify"] = __check_security_property("-s", abspath, r"_chk@GLIBC") is True
            self.__checksec["Partial RelRO"] = __check_security_property("-l", abspath, r"GNU_RELRO") is True
            self.__checksec["Full RelRO"] = self.__checksec["Partial RelRO"] and __check_security_property("-d", abspath, r"BIND_NOW") is True
        return self.__checksec

    @classproperty
    @deprecated("use `Elf.Abi.X86_64`")
    def X86_64(cls) -> int: return Elf.Abi.X86_64.value # pylint: disable=no-self-argument

    @classproperty
    @deprecated("use `Elf.Abi.X86_32`")
    def X86_32(cls) -> int : return Elf.Abi.X86_32.value # pylint: disable=no-self-argument

    @classproperty
    @deprecated("use `Elf.Abi.ARM`")
    def ARM(cls) -> int : return Elf.Abi.ARM.value # pylint: disable=no-self-argument

    @classproperty
    @deprecated("use `Elf.Abi.MIPS`")
    def MIPS(cls) -> int : return Elf.Abi.MIPS.value # pylint: disable=no-self-argument

    @classproperty
    @deprecated("use `Elf.Abi.POWERPC`")
    def POWERPC(cls) -> int : return Elf.Abi.POWERPC.value # pylint: disable=no-self-argument

    @classproperty
    @deprecated("use `Elf.Abi.POWERPC64`")
    def POWERPC64(cls) -> int : return Elf.Abi.POWERPC64.value # pylint: disable=no-self-argument

    @classproperty
    @deprecated("use `Elf.Abi.SPARC`")
    def SPARC(cls) -> int : return Elf.Abi.SPARC.value # pylint: disable=no-self-argument

    @classproperty
    @deprecated("use `Elf.Abi.SPARC64`")
    def SPARC64(cls) -> int : return Elf.Abi.SPARC64.value # pylint: disable=no-self-argument

    @classproperty
    @deprecated("use `Elf.Abi.AARCH64`")
    def AARCH64(cls) -> int : return Elf.Abi.AARCH64.value  # pylint: disable=no-self-argument

    @classproperty
    @deprecated("use `Elf.Abi.RISCV`")
    def RISCV(cls) -> int : return Elf.Abi.RISCV.value # pylint: disable=no-self-argument

bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling

[rehan malak]

[-- Attachment #1: Type: text/plain, Size: 14703 bytes --]

Hi,

Deleting all the "#" in the f-string of the gef.py file removes the problem
on my side.
 sed -i -e 's/#0{align/0{align/g' gef.py
 sed -i -e 's/#0{width/0{width/g' gef.py
 sed -i -e 's/#07x}/07x}/g' gef.py
 sed -i -e 's/#06x}/06x}/g' gef.py
 sed -i -e 's/#04x}/04x}/g' gef.py
 sed -i -e 's/#4x}/4x}/g' gef.py
 sed -i -e 's/#8x}/8x}/g' gef.py
 sed -i -e 's/#10x}/10x}/g' gef.py
 sed -i -e 's/#x}/x}/g' gef.py

Your smaller example Eli also contains a f-string with the "#".

This python f-string format specifier is described here:
https://docs.python.org/3/library/string.html#format-specification-mini-language

Is this format specifier supported from the beginning ?

For example :

value = 0xab
print(f"{value:x} is a value")

=> color syntax : *is a value*    has the color defined by
font-lock-string-face

while

value = 0xab
print(f"{value:#x} is a value")

=> color are messed up... *is* has color defined by font-lock-keyword-face
and *a value* has color defined by default face

Best,

Rehan

On Sun, Dec 8, 2024 at 6:37 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: rehan malak <rehan.malak@gmail.com>
> > Date: Sun, 8 Dec 2024 14:05:36 +0100
> >
> > I can reproduce systematically this freeze dealing with a 10000lines
> > Python file :
> >
> > wget https://raw.githubusercontent.com/hugsy/gef/refs/heads/main/gef.py
> > emacs -Q gef.py
>
> A much smaller reproducer is attached below.  Just visiting it freezes
> Emacs.  But if I change the line marked below:
>
>     @classmethod
>     def is_valid(cls, _: pathlib.Path) -> bool:
>         raise NotImplementedError
>
>     def __str__(self) -> str:
>   >>>>> return f"{self.name}('{self.path.absolute()}', entry @
> {self.entry_point:#x})"
>
> to say this instead:
>
>         return self.name
>
> the problem goes away.  So it is something in that complex expression
> that trips syntax-ppss.
>
> Stefan and kobarity, any suggestions or ideas?
>
> > PageDown several times or scrolling with the mouse
> >
> > Emacs freezes, Ctrl-g not working, CPU 100%
> >
> > It works also by searching : Ctrl-s show RET
> > then Ctrl-s several times
> >
> > With a minimal .emacs :
> >
> > (set debug-on-error t)
> > (set debug-on-quit t)
> >
> > and before scrolling
> >
> > M-x profiler-start RET
> >
> > then in an external terminal
> >
> > pkill -SIGUSR2 emacs
> >
> > I get the backtrace :
> >
> > Debugger entered--entering a function:
> > * #f(compiled-function () #<bytecode -0x179dcd1db31182ae>)()
> >   syntax-ppss()
> >   python-syntax-context-type()
> >   python-nav-forward-block(-1)
> >   python-nav-backward-block()
> >   python-nav-beginning-of-block()
> >   python-nav-end-of-block()
> >   python-info-statement-ends-block-p()
> >   python-info-end-of-block-p()
> >   python-nav--forward-sexp(1 nil nil)
> >   python-nav-forward-sexp(1)
> >   forward-sexp(1)
> >   up-list(1)
> >   python--font-lock-f-strings(30419)
> >   font-lock-fontify-keywords-region(28914 30419 nil)
> >   font-lock-default-fontify-region(28914 30414 nil)
> >   font-lock-fontify-region(28914 30414)
> >   #f(compiled-function (fun) #<bytecode
> -0x8a96d65e1315e71>)(font-lock-fontify-region)
> >   run-hook-wrapped(#f(compiled-function (fun) #<bytecode
> -0x8a96d65e1315e71>) font-lock-fontify-region)
> >   jit-lock--run-functions(28914 30414)
> >   jit-lock-fontify-now(28914 30414)
> >   jit-lock-function(28914)
>
> I see something different, in GDB:
>
>   Lisp Backtrace:
>   "parse-partial-sexp" (0x9f1a788)
>   "syntax-ppss" (0x9f1a710)
>   "python-info-line-ends-backslash-p" (0x9f1a6b8)
>   "python-nav-end-of-statement" (0x9f1a670)
>   "python-nav-end-of-block" (0x9f1a630)
>   "python-info-statement-ends-block-p" (0x9f1a610)
>   "python-info-end-of-block-p" (0x9f1a5c0)
>   "python-nav--forward-sexp" (0x9f1a548)
>   "python-nav-forward-sexp" (0x9f1a4f8)
>   "forward-sexp" (0x9f1a4a8)
>   "up-list" (0x9f1a438)
>   "python--font-lock-f-strings" (0x9f1a3a0)
>   "font-lock-fontify-keywords-region" (0x9f1a308)
>   "font-lock-default-fontify-region" (0x9f1a2a0)
>   "font-lock-fontify-region" (0x9f1a230)
>   0xb9117d0 PVEC_CLOSURE
>   "run-hook-wrapped" (0x9f1a1c0)
>   "jit-lock--run-functions" (0x9f1a0e8)
>   "jit-lock-fontify-now" (0x9f1a058)
>   "jit-lock-function" (0x5ffba98)
>   "redisplay_internal (C function)" (0x0)
>
> Here's the file with which I can reproduce the problem?
>
>
> class FileFormat:
>     name: str
>     path: pathlib.Path
>     entry_point: int
>     checksec: dict[str, bool]
>     sections: list[FileFormatSection]
>
>     def __init__(self, path: str | pathlib.Path) -> None:
>         raise NotImplementedError
>
>     def __init_subclass__(cls: Type["FileFormat"], **kwargs):
>         global __registered_file_formats__
>         super().__init_subclass__(**kwargs)
>         required_attributes = ("name", "entry_point", "is_valid",
> "checksec",)
>         for attr in required_attributes:
>             if not hasattr(cls, attr):
>                 raise NotImplementedError(f"File format '{cls.__name__}'
> is invalid: missing attribute '{attr}'")
>         __registered_file_formats__.add(cls)
>         return
>
>     @classmethod
>     def is_valid(cls, _: pathlib.Path) -> bool:
>         raise NotImplementedError
>
>     def __str__(self) -> str:
>         return f"{self.name}('{self.path.absolute()}', entry @
> {self.entry_point:#x})"
>
>
> class Elf(FileFormat):
>     """Basic ELF parsing.
>     Ref:
>     - http://www.skyfree.org/linux/references/ELF_Format.pdf
>     - https://refspecs.linuxfoundation.org/elf/elfspec_ppc.pdf
>     - https://refspecs.linuxfoundation.org/ELF/ppc64/PPC-elf64abi.html
>     """
>     class Class(enum.Enum):
>         ELF_32_BITS       = 0x01
>         ELF_64_BITS       = 0x02
>
>     ELF_MAGIC         = 0x7f454c46
>
>     class Abi(enum.Enum):
>         X86_64            = 0x3e
>         X86_32            = 0x03
>         ARM               = 0x28
>         MIPS              = 0x08
>         POWERPC           = 0x14
>         POWERPC64         = 0x15
>         SPARC             = 0x02
>         SPARC64           = 0x2b
>         AARCH64           = 0xb7
>         RISCV             = 0xf3
>         IA64              = 0x32
>         M68K              = 0x04
>
>     class Type(enum.Enum):
>         ET_RELOC          = 1
>         ET_EXEC           = 2
>         ET_DYN            = 3
>         ET_CORE           = 4
>
>     class OsAbi(enum.Enum):
>         SYSTEMV     = 0x00
>         HPUX        = 0x01
>         NETBSD      = 0x02
>         LINUX       = 0x03
>         SOLARIS     = 0x06
>         AIX         = 0x07
>         IRIX        = 0x08
>         FREEBSD     = 0x09
>         OPENBSD     = 0x0C
>
>     e_magic: int                = ELF_MAGIC
>     e_class: "Elf.Class"        = Class.ELF_32_BITS
>     e_endianness: Endianness    = Endianness.LITTLE_ENDIAN
>     e_eiversion: int
>     e_osabi: "Elf.OsAbi"
>     e_abiversion: int
>     e_pad: bytes
>     e_type: "Elf.Type"          = Type.ET_EXEC
>     e_machine: Abi              = Abi.X86_32
>     e_version: int
>     e_entry: int
>     e_phoff: int
>     e_shoff: int
>     e_flags: int
>     e_ehsize: int
>     e_phentsize: int
>     e_phnum: int
>     e_shentsize: int
>     e_shnum: int
>     e_shstrndx: int
>
>     path: pathlib.Path
>     phdrs : list["Phdr"]
>     shdrs : list["Shdr"]
>     name: str = "ELF"
>
>     __checksec : dict[str, bool]
>
>     def __init__(self, path: str | pathlib.Path) -> None:
>         """Instantiate an ELF object. A valid ELF must be provided, or an
> exception will be thrown."""
>
>         if isinstance(path, str):
>             self.path = pathlib.Path(path).expanduser()
>         elif isinstance(path, pathlib.Path):
>             self.path = path
>         else:
>             raise TypeError
>
>         if not self.path.exists():
>             raise FileNotFoundError(f"'{self.path}' not found/readable,
> most gef features will not work")
>
>         self.__checksec = {}
>
>         with self.path.open("rb") as self.fd:
>             # off 0x0
>             self.e_magic, e_class, e_endianness, self.e_eiversion =
> self.read_and_unpack(">IBBB")
>             if self.e_magic != Elf.ELF_MAGIC:
>                 # The ELF is corrupted, GDB won't handle it, no point
> going further
>                 raise RuntimeError("Not a valid ELF file (magic)")
>
>             self.e_class, self.e_endianness = Elf.Class(e_class),
> Endianness(e_endianness)
>
>             if self.e_endianness != gef.arch.endianness:
>                 warn("Unexpected endianness for architecture")
>
>             endian = self.e_endianness
>
>             # off 0x7
>             e_osabi, self.e_abiversion =
> self.read_and_unpack(f"{endian}BB")
>             self.e_osabi = Elf.OsAbi(e_osabi)
>
>             # off 0x9
>             self.e_pad = self.read(7)
>
>             # off 0x10
>             e_type, e_machine, self.e_version =
> self.read_and_unpack(f"{endian}HHI")
>             self.e_type, self.e_machine = Elf.Type(e_type),
> Elf.Abi(e_machine)
>
>             # off 0x18
>             if self.e_class == Elf.Class.ELF_64_BITS:
>                 self.e_entry, self.e_phoff, self.e_shoff =
> self.read_and_unpack(f"{endian}QQQ")
>             else:
>                 self.e_entry, self.e_phoff, self.e_shoff =
> self.read_and_unpack(f"{endian}III")
>
>             self.e_flags, self.e_ehsize, self.e_phentsize, self.e_phnum =
> self.read_and_unpack(f"{endian}IHHH")
>             self.e_shentsize, self.e_shnum, self.e_shstrndx =
> self.read_and_unpack(f"{endian}HHH")
>
>             self.phdrs = []
>             for i in range(self.e_phnum):
>                 self.phdrs.append(Phdr(self, self.e_phoff +
> self.e_phentsize * i))
>
>             self.shdrs = []
>             for i in range(self.e_shnum):
>                 self.shdrs.append(Shdr(self, self.e_shoff +
> self.e_shentsize * i))
>         return
>
>     def read(self, size: int) -> bytes:
>         return self.fd.read(size)
>
>     def read_and_unpack(self, fmt: str) -> tuple[Any, ...]:
>         size = struct.calcsize(fmt)
>         data = self.fd.read(size)
>         return struct.unpack(fmt, data)
>
>     def seek(self, off: int) -> None:
>         self.fd.seek(off, 0)
>
>     def __str__(self) -> str:
>         return f"ELF('{self.path.absolute()}', {self.e_class.name}, {
> self.e_machine.name})"
>
>     def __repr__(self) -> str:
>         return f"ELF('{self.path.absolute()}', {self.e_class.name}, {
> self.e_machine.name})"
>
>     @property
>     def entry_point(self) -> int:
>         return self.e_entry
>
>     @classmethod
>     def is_valid(cls, path: pathlib.Path) -> bool:
>         return u32(path.open("rb").read(4), e = Endianness.BIG_ENDIAN) ==
> Elf.ELF_MAGIC
>
>     @property
>     def checksec(self) -> dict[str, bool]:
>         """Check the security property of the ELF binary. The following
> properties are:
>         - Canary
>         - NX
>         - PIE
>         - Fortify
>         - Partial/Full RelRO.
>         Return a dict() with the different keys mentioned above, and the
> boolean
>         associated whether the protection was found."""
>         if not self.__checksec:
>             def __check_security_property(opt: str, filename: str,
> pattern: str) -> bool:
>                 cmd   = [readelf,]
>                 cmd  += opt.split()
>                 cmd  += [filename,]
>                 lines = gef_execute_external(cmd, as_list=True)
>                 for line in lines:
>                     if re.search(pattern, line):
>                         return True
>                 return False
>
>             abspath = str(self.path.absolute())
>             readelf = gef.session.constants["readelf"]
>             self.__checksec["Canary"] = __check_security_property("-rs",
> abspath, r"__stack_chk_fail") is True
>             has_gnu_stack = __check_security_property("-W -l", abspath,
> r"GNU_STACK") is True
>             if has_gnu_stack:
>                 self.__checksec["NX"] = __check_security_property("-W -l",
> abspath, r"GNU_STACK.*RWE") is False
>             else:
>                 self.__checksec["NX"] = False
>             self.__checksec["PIE"] = __check_security_property("-h",
> abspath, r":.*EXEC") is False
>             self.__checksec["Fortify"] = __check_security_property("-s",
> abspath, r"_chk@GLIBC") is True
>             self.__checksec["Partial RelRO"] =
> __check_security_property("-l", abspath, r"GNU_RELRO") is True
>             self.__checksec["Full RelRO"] = self.__checksec["Partial
> RelRO"] and __check_security_property("-d", abspath, r"BIND_NOW") is True
>         return self.__checksec
>
>     @classproperty
>     @deprecated("use `Elf.Abi.X86_64`")
>     def X86_64(cls) -> int: return Elf.Abi.X86_64.value # pylint:
> disable=no-self-argument
>
>     @classproperty
>     @deprecated("use `Elf.Abi.X86_32`")
>     def X86_32(cls) -> int : return Elf.Abi.X86_32.value # pylint:
> disable=no-self-argument
>
>     @classproperty
>     @deprecated("use `Elf.Abi.ARM`")
>     def ARM(cls) -> int : return Elf.Abi.ARM.value # pylint:
> disable=no-self-argument
>
>     @classproperty
>     @deprecated("use `Elf.Abi.MIPS`")
>     def MIPS(cls) -> int : return Elf.Abi.MIPS.value # pylint:
> disable=no-self-argument
>
>     @classproperty
>     @deprecated("use `Elf.Abi.POWERPC`")
>     def POWERPC(cls) -> int : return Elf.Abi.POWERPC.value # pylint:
> disable=no-self-argument
>
>     @classproperty
>     @deprecated("use `Elf.Abi.POWERPC64`")
>     def POWERPC64(cls) -> int : return Elf.Abi.POWERPC64.value # pylint:
> disable=no-self-argument
>
>     @classproperty
>     @deprecated("use `Elf.Abi.SPARC`")
>     def SPARC(cls) -> int : return Elf.Abi.SPARC.value # pylint:
> disable=no-self-argument
>
>     @classproperty
>     @deprecated("use `Elf.Abi.SPARC64`")
>     def SPARC64(cls) -> int : return Elf.Abi.SPARC64.value # pylint:
> disable=no-self-argument
>
>     @classproperty
>     @deprecated("use `Elf.Abi.AARCH64`")
>     def AARCH64(cls) -> int : return Elf.Abi.AARCH64.value  # pylint:
> disable=no-self-argument
>
>     @classproperty
>     @deprecated("use `Elf.Abi.RISCV`")
>     def RISCV(cls) -> int : return Elf.Abi.RISCV.value # pylint:
> disable=no-self-argument
>
>

[-- Attachment #2: Type: text/html, Size: 19416 bytes --]

bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling

[kobarity]

[-- Attachment #1: Type: text/plain, Size: 1682 bytes --]

rehan malak wrote:
> Deleting all the "#" in the f-string of the gef.py file removes the problem on my side.
>  sed -i -e 's/#0{align/0{align/g' gef.py
>  sed -i -e 's/#0{width/0{width/g' gef.py
>  sed -i -e 's/#07x}/07x}/g' gef.py
>  sed -i -e 's/#06x}/06x}/g' gef.py
>  sed -i -e 's/#04x}/04x}/g' gef.py
>  sed -i -e 's/#4x}/4x}/g' gef.py
>  sed -i -e 's/#8x}/8x}/g' gef.py
>  sed -i -e 's/#10x}/10x}/g' gef.py
>  sed -i -e 's/#x}/x}/g' gef.py
> 
> Your smaller example Eli also contains a f-string with the "#".
> 
> This python f-string format specifier is described here:
> https://docs.python.org/3/library/string.html#format-specification-mini-language
> 
> Is this format specifier supported from the beginning ?
> 
> For example :
> 
> value = 0xab
> print(f"{value:x} is a value")
> 
> => color syntax : is a value    has the color defined by font-lock-string-face
> 
> while
> 
> value = 0xab
> print(f"{value:#x} is a value")
> 
> => color are messed up... is has color defined by font-lock-keyword-face and a value has color defined by default face

The mechanism of the hang is still unclear, but I found the cause of
the font-lock malfunction: when using `up-list' to find the
corresponding closing brace for an opening brace in an f-string, if
the format specifier contains "#", the rest of the string is
considered a comment and the search fails.  Therefore, it can be
worked around by temporarily binding `parse-sexp-ignore-comments' to
nil, as in the attached patch.

Hangs on gef.py can also be avoided with this patch, but it assumes
that the braces are properly closed.  If you remove the closing braces
of expressions containing "#", it will eventually hang.

[-- Attachment #2: 0001-Fix-font-lock-of-Python-f-strings.patch --]
[-- Type: text/plain, Size: 1999 bytes --]

From 697f942d70e40bcd9bd32596313a9a371fffe35c Mon Sep 17 00:00:00 2001
From: kobarity <kobarity@gmail.com>
Date: Wed, 11 Dec 2024 23:21:04 +0900
Subject: [PATCH] Fix font-lock of Python f-strings

* lisp/progmodes/python.el (python--font-lock-f-strings): Bind
'parse-sexp-ignore-comments' to nil so that we can look for
closing braces even if a hash is used in the format specifier.
* test/lisp/progmodes/python-tests.el
(python-font-lock-f-string-1): New test.  (Bug#74738)
---
 lisp/progmodes/python.el            | 4 +++-
 test/lisp/progmodes/python-tests.el | 8 ++++++++
 2 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index cfa3cc59568..827f437013f 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -631,7 +631,9 @@ python--font-lock-f-strings
               (forward-char 1)          ;Just skip over {{
             (let ((beg (match-beginning 0))
                   (end (condition-case nil
-                           (progn (up-list 1) (min send (point)))
+                           (let ((parse-sexp-ignore-comments))
+                             (up-list 1)
+                             (min send (point)))
                          (scan-error send))))
               (goto-char end)
               (put-text-property beg end 'face nil))))
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index 4cc4040d0ff..ac17c5b1263 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -709,6 +709,14 @@ python-font-lock-string-literal-concatenation
      (17 . font-lock-operator-face) (18)
      (19 . font-lock-string-face))))
 
+(ert-deftest python-font-lock-f-string-1 ()
+  "Test for bug#74738."
+  (python-tests-assert-faces
+   "print(f\"{value:#x} is a value\")"
+   '((1 . font-lock-builtin-face) (6)
+     (8 . font-lock-string-face) (9)
+     (19 . font-lock-string-face) (31))))
+
 \f
 ;;; Indentation
 
-- 
2.43.0

bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling

[rehan malak]

[-- Attachment #1: Type: text/plain, Size: 2064 bytes --]

thanks kobarity, with your patch, I confirm that :
- I can open gef.py and scroll
- I get the correct font-lock-string-face in the f-string after a "#" in
the format

On Wed, Dec 11, 2024 at 3:24 PM kobarity <kobarity@gmail.com> wrote:

> rehan malak wrote:
> > Deleting all the "#" in the f-string of the gef.py file removes the
> problem on my side.
> >  sed -i -e 's/#0{align/0{align/g' gef.py
> >  sed -i -e 's/#0{width/0{width/g' gef.py
> >  sed -i -e 's/#07x}/07x}/g' gef.py
> >  sed -i -e 's/#06x}/06x}/g' gef.py
> >  sed -i -e 's/#04x}/04x}/g' gef.py
> >  sed -i -e 's/#4x}/4x}/g' gef.py
> >  sed -i -e 's/#8x}/8x}/g' gef.py
> >  sed -i -e 's/#10x}/10x}/g' gef.py
> >  sed -i -e 's/#x}/x}/g' gef.py
> >
> > Your smaller example Eli also contains a f-string with the "#".
> >
> > This python f-string format specifier is described here:
> >
> https://docs.python.org/3/library/string.html#format-specification-mini-language
> >
> > Is this format specifier supported from the beginning ?
> >
> > For example :
> >
> > value = 0xab
> > print(f"{value:x} is a value")
> >
> > => color syntax : is a value    has the color defined by
> font-lock-string-face
> >
> > while
> >
> > value = 0xab
> > print(f"{value:#x} is a value")
> >
> > => color are messed up... is has color defined by font-lock-keyword-face
> and a value has color defined by default face
>
> The mechanism of the hang is still unclear, but I found the cause of
> the font-lock malfunction: when using `up-list' to find the
> corresponding closing brace for an opening brace in an f-string, if
> the format specifier contains "#", the rest of the string is
> considered a comment and the search fails.  Therefore, it can be
> worked around by temporarily binding `parse-sexp-ignore-comments' to
> nil, as in the attached patch.
>
> Hangs on gef.py can also be avoided with this patch, but it assumes
> that the braces are properly closed.  If you remove the closing braces
> of expressions containing "#", it will eventually hang.
>

[-- Attachment #2: Type: text/html, Size: 2855 bytes --]

bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling

[kobarity]

[-- Attachment #1: Type: text/plain, Size: 1979 bytes --]


rehan malak wrote:
> thanks kobarity, with your patch, I confirm that :
> - I can open gef.py and scroll
> - I get the correct font-lock-string-face in the f-string after a "#" in the format

Thank you for your confirmation.

Stefan Monnier wrote:
> > The mechanism of the hang is still unclear, but I found the cause of
> > the font-lock malfunction: when using `up-list' to find the
> > corresponding closing brace for an opening brace in an f-string, if
> > the format specifier contains "#", the rest of the string is
> > considered a comment and the search fails.  Therefore, it can be
> > worked around by temporarily binding `parse-sexp-ignore-comments' to
> > nil, as in the attached patch.
> 
> Maybe a cleaner solution would be to locally use a different
> syntax-table (where # doesn't start a comment), but I'm not sure it's
> worth the trouble.  So I installed your patch into `master`, thank you.

Thank you.  I agree with you about the syntax-table.

> Maybe we should also temporarily setup a narrowing so `up-list` can't
> look past `send`.
> 
> BTW, beside removing the `font-lock-string-face` from the `face`
> property, `python--font-lock-f-strings` should probably also add
> a special face for the ":<FMT>" thingies found (optionally) at the end.
> 
> > Hangs on gef.py can also be avoided with this patch, but it assumes
> > that the braces are properly closed.  If you remove the closing braces
> > of expressions containing "#", it will eventually hang.
> 
> Hmm...

Narrowing might improve it, but what about binding
`forward-sexp-function' to nil as well, as in the attached patch?  It
seems to resolve the hang caused by the unbalanced braces in
f-strings.

1. wget https://raw.githubusercontent.com/hugsy/gef/refs/heads/main/gef.py
2. sed -i -e 's/\(:#\d*x\)}/\1/g' gef.py
3. emacs -Q gef.py
4. Scroll down the file.

I don't think `python-nav-forward-sexp' is necessary here, since the
inside of the braces should be an expression, not statements.

[-- Attachment #2: 0001-Fix-hangs-caused-by-unbalanced-braces-in-Python-f-st.patch --]
[-- Type: text/plain, Size: 1148 bytes --]

From bb2c1a3e03416532fb97d6fa7ef276c5feebb356 Mon Sep 17 00:00:00 2001
From: kobarity <kobarity@gmail.com>
Date: Sun, 15 Dec 2024 23:02:51 +0900
Subject: [PATCH] Fix hangs caused by unbalanced braces in Python f-strings

* lisp/progmodes/python.el (python--font-lock-f-strings):
Temporarily bind forward-sexp-function to nil when calling
'up-list'.  (Bug#74738)
---
 lisp/progmodes/python.el | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index 5729b12573f..493eab5e611 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -623,7 +623,8 @@ python--font-lock-f-strings
               (forward-char 1)          ;Just skip over {{
             (let ((beg (match-beginning 0))
                   (end (condition-case nil
-                           (let ((parse-sexp-ignore-comments))
+                           (let ((forward-sexp-function)
+                                 (parse-sexp-ignore-comments))
                              (up-list 1)
                              (min send (point)))
                          (scan-error send))))
-- 
2.43.0

bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> > Hangs on gef.py can also be avoided with this patch, but it assumes
>> > that the braces are properly closed.  If you remove the closing braces
>> > of expressions containing "#", it will eventually hang.
>> 
>> Hmm...
>
> Narrowing might improve it, but what about binding
> `forward-sexp-function' to nil as well, as in the attached patch?

Oh, of course, yes!
Thanks, pushed!


        Stefan

bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: rehan malak <rehan.malak@gmail.com>,  Eli Zaretskii <eliz@gnu.org>,
>   74738@debbugs.gnu.org
> Date: Sun, 15 Dec 2024 10:03:33 -0500
> 
> >> > Hangs on gef.py can also be avoided with this patch, but it assumes
> >> > that the braces are properly closed.  If you remove the closing braces
> >> > of expressions containing "#", it will eventually hang.
> >> 
> >> Hmm...
> >
> > Narrowing might improve it, but what about binding
> > `forward-sexp-function' to nil as well, as in the attached patch?
> 
> Oh, of course, yes!
> Thanks, pushed!

Thanks, should we now close this bug?

bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Thanks, should we now close this bug?

I believe so, yes, closing,


        Stefan

bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> The mechanism of the hang is still unclear, but I found the cause of
> the font-lock malfunction: when using `up-list' to find the
> corresponding closing brace for an opening brace in an f-string, if
> the format specifier contains "#", the rest of the string is
> considered a comment and the search fails.  Therefore, it can be
> worked around by temporarily binding `parse-sexp-ignore-comments' to
> nil, as in the attached patch.

Maybe a cleaner solution would be to locally use a different
syntax-table (where # doesn't start a comment), but I'm not sure it's
worth the trouble.  So I installed your patch into `master`, thank you.

Maybe we should also temporarily setup a narrowing so `up-list` can't
look past `send`.

BTW, beside removing the `font-lock-string-face` from the `face`
property, `python--font-lock-f-strings` should probably also add
a special face for the ":<FMT>" thingies found (optionally) at the end.

> Hangs on gef.py can also be avoided with this patch, but it assumes
> that the braces are properly closed.  If you remove the closing braces
> of expressions containing "#", it will eventually hang.

Hmm...


        Stefan

bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling

[kobarity]

rehan malak wrote:
> Please tell me if there is a workaround without losing python syntax color ?

If you can configure Emacs to use tree-sitter, python-ts-mode would
not be affected by this problem.

Eli Zaretskii wrote:
> 
> > From: rehan malak <rehan.malak@gmail.com>
> > Date: Sun, 8 Dec 2024 14:05:36 +0100
> > 
> > I can reproduce systematically this freeze dealing with a 10000lines
> > Python file :
> > 
> > wget https://raw.githubusercontent.com/hugsy/gef/refs/heads/main/gef.py
> > emacs -Q gef.py

It seems to be reproducible after the following commit:

commit 3b3274a85c2f5df21d76d82e0d7740005aa84fdf
Author: Stefan Monnier <monnier@iro.umontreal.ca>
Date:   Fri Oct 16 14:03:59 2020 -0400

    * lisp/progmodes/python.el: Teach f-strings to `font-lock`

    (python--f-string-p, python--font-lock-f-strings): New functions.
    (python-font-lock-keywords-maximum-decoration): Use them.

> A much smaller reproducer is attached below.  Just visiting it freezes
> Emacs.  But if I change the line marked below:
> 
>     @classmethod
>     def is_valid(cls, _: pathlib.Path) -> bool:
>         raise NotImplementedError
> 
>     def __str__(self) -> str:
>   >>>>> return f"{self.name}('{self.path.absolute()}', entry @ {self.entry_point:#x})"
> 
> to say this instead:
> 
>         return self.name
> 
> the problem goes away.  So it is something in that complex expression
> that trips syntax-ppss.
> 
> Stefan and kobarity, any suggestions or ideas?

I could not reproduce the freeze, but it took several seconds to visit
the reproducer.  It appears to be possible to work around this by
simply removing the format specifier ":#x".

return f"{self.name}('{self.path.absolute()}', entry @ {self.entry_point})"

So there may be a problem with font-lock when there is an f-string
format specifier, but I have not found the root cause yet.