bug#67718: 30.0.50; prog-fill-reindent-defun vs. fill-paragraph for strings and comments

bug#67718: 30.0.50; prog-fill-reindent-defun vs. fill-paragraph for strings and comments

[Jens Schmidt via Bug reports for GNU Emacs, the Swiss army knife of text editors]

While preparing a reply for bug#67462 I came accross another issue where
the new binding of M-q to `prog-fill-reindent-defun' in Emacs 30 causes
a change in behavior compared to Emacs 29.

Here is a test case starting at "emacs -Q":

- In *scratch* goto EOB, then insert the following text all on one line
  (including double quotes, without leading blanks):

  "This buffer is for text that is not saved, and for Lisp evaluation. To create a file, visit it with ‘C-x C-f’ and enter text in its buffer."

- Then C-a and M-q: No reaction

    vs.

- Then C-a and M-x fill-paragraph RET: The string gets filled to

  "This buffer is for text that is not saved, and for Lisp
   evaluation. To create a file, visit it with ‘C-x C-f’ and enter
   text in its buffer."


The problem here being that `prog-fill-reindent-defun' has a rather
narrow range where it decides to fill intead of indenting:

  (or treesit-text-node
      (nth 8 (syntax-ppss))
      (re-search-forward "\\s-*\\s<" (line-end-position) t))

That is when point is

  - (not sure about the treesit condition) OR
  - *in* a text node OR
  - *in* a comment node OR
  - *before* whitespace => <-comment syntax

I guess that heuristics needs to be improved a bit, at least if you
intend to be compatible with the Emacs 29 behavior of M-q.


Here are some more examples where there are differences or
inconsistencies, with "pipe" denoting point:

  "any" prog mode

    "after a string"|
    |
    "on an empty line before a string"

      This one could differ in a treesitter mode, I guess.

  python-mode

    |"""doc string"""

      Same on as the initial example, but particularly nasty, since
      python-mode has a whole set of rules how to fill doc strings.

  c-mode

    |/* before a non-<-comment */
    /* after a non-<-comment */|

      Well, that doesn't really count, since c-mode and derivatives
      don't use prog-fill-reindent-defun, do they?  So let's go for
      good-old:

  sql-mode

    |/* before a non-<-comment */
    /* after a non-<-comment */|
    |-- before a non-<-comment

      sql-mode doesn't even have <-comments, so you always have to move
      point *into* a comment to get it filled.  BTW, exactly like
      javascript-mode, which is a bit more important, probably:

  javascript-mode

     |// before a non-<-comment

  emacs-lisp-mode

    (foo bar baz ?\|; baz bar foo)

      So this goes the other direction: If you M-q here and expect the
      line or surrounding function to be indented, you will get
      disappointed, because the heuristics detects this as a comment
      that needs to be filled.

      This one I also find questionable:

    (foo bar baz "string|" baz bar foo)

      Why would I want to fill here?  Here I'd like to indent.

Actually, now I think the fill-vs-indent heuristics of
`prog-fill-reindent-defun' needs to be improved a *lot* to make this
feature predictable and usable!  (And yes, I'll still follow up on
bug#67462 as well.)


In GNU Emacs 30.0.50 (build 2, x86_64-pc-linux-gnu, GTK+ Version
 3.24.24, cairo version 1.16.0) of 2023-12-08 built on sappc2
Repository revision: 3b1fd42732f7ca5b2db6ad6e75af1c037e1c54e4
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12011000
System Description: Debian GNU/Linux 11 (bullseye)

Configured using:
 'configure -C --with-native-compilation --with-mailutils'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES
NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF
TOOLKIT_SCROLL_BARS X11 XDBE XIM XINPUT2 XPM GTK3 ZLIB

Important settings:
  value of $LC_COLLATE: POSIX
  value of $LC_TIME: POSIX
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  display-time-mode: t
  delete-selection-mode: t
  show-paren-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  minibuffer-regexp-mode: t
  line-number-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
/home/jschmidt/work/org-mode/lisp/org-attach hides /home/jschmidt/work/emacs-master/lisp/org/org-attach
/home/jschmidt/work/org-mode/lisp/ob-scheme hides /home/jschmidt/work/emacs-master/lisp/org/ob-scheme
/home/jschmidt/work/org-mode/lisp/ob-processing hides /home/jschmidt/work/emacs-master/lisp/org/ob-processing
/home/jschmidt/work/org-mode/lisp/ob-gnuplot hides /home/jschmidt/work/emacs-master/lisp/org/ob-gnuplot
/home/jschmidt/work/org-mode/lisp/org-element hides /home/jschmidt/work/emacs-master/lisp/org/org-element
/home/jschmidt/work/org-mode/lisp/ob-org hides /home/jschmidt/work/emacs-master/lisp/org/ob-org
/home/jschmidt/work/org-mode/lisp/ob-sql hides /home/jschmidt/work/emacs-master/lisp/org/ob-sql
/home/jschmidt/work/org-mode/lisp/oc-bibtex hides /home/jschmidt/work/emacs-master/lisp/org/oc-bibtex
/home/jschmidt/work/org-mode/lisp/ox-koma-letter hides /home/jschmidt/work/emacs-master/lisp/org/ox-koma-letter
/home/jschmidt/work/org-mode/lisp/ox-icalendar hides /home/jschmidt/work/emacs-master/lisp/org/ox-icalendar
/home/jschmidt/work/org-mode/lisp/ol-gnus hides /home/jschmidt/work/emacs-master/lisp/org/ol-gnus
/home/jschmidt/work/org-mode/lisp/ob-sqlite hides /home/jschmidt/work/emacs-master/lisp/org/ob-sqlite
/home/jschmidt/work/org-mode/lisp/ob-clojure hides /home/jschmidt/work/emacs-master/lisp/org/ob-clojure
/home/jschmidt/work/org-mode/lisp/org-indent hides /home/jschmidt/work/emacs-master/lisp/org/org-indent
/home/jschmidt/work/org-mode/lisp/org-persist hides /home/jschmidt/work/emacs-master/lisp/org/org-persist
/home/jschmidt/work/org-mode/lisp/ol-eww hides /home/jschmidt/work/emacs-master/lisp/org/ol-eww
/home/jschmidt/work/org-mode/lisp/ol-info hides /home/jschmidt/work/emacs-master/lisp/org/ol-info
/home/jschmidt/work/org-mode/lisp/ol-rmail hides /home/jschmidt/work/emacs-master/lisp/org/ol-rmail
/home/jschmidt/work/org-mode/lisp/oc-basic hides /home/jschmidt/work/emacs-master/lisp/org/oc-basic
/home/jschmidt/work/org-mode/lisp/ob-forth hides /home/jschmidt/work/emacs-master/lisp/org/ob-forth
/home/jschmidt/work/org-mode/lisp/org-timer hides /home/jschmidt/work/emacs-master/lisp/org/org-timer
/home/jschmidt/work/org-mode/lisp/ob-makefile hides /home/jschmidt/work/emacs-master/lisp/org/ob-makefile
/home/jschmidt/work/org-mode/lisp/ob-fortran hides /home/jschmidt/work/emacs-master/lisp/org/ob-fortran
/home/jschmidt/work/org-mode/lisp/ox-html hides /home/jschmidt/work/emacs-master/lisp/org/ox-html
/home/jschmidt/work/org-mode/lisp/ob-lob hides /home/jschmidt/work/emacs-master/lisp/org/ob-lob
/home/jschmidt/work/org-mode/lisp/org-plot hides /home/jschmidt/work/emacs-master/lisp/org/org-plot
/home/jschmidt/work/org-mode/lisp/ob-js hides /home/jschmidt/work/emacs-master/lisp/org/ob-js
/home/jschmidt/work/org-mode/lisp/ob-R hides /home/jschmidt/work/emacs-master/lisp/org/ob-R
/home/jschmidt/work/org-mode/lisp/org-inlinetask hides /home/jschmidt/work/emacs-master/lisp/org/org-inlinetask
/home/jschmidt/work/org-mode/lisp/ol-mhe hides /home/jschmidt/work/emacs-master/lisp/org/ol-mhe
/home/jschmidt/work/org-mode/lisp/ob-dot hides /home/jschmidt/work/emacs-master/lisp/org/ob-dot
/home/jschmidt/work/org-mode/lisp/ol-docview hides /home/jschmidt/work/emacs-master/lisp/org/ol-docview
/home/jschmidt/work/org-mode/lisp/ob-C hides /home/jschmidt/work/emacs-master/lisp/org/ob-C
/home/jschmidt/work/org-mode/lisp/ol-man hides /home/jschmidt/work/emacs-master/lisp/org/ol-man
/home/jschmidt/work/org-mode/lisp/org-cycle hides /home/jschmidt/work/emacs-master/lisp/org/org-cycle
/home/jschmidt/work/org-mode/lisp/org-pcomplete hides /home/jschmidt/work/emacs-master/lisp/org/org-pcomplete
/home/jschmidt/work/org-mode/lisp/org-faces hides /home/jschmidt/work/emacs-master/lisp/org/org-faces
/home/jschmidt/work/org-mode/lisp/org hides /home/jschmidt/work/emacs-master/lisp/org/org
/home/jschmidt/work/org-mode/lisp/ol hides /home/jschmidt/work/emacs-master/lisp/org/ol
/home/jschmidt/work/org-mode/lisp/ob-haskell hides /home/jschmidt/work/emacs-master/lisp/org/ob-haskell
/home/jschmidt/work/org-mode/lisp/ob-lisp hides /home/jschmidt/work/emacs-master/lisp/org/ob-lisp
/home/jschmidt/work/org-mode/lisp/org-mobile hides /home/jschmidt/work/emacs-master/lisp/org/org-mobile
/home/jschmidt/work/org-mode/lisp/org-agenda hides /home/jschmidt/work/emacs-master/lisp/org/org-agenda
/home/jschmidt/work/org-mode/lisp/ob-perl hides /home/jschmidt/work/emacs-master/lisp/org/ob-perl
/home/jschmidt/work/org-mode/lisp/org-lint hides /home/jschmidt/work/emacs-master/lisp/org/org-lint
/home/jschmidt/work/org-mode/lisp/org-id hides /home/jschmidt/work/emacs-master/lisp/org/org-id
/home/jschmidt/work/org-mode/lisp/ox-man hides /home/jschmidt/work/emacs-master/lisp/org/ox-man
/home/jschmidt/work/org-mode/lisp/ol-bbdb hides /home/jschmidt/work/emacs-master/lisp/org/ol-bbdb
/home/jschmidt/work/org-mode/lisp/ob-lilypond hides /home/jschmidt/work/emacs-master/lisp/org/ob-lilypond
/home/jschmidt/work/org-mode/lisp/org-archive hides /home/jschmidt/work/emacs-master/lisp/org/org-archive
/home/jschmidt/work/org-mode/lisp/ox-publish hides /home/jschmidt/work/emacs-master/lisp/org/ox-publish
/home/jschmidt/work/org-mode/lisp/ob-core hides /home/jschmidt/work/emacs-master/lisp/org/ob-core
/home/jschmidt/work/org-mode/lisp/ob-groovy hides /home/jschmidt/work/emacs-master/lisp/org/ob-groovy
/home/jschmidt/work/org-mode/lisp/ox-org hides /home/jschmidt/work/emacs-master/lisp/org/ox-org
/home/jschmidt/work/org-mode/lisp/org-entities hides /home/jschmidt/work/emacs-master/lisp/org/org-entities
/home/jschmidt/work/org-mode/lisp/org-goto hides /home/jschmidt/work/emacs-master/lisp/org/org-goto
/home/jschmidt/work/org-mode/lisp/ob-awk hides /home/jschmidt/work/emacs-master/lisp/org/ob-awk
/home/jschmidt/work/org-mode/lisp/ol-eshell hides /home/jschmidt/work/emacs-master/lisp/org/ol-eshell
/home/jschmidt/work/org-mode/lisp/org-num hides /home/jschmidt/work/emacs-master/lisp/org/org-num
/home/jschmidt/work/org-mode/lisp/oc-csl hides /home/jschmidt/work/emacs-master/lisp/org/oc-csl
/home/jschmidt/work/org-mode/lisp/org-capture hides /home/jschmidt/work/emacs-master/lisp/org/org-capture
/home/jschmidt/work/org-mode/lisp/ob-ref hides /home/jschmidt/work/emacs-master/lisp/org/ob-ref
/home/jschmidt/work/org-mode/lisp/org-list hides /home/jschmidt/work/emacs-master/lisp/org/org-list
/home/jschmidt/work/org-mode/lisp/org-macro hides /home/jschmidt/work/emacs-master/lisp/org/org-macro
/home/jschmidt/work/org-mode/lisp/org-clock hides /home/jschmidt/work/emacs-master/lisp/org/org-clock
/home/jschmidt/work/org-mode/lisp/ob-table hides /home/jschmidt/work/emacs-master/lisp/org/ob-table
/home/jschmidt/work/org-mode/lisp/org-datetree hides /home/jschmidt/work/emacs-master/lisp/org/org-datetree
/home/jschmidt/work/org-mode/lisp/org-mouse hides /home/jschmidt/work/emacs-master/lisp/org/org-mouse
/home/jschmidt/work/org-mode/lisp/ob-latex hides /home/jschmidt/work/emacs-master/lisp/org/ob-latex
/home/jschmidt/work/org-mode/lisp/org-keys hides /home/jschmidt/work/emacs-master/lisp/org/org-keys
/home/jschmidt/work/org-mode/lisp/org-compat hides /home/jschmidt/work/emacs-master/lisp/org/org-compat
/home/jschmidt/work/org-mode/lisp/org-habit hides /home/jschmidt/work/emacs-master/lisp/org/org-habit
/home/jschmidt/work/org-mode/lisp/org-tempo hides /home/jschmidt/work/emacs-master/lisp/org/org-tempo
/home/jschmidt/work/org-mode/lisp/org-refile hides /home/jschmidt/work/emacs-master/lisp/org/org-refile
/home/jschmidt/work/org-mode/lisp/ob-ruby hides /home/jschmidt/work/emacs-master/lisp/org/ob-ruby
/home/jschmidt/work/org-mode/lisp/org-attach-git hides /home/jschmidt/work/emacs-master/lisp/org/org-attach-git
/home/jschmidt/work/org-mode/lisp/org-loaddefs hides /home/jschmidt/work/emacs-master/lisp/org/org-loaddefs
/home/jschmidt/work/org-mode/lisp/org-duration hides /home/jschmidt/work/emacs-master/lisp/org/org-duration
/home/jschmidt/work/org-mode/lisp/ob-ocaml hides /home/jschmidt/work/emacs-master/lisp/org/ob-ocaml
/home/jschmidt/work/org-mode/lisp/org-fold hides /home/jschmidt/work/emacs-master/lisp/org/org-fold
/home/jschmidt/work/org-mode/lisp/ox-ascii hides /home/jschmidt/work/emacs-master/lisp/org/ox-ascii
/home/jschmidt/work/org-mode/lisp/ob-css hides /home/jschmidt/work/emacs-master/lisp/org/ob-css
/home/jschmidt/work/org-mode/lisp/ob-tangle hides /home/jschmidt/work/emacs-master/lisp/org/ob-tangle
/home/jschmidt/work/org-mode/lisp/ob-python hides /home/jschmidt/work/emacs-master/lisp/org/ob-python
/home/jschmidt/work/org-mode/lisp/org-crypt hides /home/jschmidt/work/emacs-master/lisp/org/org-crypt
/home/jschmidt/work/org-mode/lisp/ol-bibtex hides /home/jschmidt/work/emacs-master/lisp/org/ol-bibtex
/home/jschmidt/work/org-mode/lisp/oc-biblatex hides /home/jschmidt/work/emacs-master/lisp/org/oc-biblatex
/home/jschmidt/work/org-mode/lisp/org-protocol hides /home/jschmidt/work/emacs-master/lisp/org/org-protocol
/home/jschmidt/work/org-mode/lisp/org-feed hides /home/jschmidt/work/emacs-master/lisp/org/org-feed
/home/jschmidt/work/org-mode/lisp/ob-maxima hides /home/jschmidt/work/emacs-master/lisp/org/ob-maxima
/home/jschmidt/work/org-mode/lisp/org-colview hides /home/jschmidt/work/emacs-master/lisp/org/org-colview
/home/jschmidt/work/org-mode/lisp/ol-w3m hides /home/jschmidt/work/emacs-master/lisp/org/ol-w3m
/home/jschmidt/work/org-mode/lisp/ob-ditaa hides /home/jschmidt/work/emacs-master/lisp/org/ob-ditaa
/home/jschmidt/work/org-mode/lisp/ob-plantuml hides /home/jschmidt/work/emacs-master/lisp/org/ob-plantuml
/home/jschmidt/work/org-mode/lisp/org-src hides /home/jschmidt/work/emacs-master/lisp/org/org-src
/home/jschmidt/work/org-mode/lisp/ob-sed hides /home/jschmidt/work/emacs-master/lisp/org/ob-sed
/home/jschmidt/work/org-mode/lisp/ox-latex hides /home/jschmidt/work/emacs-master/lisp/org/ox-latex
/home/jschmidt/work/org-mode/lisp/ob-exp hides /home/jschmidt/work/emacs-master/lisp/org/ob-exp
/home/jschmidt/work/org-mode/lisp/ob-lua hides /home/jschmidt/work/emacs-master/lisp/org/ob-lua
/home/jschmidt/work/org-mode/lisp/ox hides /home/jschmidt/work/emacs-master/lisp/org/ox
/home/jschmidt/work/org-mode/lisp/org-footnote hides /home/jschmidt/work/emacs-master/lisp/org/org-footnote
/home/jschmidt/work/org-mode/lisp/ol-doi hides /home/jschmidt/work/emacs-master/lisp/org/ol-doi
/home/jschmidt/work/org-mode/lisp/ob-emacs-lisp hides /home/jschmidt/work/emacs-master/lisp/org/ob-emacs-lisp
/home/jschmidt/work/org-mode/lisp/ox-odt hides /home/jschmidt/work/emacs-master/lisp/org/ox-odt
/home/jschmidt/work/org-mode/lisp/ob-eval hides /home/jschmidt/work/emacs-master/lisp/org/ob-eval
/home/jschmidt/work/org-mode/lisp/ob-matlab hides /home/jschmidt/work/emacs-master/lisp/org/ob-matlab
/home/jschmidt/work/org-mode/lisp/ob-sass hides /home/jschmidt/work/emacs-master/lisp/org/ob-sass
/home/jschmidt/work/org-mode/lisp/ob-java hides /home/jschmidt/work/emacs-master/lisp/org/ob-java
/home/jschmidt/work/org-mode/lisp/ob-julia hides /home/jschmidt/work/emacs-master/lisp/org/ob-julia
/home/jschmidt/work/org-mode/lisp/org-version hides /home/jschmidt/work/emacs-master/lisp/org/org-version
/home/jschmidt/work/org-mode/lisp/ob-calc hides /home/jschmidt/work/emacs-master/lisp/org/ob-calc
/home/jschmidt/work/org-mode/lisp/org-table hides /home/jschmidt/work/emacs-master/lisp/org/org-table
/home/jschmidt/work/org-mode/lisp/ol-irc hides /home/jschmidt/work/emacs-master/lisp/org/ol-irc
/home/jschmidt/work/org-mode/lisp/ob-eshell hides /home/jschmidt/work/emacs-master/lisp/org/ob-eshell
/home/jschmidt/work/org-mode/lisp/org-fold-core hides /home/jschmidt/work/emacs-master/lisp/org/org-fold-core
/home/jschmidt/work/org-mode/lisp/org-macs hides /home/jschmidt/work/emacs-master/lisp/org/org-macs
/home/jschmidt/work/org-mode/lisp/ob-comint hides /home/jschmidt/work/emacs-master/lisp/org/ob-comint
/home/jschmidt/work/org-mode/lisp/ox-texinfo hides /home/jschmidt/work/emacs-master/lisp/org/ox-texinfo
/home/jschmidt/work/org-mode/lisp/oc-natbib hides /home/jschmidt/work/emacs-master/lisp/org/oc-natbib
/home/jschmidt/work/org-mode/lisp/ob-screen hides /home/jschmidt/work/emacs-master/lisp/org/ob-screen
/home/jschmidt/work/org-mode/lisp/ox-beamer hides /home/jschmidt/work/emacs-master/lisp/org/ox-beamer
/home/jschmidt/work/org-mode/lisp/ob-octave hides /home/jschmidt/work/emacs-master/lisp/org/ob-octave
/home/jschmidt/work/org-mode/lisp/ob hides /home/jschmidt/work/emacs-master/lisp/org/ob
/home/jschmidt/work/org-mode/lisp/ob-shell hides /home/jschmidt/work/emacs-master/lisp/org/ob-shell
/home/jschmidt/work/org-mode/lisp/ox-md hides /home/jschmidt/work/emacs-master/lisp/org/ox-md
/home/jschmidt/work/org-mode/lisp/oc hides /home/jschmidt/work/emacs-master/lisp/org/oc
/home/jschmidt/work/org-mode/lisp/org-ctags hides /home/jschmidt/work/emacs-master/lisp/org/org-ctags

Features:
(shadow sort mail-extr warnings icons emacsbug message yank-media puny
dired dired-loaddefs rfc822 mml mml-sec epa derived epg rfc6068
epg-config gnus-util time-date mm-decode mm-bodies mm-encode mail-parse
rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045
ietf-drums mm-util mail-prsvr mail-utils compile text-property-search
comint ansi-osc ansi-color ring comp-run comp-common rx time delsel
cus-load advice finder-inf info simple-httpd-autoloads xr-autoloads
yaml-mode-autoloads package browse-url url url-proxy url-privacy
url-expand url-methods url-history url-cookie generate-lisp-file
url-domsuf url-util mailcap url-handlers url-parse auth-source cl-seq
eieio eieio-core cl-macs password-cache json subr-x map byte-opt gv
bytecomp byte-compile url-vars cl-loaddefs cl-lib rmc iso-transl tooltip
cconv eldoc paren electric uniquify ediff-hook vc-hooks lisp-float-type
elisp-mode mwheel term/x-win x-win term/common-win x-dnd touch-screen
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock
font-lock syntax font-core term/tty-colors frame minibuffer nadvice seq
simple cl-generic indonesian philippine cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite emoji-zwj charscript charprop case-table
epa-hook jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button
loaddefs theme-loaddefs faces cus-face macroexp files window
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget keymap hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
cairo gtk x-toolkit xinput2 x multi-tty move-toolbar
make-network-process native-compile emacs)

Memory information:
((conses 16 164401 15077) (symbols 48 10806 0)
 (strings 32 49484 1976) (string-bytes 1 1619889)
 (vectors 16 29939) (vector-slots 8 496302 15958)
 (floats 8 54 52) (intervals 56 337 0) (buffers 984 12))

bug#67718: 30.0.50; prog-fill-reindent-defun vs. fill-paragraph for strings and comments

[Dmitry Gutov]

On 08/12/2023 23:53, Jens Schmidt wrote:
>    "any" prog mode
> 
>      "after a string"|

This might be a good idea.

>      |
>      "on an empty line before a string"

It's not obvious to me that filling would be preferable to reindenting, 
in this case.

>        This one could differ in a treesitter mode, I guess.
> 
>        sql-mode doesn't even have <-comments, so you always have to move
>        point*into*  a comment to get it filled.  BTW, exactly like
>        javascript-mode, which is a bit more important, probably:

So we could look for 1-2 more different comment syntaxes, right? And 
maybe string openers.

Would you like to propose a small patch?

>      (foo bar baz ?\|; baz bar foo)
> 
>        So this goes the other direction: If you M-q here and expect the
>        line or surrounding function to be indented, you will get
>        disappointed, because the heuristics detects this as a comment
>        that needs to be filled.

In this case it's easy enough to understand what's going on, I think, 
and move out of the string. But other suggestions welcome.

bug#67718: 30.0.50; prog-fill-reindent-defun vs. fill-paragraph for strings and comments

[Jens Schmidt via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Dmitry Gutov <dmitry@gutov.dev> writes:

>>      |
>>      "on an empty line before a string"
>
> It's not obvious to me that filling would be preferable to
> reindenting, in this case.

This is to retain UX compatibility with the Emacs 29 behavior, where
`fill-paragraph' would fill the current paragraph or the next one, when
on whitespace-only lines before a paragraph.

To what extent does UX compatibility concern you, BTW?

>>        This one could differ in a treesitter mode, I guess.  sql-mode
>>        doesn't even have <-comments, so you always have to move
>>        point*into* a comment to get it filled.  BTW, exactly like
>>        javascript-mode, which is a bit more important, probably:
>
> So we could look for 1-2 more different comment syntaxes, right?
> And maybe string openers.
>
> Would you like to propose a small patch?

No, I only start working for big patches :-)

But more seriously, that might not be fixable by a "small patch", since
AFAICT there is no simple regexp to match an arbitrary two-character
comment starter.  To some extent that is also reflected by the history
of that function, where various generic approaches to detect comments
before point have been tried, but none really successfully, IMO.

So in other words: I can start working on this, but then I'd like to
have a commitment that my work is not spent in vain, even if it comes
out more complex than you'd currently expect.  See below.

>>      (foo bar baz ?\|; baz bar foo)
>>        So this goes the other direction: If you M-q here and
>> expect the
>>        line or surrounding function to be indented, you will get
>>        disappointed, because the heuristics detects this as a comment
>>        that needs to be filled.
>
> In this case it's easy enough to understand what's going on, I
> think, and move out of the string. But other suggestions welcome.

I tried to come up with something easy that'd avoid the edge cases from
my previous post, but that turned out to be harder than I thought.
So the rules are more complex than what we have right now in Emacs 30,
but I hope they fulfill at least the principle of least surprise.

So I'd fill in function `prog-fill-reindent-defun'

  - unless the user has opted out from that (still to be discussed in
    bug#67462) or

  - if point is on or before (UX compatibility!) a line consisting
    solely of one, possibly partial contiguous string or comment or

  - if point is properly inside a string or a /* ... */-style comment
    already spanning more than one line or

  - if point is properly inside a //-style or ;-style comment.

Otherwise I'd reindent.  Examples:

  |
  "fill this
  string"

  "fill |this
  string"

  "fill this
  string"|

  "fill |this string"

  "fill this string"|

  |
  // fill this
  // comment

  // fill |this
  // comment

  // fill this
  // comment|

  // fill |this comment

  // fill this comment|

  |
  /* fill this
  comment */

  /* fill |this
  comment */

  /* fill this
  comment */|

  /* fill |this comment */

  /* fill this comment */|

  /* do |not fill */ /* non-contiguous comment */

  foo = "fill this|
  string";

  foo = /* fill this|
  comment */ 123;

  foo = "do not |fill";

  foo = /* do not |fill */ 123;

  (foo bar baz) |; do not fill

  (foo bar baz) ;| fill this comment

  select 1 from foo |-- do not fill

  select 1 from foo -|- do not fill

  select 1 from foo --| fill this comment

What do you think?

It should be possible to implement that with reasonable effort, but the
result would definitely look less appealing than what we currently have.

Plus we haven't discussed the treesitter part yet ...