bug#56796: 29.0.50; Hard newlines not respected in code comments?

bug#56796: 29.0.50; Hard newlines not respected in code comments?

[Visuwesh]

Are hard newlines supposed to be respected when they are in code
comments?  To demonstrate what I mean consider the two following cases
in emacs -Q and after M-x use-hard-newlines RET,

1. Paste the following lisp comment in the *scratch* buffer

       ;; This buffer is for text that is not saved, and for Lisp.
       ;; To create a file, visit it with C-x C-f and enter text in its buffer.
       
   And make the newline after "Lisp." hard (RET C-k works).  Then
   say M-q, "To create" creeps up to the first line.

2. Paste the following text in a text-mode buffer

       This buffer is for text that is not saved, and for Lisp.
       To create a file, visit it with C-x C-f and enter text in its buffer.

   Then make the newline after "Lisp." hard again.  Say M-q now, "To
   create" stays in the second line.

(1) happens in python-mode and sh-mode too so I don't think it is
major-mode specific?

In GNU Emacs 29.0.50 (build 26, x86_64-pc-linux-gnu, X toolkit, Xaw scroll bars)
 of 2022-07-22 built on astatine
Repository revision: 12a3137cd381cb743768033e789b900b015041d7
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12101003
System Description: Debian GNU/Linux bookworm/sid

Configured using:
 'configure --with-sound=alsa --with-x-toolkit=lucid --with-json
 --without-xaw3d --without-gconf --without-libsystemd --without-cairo'
Configured features:
ACL DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG JSON
LIBOTF LIBSELINUX LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PNG RSVG
SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS WEBP X11 XDBE XFT
XIM XINPUT2 XPM LUCID ZLIB
Important settings:
  value of $LC_MONETARY: ta_IN.UTF-8
  value of $LC_NUMERIC: ta_IN.UTF-8
  value of $LANG: en_GB.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Info

Minor modes in effect:
  recentf-mode: t
  text-scale-mode: t
  shell-dirtrack-mode: t
  eros-mode: t
  pdf-occur-global-minor-mode: t
  minibuffer-depth-indicate-mode: t
  repeat-mode: t
  display-time-mode: t
  display-battery-mode: t
  winner-mode: t
  delete-selection-mode: t
  straight-use-package-mode: t
  straight-package-neutering-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  use-hard-newlines: t
  tab-bar-mode: t
  file-name-shadow-mode: t
  isearch-fold-quotes-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  undelete-frame-mode: t
  buffer-read-only: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
/home/viz/lib/emacs/straight/build/faceup/faceup hides /home/viz/lib/ports/emacs/lisp/emacs-lisp/faceup

Features:
(shadow ecomplete emacsbug view xref expand-region text-mode-expansions
cc-mode-expansions the-org-mode-expansions er-basic-expansions
expand-region-core expand-region-custom org-element avl-tree generator
org-capture doct pdf-sync pdf-annot facemenu pdf-outline pdf-links
pdf-history descr-text wdired misc tramp-archive tramp-gvfs tramp
tramp-loaddefs trampver tramp-integration cus-start tramp-compat ls-lisp
nov ob-C cc-mode cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align
cc-engine cc-vars cc-defs ob-shell ob-racket async ob-async cdlatex
texmathp ol-eww ol-rmail ol-mhe ol-irc ol-info ol-gnus nnselect
ol-docview doc-view ol-bibtex ol-bbdb ol-w3m ol-doi org-link-doi
org-tempo tempo org-id org-refile ol-man org ob ob-tangle ob-ref ob-lob
ob-table ob-exp org-macro org-footnote org-src ob-comint org-pcomplete
org-list org-faces org-entities org-version ob-emacs-lisp ob-core
ob-eval org-table oc-basic bibtex ol org-keys oc org-compat org-macs
org-loaddefs esxml-query dired-aux gnus-dired timezone
display-line-numbers ement-room-list ement ement-notify notifications
ement-room ement-lib ement-api ement-structs plz ement-macros
taxy-magit-section magit-section taxy ewoc dns rfc1345 dabbrev
smerge-mode log-edit add-log avy flyspell ispell time-stamp shortdoc
completion pulse color reveal noutline outline bug-reference recentf
tree-widget misearch multi-isearch cl-print help-fns radix-tree
mule-util apropos url-http url-gw url-cache url-auth eww xdg url-queue
mm-url gnus-cite mm-archive mail-extr textsec uni-scripts idna-mapping
ucs-normalize uni-confusable textsec-check gnus-bcklg qp network-stream
nsm gnus-async sort gnus-ml nndraft nnmh nnfolder nnmaildir nnagent nnml
vc-backup log-view pcvs-util vc vc-git diff-mode vc-dispatcher nnnil
gnus-agent gnus-srvr gnus-score score-mode nnvirtual gnus-msg gnus-art
mm-uu mml2015 mm-view mml-smime smime gnutls dig nntp gnus-cache
gnus-sum shr pixel-fill kinsoku url-file url-dired svg gnus-group
gnus-undo gnus-start gnus-dbus gnus-cloud nnimap nnmail mail-source utf7
netrc nnoo parse-time iso8601 gnus-spec gnus-int gnus-range message
sendmail yank-media puny rfc822 mml mml-sec epa epg rfc6068 epg-config
mm-decode mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045
ietf-drums mailabbrev gmm-utils mailheader gnus-win gnus nnheader
gnus-util mail-utils range mm-util mail-prsvr face-remap sh-script smie
executable files-x shell-command+ diff cursor-sensor shell pcomplete
server paredit edmacro kmacro eros time-date checkdoc lisp-mnt
flymake-proc flymake project warnings thingatpt wordel-autoloads
sokoban-autoloads ement-autoloads svg-lib-autoloads
taxy-magit-section-autoloads magit-section-autoloads dash-autoloads
taxy-autoloads plz-autoloads nov-autoloads esxml-autoloads kv-autoloads
transmission-autoloads lua-mode-autoloads nix-mode-autoloads
racket-mode-autoloads pos-tip-autoloads faceup-autoloads eros-autoloads
flymake-shellcheck-autoloads writegood-mode-autoloads
siege-mode-autoloads paredit-autoloads puni-autoloads
expand-region-autoloads filladapt-autoloads compose quail
scroll-other-window org-pdftools-autoloads org-noter-autoloads
change-env-autoloads math-delimiters-autoloads doct-autoloads
ob-async-autoloads async-autoloads emacs-ob-racket-autoloads
valign-autoloads cdlatex-autoloads auctex-autoloads tex-site pdf-occur
ibuf-ext ibuffer ibuffer-loaddefs tablist advice tablist-filter
semantic/wisent/comp semantic/wisent semantic/wisent/wisent
semantic/util-modes semantic/util semantic semantic/tag semantic/lex
semantic/fw mode-local cedet pdf-isearch let-alist pdf-misc imenu
pdf-tools package browse-url url url-proxy url-privacy url-expand
url-methods url-history url-cookie generate-lisp-file url-domsuf
url-util mailcap url-handlers url-parse auth-source eieio eieio-core
json map byte-opt url-vars compile comint ansi-color cus-edit edebug
debug backtrace find-func wid-edit pdf-view password-cache jka-compr
pdf-cache pdf-info tq pdf-util pdf-macs image-mode dired-x dired
dired-loaddefs exif pdf-tools-autoloads tablist-autoloads mb-depth
repeat visual-fill-autoloads olivetti-autoloads time format-spec battery
dbus filenotify xml dom disp-table lacarte-autoloads
shell-command-plus-autoloads winner ring delsel cus-load easy-mmode
avy-autoloads finder-inf vc-backup-autoloads compat-autoloads icalendar
diary-lib diary-loaddefs cal-menu calendar cal-loaddefs rx filecache
imenu-xref-autoloads derived chemtable-autoloads molar-mass-autoloads
saveplace-pdf-view saveplace bookmark text-property-search pp
saveplace-pdf-view-autoloads pcase inspector-autoloads xr-autoloads
straight-autoloads cl-seq info cl-extra help-mode straight subr-x
cl-macs gv cl-loaddefs cl-lib bytecomp byte-compile cconv vz-nh-theme
vz-options-theme rmc iso-transl tooltip eldoc paren electric uniquify
ediff-hook vc-hooks lisp-float-type elisp-mode mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode register
page tab-bar menu-bar rfn-eshadow isearch easymenu timer select
scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer nadvice seq simple cl-generic indonesian philippine
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs faces cus-face macroexp files window
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget keymap hashtable-print-readable backquote threads dbusbind
inotify dynamic-setting system-font-setting font-render-setting
x-toolkit xinput2 x multi-tty make-network-process emacs)

Memory information:
((conses 16 1185460 222926) (symbols ?0 54909 174) (strings 32 331022 18751) (string-bytes 1 15084068) (vectors 16 124750) (vector-slots 8 2627127 187625) (floats 8 1200 3465) (intervals ?8 120571 4446) (buffers 992 ?P))

bug#56796: 29.0.50; Hard newlines not respected in code comments?

[Eli Zaretskii]

> From: Visuwesh <visuweshm@gmail.com>
> Date: Wed, 27 Jul 2022 21:19:29 +0530
> 
> Are hard newlines supposed to be respected when they are in code
> comments?

The answer is "it depends", AFAICS.

The main problem is that fill-comment-paragraph doesn't seem to honor
use-hard-newlines.  So any major mode whose fill-paragraph-function
uses that for filling comments will fail to pay attention to hard
newlines in comments.  In your scenario, if I set both
fill-paragraph-function and fill-paragraph-handle-comment to nil, hard
newlines in comments are honored as expected.

use-hard-newlines is weird: it is documented only for Enriched mode,
but that is at least incomplete, because we call functions from
fill.el high and low in many major modes, so at least some of them
inherit the use-hard-newlines-specific code when they call functions
which do.  For example, AFAICT use-hard-newlines is supported in Lisp
doc strings.

Adding Stefan to CC, who write fill-comment-paragraph.  If we want
use-hard-newlines to be supported in comments, we should modify
fill-comment-paragraph to honor it in some way, perhaps simply
deferring to fill-region in that case.

bug#56796: 29.0.50; Hard newlines not respected in code comments?

[Visuwesh]

[வியாழன் ஜூலை 28, 2022] Eli Zaretskii wrote:

>> From: Visuwesh <visuweshm@gmail.com>
>> Date: Wed, 27 Jul 2022 21:19:29 +0530
>> 
>> Are hard newlines supposed to be respected when they are in code
>> comments?
>
> The answer is "it depends", AFAICS.
>
> The main problem is that fill-comment-paragraph doesn't seem to honor
> use-hard-newlines.  So any major mode whose fill-paragraph-function
> uses that for filling comments will fail to pay attention to hard
> newlines in comments.  In your scenario, if I set both
> fill-paragraph-function and fill-paragraph-handle-comment to nil, hard
> newlines in comments are honored as expected.

Thanks for the explanation, I figured that was the case.

> If we want use-hard-newlines to be supported in comments, we should
> modify fill-comment-paragraph to honor it in some way, perhaps simply
> deferring to fill-region in that case.

That would be most welcome.  Hard newlines was the "missing key" in
making writing plain text more easy and enjoyable to me.

bug#56796: 29.0.50; Hard newlines not respected in code comments?

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Adding Stefan to CC, who write fill-comment-paragraph.  If we want
> use-hard-newlines to be supported in comments, we should modify
> fill-comment-paragraph to honor it in some way, perhaps simply
> deferring to fill-region in that case.

It was many years ago, but IIRC the autor of that code presumed it was
used in "code" and he also presumed that hard newlines are only for
"text", and furthermore it presumed that the two are mutually exclusive.
He was apparently a quite naive young fellow,

At the same time, I'm not sure how important it is to handle hard
newlines in ELisp comments, so I'm not sure how important it is to
fix this.  This bug report only gives a (good) recipe but not a good
reason for doing such a thing, so maybe some context explaining how/why
such hard newlines can appear in comments would help motivate a fix.


        Stefan

bug#56796: 29.0.50; Hard newlines not respected in code comments?

[Visuwesh]

[வெள்ளி ஜூலை 29, 2022] Stefan Monnier via "Bug reports for GNU Emacs, the Swiss army knife of text editors" wrote:

> At the same time, I'm not sure how important it is to handle hard
> newlines in ELisp comments, so I'm not sure how important it is to
> fix this.  This bug report only gives a (good) recipe but not a good
> reason for doing such a thing, so maybe some context explaining how/why
> such hard newlines can appear in comments would help motivate a fix.

Well, wrt Elisp comments at least, we have the Commentary section where
we write quite a lot about the package in question where having hard
newlines would be really handy since you write those text like you do in
a regular text buffer.

More generally: I'm not sure how common this pattern is but I tend to do

    ;; TODO/FIXME: Something...
    ;; Ideas and thoughts on how to clear it here.

Now if you do M-q there, your neatly arranged text is destroyed.  (I am
probably biased) I also catch this pattern in the Emacs source tree as
well.

[ Although not related to Elisp: Documentation for functions/whatever in
  Go are written as comments.  ]

bug#56796: 29.0.50; Hard newlines not respected in code comments?

[Visuwesh]

[வெள்ளி ஜூலை 29, 2022] Visuwesh wrote:

>> At the same time, I'm not sure how important it is to handle hard
>> newlines in ELisp comments, so I'm not sure how important it is to
>> fix this.  This bug report only gives a (good) recipe but not a good
>> reason for doing such a thing, so maybe some context explaining how/why
>> such hard newlines can appear in comments would help motivate a fix.

But maybe me using hard newlines to tame adaptive-fill-mode is the wrong
approach: I would like to write lists such as

    . blah blah blah
    . blah blah blah

without it getting filled to

    . blah blah blah. blah blah blah

I can't blame adaptive-fill-mode for it: it can only be ever so smart
and hard newlines seemed like the solution for "when I say newline, I
MEAN IT".

bug#56796: 29.0.50; Hard newlines not respected in code comments?

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> But maybe me using hard newlines to tame adaptive-fill-mode is the wrong
> approach: I would like to write lists such as
>
>     . blah blah blah
>     . blah blah blah

I'm not sure "." should be accepted by default, but...

> I can't blame adaptive-fill-mode for it: it can only be ever so smart
> and hard newlines seemed like the solution for "when I say newline, I
> MEAN IT".

I do think it's worth a bug report, because we should handle at least
some "common" cases of itemized lists in ELisp comments, and AFAIK we
currently don't handle any at all.


        Stefan

bug#56796: 29.0.50; Hard newlines not respected in code comments?

[Visuwesh]

[வெள்ளி ஜூலை 29, 2022] Stefan Monnier via "Bug reports for GNU Emacs, the Swiss army knife of text editors" wrote:

>> More generally: I'm not sure how common this pattern is but I tend to do
>>
>>     ;; TODO/FIXME: Something...
>>     ;; Ideas and thoughts on how to clear it here.
>>
>> Now if you do M-q there, your neatly arranged text is destroyed.  (I am
>> probably biased) I also catch this pattern in the Emacs source tree as
>> well.
>
> It never occurred to me to use hard newlines for such cases.  What I use
> instead is either:
> - add extra newlines to mark the paragraph boundaries to use
>   (that's just as easy as adding hard newlines, but requires removing
>   those extra newlines afterward).

This is what I did but frankly I am really tired of this pattern since
when you go back to revise the text, you have to do the thing yet again.
(Granted, you only avoid this when you haven't killed the buffer or
Emacs but we don't do that here, do we?  ;-)

> - Select the intended paragraph rather than rely on M-q's automatic
>   decision of what's a paragraph.

I always forget that M-q takes an active region, this is a viable option
as well.  I think the first option is quicker though.

> Hard newlines are a good idea, for this use-case, indeed, tho it would
> be even better if we could somehow represent that info in the text
> itself so it's properly saved into the file.

You're not the first one: https://yhetil.org/emacs-devel/CAArVCkQHmwqHvcOEurBYLcWS74nov8OAH4fnX8XbUr2JY2nCKA@mail.gmail.com/

[வெள்ளி ஜூலை 29, 2022] Stefan Monnier via "Bug reports for GNU Emacs, the Swiss army knife of text editors" wrote:

>> But maybe me using hard newlines to tame adaptive-fill-mode is the wrong
>> approach: I would like to write lists such as
>>
>>     . blah blah blah
>>     . blah blah blah
>
> I'm not sure "." should be accepted by default, but...

Indeed, it's a bit too "out there" if you know what I mean.

>> I can't blame adaptive-fill-mode for it: it can only be ever so smart
>> and hard newlines seemed like the solution for "when I say newline, I
>> MEAN IT".
>
> I do think it's worth a bug report, because we should handle at least
> some "common" cases of itemized lists in ELisp comments, and AFAIK we
> currently don't handle any at all.

When I was searching for discussions on adaptive-fill-mode in
emacs-devel to make sense of it, I found the past you agreeing with me
in the wild.  The discussion can be found here: https://yhetil.org/emacs-devel/jwvk5wvx5jv.fsf-monnier+emacs@gnu.org/
but I'm not sure what happened afterwards.  Hmm... but now that I
re-read it, it is not the same problem but it is a problem that
often bites me in the back.

bug#56796: 29.0.50; Hard newlines not respected in code comments?

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> More generally: I'm not sure how common this pattern is but I tend to do
>
>     ;; TODO/FIXME: Something...
>     ;; Ideas and thoughts on how to clear it here.
>
> Now if you do M-q there, your neatly arranged text is destroyed.  (I am
> probably biased) I also catch this pattern in the Emacs source tree as
> well.

It never occurred to me to use hard newlines for such cases.  What I use
instead is either:
- add extra newlines to mark the paragraph boundaries to use
  (that's just as easy as adding hard newlines, but requires removing
  those extra newlines afterward).
- Select the intended paragraph rather than rely on M-q's automatic
  decision of what's a paragraph.

Hard newlines are a good idea, for this use-case, indeed, tho it would
be even better if we could somehow represent that info in the text
itself so it's properly saved into the file.


        Stefan