bug#31656: 26.1; `fill-paragraph' malformats in emacs-lisp-mode

bug#31656: 26.1; `fill-paragraph' malformats in emacs-lisp-mode

[Stefan Guath]

[-- Attachment #1: Type: text/plain, Size: 2032 bytes --]

VERSION
GNU Emacs 26.1 (build 1, x86_64-apple-darwin14.5.0, NS appkit-1348.17
Version 10.10.5 (Build 14F2511)) of 2018-05-28

DESCRIPTION
`emacs-lisp-docstring-fill-column' shadows `fill-column' in too many cases
in emacs-lisp-mode. The documentation of `emacs-lisp-docstring-fill-column'
states: "Value of ‘fill-column’ to use when filling a docstring". But it
incorrectly seems to be used in a lot more cases than just in a docstring
(the only case that I've found where ‘fill-column’ is actually respected is
within comments). A work-around is to set
`emacs-lisp-docstring-fill-column' to nil, but it would be nice to have it
working properly instead.

I might be missing something, but think the incorrect behavior is to be
found in `lisp-fill-paragraph' that is invoked by `fill-paragraph' through
`fill-paragraph-function'. It seems like `lisp-fill-paragraph'
unconditionally sets `fill-column' to `emacs-lisp-docstring-fill-column'
without checking whether point is within a doc string first. The only
requirements for enable shadowing currently seems to be "(and (integerp
emacs-lisp-docstring-fill-column) (derived-mode-p 'emacs-lisp-mode))",
which doesn't seems sufficient.

TO REPRODUCE
* emacs -q
* C-x b <return> (i.e. switch to *scratch* buffer)
* M-x emacs-lisp-mode
* M-x column-number-mode (optional)
* (setq fill-column 80) ;used by `fill-paragraph'
* (setq emacs-lisp-docstring-fill-column 40) ;should override `fill-column'
in doc strings only!
* Paste in the line "word00 word01 word02 word03 word04 word05 word06
word07 word08 word09 word10 word11" (i.e. 12 words, 83 chars)
* M-x fill-paragraph (with point on pasted line above)
* The value of `emacs-lisp-docstring-fill-column' (40) is incorrectly used
when formatting. The value of `fill-column' (80) should have been used
instead, since the point is not in a doc string.

Note that the default value of `emacs-lisp-docstring-fill-column' is 65
(rather than nil), so this malformatting is enabled by default.

[-- Attachment #2: Type: text/html, Size: 2391 bytes --]

bug#31656: 26.1; `fill-paragraph' malformats in emacs-lisp-mode

[Eli Zaretskii]

> From: Stefan Guath <stefan@automata.se>
> Date: Wed, 30 May 2018 14:50:01 +0200
> 
> `emacs-lisp-docstring-fill-column' shadows `fill-column' in too many cases in emacs-lisp-mode. The
> documentation of `emacs-lisp-docstring-fill-column' states: "Value of ‘fill-column’ to use when filling a
> docstring". But it incorrectly seems to be used in a lot more cases than just in a docstring (the only case that
> I've found where ‘fill-column’ is actually respected is within comments). A work-around is to set
> `emacs-lisp-docstring-fill-column' to nil, but it would be nice to have it working properly instead.
> 
> I might be missing something, but think the incorrect behavior is to be found in `lisp-fill-paragraph' that is
> invoked by `fill-paragraph' through `fill-paragraph-function'. It seems like `lisp-fill-paragraph' unconditionally
> sets `fill-column' to `emacs-lisp-docstring-fill-column' without checking whether point is within a doc string
> first. The only requirements for enable shadowing currently seems to be "(and (integerp
> emacs-lisp-docstring-fill-column) (derived-mode-p 'emacs-lisp-mode))", which doesn't seems sufficient.

AFAICT, this behavior was in Emacs since about forever (since 1995, t
be precise).  So maybe we just need to adjust the doc string to
reflect the reality?

Or are there real-life use cases where this behavior is grossly
inappropriate?

bug#31656: 26.1; `fill-paragraph' malformats in emacs-lisp-mode

[Noam Postavsky]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Guath <stefan@automata.se>
>> Date: Wed, 30 May 2018 14:50:01 +0200
>> 
>> `emacs-lisp-docstring-fill-column' shadows `fill-column' in too many cases in emacs-lisp-mode. The
>> documentation of `emacs-lisp-docstring-fill-column' states: "Value of ‘fill-column’ to use when filling a
>> docstring". But it incorrectly seems to be used in a lot more cases than just in a docstring (the only case that
>> I've found where ‘fill-column’ is actually respected is within comments). A work-around is to set
>> `emacs-lisp-docstring-fill-column' to nil, but it would be nice to have it working properly instead.
>> 
>> I might be missing something, but think the incorrect behavior is to be found in `lisp-fill-paragraph' that is
>> invoked by `fill-paragraph' through `fill-paragraph-function'. It seems like `lisp-fill-paragraph' unconditionally
>> sets `fill-column' to `emacs-lisp-docstring-fill-column' without checking whether point is within a doc string
>> first. The only requirements for enable shadowing currently seems to be "(and (integerp
>> emacs-lisp-docstring-fill-column) (derived-mode-p 'emacs-lisp-mode))", which doesn't seems sufficient.
>
> AFAICT, this behavior was in Emacs since about forever (since 1995, t
> be precise).  So maybe we just need to adjust the doc string to
> reflect the reality?
>
> Or are there real-life use cases where this behavior is grossly
> inappropriate?

I don't think it makes sense to apply normal plain text filling rules to
code.  Maybe it doesn't come up much because people don't usually call
M-q on code, and usually lines of code are kept short enough that they
wouldn't get filled anyway.  But picking a random example from rgrep
^.\{100,\}$ on the Emacs code base:

(defun feedmail-default-date-generator (maybe-file)
  "Default function for generating Date: header contents."
  (feedmail-say-debug ">in-> feedmail-default-date-generator")
  (when maybe-file
    (feedmail-say-debug (concat "4 cre " (feedmail-rfc822-date (nth 4 (file-attributes maybe-file)))))
    (feedmail-say-debug (concat "5 mod " (feedmail-rfc822-date (nth 5 (file-attributes maybe-file)))))
    (feedmail-say-debug (concat "6 sta " (feedmail-rfc822-date (nth 6 (file-attributes maybe-file))))))
  (let ((date-time))
    (if (and (not feedmail-queue-use-send-time-for-date) maybe-file)
	(setq date-time (nth 5 (file-attributes maybe-file))))
    (feedmail-rfc822-date date-time))
  )

Running M-q on every line turns it into this nonsense:

(defun feedmail-default-date-generator (maybe-file)
  "Default function for generating Date: header contents."
  (feedmail-say-debug ">in-> feedmail-default-date-generator")
  (when maybe-file
    (feedmail-say-debug (concat "4
    cre " (feedmail-rfc822-date (nth 4 (file-attributes
    maybe-file)))))
    (feedmail-say-debug (concat "5
    mod " (feedmail-rfc822-date (nth 5 (file-attributes
    maybe-file)))))
    (feedmail-say-debug (concat "6
    sta " (feedmail-rfc822-date (nth 6 (file-attributes
    maybe-file))))))
  (let ((date-time))
    (if (and (not feedmail-queue-use-send-time-for-date)
    maybe-file)
	(setq date-time (nth 5 (file-attributes maybe-file))))
    (feedmail-rfc822-date date-time)) )

bug#31656: 26.1; `fill-paragraph' malformats in emacs-lisp-mode

[Stefan Guath]

[-- Attachment #1: Type: text/plain, Size: 4801 bytes --]

The very idea of shadowing the global `fill-column' in the first place
breaks the Principle Of Least Astonishment and introduces annoyance and
extra work for the user when trying to understand why the explicit setting
of `fill-column' is not respected. If it is to be done anyway, it needs to
be well motivated and also executed in a correct way.

So, I guess my basic question is what value
`emacs-lisp-docstring-fill-column' tries to add? It's obviously not aiming
to introduce a general shadowing for elisp files only, since that mechanism
is already present due to `fill-column' being buffer local. Rather,
according to its docs, it tries to add selective shadowing for elisp-files
in doc strings only. Fair enough, that is an edge case where a shadowing
mechanism could be motivated. But instead it implements some kind of
unpredictable shadowing - I don't even understand in what exact
circumstances the shadowing takes place. If we were to just update its
docs, what should it be updated to?

To me this entire functionality should either be removed or fixed so that
its behavior reflects its current docs. In its current state it just
malformats elisp files in unexpected ways. Or am I missing something?


On Fri, Jun 1, 2018 at 11:39 AM Noam Postavsky <npostavs@gmail.com> wrote:

> Eli Zaretskii <eliz@gnu.org> writes:
>
> >> From: Stefan Guath <stefan@automata.se>
> >> Date: Wed, 30 May 2018 14:50:01 +0200
> >>
> >> `emacs-lisp-docstring-fill-column' shadows `fill-column' in too many
> cases in emacs-lisp-mode. The
> >> documentation of `emacs-lisp-docstring-fill-column' states: "Value of
> ‘fill-column’ to use when filling a
> >> docstring". But it incorrectly seems to be used in a lot more cases
> than just in a docstring (the only case that
> >> I've found where ‘fill-column’ is actually respected is within
> comments). A work-around is to set
> >> `emacs-lisp-docstring-fill-column' to nil, but it would be nice to have
> it working properly instead.
> >>
> >> I might be missing something, but think the incorrect behavior is to be
> found in `lisp-fill-paragraph' that is
> >> invoked by `fill-paragraph' through `fill-paragraph-function'. It seems
> like `lisp-fill-paragraph' unconditionally
> >> sets `fill-column' to `emacs-lisp-docstring-fill-column' without
> checking whether point is within a doc string
> >> first. The only requirements for enable shadowing currently seems to be
> "(and (integerp
> >> emacs-lisp-docstring-fill-column) (derived-mode-p 'emacs-lisp-mode))",
> which doesn't seems sufficient.
> >
> > AFAICT, this behavior was in Emacs since about forever (since 1995, t
> > be precise).  So maybe we just need to adjust the doc string to
> > reflect the reality?
> >
> > Or are there real-life use cases where this behavior is grossly
> > inappropriate?
>
> I don't think it makes sense to apply normal plain text filling rules to
> code.  Maybe it doesn't come up much because people don't usually call
> M-q on code, and usually lines of code are kept short enough that they
> wouldn't get filled anyway.  But picking a random example from rgrep
> ^.\{100,\}$ on the Emacs code base:
>
> (defun feedmail-default-date-generator (maybe-file)
>   "Default function for generating Date: header contents."
>   (feedmail-say-debug ">in-> feedmail-default-date-generator")
>   (when maybe-file
>     (feedmail-say-debug (concat "4 cre " (feedmail-rfc822-date (nth 4
> (file-attributes maybe-file)))))
>     (feedmail-say-debug (concat "5 mod " (feedmail-rfc822-date (nth 5
> (file-attributes maybe-file)))))
>     (feedmail-say-debug (concat "6 sta " (feedmail-rfc822-date (nth 6
> (file-attributes maybe-file))))))
>   (let ((date-time))
>     (if (and (not feedmail-queue-use-send-time-for-date) maybe-file)
>         (setq date-time (nth 5 (file-attributes maybe-file))))
>     (feedmail-rfc822-date date-time))
>   )
>
> Running M-q on every line turns it into this nonsense:
>
> (defun feedmail-default-date-generator (maybe-file)
>   "Default function for generating Date: header contents."
>   (feedmail-say-debug ">in-> feedmail-default-date-generator")
>   (when maybe-file
>     (feedmail-say-debug (concat "4
>     cre " (feedmail-rfc822-date (nth 4 (file-attributes
>     maybe-file)))))
>     (feedmail-say-debug (concat "5
>     mod " (feedmail-rfc822-date (nth 5 (file-attributes
>     maybe-file)))))
>     (feedmail-say-debug (concat "6
>     sta " (feedmail-rfc822-date (nth 6 (file-attributes
>     maybe-file))))))
>   (let ((date-time))
>     (if (and (not feedmail-queue-use-send-time-for-date)
>     maybe-file)
>         (setq date-time (nth 5 (file-attributes maybe-file))))
>     (feedmail-rfc822-date date-time)) )
>

[-- Attachment #2: Type: text/html, Size: 5712 bytes --]

bug#31656: 26.1; `fill-paragraph' malformats in emacs-lisp-mode

[Eli Zaretskii]

> From: Stefan Guath <stefan@automata.se>
> Date: Fri, 1 Jun 2018 12:36:00 +0200
> Cc: eliz@gnu.org, 31656@debbugs.gnu.org
> 
> The very idea of shadowing the global `fill-column' in the first place breaks the Principle Of Least
> Astonishment

If that is so, then we should have gobs of astonished users since 1995.

> and introduces annoyance and extra work for the user when trying to understand why the
> explicit setting of `fill-column' is not respected.

Documenting the current behavior should go a long way towards helping
users understand that, right?

> So, I guess my basic question is what value `emacs-lisp-docstring-fill-column' tries to add?

From my POV, the answer is clear: it allows users to have different
customizable defaults for fill-column in Emacs Lisp and elsewhere.
E.g., in text modes, it is customary to enlarge the default to 79 or
thereabouts, but in Emacs Lisp we generally say that good style is to
make lines in doc strings no wider than 60 characters (see the ELisp
manual).

> To me this entire functionality should either be removed or fixed so that its behavior reflects its current docs. In
> its current state it just malformats elisp files in unexpected ways. Or am I missing something?

I'm sorry, but after so many years we had this working, removing this
functionality is out of the question.  I suggested to fix the docs to
match the implementation; if someone has other suggestions that don't
break existing behavior, or at least let users get back existing
behavior, please describe those suggestions.

Thanks.

bug#31656: 26.1; `fill-paragraph' malformats in emacs-lisp-mode

[Stefan Guath]

[-- Attachment #1: Type: text/plain, Size: 3659 bytes --]

On Fri, Jun 1, 2018 at 2:52 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Stefan Guath <stefan@automata.se>
> > Date: Fri, 1 Jun 2018 12:36:00 +0200
> > Cc: eliz@gnu.org, 31656@debbugs.gnu.org
> >
> > The very idea of shadowing the global `fill-column' in the first place
> breaks the Principle Of Least
> > Astonishment
>
> If that is so, then we should have gobs of astonished users since 1995.
>

Yes, the streets are flooded with them! But Emacs users are a tough crowd
that don't complain :)


> > and introduces annoyance and extra work for the user when trying to
> understand why the
> > explicit setting of `fill-column' is not respected.
>
> Documenting the current behavior should go a long way towards helping
> users understand that, right?
>

Sure, but I guess I don't understand exactly what that behavior is though
(see below)...


> > So, I guess my basic question is what value
> `emacs-lisp-docstring-fill-column' tries to add?
>
> From my POV, the answer is clear: it allows users to have different
> customizable defaults for fill-column in Emacs Lisp and elsewhere.
>

If that was the only intention, that functionality was already present (I
guess way long before 1995) by just doing:
(add-hook 'emacs-lisp-mode-hook (lambda () (setq fill-column 80)))
;fill-column is buffer-local


> E.g., in text modes, it is customary to enlarge the default to 79 or
> thereabouts, but in Emacs Lisp we generally say that good style is to
> make lines in doc strings no wider than 60 characters (see the ELisp
> manual).
>

Doc strings, sure. But outside doc string, as it behaves now? Nah. But even
if that was the case, then just use the buffer local fill-column in a hook
(as above). No need to introduce redundant mechanisms.


> > To me this entire functionality should either be removed or fixed so
> that its behavior reflects its current docs. In
> > its current state it just malformats elisp files in unexpected ways. Or
> am I missing something?
>
> I'm sorry, but after so many years we had this working,


But it has never been working. If we by "working" mean "behaves as its name
suggest, and as the doc describes" that is.

removing this
> functionality is out of the question.


Yes, I can see that. Sorry, that was a bad suggestion from me.


>   I suggested to fix the docs to
> match the implementation; if someone has other suggestions that don't
> break existing behavior, or at least let users get back existing
> behavior, please describe those suggestions.
>
> Thanks.
>

Ok, I'll take a stab at the easy way out and just update the docs. When
looking at the implementation in the function lisp-fill-paragraph, the
outer or-clause seems to separate two cases: 1) if in a comment use
fill-column, else 2) bind fill-column to emacs-lisp-docstring-fill-column
and call fill-paragraph. Does that seem to be correct? In that case the
current doc could be changed from "Value of `fill-column' to use when
filling a docstring..." to "Value of `fill-column' to use in
emacs-elisp-mode except in comments".

But I still think the original intent of emacs-lisp-docstring-fill-column
(as described in its current doc) is useful, and would of course prefer if
we rather could update the implementation to reflect that functionality
instead. I'm just not knowledgeable enough to do a PR. Would it be
difficult?

BTW, as a side note, I just wanted to add that this bug report is of course
a very small detail. Also, if my language sounds a bit harsh, its just
because I'm in a hurry. I'm really grateful to you and the community for
putting in all the hard work so that people like me can use such a superior
tool - thanks a million!

[-- Attachment #2: Type: text/html, Size: 5419 bytes --]

bug#31656: 26.1; `fill-paragraph' malformats in emacs-lisp-mode

[Eli Zaretskii]

> From: Stefan Guath <stefan@automata.se>
> Date: Fri, 1 Jun 2018 16:34:31 +0200
> Cc: npostavs@gmail.com, 31656@debbugs.gnu.org
> 
>  > The very idea of shadowing the global `fill-column' in the first place breaks the Principle Of Least
>  > Astonishment
> 
>  If that is so, then we should have gobs of astonished users since 1995.
> 
> Yes, the streets are flooded with them! But Emacs users are a tough crowd that don't complain :)

I guess so.

>  >From my POV, the answer is clear: it allows users to have different
>  customizable defaults for fill-column in Emacs Lisp and elsewhere.
> 
> If that was the only intention, that functionality was already present (I guess way long before 1995) by just
> doing:
> (add-hook 'emacs-lisp-mode-hook (lambda () (setq fill-column 80))) ;fill-column is buffer-local

Sure, but then everyone would need to write a mode hook, don't you
think?  With a separate option, things "just work".

>  E.g., in text modes, it is customary to enlarge the default to 79 or
>  thereabouts, but in Emacs Lisp we generally say that good style is to
>  make lines in doc strings no wider than 60 characters (see the ELisp
>  manual).
> 
> Doc strings, sure. But outside doc string, as it behaves now? Nah.

Maybe we should simply advise against using M-q outside of doc strings
and comments?  I agree with Noam that M-q elsewhere makes very little
sense.

> But even if that was the case, then just use
> the buffer local fill-column in a hook (as above). No need to introduce redundant mechanisms.

It isn't redundant, because with the additional option we can set up
things so that Emacs does TRT by default out of the box, and in most
cases no customization by users is necessary.

> Ok, I'll take a stab at the easy way out and just update the docs. When looking at the implementation in the
> function lisp-fill-paragraph, the outer or-clause seems to separate two cases: 1) if in a comment use
> fill-column, else 2) bind fill-column to emacs-lisp-docstring-fill-column and call fill-paragraph. Does that seem
> to be correct?

Yes, I think so.

> In that case the current doc could be changed from "Value of `fill-column' to use when filling a
> docstring..." to "Value of `fill-column' to use in emacs-elisp-mode except in comments".

Sounds good, but I think we should also advise against using this
except inside doc strings (and perhaps any other strings).

> But I still think the original intent of emacs-lisp-docstring-fill-column (as described in its current doc) is useful,
> and would of course prefer if we rather could update the implementation to reflect that functionality instead. I'm
> just not knowledgeable enough to do a PR. Would it be difficult?

You mean, make it work only inside a doc string?  I guess that would
be possible, yes.

> BTW, as a side note, I just wanted to add that this bug report is of course a very small detail. Also, if my
> language sounds a bit harsh, its just because I'm in a hurry. I'm really grateful to you and the community for
> putting in all the hard work so that people like me can use such a superior tool - thanks a million!

No need to apologize, there's nothing wrong with your language.

Thanks.

bug#31656: 26.1; `fill-paragraph' malformats in emacs-lisp-mode

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@gmail.com>
> Cc: Stefan Guath <stefan@automata.se>,  31656@debbugs.gnu.org
> Date: Fri, 01 Jun 2018 05:39:39 -0400
> 
> > Or are there real-life use cases where this behavior is grossly
> > inappropriate?
> 
> I don't think it makes sense to apply normal plain text filling rules to
> code.  Maybe it doesn't come up much because people don't usually call
> M-q on code, and usually lines of code are kept short enough that they
> wouldn't get filled anyway.  But picking a random example from rgrep
> ^.\{100,\}$ on the Emacs code base:
> 
> (defun feedmail-default-date-generator (maybe-file)
>   "Default function for generating Date: header contents."
>   (feedmail-say-debug ">in-> feedmail-default-date-generator")
>   (when maybe-file
>     (feedmail-say-debug (concat "4 cre " (feedmail-rfc822-date (nth 4 (file-attributes maybe-file)))))
>     (feedmail-say-debug (concat "5 mod " (feedmail-rfc822-date (nth 5 (file-attributes maybe-file)))))
>     (feedmail-say-debug (concat "6 sta " (feedmail-rfc822-date (nth 6 (file-attributes maybe-file))))))
>   (let ((date-time))
>     (if (and (not feedmail-queue-use-send-time-for-date) maybe-file)
> 	(setq date-time (nth 5 (file-attributes maybe-file))))
>     (feedmail-rfc822-date date-time))
>   )
> 
> Running M-q on every line turns it into this nonsense:
> 
> (defun feedmail-default-date-generator (maybe-file)
>   "Default function for generating Date: header contents."
>   (feedmail-say-debug ">in-> feedmail-default-date-generator")
>   (when maybe-file
>     (feedmail-say-debug (concat "4
>     cre " (feedmail-rfc822-date (nth 4 (file-attributes
>     maybe-file)))))
>     (feedmail-say-debug (concat "5
>     mod " (feedmail-rfc822-date (nth 5 (file-attributes
>     maybe-file)))))
>     (feedmail-say-debug (concat "6
>     sta " (feedmail-rfc822-date (nth 6 (file-attributes
>     maybe-file))))))
>   (let ((date-time))
>     (if (and (not feedmail-queue-use-send-time-for-date)
>     maybe-file)
> 	(setq date-time (nth 5 (file-attributes maybe-file))))
>     (feedmail-rfc822-date date-time)) )

And if you set the value of emacs-lisp-docstring-fill-column to 80,
the value of fill-column that, according to the OP, should have been
used instead, do you get a better-looking nonsense?

IOW, if indeed it makes no sense to apply normal plain text filling
rules to code, then why does it matter what is the value of
fill-column when code is being filled?

bug#31656: 26.1; `fill-paragraph' malformats in emacs-lisp-mode

[Noam Postavsky]

Eli Zaretskii <eliz@gnu.org> writes:

> And if you set the value of emacs-lisp-docstring-fill-column to 80,
> the value of fill-column that, according to the OP, should have been
> used instead, do you get a better-looking nonsense?

No, I was thinking more along the aligns of:

--- i/lisp/emacs-lisp/lisp-mode.el
+++ w/lisp/emacs-lisp/lisp-mode.el
@@ -1314,6 +1314,7 @@ lisp-fill-paragraph
       ;;
       ;; The `fill-column' is temporarily bound to
       ;; `emacs-lisp-docstring-fill-column' if that value is an integer.
+      (when (nth 3 (syntax-ppss))       ; Only fill inside strings.
         (let ((paragraph-start
                (concat paragraph-start
                        "\\|\\s-*\\([(;\"]\\|\\s-:\\|`(\\|#'(\\)"))
@@ -1323,7 +1324,7 @@ lisp-fill-paragraph
                                     (derived-mode-p 'emacs-lisp-mode))
                                emacs-lisp-docstring-fill-column
                              fill-column)))
-	(fill-paragraph justify))
+	  (fill-paragraph justify)))
       ;; Never return nil.
       t))


I think this covers the scenario in the OP.

bug#31656: 26.1; `fill-paragraph' malformats in emacs-lisp-mode

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@gmail.com>
> Cc: 31656@debbugs.gnu.org,  stefan@automata.se
> Date: Fri, 01 Jun 2018 21:45:37 -0400
> 
> --- i/lisp/emacs-lisp/lisp-mode.el
> +++ w/lisp/emacs-lisp/lisp-mode.el
> @@ -1314,6 +1314,7 @@ lisp-fill-paragraph
>        ;;
>        ;; The `fill-column' is temporarily bound to
>        ;; `emacs-lisp-docstring-fill-column' if that value is an integer.
> +      (when (nth 3 (syntax-ppss))       ; Only fill inside strings.
>          (let ((paragraph-start
>                 (concat paragraph-start
>                         "\\|\\s-*\\([(;\"]\\|\\s-:\\|`(\\|#'(\\)"))
> @@ -1323,7 +1324,7 @@ lisp-fill-paragraph
>                                      (derived-mode-p 'emacs-lisp-mode))
>                                 emacs-lisp-docstring-fill-column
>                               fill-column)))
> -	(fill-paragraph justify))
> +	  (fill-paragraph justify)))
>        ;; Never return nil.
>        t))
> 
> 
> I think this covers the scenario in the OP.

Are there no use cases where we would want M-q outside of syntactic
strings?

bug#31656: 26.1; `fill-paragraph' malformats in emacs-lisp-mode

[Noam Postavsky]

Eli Zaretskii <eliz@gnu.org> writes:

> Are there no use cases where we would want M-q outside of syntactic
> strings?

Okay, I can think of one possibility: if you've pasted a long single
line consisting of a quoted list of symbols, then M-q could be a
convenient way of breaking it up into multiple lines.  With M-q disabled
in Lisp code, you would need to copy the data to a temp non-lisp-mode
buffer in order to do that, so a bit more inconvenient.

bug#31656: 26.1; `fill-paragraph' malformats in emacs-lisp-mode

[martin rudalics]

 >> Are there no use cases where we would want M-q outside of syntactic
 >> strings?
 >
 > Okay, I can think of one possibility: if you've pasted a long single
 > line consisting of a quoted list of symbols, then M-q could be a
 > convenient way of breaking it up into multiple lines.  With M-q disabled
 > in Lisp code, you would need to copy the data to a temp non-lisp-mode
 > buffer in order to do that, so a bit more inconvenient.

Don't we want to fill comments as well?

martin

bug#31656: 26.1; `fill-paragraph' malformats in emacs-lisp-mode

[Noam Postavsky]

martin rudalics <rudalics@gmx.at> writes:

>>> Are there no use cases where we would want M-q outside of syntactic
>>> strings?
>>
>> Okay, I can think of one possibility: if you've pasted a long single
>> line consisting of a quoted list of symbols, then M-q could be a
>> convenient way of breaking it up into multiple lines.  With M-q disabled
>> in Lisp code, you would need to copy the data to a temp non-lisp-mode
>> buffer in order to do that, so a bit more inconvenient.
>
> Don't we want to fill comments as well?

Yes, we lost a bit of context there; comment filling is taken care of at
the top of lisp-fill-paragraph, I wasn't proposing to disable that.

    (defun lisp-fill-paragraph (&optional justify)
      [...]
      (or (fill-comment-paragraph justify)
          ;; Since fill-comment-paragraph returned nil, that means we're not in
          ;; a comment: Point is on a program line; we are interested
          ;; particularly in docstring lines.
          ;; [...] This setting has the consequence of inhibiting
          ;; filling many program lines that are not docstrings, which is sensible,
          ;; because the user probably asked to fill program lines by accident, or
          ;; expecting indentation (perhaps we should try to do indenting in that
          ;; case).[...]
+      (when (nth 3 (syntax-ppss))       ; Only fill inside strings.
          (let ((paragraph-start
                 (concat paragraph-start
                         "\\|\\s-*\\([(;\"]\\|\\s-:\\|`(\\|#'(\\)"))
            (paragraph-separate
             (concat paragraph-separate "\\|\\s-*\".*[,\\.]$"))
                (fill-column (if (and (integerp emacs-lisp-docstring-fill-column)
                                      (derived-mode-p 'emacs-lisp-mode))
                                 emacs-lisp-docstring-fill-column
                               fill-column)))
-	(fill-paragraph justify))
+	  (fill-paragraph justify)))
          ;; Never return nil.
          t))

bug#31656: 26.1; `fill-paragraph' malformats in emacs-lisp-mode

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@gmail.com>
> Cc: 31656@debbugs.gnu.org,  stefan@automata.se
> Date: Sat, 02 Jun 2018 09:07:10 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Are there no use cases where we would want M-q outside of syntactic
> > strings?
> 
> Okay, I can think of one possibility: if you've pasted a long single
> line consisting of a quoted list of symbols, then M-q could be a
> convenient way of breaking it up into multiple lines.  With M-q disabled
> in Lisp code, you would need to copy the data to a temp non-lisp-mode
> buffer in order to do that, so a bit more inconvenient.

Then maybe just override the value of fill-column when inside strings?

bug#31656: 26.1; `fill-paragraph' malformats in emacs-lisp-mode

[Stefan Guath]

[-- Attachment #1: Type: text/plain, Size: 1837 bytes --]

I agree that overriding the value of `fill-column' _only_ inside doc
strings is the sensible way to go. The only reason
`emacs-lisp-docstring-fill-column' exists in the first place is to cover
the edge case of doc strings as recommended in
https://www.gnu.org/software/emacs/manual/html_node/elisp/Documentation-Tips.html
(e.g. for apropos) which is fine, but overriding it in any other situation
is confusing. We already have a general mechanism for controlling the fill
column width on a major-mode basis through hooks and `fill-column' being
buffer-local, and should as a rule encourage the use of these consistent
control mechanisms across all major-modes - not introduce redundant
functionalities that fight each other.

The proposed solution in this email thread by Noam Postavsky is already too
greedy in that it affects _all_ strings, not only doc strings (or maybe I
misunderstood "(nth 3 (syntax-ppss))" ?). Still, it's better than the
current behavior, and I guess that identifying doc strings only (and not
strings in general) is too difficult.

On Sat, Jun 2, 2018 at 4:09 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Noam Postavsky <npostavs@gmail.com>
> > Cc: 31656@debbugs.gnu.org,  stefan@automata.se
> > Date: Sat, 02 Jun 2018 09:07:10 -0400
> >
> > Eli Zaretskii <eliz@gnu.org> writes:
> >
> > > Are there no use cases where we would want M-q outside of syntactic
> > > strings?
> >
> > Okay, I can think of one possibility: if you've pasted a long single
> > line consisting of a quoted list of symbols, then M-q could be a
> > convenient way of breaking it up into multiple lines.  With M-q disabled
> > in Lisp code, you would need to copy the data to a temp non-lisp-mode
> > buffer in order to do that, so a bit more inconvenient.
>
> Then maybe just override the value of fill-column when inside strings?
>

[-- Attachment #2: Type: text/html, Size: 2685 bytes --]

bug#31656: 26.1; `fill-paragraph' malformats in emacs-lisp-mode

[Lars Ingebrigtsen]

Noam Postavsky <npostavs@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Are there no use cases where we would want M-q outside of syntactic
>> strings?
>
> Okay, I can think of one possibility: if you've pasted a long single
> line consisting of a quoted list of symbols, then M-q could be a
> convenient way of breaking it up into multiple lines.  With M-q disabled
> in Lisp code, you would need to copy the data to a temp non-lisp-mode
> buffer in order to do that, so a bit more inconvenient.

I do use `M-q' on long lists of symbols sometimes (mostly because `pp'
is so unhelpful when rendering those), so it would be annoying to see
that go.

What about just a new, simple filling method for Lisp-like text?  For
instance, if you `M-q' outside of text, it's just convert this:

(defun feedmail-default-date-generator (maybe-file)
  "Default function for generating Date: header contents."
  (feedmail-say-debug ">in-> feedmail-default-date-generator")
  (when maybe-file
    (feedmail-say-debug (concat "4 cre " (feedmail-rfc822-date (nth 4 (file-attributes maybe-file)))))
    (feedmail-say-debug (concat "5 mod " (feedmail-rfc822-date (nth 5 (file-attr

into this:

(defun feedmail-default-date-generator (maybe-file)
  "Default function for generating Date: header contents."
  (feedmail-say-debug ">in-> feedmail-default-date-generator")
  (when maybe-file
    (feedmail-say-debug (concat "4 cre " (feedmail-rfc822-date (nth 4
(file-attributes maybe-file)))))
    (feedmail-say-debug (concat "5 mod " (feedmail-rfc822-date (nth 5
(file-attr

It'd still be wrong, but at least it wouldn't destroy the strings...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#31656: 26.1; `fill-paragraph' malformats in emacs-lisp-mode

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I do use `M-q' on long lists of symbols sometimes (mostly because `pp'
> is so unhelpful when rendering those), so it would be annoying to see
> that go.
>
> What about just a new, simple filling method for Lisp-like text?  For
> instance, if you `M-q' outside of text, it's just convert this:

I've now done basically this in Emacs 29 -- `M-q' in Emacs Lisp mode
fills strings if we're inside strings, but otherwise does very simple
sexp-based filling.  Tweaking this further is probably a good idea, but
it seems to give pretty logical results now, and doesn't destroy
anything like it used to.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no