bug#71017: fill-flowed-encode

bug#71017: fill-flowed-encode

[Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi y'all.

flow-fill.el.gz has a pair of functions, fill-flowed-encode and 
fill-flowed-fill-buffer (the latter is only called from the 
former).

Here is a fixed version of the former that then also deprecates 
the latter (I have signed FSF copyright papers):

(defun fill-flowed-encode (&optional buffer)
  (with-current-buffer (or buffer (current-buffer))
    ;; No point in doing this unless hard newlines is used.
    (when use-hard-newlines
      (let ((start (point-min)) end)
	;; Go through each paragraph, filling it and adding SPC
	;; as the last character on each line.
	(while (and (< start (point-max))
		    (setq end (or (text-property-any start (point-max) 'hard 't)
				  (point-max))))
	  (save-restriction
	    (narrow-to-region start end)
	    (let ((fill-column (eval fill-flowed-encode-column t))
		  (prefix
		   (concat "\n"
			   (or (and (looking-at ">[> ]*")
				    (match-string 0)) ""))))
	      (while (search-forward prefix nil t)
		(replace-match " " t t))
	      (goto-char start)
	      (while (< (+ (point) fill-column) (point-max))
		(forward-char fill-column)
		(search-backward " ")
		(forward-char)
		(insert prefix)))
	    (setq start (1+ (point-max))))))
      t)))

This fixes two bugs when sending RFC 2646–formatted email.

First, the old code didn't refill or encode the last paragraph at 
all unless there was at least one hard newline EOF.

Second, the old code borked up code indented with tabs and spaces 
(iff that code had overly long lines), such as the Lisp code in 
this email. It could sometimes insert extra in the middle of such 
long lines.

Here is an example of what it would do. It would turn this:

(defun lorem (ipsum)
  (dolor sit amet)
  (consectetur adipiscing elit (sed do eiusmod tempor incididunt 
ut labore et dolore magna aliqua))
  (ut enim ad minim veniam
      (quis nostrud exercitation ullamco laboris nisi ut aliquip 
ex ea commodo consequat
	    (duis aute irure dolor in reprehenderit in
		  voluptate velit esse cillum dolore
		  eu fugiat nulla pariatur)
	    excepteur sint occaecat cupidatat non proident
	    (sunt in culpa qui officia deserunt mollit anim id est 
laborumd))))

into this:

(defun lorem (ipsum)
  (dolor sit amet)
  (consectetur adipiscing elit (sed do eiusmod tempor incididunt
  ut labore et dolore magna aliqua))
  (ut enim ad minim veniam
      (quis nostrud exercitation ullamco laboris nisi ut aliquip
      ex ea commodo consequat
	    (duis aute irure dolor in reprehenderit in
		  voluptate velit esse cillum dolore
		  eu fugiat nulla pariatur)
	    excepteur sint occaecat cupidatat non proident
	    (sunt in culpa qui officia deserunt mollit anim id est
	    laborumd))))

It was breaking lines awkwardly so when they're reconnected they 
have extra whitespace in the mkddle of lines. 

bug#71017: fill-flowed-encode

[Stefan Kangas]

Sandra Snan <sandra.snan@idiomdrottning.org> writes:

> Hi y'all.

Hi Sandra,

> flow-fill.el.gz has a pair of functions, fill-flowed-encode and
> fill-flowed-fill-buffer (the latter is only called from the former).
>
> Here is a fixed version of the former that then also deprecates the latter (I
> have signed FSF copyright papers):
>
> (defun fill-flowed-encode (&optional buffer)
>  (with-current-buffer (or buffer (current-buffer))
>    ;; No point in doing this unless hard newlines is used.
>    (when use-hard-newlines
>      (let ((start (point-min)) end)
> 	;; Go through each paragraph, filling it and adding SPC
> 	;; as the last character on each line.
> 	(while (and (< start (point-max))
> 		    (setq end (or (text-property-any start (point-max) 'hard 't)
> 				  (point-max))))
> 	  (save-restriction
> 	    (narrow-to-region start end)
> 	    (let ((fill-column (eval fill-flowed-encode-column t))
> 		  (prefix
> 		   (concat "\n"
> 			   (or (and (looking-at ">[> ]*")
> 				    (match-string 0)) ""))))
> 	      (while (search-forward prefix nil t)
> 		(replace-match " " t t))
> 	      (goto-char start)
> 	      (while (< (+ (point) fill-column) (point-max))
> 		(forward-char fill-column)
> 		(search-backward " ")
> 		(forward-char)
> 		(insert prefix)))
> 	    (setq start (1+ (point-max))))))
>      t)))
>
> This fixes two bugs when sending RFC 2646–formatted email.
>
> First, the old code didn't refill or encode the last paragraph at all unless
> there was at least one hard newline EOF.
>
> Second, the old code borked up code indented with tabs and spaces (iff that code
> had overly long lines), such as the Lisp code in this email. It could sometimes
> insert extra in the middle of such long lines.
>
> Here is an example of what it would do. It would turn this:
>
> (defun lorem (ipsum)
>  (dolor sit amet)
>  (consectetur adipiscing elit (sed do eiusmod tempor incididunt ut labore et
> dolore magna aliqua))
>  (ut enim ad minim veniam
>      (quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
>     consequat
> 	    (duis aute irure dolor in reprehenderit in
> 		  voluptate velit esse cillum dolore
> 		  eu fugiat nulla pariatur)
> 	    excepteur sint occaecat cupidatat non proident
> 	    (sunt in culpa qui officia deserunt mollit anim id est laborumd))))
>
> into this:
>
> (defun lorem (ipsum)
>  (dolor sit amet)
>  (consectetur adipiscing elit (sed do eiusmod tempor incididunt
>  ut labore et dolore magna aliqua))
>  (ut enim ad minim veniam
>      (quis nostrud exercitation ullamco laboris nisi ut aliquip
>      ex ea commodo consequat
> 	    (duis aute irure dolor in reprehenderit in
> 		  voluptate velit esse cillum dolore
> 		  eu fugiat nulla pariatur)
> 	    excepteur sint occaecat cupidatat non proident
> 	    (sunt in culpa qui officia deserunt mollit anim id est
> 	    laborumd))))
>
> It was breaking lines awkwardly so when they're reconnected they have extra
> whitespace in the mkddle of lines.

Could you please resend the above as patches formatted by

    git format-patch -1

?

It will make it easier for us to review and install the proposed
changes.  Thanks in advance.

bug#71017: fill-flowed-encode

[Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Stefan Kangas <stefankangas@gmail.com> writes:

> Could you please resend the above as patches formatted by
>
>     git format-patch -1
>
> ?
>

I'm glad I'm gonna get a chance to resend because I've fixed some 
more bugs in the version I've been dogfooding since I first sent 
that so I wanna send an updated version that has that.

Normally when I'm sending Emacs lisp patches, it's on packages and 
they're usually in git already. Here it's a core file so the 
question I've got is what is the git repo I should make the change 
in? Then I could git send-email to this In-Reply-To thread id.

Stefan Kangas <stefankangas@gmail.com> writes:

> Sandra Snan <sandra.snan@idiomdrottning.org> writes:
>
>> Hi y'all.
>
> Hi Sandra,
>
>> flow-fill.el.gz has a pair of functions, fill-flowed-encode and 
>> fill-flowed-fill-buffer (the latter is only called from the 
>> former).
>> Here is a fixed version of the former that then also 
>> deprecates the latter (I have signed FSF copyright papers):
>> 
>> (defun fill-flowed-encode (&optional buffer)  (with-current-buffer 
>> (or buffer (current-buffer))    ;; No point in doing this unless 
>> hard newlines is used.    (when use-hard-newlines      (let ((start 
>> (point-min)) end) 	;; Go through each paragraph, filling it and 
>> adding SPC 	;; as the last character on each line. 	(while (and (< 
>> start (point-max)) 		    (setq end (or (text-property-any start 
>> (point-max) 'hard 't) 				  (point-max)))) 	  (save-restriction 	   
>>  (narrow-to-region start end) 	    (let ((fill-column (eval 
>> fill-flowed-encode-column t)) 		  (prefix 		   (concat "\n" 			   
>> (or (and (looking-at ">[> ]*") 				    (match-string 0)) "")))) 	   
>>    (while (search-forward prefix nil t) 		(replace-match " " t t)) 
>> 	      (goto-char start) 	      (while (< (+ (point) fill-column) 
>> (point-max)) 		(forward-char fill-column) 		(search-backward " ") 
>> 		(forward-char) 		(insert prefix))) 	    (setq start (1+ 
>> (point-max))))))      t)))
>> This fixes two bugs when sending RFC 
>> 2646–formatted email.
>> First, the old code didn't refill or 
>> encode the last paragraph at all unless there was at least one hard 
>> newline EOF.
>> Second, the old code borked up code indented with 
>> tabs and spaces (iff that code had overly long lines), such as the 
>> Lisp code in this email. It could sometimes insert extra in the 
>> middle of such long lines.
>> Here is an example of what it would 
>> do. It would turn this:
>> (defun lorem (ipsum)  (dolor sit amet)  
>> (consectetur adipiscing elit (sed do eiusmod tempor incididunt ut 
>> labore et dolore magna aliqua))  (ut enim ad minim veniam      
>> (quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea 
>> commodo     consequat 	    (duis aute irure dolor in reprehenderit 
>> in 		  voluptate velit esse cillum dolore 		  eu fugiat nulla 
>> pariatur) 	    excepteur sint occaecat cupidatat non proident 	    
>> (sunt in culpa qui officia deserunt mollit anim id est 
>> laborumd))))
>> into this:
>> (defun lorem (ipsum)  (dolor sit 
>> amet)  (consectetur adipiscing elit (sed do eiusmod tempor 
>> incididunt  ut labore et dolore magna aliqua))  (ut enim ad minim 
>> veniam      (quis nostrud exercitation ullamco laboris nisi ut 
>> aliquip      ex ea commodo consequat 	    (duis aute irure dolor in 
>> reprehenderit in 		  voluptate velit esse cillum dolore 		  eu 
>> fugiat nulla pariatur) 	    excepteur sint occaecat cupidatat non 
>> proident 	    (sunt in culpa qui officia deserunt mollit anim id 
>> est 	    laborumd))))
>> It was breaking lines awkwardly so when 
>> they're reconnected they have extra whitespace in the mkddle of 
>> lines.
>
> Could you please resend the above as patches formatted by
>
>     git format-patch -1
>
> ?
>
> It will make it easier for us to review and install the proposed 
> changes.  Thanks in advance.

bug#71017: fill-flowed-encode

[Stefan Kangas]

Sandra Snan <sandra.snan@idiomdrottning.org> writes:

> I'm glad I'm gonna get a chance to resend because I've fixed some
> more bugs in the version I've been dogfooding since I first sent
> that so I wanna send an updated version that has that.

Great, thanks.

> Normally when I'm sending Emacs lisp patches, it's on packages and
> they're usually in git already. Here it's a core file so the
> question I've got is what is the git repo I should make the change
> in? Then I could git send-email to this In-Reply-To thread id.

https://git.savannah.gnu.org/cgit/emacs.git

Please have a look at the CONTRIBUTE file in the Emacs source
distribution.

https://git.savannah.gnu.org/cgit/emacs.git/tree/CONTRIBUTE

bug#71017: [PATCH] Flow single-paragraph messages

[Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors]

This fixes two bugs when sending RFC 2646–formatted email.

First, the old code didn't refill or encode the last paragraph at all
unless there was at least one hard newline EOF. This was a bee to track
down because there were two separate issues at play. One was a a bug in
flow-fill.el where every paragraph except the last paragraph was
reflowed, but the last paragraph would stay hardwrapped. Manually
placing a hard newline at the end of the file was a workaround but I
don't always remember to do that. I managed to fix that bug a few months
ago.

Second, the old code borked up code indented with tabs and spaces
(iff that code had overly long lines), such as Lisp code. It could
sometimes insert extra whitespace in the middle of such long lines.
I fixed that bug shortly after the first one.

But for months dogfooding those two changes, sometimes a hardwrapped
email would still be sent. I finally managed to debug and figure it out
and it took all day. Turns out mml just plain didn't call the
fill-flowed-encode function if the message doesn't have any hard
newlines (newlines with the hard text property). Well, of course a
single-paragraph email isn't gonna have any hard newlines! But it still
needs reflowing!

So I've now changed that and updated the documentation to match those
news semantics. I went all the way, but a possible compromise might be
to not-flow a message that has \n\n but no hard text props, since that's
a sign that something is wrong.

Since the use-hard-newlines variable is buffer local and all this
reflowing is being done in a temp buffer, that variable is more than
useless so I've removed references to it.
---
 doc/misc/emacs-mime.texi |  7 ++---
 lisp/gnus/mml.el         | 29 ++++++++----------
 lisp/mail/flow-fill.el   | 65 +++++++++++++++++-----------------------
 3 files changed, 41 insertions(+), 60 deletions(-)

diff --git a/doc/misc/emacs-mime.texi b/doc/misc/emacs-mime.texi
index ef7ea61..7621a9a 100644
--- a/doc/misc/emacs-mime.texi
+++ b/doc/misc/emacs-mime.texi
@@ -1087,13 +1087,10 @@ terminated by soft newline characters are filled together and wrapped
 after the column decided by @code{fill-flowed-encode-column}.
 Quotation marks (matching @samp{^>* ?}) are respected.  The variable
 controls how the text will look in a client that does not support
-flowed text, the default is to wrap after 66 characters.  If hard
-newline characters are not present in the buffer, no flow encoding
-occurs.
+flowed text, the default is to wrap after 66 characters.
 
 You can customize the value of the @code{mml-enable-flowed} variable
-to enable or disable the flowed encoding usage when newline
-characters are present in the buffer.
+to enable or disable the flowed encoding usage.
 
 On decoding flowed text, lines with soft newline characters are filled
 together and wrapped after the column decided by
diff --git a/lisp/gnus/mml.el b/lisp/gnus/mml.el
index e3bc393..2db39dc 100644
--- a/lisp/gnus/mml.el
+++ b/lisp/gnus/mml.el
@@ -691,23 +691,18 @@ type detected."
 		   (t
 		    ;; Only perform format=flowed filling on text/plain
 		    ;; parts where there either isn't a format parameter
-		    ;; in the mml tag or it says "flowed" and there
-		    ;; actually are hard newlines in the text.
-		    (let (use-hard-newlines)
-		      (when (and mml-enable-flowed
-				 (string= type "text/plain")
-				 (not (string= (cdr (assq 'sign cont)) "pgp"))
-				 (or (null (assq 'format cont))
-				     (string= (cdr (assq 'format cont))
-					      "flowed"))
-				 (setq use-hard-newlines
-				       (text-property-any
-					(point-min) (point-max) 'hard 't)))
-			(fill-flowed-encode)
-			;; Indicate that `mml-insert-mime-headers' should
-			;; insert a "; format=flowed" string unless the
-			;; user has already specified it.
-			(setq flowed (null (assq 'format cont)))))
+		    ;; in the mml tag or it says "flowed".
+		    (when (and mml-enable-flowed
+			       (string= type "text/plain")
+			       (not (string= (cdr (assq 'sign cont)) "pgp"))
+			       (or (null (assq 'format cont))
+				   (string= (cdr (assq 'format cont))
+					    "flowed")))
+		      (fill-flowed-encode)
+		      ;; Indicate that `mml-insert-mime-headers' should
+		      ;; insert a "; format=flowed" string unless the
+		      ;; user has already specified it.
+		      (setq flowed (null (assq 'format cont))))
 		    ;; Prefer `utf-8' for text/calendar parts.
 		    (if (or charset
 			    (not (string= type "text/calendar")))
diff --git a/lisp/mail/flow-fill.el b/lisp/mail/flow-fill.el
index 919490e..5c9ae21 100644
--- a/lisp/mail/flow-fill.el
+++ b/lisp/mail/flow-fill.el
@@ -73,50 +73,39 @@ RFC 2646 suggests 66 characters for readability."
 ;;;###autoload
 (defun fill-flowed-encode (&optional buffer)
   (with-current-buffer (or buffer (current-buffer))
-    ;; No point in doing this unless hard newlines is used.
-    (when use-hard-newlines
-      (let ((start (point-min)) end)
-	;; Go through each paragraph, filling it and adding SPC
-	;; as the last character on each line.
-	(while (setq end (text-property-any start (point-max) 'hard 't))
-	  (save-restriction
-	    (narrow-to-region start end)
-	    (let ((fill-column (eval fill-flowed-encode-column t)))
-	      (fill-flowed-fill-buffer))
-	    (goto-char (point-min))
-	    (while (re-search-forward "\n" nil t)
-	      (replace-match " \n" t t))
-	    (goto-char (setq start (1+ (point-max)))))))
-      t)))
-
-(defun fill-flowed-fill-buffer ()
-  (let ((prefix nil)
-	(prev-prefix nil)
-	(start (point-min)))
-    (goto-char (point-min))
-    (while (not (eobp))
-      (setq prefix (and (looking-at "[> ]+")
-			(match-string 0)))
-      (if (equal prefix prev-prefix)
-	  (forward-line 1)
+    (let ((fill-column (eval fill-flowed-encode-column t))
+	  (start (point-min))
+	  end)
+      ;; Go through each paragraph, filling it and adding SPC
+      ;; as the last character on each line.
+      (while (and (< start (point-max))
+		  (setq end (or (text-property-any start (point-max) 'hard 't)
+				(point-max))))
 	(save-restriction
-	  (narrow-to-region start (point))
-	  (let ((fill-prefix prev-prefix))
-	    (fill-region (point-min) (point-max) t 'nosqueeze 'to-eop))
-	  (goto-char (point-max)))
-	(setq prev-prefix prefix
-	      start (point))))
-    (save-restriction
-      (narrow-to-region start (point))
-      (let ((fill-prefix prev-prefix))
-	(fill-region (point-min) (point-max) t 'nosqueeze 'to-eop)))))
+	  (narrow-to-region start end)
+	  (let ((prefix
+		 (concat "\n"
+			 (or (and (looking-at ">[> ]*")
+				  (match-string 0)) ""))))
+	    (goto-char start)
+	    (while (search-forward prefix nil t)
+	      (replace-match " " t t))
+	    (goto-char start)
+	    (while (< (+ (point) fill-column) (point-max))
+	      (let ((start (point)))
+		(forward-char fill-column)
+		(when (search-backward " " start t)
+		  (forward-char)
+		  (insert prefix)))))
+	  (setq start (1+ (point-max))))))
+    t))
 
 ;;;###autoload
 (defun fill-flowed (&optional buffer delete-space)
   "Apply RFC2646 decoding to BUFFER.
 If BUFFER is nil, default to the current buffer.
 
-If DELETE-SPACE, delete RFC2646 spaces padding at the end of
+If DELETE-SPACE, delete RFC3676 spaces padding at the end of
 lines."
   (with-current-buffer (or buffer (current-buffer))
     (let ((fill-column  (eval fill-flowed-display-column t)))
@@ -154,7 +143,7 @@ lines."
           ;; Delete the newline.
           (when (eq (following-char) ?\s)
             (delete-char 1))
-          ;; Hack: Don't do the flowing on the signature line.
+          ;; As per RFC3767: Don't do the flowing on the signature line.
           (when (and (not (looking-at "-- $"))
                      (eq (char-before (line-end-position)) ?\s))
             (while (and (not (eobp))
-- 
2.39.2

bug#71017: [PATCH] Flow single-paragraph messages

[Eli Zaretskii]

> Cc: Sandra Snan <sandra.snan@idiomdrottning.org>
> Date: Sat,  6 Jul 2024 22:49:50 +0200
> From:  Sandra Snan via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> This fixes two bugs when sending RFC 2646–formatted email.

Thanks.  (I also see RFC 3676 mentioned -- what is this about?)

> First, the old code didn't refill or encode the last paragraph at all
> unless there was at least one hard newline EOF.

Isn't this the documented behavior?

> But for months dogfooding those two changes, sometimes a hardwrapped
> email would still be sent. I finally managed to debug and figure it out
> and it took all day. Turns out mml just plain didn't call the
> fill-flowed-encode function if the message doesn't have any hard
> newlines (newlines with the hard text property). Well, of course a
> single-paragraph email isn't gonna have any hard newlines! But it still
> needs reflowing!
> 
> So I've now changed that and updated the documentation to match those
> news semantics. I went all the way, but a possible compromise might be
> to not-flow a message that has \n\n but no hard text props, since that's
> a sign that something is wrong.

The change seems to be an incompatible behavior change, so I wonder
whether we'd need some way for users to get back old behavior.  Eric,
WDYT?

I'll leave it to Eric to comment on the code changes.

bug#71017: [PATCH] Flow single-paragraph messages

[Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> Thanks.  (I also see RFC 3676 mentioned -- what is this about?)

I mentioned RFC 3676 twice for different reasons.

First, it's introduced the delsp parameter which wasn't in 2646 so 
code related to that parameter shouldn't talk about 2646 but rather 
3676.

Second, I saw a comment referring to not tampering with the sig 
line as a hack. I happened to have the section of RFC 3676 open 
that mandates that behavior so I changed the comment, however, this 
might be in RFC 2646 also, I don't know that, so in that case that 
comment might better change to refer to 2646 instead.

>> First, the old code didn't refill or encode the last paragraph 
>> at all unless there was at least one hard newline EOF.
>
> Isn't this the documented behavior?

Re multi-paragraph messages:

No, it's not the documented behavior, it was an unrelated bug in 
fill-flowed-encode. It would refill all the other paragraphs, 
separated by hard newlines, just not the last one. That was a bug 
and broke documented behavior. I fixed that bug + another unrelated 
reflow bug. That was in fill-flowed-encode and that bugfix doesn't 
rely on the change in mml. With my fix in fill-flowed-encode, 
multi-paragraph-messages started working fine.

Re single-paragraph messages:

However, according to the old documented behavior, a message that 
contained no hard newlines should not be refilled. This documented 
behavior in mml-generate-mime-1 meant that single-paragraph 
messages would not be filled even with the fill-flowed-encode bug 
fixed. That is an unintended bad consequence of the documented 
behavior, a "bug in the design".

I did change that but I updated the documentation to match.

> The change seems to be an incompatible behavior change, so I 
> wonder whether we'd need some way for users to get back old 
> behavior.

There is still the (neglected) defcustom mml-enable-flowed which 
now becomes more relevant since it's a way to turn off all this 
meddling and reflowing in the first place.

The old documented behavior was bugged-by-design. It's not right 
that single paragraph messages are hardwrapped and not reflowed. In 
my day-to-day I write many messages in Emacs that I later see in 
threads in another MUA (Delta Chat) and these messages stand out in 
a way that something is wrong with them.

However, one intent behind the old behavior was, in spirit, good: 
It'd be good to detect whether or not users with mml-enable-flowed 
on have remembered to also turn on use-hard-newlines, which is 
important for users with that on to do, especially since 
mml-enable-flowed defaults to t. The old attempt at doing that was 
flawed since it only worked reliably for multi-paragraph messages.

Unfortunately there's currently no way to detect in a 
single-paragraph message whether or not use-hard-newlines have been 
turned on, since the variable it sets is buffer local. (One 
extremely klugy workaround would be to change the 
message-send-and-exit command to check whether use-hard-newlines is 
on and if it is, add an extra hard newline EOT just for detecting 
this. Not super into that solution so hopefully there are other 
ways.)

With this patch, the defcustom mml-enable-flowed becomes _the_ 
setting for this, which does match a lot of documentation on the 
books. Perhaps it shall no longer default to t though since it 
completely borks messages up if it's t but use-hard-newlines are 
not on!

So here we are:

The new behavior has a problem: messages will get reflowed if 
mml-enable-flowed is t (the default!) even when use-hard-newlines 
is off, meaning that even separate "\n\n" paragraphs will get 
flowed together which is not what people want. Use-hard-newlines 
should be mandatory whenever mml-enable-flowed is on.

The old behavior is not OK since single-paragraph messages will get 
messed up, hardwrapped even when those newlines were advertised as 
"soft", or not softwrapped even when the paragraph consists of just 
one single super long unbroken line.

I also have sent a patch to the messages-are-flowing project 
highlighting the importance of this variable:

https://github.com/legoscia/messages-are-flowing/pull/15/commits/ae432723c2565ceced5d01d9aa2d314bd42aaa3c

So how about this idea:

If mml-enable-flowed is on but fill-flowed-encode is asked to flow 
a message that doesn't have any hard newlines, assume Markdown 
semantics, i.e. special treatment for "\n\n+", "  $", and "^    ". 
I'll see if I have time to implement that this morning. I think I'd 
place that change in fill-flowed-encode.

That wouldn't affect people with mml-enable-flowed off, or people 
with both mml-enable-flowed and use-hard-newlines on, it'd just be 
a sort of DWIM fallback based on the guess that markdown semantics 
are somewhat widely known or expected in 2024, to prevent separate 
paragraphs to be flowed together for people with "incorrect" 
settings.

Again, if there were a cross-buffer way to reliably detect whether 
use-hard-newlines is on, that dwimmy fallback wouldn't be needed.

I thought about whether it'd have been better if it was instead 
soft newlines that were marked with a text property, not hard ones, 
but in the end that wouldn't properly softwrap messages with just 
one single overly long line. 

bug#71017: [PATCH] Flow single-paragraph messages

[Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors]

See discussion in #71017.

In short:

if mml-flowed-enable is nil: no change to behavior

if mml-flowed-enable is t and use-hard-newlines is on: several bugs
fixed, some (multi-paragraph messages) of which were counter to
documentation and some (single-paragraph messages) that were
along documentation lines but which messed up messages royally.

if mml-flowed-enable is t but use-hard-newlines was accidentally
forgotten by user: new behavior; reflow but don't reflow together
separate paragraphs nor reflow across lines that markdown would've
considered hard.
---
 doc/misc/emacs-mime.texi |  7 +---
 lisp/gnus/mml.el         | 29 ++++++--------
 lisp/mail/flow-fill.el   | 86 ++++++++++++++++++++++------------------
 3 files changed, 62 insertions(+), 60 deletions(-)

diff --git a/doc/misc/emacs-mime.texi b/doc/misc/emacs-mime.texi
index ef7ea61..7621a9a 100644
--- a/doc/misc/emacs-mime.texi
+++ b/doc/misc/emacs-mime.texi
@@ -1087,13 +1087,10 @@ terminated by soft newline characters are filled together and wrapped
 after the column decided by @code{fill-flowed-encode-column}.
 Quotation marks (matching @samp{^>* ?}) are respected.  The variable
 controls how the text will look in a client that does not support
-flowed text, the default is to wrap after 66 characters.  If hard
-newline characters are not present in the buffer, no flow encoding
-occurs.
+flowed text, the default is to wrap after 66 characters.
 
 You can customize the value of the @code{mml-enable-flowed} variable
-to enable or disable the flowed encoding usage when newline
-characters are present in the buffer.
+to enable or disable the flowed encoding usage.
 
 On decoding flowed text, lines with soft newline characters are filled
 together and wrapped after the column decided by
diff --git a/lisp/gnus/mml.el b/lisp/gnus/mml.el
index e3bc393..2db39dc 100644
--- a/lisp/gnus/mml.el
+++ b/lisp/gnus/mml.el
@@ -691,23 +691,18 @@ type detected."
 		   (t
 		    ;; Only perform format=flowed filling on text/plain
 		    ;; parts where there either isn't a format parameter
-		    ;; in the mml tag or it says "flowed" and there
-		    ;; actually are hard newlines in the text.
-		    (let (use-hard-newlines)
-		      (when (and mml-enable-flowed
-				 (string= type "text/plain")
-				 (not (string= (cdr (assq 'sign cont)) "pgp"))
-				 (or (null (assq 'format cont))
-				     (string= (cdr (assq 'format cont))
-					      "flowed"))
-				 (setq use-hard-newlines
-				       (text-property-any
-					(point-min) (point-max) 'hard 't)))
-			(fill-flowed-encode)
-			;; Indicate that `mml-insert-mime-headers' should
-			;; insert a "; format=flowed" string unless the
-			;; user has already specified it.
-			(setq flowed (null (assq 'format cont)))))
+		    ;; in the mml tag or it says "flowed".
+		    (when (and mml-enable-flowed
+			       (string= type "text/plain")
+			       (not (string= (cdr (assq 'sign cont)) "pgp"))
+			       (or (null (assq 'format cont))
+				   (string= (cdr (assq 'format cont))
+					    "flowed")))
+		      (fill-flowed-encode)
+		      ;; Indicate that `mml-insert-mime-headers' should
+		      ;; insert a "; format=flowed" string unless the
+		      ;; user has already specified it.
+		      (setq flowed (null (assq 'format cont))))
 		    ;; Prefer `utf-8' for text/calendar parts.
 		    (if (or charset
 			    (not (string= type "text/calendar")))
diff --git a/lisp/mail/flow-fill.el b/lisp/mail/flow-fill.el
index 919490e..932aa8f 100644
--- a/lisp/mail/flow-fill.el
+++ b/lisp/mail/flow-fill.el
@@ -73,50 +73,60 @@ RFC 2646 suggests 66 characters for readability."
 ;;;###autoload
 (defun fill-flowed-encode (&optional buffer)
   (with-current-buffer (or buffer (current-buffer))
-    ;; No point in doing this unless hard newlines is used.
-    (when use-hard-newlines
-      (let ((start (point-min)) end)
-	;; Go through each paragraph, filling it and adding SPC
-	;; as the last character on each line.
-	(while (setq end (text-property-any start (point-max) 'hard 't))
-	  (save-restriction
-	    (narrow-to-region start end)
-	    (let ((fill-column (eval fill-flowed-encode-column t)))
-	      (fill-flowed-fill-buffer))
-	    (goto-char (point-min))
-	    (while (re-search-forward "\n" nil t)
-	      (replace-match " \n" t t))
-	    (goto-char (setq start (1+ (point-max)))))))
-      t)))
-
-(defun fill-flowed-fill-buffer ()
-  (let ((prefix nil)
-	(prev-prefix nil)
-	(start (point-min)))
-    (goto-char (point-min))
-    (while (not (eobp))
-      (setq prefix (and (looking-at "[> ]+")
-			(match-string 0)))
-      (if (equal prefix prev-prefix)
-	  (forward-line 1)
+    (let ((fill-column (eval fill-flowed-encode-column t))
+	  (start (point-min))
+	  (end (point-max)))
+      ;; Only when we've been called to reflow a buffer that doesn't
+      ;; have any hard newlines:
+      (unless (text-property-any start end 'hard 't)
+	(save-excursion
+	  ;; Harden between paras:
+	  (goto-char start)
+	  (while (search-forward "\n\n" end t)
+	    (set-hard-newline-properties
+	     (- (point) 2) (point)))
+	  ;; Harden "  $"
+	  (goto-char start)
+	  (while (search-forward "  \n" end t)
+	    (backward-char)
+	    (backward-delete-char 2)
+	    (set-hard-newline-properties
+	     (point) (1+ (point))))
+	  ;; Harden "^    "
+	  (goto-char start)
+	  (while (search-forward "\n    " end t)
+	    (set-hard-newline-properties
+	     (- (point) 5) (- (point) 4)))))
+      ;; Go through each paragraph, filling it and adding SPC
+      ;; as the last character on each line.
+      (while (and (< start (point-max))
+		  (setq end (or (text-property-any start (point-max) 'hard 't)
+				(point-max))))
 	(save-restriction
-	  (narrow-to-region start (point))
-	  (let ((fill-prefix prev-prefix))
-	    (fill-region (point-min) (point-max) t 'nosqueeze 'to-eop))
-	  (goto-char (point-max)))
-	(setq prev-prefix prefix
-	      start (point))))
-    (save-restriction
-      (narrow-to-region start (point))
-      (let ((fill-prefix prev-prefix))
-	(fill-region (point-min) (point-max) t 'nosqueeze 'to-eop)))))
+	  (narrow-to-region start end)
+	  (let ((prefix
+		 (concat "\n"
+			 (or (and (looking-at ">[> ]*")
+				  (match-string 0)) ""))))
+	    (goto-char start)
+	    (while (search-forward prefix nil t)
+	      (replace-match " " t t))
+	    (goto-char start)
+	    (while (< (+ (point) fill-column) (point-max))
+	      (let ((start (point)))
+		(forward-char fill-column)
+		(when (search-backward " " start t)
+		  (forward-char)
+		  (insert prefix)))))
+	  (setq start (1+ (point-max))))))
+    t))
 
 ;;;###autoload
 (defun fill-flowed (&optional buffer delete-space)
   "Apply RFC2646 decoding to BUFFER.
 If BUFFER is nil, default to the current buffer.
 
-If DELETE-SPACE, delete RFC2646 spaces padding at the end of
+If DELETE-SPACE, delete RFC3676 spaces padding at the end of
 lines."
   (with-current-buffer (or buffer (current-buffer))
     (let ((fill-column  (eval fill-flowed-display-column t)))
@@ -154,7 +164,7 @@ lines."
           ;; Delete the newline.
           (when (eq (following-char) ?\s)
             (delete-char 1))
-          ;; Hack: Don't do the flowing on the signature line.
+          ;; As per RFC3767: Don't do the flowing on the signature line.
           (when (and (not (looking-at "-- $"))
                      (eq (char-before (line-end-position)) ?\s))
             (while (and (not (eobp))
-- 
2.39.2

bug#71017: [PATCH] Flow single-paragraph messages

[Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eww, I forgot about quote handling for this autoharden fallback. 

bug#71017: [PATCH] Flow single-paragraph messages

[Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors]

See discussion in #71017.

In short:

if mml-flowed-enable is nil: no change to behavior

if mml-flowed-enable is t and use-hard-newlines is on: several bugs
fixed, some (multi-paragraph messages) of which were counter to
documentation and some (single-paragraph messages) that were
along documentation lines but which messed up messages royally.

if mml-flowed-enable is t but use-hard-newlines was accidentally
forgotten by user: new behavior; reflow but don't reflow together
separate paragraphs nor reflow across lines that markdown would've
considered hard.
---
 doc/misc/emacs-mime.texi |  7 +--
 lisp/gnus/mml.el         | 29 ++++++-------
 lisp/mail/flow-fill.el   | 93 ++++++++++++++++++++++++----------------
 3 files changed, 69 insertions(+), 60 deletions(-)

diff --git a/doc/misc/emacs-mime.texi b/doc/misc/emacs-mime.texi
index ef7ea61..7621a9a 100644
--- a/doc/misc/emacs-mime.texi
+++ b/doc/misc/emacs-mime.texi
@@ -1087,13 +1087,10 @@ terminated by soft newline characters are filled together and wrapped
 after the column decided by @code{fill-flowed-encode-column}.
 Quotation marks (matching @samp{^>* ?}) are respected.  The variable
 controls how the text will look in a client that does not support
-flowed text, the default is to wrap after 66 characters.  If hard
-newline characters are not present in the buffer, no flow encoding
-occurs.
+flowed text, the default is to wrap after 66 characters.
 
 You can customize the value of the @code{mml-enable-flowed} variable
-to enable or disable the flowed encoding usage when newline
-characters are present in the buffer.
+to enable or disable the flowed encoding usage.
 
 On decoding flowed text, lines with soft newline characters are filled
 together and wrapped after the column decided by
diff --git a/lisp/gnus/mml.el b/lisp/gnus/mml.el
index e3bc393..2db39dc 100644
--- a/lisp/gnus/mml.el
+++ b/lisp/gnus/mml.el
@@ -691,23 +691,18 @@ type detected."
 		   (t
 		    ;; Only perform format=flowed filling on text/plain
 		    ;; parts where there either isn't a format parameter
-		    ;; in the mml tag or it says "flowed" and there
-		    ;; actually are hard newlines in the text.
-		    (let (use-hard-newlines)
-		      (when (and mml-enable-flowed
-				 (string= type "text/plain")
-				 (not (string= (cdr (assq 'sign cont)) "pgp"))
-				 (or (null (assq 'format cont))
-				     (string= (cdr (assq 'format cont))
-					      "flowed"))
-				 (setq use-hard-newlines
-				       (text-property-any
-					(point-min) (point-max) 'hard 't)))
-			(fill-flowed-encode)
-			;; Indicate that `mml-insert-mime-headers' should
-			;; insert a "; format=flowed" string unless the
-			;; user has already specified it.
-			(setq flowed (null (assq 'format cont)))))
+		    ;; in the mml tag or it says "flowed".
+		    (when (and mml-enable-flowed
+			       (string= type "text/plain")
+			       (not (string= (cdr (assq 'sign cont)) "pgp"))
+			       (or (null (assq 'format cont))
+				   (string= (cdr (assq 'format cont))
+					    "flowed")))
+		      (fill-flowed-encode)
+		      ;; Indicate that `mml-insert-mime-headers' should
+		      ;; insert a "; format=flowed" string unless the
+		      ;; user has already specified it.
+		      (setq flowed (null (assq 'format cont))))
 		    ;; Prefer `utf-8' for text/calendar parts.
 		    (if (or charset
 			    (not (string= type "text/calendar")))
diff --git a/lisp/mail/flow-fill.el b/lisp/mail/flow-fill.el
index 919490e..793088a 100644
--- a/lisp/mail/flow-fill.el
+++ b/lisp/mail/flow-fill.el
@@ -73,50 +73,67 @@ RFC 2646 suggests 66 characters for readability."
 ;;;###autoload
 (defun fill-flowed-encode (&optional buffer)
   (with-current-buffer (or buffer (current-buffer))
-    ;; No point in doing this unless hard newlines is used.
-    (when use-hard-newlines
-      (let ((start (point-min)) end)
-	;; Go through each paragraph, filling it and adding SPC
-	;; as the last character on each line.
-	(while (setq end (text-property-any start (point-max) 'hard 't))
-	  (save-restriction
-	    (narrow-to-region start end)
-	    (let ((fill-column (eval fill-flowed-encode-column t)))
-	      (fill-flowed-fill-buffer))
-	    (goto-char (point-min))
-	    (while (re-search-forward "\n" nil t)
-	      (replace-match " \n" t t))
-	    (goto-char (setq start (1+ (point-max)))))))
-      t)))
-
-(defun fill-flowed-fill-buffer ()
-  (let ((prefix nil)
-	(prev-prefix nil)
-	(start (point-min)))
-    (goto-char (point-min))
-    (while (not (eobp))
-      (setq prefix (and (looking-at "[> ]+")
-			(match-string 0)))
-      (if (equal prefix prev-prefix)
-	  (forward-line 1)
+    (let ((fill-column (eval fill-flowed-encode-column t))
+	  (start (point-min))
+	  (end (point-max)))
+      ;; Only when we've been called to reflow a buffer that doesn't
+      ;; have any hard newlines:
+      (unless (text-property-any start end 'hard 't)
+	(save-excursion
+	  ;; Harden between paras:
+	  (goto-char start)
+	  (while (re-search-forward "\n[> ]*\n" end t)
+	    (set-hard-newline-properties
+	     (match-beginning 0) (point))
+	    (backward-char))
+	  ;; Harden before a quote starts:
+	  (goto-char start)
+	  (while (re-search-forward "\n[^>][^\n]*\n>" end t)
+	    (backward-char)
+	    (set-hard-newline-properties
+	     (1- (point)) (point)))
+	  ;; Harden "  $"
+	  (goto-char start)
+	  (while (search-forward "  \n" end t)
+	    (backward-char)
+	    (backward-delete-char 2)
+	    (set-hard-newline-properties
+	     (point) (1+ (point))))
+	  ;; Harden "^    "
+	  (goto-char start)
+	  (while (search-forward "\n    " end t)
+	    (set-hard-newline-properties
+	     (- (point) 5) (- (point) 4)))))
+      ;; Go through each paragraph, filling it and adding SPC
+      ;; as the last character on each line.
+      (while (and (< start (point-max))
+		  (setq end (or (text-property-any start (point-max) 'hard 't)
+				(point-max))))
 	(save-restriction
-	  (narrow-to-region start (point))
-	  (let ((fill-prefix prev-prefix))
-	    (fill-region (point-min) (point-max) t 'nosqueeze 'to-eop))
-	  (goto-char (point-max)))
-	(setq prev-prefix prefix
-	      start (point))))
-    (save-restriction
-      (narrow-to-region start (point))
-      (let ((fill-prefix prev-prefix))
-	(fill-region (point-min) (point-max) t 'nosqueeze 'to-eop)))))
+	  (narrow-to-region start end)
+	  (let ((prefix
+		 (concat "\n"
+			 (or (and (looking-at ">[> ]*")
+				  (match-string 0)) ""))))
+	    (goto-char start)
+	    (while (search-forward prefix nil t)
+	      (replace-match " " t t))
+	    (goto-char start)
+	    (while (< (+ (point) fill-column) (point-max))
+	      (let ((start (point)))
+		(forward-char fill-column)
+		(when (search-backward " " start t)
+		  (forward-char)
+		  (insert prefix)))))
+	  (setq start (1+ (point-max))))))
+    t))
 
 ;;;###autoload
 (defun fill-flowed (&optional buffer delete-space)
   "Apply RFC2646 decoding to BUFFER.
 If BUFFER is nil, default to the current buffer.
 
-If DELETE-SPACE, delete RFC2646 spaces padding at the end of
+If DELETE-SPACE, delete RFC3676 spaces padding at the end of
 lines."
   (with-current-buffer (or buffer (current-buffer))
     (let ((fill-column  (eval fill-flowed-display-column t)))
@@ -154,7 +171,7 @@ lines."
           ;; Delete the newline.
           (when (eq (following-char) ?\s)
             (delete-char 1))
-          ;; Hack: Don't do the flowing on the signature line.
+          ;; As per RFC3767: Don't do the flowing on the signature line.
           (when (and (not (looking-at "-- $"))
                      (eq (char-before (line-end-position)) ?\s))
             (while (and (not (eobp))
-- 
2.39.2