* bug#71017: fill-flowed-encode
@ 2024-05-17 20:23 Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors
2024-06-30 5:44 ` Stefan Kangas
0 siblings, 1 reply; 10+ messages in thread
From: Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-05-17 20:23 UTC (permalink / raw
To: 71017
Hi y'all.
flow-fill.el.gz has a pair of functions, fill-flowed-encode and
fill-flowed-fill-buffer (the latter is only called from the
former).
Here is a fixed version of the former that then also deprecates
the latter (I have signed FSF copyright papers):
(defun fill-flowed-encode (&optional buffer)
(with-current-buffer (or buffer (current-buffer))
;; No point in doing this unless hard newlines is used.
(when use-hard-newlines
(let ((start (point-min)) end)
;; Go through each paragraph, filling it and adding SPC
;; as the last character on each line.
(while (and (< start (point-max))
(setq end (or (text-property-any start (point-max) 'hard 't)
(point-max))))
(save-restriction
(narrow-to-region start end)
(let ((fill-column (eval fill-flowed-encode-column t))
(prefix
(concat "\n"
(or (and (looking-at ">[> ]*")
(match-string 0)) ""))))
(while (search-forward prefix nil t)
(replace-match " " t t))
(goto-char start)
(while (< (+ (point) fill-column) (point-max))
(forward-char fill-column)
(search-backward " ")
(forward-char)
(insert prefix)))
(setq start (1+ (point-max))))))
t)))
This fixes two bugs when sending RFC 2646–formatted email.
First, the old code didn't refill or encode the last paragraph at
all unless there was at least one hard newline EOF.
Second, the old code borked up code indented with tabs and spaces
(iff that code had overly long lines), such as the Lisp code in
this email. It could sometimes insert extra in the middle of such
long lines.
Here is an example of what it would do. It would turn this:
(defun lorem (ipsum)
(dolor sit amet)
(consectetur adipiscing elit (sed do eiusmod tempor incididunt
ut labore et dolore magna aliqua))
(ut enim ad minim veniam
(quis nostrud exercitation ullamco laboris nisi ut aliquip
ex ea commodo consequat
(duis aute irure dolor in reprehenderit in
voluptate velit esse cillum dolore
eu fugiat nulla pariatur)
excepteur sint occaecat cupidatat non proident
(sunt in culpa qui officia deserunt mollit anim id est
laborumd))))
into this:
(defun lorem (ipsum)
(dolor sit amet)
(consectetur adipiscing elit (sed do eiusmod tempor incididunt
ut labore et dolore magna aliqua))
(ut enim ad minim veniam
(quis nostrud exercitation ullamco laboris nisi ut aliquip
ex ea commodo consequat
(duis aute irure dolor in reprehenderit in
voluptate velit esse cillum dolore
eu fugiat nulla pariatur)
excepteur sint occaecat cupidatat non proident
(sunt in culpa qui officia deserunt mollit anim id est
laborumd))))
It was breaking lines awkwardly so when they're reconnected they
have extra whitespace in the mkddle of lines.
^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#71017: fill-flowed-encode
2024-05-17 20:23 bug#71017: fill-flowed-encode Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors
@ 2024-06-30 5:44 ` Stefan Kangas
2024-06-30 8:32 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors
0 siblings, 1 reply; 10+ messages in thread
From: Stefan Kangas @ 2024-06-30 5:44 UTC (permalink / raw
To: Sandra Snan; +Cc: 71017
Sandra Snan <sandra.snan@idiomdrottning.org> writes:
> Hi y'all.
Hi Sandra,
> flow-fill.el.gz has a pair of functions, fill-flowed-encode and
> fill-flowed-fill-buffer (the latter is only called from the former).
>
> Here is a fixed version of the former that then also deprecates the latter (I
> have signed FSF copyright papers):
>
> (defun fill-flowed-encode (&optional buffer)
> (with-current-buffer (or buffer (current-buffer))
> ;; No point in doing this unless hard newlines is used.
> (when use-hard-newlines
> (let ((start (point-min)) end)
> ;; Go through each paragraph, filling it and adding SPC
> ;; as the last character on each line.
> (while (and (< start (point-max))
> (setq end (or (text-property-any start (point-max) 'hard 't)
> (point-max))))
> (save-restriction
> (narrow-to-region start end)
> (let ((fill-column (eval fill-flowed-encode-column t))
> (prefix
> (concat "\n"
> (or (and (looking-at ">[> ]*")
> (match-string 0)) ""))))
> (while (search-forward prefix nil t)
> (replace-match " " t t))
> (goto-char start)
> (while (< (+ (point) fill-column) (point-max))
> (forward-char fill-column)
> (search-backward " ")
> (forward-char)
> (insert prefix)))
> (setq start (1+ (point-max))))))
> t)))
>
> This fixes two bugs when sending RFC 2646–formatted email.
>
> First, the old code didn't refill or encode the last paragraph at all unless
> there was at least one hard newline EOF.
>
> Second, the old code borked up code indented with tabs and spaces (iff that code
> had overly long lines), such as the Lisp code in this email. It could sometimes
> insert extra in the middle of such long lines.
>
> Here is an example of what it would do. It would turn this:
>
> (defun lorem (ipsum)
> (dolor sit amet)
> (consectetur adipiscing elit (sed do eiusmod tempor incididunt ut labore et
> dolore magna aliqua))
> (ut enim ad minim veniam
> (quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
> consequat
> (duis aute irure dolor in reprehenderit in
> voluptate velit esse cillum dolore
> eu fugiat nulla pariatur)
> excepteur sint occaecat cupidatat non proident
> (sunt in culpa qui officia deserunt mollit anim id est laborumd))))
>
> into this:
>
> (defun lorem (ipsum)
> (dolor sit amet)
> (consectetur adipiscing elit (sed do eiusmod tempor incididunt
> ut labore et dolore magna aliqua))
> (ut enim ad minim veniam
> (quis nostrud exercitation ullamco laboris nisi ut aliquip
> ex ea commodo consequat
> (duis aute irure dolor in reprehenderit in
> voluptate velit esse cillum dolore
> eu fugiat nulla pariatur)
> excepteur sint occaecat cupidatat non proident
> (sunt in culpa qui officia deserunt mollit anim id est
> laborumd))))
>
> It was breaking lines awkwardly so when they're reconnected they have extra
> whitespace in the mkddle of lines.
Could you please resend the above as patches formatted by
git format-patch -1
?
It will make it easier for us to review and install the proposed
changes. Thanks in advance.
^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#71017: fill-flowed-encode
2024-06-30 5:44 ` Stefan Kangas
@ 2024-06-30 8:32 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors
2024-06-30 14:34 ` Stefan Kangas
0 siblings, 1 reply; 10+ messages in thread
From: Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-06-30 8:32 UTC (permalink / raw
To: Stefan Kangas; +Cc: 71017
Stefan Kangas <stefankangas@gmail.com> writes:
> Could you please resend the above as patches formatted by
>
> git format-patch -1
>
> ?
>
I'm glad I'm gonna get a chance to resend because I've fixed some
more bugs in the version I've been dogfooding since I first sent
that so I wanna send an updated version that has that.
Normally when I'm sending Emacs lisp patches, it's on packages and
they're usually in git already. Here it's a core file so the
question I've got is what is the git repo I should make the change
in? Then I could git send-email to this In-Reply-To thread id.
Stefan Kangas <stefankangas@gmail.com> writes:
> Sandra Snan <sandra.snan@idiomdrottning.org> writes:
>
>> Hi y'all.
>
> Hi Sandra,
>
>> flow-fill.el.gz has a pair of functions, fill-flowed-encode and
>> fill-flowed-fill-buffer (the latter is only called from the
>> former).
>> Here is a fixed version of the former that then also
>> deprecates the latter (I have signed FSF copyright papers):
>>
>> (defun fill-flowed-encode (&optional buffer) (with-current-buffer
>> (or buffer (current-buffer)) ;; No point in doing this unless
>> hard newlines is used. (when use-hard-newlines (let ((start
>> (point-min)) end) ;; Go through each paragraph, filling it and
>> adding SPC ;; as the last character on each line. (while (and (<
>> start (point-max)) (setq end (or (text-property-any start
>> (point-max) 'hard 't) (point-max)))) (save-restriction
>> (narrow-to-region start end) (let ((fill-column (eval
>> fill-flowed-encode-column t)) (prefix (concat "\n"
>> (or (and (looking-at ">[> ]*") (match-string 0)) ""))))
>> (while (search-forward prefix nil t) (replace-match " " t t))
>> (goto-char start) (while (< (+ (point) fill-column)
>> (point-max)) (forward-char fill-column) (search-backward " ")
>> (forward-char) (insert prefix))) (setq start (1+
>> (point-max)))))) t)))
>> This fixes two bugs when sending RFC
>> 2646–formatted email.
>> First, the old code didn't refill or
>> encode the last paragraph at all unless there was at least one hard
>> newline EOF.
>> Second, the old code borked up code indented with
>> tabs and spaces (iff that code had overly long lines), such as the
>> Lisp code in this email. It could sometimes insert extra in the
>> middle of such long lines.
>> Here is an example of what it would
>> do. It would turn this:
>> (defun lorem (ipsum) (dolor sit amet)
>> (consectetur adipiscing elit (sed do eiusmod tempor incididunt ut
>> labore et dolore magna aliqua)) (ut enim ad minim veniam
>> (quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea
>> commodo consequat (duis aute irure dolor in reprehenderit
>> in voluptate velit esse cillum dolore eu fugiat nulla
>> pariatur) excepteur sint occaecat cupidatat non proident
>> (sunt in culpa qui officia deserunt mollit anim id est
>> laborumd))))
>> into this:
>> (defun lorem (ipsum) (dolor sit
>> amet) (consectetur adipiscing elit (sed do eiusmod tempor
>> incididunt ut labore et dolore magna aliqua)) (ut enim ad minim
>> veniam (quis nostrud exercitation ullamco laboris nisi ut
>> aliquip ex ea commodo consequat (duis aute irure dolor in
>> reprehenderit in voluptate velit esse cillum dolore eu
>> fugiat nulla pariatur) excepteur sint occaecat cupidatat non
>> proident (sunt in culpa qui officia deserunt mollit anim id
>> est laborumd))))
>> It was breaking lines awkwardly so when
>> they're reconnected they have extra whitespace in the mkddle of
>> lines.
>
> Could you please resend the above as patches formatted by
>
> git format-patch -1
>
> ?
>
> It will make it easier for us to review and install the proposed
> changes. Thanks in advance.
^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#71017: fill-flowed-encode
2024-06-30 8:32 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors
@ 2024-06-30 14:34 ` Stefan Kangas
2024-07-06 20:49 ` bug#71017: [PATCH] Flow single-paragraph messages Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors
0 siblings, 1 reply; 10+ messages in thread
From: Stefan Kangas @ 2024-06-30 14:34 UTC (permalink / raw
To: Sandra Snan; +Cc: 71017
Sandra Snan <sandra.snan@idiomdrottning.org> writes:
> I'm glad I'm gonna get a chance to resend because I've fixed some
> more bugs in the version I've been dogfooding since I first sent
> that so I wanna send an updated version that has that.
Great, thanks.
> Normally when I'm sending Emacs lisp patches, it's on packages and
> they're usually in git already. Here it's a core file so the
> question I've got is what is the git repo I should make the change
> in? Then I could git send-email to this In-Reply-To thread id.
https://git.savannah.gnu.org/cgit/emacs.git
Please have a look at the CONTRIBUTE file in the Emacs source
distribution.
https://git.savannah.gnu.org/cgit/emacs.git/tree/CONTRIBUTE
^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#71017: [PATCH] Flow single-paragraph messages
2024-06-30 14:34 ` Stefan Kangas
@ 2024-07-06 20:49 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors
2024-07-07 5:43 ` Eli Zaretskii
0 siblings, 1 reply; 10+ messages in thread
From: Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-07-06 20:49 UTC (permalink / raw
To: 71017; +Cc: Sandra Snan
This fixes two bugs when sending RFC 2646–formatted email.
First, the old code didn't refill or encode the last paragraph at all
unless there was at least one hard newline EOF. This was a bee to track
down because there were two separate issues at play. One was a a bug in
flow-fill.el where every paragraph except the last paragraph was
reflowed, but the last paragraph would stay hardwrapped. Manually
placing a hard newline at the end of the file was a workaround but I
don't always remember to do that. I managed to fix that bug a few months
ago.
Second, the old code borked up code indented with tabs and spaces
(iff that code had overly long lines), such as Lisp code. It could
sometimes insert extra whitespace in the middle of such long lines.
I fixed that bug shortly after the first one.
But for months dogfooding those two changes, sometimes a hardwrapped
email would still be sent. I finally managed to debug and figure it out
and it took all day. Turns out mml just plain didn't call the
fill-flowed-encode function if the message doesn't have any hard
newlines (newlines with the hard text property). Well, of course a
single-paragraph email isn't gonna have any hard newlines! But it still
needs reflowing!
So I've now changed that and updated the documentation to match those
news semantics. I went all the way, but a possible compromise might be
to not-flow a message that has \n\n but no hard text props, since that's
a sign that something is wrong.
Since the use-hard-newlines variable is buffer local and all this
reflowing is being done in a temp buffer, that variable is more than
useless so I've removed references to it.
---
doc/misc/emacs-mime.texi | 7 ++---
lisp/gnus/mml.el | 29 ++++++++----------
lisp/mail/flow-fill.el | 65 +++++++++++++++++-----------------------
3 files changed, 41 insertions(+), 60 deletions(-)
diff --git a/doc/misc/emacs-mime.texi b/doc/misc/emacs-mime.texi
index ef7ea61..7621a9a 100644
--- a/doc/misc/emacs-mime.texi
+++ b/doc/misc/emacs-mime.texi
@@ -1087,13 +1087,10 @@ terminated by soft newline characters are filled together and wrapped
after the column decided by @code{fill-flowed-encode-column}.
Quotation marks (matching @samp{^>* ?}) are respected. The variable
controls how the text will look in a client that does not support
-flowed text, the default is to wrap after 66 characters. If hard
-newline characters are not present in the buffer, no flow encoding
-occurs.
+flowed text, the default is to wrap after 66 characters.
You can customize the value of the @code{mml-enable-flowed} variable
-to enable or disable the flowed encoding usage when newline
-characters are present in the buffer.
+to enable or disable the flowed encoding usage.
On decoding flowed text, lines with soft newline characters are filled
together and wrapped after the column decided by
diff --git a/lisp/gnus/mml.el b/lisp/gnus/mml.el
index e3bc393..2db39dc 100644
--- a/lisp/gnus/mml.el
+++ b/lisp/gnus/mml.el
@@ -691,23 +691,18 @@ type detected."
(t
;; Only perform format=flowed filling on text/plain
;; parts where there either isn't a format parameter
- ;; in the mml tag or it says "flowed" and there
- ;; actually are hard newlines in the text.
- (let (use-hard-newlines)
- (when (and mml-enable-flowed
- (string= type "text/plain")
- (not (string= (cdr (assq 'sign cont)) "pgp"))
- (or (null (assq 'format cont))
- (string= (cdr (assq 'format cont))
- "flowed"))
- (setq use-hard-newlines
- (text-property-any
- (point-min) (point-max) 'hard 't)))
- (fill-flowed-encode)
- ;; Indicate that `mml-insert-mime-headers' should
- ;; insert a "; format=flowed" string unless the
- ;; user has already specified it.
- (setq flowed (null (assq 'format cont)))))
+ ;; in the mml tag or it says "flowed".
+ (when (and mml-enable-flowed
+ (string= type "text/plain")
+ (not (string= (cdr (assq 'sign cont)) "pgp"))
+ (or (null (assq 'format cont))
+ (string= (cdr (assq 'format cont))
+ "flowed")))
+ (fill-flowed-encode)
+ ;; Indicate that `mml-insert-mime-headers' should
+ ;; insert a "; format=flowed" string unless the
+ ;; user has already specified it.
+ (setq flowed (null (assq 'format cont))))
;; Prefer `utf-8' for text/calendar parts.
(if (or charset
(not (string= type "text/calendar")))
diff --git a/lisp/mail/flow-fill.el b/lisp/mail/flow-fill.el
index 919490e..5c9ae21 100644
--- a/lisp/mail/flow-fill.el
+++ b/lisp/mail/flow-fill.el
@@ -73,50 +73,39 @@ RFC 2646 suggests 66 characters for readability."
;;;###autoload
(defun fill-flowed-encode (&optional buffer)
(with-current-buffer (or buffer (current-buffer))
- ;; No point in doing this unless hard newlines is used.
- (when use-hard-newlines
- (let ((start (point-min)) end)
- ;; Go through each paragraph, filling it and adding SPC
- ;; as the last character on each line.
- (while (setq end (text-property-any start (point-max) 'hard 't))
- (save-restriction
- (narrow-to-region start end)
- (let ((fill-column (eval fill-flowed-encode-column t)))
- (fill-flowed-fill-buffer))
- (goto-char (point-min))
- (while (re-search-forward "\n" nil t)
- (replace-match " \n" t t))
- (goto-char (setq start (1+ (point-max)))))))
- t)))
-
-(defun fill-flowed-fill-buffer ()
- (let ((prefix nil)
- (prev-prefix nil)
- (start (point-min)))
- (goto-char (point-min))
- (while (not (eobp))
- (setq prefix (and (looking-at "[> ]+")
- (match-string 0)))
- (if (equal prefix prev-prefix)
- (forward-line 1)
+ (let ((fill-column (eval fill-flowed-encode-column t))
+ (start (point-min))
+ end)
+ ;; Go through each paragraph, filling it and adding SPC
+ ;; as the last character on each line.
+ (while (and (< start (point-max))
+ (setq end (or (text-property-any start (point-max) 'hard 't)
+ (point-max))))
(save-restriction
- (narrow-to-region start (point))
- (let ((fill-prefix prev-prefix))
- (fill-region (point-min) (point-max) t 'nosqueeze 'to-eop))
- (goto-char (point-max)))
- (setq prev-prefix prefix
- start (point))))
- (save-restriction
- (narrow-to-region start (point))
- (let ((fill-prefix prev-prefix))
- (fill-region (point-min) (point-max) t 'nosqueeze 'to-eop)))))
+ (narrow-to-region start end)
+ (let ((prefix
+ (concat "\n"
+ (or (and (looking-at ">[> ]*")
+ (match-string 0)) ""))))
+ (goto-char start)
+ (while (search-forward prefix nil t)
+ (replace-match " " t t))
+ (goto-char start)
+ (while (< (+ (point) fill-column) (point-max))
+ (let ((start (point)))
+ (forward-char fill-column)
+ (when (search-backward " " start t)
+ (forward-char)
+ (insert prefix)))))
+ (setq start (1+ (point-max))))))
+ t))
;;;###autoload
(defun fill-flowed (&optional buffer delete-space)
"Apply RFC2646 decoding to BUFFER.
If BUFFER is nil, default to the current buffer.
-If DELETE-SPACE, delete RFC2646 spaces padding at the end of
+If DELETE-SPACE, delete RFC3676 spaces padding at the end of
lines."
(with-current-buffer (or buffer (current-buffer))
(let ((fill-column (eval fill-flowed-display-column t)))
@@ -154,7 +143,7 @@ lines."
;; Delete the newline.
(when (eq (following-char) ?\s)
(delete-char 1))
- ;; Hack: Don't do the flowing on the signature line.
+ ;; As per RFC3767: Don't do the flowing on the signature line.
(when (and (not (looking-at "-- $"))
(eq (char-before (line-end-position)) ?\s))
(while (and (not (eobp))
--
2.39.2
^ permalink raw reply related [flat|nested] 10+ messages in thread
* bug#71017: [PATCH] Flow single-paragraph messages
2024-07-06 20:49 ` bug#71017: [PATCH] Flow single-paragraph messages Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors
@ 2024-07-07 5:43 ` Eli Zaretskii
2024-07-07 8:34 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors
0 siblings, 1 reply; 10+ messages in thread
From: Eli Zaretskii @ 2024-07-07 5:43 UTC (permalink / raw
To: Sandra Snan, Eric Abrahamsen; +Cc: 71017
> Cc: Sandra Snan <sandra.snan@idiomdrottning.org>
> Date: Sat, 6 Jul 2024 22:49:50 +0200
> From: Sandra Snan via "Bug reports for GNU Emacs,
> the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
>
> This fixes two bugs when sending RFC 2646–formatted email.
Thanks. (I also see RFC 3676 mentioned -- what is this about?)
> First, the old code didn't refill or encode the last paragraph at all
> unless there was at least one hard newline EOF.
Isn't this the documented behavior?
> But for months dogfooding those two changes, sometimes a hardwrapped
> email would still be sent. I finally managed to debug and figure it out
> and it took all day. Turns out mml just plain didn't call the
> fill-flowed-encode function if the message doesn't have any hard
> newlines (newlines with the hard text property). Well, of course a
> single-paragraph email isn't gonna have any hard newlines! But it still
> needs reflowing!
>
> So I've now changed that and updated the documentation to match those
> news semantics. I went all the way, but a possible compromise might be
> to not-flow a message that has \n\n but no hard text props, since that's
> a sign that something is wrong.
The change seems to be an incompatible behavior change, so I wonder
whether we'd need some way for users to get back old behavior. Eric,
WDYT?
I'll leave it to Eric to comment on the code changes.
^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#71017: [PATCH] Flow single-paragraph messages
2024-07-07 5:43 ` Eli Zaretskii
@ 2024-07-07 8:34 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors
2024-07-07 9:04 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors
0 siblings, 1 reply; 10+ messages in thread
From: Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-07-07 8:34 UTC (permalink / raw
To: Eli Zaretskii, Eric Abrahamsen; +Cc: 71017
Eli Zaretskii <eliz@gnu.org> writes:
> Thanks. (I also see RFC 3676 mentioned -- what is this about?)
I mentioned RFC 3676 twice for different reasons.
First, it's introduced the delsp parameter which wasn't in 2646 so
code related to that parameter shouldn't talk about 2646 but rather
3676.
Second, I saw a comment referring to not tampering with the sig
line as a hack. I happened to have the section of RFC 3676 open
that mandates that behavior so I changed the comment, however, this
might be in RFC 2646 also, I don't know that, so in that case that
comment might better change to refer to 2646 instead.
>> First, the old code didn't refill or encode the last paragraph
>> at all unless there was at least one hard newline EOF.
>
> Isn't this the documented behavior?
Re multi-paragraph messages:
No, it's not the documented behavior, it was an unrelated bug in
fill-flowed-encode. It would refill all the other paragraphs,
separated by hard newlines, just not the last one. That was a bug
and broke documented behavior. I fixed that bug + another unrelated
reflow bug. That was in fill-flowed-encode and that bugfix doesn't
rely on the change in mml. With my fix in fill-flowed-encode,
multi-paragraph-messages started working fine.
Re single-paragraph messages:
However, according to the old documented behavior, a message that
contained no hard newlines should not be refilled. This documented
behavior in mml-generate-mime-1 meant that single-paragraph
messages would not be filled even with the fill-flowed-encode bug
fixed. That is an unintended bad consequence of the documented
behavior, a "bug in the design".
I did change that but I updated the documentation to match.
> The change seems to be an incompatible behavior change, so I
> wonder whether we'd need some way for users to get back old
> behavior.
There is still the (neglected) defcustom mml-enable-flowed which
now becomes more relevant since it's a way to turn off all this
meddling and reflowing in the first place.
The old documented behavior was bugged-by-design. It's not right
that single paragraph messages are hardwrapped and not reflowed. In
my day-to-day I write many messages in Emacs that I later see in
threads in another MUA (Delta Chat) and these messages stand out in
a way that something is wrong with them.
However, one intent behind the old behavior was, in spirit, good:
It'd be good to detect whether or not users with mml-enable-flowed
on have remembered to also turn on use-hard-newlines, which is
important for users with that on to do, especially since
mml-enable-flowed defaults to t. The old attempt at doing that was
flawed since it only worked reliably for multi-paragraph messages.
Unfortunately there's currently no way to detect in a
single-paragraph message whether or not use-hard-newlines have been
turned on, since the variable it sets is buffer local. (One
extremely klugy workaround would be to change the
message-send-and-exit command to check whether use-hard-newlines is
on and if it is, add an extra hard newline EOT just for detecting
this. Not super into that solution so hopefully there are other
ways.)
With this patch, the defcustom mml-enable-flowed becomes _the_
setting for this, which does match a lot of documentation on the
books. Perhaps it shall no longer default to t though since it
completely borks messages up if it's t but use-hard-newlines are
not on!
So here we are:
The new behavior has a problem: messages will get reflowed if
mml-enable-flowed is t (the default!) even when use-hard-newlines
is off, meaning that even separate "\n\n" paragraphs will get
flowed together which is not what people want. Use-hard-newlines
should be mandatory whenever mml-enable-flowed is on.
The old behavior is not OK since single-paragraph messages will get
messed up, hardwrapped even when those newlines were advertised as
"soft", or not softwrapped even when the paragraph consists of just
one single super long unbroken line.
I also have sent a patch to the messages-are-flowing project
highlighting the importance of this variable:
https://github.com/legoscia/messages-are-flowing/pull/15/commits/ae432723c2565ceced5d01d9aa2d314bd42aaa3c
So how about this idea:
If mml-enable-flowed is on but fill-flowed-encode is asked to flow
a message that doesn't have any hard newlines, assume Markdown
semantics, i.e. special treatment for "\n\n+", " $", and "^ ".
I'll see if I have time to implement that this morning. I think I'd
place that change in fill-flowed-encode.
That wouldn't affect people with mml-enable-flowed off, or people
with both mml-enable-flowed and use-hard-newlines on, it'd just be
a sort of DWIM fallback based on the guess that markdown semantics
are somewhat widely known or expected in 2024, to prevent separate
paragraphs to be flowed together for people with "incorrect"
settings.
Again, if there were a cross-buffer way to reliably detect whether
use-hard-newlines is on, that dwimmy fallback wouldn't be needed.
I thought about whether it'd have been better if it was instead
soft newlines that were marked with a text property, not hard ones,
but in the end that wouldn't properly softwrap messages with just
one single overly long line.
^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#71017: [PATCH] Flow single-paragraph messages
2024-07-07 8:34 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors
@ 2024-07-07 9:04 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors
2024-07-07 9:16 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors
0 siblings, 1 reply; 10+ messages in thread
From: Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-07-07 9:04 UTC (permalink / raw
To: 71017; +Cc: Sandra Snan
See discussion in #71017.
In short:
if mml-flowed-enable is nil: no change to behavior
if mml-flowed-enable is t and use-hard-newlines is on: several bugs
fixed, some (multi-paragraph messages) of which were counter to
documentation and some (single-paragraph messages) that were
along documentation lines but which messed up messages royally.
if mml-flowed-enable is t but use-hard-newlines was accidentally
forgotten by user: new behavior; reflow but don't reflow together
separate paragraphs nor reflow across lines that markdown would've
considered hard.
---
doc/misc/emacs-mime.texi | 7 +---
lisp/gnus/mml.el | 29 ++++++--------
lisp/mail/flow-fill.el | 86 ++++++++++++++++++++++------------------
3 files changed, 62 insertions(+), 60 deletions(-)
diff --git a/doc/misc/emacs-mime.texi b/doc/misc/emacs-mime.texi
index ef7ea61..7621a9a 100644
--- a/doc/misc/emacs-mime.texi
+++ b/doc/misc/emacs-mime.texi
@@ -1087,13 +1087,10 @@ terminated by soft newline characters are filled together and wrapped
after the column decided by @code{fill-flowed-encode-column}.
Quotation marks (matching @samp{^>* ?}) are respected. The variable
controls how the text will look in a client that does not support
-flowed text, the default is to wrap after 66 characters. If hard
-newline characters are not present in the buffer, no flow encoding
-occurs.
+flowed text, the default is to wrap after 66 characters.
You can customize the value of the @code{mml-enable-flowed} variable
-to enable or disable the flowed encoding usage when newline
-characters are present in the buffer.
+to enable or disable the flowed encoding usage.
On decoding flowed text, lines with soft newline characters are filled
together and wrapped after the column decided by
diff --git a/lisp/gnus/mml.el b/lisp/gnus/mml.el
index e3bc393..2db39dc 100644
--- a/lisp/gnus/mml.el
+++ b/lisp/gnus/mml.el
@@ -691,23 +691,18 @@ type detected."
(t
;; Only perform format=flowed filling on text/plain
;; parts where there either isn't a format parameter
- ;; in the mml tag or it says "flowed" and there
- ;; actually are hard newlines in the text.
- (let (use-hard-newlines)
- (when (and mml-enable-flowed
- (string= type "text/plain")
- (not (string= (cdr (assq 'sign cont)) "pgp"))
- (or (null (assq 'format cont))
- (string= (cdr (assq 'format cont))
- "flowed"))
- (setq use-hard-newlines
- (text-property-any
- (point-min) (point-max) 'hard 't)))
- (fill-flowed-encode)
- ;; Indicate that `mml-insert-mime-headers' should
- ;; insert a "; format=flowed" string unless the
- ;; user has already specified it.
- (setq flowed (null (assq 'format cont)))))
+ ;; in the mml tag or it says "flowed".
+ (when (and mml-enable-flowed
+ (string= type "text/plain")
+ (not (string= (cdr (assq 'sign cont)) "pgp"))
+ (or (null (assq 'format cont))
+ (string= (cdr (assq 'format cont))
+ "flowed")))
+ (fill-flowed-encode)
+ ;; Indicate that `mml-insert-mime-headers' should
+ ;; insert a "; format=flowed" string unless the
+ ;; user has already specified it.
+ (setq flowed (null (assq 'format cont))))
;; Prefer `utf-8' for text/calendar parts.
(if (or charset
(not (string= type "text/calendar")))
diff --git a/lisp/mail/flow-fill.el b/lisp/mail/flow-fill.el
index 919490e..932aa8f 100644
--- a/lisp/mail/flow-fill.el
+++ b/lisp/mail/flow-fill.el
@@ -73,50 +73,60 @@ RFC 2646 suggests 66 characters for readability."
;;;###autoload
(defun fill-flowed-encode (&optional buffer)
(with-current-buffer (or buffer (current-buffer))
- ;; No point in doing this unless hard newlines is used.
- (when use-hard-newlines
- (let ((start (point-min)) end)
- ;; Go through each paragraph, filling it and adding SPC
- ;; as the last character on each line.
- (while (setq end (text-property-any start (point-max) 'hard 't))
- (save-restriction
- (narrow-to-region start end)
- (let ((fill-column (eval fill-flowed-encode-column t)))
- (fill-flowed-fill-buffer))
- (goto-char (point-min))
- (while (re-search-forward "\n" nil t)
- (replace-match " \n" t t))
- (goto-char (setq start (1+ (point-max)))))))
- t)))
-
-(defun fill-flowed-fill-buffer ()
- (let ((prefix nil)
- (prev-prefix nil)
- (start (point-min)))
- (goto-char (point-min))
- (while (not (eobp))
- (setq prefix (and (looking-at "[> ]+")
- (match-string 0)))
- (if (equal prefix prev-prefix)
- (forward-line 1)
+ (let ((fill-column (eval fill-flowed-encode-column t))
+ (start (point-min))
+ (end (point-max)))
+ ;; Only when we've been called to reflow a buffer that doesn't
+ ;; have any hard newlines:
+ (unless (text-property-any start end 'hard 't)
+ (save-excursion
+ ;; Harden between paras:
+ (goto-char start)
+ (while (search-forward "\n\n" end t)
+ (set-hard-newline-properties
+ (- (point) 2) (point)))
+ ;; Harden " $"
+ (goto-char start)
+ (while (search-forward " \n" end t)
+ (backward-char)
+ (backward-delete-char 2)
+ (set-hard-newline-properties
+ (point) (1+ (point))))
+ ;; Harden "^ "
+ (goto-char start)
+ (while (search-forward "\n " end t)
+ (set-hard-newline-properties
+ (- (point) 5) (- (point) 4)))))
+ ;; Go through each paragraph, filling it and adding SPC
+ ;; as the last character on each line.
+ (while (and (< start (point-max))
+ (setq end (or (text-property-any start (point-max) 'hard 't)
+ (point-max))))
(save-restriction
- (narrow-to-region start (point))
- (let ((fill-prefix prev-prefix))
- (fill-region (point-min) (point-max) t 'nosqueeze 'to-eop))
- (goto-char (point-max)))
- (setq prev-prefix prefix
- start (point))))
- (save-restriction
- (narrow-to-region start (point))
- (let ((fill-prefix prev-prefix))
- (fill-region (point-min) (point-max) t 'nosqueeze 'to-eop)))))
+ (narrow-to-region start end)
+ (let ((prefix
+ (concat "\n"
+ (or (and (looking-at ">[> ]*")
+ (match-string 0)) ""))))
+ (goto-char start)
+ (while (search-forward prefix nil t)
+ (replace-match " " t t))
+ (goto-char start)
+ (while (< (+ (point) fill-column) (point-max))
+ (let ((start (point)))
+ (forward-char fill-column)
+ (when (search-backward " " start t)
+ (forward-char)
+ (insert prefix)))))
+ (setq start (1+ (point-max))))))
+ t))
;;;###autoload
(defun fill-flowed (&optional buffer delete-space)
"Apply RFC2646 decoding to BUFFER.
If BUFFER is nil, default to the current buffer.
-If DELETE-SPACE, delete RFC2646 spaces padding at the end of
+If DELETE-SPACE, delete RFC3676 spaces padding at the end of
lines."
(with-current-buffer (or buffer (current-buffer))
(let ((fill-column (eval fill-flowed-display-column t)))
@@ -154,7 +164,7 @@ lines."
;; Delete the newline.
(when (eq (following-char) ?\s)
(delete-char 1))
- ;; Hack: Don't do the flowing on the signature line.
+ ;; As per RFC3767: Don't do the flowing on the signature line.
(when (and (not (looking-at "-- $"))
(eq (char-before (line-end-position)) ?\s))
(while (and (not (eobp))
--
2.39.2
^ permalink raw reply related [flat|nested] 10+ messages in thread
* bug#71017: [PATCH] Flow single-paragraph messages
2024-07-07 9:04 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors
@ 2024-07-07 9:16 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors
2024-07-07 10:02 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors
0 siblings, 1 reply; 10+ messages in thread
From: Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-07-07 9:16 UTC (permalink / raw
To: 71017
Eww, I forgot about quote handling for this autoharden fallback.
^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#71017: [PATCH] Flow single-paragraph messages
2024-07-07 9:16 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors
@ 2024-07-07 10:02 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors
0 siblings, 0 replies; 10+ messages in thread
From: Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-07-07 10:02 UTC (permalink / raw
To: 71017; +Cc: Sandra Snan
See discussion in #71017.
In short:
if mml-flowed-enable is nil: no change to behavior
if mml-flowed-enable is t and use-hard-newlines is on: several bugs
fixed, some (multi-paragraph messages) of which were counter to
documentation and some (single-paragraph messages) that were
along documentation lines but which messed up messages royally.
if mml-flowed-enable is t but use-hard-newlines was accidentally
forgotten by user: new behavior; reflow but don't reflow together
separate paragraphs nor reflow across lines that markdown would've
considered hard.
---
doc/misc/emacs-mime.texi | 7 +--
lisp/gnus/mml.el | 29 ++++++-------
lisp/mail/flow-fill.el | 93 ++++++++++++++++++++++++----------------
3 files changed, 69 insertions(+), 60 deletions(-)
diff --git a/doc/misc/emacs-mime.texi b/doc/misc/emacs-mime.texi
index ef7ea61..7621a9a 100644
--- a/doc/misc/emacs-mime.texi
+++ b/doc/misc/emacs-mime.texi
@@ -1087,13 +1087,10 @@ terminated by soft newline characters are filled together and wrapped
after the column decided by @code{fill-flowed-encode-column}.
Quotation marks (matching @samp{^>* ?}) are respected. The variable
controls how the text will look in a client that does not support
-flowed text, the default is to wrap after 66 characters. If hard
-newline characters are not present in the buffer, no flow encoding
-occurs.
+flowed text, the default is to wrap after 66 characters.
You can customize the value of the @code{mml-enable-flowed} variable
-to enable or disable the flowed encoding usage when newline
-characters are present in the buffer.
+to enable or disable the flowed encoding usage.
On decoding flowed text, lines with soft newline characters are filled
together and wrapped after the column decided by
diff --git a/lisp/gnus/mml.el b/lisp/gnus/mml.el
index e3bc393..2db39dc 100644
--- a/lisp/gnus/mml.el
+++ b/lisp/gnus/mml.el
@@ -691,23 +691,18 @@ type detected."
(t
;; Only perform format=flowed filling on text/plain
;; parts where there either isn't a format parameter
- ;; in the mml tag or it says "flowed" and there
- ;; actually are hard newlines in the text.
- (let (use-hard-newlines)
- (when (and mml-enable-flowed
- (string= type "text/plain")
- (not (string= (cdr (assq 'sign cont)) "pgp"))
- (or (null (assq 'format cont))
- (string= (cdr (assq 'format cont))
- "flowed"))
- (setq use-hard-newlines
- (text-property-any
- (point-min) (point-max) 'hard 't)))
- (fill-flowed-encode)
- ;; Indicate that `mml-insert-mime-headers' should
- ;; insert a "; format=flowed" string unless the
- ;; user has already specified it.
- (setq flowed (null (assq 'format cont)))))
+ ;; in the mml tag or it says "flowed".
+ (when (and mml-enable-flowed
+ (string= type "text/plain")
+ (not (string= (cdr (assq 'sign cont)) "pgp"))
+ (or (null (assq 'format cont))
+ (string= (cdr (assq 'format cont))
+ "flowed")))
+ (fill-flowed-encode)
+ ;; Indicate that `mml-insert-mime-headers' should
+ ;; insert a "; format=flowed" string unless the
+ ;; user has already specified it.
+ (setq flowed (null (assq 'format cont))))
;; Prefer `utf-8' for text/calendar parts.
(if (or charset
(not (string= type "text/calendar")))
diff --git a/lisp/mail/flow-fill.el b/lisp/mail/flow-fill.el
index 919490e..793088a 100644
--- a/lisp/mail/flow-fill.el
+++ b/lisp/mail/flow-fill.el
@@ -73,50 +73,67 @@ RFC 2646 suggests 66 characters for readability."
;;;###autoload
(defun fill-flowed-encode (&optional buffer)
(with-current-buffer (or buffer (current-buffer))
- ;; No point in doing this unless hard newlines is used.
- (when use-hard-newlines
- (let ((start (point-min)) end)
- ;; Go through each paragraph, filling it and adding SPC
- ;; as the last character on each line.
- (while (setq end (text-property-any start (point-max) 'hard 't))
- (save-restriction
- (narrow-to-region start end)
- (let ((fill-column (eval fill-flowed-encode-column t)))
- (fill-flowed-fill-buffer))
- (goto-char (point-min))
- (while (re-search-forward "\n" nil t)
- (replace-match " \n" t t))
- (goto-char (setq start (1+ (point-max)))))))
- t)))
-
-(defun fill-flowed-fill-buffer ()
- (let ((prefix nil)
- (prev-prefix nil)
- (start (point-min)))
- (goto-char (point-min))
- (while (not (eobp))
- (setq prefix (and (looking-at "[> ]+")
- (match-string 0)))
- (if (equal prefix prev-prefix)
- (forward-line 1)
+ (let ((fill-column (eval fill-flowed-encode-column t))
+ (start (point-min))
+ (end (point-max)))
+ ;; Only when we've been called to reflow a buffer that doesn't
+ ;; have any hard newlines:
+ (unless (text-property-any start end 'hard 't)
+ (save-excursion
+ ;; Harden between paras:
+ (goto-char start)
+ (while (re-search-forward "\n[> ]*\n" end t)
+ (set-hard-newline-properties
+ (match-beginning 0) (point))
+ (backward-char))
+ ;; Harden before a quote starts:
+ (goto-char start)
+ (while (re-search-forward "\n[^>][^\n]*\n>" end t)
+ (backward-char)
+ (set-hard-newline-properties
+ (1- (point)) (point)))
+ ;; Harden " $"
+ (goto-char start)
+ (while (search-forward " \n" end t)
+ (backward-char)
+ (backward-delete-char 2)
+ (set-hard-newline-properties
+ (point) (1+ (point))))
+ ;; Harden "^ "
+ (goto-char start)
+ (while (search-forward "\n " end t)
+ (set-hard-newline-properties
+ (- (point) 5) (- (point) 4)))))
+ ;; Go through each paragraph, filling it and adding SPC
+ ;; as the last character on each line.
+ (while (and (< start (point-max))
+ (setq end (or (text-property-any start (point-max) 'hard 't)
+ (point-max))))
(save-restriction
- (narrow-to-region start (point))
- (let ((fill-prefix prev-prefix))
- (fill-region (point-min) (point-max) t 'nosqueeze 'to-eop))
- (goto-char (point-max)))
- (setq prev-prefix prefix
- start (point))))
- (save-restriction
- (narrow-to-region start (point))
- (let ((fill-prefix prev-prefix))
- (fill-region (point-min) (point-max) t 'nosqueeze 'to-eop)))))
+ (narrow-to-region start end)
+ (let ((prefix
+ (concat "\n"
+ (or (and (looking-at ">[> ]*")
+ (match-string 0)) ""))))
+ (goto-char start)
+ (while (search-forward prefix nil t)
+ (replace-match " " t t))
+ (goto-char start)
+ (while (< (+ (point) fill-column) (point-max))
+ (let ((start (point)))
+ (forward-char fill-column)
+ (when (search-backward " " start t)
+ (forward-char)
+ (insert prefix)))))
+ (setq start (1+ (point-max))))))
+ t))
;;;###autoload
(defun fill-flowed (&optional buffer delete-space)
"Apply RFC2646 decoding to BUFFER.
If BUFFER is nil, default to the current buffer.
-If DELETE-SPACE, delete RFC2646 spaces padding at the end of
+If DELETE-SPACE, delete RFC3676 spaces padding at the end of
lines."
(with-current-buffer (or buffer (current-buffer))
(let ((fill-column (eval fill-flowed-display-column t)))
@@ -154,7 +171,7 @@ lines."
;; Delete the newline.
(when (eq (following-char) ?\s)
(delete-char 1))
- ;; Hack: Don't do the flowing on the signature line.
+ ;; As per RFC3767: Don't do the flowing on the signature line.
(when (and (not (looking-at "-- $"))
(eq (char-before (line-end-position)) ?\s))
(while (and (not (eobp))
--
2.39.2
^ permalink raw reply related [flat|nested] 10+ messages in thread
end of thread, other threads:[~2024-07-07 10:02 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-05-17 20:23 bug#71017: fill-flowed-encode Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors
2024-06-30 5:44 ` Stefan Kangas
2024-06-30 8:32 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors
2024-06-30 14:34 ` Stefan Kangas
2024-07-06 20:49 ` bug#71017: [PATCH] Flow single-paragraph messages Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors
2024-07-07 5:43 ` Eli Zaretskii
2024-07-07 8:34 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors
2024-07-07 9:04 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors
2024-07-07 9:16 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors
2024-07-07 10:02 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/emacs.git
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.