* bug#71017: fill-flowed-encode @ 2024-05-17 20:23 Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-06-30 5:44 ` Stefan Kangas 0 siblings, 1 reply; 10+ messages in thread From: Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-05-17 20:23 UTC (permalink / raw) To: 71017 Hi y'all. flow-fill.el.gz has a pair of functions, fill-flowed-encode and fill-flowed-fill-buffer (the latter is only called from the former). Here is a fixed version of the former that then also deprecates the latter (I have signed FSF copyright papers): (defun fill-flowed-encode (&optional buffer) (with-current-buffer (or buffer (current-buffer)) ;; No point in doing this unless hard newlines is used. (when use-hard-newlines (let ((start (point-min)) end) ;; Go through each paragraph, filling it and adding SPC ;; as the last character on each line. (while (and (< start (point-max)) (setq end (or (text-property-any start (point-max) 'hard 't) (point-max)))) (save-restriction (narrow-to-region start end) (let ((fill-column (eval fill-flowed-encode-column t)) (prefix (concat "\n" (or (and (looking-at ">[> ]*") (match-string 0)) "")))) (while (search-forward prefix nil t) (replace-match " " t t)) (goto-char start) (while (< (+ (point) fill-column) (point-max)) (forward-char fill-column) (search-backward " ") (forward-char) (insert prefix))) (setq start (1+ (point-max)))))) t))) This fixes two bugs when sending RFC 2646–formatted email. First, the old code didn't refill or encode the last paragraph at all unless there was at least one hard newline EOF. Second, the old code borked up code indented with tabs and spaces (iff that code had overly long lines), such as the Lisp code in this email. It could sometimes insert extra in the middle of such long lines. Here is an example of what it would do. It would turn this: (defun lorem (ipsum) (dolor sit amet) (consectetur adipiscing elit (sed do eiusmod tempor incididunt ut labore et dolore magna aliqua)) (ut enim ad minim veniam (quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat (duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur) excepteur sint occaecat cupidatat non proident (sunt in culpa qui officia deserunt mollit anim id est laborumd)))) into this: (defun lorem (ipsum) (dolor sit amet) (consectetur adipiscing elit (sed do eiusmod tempor incididunt ut labore et dolore magna aliqua)) (ut enim ad minim veniam (quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat (duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur) excepteur sint occaecat cupidatat non proident (sunt in culpa qui officia deserunt mollit anim id est laborumd)))) It was breaking lines awkwardly so when they're reconnected they have extra whitespace in the mkddle of lines. ^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#71017: fill-flowed-encode 2024-05-17 20:23 bug#71017: fill-flowed-encode Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-06-30 5:44 ` Stefan Kangas 2024-06-30 8:32 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors 0 siblings, 1 reply; 10+ messages in thread From: Stefan Kangas @ 2024-06-30 5:44 UTC (permalink / raw) To: Sandra Snan; +Cc: 71017 Sandra Snan <sandra.snan@idiomdrottning.org> writes: > Hi y'all. Hi Sandra, > flow-fill.el.gz has a pair of functions, fill-flowed-encode and > fill-flowed-fill-buffer (the latter is only called from the former). > > Here is a fixed version of the former that then also deprecates the latter (I > have signed FSF copyright papers): > > (defun fill-flowed-encode (&optional buffer) > (with-current-buffer (or buffer (current-buffer)) > ;; No point in doing this unless hard newlines is used. > (when use-hard-newlines > (let ((start (point-min)) end) > ;; Go through each paragraph, filling it and adding SPC > ;; as the last character on each line. > (while (and (< start (point-max)) > (setq end (or (text-property-any start (point-max) 'hard 't) > (point-max)))) > (save-restriction > (narrow-to-region start end) > (let ((fill-column (eval fill-flowed-encode-column t)) > (prefix > (concat "\n" > (or (and (looking-at ">[> ]*") > (match-string 0)) "")))) > (while (search-forward prefix nil t) > (replace-match " " t t)) > (goto-char start) > (while (< (+ (point) fill-column) (point-max)) > (forward-char fill-column) > (search-backward " ") > (forward-char) > (insert prefix))) > (setq start (1+ (point-max)))))) > t))) > > This fixes two bugs when sending RFC 2646–formatted email. > > First, the old code didn't refill or encode the last paragraph at all unless > there was at least one hard newline EOF. > > Second, the old code borked up code indented with tabs and spaces (iff that code > had overly long lines), such as the Lisp code in this email. It could sometimes > insert extra in the middle of such long lines. > > Here is an example of what it would do. It would turn this: > > (defun lorem (ipsum) > (dolor sit amet) > (consectetur adipiscing elit (sed do eiusmod tempor incididunt ut labore et > dolore magna aliqua)) > (ut enim ad minim veniam > (quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo > consequat > (duis aute irure dolor in reprehenderit in > voluptate velit esse cillum dolore > eu fugiat nulla pariatur) > excepteur sint occaecat cupidatat non proident > (sunt in culpa qui officia deserunt mollit anim id est laborumd)))) > > into this: > > (defun lorem (ipsum) > (dolor sit amet) > (consectetur adipiscing elit (sed do eiusmod tempor incididunt > ut labore et dolore magna aliqua)) > (ut enim ad minim veniam > (quis nostrud exercitation ullamco laboris nisi ut aliquip > ex ea commodo consequat > (duis aute irure dolor in reprehenderit in > voluptate velit esse cillum dolore > eu fugiat nulla pariatur) > excepteur sint occaecat cupidatat non proident > (sunt in culpa qui officia deserunt mollit anim id est > laborumd)))) > > It was breaking lines awkwardly so when they're reconnected they have extra > whitespace in the mkddle of lines. Could you please resend the above as patches formatted by git format-patch -1 ? It will make it easier for us to review and install the proposed changes. Thanks in advance. ^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#71017: fill-flowed-encode 2024-06-30 5:44 ` Stefan Kangas @ 2024-06-30 8:32 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-06-30 14:34 ` Stefan Kangas 0 siblings, 1 reply; 10+ messages in thread From: Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-06-30 8:32 UTC (permalink / raw) To: Stefan Kangas; +Cc: 71017 Stefan Kangas <stefankangas@gmail.com> writes: > Could you please resend the above as patches formatted by > > git format-patch -1 > > ? > I'm glad I'm gonna get a chance to resend because I've fixed some more bugs in the version I've been dogfooding since I first sent that so I wanna send an updated version that has that. Normally when I'm sending Emacs lisp patches, it's on packages and they're usually in git already. Here it's a core file so the question I've got is what is the git repo I should make the change in? Then I could git send-email to this In-Reply-To thread id. Stefan Kangas <stefankangas@gmail.com> writes: > Sandra Snan <sandra.snan@idiomdrottning.org> writes: > >> Hi y'all. > > Hi Sandra, > >> flow-fill.el.gz has a pair of functions, fill-flowed-encode and >> fill-flowed-fill-buffer (the latter is only called from the >> former). >> Here is a fixed version of the former that then also >> deprecates the latter (I have signed FSF copyright papers): >> >> (defun fill-flowed-encode (&optional buffer) (with-current-buffer >> (or buffer (current-buffer)) ;; No point in doing this unless >> hard newlines is used. (when use-hard-newlines (let ((start >> (point-min)) end) ;; Go through each paragraph, filling it and >> adding SPC ;; as the last character on each line. (while (and (< >> start (point-max)) (setq end (or (text-property-any start >> (point-max) 'hard 't) (point-max)))) (save-restriction >> (narrow-to-region start end) (let ((fill-column (eval >> fill-flowed-encode-column t)) (prefix (concat "\n" >> (or (and (looking-at ">[> ]*") (match-string 0)) "")))) >> (while (search-forward prefix nil t) (replace-match " " t t)) >> (goto-char start) (while (< (+ (point) fill-column) >> (point-max)) (forward-char fill-column) (search-backward " ") >> (forward-char) (insert prefix))) (setq start (1+ >> (point-max)))))) t))) >> This fixes two bugs when sending RFC >> 2646–formatted email. >> First, the old code didn't refill or >> encode the last paragraph at all unless there was at least one hard >> newline EOF. >> Second, the old code borked up code indented with >> tabs and spaces (iff that code had overly long lines), such as the >> Lisp code in this email. It could sometimes insert extra in the >> middle of such long lines. >> Here is an example of what it would >> do. It would turn this: >> (defun lorem (ipsum) (dolor sit amet) >> (consectetur adipiscing elit (sed do eiusmod tempor incididunt ut >> labore et dolore magna aliqua)) (ut enim ad minim veniam >> (quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea >> commodo consequat (duis aute irure dolor in reprehenderit >> in voluptate velit esse cillum dolore eu fugiat nulla >> pariatur) excepteur sint occaecat cupidatat non proident >> (sunt in culpa qui officia deserunt mollit anim id est >> laborumd)))) >> into this: >> (defun lorem (ipsum) (dolor sit >> amet) (consectetur adipiscing elit (sed do eiusmod tempor >> incididunt ut labore et dolore magna aliqua)) (ut enim ad minim >> veniam (quis nostrud exercitation ullamco laboris nisi ut >> aliquip ex ea commodo consequat (duis aute irure dolor in >> reprehenderit in voluptate velit esse cillum dolore eu >> fugiat nulla pariatur) excepteur sint occaecat cupidatat non >> proident (sunt in culpa qui officia deserunt mollit anim id >> est laborumd)))) >> It was breaking lines awkwardly so when >> they're reconnected they have extra whitespace in the mkddle of >> lines. > > Could you please resend the above as patches formatted by > > git format-patch -1 > > ? > > It will make it easier for us to review and install the proposed > changes. Thanks in advance. ^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#71017: fill-flowed-encode 2024-06-30 8:32 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-06-30 14:34 ` Stefan Kangas 2024-07-06 20:49 ` bug#71017: [PATCH] Flow single-paragraph messages Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors 0 siblings, 1 reply; 10+ messages in thread From: Stefan Kangas @ 2024-06-30 14:34 UTC (permalink / raw) To: Sandra Snan; +Cc: 71017 Sandra Snan <sandra.snan@idiomdrottning.org> writes: > I'm glad I'm gonna get a chance to resend because I've fixed some > more bugs in the version I've been dogfooding since I first sent > that so I wanna send an updated version that has that. Great, thanks. > Normally when I'm sending Emacs lisp patches, it's on packages and > they're usually in git already. Here it's a core file so the > question I've got is what is the git repo I should make the change > in? Then I could git send-email to this In-Reply-To thread id. https://git.savannah.gnu.org/cgit/emacs.git Please have a look at the CONTRIBUTE file in the Emacs source distribution. https://git.savannah.gnu.org/cgit/emacs.git/tree/CONTRIBUTE ^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#71017: [PATCH] Flow single-paragraph messages 2024-06-30 14:34 ` Stefan Kangas @ 2024-07-06 20:49 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-07-07 5:43 ` Eli Zaretskii 0 siblings, 1 reply; 10+ messages in thread From: Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-07-06 20:49 UTC (permalink / raw) To: 71017; +Cc: Sandra Snan This fixes two bugs when sending RFC 2646–formatted email. First, the old code didn't refill or encode the last paragraph at all unless there was at least one hard newline EOF. This was a bee to track down because there were two separate issues at play. One was a a bug in flow-fill.el where every paragraph except the last paragraph was reflowed, but the last paragraph would stay hardwrapped. Manually placing a hard newline at the end of the file was a workaround but I don't always remember to do that. I managed to fix that bug a few months ago. Second, the old code borked up code indented with tabs and spaces (iff that code had overly long lines), such as Lisp code. It could sometimes insert extra whitespace in the middle of such long lines. I fixed that bug shortly after the first one. But for months dogfooding those two changes, sometimes a hardwrapped email would still be sent. I finally managed to debug and figure it out and it took all day. Turns out mml just plain didn't call the fill-flowed-encode function if the message doesn't have any hard newlines (newlines with the hard text property). Well, of course a single-paragraph email isn't gonna have any hard newlines! But it still needs reflowing! So I've now changed that and updated the documentation to match those news semantics. I went all the way, but a possible compromise might be to not-flow a message that has \n\n but no hard text props, since that's a sign that something is wrong. Since the use-hard-newlines variable is buffer local and all this reflowing is being done in a temp buffer, that variable is more than useless so I've removed references to it. --- doc/misc/emacs-mime.texi | 7 ++--- lisp/gnus/mml.el | 29 ++++++++---------- lisp/mail/flow-fill.el | 65 +++++++++++++++++----------------------- 3 files changed, 41 insertions(+), 60 deletions(-) diff --git a/doc/misc/emacs-mime.texi b/doc/misc/emacs-mime.texi index ef7ea61..7621a9a 100644 --- a/doc/misc/emacs-mime.texi +++ b/doc/misc/emacs-mime.texi @@ -1087,13 +1087,10 @@ terminated by soft newline characters are filled together and wrapped after the column decided by @code{fill-flowed-encode-column}. Quotation marks (matching @samp{^>* ?}) are respected. The variable controls how the text will look in a client that does not support -flowed text, the default is to wrap after 66 characters. If hard -newline characters are not present in the buffer, no flow encoding -occurs. +flowed text, the default is to wrap after 66 characters. You can customize the value of the @code{mml-enable-flowed} variable -to enable or disable the flowed encoding usage when newline -characters are present in the buffer. +to enable or disable the flowed encoding usage. On decoding flowed text, lines with soft newline characters are filled together and wrapped after the column decided by diff --git a/lisp/gnus/mml.el b/lisp/gnus/mml.el index e3bc393..2db39dc 100644 --- a/lisp/gnus/mml.el +++ b/lisp/gnus/mml.el @@ -691,23 +691,18 @@ type detected." (t ;; Only perform format=flowed filling on text/plain ;; parts where there either isn't a format parameter - ;; in the mml tag or it says "flowed" and there - ;; actually are hard newlines in the text. - (let (use-hard-newlines) - (when (and mml-enable-flowed - (string= type "text/plain") - (not (string= (cdr (assq 'sign cont)) "pgp")) - (or (null (assq 'format cont)) - (string= (cdr (assq 'format cont)) - "flowed")) - (setq use-hard-newlines - (text-property-any - (point-min) (point-max) 'hard 't))) - (fill-flowed-encode) - ;; Indicate that `mml-insert-mime-headers' should - ;; insert a "; format=flowed" string unless the - ;; user has already specified it. - (setq flowed (null (assq 'format cont))))) + ;; in the mml tag or it says "flowed". + (when (and mml-enable-flowed + (string= type "text/plain") + (not (string= (cdr (assq 'sign cont)) "pgp")) + (or (null (assq 'format cont)) + (string= (cdr (assq 'format cont)) + "flowed"))) + (fill-flowed-encode) + ;; Indicate that `mml-insert-mime-headers' should + ;; insert a "; format=flowed" string unless the + ;; user has already specified it. + (setq flowed (null (assq 'format cont)))) ;; Prefer `utf-8' for text/calendar parts. (if (or charset (not (string= type "text/calendar"))) diff --git a/lisp/mail/flow-fill.el b/lisp/mail/flow-fill.el index 919490e..5c9ae21 100644 --- a/lisp/mail/flow-fill.el +++ b/lisp/mail/flow-fill.el @@ -73,50 +73,39 @@ RFC 2646 suggests 66 characters for readability." ;;;###autoload (defun fill-flowed-encode (&optional buffer) (with-current-buffer (or buffer (current-buffer)) - ;; No point in doing this unless hard newlines is used. - (when use-hard-newlines - (let ((start (point-min)) end) - ;; Go through each paragraph, filling it and adding SPC - ;; as the last character on each line. - (while (setq end (text-property-any start (point-max) 'hard 't)) - (save-restriction - (narrow-to-region start end) - (let ((fill-column (eval fill-flowed-encode-column t))) - (fill-flowed-fill-buffer)) - (goto-char (point-min)) - (while (re-search-forward "\n" nil t) - (replace-match " \n" t t)) - (goto-char (setq start (1+ (point-max))))))) - t))) - -(defun fill-flowed-fill-buffer () - (let ((prefix nil) - (prev-prefix nil) - (start (point-min))) - (goto-char (point-min)) - (while (not (eobp)) - (setq prefix (and (looking-at "[> ]+") - (match-string 0))) - (if (equal prefix prev-prefix) - (forward-line 1) + (let ((fill-column (eval fill-flowed-encode-column t)) + (start (point-min)) + end) + ;; Go through each paragraph, filling it and adding SPC + ;; as the last character on each line. + (while (and (< start (point-max)) + (setq end (or (text-property-any start (point-max) 'hard 't) + (point-max)))) (save-restriction - (narrow-to-region start (point)) - (let ((fill-prefix prev-prefix)) - (fill-region (point-min) (point-max) t 'nosqueeze 'to-eop)) - (goto-char (point-max))) - (setq prev-prefix prefix - start (point)))) - (save-restriction - (narrow-to-region start (point)) - (let ((fill-prefix prev-prefix)) - (fill-region (point-min) (point-max) t 'nosqueeze 'to-eop))))) + (narrow-to-region start end) + (let ((prefix + (concat "\n" + (or (and (looking-at ">[> ]*") + (match-string 0)) "")))) + (goto-char start) + (while (search-forward prefix nil t) + (replace-match " " t t)) + (goto-char start) + (while (< (+ (point) fill-column) (point-max)) + (let ((start (point))) + (forward-char fill-column) + (when (search-backward " " start t) + (forward-char) + (insert prefix))))) + (setq start (1+ (point-max)))))) + t)) ;;;###autoload (defun fill-flowed (&optional buffer delete-space) "Apply RFC2646 decoding to BUFFER. If BUFFER is nil, default to the current buffer. -If DELETE-SPACE, delete RFC2646 spaces padding at the end of +If DELETE-SPACE, delete RFC3676 spaces padding at the end of lines." (with-current-buffer (or buffer (current-buffer)) (let ((fill-column (eval fill-flowed-display-column t))) @@ -154,7 +143,7 @@ lines." ;; Delete the newline. (when (eq (following-char) ?\s) (delete-char 1)) - ;; Hack: Don't do the flowing on the signature line. + ;; As per RFC3767: Don't do the flowing on the signature line. (when (and (not (looking-at "-- $")) (eq (char-before (line-end-position)) ?\s)) (while (and (not (eobp)) -- 2.39.2 ^ permalink raw reply related [flat|nested] 10+ messages in thread
* bug#71017: [PATCH] Flow single-paragraph messages 2024-07-06 20:49 ` bug#71017: [PATCH] Flow single-paragraph messages Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-07-07 5:43 ` Eli Zaretskii 2024-07-07 8:34 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors 0 siblings, 1 reply; 10+ messages in thread From: Eli Zaretskii @ 2024-07-07 5:43 UTC (permalink / raw) To: Sandra Snan, Eric Abrahamsen; +Cc: 71017 > Cc: Sandra Snan <sandra.snan@idiomdrottning.org> > Date: Sat, 6 Jul 2024 22:49:50 +0200 > From: Sandra Snan via "Bug reports for GNU Emacs, > the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org> > > This fixes two bugs when sending RFC 2646–formatted email. Thanks. (I also see RFC 3676 mentioned -- what is this about?) > First, the old code didn't refill or encode the last paragraph at all > unless there was at least one hard newline EOF. Isn't this the documented behavior? > But for months dogfooding those two changes, sometimes a hardwrapped > email would still be sent. I finally managed to debug and figure it out > and it took all day. Turns out mml just plain didn't call the > fill-flowed-encode function if the message doesn't have any hard > newlines (newlines with the hard text property). Well, of course a > single-paragraph email isn't gonna have any hard newlines! But it still > needs reflowing! > > So I've now changed that and updated the documentation to match those > news semantics. I went all the way, but a possible compromise might be > to not-flow a message that has \n\n but no hard text props, since that's > a sign that something is wrong. The change seems to be an incompatible behavior change, so I wonder whether we'd need some way for users to get back old behavior. Eric, WDYT? I'll leave it to Eric to comment on the code changes. ^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#71017: [PATCH] Flow single-paragraph messages 2024-07-07 5:43 ` Eli Zaretskii @ 2024-07-07 8:34 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-07-07 9:04 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors 0 siblings, 1 reply; 10+ messages in thread From: Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-07-07 8:34 UTC (permalink / raw) To: Eli Zaretskii, Eric Abrahamsen; +Cc: 71017 Eli Zaretskii <eliz@gnu.org> writes: > Thanks. (I also see RFC 3676 mentioned -- what is this about?) I mentioned RFC 3676 twice for different reasons. First, it's introduced the delsp parameter which wasn't in 2646 so code related to that parameter shouldn't talk about 2646 but rather 3676. Second, I saw a comment referring to not tampering with the sig line as a hack. I happened to have the section of RFC 3676 open that mandates that behavior so I changed the comment, however, this might be in RFC 2646 also, I don't know that, so in that case that comment might better change to refer to 2646 instead. >> First, the old code didn't refill or encode the last paragraph >> at all unless there was at least one hard newline EOF. > > Isn't this the documented behavior? Re multi-paragraph messages: No, it's not the documented behavior, it was an unrelated bug in fill-flowed-encode. It would refill all the other paragraphs, separated by hard newlines, just not the last one. That was a bug and broke documented behavior. I fixed that bug + another unrelated reflow bug. That was in fill-flowed-encode and that bugfix doesn't rely on the change in mml. With my fix in fill-flowed-encode, multi-paragraph-messages started working fine. Re single-paragraph messages: However, according to the old documented behavior, a message that contained no hard newlines should not be refilled. This documented behavior in mml-generate-mime-1 meant that single-paragraph messages would not be filled even with the fill-flowed-encode bug fixed. That is an unintended bad consequence of the documented behavior, a "bug in the design". I did change that but I updated the documentation to match. > The change seems to be an incompatible behavior change, so I > wonder whether we'd need some way for users to get back old > behavior. There is still the (neglected) defcustom mml-enable-flowed which now becomes more relevant since it's a way to turn off all this meddling and reflowing in the first place. The old documented behavior was bugged-by-design. It's not right that single paragraph messages are hardwrapped and not reflowed. In my day-to-day I write many messages in Emacs that I later see in threads in another MUA (Delta Chat) and these messages stand out in a way that something is wrong with them. However, one intent behind the old behavior was, in spirit, good: It'd be good to detect whether or not users with mml-enable-flowed on have remembered to also turn on use-hard-newlines, which is important for users with that on to do, especially since mml-enable-flowed defaults to t. The old attempt at doing that was flawed since it only worked reliably for multi-paragraph messages. Unfortunately there's currently no way to detect in a single-paragraph message whether or not use-hard-newlines have been turned on, since the variable it sets is buffer local. (One extremely klugy workaround would be to change the message-send-and-exit command to check whether use-hard-newlines is on and if it is, add an extra hard newline EOT just for detecting this. Not super into that solution so hopefully there are other ways.) With this patch, the defcustom mml-enable-flowed becomes _the_ setting for this, which does match a lot of documentation on the books. Perhaps it shall no longer default to t though since it completely borks messages up if it's t but use-hard-newlines are not on! So here we are: The new behavior has a problem: messages will get reflowed if mml-enable-flowed is t (the default!) even when use-hard-newlines is off, meaning that even separate "\n\n" paragraphs will get flowed together which is not what people want. Use-hard-newlines should be mandatory whenever mml-enable-flowed is on. The old behavior is not OK since single-paragraph messages will get messed up, hardwrapped even when those newlines were advertised as "soft", or not softwrapped even when the paragraph consists of just one single super long unbroken line. I also have sent a patch to the messages-are-flowing project highlighting the importance of this variable: https://github.com/legoscia/messages-are-flowing/pull/15/commits/ae432723c2565ceced5d01d9aa2d314bd42aaa3c So how about this idea: If mml-enable-flowed is on but fill-flowed-encode is asked to flow a message that doesn't have any hard newlines, assume Markdown semantics, i.e. special treatment for "\n\n+", " $", and "^ ". I'll see if I have time to implement that this morning. I think I'd place that change in fill-flowed-encode. That wouldn't affect people with mml-enable-flowed off, or people with both mml-enable-flowed and use-hard-newlines on, it'd just be a sort of DWIM fallback based on the guess that markdown semantics are somewhat widely known or expected in 2024, to prevent separate paragraphs to be flowed together for people with "incorrect" settings. Again, if there were a cross-buffer way to reliably detect whether use-hard-newlines is on, that dwimmy fallback wouldn't be needed. I thought about whether it'd have been better if it was instead soft newlines that were marked with a text property, not hard ones, but in the end that wouldn't properly softwrap messages with just one single overly long line. ^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#71017: [PATCH] Flow single-paragraph messages 2024-07-07 8:34 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-07-07 9:04 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-07-07 9:16 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors 0 siblings, 1 reply; 10+ messages in thread From: Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-07-07 9:04 UTC (permalink / raw) To: 71017; +Cc: Sandra Snan See discussion in #71017. In short: if mml-flowed-enable is nil: no change to behavior if mml-flowed-enable is t and use-hard-newlines is on: several bugs fixed, some (multi-paragraph messages) of which were counter to documentation and some (single-paragraph messages) that were along documentation lines but which messed up messages royally. if mml-flowed-enable is t but use-hard-newlines was accidentally forgotten by user: new behavior; reflow but don't reflow together separate paragraphs nor reflow across lines that markdown would've considered hard. --- doc/misc/emacs-mime.texi | 7 +--- lisp/gnus/mml.el | 29 ++++++-------- lisp/mail/flow-fill.el | 86 ++++++++++++++++++++++------------------ 3 files changed, 62 insertions(+), 60 deletions(-) diff --git a/doc/misc/emacs-mime.texi b/doc/misc/emacs-mime.texi index ef7ea61..7621a9a 100644 --- a/doc/misc/emacs-mime.texi +++ b/doc/misc/emacs-mime.texi @@ -1087,13 +1087,10 @@ terminated by soft newline characters are filled together and wrapped after the column decided by @code{fill-flowed-encode-column}. Quotation marks (matching @samp{^>* ?}) are respected. The variable controls how the text will look in a client that does not support -flowed text, the default is to wrap after 66 characters. If hard -newline characters are not present in the buffer, no flow encoding -occurs. +flowed text, the default is to wrap after 66 characters. You can customize the value of the @code{mml-enable-flowed} variable -to enable or disable the flowed encoding usage when newline -characters are present in the buffer. +to enable or disable the flowed encoding usage. On decoding flowed text, lines with soft newline characters are filled together and wrapped after the column decided by diff --git a/lisp/gnus/mml.el b/lisp/gnus/mml.el index e3bc393..2db39dc 100644 --- a/lisp/gnus/mml.el +++ b/lisp/gnus/mml.el @@ -691,23 +691,18 @@ type detected." (t ;; Only perform format=flowed filling on text/plain ;; parts where there either isn't a format parameter - ;; in the mml tag or it says "flowed" and there - ;; actually are hard newlines in the text. - (let (use-hard-newlines) - (when (and mml-enable-flowed - (string= type "text/plain") - (not (string= (cdr (assq 'sign cont)) "pgp")) - (or (null (assq 'format cont)) - (string= (cdr (assq 'format cont)) - "flowed")) - (setq use-hard-newlines - (text-property-any - (point-min) (point-max) 'hard 't))) - (fill-flowed-encode) - ;; Indicate that `mml-insert-mime-headers' should - ;; insert a "; format=flowed" string unless the - ;; user has already specified it. - (setq flowed (null (assq 'format cont))))) + ;; in the mml tag or it says "flowed". + (when (and mml-enable-flowed + (string= type "text/plain") + (not (string= (cdr (assq 'sign cont)) "pgp")) + (or (null (assq 'format cont)) + (string= (cdr (assq 'format cont)) + "flowed"))) + (fill-flowed-encode) + ;; Indicate that `mml-insert-mime-headers' should + ;; insert a "; format=flowed" string unless the + ;; user has already specified it. + (setq flowed (null (assq 'format cont)))) ;; Prefer `utf-8' for text/calendar parts. (if (or charset (not (string= type "text/calendar"))) diff --git a/lisp/mail/flow-fill.el b/lisp/mail/flow-fill.el index 919490e..932aa8f 100644 --- a/lisp/mail/flow-fill.el +++ b/lisp/mail/flow-fill.el @@ -73,50 +73,60 @@ RFC 2646 suggests 66 characters for readability." ;;;###autoload (defun fill-flowed-encode (&optional buffer) (with-current-buffer (or buffer (current-buffer)) - ;; No point in doing this unless hard newlines is used. - (when use-hard-newlines - (let ((start (point-min)) end) - ;; Go through each paragraph, filling it and adding SPC - ;; as the last character on each line. - (while (setq end (text-property-any start (point-max) 'hard 't)) - (save-restriction - (narrow-to-region start end) - (let ((fill-column (eval fill-flowed-encode-column t))) - (fill-flowed-fill-buffer)) - (goto-char (point-min)) - (while (re-search-forward "\n" nil t) - (replace-match " \n" t t)) - (goto-char (setq start (1+ (point-max))))))) - t))) - -(defun fill-flowed-fill-buffer () - (let ((prefix nil) - (prev-prefix nil) - (start (point-min))) - (goto-char (point-min)) - (while (not (eobp)) - (setq prefix (and (looking-at "[> ]+") - (match-string 0))) - (if (equal prefix prev-prefix) - (forward-line 1) + (let ((fill-column (eval fill-flowed-encode-column t)) + (start (point-min)) + (end (point-max))) + ;; Only when we've been called to reflow a buffer that doesn't + ;; have any hard newlines: + (unless (text-property-any start end 'hard 't) + (save-excursion + ;; Harden between paras: + (goto-char start) + (while (search-forward "\n\n" end t) + (set-hard-newline-properties + (- (point) 2) (point))) + ;; Harden " $" + (goto-char start) + (while (search-forward " \n" end t) + (backward-char) + (backward-delete-char 2) + (set-hard-newline-properties + (point) (1+ (point)))) + ;; Harden "^ " + (goto-char start) + (while (search-forward "\n " end t) + (set-hard-newline-properties + (- (point) 5) (- (point) 4))))) + ;; Go through each paragraph, filling it and adding SPC + ;; as the last character on each line. + (while (and (< start (point-max)) + (setq end (or (text-property-any start (point-max) 'hard 't) + (point-max)))) (save-restriction - (narrow-to-region start (point)) - (let ((fill-prefix prev-prefix)) - (fill-region (point-min) (point-max) t 'nosqueeze 'to-eop)) - (goto-char (point-max))) - (setq prev-prefix prefix - start (point)))) - (save-restriction - (narrow-to-region start (point)) - (let ((fill-prefix prev-prefix)) - (fill-region (point-min) (point-max) t 'nosqueeze 'to-eop))))) + (narrow-to-region start end) + (let ((prefix + (concat "\n" + (or (and (looking-at ">[> ]*") + (match-string 0)) "")))) + (goto-char start) + (while (search-forward prefix nil t) + (replace-match " " t t)) + (goto-char start) + (while (< (+ (point) fill-column) (point-max)) + (let ((start (point))) + (forward-char fill-column) + (when (search-backward " " start t) + (forward-char) + (insert prefix))))) + (setq start (1+ (point-max)))))) + t)) ;;;###autoload (defun fill-flowed (&optional buffer delete-space) "Apply RFC2646 decoding to BUFFER. If BUFFER is nil, default to the current buffer. -If DELETE-SPACE, delete RFC2646 spaces padding at the end of +If DELETE-SPACE, delete RFC3676 spaces padding at the end of lines." (with-current-buffer (or buffer (current-buffer)) (let ((fill-column (eval fill-flowed-display-column t))) @@ -154,7 +164,7 @@ lines." ;; Delete the newline. (when (eq (following-char) ?\s) (delete-char 1)) - ;; Hack: Don't do the flowing on the signature line. + ;; As per RFC3767: Don't do the flowing on the signature line. (when (and (not (looking-at "-- $")) (eq (char-before (line-end-position)) ?\s)) (while (and (not (eobp)) -- 2.39.2 ^ permalink raw reply related [flat|nested] 10+ messages in thread
* bug#71017: [PATCH] Flow single-paragraph messages 2024-07-07 9:04 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-07-07 9:16 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-07-07 10:02 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors 0 siblings, 1 reply; 10+ messages in thread From: Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-07-07 9:16 UTC (permalink / raw) To: 71017 Eww, I forgot about quote handling for this autoharden fallback. ^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#71017: [PATCH] Flow single-paragraph messages 2024-07-07 9:16 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-07-07 10:02 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors 0 siblings, 0 replies; 10+ messages in thread From: Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-07-07 10:02 UTC (permalink / raw) To: 71017; +Cc: Sandra Snan See discussion in #71017. In short: if mml-flowed-enable is nil: no change to behavior if mml-flowed-enable is t and use-hard-newlines is on: several bugs fixed, some (multi-paragraph messages) of which were counter to documentation and some (single-paragraph messages) that were along documentation lines but which messed up messages royally. if mml-flowed-enable is t but use-hard-newlines was accidentally forgotten by user: new behavior; reflow but don't reflow together separate paragraphs nor reflow across lines that markdown would've considered hard. --- doc/misc/emacs-mime.texi | 7 +-- lisp/gnus/mml.el | 29 ++++++------- lisp/mail/flow-fill.el | 93 ++++++++++++++++++++++++---------------- 3 files changed, 69 insertions(+), 60 deletions(-) diff --git a/doc/misc/emacs-mime.texi b/doc/misc/emacs-mime.texi index ef7ea61..7621a9a 100644 --- a/doc/misc/emacs-mime.texi +++ b/doc/misc/emacs-mime.texi @@ -1087,13 +1087,10 @@ terminated by soft newline characters are filled together and wrapped after the column decided by @code{fill-flowed-encode-column}. Quotation marks (matching @samp{^>* ?}) are respected. The variable controls how the text will look in a client that does not support -flowed text, the default is to wrap after 66 characters. If hard -newline characters are not present in the buffer, no flow encoding -occurs. +flowed text, the default is to wrap after 66 characters. You can customize the value of the @code{mml-enable-flowed} variable -to enable or disable the flowed encoding usage when newline -characters are present in the buffer. +to enable or disable the flowed encoding usage. On decoding flowed text, lines with soft newline characters are filled together and wrapped after the column decided by diff --git a/lisp/gnus/mml.el b/lisp/gnus/mml.el index e3bc393..2db39dc 100644 --- a/lisp/gnus/mml.el +++ b/lisp/gnus/mml.el @@ -691,23 +691,18 @@ type detected." (t ;; Only perform format=flowed filling on text/plain ;; parts where there either isn't a format parameter - ;; in the mml tag or it says "flowed" and there - ;; actually are hard newlines in the text. - (let (use-hard-newlines) - (when (and mml-enable-flowed - (string= type "text/plain") - (not (string= (cdr (assq 'sign cont)) "pgp")) - (or (null (assq 'format cont)) - (string= (cdr (assq 'format cont)) - "flowed")) - (setq use-hard-newlines - (text-property-any - (point-min) (point-max) 'hard 't))) - (fill-flowed-encode) - ;; Indicate that `mml-insert-mime-headers' should - ;; insert a "; format=flowed" string unless the - ;; user has already specified it. - (setq flowed (null (assq 'format cont))))) + ;; in the mml tag or it says "flowed". + (when (and mml-enable-flowed + (string= type "text/plain") + (not (string= (cdr (assq 'sign cont)) "pgp")) + (or (null (assq 'format cont)) + (string= (cdr (assq 'format cont)) + "flowed"))) + (fill-flowed-encode) + ;; Indicate that `mml-insert-mime-headers' should + ;; insert a "; format=flowed" string unless the + ;; user has already specified it. + (setq flowed (null (assq 'format cont)))) ;; Prefer `utf-8' for text/calendar parts. (if (or charset (not (string= type "text/calendar"))) diff --git a/lisp/mail/flow-fill.el b/lisp/mail/flow-fill.el index 919490e..793088a 100644 --- a/lisp/mail/flow-fill.el +++ b/lisp/mail/flow-fill.el @@ -73,50 +73,67 @@ RFC 2646 suggests 66 characters for readability." ;;;###autoload (defun fill-flowed-encode (&optional buffer) (with-current-buffer (or buffer (current-buffer)) - ;; No point in doing this unless hard newlines is used. - (when use-hard-newlines - (let ((start (point-min)) end) - ;; Go through each paragraph, filling it and adding SPC - ;; as the last character on each line. - (while (setq end (text-property-any start (point-max) 'hard 't)) - (save-restriction - (narrow-to-region start end) - (let ((fill-column (eval fill-flowed-encode-column t))) - (fill-flowed-fill-buffer)) - (goto-char (point-min)) - (while (re-search-forward "\n" nil t) - (replace-match " \n" t t)) - (goto-char (setq start (1+ (point-max))))))) - t))) - -(defun fill-flowed-fill-buffer () - (let ((prefix nil) - (prev-prefix nil) - (start (point-min))) - (goto-char (point-min)) - (while (not (eobp)) - (setq prefix (and (looking-at "[> ]+") - (match-string 0))) - (if (equal prefix prev-prefix) - (forward-line 1) + (let ((fill-column (eval fill-flowed-encode-column t)) + (start (point-min)) + (end (point-max))) + ;; Only when we've been called to reflow a buffer that doesn't + ;; have any hard newlines: + (unless (text-property-any start end 'hard 't) + (save-excursion + ;; Harden between paras: + (goto-char start) + (while (re-search-forward "\n[> ]*\n" end t) + (set-hard-newline-properties + (match-beginning 0) (point)) + (backward-char)) + ;; Harden before a quote starts: + (goto-char start) + (while (re-search-forward "\n[^>][^\n]*\n>" end t) + (backward-char) + (set-hard-newline-properties + (1- (point)) (point))) + ;; Harden " $" + (goto-char start) + (while (search-forward " \n" end t) + (backward-char) + (backward-delete-char 2) + (set-hard-newline-properties + (point) (1+ (point)))) + ;; Harden "^ " + (goto-char start) + (while (search-forward "\n " end t) + (set-hard-newline-properties + (- (point) 5) (- (point) 4))))) + ;; Go through each paragraph, filling it and adding SPC + ;; as the last character on each line. + (while (and (< start (point-max)) + (setq end (or (text-property-any start (point-max) 'hard 't) + (point-max)))) (save-restriction - (narrow-to-region start (point)) - (let ((fill-prefix prev-prefix)) - (fill-region (point-min) (point-max) t 'nosqueeze 'to-eop)) - (goto-char (point-max))) - (setq prev-prefix prefix - start (point)))) - (save-restriction - (narrow-to-region start (point)) - (let ((fill-prefix prev-prefix)) - (fill-region (point-min) (point-max) t 'nosqueeze 'to-eop))))) + (narrow-to-region start end) + (let ((prefix + (concat "\n" + (or (and (looking-at ">[> ]*") + (match-string 0)) "")))) + (goto-char start) + (while (search-forward prefix nil t) + (replace-match " " t t)) + (goto-char start) + (while (< (+ (point) fill-column) (point-max)) + (let ((start (point))) + (forward-char fill-column) + (when (search-backward " " start t) + (forward-char) + (insert prefix))))) + (setq start (1+ (point-max)))))) + t)) ;;;###autoload (defun fill-flowed (&optional buffer delete-space) "Apply RFC2646 decoding to BUFFER. If BUFFER is nil, default to the current buffer. -If DELETE-SPACE, delete RFC2646 spaces padding at the end of +If DELETE-SPACE, delete RFC3676 spaces padding at the end of lines." (with-current-buffer (or buffer (current-buffer)) (let ((fill-column (eval fill-flowed-display-column t))) @@ -154,7 +171,7 @@ lines." ;; Delete the newline. (when (eq (following-char) ?\s) (delete-char 1)) - ;; Hack: Don't do the flowing on the signature line. + ;; As per RFC3767: Don't do the flowing on the signature line. (when (and (not (looking-at "-- $")) (eq (char-before (line-end-position)) ?\s)) (while (and (not (eobp)) -- 2.39.2 ^ permalink raw reply related [flat|nested] 10+ messages in thread
end of thread, other threads:[~2024-07-07 10:02 UTC | newest] Thread overview: 10+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-05-17 20:23 bug#71017: fill-flowed-encode Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-06-30 5:44 ` Stefan Kangas 2024-06-30 8:32 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-06-30 14:34 ` Stefan Kangas 2024-07-06 20:49 ` bug#71017: [PATCH] Flow single-paragraph messages Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-07-07 5:43 ` Eli Zaretskii 2024-07-07 8:34 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-07-07 9:04 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-07-07 9:16 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-07-07 10:02 ` Sandra Snan via Bug reports for GNU Emacs, the Swiss army knife of text editors
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).