bug#46507: 26.1; bold attribute copied into enriched-mode text is not saved

bug#46507: 26.1; bold attribute copied into enriched-mode text is not saved

[Dan Hitt]

[-- Attachment #1: Type: text/plain, Size: 715 bytes --]

I am using emacs 26.1 on debian 10.3.

The bug is that when you copy bolded-colored text from an eshell buffer
into an enriched-text buffer, the bold shows after the copy --- but if you
then close the enriched-text file, and re-open it, the bolding is gone.

Steps to reproduce:

(1) emacs -Q
(2) M-x eshell
(3) cd /etc
(4) ls
(5) control-X 2
(6) scroll up a dozen lines using control P; then type escape-w to capture
the text
(7) find-file /tmp/blah.txt
(8) yank in the captured text (control-y)
(9) M-x enriched-text  (so make /tmp/blah.txt into an enriched text file)
(10) kill the buffer (control-x k)
(11) reopen the file (find-file)
(12) observe that although the colors are still there, the bolding is gone.

[-- Attachment #2: Type: text/html, Size: 976 bytes --]

bug#46507: 26.1; bold attribute copied into enriched-mode text is not saved

[Tomas Nordin]

Hello Dan

Dan Hitt <dan.hitt@gmail.com> writes:

> I am using emacs 26.1 on debian 10.3.
>
> The bug is that when you copy bolded-colored text from an eshell buffer
> into an enriched-text buffer, the bold shows after the copy --- but if you
> then close the enriched-text file, and re-open it, the bolding is gone.

I have no experience with Enriched mode but I read this from [1]:

    Enriched mode is typically used with Text mode (see Text Mode). It
    is not compatible with Font Lock mode, which is used by many major
    modes, including most programming language modes, for syntax
    highlighting (see Font Lock). Unlike Enriched mode, Font Lock mode
    assigns text properties automatically, based on the current buffer
    contents; those properties are not saved to disk.

So if the "bolded-colored" text is a font lock thing, it will not be
saved to disc. Could that be it?

[1] https://www.gnu.org/software/emacs/manual/html_node/emacs/Enriched-Text.html#Enriched-Text-1

--
Tomas

bug#46507: 26.1; bold attribute copied into enriched-mode text is not saved

[Dan Hitt]

[-- Attachment #1: Type: text/plain, Size: 2226 bytes --]

On Sun, Mar 7, 2021 at 4:46 AM Tomas Nordin <tomasn@posteo.net> wrote:

> Hello Dan
>
> Dan Hitt <dan.hitt@gmail.com> writes:
>
> > I am using emacs 26.1 on debian 10.3.
> >
> > The bug is that when you copy bolded-colored text from an eshell buffer
> > into an enriched-text buffer, the bold shows after the copy --- but if
> you
> > then close the enriched-text file, and re-open it, the bolding is gone.
>
> I have no experience with Enriched mode but I read this from [1]:
>
>     Enriched mode is typically used with Text mode (see Text Mode). It
>     is not compatible with Font Lock mode, which is used by many major
>     modes, including most programming language modes, for syntax
>     highlighting (see Font Lock). Unlike Enriched mode, Font Lock mode
>     assigns text properties automatically, based on the current buffer
>     contents; those properties are not saved to disk.
>
> So if the "bolded-colored" text is a font lock thing, it will not be
> saved to disc. Could that be it?
>
> [1]
> https://www.gnu.org/software/emacs/manual/html_node/emacs/Enriched-Text.html#Enriched-Text-1
>
> --
> Tomas
>

Thanks Thomas for your mail.

I think something like that might explain the behavior: i believe that a
command in eshell (or a shell buffer also) gets bolded as a result of
fontlock.

But when the bolded text is copied to the enriched buffer, it stays
bolded, even though there's no font-lock in enriched text.

Presumably the bolded state of the text is visible to elisp---it must just
be some kind of text property.

So, as part of the saving process, enriched-text-mode could certainly
traverse the buffer and note all the places where text is bold and save
those as well as being bold.

It already does something like that because when you manually bold text
(via M-o b), that bolded text is remembered.

And, as an analogy: when the text changes color in eshell --- for example,
the response to a command gets colored orange, that color change is also
due to font-lock.

When the colored text is copied into an enriched-mode buffer, the text
stays orange.  And when the enriched buffer is saved, the orange color gets
saved with it.

So sometimes, at least, enriched saves on a WYSIWYG basis.

dan

[-- Attachment #2: Type: text/html, Size: 3095 bytes --]

bug#46507: 26.1; bold attribute copied into enriched-mode text is not saved

[Tomas Nordin]

Dan Hitt <dan.hitt@gmail.com> writes:

> So sometimes, at least, enriched saves on a WYSIWYG basis.

I'm on weak ground here, but it has something to do with what you get
with M-: (text-properties-at (point)) with point on some directory name from
the eshell ls output:

--> '(font-lock-face eshell-ls-directory)'

and what is defined by variable 'enriched-translations'.

Try the same (your recipe) but using M-x shell, and see if it works as
expected. It did here, I mean the boldness and the color stuck with the
enriched file. So maybe it is not a bug but just a limitation of the
enriched-mode, not understanding all possible kinds of text-properties.

bug#46507: 26.1; bold attribute copied into enriched-mode text is not saved

[Eli Zaretskii]

> From: Tomas Nordin <tomasn@posteo.net>
> Date: Sun, 07 Mar 2021 20:23:05 +0100
> Cc: 46507@debbugs.gnu.org
> 
> --> '(font-lock-face eshell-ls-directory)'
> 
> and what is defined by variable 'enriched-translations'.
> 
> Try the same (your recipe) but using M-x shell, and see if it works as
> expected. It did here, I mean the boldness and the color stuck with the
> enriched file. So maybe it is not a bug but just a limitation of the
> enriched-mode, not understanding all possible kinds of text-properties.

We control which text properties enriched-mode knows about.  If
there's some property it needs to support, but doesn't currently, we
can teach it to support that property as well.

bug#46507: 26.1; bold attribute copied into enriched-mode text is not saved

[Dan Hitt]

[-- Attachment #1: Type: text/plain, Size: 2050 bytes --]

On Sun, Mar 7, 2021 at 11:29 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Tomas Nordin <tomasn@posteo.net>
> > Date: Sun, 07 Mar 2021 20:23:05 +0100
> > Cc: 46507@debbugs.gnu.org
> >
> > --> '(font-lock-face eshell-ls-directory)'
> >
> > and what is defined by variable 'enriched-translations'.
> >
> > Try the same (your recipe) but using M-x shell, and see if it works as
> > expected. It did here, I mean the boldness and the color stuck with the
> > enriched file. So maybe it is not a bug but just a limitation of the
> > enriched-mode, not understanding all possible kinds of text-properties.
>
> We control which text properties enriched-mode knows about.  If
> there's some property it needs to support, but doesn't currently, we
> can teach it to support that property as well.
>

Thanks Eli and Thomas.

Thomas: when i repeat the exercise with the shell, everything is lost: both
bolding and color.

The variable enriched-translations is
((face (bold-italic "bold" "italic") (bold "bold") (italic "italic")
(underline "underline") (fixed "fixed") (excerpt "excerpt") (default) (nil
enriched-encode-other-face)) (left-margin (4 "indent")) (right-margin (4
"indentright")) (justification (none "nofill") (right "flushright") (left
"flushleft") (full "flushboth") (center "center")) (PARAMETER (t "param"))
(FUNCTION (enriched-decode-foreground "x-color")
(enriched-decode-background "x-bg-color") (enriched-decode-display-prop
"x-display")) (read-only (t "x-read-only")) (display (nil
enriched-handle-display-prop)) (unknown (nil format-annotate-value)))

This comes from starting emacs with -Q, and evaluating the variable
enriched-translations in the scratch buffer.

I went through the exercise with  eshell, to make sure that nothing
changed, and it doesn't seem to have: colors get saved, but not boldness.

Eli: Maybe it is wrong for me to say "needs" :) :).  It would be very
convenient though, and i could tweak some variables in my start up (for
testing with 'emacs -Q' though i would have to finagle it somehow i guess?).

dan

[-- Attachment #2: Type: text/html, Size: 2977 bytes --]

bug#46507: 26.1; bold attribute copied into enriched-mode text is not saved

[Eli Zaretskii]

> From: Dan Hitt <dan.hitt@gmail.com>
> Date: Sun, 7 Mar 2021 12:18:24 -0800
> Cc: Tomas Nordin <tomasn@posteo.net>, 46507@debbugs.gnu.org
> 
> Thomas: when i repeat the exercise with the shell, everything is lost: both bolding and color.
> 
> The variable enriched-translations is
> ((face (bold-italic "bold" "italic") (bold "bold") (italic "italic") (underline "underline") (fixed "fixed") (excerpt
> "excerpt") (default) (nil enriched-encode-other-face)) (left-margin (4 "indent")) (right-margin (4 "indentright"))
> (justification (none "nofill") (right "flushright") (left "flushleft") (full "flushboth") (center "center")) (PARAMETER
> (t "param")) (FUNCTION (enriched-decode-foreground "x-color") (enriched-decode-background "x-bg-color")
> (enriched-decode-display-prop "x-display")) (read-only (t "x-read-only")) (display (nil
> enriched-handle-display-prop)) (unknown (nil format-annotate-value)))
> 
> This comes from starting emacs with -Q, and evaluating the variable enriched-translations in the scratch
> buffer.
> 
> I went through the exercise with  eshell, to make sure that nothing changed, and it doesn't seem to have:
> colors get saved, but not boldness.

So if the problem is with the font-lock-face vs face property, why is
it that enriched-mode does preserve the color attribute, but not the
both attribute?  They both come from the same font-lock-face property,
no?

> Eli: Maybe it is wrong for me to say "needs" :) :).  It would be very convenient though, and i could tweak some
> variables in my start up (for testing with 'emacs -Q' though i would have to finagle it somehow i guess?).

I don't think there's argument that enriched-mode should support this
use case.

bug#46507: 26.1; bold attribute copied into enriched-mode text is not saved

[Eli Zaretskii]

> Date: Mon, 08 Mar 2021 15:16:07 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: tomasn@posteo.net, 46507@debbugs.gnu.org
> 
> > I went through the exercise with  eshell, to make sure that nothing changed, and it doesn't seem to have:
> > colors get saved, but not boldness.
> 
> So if the problem is with the font-lock-face vs face property, why is
> it that enriched-mode does preserve the color attribute, but not the
> both attribute?  They both come from the same font-lock-face property,
> no?
> 
> > Eli: Maybe it is wrong for me to say "needs" :) :).  It would be very convenient though, and i could tweak some
> > variables in my start up (for testing with 'emacs -Q' though i would have to finagle it somehow i guess?).
> 
> I don't think there's argument that enriched-mode should support this
> use case.

I've now attempted to extend enriched.el on the master branch to
support this use case.  Please test.

bug#46507: 26.1; bold attribute copied into enriched-mode text is not saved

[Dan Hitt]

[-- Attachment #1: Type: text/plain, Size: 1493 bytes --]

Hi Eli,

Thanks for your work.

The debian system i originally used is unavailable, and i'm using xubuntu
20.04 now, emacs 26.3.

Nevertheless, i would be happy to test if i can.

I guess this means that i have to get a copy of your modified version of
enriched.el, and instruct emacs to use that instead of
/usr/share/emacs/26.3/lisp/textmodes/enriched.elc.

So i need instructions on how to do all this in order to test.

Thanks again for your work!!

dan

On Sat, Mar 13, 2021 at 2:28 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > Date: Mon, 08 Mar 2021 15:16:07 +0200
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: tomasn@posteo.net, 46507@debbugs.gnu.org
> >
> > > I went through the exercise with  eshell, to make sure that nothing
> changed, and it doesn't seem to have:
> > > colors get saved, but not boldness.
> >
> > So if the problem is with the font-lock-face vs face property, why is
> > it that enriched-mode does preserve the color attribute, but not the
> > both attribute?  They both come from the same font-lock-face property,
> > no?
> >
> > > Eli: Maybe it is wrong for me to say "needs" :) :).  It would be very
> convenient though, and i could tweak some
> > > variables in my start up (for testing with 'emacs -Q' though i would
> have to finagle it somehow i guess?).
> >
> > I don't think there's argument that enriched-mode should support this
> > use case.
>
> I've now attempted to extend enriched.el on the master branch to
> support this use case.  Please test.
>

[-- Attachment #2: Type: text/html, Size: 2265 bytes --]

bug#46507: 26.1; bold attribute copied into enriched-mode text is not saved

[Eli Zaretskii]

> From: Dan Hitt <dan.hitt@gmail.com>
> Date: Sat, 13 Mar 2021 10:47:23 -0800
> Cc: Tomas Nordin <tomasn@posteo.net>, 46507@debbugs.gnu.org
> 
> Thanks for your work.

You are welcome.

> The debian system i originally used is unavailable, and i'm using xubuntu 20.04 now, emacs 26.3.
> 
> Nevertheless, i would be happy to test if i can.
> 
> I guess this means that i have to get a copy of your modified version of enriched.el, and instruct emacs to
> use that instead of /usr/share/emacs/26.3/lisp/textmodes/enriched.elc.
> 
> So i need instructions on how to do all this in order to test.

Here's the patch which should fix the issue:

diff --git a/lisp/textmodes/enriched.el b/lisp/textmodes/enriched.el
index c44b69c..e43370c 100644
--- a/lisp/textmodes/enriched.el
+++ b/lisp/textmodes/enriched.el
@@ -389,6 +389,8 @@ enriched-face-ans
 	 (apply 'append (mapcar 'enriched-face-ans face)))
 	((let* ((fg (face-attribute face :foreground))
 		(bg (face-attribute face :background))
+                (weight (face-attribute face :weight))
+                (slant (face-attribute face :slant))
 		(props (face-font face t))
 		(ans (cdr (format-annotate-single-property-change
 			   'face nil props enriched-translations))))
@@ -396,6 +398,10 @@ enriched-face-ans
 	     (setq ans (cons (list "x-color" fg) ans)))
 	   (unless (eq bg 'unspecified)
 	     (setq ans (cons (list "x-bg-color" bg) ans)))
+           (if (eq weight 'bold)
+               (setq ans (cons (list "bold") ans)))
+           (if (eq slant 'italic)
+               (setq ans (cons (list "italic") ans)))
 	   ans))))
 
 ;;;

bug#46507: 26.1; bold attribute copied into enriched-mode text is not saved

[Dan Hitt]

[-- Attachment #1: Type: text/plain, Size: 2662 bytes --]

On Sat, Mar 13, 2021 at 11:22 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Dan Hitt <dan.hitt@gmail.com>
> > Date: Sat, 13 Mar 2021 10:47:23 -0800
> > Cc: Tomas Nordin <tomasn@posteo.net>, 46507@debbugs.gnu.org
> >
> > Thanks for your work.
>
> You are welcome.
>
> > The debian system i originally used is unavailable, and i'm using
> xubuntu 20.04 now, emacs 26.3.
> >
> > Nevertheless, i would be happy to test if i can.
> >
> > I guess this means that i have to get a copy of your modified version of
> enriched.el, and instruct emacs to
> > use that instead of /usr/share/emacs/26.3/lisp/textmodes/enriched.elc.
> >
> > So i need instructions on how to do all this in order to test.
>
> Here's the patch which should fix the issue:
>
> diff --git a/lisp/textmodes/enriched.el b/lisp/textmodes/enriched.el
> index c44b69c..e43370c 100644
> --- a/lisp/textmodes/enriched.el
> +++ b/lisp/textmodes/enriched.el
> @@ -389,6 +389,8 @@ enriched-face-ans
>          (apply 'append (mapcar 'enriched-face-ans face)))
>         ((let* ((fg (face-attribute face :foreground))
>                 (bg (face-attribute face :background))
> +                (weight (face-attribute face :weight))
> +                (slant (face-attribute face :slant))
>                 (props (face-font face t))
>                 (ans (cdr (format-annotate-single-property-change
>                            'face nil props enriched-translations))))
> @@ -396,6 +398,10 @@ enriched-face-ans
>              (setq ans (cons (list "x-color" fg) ans)))
>            (unless (eq bg 'unspecified)
>              (setq ans (cons (list "x-bg-color" bg) ans)))
> +           (if (eq weight 'bold)
> +               (setq ans (cons (list "bold") ans)))
> +           (if (eq slant 'italic)
> +               (setq ans (cons (list "italic") ans)))
>            ans))))
>
>  ;;;
>

Thanks Eli!

For the sake of anybody who stumbles across this thread, i tested it by
uncompressing  /usr/share/emacs/26.3/lisp/textmodes/enriched.el.gz into a
new empty directory.

Then i hand-patched in the diffs (i.e., i patched it by cut-and-paste,
using emacs, not using any program that understands diffs).

Then i ran 'emacs -Q', and did M-x load-file to load in the modified
enriched.el file.

Then i did the test: copying in chunks of text from eshell and shell,
setting the mode to enriched, saving, closing, and reopening, and it worked
in all cases.

Presumably what will happen next is that this fix eventually gets pushed
out and adopted by debian and ubuntu and whoever else is downstream, but in
the meantime i can just the modified enriched.el.

Thanks again for figuring out how to fix it.

dan

[-- Attachment #2: Type: text/html, Size: 3701 bytes --]

bug#46507: 26.1; bold attribute copied into enriched-mode text is not saved

[Eli Zaretskii]

> From: Dan Hitt <dan.hitt@gmail.com>
> Date: Sat, 13 Mar 2021 23:32:51 -0800
> Cc: Tomas Nordin <tomasn@posteo.net>, 46507@debbugs.gnu.org
> 
> For the sake of anybody who stumbles across this thread, i tested it by uncompressing 
> /usr/share/emacs/26.3/lisp/textmodes/enriched.el.gz into a new empty directory.
> 
> Then i hand-patched in the diffs (i.e., i patched it by cut-and-paste, using emacs, not using any program that
> understands diffs).
> 
> Then i ran 'emacs -Q', and did M-x load-file to load in the modified enriched.el file.
> 
> Then i did the test: copying in chunks of text from eshell and shell, setting the mode to enriched, saving,
> closing, and reopening, and it worked in all cases.
> 
> Presumably what will happen next is that this fix eventually gets pushed out and adopted by debian and
> ubuntu and whoever else is downstream, but in the meantime i can just the modified enriched.el.
> 
> Thanks again for figuring out how to fix it.

Thanks for testing, I'm therefore closing this bug report.