Variable-width font indentation: pasting outside Emacs

Variable-width font indentation: pasting outside Emacs

[Drew Adams]

FWIW -

I see lots of discussion about different ways that Emacs might
implement appropriate display wrt indentation and alignment, for
code and other text, and in particular, for variable-width fonts.

But all of that is only about how the text appears inside Emacs.
What about copying text and pasting it into other applications?

Today, with a fixed-width font, you can copy and paste text,
including code, into other apps and have it appear pretty much
as you might expect/hope.  (At least, that is, provided the
text uses only SPC and not tab chars for indentation, or
provided you first use `untabify'.)

Should some consideration be given to pasting variable-width
text outside Emacs in a way that pretty much does what a user
might want/expect?  If so, how to do that for variable-width
fonts?

Could we, for example, (optionally) affect copy or paste
operations, to automatically try to compensate by inserting
the (more or less) right number of SPC chars of the
variable-width font (of the first non-whitespace char on the
line)?

Should we perhaps have different such conversions available
while copying, depending on the destination (paste) environment,
i.e., whether or not it handles variable-width text?

People communicate about code and other text in more and more
ways, many of which are and will remain outside Emacs.  Can we
try to DTRT for variable-width text, so that the result of
pasting into another app gives indentation and alignment that
at least approximates what one would want/expect?  If so,
should we try to do that?

Shouldn't we take a point of view in which Emacs is part of
the wider world and text is exchanged between Emacs and other
apps?  Thinking about that fact of exchange should maybe
affect decisions about how to indent and align variable-width
fonts inside Emacs.

Or if it doesn't, maybe we should at least think about how
we might convert text for exchange, from whatever design we
choose to solve the inside-Emacs problem of displaying
variable-width text with appropriate indentation and alignment.

Put differently, if we choose this or that implementation to
show variable-width text appropriately in Emacs, that might
affect how well the result of pasting outside Emacs reflects
the appearance inside Emacs.

Re: Variable-width font indentation: pasting outside Emacs

[Yuri Khan]

On Wed, Mar 7, 2018 at 1:21 AM, Drew Adams <drew.adams@oracle.com> wrote:

> Could we, for example, (optionally) affect copy or paste
> operations, to automatically try to compensate by inserting
> the (more or less) right number of SPC chars of the
> variable-width font (of the first non-whitespace char on the
> line)?

You mean changing the number of spaces used as indentation, depending
on what font the receiving application uses?

> People communicate about code and other text in more and more
> ways, many of which are and will remain outside Emacs.  Can we
> try to DTRT for variable-width text, so that the result of
> pasting into another app gives indentation and alignment that
> at least approximates what one would want/expect?  If so,
> should we try to do that?

Oh please don’t.

Why? Because the widget where the text is going to be pasted from
Emacs is not the widget that will ultimately display the text.

How is that? Imagine a chat application such as Mattermost. Its UI is
a web page with a text input widget on the bottom. That widget
normally uses a variable width font, and accepts Markdown syntax. The
user will normally type three grave accent characters ``` and a hard
newline, then paste a snippet of code from clipboard, then close with
another ```. On the server, Markdown will be interpreted and the
recipient will receive a syntax-highlighted, fixed-width-formatted
fragment of code.

With your suggestion, that fragment will have the wrong amount of
indentation. (Where “wrong” denotes “other than the sender intended or
expected”.)


Relatedly, I have worked with several applications that support
copying and pasting HTML markup. In their striving to make the result
“intuitive”, they introduce abominations on the receiving side.

Example: You are reading a web page. You copy a fragment of text. The
source page is styled with CSS that specifies a blue-black foreground
color and a fancy font for the body text. When you paste that into a
blog post, the receiving widget attempts to preserve that color. But,
since the body text style of the target article specifies dark gray
foreground color and a different fancy font, it applies direct
formatting to the pasted text: <span style="color: rgb(0,51,102);
font-family: Proxima Nova, …">Lorem ipsum…</span>.

Now the post text is tainted. It will appear in blue-black and in
Proxima Nova for every viewer, regardless of their preferred color
scheme and font. A user who prefers green text on a dark gray
background will break his/her eyes trying to read that.

Copying and pasting WYSIWYG formatting: Just say no. Doing things
behind the user’s back on copy/paste: Just say no.

RE: Variable-width font indentation: pasting outside Emacs

[Drew Adams]

> > Could we, for example, (optionally) affect copy or paste
> > operations, to automatically try to compensate by inserting
> > the (more or less) right number of SPC chars of the
> > variable-width font (of the first non-whitespace char on the
> > line)?
> 
> You mean changing the number of spaces used as indentation,
> depending on what font the receiving application uses?

Perhaps.  If the design chosen for solving this inside Emacs were
just to add lots of variable-width SPC chars, and if that were
totally inappropriate in some paste context (fixed-width, for
example), a user should be able to copy the text in a way that
pasting it didn't produce something awful.  I said "optionally".
(And I'm not supposing that that would be the design choice for
Emacs.)

But _I'm not proposing_ any design or implementation.  I just
wanted to mention that we might want to think ahead to the fact
that users will want to paste text, including code, into other
apps.  Knowing that fact might affect decisions for how we deal
with the indentation and alignment inside Emacs.

> > People communicate about code and other text in more and more
> > ways, many of which are and will remain outside Emacs.  Can we
> > try to DTRT for variable-width text, so that the result of
> > pasting into another app gives indentation and alignment that
> > at least approximates what one would want/expect?  If so,
> > should we try to do that?
> 
> Oh please don’t.
> 
> Why? Because the widget where the text is going to be pasted from
> Emacs is not the widget that will ultimately display the text.
> 
> How is that? Imagine a chat application such as Mattermost. Its UI is
> a web page with a text input widget on the bottom. That widget
> normally uses a variable width font, and accepts Markdown syntax. The
> user will normally type three grave accent characters ``` and a hard
> newline, then paste a snippet of code from clipboard, then close with
> another ```. On the server, Markdown will be interpreted and the
> recipient will receive a syntax-highlighted, fixed-width-formatted
> fragment of code.
> 
> With your suggestion, that fragment will have the wrong amount of
> indentation. (Where “wrong” denotes “other than the sender intended or
> expected”.)

I didn't suggest any such thing.  You've gone off half-cocked.
Try reading what I wrote again, please.

There are any number of different paste contexts.  Clearly it is
the _user_ who should be able to control what the paste operation,
which really means the Emacs copy operation, does.

Obviously, if an HTML text input box accepts input that is then
displayed in some particular way (using a markup convention or
whatever), the user is the one to know about that and choose the
right kind of copy from Emacs.  Emacs itself cannot know about
the paste context.  But the user can, if anyone can.

If Emacs doesn't offer a user any control over this then the
result will mostly be bad, not good, I'm afraid.  One size does
not fit all.

> Relatedly, I have worked with several applications that support
> copying and pasting HTML markup. In their striving to make the result
> “intuitive”, they introduce abominations on the receiving side.

I didn't suggest trying to make the result intuitive or automatic.
Just the opposite.

Here are two limited/bad approaches, neither of which I'd like to
see: (1) do something in Emacs that makes display great but makes
copy+paste to other apps awful, and (2) do something in Emacs that
tries to provide a one-size-fits-all "intuitive" guess about what
the paste needs might be.

We can imagine other problematic designs, when it comes to their
effect on pasting.  If _all_ indentation is realized in Emacs
only by, say, a `display' property, what will that give when the
text is pasted somewhere?  There are any number of designs that
might have negative consequences for pasting.  Let's keep pasting
in the back of the mind when thinking about designing display of
indentation and alignment in Emacs.  That's all I wanted to say.

Only the user will know (might know) what is needed wrt pasting.
The question I'm raising is whether we can give users some control
over that.

And even if it's decided that we cannot or should not try, I expect
that ignoring the fact that people will paste copied text may lead
to a pretty indentation and alignment display inside Emacs but may
make pasting outside it problematic.  Users will then need to do a
fair amount of manual whitespace cleanup after pasting, and such
cleanup is a bother in most contexts outside Emacs.

I'm contrasting that with the case we have today with fixed-width
fonts and using only SPC (not tab chars) for indentation.  Piece
of cake, in general.

If we don't offer some control over this then providing pretty
display inside Emacs may make pasting outside Emacs so bad that
some users will just stick with fixed-width fonts.  (I may be one
of them.)

> Example: You are reading a web page. You copy a fragment of text. The
> source page is styled with CSS that specifies a blue-black foreground
> color and a fancy font for the body text. When you paste that into a
> blog post, the receiving widget attempts to preserve that color. But,
> since the body text style of the target article specifies dark gray
> foreground color and a different fancy font, it applies direct
> formatting to the pasted text: <span style="color: rgb(0,51,102);
> font-family: Proxima Nova, …">Lorem ipsum…</span>.
> 
> Now the post text is tainted. It will appear in blue-black and in
> Proxima Nova for every viewer, regardless of their preferred color
> scheme and font. A user who prefers green text on a dark gray
> background will break his/her eyes trying to read that.
>
> Copying and pasting WYSIWYG formatting: Just say no. Doing things
> behind the user’s back on copy/paste: Just say no.

You have misread me completely, I'm afraid.  I am 100% against
doing any such thing, especially behind the user's back.

Ignoring the paste problem (like ignoring the serialization problem,
which is similar) won't help Emacs find the right design for solving
the problem of displaying text with proper indentation and alignment.

Re: Variable-width font indentation: pasting outside Emacs

[Eli Zaretskii]

> Date: Tue, 6 Mar 2018 10:21:47 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: Paul Eggert <eggert@cs.ucla.edu>
> 
> Could we, for example, (optionally) affect copy or paste
> operations, to automatically try to compensate by inserting
> the (more or less) right number of SPC chars of the
> variable-width font (of the first non-whitespace char on the
> line)?

That shouldn't be needed, because those whitespace characters should
be there in the first place.  The display features should be applied
on top of that, not instead of that.

Re: Variable-width font indentation: pasting outside Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > But all of that is only about how the text appears inside Emacs.
  > What about copying text and pasting it into other applications?

This is am important area for work.  The first question
is to figure out what behavior is right.

Would you like to work on that?

It would be useful to look at what happens when copying
text in and out of LibreOffice, and between two windows
of LibreOffice.  That could give an idea of what we
would want Emacs to do, at least in modes for documents.

We might want to do different things in modes for software.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

RE: Variable-width font indentation: pasting outside Emacs

[Drew Adams]

>   > But all of that is only about how the text appears inside Emacs.
>   > What about copying text and pasting it into other applications?
> 
> This is am important area for work.  The first question
> is to figure out what behavior is right.
> 
> Would you like to work on that?

Not I, but I hope that those who design the new treatment
of indentation and alignment think about this aspect too.

> It would be useful to look at what happens when copying
> text in and out of LibreOffice, and between two windows
> of LibreOffice.  That could give an idea of what we
> would want Emacs to do, at least in modes for documents.
> 
> We might want to do different things in modes for software.

RE: Variable-width font indentation: pasting outside Emacs

[Drew Adams]

Maybe this will clarify my raising of this pasting question:

Today, I copy some code from a Lisp file included in Emacs,
or from an Info node.  Then I modify it, add to it, subtract
from it, etc., to come up with something I think might help
answer a user question on some web site.

It happens sometimes that the code I copied uses tab chars
for indentation.  I use a nil value of `indent-tabs-mode'
for my own use, but the copied code introduces some tab
chars (alas).   I also use a fixed-width font where I'm
working on the copied code.

Depending on the particular context where I might end up
pasting (outside Emacs) my code that now includes some tab
chars, the result might mess up indentation because the
paste context handles tab chars differently.  This is so
even in the common case where the destination expects code
and a fixed-width font and handles such input well.

So today, I'm in the habit of first using `untabify' to
convert such tab chars to SPC.  End of story - great and
easy solution.

In effect, I performed a simple operation before copying.
I could just as well have used a custom command that
copied the text after converting (perhaps a copy of) it
by replacing the tab chars.

Without the simple `untabify' function that Emacs provides,
or something similar (e.g. `query-replace'), I might have
pasted the text with tab chars into the destination, and
then have had to manually clean it up using the inferior
editor of the destination context.  Ugh.

Tab chars are, in effect, variable-width text.  What we
do now to handle tab chars across different environments
can perhaps give us some idea of the problem, and perhaps
of possible solutions, about variable-width text in general.

When using a variable-width font, SPC as well as tab chars
have variable width.  Other chars do too, of course, but
SPC and tab chars are typically used for indentation and
alignment.

Do we have such a simple solution, analogous to using
`untabify' before pasting, for a case where copied code
(or other text) is variable-width?  And where perhaps
the paste destination also handles variable-width text?

Should we?  If we don't, how much trouble will users have
to go through to work around the problem or clean up when
they see crazy results from pasting?

That's the question I'm suggesting we think about.  And
this context is likely more complex, as there are different
kinds of destinations, which may handle variable-width text
differently.  IOW, there may not be a single, simple,
one-size-fits-all `untabify' analog.  We might need more
than one approach.

Yes, it is the user who knows the destination (better than
Emacs knows it, at least).  It is the user whom I would
want to be able to choose how to handle it.

How we can best respond to this question/problem, I don't
know.  I was only asking that we take it into account while
designing an improved-variable-text-indentation-&-alignment
approach for use inside Emacs.  It's not just about how
things look in Emacs.  Text copied from Emacs sometimes
gets pasted outside Emacs.  Let's think about that too;
that's all.

Re: Variable-width font indentation: pasting outside Emacs

[Yuri Khan]

On Wed, Mar 7, 2018 at 5:15 AM, Drew Adams <drew.adams@oracle.com> wrote:

> It happens sometimes that the code I copied uses tab chars
> for indentation.  I use a nil value of `indent-tabs-mode'
> for my own use, but the copied code introduces some tab
> chars (alas).

Okay, that is the essence of the problem: you are copying and pasting
code across a formatting convention boundary and you have to realize
and acknowledge the fact and reformat the code according to the
conventions of the target (or have code reformatted for you).

That problem is not limited to tabs, spaces, indentation, and
alignment. It involves also:

* maximum line length;
* spaces around punctuation;
* omitting or including optional grammar elements (e.g. quotes around
HTML attribute values, or the trailing semicolon in a Pascal compound
statement);
* preference for one of several equivalent forms of expressing an idea
(e.g. in C++, “const Foo” vs “Foo const”; or in HTML, using a Unicode
character directly vs character reference vs entity reference);
* …


There are some modes (web-mode comes to mind) that automatically
reindent after a paste operation. That could be made a universal
feature. I imagine it could be quite convenient if (a) your preferred
indentation style matches their defaults, or (b) you take the effort
to teach them your style; and quite annoying otherwise.

A good first step would be to make reindentation also tabify or
untabify the leading whitespace of each reindented line according to
the current value of indent-tabs-mode. (Now, if the line is to be
indented at 4, indent-tabs-mode is nil, tab-width is 4, and there is a
single tab character at the start of the line, it will be kept with a
tab.)