Re: [Emacs-diffs] master c4151eb: Improve the optional translation of quotes

Re: [Emacs-diffs] master c4151eb: Improve the optional translation of quotes

[Dmitry Gutov]

Hi Paul,

On 06/19/2015 10:39 AM, Paul Eggert wrote:
> branch: master
> commit c4151ebe15479de4c2e511b068cdf9af6a4576cf
> Author: Paul Eggert <eggert@cs.ucla.edu>

> -    (princ (format "\n     ‘%s’ value is\n       " symbol))
> +    (princ (format (substitute-command-keys "\n     ‘%s’ value is\n       ")
> +                   symbol))
>       (if (and value (symbolp value))
> -        (princ (format "‘%s’" value))
> +        (princ (format (substitute-command-keys "‘%s’") value))
> ...etc

With this change, the source files might as well contain ASCII quotes, 
couldn't they?

Re: [Emacs-diffs] master c4151eb: Improve the optional translation of quotes

[Paul Eggert]

Dmitry Gutov wrote:
> With this change, the source files might as well contain ASCII quotes, couldn't
> they?

No, as it's more efficient when substitute-command-keys is a no-op, as this puts 
less pressure on the garbage collector.  Besides, it's no longer important to 
insist on ASCII-only strings in Emacs Lisp files.

Re: [Emacs-diffs] master c4151eb: Improve the optional translation of quotes

[Dmitry Gutov]

On 06/21/2015 10:56 PM, Paul Eggert wrote:
> Besides, it's no
> longer important to insist on ASCII-only strings in Emacs Lisp files.

Since when?

Re: [Emacs-diffs] master c4151eb: Improve the optional translation of quotes

[Paul Eggert]

Dmitry Gutov wrote:
>> it's no
>> longer important to insist on ASCII-only strings in Emacs Lisp files.
>
> Since when?

We've had non-ASCII strings in GNU Emacs Lisp source-code strings for several 
years.  It hasn't been a problem in practice.  Obviously we can't use any 
Unicode character willy-nilly in any context, but it's quite safe now to use 
curved single quotes in docstrings.

Re: [Emacs-diffs] master c4151eb: Improve the optional translation of quotes

[Dmitry Gutov]

On 06/21/2015 11:03 PM, Paul Eggert wrote:

> We've had non-ASCII strings in GNU Emacs Lisp source-code strings for
> several years.  It hasn't been a problem in practice.  Obviously we
> can't use any Unicode character willy-nilly in any context, but it's
> quite safe now to use curved single quotes in docstrings.

Okay, rather than ASCII or not in Elisp code, my concern is whether 
`substitute-command-keys' should translate *from* curved quotes. It's 
not hard to imagine `substitute-command-keys' being called on its result 
again, and then there will be risk of matching different pairs or quotes.

I'd rather keep them in the destination strings, if at all. Like 
mentioned in the other thread, we can switch to simply using a face to 
highlight code segments, and then the curved quotes won't be needed at 
all (at least if that's the user's preference).

Regarding the GC argument, I think you'll be hard-pressed to find a 
machine where it'll make a noticeable difference when processing 
read-life docstrings.

Re: [Emacs-diffs] master c4151eb: Improve the optional translation of quotes

[Paul Eggert]

Dmitry Gutov wrote:
> It's not hard to imagine `substitute-command-keys' being called on its result again

True, but that's always been a bug, even before the recent changes. 
substitute-command-keys has never been idempotent.

> I'd rather keep them [curved quotes] in the destination strings, if at all.

If I understand you correctly, curved quotes will normally be kept in 
destination strings so it should be OK.  The only exceptions will be if users 
customize help-quote-translation to say that they prefer straight quotes, or if 
they run in an obsolescent environment that can't display curved quotes. 
Neither case should be common.

Re: [Emacs-diffs] master c4151eb: Improve the optional translation of quotes

[Dmitry Gutov]

On 06/21/2015 11:46 PM, Paul Eggert wrote:

> True, but that's always been a bug, even before the recent changes.
> substitute-command-keys has never been idempotent.

Shouldn't we try not to make things worse?

> If I understand you correctly, curved quotes will normally be kept in
> destination strings so it should be OK.

They would make little no to sense in the source strings. Have you read 
this subthread?

http://lists.gnu.org/archive/html/emacs-devel/2015-06/msg00358.html

Re: [Emacs-diffs] master c4151eb: Improve the optional translation of quotes

[Paul Eggert]

Dmitry Gutov wrote:

>> substitute-command-keys has never been idempotent.
>
> Shouldn't we try not to make things worse?

I don't see the point.  Code should never invoke substitute-command-keys on the 
output of substitute-command-keys; the function is not designed that way. 
That's always been true.  It's not "making things worse" to rely on a property 
that's been in Emacs for ages.

>> If I understand you correctly, curved quotes will normally be kept in
>> destination strings so it should be OK.
>
> They would make little no to sense in the source strings. Have you read this
> subthread?
>
> http://lists.gnu.org/archive/html/emacs-devel/2015-06/msg00358.html

Yes, I've been reading and participating in these subthreads.  I don't follow 
your point here, though.  Normally, curved quotes pass through 
substitute-command-keys unchanged.  This is simple and intuitive and is what 
Emacs has always normally done.  Perhaps we'll come up with a better way to 
highlight the output of substitute-command-keys at some point, a way that treats 
quotes and key substitutions differently.  That might be nice, but it's not 
clear how it would work exactly, and in the meantime we have a simple approach 
that does work.

Re: [Emacs-diffs] master c4151eb: Improve the optional translation of quotes

[Dmitry Gutov]

On 06/22/2015 09:50 AM, Paul Eggert wrote:

> It's not "making things worse" to
> rely on a property that's been in Emacs for ages.

All right, I guess that's true.

> Yes, I've been reading and participating in these subthreads.  I don't
> follow your point here, though.  Normally, curved quotes pass through
> substitute-command-keys unchanged.

I don't see what's "normally" have to do with anything.

In the input, we don't need quotes that usually pass unchanged to the 
output. We need some output-independent markup that will eventually 
translate into whatever presentation we choose. The more we're able to 
defer that choice, the better. It's why I'm liking the original 
suggestion to do it via font-lock better and better now.

> This is simple and intuitive and is
> what Emacs has always normally done.  Perhaps we'll come up with a
> better way to highlight the output of substitute-command-keys at some
> point, a way that treats quotes and key substitutions differently.  That
> might be nice, but it's not clear how it would work exactly, and in the
> meantime we have a simple approach that does work.

Please recall the suggestion to do everything via font-lock. It would 
also allow us to use some different quoting mechanism other than \\=, 
which Stefan complained about.

Re: [Emacs-diffs] master c4151eb: Improve the optional translation of quotes

[Paul Eggert]

Dmitry Gutov wrote:
> We need some output-independent markup that will eventually translate into
> whatever presentation we choose.

Yes, and we have that now.  The simplest way to use it is to use the same curved 
quotes in input as in output, but we can add more complex ways as needed.

> Please recall the suggestion to do everything via font-lock. It would also allow
> us to use some different quoting mechanism other than \\=, which Stefan
> complained about.

Of course.  Font-lock is pretty much independent of the quoting mechanism that 
docstrings use.  For example, if we decide to support only backslash escapes to 
quote in docstrings (yuck!), we could get that to work with font-lock; or if we 
decide to support only curved quotes to quote in docstrings (too aggressive, if 
you ask me), we could get that to work with font-lock too.

When I tried font-lock mode in this area I came away baffled (see Bug#20385 
Message#289, and Bug#20613), so I don't know how well it'd really work in 
practice for this problem.  It is a poorly documented area, unfortunately.

Re: [Emacs-diffs] master c4151eb: Improve the optional translation of quotes

[Dmitry Gutov]

On 06/23/2015 08:36 AM, Paul Eggert wrote:

> Yes, and we have that now.  The simplest way to use it is to use the
> same curved quotes in input as in output, but we can add more complex
> ways as needed.

It may look easiest, but I wouldn't call it "simplest". It would be 
better to have a clear separation between the input and output. If you 
don't like the `' quotes so much, why not borrow a quoting method from 
some other popular markup language, like tildes from org?

For the other options, see "fixed width text" in 
http://hyperpolyglot.org/lightweight-markup.

> Of course.  Font-lock is pretty much independent of the quoting
> mechanism that docstrings use.

That's not true. For better results, font-lock will need to handle the 
escaping and quote conversion.

Otherwise, it won't know whether a given character was escaped in the 
docstring, or was present there verbatim.

>  For example, if we decide to support
> only backslash escapes to quote in docstrings (yuck!), we could get that
> to work with font-lock; or if we decide to support only curved quotes to
> quote in docstrings (too aggressive, if you ask me), we could get that
> to work with font-lock too.
>
> When I tried font-lock mode in this area I came away baffled (see
> Bug#20385 Message#289, and Bug#20613), so I don't know how well it'd
> really work in practice for this problem.  It is a poorly documented
> area, unfortunately.

I gave you a recipe for a fix in 
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=20385#292, and it worked 
fine for me locally. (I thought I've also sent a patch, but apparently not).

So if you had trouble applying that, maybe you should've asked for the 
proper patch instead of rushing the other way.

Re: [Emacs-diffs] master c4151eb: Improve the optional translation of quotes

[Paul Eggert]

Dmitry Gutov wrote:
> I gave you a recipe for a fix in
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=20385#292, and it worked fine for
> me locally. (I thought I've also sent a patch, but apparently not).
>
> So if you had trouble applying that, maybe you should've asked for the proper
> patch instead of rushing the other way.

As I mentioned earlier I never fully understood the font-lock proposal.  I could 
not easily decipher most of the abovementioned message.  Luckily, though, the 
abovementioned message briefly mentioned not using font-locking and said of it 
"...but indeed, this approach could be the simpler one." -- an assessmeent that 
seemed sound to me -- and so I took the simpler approach.

Re: [Emacs-diffs] master c4151eb: Improve the optional translation of quotes

[Dmitry Gutov]

On 06/24/2015 08:24 AM, Paul Eggert wrote:

> As I mentioned earlier I never fully understood the font-lock proposal.
> I could not easily decipher most of the abovementioned message.

Asking questions might help. I can't help but feel frustrated that you 
haven't made an informed decision. See the bottom of this email for the 
patch against f743819 (which was the version in master at the time). It 
only adds one line, the one I've mentioned in that message.

I'm not sure what more to explain, but the undesirable effect you've 
seen was due to 
http://www.gnu.org/software/emacs/manual/html_node/elisp/Syntactic-Font-Lock.html

> Luckily, though, the abovementioned message briefly mentioned not using
> font-locking and said of it "...but indeed, this approach could be the
> simpler one." -- an assessmeent that seemed sound to me -- and so I took
> the simpler approach.

I did say that, tentatively, and I've changed my mind since.

- Using font-lock will allow to change the output significantly in the 
future, without breaking the substitute-command-keys API.

- This functionality is closely related to linkification of identifiers 
between the quotes, also performed in Lisp (see the usages of 
help-xref-symbol-regexp). If substitute-command-keys handles escapes, 
help-xref-symbol-regexp (and the function using it) don't know whether a 
given quote was escaped or not). Especially if the source file can 
contain curly quotes, and substitute-command-keys also translates into them.

- "too tempting to implement complicated heuristics that would have been 
a pain to document" is not a good argument not to do something in a 
high-level language.

Now, there are still problems with that patch:

- It doesn't handle escaping either. Someone just needs to pick the 
escaping syntax, and we can handle it in Lisp just as well as in C. Why 
did we decide that "\\" isn't good enough?

- Like you said: "describe-variable should curve the quotes in the doc 
string, but not in the contents of the variable". The code that prints 
the value will need to add some text property to it ("verbatim"? 
"font-lock-ignore"?), and the font-lock rule can look it up and skip 
those regions. Nothing too hard.

Re: [Emacs-diffs] master c4151eb: Improve the optional translation of quotes

[Paul Eggert]

Dmitry Gutov wrote:

> See the bottom of this email for the patch

I didn't see any patch in that email, unfortunately -- perhaps you forgot to 
include it?

> - "too tempting to implement complicated heuristics that would have been a pain
> to document" is not a good argument not to do something in a high-level language.

True.  But it's still a temptation we should strive to avoid.

> - Like you said: "describe-variable should curve the quotes in the doc string,
> but not in the contents of the variable". The code that prints the value will
> need to add some text property to it ("verbatim"? "font-lock-ignore"?), and the
> font-lock rule can look it up and skip those regions. Nothing too hard.

As you'll recall, I don't understand font-lock.  That being said, I worry that 
all this stuff would be complicated and would involve using undocumented or 
poorly-documented features (see Bug#20613).

Also, it's not clear how the overall approach would work on limited displays 
that don't have alternate fonts and/or colors.  Suppose, for example, that the 
user is running 'emacs --color=never' on a character terminal?

Re: [Emacs-diffs] master c4151eb: Improve the optional translation of quotes

[Dmitry Gutov]

On 06/25/2015 05:05 PM, Paul Eggert wrote:

> I didn't see any patch in that email, unfortunately -- perhaps you
> forgot to include it?

Sorry, it's attached now.

> True.  But it's still a temptation we should strive to avoid.

Depends on what we call complicated. Like Richard mentioned, at least we 
can try to leave unpaired quotes alone. Or not, I don't really care that 
much personally.

>> - Like you said: "describe-variable should curve the quotes in the doc
>> string,
>> but not in the contents of the variable". The code that prints the
>> value will
>> need to add some text property to it ("verbatim"?
>> "font-lock-ignore"?), and the
>> font-lock rule can look it up and skip those regions. Nothing too hard.
>
> As you'll recall, I don't understand font-lock.

There's no requirement for you to implement all of it yourself. You can 
take my patch, look for problems, and ask for them to be fixed.

You'll have to do the substitute-command-keys part, though.

> That being said, I
> worry that all this stuff would be complicated and would involve using
> undocumented or poorly-documented features (see Bug#20613).

I've enumerated practical considerations of having it implemented *not* 
in font-lock. If some font-lock features lack documentation, demand it 
to be added, then.

> Also, it's not clear how the overall approach would work on limited
> displays that don't have alternate fonts and/or colors.  Suppose, for
> example, that the user is running 'emacs --color=never' on a character
> terminal?

diff --git a/lisp/help-mode.el b/lisp/help-mode.el
index f99e916..6256513 100644
--- a/lisp/help-mode.el
+++ b/lisp/help-mode.el
@@ -287,6 +287,13 @@ Commands:
  \\{help-mode-map}"
    (set (make-local-variable 'revert-buffer-function)
         'help-mode-revert-buffer)
+  (setq font-lock-defaults '(nil t))
+  (font-lock-add-keywords
+   nil '(("`\\([^[:space:]'`‘’][^'`‘’]*\\)?'"
+	  . (0 (progn (compose-region (match-beginning 0)
+				      (1+ (match-beginning 0)) ?‘)
+		      (compose-region (1- (match-end 0)) (match-end 0) ?’)
+		      nil)))))
    (set (make-local-variable 'bookmark-make-record-function)
         'help-bookmark-make-record))

Re: [Emacs-diffs] master c4151eb: Improve the optional translation of quotes

[Paul Eggert]

Dmitry Gutov wrote:

> If some font-lock features lack documentation, demand it to be added, then.

"Demand" is a pretty strong word.  I did ask for better documentation in 
Bug#20613.  I realize it's not always easy to write good documentation.

Re: [Emacs-diffs] master c4151eb: Improve the optional translation of quotes

[Dmitry Gutov]

On 07/31/2015 09:51 PM, Paul Eggert wrote:

> "Demand" is a pretty strong word.  I did ask for better documentation in
> Bug#20613.  I realize it's not always easy to write good documentation.

I thought I addressed that bug report more or less adequately.

The piece of code you've quoted does not follow the idiomatic way 
font-lock-keywords should be used, but it plays by the documented rules, 
and simply adds compose-region as a side-effect. We can't document every 
way people might want to write a Lisp program.

In any case, I thought we've moved away from the idea of using 
font-lock, to translating the buffer contents directly, in Lisp?

Re: [Emacs-diffs] master c4151eb: Improve the optional translation of quotes

[Paul Eggert]

Dmitry Gutov wrote:
> I thought we've moved away from the idea of using font-lock, to translating the
> buffer contents directly, in Lisp?

Ah, I wasn't aware of that.  At least, not the Lisp part.

Re: [Emacs-diffs] master c4151eb: Improve the optional translation of quotes

[Dmitry Gutov]

On 08/01/2015 04:16 AM, Paul Eggert wrote:

> Ah, I wasn't aware of that.  At least, not the Lisp part.

I hoped to move, at least.

http://lists.gnu.org/archive/html/emacs-devel/2015-07/msg00063.html

Re: [Emacs-diffs] master c4151eb: Improve the optional translation of quotes

[Dmitry Gutov]

On 06/25/2015 05:05 PM, Paul Eggert wrote:

> Also, it's not clear how the overall approach would work on limited
> displays that don't have alternate fonts and/or colors.  Suppose, for
> example, that the user is running 'emacs --color=never' on a character
> terminal?

I thought this was implemented somewhere in the display engine? If not, 
it could be.

Or someone could tell me how to check whether curly quotes are 
undisplayable in Lisp, and the rule will perform no substitutions, for 
instance.

And colors -- what about them?

Re: [Emacs-diffs] master c4151eb: Improve the optional translation of quotes

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 25 Jun 2015 17:59:58 +0300
> 
> On 06/25/2015 05:05 PM, Paul Eggert wrote:
> 
> > Also, it's not clear how the overall approach would work on limited
> > displays that don't have alternate fonts and/or colors.  Suppose, for
> > example, that the user is running 'emacs --color=never' on a character
> > terminal?
> 
> I thought this was implemented somewhere in the display engine? If not, 
> it could be.

I didn't track this discussion, so forgive me if what I say below
makes no sense.

With that caveat, since you (AFAIU) are talking about using the
font-lock mechanism for producing effects other than color, whether
the terminal supports colors should not bother you.  That's because
the actual expression of the faces that have color attributes happens
on the lowest level of terminal-specific parts of the display engine,
and the rest of display doesn't care -- it manipulates faces, not
colors.

> Or someone could tell me how to check whether curly quotes are 
> undisplayable in Lisp, and the rule will perform no substitutions, for 
> instance.

I think you want char-displayable-p.

Re: [Emacs-diffs] master c4151eb: Improve the optional translation of quotes

[Dmitry Gutov]

On 06/25/2015 06:11 PM, Eli Zaretskii wrote:

> With that caveat, since you (AFAIU) are talking about using the
> font-lock mechanism for producing effects other than color, whether
> the terminal supports colors should not bother you.

Yes, that's why I didn't understand why mention colors. The question of 
supported characters seems to be a pertinent one, though.

> I think you want char-displayable-p.

Seems so. Thanks.