bug#31231: Symbol search broken in NEWS and ChangeLogs

bug#31231: Symbol search broken in NEWS and ChangeLogs

[Juri Linkov]

Symbol search ‘M-s _’ fails to find symbols in NEWS and ChangeLogs where
symbols are quoted by apostrophes whose syntax is word-constituent.

I see two solutions:

1. Modify syntax tables to redefine syntax of ‘'’ back to its default
   that is punctuation or to expression quote:

   (defvar change-log-mode-syntax-table
     (let ((st (make-syntax-table)))
       (modify-syntax-entry ?' "'   " st)
       st)
     "Syntax table used while in `change-log-mode'.")

This solution is not applicable to NEWS files where
text-mode syntax tables are inherited via outline-mode
whose syntax table can't be changed.

2. In these files replace apostrophes by “LEFT SINGLE QUOTATION MARK” and
   “RIGHT SINGLE QUOTATION MARK” like are used in Info and Help buffers.

bug#31231: Symbol search broken in NEWS and ChangeLogs

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Date: Sat, 21 Apr 2018 23:21:05 +0300
> 
> Symbol search ‘M-s _’ fails to find symbols in NEWS and ChangeLogs where
> symbols are quoted by apostrophes whose syntax is word-constituent.

These files use modes that are descendants of text-mode, so the notion
of a symbol in general makes little sense there.  IOW, this is plain
text, not a program.

> 1. Modify syntax tables to redefine syntax of ‘'’ back to its default
>    that is punctuation or to expression quote:
> 
>    (defvar change-log-mode-syntax-table
>      (let ((st (make-syntax-table)))
>        (modify-syntax-entry ?' "'   " st)
>        st)
>      "Syntax table used while in `change-log-mode'.")
> 
> This solution is not applicable to NEWS files where
> text-mode syntax tables are inherited via outline-mode
> whose syntax table can't be changed.
> 
> 2. In these files replace apostrophes by “LEFT SINGLE QUOTATION MARK” and
>    “RIGHT SINGLE QUOTATION MARK” like are used in Info and Help buffers.

Thanks, but I don't like any of these 2 alternatives.  The first one
will get in the way of spell-checking words with an apostrophe, the
second one will make it harder to view the files on text terminals
that don't support UTF-8.

Is this really such a bad problem that it requires us to tweak
text-based modes?  What happens if you try searching for the symbol as
a simple string? do you get many false positives?

bug#31231: Symbol search broken in NEWS and ChangeLogs

[Juri Linkov]

>> 2. In these files replace apostrophes by “LEFT SINGLE QUOTATION MARK” and
>>    “RIGHT SINGLE QUOTATION MARK” like are used in Info and Help buffers.
>
> Thanks, but I don't like any of these 2 alternatives.  The first one
> will get in the way of spell-checking words with an apostrophe, the
> second one will make it harder to view the files on text terminals
> that don't support UTF-8.

But how these text terminals display Unicode quotes in Info manuals?

> Is this really such a bad problem that it requires us to tweak
> text-based modes?  What happens if you try searching for the symbol as
> a simple string? do you get many false positives?

Yes, this is a real problem.  Now I have a good habit of using
symbol search mode everywhere when searching for symbols.
But the search failure reminds that NEWS and ChangeLogs are still
problematic and I had to try again after switching to literal search.

bug#31231: Symbol search broken in NEWS and ChangeLogs

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: 31231@debbugs.gnu.org
> Date: Mon, 23 Apr 2018 01:12:31 +0300
> 
> >> 2. In these files replace apostrophes by “LEFT SINGLE QUOTATION MARK” and
> >>    “RIGHT SINGLE QUOTATION MARK” like are used in Info and Help buffers.
> >
> > Thanks, but I don't like any of these 2 alternatives.  The first one
> > will get in the way of spell-checking words with an apostrophe, the
> > second one will make it harder to view the files on text terminals
> > that don't support UTF-8.
> 
> But how these text terminals display Unicode quotes in Info manuals?

Info manuals are displayed by an Info reader, and each Info reader has
its own solution for this problem (generally, substitution of ASCII
characters for undisplayable Unicode ones).

I was talking about reading NEWS in a general-purpose text-browsing
program, such as Less.  It is IMO not a good idea to prevent people
from doing that and requiring them to only read NEWS and ChangeLog
files in Emacs.

> > Is this really such a bad problem that it requires us to tweak
> > text-based modes?  What happens if you try searching for the symbol as
> > a simple string? do you get many false positives?
> 
> Yes, this is a real problem.  Now I have a good habit of using
> symbol search mode everywhere when searching for symbols.
> But the search failure reminds that NEWS and ChangeLogs are still
> problematic and I had to try again after switching to literal search.

Can you show a real-life example, with a particular symbol, where
using a literal search is inconvenient?

bug#31231: Symbol search broken in NEWS and ChangeLogs

[Juri Linkov]

>> >> 2. In these files replace apostrophes by “LEFT SINGLE QUOTATION MARK” and
>> >>    “RIGHT SINGLE QUOTATION MARK” like are used in Info and Help buffers.
>> >
>> > Thanks, but I don't like any of these 2 alternatives.  The first one
>> > will get in the way of spell-checking words with an apostrophe, the
>> > second one will make it harder to view the files on text terminals
>> > that don't support UTF-8.
>> 
>> But how these text terminals display Unicode quotes in Info manuals?
>
> Info manuals are displayed by an Info reader, and each Info reader has
> its own solution for this problem (generally, substitution of ASCII
> characters for undisplayable Unicode ones).

Does the Emacs Info reader implement such solution?

> I was talking about reading NEWS in a general-purpose text-browsing
> program, such as Less.  It is IMO not a good idea to prevent people
> from doing that and requiring them to only read NEWS and ChangeLog
> files in Emacs.

When I tried to read NEWS using Less in xterm, I can see all these
quotes in UTF-8, for example, the quotes in “like this”.

>> > Is this really such a bad problem that it requires us to tweak
>> > text-based modes?  What happens if you try searching for the symbol as
>> > a simple string? do you get many false positives?
>> 
>> Yes, this is a real problem.  Now I have a good habit of using
>> symbol search mode everywhere when searching for symbols.
>> But the search failure reminds that NEWS and ChangeLogs are still
>> problematic and I had to try again after switching to literal search.
>
> Can you show a real-life example, with a particular symbol, where
> using a literal search is inconvenient?

Every symbol in NEWS quoted by either ‘straight’ quoting style 'like this',
or ‘grave’ `like this'.  For example, for 'electric-quote-replace-double'
starting Isearch with ‘M-s .’, and repeating the same symbol search in the
source code.

bug#31231: Symbol search broken in NEWS and ChangeLogs

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: 31231@debbugs.gnu.org
> Date: Mon, 23 Apr 2018 23:16:09 +0300
> 
> >> But how these text terminals display Unicode quotes in Info manuals?
> >
> > Info manuals are displayed by an Info reader, and each Info reader has
> > its own solution for this problem (generally, substitution of ASCII
> > characters for undisplayable Unicode ones).
> 
> Does the Emacs Info reader implement such solution?

Emacs itself does, not just in Info.  The standalone Info reader (part
of the Texinfo package) has a slightly different solution.

> > I was talking about reading NEWS in a general-purpose text-browsing
> > program, such as Less.  It is IMO not a good idea to prevent people
> > from doing that and requiring them to only read NEWS and ChangeLog
> > files in Emacs.
> 
> When I tried to read NEWS using Less in xterm, I can see all these
> quotes in UTF-8, for example, the quotes in “like this”.

I guess that's because your text terminal supports UTF-8.

> Every symbol in NEWS quoted by either ‘straight’ quoting style 'like this',
> or ‘grave’ `like this'.  For example, for 'electric-quote-replace-double'
> starting Isearch with ‘M-s .’, and repeating the same symbol search in the
> source code.

I'm asking what us the problem not to use "M-s ." in those cases, and
instead to look for the symbol as a literal string.  I think in a file
like NEWS this shouldn't yield too many false hits.

bug#31231: Symbol search broken in NEWS and ChangeLogs

[Eli Zaretskii]

On April 24, 2018 5:33:30 AM GMT+03:00, Eli Zaretskii <eliz@gnu.org> wrote:
> > From: Juri Linkov <juri@linkov.net>
> > Cc: 31231@debbugs.gnu.org
> > Date: Mon, 23 Apr 2018 23:16:09 +0300
> > 
> > >> But how these text terminals display Unicode quotes in Info
> manuals?
> > >
> > > Info manuals are displayed by an Info reader, and each Info reader
> has
> > > its own solution for this problem (generally, substitution of
> ASCII
> > > characters for undisplayable Unicode ones).
> > 
> > Does the Emacs Info reader implement such solution?
> 
> Emacs itself does, not just in Info.  The standalone Info reader (part
> of the Texinfo package) has a slightly different solution.
> 
> > > I was talking about reading NEWS in a general-purpose
> text-browsing
> > > program, such as Less.  It is IMO not a good idea to prevent
> people
> > > from doing that and requiring them to only read NEWS and ChangeLog
> > > files in Emacs.
> > 
> > When I tried to read NEWS using Less in xterm, I can see all these
> > quotes in UTF-8, for example, the quotes in “like this”.
> 
> I guess that's because your text terminal supports UTF-8.
> 
> > Every symbol in NEWS quoted by either ‘straight’ quoting style 'like
> this',
> > or ‘grave’ `like this'.  For example, for
> 'electric-quote-replace-double'
> > starting Isearch with ‘M-s .’, and repeating the same symbol search
> in the
> > source code.
> 
> I'm asking what us the problem not to use "M-s ." in those cases, and
> instead to look for the symbol as a literal string.  I think in a file
> like NEWS this shouldn't yield too many false hits.

Ah, I see I've misunderstood.  You originally talked about "M-s _", not about
"M-s .".

Is "M-s ." the main/only problem?

bug#31231: Symbol search broken in NEWS and ChangeLogs

[Juri Linkov]

>> >> But how these text terminals display Unicode quotes in Info manuals?
>> >
>> > Info manuals are displayed by an Info reader, and each Info reader has
>> > its own solution for this problem (generally, substitution of ASCII
>> > characters for undisplayable Unicode ones).
>> 
>> Does the Emacs Info reader implement such solution?
>
> Emacs itself does, not just in Info.  The standalone Info reader (part
> of the Texinfo package) has a slightly different solution.

Then using the same solution as not just for Info, Emacs will automatically
substitute ASCII characters for undisplayable Unicode ones in all
other places like NEWS and ChangeLogs.

>> Every symbol in NEWS quoted by either ‘straight’ quoting style 'like this',
>> or ‘grave’ `like this'.  For example, for 'electric-quote-replace-double'
>> starting Isearch with ‘M-s .’, and repeating the same symbol search in the
>> source code.
>
> I'm asking what us the problem not to use "M-s ." in those cases, and
> instead to look for the symbol as a literal string.  I think in a file
> like NEWS this shouldn't yield too many false hits.
>
> Ah, I see I've misunderstood.  You originally talked about "M-s _", not about
> "M-s .".
>
> Is "M-s ." the main/only problem?

Not only ‘M-s .’, more so ‘M-s _’ where neither ‘M-s _ C-M-y’
nor ‘M-s _ C-M-w’ work when used at the beginning of the symbol.

It's a good habit to use symbol search to search symbols because
it reduces the likelihood of errors, so using it should be
encouraged in NEWS and ChangeLogs.

bug#31231: Symbol search broken in NEWS and ChangeLogs

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: 31231@debbugs.gnu.org
> Date: Tue, 24 Apr 2018 22:16:16 +0300
> 
> > Emacs itself does, not just in Info.  The standalone Info reader (part
> > of the Texinfo package) has a slightly different solution.
> 
> Then using the same solution as not just for Info, Emacs will automatically
> substitute ASCII characters for undisplayable Unicode ones in all
> other places like NEWS and ChangeLogs.

You for some reason elided the part where I explained why an
Emacs-only solution is unacceptable.

> > Is "M-s ." the main/only problem?
> 
> Not only ‘M-s .’, more so ‘M-s _’ where neither ‘M-s _ C-M-y’
> nor ‘M-s _ C-M-w’ work when used at the beginning of the symbol.
> 
> It's a good habit to use symbol search to search symbols because
> it reduces the likelihood of errors, so using it should be
> encouraged in NEWS and ChangeLogs.

I think you are trying to use features that make no sense in
text-based modes.  Just because the text mentions symbols quoted in
some way doesn't yet mean that text is like a program source.

Maybe we should have separate (sub)commands for when symbols are
mentioned within human-readable text.  Tweaking what a word is due to
this is a too large hammer, IMO.

bug#31231: Symbol search broken in NEWS and ChangeLogs

[Eli Zaretskii]

> Date: Wed, 25 Apr 2018 05:31:12 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 31231@debbugs.gnu.org
> 
> Maybe we should have separate (sub)commands for when symbols are
> mentioned within human-readable text.

Another possibility would be to redefine what the various "M-s"
subcommands do in NEWS and in ChangeLog files.

bug#31231: Symbol search broken in NEWS and ChangeLogs

[Juri Linkov]

>> > Emacs itself does, not just in Info.  The standalone Info reader (part
>> > of the Texinfo package) has a slightly different solution.
>>
>> Then using the same solution as not just for Info, Emacs will automatically
>> substitute ASCII characters for undisplayable Unicode ones in all
>> other places like NEWS and ChangeLogs.
>
> You for some reason elided the part where I explained why an
> Emacs-only solution is unacceptable.

You said that using Unicode quotes will make it harder to view the files
on text terminals that don't support UTF-8.  This means that we have
to find another solution.  Maybe just removing quote characters is enough.
If not, then modifying the syntax of ‘'’ to ‘expression quote’ looks good
since it will correctly handle spell-checking words with an apostrophe
with this patch that solves the problem for ChangeLogs:

diff --git a/lisp/vc/add-log.el b/lisp/vc/add-log.el
index 4c64ae1..629470a 100644
--- a/lisp/vc/add-log.el
+++ b/lisp/vc/add-log.el
@@ -1024,6 +1024,12 @@ change-log-indent
 (defvar smerge-resolve-function)
 (defvar copyright-at-end-flag)
 
+(defvar change-log-mode-syntax-table
+  (let ((st (make-syntax-table)))
+    (modify-syntax-entry ?' "'   " st)
+    st)
+  "Syntax table used while in `change-log-mode'.")
+
 ;;;###autoload
 (define-derived-mode change-log-mode text-mode "Change Log"
   "Major mode for editing change logs; like Indented Text mode.

>> It's a good habit to use symbol search to search symbols because
>> it reduces the likelihood of errors, so using it should be
>> encouraged in NEWS and ChangeLogs.
>
> I think you are trying to use features that make no sense in
> text-based modes.  Just because the text mentions symbols quoted in
> some way doesn't yet mean that text is like a program source.

NEWS and ChangeLogs are not just conventional prose, they are
technical writings with symbol references, so they should be
handled exactly the same way as Emacs Info manuals, docstrings
and code comments.

> Maybe we should have separate (sub)commands for when symbols are
> mentioned within human-readable text.  Tweaking what a word is due to
> this is a too large hammer, IMO.

Maybe creating a new minor mode for NEWS files that in addition
to modifying the syntax of quotes will also highlight quoted symbols
with a special face like is used currently in Info.

bug#31231: Symbol search broken in NEWS and ChangeLogs

[Juri Linkov]

> Modifying the syntax of ‘'’ to ‘expression quote’ looks good since it
> will correctly handle spell-checking words with an apostrophe with
> this patch that solves the problem for ChangeLogs:

Pushed to master as bcee160.

> Maybe creating a new minor mode for NEWS files that in addition
> to modifying the syntax of quotes will also highlight quoted symbols
> with a special face like is used currently in Info.

I'm not sure if implementing such mode just for NEWS is worthwhile enough.

bug#31231: Symbol search broken in NEWS and ChangeLogs

[Lars Ingebrigtsen]

Juri Linkov <juri@linkov.net> writes:

>> Modifying the syntax of ‘'’ to ‘expression quote’ looks good since it
>> will correctly handle spell-checking words with an apostrophe with
>> this patch that solves the problem for ChangeLogs:
>
> Pushed to master as bcee160.
>
>> Maybe creating a new minor mode for NEWS files that in addition
>> to modifying the syntax of quotes will also highlight quoted symbols
>> with a special face like is used currently in Info.
>
> I'm not sure if implementing such mode just for NEWS is worthwhile enough.

Skimming this thread, it sounds like bcee160 fixed the reported bug?
And a mode for NEWS was deemed not worthwhile, so it seems like
everything was covered?  So I'm closing this bug report.  If this is
wrong, please reopen.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#31231: Symbol search broken in NEWS and ChangeLogs

[Phil Sainty]

On 2018-04-22 08:21, Juri Linkov wrote:
> This solution is not applicable to NEWS files where
> text-mode syntax tables are inherited via outline-mode
> whose syntax table can't be changed.

Of course a new derived major mode could be defined with such
a syntax table.

The NEWS files set their mode with a local variable, so that
could be trivially changed if need be.

(Not arguing for or against the change, though.)

bug#31231: Symbol search broken in NEWS and ChangeLogs

[Eli Zaretskii]

> Date: Mon, 23 Apr 2018 11:34:17 +1200
> From: Phil Sainty <psainty@orcon.net.nz>
> Cc: 31231@debbugs.gnu.org,
> 	bug-gnu-emacs <bug-gnu-emacs-bounces+psainty=orcon.net.nz@gnu.org>
> 
> On 2018-04-22 08:21, Juri Linkov wrote:
> > This solution is not applicable to NEWS files where
> > text-mode syntax tables are inherited via outline-mode
> > whose syntax table can't be changed.
> 
> Of course a new derived major mode could be defined with such
> a syntax table.
> 
> The NEWS files set their mode with a local variable, so that
> could be trivially changed if need be.

Technically, the solution is simple.  The problem is elsewhere, IMO.