jinx

jinx

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I saw the announcement about the jinx spelling package.  It sounds
very powerful.  Could it replace ispell.el?  If so, do people think
that would be a good idea?  If not, why not?

Same questions for flyspell.el.

What languages is jinx written in?  What does the libenchant library
do, and how does it combine or work with Emacs?  How does it combine
or work with aspell?

The description talks about "native modules" but doesn't say what they
are or what they do.  Can anyone tell us?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: jinx

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> I saw the announcement about the jinx spelling package.  It sounds
> very powerful.  Could it replace ispell.el?  If so, do people think
> that would be a good idea?  If not, why not?

I think it would be an idea worth considering, but that would mean that
Emacs would have another compile-time dependency.

> Same questions for flyspell.el.
>
> What languages is jinx written in?  

https://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/?h=externals/jinx

jinx is written as a Emacs module, so one part is Emacs Lisp and the
other one is C.

>                                     What does the libenchant library
> do, and how does it combine or work with Emacs?  How does it combine
> or work with aspell?

https://abiword.github.io/enchant/

"Enchant is a library (and command-line program) that wraps a number of
different spelling libraries and programs with a consistent
interface. By using Enchant, you can use a wide range of spelling
libraries, including some specialised for particular languages, without
needing to program to each library's interface."

ispell already supports enchant, but it communicates via IPC instead of
the library interface.

> The description talks about "native modules" but doesn't say what they
> are or what they do.  Can anyone tell us?

I am guessing that is just a typo, and Daniel means (elisp) Writing
Dynamic Modules.

-- 
Philip Kaludercic

Re: jinx

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > ispell already supports enchant, but it communicates via IPC instead of
  > the library interface.

Would it be possible for jinx to do this?
Would it be better overall for jinx to do this?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: jinx

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > ispell already supports enchant, but it communicates via IPC instead of
>   > the library interface.
>
> Would it be possible for jinx to do this?
> Would it be better overall for jinx to do this?

Do what?  Jinx /does/ use the enchant library, ispell /does/ use IPC.

-- 
Philip Kaludercic

Re: jinx

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >   > ispell already supports enchant, but it communicates via IPC instead of
  > >   > the library interface.
  > >
  > > Would it be possible for jinx to do this?
  > > Would it be better overall for jinx to do this?

  > Do what?

Communicate with enchant via IPC.

I understood the other person to be saying that communicating that way
is an advantage for ispell.  If it is an advantage, could jinx be made
to do it that way?

Maybe perse did not mean that this was an advantage.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: jinx

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 31 Mar 2023 23:11:57 -0400
> 
>   > >   > ispell already supports enchant, but it communicates via IPC instead of
>   > >   > the library interface.
>   > >
>   > > Would it be possible for jinx to do this?
>   > > Would it be better overall for jinx to do this?
> 
>   > Do what?
> 
> Communicate with enchant via IPC.
> 
> I understood the other person to be saying that communicating that way
> is an advantage for ispell.  If it is an advantage, could jinx be made
> to do it that way?
> 
> Maybe perse did not mean that this was an advantage.

AFAIU, the IPC interface that is bypassed is the one between Emacs and
Enchant, implemented in ispell.el.  The IPC that is NOT bypassed is
the one between Enchant and the speller used as its back-end: Aspell
or Hunspell.

Personally, I don't think this is a significant matter: the IPC
interface used by ispell.el is not slow, because we keep the
spell-checking process running at all times.  What might be slow is
the Flyspell's reliance on post-command-hook to spell-check as you
type: this slows down Emacs in some important scenarios.  But again:
if we want to improve that aspect, we don't need jinx, we should
redesign and reimplement how we pass words to the speller using more
efficient mechanisms, such as JIT font-lock hooks.  I believe there's
already an ELPA package which attempts to do something like that.

Re: jinx

[Peter Oliver]

On Sat, 1 Apr 2023, Eli Zaretskii wrote:

> AFAIU, the IPC interface that is bypassed is the one between Emacs and
> Enchant, implemented in ispell.el.  The IPC that is NOT bypassed is
> the one between Enchant and the speller used as its back-end: Aspell
> or Hunspell.

With libenchant (and hence Jinx), both IPCs are avoided for most spellchecking providers (including Aspell, Hunspell, Nuspell).

https://github.com/AbiWord/enchant/tree/master/providers

-- 
Peter Oliver

Re: jinx

[Eli Zaretskii]

> Date: Sat, 1 Apr 2023 13:43:05 +0100 (BST)
> From: Peter Oliver <p.d.oliver@mavit.org.uk>
> cc: emacs-devel@gnu.org
> 
> On Sat, 1 Apr 2023, Eli Zaretskii wrote:
> 
> > AFAIU, the IPC interface that is bypassed is the one between Emacs and
> > Enchant, implemented in ispell.el.  The IPC that is NOT bypassed is
> > the one between Enchant and the speller used as its back-end: Aspell
> > or Hunspell.
> 
> With libenchant (and hence Jinx), both IPCs are avoided for most spellchecking providers (including Aspell, Hunspell, Nuspell).
> 
> https://github.com/AbiWord/enchant/tree/master/providers

That's a two-edge sword: it means an Emacs built with that will be
wedded to a particular version of the speller.  I'm not sure this is a
good idea (but won't object to supporting such a build, if someone
wants that).

Spell-checking is an interactive process, which proceeds word by word,
and most of the time is taken by the user deciding which alternative
to choose.  So the speed of communicating with the spell-checker
should not by itself be an issue.

Re: jinx

[Peter Oliver]

[-- Attachment #1: Type: text/plain, Size: 785 bytes --]

On Sat, 1 Apr 2023, Eli Zaretskii wrote:

>> With libenchant (and hence Jinx), both IPCs are avoided for most spellchecking providers (including Aspell, Hunspell, Nuspell).
>>
>> https://github.com/AbiWord/enchant/tree/master/providers
>
> That's a two-edge sword: it means an Emacs built with that will be
> wedded to a particular version of the speller.  I'm not sure this is a
> good idea (but won't object to supporting such a build, if someone
> wants that).

I’d be fine with that, since I get all of this pre-built from my distribution.

But anyway, I’m not sure that that’s true.  If you want to take a major upgrade to your spellchecker, I would expect it to be sufficient to update its dynamically loaded Enchant plugin, not everything that uses libenchant.

-- 
Peter Oliver

Re: jinx

[Augusto Stoffel]

On Fri, 31 Mar 2023 at 23:11, Richard Stallman wrote:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > >   > ispell already supports enchant, but it communicates via IPC instead of
>   > >   > the library interface.
>   > >
>   > > Would it be possible for jinx to do this?
>   > > Would it be better overall for jinx to do this?
>
>   > Do what?
>
> Communicate with enchant via IPC.
>
> I understood the other person to be saying that communicating that way
> is an advantage for ispell.  If it is an advantage, could jinx be made
> to do it that way?
>
> Maybe perse did not mean that this was an advantage.

It is totally feasible to write a decent spell-checking package relying
just on IPC, but it would be very helpful (for this an many other
purposes) to optimize Emacs's subprocess communication routines, which
are clearly pretty slow.

In fact, jinx is a fork of my (also recent) jit-spell package, which
communicates with the spellchecker through pipes using the `ispell -a'
interface.  Jinx was created because its author experienced some latency
issues while using jit-spell.  I was unable to reproduce his
observations, although I do observe relatively high CPU usage while
typing with jit-spell mode on.

Re: jinx

[Michael Eliachevitch]

[-- Attachment #1: Type: text/plain, Size: 3548 bytes --]


The technical questions where answered by Philip. I tried the package today and can comment from a user perspective:

> I saw the announcement about the jinx spelling package.  It sounds
> very powerful.  Could it replace ispell.el?  If so, do people think
> that would be a good idea?  If not, why not?
>
> Same questions for flyspell.el.

These packages are used in different ways and may complement each other.

jinx spell-checks the visible text and underlines errors. It's very fast at finding errors due to focusing on the visible text and using the libenchant API instead of subprocess calls. But it currently lacks the ability to check the entire buffer and e.g. jump to the next error or list all errors.

In its README, the jinx author writes that flyspell only checks the work under point. However, it doesn't mention that one can run flyspell-buffer (or flycheck-region) to check the entire buffer for errors, so that's not a huge issue. But jinx just feels more snappy in that regard, incorrect are just highlighted immediately without having to do anything except having the mode enabled.

But as far as I know it doesn't have yet the ability to check and list the spelling mistakes in the entire buffer. Therefore, I will probably still be using ispell-buffer to iterate through all spelling errors.

What I personally enjoy most about jinx are the ergonomics of being able to exclude certain syntactic categories from spell-checking. One convenient way are  `jinx-include-faces' and `jinx-exclude-faces', which are alists of major modes to faces to include/ignore. Also there's `jinx-exclude-regexps' which is an alist from major modes to regexes. For example made it easy for me to get rid of false-positives on macros in LaTeX [fn 1]. Some spell-checking programs (aspell, hunspell) have LaTeX-detection built-in, but it doesn't always work reliably and is not as easily configurable. And that functionality doesn't extend to other markup languages. This is also useful for e.g. org-mode or checking comments and docstrings in programming languages.


Ergonomic-wise I also enjoyed that `jinx-languages', which configures the language, can be a list of language (locale) strings (or just a single string).. I'll use that for my private notes, which tend to be a mix of German and English. `ispell-dictionary' in comparison can only be a single string. (Though through online search I found that when used with hunspell one can provide a comma-seperated list of locales to ispell-dictionary and hunspell will recognise that).

I feel like from a user-perspective, jinx, possibly extended, could replace most use-cases of flyspell. Alternatively, one could also try to add a feature to make flyspell always check the visible text in the buffer, which would need to happen asynchronously I don't know whether technically that would be possible  One could also consider trying to port some of its other ideas back to the existing spell-checking packages in emacs, like using mode-alists for the faces-configuration.

[fn 1]: While writing the mail I looked into the flyspell documentation and that is has similar features via `flyspell-prog-text-faces' and `flyspell-tex-command-regexp', but those are less flexible. `flyspell-prog-text-faces' doesn't work for modes that do not inherit from prog-mode and I can't configure it on a mode-specific granularity, except by using major-mode-hooks and dir- or file-local variables.

--
Michael Eliachevitch
Public PGP Key: https://keyoxide.org/hkp/546908c782383ad0e7d894ec1b8f95c8125dce31

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

Re: jinx

[João Pedro]

Hey Michael,

On Wed, March 29 2023 22:46, Michael Eliachevitch <m.eliachevitch@posteo.de> wrote:

> But it currently lacks the ability to check the entire buffer

Little correction here. It does have a "check the entire buffer"
functionality, just call jinx-correct with a universal-argument (C-u).

Best regards,

-- 
João Pedro de A. Paula

Re: jinx

[Eli Zaretskii]

> From: Michael Eliachevitch <m.eliachevitch@posteo.de>
> Cc: emacs-devel@gnu.org
> Date: Wed, 29 Mar 2023 22:46:01 +0000
> 
> [fn 1]: While writing the mail I looked into the flyspell documentation and that is has similar features via `flyspell-prog-text-faces' and `flyspell-tex-command-regexp', but those are less flexible. `flyspell-prog-text-faces' doesn't work for modes that do not inherit from prog-mode and I can't configure it on a mode-specific granularity, except by using major-mode-hooks and dir- or file-local variables.

It should be easy to extend these flyspell features to be more
flexible.  Patches welcome.

Re: jinx

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > But as far as I know it doesn't have yet the ability to check and
  > list the spelling mistakes in the entire buffer. Therefore, I will
  > probably still be using ispell-buffer to iterate through all
  > spelling errors.

It was pointed out that there is a way to do this.  Of course, if we
install it in Emacs we could adjust the calling conventions however we
think users will like.  We could make `ispell-buffer' call jinx
if that's what we want.

Given that, do you still think you would have a need for the existing
`ispell-buffer' that uses ispell.el, if jinx were always available?


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: jinx

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 31 Mar 2023 00:29:23 -0400
> 
>   > But as far as I know it doesn't have yet the ability to check and
>   > list the spelling mistakes in the entire buffer. Therefore, I will
>   > probably still be using ispell-buffer to iterate through all
>   > spelling errors.
> 
> It was pointed out that there is a way to do this.  Of course, if we
> install it in Emacs we could adjust the calling conventions however we
> think users will like.  We could make `ispell-buffer' call jinx
> if that's what we want.
> 
> Given that, do you still think you would have a need for the existing
> `ispell-buffer' that uses ispell.el, if jinx were always available?

We cannot assume that "jinx is always available", since it requires
Emacs to be linked against the enchant library.

Anyway, after looking at jinx, I must say that I don't see why we
should replace ispell and flyspell with jinx.  From my POV, ispell and
flyspell provide all the required spell-checking functionalities:

  . spell-check a word
  . spell-check a region
  . spell-check the entire buffer
  . spell-check comments and strings in a prog-mode buffer
  . spell-check as you type

jinx spell-checks the visible portion of the window.  I don't see why
we would need that.  Suppose I scroll the window by 2 lines: why would
I want the new visible portion to be spell-checked again, when the
overlap with the last check is so large?  In any case, if someone
wants a handy command to spell-check only the part that is currently
visible in the window, we can add such a command to ispell.el.  (Btw,
jinx relies on window-end for this, and window-end is a problematic
function which cannot be trusted.)

Another feature of jinx is the ability to skip some buffer text, e.g.,
relying on font-lock faces or user-defined regexps to identify the
parts that should be skipped.  We have similar features:
spell-checking of comments and strings, but if that is not enough, we
could extend these capabilities in ispell and flyspell by simple
enough additions.

All those are minor extensions of the existing spell-checking
capabilities that require no deep changes in the infrastructure and
documentation, which have been stable and useful for many years.

Having jinx as an ELPA package is fine, of course.  Likewise, it's
fine if someone wants to work on redesigning and reimplementing the
low level of ispell.el so that it could optionally use libenchant if
that is available.  But I see no reason to throw out the window the
whole spelling infrastructure we have and replacing it with what jinx
provides.  That would require not just coding and testing, but also
rewriting the manual and will require people to re-learn how they use
spell-checking features in Emacs.  I see no justification for that,
especially since jinx doesn't bring any new spell-checking technology
to Emacs -- libenchant uses those same spell-checking programs, aspell
and hunspell, that we have been using for ages.

As an aside, a technology I would like to see in Emacs is grammar
checking.  Last I looked, there are no free grammar-checking libraries
or services out there, even just for the English language.  But if
that's no longer true, then adding such capabilities to Emacs will be
very welcome, I think, since word-based spell-checking is clearly not
enough these days.

Re: jinx

[Gregory Heytings]

>
> As an aside, a technology I would like to see in Emacs is grammar 
> checking.  Last I looked, there are no free grammar-checking libraries 
> or services out there, even just for the English language.  But if 
> that's no longer true, then adding such capabilities to Emacs will be 
> very welcome, I think, since word-based spell-checking is clearly not 
> enough these days.
>

I think that's no longer true, LanguageTool is available (and has been for 
~20 years): https://github.com/languagetool-org/languagetool .  But it's 
written in Java...

Re: Grammar checking (was: jinx)

[Eli Zaretskii]

> Date: Fri, 31 Mar 2023 07:10:11 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: rms@gnu.org, m.eliachevitch@posteo.de, emacs-devel@gnu.org
> 
> > As an aside, a technology I would like to see in Emacs is grammar 
> > checking.  Last I looked, there are no free grammar-checking libraries 
> > or services out there, even just for the English language.  But if 
> > that's no longer true, then adding such capabilities to Emacs will be 
> > very welcome, I think, since word-based spell-checking is clearly not 
> > enough these days.
> 
> I think that's no longer true, LanguageTool is available (and has been for 
> ~20 years): https://github.com/languagetool-org/languagetool .  But it's 
> written in Java...

Thanks.  Maybe someone will figure out how to integrate this with
Emacs.

Re: Grammar checking

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Fri, 31 Mar 2023 07:10:11 +0000
>> From: Gregory Heytings <gregory@heytings.org>
>> cc: rms@gnu.org, m.eliachevitch@posteo.de, emacs-devel@gnu.org
>> 
>> > As an aside, a technology I would like to see in Emacs is grammar 
>> > checking.  Last I looked, there are no free grammar-checking libraries 
>> > or services out there, even just for the English language.  But if 
>> > that's no longer true, then adding such capabilities to Emacs will be 
>> > very welcome, I think, since word-based spell-checking is clearly not 
>> > enough these days.
>> 
>> I think that's no longer true, LanguageTool is available (and has been for 
>> ~20 years): https://github.com/languagetool-org/languagetool .  But it's 
>> written in Java...
>
> Thanks.  Maybe someone will figure out how to integrate this with
> Emacs.

There is an implementation[0], but I found it to be awkward to work
with.

Looking around, I found a second implementation[1], but I cannot comment
on how well it works.

One of the main annoyances is that LanguageTool does not appear to be
widely packaged[2].

[0] https://github.com/mhayashi1120/Emacs-langtool/
[1] https://github.com/PillFall/languagetool.el
[2] https://repology.org/project/languagetool/versions

-- 
Philip Kaludercic

Re: Grammar checking

[Gregory Heytings]

>
> One of the main annoyances is that LanguageTool does not appear to be 
> widely packaged[2].
>

Why would it be necessary to package it?  It's a Java app, you can 
download it from here:

https://languagetool.org/download/LanguageTool-stable.zip

and run it everywhere (provided you have a JVM installed, of course).

Re: Grammar checking

[Philip Kaludercic]

Gregory Heytings <gregory@heytings.org> writes:

>> One of the main annoyances is that LanguageTool does not appear to
>> be widely packaged[2].
>
> Why would it be necessary to package it?  It's a Java app, you can
> download it from here:
>
> https://languagetool.org/download/LanguageTool-stable.zip
>
> and run it everywhere (provided you have a JVM installed, of course).

I am not saying it is necessary, just that it would be more convenient
if I wouldn't have to manually set it up and keep it up to date.

-- 
Philip Kaludercic

Re: Grammar checking

[Gregory Heytings]

>> Why would it be necessary to package it?  It's a Java app, you can 
>> download it from here:
>>
>> https://languagetool.org/download/LanguageTool-stable.zip
>>
>> and run it everywhere (provided you have a JVM installed, of course).
>
> I am not saying it is necessary, just that it would be more convenient 
> if I wouldn't have to manually set it up and keep it up to date.
>

That's true.  I guess the reason why distros package only a few Java apps 
is that the "added value" would be minimal, and the cost non-negligible 
(bandwidth and disk space).  E.g. Debian packages only about 100 Java 
apps, and if you subtract from that number the compilers, parser 
generators, linters, build tools, etc., the number is probably about 30.

Re: Grammar checking

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 702 bytes --]

On Fri, Mar 31, 2023, 4:10 AM Gregory Heytings <gregory@heytings.org> wrote:

>
> >
> > One of the main annoyances is that LanguageTool does not appear to be
> > widely packaged[2].
> >
>
> Why would it be necessary to package it?  It's a Java app, you can
> download it from here:
>
> https://languagetool.org/download/LanguageTool-stable.zip
>
> and run it everywhere (provided you have a JVM installed, of course).
>

Do you use it?  It appears to be designed as a full-blown, multi-connection
server.  Even if it is restricted to a local address, if it were a
multi-user machine I'd be concerned about whether other users might gain
access to the text I've sent to the server for processing.

Lynn

[-- Attachment #2: Type: text/html, Size: 1440 bytes --]

Re: Grammar checking

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 452 bytes --]


>> https://languagetool.org/download/LanguageTool-stable.zip
> 
> It appears to be designed as a full-blown, multi-connection server.  
> Even if it is restricted to a local address, if it were a multi-user 
> machine I'd be concerned about whether other users might gain access to 
> the text I've sent to the server for processing.
>

The zip file above contains the three versions of the program: GUI, 
client-server, and command-line.

Re: Grammar checking

[Peter Oliver]

[-- Attachment #1: Type: text/plain, Size: 745 bytes --]

On Fri, 31 Mar 2023, Gregory Heytings wrote:

>>>  https://languagetool.org/download/LanguageTool-stable.zip
>>
>>  It appears to be designed as a full-blown, multi-connection server.  Even
>>  if it is restricted to a local address, if it were a multi-user machine
>>  I'd be concerned about whether other users might gain access to the text
>>  I've sent to the server for processing.

If it leaked text between requests, that would be a bug.  But yes, it would be tidier if it was possible to communicate with the server via, say, a socket.

> The zip file above contains the three versions of the program: GUI, 
> client-server, and command-line.

The command-line tool takes too long to start to be of practical use to us.

-- 
Peter Oliver

Re: Grammar checking

[Philip Kaludercic]

Peter Oliver <p.d.oliver@mavit.org.uk> writes:

>> The zip file above contains the three versions of the program: GUI,
>> client-server, and command-line.
>
> The command-line tool takes too long to start to be of practical use to us.

Why?  If it were to be used asynchronously, it could start up in the
background and the results would be displayed when it finds something?

-- 
Philip Kaludercic

Re: Grammar checking

[Peter Oliver]

[-- Attachment #1: Type: text/plain, Size: 722 bytes --]

On Fri, 31 Mar 2023, Philip Kaludercic wrote:

> Peter Oliver <p.d.oliver@mavit.org.uk> writes:
>
>>> The zip file above contains the three versions of the program: GUI,
>>> client-server, and command-line.
>>
>> The command-line tool takes too long to start to be of practical use to us.
>
> Why?  If it were to be used asynchronously, it could start up in the
> background and the results would be displayed when it finds something?

On my system, “java -jar languagetool-commandline.jar /dev/null” takes 15 CPU-seconds in 4 real seconds.  The noise from my CPU fan alone would be enough for me to not want to run that all the time.

If we use the server, we only have to pay the startup cost once.

-- 
Peter Oliver

Re: Grammar checking

[Peter Oliver]

On Fri, 31 Mar 2023, Gregory Heytings wrote:

>>  One of the main annoyances is that LanguageTool does not appear to be
>>  widely packaged[2].
>
> Why would it be necessary to package it?

If a user has to jump through hoops to download, install, and configure it, could we honestly claim that the functionality was part of Emacs?  If users are willing to do all that, I would expect them to be happy to use one of the existing third-party Emacs packages for the task.

-- 
Peter Oliver

Re: Grammar checking

[Gregory Heytings]

>>> One of the main annoyances is that LanguageTool does not appear to be 
>>> widely packaged[2].
>> 
>> Why would it be necessary to package it?
>
> If a user has to jump through hoops to download, install, and configure 
> it, could we honestly claim that the functionality was part of Emacs?
>

A functionality of Emacs would depend on an externally installed tool, 
that's all.  Just like M-x ispell depends on an externally installed tool. 
And a user does not have to "jump through hoops to download, install, and 
configure it", it's a single (but large: 220 MB) zip file to download and 
unzip somewhere.  There is also an optional per-language ngram data set, 
which is much bigger (8.5 GB for English) but not available for all 
languages that LanguageTool supports.

Re: Grammar checking

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: Gregory Heytings <gregory@heytings.org>,  rms@gnu.org,
>   m.eliachevitch@posteo.de,  emacs-devel@gnu.org
> Date: Fri, 31 Mar 2023 07:47:15 +0000
> 
> There is an implementation[0], but I found it to be awkward to work
> with.

Can you share more details? what is "awkward" about it?

In any case, we don't have to use the Emacs interface as-is, we could
write our own.

Re: Grammar checking

[Peter Oliver]

On Fri, 31 Mar 2023, Eli Zaretskii wrote:

> In any case, we don't have to use the Emacs interface as-is, we could
> write our own.

There are already five LanguageTool integrations to choose from on MELPA.  I use flycheck-languagetool (for no other reason than because I happen to already use flycheck for other things).

https://melpa.org/#/?q=languagetool

-- 
Peter Oliver

Re: Grammar checking

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: Gregory Heytings <gregory@heytings.org>,  rms@gnu.org,
>>   m.eliachevitch@posteo.de,  emacs-devel@gnu.org
>> Date: Fri, 31 Mar 2023 07:47:15 +0000
>> 
>> There is an implementation[0], but I found it to be awkward to work
>> with.
>
> Can you share more details? what is "awkward" about it?

From what I recall, getting the server to run and recognise additional
grammar-rule databases like German wasn't that easy (though I might have
also just messed something up).  That if all I had to do was run

     $ apt install language-tool language-tool-de

that this part would have been easier.  The Emacs package did the job,
but had a ispell instead of a flyspell-like interface.  Better than
nothing, but a fly... interface would be nice to have, if it is
technically possible. 

> In any case, we don't have to use the Emacs interface as-is, we could
> write our own.

-- 
Philip Kaludercic

Re: Grammar checking

[Nasser Alkmim]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Fri, 31 Mar 2023 07:10:11 +0000
>> From: Gregory Heytings <gregory@heytings.org>
>> cc: rms@gnu.org, m.eliachevitch@posteo.de, emacs-devel@gnu.org
>> 
>> > As an aside, a technology I would like to see in Emacs is grammar 
>> > checking.  Last I looked, there are no free grammar-checking libraries 
>> > or services out there, even just for the English language.  But if 
>> > that's no longer true, then adding such capabilities to Emacs will be 
>> > very welcome, I think, since word-based spell-checking is clearly not 
>> > enough these days.
>> 
>> I think that's no longer true, LanguageTool is available (and has been for 
>> ~20 years): https://github.com/languagetool-org/languagetool .  But it's 
>> written in Java...
>
> Thanks.  Maybe someone will figure out how to integrate this with
> Emacs.
>
>

I tried a couple of grammar checkers in Emacs, integrated via Flymake and Eglot.
The one that works best for me is this Grammarly language server [1].
Best in terms of diagnostics.
Some users may find it too intrusive, so it may be a personal preference thing.
However, it is not usable for larger buffers because it is too slow.
Maybe some narrowing functionality can help.
Also, there may be other implications regarding using Grammarly.
The LanguageTool language server is available as well, but It is not as effective as Grammarly to catch mistakes and offer suggestions.

[0] https://github.com/emacs-languagetool/flymake-languagetool
[1] https://github.com/emacs-grammarly/eglot-grammarly

-- 
Nasser Alkmim 
 +43 677 6408 9171

Re: Grammar checking

[Michael Eliachevitch]

[-- Attachment #1: Type: text/plain, Size: 2711 bytes --]


> I tried a couple of grammar checkers in Emacs, integrated via Flymake and Eglot.
> The one that works best for me is this Grammarly language server [1].
> Best in terms of diagnostics.
> Some users may find it too intrusive, so it may be a personal preference thing.
> However, it is not usable for larger buffers because it is too slow.
> Maybe some narrowing functionality can help.
> Also, there may be other implications regarding using Grammarly.
> The LanguageTool language server is available as well, but It is not as
> effective as Grammarly to catch mistakes and offer suggestions.
>
> [0] https://github.com/emacs-languagetool/flymake-languagetool
> [1] https://github.com/emacs-grammarly/eglot-grammarly

Most recently I also reverted to grammar checking via LSP servers with Eglot. Grammarly is powerful an I like knowing that it's powered by Common Lisp, but due to its proprietary nature it's not something that fits well with the philosophy of Emacs and the FSF.

I'm checking a LaTeX-file, where I found the "ltex" [3] language server to be useful [3]. It's basically a Languagetool language server, but aware of markdown and LaTeX syntax. But in practice it doesn't seem to do anything much smarter than just ignoring certain syntactic commands or macros. So plain Languagetool might be sufficient if one could configure it with text-faces like in `flyspell-prog-mode' or faces to ignore.

One issue with Eglot is that afaik it currently only allows for one language server to be active at a time, so if I want to check the grammar I need to switch the active LSP server. I think it was mentioned as a possible feature in the future to allow different simultaneous language servers for different LSP capabilities, maybe one main one for completion and thinks like rename and two LSP's for linting, for instance. I don't know how far that is, but I'm aware that Eglot development is very active and there are many other issues.

Writing text is one of the core functionalities of an editor and therefore I think that grammar checking could be integrated more tightly than just relying on Eglot. And as I just discussed it's something that might used in parallel to a language server. However, the strength of using LSP servers for linting is that it's easy to change grammar checking LSP servers without _any_ additional work required on Emacs side. So I'm not sure if coupling Emacs tightly to Languagetool and not allowing for alternatives is the way to go.

[3]: https://valentjn.github.io/ltex. Actually there's even a special package for that https://github.com/emacs-languagetool/eglot-ltex, but in my opinion that's not really necessary, it just adds some custom eglot configuration.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

Re: Grammar checking

[Felician Nemeth]

> One issue with Eglot is that afaik it currently only allows for one
> language server to be active at a time, so if I want to check the
> grammar I need to switch the active LSP server. I think it was
> mentioned as a possible feature in the future to allow different
> simultaneous language servers for different LSP capabilities, maybe
> one main one for completion and thinks like rename and two LSP's for
> linting, for instance.

Instead of enhancing Eglot to work with multiple LSP servers in
parallel, we talked in the past about the idea of an
lsp-proxy/lsp-multiplexer that is independent of Eglot, shows itself as
an ordinary LSP client to Eglot, but also starts multiple LSP severs and
behaves as a client to these servers.

Now I'm only able to find these pointers about this:
  https://github.com/joaotavora/eglot/issues/249#issuecomment-482561502
  https://github.com/joaotavora/eglot/issues/90#issuecomment-443906068

This approach would keep Eglot relatively simple, and may result in a
project that cloud also attract contributors outside of the Emacs
community.

Re: Grammar checking

[Peter Oliver]

On Fri, 31 Mar 2023, Michael Eliachevitch wrote:

> I'm checking a LaTeX-file, where I found the "ltex" [3] language server to be 
> useful [3].

Also worth a mention is YaLafi, which is a program that sits in between a LanguageTool client and server, filtering out LaTeX markup so that LangugageTool just sees the text.  https://github.com/torik42/YaLafi

-- 
Peter Oliver

Re: Grammar checking

[Gregory Heytings]

>
> The one that works best for me is this Grammarly language server [1].
>

It might give better results, but it's proprietary software.

Re: Grammar checking

[Lynn Winebarger]

On Fri, Mar 31, 2023 at 4:49 AM Gregory Heytings <gregory@heytings.org> wrote:
> > The one that works best for me is this Grammarly language server [1].
> It might give better results, but it's proprietary software.

LanguageTool isn't proprietary software, but it is used for SaaSS
where the "premium" tier appears to use rule sets that are not
released with the base tool.  LanguageTool.org is very clear in using
the term "open source" rather than "free" to describe the software
(and configuration files) for its "Basic" service.

Given discussions that have happened in the last few months on this
list, I'm surprised the software maintained by the languagetool
project would be considered suitable for introducing a dependency into
core emacs.  Actually, I'm surprised that sort of dependency would be
ok for a package included in GNU ELPA, or possibly even non-GNU ELPA.

I'm not personally a stickler for using free-vs-OS, but even with my
promiscuous package usage I avoided installing the languagetool
package /  I would be concerned that emacs might start communicating
to languagetool servers by default unless I took extreme care. in
reviewing the package code and committed to doing so for every update.
I'm sure any functionality built into core emacs wouldn't do that, but
I don't know that I would trust the languagetool software project to
never introduce such transactions at some future point.

Lynn

Re: Grammar checking

[Gregory Heytings]

>>> The one that works best for me is this Grammarly language server [1].
>>
>> It might give better results, but it's proprietary software.
>
> LanguageTool isn't proprietary software,
>

It's Grammarly that is proprietary software.

>
> but it is used for SaaSS
>

The free version does not communicate with the LanguageTool servers.

>
> where the "premium" tier appears to use rule sets that are not released 
> with the base tool.
>

Yes, for an additional cost you can use an extended set of rules.  But the 
free version works (and you can add your own set of rules if you want).

>
> Given discussions that have happened in the last few months on this 
> list, I'm surprised the software maintained by the languagetool project 
> would be considered suitable for introducing a dependency into core 
> emacs.
>

LanguageTool's license is the LGPL, why would it wrong to use it?

Re: Grammar checking

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Sat, 1 Apr 2023 08:59:50 -0400
> Cc: Nasser Alkmim <nasser.alkmim@gmail.com>, Eli Zaretskii <eliz@gnu.org>, rms@gnu.org, 
> 	m.eliachevitch@posteo.de, emacs-devel@gnu.org
> 
> On Fri, Mar 31, 2023 at 4:49 AM Gregory Heytings <gregory@heytings.org> wrote:
> > > The one that works best for me is this Grammarly language server [1].
> > It might give better results, but it's proprietary software.
> 
> LanguageTool isn't proprietary software, but it is used for SaaSS
> where the "premium" tier appears to use rule sets that are not
> released with the base tool.  LanguageTool.org is very clear in using
> the term "open source" rather than "free" to describe the software
> (and configuration files) for its "Basic" service.

I don't understand what you are trying to say.  LanguageTool is
distributed with its sources, and the code is under LGPL 2.  The
"premium" version is not what we are considering.  So where is the
problem with that?

> Given discussions that have happened in the last few months on this
> list, I'm surprised the software maintained by the languagetool
> project would be considered suitable for introducing a dependency into
> core emacs.

We are discussing a possibility of using LanguageTool as an external
tool with which Emacs will communicate, not as a dependency for the
Emacs core.

Re: Grammar checking

[Lynn Winebarger]

On Sat, Apr 1, 2023 at 9:37 AM Eli Zaretskii <eliz@gnu.org> wrote:
> > From: Lynn Winebarger <owinebar@gmail.com>
> > Date: Sat, 1 Apr 2023 08:59:50 -0400
> > Cc: Nasser Alkmim <nasser.alkmim@gmail.com>, Eli Zaretskii <eliz@gnu.org>, rms@gnu.org,
> >       m.eliachevitch@posteo.de, emacs-devel@gnu.org
> >
> > On Fri, Mar 31, 2023 at 4:49 AM Gregory Heytings <gregory@heytings.org> wrote:
> > > > The one that works best for me is this Grammarly language server [1].
> > > It might give better results, but it's proprietary software.
> >
> > LanguageTool isn't proprietary software, but it is used for SaaSS
> > where the "premium" tier appears to use rule sets that are not
> > released with the base tool.  LanguageTool.org is very clear in using
> > the term "open source" rather than "free" to describe the software
> > (and configuration files) for its "Basic" service.
>
> I don't understand what you are trying to say.  LanguageTool is
> distributed with its sources, and the code is under LGPL 2.  The
> "premium" version is not what we are considering.  So where is the
> problem with that?
>
> > Given discussions that have happened in the last few months on this
> > list, I'm surprised the software maintained by the languagetool
> > project would be considered suitable for introducing a dependency into
> > core emacs.
>
> We are discussing a possibility of using LanguageTool as an external
> tool with which Emacs will communicate, not as a dependency for the
> Emacs core.

I'll respond to this point first, then the question above, so the
context is clear.

I might have misunderstood - this thread arose because you mentioned
that grammar correction was a higher priority than reimplementing
spell-checking support, so I took it to mean the support would be
included in core in the same way spell-c,hecker support is
incorporated in the core, even though the actual spell checking is
dependent on an external tool being available.

Re "So where is the problem with that"?  There may not be one - it's
possible I'm misunderstanding the implications of the GNU coding
standards.  It seems to me there are three potential issues:

1) Violations of
https://www.gnu.org/prep/standards/html_node/References.html#References
if it is read to include references to SaaSS as equivalent to
references to nonfree software.
2)  Long term implications with regard to the set of grammar rules
emacs encourages using
3)  Concerns about trademarks and confusion as to whether
"languagetool" is intended to reference the software or the product -
not just legally, but with end users

Regarding (1):  Based on previous discussions on this list and
https://www.gnu.org/philosophy/who-does-that-server-really-serve.html
, I am extrapolating that the prohibition against linking to websites
that promote non-free software also apply to websites that promote
SaaSS.  In particular:

[[ BEGIN QUOTE ]]
What about chains of links? Following links from nearly any web site
can lead eventually to promotion of non-free software; this is
inherent in the nature of the web. Here’s how we treat that.

You should not refer to AT&T’s web site if that recommends AT&T’s
non-free software packages; you should not refer to a page p that
links to AT&T’s site presenting it as a place to get some non-free
program, because that part of the page p itself recommends and
legitimizes the non-free program.

However, if p contains a link to AT&T’s web site for some other
purpose (such as long-distance telephone service), that is no reason
you should not link to p.
[[ END QUOTE ]]

In this case, the language tool repository page clearly refers to the
languagetool.org website.  Furthermore, the basic grammar rules that
are free are used as an enticement to consider the SaaSS.  On
languagetool.org's webpage, there is a FAQ, which includes the Q&A:

[[ BEGIN QUOTE ]]
Is LanguageTool free?
LanguageTool is available in two versions. The free version corrects
spelling as well as simple punctuation and some style mistakes. Only
the Premium version will show you all errors and give you the best
possible text. Team accounts for companies are also available.
[[ END QUOTE ]]
and then links https://languagetool.org/insights/post/premium-vs-free-product-updates/
with the text "Read more"

So, it would seem to me referencing the languagetool git repository
would violate the general prohibition.  Of course, someone could fork
the project to provide the source in a way that excises all such
references.

Regarding (2):  Based on the text I just quoted about the
(intentional) limitations of the free rule set, it seems that if emacs
encourages the use of (local) languagetool servers, users will
inevitably want to improve its performance.  There are at least two
ways that can play out:

(a) that activity go into the repo languagetool.org maintains for
"community" rulesets under the LGPL, so they may be included in
languagetool.org's SaaSS
(b) emacs developers/users establish a separate repository distributed
under a copyleft, probably Affero-type, license to prevent their
inclusion in languagetool.org's SaaSS

I'm sure there are other ways, but these are the primary ones I can
foresee.  I guess my question is whether that is a position the GNU
project is content to put its users in, especially as many users will
probably be content with (a) or simply subscribing to the Premium
service.

(3)  Presumably the trademark confusion can be worked around, it would
simply take some care, i.e. be a slight headache.

For me, this is primarily an intellectual exercise as I am not a
contributor (due to legal ambiguities about my right to assign
copyrights, not my willingness to do so), but I would like to
understand how the standards are applied in this situation.  Am I
wrong about treating SaaSS consistently with nonfree software when it
comes to promoting its use?

Lynn

Re: Grammar checking

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Sat, 1 Apr 2023 13:30:06 -0400
> Cc: gregory@heytings.org, nasser.alkmim@gmail.com, rms@gnu.org, 
> 	m.eliachevitch@posteo.de, emacs-devel@gnu.org
> 
> I might have misunderstood - this thread arose because you mentioned
> that grammar correction was a higher priority than reimplementing
> spell-checking support

You have misunderstood.  There's no precedence here at all.  We need
(and already have) spell-checking capabilities in Emacs, and will
continue having and developing them.  We _also_ need
grammar-correcting capabilities, independently of spell-checking.

> For me, this is primarily an intellectual exercise as I am not a
> contributor (due to legal ambiguities about my right to assign
> copyrights, not my willingness to do so), but I would like to
> understand how the standards are applied in this situation.  Am I
> wrong about treating SaaSS consistently with nonfree software when it
> comes to promoting its use?

We are not discussing addition to Emacs of any features based on
SaaSS.

Re: Grammar checking

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > LanguageTool isn't proprietary software, but it is used for SaaSS
  > where the "premium" tier appears to use rule sets that are not
  > released with the base tool.

We certainly don't want Emacs to lead people to submit to practices
like that.

If the released (and free) LanguageTool _program_ gives adequate
results, we could make Emacs support working with that.  But we should
take pains _not_ to support the kind of communication that that SaaSS
server offers.


                                  LanguageTool.org is very clear in using
  > the term "open source" rather than "free" to describe the software
  > (and configuration files) for its "Basic" service.

That means they don't agree with our philosophy.  Given the facts
you've described, that is no surprise.  But we don't need to reject a
free program just because its developers disagree with our philosophy.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Grammar checking

[Lynn Winebarger]

On Sat, Apr 1, 2023 at 11:12 PM Richard Stallman <rms@gnu.org> wrote:
>   > LanguageTool isn't proprietary software, but it is used for SaaSS
>   > where the "premium" tier appears to use rule sets that are not
>   > released with the base tool.
>
> We certainly don't want Emacs to lead people to submit to practices
> like that.
>
> If the released (and free) LanguageTool _program_ gives adequate
> results, we could make Emacs support working with that.  But we should
> take pains _not_ to support the kind of communication that that SaaSS
> server offers.
They may not make it easy, see this complaint on their forum:
https://forum.languagetool.org/t/about-the-premium-version-of-languagetool/8469
Although that appears to only apply to the "grammar checker" on the
languagetool.org site, so presumably users of a locally installed free
server won't see such comments.
There's also this kind of thing in the code base:
https://github.com/languagetool-org/languagetool/blob/master/languagetool-core/src/main/java/org/languagetool/Premium.java

>
>
>                                   LanguageTool.org is very clear in using
>   > the term "open source" rather than "free" to describe the software
>   > (and configuration files) for its "Basic" service.
>
> That means they don't agree with our philosophy.  Given the facts
> you've described, that is no surprise.  But we don't need to reject a
> free program just because its developers disagree with our philosophy.

I agree the concerns I've raised are of a "second-order" nature.  I
haven't claimed the program itself is non-free, just that there are
foreseeable complications of the sort highlighted in Section 8 of the
GNU coding standards.   There was an extensive discussion around how
to incorporate the grammars for tree-sitter modes *after* all the work
to get the functionality working, and I'm not sure that even involved
any truly non-free software.  The issues around how to advise users to
set up and make use of languagetool seem to me to be much stickier.

* The documentation is free but definitely refers to the SaaSS in
meaningful ways in the sense of Section 8.

* The process for contributing "rules" to the free version is to go
through the SaaSS's forum sites.
https://community.languagetool.org/rule/list?lang=en shows 5919 rules
for english, presumably in the basic version.  I have no idea where to
find these rules in the code base, nor whether submitting a rule that
is already in the premium version will actually get incorporated in
the free version.   Looking at the java code makes it appear there are
many hard-coded rules, but I don't know if that is really the case.
That is whether the code for the rules are some generic implementation
of the rules coded in XML, or if the XML rule sets are being
translated into java code at some point in the build process.

The second one is going to put users in a sticky situation if they
want to enhance emacs's grammar checking once there is an official
reliance on this external tool.  I'm not really a purist, but I don't
like dealing with these kinds of ambiguous situations where the
gatekeeper for community contributions has such a conflict of interest
in whether and how those contributions are incorporated and shared.
Without impugning the developer and gatekeepers, there's no
*guarantee* a contributed rule via those community forums will not be
incorporated in the premium ruleset without being incorporated in the
"basic" ruleset, since such contributions are presumably provided to
the project maintainer under the LGPL.

There are workarounds to these issues, but none of them that I can see
are trivial.  I don't know what benefit there is to delaying dealing
with these issues until the functionality is already implemented in
master.

Lynn

Re: Grammar checking

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > If the released (and free) LanguageTool _program_ gives adequate
  > > results, we could make Emacs support working with that.  But we should
  > > take pains _not_ to support the kind of communication that that SaaSS
  > > server offers.
  > They may not make it easy, see this complaint on their forum:

Would you please spell out what it is
that they "may not make easy"? 

  > https://forum.languagetool.org/t/about-the-premium-version-of-languagetool/8469

I looked at that page, but lacking the context, I can't understand it
well enough to divine the point this hints at.

  > * The documentation is free but definitely refers to the SaaSS in
  > meaningful ways in the sense of Section 8.

Which documentation is that?  Documentation of what?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Grammar checking

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > If the released (and free) LanguageTool _program_ gives adequate
  > > results, we could make Emacs support working with that.  But we should
  > > take pains _not_ to support the kind of communication that that SaaSS
  > > server offers.
  > They may not make it easy, see this complaint on their forum:

Would you please spell out what it is
that they "may not make easy"? 

  > https://forum.languagetool.org/t/about-the-premium-version-of-languagetool/8469

I looked at that page, but lacking the context, I can't understand it
well enough to divine the point that your message hints at.

  > * The process for contributing "rules" to the free version is to go
  > through the SaaSS's forum sites.
  > https://community.languagetool.org/rule/list?lang=en shows 5919 rules
  > for english, presumably in the basic version.

We could consider forking that code in a limited way: adding new rules.

In general, we should cooperate with upstream developers, but we don't
have to jump through hoops to do so.

  >    Looking at the java code makes it appear there are
  > many hard-coded rules, but I don't know if that is really the case.
  > That is whether the code for the rules are some generic implementation
  > of the rules coded in XML, or if the XML rule sets are being
  > translated into java code at some point in the build process.

I can only guess at the context this is about, but it sounds like
you're suggesting that it may not be clear what form of the code is
the real source code.  Do they not say?  Does their source release
include the XML?  Does it include Make rules to translate the XML into
Java?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Grammar checking

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 6984 bytes --]

On Sun, Apr 2, 2023, 11:05 PM Richard Stallman <rms@gnu.org> wrote:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > If the released (and free) LanguageTool _program_ gives adequate
>   > > results, we could make Emacs support working with that.  But we
> should
>   > > take pains _not_ to support the kind of communication that that SaaSS
>   > > server offers.
>   > They may not make it easy, see this complaint on their forum:
>
> Would you please spell out what it is
> that they "may not make easy"?
>
>   >
> https://forum.languagetool.org/t/about-the-premium-version-of-languagetool/8469
>
> I looked at that page, but lacking the context, I can't understand it
> well enough to divine the point that your message hints at.
>


I may have been mistaken in my first reading. I read the message as saying
that any process using the free service would receive an advertisement of
how many corrections would be found by the premium service.  I am assuming
that at the least emacs maintainers would want to filter that out by
default. The forum message may only refer to using the web user interface
for checking sample text, though.

If the former is true though, it could be difficult to ensure such
advertising is always filtered. It really depends on the owners of that
service, who can change over time.


 > * The process for contributing "rules" to the free version is to go
>   > through the SaaSS's forum sites.
>   > https://community.languagetool.org/rule/list?lang=en shows 5919 rules
>   > for english, presumably in the basic version.
>

I found a more on-point reference addressing my concern, i.e. how will
contributions replicating the rules implemented in the premium version be
treated by the project developers:

https://forum.languagetool.org/t/free-lt-premium-for-contributors/8639

Since the exact nature of those premium rules is presumably not disclosed
just by virtue of having a premium subscription, I can only guess this
reverse engineering would happen by following a process like:

1) Take a large corpus of texts with known grammatical errors, e.g.
https://www.cl.cam.ac.uk/research/nl/bea2019st/ or
https://ai.googleblog.com/2021/08/the-c4200m-synthetic-dataset-for.html?m=1
2) Record the results produced by the free and premium versions on each
test case
3) Formulate rules that specifically fix issues found by the premium
version and not the free version.

Perhaps the LanguageTool.org owners would consider this a violation of
their service's terms and conditions as a justification for not accepting
contributions of source code to the project.

OTOH, if an emacs developer or user simply wants to systematically improve
the free version of LanguageTool, the most obvious method for doing so
would be

1) Take a large corpus of texts with known grammatical errors, see above
2) Record the results produced by the free rule set
3) Formulate rules that specifically fix issues found, prioritizing issues
by some measure of expected frequency in real text

Presumably the additional rules in the premium version have been added
precisely according to some measure of their expected frequency, possibly
by analysis of real-world text from users over the years the service has
been available.

It would be surprising if any attempt to systematically improve my the
rules in LanguageTool did not have significant overlap with the rules found
in the premium version, if that attempt was successful, just due to the
definition of "successful" in statistical terms and the assumption that the
premium rule set is likewise "successful".

We could consider forking that code in a limited way: adding new rules.
>
> In general, we should cooperate with upstream developers, but we don't
> have to jump through hoops to do so.
>

I'm not personally very pure in the software I use, so I'm surprised at how
much the issues I perceive seem to bother me.  I've been an emacs user
since the 90s, and it would never have occurred to me that I would ever be
concerned about contributing code to improve emacs, whether directly to the
emacs projects, or indirectly through one of its dependencies.  From what I
see now, that will not be the case if grammar checking support is added
that depends on languagetool.

I suppose there's another, even more abstract concern with open source
software that is developed specifically in conjunction with a SaaSS
business, which is:  To what extent does data from users of the SaaSS drive
development, or even get incorporated in some (aggregated or statistical)
form in the source code.  For example, what if a grammar checker
incorporated a "deep learning" system that had been trained on such data.
In most cases, it would be impossible to reconstruct the training data set
starting from the data specifying the trained model.  But, would it be
acceptable for a GNU software project to depend on such software?  I don't
know the answer, but I think it's a real question when dealing with open
source software from projects like LanguageTool.  I also don't know or
allege that there's anything like that in LanguageTool, but neither can I
be certain that there is not.  I can't help but think this business model -
maintaining an open source version as a loss leader for a proprietary or
SaaSS version - is only going to continue growing, and hence the need to
address it in the GNU coding manual section 8 or otherwise.


>   >    Looking at the java code makes it appear there are
>   > many hard-coded rules, but I don't know if that is really the case.
>   > That is whether the code for the rules are some generic implementation
>   > of the rules coded in XML, or if the XML rule sets are being
>   > translated into java code at some point in the build process.
>
> I can only guess at the context this is about, but it sounds like
> you're suggesting that it may not be clear what form of the code is
> the real source code.  Do they not say?  Does their source release
> include the XML?  Does it include Make rules to translate the XML into
> Java?


I don't do a lot of Java coding, and it was a cursory examination.  I did
eventually find the xml rulesets linked to from
https://dev.languagetool.org/languages, which is classified as "user
documentation". It appears most rules in well-supported languages are in
XML, with some coded in Java.  Whether the coding in Java is for speed or
to overcome limitations of the semantics of rules expressed in XML, I have
no idea.

I'm going to leave my concerns at that.  I've already spent too much time
on this as it is.  I just thought the last-minute hair-pulling discussion
of tree-sitter grammar files, which frankly seem to have much less ethical
baggage, should not be repeated after grammar checking support depending on
LanguageTool is already implemented and adopted.

Lynn

[-- Attachment #2: Type: text/html, Size: 9675 bytes --]

Re: Grammar checking

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I may have been mistaken in my first reading. I read the message as saying
  > that any process using the free service would receive an advertisement of
  > how many corrections would be found by the premium service.  I am assuming
  > that at the least emacs maintainers would want to filter that out by
  > default. The forum message may only refer to using the web user interface
  > for checking sample text, though.

  > If the former is true though, it could be difficult to ensure such
  > advertising is always filtered. It really depends on the owners of that
  > service, who can change over time.

We would not want to release Emacs with code that would talk to the
premium dis-service.  We would want to write the code we release so
that it would not try to use the dis-service.

Since Emacs is free, users could change the code so it would talk
to the LanguageTool dis-service.  We have no say over that.
But we would never release or recommend that code.

This being so, I think we don't need to worry about whether the dis-service
sends advertisements to users of software that talks to it.   Our
software will not talk to it anyway.





-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Grammar checking

[Lynn Winebarger]

On Fri, Apr 7, 2023 at 11:28 PM Richard Stallman <rms@gnu.org> wrote:
> This being so, I think we don't need to worry about whether the dis-service
> sends advertisements to users of software that talks to it.   Our
> software will not talk to it anyway.

The concern is whether their *free* service sends the advertisements.
Maybe someone who actually uses LanguageTool's free service now can
say whether they see such ads in the feedback they receive ("the
premium version would have identified X additional problems").

Lynn

Re: Grammar checking

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Sat, 8 Apr 2023 09:33:55 -0400
> Cc: emacs-devel@gnu.org
> 
> On Fri, Apr 7, 2023 at 11:28 PM Richard Stallman <rms@gnu.org> wrote:
> > This being so, I think we don't need to worry about whether the dis-service
> > sends advertisements to users of software that talks to it.   Our
> > software will not talk to it anyway.
> 
> The concern is whether their *free* service sends the advertisements.
> Maybe someone who actually uses LanguageTool's free service now can
> say whether they see such ads in the feedback they receive ("the
> premium version would have identified X additional problems").

This is all besides the point, AFAIU, because we are talking about
installing the stand-alone application, which AFAICT doesn't display
any advertisements and doesn't use any services to begin with.

Re: Grammar checking

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Perhaps the LanguageTool.org owners would consider this a violation of
  > their service's terms and conditions as a justification for not accepting
  > contributions of source code to the project.

This seems to assume that a developer has an obligation to install any change
that "is an improvement" and must submit a "justification" for not adding it.

That seems very strange to me.  I think any developer is entitled to
reject a change and does not have to justify rejecting it.

  > We could consider forking that code in a limited way: adding new rules.

We can consider forking in any manner whatsoever.  Of course, forking
the free version of LanguageTool has a practical downside (more work
for us!) as well as an upside.  We might decide it is not worth the
trouble.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Grammar checking

[Lynn Winebarger]

On Fri, Apr 7, 2023 at 11:28 PM Richard Stallman <rms@gnu.org> wrote:
>   > Perhaps the LanguageTool.org owners would consider this a violation of
>   > their service's terms and conditions as a justification for not accepting
>   > contributions of source code to the project.

I recall writing this, but I thought there was additional context.
Unfortunately I can't locate my original email for reference.  I am
sure I did not intend the following inference.

> This seems to assume that a developer has an obligation to install any change
> that "is an improvement" and must submit a "justification" for not adding it.
>
> That seems very strange to me.  I think any developer is entitled to
> reject a change and does not have to justify rejecting it.

Sure.  My question was whether the GNU project condones introducing
dependencies in GNU software X on a software *project* Y such that
attempts to improve functionality in X will only be incorporated by Y
if they do not reduce the value of proprietary or SaaSS offering Z
which Y promotes.  While the developer has every right to control
their distribution of software, potential contributors also have every
right to consider whether or how a contribution will be considered.
For me, I simply would not consider making a contribution under the
LGPL to a project that might reject it for the free project but
incorporate it into their SaaSS.  It's not like there would be any way
to tell if that happened.  The only way I would contribute to such
software would be in a fork under an Affero-style copyleft license,
which as you note below, can be quite a bit of trouble.

>   > We could consider forking that code in a limited way: adding new rules.
>
> We can consider forking in any manner whatsoever.  Of course, forking
> the free version of LanguageTool has a practical downside (more work
> for us!) as well as an upside.  We might decide it is not worth the
> trouble.

I appear to have messed up the quoting here - that sentence was from
one of your earlier replies.

Lynn

Re: Grammar checking

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I suppose there's another, even more abstract concern with open source
  > software that is developed specifically in conjunction with a SaaSS
  > business,

I can't be certain which program(s) you are talking about, but I think
you are referring to the free version of LanguageTool as "open source"
rather than "free".  Nearly all free programs are open source and
nearly all open source programs are free, but normally we care whather
a program is free, and we do NOT care whether it is open source.
https://gnu.org/philosophy/open-source-misses-the-point.html.

I think you must have chosen to write "open source" there to convey a
point, but what point was it?

For any program want Emacs to incorporate or use, we insist it be
free.  Whether that program is open source is not a question we are
concerned about.

  >   But, would it be
  > acceptable for a GNU software project to depend on such software?

If we have free software to do further training on the neural network,
we can accept it.

Does LanguageTool use a neural network?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Grammar checking

[Lynn Winebarger]

On Fri, Apr 7, 2023 at 11:28 PM Richard Stallman <rms@gnu.org> wrote:
>   > I suppose there's another, even more abstract concern with open source
>   > software that is developed specifically in conjunction with a SaaSS
>   > business,
>
> I can't be certain which program(s) you are talking about, but I think
> you are referring to the free version of LanguageTool as "open source"
> rather than "free".  Nearly all free programs are open source and
> nearly all open source programs are free, but normally we care whather
> a program is free, and we do NOT care whether it is open source.
> https://gnu.org/philosophy/open-source-misses-the-point.html.
>
> I think you must have chosen to write "open source" there to convey a
> point, but what point was it?
>

I'm attempting to refer to the project and/or development process
maintaining the primary distribution of the software.  When a program
is distributed as free software, but the project that
develops/maintains/distributes that program exists in part or in whole
to promote either a proprietary or SaaSS version of the software, I do
not understand how the program and the project are effectively
distinguished for the purpose of Section 8 of the GNU coding
standards.  That is, I do not know how a GNU program can depend on
such software without promoting the associated project and effectively
the proprietary software or SaaSS, under the two-step transitivity
rule described in Section 8.

Of course, my understanding is not required.  I believe my question
has been resolved - Section 8 is not intended to prohibit reliance on
such software projects.

>   >   But, would it be
>   > acceptable for a GNU software project to depend on such software?
>
> If we have free software to do further training on the neural network,
> we can accept it.

On further reflection, I had thought this situation would be covered
under the "preferred form of modification" rule, in that the dataset
used to train the algorithm (NN or otherwise) would be considered a
form of "source code".  It seems I was mistaken on that as well.

> Does LanguageTool use a neural network?

I do not know.

Lynn

Re: Grammar checking

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I'm attempting to refer to the project and/or development process
  > maintaining the primary distribution of the software.

I can see various meanings for that.
Do you mean that you want to talk about those things now
so you want to refer to them in your email?
Or do you mean that you want a free program's release materials
and documentation to refer to those things?

  > When a program
  > is distributed as free software, but the project that
  > develops/maintains/distributes that program exists in part or in whole
  > to promote either a proprietary or SaaSS version of the software, I do
  > not understand how the program and the project are effectively
  > distinguished for the purpose of Section 8 of the GNU coding
  > standards.

I think you have a valid point.  We could call it a "free program
with a moral shadow."

But we don't always have the luxury to reject them.
It could mean we can't compile the program we care about, not at all.
That would be throwing the baby out with the bathwater.

At most drastic, we could recommend that people avoid certain
languages whose tools have problems of this sort.

But just saying "compile Foo.blahlang" and including a make rule to do
it does not promote the Blah compiler very much.  So if the Blah
compiler has a moral shadow, it doesn't fall on the Foo program very
much.  Not if you don't talk much about the Blah language in the Foo
documentation (and why would/should you?).

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Grammar checking

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

>   >   But, would it be
>   > acceptable for a GNU software project to depend on such software?
>
> If we have free software to do further training on the neural network,
> we can accept it.
>
> Does LanguageTool use a neural network?

No, according to [0] as of 2010 the approach they were using was
rule-based, as opposed to statistical (neutral network).

[0] https://www.researchgate.net/profile/Marcin-Milkowski/publication/220282022_Developing_an_open-source_rule-based_proofreading_tool/links/5b030f720f7e9be94bdabb60/Developing-an-open-source-rule-based-proofreading-tool.pdf

-- 
Philip Kaludercic

Re: Grammar checking

[Lynn Winebarger]

On Sun, Apr 9, 2023 at 5:02 AM Philip Kaludercic <philipk@posteo.net> wrote:
> Richard Stallman <rms@gnu.org> writes:
>
> >   >   But, would it be
> >   > acceptable for a GNU software project to depend on such software?
> >
> > If we have free software to do further training on the neural network,
> > we can accept it.
> >
> > Does LanguageTool use a neural network?
>
> No, according to [0] as of 2010 the approach they were using was
> rule-based, as opposed to statistical (neutral network).
>
> [0] https://www.researchgate.net/profile/Marcin-Milkowski/publication/220282022_Developing_an_open-source_rule-based_proofreading_tool/links/5b030f720f7e9be94bdabb60/Developing-an-open-source-rule-based-proofreading-tool.pdf

2010 is a long time ago in terms of machine learning.  It's also
possible to construct a rule semantics that incorporates machine
learning.

There's this directory in languagetool's source code that appears to
provide infrastructure for using some RPC protocol for "MLServer"s:
https://github.com/languagetool-org/languagetool/tree/master/languagetool-core/src/main/java/org/languagetool/rules/ml.
One common expansion of "ML" is "machine learning", so it's *possible*
that this provides infrastructure for consulting some machine learning
service.  Determining what it is actually used for, whether the RPC
services are local or remote, and whether it is used in the free
version or is simply infrastructure for the premium service, would
require much more investigation than I am willing to perform any time
soon.  Hence, I don't know.

Lynn

Re: Grammar checking

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > There's this directory in languagetool's source code that appears to
  > provide infrastructure for using some RPC protocol for "MLServer"s:
  > https://github.com/languagetool-org/languagetool/tree/master/languagetool-core/src/main/java/org/languagetool/rules/ml.

If we wamt to use the free version of Languagetool, this raiss
important questions.  First of all, does the program as currently
released try by default to use tha remote protocol?

Secondarily, how much of the usefulness of that program as currently
released if we turn that off?  And how would we modify the release
to turn that off?

If the program as currently released does NOT try ny default to use
tha protocol, is there something that they invite users to do so as
to start using it?  We might want to delete that feature from the version
we distribute, so we don't end up distributing SaaSS unknowingly.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Grammar checking

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I wrote:

  > If we wamt to use the free version of Languagetool, this raiss
  > important questions.  First of all, does the program as currently
  > released try by default to use tha remote protocol?

  > Secondarily, how much of the usefulness of that program as currently
  > released if we turn that off?  And how would we modify the release
  > to turn that off?

but the second graf should have said

  > Secondarily, how much of the usefulness of that program as
  > currently released is due to using that dis-service?  And how
  > would we modify the release to turn that off?


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Grammar checking

[Lynn Winebarger]

On Fri, Apr 21, 2023 at 10:22 PM Richard Stallman <rms@gnu.org> wrote:
>   > There's this directory in languagetool's source code that appears to
>   > provide infrastructure for using some RPC protocol for "MLServer"s:
>   > https://github.com/languagetool-org/languagetool/tree/master/languagetool-core/src/main/java/org/languagetool/rules/ml.
>
> If we wamt to use the free version of Languagetool, this raiss
> important questions.  First of all, does the program as currently
> released try by default to use tha remote protocol?
>
[From subsequent amending email]
>   > Secondarily, how much of the usefulness of that program as
>   > currently released is due to using that dis-service?  And how
>   > would we modify the release to turn that off?
>
> If the program as currently released does NOT try ny default to use
> tha protocol, is there something that they invite users to do so as
> to start using it?  We might want to delete that feature from the version
> we distribute, so we don't end up distributing SaaSS unknowingly.
>

The only concrete description I recall seeing (from an emacs user) is
from Ihor Radchenko in
https://lists.gnu.org/archive/html/emacs-devel/2023-03/msg01044.html

The design of the software reflects its intended use in a multi-user
SaaSS server.  If emacs maintainer time is going to be consumed by
ensuring an acceptable version and configuration is installed on user
machines, I hope some consideration (by that maintainer) would be
whether the bulk of the functionality could be implemented in a
simpler, single-user, form in some incremental way - say by an emacs
lisp program running in a batch emacs process.
https://dev.languagetool.org/languages classifies the rules enforced
by LanguageTool into XML, Java-based, spell-checking, and confusion
pairs.  There's a separate 8GB of n-gram data, covering the languages
supported by the languagetool organization.  So, how much work would
be required to meaningfully use the confusion pairs, and what would it
provide.  Or interpreting and applying the XML rules.  Or be more
efficient in the use of the 8GB of n-gram data
(https://dev.languagetool.org/finding-errors-using-n-gram-data).

Maybe the effort wouldn't be worth it, but maybe it would.  I don't
think it's obvious either way.

Lynn

Re: Grammar checking

[Eli Zaretskii]

> From: Nasser Alkmim <nasser.alkmim@gmail.com>
> Cc: Gregory Heytings <gregory@heytings.org>,  rms@gnu.org,
>   m.eliachevitch@posteo.de,  emacs-devel@gnu.org
> Date: Fri, 31 Mar 2023 10:40:28 +0200
> 
> I tried a couple of grammar checkers in Emacs, integrated via Flymake and Eglot.
> The one that works best for me is this Grammarly language server [1].

Grammarly is not Free Software, AFAIK, so using it in Emacs is out of
the question.

Re: Grammar checking

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Grammarly is not Free Software, AFAIK, so using it in Emacs is out of
  > the question.

Exactly right.  The purpose of GNU Emacs, and the purpose of improving
it, is to advance what people can do in freedom, and spread freedom.
We shouldn't let the means run away with the ends.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Grammar checking

[Emanuel Berg]

Richard Stallman wrote:

>> Grammarly is not Free Software, AFAIK, so using it in Emacs
>> is out of the question.
>
> Exactly right. The purpose of GNU Emacs, and the purpose of
> improving it, is to advance what people can do in freedom,
> and spread freedom. We shouldn't let the means run away with
> the ends.

That's right, if it's possible to do the right way, no excuse
not to do it the right way. All it takes is time and effort.
And if it doesn't happen, ever - this is a clear indication no
one cared about it enough to actually do it - and if not
a single one, no one ever did it, maybe that says something
how important it was to begin with ...

I've seen this so many times, in so many places. Compromise,
deadline, double standards, it always blows up in the end.

Little by little against the goal always beats trying to
outsmart the problem to achieve a quick fix, compromising
standards and ideals, just because of the perception it "has
to" happen this instant ...

No shortcuts to the top!

-- 
underground experts united
https://dataswamp.org/~incal

Re: Grammar checking (was: jinx)

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1093 bytes --]

On Fri, Mar 31, 2023, 08:15 Eli Zaretskii <eliz@gnu.org> wrote:

> > Date: Fri, 31 Mar 2023 07:10:11 +0000
> > From: Gregory Heytings <gregory@heytings.org>
> > cc: rms@gnu.org, m.eliachevitch@posteo.de, emacs-devel@gnu.org
> >
> > > As an aside, a technology I would like to see in Emacs is grammar
> > > checking.  Last I looked, there are no free grammar-checking libraries
> > > or services out there, even just for the English language.  But if
> > > that's no longer true, then adding such capabilities to Emacs will be
> > > very welcome, I think, since word-based spell-checking is clearly not
> > > enough these days.
> >
> > I think that's no longer true, LanguageTool is available (and has been
> for
> > ~20 years): https://github.com/languagetool-org/languagetool .  But
> it's
> > written in Java...
>
> Thanks.  Maybe someone will figure out how to integrate this with
> Emacs.
>

Flymake, or LSP, could be a starting point. Regarding the latter, Eglot
doesn't currently allow more than one server to manage a buffer.
But I'm working on that.

João

[-- Attachment #2: Type: text/html, Size: 1985 bytes --]

Re: jinx

[Rudolf Adamkovič]

Gregory Heytings <gregory@heytings.org> writes:

> I think that's no longer true, LanguageTool is available (and has been for 
> ~20 years): https://github.com/languagetool-org/languagetool .  But it's 
> written in Java...

There is also Vale [1], a program written in Go and released under the
MIT license.  It now even supports the Org mode format (along with
Markdown, AsciiDoc, reStructuredText, HTML, and XML).

[1] https://github.com/errata-ai/vale
-- 
"'Contrariwise,' continued Tweedledee, 'if it was so, it might be; and
if it were so, it would be; but as it isn't, it ain't.  That's logic.'"
-- Lewis Carroll, Through the Looking Glass, 1871/1872

Rudolf Adamkovič <salutis@me.com> [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: jinx

[Philip Kaludercic]

Rudolf Adamkovič <salutis@me.com> writes:

> Gregory Heytings <gregory@heytings.org> writes:
>
>> I think that's no longer true, LanguageTool is available (and has been for 
>> ~20 years): https://github.com/languagetool-org/languagetool .  But it's 
>> written in Java...
>
> There is also Vale [1], a program written in Go and released under the
> MIT license.  It now even supports the Org mode format (along with
> Markdown, AsciiDoc, reStructuredText, HTML, and XML).
>
> [1] https://github.com/errata-ai/vale

This seems to be on a different level than what LanguageTool does.  Vale
and other alternatives are style-checkers that catch mistakes matching
some fixed rules, without any "understanding" (linguistic model) of the
underlying language.  IIUC it could be compared to font-locking using
regular expressions or a library like Tree Sitter.

Re: jinx

[Arash Esbati]

Eli Zaretskii <eliz@gnu.org> writes:

> Another feature of jinx is the ability to skip some buffer text, e.g.,
> relying on font-lock faces or user-defined regexps to identify the
> parts that should be skipped.  We have similar features:
> spell-checking of comments and strings, but if that is not enough, we
> could extend these capabilities in ispell and flyspell by simple
> enough additions.

Somewhat off-topic, but it would be great if ispell and flyspell could
use the same logic for skipping portions of buffer text.  Take for
example (La)TeX mode: ispell uses `ispell-tex-skip-alists' and flyspell
uses `flyspell-tex-command-regexp' (which is not very powerful) and then
a function added to `flyspell-generic-check-word-predicate' (AFAIU).  I
once wrote an addition for AUCTeX (tex-ispell.el) which extends
`ispell-tex-skip-alists' for many LaTeX packages but had hard times
making it work with flyspell as well, skipping multi-line text portions
was something I couldn't solve for flyspell, e.g.:

\begin{verbatim}
Please skip the entire text in this environment.
\end{verbatim}

Best, Arash

Re: jinx

[Eli Zaretskii]

> From: Arash Esbati <arash@gnu.org>
> Cc: rms@gnu.org,  m.eliachevitch@posteo.de,  emacs-devel@gnu.org
> Date: Fri, 31 Mar 2023 20:33:02 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Another feature of jinx is the ability to skip some buffer text, e.g.,
> > relying on font-lock faces or user-defined regexps to identify the
> > parts that should be skipped.  We have similar features:
> > spell-checking of comments and strings, but if that is not enough, we
> > could extend these capabilities in ispell and flyspell by simple
> > enough additions.
> 
> Somewhat off-topic, but it would be great if ispell and flyspell could
> use the same logic for skipping portions of buffer text.

Please submit patches to do that, it shouldn't be hard.

(I wonder why the built-in TeX support in the spell-checker doesn't do
this job, it's supposed to be adequate, and all Emacs needs to do is
to tell the speller it is working on TeX text.  So I don't even
understand why we need to skip something on our own in these cases.)

Re: jinx

[Arash Esbati]

Eli Zaretskii <eliz@gnu.org> writes:

> Please submit patches to do that, it shouldn't be hard.

I tried it as I wrote tex-ispell.el but I didn't understand flyspell.el
enough back then.  Maybe I give it a new try, but I wouldn't hold my
breath for that.

> (I wonder why the built-in TeX support in the spell-checker doesn't do
> this job, it's supposed to be adequate, and all Emacs needs to do is
> to tell the speller it is working on TeX text.  So I don't even
> understand why we need to skip something on our own in these cases.)

I can only tell for hunspell: It has built-in support for skipping of
many macro arguments and environments[1], but:

  a) It is not complete (and probably will never be)
  b) It cannot work for user defined macros/environments, and this is
  where AUCTeX/ispell shine: AUCTeX can parse user defined
  macros/environments and add them automatically to
  `ispell-tex-skip-alists'.
  c) Another advantage of `ispell-tex-skip-alists' is that one can write
  a function and put that in `ispell-tex-skip-alists' for certain
  macros/environments: That gives us a lot of flexibility to precisely
  skip or check.

Best, Arash

Footnotes:
[1]  https://github.com/hunspell/hunspell/blob/e7ccb97fde9d7e9a81c920a3e255ad98426692f8/src/parsers/latexparser.cxx#L53

Re: jinx

[Eli Zaretskii]

> From: Arash Esbati <arash@gnu.org>
> Cc: rms@gnu.org,  m.eliachevitch@posteo.de,  emacs-devel@gnu.org
> Date: Fri, 31 Mar 2023 21:35:25 +0200
> 
> > (I wonder why the built-in TeX support in the spell-checker doesn't do
> > this job, it's supposed to be adequate, and all Emacs needs to do is
> > to tell the speller it is working on TeX text.  So I don't even
> > understand why we need to skip something on our own in these cases.)
> 
> I can only tell for hunspell: It has built-in support for skipping of
> many macro arguments and environments[1], but:
> 
>   a) It is not complete (and probably will never be)
>   b) It cannot work for user defined macros/environments, and this is
>   where AUCTeX/ispell shine: AUCTeX can parse user defined
>   macros/environments and add them automatically to
>   `ispell-tex-skip-alists'.
>   c) Another advantage of `ispell-tex-skip-alists' is that one can write
>   a function and put that in `ispell-tex-skip-alists' for certain
>   macros/environments: That gives us a lot of flexibility to precisely
>   skip or check.

These issues are not TeX-specific, so they should be handled in a more
general way, not bound to TeX/LaTeX.  In particular, any "tex-skip"
features should not supplant the TeX support built into the speller,
they should only _add_ skip regexps for stuff that the speller cannot
support, like user-defined macros.

I'm not sure I understand item c), though: when and why would one want
to spell-check macros and environments?

Re: jinx

[Arash Esbati]

Eli Zaretskii <eliz@gnu.org> writes:

> These issues are not TeX-specific, so they should be handled in a more
> general way, not bound to TeX/LaTeX.  In particular, any "tex-skip"
> features should not supplant the TeX support built into the speller,
> they should only _add_ skip regexps for stuff that the speller cannot
> support, like user-defined macros.

Agreed, in general.  The only thing is that we are dealing with
different spellers (ispell, aspell, hunspell, nuspell?) and I assume
they have different level of built-in support for "tex-skips" (I haven't
checked though).  My example with verbatim environment was related to
"multi-line skip" in flyspell; it can be replaced with any other
environment which does a similar thing.

> I'm not sure I understand item c), though: when and why would one want
> to spell-check macros and environments?

Sorry, I meant to say arguments/bodies of macros/environments, not the
name of macros and environments.

Best, Arash

Re: jinx

[Eli Zaretskii]

> From: Arash Esbati <arash@gnu.org>
> Cc: rms@gnu.org,  m.eliachevitch@posteo.de,  emacs-devel@gnu.org
> Date: Sat, 01 Apr 2023 09:42:40 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I'm not sure I understand item c), though: when and why would one want
> > to spell-check macros and environments?
> 
> Sorry, I meant to say arguments/bodies of macros/environments, not the
> name of macros and environments.

This is again not TeX-specific.  Similar issues exist in Texinfo and
in other markup languages.  The annoyance is usually minor: it's
enough to tell the speller to "accept" a word just once, since the
number of markup codes is usually very small.  But it would be nicer,
of course, if Emacs could automatically skip markup in each major
mode.

We should keep in mind that spell-checking technical text will
inevitably produce quite a few false positives, due to the jargon,
acronyms, file names, and other similar stuff.  I don't see how this
could be solved in principle without risking false negatives, which is
worse.

Re: jinx

[Arash Esbati]

[-- Attachment #1: Type: text/plain, Size: 1257 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

> This is again not TeX-specific.  Similar issues exist in Texinfo and
> in other markup languages.  The annoyance is usually minor: it's
> enough to tell the speller to "accept" a word just once, since the
> number of markup codes is usually very small.

I see your point, but I think Emacs could do better.

> But it would be nicer, of course, if Emacs could automatically skip
> markup in each major mode.

I second this: Emacs should skip markup in each major mode, and it is a
pity that the support for things like texinfo and/or org-mode is
somewhat poor.  This was my motivation to add a library to AUCTeX for
ispell and adjust most of it style files to use it.

> We should keep in mind that spell-checking technical text will
> inevitably produce quite a few false positives, due to the jargon,
> acronyms, file names, and other similar stuff.  I don't see how this
> could be solved in principle without risking false negatives, which is
> worse.

True.  OTOH, Emacs could skip the markup where we know it can be
ignored, take for example @file{filename}.  Currently, both ispell and
flyspell try to correct the filename argument.  Here a nonsensical
example how flyspell looks in texinfo-mode with Emacs -Q:

[-- Attachment #2: flyspell-texinfo.png --]
[-- Type: image/png, Size: 2525 bytes --]

[-- Attachment #3: Type: text/plain, Size: 298 bytes --]


So if I could make a wish, I'd like to see a general infrastructure
where one can do things like in `ispell-skip-region-alist', looking for
font-lock and all the other techniques lying around and ispell, flyspell
and others would use this infrastructure and not bake their own stuff.

Best, Arash

Re: jinx

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> > From: Arash Esbati <arash@gnu.org>
> > Cc: rms@gnu.org,  m.eliachevitch@posteo.de,  emacs-devel@gnu.org
> > Date: Sat, 01 Apr 2023 09:42:40 +0200
> >
> > Eli Zaretskii <eliz@gnu.org> writes:
> >
> > > I'm not sure I understand item c), though: when and why would one want
> > > to spell-check macros and environments?
> >
> > Sorry, I meant to say arguments/bodies of macros/environments, not the
> > name of macros and environments.
>
> This is again not TeX-specific.  Similar issues exist in Texinfo and
> in other markup languages.  The annoyance is usually minor: it's
> enough to tell the speller to "accept" a word just once, since the
> number of markup codes is usually very small.  But it would be nicer,
> of course, if Emacs could automatically skip markup in each major
> mode.

Wasn't that the purpose of `flyspell-prog-mode'?

Michael.

Re: jinx

[Michael Eliachevitch]

[-- Attachment #1: Type: text/plain, Size: 2670 bytes --]


On 2023-04-03 at 14:32 +02, Michael Heerdegen <michael_heerdegen@web.de> wrote:

>> This is again not TeX-specific.  Similar issues exist in Texinfo and
>> in other markup languages.  The annoyance is usually minor: it's
>> enough to tell the speller to "accept" a word just once, since the
>> number of markup codes is usually very small.  But it would be nicer,
>> of course, if Emacs could automatically skip markup in each major
>> mode.
>
> Wasn't that the purpose of `flyspell-prog-mode'?

Yes, `flyspell-prog-mode' and `flyspell-prog-text-faces' together offer similar functionality to what jinx does by default, only that it allows configuring the faces on per-major-mode basis and has include- and exclude-faces. As I understood the discussion, ideally we could add this to flyspell and ispell in a consistent way.

I'm also not sure if the distinction between `flyspell-mode' and `flyspell-prog-mode' is still needed, you can just customize the text-mode to not exclude any faces for instance. It doesn't hurt, we can just enable `flyspell-prog-mode' everywhere. But ispell doesn't have a prog-mode. If we want to have common include/exclude faces for ispell and flyspell and be consistent with those packages, one could think about adding an ispell-prog-mode for consistency, though that seems weird to me.

Of course `flyspell-prog-mode' one can achieve very similar functionality to jinx by using per-major-mode hooks and dir-local variables to customize `flyspell-prog-text-faces'  for each mode, but it would be nice for that to be simpler. And naming wise, initially I was confused by `flyspell-prog-mode' because I thought this is programming-mode-specific and might need for the mode to inherit from `prog-mode', but we have plenty of modes with font-lock that inherit from text-mode or conf-mode.

An example: I noticed that both jinx and flyspell underline "pgpmime" string in the MML signature snippet in my email buffer, which has the major mode `mu4e-compose-mode' inheriting from `message-mode' and in turn from text-mode. In jinx, I can fix the error message by a simple

    (cl-pushnew 'message-mml (alist-get 'message-mode jinx-exclude-faces))

The cited text is still checked, because for non-prog-modes I didn't specify any jinx-include-faces and thus it checks all non-exclude faces. In flyspell, I could add `flyspell-prog-mode' to `message-mode-hook' and also hook a function that locally sets the `flyspell-prog-text-faces' to the default text-face and the four message-cited-faces, hoping I didn't forget any other faces that text might have in the email compose buffer. So it feels a bit more annoying to configure.

Cheers, Michael

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

Re: jinx

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: Arash Esbati <arash@gnu.org>,  rms@gnu.org,  m.eliachevitch@posteo.de,
>   emacs-devel@gnu.org
> Date: Mon, 03 Apr 2023 14:32:36 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > > From: Arash Esbati <arash@gnu.org>
> > > Cc: rms@gnu.org,  m.eliachevitch@posteo.de,  emacs-devel@gnu.org
> > > Date: Sat, 01 Apr 2023 09:42:40 +0200
> > >
> > > Eli Zaretskii <eliz@gnu.org> writes:
> > >
> > > > I'm not sure I understand item c), though: when and why would one want
> > > > to spell-check macros and environments?
> > >
> > > Sorry, I meant to say arguments/bodies of macros/environments, not the
> > > name of macros and environments.
> >
> > This is again not TeX-specific.  Similar issues exist in Texinfo and
> > in other markup languages.  The annoyance is usually minor: it's
> > enough to tell the speller to "accept" a word just once, since the
> > number of markup codes is usually very small.  But it would be nicer,
> > of course, if Emacs could automatically skip markup in each major
> > mode.
> 
> Wasn't that the purpose of `flyspell-prog-mode'?

Markup languages are usually not descendants of prog-modes.

Or maybe I misunderstand what you are saying.

Re: jinx

[Michael Eliachevitch]

On 2023-04-03 at 17:26 +03, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Michael Heerdegen <michael_heerdegen@web.de>
>> Wasn't that the purpose of `flyspell-prog-mode'?
>
> Markup languages are usually not descendants of prog-modes.
>
> Or maybe I misunderstand what you are saying.

`flyspell-prog-mode' doesn't have any relation to prog-mode. It is a function (not really a mode) that enables flyspell-mode and sets the buffer-local variable `flyspell-generic-check-word-predicate'.

--8<---------------cut here---------------start------------->8---
  (defun flyspell-prog-mode ()
    "Turn on `flyspell-mode' for comments and strings."
    (interactive)
    (setq flyspell-generic-check-word-predicate
          #'flyspell-generic-progmode-verify)
    (flyspell-mode 1)
    (run-hooks 'flyspell-prog-mode-hook))
--8<---------------cut here---------------end--------------->8---

The function `flyspell-generic-progmode-verify' checks whether the face at point is a member of `flyspell-prog-text-faces'. By default it only checks comment and string faces, which is useful in programming modes. But flyspell-prog-mode can be used in any other mode as well `flyspell-prog-text-faces', and one can set `flyspell-prog-text-faces' buffer-locally via setq-local to e.g. only include non-markup. Though the flyspell documentation talks about it being for programming modes, so that seems to be a hack.

Until I read the code above today, I wasn't aware of `flyspell-generic-check-word-predicate', as it's not a defcustom-variable. It might serve as a customization entrypoint, i.e. one could bind it to a function that e.g. checks for include- and exclude-lists like jinx.

Re: jinx

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > `flyspell-prog-mode' doesn't have any relation to prog-mode. It is
  > a function (not really a mode) that enables flyspell-mode and sets
  > the buffer-local variable `flyspell-generic-check-word-predicate'.

It seems to act somewhat like a major mode but not entirely.
Also, its name seems to suggest something different.  That can
cause confusion.

Is it possible to rename this to a clearer name?  Perhaps
program-text-flyspell-mode, or something else?  And then make it a
proper major mode?, following all the conventions for major modes?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: jinx

[Michael Heerdegen]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > `flyspell-prog-mode' doesn't have any relation to prog-mode. It is
>   > a function (not really a mode) that enables flyspell-mode and sets
>   > the buffer-local variable `flyspell-generic-check-word-predicate'.
>
> It seems to act somewhat like a major mode but not entirely.
> Also, its name seems to suggest something different.  That can
> cause confusion.
>
> Is it possible to rename this to a clearer name?  Perhaps
> program-text-flyspell-mode, or something else?  And then make it a
> proper major mode?, following all the conventions for major modes?

A major mode?? I think you misunderstood what it does.  It's simply a
slightly modified flyspell-mode, conceptually clearly a minor-mode, see
the doc string ("Turn on `flyspell-mode' for comments and strings").

What the "don't spell-check language keywords" functionality skips
depends on the major-mode.  That's all.

Michael.

Re: jinx

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Is it possible to rename this to a clearer name?  Perhaps
  > > program-text-flyspell-mode, or something else?  And then make it a
  > > proper major mode?, following all the conventions for major modes?

  > A major mode?? I think you misunderstood what it does.  It's simply a
  > slightly modified flyspell-mode, conceptually clearly a minor-mode, 

I stand corrected.

But the name is still unhelpful.  It seems to enable Flyspell mode for
strings and comments only.  But you'd never guess that from its name.
The name suggests it is a mode function, but I doesn't follow the
conventions for those.

Can those things be corrected?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: jinx

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > Is it possible to rename this to a clearer name?  Perhaps
>   > > program-text-flyspell-mode, or something else?  And then make it a
>   > > proper major mode?, following all the conventions for major modes?
>
>   > A major mode?? I think you misunderstood what it does.  It's simply a
>   > slightly modified flyspell-mode, conceptually clearly a minor-mode, 
>
> I stand corrected.
>
> But the name is still unhelpful.  It seems to enable Flyspell mode for
> strings and comments only.  But you'd never guess that from its name.
> The name suggests it is a mode function, but I doesn't follow the
> conventions for those.
>
> Can those things be corrected?

It should be possible to add a defalias with a different name, but
personally I don't think the name is that confusing.  The docstring is
pretty clear as well.  Then again, I know about prog-mode which is
something that newcomers might not?

Re: jinx

[Michael Heerdegen]

Philip Kaludercic <philipk@posteo.net> writes:

> > But the name is still unhelpful.  It seems to enable Flyspell mode for
> > strings and comments only.  But you'd never guess that from its name.
> > The name suggests it is a mode function, but I doesn't follow the
> > conventions for those.
> >
> > Can those things be corrected?
>
> It should be possible to add a defalias with a different name, but
> personally I don't think the name is that confusing.  The docstring is
> pretty clear as well.  Then again, I know about prog-mode which is
> something that newcomers might not?

Could be the intention was to abbreviate something like "flyspell mode
for programming" and that the name doesn't refer to `prog-mode' at all.

If that is true (I don't know) - it would be a very bad name indeed.

Michael.

Re: jinx

[Michael Heerdegen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Could be the intention was to abbreviate something like "flyspell mode
> for programming" and that the name doesn't refer to `prog-mode' at all.

Seems `flyspell-prog-mode' is even older than `prog-mode'.

Michael.

Re: jinx

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Date: Wed, 05 Apr 2023 13:25:54 +0200
> 
> Michael Heerdegen <michael_heerdegen@web.de> writes:
> 
> > Could be the intention was to abbreviate something like "flyspell mode
> > for programming" and that the name doesn't refer to `prog-mode' at all.
> 
> Seems `flyspell-prog-mode' is even older than `prog-mode'.

Since flyspell-prog-mode just sets up a mechanism for skipping
user-defined words, I actually see no reason to have "prog" in that
name, none at all.  E.g., as already mentioned, text markup modes may
wish using that, and they have nothing to do with programming
languages.  Likewise, spell-checking some other formatted text, like
email messages.

It's a bad name, that's all.  The question is, can we come up with a
significantly better one, given the generality of the feature?

Re: jinx

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> It's a bad name, that's all.  The question is, can we come up with a
> significantly better one, given the generality of the feature?

I have created bug#62677: "Need to find a better name  for
flyspell-prog-mode" for this issue.

Yes, we only need a better name.  "flyspell-restricted-mode"?

But maybe let's better continue in that bug report.

Thx,

Michael.

Re: jinx

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Is it possible to rename this to a clearer name?  Perhaps
  > program-text-flyspell-mode, or something else?  And then make it a
  > proper major mode?, following all the conventions for major modes?

Another idea is, text-within-code-flyspell-mode.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: jinx

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

> Another idea is, text-within-code-flyspell-mode.

How about `syntactical-flyspell-mode'?

Re: jinx

[Arash Esbati]

Richard Stallman <rms@gnu.org> writes:

> Another idea is, text-within-code-flyspell-mode.

ispell.el has `ispell-comments-and-strings', so I'd suggest
`flyspell-comments-and-strings-mode'.

Best, Arash

Re: jinx

[Emanuel Berg]

Arash Esbati wrote:

>> Another idea is, text-within-code-flyspell-mode.
>
> ispell.el has `ispell-comments-and-strings', so I'd suggest
> `flyspell-comments-and-strings-mode'.

Can any of them spell a single word? If not, here is code
for that.

(defun spell-word (word)
  (with-temp-buffer
    (save-excursion
      (insert word) )
    (condition-case nil
        (not (ispell-word))
      (error nil) )))
;; (spell-word "length") ; t
;; (spell-word "lenght") ; nil

(defun spell-this-word (dict)
  "`ispell-word' the word with DICT."
  (ispell-change-dictionary dict)
  (ispell-word) )

(let ((eng-dict "american-insane"))
  (defun spell-english ()
    (interactive)
    (spell eng-dict) )

  (defun word ()
    (interactive)
    (spell-this-word eng-dict) ))
;; test: hello / hxlle

(let ((swe-dict "svenska"))
  (defun spell-swedish ()
    (interactive)
    (spell swe-dict) )

  (defun ord ()
    (interactive)
    (spell-this-word swe-dict) ))
;; test: hej / hxj

Full source:
  https://dataswamp.org/~incal/emacs-init/spell.el

-- 
underground experts united
https://dataswamp.org/~incal

Re: jinx

[Lynn Winebarger]

On Sat, Apr 1, 2023 at 3:20 AM Eli Zaretskii <eliz@gnu.org> wrote:
> > From: Arash Esbati <arash@gnu.org>
> > Cc: rms@gnu.org,  m.eliachevitch@posteo.de,  emacs-devel@gnu.org
> > Date: Fri, 31 Mar 2023 21:35:25 +0200
> >
> > > (I wonder why the built-in TeX support in the spell-checker doesn't do
> > > this job, it's supposed to be adequate, and all Emacs needs to do is
> > > to tell the speller it is working on TeX text.  So I don't even
> > > understand why we need to skip something on our own in these cases.)
> >
> > I can only tell for hunspell: It has built-in support for skipping of
> > many macro arguments and environments[1], but:
> >
> >   a) It is not complete (and probably will never be)
> >   b) It cannot work for user defined macros/environments, and this is
> >   where AUCTeX/ispell shine: AUCTeX can parse user defined
> >   macros/environments and add them automatically to
> >   `ispell-tex-skip-alists'.
> >   c) Another advantage of `ispell-tex-skip-alists' is that one can write
> >   a function and put that in `ispell-tex-skip-alists' for certain
> >   macros/environments: That gives us a lot of flexibility to precisely
> >   skip or check.
>
> These issues are not TeX-specific, so they should be handled in a more
> general way, not bound to TeX/LaTeX.  In particular, any "tex-skip"
> features should not supplant the TeX support built into the speller,
> they should only _add_ skip regexps for stuff that the speller cannot
> support, like user-defined macros.
>
> I'm not sure I understand item c), though: when and why would one want
> to spell-check macros and environments?

If's completely possible to have TeX files with typos in the macros or
environments.  I wouldn't mind having some mechanism for detecting
such typos and suggesting correct possibilities.  Whether that should
be done through a spell-checking interface or not, I don't know.
Obviously I'd want the dictionary for spell-checking those lexemes to
be distinct from the ones used for checking ordinary text.

Please note I am not advocating for such a feature, I am only
responding to your question.

Lynn

Re: jinx

[Augusto Stoffel]

On Fri, 31 Mar 2023 at 21:35, Arash Esbati wrote:

>> (I wonder why the built-in TeX support in the spell-checker doesn't do
>> this job, it's supposed to be adequate, and all Emacs needs to do is
>> to tell the speller it is working on TeX text.  So I don't even
>> understand why we need to skip something on our own in these cases.)
>
> I can only tell for hunspell: It has built-in support for skipping of
> many macro arguments and environments[1], but:
>
>   a) It is not complete (and probably will never be)
>   b) It cannot work for user defined macros/environments, and this is
>   where AUCTeX/ispell shine: AUCTeX can parse user defined
>   macros/environments and add them automatically to
>   `ispell-tex-skip-alists'.
>   c) Another advantage of `ispell-tex-skip-alists' is that one can write
>   a function and put that in `ispell-tex-skip-alists' for certain
>   macros/environments: That gives us a lot of flexibility to precisely
>   skip or check.

A more basic reason why this couldn't work is, the parser is stateful
and you need to look at an arbitrary amount of preceding text to figure
out the context at point.  So you can't rely on the spellchecker TeX
parser if you feed single words or lines into it, as Flyspell et al do.

Re: jinx

[Augusto Stoffel]

On Fri, 31 Mar 2023 at 20:33, Arash Esbati wrote:

> Somewhat off-topic, but it would be great if ispell and flyspell could
> use the same logic for skipping portions of buffer text.  Take for
> example (La)TeX mode: ispell uses `ispell-tex-skip-alists' and flyspell
> uses `flyspell-tex-command-regexp' (which is not very powerful) and then
> a function added to `flyspell-generic-check-word-predicate' (AFAIU).  I
> once wrote an addition for AUCTeX (tex-ispell.el) which extends
> `ispell-tex-skip-alists' for many LaTeX packages but had hard times
> making it work with flyspell as well, skipping multi-line text portions
> was something I couldn't solve for flyspell, e.g.:
>
> \begin{verbatim}
> Please skip the entire text in this environment.
> \end{verbatim}
>
> Best, Arash

The logic used by Flyspell could easily be used by ispell.el, but not
vice-versa.  However, the flyspell-generic-check-word-predicate API also
has some issues (e.g., a minor mode can't plug into it easily).  Also,
it's quite messy in that some modes are supported in flyspell.el, while
all other are expected to implement their own predicate (of course, the
latter is a better approach).

So I'd suggest a generic API on these lines:

#+begin_src emacs-lisp
(defvar spelling-ignore-functions nil
  "Hook used to determine if a word should be excluded from spell checking.
These functions are called with two arguments, the start and end
positions of a word, until a non-nil value is returned.  Any
result other than nil or `never' indicates that the word should
be excluded from spell checking.

Modes derived from `prog-mode' should leave it up to the spell
checker to decide whether or not to ignore comments, strings or
code regions.  They may provide more specific rules, if desired,
but this typically not needed.

MAYBE ALSO: These functions should make no assumptions about the
point position, and are free to move it.")

(defun spelling-ignored-p (start end)
  "Return non-nil if the word between START and END should not be spell checked.
See `spelling-ignore-functions' for information on how this is
determined."
  (save-excursion
    (memq (run-hook-with-args-until-success spelling-ignore-functions start end)
          '(nil never))))
#+end_src

The above would go into a “neutral” place like simple.el.  Each mode
would be responsible to define its own hooks and each spell-checking
package could use it as it sees fit.

Re: jinx

[Eli Zaretskii]

> From: Augusto Stoffel <arstoffel@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  rms@gnu.org,  m.eliachevitch@posteo.de,
>   emacs-devel@gnu.org
> Date: Sat, 01 Apr 2023 10:29:13 +0200
> 
> So I'd suggest a generic API on these lines:
> 
> #+begin_src emacs-lisp
> (defvar spelling-ignore-functions nil
>   "Hook used to determine if a word should be excluded from spell checking.
> These functions are called with two arguments, the start and end
> positions of a word, until a non-nil value is returned.  Any
> result other than nil or `never' indicates that the word should
> be excluded from spell checking.

If we pass START and END, why does the region have to be a single
"word"? why not let the caller specify a larger region to be skipped?

> Modes derived from `prog-mode' should leave it up to the spell
> checker to decide whether or not to ignore comments, strings or
> code regions.  They may provide more specific rules, if desired,
> but this typically not needed.

Why?  I thought the purpose of this is to override what the
spell-checker knows, and that includes comments and strings, if
needed.  So I see no reason for such a recommendation.

> (defun spelling-ignored-p (start end)
>   "Return non-nil if the word between START and END should not be spell checked.
> See `spelling-ignore-functions' for information on how this is
> determined."
>   (save-excursion

Why do we need save-excursion?

> The above would go into a “neutral” place like simple.el.

Why not ispell.el?  All the spelling functions use ispell.el anyway.
simple.el is preloaded, so having this there bloats every Emacs
session for no good reason, IMO.

Re: jinx

[Augusto Stoffel]

On Sat,  1 Apr 2023 at 14:21, Eli Zaretskii wrote:

>> From: Augusto Stoffel <arstoffel@gmail.com>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  rms@gnu.org,  m.eliachevitch@posteo.de,
>>   emacs-devel@gnu.org
>> Date: Sat, 01 Apr 2023 10:29:13 +0200
>> 
>> So I'd suggest a generic API on these lines:
>> 
>> #+begin_src emacs-lisp
>> (defvar spelling-ignore-functions nil
>>   "Hook used to determine if a word should be excluded from spell checking.
>> These functions are called with two arguments, the start and end
>> positions of a word, until a non-nil value is returned.  Any
>> result other than nil or `never' indicates that the word should
>> be excluded from spell checking.
>
> If we pass START and END, why does the region have to be a single
> "word"?

This is a tricky point.  The notion of "word", for spell checking
purposes, is outside of the major mode's control.  For instance, some
languages include the apostrophe or hyphen as a word char, some don't.

Passing START and END means that the major mode can use the information
of how the spell-checker did the tokenization.  Typically, though, I
expect most function will look only at START or END.

>  why not let the caller specify a larger region to be skipped?

Then the return value can't be a boolean; it has to be a list of
regions, which is too complicated.

>> Modes derived from `prog-mode' should leave it up to the spell
>> checker to decide whether or not to ignore comments, strings or
>> code regions.  They may provide more specific rules, if desired,
>> but this typically not needed.
>
> Why?  I thought the purpose of this is to override what the
> spell-checker knows, and that includes comments and strings, if
> needed.  So I see no reason for such a recommendation.

Maybe.  But the above is exactly how Flyspell's
flyspell-generic-check-word-predicate works, and it makes sense IMO.

>> (defun spelling-ignored-p (start end)
>>   "Return non-nil if the word between START and END should not be spell checked.
>> See `spelling-ignore-functions' for information on how this is
>> determined."
>>   (save-excursion
>
> Why do we need save-excursion?

If we remove it here, then most spelling-ignore-functions will need to
call save excursion on their own, which would probably be fine as well.

>> The above would go into a “neutral” place like simple.el.
>
> Why not ispell.el?  All the spelling functions use ispell.el anyway.
> simple.el is preloaded, so having this there bloats every Emacs
> session for no good reason, IMO.

jit-spell only uses ispell.el to start a process and jinx doesn't use it
at all.  To me it makes sense to untangle the spellchecking support API
for major modes from any particular implementation.  But I guess this is
not a major issue.

Re: jinx

[Eli Zaretskii]

> From: Augusto Stoffel <arstoffel@gmail.com>
> Cc: arash@gnu.org,  rms@gnu.org,  m.eliachevitch@posteo.de,
>   emacs-devel@gnu.org
> Date: Sat, 01 Apr 2023 13:39:40 +0200
> 
> On Sat,  1 Apr 2023 at 14:21, Eli Zaretskii wrote:
> 
> > If we pass START and END, why does the region have to be a single
> > "word"?
> 
> This is a tricky point.  The notion of "word", for spell checking
> purposes, is outside of the major mode's control.  For instance, some
> languages include the apostrophe or hyphen as a word char, some don't.
> 
> Passing START and END means that the major mode can use the information
> of how the spell-checker did the tokenization.  Typically, though, I
> expect most function will look only at START or END.
> 
> >  why not let the caller specify a larger region to be skipped?
> 
> Then the return value can't be a boolean; it has to be a list of
> regions, which is too complicated.

I'm afraid I don't quite follow.  I actually don't understand why we
need END here.  Why not call the function with some buffer position,
and let it return nil (meaning don't skip) or a buffer position, which
means skip until that position?

IOW, skipping text in at least some situation needs to skip multiple
words, perhaps even multiple lines, and the skip function should be
allowed to specify that in one go.  Right?

> >> (defun spelling-ignored-p (start end)
> >>   "Return non-nil if the word between START and END should not be spell checked.
> >> See `spelling-ignore-functions' for information on how this is
> >> determined."
> >>   (save-excursion
> >
> > Why do we need save-excursion?
> 
> If we remove it here, then most spelling-ignore-functions will need to
> call save excursion on their own, which would probably be fine as well.

Yes, I don't think we should second-guess what those functions will
do.

> >> The above would go into a “neutral” place like simple.el.
> >
> > Why not ispell.el?  All the spelling functions use ispell.el anyway.
> > simple.el is preloaded, so having this there bloats every Emacs
> > session for no good reason, IMO.
> 
> jit-spell only uses ispell.el to start a process and jinx doesn't use it
> at all.

jinx is not in Emacs, so we don't have to solve its problems.  And
jit-spell uses ispell.el, so it will be able to use any function
there.

> To me it makes sense to untangle the spellchecking support API
> for major modes from any particular implementation.  But I guess this is
> not a major issue.

Not a major issue indeed, at least not yet.  If and when needed, we
could have a general spell.el file that would hold all the
spell-checking stuff not related to particular implementations.
Although in my mind ispell.el is such a "base-line" file already, as
every spelling feature basically uses some of its infrastructure.

Re: jinx

[Augusto Stoffel]

On Sat,  1 Apr 2023 at 14:54, Eli Zaretskii wrote:

> I'm afraid I don't quite follow.  I actually don't understand why we
> need END here.  Why not call the function with some buffer position,
> and let it return nil (meaning don't skip) or a buffer position, which
> means skip until that position?
>
> IOW, skipping text in at least some situation needs to skip multiple
> words, perhaps even multiple lines, and the skip function should be
> allowed to specify that in one go.  Right?

You are making too many assumptions about how the spell checker logic
works.  What you propose would be really handy for ispell-region and the
like but is not usable by Flyspell -- for the same reason Flyspell
doesn't use ispell-skip-region-alist.

We should come up with an API that any spell-checking package, present
of future, could use.

>> > Why not ispell.el?  All the spelling functions use ispell.el anyway.
>> > simple.el is preloaded, so having this there bloats every Emacs
>> > session for no good reason, IMO.
>> 
>> jit-spell only uses ispell.el to start a process and jinx doesn't use it
>> at all.
>
> jinx is not in Emacs, so we don't have to solve its problems.  And
> jit-spell uses ispell.el, so it will be able to use any function
> there.

We don't need to solve jinx's problems, but the API should be convenient
for any third-party package to integrate with, and that's why I brought
up the example.

Re: jinx

[Eli Zaretskii]

> From: Augusto Stoffel <arstoffel@gmail.com>
> Cc: arash@gnu.org,  rms@gnu.org,  m.eliachevitch@posteo.de,
>   emacs-devel@gnu.org
> Date: Sat, 01 Apr 2023 14:32:33 +0200
> 
> On Sat,  1 Apr 2023 at 14:54, Eli Zaretskii wrote:
> 
> > I'm afraid I don't quite follow.  I actually don't understand why we
> > need END here.  Why not call the function with some buffer position,
> > and let it return nil (meaning don't skip) or a buffer position, which
> > means skip until that position?
> >
> > IOW, skipping text in at least some situation needs to skip multiple
> > words, perhaps even multiple lines, and the skip function should be
> > allowed to specify that in one go.  Right?
> 
> You are making too many assumptions about how the spell checker logic
> works.  What you propose would be really handy for ispell-region and the
> like but is not usable by Flyspell -- for the same reason Flyspell
> doesn't use ispell-skip-region-alist.

This depends on where we decide to plug this feature into Flyspell, I
think.  If we plug this into it before it gets to process the result
of flyspell-get-word, then all we need to do is make sure
flyspell-get-word returns nil for stretches of text that need to be
skipped.

> We should come up with an API that any spell-checking package, present
> of future, could use.

I think what I proposed fits the bill.  But it's just an idea.

> >> jit-spell only uses ispell.el to start a process and jinx doesn't use it
> >> at all.
> >
> > jinx is not in Emacs, so we don't have to solve its problems.  And
> > jit-spell uses ispell.el, so it will be able to use any function
> > there.
> 
> We don't need to solve jinx's problems, but the API should be convenient
> for any third-party package to integrate with, and that's why I brought
> up the example.

I don't see how loading a single Lisp file could be of any
inconvenience.

Adding functions to preloaded packages just because we think they will
make it convenient to some third-party package is not TRT.

Re: jinx

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > jinx spell-checks the visible portion of the window.  I don't see why
  > we would need that.  Suppose I scroll the window by 2 lines: why would
  > I want the new visible portion to be spell-checked again, when the
  > overlap with the last check is so large?

You mean, it doesn't remember which part has already been checked?
I took for granted it would optimize by not rechecking what was
already on the window.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: jinx

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: m.eliachevitch@posteo.de, emacs-devel@gnu.org
> Date: Fri, 31 Mar 2023 23:11:55 -0400
> 
>   > jinx spell-checks the visible portion of the window.  I don't see why
>   > we would need that.  Suppose I scroll the window by 2 lines: why would
>   > I want the new visible portion to be spell-checked again, when the
>   > overlap with the last check is so large?
> 
> You mean, it doesn't remember which part has already been checked?

It seems to rely on JIT font-lock mechanism to keep track of what was
already spell-checked.

But my point was not about efficiency, it was about the user
experience: why would I want to have the visible portion marked with
mis-spelling indications at all times?  This is not a natural way of
dealing with spelling mistakes.

And my main point was that if someone does want to have a mode or a
command to mark spelling mistakes in a visible portion of the buffer,
we can easily add that to the existing spell-checking commands and
modes.  No need to switch to a completely different implementation to
have that feature.

Re: jinx

[Augusto Stoffel]

On Sat,  1 Apr 2023 at 08:56, Eli Zaretskii wrote:

>> From: Richard Stallman <rms@gnu.org>
>> Cc: m.eliachevitch@posteo.de, emacs-devel@gnu.org
>> Date: Fri, 31 Mar 2023 23:11:55 -0400
>> 
>>   > jinx spell-checks the visible portion of the window.  I don't see why
>>   > we would need that.  Suppose I scroll the window by 2 lines: why would
>>   > I want the new visible portion to be spell-checked again, when the
>>   > overlap with the last check is so large?
>> 
>> You mean, it doesn't remember which part has already been checked?
>
> It seems to rely on JIT font-lock mechanism to keep track of what was
> already spell-checked.
>
> But my point was not about efficiency, it was about the user
> experience: why would I want to have the visible portion marked with
> mis-spelling indications at all times?  This is not a natural way of
> dealing with spelling mistakes.

FWIW, this is what any word processor out there does.  Personally I find
it much more natural.

Re: jinx

[Emanuel Berg]

Eli Zaretskii wrote:

> We cannot assume that "jinx is always available", since it
> requires Emacs to be linked against the enchant library.
>
> Anyway, after looking at jinx, I must say that I don't see
> why we should replace ispell and flyspell with jinx. From my
> POV, ispell and flyspell provide all the required
> spell-checking functionalities:
>
>   . spell-check a word
>   . spell-check a region
>   . spell-check the entire buffer
>   . spell-check comments and strings in a prog-mode buffer
>   . spell-check as you type

Had to do some work on some of that back when I set it up,
don't know if that has since then been moved into vanilla
Emacs - here is the file anyway

;;; -*- lexical-binding: t -*-
;;
;; this file:
;;   https://dataswamp.org/~incal/emacs-init/spell.el
;;
;; Debian bug:
;;   ispell-init-process: Illegal format hash table
;;   /usr/lib/ispell/svenska.hash - expected magic2 0x9602, got 0x414c
;;   $ sudo buildhash /usr/share/dict/svenska /usr/lib/ispell/svenska.aff /usr/lib/ispell/svenska.hash

(require 'ispell)

(when (eq system-type 'gnu/linux)
  (load-file "/usr/share/dictionaries-common/site-elisp/debian-ispell.el")
  (load-file "/var/cache/dictionaries-common/emacsen-ispell-dicts.el") )

(setq ispell-program-name   "ispell")
(setq ispell-silently-savep t)

;; single word

(defun spell-word (word)
  (with-temp-buffer
    (save-excursion
      (insert word) )
    (condition-case nil
        (not (ispell-word))
      (error nil) )))
;; (spell-word "length") ; t
;; (spell-word "lenght") ; nil

(defun spell-this-word (dict)
  "`ispell-word' the preceding word with DICT."
  (ispell-change-dictionary dict)
  (ispell-word) )

(let ((eng-dict "american-insane"))
  (defun spell-english ()
    (interactive)
    (spell eng-dict) )

  (defun word ()
    (interactive)
    (spell-this-word eng-dict) ))
;; test: hello / hxllo

(let ((swe-dict "svenska"))
  (defun spell-swedish ()
    (interactive)
    (spell swe-dict) )

  (defun ord ()
    (interactive)
    (spell-this-word swe-dict) ))
;; test: hej / hxj

;; buffer / whole / dwim

(defun is-code ()
  (member major-mode '(c++-mode
                       c-mode
                       emacs-lisp-mode
                       prolog-mode
                       python-mode
                       sh-mode
                       Shell-script-mode) )) ; add more ...

(defun is-message ()
  (eq major-mode 'message-mode) )

(defun spell (dict)
  "Spell with DICT.
If region is active, use `ispell-region';
if editing code, `ispell-comments-and-strings';
if writing a message, `ispell-message';
else, `ispell-buffer'"
  (ispell-change-dictionary dict)
  (save-window-excursion
    (save-excursion
      (cond
       ((use-region-p) (ispell-region (region-beginning) (region-end)))
       ((is-message)   (ispell-message))
       ((is-code)      (ispell-comments-and-strings))
       (t              (ispell-buffer)) ))))

-- 
underground experts united
https://dataswamp.org/~incal