bug#51843: Ispell personal dictionary breaks other dictionaries

bug#51843: Ispell personal dictionary breaks other dictionaries

[Tor Kringeland]

Using e.g. `ispell' with an English dictionary and setting
`ispell-personal-dictionary' to, say, `/tmp/words.txt' will add new
words to this file when using this dictionary.  However, if I change the
dictionary to something else (like a Norwegian dictionary), I get the
following output

  Starting new Ispell process /usr/local/bin/aspell with english dictionary...done
  ispell-init-process: Error: Expected language "nn" but got "en".

when running `ispell-word' on a misspelled word.  By extension this
breaks `flyspell-mode'.  If I set `ispell-personal-dictionary' back to
nil, though, things works as usual.

For reference this happens on a build of Emacs 29 from 11 November using
`-Q' on macOS Catalina, though I remember having this problem for quite
some time.

bug#51843: Ispell personal dictionary breaks other dictionaries

[Eli Zaretskii]

> From: Tor Kringeland <tor.a.s.kringeland@ntnu.no>
> Date: Sun, 14 Nov 2021 14:50:01 +0100
> 
> Using e.g. `ispell' with an English dictionary and setting
> `ispell-personal-dictionary' to, say, `/tmp/words.txt' will add new
> words to this file when using this dictionary.  However, if I change the
> dictionary to something else (like a Norwegian dictionary), I get the
> following output
> 
>   Starting new Ispell process /usr/local/bin/aspell with english dictionary...done
>   ispell-init-process: Error: Expected language "nn" but got "en".
> 
> when running `ispell-word' on a misspelled word.  By extension this
> breaks `flyspell-mode'.  If I set `ispell-personal-dictionary' back to
> nil, though, things works as usual.
> 
> For reference this happens on a build of Emacs 29 from 11 November using
> `-Q' on macOS Catalina, though I remember having this problem for quite
> some time.

Isn't this the same problem with the LANG setting in the environment?

bug#51843: Ispell personal dictionary breaks other dictionaries

[Tor Kringeland]

Eli Zaretskii <eliz@gnu.org> writes:

> Isn't this the same problem with the LANG setting in the environment?

I don't think so.  The issues seems to be a disparity between the
language codes for the personal dictionary (you can only have a personal
dictionary for one language) and the main dictionary.  The header of
`/tmp/words.txt' is

  personal_ws-1.1 en 1

where the `en' refers to the language/dictionary and `1' refers to the
number of words in your personal dictionary.  If I initiate a personal
dictionary with, say, Norwegian, the header looks like

  personal_ws-1.1 nn 1

and I get the opposite problem (that it works for Norwegian but not
English).

I suspect the error comes from `ispell' feeding these two disparate
language codes to `aspell': it tries to combine two dictionaries for two
different languages and `aspell' complains.  But I'm not certain this is
the case, as I don't understand well the code in `ispell.el'.

A possible fix would be to have an `ispell-personal-dictionaries-alist'
in which one could specify different personal dictionaries for different
language codes/dictionaries, which would be a feature improvement.
Another fix would be for `ispell' to check for disparity in language
code of the current dictionary and the personal dictionary, and if there
is one, to not feed the personal dictionary into `ispell-program-name'.

Are you able to reproduce the issue?

(Sorry for re-sending, I forgot to CC the mailing list.)

bug#51843: Ispell personal dictionary breaks other dictionaries

[Eli Zaretskii]

[Please use Reply All to reply, so that the bug tracker gets a copy.]

> From: Tor Kringeland <tor.a.s.kringeland@ntnu.no>
> Date: Sun, 14 Nov 2021 16:28:04 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Isn't this the same problem with the LANG setting in the environment?
> 
> I don't think so.  The issues seems to be a disparity between the
> language codes for the personal dictionary (you can only have a personal
> dictionary for one language) and the main dictionary.  The header of
> `/tmp/words.txt' is
> 
>   personal_ws-1.1 en 1
> 
> where the `en' refers to the language/dictionary and `1' refers to the
> number of words in your personal dictionary.  If I initiate a personal
> dictionary with, say, Norwegian, the header looks like
> 
>   personal_ws-1.1 nn 1
> 
> and I get the opposite problem (that it works for Norwegian but not
> English).

How do you initiate the personal dictionary?

> I suspect the error comes from `ispell' feeding these two disparate
> language codes to `aspell': it tries to combine two dictionaries for two
> different languages and `aspell' complains.  But I'm not certain this is
> the case, as I don't understand well the code in `ispell.el'.

Does the same problem happen if you invoke Aspell from the shell,
passing it the Norwegian language and the personal dictionary for
English?

> A possible fix would be to have an `ispell-personal-dictionaries-alist'
> in which one could specify different personal dictionaries for different
> language codes/dictionaries, which would be a feature improvement.
> Another fix would be for `ispell' to check for disparity in language
> code of the current dictionary and the personal dictionary, and if there
> is one, to not feed the personal dictionary into `ispell-program-name'.
> 
> Are you able to reproduce the issue?

I don't use Aspell.  I use Hunspell, where this problem doesn't exist,
and where you can have a separate personal dictionary for each
language.

bug#51843: Ispell personal dictionary breaks other dictionaries

[Tor Kringeland]

Eli Zaretskii <eliz@gnu.org> writes:

> [Please use Reply All to reply, so that the bug tracker gets a copy.]
>
>> From: Tor Kringeland <tor.a.s.kringeland@ntnu.no>
>> Date: Sun, 14 Nov 2021 16:28:04 +0100
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Isn't this the same problem with the LANG setting in the environment?
>> 
>> I don't think so.  The issues seems to be a disparity between the
>> language codes for the personal dictionary (you can only have a personal
>> dictionary for one language) and the main dictionary.  The header of
>> `/tmp/words.txt' is
>> 
>>   personal_ws-1.1 en 1
>> 
>> where the `en' refers to the language/dictionary and `1' refers to the
>> number of words in your personal dictionary.  If I initiate a personal
>> dictionary with, say, Norwegian, the header looks like
>> 
>>   personal_ws-1.1 nn 1
>> 
>> and I get the opposite problem (that it works for Norwegian but not
>> English).
>
> How do you initiate the personal dictionary?

By setting `ispell-personal-dictionary' to point at a non-existent file.
It then creates the file with the language code for the current
dictionary.

>> I suspect the error comes from `ispell' feeding these two disparate
>> language codes to `aspell': it tries to combine two dictionaries for two
>> different languages and `aspell' complains.  But I'm not certain this is
>> the case, as I don't understand well the code in `ispell.el'.
>
> Does the same problem happen if you invoke Aspell from the shell,
> passing it the Norwegian language and the personal dictionary for
> English?

Yes.  If `/tmp/words.txt' is

  personal_ws-1.1 en 1
  mysterious

running

  echo -n mystiks | aspell -a --lang=nn_NO -p /tmp/words.txt

outputs `Error: Expected language "nn" but got "en".' similar to the
error using `ispell' in Emacs.  Running the following similar command
with `hunspell'

  echo -n mystiks | hunspell -a -d nn_NO -p /tmp/words.txt

gives no complaints and suggests the word is a typo for `mystisk'.

One could argue that the behavior of `hunspell' is not the correct one
here, since we're using dictionaries of two different languages and it
could have yielded a warning at least.  But this has nothing to do with
Emacs.

But given that this is "undefined behavior" for spell checkers one can
use with `ispell', it would be good if Emacs could sanity check the
language code of the personal dictionary with that of the main one and
not include the former if there is a disparity --- or optionally being
able to set personal dictionaries for multiple language codes.