bug#32319: 27.0.50; flyspell-mode error on start

bug#32319: 27.0.50; flyspell-mode error on start

[Eric Abrahamsen]

Turning on flyspell-mode started giving me an error a few days ago, on
two different machines I use. Both are Arch linux, both running Emacs
master.

My only ispell customizations:

(setq ispell-program-name (executable-find "hunspell"))
(setq ispell-personal-dictionary "~/.aspell.en_US.pws")

Starting flyspell-mode gives me:

Debugger entered--Lisp error: (wrong-type-argument stringp nil)
  string-match("," nil 0)
  split-string(nil "," t)
  ispell-parse-hunspell-affix-file(nil)
  ispell-hunspell-fill-dictionary-entry(nil)
  ispell-start-process()
  ispell-init-process()
  ispell-buffer-local-words()
  ispell-accept-buffer-local-defs()
  flyspell-accept-buffer-local-defs(force)
  flyspell-mode-on()

From "emacs -Q", setting `ispell-program-name' to (executable-find
"hunspell") (the first line of my customizations above) is sufficient to
trigger the error.

`ispell-current-dictionary' seems to be nil during the whole process,
which causes the eventual error.

The Hunspell version on Arch is "1.6.2", which is the value of
`ispell-really-hunspell'.

Hope that's enough information!

In GNU Emacs 27.0.50 (build 37, x86_64-pc-linux-gnu, GTK+ Version 3.22.30)
 of 2018-07-30 built on clem
Repository revision: 63ef79329935b790b9c8107125bce66e1f272c2e
Windowing system distributor 'The X.Org Foundation', version 11.0.12000000
System Description: Arch Linux

Recent messages:
Entering debugger...
Debug on Error disabled globally
Mark set
Mark activated
Back to top level
Mark set
Quit [2 times]
Mark saved where search started
"/usr/bin/hunspell"
"1.6.2"

bug#32319: 27.0.50; flyspell-mode error on start

[Eli Zaretskii]

> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> Date: Mon, 30 Jul 2018 12:42:24 -0700
> 
> 
> Turning on flyspell-mode started giving me an error a few days ago, on
> two different machines I use. Both are Arch linux, both running Emacs
> master.
> 
> My only ispell customizations:
> 
> (setq ispell-program-name (executable-find "hunspell"))
> (setq ispell-personal-dictionary "~/.aspell.en_US.pws")
> 
> Starting flyspell-mode gives me:
> 
> Debugger entered--Lisp error: (wrong-type-argument stringp nil)
>   string-match("," nil 0)
>   split-string(nil "," t)
>   ispell-parse-hunspell-affix-file(nil)
>   ispell-hunspell-fill-dictionary-entry(nil)
>   ispell-start-process()
>   ispell-init-process()
>   ispell-buffer-local-words()
>   ispell-accept-buffer-local-defs()
>   flyspell-accept-buffer-local-defs(force)
>   flyspell-mode-on()
> 
> >From "emacs -Q", setting `ispell-program-name' to (executable-find
> "hunspell") (the first line of my customizations above) is sufficient to
> trigger the error.

You mean, just evaluating

  (setq ispell-program-name (executable-find "hunspell"))

signals an error??

> `ispell-current-dictionary' seems to be nil during the whole process,
> which causes the eventual error.

Yes, and you need to find out why does that happen.  FWIW, I cannot
reproduce the problem here.  Is ispell-internal-change-dictionary
called when you start spell-checking?  What if you manually set the
dictionary with "M-x ispell-change-dictionary"?

> The Hunspell version on Arch is "1.6.2", which is the value of
> `ispell-really-hunspell'.

Did you upgrade Hunspell lately?

If nothing gives a clue, I'm afraid you will have to step in a
debugger through the ispell.el initialization code and see what's
going wrong there.

bug#32319: 27.0.50; flyspell-mode error on start

[Eric Abrahamsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>> Date: Mon, 30 Jul 2018 12:42:24 -0700
>> 
>> 
>> Turning on flyspell-mode started giving me an error a few days ago, on
>> two different machines I use. Both are Arch linux, both running Emacs
>> master.
>> 
>> My only ispell customizations:
>> 
>> (setq ispell-program-name (executable-find "hunspell"))
>> (setq ispell-personal-dictionary "~/.aspell.en_US.pws")
>> 
>> Starting flyspell-mode gives me:
>> 
>> Debugger entered--Lisp error: (wrong-type-argument stringp nil)
>>   string-match("," nil 0)
>>   split-string(nil "," t)
>>   ispell-parse-hunspell-affix-file(nil)
>>   ispell-hunspell-fill-dictionary-entry(nil)
>>   ispell-start-process()
>>   ispell-init-process()
>>   ispell-buffer-local-words()
>>   ispell-accept-buffer-local-defs()
>>   flyspell-accept-buffer-local-defs(force)
>>   flyspell-mode-on()
>> 
>> >From "emacs -Q", setting `ispell-program-name' to (executable-find
>> "hunspell") (the first line of my customizations above) is sufficient to
>> trigger the error.
>
> You mean, just evaluating
>
>   (setq ispell-program-name (executable-find "hunspell"))
>
> signals an error??

No, sorry, I meant setting the program name to "hunspell" and then doing
anything spelling-related caused an error.

But! This was a false alarm: the problem is a bug in the hunspell_en
package for Arch linux, which is missing some symlinks, and causing
several other programs to fail: 

https://bugs.archlinux.org/task/59401

It might have been nice if ispell allowed the real error to reach me
(during edebugging I did see something like "no affix file found for
"en_US""), but I suppose it isn't a big deal.

Eric

bug#32319: 27.0.50; flyspell-mode error on start

[Eli Zaretskii]

> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> Date: Tue, 31 Jul 2018 19:03:26 -0700
> 
> But! This was a false alarm: the problem is a bug in the hunspell_en
> package for Arch linux, which is missing some symlinks, and causing
> several other programs to fail: 
> 
> https://bugs.archlinux.org/task/59401

So the problem was that the files en_US.*, which spell-checking was
supposed to use, didn't exist, is that right?

> It might have been nice if ispell allowed the real error to reach me
> (during edebugging I did see something like "no affix file found for
> "en_US""), but I suppose it isn't a big deal.

At some point in the initialization, ispell.el looks for the
dictionary file, and if not found, falls back to a user option (which
was nil in your case).  Can you spot where both are nil, and propose a
patch in that place to display a warning or error message?

Thanks.

bug#32319: 27.0.50; flyspell-mode error on start

[Eric Abrahamsen]

On 08/01/18 09:00 AM, Eli Zaretskii wrote:
>> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>> Date: Tue, 31 Jul 2018 19:03:26 -0700
>> 
>> But! This was a false alarm: the problem is a bug in the hunspell_en
>> package for Arch linux, which is missing some symlinks, and causing
>> several other programs to fail: 
>> 
>> https://bugs.archlinux.org/task/59401
>
> So the problem was that the files en_US.*, which spell-checking was
> supposed to use, didn't exist, is that right?

That's correct. My LANG is set to en_US.UTF-8, so I guess hunspell
automatically tries to load the en_US dictionary. With the correct
symlink in place, "hunspell -D" produces:

SEARCH PATH:
.:: <elided>
AVAILABLE DICTIONARIES
/usr/share/hunspell/en_TT
<more>
LOADED DICTIONARY:
/usr/share/hunspell/en_US.aff
/usr/share/hunspell/en_US.dic

`ispell-find-hunspell-dictionaries' cycles over each line of the above
output and looks for "\\.aff$" (ispell.el:1120). If it finds it, the
local variable `hunspell-default-dictionary' is set from that value,
which eventually finds its way into `ispell-hunspell-dictionary-alist',
and things proceed normally from there.

If the symlink is not present, the "hunspell -D" output looks like:

SEARCH PATH:.:: <elided>
AVAILABLE DICTIONARIES
/usr/share/hunspell/en_TT
<more>
Can't open affix or dictionary files for dictionary named "en_US".

IOW, there's no line that matches "\\.aff$", and
`hunspell-default-dictionary' is never set, and the error is raised in
line 1177, where it's assumed that that variable has a value.

So things go wrong pretty early on, as soon as Hunspell fails to load a
dictionary.

>> It might have been nice if ispell allowed the real error to reach me
>> (during edebugging I did see something like "no affix file found for
>> "en_US""), but I suppose it isn't a big deal.
>
> At some point in the initialization, ispell.el looks for the
> dictionary file, and if not found, falls back to a user option (which
> was nil in your case).  Can you spot where both are nil, and propose a
> patch in that place to display a warning or error message?

I suppose if the user has asked to use Hunspell, and Hunspell has failed
to load a dictionary, we should bail with a warning at that stage?

Alternately, I tried wrapping lines 1176-1180, at the bottom of
`ispell-find-hunspell-dictionaries', in a (when
hunspell-default-dictionary ...), so that essentially nothing gets set.
When I then ran `ispell-buffer', I got:

Can't open affix or dictionary files for dictionary named "en_US".

Which is the error returned by hunspell itself, and useful information.

What I didn't test was whether having a user option in place, specifying
a dictionary to use, would prevent even that error from being signaled.
Mostly because I don't know which option that would be -- I find the
proliferation of dictionary-holding variables pretty confusing. But I
suppose the ideal situation would be: having a valid option set would
swallow the error and allow the process to continue as normal.

Eric

bug#32319: 27.0.50; flyspell-mode error on start

[Eli Zaretskii]

> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> Cc: 32319@debbugs.gnu.org
> Date: Wed, 01 Aug 2018 12:51:54 -0700
> 
> If the symlink is not present, the "hunspell -D" output looks like:
> 
> SEARCH PATH:.:: <elided>
> AVAILABLE DICTIONARIES
> /usr/share/hunspell/en_TT
> <more>
> Can't open affix or dictionary files for dictionary named "en_US".
> 
> IOW, there's no line that matches "\\.aff$", and
> `hunspell-default-dictionary' is never set, and the error is raised in
> line 1177, where it's assumed that that variable has a value.
> 
> So things go wrong pretty early on, as soon as Hunspell fails to load a
> dictionary.

Does the patch below produce a useful diagnostic?

> What I didn't test was whether having a user option in place, specifying
> a dictionary to use, would prevent even that error from being signaled.
> Mostly because I don't know which option that would be -- I find the
> proliferation of dictionary-holding variables pretty confusing. But I
> suppose the ideal situation would be: having a valid option set would
> swallow the error and allow the process to continue as normal.

AFAIK, with Hunspell we get the default dictionary from Hunspell
itself, not from the user.

Here's the proposed patch, let me know what you think:

diff --git a/lisp/textmodes/ispell.el b/lisp/textmodes/ispell.el
index e77bc7e..321d7dd 100644
--- a/lisp/textmodes/ispell.el
+++ b/lisp/textmodes/ispell.el
@@ -1180,6 +1180,8 @@ ispell-find-hunspell-dictionaries
     ;; Parse and set values for default dictionary.
     (setq hunspell-default-dict (or hunspell-multi-dict
 				    (car hunspell-default-dict)))
+    (or hunspell-default-dict
+        (error "Can't find Hunspell dictionary with an .aff affix file"))
     (setq hunspell-default-dict-entry
 	  (ispell-parse-hunspell-affix-file hunspell-default-dict))
     ;; Create an alist of found dicts with only names, except for default dict.

bug#32319: 27.0.50; flyspell-mode error on start

[Eric Abrahamsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>> Cc: 32319@debbugs.gnu.org
>> Date: Wed, 01 Aug 2018 12:51:54 -0700
>> 
>> If the symlink is not present, the "hunspell -D" output looks like:
>> 
>> SEARCH PATH:.:: <elided>
>> AVAILABLE DICTIONARIES
>> /usr/share/hunspell/en_TT
>> <more>
>> Can't open affix or dictionary files for dictionary named "en_US".
>> 
>> IOW, there's no line that matches "\\.aff$", and
>> `hunspell-default-dictionary' is never set, and the error is raised in
>> line 1177, where it's assumed that that variable has a value.
>> 
>> So things go wrong pretty early on, as soon as Hunspell fails to load a
>> dictionary.
>
> Does the patch below produce a useful diagnostic?

This patch is just fine, and would certainly be a lot better than the
current situation. I only note that, if we allow this function to fail
silently, the error you get *later on*, when you try to use flyspell to
do something, comes directly from hunspell and says exactly what the
problem is (ie, which dictionary it tried and failed to load). That's
even more helpful. If we did that, this function could just call
`ispell-print-if-debug' and log the failure quietly.

But raising an error directly is also fine, and might be preferable in
order to avoid creating weird problems further down the line.

Eric

bug#32319: 27.0.50; flyspell-mode error on start

[Eli Zaretskii]

> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> Cc: 32319@debbugs.gnu.org
> Date: Sat, 04 Aug 2018 22:21:47 -0700
> 
> > Does the patch below produce a useful diagnostic?
> 
> This patch is just fine, and would certainly be a lot better than the
> current situation. I only note that, if we allow this function to fail
> silently, the error you get *later on*, when you try to use flyspell to
> do something, comes directly from hunspell and says exactly what the
> problem is (ie, which dictionary it tried and failed to load). That's
> even more helpful. If we did that, this function could just call
> `ispell-print-if-debug' and log the failure quietly.
> 
> But raising an error directly is also fine, and might be preferable in
> order to avoid creating weird problems further down the line.

OK, so I installed my patch on the master branch, and I'm marking this
bug done.  Thanks for the analysis and the feedback.