Debugging aspell problems

Debugging aspell problems

[Juan José García-Ripoll]

Hi,

I have faced this same problem with Mingw64's aspell.exe and Latex mode
https://www.reddit.com/r/emacs/comments/fuf0fp/aspell_broke_latex/
when running Emacs on Windows 10.

Actually, the problem can be simplified to this:
- Set ispell-program-name to "c:/msys64/mingw64/bin/aspell.exe"
- Create two files, foo.txt and foo.tex, in fundamental and latex modes
- In both, write the word "buffer" (without quotes)
- Open foo.txt and type M-$. Ispell outputs that the word is correct
- Open foo.tex and do the same. Ispell locks with no output

I have set traces on ispell-filter and process-send-string and the
problem is that in text mode the whole process succeeds (see log below),
but when using latex mode, ispell-filter never gets called. My suspicion
is that process-send-string does not flush the process' input buffer,
and aspell.exe waits forever but I do not know how to debug this issue.

This problem dates back to 26.3 and is stil present in 27.rc1 Some help
on how to debug it would be appreciated.

Juanjo

--- Running on foo.txt

======================================================================
1 -> (ispell-filter #<process ispell> "@(#) International Ispell Version 3.1.20 (but really Aspell 0.60.7)
")
1 <- ispell-filter: nil
======================================================================
1 -> (process-send-string #<process ispell> "!
")
1 <- process-send-string: nil
======================================================================
1 -> (process-send-string #<process ispell> "-
")
1 <- process-send-string: nil
======================================================================
1 -> (ispell--run-on-word "buffer")
| 2 -> (process-send-string #<process ispell> "%
")
| 2 <- process-send-string: nil
| 2 -> (process-send-string #<process ispell> "^buffer
")
| 2 <- process-send-string: nil
| 2 -> (ispell-filter #<process ispell> "*

")
| 2 <- ispell-filter: nil
1 <- ispell--run-on-word: t

--- Running on foo.tex

======================================================================
1 -> (process-send-string #<process ispell> "!
")
1 <- process-send-string: nil
======================================================================
1 -> (process-send-string #<process ispell> "+
")
1 <- process-send-string: nil
======================================================================
1 -> (ispell--run-on-word "buffer")
| 2 -> (process-send-string #<process ispell> "%
")
| 2 <- process-send-string: nil
| 2 -> (process-send-string #<process ispell> "^buffer
")
| 2 <- process-send-string: nil
1 <- ispell--run-on-word: !non-local\ exit!

I had to break using Ctrl-G

-- 
Juan José García Ripoll
http://juanjose.garciaripoll.com
http://quinfog.hbar.es

Re: Debugging aspell problems

[Eli Zaretskii]

> From: Juan José García-Ripoll
>  <juanjose.garciaripoll@gmail.com>
> Date: Wed, 05 Aug 2020 13:04:38 +0200
> 
> - Set ispell-program-name to "c:/msys64/mingw64/bin/aspell.exe"
> - Create two files, foo.txt and foo.tex, in fundamental and latex modes
> - In both, write the word "buffer" (without quotes)
> - Open foo.txt and type M-$. Ispell outputs that the word is correct
> - Open foo.tex and do the same. Ispell locks with no output
> 
> I have set traces on ispell-filter and process-send-string and the
> problem is that in text mode the whole process succeeds (see log below),
> but when using latex mode, ispell-filter never gets called. My suspicion
> is that process-send-string does not flush the process' input buffer,
> and aspell.exe waits forever but I do not know how to debug this issue.

This works for me with Hunspell (I don't have Aspell installed), so
it's unlikely to be a problem on the Emacs side.  I'm guessing that
your Aspell is actually a MSYS2 program, not a MinGW (a.k.a. "native
Windows") program, so you have these buffering inconsistencies.

Or, if the above is not in "emacs -Q", perhaps some of your
customizations get in the way.

Re: Debugging aspell problems

[Juan José García-Ripoll]

I agree that it may be a problem with the program itself, but it comes
from mingw, not from msys, and I am running emacs.exe -Q and I am seeing
buffering problems also with standard scripts from Windows.

I have created two Windows scripts to make it more reproducible. One is
called chain.cmd and the other one is myaspell.cmd. The first one simply
pulls lines of text and copies them to a file

@echo off
rem chain.cmd
:begin
set /p line= || goto :end
echo %line% >> %1
echo %line%
goto :begin
:end
echo EOF >> %1

The second one calls aspell.exe, using this tool to save the input
(inlog) and the output (outlog)

@echo off
rem myaspell.cmd
set program=c:\msys64\mingw64\bin\aspell.exe
set log=c:\Users\juanj\tmp\inlog
echo %program%  %1 %2 %3 %4 %5 %6 >> %log%
echo Emacs input >> %log%
c:\Users\juanj\tmp\chain.cmd %log% | %program%  %1 %2 %3 %4 %5 %6 | c:\Users\juanj\tmp\chain.cmd c:\Users\juanj\tmp\outlog

Now I use the Lisp file below to run the spell checker on a fresh copy
of Emacs, as in

runemacs -Q --load test.el

The output in alllog is this:

c:\Users\juanj\tmp>type inlog
type inlog
c:\msys64\mingw64\bin\aspell.exe  -vv
Emacs input
EOF
c:\msys64\mingw64\bin\aspell.exe  dicts
Emacs input
EOF
c:\msys64\mingw64\bin\aspell.exe  config dict-dir
Emacs input
EOF
c:\msys64\mingw64\bin\aspell.exe  config data-dir
Emacs input
EOF
c:\msys64\mingw64\bin\aspell.exe  -vv
Emacs input
EOF
c:\msys64\mingw64\bin\aspell.exe  -a -m -B --encoding utf-8
Emacs input
!
+

The Emacs copy is wating for output but has not flushed out the line
buffer that contains the word. Note that here the script that is waiting
for input is chain.cmd, not aspell. Hence, one cannot really claim that
the problem is msys.

---
(require 'ispell)
;;(setq ispell-program-name (expand-file-name "~/scoop/apps/hunspell/bin/hunspell.exe"))
;;(setq ispell-program-name (expand-file-name "c:/msys64/mingw64/bin/aspell.exe"))
(setq ispell-program-name (expand-file-name "myaspell.cmd"))
(setq ispell-async-processp t)

(defun mydebug (&rest args)
  (apply 'print args))

(defadvice ispell-filter (before show first (_process output) activate)
  (mydebug (format "Receiving: %s" output)))

(defadvice ispell-call-process-region (before show first (start end &rest args) activate)
  (mydebug (format "Sending: %s" (buffer-substring start end))))

(trace-function 'mydebug)
(trace-function 'start-process)
(trace-function 'process-send-string)
(trace-function 'ispell-filter)
(trace-function 'ispell--run-on-word)
(trace-function 'ispell-call-process-region)
(trace-function 'call-process)
(trace-function 'call-process-region)

(find-file "foo.tex")
(switch-to-buffer "foo.tex")
(goto-char 1)
(insert "hello ")
(goto-char 1)
(ispell-word)

(kill-buffer "foo.tex")


--
Juan José García Ripoll
http://juanjose.garciaripoll.com
http://quinfog.hbar.es

Re: Debugging aspell problems

[Eli Zaretskii]

> From: Juan José García-Ripoll
>  <juanjose.garciaripoll@gmail.com>
> Date: Wed, 05 Aug 2020 21:32:48 +0200
> 
> I have created two Windows scripts to make it more reproducible. One is
> called chain.cmd and the other one is myaspell.cmd. The first one simply
> pulls lines of text and copies them to a file
> [...]
> c:\msys64\mingw64\bin\aspell.exe  -a -m -B --encoding utf-8
> Emacs input
> !
> +
> 
> The Emacs copy is wating for output but has not flushed out the line
> buffer that contains the word. Note that here the script that is waiting
> for input is chain.cmd, not aspell. Hence, one cannot really claim that
> the problem is msys.

Thanks for the data points, but I'm afraid I'm not convinced.  You've
used a very complicated setup, introducing many additional unknowns
into the equation: cmd.exe, batch files, redirection, more pipes, etc.
This could completely hide the real issues, and at best complicate the
investigation to a great extent.

Let's use much more direct method: Process Explorer from Sysinternals.
Please configure the Process Explorer (via View->Select Columns) so
that it shows the following attributes for each process:

 . I/O Writes
 . I/O Write Bytes

Then start "emacs -Q", configure it to use Aspell, create the
LaTeX-mode buffer with the word "buffer", type "M-$" several times,
and watch the above 2 counters.

What I see on my system (with Hunspell) is that each time I type M-$,
the Emacs process's I/O Writes counter increases by 4, and the I/O
Write Bytes increases by 18.  Which is exactly what I would expect,
since ispell.el sends these 4 strings to the speller:

   !\n
   +\n
   %\n
   ^buffer\n

and each \n character gets converted into a \r\n pair by low-level
I/O code in w32.c.

These results tell me that all the stuff that needs to be written to
the pipe does indeed get written, and no buffering issue is involved.
Which makes sense, since the Emacs implementation of subprocess I/O on
MS-Windows uses unbuffered writes, and on top of that each command we
send to the speller is terminated by a newline, so flushing buffers
should not be relevant to this scenario.

Do you see the same results, or do you see something different?

Stepping back a notch, I don't really see how buffering could be the
first suspect, given your description.  You are saying that the same
procedure works with foo.txt buffer in Text mode, and fails in LaTeX
mode, but the low-level I/O code which could be responsible for some
kind of buffering has no idea about the major mode of the buffer which
is being spell-checked.  Only ispell.el, on the one side, and the
speller itself on the other side, are aware of the differences between
Text mode and LaTeX mode.  So the explanation is likely to be found in
one of these two places, not in the low-level buffering machinery.
Since your tracing shows that process-send-string did get called with
"^buffer" in the LaTeX case, IMO the first possible reasons to explore
are:

  . does Aspell respond to the ^buffer request, when in "tex" mode?
  . do you perhaps have some local/private dictionary that affects the
    issue?

Re: Debugging aspell problems

[Juan José García-Ripoll]

Eli Zaretskii <eliz@gnu.org> writes:
> Let's use much more direct method: Process Explorer from Sysinternals.
> Please configure the Process Explorer (via View->Select Columns) so
> that it shows the following attributes for each process:

Thanks, this is indeed more direct. I did not know it existed. And I can
confirm your results.

> What I see on my system (with Hunspell) is that each time I type M-$,
> the Emacs process's I/O Writes counter increases by 4, and the I/O
> Write Bytes increases by 18.  Which is exactly what I would expect,
> since ispell.el sends these 4 strings to the speller:
>
>    !\n
>    +\n
>    %\n
>    ^buffer\n

I see now the problem. Emacs uses +\n to set aspell into tex mode while
it uses -\n for fundamental or other text modes. When +\n is entered,
mingw64's aspell crashes.

>   . do you perhaps have some local/private dictionary that affects the
>     issue?

No, everything is a pretty much stock configuration.

Sorry for the noise.

-- 
Juan José García Ripoll
http://juanjose.garciaripoll.com
http://quinfog.hbar.es

Re: Debugging aspell problems

[Eli Zaretskii]

> From: Juan José García-Ripoll
>  <juanjose.garciaripoll@gmail.com>
> Date: Thu, 06 Aug 2020 17:12:56 +0200
> 
> I see now the problem. Emacs uses +\n to set aspell into tex mode while
> it uses -\n for fundamental or other text modes. When +\n is entered,
> mingw64's aspell crashes.

OK, so I guess this needs to be reported to aspell developers, either
to whoever did the MinGW64 port or to upstream.

Thanks.

Re: Debugging aspell problems

[Juan José García-Ripoll]

Done. I leave here the link, just for the record
   https://github.com/msys2/MSYS2-packages/issues/2088
Cheers,

Juanjo

Eli Zaretskii <eliz@gnu.org> writes:
>> From: Juan José García-Ripoll
>>  <juanjose.garciaripoll@gmail.com>
>> Date: Thu, 06 Aug 2020 17:12:56 +0200
>> 
>> I see now the problem. Emacs uses +\n to set aspell into tex mode while
>> it uses -\n for fundamental or other text modes. When +\n is entered,
>> mingw64's aspell crashes.
>
> OK, so I guess this needs to be reported to aspell developers, either
> to whoever did the MinGW64 port or to upstream.
>
> Thanks.
>
>

-- 
Juan José García Ripoll
http://juanjose.garciaripoll.com
http://quinfog.hbar.es