Hunspell 1.6 on Msys2

Hunspell 1.6 on Msys2

[Arash Esbati]

[-- Attachment #1: Type: text/plain, Size: 1311 bytes --]

Dear all,

Msys2 is delivering Hunspell 1.6 and I ran into this problem on Windows
10.  Recipe:

1) emacs -q &

2) Open a file like

    \documentclass{article}

    \begin{document}
    Übersetzugn
    \end{document}

3)  M-x ispell-change-dictionary RET deutsch8 RET

4)  M-x toggle-debug-on-error RET

5)  M-x ispell

Debugger says:

--8<---------------cut here---------------start------------->8---
Debugger entered--Lisp error: (error "Ispell misalignment: word ‘\334bersetzugn’ point 43; probably incompatible versions")
  signal(error ("Ispell misalignment: word ‘\334bersetzugn’ point 43; probably incompatible versions"))
  error("Ispell misalignment: word `%s' point %d; probably incompatible versions" "\334bersetzugn" 43)
  ispell-process-line("^Übersetzugn\n" nil)
  ispell-region(1 71)
  ispell-buffer()
  ispell()
  funcall-interactively(ispell)
  call-interactively(ispell record nil)
  command-execute(ispell record)
  execute-extended-command(nil "ispell" nil)
  funcall-interactively(execute-extended-command nil "ispell" nil)
  call-interactively(execute-extended-command nil nil)
  command-execute(execute-extended-command)
--8<---------------cut here---------------end--------------->8---

The same procedure works with hunspell 1.4.1.


[-- Attachment #2: ispell_1.4.1.jpg --]
[-- Type: image/jpeg, Size: 20224 bytes --]

[-- Attachment #3: Type: text/plain, Size: 43 bytes --]


Can somebody reproduce this?

Best, Arash

Re: Hunspell 1.6 on Msys2

[Eli Zaretskii]

> From: Arash Esbati <arash@gnu.org>
> Date: Tue, 17 Jan 2017 23:16:51 +0100
> 
> Msys2 is delivering Hunspell 1.6 and I ran into this problem on Windows
> 10.  Recipe:
> 
> 1) emacs -q &
> 
> 2) Open a file like
> 
>     \documentclass{article}
> 
>     \begin{document}
>     Übersetzugn
>     \end{document}
> 
> 3)  M-x ispell-change-dictionary RET deutsch8 RET
> 
> 4)  M-x toggle-debug-on-error RET
> 
> 5)  M-x ispell
> 
> Debugger says:
> 
> --8<---------------cut here---------------start------------->8---
> Debugger entered--Lisp error: (error "Ispell misalignment: word ‘\334bersetzugn’ point 43; probably incompatible versions")
>   signal(error ("Ispell misalignment: word ‘\334bersetzugn’ point 43; probably incompatible versions"))

If you turn on ispell.el's debugging, what does it tell about this?

Re: Hunspell 1.6 on Msys2

[Arash Esbati]

[-- Attachment #1: Type: text/plain, Size: 1073 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arash Esbati <arash@gnu.org>
>> Date: Tue, 17 Jan 2017 23:16:51 +0100
>> 
>> Debugger says:
>> 
>> --8<---------------cut here---------------start------------->8---
>> Debugger entered--Lisp error: (error "Ispell misalignment: word ‘\334bersetzugn’ point 43; probably incompatible versions")
>>   signal(error ("Ispell misalignment: word ‘\334bersetzugn’ point 43; probably incompatible versions"))
>
> If you turn on ispell.el's debugging, what does it tell about this?

Thanks for your response.  I think the relevant part of running
`ispell-buffer-with-debug' is:

--8<---------------cut here---------------start------------->8---
ispell-region: string52 pos (43->54), eol: 54, [in-comment]: [nil], [add-comment]: [nil], [string]: [^Übersetzugn
]
ispell-process-line: Ispell misalignment error:
  [Word from ispell pipe]: [\334bersetzugn], actual (point,line,column): (43,4,0)
--8<---------------cut here---------------end--------------->8---

Complete output is attached.

Best, Arash


[-- Attachment #2: ispell-debug --]
[-- Type: application/octet-stream, Size: 2265 bytes --]

Re: Hunspell 1.6 on Msys2

[Eli Zaretskii]

> From: Arash Esbati <arash@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Wed, 18 Jan 2017 11:22:55 +0100
> 
> --8<---------------cut here---------------start------------->8---
> ispell-region: string52 pos (43->54), eol: 54, [in-comment]: [nil], [add-comment]: [nil], [string]: [^Übersetzugn
> ]
> ispell-process-line: Ispell misalignment error:
>   [Word from ispell pipe]: [\334bersetzugn], actual (point,line,column): (43,4,0)
> --8<---------------cut here---------------end--------------->8---

Looks like some encoding issue?  Can you tell what is the command line
with which Hunspell is invoked?  (You should be able to see it in Task
Manager or in Process Explorer.)

Also, is this in "emacs -Q", or do you have any customizations
relevant to spell-checking?

Re: Hunspell 1.6 on Msys2

[Arash Esbati]

Eli Zaretskii <eliz@gnu.org> writes:

> Looks like some encoding issue?  Can you tell what is the command line
> with which Hunspell is invoked?

`M-x list-processes RET' says:

    c:/msys64/mingw64/bin/hunspell.exe -a  -d de_DE -i UTF-8

Looks ok to me (-t flag for TeX missing, but that should not be
relevant, I think)

> Also, is this in "emacs -Q", or do you have any customizations
> relevant to spell-checking?

My first try was with "emacs -q", now I tried with "emacs -Q", same
result.  My customizations were not involved.

Best, Arash

Re: Hunspell 1.6 on Msys2

[Eli Zaretskii]

> From: Arash Esbati <arash@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Wed, 18 Jan 2017 18:57:35 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Looks like some encoding issue?  Can you tell what is the command line
> > with which Hunspell is invoked?
> 
> `M-x list-processes RET' says:
> 
>     c:/msys64/mingw64/bin/hunspell.exe -a  -d de_DE -i UTF-8
> 
> Looks ok to me (-t flag for TeX missing, but that should not be
> relevant, I think)
> 
> > Also, is this in "emacs -Q", or do you have any customizations
> > relevant to spell-checking?
> 
> My first try was with "emacs -q", now I tried with "emacs -Q", same
> result.  My customizations were not involved.

Very strange.  What version of Emacs is that?

Re: Hunspell 1.6 on Msys2

[Arash Esbati]

Eli Zaretskii <eliz@gnu.org> writes:

> Very strange.  What version of Emacs is that?

25.1.91.1

I build Emacs myself, the value of `system-configuration-options' is:

"--host=x86_64-w64-mingw32
 --target=x86_64-w64-mingw32
 --build=x86_64-w64-mingw32
 --with-jpeg
 --with-xpm
 --with-png
 --with-tiff
 --with-rsvg
 --with-xml2
 --with-gnutls
 --with-xft
 --without-imagemagick
 --without-dbus
 'CFLAGS=-O3 -g0 -pipe'
 CPPFLAGS=-DNDEBUG
 'LDFLAGS=-s -Wl,-s'"

Best, Arash

Re: Hunspell 1.6 on Msys2

[Eli Zaretskii]

> From: Arash Esbati <arash@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Wed, 18 Jan 2017 19:39:41 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Very strange.  What version of Emacs is that?
> 
> 25.1.91.1

Then I'm out of ideas: I downloaded the MSYS2 Hunspell 1.6, and
couldn't reproduce this with Emacs 25.1.91.  So the only other factor
I could think of is the dictionary you are using (although I cannot
imagine how that could cause such problems), because otherwise I
simply don't see any problems with this combination.

Sorry.

Re: Hunspell 1.6 on Msys2

[Eli Zaretskii]

> Date: Wed, 18 Jan 2017 21:15:45 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > From: Arash Esbati <arash@gnu.org>
> > Cc: emacs-devel@gnu.org
> > Date: Wed, 18 Jan 2017 19:39:41 +0100
> > 
> > Eli Zaretskii <eliz@gnu.org> writes:
> > 
> > > Very strange.  What version of Emacs is that?
> > 
> > 25.1.91.1
> 
> Then I'm out of ideas:

Actually, one other idea: do you have any LC_* variables in the
environment?  (MinGW programs shouldn't pay attention, but maybe
MinGW64/MSYS2 guys did something so they do.)

Re: Hunspell 1.6 on Msys2

[Arash Esbati]

Eli Zaretskii <eliz@gnu.org> writes:

> Actually, one other idea: do you have any LC_* variables in the
> environment?  (MinGW programs shouldn't pay attention, but maybe
> MinGW64/MSYS2 guys did something so they do.)

Many thanks for taking your time and checking!  locale returns:

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_ALL=

Do you have other settings?

Best, Arash

Re: Hunspell 1.6 on Msys2

[Eli Zaretskii]

> From: Arash Esbati <arash@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Wed, 18 Jan 2017 20:54:29 +0100
> 
> > Actually, one other idea: do you have any LC_* variables in the
> > environment?  (MinGW programs shouldn't pay attention, but maybe
> > MinGW64/MSYS2 guys did something so they do.)
> 
> Many thanks for taking your time and checking!  locale returns:
> 
> LANG=en_US.UTF-8
> LC_CTYPE="en_US.UTF-8"
> LC_NUMERIC="en_US.UTF-8"
> LC_TIME="en_US.UTF-8"
> LC_COLLATE="en_US.UTF-8"
> LC_MONETARY="en_US.UTF-8"
> LC_MESSAGES="en_US.UTF-8"
> LC_ALL=
> 
> Do you have other settings?

Is this in the MSYS Bash?  I meant inside Emacs.

I have none of these.  Windows runtime ignores these variables, but
maybe MSYS2 Hunspell doesn't.  What happens if you set LC_ALL=C and
try your recipe again?

Re: Hunspell 1.6 on Msys2

[Arash Esbati]

[-- Attachment #1: Type: text/plain, Size: 899 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

> Is this in the MSYS Bash?  I meant inside Emacs.

I tried all of the following:

1)  Started Msys bash by clicking on mingw64.exe

2)  Made a test.bat containing
    msys2_shell.cmd -mingw64 -mintty

3)  At Windows cmd, typed
    bash -c 'MSYSTEM=MINGW64 exec /usr/bin/bash --login -i'

How do you start Msys?

> I have none of these.  Windows runtime ignores these variables, but
> maybe MSYS2 Hunspell doesn't.  What happens if you set LC_ALL=C and
> try your recipe again?

I set that in .bash_profile with the combinations mentioned above, but
no avail.  I'm out of ideas, but since it works for you, the problem
must be at my end.  For now, I will downgrade Hunspell and keeping
searching.  Many thanks again for looking at this.

While we're at it, if found a small typo in the doc-string of
`ispell-buffer-with-debug', patch is attached.

Best, Arash


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-lisp-textmodes-ispell.el-ispell-buffer-with-debug-De.patch --]
[-- Type: text/x-patch, Size: 916 bytes --]

From 3bebc3d473884fa27392ed45d7707ec9632fe659 Mon Sep 17 00:00:00 2001
From: Arash Esbati <arash@gnu.org>
Date: Thu, 19 Jan 2017 09:26:15 +0100
Subject: [PATCH] ; * lisp/textmodes/ispell.el (ispell-buffer-with-debug):
 Delete space.

---
 lisp/textmodes/ispell.el | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lisp/textmodes/ispell.el b/lisp/textmodes/ispell.el
index d9a1c71..6920de4 100644
--- a/lisp/textmodes/ispell.el
+++ b/lisp/textmodes/ispell.el
@@ -3803,7 +3803,7 @@ ispell-buffer
 ;;;###autoload
 (defun ispell-buffer-with-debug (&optional append)
   "`ispell-buffer' with some output sent to `ispell-debug-buffer' buffer.
-If APPEND is non-n il, append the info to previous buffer if exists."
+If APPEND is non-nil, append the info to previous buffer if exists."
   (interactive)
   (let ((ispell-debug-buffer (ispell-create-debug-buffer append)))
     (ispell-buffer)))
-- 
2.10.2

Re: Hunspell 1.6 on Msys2

[Eli Zaretskii]

> From: Arash Esbati <arash@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Thu, 19 Jan 2017 10:04:54 +0100
> 
> Debugger entered--Lisp error: (error "Ispell misalignment: word 
> ‘\334bersetzugn’ point 43; probably incompatible versions")

Forget about all I said, I've succeeded in reproducing this now.
(Previously, I used "M-$" instead of "M-x ispell", and ispell-word
somehow succeeds to work in this case.)

This is a bug in Hunspell 1.6: it seems to ignore the "-i UTF-8"
command-line switch, and sends its output in Latin-1 (perhaps because
the de_DE dictionary uses that encoding).  With a file to-spell.tex
using the same text you show in your recipe and encoded in UTF-8, try
this from the Windows command line:

  cat to-spell.tex | hunspell -a "" -d de_DE -i UTF-8 > hunspell-1.6.txt

Then visit the file hunspell-1.6.txt -- you will see that it shows
Übersetzugn in Latin-1 encoding, although the -i switch requested that
the UI be in UTF-8.

It's possible that this is a kludgey "feature" in Hunspell 1.6, meant
as a stop-gap for the long-standing bug in Hunspell, whereby it
reports offsets in bytes, not in characters.  It could be that the
Hunspell developers made this change in behavior to make the problem
less acute.  But it's a bug anyway.

The solution is to fix Hunspell, of course.  Failing that, a
workaround would be to customize your Emacs to use single-byte
encodings for dictionaries with which you need to work.  One way to do
that is to set up the coding-systems of the Hunspell process
accordingly.  But that could be messy, as ispell.el is quite set on
using UTF-8 with Hunspell and Aspell.

Re: Hunspell 1.6 on Msys2

[Arash Esbati]

Eli Zaretskii <eliz@gnu.org> writes:

> This is a bug in Hunspell 1.6: it seems to ignore the "-i UTF-8"
> command-line switch, and sends its output in Latin-1 (perhaps because
> the de_DE dictionary uses that encoding).

Many thanks for looking into this again and the description.  I will see
how to tell Hunspell developers about this (I don't have a Github
account).

Best, Arash