bug#7668: ispell and dictionary encodings

bug#7668: ispell and dictionary encodings

[Reuben Thomas]

I've just been puzzling my way through ispell.gz's dictionary encoding
code, after switching from aspell to hunspell in order to be able to
treat Unicode curly single quotes as normal intraword punctuation
(which it seems aspell cannot be persuaded to do, but that's another
story).

I noticed a feature of ispell-dictionary-base-alist, which I don't
understand: the last (7th) element of each dictionary definition is
called "Coding System", which seems to be the coding system of the
case character and non-case-character strings, but it is also passed
to the spelling program as the input encoding, which is wrong, since
the input encoding depends on the file to be checked.

I currently use the classic workaround of making up my own dictionary
definition which includes accented characters that I want to be able
to use in words (which is necessary anyway), and which specifies utf-8
as the coding system. This only works because I use utf-8 for all my
text files.

It seems, therefore, that the argument to follow
ispell-encoding8-command (which itself is mis-documented:

Command line option prefix to select UTF-8 if supported, nil otherwise.
If UTF-8 if supported by spellchecker and is selectable from the command line
this variable will contain \"--encoding=\" for aspell and \"-i \" for hunspell,
so UTF-8 or other mime charsets can be selected.  That will be set for hunspell
>=1.1.6 or aspell >= 0.60 in `ispell-check-version'.

It is not just for selecting UTF-8; indeed, that's the irony: in the
default configuration it's used mostly to select 8-bit character sets!
And there are one or two other typos. How about (suitably rewrapped):

Command line option prefix to select coding system if supported, nil otherwise.
If the coding system is selectable from the command line
this variable will contain \"--encoding=\" for aspell and \"-i \" for hunspell,
so that the input encoding can be selected.  That will be set for hunspell
>= 1.1.6 or aspell >= 0.60 in `ispell-check-version'.

Then, the following code in ispell-start-process:

    ;; If we are using recent aspell or hunspell, make sure we use the
right encoding
    ;; for communication. ispell or older aspell/hunspell does not support this
    (if ispell-encoding8-command
	(setq args
	      (append args
		      (list
		       (concat ispell-encoding8-command
			       (symbol-name (ispell-get-coding-system)))))))

needs fixing: rather than using ispell-get-coding-system, it should
use a prefix of buffer-file-coding-system (without the suffix that
specifies the line ending).

I'm sure I'm missing things here, but if what I've said above makes
any sense, I'd like to help refine it into a sensible proposal to
improve ispell.el.

-- 
http://rrt.sc3d.org

bug#7668: ispell and dictionary encodings

[Agustin Martin]

On Fri, Dec 17, 2010 at 06:30:14PM +0000, Reuben Thomas wrote:
> I've just been puzzling my way through ispell.gz's dictionary encoding
> code, after switching from aspell to hunspell in order to be able to
> treat Unicode curly single quotes as normal intraword punctuation
> (which it seems aspell cannot be persuaded to do, but that's another
> story).
> 
> I noticed a feature of ispell-dictionary-base-alist, which I don't
> understand: the last (7th) element of each dictionary definition is
> called "Coding System", which seems to be the coding system of the
> case character and non-case-character strings, but it is also passed
> to the spelling program as the input encoding, which is wrong, since
> the input encoding depends on the file to be checked.

That element represents the language that will be used for communication
with the dictionary. case-character and non-case-character strings should 
be in the same encoding as it.

> I currently use the classic workaround of making up my own dictionary
> definition which includes accented characters that I want to be able
> to use in words (which is necessary anyway), and which specifies utf-8
> as the coding system. This only works because I use utf-8 for all my
> text files.

If you are not going to use XEmacs, but only FSF Emacs, just use [:alpha:]
for the case-character and non-case-character strings along with utf-8. That
is already done automatically for aspell dictionaries, where is easy to get
a list of installed dictionaries and additional info.

> It seems, therefore, that the argument to follow
> ispell-encoding8-command (which itself is mis-documented:
> 
> Command line option prefix to select UTF-8 if supported, nil otherwise.
> If UTF-8 if supported by spellchecker and is selectable from the command line
> this variable will contain \"--encoding=\" for aspell and \"-i \" for hunspell,
> so UTF-8 or other mime charsets can be selected.  That will be set for hunspell
> >=1.1.6 or aspell >= 0.60 in `ispell-check-version'.
> 
> It is not just for selecting UTF-8; indeed, that's the irony: in the
> default configuration it's used mostly to select 8-bit character sets!
> And there are one or two other typos. How about (suitably rewrapped):
> 
> Command line option prefix to select coding system if supported, nil otherwise.
> If the coding system is selectable from the command line
> this variable will contain \"--encoding=\" for aspell and \"-i \" for hunspell,
> so that the input encoding can be selected.  That will be set for hunspell
> >= 1.1.6 or aspell >= 0.60 in `ispell-check-version'.

Agreed, thanks

> Then, the following code in ispell-start-process:
> 
>     ;; If we are using recent aspell or hunspell, make sure we use the
> right encoding
>     ;; for communication. ispell or older aspell/hunspell does not support this
>     (if ispell-encoding8-command
> 	(setq args
> 	      (append args
> 		      (list
> 		       (concat ispell-encoding8-command
> 			       (symbol-name (ispell-get-coding-system)))))))
> 
> needs fixing: rather than using ispell-get-coding-system, it should
> use a prefix of buffer-file-coding-system (without the suffix that
> specifies the line ending).

No, current code is correct. It is telling the spellchecker that
communication with the dictionary will be done in (ispell-get-coding-system) 
coding system. ispell.el will do the internal conversions needed for that in 
a diferent place, so everything is transparent to the user.

> I'm sure I'm missing things here, but if what I've said above makes
> any sense, I'd like to help refine it into a sensible proposal to
> improve ispell.el.

Thanks for looking into this. Will prepare a change with the
`ispell-encoding8-command' documentation fix.

Regards,

-- 
Agustin

bug#7668: ispell and dictionary encodings

[Reuben Thomas]

On 20 December 2010 11:31, Agustin Martin <agustin.martin@hispalinux.es> wrote:

[a very helpful reply; thanks]

> On Fri, Dec 17, 2010 at 06:30:14PM +0000, Reuben Thomas wrote:
> If you are not going to use XEmacs, but only FSF Emacs, just use [:alpha:]
> for the case-character and non-case-character strings along with utf-8. That
> is already done automatically for aspell dictionaries, where is easy to get
> a list of installed dictionaries and additional info.

So, the built-in entries of ispell-dictionary-base-alist are
specifically for ispell? In that case, it seems a bit odd that they
are used for hunspell, but perhaps the problem is that you can't get
hunspell to give you that information about its dictionaries? But is
there in any case a reason not to default to using [:alpha:] for
case-chars and ^[:alpha:] for non-case-chars with hunspell?

In case I'm getting too confused, I'll just restate the basic
objective I have: I want to be able to spell-check (in my case,
British, but I don't think it matters for this purpose) English with
a) accents and b) fancy quotes. In these days of utf-8 being widely
used for English, it seems it should be possible to do at least b) out
of the box, which currently it isn't, as far as I can see.

--
http://rrt.sc3d.org

bug#7668: ispell and dictionary encodings

[Agustin Martin]

On Mon, Dec 20, 2010 at 03:40:18PM +0000, Reuben Thomas wrote:
> On 20 December 2010 11:31, Agustin Martin <agustin.martin@hispalinux.es> wrote:
> 
> [a very helpful reply; thanks]
> 
> > On Fri, Dec 17, 2010 at 06:30:14PM +0000, Reuben Thomas wrote:
> > If you are not going to use XEmacs, but only FSF Emacs, just use [:alpha:]
> > for the case-character and non-case-character strings along with utf-8. That
> > is already done automatically for aspell dictionaries, where is easy to get
> > a list of installed dictionaries and additional info.
> 
> So, the built-in entries of ispell-dictionary-base-alist are
> specifically for ispell? 

Or generally for versions of the spellcheckers that do not properly support
different encodings, old aspells and hunspells, there are still some of them
flying around.

> In that case, it seems a bit odd that they
> are used for hunspell, but perhaps the problem is that you can't get
> hunspell to give you that information about its dictionaries? 

That is indeed part of the problem. Otherwise something like
(ispell-aspell-find-dictionaries) and friends could be used. 'hunspell -D'
does not provide all the info, and does not return control until ^C. 

> But is
> there in any case a reason not to default to using [:alpha:] for
> case-chars and ^[:alpha:] for non-case-chars with hunspell?

Besides old aspells and hunspells, I am trying to improve XEmacs
compatibility for ispell.el and flyspell.el. I keep patched versions for
Debian, so all Emacs flavours use the same ispell.el and flyspell.el. In its
current incarnation, even Emacs >=21.3 is supported by Debian patched files. 
I am currently removing all that compatibility leaving only Emacs23 and 
XEmacs, and would like to keep FSF Emacs ispell.el and flyspell.el 
reasonably close to those I use, so I need less changes. And XEmacs do not 
support [:alpha:]. 

An intermediate possibility could be to use a hunspell specific default 
dictionary list built on the fly from base-alist with encoding set to utf8
and case/not-case changed to [:alpha:] for FSF Emacs and recent enough
hunspell. Since this would only be done first time ispell.el invokes
hunspell spellchecking, seems be reasonable. But I have to think about this.

> In case I'm getting too confused, I'll just restate the basic
> objective I have: I want to be able to spell-check (in my case,
> British, but I don't think it matters for this purpose) English with
> a) accents and b) fancy quotes. In these days of utf-8 being widely
> used for English, it seems it should be possible to do at least b) out
> of the box, which currently it isn't, as far as I can see.

Putting those fancy quotes in 'otherchars' section in dictionary definition 
for ispell.el should make ispell.el consider them part of the word, but 
IIRC will not affect hunspell unless they are defined in TRY section of 
.aff file.

-- 
Agustin

bug#7668: ispell and dictionary encodings

[Reuben Thomas]

On 21 December 2010 11:30, Agustin Martin <agustin.martin@hispalinux.es> wrote:
>
> Or generally for versions of the spellcheckers that do not properly support
> different encodings, old aspells and hunspells, there are still some of them
> flying around.

Presumably for the Debian version of ispell.el you don't have to cater
to old versions?

> Putting those fancy quotes in 'otherchars' section in dictionary definition
> for ispell.el should make ispell.el consider them part of the word,

Well, it definitely doesn't work with aspell. It seems to work fine
with hunspell (for whatever reason).

Thanks again; if I can be of any assistance don't hesitate to ask.

One other confusion arises: I was reporting a bug about FSF Emacs
(i.e. upstream) and I am currently, because of fixed session support
bugs, using the tip of the emacs-23 source branch, not the Ubuntu
maverick version (which is the distro I'm using).

-- 
http://rrt.sc3d.org