Hunspell and contractions with apostrophes

Hunspell and contractions with apostrophes

[Eric Abrahamsen]

I've battled with this for years now: Hunspell marks any contraction
with an apostrophe (eg the "I've" that starts this sentence) as
misspelled.

It used to be that I could edit /usr/share/hunspell/en_US.aff and add
the apostrophe to WORDCHARS (and also "ICONV ’ '"), and that would do
it. Until the next time the hunspell package updated, and over-wrote its
config files (I'm running Arch linux), and I would have to do it again.

As of six months or a year or so ago, that trick no longer works. I add
the apostrophe to the above-mentioned file, and to
"/usr/share/myspell/dicts/en_US.aff", but it has no effect.

I've got this in my init:

(set-language-environment "utf-8")

And my shell $LANG variable is en_US.UTF-8. The
"/usr/share/myspell/dicts" directory contains *.aff and *.dic files for
all the en_* variants.

I don't know what else to edit! I'm sort of amazed that this problem is
so persistent, and so difficult to fix.

Any pointers very welcome!

Eric

Re: Hunspell and contractions with apostrophes

[Eli Zaretskii]

> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> Date: Tue, 26 May 2020 10:48:34 -0700
> 
> I've battled with this for years now: Hunspell marks any contraction
> with an apostrophe (eg the "I've" that starts this sentence) as
> misspelled.

Which apostrophe are you talking about: the ASCII one as in "I've", or
the non-ASCII one as in "I’ve"?  (The latter is not an "apostrophe",
to be exact, that character has a different name.)

Re: Hunspell and contractions with apostrophes

[Eric Abrahamsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>> Date: Tue, 26 May 2020 10:48:34 -0700
>> 
>> I've battled with this for years now: Hunspell marks any contraction
>> with an apostrophe (eg the "I've" that starts this sentence) as
>> misspelled.
>
> Which apostrophe are you talking about: the ASCII one as in "I've", or
> the non-ASCII one as in "I’ve"?  (The latter is not an "apostrophe",
> to be exact, that character has a different name.)

I'm talking about the ASCII apostrophe, 0x27. But I get the same result
with RIGHT SINGLE QUOTATION MARK (0x2019). My understanding is that the
ICONV line should unify the two.

I realize that this is really an OS question, not an Emacs question, and
I appreciate you looking into it. Other hunspell consumers (Libreoffice,
Firefox) don't have this problem, though, so it might be something Emacs
wants to be aware of.

Thanks!
Eric

Re: Hunspell and contractions with apostrophes

[Dmitry Alexandrov]

[-- Attachment #1: Type: text/plain, Size: 3129 bytes --]

Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
> I've battled with this for years now: Hunspell marks any contraction with an apostrophe (eg the "I've" that starts this sentence) as misspelled.

Iʼd say that this is an obvious bug in the dictionary, that should be reported.

FWIW, it is present in Debian 10 as well:

	$ HOME=/tmp DICPATH='' hunspell -d en_US
	Hunspell 1.7.0
	I've
	*
	& ve 15 2: be, v, e, eve, vie, ave, vet, veg, Eve, Ave, vs, vi, re, me, he

and Iʼve never noticed it only because I use en_GB dictionary, which is fine:

	$ HOME=/tmp DICPATH='' hunspell -d en_GB
	Hunspell 1.7.0
	I've
        *

> It used to be that I could edit /usr/share/hunspell/en_US.aff and add the apostrophe to WORDCHARS (and also "ICONV ’ '")

First and foremost, your ’ is *not* an apostrophe, itʼs a right quote.  Apostrophe is ʼ.

This does matter, just check how do word-moving commands act on weird “I’ve” vs proper “Iʼve” and ascii (but no less proper) “I've”.

> and that would do it. Until the next time the hunspell package updated, and over-wrote its config files (I'm running Arch linux), and I would have to do it again.

Sure.  You are not supposed to tamper with files under package management.  Put your customized dictionaries somewhere else (in /etc, in your home directory).  I do not remember, whether hunspell(1) have any non-/usr paths hardcoded, but these lines in my ~/.profile suggest, that it does not:

	if [ -z "$DICPATH" ]; then
	    if [ -d '/usr/share/hunspell' ]; then
	        DICPATH='/usr/share/hunspell'
	    fi
	fi
	
	if [ -d "$HOME/.share/hunspell" ]; then
	    DICPATH="$HOME/.share/hunspell:$DICPATH"
	fi
	
	if [ -d "$HOME/.local/share/hunspell" ]; then
	    DICPATH="$HOME/.local/share/hunspell:$DICPATH"
	fi
	
	export DICPATH

> As of six months or a year or so ago, that trick no longer works.

It seems, that the question have changed in meanwhile.  Whether a right single quote is recognized as apostrophe is orthogonal to whether “I've” is recognized as a correct English word.

Things like “I've” or “I'm” are normally explicitly mentioned in the dictionary (since something like “pointʼve” is not entirely okay, afaiu).

If they are there, then double-check, that the used affix file does have apostrophe among WORDCHARS:

	WORDCHARS 0123456789'

(Thatʼs what is wrong with en_US.aff in Debian.)

If “I've” had still been recognized as a mistake, that would be pretty odd.


Now to Unicode.

Make sure, that the affix file correctly declares its encoding:

	SET UTF-8

Make Unicode apostrophe recognized as apostrophe:

	ICONV 1
	ICONV ʼ '

Optionally, make Unicode preferred to ASCII one for suggestions:

	OCONV 1
	OCONV ' ʼ


Now to Emacs.

If all of the above works with hunspell(1) itself, no configurations besides (setq ispell-program-name "hunspell") should be required.

I you insist on using right single quote as apostrophe, though, I have no idea, how to make ispell.el pass it to hunspell(1) as a part of a word.  Neither why ever do that.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

Re: Hunspell and contractions with apostrophes

[Eli Zaretskii]

> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> Date: Tue, 26 May 2020 13:03:52 -0700
> 
> > Which apostrophe are you talking about: the ASCII one as in "I've", or
> > the non-ASCII one as in "I’ve"?  (The latter is not an "apostrophe",
> > to be exact, that character has a different name.)
> 
> I'm talking about the ASCII apostrophe, 0x27.

Then I cannot reproduce this, it works for me, ispell.el (using
Hunspell as the speller) says "I'VE is correct".  I guess your
dictionary has a problem or something.  Or maybe you set up
spell-checking incorrectly (try in "emacs -Q").  There should be no
need to use ICONV at all.

> Other hunspell consumers (Libreoffice, Firefox) don't have this
> problem, though, so it might be something Emacs wants to be aware
> of.

I see no problem in Emacs using Hunspell.

Re: Hunspell and contractions with apostrophes

[Eli Zaretskii]

> From: Dmitry Alexandrov <dag@gnui.org>
> Date: Wed, 27 May 2020 03:23:16 +0300
> Cc: help-gnu-emacs@gnu.org
> 
> Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
> > I've battled with this for years now: Hunspell marks any contraction with an apostrophe (eg the "I've" that starts this sentence) as misspelled.
> 
> Iʼd say that this is an obvious bug in the dictionary, that should be reported.

Maybe with the dictionary you have, not with the one I have.

> FWIW, it is present in Debian 10 as well:
> 
> 	$ HOME=/tmp DICPATH='' hunspell -d en_US
> 	Hunspell 1.7.0
> 	I've
> 	*
> 	& ve 15 2: be, v, e, eve, vie, ave, vet, veg, Eve, Ave, vs, vi, re, me, he

I get just "*", meaning there's no spelling errors.

Re: Hunspell and contractions with apostrophes

[Yuri Khan]

On Wed, 27 May 2020 at 07:23, Dmitry Alexandrov <dag@gnui.org> wrote:

> First and foremost, your ’ is *not* an apostrophe, itʼs a right quote.  Apostrophe is ʼ.
> I you insist on using right single quote as apostrophe, though, I have no idea, how to make ispell.el pass it to hunspell(1) as a part of a word.  Neither why ever do that.

Because the Unicode standard says so.
<https://www.unicode.org/charts/PDF/U2000.pdf>, entry for U+2019:

    2019 ’ RIGHT SINGLE QUOTATION MARK
           = single comma quotation mark
           • this is the preferred character to use for apostrophe
           → 0027 ' apostrophe
           → 02BC ʼ modifier letter apostrophe
           → 275C ❜ heavy single comma quotation mark ornament

(Yes, it’s unfortunate for word boundary algorithms.)

Also, <https://www.unicode.org/charts/PDF/U02B0.pdf>:

    02BC ʼ MODIFIER LETTER APOSTROPHE
           = apostrophe
           • glottal stop, glottalization, ejective
           • many languages use this as a letter of their alphabets
           • used as a tone marker in Bodo, Dogri, and Maithili
           • 2019 ’ is the preferred character for a punctuation apostrophe
           → 0027 ' apostrophe
           → 0313  ̓ combining comma above
           → 0315  ̕ combining comma above right
           → 055A ՚ armenian apostrophe
           → 07F4 ߴ nko high tone apostrophe
           → 1FBF ̓ greek psili
           → 2019 ’ right single quotation mark

Re: (Mis?)using quote as apostrophe (was: Hunspell and contractions with apostrophes)

[Dmitry Alexandrov]

[-- Attachment #1: Type: text/plain, Size: 1666 bytes --]

Yuri Khan <yuri.v.khan@gmail.com> wrote:
> On Wed, 27 May 2020 at 07:23, Dmitry Alexandrov <dag@gnui.org> wrote:
>
>> First and foremost, your ’ is *not* an apostrophe, itʼs a right quote.  Apostrophe is ʼ.

>> I you insist on using right single quote as apostrophe, though, I have no idea, how to make ispell.el pass it to hunspell(1) as a part of a word.  Neither why ever do that.
>
> Because the Unicode standard says so.
>
> <https://www.unicode.org/charts/PDF/U2000.pdf>, entry for U+2019:
>
>     2019 ’ RIGHT SINGLE QUOTATION MARK
>            = single comma quotation mark
>            • this is the preferred character to use for apostrophe
>            → 0027 ' apostrophe
>            → 02BC ʼ modifier letter apostrophe
>            → 275C ❜ heavy single comma quotation mark ornament
>
> (Yes, it’s unfortunate for word boundary algorithms.)

Heh.  Iʼm afraid, itʼs not merely unfortunate, itʼs totally in spite of the whole spirit of Unicode, and simply silly.  Just as if it said that ” (RIGHT DOUBLE QUOTATION MARK) were the preferred character for inches sign.

> Also, <https://www.unicode.org/charts/PDF/U02B0.pdf>:
>
>     02BC ʼ MODIFIER LETTER APOSTROPHE
>            = apostrophe
>            • glottal stop, glottalization, ejective
>            • many languages use this as a letter of their alphabets
>            • used as a tone marker in Bodo, Dogri, and Maithili
>            • 2019 ’ is the preferred character for a punctuation apostrophe

A!  _Punctuation_ apostrophe.  Whatever is it, itʼs not what we are taking about here.  hunspell does not check punctuation.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

Re: (Mis?)using quote as apostrophe (was: Hunspell and contractions with apostrophes)

[Yuri Khan]

On Wed, 27 May 2020 at 13:05, Dmitry Alexandrov <dag@gnui.org> wrote:
> Heh.  Iʼm afraid, itʼs not merely unfortunate, itʼs totally in spite of the whole spirit of Unicode, and simply silly.  Just as if it said that ” (RIGHT DOUBLE QUOTATION MARK) were the preferred character for inches sign.

FWIW, the preferred character for inches is U+2033 DOUBLE PRIME. (Not
U+0022 as some say.)

> A!  _Punctuation_ apostrophe.  Whatever is it, itʼs not what we are taking about here.  hunspell does not check punctuation.

Also FWIW, many English grammar guides out there call the apostrophe a
punctuation mark, whether it indicates possession or contraction.

Re: (Mis?)using quote as apostrophe

[Dmitry Alexandrov]

[-- Attachment #1: Type: text/plain, Size: 1560 bytes --]

Yuri Khan <yuri.v.khan@gmail.com> wrote:
> On Wed, 27 May 2020 at 13:05, Dmitry Alexandrov <dag@gnui.org> wrote:
>> Heh.  Iʼm afraid, itʼs not merely unfortunate, itʼs totally in spite of the whole spirit of Unicode, and simply silly.  Just as if it said that ” (RIGHT DOUBLE QUOTATION MARK) were the preferred character for inches sign.
>
> FWIW, the preferred character for inches is U+2033 DOUBLE PRIME. (Not U+0022 as some say.)

Yep.  Thatʼs one of the objectives of Unicode: to split symbols, that were conflated once due to the space shortage, _semantically_, no matter how similar they look.  So overloading the right single quote with apostrophe (as itʼs used in English or, say, Bielorussian) is absolutely inapt.

>> A!  _Punctuation_ apostrophe.  Whatever is it, itʼs not what we are taking about here.  hunspell does not check punctuation.
>
> Also FWIW, many English grammar guides out there call the apostrophe a punctuation mark, whether it indicates possession or contraction.

Perfectly plausible¹, yet it does not change the fact that compilers of English dictionaries for hunspell clearly consider apostrophe as a part of a word.

-
¹ Grammarians do not seem to be very much concerned of the reality.  For instance, you are perhaps aware, that nearly any book from primers onwards positively state that a grapheme ‘ё’ is an integral part of Russian alphabet, thatʼs while ‘е́’ is not, despite that in any practical application (lexical sorting, searching, etc) they should be treated exactly the same.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

Re: Hunspell and contractions with apostrophes

[Eric Abrahamsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>> Date: Tue, 26 May 2020 13:03:52 -0700
>> 
>> > Which apostrophe are you talking about: the ASCII one as in "I've", or
>> > the non-ASCII one as in "I’ve"?  (The latter is not an "apostrophe",
>> > to be exact, that character has a different name.)
>> 
>> I'm talking about the ASCII apostrophe, 0x27.
>
> Then I cannot reproduce this, it works for me, ispell.el (using
> Hunspell as the speller) says "I'VE is correct".  I guess your
> dictionary has a problem or something.  Or maybe you set up
> spell-checking incorrectly (try in "emacs -Q").  There should be no
> need to use ICONV at all.

Ack! You're right, "emacs -Q" works fine. I should've checked that
first. My only other spelling-related customization is setting
`ispell-personal-dictionary', and setting `flyspell-use-meta-tab' to
nil. I'll try changing those and figuring out what's going on.

Sorry for the noise,
Eric

Re: Hunspell and contractions with apostrophes

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>>> Date: Tue, 26 May 2020 13:03:52 -0700
>>> 
>>> > Which apostrophe are you talking about: the ASCII one as in "I've", or
>>> > the non-ASCII one as in "I’ve"?  (The latter is not an "apostrophe",
>>> > to be exact, that character has a different name.)
>>> 
>>> I'm talking about the ASCII apostrophe, 0x27.
>>
>> Then I cannot reproduce this, it works for me, ispell.el (using
>> Hunspell as the speller) says "I'VE is correct".  I guess your
>> dictionary has a problem or something.  Or maybe you set up
>> spell-checking incorrectly (try in "emacs -Q").  There should be no
>> need to use ICONV at all.
>
> Ack! You're right, "emacs -Q" works fine. I should've checked that
> first. My only other spelling-related customization is setting
> `ispell-personal-dictionary', and setting `flyspell-use-meta-tab' to
> nil. I'll try changing those and figuring out what's going on.

I've unset `ispell-personal-dictionary' it's working correctly now.
Obviously there's something about personal dictionaries that I'm not
understanding.

Re: Hunspell and contractions with apostrophes

[Eli Zaretskii]

> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> Cc: help-gnu-emacs@gnu.org
> Date: Wed, 27 May 2020 14:53:54 -0700
> 
> > Ack! You're right, "emacs -Q" works fine. I should've checked that
> > first. My only other spelling-related customization is setting
> > `ispell-personal-dictionary', and setting `flyspell-use-meta-tab' to
> > nil. I'll try changing those and figuring out what's going on.
> 
> I've unset `ispell-personal-dictionary' it's working correctly now.
> Obviously there's something about personal dictionaries that I'm not
> understanding.

ispell-personal-dictionary was my guess when I read your previous
message.

Can you show that setting?  Maybe we will be able to tell you what was
wrong about it.

Re: Hunspell and contractions with apostrophes

[Eric Abrahamsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>> Cc: help-gnu-emacs@gnu.org
>> Date: Wed, 27 May 2020 14:53:54 -0700
>> 
>> > Ack! You're right, "emacs -Q" works fine. I should've checked that
>> > first. My only other spelling-related customization is setting
>> > `ispell-personal-dictionary', and setting `flyspell-use-meta-tab' to
>> > nil. I'll try changing those and figuring out what's going on.
>> 
>> I've unset `ispell-personal-dictionary' it's working correctly now.
>> Obviously there's something about personal dictionaries that I'm not
>> understanding.
>
> ispell-personal-dictionary was my guess when I read your previous
> message.
>
> Can you show that setting?  Maybe we will be able to tell you what was
> wrong about it.

The setting was just:

(setq ispell-personal-dictionary "~/.aspell.en_US.pws")

That file was just a plain word list. Above the setting I'd left this
comment:

;; ispell.el claims that the default personal dictionary should be
;; ~/.aspell.en.pws, while aspell itself thinks it should be
;; ~/.aspell.en_US.pws Right now I've changed the dictionary to what
;; aspell thinks it should be, and we'll see if that confuses ispell.el
;; at all.

Though TBH at a remove of several years, I don't really remember where
these ideas came from.

Thanks,
Eric

Re: Hunspell and contractions with apostrophes

[Eli Zaretskii]

> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> Cc: help-gnu-emacs@gnu.org
> Date: Thu, 28 May 2020 07:00:48 -0700
> 
> > Can you show that setting?  Maybe we will be able to tell you what was
> > wrong about it.
> 
> The setting was just:
> 
> (setq ispell-personal-dictionary "~/.aspell.en_US.pws")

So if you now set that variable again to the name of an empty file,
and then restart Emacs, you again have the problem?  Or is the problem
happening only with some specific contents of the personal dictionary?
If the latter, perhaps some word in your personal dictionary causes
that?

Re: Hunspell and contractions with apostrophes

[Eric Abrahamsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>> Cc: help-gnu-emacs@gnu.org
>> Date: Thu, 28 May 2020 07:00:48 -0700
>> 
>> > Can you show that setting?  Maybe we will be able to tell you what was
>> > wrong about it.
>> 
>> The setting was just:
>> 
>> (setq ispell-personal-dictionary "~/.aspell.en_US.pws")
>
> So if you now set that variable again to the name of an empty file,
> and then restart Emacs, you again have the problem?  Or is the problem
> happening only with some specific contents of the personal dictionary?
> If the latter, perhaps some word in your personal dictionary causes
> that?

Okay, looks like there is something bogus in my personal dictionary. I
removed the customization altogether (which caused the contraction
problem to go away), saved a few words to my personal dictionary, and
that created the file ~/.hunspell_en_US.

Then I restarted, set `ispell-personal-dictionary' to a non-existent
file, restarted, and saved some more words. They went into the new file,
but interestingly ~/.hunspell_en_US was still honored. I've _never_
understood the interaction between all the various options here, so this
was good information. I guess there's no need to set
`ispell-personal-dictionary' at all, unless I want to keep the
dictionary in a git repo or something.

I'll figure out what's wrong with my old dictionary. It's got Chinese
and Arabic in it (and I note is saved as UTF-8 whereas ~/.hunspell_en_US
is ASCII) so maybe that has something to do with it. I tried
spellchecking a Chinese word and hunspell just errored, so there's no
point putting Chinese in there to begin with.

Anyway... thanks for walking me through this.

Eric

Re: Hunspell and contractions with apostrophes

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>>> Cc: help-gnu-emacs@gnu.org
>>> Date: Thu, 28 May 2020 07:00:48 -0700
>>> 
>>> > Can you show that setting?  Maybe we will be able to tell you what was
>>> > wrong about it.
>>> 
>>> The setting was just:
>>> 
>>> (setq ispell-personal-dictionary "~/.aspell.en_US.pws")
>>
>> So if you now set that variable again to the name of an empty file,
>> and then restart Emacs, you again have the problem?  Or is the problem
>> happening only with some specific contents of the personal dictionary?
>> If the latter, perhaps some word in your personal dictionary causes
>> that?
>
> Okay, looks like there is something bogus in my personal dictionary. I
> removed the customization altogether (which caused the contraction
> problem to go away), saved a few words to my personal dictionary, and
> that created the file ~/.hunspell_en_US.
>
> Then I restarted, set `ispell-personal-dictionary' to a non-existent
> file, restarted, and saved some more words. They went into the new file,
> but interestingly ~/.hunspell_en_US was still honored. I've _never_
> understood the interaction between all the various options here, so this
> was good information. I guess there's no need to set
> `ispell-personal-dictionary' at all, unless I want to keep the
> dictionary in a git repo or something.
>
> I'll figure out what's wrong with my old dictionary. It's got Chinese
> and Arabic in it (and I note is saved as UTF-8 whereas ~/.hunspell_en_US
> is ASCII) so maybe that has something to do with it. I tried
> spellchecking a Chinese word and hunspell just errored, so there's no
> point putting Chinese in there to begin with.

Yes, removing Chinese and Arabic did the trick. The encoding of the
dictionary file didn't matter.