bug#7230: 24.0.50; Regression: Many dictionaries not found by ispell.el

bug#7230: 24.0.50; Regression: Many dictionaries not found by ispell.el

[Knut Anders Hatlen]

On a system with Aspell 0.60.6 with English dictionaries installed,
evaluating the following expression successfully changes the dictionary
on Emacs 23.2:

(progn
  (require 'ispell)
  (ispell-change-dictionary "british-ize"))

On Emacs 24 it fails:

Debugger entered--Lisp error: (error "Undefined dictionary: british-ize")
  signal(error ("Undefined dictionary: british-ize"))
  error("Undefined dictionary: %s" "british-ize")
  ispell-change-dictionary("british-ize")
  (progn (require (quote ispell)) (ispell-change-dictionary "british-ize"))

Comparing ispell-dictionary-alist on Emacs 23.2 and Emacs 24, it looks
like the following dictionaries are not found by ispell.el on Emacs 24:

  canadian
  canadian-wo_accents
  canadian-w_accents
  british-wo_accents
  british-w_accents
  british-ize
  british-ize-wo_accents
  british-ize-w_accents
  british-ise
  british-ise-wo_accents
  british-ise-w_accents
  american-wo_accents
  american-w_accents
  en_CA
  en_CA-w_accents
  en_CA-wo_accents
  en_GB
  en_GB-ise
  en_GB-ise-w_accents
  en_GB-ise-wo_accents
  en_GB-ize
  en_GB-ize-w_accents
  en_GB-ize-wo_accents
  en_GB-w_accents
  en_GB-wo_accents
  en_US
  en_US-w_accents
  en_US-wo_accents

The new behaviour was introduced by this change:

------------------------------------------------------------
revno: 100614
committer: Agustin martin <agustin.martin@hispalinux.es>
branch nick: trunk
timestamp: Thu 2010-06-17 16:44:34 +0200
message:
  ispell.el (ispell-aspell-find-dictionary): Fix regexp for full xx_YY search.
  
  This is needed for languages like Portuguese where there are aspell dicts
  for pt_BR and pt_PT, not for plain pt. If we strip country part 
  `ispell-aspell-find-dictionary' does not find dicts for portuguese.
------------------------------------------------------------




In GNU Emacs 24.0.50.1 (x86_64-pc-solaris2.11, GTK+ Version 2.20.0)
 of 2010-10-17 on ugle
Windowing system distributor `Oracle Corporation, based on X.Org Foundation sources', version 11.0.10707000
configured using `configure  '--build=x86_64-pc-solaris2.11' '--prefix=/opt/emacs24' '--with-imagemagick' 'PKG_CONFIG_PATH=/usr/lib/64/pkgconfig' 'CC=/usr/gcc/4.3/bin/gcc -m64' 'build_alias=x86_64-pc-solaris2.11''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=iiimx
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
<escape> x r e p o r t - e m a c s <tab> <return>

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr message sendmail regexp-opt rfc822 mml
easymenu mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231
rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mailabbrev mail-utils
gmm-utils mailheader emacsbug tooltip ediff-hook vc-hooks
lisp-float-type mwheel x-win x-dnd tool-bar dnd fontset image fringe
lisp-mode register page menu-bar rfn-eshadow timer select scroll-bar
mouse jit-lock font-lock syntax facemenu font-core frame cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese case-table epa-hook jka-cmpr-hook help simple abbrev
loaddefs button minibuffer faces cus-face files text-properties overlay
md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote make-network-process dbusbind
dynamic-setting system-font-setting font-render-setting move-toolbar gtk
x-toolkit x multi-tty emacs)

-- 
Knut Anders

bug#7230: 24.0.50; Regression: Many dictionaries not found by ispell.el

[Agustin Martin]

[-- Attachment #1: Type: text/plain, Size: 907 bytes --]

On Sun, Oct 17, 2010 at 07:41:16PM +0200, Knut Anders Hatlen wrote:
> 
> On a system with Aspell 0.60.6 with English dictionaries installed,
> evaluating the following expression successfully changes the dictionary
> on Emacs 23.2:
> 
> (progn
>   (require 'ispell)
>   (ispell-change-dictionary "british-ize"))
> 
> On Emacs 24 it fails:

Thanks for noticing and sorry for the bug. I thought it was behaving 
correctly since a lot of English variants were listed and unfortunately 
did not notice that some of them were missing, my fault. I really wonder 
how no one found this before.

This change was introduced to deal with portuguese, that has no pt.dat, but
pt_BR or pt_PT. 

I am currently playing with attached patch, that tries different .dat
matches. This should be better than before, and deal with former case plus
portuguese. Will test a bit more and commit if no problems appear.

-- 
Agustin

[-- Attachment #2: ispell-el_diferent-dat-files.diff --]
[-- Type: text/x-diff, Size: 1478 bytes --]

--- ispell.el.orig	2010-10-18 13:20:35.000000000 +0200
+++ ispell.el	2010-10-18 13:20:45.000000000 +0200
@@ -983,17 +983,30 @@
     (car (split-string (buffer-string)))))
 
 (defun ispell-aspell-find-dictionary (dict-name)
+  "Create a list of properties for a given aspell dictionary."
+  ;; Make sure this is defined
+  (or ispell-aspell-data-dir
+      (setq ispell-aspell-data-dir
+	    (ispell-get-aspell-config-value "data-dir")))
   ;; This returns nil if the data file does not exist.
   ;; Can someone please explain the return value format when the
   ;; file does exist -- rms?
-  (let* ((lang ;; Strip out variant, etc.
-	  (and (string-match "^[[:alpha:]_]+" dict-name)
-	       (match-string 0 dict-name)))
+  (let* ((datafile1
+	  (concat ispell-aspell-data-dir "/"
+		  ;; Strip out variant, country code, etc.
+		  (and (string-match "^[[:alpha:]]+" dict-name)
+		       (match-string 0 dict-name)) ".dat"))
+	 (datafile2
+	  (concat ispell-aspell-data-dir "/"
+		  ;; Strip out anything but xx_YY.
+		  (and (string-match "^[[:alpha:]_]+" dict-name)
+		       (match-string 0 dict-name)) ".dat"))
 	 (data-file
-	  (concat (or ispell-aspell-data-dir
-		      (setq ispell-aspell-data-dir
-			    (ispell-get-aspell-config-value "data-dir")))
-		  "/" lang ".dat"))
+	  (if (file-readable-p datafile1)
+	      datafile1
+	    (if (file-readable-p datafile2)
+		datafile2
+	      "No dat file found")))
 	 otherchars)
     (condition-case ()
 	(with-temp-buffer

bug#7230: 24.0.50; Regression: Many dictionaries not found by ispell.el

[Agustin Martin]

On Mon, Oct 18, 2010 at 01:22:47PM +0200, Agustin Martin wrote:
> On Sun, Oct 17, 2010 at 07:41:16PM +0200, Knut Anders Hatlen wrote:
> > 
> > On a system with Aspell 0.60.6 with English dictionaries installed,
> > evaluating the following expression successfully changes the dictionary
> > on Emacs 23.2:
> > 
> > (progn
> >   (require 'ispell)
> >   (ispell-change-dictionary "british-ize"))
> > 
> > On Emacs 24 it fails:
> 
> Thanks for noticing and sorry for the bug. I thought it was behaving 
> correctly since a lot of English variants were listed and unfortunately 
> did not notice that some of them were missing, my fault. I really wonder 
> how no one found this before.
> 
> This change was introduced to deal with portuguese, that has no pt.dat, but
> pt_BR or pt_PT. 
> 
> I am currently playing with attached patch, that tries different .dat
> matches. This should be better than before, and deal with former case plus
> portuguese. Will test a bit more and commit if no problems appear.

Just committed a change that should look for both xx.dat and xx_YY.dat files
for xx_YY.. aspell dicts. Includes also some function reorganization and a
proper docstring. 

This should fix this problem, so I am closing the bug report. Please reopen
if needed.

>  (defun ispell-aspell-find-dictionary (dict-name)

Function name is a bit misleading. Fortunately, seems it is only used 
internally by ispell.el and is not intended for external use. I left it with 
its current name, but something like `ispell-aspell-find-dict-params' looks 
more appropriate. Will think if worths renaming and declaring old name 
obsolete.

Thanks for your feedback,

-- 
Agustin