flyspell bug

flyspell bug

[Stephen Eglen]

In today's CVS emacs:
emacs -q
C-h N
M-x flyspell-buffer

I get:  Error: Can't check region

and *flyspell-region* contains:
Error: Could not open the file "list" for reading.


This is compatibility problem between ispell and aspell.
I have:

$ ispell -v
@(#) International Ispell Version 3.1.20 (but really Aspell 0.50.5)

Relevant settings:
ispell-really-aspell t
ispell-program-name "ispell"

The problem is that ispell doesn't allow the "list" command:

$ echo "hello hhh" | ispell list
Error: Could not open the file "list" for reading.

whereas all the followng forms work (flagging hhh as a typo):

$ echo "hello hhh" | ispell  -l
$ echo "hello hhh" | aspell list
$ echo "hello hhh" | aspell -l


So can I suggest we make the simple change within:
(defun flyspell-large-region (beg end)

from 

(if ispell-really-aspell "list" "-l") (line 1332)

to

"-l"

Or are thing likely to be more complex depending on ispell/aspell version?

Stephen

Re: flyspell bug

[Peter Heslin]

On 2005-04-25, Stephen Eglen <S.J.Eglen@damtp.cam.ac.uk> wrote:
>  Relevant settings:
>  ispell-really-aspell t
>  ispell-program-name "ispell"

Aren't these contradictory settings?  If you have

  ispell-really-aspell t
  ispell-program-name "aspell"

your problem will go away, won't it?

I never understood why there are two user-configurable variables here,
myself.  Shouldn't setting ispell-really-aspell to t automatically set
ispell-program-name to "aspell"?

>  Or are thing likely to be more complex depending on ispell/aspell
>  version?

Seems likely, and a good reason not to change that line willy-nilly.

Peter

Re: flyspell bug

[Stephen Eglen]

Peter Heslin writes:
 > On 2005-04-25, Stephen Eglen <S.J.Eglen@damtp.cam.ac.uk> wrote:
 > >  Relevant settings:
 > >  ispell-really-aspell t
 > >  ispell-program-name "ispell"
 > 
 > Aren't these contradictory settings?  If you have
 > 
 >   ispell-really-aspell t
 >   ispell-program-name "aspell"
 > 
 > your problem will go away, won't it?

Yes, I think so, in this case.  However, I didn't set those variables,
so maybe ispell.el should be looked at.

 > I never understood why there are two user-configurable variables here,
 > myself.  Shouldn't setting ispell-really-aspell to t automatically set
 > ispell-program-name to "aspell"?
Indeed.

 > >  Or are thing likely to be more complex depending on ispell/aspell
 > >  version?
 > 
 > Seems likely, and a good reason not to change that line willy-nilly.

Okay!

I'll see if Ken Stevens (author of ispell) is aware of this.

Stephen

Re: flyspell bug

[Richard Stallman]

     > Aren't these contradictory settings?  If you have
     > 
     >   ispell-really-aspell t
     >   ispell-program-name "aspell"
     > 
     > your problem will go away, won't it?

    Yes, I think so, in this case.  However, I didn't set those variables,
    so maybe ispell.el should be looked at.

Could you debug why it sets the variables in an inconsistent manner?
It would be hard to debug this without being able to see it fail.

Re: flyspell bug

[Stephen Eglen]

Richard Stallman writes:
 >      > Aren't these contradictory settings?  If you have
 >      > 
 >      >   ispell-really-aspell t
 >      >   ispell-program-name "aspell"
 >      > 
 >      > your problem will go away, won't it?
 > 
 >     Yes, I think so, in this case.  However, I didn't set those variables,
 >     so maybe ispell.el should be looked at.
 > 
 > Could you debug why it sets the variables in an inconsistent manner?
 > It would be hard to debug this without being able to see it fail.

Having looked at ispell.el, I don't think they are inconsistent; I
think the logic of ispell.el is that if ispell-program-name is
"ispell", it just checks the output of ispell -v:

$ ispell -v
@(#) International Ispell Version 3.1.20 (but really Aspell 0.50.5)

and that parenthetic comment triggers ispell-really-aspell to t.

I suggest we just fix flyspell to always use the -l flag in this case,
rather than "list".

Stephen

Re: flyspell bug

[Magnus Henoch]

Stephen Eglen <S.J.Eglen@damtp.cam.ac.uk> writes:

> Having looked at ispell.el, I don't think they are inconsistent; I
> think the logic of ispell.el is that if ispell-program-name is
> "ispell", it just checks the output of ispell -v:
>
> $ ispell -v
> @(#) International Ispell Version 3.1.20 (but really Aspell 0.50.5)
>
> and that parenthetic comment triggers ispell-really-aspell to t.
>
> I suggest we just fix flyspell to always use the -l flag in this case,
> rather than "list".

I disagree.  I was the one who asked for the change from -l to list in
the first place, because it wouldn't work for me.  The situation seems
to be that your version of aspell installs a compatibility "ispell"
command which supports the -l option meaning "list", while my version
of aspell (0.60.2) doesn't install any such alias and only understands
"list" when invoked as aspell.  (-l specifies the language to use)

So it seems that the variable ispell-really-aspell is moderately
useful, as there are three situations: ispell, aspell in ispell
compatibility mode, and "real" aspell.

The following patch eliminates the middle case.  The documentation of
aspell 0.50.5 claims that it supports the "list" command, but I
haven't tested it.

--- orig/lisp/textmodes/ispell.el
+++ mod/lisp/textmodes/ispell.el
@@ -301,7 +301,8 @@
   :type 'integer
   :group 'ispell)
 
-(defcustom ispell-program-name "ispell"
+(defcustom ispell-program-name (or (executable-find "aspell")
+				   "ispell")
   "Program invoked by \\[ispell-word] and \\[ispell-region] commands."
   :type 'string
   :group 'ispell)

Magnus

Re: flyspell bug

[Stephen Eglen]

hi Magnus, thanks for explaining the version differences in aspell's
behaviour.  

 > The following patch eliminates the middle case.  The documentation of
 > aspell 0.50.5 claims that it supports the "list" command, but I
 > haven't tested it.
 > 
 > --- orig/lisp/textmodes/ispell.el
 > +++ mod/lisp/textmodes/ispell.el
 > @@ -301,7 +301,8 @@
 >    :type 'integer
 >    :group 'ispell)
 >  
 > -(defcustom ispell-program-name "ispell"
 > +(defcustom ispell-program-name (or (executable-find "aspell")
 > +				   "ispell")
 >    "Program invoked by \\[ispell-word] and \\[ispell-region] commands."
 >    :type 'string
 >    :group 'ispell)

Sounds like a good way to solve this, thanks.   Stephen

Re: flyspell bug

[Juri Linkov]

>> To: emacs-devel@gnu.org
>> From: Magnus Henoch <mange@freemail.hu>
>> Date: Wed, 27 Apr 2005 16:04:45 +0200
>> Subject: Re: flyspell bug
>> 
>> So it seems that the variable ispell-really-aspell is moderately
>> useful, as there are three situations: ispell, aspell in ispell
>> compatibility mode, and "real" aspell.
>> 
>> The following patch eliminates the middle case.  The documentation of
>> aspell 0.50.5 claims that it supports the "list" command, but I
>> haven't tested it.
>> 
>> - --- orig/lisp/textmodes/ispell.el
>> +++ mod/lisp/textmodes/ispell.el
>> @@ -301,7 +301,8 @@
>>    :type 'integer
>>    :group 'ispell)
>>  
>> - -(defcustom ispell-program-name "ispell"
>> +(defcustom ispell-program-name (or (executable-find "aspell")
>> +				   "ispell")
>>    "Program invoked by \\[ispell-word] and \\[ispell-region] commands."
>>    :type 'string
>>    :group 'ispell)
>
> Thanks, I installed a slightly different change (using locate-file
> instead of executable-find).

Now with using a non-default dictionary ispell.el fails with the
error message telling that ispell can't find a dictionary in
/usr/lib/aspell-0.60/.  That's becuase ispell and aspell have
different sets of dictionaries with different names.

ispell.el shouldn't insist on using "aspell" instead of "ispell"
even when aspell executable happens to be in exec-path.
Perhaps the following patch will do what was intended to fix the
original problem:

Index: lisp/textmodes/ispell.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/textmodes/ispell.el,v
retrieving revision 1.162
diff -u -r1.162 ispell.el
--- lisp/textmodes/ispell.el	7 May 2005 16:04:39 -0000	1.162
+++ lisp/textmodes/ispell.el	10 May 2005 00:05:05 -0000
@@ -302,7 +302,8 @@
   :group 'ispell)
 
 (defcustom ispell-program-name
-  (or (locate-file "aspell" exec-path exec-suffixes 'file-executable-p)
+  (or (and ispell-really-aspell
+	   (locate-file "aspell" exec-path exec-suffixes 'file-executable-p))
       "ispell")
   "Program invoked by \\[ispell-word] and \\[ispell-region] commands."
   :type 'string
@@ -815,7 +816,11 @@
         (goto-char (point-min))
         (let (case-fold-search)
           (setq ispell-really-aspell
-                (and (search-forward "(but really Aspell " nil t) t))))
+                (and (search-forward "(but really Aspell " nil t) t))
+	  (if ispell-really-aspell
+	      (let ((exec (locate-file "aspell" exec-path exec-suffixes
+				       'file-executable-p)))
+		(if exec (setq ispell-program-name exec))))))
       (kill-buffer (current-buffer)))
     result))
 
-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: flyspell bug

[Eli Zaretskii]

> From: Juri Linkov <juri@jurta.org>
> Date: Tue, 10 May 2005 03:06:48 +0300
> Cc: public@heslin.eclipse.co.uk, mange@freemail.hu, S.J.Eglen@damtp.cam.ac.uk
> 
> > Thanks, I installed a slightly different change (using locate-file
> > instead of executable-find).
> 
> Now with using a non-default dictionary ispell.el fails with the
> error message telling that ispell can't find a dictionary in
> /usr/lib/aspell-0.60/.  That's becuase ispell and aspell have
> different sets of dictionaries with different names.

Sorry about that.  I don't have aspell installed so I couldn't test
that, and people who were participating in the thread suggested that
this was the right solution.

> ispell.el shouldn't insist on using "aspell" instead of "ispell"
> even when aspell executable happens to be in exec-path.

I asked Richard and he said that's what we want, since aspell is newer
and is a GNU package.

> Perhaps the following patch will do what was intended to fix the
> original problem:
> 
> Index: lisp/textmodes/ispell.el
> ===================================================================
> RCS file: /cvsroot/emacs/emacs/lisp/textmodes/ispell.el,v
> retrieving revision 1.162
> diff -u -r1.162 ispell.el
> --- lisp/textmodes/ispell.el	7 May 2005 16:04:39 -0000	1.162
> +++ lisp/textmodes/ispell.el	10 May 2005 00:05:05 -0000
> @@ -302,7 +302,8 @@
>    :group 'ispell)
>  
>  (defcustom ispell-program-name
> -  (or (locate-file "aspell" exec-path exec-suffixes 'file-executable-p)
> +  (or (and ispell-really-aspell
> +	   (locate-file "aspell" exec-path exec-suffixes 'file-executable-p))
>        "ispell")

??? Does this really work?  AFAIK, ispell-really-aspell is set when
ispell.el invokes the speller for the first time, not before that.  So
this defcustom wuill never try aspell, I think.

Am I missing something?

Re: flyspell bug

[Juri Linkov]

>> Index: lisp/textmodes/ispell.el
>> ===================================================================
>> RCS file: /cvsroot/emacs/emacs/lisp/textmodes/ispell.el,v
>> retrieving revision 1.162
>> diff -u -r1.162 ispell.el
>> --- lisp/textmodes/ispell.el	7 May 2005 16:04:39 -0000	1.162
>> +++ lisp/textmodes/ispell.el	10 May 2005 00:05:05 -0000
>> @@ -302,7 +302,8 @@
>>    :group 'ispell)
>>  
>>  (defcustom ispell-program-name
>> -  (or (locate-file "aspell" exec-path exec-suffixes 'file-executable-p)
>> +  (or (and ispell-really-aspell
>> +	   (locate-file "aspell" exec-path exec-suffixes 'file-executable-p))
>>        "ispell")
>
> ??? Does this really work?  AFAIK, ispell-really-aspell is set when
> ispell.el invokes the speller for the first time, not before that.  So
> this defcustom wuill never try aspell, I think.

This may be needed when users set `ispell-really-aspell' explicitly
in .emacs.  However, after looking at ispell.el it seems that
`ispell-check-version' always is called before invoking the speller
for the first time.  So maybe the second part of my patch makes
the search for aspell in defcustom unnecessary, and its initial
value should be reverted to "ispell".

Note that automatic changing the value of `ispell-program-name'
in `ispell-check-version' might surprise users of the aspell's
implementation of ispell.  Anyhow, this is the better place to
change ispell program name to "aspell" than defcustom.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: flyspell bug

[Kai Großjohann]

Peter Heslin <public@heslin.eclipse.co.uk> writes:

> I never understood why there are two user-configurable variables here,
> myself.  Shouldn't setting ispell-really-aspell to t automatically set
> ispell-program-name to "aspell"?

Some folks might set ispell-program-name to "/sw/foo/bin/aspell", so
the variable is not redundant.

Kai

Re: flyspell bug

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, public@heslin.eclipse.co.uk,
> 	mange@freemail.hu, s.j.eglen@damtp.cam.ac.uk
> From: Juri Linkov <juri@jurta.org>
> Date: Tue, 10 May 2005 08:04:07 +0300
> 
> Note that automatic changing the value of `ispell-program-name'
> in `ispell-check-version' might surprise users of the aspell's
> implementation of ispell.  Anyhow, this is the better place to
> change ispell program name to "aspell" than defcustom.

I think the right thing to do is to modify what's currently in CVS so
that it doesn't fail with non-English dictionaries.  Can you describe
how it fails in that case, and why?

Re: flyspell bug

[Juri Linkov]

>> Cc: emacs-devel@gnu.org, public@heslin.eclipse.co.uk,
>> 	mange@freemail.hu, s.j.eglen@damtp.cam.ac.uk
>> From: Juri Linkov <juri@jurta.org>
>> Date: Tue, 10 May 2005 08:04:07 +0300
>> 
>> Note that automatic changing the value of `ispell-program-name'
>> in `ispell-check-version' might surprise users of the aspell's
>> implementation of ispell.  Anyhow, this is the better place to
>> change ispell program name to "aspell" than defcustom.
>
> I think the right thing to do is to modify what's currently in CVS so
> that it doesn't fail with non-English dictionaries.  Can you describe
> how it fails in that case, and why?

ispell and aspell use different dictionaries.  Changing the default
ispell program name from "ispell" to "aspell" is an incompatible
change for users configured Emacs to use ispell dictionaries,
not aspell dictionaries.  `ispell-program-name' can be changed
automatically to "aspell" only when `ispell-check-version' detects
aspell's implemention of ispell, i.e. a program for which `ispell -v'
returns:

@(#) International Ispell Version 3.1.20 (but really Aspell 0.50.5)

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: flyspell bug

[Eli Zaretskii]

> From: Juri Linkov <juri@jurta.org>
> Date: Wed, 11 May 2005 11:23:48 +0300
> 
> ispell and aspell use different dictionaries.  Changing the default
> ispell program name from "ispell" to "aspell" is an incompatible
> change for users configured Emacs to use ispell dictionaries,
> not aspell dictionaries.

I understand that.  What I need is a way to distinguish between three
situations:

  (1) user doesn't have aspell installed
  (2) user has aspell installed, but configured it to use ispell's
      dictionaries
  (3) user has aspell installed and wants to use aspell's dictionaries

I know how to detect (1) -- use locate-file or similar technique to
look for aspell.  But how to distinguish between (2) and (3)?

> `ispell-program-name' can be changed
> automatically to "aspell" only when `ispell-check-version' detects
> aspell's implemention of ispell, i.e. a program for which `ispell -v'
> returns:
> 
> @(#) International Ispell Version 3.1.20 (but really Aspell 0.50.5)

So the conditions for (2) should be:

  (a) locate-file finds an executable `ispell'
  (b) that `ispell', when invoked with -v outputs the above banner

Is that right?  Are we sure that no user will use aspell with ispell's
dictionaries _except_ through the `ispell' alias?

Re: flyspell bug

[Richard Stallman]

    ispell.el shouldn't insist on using "aspell" instead of "ispell"
    even when aspell executable happens to be in exec-path.

Of course it should.  aspell is a GNU package.

What led you to think the contrary?

Re: flyspell bug

[Juri Linkov]

>     ispell.el shouldn't insist on using "aspell" instead of "ispell"
>     even when aspell executable happens to be in exec-path.
>
> Of course it should.  aspell is a GNU package.
>
> What led you to think the contrary?

I had the impression that the intention was to fix a problem of using
aspell with ispell.el.  However, the installed change is too radical.
It affects all users of ispell having aspell in exec-path.

If switching from ispell to aspell was intentional (I can't find
a discussion about this on emacs-devel) then we need to document
it in the NEWS file with a warning about possible problems, to inform
maintainers of ispell.el and aspell about this change asking them to
update documentation, web pages, and to test aspell dictionaries
to confirm their compatibility with ispell.el.

But perhaps switching from ispell to aspell was too hasty change.
According to aspell's TODO list, ispell.el is not quite ready for
using aspell.  The Info node (info "(aspell)Important Items") has:

  Things that need to be done
  ---------------------------

     * Enhance *ispell.el* so that it will work better with GNU Aspell.
       _[In Progress]_.

I don't know what is the current status of this TODO item.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: flyspell bug

[Juri Linkov]

>   (1) user doesn't have aspell installed
>   (2) user has aspell installed, but configured it to use ispell's
>       dictionaries
>   (3) user has aspell installed and wants to use aspell's dictionaries

There is also a fourth situation:

    (4) user has aspell installed, but uses ispell with ispell's dictionaries

> I know how to detect (1) -- use locate-file or similar technique to
> look for aspell.  But how to distinguish between (2) and (3)?

AFAIK, the situation (2) doesn't exist.  There are the following
situations instead:

    (2) user has aspell installed and uses the compatibility script
        `ispell' as a wrapper around aspell with aspell's dictionaries

    (3) user has aspell installed and uses `aspell' executable with
        aspell's dictionaries

(2) is detected by checking the output of `ispell -v'.
(3) is configured by users by changing ispell-program-name to "aspell".

After the change in flyspell.el it doesn't work with the `ispell'
compatibility script.  Maybe ispell-really-aspell should have more
values to distinguish between (2) and (3).  (3) can be detected
by matching ispell-program-name against the regexp "aspell$".

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: flyspell bug

[Juri Linkov]

To my surprise I just discovered that the latest development release of
ispell.el at http://kdstevens.com/~stevens/ispell-page.html already
supports the option to choose between ispell and aspell:

(defcustom ispell-prefer-aspell nil
  "*Select preference between using the `ispell' or `aspell' program.
`ispell' is used by default.
When this variable is set, `aspell' is used if it is installed on the system."
  :type 'boolean
  :group 'ispell)

(defcustom ispell-program-name
  (or (and (exec-installed-p "aspell")
	   (or ispell-prefer-aspell
	       (not (exec-installed-p "ispell")))
	   "aspell")
      "ispell")
  "Program invoked by \\[ispell-word] and \\[ispell-region] commands."
  :type 'string
  :group 'ispell)

This seems right to me.  Are there any reasons why the latest
development release of ispell.el is not synced with Emacs CVS?

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: flyspell bug

[Eli Zaretskii]

> Cc: s.j.eglen@damtp.cam.ac.uk, mange@freemail.hu,
> 	public@heslin.eclipse.co.uk, emacs-devel@gnu.org
> From: Juri Linkov <juri@jurta.org>
> Date: Thu, 12 May 2005 08:29:55 +0300
> 
> >   (1) user doesn't have aspell installed
> >   (2) user has aspell installed, but configured it to use ispell's
> >       dictionaries
> >   (3) user has aspell installed and wants to use aspell's dictionaries
> 
> There is also a fourth situation:
> 
>     (4) user has aspell installed, but uses ispell with ispell's dictionaries

That is the situation in which Richard wanted Emacs to prefer aspell.

Re: flyspell bug

[Richard Stallman]

    ispell and aspell use different dictionaries.  Changing the default
    ispell program name from "ispell" to "aspell" is an incompatible
    change for users configured Emacs to use ispell dictionaries,
    not aspell dictionaries.

Maybe aspell could support ispell dictionaries.  Or, aspell could see
that it has been given an ispell dictionary file for a certain
language, and automatically use the aspell dictionary for the same
language (or report an error if it doesn't have one available).

Kevin, what do you think?

Re: flyspell bug

[Eli Zaretskii]

> From: Juri Linkov <juri@jurta.org>
> Date: Thu, 12 May 2005 09:14:47 +0300
> Cc: public@heslin.eclipse.co.uk, mange@freemail.hu, s.j.eglen@damtp.cam.ac.uk,
> 	k.stevens@ieee.org
> 
> To my surprise I just discovered that the latest development release of
> ispell.el at http://kdstevens.com/~stevens/ispell-page.html already
> supports the option to choose between ispell and aspell:

This uses exec-installed-p, which we don't have.  Where is it from?

Re: flyspell bug

[Ken Stevens]

"Eli Zaretskii" writes: 

> This uses exec-installed-p, which we don't have.  Where is it from?

I have been the developer and maintainer of this for years - however, I
have become neglectful of late due to work, family, and other
volunteering loads.

I have a version that doesn't use exec-installed-p, but have not put it
out for testing.

regards		-Ken

Re: flyspell bug

[Richard Stallman]

    This seems right to me.  Are there any reasons why the latest
    development release of ispell.el is not synced with Emacs CVS?

In principle there is no reason--it would be good to install
the latest code.  Ken, could you do this soon?

Re: flyspell bug

[Richard Stallman]

      However, the installed change is too radical.
    It affects all users of ispell having aspell in exec-path.

Preferring aspell to ispell will inevitably do that.
It is the price that must be paid for a change that we will make
sooner or later.

We are going to switch to preferring aspell sometime.  If aspell is
not ready, we can't do it.  If aspell is ready now, sooner is better
than later.

    If switching from ispell to aspell was intentional (I can't find
    a discussion about this on emacs-devel) then we need to document
    it in the NEWS file with a warning about possible problems, to inform
    maintainers of ispell.el and aspell about this change asking them to
    update documentation, web pages, and to test aspell dictionaries
    to confirm their compatibility with ispell.el.

We're not building airplanes, you know.  Ideally in the absence of
resource constraints we would do all these things, but we don't have
to do them all.  We should not evaluate development choices as if we
were going to do them with the level of care that would be required
for building airplanes.

    But perhaps switching from ispell to aspell was too hasty change.
    According to aspell's TODO list, ispell.el is not quite ready for
    using aspell.

I would not rely on aspell's TODO list for the very latest information
on the status of ispell.el development.  The facts we are observing
directly about ispell are much more authoritative than a report
that might be old.

Re: flyspell bug

[Richard Stallman]

    I have a version that doesn't use exec-installed-p, but have not put it
    out for testing.

Could you email it to us?  We can look at the differences from your
previous version, and also test it.

Re: flyspell bug

[Ken Stevens]

Richard Stallman writes: 

>     I have a version that doesn't use exec-installed-p, but have not
>     put it out for testing.
> 
> Could you email it to us?  We can look at the differences from your
> previous version, and also test it.

The current version I have does not try to enforce a particular speller
- both ispell and aspell are supported.  The choice is determined by the
setting of the customized variable `ispell-prefer-aspell' when both ispell
and aspell exist on the system.

I had used the function `exec-installed-p' which had it's roots in APEL
(A Portable Emacs Library).  In digging deeper, it appears that this is
not necessary as the standard emacs function `executable-find' appears
to have the same desired functionality.

Here are the variables:


(defcustom ispell-prefer-aspell nil
  "*Select preference between using the `ispell' or `aspell' program.
`ispell' is used by default.
When this variable is set, `aspell' is used if it is installed on the system."
  :type 'boolean
  :group 'ispell)


(defcustom ispell-program-name
  (or (and (executable-find "aspell")
	   (or ispell-prefer-aspell
	       (not (executable-find "ispell")))
	   "aspell")
      "ispell")
  "Program invoked by \\[ispell-word] and \\[ispell-region] commands."
  :type 'string
  :group 'ispell)


I'm sure you'll argue to make `ispell-prefer-aspell' set by default ;-)
It is only used in the above location in the code.


I have several other patches, improvements to integrate into the current
version of ispell as well.

regards	   -Ken

Re: flyspell bug

[Richard Stallman]

    The current version I have does not try to enforce a particular speller
    - both ispell and aspell are supported.  The choice is determined by the
    setting of the customized variable `ispell-prefer-aspell' when both ispell
    and aspell exist on the system.

That sounds good.

    I have several other patches, improvements to integrate into the current
    version of ispell as well.

How about sending us this version in some form or other?

Re: flyspell bug

[Kevin Atkinson]

On Thu, 12 May 2005, Richard Stallman wrote:

>     ispell and aspell use different dictionaries.  Changing the default
>     ispell program name from "ispell" to "aspell" is an incompatible
>     change for users configured Emacs to use ispell dictionaries,
>     not aspell dictionaries.
> 
> Maybe aspell could support ispell dictionaries.  

They are in a compiled binary format and doing such would be rather 
difficult.

> Or, aspell could see that it has been given an ispell dictionary file
> for a certain language, and automatically use the aspell dictionary for
> the same language (or report an error if it doesn't have one available).

If you use "-d DICT" where DICT does NOT end in ".hash" or is a complete 
path to an Ispell dictionary than it should work as you suggested.  This 
strategy should work for both Ispell and Aspell as Ispell.  If it did not 
find one it should print an error to standard out and return a non-zero 
exit code.

If a complete path is provided and the supplied dictionary is not an 
Aspell one it will also report an error to standard out and return a 
non-zero exit code.

-- 
http://kevin.atkinson.dhs.org

Re: flyspell bug

[Richard Stallman]

    > Or, aspell could see that it has been given an ispell dictionary file
    > for a certain language, and automatically use the aspell dictionary for
    > the same language (or report an error if it doesn't have one available).

    If you use "-d DICT" where DICT does NOT end in ".hash" or is a complete 
    path to an Ispell dictionary than it should work as you suggested.

If this already works, how come the problem happens?
Why doesn't this make everything work right already?

Re: flyspell bug

[Kevin Atkinson]

On Tue, 24 May 2005, Richard Stallman wrote:

>     > Or, aspell could see that it has been given an ispell dictionary file
>     > for a certain language, and automatically use the aspell dictionary for
>     > the same language (or report an error if it doesn't have one available).
> 
>     If you use "-d DICT" where DICT does NOT end in ".hash" or is a complete 
>     path to an Ispell dictionary than it should work as you suggested.
> 
> If this already works, how come the problem happens?
> Why doesn't this make everything work right already?

Are you asking me or some of the peole on the CC list?

-- 
http://kevin.atkinson.dhs.org

Re: flyspell bug

[Juri Linkov]

>     If you use "-d DICT" where DICT does NOT end in ".hash" or is a complete 
>     path to an Ispell dictionary than it should work as you suggested.
>
> If this already works, how come the problem happens?
> Why doesn't this make everything work right already?

Users may have different sets of dictionaries installed for Ispell
and Aspell.  After replacing the default program name "ispell" by
"aspell", ispell.el may fail when a configured Ispell's dictionary
name doesn't match any of installed Aspell dictionary names.

To avoid such failure, ispell.el could check the Aspell's exit code
and in case when Aspell can't find a dictionary name among its list
of available dictionaries, ispell.el could change the default program
name from "aspell" to "ispell" and try to run Ispell with the same
dictionary name.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: flyspell bug

[Kevin Atkinson]

On Wed, 25 May 2005, Juri Linkov wrote:

> >     If you use "-d DICT" where DICT does NOT end in ".hash" or is a complete 
> >     path to an Ispell dictionary than it should work as you suggested.
> >
> > If this already works, how come the problem happens?
> > Why doesn't this make everything work right already?
> 
> Users may have different sets of dictionaries installed for Ispell
> and Aspell.  After replacing the default program name "ispell" by
> "aspell", ispell.el may fail when a configured Ispell's dictionary
> name doesn't match any of installed Aspell dictionary names.
> 
> To avoid such failure, ispell.el could check the Aspell's exit code
> and in case when Aspell can't find a dictionary name among its list
> of available dictionaries, ispell.el could change the default program
> name from "aspell" to "ispell" and try to run Ispell with the same
> dictionary name.

Most Aspell dictionaries are installed with names that the orignal Ispell 
dictionarys use.  Just make sure it isn't a complete path or the name 
doesn't end if ".hash".

-- 
http://kevin.atkinson.dhs.org

Re: flyspell bug

[Richard Stallman]

    > If this already works, how come the problem happens?
    > Why doesn't this make everything work right already?

    Are you asking me or some of the peole on the CC list?

Whoever knows the answer, I guess.

Re: flyspell bug

[Richard Stallman]

    To avoid such failure, ispell.el could check the Aspell's exit code
    and in case when Aspell can't find a dictionary name among its list
    of available dictionaries, ispell.el could change the default program
    name from "aspell" to "ispell" and try to run Ispell with the same
    dictionary name.

That is an ugly temporary workaround.  What I would rather do is give
a clear error message that points the user at the ways to reconfigure.

flyspell bug

[Richard M. Stallman]

It seems to me that there is no reason for flyspell.el to check the
buffer-local specifications every time it checks or corrects a word;
doing this when the user enables Flyspell mode ought to be enough.

Which suggests making this change.  Does it fix the problem?


*** flyspell.el	20 Sep 2005 19:31:14 -0400	1.76
--- flyspell.el	02 Oct 2005 01:38:17 -0400	
***************
*** 981,987 ****
    (interactive (list ispell-following-word))
    (save-excursion
      ;; use the correct dictionary
!     (flyspell-accept-buffer-local-defs)
      (let* ((cursor-location (point))
  	  (flyspell-word (flyspell-get-word following))
  	  start end poss word)
--- 981,987 ----
    (interactive (list ispell-following-word))
    (save-excursion
      ;; use the correct dictionary
! ;;;    (flyspell-accept-buffer-local-defs)
      (let* ((cursor-location (point))
  	  (flyspell-word (flyspell-get-word following))
  	  start end poss word)
***************
*** 1715,1721 ****
    (let ((pos     (point))
  	(old-max (point-max)))
      ;; use the correct dictionary
!     (flyspell-accept-buffer-local-defs)
      (if (and (eq flyspell-auto-correct-pos pos)
  	     (consp flyspell-auto-correct-region))
  	;; we have already been using the function at the same location
--- 1715,1721 ----
    (let ((pos     (point))
  	(old-max (point-max)))
      ;; use the correct dictionary
! ;;;    (flyspell-accept-buffer-local-defs)
      (if (and (eq flyspell-auto-correct-pos pos)
  	     (consp flyspell-auto-correct-region))
  	;; we have already been using the function at the same location
***************
*** 1886,1892 ****
  The word checked is the word at the mouse position."
    (interactive "e")
    ;; use the correct dictionary
!   (flyspell-accept-buffer-local-defs)
    ;; retain cursor location (I don't know why but save-excursion here fails).
    (let ((save (point)))
      (mouse-set-point event)
--- 1886,1892 ----
  The word checked is the word at the mouse position."
    (interactive "e")
    ;; use the correct dictionary
! ;;;  (flyspell-accept-buffer-local-defs)
    ;; retain cursor location (I don't know why but save-excursion here fails).
    (let ((save (point)))
      (mouse-set-point event)

Re: flyspell bug

[Slawomir Nowaczyk]

On Sun, 02 Oct 2005 13:08:27 -0400
"Richard M. Stallman" <rms@gnu.org> wrote:

#> It seems to me that there is no reason for flyspell.el to check the
#> buffer-local specifications every time it checks or corrects a word;
#> doing this when the user enables Flyspell mode ought to be enough.

Not really. I work sometimes with several buffers where each is
written in a different language. After your change, the ispell
dictionary isn't automatically adjusted anymore.

I need to manually run, for example, ispell-word in order for
ispell/flyspell to notice it should change language/dictionary.

It seems, therefore, that flyspell should read buffer local
specifications at least after changing buffers.

#> Which suggests making this change. Does it fix the problem?

Your change does seem to fix the problem with transient mark mode for
me, though.

-- 
 Best wishes,
   Slawomir Nowaczyk
     ( slawomir.nowaczyk.847@student.lu.se )

The first myth of good management is that it exists.

Re: flyspell bug

[Kim F. Storm]

Slawomir Nowaczyk <slawomir.nowaczyk.847@student.lu.se> writes:

> On Sun, 02 Oct 2005 13:08:27 -0400
> "Richard M. Stallman" <rms@gnu.org> wrote:
>
> #> It seems to me that there is no reason for flyspell.el to check the
> #> buffer-local specifications every time it checks or corrects a word;
> #> doing this when the user enables Flyspell mode ought to be enough.
>
> Not really. I work sometimes with several buffers where each is
> written in a different language. After your change, the ispell
> dictionary isn't automatically adjusted anymore.
>
> I need to manually run, for example, ispell-word in order for
> ispell/flyspell to notice it should change language/dictionary.
>
> It seems, therefore, that flyspell should read buffer local
> specifications at least after changing buffers.

I guess you mean after switching buffers (in the emacs terminology)?

I often wondered why there are no hooks associated with switching
buffers, windows, and frames.

E.g. 
switch-buffer-functions (args: old buffer, new buffer)

select-window-functions (args: old window, new window)

select-frame-functions (args: old frame, new frame)

But the problem might be to decide when to actually run these
hooks -- I think somewhere in the main loop...?

>
> #> Which suggests making this change. Does it fix the problem?
>
> Your change does seem to fix the problem with transient mark mode for
> me, though.

Which is one good reason to install it unless somebody can make
a better fix soon!!

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: flyspell bug

[Slawomir Nowaczyk]

On Sun, 02 Oct 2005 23:45:26 +0200
storm@cua.dk (Kim F. Storm) wrote:

#> > It seems, therefore, that flyspell should read buffer local
#> > specifications at least after changing buffers.

#> I guess you mean after switching buffers (in the emacs terminology)?

Yes... sorry for being unclear.

#> I often wondered why there are no hooks associated with switching
#> buffers, windows, and frames.

Having those would be a great idea, IMHO.

I like to my cursor change colour to red when in overwrite mode, and
currently my solution uses post-command-hook. Having a hook run after
buffer switch would simplify things :)

#> > #> Which suggests making this change. Does it fix the problem?
#> >
#> > Your change does seem to fix the problem with transient mark mode for
#> > me, though.
#> 
#> Which is one good reason to install it unless somebody can make
#> a better fix soon!!

Yes, definitely. I have applied the fix to my personal copy of Emacs
already :)

-- 
 Best wishes,
   Slawomir Nowaczyk
     ( slawomir.nowaczyk.847@student.lu.se )

Be nice to other people--they outnumber you six billion to one.

Re: flyspell bug

[Richard M. Stallman]

    I often wondered why there are no hooks associated with switching
    buffers, windows, and frames.

These activities are too low level.  Certainly in the past it was not
safe to run Lisp code for switching buffers, and maybe not for windows
either.  Nowadays, since Lisp code can run during redisplay, maybe
it would be safe.  But I do not like the idea.  Do you really want
set-buffer to run Lisp code you don't know about?

Re: flyspell bug

[Richard M. Stallman]

Do these changes give good results?

*** ispell.el	24 Sep 2005 22:42:55 -0400	1.177
--- ispell.el	03 Oct 2005 11:33:22 -0400	
***************
*** 1558,1563 ****
--- 1558,1564 ----
    (interactive (list ispell-following-word ispell-quietly current-prefix-arg))
    (if continue
        (ispell-continue)
+     (ispell-maybe-find-aspell-dictionaries) 
      (ispell-accept-buffer-local-defs)	; use the correct dictionary
      (let ((cursor-location (point))	; retain cursor location
  	  (word (ispell-get-word following))
***************
*** 2616,2621 ****
--- 2617,2623 ----
  Return nil if spell session is quit,
   otherwise returns shift offset amount for last line processed."
    (interactive "r")			; Don't flag errors on read-only bufs.
+   (ispell-maybe-find-aspell-dictionaries)
    (if (not recheckp)
        (ispell-accept-buffer-local-defs)) ; set up dictionary, local words, etc.
    (let ((skip-region-start (make-marker))
***************
*** 3577,3583 ****
  
  (defun ispell-accept-buffer-local-defs ()
    "Load all buffer-local information, restarting Ispell when necessary."
-   (ispell-maybe-find-aspell-dictionaries)
    (ispell-buffer-local-dict)		; May kill ispell-process.
    (ispell-buffer-local-words)		; Will initialize ispell-process.
    (ispell-buffer-local-parsing))
--- 3579,3584 ----
*** flyspell.el	20 Sep 2005 19:31:14 -0400	1.76
--- flyspell.el	03 Oct 2005 11:34:03 -0400	
***************
*** 536,541 ****
--- 536,542 ----
  ;*---------------------------------------------------------------------*/
  (defun flyspell-mode-on ()
    "Turn Flyspell mode on.  Do not use this; use `flyspell-mode' instead."
+   (ispell-maybe-find-aspell-dictionaries)
    (setq ispell-highlight-face 'flyspell-incorrect)
    ;; local dictionaries setup
    (or ispell-local-dictionary ispell-dictionary

Re: flyspell bug

[Kim F. Storm]

"Richard M. Stallman" <rms@gnu.org> writes:

>     I often wondered why there are no hooks associated with switching
>     buffers, windows, and frames.
>
> These activities are too low level.  Certainly in the past it was not
> safe to run Lisp code for switching buffers, and maybe not for windows
> either.  Nowadays, since Lisp code can run during redisplay, maybe
> it would be safe.  But I do not like the idea.  Do you really want
> set-buffer to run Lisp code you don't know about?

I was not suggesting to do this at the low-level.

Rather I would do it in the top-level command loop, e.g.
by saving the current window/buffer/frame before running
the pre-command hook and compare them to the value after
running the post-command-hook -- and run the appropriate
hooks at that time.

That way, e.g. set-buffer on its own won't run any unknown Lisp code.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: flyspell bug

[Richard M. Stallman]

    Rather I would do it in the top-level command loop, e.g.
    by saving the current window/buffer/frame before running
    the pre-command hook and compare them to the value after
    running the post-command-hook -- and run the appropriate
    hooks at that time.

    That way, e.g. set-buffer on its own won't run any unknown Lisp code.

I think it would be extremely confusing if switching windows to look
around in another buffer were likely to run some Lisp code.
It would make debugging very painful, if you could not look at buffers
without changing them.

Re: flyspell bug

[Kim F. Storm]

"Richard M. Stallman" <rms@gnu.org> writes:

>     Rather I would do it in the top-level command loop, e.g.
>     by saving the current window/buffer/frame before running
>     the pre-command hook and compare them to the value after
>     running the post-command-hook -- and run the appropriate
>     hooks at that time.
>
>     That way, e.g. set-buffer on its own won't run any unknown Lisp code.
>
> I think it would be extremely confusing if switching windows to look
> around in another buffer were likely to run some Lisp code.

How would you notice if flyspell did some "behind the scenes" 
dictionary setup ?

It would be more confusing if switching to another buffer did
not do that ... as I understand the current problem was about?

> It would make debugging very painful, if you could not look at buffers
> without changing them.

Don't load modes that use those hooks then...

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: flyspell bug

[Richard M. Stallman]

    > I think it would be extremely confusing if switching windows to look
    > around in another buffer were likely to run some Lisp code.

    How would you notice if flyspell did some "behind the scenes" 
    dictionary setup ?

It might cause a lot of trouble if it had a bug of some kind, or if it
did things with processes at a time you did not expect.

Anyway, my intuition, based on many years of programming, says
that it is a bad idea for switching buffers to run a hook.

Re: flyspell bug

[David Kastrup]

"Richard M. Stallman" <rms@gnu.org> writes:

>     > I think it would be extremely confusing if switching windows to look
>     > around in another buffer were likely to run some Lisp code.
>
>     How would you notice if flyspell did some "behind the scenes" 
>     dictionary setup ?
>
> It might cause a lot of trouble if it had a bug of some kind, or if
> it did things with processes at a time you did not expect.
>
> Anyway, my intuition, based on many years of programming, says that
> it is a bad idea for switching buffers to run a hook.

Switching windows to look around in another buffer certainly can run
hooks right now.  We have pre-command-hook, post-command-hook, we have
focus events and of course we have the "display" property which can
run Lisp code, and which will get triggered upon redisplay.

All of those however are connected with windows and the user interface
and the current _window_.  The current _buffer_ is something which is
not visible and which can be switched around from C.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: flyspell bug

[Slawomir Nowaczyk]

On Mon, 03 Oct 2005 11:35:27 -0400
"Richard M. Stallman" <rms@gnu.org> wrote:

#> Do these changes give good results?

Not really. For some reason flyspell still does not notice
automatically when it needs to restart ispell because current buffer
uses different dictionary.

Anyway, I think it is important to apply your initial patch ASAP,
because flyspell not working with transient mode is much more annoying
than those minor problems with multi-language spellchecking.

In addition, there seem to be something seriously wrong with flyspell
mode, in particular with flyspell-buffer function. It doesn't check
the whole buffer anymore. It seems to work near the beginning, or when
buffer is short, but further down it does not mark words as
misspelled. flyspell-region has the same problem.

To reproduce, visit emacs/etc/MACHINES, M-x flyspell-buffer, go to end
of the buffer and notice that, for example, "Smail" isn't marked as
misspelled word. If you place the point over this word, flyspell will
mark it as misspelled.

Unfortunately I do not understand the code and so not have time right
now to track down this bug :( I also do not know when this bug was
introduced -- all I can say is that the version of CVS emacs that I
compiled on 2005-05-03 works fine.

-- 
 Best wishes,
   Slawomir Nowaczyk
     ( slawomir.nowaczyk.847@student.lu.se )

Meddle not in the affairs of dragons, for thou art crunchy,
and good with ketchup

Re: flyspell bug

[Slawomir Nowaczyk]

On Wed, 05 Oct 2005 12:41:15 +0200
Slawomir Nowaczyk <slawomir.nowaczyk.847@student.lu.se> wrote:

#> On Mon, 03 Oct 2005 11:35:27 -0400
#> "Richard M. Stallman" <rms@gnu.org> wrote:
#> 
#> #> Do these changes give good results?
#> 
#> Not really. For some reason flyspell still does not notice
#> automatically when it needs to restart ispell because current buffer
#> uses different dictionary.

Ugh... scratch that... the current CVS code changes dictionaries
perfectly. My local emacs source, on which I tried to check your
change before, must have gotten corrupted somehow during my feeble
attempts to figure out what was the problem.

However, the bug described below is still present:

#> In addition, there seem to be something seriously wrong with flyspell
#> mode, in particular with flyspell-buffer function. It doesn't check
#> the whole buffer anymore. It seems to work near the beginning, or when
#> buffer is short, but further down it does not mark words as
#> misspelled. flyspell-region has the same problem.
#> 
#> To reproduce, visit emacs/etc/MACHINES, M-x flyspell-buffer, go to end
#> of the buffer and notice that, for example, "Smail" isn't marked as
#> misspelled word. If you place the point over this word, flyspell will
#> mark it as misspelled.
#> 
#> Unfortunately I do not understand the code and so not have time right
#> now to track down this bug :( I also do not know when this bug was
#> introduced -- all I can say is that the version of CVS emacs that I
#> compiled on 2005-05-03 works fine.

-- 
 Best wishes,
   Slawomir Nowaczyk
     ( slawomir.nowaczyk.847@student.lu.se )

COMMAND: A suggestion made to a computer.

Re: flyspell bug

[Richard M. Stallman]

I installed a fix that seems to make flyspell-buffer work right.
Does it work now?

Re: flyspell bug

[Slawomir Nowaczyk]

On Mon, 10 Oct 2005 00:15:13 -0400
"Richard M. Stallman" <rms@gnu.org> wrote:

#> I installed a fix that seems to make flyspell-buffer work right.
#> Does it work now?

Thank you very much, it works better... but still not perfect :(

flyspell-buffer doesn't work in (La)TeX mode (maybe also in some
others): open etc/refcard.tex M-x flyspell-buffer, go to the end of
the buffer and notice that, for example, recenter isn't marked as
misspelled.

Also note that changing major mode to text-mode before running
flyspell-buffer still doesn't make it check the file correctly
(although "recenter" is correctly marked as misspelled, "endexample"
is not).

Again sorry I do not have time to track down the bug :(

-- 
 Best wishes,
   Slawomir Nowaczyk
     ( slawomir.nowaczyk.847@student.lu.se )

Anybody who doesn't cut his speed at the sight of a police car
is probably parked.

Re: flyspell bug

[Richard M. Stallman]

    flyspell-buffer doesn't work in (La)TeX mode (maybe also in some
    others): open etc/refcard.tex M-x flyspell-buffer, go to the end of
    the buffer and notice that, for example, recenter isn't marked as
    misspelled.

Why should it be?  It is correct.

As far as I can see, flyspell-buffer marks the words that are
reported as incorrect.

Re: flyspell bug

[Slawomir Nowaczyk]

On Thu, 13 Oct 2005 00:52:41 -0400
"Richard M. Stallman" <rms@gnu.org> wrote:

#>     flyspell-buffer doesn't work in (La)TeX mode (maybe also in some
#>     others): open etc/refcard.tex M-x flyspell-buffer, go to the end of
#>     the buffer and notice that, for example, recenter isn't marked as
#>     misspelled.
#> 
#> Why should it be?  It is correct.

Well, my ispell, using English dictionary, reports "recenter" as
incorrect. Sorry, I didn't notice it is, indeed, a correct word :) How
about "defun", four lines above?

#> As far as I can see, flyspell-buffer marks the words that are
#> reported as incorrect.

Yes, that's exactly what it is supposed to be doing.

One thing I can add is that things like "LocalWords" can change the
behaviour of flyspell-buffer. I have a file for which flyspell-buffer
works fine if there are no "LocalWords", and only checks part of the
buffer if I add " LocalWords: Shawn" definition.

Also, it is possible to notice if flyspell-buffer is working OK or
not, by looking at the echo area... flyspell reports percent of buffer
already checked there -- if this goes steadily from 0% to 100%, all is
fine.

But sometimes it seems to behave strangely, like taking 10 seconds to
go from 0% to 10%, then immediately jumping to 40%, and then
immediately reporting "Spell Checking completed." And in such cases
the buffer is not checked correctly.

-- 
 Best wishes,
   Slawomir Nowaczyk
     ( slawomir.nowaczyk.847@student.lu.se )

One cannot make an omelette without breaking eggs -- but it is
amazing how many eggs one can break without making a decent omelette.

Re: flyspell bug

[Richard M. Stallman]

    But sometimes it seems to behave strangely, like taking 10 seconds to
    go from 0% to 10%, then immediately jumping to 40%, and then
    immediately reporting "Spell Checking completed." And in such cases
    the buffer is not checked correctly.

I agree.  If you send us a test case that fails, we can debug it.

Re: flyspell bug

[Slawomir Nowaczyk]

On Fri, 14 Oct 2005 01:11:05 -0400
"Richard M. Stallman" <rms@gnu.org> wrote:

#>     But sometimes it seems to behave strangely, like taking 10 seconds to
#>     go from 0% to 10%, then immediately jumping to 40%, and then
#>     immediately reporting "Spell Checking completed." And in such cases
#>     the buffer is not checked correctly.
#> 
#> I agree.  If you send us a test case that fails, we can debug it.

That's what I did earlier... Sorry if I was unclear. Can you reproduce
the bug on your machine? The stuff below exposes the bug for me, after
"runemacs --no-init-file --no-site-file"

open etc/refcard.tex, M-x flyspell-buffer, go to the end of the buffer
and notice that, for example, defun (21 lines from the bottom of the
buffer) isn't marked as misspelled.

-- 
 Best wishes,
   Slawomir Nowaczyk
     ( slawomir.nowaczyk.847@student.lu.se )

Live in the past and future only.

Re: flyspell bug

[Richard M. Stallman]

    #>     But sometimes it seems to behave strangely, like taking 10 seconds to
    #>     go from 0% to 10%, then immediately jumping to 40%, and then
    #>     immediately reporting "Spell Checking completed." And in such cases
    #>     the buffer is not checked correctly.

    open etc/refcard.tex, M-x flyspell-buffer, go to the end of the buffer
    and notice that, for example, defun (21 lines from the bottom of the
    buffer) isn't marked as misspelled.

I do see that--but I am not sure it is a bug.  When I type M-$ on
`defun', it says that word is correct.  And there was no jumping in
the percentages in the progress reports.

Can anyone else see if there is a real bug here?

Re: flyspell bug

[Piet van Oostrum]

>>>>> "Richard M. Stallman" <rms@gnu.org> (RMS) wrote:

>RMS>     open etc/refcard.tex, M-x flyspell-buffer, go to the end of the buffer
>RMS>     and notice that, for example, defun (21 lines from the bottom of the
>RMS>     buffer) isn't marked as misspelled.

>RMS> I do see that--but I am not sure it is a bug.  When I type M-$ on
>RMS> `defun', it says that word is correct.  And there was no jumping in
>RMS> the percentages in the progress reports.

>RMS> Can anyone else see if there is a real bug here?

No, I think this is the intended behavior. Flyspell is not spell-checking
the whole buffer for you, only the words that you enter, edit, or are about
to edit (i.e. that the cursor is on). It is not like in MS WORD where all
misspelled words are marked. See the Emacs manual:

       Flyspell mode is a fully-automatic way to check spelling as you edit
    in Emacs.  It operates by checking words as you change or insert them.
    When it finds a word that it does not recognize, it highlights that
    word.  This does not interfere with your editing, but when you see the
    highlighted word, you can move to it and fix it.  Type `M-x
    flyspell-mode' to enable or disable this mode in the current buffer.

-- 
Piet van Oostrum <piet@cs.uu.nl>
URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4]
Private email: piet@vanoostrum.org

Re: flyspell bug

[Richard M. Stallman]

    No, I think this is the intended behavior. Flyspell is not spell-checking
    the whole buffer for you, only the words that you enter, edit, or are about
    to edit (i.e. that the cursor is on). It is not like in MS WORD where all
    misspelled words are marked. See the Emacs manual:

That is true of Flyspell mode; but he used flyspell-buffer,
which does check the whole buffer.

I think the explanation is that "defun" is not considered a misspelling.

Re: flyspell bug

[Piet van Oostrum]

>>>>> "Richard M. Stallman" <rms@gnu.org> (RMS) wrote:

>RMS>     No, I think this is the intended behavior. Flyspell is not
>RMS>     spell-checking the whole buffer for you, only the words that you
>RMS>     enter, edit, or are about to edit (i.e. that the cursor is on).
>RMS>     It is not like in MS WORD where all misspelled words are marked.
>RMS>     See the Emacs manual:

>RMS> That is true of Flyspell mode; but he used flyspell-buffer,
>RMS> which does check the whole buffer.

You are right. I overlooked that.

>RMS> I think the explanation is that "defun" is not considered a misspelling.

When I check it, it is considered a misspelling. Maybe it is in your
personal dictionary?
-- 
Piet van Oostrum <piet@cs.uu.nl>
URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4]
Private email: piet@vanoostrum.org

Re: flyspell bug

[Piet van Oostrum]

I have found the bug.

Explanation:

flyspell-buffer (actually flyspell-region) works in two different ways,
depending on a small or large region to be checked. For a small region each
word is subjected to flyspell-word. No problem. For a large region
(default > 1000 characters) first the region is piped through ispell which
returns a list of misspelled words. Then each of these words is searched
for in the region (sequentially) and then flyspell-word is applied to it.
This searching and spelling is done in a loop until the misspelling is
found. 

In a TeX file, ispell should be called with the "-t" option and this is not
done. That is the bug. What happens is that ispell finds a misspelled word,
in the refcard.tex for example "newcount". The searching loop finds
"newcount", applies flyspell-word, but flyspell-word accepts the word
because it is in a TeX command. Therefore the loop continues until a really
misspelled "newcount" is found. In this case there is none, so the loop
stops after the second "\newcount". This looping is necessary, because (1)
there could really be a word "newcount" without the backslash, (2) the word
could be found as a substring of a correctly spelled word (e.g. when
"redist" would appear after "redistribute").

The solution is to give ispell the "-t" option.

diff -u ~/Projects/cvs/emacs/lisp/textmodes/flyspell.el flyspell.el
--- /Users/piet/Projects/cvs/emacs/lisp/textmodes/flyspell.el	2005-10-24 09:17:22.000000000 +0200
+++ flyspell.el	2005-10-24 15:38:27.000000000 +0200
@@ -1389,6 +1389,8 @@
 		      (if ispell-local-dictionary
 			  (setq ispell-dictionary ispell-local-dictionary))
 		      (setq args (ispell-get-ispell-args))
+		      (if (eq ispell-parser 'tex)
+			  (setq args (cons "-t" args)))
 		      (if ispell-dictionary ; use specified dictionary
 			  (setq args
 				(append (list "-d" ispell-dictionary) args)))

-- 
Piet van Oostrum <piet@cs.uu.nl>
URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4]
Private email: piet@vanoostrum.org

Re: flyspell bug

[Piet van Oostrum]

>>>>> Piet van Oostrum <piet@cs.uu.nl> (PvO) wrote:

>PvO> misspelled "newcount" is found. In this case there is none, so the loop
>PvO> stops after the second "\newcount". 

I should have added that now the wordlist and the buffer are out of sync,
and therefore potentially many wrong spelling can be missed.
-- 
Piet van Oostrum <piet@cs.uu.nl>
URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4]
Private email: piet@vanoostrum.org

Re: flyspell bug

[Richard M. Stallman]

    When I check it, it is considered a misspelling. Maybe it is in your
    personal dictionary?

That was indeed the reason.  Trying it with an empty personal
dictionary, I observed the bug.  This patch seems to work; does it
give good results?


*** flyspell.el	20 Oct 2005 00:27:34 -0400	1.80
--- flyspell.el	24 Oct 2005 11:50:52 -0400	
***************
*** 1333,1339 ****
      ;; Loop over incorrect words.
      (while (re-search-forward "\\([^\n]+\\)\n" (point-max) t)
        ;; Bind WORD to the next one.
!       (let ((word (match-string 1)))
  	;; Here there used to be code to see if WORD is the same
  	;; as the previous iteration, and count the number of consecutive
  	;; identical words, and the loop below would search for that many.
--- 1333,1339 ----
      ;; Loop over incorrect words.
      (while (re-search-forward "\\([^\n]+\\)\n" (point-max) t)
        ;; Bind WORD to the next one.
!       (let ((word (match-string 1)) (wordpos (point)))
  	;; Here there used to be code to see if WORD is the same
  	;; as the previous iteration, and count the number of consecutive
  	;; identical words, and the loop below would search for that many.
***************
*** 1349,1360 ****
  	;; but I didn't want to change it. -- rms.)
  	(with-current-buffer flyspell-large-region-buffer
  	  (goto-char flyspell-large-region-beg)
! 	  (let ((keep t))
! 	    (while (and keep
! 			(search-forward word flyspell-large-region-end t))
! 	      (goto-char (- (point) 1))
! 	      (setq keep (flyspell-word)))
! 	    (setq flyspell-large-region-beg (point))))))
      ;; we are done
      (if flyspell-issue-message-flag (message "Spell Checking completed.")))
    ;; Kill and forget the buffer with the list of incorrect words.
--- 1349,1362 ----
  	;; but I didn't want to change it. -- rms.)
  	(with-current-buffer flyspell-large-region-buffer
  	  (goto-char flyspell-large-region-beg)
! 	  (if (search-forward word flyspell-large-region-end t)
! 	      (progn
! 		(setq flyspell-large-region-beg (point))
! 		(goto-char (- (point) 1))
! 		;; (setq keep (flyspell-word))
! 		(flyspell-word))
! 	    (error "Bug: misspelled word `%s' (output pos %d) not found in buffer"
! 		   word wordpos)))))
      ;; we are done
      (if flyspell-issue-message-flag (message "Spell Checking completed.")))
    ;; Kill and forget the buffer with the list of incorrect words.

Re: flyspell bug

[Piet van Oostrum]

>>>>> "Richard M. Stallman" <rms@gnu.org> (RMS) wrote:

>RMS>     When I check it, it is considered a misspelling. Maybe it is in your
>RMS>     personal dictionary?

>RMS> That was indeed the reason.  Trying it with an empty personal
>RMS> dictionary, I observed the bug.  This patch seems to work; does it
>RMS> give good results?

No, not completely. For example in the following (as part of a larger text):


% GNU Emacs is free software; you can redistribute it and/or modify
% it under the redist terms of the GNU General Public License as published by
% the Free Software Foundation; either version 2, or (at your option)
% any later version.

I inserted the word "redist" in the second line, which is misspelled.
However, your code misses this word, because it finds
"redistributable" first and that is accepted by flyspell-word. The
original code continues and finds "redist" the next time in the inner
loop. However, that code can only work correctly when both ispell the
program, and ispell.el have the same ideas about TeX commands to skip.

I can even imagine that the checking loop goes out of sync with a
somewhat more contorted example.

Maybe a solution is to change the search-forward into an exact-word
search, i.e. change word into a regexp with word-boundary matches.
Then it should use the same ideas about words as ispell, and the
regexp would slow it down, unfortunately.

And now I think of it, are there no problems to be expected with
capitalized words where one form is correct and the other (e.g. the
lowercase one) is incorrect, with case-fold-search = true?
-- 
Piet van Oostrum <piet@cs.uu.nl>
URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4]
Private email: piet@vanoostrum.org

Re: flyspell bug

[Slawomir Nowaczyk]

On Mon, 24 Oct 2005 12:27:32 -0400
"Richard M. Stallman" <rms@gnu.org> wrote:

#>     When I check it, it is considered a misspelling. Maybe it is in your
#>     personal dictionary?
#> 
#>  That was indeed the reason.  Trying it with an empty personal
#>  dictionary, I observed the bug.  This patch seems to work; does it
#>  give good results?

I have tested this patch on three files which were causing me problems
before, and it seems to work OK. Thanks a lot :)

The problem Piet van Oostrum mentioned, however, is still present: the
word "redist" is not marked as misspelled in this text:
                                                            .
% redistribute redist redistribut                           .
                                                            .
Anyway, it definitely makes sense to apply your patch, but it is
probably worthwhile to investigate the matter a little bit more.

-- 
 Best wishes,
   Slawomir Nowaczyk
     ( slawomir.nowaczyk.847@student.lu.se )

Remember.... Marriage is the number one cause of divorce.
Statistically, 100% of all divorces started with marriage.

Re: flyspell bug

[Slawomir Nowaczyk]

On Mon, 24 Oct 2005 15:39:43 +0200
Piet van Oostrum <piet@cs.uu.nl> wrote:

#> The solution is to give ispell the "-t" option.

I am still analysing what you have written -- thanks for your time, by
the way -- but the above is definitely not the complete solution. I do
encounter the same/similar bug in other modes, not only in LaTeX. So I
do believe the bug must be in mode-independent code.

RMS's patch is a vast improvement, though.

-- 
 Best wishes,
   Slawomir Nowaczyk
     ( slawomir.nowaczyk.847@student.lu.se )

Diplomacy is the art of saying "nice doggy" until you can find a rock.

Re: flyspell bug

[Agustin Martin]

Piet van Oostrum <piet <at> cs.uu.nl> writes:

> 
> I have found the bug.
> 
> Explanation:
> 
> flyspell-buffer (actually flyspell-region) works in two different ways,
> depending on a small or large region to be checked. For a small region each
> word is subjected to flyspell-word. No problem. For a large region
> (default > 1000 characters) first the region is piped through ispell which
> returns a list of misspelled words. Then each of these words is searched
> for in the region (sequentially) and then flyspell-word is applied to it.
> This searching and spelling is done in a loop until the misspelling is
> found. 
> 
> In a TeX file, ispell should be called with the "-t" option and this is not
> done. That is the bug. What happens is that ispell finds a misspelled word,
> in the refcard.tex for example "newcount". The searching loop finds
> "newcount", applies flyspell-word, but flyspell-word accepts the word
> because it is in a TeX command. Therefore the loop continues until a really
> misspelled "newcount" is found. In this case there is none, so the loop
> stops after the second "\newcount". This looping is necessary, because (1)
> there could really be a word "newcount" without the backslash, (2) the word
> could be found as a substring of a correctly spelled word (e.g. when
> "redist" would appear after "redistribute").

Not sure if this will also affect, but pointing out about it, so people is aware

When a file not explicitely tex, but containing tex commands is passed to
ispell, ispell's default deformatter is nroff, so the beginning of some of
the tex commands might be considered as nroff control sequences. That will
happen for things starting by \sp and other ones. For instance

$ cat test.txt
\special
\spanish

$ cat test.txt | ispell -l -d american
ecial
anish

I do not know if this might affect the scanning mechanism for files containing
tex commands (e.g., an sgml document about something tex-related), but since
flyspell-large-region calls ispell without -t it might be.

http://bugs.debian.org/310844

Re: flyspell bug

[Richard M. Stallman]

    The solution is to give ispell the "-t" option.

I will use that.  Thanks.

Re: flyspell bug

[Richard M. Stallman]

I installed vab Oostrum's patch and a part of mine.
Things seem to work now for me.  Do they work for you?]

Re: flyspell bug

[Slawomir Nowaczyk]

On Wed, 26 Oct 2005 12:46:24 -0400
"Richard M. Stallman" <rms@gnu.org> wrote:

#> I installed vab Oostrum's patch and a part of mine. Things seem to
#> work now for me. Do they work for you?]

No, they do not. Your original patch from Mon, 24 Oct 2005 12:27:32
-0400 worked much better than current CVS does.

But I do not see any recent changes to flyspell.el in the CVS... the
last one is "revision 1.80
date: 2005/10/23 18:22:49;  author: rms;  state: Exp;  lines: +1 -0
(flyspell-large-region): Call ispell-check-version."

Is there something wrong with my CVS access? Did you put your changes
in some other file?

-- 
 Best wishes,
   Slawomir Nowaczyk
     ( slawomir.nowaczyk.847@student.lu.se )

One cannot make an omelette without breaking eggs -- but it is
amazing how many eggs one can break without making a decent omelette.

Re: flyspell bug

[Agustin Martin]

Richard M. Stallman <rms <at> gnu.org> writes:

> 
>     When I check it, it is considered a misspelling. Maybe it is in your
>     personal dictionary?
> 
> That was indeed the reason.  Trying it with an empty personal
> dictionary, I observed the bug.  This patch seems to work; does it
> give good results?

I have tested that in a sample file and seems to work the same as the old
flyspell-external-point-words function from flyspell.el-1.6h (what was present
in emacs21.4). That means that works much better than what was previously
included in emacs-cvs flyspell.el and in any further flyspell.el version
(seemed to be the same code).

The only pending problem I see is that mentioned by Piet van Oostrum.
Reading the flyspell page seems that trying to fix it was the cause for all
those strange double loops, introduced in flyspell.el 1.6i and 1.6j (from
Sep 2002). Unfortunately seems that the fix did more harm than good, and never
really worked. I am surprised that this has been unnoticed for this long time
(since 2002!!)

I am adding the relevant flyspell changelog entries,

---------------------------------------------------------------------------
Release 1.6j, Fri Sep 20 08:52:00 CEST 2002 (Pasi Ryhanen, Manuel Serrano):
      *** The fix of 1.6i was actually incorrect. It has thus been stripped
      off. The new fix iterate over a search word until an actual match
      is found.      

Release 1.6i, Thu Sep 19 10:23:12 CEST 2002 (Pasi Ryhanen, Manuel Serrano):

      *** Improves the performance of flyspell-buffer. The spell checking
        of the whole buffer is now much faster (many thanks to Pasi).

        *** Fix a bug in buffer checking. The search for incorrect words
        was incorrect because flyspell.el was not searching "words" but
        "strings". The consequence was that if the incorrect word appeared
        fist, as a prefix of a correct word, it was not highlighted at the
        location it was misspelled.
----------------------------------------------------------------------------

Re: flyspell bug

[Agustin Martin]

Richard M. Stallman <rms <at> gnu.org> writes:

> 
> I installed vab Oostrum's patch and a part of mine.
> Things seem to work now for me.  Do they work for you?]
> 

I did not notice that there was fresh stuff at the CVS. While testing it I
am finding some problems (flyspell-large-region customized to 1, so is
always used by flyspell-{buffer,region}). In the file

--------------------------------------------------------

local.h

distribution dist distr

Local Variables:
ispell-local-dictionary: "francais"
End:
--------------------------------------------------------

I get on flyspell-buffer command


Checking region...
Spell Checking...15% [local.h]
if: Bug: misspelled word `local.h' (output pos 9) not found in buffer

Re: flyspell bug

[Richard M. Stallman]

It turns out I didn't install the change then.  Please try again now.

Re: flyspell bug

[Piet van Oostrum]

>>>>> Agustin Martin <agustin.martin@hispalinux.es> (AM) wrote:

>AM> I did not notice that there was fresh stuff at the CVS. While testing it I
>AM> am finding some problems (flyspell-large-region customized to 1, so is
>AM> always used by flyspell-{buffer,region}). In the file

>AM> --------------------------------------------------------

>AM> local.h

>AM> distribution dist distr

>AM> Local Variables:
>AM> ispell-local-dictionary: "francais"
>AM> End:
>AM> --------------------------------------------------------

>AM> I get on flyspell-buffer command


>AM> Checking region...
>AM> Spell Checking...15% [local.h]
>AM> if: Bug: misspelled word `local.h' (output pos 9) not found in buffer

For me it works (with flyspell-large-region set to 1). Both with ispell and
with aspell. And todays CVS version.
-- 
Piet van Oostrum <piet@cs.uu.nl>
URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4]
Private email: piet@vanoostrum.org

Re: flyspell bug

[Slawomir Nowaczyk]

On Fri, 28 Oct 2005 13:32:05 +0000 (UTC)
Agustin Martin <agustin.martin@hispalinux.es> wrote:

#> Richard M. Stallman <rms <at> gnu.org> writes:
#> 
#> > I installed vab Oostrum's patch and a part of mine.
#> > Things seem to work now for me.  Do they work for you?]

Not really. Again, your original patch from Mon, 24 Oct 2005 12:27:32
-0400 worked much better than current CVS does.

#> I did not notice that there was fresh stuff at the CVS. While testing it I
#> am finding some problems (flyspell-large-region customized to 1, so is
#> always used by flyspell-{buffer,region}). In the file

<snip>

#> Checking region...
#> Spell Checking...15% [local.h]
#> if: Bug: misspelled word `local.h' (output pos 9) not found in buffer

Your example works just fine for me, but I can see the same error
message on some of my files... I am unable to find a short example
reproducing it at the moment, though -- will try tomorrow.

-- 
 Best wishes,
   Slawomir Nowaczyk
     ( slawomir.nowaczyk.847@student.lu.se )

A seminar on Time Travel will be held two weeks ago.

Re: flyspell bug

[Richard M. Stallman]

    local.h

    distribution dist distr

    Local Variables:
    ispell-local-dictionary: "francais"
    End:

I can't test this, because I don't have a French dictionary.
I hope someone else can debug it, if indeed it fails with the
latest code.

Re: flyspell bug

[Piet van Oostrum]

>>>>> "Richard M. Stallman" <rms@gnu.org> (RMS) wrote:

>RMS>     local.h

>RMS>     distribution dist distr

>RMS>     Local Variables:
>RMS>     ispell-local-dictionary: "francais"
>RMS>     End:

>RMS> I can't test this, because I don't have a French dictionary.
>RMS> I hope someone else can debug it, if indeed it fails with the
>RMS> latest code.

Well, I  was too fast saying that it worked for me: I also didn't have a
french dictionary, I don't write french too often. Moreover I copied the
text in a buffer, so the local dictionary wasn't obeyed.

I installed the french dictionary and found the bug.

The problem is that ispell considers "local.h" an incorrect word, but
flyspell-word (and ispell-word) when checking consider "h" the word in
question and consider it good. The reason is that ispell.el and ispell the
program don't agree on what's a word. In francais.aff the boundarychars are
[-'.@] and in the ispell-dictionary-alist only [-']. Therefore ispell the
program considers "local.h" one word and ispell.el considers it two words.
I don't know why "." and "@" are included in the affix file (especially "@"
seems strange) but I guess a French person might explain it. Anyway it
helps to add "." and "@" to otherchars in the ispell-dictionary-alist entry
for francais. By the way, I don't think aspell considers "@" to be a
boundarychar.


-- 
Piet van Oostrum <piet@cs.uu.nl>
URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4]
Private email: piet@vanoostrum.org

Re: flyspell bug

[David Kastrup]

Piet van Oostrum <piet@cs.uu.nl> writes:

> I installed the french dictionary and found the bug.
>
> The problem is that ispell considers "local.h" an incorrect word,
> but flyspell-word (and ispell-word) when checking consider "h" the
> word in question and consider it good. The reason is that ispell.el
> and ispell the program don't agree on what's a word. In francais.aff
> the boundarychars are [-'.@] and in the ispell-dictionary-alist only
> [-']. Therefore ispell the program considers "local.h" one word and
> ispell.el considers it two words.  I don't know why "." and "@" are
> included in the affix file (especially "@" seems strange) but I
> guess a French person might explain it. Anyway it helps to add "."
> and "@" to otherchars in the ispell-dictionary-alist entry for
> francais. By the way, I don't think aspell considers "@" to be a
> boundarychar.

That does not sound like a robust solution, and it will change along
with languages.  There is no good way to make mostly irrelevant
whether Emacs and flyspell don't quite agree on word boundaries?

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: flyspell bug

[Piet van Oostrum]

>>>>> David Kastrup <dak@gnu.org> (DK) wrote:

>DK> That does not sound like a robust solution, and it will change along
>DK> with languages.  There is no good way to make mostly irrelevant
>DK> whether Emacs and flyspell don't quite agree on word boundaries?

The boundarychars are defined for each language separately. In emacs in
ispell-dictionary-alist which has an entry for each language, and in ispell
the program in the affix file. The two should agree, otherwise you may get
problems, not only with flyspell, but also with ispell itself. For some
reason there are different values in emacs and ispell the program for
French. Of course having the same information in two places is not a
recommended practice in computer science. It would be better if for example
emacs could query ispell for the required parameters, but ispell doesn't
offer that functionality AFAIK.
-- 
Piet van Oostrum <piet@cs.uu.nl>
URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4]
Private email: piet@vanoostrum.org

Re: flyspell bug

[Richard M. Stallman]

    Your example works just fine for me, but I can see the same error
    message on some of my files... I am unable to find a short example
    reproducing it at the moment, though -- will try tomorrow.

The example does not HAVE to be short.  A shorter one
will be less work to debug, but if you can't produce a short one,
please send a long one.

Re: flyspell bug

[Richard M. Stallman]

    question and consider it good. The reason is that ispell.el and ispell the
    program don't agree on what's a word. In francais.aff the boundarychars are
    [-'.@] and in the ispell-dictionary-alist only [-']. Therefore ispell the
    program considers "local.h" one word and ispell.el considers it two words.
    I don't know why "." and "@" are included in the affix file (especially "@"
    seems strange) but I guess a French person might explain it. Anyway it
    helps to add "." and "@" to otherchars in the ispell-dictionary-alist entry
    for francais.

It is not obvious to me how to do that.  Could you send the patch that
you are proposing?

One possible problem occurs to me: would adding . to this
cause incorrect behavior at the end of nearly every sentence?

Meanwhile, does this patch fix the problem?  It tries to detect
such cases, and recognize that the word has been found but can't
really be checked.

*** flyspell.el	25 Oct 2005 09:08:25 -0400	1.81
--- flyspell.el	29 Oct 2005 12:06:29 -0400	
***************
*** 1356,1362 ****
  		  (progn
  		    (setq flyspell-large-region-beg (point))
  		    (goto-char (- (point) 1))
! 		    (setq keep (flyspell-word)))
  		(error "Bug: misspelled word `%s' (output pos %d) not found in buffer"
  		       word wordpos)))))))
      ;; we are done
--- 1356,1365 ----
  		  (progn
  		    (setq flyspell-large-region-beg (point))
  		    (goto-char (- (point) 1))
! 		    (if (< (length (flyspell-get-word following))
! 			   (length word))
! 			(setq keep nil)
! 		      (setq keep (flyspell-word))))
  		(error "Bug: misspelled word `%s' (output pos %d) not found in buffer"
  		       word wordpos)))))))
      ;; we are done

Re: flyspell bug

[Agustin Martin]

On Sat, Oct 29, 2005 at 01:13:30AM -0400, Richard M. Stallman wrote:
> 
>     local.h
> 
>     distribution dist distr
> 
>     Local Variables:
>     ispell-local-dictionary: "francais"
>     End:
> 
> I can't test this, because I don't have a French dictionary.
> I hope someone else can debug it, if indeed it fails with the
> latest code.

I think the problem is that french ispell dict (Hydro Quebec version)
considers dot as a boundary char, so finds "local.h" as a single
mispelled word, and this seems to not match emacs concept of word. In the
aff file that is suggested to be appplied to network names.

Browsing my /usr/lib/ispell dict, seems that the same should happen for
norwegian nynorsk and bokmal, italian and polish, all them having dot as a
boundary char, and indeed I get the same error.

Other languages seem to consider local.h as two words and error does not
appear.

I am in low bandwith for some days, so I cannot check that for other dicts
like the other french ispell dict (gutenberg version) or the aspell-fr or
any other ispell or aspell dict.

-- 
Agustin

Re: flyspell bug

[Piet van Oostrum]

>>>>> Agustin Martin <agustin.martin@hispalinux.es> (AM) wrote:

>AM> I think the problem is that french ispell dict (Hydro Quebec version)
>AM> considers dot as a boundary char, so finds "local.h" as a single
>AM> mispelled word, and this seems to not match emacs concept of word. In the
>AM> aff file that is suggested to be appplied to network names.

The French dictionary contains "." in abbreviations, like "i.e". I haven't
found any @ in the dictionary, so I find it strange that it is defined as a
boundarychar.
>>>>> Agustin Martin <agustin.martin@hispalinux.es> (AM) wrote:

>AM> Browsing my /usr/lib/ispell dict, seems that the same should happen for
>AM> norwegian nynorsk and bokmal, italian and polish, all them having dot as a
>AM> boundary char, and indeed I get the same error.

I haven't checked these others (I don't have them, and I have no need for them).
-- 
Piet van Oostrum <piet@cs.uu.nl>
URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4]
Private email: piet@vanoostrum.org

Re: flyspell bug

[Piet van Oostrum]

>>>>> "Richard M. Stallman" <rms@gnu.org> (RMS) wrote:

>RMS> It is not obvious to me how to do that.  Could you send the patch that
>RMS> you are proposing?

,----
| --- /Users/piet/Projects/cvs/emacs/lisp/textmodes/ispell.el.~1~	2005-10-24 09:17:22.000000000 +0200
| +++ /Users/piet/Projects/cvs/emacs/lisp/textmodes/ispell.el	2005-10-30 09:35:28.000000000 +0100
| @@ -579,7 +579,7 @@
|     ("francais"				; Francais.aff
|      "[A-Za-z\300\302\306\307\310\311\312\313\316\317\324\331\333\334\340\342\347\350\351\352\353\356\357\364\371\373\374]"
|      "[^A-Za-z\300\302\306\307\310\311\312\313\316\317\324\331\333\334\340\342\347\350\351\352\353\356\357\364\371\373\374]"
| -    "[-']" t nil "~list" iso-8859-1)
| +    "[-'.@]" t nil "~list" iso-8859-1)
|     ("francais-tex"			; Francais.aff
|      "[A-Za-z\300\302\306\307\310\311\312\313\316\317\324\331\333\334\340\342\347\350\351\352\353\356\357\364\371\373\374\\]"
|      "[^A-Za-z\300\302\306\307\310\311\312\313\316\317\324\331\333\334\340\342\347\350\351\352\353\356\357\364\371\373\374\\]"
`----

>RMS> One possible problem occurs to me: would adding . to this
>RMS> cause incorrect behavior at the end of nearly every sentence?

No. The `boundarychars' are only accepted between letters of the word, not
at the ends.

>RMS> Meanwhile, does this patch fix the problem?  It tries to detect
>RMS> such cases, and recognize that the word has been found but can't
>RMS> really be checked.

>RMS> *** flyspell.el	25 Oct 2005 09:08:25 -0400	1.81
>RMS> --- flyspell.el	29 Oct 2005 12:06:29 -0400	
>RMS> ***************
>RMS> *** 1356,1362 ****
>RMS>   		  (progn
>RMS>   		    (setq flyspell-large-region-beg (point))
>RMS>   		    (goto-char (- (point) 1))
>RMS> ! 		    (setq keep (flyspell-word)))
>RMS>   		(error "Bug: misspelled word `%s' (output pos %d) not found in buffer"
>RMS>   		       word wordpos)))))))
>RMS>       ;; we are done
>RMS> --- 1356,1365 ----
>RMS>   		  (progn
>RMS>   		    (setq flyspell-large-region-beg (point))
>RMS>   		    (goto-char (- (point) 1))
>RMS> ! 		    (if (< (length (flyspell-get-word following))
That should be:             (if (< (length (car (flyspell-get-word nil)))
flyspell-word returns a triple (word startpos endpos)
>RMS> ! 			   (length word))
>RMS> ! 			(setq keep nil)
>RMS> ! 		      (setq keep (flyspell-word))))
>RMS>   		(error "Bug: misspelled word `%s' (output pos %d) not found in buffer"
>RMS>   		       word wordpos)))))))
>RMS>       ;; we are done

You can also factor out the "setq keep":

		    (setq keep
			  (if (< (length (car (flyspell-get-word nil)))
				 (length word))
			      nil
			    (flyspell-word))))

It solves the problem at hand. I can't think of any bad effects this would
have otherwise than missing some misspellings with a "." or "@" in it. I
wouldn't care much for the "@" because I think it shouldn't be there.

Anyhow, the ispell-dictionary-alist should be made compatible with the
definitions in the dictionaries.

-- 
Piet van Oostrum <piet@cs.uu.nl>
URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4]
Private email: piet@vanoostrum.org

Re: flyspell bug

[Richard M. Stallman]

I fixed the boundarychars for both forms of French, and added period
for Italian and Polish.  ispell.el doesn't have elements for nynorsk
and bokmal, and I didn't do anything about the ones for norsk.  (I'm
guessing that is really samnorsk.)

Would someone else like to check for any further corrections needed?

Re: flyspell bug

[Slawomir Nowaczyk]

On Sat, 29 Oct 2005 16:33:48 -0400
"Richard M. Stallman" <rms@gnu.org> wrote:

#>     Your example works just fine for me, but I can see the same error
#>     message on some of my files... I am unable to find a short example
#>     reproducing it at the moment, though -- will try tomorrow.
#> 
#> The example does not HAVE to be short. A shorter one will be less
#> work to debug, but if you can't produce a short one, please send a
#> long one.

One problem I see right now is with LocalWords -- when there are any
local words for a buffer, I get (error "Bug: misspelled word `blabla'
(output pos 36) not found in buffer")

For example, run the below in this buffer:

(setq flyspell-large-region 20)
(flyspell-buffer)

If you remove the line below, it still doesn't work. But if you remove
the line, kill the buffer and revisit it, everything will work fine.

 LocalWords:  blabla

-- 
 Best wishes,
   Slawomir Nowaczyk
     ( slawomir.nowaczyk.847@student.lu.se )

Have no fear of perfection - you'll never reach it  (S. Dali)

Re: flyspell bug

[Piet van Oostrum]

There is a bug in the current cvs version of flyspell.el (there is no
variable "following" in the function).

*** flyspell.el	31 Oct 2005 16:31:52 +0100	1.82
--- flyspell.el	01 Nov 2005 11:17:52 +0100	
***************
*** 1360,1366 ****
  			  ;; Detect when WORD can't be checked properly
  			  ;; because flyspell-get-word finds
  			  ;; just part of it, and treat that as ok.
! 			  (if (< (length (flyspell-get-word following))
  				 (length word))
  			      nil
  			    (flyspell-word))))
--- 1360,1366 ----
  			  ;; Detect when WORD can't be checked properly
  			  ;; because flyspell-get-word finds
  			  ;; just part of it, and treat that as ok.
! 			  (if (< (length (flyspell-get-word nil))
  				 (length word))
  			      nil
  			    (flyspell-word))))

-- 
Piet van Oostrum <piet@cs.uu.nl>
URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4]
Private email: piet@vanoostrum.org

Re: flyspell bug

[Slawomir Nowaczyk]

On Tue, 01 Nov 2005 11:19:32 +0100
Piet van Oostrum <piet@cs.uu.nl> wrote:

#> There is a bug in the current cvs version of flyspell.el (there is no
#> variable "following" in the function).

Thanks, it works better now... however, flyspell-buffer still doesn't
check all the words. In fact, it seems to miss most of them -- try the
following in this buffer:

(setq flyspell-large-region 100)
(flyspell-buffer)

This is a siimple tesst to showw how mostt words are misssed by
flyspell-buffer... not a single word is marked as misspelled in this
paragraph.

-- 
 Best wishes,
   Slawomir Nowaczyk
     ( slawomir.nowaczyk.847@student.lu.se )

Don't wake me for the end of the world unless it has very good special effects -- Roger Zelazny

Re: flyspell bug

[Richard M. Stallman]

Thanks.

Re: flyspell bug

[Piet van Oostrum]

>>>>> Slawomir Nowaczyk <slawomir.nowaczyk.847@student.lu.se> (SN) wrote:

>SN> Thanks, it works better now... however, flyspell-buffer still doesn't
>SN> check all the words. In fact, it seems to miss most of them -- try the
>SN> following in this buffer:

For me it works, with both ispell and aspell (english dictionary).
See http://www.cs.uu.nl/~piet/flyspell.png
-- 
Piet van Oostrum <piet@cs.uu.nl>
URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4]
Private email: piet@vanoostrum.org

Re: flyspell bug

[Slawomir Nowaczyk]

On Wed, 02 Nov 2005 22:30:52 +0100
Piet van Oostrum <piet@cs.uu.nl> wrote:

#> >>>>> Slawomir Nowaczyk <slawomir.nowaczyk.847@student.lu.se> (SN) wrote:
#> 
#> >SN> Thanks, it works better now... however, flyspell-buffer still doesn't
#> >SN> check all the words. In fact, it seems to miss most of them -- try the
#> >SN> following in this buffer:
#> 
#> For me it works, with both ispell and aspell (english dictionary).
#> See http://www.cs.uu.nl/~piet/flyspell.png

Well, that's strange... for me it doesn't:
http://www.cs.lth.se/~slawek/flyspell.jpg

running "emacs --no-init-file --no-site-file" on Windows (I don't see
how it could be OS-related, but maybe?), English dictionary and
ispell.

-- 
 Best wishes,
   Slawomir Nowaczyk
     ( slawomir.nowaczyk.847@student.lu.se )

To err is human, but to really fuck things up takes a computer.

Re: flyspell bug

[Piet van Oostrum]

>>>>> Slawomir Nowaczyk <slawomir.nowaczyk.847@student.lu.se> (SN) wrote:

>SN> Well, that's strange... for me it doesn't:
>SN> http://www.cs.lth.se/~slawek/flyspell.jpg

>SN> running "emacs --no-init-file --no-site-file" on Windows (I don't see
>SN> how it could be OS-related, but maybe?), English dictionary and
>SN> ispell.

I found that I used my own code rather than the CVS code. I forgot a part
of my patch. Here is the rest.

diff -u /Users/piet/Projects/cvs/emacs/lisp/textmodes/flyspell.el.\~1.84.\~ /Users/piet/Projects/cvs/emacs/lisp/textmodes/flyspell.el
--- /Users/piet/Projects/cvs/emacs/lisp/textmodes/flyspell.el.~1.84.~	2005-11-04 10:13:37.000000000 +0100
+++ /Users/piet/Projects/cvs/emacs/lisp/textmodes/flyspell.el	2005-11-04 10:16:39.000000000 +0100
@@ -1362,7 +1362,7 @@
 			  ;; Detect when WORD can't be checked properly
 			  ;; because flyspell-get-word finds
 			  ;; just part of it, and treat that as ok.
-			  (if (< (length (flyspell-get-word nil))
+			  (if (< (length (car (flyspell-get-word nil)))
 				 (length word))
 			      nil
 			    (flyspell-word))))


-- 
Piet van Oostrum <piet@cs.uu.nl>
URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4]
Private email: piet@vanoostrum.org

Re: flyspell bug

[Agustin Martin]

Piet van Oostrum writes:

> 
> >>>>> Slawomir Nowaczyk (SN) wrote:
> 
> >SN> Well, that's strange... for me it doesn't:
> >SN> http://www.cs.lth.se/~slawek/flyspell.jpg
> 
> >SN> running "emacs --no-init-file --no-site-file" on Windows (I don't see
> >SN> how it could be OS-related, but maybe?), English dictionary and
> >SN> ispell.
> 
> I found that I used my own code rather than the CVS code. I forgot a part
> of my patch. Here is the rest.

While not yet extensively tested, this seems to work much better in the files I
tried, thanks

Re: flyspell bug

[Slawomir Nowaczyk]

On Fri, 04 Nov 2005 10:20:50 +0100
Piet van Oostrum <piet@cs.uu.nl> wrote:

#> I found that I used my own code rather than the CVS code. I forgot a part
#> of my patch. Here is the rest.

Thanks, it works much better now. However, I still have occasional
problems with flyspell, mainly when using Local-Words... for example,
run flyspell-buffer in this buffer.

This is a test. blablabla is buffer local word, so it shouldn't be a
problem... but it apparently causes some missspelled word not to be
detected correctly. blablabla's fault, apparently.

I get an error: "misspelled word `missspelled' (output pos 96) not
found in buffer". Observe that removing Local-Words definition fixes
the problem, as does removing the "blablabla" OR "blablabla's" from
the paragraph above (not completely, as blablabla in the Local-Words
definition itself causes problems, but at least it doesn't complain
about misspelled any more).

 LocalWords:  blablabla

-- 
 Best wishes,
   Slawomir Nowaczyk
     ( slawomir.nowaczyk.847@student.lu.se )

The true opposite of love is indifference.

Re: flyspell bug

[Agustin Martin]

Slawomir Nowaczyk writes:
> 
> Piet van Oostrum wrote:
> 
> #> I found that I used my own code rather than the CVS code. I forgot a part
> #> of my patch. Here is the rest.
> 
> Thanks, it works much better now. However, I still have occasional
> problems with flyspell, mainly when using Local-Words... for example,
> run flyspell-buffer in this buffer.

[Note: tests below with only the flyspell-external-point-words function
changed, but with the rest of flyspell.el that of emacs 21.4]

Strange, seems to work for me with that buffer. But I am also getting
ocasional problems. They seem to be really obscure, e.g. in the file containing

------------------------------ snip ------------------------------------
<!DOCTYPE linuxdoc SYSTEM>
  <article>
      <title>A dummy text file</title>
      <author>Anonymous
      <date> today
	
      <abstract>
        text text text text text text text text text 
        text text text text text text text text text 
        text text text text text text text text text 
        text text text text text text text text text 
        text text text text text text text text text 
        text text text text text text text text text 
        text text text text text text text text text 
        text text text text text text text text text 
        text text text text text text text text text 
        text text text text text text text text text 
        text text text text text text text text text
        text text text text text text text text text
        text text text text text text text text text
        text text text text text text text text text
        text text text text text text text text text
      </abstract>
</article>

Local Variables:
ispell-local-dictionary: "american"
End:
------------------------------ snip ------------------------------------

if I run flyspell-buffer in sgml-mode (psgml) I get

flyspell-external-point-words:
Bug: misspelled word `DOCTYPE' (output pos 9) not found in buffer

but the funny thing is that if I do it in text mode seems to work as
expected with no error message. 

Also, in sgml mode, if I remove one of the "text" lines, I get no error message
at all, but mispelling marks appear over some of the 'text' strings, but not on
others.

Re: flyspell bug

[Richard M. Stallman]

    +			  (if (< (length (car (flyspell-get-word nil)))

Thanks.

Re: flyspell bug

[Richard M. Stallman]

    For example, run the below in this buffer:

    (setq flyspell-large-region 20)
    (flyspell-buffer)

    If you remove the line below, it still doesn't work. But if you remove
    the line, kill the buffer and revisit it, everything will work fine.

     LocalWords:  blabla

The bug in this case is that the LocalWords are not known to the
ispell process that is used by flyspell-large-region.  It clearly
needs the change below.

With that change, ispell-buffer-local-words gets called at the right
time, finds blabla, and sends it to the ispell process.
ispell-kill-ispell is not called again before flyspell-large-region
sends the text to the process.  Despite that, blabla is still included
in the list of misspelled words in the buffer.

I am stumped now.  Ken, can you help?


*** flyspell.el	04 Nov 2005 17:32:58 -0500	1.85
--- flyspell.el	06 Nov 2005 21:29:35 -0500	
***************
*** 1384,1389 ****
--- 1384,1390 ----
      (setq flyspell-large-region-buffer curbuf)
      (setq flyspell-large-region-beg beg)
      (setq flyspell-large-region-end end)
+     (flyspell-accept-buffer-local-defs)
      (set-buffer buffer)
      (erase-buffer)
      ;; this is done, we can start checking...

Re: flyspell bug

[Slawomir Nowaczyk]

OK, I finally had some time (not enough, though) to debug the problem,
and here is what I have found.

First of all, there is problem with flyspell-buffer not taking into
account the LocalWords: definition (the list of words assumed to be
correct in particular buffer). Words from this list are reported in
flyspell-external-ispell-buffer by ispell, but are not considered to
be misspelled by flyspell-word. This causes the logic in
flyspell-external-point-words to get out of sync with the buffer
contents.

I do not know how to fix this, though. There is
ispell-buffer-local-words in ispell.el, but I do not know enough about
ispell to understand what it is doing, or even if this is a right
place to start.

As an ugly kludge, the following patch does seem to fix things for me:

*** M:\EmacsCVS\EmacsCVS\lisp\textmodes\flyspell.el     Mon Nov  7 15:00:13 2005
--- c:\Emacs\lisp\textmodes\flyspell.el Mon Nov  7 16:39:45 2005
***************
*** 1365,1371 ****
                          (if (< (length (car (flyspell-get-word nil)))
                                 (length word))
                              nil
!                           (flyspell-word))))
                (error "Bug: misspelled word `%s' (output pos %d) not found in buffer"
                       word wordpos)))))))
      ;; we are done
--- 1365,1374 ----
                          (if (< (length (car (flyspell-get-word nil)))
                                 (length word))
                              nil
!                             (if (= (length (car (flyspell-get-word nil)))
!                                    (length word))
!                                 (and (flyspell-word) nil)  ;; flyspell-word but exit the loop unconditionally
!                               (flyspell-word)))))
                (error "Bug: misspelled word `%s' (output pos %d) not found in buffer"
                       word wordpos)))))))
      ;; we are done
***************

But I would prefer to have LocalWords taken care of by ispell -- if
only due to the fact that it should make flyspell-buffer significantly
faster.

Besides that, I believe the logic in flyspell-external-point-words
should be simplified significantly -- we should get rid of the "while
keep" loop, just find the first place where
     			  (if (= (length (flyspell-get-word nil))
				 (length word))
and flyspell-word that. And consider cases where ispell and flyspell
have different ideas about word boundaries to be bugs and fix those.

-- 
 Best wishes,
   Slawomir Nowaczyk
     ( slawomir.nowaczyk.847@student.lu.se )

A computer DOES save time at work. I can play solitaire without
having to spend all that time shuffling real cards.

Re: flyspell bug

[Agustin Martin]

Slawomir Nowaczyk <slawomir.nowaczyk.847 <at> student.lu.se> writes:

> But I would prefer to have LocalWords taken care of by ispell -- if
> only due to the fact that it should make flyspell-buffer significantly
> faster.

This should already work with the new localwords patch.

> 
> Besides that, I believe the logic in flyspell-external-point-words
> should be simplified significantly -- we should get rid of the "while
> keep" loop, just find the first place where
>      			  (if (= (length (flyspell-get-word nil))
> 				 (length word))
> and flyspell-word that.

I think you need the (while keep) loop for this. 

> And consider cases where ispell and flyspell
> have different ideas about word boundaries to be bugs and fix those.

Some of these might require a not found word be signalled by something softer
than an error. 

I am also taking a look at this function making keep nil if any of some
conditions  is matched. Will send something as soon as looks better (if so),
even if still full of debugging code

-- 
Agustin

Re: flyspell bug

[Agustin Martin]

[-- Attachment #1: Type: text/plain, Size: 2606 bytes --]

On Wed, Nov 16, 2005 at 03:22:42PM +0000, Agustin Martin wrote:
> Slawomir Nowaczyk <slawomir.nowaczyk.847 <at> student.lu.se> writes:
> > 
> > Besides that, I believe the logic in flyspell-external-point-words
> > should be simplified significantly -- we should get rid of the "while
> > keep" loop, just find the first place where
> >      			  (if (= (length (flyspell-get-word nil))
> > 				 (length word))
> > and flyspell-word that.
> 
> I think you need the (while keep) loop for this. 
> 
> > And consider cases where ispell and flyspell
> > have different ideas about word boundaries to be bugs and fix those.
> 
> Some of these might require a not found word be signalled by something softer
> than an error. 
> 
> I am also taking a look at this function making keep nil if any of some
> conditions  is matched. Will send something as soon as looks better (if so),
> even if still full of debugging code

I am attaching the function as currently looks in my system. It is still
plenty of debugging code and could surely be simplified, but seems to work
reasonably. However much more test is needed.

The basic check is the one you proposed, check lengths (seems to suffice,
but this might be raised to a 'string=' comparison if needed), but some
other checks are added, mostly for ispell benefit:

   (member word
       (split-string flyword-prev ispell-otherchars))

will try to see if word has been found as part of a boundary-chars
separated compound word. If so, skip word and move point to the match
point, not to the end of the flyspell-get-word word found, just in case
next element in that compound word is also misspelled (we would lose it
otherwise and might unsync the process).

  (and (not ispell-really-aspell)
   (save-excursion
      (goto-char (- (nth 1 flyword-prev-l) 1))
	      (if (looking-at "[\\]" )
		  (progn
      			(when mydebug (message "[%s] %s has nroff problems in \\%s" 
					(match-string 0) word flyword-prev))
			 t)
			nil
		))

ispell (not aspell) works in nroff mode as default, for that reason if a
tex string is in a non tex file, it might get stripped of the first chars,
because they can match a nroff command,

\special -> ecial

thus making the mispelled word 'ecial' not be found in buffer, or unsync the
process if 'ecial' would be a good word present later.

If any of these tests matches, word is considered as found and
(flyspell-word) is run on it.

I still find some problems in tex mode with the '-t' option introduced in
flyspell-large-region, but this might be independent of this change. I still
have to look better at this.

-- 
Agustin

[-- Attachment #2: flyspell-external-point-words.debug.el --]
[-- Type: text/plain, Size: 3933 bytes --]


(defun flyspell-external-point-words ()
  "Mark words from a buffer listing incorrect words in order of appearance.
The list of incorrect words should be in `flyspell-external-ispell-buffer'.
\(We finish by killing that buffer and setting the variable to nil.)
The buffer to mark them in is `flyspell-large-region-buffer'."
  
  (let (words-not-found
	(ispell-otherchars (ispell-get-otherchars))
	(mydebug nil))    
    (with-current-buffer flyspell-external-ispell-buffer
      (goto-char (point-min))
      ;; Loop over incorrect words.
      (while (re-search-forward "\\([^\n]+\\)\n" (point-max) t)
	;; Bind WORD to the next one.
	(let ((word (match-string 1)) (wordpos (point)))
	  ;; Here there used to be code to see if WORD is the same
	  ;; as the previous iteration, and count the number of consecutive
	  ;; identical words, and the loop below would search for that many.
	  ;; That code seemed to be incorrect, and on principle, should
	  ;; be unnecessary too. -- rms.
	  (if flyspell-issue-message-flag
	      (message "Spell Checking...%d%% [%s]"
		       (* 100 (/ (float (point)) (point-max)))
		       word))
	  ;; Search the other buffer for occurrences of this word,
	  ;; and check them.  Stop when we find one that reports "incorrect".
	  ;; (I don't understand the reason for that logic,
	  ;; but I didn't want to change it. -- rms.)
	  (with-current-buffer flyspell-large-region-buffer
	    (goto-char flyspell-large-region-beg)
	    (let ((keep t))
	      ;; Iterate on string search until string is found as word,
	      ;; not as substring, 
	      (while keep
		(if (search-forward word
				    flyspell-large-region-end t)
		    (progn
		      (goto-char (- (point) 1))
		      (let* ((match-point (point)) ; flyspell-get-word might move it
			     (flyword-prev-l (flyspell-get-word nil))
			     (flyword-endpoint (point))
			     (flyword-prev (car flyword-prev-l)) 
			     (size-match (= (length flyword-prev) (length word)))
			     )
			
			(when mydebug ;; --- debugging code starts
			  (message "word[%s], flyword[%s], wp[%d] flrb[%d] p[%d]"
				   word
				   flyword-prev
				   wordpos
				   flyspell-large-region-beg
				   (point))
			  (when (not size-match)
			    (message "Size mismatch f:[%s] m:[%s] %s"
				     flyword-prev word flyword-prev-l))
			  ) ;; --- debugging code ends
			(when (or
			       size-match
			       (member word
				       (split-string flyword-prev ispell-otherchars))
			       (and (not ispell-really-aspell)
				    (save-excursion
				      (goto-char (- (nth 1 flyword-prev-l) 1))
				      (if (looking-at "[\\]" )
					  (progn
					    (when mydebug (message "[%s] %s has nroff problems in \\%s" 
								   (match-string 0) word flyword-prev))
					    t)
					nil
					))
				    )
			       ) ; end-or
			  (setq keep nil)
			  (flyspell-word)
			  (setq flyspell-large-region-beg
				(if size-match flyword-endpoint match-point))
			  ;; ---------------------------
			  (when mydebug
			    (if size-match
				(message "+ Moving to %d (match on %d)"
					 flyword-endpoint match-point)
			      (message "+ Moving to %d (point on %d)"
				       match-point flyword-endpoint)))
			  ;; ------------------------------
			  ) ; end-when
			) ; end-let
		      ) ; end-progn
		  (when mydebug 
		    (error "Error for word [%s] on flrb[%d] p[%d]"
			   word flyspell-large-region-beg (point)))
		  (add-to-list 'words-not-found
			       (concat 
				" -> "
				word
				" - "
				(int-to-string wordpos)))
		  (setq keep nil)
		  ))))))
      ;; we are done
      (if flyspell-issue-message-flag (message "Spell Checking completed.")))
    (dolist (word words-not-found)
      (message "%s: word not found" word))
    ;;) ; let keep
    ;; Kill and forget the buffer with the list of incorrect words.
    (when (not mydebug)
      (kill-buffer flyspell-external-ispell-buffer)
      (setq flyspell-external-ispell-buffer nil))
    )
  ); let

[-- Attachment #3: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: flyspell bug

[Agustin Martin]

Piet van Oostrum <piet <at> cs.uu.nl> writes:

> 
> I have found the bug.
> 
> Explanation:
...
> 
> In a TeX file, ispell should be called with the "-t" option and this is not
> done. That is the bug. What happens is that ispell finds a misspelled word,
> in the refcard.tex for example "newcount". The searching loop finds
> "newcount", applies flyspell-word, but flyspell-word accepts the word
> because it is in a TeX command. Therefore the loop continues until a really
> misspelled "newcount" is found. In this case there is none, so the loop
> stops after the second "\newcount". This looping is necessary, because (1)
> there could really be a word "newcount" without the backslash, (2) the word
> could be found as a substring of a correctly spelled word (e.g. when
> "redist" would appear after "redistribute").
> 
> The solution is to give ispell the "-t" option.
> 
> diff -u ~/Projects/cvs/emacs/lisp/textmodes/flyspell.el flyspell.el
> --- /Users/piet/Projects/cvs/emacs/lisp/textmodes/flyspell.el	2005-10-24
09:17:22.000000000 +0200
> +++ flyspell.el	2005-10-24 15:38:27.000000000 +0200
>  <at>  <at>  -1389,6 +1389,8  <at>  <at> 
>  		      (if ispell-local-dictionary
>  			  (setq ispell-dictionary ispell-local-dictionary))
>  		      (setq args (ispell-get-ispell-args))
> +		      (if (eq ispell-parser 'tex)
> +			  (setq args (cons "-t" args)))
>  		      (if ispell-dictionary ; use specified dictionary
>  			  (setq args
>  				(append (list "-d" ispell-dictionary) args)))
> 

This seems to have some side effects with ispell, save

---------------------
óptica
---------------------

to something.tex, make sure flyspell-large-region limit is set to 1, select
castellano8 dictionary and run flyspell-buffer

if: Bug: misspelled word `ptica' (output pos 7) not found in buffer

and contents of flyspell-region buffer (containing words ispell considers
misspelled) is

ptica

Seems that by doing that, ispell is also setting to the tex charset, e.g.

$ echo français | ispell -l -t -d francais
fran
ais

with the word "français" and the francais dict (they could also be used to test
this problem)

aspell seems to accept that.

-- 
Agustin

Re: flyspell bug

[Richard M. Stallman]

I have run out of time to work on flyspell myself--I am overwhelmed
with mail and can't keep up.

Can someone please volunteer to take charge of this, and debug the
bugs that remain?  Please respond if you will do this.

Re: flyspell bug

[Agustin Martin]

On Sat, Nov 19, 2005 at 06:26:39PM -0500, Richard M. Stallman wrote:
> I have run out of time to work on flyspell myself--I am overwhelmed
> with mail and can't keep up.
> 
> Can someone please volunteer to take charge of this, and debug the
> bugs that remain?  Please respond if you will do this.

I am sorry I cannot take care of that,

My lisp skills are limited and all last code I sent was preceded by a lot of
time dedicated to searching about "How could I do in lisp what I want to do?"
for things that for a lisp programmer should have probably been
straightforward. And unfortunately, I do not have the time to improve that
situation from my side. I also think that the person coordinating
flyspell.el and ispell.el should be somebody fluent with lisp.

I will however be happy on testing proposed patches, and if possible on
debugging some problems and sending possible patches, always depending on
my available time.

I already subscribed to emacs-devel, and will read all mails having the
"spell" string in the subject. Other mail from emacs-devel will go to a
delayed mailbox I will only read occasionally. I am also in the limits
of my mail processing capabilities, and I really need to shorten the
large pending TODO list I have in my "real life" work. 

-- 
Agustin

Re: flyspell bug

[Richard M. Stallman]

I have run out of time to work on flyspell myself--I am overwhelmed
with mail and can't keep up.

Can someone please volunteer to take charge of this, and debug the
bugs that remain?  Please respond if you will do this.

If you are not offering to do this, please DO NOT respond to this
message.  It has a reminder on it, and if you send a spurious
response, I will have to send this message again.

Re: flyspell bug [possible patch]

[Agustin Martin]

[-- Attachment #1: Type: text/plain, Size: 1590 bytes --]

On Wed, Nov 16, 2005 at 03:22:42PM +0000, Agustin Martin wrote:
> Slawomir Nowaczyk <slawomir.nowaczyk.847 <at> student.lu.se> writes:
> > 
> > Besides that, I believe the logic in flyspell-external-point-words
> > should be simplified significantly -- we should get rid of the "while
> > keep" loop, just find the first place where
> >      			  (if (= (length (flyspell-get-word nil))
> > 				 (length word))
> > and flyspell-word that.
> 
> I think you need the (while keep) loop for this. 
> 
> > And consider cases where ispell and flyspell
> > have different ideas about word boundaries to be bugs and fix those.
> 
> Some of these might require a not found word be signalled by something softer
> than an error. 
> 
> I am also taking a look at this function making keep nil if any of some
> conditions  is matched. Will send something as soon as looks better (if so),
> even if still full of debugging code

I am attaching the patch as is currently in my box, seems to work well with
aspell, and in my tests failed only with ispell in rather uncommon things
like

\distribute dist distr

where dist is not tagged as misspelled. It however worked well with

distribute dist distr

In the former case, unsync is backwards, what is not that critical, as
opposed to an forward unsync, and since this is also a rather unfrequent
example, I think this is acceptable, and anyway better than current
situation. 

Patch includes removal of the "-t" stuff in flyspell-large-region, I think
it is no longer needed and was causing problems with ispell.

Please check in your boxes

-- 
Agustin

[-- Attachment #2: flyspell-external-point-words.diff --]
[-- Type: text/plain, Size: 5520 bytes --]

--- flyspell.el.emacs-cvs	2005-11-22 12:29:26.000000000 +0100
+++ flyspell.el	2005-11-22 12:32:19.000000000 +0100
@@ -1304,50 +1304,70 @@
 The list of incorrect words should be in `flyspell-external-ispell-buffer'.
 \(We finish by killing that buffer and setting the variable to nil.)
 The buffer to mark them in is `flyspell-large-region-buffer'."
-
-  (with-current-buffer flyspell-external-ispell-buffer
-    (goto-char (point-min))
-    ;; Loop over incorrect words.
-    (while (re-search-forward "\\([^\n]+\\)\n" (point-max) t)
-      ;; Bind WORD to the next one.
-      (let ((word (match-string 1)) (wordpos (point)))
-	;; Here there used to be code to see if WORD is the same
-	;; as the previous iteration, and count the number of consecutive
-	;; identical words, and the loop below would search for that many.
-	;; That code seemed to be incorrect, and on principle, should
-	;; be unnecessary too. -- rms.
-	(if flyspell-issue-message-flag
-	    (message "Spell Checking...%d%% [%s]"
-		     (* 100 (/ (float (point)) (point-max)))
-		     word))
-	;; Search the other buffer for occurrences of this word,
-	;; and check them.  Stop when we find one that reports "incorrect".
-	;; (I don't understand the reason for that logic,
-	;; but I didn't want to change it. -- rms.)
-	(with-current-buffer flyspell-large-region-buffer
-	  (goto-char flyspell-large-region-beg)
-	  (let ((keep t))
-	    (while keep
-	      (if (search-forward word
-				     flyspell-large-region-end t)
-		  (progn
-		    (setq flyspell-large-region-beg (point))
-		    (goto-char (- (point) 1))
-		    (setq keep
-			  ;; Detect when WORD can't be checked properly
-			  ;; because flyspell-get-word finds
-			  ;; just part of it, and treat that as ok.
-			  (if (< (length (car (flyspell-get-word nil)))
-				 (length word))
-			      nil
-			    (flyspell-word))))
-		(error "Bug: misspelled word `%s' (output pos %d) not found in buffer"
-		       word wordpos)))))))
-    ;; we are done
-    (if flyspell-issue-message-flag (message "Spell Checking completed.")))
-  ;; Kill and forget the buffer with the list of incorrect words.
-  (kill-buffer flyspell-external-ispell-buffer)
-  (setq flyspell-external-ispell-buffer nil))
+  (let (words-not-found
+	(ispell-otherchars (ispell-get-otherchars)))    
+    (with-current-buffer flyspell-external-ispell-buffer
+      (goto-char (point-min))
+      ;; Loop over incorrect words.
+      (while (re-search-forward "\\([^\n]+\\)\n" (point-max) t)
+	;; Bind WORD to the next one.
+	(let ((word (match-string 1)) (wordpos (point)))
+	  ;; Here there used to be code to see if WORD is the same
+	  ;; as the previous iteration, and count the number of consecutive
+	  ;; identical words, and the loop below would search for that many.
+	  ;; That code seemed to be incorrect, and on principle, should
+	  ;; be unnecessary too. -- rms.
+	  (if flyspell-issue-message-flag
+	      (message "Spell Checking...%d%% [%s]"
+		       (* 100 (/ (float (point)) (point-max)))
+		       word))
+	  (with-current-buffer flyspell-large-region-buffer
+	    (goto-char flyspell-large-region-beg)
+	    (let ((keep t))
+	      ;; Iterate on string search until string is found as word,
+	      ;; not as substring, 
+	      (while keep
+		(if (search-forward word
+				    flyspell-large-region-end t)
+		    (progn
+		      (goto-char (- (point) 1))
+		      (let* ((match-point (point)) ; flyspell-get-word might move it
+			     (flyword-prev-l (flyspell-get-word nil))
+			     (flyword-endpoint (point))
+			     (flyword-prev (car flyword-prev-l)) 
+			     (size-match (= (length flyword-prev) (length word))))
+			(when (or
+			       size-match
+			       ;; Matches as part of a boundary-char separated word
+			       ;; In this case move to end of match, not to end of word
+			       (member word 
+				       (split-string flyword-prev ispell-otherchars))
+			       ;; ispell treats beginning of some TeX commands as nroff
+			       ;; control sequences and strips them in the list of
+			       ;; misspelled words thus giving a non-existent word.
+			       ;; Skip if ispell is used, string is a TeX command and
+			       ;; none of the previous contitions match
+			       (and (not ispell-really-aspell)
+				    (save-excursion
+				      (goto-char (- (nth 1 flyword-prev-l) 1))
+				      (if (looking-at "[\\]" )
+					  t
+					nil))))
+			  (setq keep nil)
+			  (flyspell-word)
+			  (setq flyspell-large-region-beg
+				(if size-match flyword-endpoint match-point)))))
+		  (add-to-list 'words-not-found
+			       (concat " -> " word " - " (int-to-string wordpos)))
+		  (setq keep nil)
+		  ))))))
+      ;; we are done
+      (if flyspell-issue-message-flag (message "Spell Checking completed.")))
+    (dolist (word words-not-found)
+      (message "%s: word not found" word))
+    ;; Kill and forget the buffer with the list of incorrect words.
+    (kill-buffer flyspell-external-ispell-buffer)
+    (setq flyspell-external-ispell-buffer nil)))
 
 ;;*---------------------------------------------------------------------*/
 ;;*    flyspell-process-localwords ...                                  */
@@ -1416,8 +1436,6 @@
 		      (if ispell-local-dictionary
 			  (setq ispell-dictionary ispell-local-dictionary))
 		      (setq args (ispell-get-ispell-args))
-		      (if (eq ispell-parser 'tex)
-			  (setq args (cons "-t" args)))
 		      (if ispell-dictionary ; use specified dictionary
 			  (setq args
 				(append (list "-d" ispell-dictionary) args)))

[-- Attachment #3: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: flyspell bug [possible patch]

[Richard M. Stallman]

I asked Eli Z to install your patch.  But could you provide a change log
explaining the changes?

Also, I think a few more comments are needed in the new code
to make it really clear.

Re: flyspell bug [possible patch]

[Agustin Martin]

[-- Attachment #1: Type: text/plain, Size: 2268 bytes --]

On Wed, Nov 23, 2005 at 10:58:43PM -0500, Richard M. Stallman wrote:
> I asked Eli Z to install your patch.  But could you provide a change log
> explaining the changes?

A short summary can be something like

(flyspell-external-point-words) Improve misspelling match reliability. Warn
                                about non found misspellings
(flyspell-large-region)  No longer needs setting ispell-parser to tex in TeX mode.

This is a verbose summary of the changes, so the above can be improved:

 *  (flyspell-external-point-words) Consider a misspelling as found in the
    string search if any of:

    (a) misspelling and found string lengths match
    (b) misspelling is found as element in a boundary-chars separated
        longer string (possible boundary-chars mismatch)
    (c) ispell-program-name is really ispell and misspelling is found as
        part of a TeX string (in base mode ispell considers the beginning of
        some of those TeX strings a nroff sequences and strips them when
	writing, thus producing not-found misspellings or, worse, bad
        unsyncs. Setting mode to TeX produces other problems).
  
    After successful match move beginning of search region to end of match

    Warn about not found misspellings once the process is done. (This
    should mostly affect ispell when ispell dict allows more >7bit chars
    than those in (not-)casechars (or are passed to ispell via -w) and 
    there are words in the text with them.
  
 *  (flyspell-large-region) Do not set ispell-parser to tex if in TeX mode.
    This is no longer needed for (flyspell-external-point-words) and has
    some drawbacks when real ispell is used and TeX files use an 8bit
    encoding.
  
> Also, I think a few more comments are needed in the new code
> to make it really clear.

I am attaching a new version of the patch, slightly simplified, just move
always to end of match if a misspelling is considered found by any of (a),
(b) or (c). In the only case (a) where end of word was used, both end of
match and end of word should be the same, so seems simpler to use end of
match for all.

I have added some extra comments, but you might want to add some more. I
also left the coments about

;; Here there used to be ...

present,

-- 
Agustin

[-- Attachment #2: flyspell-external-point-words.diff3 --]
[-- Type: text/plain, Size: 5627 bytes --]

--- flyspell.el.emacs-cvs	2005-11-22 12:29:26.000000000 +0100
+++ flyspell.el	2005-11-24 13:30:19.000000000 +0100
@@ -1304,50 +1304,71 @@
 The list of incorrect words should be in `flyspell-external-ispell-buffer'.
 \(We finish by killing that buffer and setting the variable to nil.)
 The buffer to mark them in is `flyspell-large-region-buffer'."
-
-  (with-current-buffer flyspell-external-ispell-buffer
-    (goto-char (point-min))
-    ;; Loop over incorrect words.
-    (while (re-search-forward "\\([^\n]+\\)\n" (point-max) t)
-      ;; Bind WORD to the next one.
-      (let ((word (match-string 1)) (wordpos (point)))
-	;; Here there used to be code to see if WORD is the same
-	;; as the previous iteration, and count the number of consecutive
-	;; identical words, and the loop below would search for that many.
-	;; That code seemed to be incorrect, and on principle, should
-	;; be unnecessary too. -- rms.
-	(if flyspell-issue-message-flag
-	    (message "Spell Checking...%d%% [%s]"
-		     (* 100 (/ (float (point)) (point-max)))
-		     word))
-	;; Search the other buffer for occurrences of this word,
-	;; and check them.  Stop when we find one that reports "incorrect".
-	;; (I don't understand the reason for that logic,
-	;; but I didn't want to change it. -- rms.)
-	(with-current-buffer flyspell-large-region-buffer
-	  (goto-char flyspell-large-region-beg)
-	  (let ((keep t))
-	    (while keep
-	      (if (search-forward word
-				     flyspell-large-region-end t)
-		  (progn
-		    (setq flyspell-large-region-beg (point))
-		    (goto-char (- (point) 1))
-		    (setq keep
-			  ;; Detect when WORD can't be checked properly
-			  ;; because flyspell-get-word finds
-			  ;; just part of it, and treat that as ok.
-			  (if (< (length (car (flyspell-get-word nil)))
-				 (length word))
-			      nil
-			    (flyspell-word))))
-		(error "Bug: misspelled word `%s' (output pos %d) not found in buffer"
-		       word wordpos)))))))
-    ;; we are done
-    (if flyspell-issue-message-flag (message "Spell Checking completed.")))
-  ;; Kill and forget the buffer with the list of incorrect words.
-  (kill-buffer flyspell-external-ispell-buffer)
-  (setq flyspell-external-ispell-buffer nil))
+  (let (words-not-found
+	(ispell-otherchars (ispell-get-otherchars)))    
+    (with-current-buffer flyspell-external-ispell-buffer
+      (goto-char (point-min))
+      ;; Loop over incorrect words.
+      (while (re-search-forward "\\([^\n]+\\)\n" (point-max) t)
+	;; Bind WORD to the next one.
+	(let ((word (match-string 1)) (wordpos (point)))
+	  ;; Here there used to be code to see if WORD is the same
+	  ;; as the previous iteration, and count the number of consecutive
+	  ;; identical words, and the loop below would search for that many.
+	  ;; That code seemed to be incorrect, and on principle, should
+	  ;; be unnecessary too. -- rms.
+	  (if flyspell-issue-message-flag
+	      (message "Spell Checking...%d%% [%s]"
+		       (* 100 (/ (float (point)) (point-max)))
+		       word))
+	  (with-current-buffer flyspell-large-region-buffer
+	    (goto-char flyspell-large-region-beg)
+	    (let ((keep t))
+	      ;; Iterate on string search until string is found as word,
+	      ;; not as substring, 
+	      (while keep
+		(if (search-forward word
+				    flyspell-large-region-end t)
+		    (progn
+		      (goto-char (- (point) 1))
+		      (let* ((match-point (point)) ; flyspell-get-word might move it
+			     (flyword-prev-l (flyspell-get-word nil))
+			     (flyword-prev (car flyword-prev-l)) 
+			     (size-match (= (length flyword-prev) (length word))))
+			(when (or
+			       ;; size matches, we are done
+			       size-match
+			       ;; Matches as part of a boundary-char separated word
+			       (member word 
+				       (split-string flyword-prev ispell-otherchars))
+			       ;; ispell treats beginning of some TeX commands as
+			       ;; nroff control sequences and strips them in the list
+			       ;; of misspelled words thus giving a non-existent word.
+			       ;; Skip if ispell is used, string is a TeX command
+			       ;; (char before beginning of word is backslash) and
+			       ;; none of the previous contitions match
+			       (and (not ispell-really-aspell)
+				    (save-excursion
+				      (goto-char (- (nth 1 flyword-prev-l) 1))
+				      (if (looking-at "[\\]" )
+					  t
+					nil))))
+			  (setq keep nil)
+			  (flyspell-word)
+			  ;; Next search will begin from end of last match
+			  (setq flyspell-large-region-beg match-point))))
+		  ;; Record if misspelling is not found and try new one
+		  (add-to-list 'words-not-found
+			       (concat " -> " word " - " (int-to-string wordpos)))
+		  (setq keep nil)))))))
+      ;; we are done
+      (if flyspell-issue-message-flag (message "Spell Checking completed.")))
+    ;; Warn about not found misspellings
+    (dolist (word words-not-found)
+      (message "%s: word not found" word))
+    ;; Kill and forget the buffer with the list of incorrect words.
+    (kill-buffer flyspell-external-ispell-buffer)
+    (setq flyspell-external-ispell-buffer nil)))
 
 ;;*---------------------------------------------------------------------*/
 ;;*    flyspell-process-localwords ...                                  */
@@ -1416,8 +1437,6 @@
 		      (if ispell-local-dictionary
 			  (setq ispell-dictionary ispell-local-dictionary))
 		      (setq args (ispell-get-ispell-args))
-		      (if (eq ispell-parser 'tex)
-			  (setq args (cons "-t" args)))
 		      (if ispell-dictionary ; use specified dictionary
 			  (setq args
 				(append (list "-d" ispell-dictionary) args)))

[-- Attachment #3: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel