Adding ispell-multi.el and friends to Emacs

Adding ispell-multi.el and friends to Emacs

[Peter Heslin]

In another thread, it was suggested that this might be a good time to
suggest including a package of mine, ispell-multi.el, in Emacs, possibly
along with two other packages that depend on it.  When I posted
ispell-multi to emacs.sources a few years ago, RMS expressed an interest
in including it in Emacs.  He noted a few things that needed fixing, and I
believe these have been fixed in the interim.

ispell-multi.el extends ispell.el so that it maintains a collection of
running ispell processes instead of just one.  So if you have one buffer
in English and another in French, Emacs no longer kills the old process
and starts a new one every time you switch buffers.  This is a useful
feature, but the real reason I wrote it is to support the ability to
spell-check multiple languages in the same buffer using flyspell.
Without the ability to maintain several running processes, there would
be a very noticeable delay when moving the cursor from one language region 
to another.

ispell-multi therefore also provides a hook that tells flyspell to
switch languages depending on the value of a particular text property,
and provides hooks to trigger re-parsing of the buffer in order to set
that text property to the correct language.  This facility is used by
two other packages of mine, both of which allow you to mix text in
different languages in the same buffer, and have flyspell check it all.

These two packages might also be considered for inclusion:

* flyspell-babel.el: It roughly parses a LaTeX buffer and sets the text
  property to the correct language for flyspell, depending on the Babel
  language-switching commands in the buffer.  These are configurable, so
  if you use custom LaTeX commands instead of Babel, you can still use
  the package.  Of course, TeX can only be parsed by TeX, so it is far
  from foolproof; but it does work well for normal usage.

* flyspell-xml-multi.el:  As above, but changes the language for
  flyspell depending on the xml:lang attribute of the current element in
  an XML buffer.  If the major mode is nxml-mode, it should understand XML
  pretty well; if not, it falls back on an even rougher sort of parsing
  than what is used for LaTeX.

I believe the only change that would be required in Emacs to integrate
these would be to provide a hook in ispell-accept-buffer-local-defs in
ispell.el where ispell-multi could be called.  This is currently
achieved by means of a defadvice, which would have to be changed.

I don't know if the functionality of ispell-multi should be enabled by
default, or only if the user requests it.

All three files are available here:
http://www.dur.ac.uk/p.j.heslin/Software/Emacs/index.php

Peter

-- 
Peter Heslin (http://www.dur.ac.uk/p.j.heslin)

Re: Adding ispell-multi.el and friends to Emacs

[Richard Stallman]

Would it make the code simpler to merge ispell-multi.el into ispell.el?

Re: Adding ispell-multi.el and friends to Emacs

[martin rudalics]

 > ispell-multi.el extends ispell.el so that it maintains a collection of
 > running ispell processes instead of just one.  So if you have one buffer
 > in English and another in French, Emacs no longer kills the old process
 > and starts a new one every time you switch buffers.

AFAICT flyspell-mode has the following bug: When I check two buffers
with different languages and switch buffers, words that have been
"accepted for the session" are lost.  Does ispell-multi handle this?

Re: Adding ispell-multi.el and friends to Emacs

[Peter Heslin]

martin rudalics <rudalics@gmx.at> writes:

> AFAICT flyspell-mode has the following bug: When I check two buffers
> with different languages and switch buffers, words that have been
> "accepted for the session" are lost.  Does ispell-multi handle this?

I wasn't aware of the problem and I haven't looked into it closely, but
a quick test seems to indicate that loading ispell-multi.el makes this
bug go away.  This would make sense, since ispell-multi tries to stop
the killing and restarting of ispell processes that would cause the
words accepted for the session to be lost.  A corollary is that with
ispell-multi, words accepted for the session are also accepted for the
session in other buffers in the same language.  I would tend to regard
that as a feature rather than a bug.

Could you confirm that loading ispell-multi fixes the problem for you?

Peter

-- 
Peter Heslin (http://www.dur.ac.uk/p.j.heslin)

Re: Adding ispell-multi.el and friends to Emacs

[Peter Heslin]

Richard Stallman <rms@gnu.org> writes:

> Would it make the code simpler to merge ispell-multi.el into ispell.el?

Not by much.  But if its functionality were loaded by default rather
than as a user option, it might be conceptually clearer to merge it.

-- 
Peter Heslin (http://www.dur.ac.uk/p.j.heslin)

Re: Adding ispell-multi.el and friends to Emacs

[martin rudalics]

 > I wasn't aware of the problem and I haven't looked into it closely, but
 > a quick test seems to indicate that loading ispell-multi.el makes this
 > bug go away.  This would make sense, since ispell-multi tries to stop
 > the killing and restarting of ispell processes that would cause the
 > words accepted for the session to be lost.  A corollary is that with
 > ispell-multi, words accepted for the session are also accepted for the
 > session in other buffers in the same language.  I would tend to regard
 > that as a feature rather than a bug.

Sounds reasonable.  BTW, what happens when you "accept a word for the
buffer" and there's a shared process for the current and another buffer?

 > Could you confirm that loading ispell-multi fixes the problem for you?

FWIW it does but I can't test reliably.  I don't use multiple languages
with `ispell-word' because it usually complains about different
character maps.  `flyspell-mode' is a real pain on my system since it
makes my cursor crawl.  I wrote my own mode to avoid all that hassle.

Re: Adding ispell-multi.el and friends to Emacs

[Eli Zaretskii]

> From: Peter Heslin <pj@heslin.eclipse.co.uk>
> Date: Fri, 15 Jun 2007 23:53:14 +0100
> 
> A corollary is that with ispell-multi, words accepted for the
> session are also accepted for the session in other buffers in the
> same language.  I would tend to regard that as a feature rather than
> a bug.

In some situations it could actually be a misfeature.  Imagine that I
tell flyspell to accept words that are special to some programming
language.  For example, in a Texinfo buffer, I might allow some
@-commands.  Another similar case is that some rare acronyms or
personal names turn up frequently in some text, so I want to allow
them in that one buffer.

So it sounds like this feature should be optional, although I don't
mind it being on by default.

Re: Adding ispell-multi.el and friends to Emacs

[martin rudalics]

 >>A corollary is that with ispell-multi, words accepted for the
 >>session are also accepted for the session in other buffers in the
 >>same language.  I would tend to regard that as a feature rather than
 >>a bug.
 >
 >
 > In some situations it could actually be a misfeature.  Imagine that I
 > tell flyspell to accept words that are special to some programming
 > language.  For example, in a Texinfo buffer, I might allow some
 > @-commands.  Another similar case is that some rare acronyms or
 > personal names turn up frequently in some text, so I want to allow
 > them in that one buffer.

That's what flyspell's "accept for buffer" feature is for.

 > So it sounds like this feature should be optional, although I don't
 > mind it being on by default.

AFAICT flyspell throws away all words accepted for the session _and_ the
buffer when you switch buffers.  Hence Peter's solution is certainly not
a regression.  Making the feature optional means you still can choose
only between accepting a word for all buffers sharing the same process
or throw away any words you accepted previously and did not add to the
personal word list.

I solved the problem by using a hash table for every buffer I
spell-check and keep all the words there (together with the
corresponding language identifiers and whether they are buffer- or
session-local).

Re: Adding ispell-multi.el and friends to Emacs

[Richard Stallman]

    Not by much.  But if its functionality were loaded by default rather
    than as a user option, it might be conceptually clearer to merge it.

I think it should be the default.  Most users won't make real use of
more than one dictionary, so they won't lose anything.  Those that do
should get the benefit of this.

Re: Adding ispell-multi.el and friends to Emacs

[Frank Schmitt]

Richard Stallman <rms@gnu.org> writes:

>     Not by much.  But if its functionality were loaded by default rather
>     than as a user option, it might be conceptually clearer to merge it.
>
> I think it should be the default.  Most users won't make real use of
> more than one dictionary, so they won't lose anything.  Those that do
> should get the benefit of this.

I think almost everyone whose native language isn't English does make
use of more than one dictionary...

-- 
Did you ever realize how much text fits in eighty columns? If you now consider
that a signature usually consists of up to four lines, this gives you enough
space to spread a tremendous amount of information with your messages. So seize
this opportunity and don't waste your signature with bullshit nobody will read.

Re: Adding ispell-multi.el and friends to Emacs

[Agustin Martin]

On Sat, Jun 16, 2007 at 09:46:15PM +0200, Frank Schmitt wrote:
> Richard Stallman <rms@gnu.org> writes:
> 
> >     Not by much.  But if its functionality were loaded by default rather
> >     than as a user option, it might be conceptually clearer to merge it.
> >
> > I think it should be the default.  Most users won't make real use of
> > more than one dictionary, so they won't lose anything.  Those that do
> > should get the benefit of this.
> 
> I think almost everyone whose native language isn't English does make
> use of more than one dictionary...

But usually not at the same time.

-- 
Agustin

Re: Adding ispell-multi.el and friends to Emacs

[Juri Linkov]

>> > I think it should be the default.  Most users won't make real use of
>> > more than one dictionary, so they won't lose anything.  Those that do
>> > should get the benefit of this.
>> 
>> I think almost everyone whose native language isn't English does make
>> use of more than one dictionary...
>
> But usually not at the same time.

Spell-checking of mixed multi-language texts is not a rare need.

How well does ispell-multi.el deal with this?  Can it detect different
languages in the same buffer and check them with different ispell
dictionaries? 

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: Adding ispell-multi.el and friends to Emacs

[Peter Heslin]

Juri Linkov <juri@jurta.org> writes:

>>> > I think it should be the default.  Most users won't make real use of
>>> > more than one dictionary, so they won't lose anything.  Those that do
>>> > should get the benefit of this.
>>> 
>>> I think almost everyone whose native language isn't English does make
>>> use of more than one dictionary...
>>
>> But usually not at the same time.
>
> Spell-checking of mixed multi-language texts is not a rare need.
>
> How well does ispell-multi.el deal with this?  Can it detect different
> languages in the same buffer and check them with different ispell
> dictionaries? 

Yes, but only for flyspell (this is exactly what flyspell-babel and
flyspell-xml-lang do).  Flyspell provides a hook it calls before
spell-checking each word, which makes it easy ensure the correct
dictionary for each word in the buffer.

I don't use ispell on its own, so I did not look previously into what it
would take to add this feature to ispell-buffer and friends. I have
taken a quick look now at ispell.el and it seems that ispell-buffer
sends the text of the buffer to the ispell process one line at a time.
This design makes it impossible to switch language in mid-line.

Perhaps it would be a simple change to modify ispell-process-line so
that it loops over the words in a line and sends each one to ispell
individually, switching processes if the language has changed.  Or it
might be hairy to change that function -- I don't know the code well
enough to guess what the complications might be.

Peter

-- 
Peter Heslin (http://www.dur.ac.uk/p.j.heslin)

Re: Adding ispell-multi.el and friends to Emacs

[Agustin Martin]

On Mon, Jun 18, 2007 at 12:11:59AM +0100, Peter Heslin wrote:
> I don't use ispell on its own, so I did not look previously into what it
> would take to add this feature to ispell-buffer and friends. I have
> taken a quick look now at ispell.el and it seems that ispell-buffer
> sends the text of the buffer to the ispell process one line at a time.
> This design makes it impossible to switch language in mid-line.
> 
> Perhaps it would be a simple change to modify ispell-process-line so
> that it loops over the words in a line and sends each one to ispell
> individually, switching processes if the language has changed.  Or it
> might be hairy to change that function -- I don't know the code well
> enough to guess what the complications might be.

I think I have read somewhere that current ispell.el
ispell-process-line use was made for efficiency, looping over
individual words was far slower. Something similar to flyspell-region over
large regions.

A mixed approach would be to first parse line and loop over words if a
language change is detected or active and send lines otherwise. I only
looked once at the line handling code to try doing something like that in
case of misalignments (doing line per words in case a misalignment is
found), but I had no success at all.

-- 
Agustin

Re: Adding ispell-multi.el and friends to Emacs

[Agustin Martin]

On Fri, Jun 15, 2007 at 11:53:14PM +0100, Peter Heslin wrote:
> martin rudalics <rudalics@gmx.at> writes:
> 
> > AFAICT flyspell-mode has the following bug: When I check two buffers
> > with different languages and switch buffers, words that have been
> > "accepted for the session" are lost.  Does ispell-multi handle this?
> 
> I wasn't aware of the problem and I haven't looked into it closely, but
> a quick test seems to indicate that loading ispell-multi.el makes this
> bug go away.  This would make sense, since ispell-multi tries to stop
> the killing and restarting of ispell processes that would cause the
> words accepted for the session to be lost.  A corollary is that with
> ispell-multi, words accepted for the session are also accepted for the
> session in other buffers in the same language.  I would tend to regard
> that as a feature rather than a bug.

Current ispell.el behavior wrt localwords when switching buffers with the
same language is to restart ispell if localwords are present, to avoid the
problem of conflicting localwords. If ispell-multi.el starts a new ispell
process in this case, keeping two processes for the same language, it might
also be made aware that words are accepted for that buffer session.

-- 
Agustin

Re: Adding ispell-multi.el and friends to Emacs

[martin rudalics]

 > A mixed approach would be to first parse line and loop over words if a
 > language change is detected or active and send lines otherwise. I only
 > looked once at the line handling code to try doing something like that in
 > case of misalignments (doing line per words in case a misalignment is
 > found), but I had no success at all.

FWIW, I have misalignment clashes only when Emacs and the spell system
disagree about the coding system.  In that case, however, the output
of the spell system is unusable for single words too.

Re: Adding ispell-multi.el and friends to Emacs

[martin rudalics]

 > Current ispell.el behavior wrt localwords when switching buffers with the
 > same language is to restart ispell if localwords are present, to avoid the
 > problem of conflicting localwords.

With the consequence that all "session-local" words are lost.

 > If ispell-multi.el starts a new ispell
 > process in this case, keeping two processes for the same language, it might
 > also be made aware that words are accepted for that buffer session.

ispell-multi.el seems to be able to retain "session-local" words for the
original buffer.  If they are present (or introduced) it should probably
start a new process for the second buffer.

A different problem occurs when you check one and the same buffer with
multiple processes: In this case "session-local" words should be shared
by all processes checking this buffer.  Otherwise, an abbreviation would
be marked erroneous by the French process and not marked by the Italian
one.  As I mentioned earlier, it's much simpler to have Emacs maintain
these words in buffer-local hash-lists and have Emacs override the
results reported by the spell processes.

Re: Adding ispell-multi.el and friends to Emacs

[Richard Stallman]

Can you send a patch to install ispell-multi's code
into ispell.el?

Re: Adding ispell-multi.el and friends to Emacs

[Agustin Martin]

On Mon, Jun 18, 2007 at 02:35:14PM +0200, martin rudalics wrote:
> > A mixed approach would be to first parse line and loop over words if a
> > language change is detected or active and send lines otherwise. I only
> > looked once at the line handling code to try doing something like that in
> > case of misalignments (doing line per words in case a misalignment is
> > found), but I had no success at all.
> 
> FWIW, I have misalignment clashes only when Emacs and the spell system
> disagree about the coding system.  In that case, however, the output
> of the spell system is unusable for single words too.

Unless communication is done in utf8, if you have an utf8 buffer with a char
not in the dict charset ispell-region will fail with a misalingment (at
least with ispell, most aspell communications are done in utf8), but if you
try ispell-word in that word, the foreign char will not be recognized as
part of a word and only part of it will be sent to ispell for testing. The
difference is that ispell-region will fail for all the region if a
misalignment appear while ispell-word will only do things differently for
the affected word. That was the intention.

-- 
Agustin

Re: Adding ispell-multi.el and friends to Emacs

[Agustin Martin]

[-- Attachment #1: Type: text/plain, Size: 2551 bytes --]

On Mon, Jun 18, 2007 at 02:37:20PM +0200, martin rudalics wrote:
> > Current ispell.el behavior wrt localwords when switching buffers with the
> > same language is to restart ispell if localwords are present, to avoid the
> > problem of conflicting localwords.
> 
> With the consequence that all "session-local" words are lost.

But the underlying problem is the lack of full support for buffer
session-local words. I have been playing a bit with ispell.el and flyspell.el
to try dealing with this. I am attaching a couple of patches with the current
status of my experiments. Note that they are minimally tested, but seem to
work reasonably.

A new buffer local variable `ispell-buffer-session-localwords' is used that
will contain a list of words accepted for session in each buffer. Each time
a buffer is parsed for localwords, those sesion localwords are sent to the
ispell process as accepted and the buffer marked as having localwords. same
when a word is accepted for session by either ispell.el or flyspell.el.

A new option CLEAR is added to (ispell-kill-ispell). For a buffer
`ispell-buffer-session-localwords' is cleaned when dict is explicitely changed
for that buffer with (ispell-change-dictionary) or when ispell process is
killed, either interactively or through the pop-up menu (using the 'clear).

flyspell.el is modified for the same purpose.

I will try to test this more in depth, but I think the ideas behind the
changes are reasonable.
 
> > If ispell-multi.el starts a new ispell
> > process in this case, keeping two processes for the same language, it 
> might
> > also be made aware that words are accepted for that buffer session.
> 
> ispell-multi.el seems to be able to retain "session-local" words for the
> original buffer.  If they are present (or introduced) it should probably
> start a new process for the second buffer.

As long as the ispell process is not killed those words will remain
accepted.

> A different problem occurs when you check one and the same buffer with
> multiple processes: In this case "session-local" words should be shared
> by all processes checking this buffer.  Otherwise, an abbreviation would
> be marked erroneous by the French process and not marked by the Italian
> one.  As I mentioned earlier, it's much simpler to have Emacs maintain
> these words in buffer-local hash-lists and have Emacs override the
> results reported by the spell processes.

I do not think there is currently support for multi-language buffer accepted
localwords, so that seems indeed simpler.

-- 
Agustin

[-- Attachment #2: flyspell.el_session_localwords.diff --]
[-- Type: text/x-diff, Size: 1024 bytes --]

--- flyspell.el.orig	2007-06-19 15:04:09.000000000 +0200
+++ flyspell.el	2007-06-19 15:04:21.000000000 +0200
@@ -1445,7 +1445,8 @@
 ;;*    declared correct.                                                */
 ;;*---------------------------------------------------------------------*/
 (defun flyspell-process-localwords (misspellings-buffer)
-  (let (localwords case-fold-search
+  (let ((localwords ispell-buffer-session-localwords)
+	case-fold-search
 	(ispell-casechars (ispell-get-casechars)))
     ;; Get localwords from the original buffer
     (save-excursion
@@ -2085,6 +2086,9 @@
 	 (setq ispell-pdict-modified-p '(t)))
 	((or (eq replace 'buffer) (eq replace 'session))
 	 (ispell-send-string (concat "@" word "\n"))
+	 (add-to-list 'ispell-buffer-session-localwords word)
+	 (or ispell-buffer-local-name ; session localwords might conflict
+	     (setq ispell-buffer-local-name (buffer-name)))
 	 (flyspell-unhighlight-at cursor-location)
 	 (if (null ispell-pdict-modified-p)
 	     (setq ispell-pdict-modified-p

[-- Attachment #3: ispell.el_session_localwords.diff --]
[-- Type: text/x-diff, Size: 3609 bytes --]

--- ispell.el.orig	2007-06-19 15:04:13.000000000 +0200
+++ ispell.el	2007-06-19 15:04:37.000000000 +0200
@@ -1050,7 +1050,8 @@
 	'(menu-item "Change Dictionary..." ispell-change-dictionary
 		    :help "Supply explicit dictionary file name"))
       (define-key ispell-menu-map [ispell-kill-ispell]
-	'(menu-item "Kill Process" ispell-kill-ispell
+	'(menu-item "Kill Process"
+		    (lambda () (interactive) (ispell-kill-ispell nil 'clear))
 		    :enable (and (boundp 'ispell-process) ispell-process
 				 (eq (ispell-process-status) 'run))
 		    :help "Terminate Ispell subprocess"))
@@ -1132,7 +1133,7 @@
 	     ["Continue Check"	ispell-continue			t]
 	     ["Complete Word Frag"ispell-complete-word-interior-frag t]
 	     ["Complete Word"	ispell-complete-word		t]
-	     ["Kill Process"	ispell-kill-ispell		t]
+	     ["Kill Process"	ispell-kill-ispell       t 'clear]
 	     ["Customize..."	(customize-group 'ispell)	t]
 	     ;; flyspell-mode may not be bound...
 	     ;;["flyspell"	flyspell-mode
@@ -1397,6 +1398,11 @@
   "Contains the buffer name if local word definitions were used.
 Ispell is then restarted because the local words could conflict.")
 
+(defvar ispell-buffer-session-localwords nil
+  "List of words accepted for session in this buffer.")
+
+(make-variable-buffer-local 'ispell-buffer-session-localwords)
+
 (defvar ispell-parser 'use-mode-name
    "*Indicates whether ispell should parse the current buffer as TeX Code.
 Special value `use-mode-name' tries to guess using the name of `major-mode'.
@@ -1875,6 +1881,9 @@
 		    nil)
 		   ((or (= char ?a) (= char ?A)) ; accept word without insert
 		    (ispell-send-string (concat "@" word "\n"))
+		    (add-to-list 'ispell-buffer-session-localwords word)
+		    (or ispell-buffer-local-name ; session localwords might conflict
+			(setq ispell-buffer-local-name (buffer-name)))
 		    (if (null ispell-pdict-modified-p)
 			(setq ispell-pdict-modified-p
 			      (list ispell-pdict-modified-p)))
@@ -2554,13 +2563,16 @@
 	(set-process-query-on-exit-flag ispell-process nil))))
 
 ;;;###autoload
-(defun ispell-kill-ispell (&optional no-error)
+(defun ispell-kill-ispell (&optional no-error clear)
   "Kill current Ispell process (so that you may start a fresh one).
-With NO-ERROR, just return non-nil if there was no Ispell running."
+With NO-ERROR, just return non-nil if there was no Ispell running.
+With CLEAR, buffer session localwords are cleaned."
   (interactive)
   ;; This hook is typically used by flyspell to flush some variables used
   ;; to optimize the common cases.
   (run-hooks 'ispell-kill-ispell-hook)
+  (if (or clear (interactive-p))
+      (setq ispell-buffer-session-localwords nil))
   (if (not (and ispell-process
 		(eq (ispell-process-status) 'run)))
       (or no-error
@@ -2622,6 +2634,7 @@
 	       (setq ispell-local-dictionary-overridden t))
 	   (error "Undefined dictionary: %s" dict))
 	 (ispell-internal-change-dictionary)
+	 (setq ispell-buffer-session-localwords nil)
 	 (message "%s Ispell dictionary set to %s"
 		  (if arg "Global" "Local")
 		  dict))))
@@ -3708,6 +3721,11 @@
   ;; Actually start a new ispell process, because we need
   ;; to send commands now to specify the local words to it.
   (ispell-init-process)
+  (dolist (session-localword ispell-buffer-session-localwords)
+    (ispell-send-string (concat "@" session-localword "\n")))
+  (or ispell-buffer-local-name
+      (if ispell-buffer-session-localwords
+	  (setq ispell-buffer-local-name (buffer-name))))
   (save-excursion
     (goto-char (point-min))
     (while (search-forward ispell-words-keyword nil t)

[-- Attachment #4: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: Adding ispell-multi.el and friends to Emacs

[martin rudalics]

 >>FWIW, I have misalignment clashes only when Emacs and the spell system
 >>disagree about the coding system.  In that case, however, the output
 >>of the spell system is unusable for single words too.
 >
 >
 > Unless communication is done in utf8, if you have an utf8 buffer with a char
 > not in the dict charset ispell-region will fail with a misalingment (at
 > least with ispell, most aspell communications are done in utf8), but if you
 > try ispell-word in that word, the foreign char will not be recognized as
 > part of a word and only part of it will be sent to ispell for testing. The
 > difference is that ispell-region will fail for all the region if a
 > misalignment appear while ispell-word will only do things differently for
 > the affected word. That was the intention.

Personally, I don't care whether a word, line, region or buffer appears
corrupted.  I'd discontinue spell-checking in any of these cases.  The
interface (ispell.el) has to check which coding system should get
applied.  The user has to check whether the interface got it right and,
if necessary, be able to override wrong settings.  BTW, I don't use utf8
to communicate with Aspell and did not encounter any problems so far.

Re: Adding ispell-multi.el and friends to Emacs

[martin rudalics]

 > But the underlying problem is the lack of full support for buffer
 > session-local words.

The underlying problem is that you have to pass all words to the process
before it can resume checking.  Currently, you have to save the
session-local words within Emacs and pass them to the process whenever
you change buffers or dictionaries.  In addition you have to re-parse
the local words section to find all buffer-local words and pass them to
the process too.  ispell-multi would help here because it makes restarts
less frequent.

 >>A different problem occurs when you check one and the same buffer with
 >>multiple processes: In this case "session-local" words should be shared
 >>by all processes checking this buffer.  Otherwise, an abbreviation would
 >>be marked erroneous by the French process and not marked by the Italian
 >>one.  As I mentioned earlier, it's much simpler to have Emacs maintain
 >>these words in buffer-local hash-lists and have Emacs override the
 >>results reported by the spell processes.
 >
 >
 > I do not think there is currently support for multi-language buffer accepted
 > localwords, so that seems indeed simpler.

These are two different issues: There should be an option to add to
entries in the local words section of a file the language (-code) to
ensure that the word was accepted only for that language.  A word
without that code would be accepted for all languages.

I talked about session-only words, words that are not written to the
local words section.  Adding, for example, an abbreviation to these
words should render it correct for all languages in that buffer.

Re: Adding ispell-multi.el and friends to Emacs

[Agustin Martin]

On Wed, Jun 20, 2007 at 02:46:03PM +0200, martin rudalics wrote:
> >>FWIW, I have misalignment clashes only when Emacs and the spell system
> >>disagree about the coding system.  In that case, however, the output
> >>of the spell system is unusable for single words too.
> >
> >
> > Unless communication is done in utf8, if you have an utf8 buffer with a 
> char
> > not in the dict charset ispell-region will fail with a misalingment (at
> > least with ispell, most aspell communications are done in utf8), but if 
> you
> > try ispell-word in that word, the foreign char will not be recognized as
> > part of a word and only part of it will be sent to ispell for testing. The
> > difference is that ispell-region will fail for all the region if a
> > misalignment appear while ispell-word will only do things differently for
> > the affected word. That was the intention.
> 
> Personally, I don't care whether a word, line, region or buffer appears
> corrupted.  I'd discontinue spell-checking in any of these cases. 

Sometimes you know what happened and can continue.

> The interface (ispell.el) has to check which coding system should get
> applied.

Agreed, and the interface should avoid sending words with non supported
chars in case a word with them is present (as with a foreign word) or at
least discriminate properly. What I mean is that is difficult to do that
when a line is sent as a whole to the ispell process. When things are
done word-by-word that is possible, but this is of small efficiency in
large buffers.

> The user has to check whether the interface got it right and,
> if necessary, be able to override wrong settings.  BTW, I don't use utf8
> to communicate with Aspell and did not encounter any problems so far.

You probably use it implicitely unless you use an explicitely user defined
non-utf8 dict entry.

>From ispell.el,

(defun ispell-aspell-find-dictionary (dict-name)
...
(list "-d" dict-name "--encoding=utf-8")
...
'utf-8))

BTW, I have just reproduced it with emacs21, aspell and the old ispell.el.
I admit that this is currently (emacs 22) a problem only with ispell or with
buggy user-defined entries.

Anyway, this subthread started when Peter considered using the word-by-word
mechanism and I pointed out that it might not be efficient in large buffers.
I do not think this is now adding much to that.

-- 
Agustin

Re: Adding ispell-multi.el and friends to Emacs

[Agustin Martin]

On Wed, Jun 20, 2007 at 02:47:15PM +0200, martin rudalics wrote:
> > But the underlying problem is the lack of full support for buffer
> > session-local words.
> 
> The underlying problem is that you have to pass all words to the process
> before it can resume checking.  Currently, you have to save the
> session-local words within Emacs and pass them to the process whenever
> you change buffers or dictionaries.  In addition you have to re-parse
> the local words section to find all buffer-local words and pass them to
> the process too.  ispell-multi would help here because it makes restarts
> less frequent.

Sorry, as I wrote things might seem that I am reluctant about using
ispell-multi.el, while I am in favour of that because it is cleaner and because
the old mechanism comes from a time where opening parallel ispell processes
was not cheap.

But unless ispell-multi.el is made *the* way of handling things (something I
would like) if ispell.el can be used without ispell-multi.el it should also
handle session-localwords properly and standalone. I admit this is not a big
problem if ispell-multi is enabled by default, but even if ispell-multi is
integrated into ispell.el, at least for flyspell-large-region is still good
to have ispell.el and flyspell.el keelping a record of buffer
session-accepted words and handle them properly (f-l-a uses a completely
different ispell process).

-- 
Agustin

Re: Adding ispell-multi.el and friends to Emacs

[Richard Stallman]

    But unless ispell-multi.el is made *the* way of handling things (something I
    would like)

I want ispell-multi to be integrated into ispell.el and to become the
default.  Whether the current behavior should remain as an option
is a question on which I don't have an opinion.

Re: Adding ispell-multi.el and friends to Emacs

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

>     But unless ispell-multi.el is made *the* way of handling things
>     (something I would like)
>
> I want ispell-multi to be integrated into ispell.el and to become
> the default.  Whether the current behavior should remain as an
> option is a question on which I don't have an opinion.

Seems sort of pointless two offer two different possibilities of
basically achieving the same result, and in the same file, to boot.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Adding ispell-multi.el and friends to Emacs

[Davis Herring]

> A mixed approach would be to first parse line and loop over words if a
> language change is detected or active and send lines otherwise. I only
> looked once at the line handling code to try doing something like that in
> case of misalignments (doing line per words in case a misalignment is
> found), but I had no success at all.

This could probably be made simple and efficient by using

(next-single-char-property-change (point) 'ispell-multi-lang
                                  nil (line-end-position))

to grab strings to send.  I'm not at all familiar with ispell, but if this
idea looks good I can try my hand at putting it in.

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: Adding ispell-multi.el and friends to Emacs

[Peter Heslin]

Agustin Martin <agustin.martin@hispalinux.es> writes:

> but even if ispell-multi is
> integrated into ispell.el, at least for flyspell-large-region is still good
> to have ispell.el and flyspell.el keelping a record of buffer
> session-accepted words and handle them properly (f-l-a uses a completely
> different ispell process).

That reminds me -- flyspell-babel and flyspell-xml-lang switch off
flyspell-large-region, because by design it starts a single dedicated
ispell process.  So flyspell-large-region would need to be rewritten to
iterate over language-regions if ispell-multi were integrated into Emacs.

Peter

-- 
Peter Heslin (http://www.dur.ac.uk/p.j.heslin)

Re: Adding ispell-multi.el and friends to Emacs

[Peter Heslin]

Richard Stallman <rms@gnu.org> writes:

>   But unless ispell-multi.el is made *the* way of handling things
>   (something I would like)
>
> I want ispell-multi to be integrated into ispell.el and to become the
> default.

This thread has brought up a few functions that will need to be reworked
to cope with multiple languages: ispell-buffer and
flyspell-large-region.  So it will be a bigger job than I originally
thought, and it will take time.

Peter

Re: Adding ispell-multi.el and friends to Emacs

[Richard Stallman]

    This thread has brought up a few functions that will need to be reworked
    to cope with multiple languages: ispell-buffer and
    flyspell-large-region.  So it will be a bigger job than I originally
    thought, and it will take time.

Do you think you can you do it in a few weeks?

Re: Adding ispell-multi.el and friends to Emacs

[Peter Heslin]

Richard Stallman <rms@gnu.org> writes:

>     This thread has brought up a few functions that will need to be reworked
>     to cope with multiple languages: ispell-buffer and
>     flyspell-large-region.  So it will be a bigger job than I originally
>     thought, and it will take time.
>
> Do you think you can you do it in a few weeks?

No.  I see now that modifying and testing all of the functionality of
both ispell and flyspell to work consistently with multiple ispell
processes may take some considerable effort, and I have other things on
my plate at the moment.

Peter

-- 
Peter Heslin (http://www.dur.ac.uk/p.j.heslin)

Re: Adding ispell-multi.el and friends to Emacs

[Richard Stallman]

    No.  I see now that modifying and testing all of the functionality of
    both ispell and flyspell to work consistently with multiple ispell
    processes may take some considerable effort, and I have other things on
    my plate at the moment.

In that case, we should install your code soon, thus
giving someone else a chance to do the rest.

Re: Adding ispell-multi.el and friends to Emacs

[Dan Nicolaescu]

Richard Stallman <rms@gnu.org> writes:

  >     No.  I see now that modifying and testing all of the functionality of
  >     both ispell and flyspell to work consistently with multiple ispell
  >     processes may take some considerable effort, and I have other things on
  >     my plate at the moment.
  > 
  > In that case, we should install your code soon, thus
  > giving someone else a chance to do the rest.

Hi,

Given that RMS agreed to have your code installed a while ago, don't you
want to go ahead and install it?

Thanks
        --dan