comint-carriage-motion causes severe problems.

comint-carriage-motion causes severe problems.

[Luc Teirlinck]

I do not know what problem(s) comint-carriage-motion is supposed to fix.
I use ielm extensively and M-x run-lisp and M-x shell frequently.  I have
never experienced any problems in Emacs20.7 before this function was 
introduced, but started experiencing plenty of them in Emacs21.

(remove-hook 'comint-output-filter-functions 'comint-carriage-motion) 

makes all these problems disappear and does not seem to introduce any
new ones.

The worst problems occur in ielm. (See the run below.)  In my own
usage of ielm, they are so bad that ielm is totally unusable without
executing the above line of code.  Trying to compile a function will
mess up the syntax badly and mess up everything unless one takes
specific action to manually correct the syntax situation.  Playing
around with any type of code involving key bindings or keyboard macros
just keeps printing out the wrong results and keeps messing up the
syntax, which has to be continuously manually corrected.  In fact any
string containing ^H or ^M characters causes trouble.  Similar results
occur in other comint applications as well, although the problem is
less acute there for the simple reason that strings containing ^H or
^M occur less often there than in ielm.

I believe that at the very least this function should be removed from
ielm.  I do not believe that there is really any reason to have
comint-output-filter-functions mess around with the string returned by
pp-to-string.  If there would be, then there would be something wrong
with pp-to-string.

Whether it has a place in other comint applications depends on the
severity of the problems it is supposed to fix.  I looked at old bug
reports posted around the time this function was introduced and the
only thing I could find was a posting concerning ls --color, which
seemed to be a problem specific to telnet-filter, not a general comint
problem.  I know programs which, when ran from a shell-buffer will
intersperse their output with repeats and ^H's.  However,
comint-carriage-motion makes these problems worse rather than solving
them.  Without the function, the output looks ugly, but is
readable. With the function, the output is unrecognizable.

Again, I am not necessarily saying that this function is a mistake
for all comint applications (although I definitely believe it is for
ielm).  It all depends on the severity of the problems it fixes and
whether or not there are better ways to fix them.

Does anybody remember which problem(s) this function is supposed to fix?

If this function fixes problems that are limited to particular comint 
applications, then it should be added to the local value of
comint-output-filter-functions for those particular applications, not
to the global value, as is presently the case.

Below is an ielm run, with comint-carriage-motion enabled.  To
understand the problems fully, one should run this example with
font-lock-mode enabled.  Redoing the example after executing:

(remove-hook 'comint-output-filter-functions 'comint-carriage-motion)

will give the correct results.

*** Welcome to IELM ***  Type (describe-mode) for help.
ELISP> (read-kbd-macro "\C-ca\C-m")
"              ; self-explanatory
ELISP> (defun aa (x y) (set x y))
aa
ELISP> (aa 'b 9)
9
ELISP> b
9
ELISP> (byte-compile 'aa)
#[(x y)
     L\207"    ; a piece of this string is missing
  [x y]
  2]

ELISP> "\bbb"
bb"            ; self-explanatory
ELISP> "aa\rbb"
bb"            ; self-explanatory
ELISP> 


Sincerely,

Luc Teirlinck.

Re: comint-carriage-motion causes severe problems.

[Miles Bader]

Luc Teirlinck <teirllm@dms.auburn.edu> writes:
> I do not know what problem(s) comint-carriage-motion is supposed to fix.

Many interactive programs in unix use CR and BS characters in their
output to try to update a single line of output (e.g., in progress
indicators).

> I believe that at the very least this function should be removed from
> ielm.

It's probably safe to do so, since probably nobody tries to do such
`carriage motion tricks' in elisp.

However ielm is a very atypical use of comint, and
`comint-carriage-motion' should definitely be turned on by default.  It
doesn't work to `turn it on for specific applications' because it's
almost always the case that you really have no idea what application in
a subprocess is going to be producing the output (e.g., if the program
directly invoked by comint is a shell, the output comes from whatever
the user invokes).

-Miles
-- 
[|nurgle|]  ddt- demonic? so quake will have an evil kinda setting? one that 
            will  make every christian in the world foamm at the mouth? 
[iddt]      nurg, that's the goal 

Re: comint-carriage-motion causes severe problems.

[Kai Großjohann]

Miles Bader <miles@lsi.nec.co.jp> writes:

> However ielm is a very atypical use of comint, and
> `comint-carriage-motion' should definitely be turned on by default.  It
> doesn't work to `turn it on for specific applications' because it's
> almost always the case that you really have no idea what application in
> a subprocess is going to be producing the output (e.g., if the program
> directly invoked by comint is a shell, the output comes from whatever
> the user invokes).

It seems, however, that M-x ielm RET should lead to a buffer where
comint-carriage-motion is off.  Is there a way to do that?

kai
-- 
A large number of young women don't trust men with beards.  (BFBS Radio)

Re: comint-carriage-motion causes severe problems.

[Miles Bader]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
> It seems, however, that M-x ielm RET should lead to a buffer where
> comint-carriage-motion is off.  Is there a way to do that?

I guess it could be done by having ielm make
`comint-output-filter-functions' a buffer-local copy of the global value,
excluding `comint-carriage-motion'.

`remove-hook' actually contains commented-out code that would do this sort
of thing more elegantly, by adding a local `not'-hook that would
effectively override a global hook.  I don't know why that code is
commented out; perhaps it doesn't work.

-Miles
-- 
I'm beginning to think that life is just one long Yoko Ono album; no rhyme
or reason, just a lot of incoherent shrieks and then it's over.  --Ian Wolff

Re: comint-carriage-motion causes severe problems.

[Stefan Monnier]

> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
> > It seems, however, that M-x ielm RET should lead to a buffer where
> > comint-carriage-motion is off.  Is there a way to do that?
> 
> I guess it could be done by having ielm make
> `comint-output-filter-functions' a buffer-local copy of the global value,
> excluding `comint-carriage-motion'.

But we need to be careful that this local copy does not contain t.
And if the global hook is modified afterwards, the local copy will not
see the change.

> `remove-hook' actually contains commented-out code that would do this sort
> of thing more elegantly, by adding a local `not'-hook that would
> effectively override a global hook.  I don't know why that code is
> commented out; perhaps it doesn't work.

Because it requires support in the C code that I have not installed (yet)
because I haven't felt like arguing about its usefulness.


	Stefan


Index: eval.c
===================================================================
RCS file: /cvsroot/emacs/emacs/src/eval.c,v
retrieving revision 1.189
diff -u -r1.189 eval.c
--- eval.c	28 May 2002 20:24:32 -0000	1.189
+++ eval.c	2 Jul 2002 15:30:55 -0000
@@ -90,7 +90,7 @@
 Lisp_Object Qautoload, Qmacro, Qexit, Qinteractive, Qcommandp, Qdefun;
 Lisp_Object Qinhibit_quit, Vinhibit_quit, Vquit_flag;
 Lisp_Object Qand_rest, Qand_optional;
-Lisp_Object Qdebug_on_error;
+Lisp_Object Qdebug_on_error, Qnot;
 Lisp_Object Qdeclare;
 
 /* This holds either the symbol `run-hooks' or nil.
@@ -2338,7 +2366,8 @@
 {
   Lisp_Object sym, val, ret;
   Lisp_Object globals;
-  struct gcpro gcpro1, gcpro2, gcpro3;
+  Lisp_Object negatives = Qnil;
+  struct gcpro gcpro1, gcpro2, gcpro3, gcpro4;
 
   /* If we are dying or still initializing,
      don't do anything--it would probably crash if we tried.  */
@@ -2359,7 +2388,7 @@
   else
     {
       globals = Qnil;
-      GCPRO3 (sym, val, globals);
+      GCPRO4 (sym, val, globals, negatives);
 
       for (;
 	   CONSP (val) && ((cond == to_completion)
@@ -2381,10 +2410,15 @@
 		  args[0] = XCAR (globals);
 		  /* In a global value, t should not occur.  If it does, we
 		     must ignore it to avoid an endless loop.  */
-		  if (!EQ (args[0], Qt))
+		  if (!(EQ (args[0], Qt))
+		      && NILP (Fmember (args[0], negatives)))
 		    ret = Ffuncall (nargs, args);
 		}
 	    }
+	  else if (CONSP (XCAR (val)) && (EQ (XCAR (XCAR (val)), Qnot)))
+	    /* (not . FUNCTION) indicates that any subsequent FUNCTION
+	       should be ignored.  */
+	    negatives = Fcons (XCDR (XCAR (val)), negatives);
 	  else
 	    {
 	      args[0] = XCAR (val);
@@ -3276,6 +3310,9 @@
   Qdebug_on_error = intern ("debug-on-error");
   staticpro (&Qdebug_on_error);
 
+  Qnot = intern ("not");
+  staticpro (&Qnot);
+
   Qmacro = intern ("macro");
   staticpro (&Qmacro);
 
Index: subr.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/subr.el,v
retrieving revision 1.311
diff -u -r1.311 subr.el
--- subr.el	27 Jun 2002 16:08:56 -0000	1.311
+++ subr.el	2 Jul 2002 15:32:59 -0000
@@ -58,6 +58,17 @@
 
 (defalias 'not 'null)
 
+(if (not (fboundp 'while))
+    (defmacro while (test &rest body)
+      "Like `until' except TEST is negated."
+      `(progn (until (not ,test) ,@body) nil)))
+(if (not (fboundp 'until))
+    (defmacro until (test &rest body)
+      "Like `while' except TEST is negated and the form returns non-nil."
+      `(let (-until-res)
+	 (while (not (setq -until-res ,test)) ,@body)
+	 -until-res)))
+
 (defmacro lambda (&rest cdr)
   "Return a lambda expression.
 A call of the form (lambda ARGS DOCSTRING INTERACTIVE BODY) is
@@ -833,9 +844,13 @@
 	(if (equal hook-value function) (setq hook-value nil))
       (setq hook-value (delete function (copy-sequence hook-value))))
     ;; If the function is on the global hook, we need to shadow it locally
-    ;;(when (and local (member function (default-value hook))
-    ;;	       (not (member (cons 'not function) hook-value)))
-    ;;  (push (cons 'not function) hook-value))
+    (when (and local (local-variable-if-set-p hook)
+	       (member function
+		       (let ((val (default-value hook)))
+			 (if (or (not (listp val)) (functionp val))
+			     (list val) val)))
+    	       (not (member (cons 'not function) hook-value)))
+      (push (cons 'not function) hook-value))
     ;; Set the actual variable
     (if (not local)
 	(set-default hook hook-value)

Re: comint-carriage-motion causes severe problems.

[Luc Teirlinck]

I now agree that comint-carriage-motion is appropriate for the global
value of comint-output-filter-functions.  I believe that there seems
to be consensus that it is not appropriate for the local ielm value.

The following code would implement this, but has some drawbacks:

 (setq comint-output-filter-functions
       (default-value 'comint-output-filter-functions))

 (remove-hook 'comint-output-filter-functions 'comint-carriage-motion t)

The drawbacks are that this creates an "old-style" local hook without
t and the Emacs Lisp manual as well as plenty of documentation strings
of hook-related functions seem to very emphatically state that this is
stylistically unacceptable.  (Or else I am misunderstanding them.)

The removal of t from the local value also means, as Stefan pointed
out, that changes in the global value will no longer be seen by the
local value.  Fortunately, this would be limited to the duration of
the present ielm run.

The question of how to best remove unwanted global functions from
the local values of hooks is, however, way more general and far
reaching than the particular question we are discussing.

I would argue for the desirability of Stefan's suggested additions to
the hook machinery.  I believe they are absolutely necessary.  We have
right here a very good example why.

Sincerely,

Luc Teirlinck.

Re: comint-carriage-motion causes severe problems.

[Richard Stallman]

    I would argue for the desirability of Stefan's suggested additions to
    the hook machinery.  I believe they are absolutely necessary.  We have
    right here a very good example why.

I think such a feature could be a good idea.  Stefan, could you state
the proposed specs (don't worry about implementation at this point)
for the feature?

Re: comint-carriage-motion causes severe problems.

[Stefan Monnier]

>     I would argue for the desirability of Stefan's suggested additions to
>     the hook machinery.  I believe they are absolutely necessary.  We have
>     right here a very good example why.
> 
> I think such a feature could be a good idea.  Stefan, could you state
> the proposed specs (don't worry about implementation at this point)
> for the feature?

The spec corresponding (barring bugs) to my current implementation
is as follows:

Among the list of functions specified in a hook, you can now have
one of the form (not . FOO) which says "if you come across FOO
in the rest of the list, just ignore it".

I.e. if the value of a hook is (a (not . b) c b), then `run-hooks'
will only run `a' and `c' but not `b'.  By "value of the hook" I mean
the combined local+global lists, so the above example could happen
in the case where the local value of the hook variable is
(a (not . b) c b) but could also happen if the local value is
(a (not . b) t) and the global value is (c b).

If you use remove-hook, add-hook, run-hooks, this new functionality
means that calling `remove-hook' will make sure that a subsequent
`run-hooks' will not run this code.
Actually, there are some unclear semantics when you do things like

	(add-hook 'foo 'bar)
	(add-hook 'foo 'bar nil 'local)
	(remove-hook 'foo 'bar 'local)

The current code removes `bar' from the local value of `foo' but
leaves it in the global one.  My new code does the same and only a second
(remove-hook 'foo 'bar 'local) will add a (not . bar) such that the
global setting is overridden.  This decision was arbitrary (mostly
trying to minimize the change in behavior for better compatibility).


	Stefan

Re: comint-carriage-motion causes severe problems.

[Luc Teirlinck]

Stefan Monnier wrote:

   Actually, there are some unclear semantics when you do things like

	   (add-hook 'foo 'bar)
	   (add-hook 'foo 'bar nil 'local)
	   (remove-hook 'foo 'bar 'local)

   The current code removes `bar' from the local value of `foo' but
   leaves it in the global one.  My new code does the same and only a second
   (remove-hook 'foo 'bar 'local) will add a (not . bar) such that the
   global setting is overridden.  This decision was arbitrary (mostly
   trying to minimize the change in behavior for better compatibility).

I believe that in the given situation it is actually the only choice.
If bar occurs in both the global and the local value it will get
executed twice each time foo is ran.  This is very likely to be
unintended.  The user should be able to correct the situation with a
single 

(remove-hook 'foo 'bar 'local)

rather than still having to re-add it to the global value.

However, I now believe that there is an even better solution which not
only removes the arbitrariness in the above example, but moreover
removes any danger of breaking existing code, which otherwise seems
to be a very real danger, and which also adds useful flexibility.

Indeed, I now start to get really weary about breaking existing code
where people rely on the fact that remove-hook "does no harm anyway"
if bar is not in the local value of foo and hence it does not matter
whether

(remove-hook 'foo 'bar 'local) 

is executed several times if a file is accidentally loaded more than
once.

I am starting to believe that it might be good to keep the present
behavior of remove-hook, unless the third argument is the symbol
'override, in which case it would override any function later in the
hook, including global functions.

So:

(remove 'foo 'bar 'local) completely keeps the present behavior:

remove from local value if bar is in the local value.  Do nothing
otherwise.  Same if 'local is replaced by t or any value other than
nil or override.

(remove 'foo 'bar 'override) overrides any other occurrences of bar
further down the list, local or global.


Depending on peoples' style, current code may contain third non-nil
arguments of t or 'local, as you use, but I can not possibly see how
people would have used a third argument 'override anywhere.  (Unless
they were actually anticipating your code.)  Hence, all worries about
breaking existing code would be gone, the worry about arbitrariness in
cases of the type you describe is gone, the worry about bad things
happening if files get accidentally loaded more than once is totally
gone and we have, moreover, better flexibility.

Sincerely,

Luc Teirlinck.

Re: comint-carriage-motion causes severe problems.

[Stefan Monnier]

> However, I now believe that there is an even better solution which not
> only removes the arbitrariness in the above example, but moreover
> removes any danger of breaking existing code, which otherwise seems
> to be a very real danger, and which also adds useful flexibility.
> 
> Indeed, I now start to get really weary about breaking existing code
> where people rely on the fact that remove-hook "does no harm anyway"
> if bar is not in the local value of foo and hence it does not matter
> whether
> 
> (remove-hook 'foo 'bar 'local) 
> 
> is executed several times if a file is accidentally loaded more than
> once.

This fear is unfounded.  I have been using my hacked Emacs with this
feature for at least a year with no problem and I have grepped
through the code to try and see what kind of behavior is expected,
and the fact is that hooks are added either only globally or only
locally.  And since my code only adds a (not . BAR) if we're removing
it locally and it is present globally, that means that my code is
almost never activated.  I hactually have a (debug) call in my code
to see when a (not . BAR) happens, and the very few times it has
happened have all been right.

> Depending on peoples' style, current code may contain third non-nil
> arguments of t or 'local, as you use, but I can not possibly see how
> people would have used a third argument 'override anywhere.  (Unless
> they were actually anticipating your code.)  Hence, all worries about
> breaking existing code would be gone, the worry about arbitrariness in
> cases of the type you describe is gone, the worry about bad things
> happening if files get accidentally loaded more than once is totally
> gone and we have, moreover, better flexibility.

I'm not sure what you mean by "better flexibility".  My "arbitrary"
choice is not so arbitrary in that it tries to maintain compatibility
and it also has the advantage of still allowing the other behavior.
I.e. if you really want to make sure that the hook function will not
be run, like your `override' argument would do, you can just do

	(remove 'foo 'bar 'local)
	(remove 'foo 'bar 'local)

where the first call will remove any local ocurrence and the second
will override any remaining global ocurrence.

So I'm not sure if we really get more flexibility and if the
added complexity is warranted.  I see the current behavior of
(remove 'foo 'bar 'local) as a bug and my patch as a bug-fix.
This impression is partly based on the (very few) cases where the
code was triggered to insert (not . BAR).


	Stefan

Re: comint-carriage-motion causes severe problems.

[Luc Teirlinck]

Dear Stefan,

My main concern with the 'override stuff was the danger of breaking
existing code and the danger of repeated loading having unwanted
effects.  If you checked out things carefully, then I guess it is not
needed.

This makes, of course, the other change a little bit more dangerous,
since its effect on existing code has not been tested yet.

Let me just tell you about my concern.  Through autoloads, :load and
:require keywords in defgroups and defcustoms and so on, plenty of
functions can be unpredictably added to the global values of hooks,
behind the user's back.  This actually happens all the time in
practice.  Just searching for some item using customize-browse can add
plenty of functions to hooks, even if no custumizations are actually
made.  Some of these functions may be globally necessary, but cause
trouble locally.  We may want to exclude them from the local value
before they appear in the global one because we have no idea when that
might happen.  

   After all

	      (remove-hook 'foo 'bar)
	      (add-hook 'foo 'bar)

      does not prevent `bar' from being run (the `remove-hook' above ends up
      being a noop).

This is not the same situation. We remove bar fom the global value
and add it right back.  Clearly that should be a no-op.

On the other hand:

(remove-hook 'foo 'bar 'local)
(add-hook 'foo 'bar)

first removes it from the local value and then adds it to the global.
This is not an inverse operation, as in the previous example, and
hence, there is no logical reason why it should be a no-op.

I will wait for your comments before making any additional
suggestions.

Sincerely,

Luc.

Re: comint-carriage-motion causes severe problems.

[Stefan Monnier]

> Let me just tell you about my concern.  Through autoloads, :load and
> :require keywords in defgroups and defcustoms and so on, plenty of
> functions can be unpredictably added to the global values of hooks,
> behind the user's back.  This actually happens all the time in
> practice.  Just searching for some item using customize-browse can add
> plenty of functions to hooks, even if no custumizations are actually
> made.  Some of these functions may be globally necessary, but cause
> trouble locally.  We may want to exclude them from the local value
> before they appear in the global one because we have no idea when that
> might happen.

I haven't seen any evidence for the need of such a thing.
The only cases I've seen where my patch is useful is when you want to
turn a feature on globally but turn it off at a few places.
Global customizations normally occur before buffers are created, so
before the (remove-hook 'foo 'bar 'local) has even a chance of being run.
If you could show me something more concrete, I might better understand
your concern.

>    After all
> 
> 	      (remove-hook 'foo 'bar)
> 	      (add-hook 'foo 'bar)
> 
>       does not prevent `bar' from being run (the `remove-hook' above ends up
>       being a noop).
> 
> This is not the same situation. We remove bar fom the global value
> and add it right back.  Clearly that should be a no-op.

I assumed that the hook was nil before the above two commands were executed,
which is why the `remove-hook' is a noop.


	Stefan

Re: comint-carriage-motion causes severe problems.

[Kai Großjohann]

"Stefan Monnier" <monnier+gnu/emacs@rum.cs.yale.edu> writes:

> I haven't seen any evidence for the need of such a thing.
> The only cases I've seen where my patch is useful is when you want to
> turn a feature on globally but turn it off at a few places.
> Global customizations normally occur before buffers are created,

Can you really predict when the user might say M-x customize-variable
RET?  I think this can happen at any time.

> so before the (remove-hook 'foo 'bar 'local) has even a chance of
> being run.  If you could show me something more concrete, I might
> better understand your concern.

kai
-- 
A large number of young women don't trust men with beards.  (BFBS Radio)

Re: comint-carriage-motion causes severe problems.

[Luc Teirlinck]

   I haven't seen any evidence for the need of such a thing.
   The only cases I've seen where my patch is useful is when you want
   to
   turn a feature on globally but turn it off at a few places.
   Global customizations normally occur before buffers are created, so
   before the (remove-hook 'foo 'bar 'local) has even a chance of
   being run.
   If you could show me something more concrete, I might better
   understand
   your concern.

OK.  Here is an ielm run:

*** Welcome to IELM ***  Type (describe-mode) for help.
ELISP> (default-value 'comint-output-filter-functions)
(comint-carriage-motion comint-postoutput-scroll-to-bottom)

ELISP> (default-value 'comint-output-filter-functions)
(ansi-color-process-output comint-carriage-motion
comint-postoutput-scroll-to-bottom)

ELISP>

What happened between line one and line two?
I looked at the following documentation.
Note: I definitely only looked at it, I was careful not to customize
anything.  To no avail.  Things like this happen all the time for the
reasons I outlined in my previous message.


Ansi Color For Comint Mode: Hide Value Menu Do nothing
   State: this option is unchanged from its standard setting.
Determines what to do with comint output. Hide
If nil, do nothing.
If the symbol `filter', then filter all SGR control sequences.
If anything else (such as t), then translate SGR control sequences
into text-properties.

In order for this to have any effect, `ansi-color-process-output' must
be in `comint-output-filter-functions'.

This can be used to enable colorized ls --color=yes output
in shell buffers.  You set this variable by calling one of:
M-x ansi-color-for-comint-mode-on
M-x ansi-color-for-comint-mode-off
M-x ansi-color-for-comint-mode-filter

\- Ansi Colors group end
-----------------------------------------------/

----:---F1  savefile       12:29PM 0.07   (Mail Abbrev Fill)--L32--All----------

Re: comint-carriage-motion causes severe problems.

[Miles Bader]

"Stefan Monnier" <monnier+gnu/emacs@rum.cs.yale.edu> writes:
> Global customizations normally occur before buffers are created, so
> before the (remove-hook 'foo 'bar 'local) has even a chance of being run.

Yes but as Kai noted `normally' isn't `always'.

It seems simpler and less likely to cause surprises to just always add
(not . FOO) if FOO isn't in the local hook, even if there's no FOO in
the global hook either -- what would be the disadvantage of that?

[of course, doing a local `add-hook' of FOO should remove the (not . FOO)!]

-Miles
-- 
`There are more things in heaven and earth, Horatio,
 Than are dreamt of in your philosophy.'

Re: comint-carriage-motion causes severe problems.

[Luc Teirlinck]

From my earlier message:
   
   (remove 'foo 'bar 'override) overrides any other occurrences of bar
   further down the list, local or global.

To avoid any confusion:

(remove 'foo 'bar 'override)

would do that in the local value of foo.  It would not affect the
global value.  (That is, of couse, the entire purpose.)

Sincerely,

Luc Teirlinck.

Re: comint-carriage-motion causes severe problems.

[Luc Teirlinck]

Dear Stefan,

Here is my more concrete proposal:

Here is your commented out code:


    ;; If the function is on the global hook, we need to shadow it
       locally
    ;;(when (and local (member function (default-value hook))
    ;;	           (not (member (cons 'not function) hook-value)))
    ;;  (push (cons 'not function) hook-value))

There actually is a problem with this.  We want locally nothing to do
with 'function'.  But if 'function' is not currently a member of the
default-value, any later adding to the global value will reintroduce
it.  So I would suggest to replace:

(and local (member function (default-value hook))
       (not ...

with:

(and (eq local 'override) (not ...))
               
I believe this single change might be sufficient to both implement my
previous suggestion and make sure that any later adding of 'function'
to the default-value will be shadowed locally.  If no shadowing of the
global value is wanted, we call with local equal to t or 'local,
whatever.  If we want the behavior your code implements, we call with
local equal to 
(if (member function (default-value hook)) 'override 'local)

So, unless I am missing something, I believe my suggested changes
would not force you to rewrite every detail of your code.  Just the one
small change above would suffice.  (Again, unless I am missing
something.)


What do you think?

Sincerely,

Luc Teirlinck.

Re: comint-carriage-motion causes severe problems.

[Luc Teirlinck]

Stefan Monnier wrote:

   > I believe this single change might be sufficient to both implement my
   > previous suggestion and make sure that any later adding of 'function'
   > to the default-value will be shadowed locally.  If no shadowing of the

   I thought about this alternative when I was hacking on it.
   Basically, you want that

	   (remove-hook 'foo 'bar 'local)
	   (add-hook 'foo 'bar)

   prevents `bar' from being run in the buffer.  But I'm not sure if
   it's such a good idea.  After all

	   (remove-hook 'foo 'bar)
	   (add-hook 'foo 'bar)

   does not prevent `bar' from being run (the `remove-hook' above ends up
   being a noop).


	   Stefan

Dear Stefan,

First of all, I messed up things when I sent you my previous message.
I somehow clumsily managed to accidentally remove the CC to
emacs-devel.  I resent my message to emacs-devel separately.  You
replied to the unintentionally "private" message.  So your reply did not
include a CC to emacs-devel either.  Sorry for messing things up.  I
included your entire message above, so you would not have to resend it
to emacs-devel.

I will respond to both your messages shortly after I have add some
time to think.

Sincerely,

Luc.

Re: comint-carriage-motion causes severe problems.

[Richard Stallman]

    Actually, there are some unclear semantics when you do things like

	    (add-hook 'foo 'bar)
	    (add-hook 'foo 'bar nil 'local)
	    (remove-hook 'foo 'bar 'local)

    The current code removes `bar' from the local value of `foo' but
    leaves it in the global one.  My new code does the same and only a second
    (remove-hook 'foo 'bar 'local) will add a (not . bar) such that the
    global setting is overridden.

This is semantically incoherent.  It is very dangerous for two
identical calls to remove-hook to be different in effect from one.

    I am starting to believe that it might be good to keep the present
    behavior of remove-hook, unless the third argument is the symbol
    'override, in which case it would override any function later in the
    hook, including global functions.

That gives a clean and predictable semantics for remove-hook.
However, this leaves the question of how you remove one of these
overrides.  Related question: how should add-hook deal with them?

This relates to another question: what should (remove-hook 'foo 'bar
'local) do after (remove-hook 'foo 'bar 'override)?  Nothing?  I don't
think that is right, because doing nothing would leave the override
marker in place; the state for `bar' would not be "no local hook".

There are three states that the local hook list can have, for any
function `bar':

1. Present
2. Absent
3. Overridden.

We could say that (add-hook 'foo 'bar 'local) puts it in state 1,
(remove-hook 'foo 'bar 'local) puts it in state 2, and (remove-hook
'foo 'bar 'override) puts it in state 3.  None of them changes the
global hook list, and global use of add-hook or remove-hook does not
change the local hook list.  Now we have fully coherent semantics
for both add-hook and remove-hook.

However, I think it is even cleaner to regard (not . bar) as a kind
of local hook value in its own right, and use add-hook to add that
and remove-hook to remove it.

(add-hook 'foo 'bar 'local) puts it in state 1
(remove-hook 'foo 'bar 'local) puts it in state 2
(add-hook 'foo '(not . bar) 'local) puts it in state 3
(remove-hook 'foo '(not . bar) 'local) puts it in state 2
so this is equivalent to (remove-hook 'foo 'bar 'local).

This is a coherent feature.  Is it really useful?  I am not sure.

The reason is, it is wrong to call comint-carriage-motion by adding it
globally and unconditionally to the hook.  When a function should be
called unconditionally, it should be called explicitly from the code,
not thru a hook.

Once this change is made, the way to fix the present problem is for
ielm to bind a variable that tells the code not to call
comint-carriage-motion, or tells comint-carriage-motion to do nothing.
The hook override feature won't be used here.

Do we have any other occasion to use the hook override feature?
Is that a useful feature?

Re: comint-carriage-motion causes severe problems.

[Luc Teirlinck]

Richard Stallman wrote:
   
   This is semantically incoherent.  It is very dangerous for two
   identical calls to remove-hook to be different in effect from one.

Indeed, what happens, for instance, if a file containing such a call gets
accidentally loaded a second time in the same session?

I start again to lean somewhat towards my original 'override proposal
with Richard's amendments, but I have no strong feelings about that.

Anyway, first we need to decide whether the feature is needed, then we
can worry about its concrete implementation.  I personally do not feel
like further discussing implementation before discussing desirability,
unless desirability depends on implementation.

First question:

What are hooks meant for?

From the Emacs Lisp manual:

Emacs provides hooks for the sake of customization.  Most often hooks
are set in the init file, but Lisp programs can set them also.

What Lisp programs are meant?

If these are user files or external packages or specialized very
optional packages and no major Emacs file is ever supposed to add
functions to a global hook, then indeed the new feature is not needed.

But if so, one has the problem that that convention is not followed in
the actual Emacs code.  In a previous message I gave an example of how
functions get into global hooks just by looking at documentation.

Given the fact that this is the actual behavior, I believe that we
need some form of the feature.  Let us first see whether or not we can
agree on that and then we can discuss implementation details.
   
   The reason is, it is wrong to call comint-carriage-motion by adding it
   globally and unconditionally to the hook.  When a function should be
   called unconditionally, it should be called explicitly from the code,
   not thru a hook.

I do not believe comint-carriage-motion should be called completely
unconditionally, not even in shell buffers, but I could temporarily
turn it off, using the variable in question, in shell or inferior Lisp
buffers anyway, so that is no problem.

But plenty of Emacs code would need to be changed if the above
principles should be followed rigorously everywhere.  Moreover, we
could wind up with plenty of turn-this-function-off variables.  Does
comint-postoutput-scroll-to-bottom belong in the hook or should it too
be called explicitly from code?  Is it OK for customize-browse to
unconditionally add ansi-color-process-output to the same global hook,
just because I happen to consult documentation?  (I do not mind about
ansi-color-process-output itself, it causes no harm, but it is a good
example.)

I agree that the turn-off variable solution provides an adequate
solution to the comint-ielm problem.  The new feature is more
convenient in that it only requires changes in the local specialized
package rather than in both the specialized and the general package.
However, it would make no sense to add a feature just for this one
problem, which can indeed easily be solved in other ways.

To summarize, I personally believe that there is a need for this
feature because of the ways hooks are used in practice in Emacs,
regardless of how they are theoretically supposed to be used.

Sincerely,

Luc.

Re: comint-carriage-motion causes severe problems.

[Richard Stallman]

    But plenty of Emacs code would need to be changed if the above
    principles should be followed rigorously everywhere.

This is an abstract theoretical question, and thus not relevant.
The relevant question here is a practical question:
do we need this feature *now*?

Re: comint-carriage-motion causes severe problems.

[Luc Teirlinck]

Richard Stallman wrote:

   Once this change is made, the way to fix the present problem is for
   ielm to bind a variable that tells the code not to call
   comint-carriage-motion, or tells comint-carriage-motion to do nothing.
   The hook override feature won't be used here.

   Do we have any other occasion to use the hook override feature?
   Is that a useful feature?

I am not too sure I answered this question adequately in my previous
message.

Again the proposed solution for the ielm-comint problem using a
variable is perfectly satisfactory.

Richard's question seems to be:

Do we know any other use for this feature in the Emacs source code?

I do not know any and if one crops up, we could always use the
variable trick again.

But that is not really what I believe the main question is.

The main question is: Is it useful for user customization?

If the user wants to call a hook function unconditionally, (s)he adds
it to the global hook.  If (s)he wants to add it to a few specialized
modes (s)he adds them to the corresponding local hooks.  What does
(s)he do if (s)he wants it in all but a few modes, where it
back-fires?  It seems inconvenient to add the function locally to all
modes (s)he could possibly ever use, including new modes in new emacs
releases, just to avoid enabling them in, say, one single mode.

In addition, the proposed variable trick requires changing the file
that sets the global value, which quite often is a file that is part
of Emacs.  With the feature a user (or somebody writing some
major-mode package) can easily override a locally inappropriate
function.  Using the variable trick would not work without altering
the Emacs source code.

I believe I allowed myself to get side-tracked and somewhat unfocused
in my previous message.

Sincerely,

Luc.

Re: comint-carriage-motion causes severe problems.

[Richard Stallman]

      What does
    (s)he do if (s)he wants it in all but a few modes, where it
    back-fires?  It seems inconvenient to add the function locally to all
    modes (s)he could possibly ever use, including new modes in new emacs
    releases, just to avoid enabling them in, say, one single mode.

This feature is clearly useful in theory.  If this case arises often
in practice, this feature is a good way to make it easier.  Let's
wait, before implementing it, to see if the need really arises in
practice.  There is no end of clean and theoretically useful features
we could implement, but it is not really a useful way to spend our
time.

Re: comint-carriage-motion causes severe problems.

[Luc Teirlinck]

Richard Stallman wrote (some time ago):

   The reason is, it is wrong to call comint-carriage-motion by adding it
   globally and unconditionally to the hook.  When a function should be
   called unconditionally, it should be called explicitly from the code,
   not thru a hook.

   Once this change is made, the way to fix the present problem is for
   ielm to bind a variable that tells the code not to call
   comint-carriage-motion, or tells comint-carriage-motion to do nothing.

This was written about a month ago and concerned problems caused by
comint-carriage-motion for ielm.  One thing was (and is) not
completely clear to me:

Would you like me to send  diffs for comint.el and ielm.el
implementing these changes?  I could easily do so.

Sincerely,

Luc.

Re: comint-carriage-motion causes severe problems.

[Richard Stallman]

    This was written about a month ago and concerned problems caused by
    comint-carriage-motion for ielm.  One thing was (and is) not
    completely clear to me:

    Would you like me to send  diffs for comint.el and ielm.el
    implementing these changes?  I could easily do so.

I would like someone to do it, and that could be you.
However, if they are nontrivial we would need to get new
legal papers from you.  This is because the papers you signed
(at least according to our file) in Jan 2001 only covered your
past changes.  If someone else who has assigned future changes
writes this, we won't need new papers from him.

Re: comint-carriage-motion causes severe problems.

[Luc Teirlinck]

Richard Stallman wrote:

   I would like someone to do it, and that could be you.
   However, if they are nontrivial we would need to get new
   legal papers from you.  This is because the papers you signed
   (at least according to our file) in Jan 2001 only covered your
   past changes.  If someone else who has assigned future changes
   writes this, we won't need new papers from him.

I believe the papers I signed back then concerned planned changes to
mailabbrev.el.

Current submissions or planned submissions include:

1.  Submitted changes to ielm.el concerning the variables *, ** and
    ***.  I submitted those a little over two weeks ago.  If you did not
    like the concrete changes incorporated in the diff I sent, then some
    other changes need to be made, since the two bugs I reported do need
    to be taken care of one way or the other.

2.  Maybe the comint-carriage-motion stuff, but it is not clear that
    this will take more than twenty lines, which if I remember well is
    about the limit.

3.  Combine mailalias and mailabbrev into a single package, where all
    mail alias definitions would be stored in .mailrc.

4.  A new file, tentatively called aliasfiles.el (name subject to
    change) which allows to write interactively defined mail aliases
    automatically into .mailrc without need for the user to visit that
    file or even worry about it.

5.  A new file called something like rc-mode.el (name subject to
    change) implementing a major mode to edit .mailrc directly.  (To
    get useful results from font-lock, syntax-motion, filling and the
    like.)

Timetable:

1.  Minus 18 days.

2.  A couple of days if it is decided that I am going to do it.

3, 4 and 5: Way more involved than 1. and 2. and took me way too long
(I had to interrupt my work on this to the point that I had to refresh
my Elisp knowledge when I finally found time to restart working on it,
which is how I ran into ielm bugs.)  However, I am now hopeful to
finish all of this off within about one month.

To summarize, I probably will need to sign new papers.  I believe I
remember that Gerd told me back in January 2001 that the papers I
signed would cover future work as well, but I deduce from your reply
that is not accurate.

Sincerely,

Luc.

Re: comint-carriage-motion causes severe problems.

[Luc Teirlinck]

I forgot to tell that my work on the mailalias-mailabbrev complex also
necessitates changes to sendmail.el and maybe message.el, as well as,
in addition, some small changes to rmail.el and certain other mail
related files to adapt them to the new combined package.

Sincerely,

Luc.

Re: comint-carriage-motion causes severe problems.

[Richard Stallman]

    To summarize, I probably will need to sign new papers.  I believe I
    remember that Gerd told me back in January 2001 that the papers I
    signed would cover future work as well, but I deduce from your reply
    that is not accurate.

The listing in our files says they do not cover future changes.
I will ask our clerk to verify one way or the other.

Re: comint-carriage-motion causes severe problems.

[Luc Teirlinck]

Richard Stallman wrote:

       This was written about a month ago and concerned problems caused by
       comint-carriage-motion for ielm.  One thing was (and is) not
       completely clear to me:

       Would you like me to send  diffs for comint.el and ielm.el
       implementing these changes?  I could easily do so.

   I would like someone to do it, and that could be you.
   However, if they are nontrivial we would need to get new
   legal papers from you.  This is because the papers you signed
   (at least according to our file) in Jan 2001 only covered your
   past changes.  If someone else who has assigned future changes
   writes this, we won't need new papers from him.

You should have received my signed papers by now.

Below are change logs and diffs for comint.el and ielm.el.
(For the latest CVS versions, 21.3.50)
I will send some additional remarks in a separate message.

Change logs:

2002-08-17  Luc Teirlinck  <teirllm@mail.auburn.edu>

	* comint.el: (no-carriage-motion): New variable. 
        (comint-carriage-motion, comint-output-filter-functions,
	comint-send-input, comint-output-filter): Remove
	comint-carriage-motion from comint-output-filter-functions and
	call it directly from comint-send-input an comint-output-filter.

        * ielm.el: (inferior-emacs-lisp-mode): Give no-carriage-motion
	a buffer local value of t.

Diffs:


===File ~/comint-diff=======================================
cd /usr/local/share/emacs/21.3.50/lisp/
diff -c /usr/local/share/emacs/21.3.50/lisp/comintold.el /usr/local/share/emacs/21.3.50/lisp/comint.el
*** /usr/local/share/emacs/21.3.50/lisp/comintold.el	Mon Jul  8 03:45:00 2002
--- /usr/local/share/emacs/21.3.50/lisp/comint.el	Fri Aug 16 21:42:01 2002
***************
*** 421,426 ****
--- 421,432 ----
    "Non-nil if you are accumulating input lines to send as input together.
  The command \\[comint-accumulate] sets this.")
  
+ (defvar no-carriage-motion nil
+   "If nil, `comint-carriage-motion' modifies comint output.
+ See the documentation string of that function for details.
+ You can overwrite that feature in specific comint applications or
+ buffers by giving this variable a non-nil buffer local value.")
+ 
  (put 'comint-replace-by-expanded-history 'menu-enable 'comint-input-autoexpand)
  (put 'comint-input-ring 'permanent-local t)
  (put 'comint-input-ring-index 'permanent-local t)
***************
*** 1508,1513 ****
--- 1514,1520 ----
  
  	  ;; This used to call comint-output-filter-functions,
  	  ;; but that scrolled the buffer in undesirable ways.
+ 	  (unless no-carriage-motion (comint-carriage-motion ""))
  	  (run-hook-with-args 'comint-output-filter-functions "")))))
  
  (defvar comint-preoutput-filter-functions nil
***************
*** 1545,1551 ****
  Make single carriage returns delete to the beginning of the line.
  Make backspaces delete the previous character.
  
! This function should be in the list `comint-output-filter-functions'."
    (save-match-data
      ;; We first check to see if STRING contains any magic characters, to
      ;; avoid overhead in the common case where it does not
--- 1552,1560 ----
  Make single carriage returns delete to the beginning of the line.
  Make backspaces delete the previous character.
  
! You can disable these features for specific comint applications or
! buffers by giving the variable `no-carriage-motion' a non-nil buffer
! local value."
    (save-match-data
      ;; We first check to see if STRING contains any magic characters, to
      ;; avoid overhead in the common case where it does not
***************
*** 1575,1582 ****
  	      (while (search-forward "\b" pmark t)
  		(delete-char -2)))))))))
  
- (add-hook 'comint-output-filter-functions 'comint-carriage-motion)
- 
  ;; The purpose of using this filter for comint processes
  ;; is to keep comint-last-input-end from moving forward
  ;; when output is inserted.
--- 1584,1589 ----
***************
*** 1660,1665 ****
--- 1667,1673 ----
  	    ;; Advance process-mark
  	    (set-marker (process-mark process) (point))
  
+ 	    (unless no-carriage-motion (comint-carriage-motion string))
  	    (run-hook-with-args 'comint-output-filter-functions string)
  	    (goto-char (process-mark process)) ; in case a filter moved it
  

Diff finished at Sat Aug 17 20:26:22
============================================================


===File ~/ielmdiff2=========================================
cd /usr/local/share/emacs/21.3.50/lisp/
diff -c /usr/local/share/emacs/21.3.50/lisp/ielmold.el /usr/local/share/emacs/21.3.50/lisp/ielm.el
*** /usr/local/share/emacs/21.3.50/lisp/ielmold.el	Fri Aug 16 21:06:02 2002
--- /usr/local/share/emacs/21.3.50/lisp/ielm.el	Fri Aug 16 21:14:49 2002
***************
*** 464,469 ****
--- 464,471 ----
      (comint-output-filter (ielm-process) ielm-prompt)
      (set-marker comint-last-input-start (ielm-pm))
      (set-process-filter (get-buffer-process (current-buffer)) 'comint-output-filter))
+   (make-local-variable 'no-carriage-motion)
+   (setq no-carriage-motion t)
    (run-hooks 'ielm-mode-hook))
  
  (defun ielm-get-old-input nil

Diff finished at Sat Aug 17 20:30:37
============================================================

Re: comint-carriage-motion causes severe problems.

[Luc Teirlinck]

Two more remarks on comint.el.  They are not directly related to the
changes proposed in my previous message, but I noticed them while
making those changes.

First remark:

Old version, lines 1509-1511:

          ;; This used to call comint-output-filter-functions,
          ;; but that scrolled the buffer in undesirable ways.
          (run-hook-with-args 'comint-output-filter-functions "")))))

New version:

          ;; This used to call comint-output-filter-functions,
          ;; but that scrolled the buffer in undesirable ways.
+         (unless no-carriage-motion (comint-carriage-motion ""))
          (run-hook-with-args 'comint-output-filter-functions "")))))

This seems cryptic and confusing:

"This USED to call comint-output-filter-functions":

I do not understand, it clearly still does.

Second remark:

While byte-compiling the compiler got upset over both comint.el and
ielm.el's use of directory-sep-char: 

^L
Compiling file /usr/local/share/emacs/21.3.50/lisp/comint.el at Sat
Aug 17 21:47:39 2002

/usr/local/share/emacs/21.3.50/lisp/comint.el:742:29:
Warning: directory-sep-char is an obsolete variable since 21.1; do not
use it.

In comint-dynamic-complete-as-filename:
Warning: directory-sep-char is an obsolete variable since 21.1; do not
use it.

^L
Compiling file /usr/local/share/emacs/21.3.50/lisp/ielm.el at Sat Aug
17 21:49:00 2002

In inferior-emacs-lisp-mode:
Warning: directory-sep-char is an obsolete variable since 21.1; do not
use it.

From the NEWS:

** The variable `directory-sep-char' is slated for removal.
Not really a change (yet), but a projected one that you should be
aware of: The variable `directory-sep-char' is deprecated, and should
not be used.  It was always ignored on GNU/Linux and Unix systems and
on MS-DOS, but the MS-Windows port tried to support it by adapting the
behavior of certain primitives to the value of this variable.  It
turned out that such support cannot be reliable, so it was decided to
remove this variable in the near future.  Lisp programs are well
advised not to set it to anything but '/', because any different value
will not have any effect when support for this variable is removed.


It is clear that all of this is only relevant to MS-Windows.  I do not
use MS-Windows and have very limited experience with it.  Hence I do
not know how relevant the above is and whether the references to this
variable should be removed as the compiler seems to wish.

Sincerely,

Luc.

Re: comint-carriage-motion causes severe problems.

[Luc Teirlinck]

On comint.el:

Old version, lines 1509-1511:

          ;; This used to call comint-output-filter-functions,
          ;; but that scrolled the buffer in undesirable ways.
          (run-hook-with-args 'comint-output-filter-functions "")))))

New version:

          ;; This used to call comint-output-filter-functions,
          ;; but that scrolled the buffer in undesirable ways.
+         (unless no-carriage-motion (comint-carriage-motion ""))
          (run-hook-with-args 'comint-output-filter-functions "")))))

I just realized that adding the line:

+         (unless no-carriage-motion (comint-carriage-motion ""))

here is unnecessary (though harmless), because calling
comint-carriage-motion on the empty string has no effect whatsoever.
Sorry for not noticing this earlier.  As mentioned in my previous
message, I am intrigued by the remark in this extract.

Revised change log:

2002-08-17  Luc Teirlinck  <teirllm@mail.auburn.edu>

	* comint.el: (no-carriage-motion): New variable.
        (comint-carriage-motion, comint-output-filter-functions,
        comint-output-filter): Remove comint-carriage-motion from
        comint-output-filter-functions and call it directly from
        comint-output-filter.

	* ielm.el: (inferior-emacs-lisp-mode): Give no-carriage-motion
	a buffer local value of t.


Sincerely,

Luc.

Re: comint-carriage-motion causes severe problems.

[Miles Bader]

[-- Attachment #1: Type: text/plain, Size: 244 bytes --]

Luc Teirlinck <teirllm@dms.auburn.edu> writes:
> Below are change logs and diffs for comint.el and ielm.el.

I've made equivalent changes in CVS (basically the same changes you
made, but some of the details are different).

Here's the patch:



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Patch to fix carriage motion problems in ielm --]
[-- Type: text/x-patch, Size: 6643 bytes --]

2002-08-19  Miles Bader  <miles@gnu.org>

	[original idea from Luc Teirlinck <teirllm@mail.auburn.edu>]
	* comint.el (comint-inhibit-carriage-motion): New variable.
	(comint-carriage-motion): Argument STRING removed.  New arguments
	START and END; interpret characters between START and END rather
	than using special comint state.
	(comint-output-filter): Call `comint-carriage-motion'.
	(comint-output-filter-functions): Don't add `comint-carriage-motion'.
	* ielm.el (inferior-emacs-lisp-mode): Give
	`comint-inhibit-carriage-motion' a local value of t.

Index: lisp/comint.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/comint.el,v
retrieving revision 1.281
diff -u -r1.281 comint.el
--- lisp/comint.el	8 Jul 2002 08:45:00 -0000	1.281
+++ lisp/comint.el	19 Aug 2002 05:00:08 -0000
@@ -1525,6 +1525,10 @@
 You can use `add-hook' to add functions to this list
 either globally or locally.")
 
+(defvar comint-inhibit-carriage-motion nil
+  "If nil, comint will interpret `carriage control' characters in output.
+See `comint-carriage-motion' for details.")
+
 ;; When non-nil, this is an overlay over the last recognized prompt in
 ;; the buffer; it is used when highlighting the prompt.
 (defvar comint-last-prompt-overlay nil)
@@ -1539,43 +1543,38 @@
                            (overlay-end comint-last-prompt-overlay)
                            (overlay-properties comint-last-prompt-overlay)))))
 
-(defun comint-carriage-motion (string)
-  "Handle carriage control characters in comint output.
+(defun comint-carriage-motion (start end)
+  "Interpret carriage control characters in the region from START to END.
 Translate carriage return/linefeed sequences to linefeeds.
 Make single carriage returns delete to the beginning of the line.
-Make backspaces delete the previous character.
-
-This function should be in the list `comint-output-filter-functions'."
-  (save-match-data
-    ;; We first check to see if STRING contains any magic characters, to
-    ;; avoid overhead in the common case where it does not
-    (when (string-match "[\r\b]" string)
-      (let ((pmark (process-mark (get-buffer-process (current-buffer)))))
-	(save-excursion
-	  (save-restriction
-	    (widen)
-	    (let ((inhibit-field-text-motion t)
-		  (buffer-read-only nil))
-	      ;; CR LF -> LF
-	      ;; Note that this won't work properly when the CR and LF
-	      ;; are in different output chunks, but this is probably an
-	      ;; exceedingly rare case (because they are generally
-	      ;; written as a unit), and to delay interpretation of a
-	      ;; trailing CR in a chunk would result in odd interactive
-	      ;; behavior (and this case is probably far more common).
-	      (goto-char comint-last-output-start)
-	      (while (re-search-forward "\r$" pmark t)
-		(delete-char -1))
-	      ;; bare CR -> delete preceding line
-	      (goto-char comint-last-output-start)
-	      (while (search-forward "\r" pmark t)
-		(delete-region (point) (line-beginning-position)))
-	      ;; BS -> delete preceding character
-	      (goto-char comint-last-output-start)
-	      (while (search-forward "\b" pmark t)
-		(delete-char -2)))))))))
-
-(add-hook 'comint-output-filter-functions 'comint-carriage-motion)
+Make backspaces delete the previous character."
+  (save-excursion
+    ;; First do a quick check to see if there are any applicable
+    ;; characters, so we can avoid calling save-match-data and
+    ;; save-restriction if not.
+    (when (< (skip-chars-forward "^\b\r" end) (- end start))
+      (save-match-data
+	(save-restriction
+	  (widen)
+	  (let ((inhibit-field-text-motion t)
+		(buffer-read-only nil))
+	    ;; CR LF -> LF
+	    ;; Note that this won't work properly when the CR and LF
+	    ;; are in different output chunks, but this is probably an
+	    ;; exceedingly rare case (because they are generally
+	    ;; written as a unit), and to delay interpretation of a
+	    ;; trailing CR in a chunk would result in odd interactive
+	    ;; behavior (and this case is probably far more common).
+	    (while (re-search-forward "\r$" end t)
+	      (delete-char -1))
+	    ;; bare CR -> delete preceding line
+	    (goto-char start)
+	    (while (search-forward "\r" end t)
+	      (delete-region (point) (line-beginning-position)))
+	    ;; BS -> delete preceding character
+	    (goto-char start)
+	    (while (search-forward "\b" end t)
+	      (delete-char -2))))))))
 
 ;; The purpose of using this filter for comint processes
 ;; is to keep comint-last-input-end from moving forward
@@ -1660,7 +1659,12 @@
 	    ;; Advance process-mark
 	    (set-marker (process-mark process) (point))
 
+	    (unless comint-inhibit-carriage-motion
+	      ;; Interpret any carriage motion characters (newline, backspace)
+	      (comint-carriage-motion comint-last-output-start (point)))
+
 	    (run-hook-with-args 'comint-output-filter-functions string)
+
 	    (goto-char (process-mark process)) ; in case a filter moved it
 
 	    (unless comint-use-prompt-regexp-instead-of-fields
Index: lisp/ielm.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/ielm.el,v
retrieving revision 1.26
diff -u -r1.26 ielm.el
--- lisp/ielm.el	15 Jun 2002 14:17:24 -0000	1.26
+++ lisp/ielm.el	19 Aug 2002 05:00:08 -0000
@@ -1,6 +1,6 @@
 ;;; ielm.el --- interaction mode for Emacs Lisp
 
-;; Copyright (C) 1994 Free Software Foundation, Inc.
+;; Copyright (C) 1994, 2002 Free Software Foundation, Inc.
 
 ;; Author: David Smith <maa036@lancaster.ac.uk>
 ;; Maintainer: FSF
@@ -452,18 +452,24 @@
 	'(ielm-font-lock-keywords nil nil ((?: . "w") (?- . "w") (?* . "w"))))
 
   ;; A dummy process to keep comint happy. It will never get any input
-  (if (comint-check-proc (current-buffer)) nil
+  (unless (comint-check-proc (current-buffer))
     ;; Was cat, but on non-Unix platforms that might not exist, so
     ;; use hexl instead, which is part of the Emacs distribution.
     (start-process "ielm" (current-buffer) "hexl")
     (process-kill-without-query (ielm-process))
     (goto-char (point-max))
+    
+    ;; Lisp output can include raw characters that confuse comint's
+    ;; carriage control code.
+    (set (make-local-variable 'comint-inhibit-carriage-motion) t)
+
     ;; Add a silly header
     (insert ielm-header)
     (ielm-set-pm (point-max))
     (comint-output-filter (ielm-process) ielm-prompt)
     (set-marker comint-last-input-start (ielm-pm))
     (set-process-filter (get-buffer-process (current-buffer)) 'comint-output-filter))
+
   (run-hooks 'ielm-mode-hook))
 
 (defun ielm-get-old-input nil

[-- Attachment #3: Type: text/plain, Size: 70 bytes --]



-Miles
-- 
Saa, shall we dance?  (from a dance-class advertisement)

Re: comint-carriage-motion causes severe problems.

[Luc Teirlinck]

From my earlier posting:
    
    (setq comint-output-filter-functions
	  (default-value 'comint-output-filter-functions))

    (remove-hook 'comint-output-filter-functions 'comint-carriage-motion t)

To avoid possible confusion:

These lines are, of course, meant to be included in ielm.el, where
comint-output-filter-functions is already made local.  Hence, setq
sets the local value.

Sincerely,

Luc Teirlinck.

Re: comint-carriage-motion causes severe problems.

[Richard Stallman]

You wrote,

    Many interactive programs in unix use CR and BS characters in their
    output to try to update a single line of output (e.g., in progress
    indicators).

However, Teirlinck reports this:

      I know programs which, when ran from a shell-buffer will
    intersperse their output with repeats and ^H's.  However,
    comint-carriage-motion makes these problems worse rather than solving
    them.  Without the function, the output looks ugly, but is
    readable. With the function, the output is unrecognizable.

It seems that you both agree that comint-carriage-motion
should be turned off in ielm, but disagree about what its
effects are in shell buffers.

Luc, could you please give some examples of programs where
comint-carriage-motion makes things worse, and show what the output
looks like both with and without comint-carriage-motion?

Miles, could you please show an example where the feature is
beneficial?

By looking at them, it may perhaps be possible to figure out a
heuristic for handling more of the cases in a reasonable way.

Re: comint-carriage-motion causes severe problems.

[Miles Bader]

Richard Stallman <rms@gnu.org> writes:
> Miles, could you please show an example where the feature is
> beneficial?

[BTW, Luc has already agreed, in another message, that default should be
 on, with ielm turning locallying disabling it.]

Some examples that I use frequently are `apt-get' (the debian package
updating program), and `lftp' (an ftp program).  Both of these use BS
and CR (actually, I think apt-get uses only CR, whereas lftp uses only
BS) to update progress displays.  Such usage is fairly common (for
instance most value-added ftp clients implement it).

-Miles
-- 
Would you like fries with that?

Re: comint-carriage-motion causes severe problems.

[Miles Bader]

BTW, I should note that the interpretation of BS and CR done by
comint-carriage-motion is almost guaranteed to be the right thing for
programs running in a shell, since it simulates what a terminal does.

Only programs like ielm [and it's the only one I'm aware of] that
_expect_ to be running emacs, and have non-printable characters
displayed `readably', will be screwed up by this interpretation.

-Miles
-- 
97% of everything is grunge

Re: comint-carriage-motion causes severe problems.

[Richard Stallman]

    Only programs like ielm [and it's the only one I'm aware of] that
    _expect_ to be running emacs, and have non-printable characters
    displayed `readably', will be screwed up by this interpretation.

They could locally set a variable that tells this function to do
nothing.  That requires no special mechanism.

Re: comint-carriage-motion causes severe problems.

[Luc Teirlinck]

Richard Stallman wrote:

   Luc, could you please give some examples of programs where
   comint-carriage-motion makes things worse, and show what the output
   looks like both with and without comint-carriage-motion?

As mentioned in my earlier message, I now agree with Miles that
comint-carriage-motion should be in the global value of
comint-output-filter-functions, but not in the local ielm value.

Miles says that the function is necessary for the correct functioning
of interactive Unix programs.  The program I was referring to, where
comint-carriage-motion makes a bad situation even substantially worse,
is the mathematical software package magma.  I could send you examples
of runs if this is still relevant or of interest to you or
Miles.  However, the correct running of Unix programs in M-x shell is
more important than the correct running of specialized software
packages that could only be made to run correctly by writing a
specialized mode especially for them.  (They would require their own
highly specialized and complex syntax-table and syntactic rules, among
other things.)

I was actually less concerned with making mathematical software
packages run correctly (which would require tons of work) than with
the same type of problems that occur in ielm.  But in other applications
of comint control characters meant to be printed as such occur very
infrequently, whereas in ielm they are routine.

What happens is that I (just every now and then) have C programs or
Common Lisp programs yielding control characters that need to be
printed out as such.  I can not run these programs from a shell in an
Xterm (unless I redirect their output to a file) because the shell
itself mutilates their output just like comint-carriage-motion does.
Thus I run them from M-x shell or M-x run lisp (obviously with
comint-carriage-motion disabled) to see the correct output
interactively, without having to look it up in a separate file.  We
are in effect talking here about using M-x shell and M-x run-lisp to
improve shell behavior rather than to just imitate it.  But these
improvements are actually necessary to enable Emacs to understand the
syntax correctly which is absolutely essential, certainly in M-x
run-lisp, for proper functioning of font-lock, show-paren, and
several motion and other commands relying on syntax.  (For exactly the
same reasons as in ielm.)

However, I fully agree that the above is highly specialized and
a-typical usage.  Hence I agree with Miles.

Maybe one little detail:

Would it not be somewhat better and more logical to put, in comint.el,
comint-carriage-motion in the defvar of
comint-output-filter-functions, rather than using a separate add-hook
for it? 

This would have two advantages:

1. It would make the comint.el code easier to understand.

2. As noted above, while I agree that comint-carriage-motion belongs
   in the global default value of comint-output-filter-functions,
   there are, for the reasons outlined above, definitely situations
   where one, at least temporarily, wants to remove it.  In the event
   that, in some way or another, comint.el gets loaded again, the defvar
   will not re-add comint-carriage-motion secretly behind the user's
   back, but the add-hook will.  Essentially, we are talking about the
   same difference as the one between defvar and setq. 

However, if this change would cause any problems, then it is not that
important.  Require and provide make a reloading of comint unlikely,
assuming all comint using code uses them rigidly.

Sincerely,

Luc Teirlinck.