bug in ielm: patch. Related problem with end-of-line.

bug in ielm: patch. Related problem with end-of-line.

[Luc Teirlinck]

C-c RETURN in ielm on the first input line copies the old prompt to
the current input instead of the old input.  This is related to some
strange subtleties with end-of-line.  The patch below fixes the ielm
bug.  It does so by treating the ielm header as output.  I believe
this makes sense anyway and is consistent with what M-x run-lisp does.

However, the patch fixes one symptom of a more general problem with
end-of-line (as well as maybe with some of its friends), without
really solving the more general problem itself.

The output of C-h v end-of-line contains the following line:

if N is nil or 1, and point starts at a field boundary, point does not
move.

This is not quite accurate: there is an exception.  Point does not
move *unless* the previous character has a non-nil rear-nonsticky
text-property *and* a text-property field eq to boundary.  Then point
*does* move over the field.  Comint and its derived modes rely heavily
on this exception.

The problem with ielm was that ielm-get-old-input uses
comint-skip-prompt which uses end-of-line at the beginning of a
prompt.  At the second and later prompts, comint-skip-prompt indeed
skips over the prompt, because point is not at a field boundary if
there is preceding output, and because of the above mentionned
exception, if the output was flushed.  However, because the current
ielm header has no text properties, the beginning of the first prompt
is at an "unexceptional" field boundary and point remains "frozen",
leading to the bug.  My patch treats the header as output.  Now the
first prompt is no longer at a field boundary and the bug is gone.

Not all problems are gone, however.  The ielm-type patch can be applied
to any comint-buffer with a header.  Some, like M-x run-lisp, already
treat the header as output.  The one remaining problem occurs at the
beginning of the buffer, header or no header.

In comint buffers with headers, treated as output, there now is the
problem that C-e at the beginning of the buffer stays put instead of
moving to the end of the line.  This is less serious than copying an
old (read-only) prompt into the current input, but is still annoying.
(For ordinary output, the exceptional text-property combination
assures proper C-e functionning, even at the beginning of the output.)

For *shell* buffers the problem is worse.  They start with a prompt.
One, in its own right not too serious, but still confusing, problem is
inconsistent C-e behavior.  Go to a shell buffer, with at least two
prompts.  Do C-e at the first prompt.  Point stays put.  However, at
all other prompts, C-e moves over the prompt to the input field.

Much more serious than this are the potential related problems with
end-of-line called from programs.  Although I currently know of no
concrete bugs resulting from the situation, other than the
inconsistent C-e behavior, the danger for future (or undiscovered
present) bugs is very great.  Indeed, any calling of end-of-line, with
inhibit-field-text-motion eq to nil, at the beginning of a prompt in a
*shell*-type buffer, without checking whether the prompt is the first
prompt or not, is a bug: either it expects point to move over the
prompt and then it malfunctions when called at the first prompt, or it
expects point to stay put and then it malfunctions at all other
prompts.

One solution would be a special text-property at the beginning of the
buffer, determining whether the beginning of the buffer should be
treated as a field boundary (for purposes of C-e and friends) or
not. That would solve all above small actual (C-e at the beginning of
a comint buffer) and potential larger problems.  I do not know how
easy or difficult this would be to implement.  Another solution would
be not to introduce any new text properties, but to allow
text-properties "at position 0", which of course would only be
relevant for inheritance and problems of the above type.  Maybe the
problem can be solved in an easier way.

On a related note, finding documentation for text properties seems to
be quite non-trivial.  My patch uses a text property
inhibit-line-move-field-capture.  I am sure it must be used, because
it is used for all comint output fields, but I was unable to find any
information about that property in the C-h n output or in the latest
CVS Elisp manual, or in the documentation strings of several
field-motion related functions, or anywhere else.  Where can I find
some documentation about this text property?

Next, change log and patch:

Change log:

2002-09-22  Luc Teirlinck  <teirllm@mail.auburn.edu>

       * ielm.el (inferior-emacs-lisp-mode): Treat the header as output.

Patch:


===File ~/ielm-diff=========================================
cd /usr/local/share/emacs/21.3.50/lisp/
diff -c /usr/local/share/emacs/21.3.50/lisp/ielmold.el /usr/local/share/emacs/21.3.50/lisp/ielm.el
*** /usr/local/share/emacs/21.3.50/lisp/ielmold.el	Thu Sep 19 10:35:10 2002
--- /usr/local/share/emacs/21.3.50/lisp/ielm.el	Sun Sep 22 18:02:21 2002
***************
*** 510,515 ****
--- 510,518 ----
      ;; Add a silly header
      (insert ielm-header)
      (ielm-set-pm (point-max))
+     (add-text-properties
+      1 (point-max)
+      '(rear-nonsticky t field output inhibit-line-move-field-capture t))
      (comint-output-filter (ielm-process) ielm-prompt)
      (set-marker comint-last-input-start (ielm-pm))
      (set-process-filter (get-buffer-process (current-buffer)) 'comint-output-filter))

Diff finished at Sun Sep 22 19:05:41
============================================================

Re: bug in ielm: patch. Related problem with end-of-line.

[Luc Teirlinck]

Sorry, I forgot to properly account for
comint-use-prompt-regexp-instead-of-fields in my previous patch.

Corrected change log and patch:

Change log:

2002-09-22  Luc Teirlinck  <teirllm@mail.auburn.edu>

       * ielm.el (inferior-emacs-lisp-mode): Treat the header as
         output, if comint-use-prompt-regexp-instead-of-fields is nil.

Patch:

===File ~/ielm-diff2========================================
cd /usr/local/share/emacs/21.3.50/lisp/
diff -c /usr/local/share/emacs/21.3.50/lisp/ielmold.el /usr/local/share/emacs/21.3.50/lisp/ielm.el
*** /usr/local/share/emacs/21.3.50/lisp/ielmold.el	Thu Sep 19 10:35:10 2002
--- /usr/local/share/emacs/21.3.50/lisp/ielm.el	Sun Sep 22 22:26:24 2002
***************
*** 510,515 ****
--- 510,519 ----
      ;; Add a silly header
      (insert ielm-header)
      (ielm-set-pm (point-max))
+     (unless comint-use-prompt-regexp-instead-of-fields
+       (add-text-properties
+        1 (point-max)
+        '(rear-nonsticky t field output inhibit-line-move-field-capture t)))
      (comint-output-filter (ielm-process) ielm-prompt)
      (set-marker comint-last-input-start (ielm-pm))
      (set-process-filter (get-buffer-process (current-buffer)) 'comint-output-filter))

Diff finished at Sun Sep 22 22:39:51
============================================================

Re: bug in ielm: patch. Related problem with end-of-line.

[Luc Teirlinck]

   The output of C-h v end-of-line contains the following line:

   if N is nil or 1, and point starts at a field boundary, point does not
   move.

   This is not quite accurate: there is an exception.  Point does not
   move *unless* the previous character has a non-nil rear-nonsticky
   text-property *and* a text-property field eq to boundary.  Then point
   *does* move over the field.  Comint and its derived modes rely heavily
   on this exception.

Actually, although the description of the situation is correct, my
interpretation of it was, I believe wrong.

If point is at the beginning of a buffer, or after a character with a
rear-nonsticky field property, different from the following
character's field property, then I believe C-e should move point over
the following field, regardless of whether the previous character's
field property is `boundary'.  I checked that this actually was the
behavior in emacs-21.2.90.  The problems I described do not occur in
21.2.90.  So maybe it might be good to revert to the 21.2.90 behavior.

Sincerely,

Luc.

Re: bug in ielm: patch. Related problem with end-of-line.

[Luc Teirlinck]

More precisely, the function that got modified between 21.2.90 and the
latest CVS is `field-end'.  At the beginning of a shell buffer,
(field-end) returns 1 in the latest CVS and the position of the end of
the prompt in emacs-21.2.90.

Sincerely,

Luc.

Re: bug in ielm: patch. Related problem with end-of-line.

[Luc Teirlinck]

From my previous message:

   So maybe it might be good to revert to the 21.2.90 behavior.

Actually in 21.2.90 C-e moved over the next field and (field-end)
returned the end of the next field (at a field boundary) even if the
previous character did have a rear-sticky field property.  Maybe this
part of the 21.2.90 behavior was incorrect, which may have been why
it got changed.  However, I believe that the old behavior should have
been maintained at the beginning of a buffer or after a character with
a rear-nonsticky field property of any kind.

Sincerely,

Luc.

Re: bug in ielm: patch. Related problem with end-of-line.

[Luc Teirlinck]

From my previous message:

   If point is at the beginning of a buffer, or after a character with a
   rear-nonsticky field property, different from the following
   character's field property, then I believe C-e should move point over
   the following field, regardless of whether the previous character's
   field property is `boundary'.  I checked that this actually was the
   behavior in emacs-21.2.90.  The problems I described do not occur in
   21.2.90.  So maybe it might be good to revert to the 21.2.90 behavior.

Actually my description of the emacs-21.2.90 behavior here and in some
subsequent messages was wrong.  The emacs-21.2.90 behavior of C-e with
respect to text-properties actually seems identical to the
emacs-21.3.50 behavior.  I was confused by the fact that in
emacs-21.3.50 comint used overlays instead of text properties.  That
seems to be why the described problem did not occur in 21.2.90, at
least not in comint buffers.

Sorry for the confusion.

Sincerely,

Luc.

Re: bug in ielm: patch. Related problem with end-of-line.

[Luc Teirlinck]

From my previous message:

   I was confused by the fact that in
   emacs-21.3.50 comint used overlays instead of text properties.

Of course, I meant: in emacs-21.2.90 comint used overlays instead ot
text-properties (for fields).

Sincerely,

Luc.