bug#846: Bug in handling invisible text, and bug in Pmail.

bug#846: Bug in handling invisible text, and bug in Pmail.

[Richard M. Stallman]

In Pmail I had a message that looked like this
(I changed his host name for privacy's sake):

----------------------------------------------------------------------
To: rms@gnu.org
Subject: sorry that I was distracted with phone calls 
From: Brand@xyzxysxysbb.com
Reply-to: brand@xyzxysxysbb.com
Date: Thu, 21 Aug 2008 18:46:37 -0700
X-BABYL-V6-ATTRIBUTES: A------



Two other ideas to run past you when time permits.

/w

----------------------------------------------------------------------

and I wanted to copy that into an outgoing message.
So I typed C-x h M-w, went to *mail* and inserted it.
What this inserted appears below.  It was quite a surprise.

It seems there is a lot of invisible text at the beginning of
the buffer, and two different values of point display at the same
place on the screen.  These places are 934995 and 9351405.
(point-min) returns 934995, and that's where M-< moves point to.
Typing C-f C-b moves point to 9351405.  I gather that those two
positions surround the invisible text.	

I think this is a bug in the command loop's handling of invisible
text -- it should not be the case that C-f C-b changes point.

I think it is also a bug in Pmail.  Pmail should set the buffer boundaries
after that text, rather than making it invisible.

If Pmail is fixed, then it won't trigger the invisible text bug.
But I think it needs to be fixed anyway.


----------------------------------------------------------------------
From Brand@xyzxysxysbb.com  Fri Aug 22 01:11:22 2008
X-Coding-System: undecided-unix
Return-path: <Brand@xyzxysxysbb.com>
Envelope-to: rms@gnu.org
Delivery-date: Fri, 22 Aug 2008 01:11:22 -0400
Received: from mx10.gnu.org ([199.232.76.166]:43206)
	by fencepost.gnu.org with esmtp (Exim 4.67)
	(envelope-from <Brand@xyzxysxysbb.com>)
	id 1KWOvi-0001Cl-Jw
	for rms@gnu.org; Fri, 22 Aug 2008 01:11:22 -0400
Received: from Debian-exim by monty-python.gnu.org with spam-scanned (Exim 4.60)
	(envelope-from <Brand@xyzxysxysbb.com>)
	id 1KWOwz-0000Ti-Nv
	for rms@gnu.org; Fri, 22 Aug 2008 01:12:44 -0400
X-Spam-Checker-Version: SpamAssassin 3.1.0 (2005-09-13) on monty-python
X-Spam-Level: 
X-Spam-Status: No, score=-1.7 required=5.0 tests=AWL,BAYES_00,
	FORGED_RCVD_HELO,NO_REAL_NAME autolearn=no version=3.1.0
Received: from sh.day.org ([198.102.73.65]:47087)
	by monty-python.gnu.org with esmtp (Exim 4.60)
	(envelope-from <Brand@xyzxysxysbb.com>)
	id 1KWOwz-0000T4-E6
	for rms@gnu.org; Fri, 22 Aug 2008 01:12:41 -0400
X-Client-Authorized: MaGic Cook1e
Received: from vree.xyzxysxysbb.com (sh.day.org [198.102.73.65])
	by sh.day.org (Postfix) with ESMTPS id 18A9EC6988A39
	for <rms@gnu.org>; Thu, 21 Aug 2008 22:12:38 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1])
	by vree.xyzxysxysbb.com (8.13.7/8.13.7) with ESMTP id m7M1kbEs004491
	for <rms@gnu.org>; Thu, 21 Aug 2008 18:46:46 -0700
Message-Id: <200808220146.m7M1kbEs004491@vree.xyzxysxysbb.com>
To: rms@gnu.org
Subject: sorry that I was distracted with phone calls 
From: Brand@xyzxysxysbb.com
Reply-to: brand@xyzxysxysbb.com
Date: Thu, 21 Aug 2008 18:46:37 -0700
X-detected-kernel: by monty-python.gnu.org: Genre and OS details not recognized.
X-BABYL-V6-ATTRIBUTES: A------



Two other ideas to run past you when time permits.

/w

----------------------------------------------------------------------

bug#846: Bug in handling invisible text, and bug in Pmail.

[Stefan Monnier]

> I think this is a bug in the command loop's handling of invisible
> text -- it should not be the case that C-f C-b changes point.

There are various desirable behaviors w.r.t invisible text, and you
can't have them all at the same time.  I.e. the above "bug" is
a conscious decision.

It's been with us for a while now and hasn't caused any real bug report
yet (other than reports such as this one, that is; which complain on
principle rather than because of some real problem introduced by this
behavior).  So I'd rather leave it as is.

Now as for the Pmail-specific part of your bug-report, I can't help you
right now.  Is this different from Rmail's behavior?


        Stefan

bug#846: Bug in handling invisible text, and bug in Pmail.

[Richard M. Stallman]

    > I think this is a bug in the command loop's handling of invisible
    > text -- it should not be the case that C-f C-b changes point.

    There are various desirable behaviors w.r.t invisible text, and you
    can't have them all at the same time.

I know that better than anyone, and I thought about this a long time.
The command loop used to have code to make sure that there weren't
two different possible point values that would display as the same place.

    It's been with us for a while now

Did you change my decision?  If so, when did that happen?

				      and hasn't caused any real bug report
    yet (other than reports such as this one, that is; which complain on
    principle rather than because of some real problem

It was a real problem, even though Pmail should not use invisible text
for this.  It led to confusing behavior in this case, and it will be
confusing in every case where invisible text is used.

If people have not complained, maybe it is because the use of 
invisible text is rare.  What other modes use invisible text?

bug#846: Bug in handling invisible text, and bug in Pmail.

[Stefan Monnier]

>> I think this is a bug in the command loop's handling of invisible
>> text -- it should not be the case that C-f C-b changes point.

>     There are various desirable behaviors w.r.t invisible text, and you
>     can't have them all at the same time.

> I know that better than anyone, and I thought about this a long time.
> The command loop used to have code to make sure that there weren't
> two different possible point values that would display as the same place.

When was that?  How did it do that?

>     It's been with us for a while now
> Did you change my decision?

What decision?

> If so, when did that happen?

The relevant piece of code is adjust_point_for_property, which used to
make point move over images and char-compositions and which I extended
in Emacs-22 (IIRC) to move over invisible text as well (making it
automatically somewhat-intangible).

> 				      and hasn't caused any real bug report
>     yet (other than reports such as this one, that is; which complain on
>     principle rather than because of some real problem

> It was a real problem, even though Pmail should not use invisible text
> for this.  It led to confusing behavior in this case, and it will be
> confusing in every case where invisible text is used.

Could you re-explain the problem you encountered, then?  Re-reading your
original bug-report I still can't see where this difference between point
before and after C-f C-b introduced a problem.

> If people have not complained, maybe it is because the use of 
> invisible text is rare.  What other modes use invisible text?

Indeed, most complaints around invisible text have to do with the
invisibility itself rather than details of how point is handed in
its presence.  So invisible text is rare because it's inherently
problematic (and should hence be avoided whenever possible).


        Stefan

bug#846: Bug in handling invisible text, and bug in Pmail.

[Reiner Steib]

On Tue, Sep 02 2008, Stefan Monnier wrote:

>> If people have not complained, maybe it is because the use of 
>> invisible text is rare.  What other modes use invisible text?
>
> Indeed, most complaints around invisible text have to do with the
> invisibility itself rather than details of how point is handed in
> its presence.  So invisible text is rare because it's inherently
> problematic (and should hence be avoided whenever possible).

Gnus message mode used invisible text to hide headers while composing
a news/mail message.  But it was changed to use narrowing instead:

,----[ gnus/ChangeLog ]
| 2004-11-22  Romain Francoise  <romain@orebokech.com>
| 
| 	* message.el (message-header-format-alist): Add `From' in list
| 	so that it can be sorted.
| 	(message-fix-before-sending): Widen and sort headers before
| 	sending.
| 	(message-hide-headers): Use narrowing to hide headers by moving
| 	them to the top of the buffer and narrowing to the region
| 	underneath.
`----

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

bug#869: Bug in handling invisible text, and bug in Pmail.

[Paul Michael Reilly]

"Richard M. Stallman" <rms@gnu.org> writes:

 > I think it is also a bug in Pmail.  Pmail should set the buffer boundaries
 > after that text, rather than making it invisible.

I don't think it is that simple because "that text" is simply part of
the collection of headers, some of which have been specified to be
invisible as Emacs displays the message, i.e. the X-Coding-System:
header can, according to rfc2822, precede or follow the Date: header.
This, if I understand the issue and your desires properly, makes
setting the buffer boundaries a non-workable solution.  I don't know
what the right solution is yet.  Worst case, the issue is an
indictment of using invisible text to hide the headers.  Yuck.

-pmr


 > If Pmail is fixed, then it won't trigger the invisible text bug.
 > But I think it needs to be fixed anyway.
 > 
 > 
 > ----------------------------------------------------------------------
 > >From Brand@xyzxysxysbb.com  Fri Aug 22 01:11:22 2008
 > X-Coding-System: undecided-unix
 > Return-path: <Brand@xyzxysxysbb.com>
 > Envelope-to: rms@gnu.org
 > Delivery-date: Fri, 22 Aug 2008 01:11:22 -0400
 > Received: from mx10.gnu.org ([199.232.76.166]:43206)
 > 	by fencepost.gnu.org with esmtp (Exim 4.67)
 > 	(envelope-from <Brand@xyzxysxysbb.com>)
 > 	id 1KWOvi-0001Cl-Jw
 > 	for rms@gnu.org; Fri, 22 Aug 2008 01:11:22 -0400
 > Received: from Debian-exim by monty-python.gnu.org with spam-scanned (Exim 4.60)
 > 	(envelope-from <Brand@xyzxysxysbb.com>)
 > 	id 1KWOwz-0000Ti-Nv
 > 	for rms@gnu.org; Fri, 22 Aug 2008 01:12:44 -0400
 > X-Spam-Checker-Version: SpamAssassin 3.1.0 (2005-09-13) on monty-python
 > X-Spam-Level: 
 > X-Spam-Status: No, score=-1.7 required=5.0 tests=AWL,BAYES_00,
 > 	FORGED_RCVD_HELO,NO_REAL_NAME autolearn=no version=3.1.0
 > Received: from sh.day.org ([198.102.73.65]:47087)
 > 	by monty-python.gnu.org with esmtp (Exim 4.60)
 > 	(envelope-from <Brand@xyzxysxysbb.com>)
 > 	id 1KWOwz-0000T4-E6
 > 	for rms@gnu.org; Fri, 22 Aug 2008 01:12:41 -0400
 > X-Client-Authorized: MaGic Cook1e
 > Received: from vree.xyzxysxysbb.com (sh.day.org [198.102.73.65])
 > 	by sh.day.org (Postfix) with ESMTPS id 18A9EC6988A39
 > 	for <rms@gnu.org>; Thu, 21 Aug 2008 22:12:38 -0700 (PDT)
 > Received: from localhost (localhost [127.0.0.1])
 > 	by vree.xyzxysxysbb.com (8.13.7/8.13.7) with ESMTP id m7M1kbEs004491
 > 	for <rms@gnu.org>; Thu, 21 Aug 2008 18:46:46 -0700
 > Message-Id: <200808220146.m7M1kbEs004491@vree.xyzxysxysbb.com>
 > To: rms@gnu.org
 > Subject: sorry that I was distracted with phone calls 
 > From: Brand@xyzxysxysbb.com
 > Reply-to: brand@xyzxysxysbb.com
 > Date: Thu, 21 Aug 2008 18:46:37 -0700
 > X-detected-kernel: by monty-python.gnu.org: Genre and OS details not recognized.
 > X-BABYL-V6-ATTRIBUTES: A------
 > 
 > 
 > 
 > Two other ideas to run past you when time permits.
 > 
 > /w

bug#869: Bug in handling invisible text, and bug in Pmail.

[Stefan Monnier]

> I don't think it is that simple because "that text" is simply part of
> the collection of headers, some of which have been specified to be
> invisible as Emacs displays the message, i.e. the X-Coding-System:
> header can, according to rfc2822, precede or follow the Date: header.
> This, if I understand the issue and your desires properly, makes
> setting the buffer boundaries a non-workable solution.  I don't know
> what the right solution is yet.  Worst case, the issue is an
> indictment of using invisible text to hide the headers.  Yuck.

But is that specific to Pmail?  I.e. doesn't Rmail use the same
texhcnique (or even code) to hide the headers?
If Pmail is indeed different, could you explain to me why Pmail had to
change this part of the code?  It seems unrelated to babyl-vs-mbox.


        Stefan

bug#869: Bug in handling invisible text, and bug in Pmail.

[Richard M. Stallman]

Rmail hides some headers by making a copy of the whole header.
In that copy it deletes the uninteresting headers.
Babyl format makes special provision for that.
Pmail cannot use that method.

If we make Pmail always display the current message decoded in a
separate buffer, we could delete uninteresting header fields from that
buffer.

bug#869: Bug in handling invisible text, and bug in Pmail.

[Paul Michael Reilly]

Stefan Monnier wrote:
>> I don't think it is that simple because "that text" is simply part of
>> the collection of headers, some of which have been specified to be
>> invisible as Emacs displays the message, i.e. the X-Coding-System:
>> header can, according to rfc2822, precede or follow the Date: header.
>> This, if I understand the issue and your desires properly, makes
>> setting the buffer boundaries a non-workable solution.  I don't know
>> what the right solution is yet.  Worst case, the issue is an
>> indictment of using invisible text to hide the headers.  Yuck.
> 
> But is that specific to Pmail?  I.e. doesn't Rmail use the same
> texhcnique (or even code) to hide the headers?
> If Pmail is indeed different, could you explain to me why Pmail had to
> change this part of the code?  It seems unrelated to babyl-vs-mbox.

Apparently, Rmail/babyl rearranges the headers and then sets the buffer 
boundaries, thus eliminating the problem.  Rmail/mbox could have (and 
probably should have) used the same approach.

Now however, it is pretty clear that a presentation buffer is in order 
to deal with the coding issue, something I ill understand today but I'm 
gradually picking up.  I also think that the coding issue implicitly 
means better MIME handling within Rmail.  Not hardly a bad thing, 
especially given modern Emacs' improved graphic capabilities.

-pmr

bug#869: Bug in handling invisible text, and bug in Pmail.

[Richard M. Stallman]

     > I think it is also a bug in Pmail.  Pmail should set the buffer boundaries
     > after that text, rather than making it invisible.

    I don't think it is that simple because "that text" is simply part of
    the collection of headers, some of which have been specified to be
    invisible as Emacs displays the message, i.e. the X-Coding-System:
    header can, according to rfc2822, precede or follow the Date: header.

Why does that make it not simple?  You can move the uninteresting headers
to the top and put point-min after them.

Also, if it is necessary to display the decoded message in a separate
buffer then Pmail can delete the unwanted headers in that buffer.