require-hard-newlines to use newline

require-hard-newlines to use newline

[Chong Yidong]

(Sorry for leaving out a subject field)

>    Could this patch be checked in? When require-final-newlines is t, the
>    newline is inserted by (insert "\n"), which does not take
>    use-hard-newlines into account, so the newline is not marked as 'hard.
>
> Are you really _sure_ that a newline inserted by saving a buffer
> should be hard if use-hard-newlines is enabled?  I would guess that
> people might occasionally save their work to file while in the middle
> of a long paragraph, in which case the inserted newline should be
> soft.  I guess that if people want to mark the end of a paragraph
> before saving, they would explicitly type RET.
>
> What situation are you thinking of that would motivate making C-x C-s
> insert a hard newline?

As I see it, the rationale for turning on require-final-newline is that a
particular type of file should always end in a newline. The user should
type RET himself, but in case he forgets to do so, Emacs does it for him.
So (newline) should be used.

Nothing will go wrong when you save in the middle of a paragraph, since
the code inserts the newline inside a save-excursion. The hard newline
always goes *after* point, at the end of the buffer, where it won't get in
the way.

The specific problem I am trying to solve is with Longlines mode (which is
not part of Emacs.) It is, in principle, impossible for Longlines to
distinguish between the soft newline inserted by require-final-newline and
the soft newline inserted by filling when performing automatic line
wrapping. It therefore turns that newline into a space, with the effect
that buffer-modified-p is always t if you try to save a Longlines buffer
with require-final-newline.

Re: require-hard-newlines to use newline

[Luc Teirlinck]

Chong Yidong wrote:

   Nothing will go wrong when you save in the middle of a paragraph, since
   the code inserts the newline inside a save-excursion. The hard newline
   always goes *after* point, at the end of the buffer, where it won't get in
   the way.

I guess you may be right.  (But it is probably better if Richard still
takes a look at it.)

Sincerely,

Luc.

Re: require-hard-newlines to use newline

[Richard Stallman]

    As I see it, the rationale for turning on require-final-newline is that a
    particular type of file should always end in a newline. The user should
    type RET himself, but in case he forgets to do so, Emacs does it for him.
    So (newline) should be used.

That is true.  But those kinds of files are in special formats, not
human-language text.  What is the motive for setting use-hard-newlines
in one of those buffers?

    The specific problem I am trying to solve is with Longlines mode (which is
    not part of Emacs.) It is, in principle, impossible for Longlines to
    distinguish between the soft newline inserted by require-final-newline and
    the soft newline inserted by filling when performing automatic line
    wrapping.

"Automatic line wrapping" is not a normal Emacs term, and I am not
sure what you mean by it.  Do you mean Auto Fill mode?  If so,
longlines can distinguish the two cases because the newline inserted
by Auto Fill mode is not at the end of the line.

If it means something else, could you please say what?

Re: require-hard-newlines to use newline

[Chong Yidong]

>     As I see it, the rationale for turning on require-final-newline is
>     that a particular type of file should always end in a newline. The
>     user should type RET himself, but in case he forgets to do so, Emacs
>     does it for him. So (newline) should be used.
>
> That is true.  But those kinds of files are in special formats, not
> human-language text.  What is the motive for setting use-hard-newlines
> in one of those buffers?

Require-final-newline is a user option, and if a user customizes it to t
then it is turned on for every buffer, even for human-language text. A
user might do this if he want every file he edits to end in a newline.

>     The specific problem I am trying to solve is with Longlines mode
>     (which is not part of Emacs.) It is, in principle, impossible for
>     Longlines to distinguish between the soft newline inserted by
>     require-final-newline and the soft newline inserted by filling when
>     performing automatic line wrapping.
>
> "Automatic line wrapping" is not a normal Emacs term, and I am not
> sure what you mean by it.  Do you mean Auto Fill mode?  If so,
> longlines can distinguish the two cases because the newline inserted
> by Auto Fill mode is not at the end of the line.
>
> If it means something else, could you please say what?

Like Emacs' built-in Refill mode, Longlines does filling after every user
command, except that it uses its own filling functions (for various
reasons that have to do with simulating the behavior of "word-wrapped"
text editors.) In any case, it makes use of use-hard-newlines to keep
track of hard and soft newlines.

It is impossible to distinguish between a soft newline left at the end of
a line by require-final-newline, and a soft newline produced by, e.g., a
call to kill-line, simply by looking at the context. Both are soft, and
both occur at the end of a line. Both newlines might even occur at the end
of a buffer, if that was the final line. But the first is conceptually a
hard newline, and should not be converted into a space by filling; whereas
the second is a "real" soft newline.

Re: require-hard-newlines to use newline

[Richard Stallman]

    It is impossible to distinguish between a soft newline left at the end of
    a line by require-final-newline, and a soft newline produced by, e.g., a
    call to kill-line,

I don't see how a call to kill-line can produce a newline.
I don't understand you.

		       simply by looking at the context. Both are soft, and
    both occur at the end of a line. Both newlines might even occur at the end
    of a buffer, if that was the final line. But the first is conceptually a
    hard newline

Why do you think it is "conceptually a hard newline"?  I don't think
that is true.

As I understand it, the newline that ends the last line in a paragraph
is normally soft.  A hard newline would end the following blank line.
The newline added by require-final-newline would be at the end of the
last line in a paragraph, so it ought to be soft.

Re: require-hard-newlines to use newline

[Chong Yidong]

> As I understand it, the newline that ends the last line in a paragraph
> is normally soft.  A hard newline would end the following blank line.
> The newline added by require-final-newline would be at the end of the
> last line in a paragraph, so it ought to be soft.

Both of the newline ends the last line in a paragraph, as well as the
newline on the following blank line, are hard. You can verify this for
yourself, but I can also illustrate it. First, recall that soft newlines
are equivalent to spaces, so fill-paragraph can delete a soft newline or
replace it with one or more spaces. Now if you have a paragraph

foo foo\n
bar bar\n
\N
foo bar\n
...

where \n denotes a soft newline and \N a hard newline, then it is refilled to

foo foo\n
bar bar\N
foo bar\n
...

whereas

foo foo\n
bar bar\N
\N
foo bar\n

can't be further refilled.

>     It is impossible to distinguish between a soft newline left at the end
>     of a line by require-final-newline, and a soft newline produced by,
>     e.g., a call to kill-line,
>
> I don't see how a call to kill-line can produce a newline.
> I don't understand you.

kill-line deletes up to and including the newline. Suppose the cursor is
at the indicated position:

foo-!-bar\n[EOB]

Calling kill-lines gives

foo-!-\n[EOB]

which, since soft newlines can be converted to spaces, is refilled to

foo-!-[EOB]

The soft newline at the end is gobbled up. That's the problem.

Re: require-hard-newlines to use newline

[Luc Teirlinck]

Chong Yidong wrote:
   
   Both of the newline ends the last line in a paragraph, as well as the
   newline on the following blank line, are hard.

I believe that you are right on that one.  However, there is another
way to view things.  I believe that the newline inserted automatically
at the end of the buffer if `require-final-newline' is enabled should
be soft.

Here is my reasoning:

I believe that people often save in the middle of a paragraph, but
very rarely in the middle of a word.  C-x C-s inserts a newline at the
end of the file, but in the buffer that newline is just a nuisance.
If you call end-of-buffer you go past the newline and you have to be
careful to do C-b, or you are editing at the wrong place.  Normally,
there is nothing you can do about that, you just have to live with this
(granted small) misfeature of `require-final-newline'.  Because the
newline is soft, if you are careless and start typing at the wrong
place, you can trivially correct the situation with M-q.  However,
Longlines tries to make it convenient on you by translating the
newline back into a space.  Now you can just do end-of-buffer and
immediately continue typing, without having to worry about remembering
to type C-b.  Wonderful.  Making the newline hard would mess this up.
So what is the problem?  From what you told us, apparently that
Longlines marks the buffer modified.  Apart from that the feature (with
_soft_ newline) seems perfect.

I believe to remember that you said that Longlines is not included with
Emacs.  But you could ask the Longlines maintainer to put a function
in after-save-hook that transforms the soft newline into a space,
without marking the buffer modified.  There is no need to mark the
buffer modified, there are no changes that are going to be saved.  On
saving, filling should remove the trailing whitespace at the end of
the buffer and put the newline back in, yielding exactly the same
content as the file currently has on disk.  After this change, it
would seem that the problem disappears and everything is perfect.

Does the above make sense or are there other reasons to make the
newline hard?

Sincerely,

Luc.

Re: require-hard-newlines to use newline

[Chong Yidong]

> C-x C-s inserts a newline at the end of the file, but in the buffer that
> newline is just a nuisance. If you call end-of-buffer you go past the
> newline and you have to be careful to do C-b, or you are editing at the
> wrong place... Because the newline is soft, if you are careless and start
> typing at the wrong place, you can trivially correct the situation with
> M-q.

Typing C-b is no more difficult than typing M-q.

> Longlines tries to make it convenient on you by translating the
> newline back into a space.  Now you can just do end-of-buffer and
> immediately continue typing, without having to worry about remembering
> to type C-b...
> You could ask the Longlines maintainer to put a function
> in after-save-hook that transforms the soft newline into a space,
> without marking the buffer modified.

I *am* the Longlines maintainer. The solution you suggested is no good,
because if you edit somewhere else in the document (not at the end of the
buffer) and save, you will end up with another newline, which gets
converted into another space.

Yes, I can cook up a kludge to work around require-hard-newline if it
continues to use a soft final newline. But the reason I posted this thread
is because that would be unclean. Normal paragraphs are terminated by hard
newlines, and so should the final paragraph.

Re: require-hard-newlines to use newline

[Miles Bader]

BTW, longlines.el seems to be fairly widely used; is there a reason it
hasn't been added to the Emacs distribution?

-Miles

Re: require-hard-newlines to use newline

[Chong Yidong]

> BTW, longlines.el seems to be fairly widely used; is there a reason it
> hasn't been added to the Emacs distribution?

The main reason is the feature freeze. RMS contacted me about putting it
into Emacs a couple months back, and my guess is that it will go in after
this release. My copyright assignment is already in place.

Re: require-hard-newlines to use newline

[David Kastrup]

"Chong Yidong" <cyd@stupidchicken.com> writes:

>> BTW, longlines.el seems to be fairly widely used; is there a reason
>> it hasn't been added to the Emacs distribution?
>
> The main reason is the feature freeze. RMS contacted me about
> putting it into Emacs a couple months back, and my guess is that it
> will go in after this release. My copyright assignment is already in
> place.

Putting a limited optional package in should not really be a concern
for the feature freeze as long as its presence does not imply one
should rework the manual and stuff.  And as long as it does not keep
other people from working on the release.

I don't think this is the case here, but I'd let Richard decide this
instead of trying to interpret the general intent and rules yourself.
It is an often asked for functionality, and the package is more or
less the answer given to that even if I don't know it myself.  But
people need not use it if they don't want to.

So I'd ask Richard instead of censoring yourself.  It is certainly not
worth putting up a fight or argument for, but I don't see the harm in
him taking a look.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: require-hard-newlines to use newline

[Richard Stallman]

    First, recall that soft newlines
    are equivalent to spaces, so fill-paragraph can delete a soft newline or
    replace it with one or more spaces.

					Now if you have a paragraph

    foo foo\n
    bar bar\n
    \N
    foo bar\n
    ...

    where \n denotes a soft newline and \N a hard newline, then it is refilled to

    foo foo\n
    bar bar\N

That seems to be replacing \n with zero spaces, not with one space.
Is it correct?

Anyway, if it is a fact that both of these newlines are normally
hard, then it seems to me that there is no way of telling
whether the user wants a hard newline or a soft one
when require-final-newline adds a newline.
A user might save the file with an unfinished paragraph,
then go back to writing more of it.

Perhaps use-hard-newlines should inhibit the effect of
require-final-newline.

Re: require-hard-newlines to use newline

[Richard Stallman]

    BTW, longlines.el seems to be fairly widely used; is there a reason it
    hasn't been added to the Emacs distribution?

It would be useful for a few experienced Emacs developers to look it
over and make suggestions.

Re: require-hard-newlines to use newline

[Chong Yidong]

> Anyway, if it is a fact that both of these newlines are normally
> hard, then it seems to me that there is no way of telling
> whether the user wants a hard newline or a soft one
> when require-final-newline adds a newline.
> A user might save the file with an unfinished paragraph,
> then go back to writing more of it.

I don't think there's a problem in this situation; he just has to make
sure that point is before the final (hard) newline before he continues
editing. Furthermore, there is no need to re-position point, because the
final newline is inserted inside a save-excursion, *after* point.

Re: require-hard-newlines to use newline

[Richard Stallman]

    I don't think there's a problem in this situation; he just has to make
    sure that point is before the final (hard) newline before he continues
    editing. 

If the user has to do that, then why does it matter whether this
newline is hard or soft?

Re: require-hard-newlines to use newline

[Chong Yidong]

>     I don't think there's a problem in this situation; he just has to make
>     sure that point is before the final (hard) newline before he continues
>     editing.
>
> If the user has to do that, then why does it matter whether this
> newline is hard or soft?

Because, everywhere else in the buffer, the newline at the end of a
paragraph is hard.

To forestall this discussion going round in a circle again: one might
object that the newline at the end of the last paragraph need not be hard,
because the user may not have decided to end that paragraph yet. But
that's okay, because eventually he *has* to end that paragraph (that's
what require-final-newline says). Because the final newline is inserted in
a save-excursion, after point, there is never any inconvenience to the
user.

I will have one last try at illustrating why using a soft newline can lead
to strange behavior. If this does not work, then I will simply drop the
issue.

Consider this, scenario A:

emacs -Q
M-x use-hard-newline
M-: (setq require-final-newline t)
C-x C-f foo.txt
some text
C-x C-s

And consider another scenario, scenario B, which is the same except the
user explicitly types in a newline before C-x C-s ("some text RET").

Now suppose the user goes to another buffer to do his editing, and comes
back to this buffer a long time later. He does not remember the exact
sequence of edits he performed on that buffer -- in particular, whether he
typed RET or not. From moving point around, he observes that the buffer
contains "some text" followed by a final newline.

Now suppose he does this:

M->
some more text
M-q

There is a big difference! In scenario A, M-q causes moves "some more
text" onto the same line as "some text". In scenario B, nothing happens
(which is correct, because the purpose of use-hard-newlines is to allow a
hard break in the middle of a sentence.) This leads to an appearance of
unpredictability.

Re: require-hard-newlines to use newline

[Richard Stallman]

    Because, everywhere else in the buffer, the newline at the end of a
    paragraph is hard.

That doesn't seem like a convincing reason.

    Now suppose the user goes to another buffer to do his editing, and comes
    back to this buffer a long time later. He does not remember the exact
    sequence of edits he performed on that buffer -- in particular, whether he
    typed RET or not. From moving point around, he observes that the buffer
    contains "some text" followed by a final newline.

If he did not finish the paragraph, he will probably assume the
newline is soft.  If he did finish the paragraph, he will probably
assume the newline is hard.  Either way, he might be wrong.

So I think that use-hard-newlines should inhibit the effect of
require-final-newline.  It is the only way to get reliable results.

Re: require-hard-newlines to use newline

[Oliver Scholz]

Richard Stallman <rms@gnu.org> writes:

>     Because, everywhere else in the buffer, the newline at the end of a
>     paragraph is hard.
>
> That doesn't seem like a convincing reason.
>
>     Now suppose the user goes to another buffer to do his editing, and comes
>     back to this buffer a long time later. He does not remember the exact
>     sequence of edits he performed on that buffer -- in particular, whether he
>     typed RET or not. From moving point around, he observes that the buffer
>     contains "some text" followed by a final newline.
>
> If he did not finish the paragraph, he will probably assume the
> newline is soft.  If he did finish the paragraph, he will probably
> assume the newline is hard.  Either way, he might be wrong.
>
> So I think that use-hard-newlines should inhibit the effect of
> require-final-newline.  It is the only way to get reliable results.

Sorry if I missed something: as I understand it the problem arises
from the fact that `require-final-newline' will not only make sure
that the *file* ends with a newline, but will that newline add to the
*buffer*.

If I understand things correctly, the problem would go away, if
require-f-n would just add the newline when writing the file but not
to the buffer (a bit similar to a function in
`write-region-annotate-functions') . Then the user would only come to
see it, if she reverts the buffer; in this case it is longlines.el's
job to use its heuristics to detect whether the final newline is hard
or soft.

    Oliver
-- 
17 Ventôse an 213 de la Révolution
Liberté, Egalité, Fraternité!

Re: require-hard-newlines to use newline

[Luc Teirlinck]

Richard Stallman wrote:

       BTW, longlines.el seems to be fairly widely used; is there a reason it
       hasn't been added to the Emacs distribution?

   It would be useful for a few experienced Emacs developers to look it
   over and make suggestions.

I am not really the best person to make suggestions.  It would  be
better if longtime users did.  Here are some remarks.

I do not know what the version is that would be included.  I
dowmloaded 2.2.7 from:

//www.emacswiki.org/elisp/index.html

After putting (load "~/longlines.elc") in my .emacs, I had to edit
longlines.el and put: (defvar longlines-mode nil) at the beginning to
avoid a warnings buffer cropping up each time I started Emacs, warning
about the free variable `longlines-mode'.  Just (defvar longlines-mode)
was _not_ sufficient.

If I understood correctly, we do not like code included with the Emacs
distribution to use `defadvice'.  longlines.el uses defadvice for
`newline', `kill-region', `copy-region-as-kill', `yank' and `yank-pop'.

Sincerely,

Luc.

Re: require-hard-newlines to use newline

[Luc Teirlinck]

Oliver Scholz wrote:

   If I understand things correctly, the problem would go away, if
   require-f-n would just add the newline when writing the file but not
   to the buffer (a bit similar to a function in
   `write-region-annotate-functions') .

That would be OK for longlines to do but not for Emacs in general.
Some people are already (very) upset that certain modes add newlines
behind their back.  The added newline is not easy to spot in the
buffer.  Going trough the effort of actually _completely_ hiding it
would make matters worse.

But longlines _already_ puts newlines in the buffer that are not present
in the file and will _not_ be saved into it, so for longlines to do it
would make sense.

I would suggest that we keep writing a _soft_ newline to the end of
the file (unless a newline, soft or hard, is already present) if both
require-final-newline and use-hard-newlines are enabled, but that
_only_ longlines handle it differently.

Longlines treats soft and hard newlines similarly to the way for
instance the AbiWord word processor does, which is very different from
the way Enriched mode treats them.  Soft newlines do not exist in the
actual file.  Every newline occurring in the file is a hard newline.
Using longlines-mode for a file apparently only makes sense if you
always use it if you edit that file (for instance, if you set it
through a file local variable).  I suggest that longlines treats final
newlines the same way AbiWord does.  _Always_ add one internal final
newline to the file (_maybe_ even regardless of require-final-newline),
that _never_ shows up in the buffer.  (But _tell_ the user in the docs
that you are doing so.)  Applications that insist on a final newline
will not recognize the extra newline as internal and will be happy.

Note that Enriched mode, in contrast with AbiWord and longlines stores
soft newlines as a single newline and sequences of N hard newlines as
sequences of N + 1 newlines, in line with RFC1896.

Sincerely,

Luc.

Re: require-hard-newlines to use newline

[Luc Teirlinck]

Richard Stallman wrote:

   If he did not finish the paragraph, he will probably assume the
   newline is soft.  If he did finish the paragraph, he will probably
   assume the newline is hard.

The way I see it, if use-hard-newlines is enabled, the user finishes a
paragraph by explicitly typing a newline.  That newline _is_ hard.
Newlines inserted by Emacs for filling or standard-following purposes
are soft.

   So I think that use-hard-newlines should inhibit the effect of
   require-final-newline.  It is the only way to get reliable results.

There are two minor modes I know of that use hard newlines: Enriched
mode and Longlines.  I believe that the above solution would be OK for
Enriched mode, at least as long as it only works with text/enriched.
Unless I overlooked something, nothing in RFC1896 requires a final
newline, hard or soft.  I believe, however, that Longlines could
conceivably be used for stuff whose standard requires a final newline.

Sincerely,

Luc.

Re: require-hard-newlines to use newline

[Luc Teirlinck]

>From my previous message:

   I suggest that longlines treats final newlines the same way AbiWord does.

I should have been more explicit here (and elsewhere).  When I
referred to "what AbiWord does" I meant:  "what Abiword does when
saving in the (plain) Text format".

Sincerely,

Luc.

Re: require-hard-newlines to use newline

[Chong Yidong]

"Luc Teirlinck" <teirllm@dms.auburn.edu> wrote:
>    It would be useful for a few experienced Emacs developers to look it
>    [longlines.el] over and make suggestions.
>
> I am not really the best person to make suggestions.  It would  be
> better if longtime users did.  Here are some remarks.
> ...
> After putting (load "~/longlines.elc") in my .emacs, I had to edit
> longlines.el and put: (defvar longlines-mode nil) at the beginning to
> avoid a warnings buffer cropping up each time I started Emacs, warning
> about the free variable `longlines-mode'.  Just (defvar longlines-mode)
> was _not_ sufficient.

I can't reproduce this, and it doesn't make sense; longlines-mode is
defined using define-minor-mode, which should do the variable definition
properly. Could you look into this further? Maybe it is something in your
.emacs.

> If I understood correctly, we do not like code included with the Emacs
> distribution to use `defadvice'.  longlines.el uses defadvice for
> `newline', `kill-region', `copy-region-as-kill', `yank' and `yank-pop'.

Yes, this will have to be fixed before longlines.el can go in.

Re: require-hard-newlines to use newline

[Richard Stallman]

    If I understand things correctly, the problem would go away, if
    require-f-n would just add the newline when writing the file but not
    to the buffer (a bit similar to a function in
    `write-region-annotate-functions') . Then the user would only come to
    see it, if she reverts the buffer; in this case it is longlines.el's
    job to use its heuristics to detect whether the final newline is hard
    or soft.

That is an interesting suggestion, but I tend to suspect it
might also cause problems.  People could try it and see.

Re: require-hard-newlines to use newline

[Luc Teirlinck]

Chong Yidong wrote:

   I can't reproduce this, and it doesn't make sense; longlines-mode is
   defined using define-minor-mode, which should do the variable definition
   properly. Could you look into this further? Maybe it is something in your
   .emacs.

I believe that the following should be reproducible:

Do `emacs -Q' `M-x ielm RET", then, after the following ielm run, the
warning appears:

===File ~/longlines-ielm====================================
***nil
 Welcome to IELM ***  Type (describe-mode) for help.
ELISP> (compile-defun
	'(defun my-emacs-exit-question ()
	   "Ask for confirmation before exiting emacs, if there is a visible frame"
	   (if (visible-frame-list)
	       (y-or-n-p "Really quit Emacs? ")
	     t)))
nil
ELISP> (load "~/longlines.elc")
t
ELISP> ============================================================

Re: require-hard-newlines to use newline

[Luc Teirlinck]

The following contains less irrelevant clutter than my previous ielm run.

`emacs -Q' `M-x ielm RET', then:

***nil
 Welcome to IELM ***  Type (describe-mode) for help.
ELISP> (compile-defun (lambda ()))
nil
ELISP> (load "~/longlines.elc")
t

Message appears.

Sincerely,

Luc.

Re: require-hard-newlines to use newline

[Luc Teirlinck]

In my previous ielm run, I somehow managed to misunderstand
`compile-defun', but anyway, the message appears after:

`emacs -Q', `M-x ielm RET' and:

*** Welcome to IELM ***  Type (describe-mode) for help.
ELISP> (load "bytecomp")
t
ELISP> (load "~/longlines.elc")
t
ELISP> 

It does not appear without (load "bytecomp").

Sincerely,

Luc.

Re: require-hard-newlines to use newline

[Richard Stallman]

    If I understood correctly, we do not like code included with the Emacs
    distribution to use `defadvice'.  longlines.el uses defadvice for
    `newline', `kill-region', `copy-region-as-kill', `yank' and `yank-pop'.

Yes,it would be good to think about other mechanisms to use instead
of that.  Yidong, would you like to think about what kind of new hook
might make it possible to do the job?

Re: require-hard-newlines to use newline

[Chong Yidong]

>     If I understood correctly, we do not like code included with the Emacs
>     distribution to use `defadvice'.  longlines.el uses defadvice for
>     `newline', `kill-region', `copy-region-as-kill', `yank' and
>     `yank-pop'.
>
> Yes,it would be good to think about other mechanisms to use instead
> of that.  Yidong, would you like to think about what kind of new hook
> might make it possible to do the job?

What the advice does is mainly encoding and decoding yanked text. When a
piece of text is yanked into a longlines buffer, the newlines have to be
marked as hard. Conversely, when text is killed from a longlines buffer,
the soft newlines have to be removed before it is placed in the kill ring.

As I see it, there are two ways to get around using advice:

1. If Longlines is to be merged into Emacs, lines like
(if (and (fboundp longlines-mode) longlines-mode) blahblahblah)
could be added to kill-region and the other functions. This is what
Transient mark mode does, but I doubt that Longlines mode is "important"
enough to justify it.

2. Defining two new abnormal hooks, maybe named yank-encode-functions and
kill-encode-functions, to be called by kill-region etc (or possibly the
lower-level functions like kill-new and kill-append.) Each function would
be called with one argument, the yanked or killed string, and return an
encoded string to be passed to the next function, or to the buffer/kill
ring. For example, kill-region could do something like

 (mapcar '(lambda (fun) (setq string (funcall fun string)))
         'kill-encode-functions)

Re: require-hard-newlines to use newline

[Chong Yidong]

> ELISP> (load "bytecomp")
> t
> ELISP> (load "~/longlines.elc")
> t
> ELISP>
>
> It does not appear without (load "bytecomp").

Okay, I can reproduce it now. Moving the define-minor-mode call to the top
of the file fixes the problem; thanks.

Strangely enough, M-x byte-compile does not complain about free variables.
Why should (load "bytecomp") be necessary to produce the message?

Re: require-hard-newlines to use newline

[Stefan Monnier]

ELISP> (load "bytecomp")
>> t
ELISP> (load "~/longlines.elc")
>> t
ELISP> 
>> 
>> It does not appear without (load "bytecomp").

> Okay, I can reproduce it now. Moving the define-minor-mode call to the top
> of the file fixes the problem; thanks.

> Strangely enough, M-x byte-compile does not complain about free variables.
> Why should (load "bytecomp") be necessary to produce the message?

Maybe it's because you call M-x byte-compiler from an Emacs where longlines
was already loaded.


        Stefan

Re: require-hard-newlines to use newline

[Luc Teirlinck]

Stefan Monnier wrote:

   Maybe it's because you call M-x byte-compiler from an Emacs where longlines
   was already loaded.

I do not believe so:

[bash2.05b.0 ~ 3 1] emacs-22.0.50 -batch -f batch-byte-compile
longlines.el
Wrote /home/teirllm/longlines.elc
[bash2.05b.0 ~ 3 2] 

Maybe it is because all the offending occurrences of the variable are
in defadvice forms.

Another thing I do not understand is that (devar longlines-mode) did
not help.  Only (devar longlines-mode nil) did.

Sincerely,

Luc.

Re: require-hard-newlines to use newline

[Luc Teirlinck]

I guess that, In Chong's message, I confused byte-compile with
byte-compile-file.  What I believed was strange is that byte compiling
the file produced no warning messages, but maybe that is due to the
variable occurring in defadvice.

Sincerely,

Luc.

Re: require-hard-newlines to use newline

[Stefan Monnier]

>    Maybe it's because you call M-x byte-compiler from an Emacs where longlines
>    was already loaded.

> I do not believe so:

> [bash2.05b.0 ~ 3 1] emacs-22.0.50 -batch -f batch-byte-compile
> longlines.el
> Wrote /home/teirllm/longlines.elc
> [bash2.05b.0 ~ 3 2] 

> Maybe it is because all the offending occurrences of the variable are
> in defadvice forms.

Indeed.  The code in defadvice is (generally) not compiled during
byte-compilation.

> Another thing I do not understand is that (devar longlines-mode) did
> not help.  Only (devar longlines-mode nil) did.

A (defvar longlines-mode) doesn't have any long term effect: it only
temporarily tells the byte-compiler to shut up.  This only lasts until the
file is byte-compiled.  If you do further byte-compilation (e.g. of the
advice code or of code output by some macro), the byte-compiler won't
remember that the file where this code originally appeared had a (defvar
longlines-mode).


        Stefan

Re: require-hard-newlines to use newline

[Chong Yidong]

> 2. Defining two new abnormal hooks, maybe named yank-encode-functions and
> kill-encode-functions, to be called by kill-region etc (or possibly the
> lower-level functions like kill-new and kill-append.) Each function would
> be called with one argument, the yanked or killed string, and return an
> encoded string to be passed to the next function, or to the buffer/kill
> ring.

On further consideration, there seems to be some overlap between the above
suggestion and the yank-handler text property (new to Emacs 22). It may be
better to a variable called `yank-handlers' which is a list of yank
handlers to use. The yank functions could then read from both
yank-handlers and the yank-handler text property.

There is one problem, however: currently, the function specified by the
yank-handler text property is called *instead* of insert, to insert the
desired text. According to NEWS, this is to allow it to do things like
yanking rectanges. Such a function obviously cannot "stack", so what
happens if yank-handlers and the yank-handler text property are both
defined? My original idea was to have the functions specified by
yank-handlers return a string to be passed to the next handler, but (i)
this would be an incompatibility with the text property, and (ii) it won't
be able to perform more general operations like yanking rectangles.

I would also still need an analogous kill-handlers variable for encoding
kills, i.e. stripping out newlines from the killed text (it's not enough
to use a yank handler, because the killed text could be pasted into an
external program via the X selection or clipboard.)

Re: require-hard-newlines to use newline

[Richard Stallman]

    2. Defining two new abnormal hooks, maybe named yank-encode-functions and
    kill-encode-functions, to be called by kill-region etc (or possibly the
    lower-level functions like kill-new and kill-append.)

This would be better than using advice.

    On further consideration, there seems to be some overlap between the above
    suggestion and the yank-handler text property (new to Emacs 22).

The idea of the yank-handler text property is that certain kill
strings are encoded specially, and need to be decoded when they are
yanked.  Maybe longlines could use this instead of a new hook for
yanking.

However, it would still need a hook for killing.  However,
not just for killing.  This hook should be used in various places,
including Fdelete_and_extract_region.

Or maybe you could just use after-change-functions.
Would that work?

Re: require-hard-newlines to use newline

[Chong Yidong]

> The idea of the yank-handler text property is that certain kill
> strings are encoded specially, and need to be decoded when they are
> yanked.  Maybe longlines could use this instead of a new hook for
> yanking.

It is possible to use the yank-handler property to remove the newlines
from longlines text when copying into other Emacs buffers; however, since
the newlines are copied onto the clipboard and X selection, they will show
up when the user does a paste in another application.  So this is not a
good approach.

> Or maybe you could just use after-change-functions.
> Would that work?

The only way I can see for this to work is for the after-change function
to test this-command, which I think is not very robust.

So I guess the best solution is to have an encoding hook for kills, and a
decoding hook for yanks.

I also think we should wait till after the release before implementing
this (which hopefully shouldn't be a long wait...)

Re: require-hard-newlines to use newline

[Kim F. Storm]

"Chong Yidong" <cyd@stupidchicken.com> writes:

> So I guess the best solution is to have an encoding hook for kills, and a
> decoding hook for yanks.
>
> I also think we should wait till after the release before implementing
> this (which hopefully shouldn't be a long wait...)

If we agree that those hooks are the proper approach, and you can
write the proper doc for them, this is a trivial change, so I don't
see why we have to wait.

Especially if that means that longlines.el can go into 22.1.

I think that would be a good thing considering that 23.x is most
likely years rather than months away...  [I hope that's not true for
22.x as well]

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: require-hard-newlines to use newline

[Chong Yidong]

> If we agree that those hooks are the proper approach, and you can
> write the proper doc for them, this is a trivial change, so I don't
> see why we have to wait.
>
> Especially if that means that longlines.el can go into 22.1.

Okay then. How about this patch?

While writing it, I figured out that I only need to introduce one hook
(`before-kill-functions'), not two. The hook for yanking is not needed for
longlines.el -- I can handle that with post-command-hook (which longlines
already makes use of anyway.)

`before-kill-functions' seems like a convenient thing to have in any
event, since Emacs Lisp packages can use it to apply the yank-handler text
property on the fly.


*** simple.el~	Fri Mar 11 19:27:00 2005
--- simple.el	Fri Mar 11 21:00:45 2005
***************
*** 2275,2280 ****
--- 2275,2286 ----
  (defvar kill-ring-yank-pointer nil
    "The tail of the kill ring whose car is the last thing yanked.")

+ (defvar before-kill-functions nil
+   "List of functions called on the region before killing.
+ This abnormal hook is run before `kill-region' and
+ `copy-region-as-kill', with the beginning and end positions of
+ the killed region as the arguments.")
+
  (defun kill-new (string &optional replace yank-handler)
    "Make STRING the latest kill in the kill ring.
  Set `kill-ring-yank-pointer' to point to it.
***************
*** 2371,2376 ****
--- 2377,2385 ----
  The command \\[yank] can retrieve it from there.
  \(If you want to kill and then yank immediately, use \\[kill-ring-save].)

+ This runs the abnormal hook `before-kill-functions' with the
+ arguments BEG and END before actually killing.
+
  If you want to append the killed region to the last killed text,
  use \\[append-next-kill] before \\[kill-region].

***************
*** 2390,2395 ****
--- 2399,2405 ----
  specifies the yank-handler text property to be set on the killed
  text.  See `insert-for-yank'."
    (interactive "r")
+   (run-hook-with-args 'before-kill-functions beg end)
    (condition-case nil
        (let ((string (delete-and-extract-region beg end)))
  	(when string			;STRING is nil if BEG = END
***************
*** 2424,2431 ****
    "Save the region as if killed, but don't kill it.
  In Transient Mark mode, deactivate the mark.
  If `interprogram-cut-function' is non-nil, also save the text for a window
! system cut and paste."
    (interactive "r")
    (if (eq last-command 'kill-region)
        (kill-append (buffer-substring beg end) (< end beg))
      (kill-new (buffer-substring beg end)))
--- 2434,2445 ----
    "Save the region as if killed, but don't kill it.
  In Transient Mark mode, deactivate the mark.
  If `interprogram-cut-function' is non-nil, also save the text for a window
! system cut and paste.
!
! The abnormal hook `before-kill-functions' is run with the
! arguments BEG and END before the region is saved."
    (interactive "r")
+   (run-hook-with-args 'before-kill-functions beg end)
    (if (eq last-command 'kill-region)
        (kill-append (buffer-substring beg end) (< end beg))
      (kill-new (buffer-substring beg end)))

Re: require-hard-newlines to use newline

[Stefan Monnier]

> + (defvar before-kill-functions nil
> +   "List of functions called on the region before killing.
> + This abnormal hook is run before `kill-region' and
> + `copy-region-as-kill', with the beginning and end positions of
> + the killed region as the arguments.")

I think it's usually important to keep the property that killing doesn't
modify the buffer's text (when used in copy-as-kill), so rather than having
before-kill-functions modify the buffer's text just before it's put in the
kill-ring, it would be more convenient to make it return the string,
i.e. use it as a replacement for buffer-substring.


        Stefan

Re: require-hard-newlines to use newline

[Kim F. Storm]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> + (defvar before-kill-functions nil
>> +   "List of functions called on the region before killing.
>> + This abnormal hook is run before `kill-region' and
>> + `copy-region-as-kill', with the beginning and end positions of
>> + the killed region as the arguments.")
>
> I think it's usually important to keep the property that killing doesn't
> modify the buffer's text (when used in copy-as-kill), so rather than having
> before-kill-functions modify the buffer's text just before it's put in the
> kill-ring, it would be more convenient to make it return the string,
> i.e. use it as a replacement for buffer-substring.

I like the idea, but returning a value from a hook is rather
inconvenient -- if there are multiple functions on the hook, which
value should you use eventually?

Perhaps before-kill-functions should get START, END and STRING
as arguments where STRING is the result of buffer-substring.

Each hook could then modify that string as they please.


Or perhaps a different approach:

Run after-kill-hooks (a normal hook!) as the last thing in kill-region
and copy-region-as-kill, i.e. after putting the string into
the kill-ring.  If necessary, the hooks can modify the head of
the kill-ring as they please...?

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: require-hard-newlines to use newline

[Chong Yidong]

>> I think it's usually important to keep the property that killing doesn't
>> modify the buffer's text (when used in copy-as-kill), so rather than
>> having before-kill-functions modify the buffer's text just before it's
>> put in the kill-ring, it would be more convenient to make it return the
>> string, i.e. use it as a replacement for buffer-substring.
>
> I like the idea, but returning a value from a hook is rather
> inconvenient -- if there are multiple functions on the hook, which
> value should you use eventually?

Another possibility is not to use a hook at all, e.g.:

(defvar kill-filters nil
  "List of functions for converting a string before it is killed.
Each function should accept a single argument, a string, and
return a string.  Whenever a string is killed, it is passed as an
argument to the first function in the list, and the return value
of each function is passed as the argument to the next.  The
final return value is the string that is actually put in the kill
ring and passed to `interprogram-cut-function'.")

Re: require-hard-newlines to use newline

[Chong Yidong]

> Or perhaps a different approach:
>
> Run after-kill-hooks (a normal hook!) as the last thing in kill-region
> and copy-region-as-kill, i.e. after putting the string into
> the kill-ring.  If necessary, the hooks can modify the head of
> the kill-ring as they please...?

There would be no way for them to modify the text that's been stored into
the X selection or clipboard.

Re: require-hard-newlines to use newline

[Stefan Monnier]

> (defvar kill-filters nil
>   "List of functions for converting a string before it is killed.
> Each function should accept a single argument, a string, and
> return a string.  Whenever a string is killed, it is passed as an
> argument to the first function in the list, and the return value
> of each function is passed as the argument to the next.  The
> final return value is the string that is actually put in the kill
> ring and passed to `interprogram-cut-function'.")

But that makes it difficult to process the string in a way that depends on
its context.


        Stefan

Re: require-hard-newlines to use newline

[Stefan Monnier]

>>> + (defvar before-kill-functions nil
>>> +   "List of functions called on the region before killing.
>>> + This abnormal hook is run before `kill-region' and
>>> + `copy-region-as-kill', with the beginning and end positions of
>>> + the killed region as the arguments.")
>> 
>> I think it's usually important to keep the property that killing doesn't
>> modify the buffer's text (when used in copy-as-kill), so rather than having
>> before-kill-functions modify the buffer's text just before it's put in the
>> kill-ring, it would be more convenient to make it return the string,
>> i.e. use it as a replacement for buffer-substring.

> I like the idea, but returning a value from a hook is rather
> inconvenient -- if there are multiple functions on the hook, which
> value should you use eventually?

> Perhaps before-kill-functions should get START, END and STRING
> as arguments where STRING is the result of buffer-substring.

> Each hook could then modify that string as they please.

Fair enough.

> Or perhaps a different approach:
> Run after-kill-hooks (a normal hook!) as the last thing in kill-region
> and copy-region-as-kill, i.e. after putting the string into
> the kill-ring.  If necessary, the hooks can modify the head of
> the kill-ring as they please...?

I'm not sure if that would be reliable: the head of kill-ring might be the
result of concatenating the old head and the new text, where the old head
has already been processed by after-kill-hook, but you can't tell which part
is the old one and which part is the new one.


        Stefan

Re: require-hard-newlines to use newline

[Chong Yidong]

>> I like the idea, but returning a value from a hook is rather
>> inconvenient -- if there are multiple functions on the hook, which
>> value should you use eventually?
>
>> Perhaps before-kill-functions should get START, END and STRING
>> as arguments where STRING is the result of buffer-substring.
>
>> Each hook could then modify that string as they please.
>
> Fair enough.

Sorry, I'm probably missing something, but I don't understand how this
would work. Since each function would only be modifying their own local
STRING variable, not the one that will actually be killed... Unless it's
something like

  [in kill-region]:
  (let ((string (delete-and-extract-region beg end)))
    (run-hook-with-args 'before-kill-functions beg end #'string)
    ...

Then the hook function would have to set the string by

  (defun foo-function (symbol)
    (set symbol "replacement string"))

But that would be ridiculously arcane.

Re: require-hard-newlines to use newline

[Stefan Monnier]

> Sorry, I'm probably missing something, but I don't understand how this
> would work.

Simple: it wouldn't use run-hook* (or maybe it would introduce a new
run-hook-filter).


        Stefan

Re: require-hard-newlines to use newline

[Kim F. Storm]

"Chong Yidong" <cyd@stupidchicken.com> writes:

>>> I like the idea, but returning a value from a hook is rather
>>> inconvenient -- if there are multiple functions on the hook, which
>>> value should you use eventually?
>>
>>> Perhaps before-kill-functions should get START, END and STRING
>>> as arguments where STRING is the result of buffer-substring.
>>
>>> Each hook could then modify that string as they please.
>
> Sorry, I'm probably missing something, but I don't understand how this
> would work. Since each function would only be modifying their own local
> STRING variable, not the one that will actually be killed... 

It doesn't have to modify the variable `string', but its string value, e.g.

(defun x (s)
  (aset s 1 ?-))
(let ((s "abc"))
  (x s)
  s)
=> "a-c"

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: require-hard-newlines to use newline

[Chong Yidong]

>> Sorry, I'm probably missing something, but I don't understand how this
>> would work. Since each function would only be modifying their own local
>> STRING variable, not the one that will actually be killed...
>
> It doesn't have to modify the variable `string', but its string value,
> e.g.
>
> (defun x (s)
>   (aset s 1 ?-))
> (let ((s "abc"))
>   (x s)
>   s)
> => "a-c"

But this method restricts the possible manipulations to those that keep
the string length unchanged.  That is fine for longlines, which is just
swapping spaces and newlines, but it may not be very useful for other
(hypothetical) purposes.  Even for longlines, aset would be very
inconvenient to use.

Re: require-hard-newlines to use newline

[Chong Yidong]

>> Sorry, I'm probably missing something, but I don't understand how this
>> would work.
>
> Simple: it wouldn't use run-hook* (or maybe it would introduce a new
> run-hook-filter).

Not using a hook seems to be simpler than writing a new run-hook-filter
function.  How about the following?  It seems to be a bit overengineered,
but that's the only way to meet the various objections.


(defvar kill-filters nil
  "List of functions for converting a string before it is killed.
These are called by `kill-region' and `copy-region-as-kill' to
convert a buffer substring before putting it into the kill ring
and passing it to `interprogram-cut-function'.

Each function must accept three arguments: STRING, BEG, and END.
STRING is the string to be converted, and BEG and END are the
position arguments given to `kill-region' or
`copy-region-as-kill'.  Each function must return a string.

The buffer substring between BEG and END is passed as STRING to
the first function in the list, and the return value of each
function is passed as STRING to the next.  The final return value
is used as the killed string.

If this variable is nil, no filtering is performed.")

Re: require-hard-newlines to use newline

[Richard Stallman]

    The only way I can see for this to work is for the after-change function
    to test this-command, which I think is not very robust.

I don't understand.  Why check this-command?  Wouldn't it be correct
for longlines.el to operate on ALL text inserted in the buffer?

Re: require-hard-newlines to use newline

[Richard Stallman]

    > If we agree that those hooks are the proper approach, and you can
    > write the proper doc for them, this is a trivial change, so I don't
    > see why we have to wait.

    Okay then. How about this patch?

It only does a part of the job.  It handles kill, but not registers,
etc.  I think it is implemented at the wrong level.

Stefan wrote:

    I think it's usually important to keep the property that killing doesn't
    modify the buffer's text (when used in copy-as-kill), so rather than having
    before-kill-functions modify the buffer's text just before it's put in the
    kill-ring, it would be more convenient to make it return the string,
    i.e. use it as a replacement for buffer-substring.

That is true, too.

    Another possibility is not to use a hook at all, e.g.:

    (defvar kill-filters nil
      "List of functions for converting a string before it is killed.
    Each function should accept a single argument, a string, and
    return a string.

That is more the right idea, for the part that reads from
the buffer.  But it is not correct to associate this with
killing.  Think of it as a variant of buffer-substring.

    But that makes it difficult to process the string in a way that depends on
    its context.

Maybe so, but is it necessary to do that?

Re: require-hard-newlines to use newline

[Stefan Monnier]

>     But that makes it difficult to process the string in a way that
>     depends on its context.
> Maybe so, but is it necessary to do that?

I haven't thought really hard about it, but superficially, it would seem
like a pretty common occurrence that the transformation should be different
depending on the context (things like "was the beginning of the string at
BOL?", or maybe for message-mode "is this taken from the header or the body
of the email", ...).


        Stefan

Re: require-hard-newlines to use newline

[Chong Yidong]

> It only does a part of the job.  It handles kill, but not registers,
> etc.  I think it is implemented at the wrong level.
>
>>     (defvar kill-filters nil
>>       "List of functions for converting a string before it is killed.
>>     Each function should accept a single argument, a string, and
>>     return a string.
>
> That is more the right idea, for the part that reads from
> the buffer.  But it is not correct to associate this with
> killing.  Think of it as a variant of buffer-substring.

By this, I'm guessing you mean going to a lower level, e.g. calling the
filter functions from inside buffer-substring.  I'll look into this, but I
doubt it's worth it.

The current behavior of longlines.el, which is to filter at the level of
kill-region and copy-region-as-kill, addresses the main requirement of
users -- removing the soft newlines when copying into another program or
yanking into a non-longlines buffer.  All the bugs here have been pretty
much ironed out.  It doesn't deal with *all* the situations where
filtering might be good, like copy-to-register, but we could simply make
those commands use the filter too.

buffer-substring is called in many, many places, and I'm not confident
that filtering is appropriate everywhere.  For example, fill.el calls
buffer-substring all over the place, and one feature of longlines is that
M-q still works as usual; changing buffer-substring may screw it up.

I suggest implementing the `kill-filter' variable I proposed; then I will
change longlines.el to use it, and we can put longlines.el into Emacs. 
This version of longlines.el will be the same as the existing
"third-party" longlines.el, except it won't use advice.  A future version
can implement something more elegant. (I have other plans for longlines.el
too, including providing word wrap for files which are already full of
hard newlines.)

Re: require-hard-newlines to use newline

[Richard Stallman]

    > That is more the right idea, for the part that reads from
    > the buffer.  But it is not correct to associate this with
    > killing.  Think of it as a variant of buffer-substring.

    By this, I'm guessing you mean going to a lower level, e.g. calling the
    filter functions from inside buffer-substring.

No, buffer-substring should not call them.  That's why I said "a
variant of buffer-substring".  Some places would call this new
variant, but some would continue to call the existing, ordinary
buffer-substring function.

    I suggest implementing the `kill-filter' variable I proposed; then I will
    change longlines.el to use it, and we can put longlines.el into Emacs. 

That's not right.  It only does part of the job.  Killing is not the
only feature that needs to do this processing.

Anyway, what about my other suggestion?  The suggestion to make
longlines.el use after-change-functions to discover all new text inserted
in the buffer, and do longlines processing on that text?

Re: require-hard-newlines to use newline

[Richard Stallman]

    I haven't thought really hard about it, but superficially, it would seem
    like a pretty common occurrence that the transformation should be different
    depending on the context (things like "was the beginning of the string at
    BOL?", or maybe for message-mode "is this taken from the header or the body
    of the email", ...).

It's often a mistake to aim for generality that goes beyond the uses
we have envisioned, because any easy job can be made hard if you do
that.  Anyway, it is easy enough to have a convention to put point at
the start of where the text came from, before running the hook
functions.

Re: require-hard-newlines to use newline

[Chong Yidong]

> Anyway, what about my other suggestion?  The suggestion to make
> longlines.el use after-change-functions to discover all new text inserted
> in the buffer, and do longlines processing on that text?

I am already making longlines.el use post-command-hook to process text
yanked *into* the longlines buffer.  But I need a separate mechanism to
process text yanked *out of* the buffer, and neither post-command-hook nor
after-change-functions are suitable for this.  By the time the hook is
called, the text has already been put into the kill ring and clipboard.

> No, buffer-substring should not call them.  That's why I said "a
> variant of buffer-substring".  Some places would call this new
> variant, but some would continue to call the existing, ordinary
> buffer-substring function.

In that case, I don't think our ideas are so different.  I'll alter my
patch to implement the `kill-filters' variable, and make the appropriate
functions use it before calling buffer-substring.  So far, these functions
are `kill-region', `copy-region-as-kill', and `copy-to-register'; are
there others?

Re: require-hard-newlines to use newline

[Richard Stallman]

    > Anyway, what about my other suggestion?  The suggestion to make
    > longlines.el use after-change-functions to discover all new text inserted
    > in the buffer, and do longlines processing on that text?

    I am already making longlines.el use post-command-hook to process text
    yanked *into* the longlines buffer.

Ok.  So all that's missing is to process the text that is copied OUT OF
this buffer.

In that case, if the hook you've implemented does the job for
longlines.el, let's install it.