Longlines and insert

Longlines and insert

[Chong Yidong]

I got a bug report from a user who had longlines-mode turned on for
text-mode and all its derivatives, and found that it didn't play well
with `M-x mail'.

(The fact that longlines was originally written as a hack is sadly
apparent when situations like this arise...)

> (add-hook 'text-mode-hook 'longlines-mode)
>
> I have to disable longlines in mail-mode-hook to reenable it in
> wl-mail-setup-hook. This seems to be because of some filling
> operations when mail-mode is initialized. Don't know, is this a
> problem with longlines or mail-mode?
>
> The second problem is bothering me more. It has to do with attaching
> a binary file when editing an email. It looks like a space is turned
> into a hard newline

The problem is that mail-setup inserts the header fields by calling

  (insert "blahblah\n")

The inserted newlines are not marked as hard, so longlines gets
confused.

I could fix this by going through sendmail.el and adding (newline) to
all these places.  But maybe a more wide-reaching solution is called
for.

One possibility is to define a new variable `insert-filters', to store
a list of filter functions to run on strings before they are inserted
(analogous to `buffer-substring-filters').

I don't know if this is too invasive a change, especially before a
release, and just for the sake of longlines mode.

What do people think?

Re: Longlines and insert

[Kevin Rodgers]

Chong Yidong wrote:
 > The problem is that mail-setup inserts the header fields by calling
 >
 >   (insert "blahblah\n")
 >
 > The inserted newlines are not marked as hard, so longlines gets
 > confused.
 >
 > I could fix this by going through sendmail.el and adding (newline) to
 > all these places.  But maybe a more wide-reaching solution is called
 > for.

How about a less invasive change?  The headers are actually inserted
like this:

(insert header "\n")

Which I think could be fixed like this:

(let ((hard-newline "\n"))
   ;; see set-hard-newline-properties:
   (put-text-property 0 1 'hard t hard-newline)
   (put-text-property 0 1 'rear-nonsticky '(hard) hard-newline)
   ...
   (insert header hard-newline))

-- 
Kevin Rodgers

Re: Longlines and insert

[Paul Pogonyshev]

Kevin Rodgers wrote:
> Chong Yidong wrote:
>  > The problem is that mail-setup inserts the header fields by calling
>  >
>  >   (insert "blahblah\n")
>  >
>  > The inserted newlines are not marked as hard, so longlines gets
>  > confused.
>  >
>  > I could fix this by going through sendmail.el and adding (newline) to
>  > all these places.  But maybe a more wide-reaching solution is called
>  > for.
> 
> How about a less invasive change?  The headers are actually inserted
> like this:
> 
> (insert header "\n")
> 
> Which I think could be fixed like this:
> 
> (let ((hard-newline "\n"))
>    ;; see set-hard-newline-properties:
>    (put-text-property 0 1 'hard t hard-newline)
>    (put-text-property 0 1 'rear-nonsticky '(hard) hard-newline)
>    ...
>    (insert header hard-newline))

I think (and as far as I understood, Chong Yidong does too) that this is
a welcoming message for future problems.  You patch up a piece of code,
while there are hundreds other lisp files, many of which insert newlines,
some of which need to be hard.  We probably need something generic.  At
the very least we need a convenience function to create a newline-string
with the necessary properties already set, so we don't have to go over
the lines above each time.

Paul

Re: Longlines and insert

[Kevin Rodgers]

Paul Pogonyshev wrote:
 > I think (and as far as I understood, Chong Yidong does too) that this is
 > a welcoming message for future problems.  You patch up a piece of code,
 > while there are hundreds other lisp files, many of which insert newlines,
 > some of which need to be hard.  We probably need something generic.  At
 > the very least we need a convenience function to create a newline-string
 > with the necessary properties already set, so we don't have to go over
 > the lines above each time.

CY has identified a single Lisp file (sendmail.el) that needs to be
fixed, and we can guess that other message composition libraries do as
well.

Right now the most generic solution is the one he suggested:
(let ((use-hard-newlines t)) (newline))

But I agree with you that a convenience function for strings would be
better.  A good start would be to add an &optional OBJECT argument for
set-hard-newline-properties that gets passed to put-text-property.

-- 
Kevin Rodgers

Re: Longlines and insert

[Kim F. Storm]

Paul Pogonyshev <pogonyshev@gmx.net> writes:

>> Which I think could be fixed like this:
>> 
>> (let ((hard-newline "\n"))
>>    ;; see set-hard-newline-properties:
>>    (put-text-property 0 1 'hard t hard-newline)
>>    (put-text-property 0 1 'rear-nonsticky '(hard) hard-newline)
>>    ...
>>    (insert header hard-newline))

We could put a generic definition into subr.el
which could then be used by any code needing a hard newline.

(defconst hard-newline
          (propertize "\n"
          'hard t 'rear-nonsticky '(hard)))

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: Longlines and insert

[Stefan Monnier]

> I think (and as far as I understood, Chong Yidong does too) that this is
> a welcoming message for future problems.  You patch up a piece of code,
> while there are hundreds other lisp files, many of which insert newlines,
> some of which need to be hard.  We probably need something generic.  At
> the very least we need a convenience function to create a newline-string
> with the necessary properties already set, so we don't have to go over
> the lines above each time.

Maybe longlines-mode could provide a variable
"longlines-inserted-LF-are-hard" and then use it in an
after-change-functions hook to mark all inserted LF as hard when that
variable is non-nil.

I'd guess that the variable should be non-nil by default and only bound to
nil at a few specific spots, hopefully all of them in longlines.el.


        Stefan

Re: Longlines and insert

[Ryan Yeske]

   > Which I think could be fixed like this:
   > 
   > (let ((hard-newline "\n"))
   >    ;; see set-hard-newline-properties:
   >    (put-text-property 0 1 'hard t hard-newline)
   >    (put-text-property 0 1 'rear-nonsticky '(hard) hard-newline)
   >    ...
   >    (insert header hard-newline))

   I think (and as far as I understood, Chong Yidong does too) that this is
   a welcoming message for future problems.  You patch up a piece of code,
   while there are hundreds other lisp files, many of which insert newlines,
   some of which need to be hard.  We probably need something generic.  At
   the very least we need a convenience function to create a newline-string
   with the necessary properties already set, so we don't have to go over
   the lines above each time.

Doesn't (newline) insert the right kind of newline based on the value
of `use-hard-newlines'?

Ryan

Re: Longlines and insert

[Chong Yidong]

>    I think (and as far as I understood, Chong Yidong does too) that this is
>    a welcoming message for future problems.  You patch up a piece of code,
>    while there are hundreds other lisp files, many of which insert newlines,
>    some of which need to be hard.  We probably need something generic.  At
>    the very least we need a convenience function to create a newline-string
>    with the necessary properties already set, so we don't have to go over
>    the lines above each time.
>
> Doesn't (newline) insert the right kind of newline based on the value
> of `use-hard-newlines'?

`newline' does.  The problem is Lisp code that does (insert "foo\n")

Re: Longlines and insert

[Chong Yidong]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Maybe longlines-mode could provide a variable
> "longlines-inserted-LF-are-hard" and then use it in an
> after-change-functions hook to mark all inserted LF as hard when that
> variable is non-nil.
>
> I'd guess that the variable should be non-nil by default and only bound to
> nil at a few specific spots, hopefully all of them in longlines.el.

The problem with using an after-change-function is that the function
is only told that the text was changed in a particular region, not
what the change was.  There is no way to tell whether a soft newline
encountered in that region is a legit soft newline, or one produced by
an (insert "foo\n") call.

Re: Longlines and insert

[Chong Yidong]

Chong Yidong <cyd@stupidchicken.com> writes:

> One possibility is to define a new variable `insert-filters', to store
> a list of filter functions to run on strings before they are inserted
> (analogous to `buffer-substring-filters').

Let me be a little more specific: this would be a variable
`insert-string-filters', normally bound to nil.  insert_string (in
insdel.c) checks the value of insert-string-filters passed to it.  If
non-nil, it makes use of it as follows:

Like `buffer-substring-filters', `insert-string-filters' would contain
a list of function, each of which accepts a single argument, a string,
and returns a string.  The string argument is passed to the first
function in the list, and the return value of each function is passed
to the next.  The last return value is the one actually inserted.

Re: Longlines and insert

[Chong Yidong]

> We could put a generic definition into subr.el
> which could then be used by any code needing a hard newline.
>
> (defconst hard-newline
>           (propertize "\n"
>           'hard t 'rear-nonsticky '(hard)))

This would work.  So, which approach should I use?

Re: Longlines and insert

[Stefan Monnier]

>> Maybe longlines-mode could provide a variable
>> "longlines-inserted-LF-are-hard" and then use it in an
>> after-change-functions hook to mark all inserted LF as hard when that
>> variable is non-nil.
>> 
>> I'd guess that the variable should be non-nil by default and only bound to
>> nil at a few specific spots, hopefully all of them in longlines.el.

> The problem with using an after-change-function is that the function
> is only told that the text was changed in a particular region, not
> what the change was.  There is no way to tell whether a soft newline
> encountered in that region is a legit soft newline, or one produced by
> an (insert "foo\n") call.

If the `len' argument to the after-change-function is zero, it means it's
an insert.  Now my suggestion is probably bogus indeed 'cause I know next to
nothing about longlines-mode.  E.g. I'm not sure what you mean by "a legit
soft newline".


        Stefan

Re: Longlines and insert

[Richard M. Stallman]

    We could put a generic definition into subr.el
    which could then be used by any code needing a hard newline.

    (defconst hard-newline
	      (propertize "\n"
	      'hard t 'rear-nonsticky '(hard)))

I like that approach since it is painless and
certainly won't break anything.

Re: Longlines and insert

[Richard M. Stallman]

    > Maybe longlines-mode could provide a variable
    > "longlines-inserted-LF-are-hard" and then use it in an
    > after-change-functions hook to mark all inserted LF as hard when that
    > variable is non-nil.
    >
    > I'd guess that the variable should be non-nil by default and only bound to
    > nil at a few specific spots, hopefully all of them in longlines.el.

    The problem with using an after-change-function is that the function
    is only told that the text was changed in a particular region, not
    what the change was.  There is no way to tell whether a soft newline
    encountered in that region is a legit soft newline, or one produced by
    an (insert "foo\n") call.

A before-change-function can record the old text, and the
after-change-function can compare the new text with it.  That way it can
see whether newlines were inserted.

However, if we really want a feature that makes all inserted newlines hard,
it would be easier to implement that within `insert' itself.

    Let me be a little more specific: this would be a variable
    `insert-string-filters', normally bound to nil.  insert_string (in
    insdel.c) checks the value of insert-string-filters passed to it.  If
    non-nil, it makes use of it as follows:

I think the added generality of that feature would, in this case, be
a mistake.  It would make things slower and harder to use.

If the hard-newline variable is enough to do the job,
I would rather stick to that idea.  It is simpler and less risky.

Re: Longlines and insert

[Chong Yidong]

> If the hard-newline variable is enough to do the job,
> I would rather stick to that idea.  It is simpler and less risky.

OK, I'll do that.

Re: Longlines and insert

[Kevin Rodgers]

Richard M. Stallman wrote:
 > If the hard-newline variable is enough to do the job,
 > I would rather stick to that idea.  It is simpler and less risky.

Wouldn't the easiest thing be to add a call to
(set-hard-newline-properties (point-min) (point)) after
mail-header-separator and its trailing newline have been inserted?

-- 
Kevin Rodgers

Re: Longlines and insert

[Paul Pogonyshev]

Kevin Rodgers wrote:
> Richard M. Stallman wrote:
>  > If the hard-newline variable is enough to do the job,
>  > I would rather stick to that idea.  It is simpler and less risky.
> 
> Wouldn't the easiest thing be to add a call to
> (set-hard-newline-properties (point-min) (point)) after
> mail-header-separator and its trailing newline have been inserted?

Maybe, but that defeats the intention of providing a generic solution.

Paul

Re: Longlines and insert

[Kevin Rodgers]

Paul Pogonyshev wrote:
 >Kevin Rodgers wrote:
 >>Wouldn't the easiest thing be to add a call to
 >>(set-hard-newline-properties (point-min) (point)) after
 >>mail-header-separator and its trailing newline have been inserted?
 >
 > Maybe, but that defeats the intention of providing a generic solution.

How is the set-hard-newline-properties function any less generic than
the hard-newline variable?  (Remember, the variable proposed by Kim is
not a boolean that controls the behavior of the insert function; it is a
string that has to be explicitly passed to the insert function.)

-- 
Kevin Rodgers

Re: Longlines and insert

[Paul Pogonyshev]

Kevin Rodgers wrote:
> Paul Pogonyshev wrote:
>  >Kevin Rodgers wrote:
>  >>Wouldn't the easiest thing be to add a call to
>  >>(set-hard-newline-properties (point-min) (point)) after
>  >>mail-header-separator and its trailing newline have been inserted?
>  >
>  > Maybe, but that defeats the intention of providing a generic solution.
> 
> How is the set-hard-newline-properties function any less generic than
> the hard-newline variable?  (Remember, the variable proposed by Kim is
> not a boolean that controls the behavior of the insert function; it is a
> string that has to be explicitly passed to the insert function.)

Uh, sorry, I don't know how I was reading your message, but I managed to
seriously misread it.

Yes, `set-hard-newline-properties' looks fine as well, especially if it
is documented.  But I think we should add that `hard-newline' constant
too.  Sometimes you need to pass strings to various functions and don't
have a (clean) way to work directly with the buffer.  In such cases the
constant is useful.

Paul

Re: Longlines and insert

[Richard M. Stallman]

    How is the set-hard-newline-properties function any less generic than
    the hard-newline variable?

This question is a side issue, and not relevant to further work,
so I ask people please not to send any more mail about this.