Patch: Syntax and Hard Newlines

Patch: Syntax and Hard Newlines

[Herbert Euler]

Emacs provides a hard newline mode.  If you type M-x use-hard-newlines
RET, you turn on the hard newline mode.  In this mode, all newlines in
a buffer are re-scanned and marked as either a ``hard'' one or a
``soft'' one.  The behavior of commands inserting a newline character
is also altered: `newline' and `open-line' will add the text property
`hard' (value set to t) to newlines that they insert.

As documented in the doc string of the command `use-hard-newlines',
newlines not marked hard are internal to paragraphs.  However,
currently the syntax feature does not work well with this.  For
example, if you turn on long-lines mode with M-x longlines-mode RET in
the *scratch* buffer, you will see the word `evaluation' in the first
line is wrapped into the second line, but not recognized as a part of
the comment.

To fix this, we can let the function `forw_comment' in syntax.c skip
the soft newlines when possible.  Below is the patch.

Regards,
Guanpeng Xu

Index: syntax.c
===================================================================
RCS file: /sources/emacs/emacs/src/syntax.c,v
retrieving revision 1.166.4.20
diff -r1.166.4.20 syntax.c
2126a2127
>   register int skip_soft_newlines;
2129a2131,2137
>   /* If `use-hard-newlines' is t, only newlines with the `hard'
>      property set to t are real newlines.  Other newlines are soft and
>      should be skipped. */
>   skip_soft_newlines = !NILP (Fsymbol_value
>                               (intern
>                                ("use-hard-newlines")));
>
2145a2154,2163
>       if (skip_soft_newlines
>           && c == '\n'
>           && NILP (Fget_text_property (make_number (from_byte),
>                                        intern ("hard"),
>                                        Qnil)))
>         {
>           INC_BOTH (from, from_byte);
>           UPDATE_SYNTAX_TABLE_FORWARD (from);
>           continue;
>         }

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

RE: Patch: Syntax and Hard Newlines

[Herbert Euler]

Sorry but in the patch format:

Index: syntax.c
===================================================================
RCS file: /sources/emacs/emacs/src/syntax.c,v
retrieving revision 1.166.4.20
diff -u -p -r1.166.4.20 syntax.c
--- syntax.c    20 Sep 2006 06:04:11 -0000      1.166.4.20
+++ syntax.c    13 Nov 2006 05:51:53 -0000
@@ -2124,9 +2124,17 @@ forw_comment (from, from_byte, stop, nes
   register int c, c1;
   register enum syntaxcode code;
   register int syntax;
+  register int skip_soft_newlines;

   if (nesting <= 0) nesting = -1;

+  /* If `use-hard-newlines' is t, only newlines with the `hard'
+     property set to t are real newlines.  Other newlines are soft and
+     should be skipped. */
+  skip_soft_newlines = !NILP (Fsymbol_value
+                              (intern
+                               ("use-hard-newlines")));
+
   /* Enter the loop in the middle so that we find
      a 2-char comment ender if we start in the middle of it.  */
   syntax = prev_syntax;
@@ -2143,6 +2151,16 @@ forw_comment (from, from_byte, stop, nes
        }
       c = FETCH_CHAR_AS_MULTIBYTE (from_byte);
       syntax = SYNTAX_WITH_FLAGS (c);
+      if (skip_soft_newlines
+          && c == '\n'
+          && NILP (Fget_text_property (make_number (from_byte),
+                                       intern ("hard"),
+                                       Qnil)))
+        {
+          INC_BOTH (from, from_byte);
+          UPDATE_SYNTAX_TABLE_FORWARD (from);
+          continue;
+        }
       code = syntax & 0xff;
       if (code == Sendcomment
          && SYNTAX_FLAGS_COMMENT_STYLE (syntax) == style


>From: "Herbert Euler" <herberteuler@hotmail.com>
>To: emacs-devel@gnu.org
>Subject: Patch: Syntax and Hard Newlines
>Date: Mon, 13 Nov 2006 13:24:59 +0800
>
>Emacs provides a hard newline mode.  If you type M-x use-hard-newlines
>RET, you turn on the hard newline mode.  In this mode, all newlines in
>a buffer are re-scanned and marked as either a ``hard'' one or a
>``soft'' one.  The behavior of commands inserting a newline character
>is also altered: `newline' and `open-line' will add the text property
>`hard' (value set to t) to newlines that they insert.
>
>As documented in the doc string of the command `use-hard-newlines',
>newlines not marked hard are internal to paragraphs.  However,
>currently the syntax feature does not work well with this.  For
>example, if you turn on long-lines mode with M-x longlines-mode RET in
>the *scratch* buffer, you will see the word `evaluation' in the first
>line is wrapped into the second line, but not recognized as a part of
>the comment.
>
>To fix this, we can let the function `forw_comment' in syntax.c skip
>the soft newlines when possible.  Below is the patch.
>
>Regards,
>Guanpeng Xu
>
>Index: syntax.c
>===================================================================
>RCS file: /sources/emacs/emacs/src/syntax.c,v
>retrieving revision 1.166.4.20
>diff -r1.166.4.20 syntax.c
>2126a2127
>>   register int skip_soft_newlines;
>2129a2131,2137
>>   /* If `use-hard-newlines' is t, only newlines with the `hard'
>>      property set to t are real newlines.  Other newlines are soft and
>>      should be skipped. */
>>   skip_soft_newlines = !NILP (Fsymbol_value
>>                               (intern
>>                                ("use-hard-newlines")));
>>
>2145a2154,2163
>>       if (skip_soft_newlines
>>           && c == '\n'
>>           && NILP (Fget_text_property (make_number (from_byte),
>>                                        intern ("hard"),
>>                                        Qnil)))
>>         {
>>           INC_BOTH (from, from_byte);
>>           UPDATE_SYNTAX_TABLE_FORWARD (from);
>>           continue;
>>         }
>
>_________________________________________________________________
>Express yourself instantly with MSN Messenger! Download today it's FREE! 
>http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/
>
>
>
>_______________________________________________
>Emacs-devel mailing list
>Emacs-devel@gnu.org
>http://lists.gnu.org/mailman/listinfo/emacs-devel

_________________________________________________________________
Don't just search. Find. Check out the new MSN Search! 
http://search.msn.click-url.com/go/onm00200636ave/direct/01/

RE: Patch: Syntax and Hard Newlines

[Herbert Euler]

I'm so sorry, but I re-generated it in the format
described in the "Sending Patches" secion in Emacs
manual.

I will follow what are said there in the future.

Index: syntax.c
===================================================================
RCS file: /sources/emacs/emacs/src/syntax.c,v
retrieving revision 1.166.4.20
diff -c -r1.166.4.20 syntax.c
*** syntax.c    20 Sep 2006 06:04:11 -0000      1.166.4.20
--- syntax.c    13 Nov 2006 10:02:58 -0000
***************
*** 2124,2132 ****
--- 2124,2140 ----
    register int c, c1;
    register enum syntaxcode code;
    register int syntax;
+   register int skip_soft_newlines;

    if (nesting <= 0) nesting = -1;

+   /* If `use-hard-newlines' is t, only newlines with the `hard'
+      property set to t are real newlines.  Other newlines are soft and
+      should be skipped. */
+   skip_soft_newlines = !NILP (Fsymbol_value
+                               (intern
+                                ("use-hard-newlines")));
+
    /* Enter the loop in the middle so that we find
       a 2-char comment ender if we start in the middle of it.  */
    syntax = prev_syntax;
***************
*** 2143,2148 ****
--- 2151,2166 ----
        }
        c = FETCH_CHAR_AS_MULTIBYTE (from_byte);
        syntax = SYNTAX_WITH_FLAGS (c);
+       if (skip_soft_newlines
+           && c == '\n'
+           && NILP (Fget_text_property (make_number (from_byte),
+                                        intern ("hard"),
+                                        Qnil)))
+         {
+           INC_BOTH (from, from_byte);
+           UPDATE_SYNTAX_TABLE_FORWARD (from);
+           continue;
+         }
        code = syntax & 0xff;
        if (code == Sendcomment
          && SYNTAX_FLAGS_COMMENT_STYLE (syntax) == style

_________________________________________________________________
FREE pop-up blocking with the new MSN Toolbar - get it now! 
http://toolbar.msn.click-url.com/go/onm00200415ave/direct/01/

Re: Patch: Syntax and Hard Newlines

[Richard Stallman]

    +  skip_soft_newlines = !NILP (Fsymbol_value
    +                              (intern
    +                               ("use-hard-newlines")));
    +

It would look nicer to break the line like this:

    +  skip_soft_newlines
    +    = !NILP (Fsymbol_value (intern ("use-hard-newlines")));
    +

However, if C code is going to examine the value of that variable 
frequently, it would be more efficient to make it a C-level
variable defined with DEFVAR_PER_BUFFER in buffer.c.

Re: Patch: Syntax and Hard Newlines

[Herbert Euler]

>However, if C code is going to examine the value of that variable
>frequently, it would be more efficient to make it a C-level
>variable defined with DEFVAR_PER_BUFFER in buffer.c.

The `use-hard-newlines' mode is defined in ``paragraphs.el'' as

(define-minor-mode use-hard-newlines
  ...

So, the variable `use-hard-newlines' is defined by evaluating the
forms returned by calling macro `define-minor-mode'.  I know
if the forms like

(defvar var val "doc")

are evaluated many times, `val' will not take effect except for the
first time, but can we just define the variable `use-hard-newlines'
in buffer.c without changing the definition of `use-hard-newlines'
that defined with `define-minor-mode'?

Regards,
Guanpeng Xu

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

Re: Patch: Syntax and Hard Newlines

[Richard Stallman]

There is no conflict between use of define-minor-mode
and implementing a variable at C level.

Re: Patch: Syntax and Hard Newlines

[Stefan Monnier]

> As documented in the doc string of the command `use-hard-newlines',
> newlines not marked hard are internal to paragraphs.  However,
> currently the syntax feature does not work well with this.  For
> example, if you turn on long-lines mode with M-x longlines-mode RET in
> the *scratch* buffer, you will see the word `evaluation' in the first
> line is wrapped into the second line, but not recognized as a part of
> the comment.

It's not clear at all that it's a bug rather than a feature.
I think if it's a bug, the bug is in longlines.el since it is longlines
which decides that soft-newlines are basically "display only artifacts".

In most other contexts, soft newlines are much more real (they're soft, but
they're newlines nevertheless).


        Stefan

Re: Patch: Syntax and Hard Newlines

[Herbert Euler]

>I think if it's a bug, the bug is in longlines.el since it is longlines
>which decides that soft-newlines are basically "display only artifacts".

In fact, I wanted to adjust longlines' code by setting the `face'
property to `font-lock-comment-face' for some words.  However,
I found that the properties are cleared and reset.  And then I
traced into `parse-partial-sexp', then `scan_sexps_forward',
then `forw_comment'.

And the patch did fix the problem.

Regards,
Guanpeng Xu

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

Re: Patch: Syntax and Hard Newlines

[Stefan Monnier]

>> I think if it's a bug, the bug is in longlines.el since it is longlines
>> which decides that soft-newlines are basically "display only artifacts".

> In fact, I wanted to adjust longlines' code by setting the `face'
> property to `font-lock-comment-face' for some words.  However,
> I found that the properties are cleared and reset.  And then I
> traced into `parse-partial-sexp', then `scan_sexps_forward',
> then `forw_comment'.

> And the patch did fix the problem.

Another way to fix it, local to longlines.el is to add a syntax-table
property to soft-newlines so as to prevent them from being treated like an
end-of-comment.


        Stefan

Re: Patch: Syntax and Hard Newlines

[Herbert Euler]

>Another way to fix it, local to longlines.el is to add a syntax-table
>property to soft-newlines so as to prevent them from being treated like an
>end-of-comment.

I'd ever added

      (progn (insert-before-markers ?\n)
                (backward-char 1)
+               (add-text-properties (point)
+                                            (point)
+                                            '(syntax-table (11)))
                (delete-char -1)

to `longlines-wrap-line', but it did not work.

Regards,
Guanpeng Xu

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

Re: Patch: Syntax and Hard Newlines

[Stefan Monnier]

>> Another way to fix it, local to longlines.el is to add a syntax-table
>> property to soft-newlines so as to prevent them from being treated like an
>> end-of-comment.

> I'd ever added

>      (progn (insert-before-markers ?\n)
>                (backward-char 1)
> +               (add-text-properties (point)
> +                                            (point)
> +                                            '(syntax-table (11)))
>                (delete-char -1)

> to `longlines-wrap-line', but it did not work.

No wonder: you place the property on 0 chars.
Can you try with a second arg of (+ 1 (point))?


        Stefan

Re: Patch: Syntax and Hard Newlines

[Herbert Euler]

Ok, now we get two patches:

        * longlines.el (longlines-mode): Set
        `parse-sexp-lookup-properties' so that syntax parsing uses text
        properties.
        (longlines-wrap-line): Put the `syntax-table' text property to the
        soft newline if in a comment context.

*** longlines.el.~1.2.2.16.~    Mon Nov 13 10:58:17 2006
--- longlines.el        Wed Nov 15 09:27:18 2006
***************
*** 112,117 ****
--- 112,118 ----
          (make-local-variable 'buffer-substring-filters)
        (set (make-local-variable 'isearch-search-fun-function)
             'longlines-search-function)
+       (set (make-local-variable 'parse-sexp-lookup-properties) t)
          (add-to-list 'buffer-substring-filters 'longlines-encode-string)
          (when longlines-wrap-follows-window-size
            (set (make-local-variable 'fill-column)
***************
*** 156,161 ****
--- 157,163 ----
            (add-hook 'post-command-hook
                      'longlines-post-command-function nil t)))
      ;; Turn off longlines mode
+     (kill-local-variable 'parse-sexp-lookup-properties)
      (setq buffer-file-format (delete 'longlines buffer-file-format))
      (if longlines-showing
          (longlines-unshow-hard-newlines))
*************** (defun longlines-wrap-line ()
*** 243,248 ****
--- 245,257 ----
        (progn (insert-before-markers ?\n)
             (backward-char 1)
               (delete-char -1)
+            (let* ((point (point))
+                   (state (syntax-ppss point)))
+              (if (nth 4 state)
+                  ;; We are in a comment context.
+                  (add-text-properties point
+                                       (1+ point)
+                                       '(syntax-table (14)))))
             (forward-char 1)
               nil)
      (if (longlines-merge-lines-p)

Regards,
Guanpeng Xu

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

Re: Patch: Syntax and Hard Newlines

[Stefan Monnier]

> *************** (defun longlines-wrap-line ()
> *** 243,248 ****
> --- 245,257 ----
>        (progn (insert-before-markers ?\n)
>             (backward-char 1)
>               (delete-char -1)
> +            (let* ((point (point))
> +                   (state (syntax-ppss point)))
> +              (if (nth 4 state)
> +                  ;; We are in a comment context.
> +                  (add-text-properties point
> +                                       (1+ point)
> +                                       '(syntax-table (14)))))
>             (forward-char 1)
>               nil)
>      (if (longlines-merge-lines-p)

Why not just unconditionally do

                  (put-text-property (point)
                                     (1+ (point))
                                     'syntax-table '(0))

Or even just:

      (progn (insert-before-markers (propertize "\n" 'syntax-table '(0)))
             ...)


-- Stefan

Re: Patch: Syntax and Hard Newlines

[Herbert Euler]

> > *************** (defun longlines-wrap-line ()
> > *** 243,248 ****
> > --- 245,257 ----
> >        (progn (insert-before-markers ?\n)
> >             (backward-char 1)
> >               (delete-char -1)
> > +            (let* ((point (point))
> > +                   (state (syntax-ppss point)))
> > +              (if (nth 4 state)
> > +                  ;; We are in a comment context.
> > +                  (add-text-properties point
> > +                                       (1+ point)
> > +                                       '(syntax-table (14)))))
> >             (forward-char 1)
> >               nil)
> >      (if (longlines-merge-lines-p)
>
>Why not just unconditionally do
>
>                   (put-text-property (point)
>                                      (1+ (point))
>                                      'syntax-table '(0))
>
>Or even just:
>
>       (progn (insert-before-markers (propertize "\n" 'syntax-table '(0)))
>              ...)

Yes, that works.

Now the thing that should be considered is whether
soft newlines should be regarded the same as blank
characters.

>From the syntax's point of view, it should not.  Moreover,
since there exists the syntax text property feature, it seems
that doing something in the Lisp level will be more flexible
than doing in the C level.

I'd ever used some editors for editing formatted text.  In
such editors, the idea of soft newlines is indeed that soft
newlines and blank characters are interchangeable.
`longlines' mode also means this: it substitutes soft newlines
with spaces when saving files or quiting.  From this point
of view, the fix should be done in the C level to avoid
each code that uses soft newlines takes care of this problem.

Or in the Lisp level, perhaps we can make adding a
`syntax-table' text property default when inserting soft
newlines.

Regards,
Guanpeng Xu

_________________________________________________________________
Don't just search. Find. Check out the new MSN Search! 
http://search.msn.com/

Re: Patch: Syntax and Hard Newlines

[Richard Stallman]

    Now the thing that should be considered is whether
    soft newlines should be regarded the same as blank
    characters.

I think so.  After all, they are converted to and from spaces.

    Or in the Lisp level, perhaps we can make adding a
    `syntax-table' text property default when inserting soft
    newlines.

Since that is the method we are using, why not?

Re: Patch: Syntax and Hard Newlines

[Herbert Euler]

>     Or in the Lisp level, perhaps we can make adding a
>     `syntax-table' text property default when inserting soft
>     newlines.
>
>Since that is the method we are using, why not?

Looking through fill.el, I find that many filling functions such as
`fill-paragraph', `fill-region', `fill-individual-paragraphs' etc
finally invokes `fill-region-as-paragraph', which inserts (soft)
newlines with `fill-newline'.  It is defined as follows:

(defun fill-newline ()
  ;; Replace whitespace here with one newline, then
  ;; indent to left margin.
  (skip-chars-backward " \t")
  (insert ?\n)
  ;; Give newline the properties of the space(s) it replaces
  (set-text-properties (1- (point)) (point)
                       (fill-text-properties-at (point)))
  (and (looking-at "\\( [ \t]*\\)\\(\\c|\\)?")
       (or (aref (char-category-set (or (char-before (1- (point))) ?\000)) 
?|)
           (match-end 2))
       ;; When refilling later on, this newline would normally not be 
replaced
       ;; by a space, so we need to mark it specially to re-install the 
space
       ;; when we unfill.
       (put-text-property (1- (point)) (point) 'fill-space (match-string 
1)))
  ;; If we don't want breaks in invisible text, don't insert
  ;; an invisible newline.
  (if fill-nobreak-invisible
      (remove-text-properties (1- (point)) (point)
                              '(invisible t)))
  (if (or fill-prefix
          (not fill-indent-according-to-mode))
      (fill-indent-to-left-margin)
    (indent-according-to-mode))
  ;; Insert the fill prefix after indentation.
  (and fill-prefix (not (equal fill-prefix ""))
       ;; Markers that were after the whitespace are now at point: insert
       ;; before them so they don't get stuck before the prefix.
       (insert-before-markers-and-inherit fill-prefix)))

If all filling functions insert soft newlines with `fill-newline',
changing this function will be Ok.  And if making this change is Ok,
how about adding a customizable variable to control whether putting
the `syntax-table' property?  This would require as little functions
to be changed as possible.

Regards,
Guanpeng Xu

_________________________________________________________________
Don't just search. Find. Check out the new MSN Search! 
http://search.msn.com/

Re: Patch: Syntax and Hard Newlines

[Stefan Monnier]

> If all filling functions insert soft newlines with `fill-newline',
> changing this function will be Ok.  And if making this change is Ok,
> how about adding a customizable variable to control whether putting
> the `syntax-table' property?  This would require as little functions
> to be changed as possible.

I think it's the right idea.  But we need to make auto-fill-mode
(i.e. comment-indent-new-line) use fill-newline.


        Stefan

Re: Patch: Syntax and Hard Newlines

[Richard Stallman]

    If all filling functions insert soft newlines with `fill-newline',
    changing this function will be Ok.  And if making this change is Ok,
    how about adding a customizable variable to control whether putting
    the `syntax-table' property?  This would require as little functions
    to be changed as possible.

In general, soft newlines can be inserted by the user too,
for instance by yanking.  So I think the approach of putting
special properties on soft newlines cannot be a general solution.

Currently it is hard newlines that are special.  If we need to use
other text properties to distinguish different kinds of newlines,
putting them on hard newlines follows the existing design better.

If hard newlines should have one property, and soft newlines should
have a different property, then ordinary newlines with no property
(which is what you will get by yanking text that doesn't distinguish)
will have neither property.  In effect, they will be a third kind of
newline.

Re: Patch: Syntax and Hard Newlines

[Stefan Monnier]

>     If all filling functions insert soft newlines with `fill-newline',
>     changing this function will be Ok.  And if making this change is Ok,
>     how about adding a customizable variable to control whether putting
>     the `syntax-table' property?  This would require as little functions
>     to be changed as possible.

> In general, soft newlines can be inserted by the user too,
> for instance by yanking.  So I think the approach of putting
> special properties on soft newlines cannot be a general solution.

But longlines-mode uses an after-change-function to reformat inserted text,
so that should take care of all possible situations.


        Stefan

Re: Patch: Syntax and Hard Newlines

[Richard Stallman]

    > In general, soft newlines can be inserted by the user too,
    > for instance by yanking.  So I think the approach of putting
    > special properties on soft newlines cannot be a general solution.

    But longlines-mode uses an after-change-function to reformat inserted text,
    so that should take care of all possible situations.

Yes, but would this change be active only in Longlines mode?

Re: Patch: Syntax and Hard Newlines

[Stefan Monnier]

>> In general, soft newlines can be inserted by the user too,
>> for instance by yanking.  So I think the approach of putting
>> special properties on soft newlines cannot be a general solution.

>     But longlines-mode uses an after-change-function to reformat inserted
>     text, so that should take care of all possible situations.

> Yes, but would this change be active only in Longlines mode?

Well, that was one of the motivation for my suggestion: make sure that it
can be implemented all in longlines-mode without impacting anything else.


        Stefan

Re: Patch: Syntax and Hard Newlines

[Richard Stallman]

    >     But longlines-mode uses an after-change-function to reformat inserted
    >     text, so that should take care of all possible situations.

    > Yes, but would this change be active only in Longlines mode?

    Well, that was one of the motivation for my suggestion: make sure that it
    can be implemented all in longlines-mode without impacting anything else.

I'm talking about the proposal to change the subroutine fill-newline.
That is not in longlines.el, so unless we do something special to make
it work only when Longlines mode is enabled, it will apply all the
time.  That's what I thought the idea was.

Re: Patch: Syntax and Hard Newlines

[Stefan Monnier]

>> But longlines-mode uses an after-change-function to reformat inserted
>> text, so that should take care of all possible situations.

>> Yes, but would this change be active only in Longlines mode?

>     Well, that was one of the motivation for my suggestion: make sure that it
>     can be implemented all in longlines-mode without impacting anything else.

> I'm talking about the proposal to change the subroutine fill-newline.
> That is not in longlines.el, so unless we do something special to make
> it work only when Longlines mode is enabled, it will apply all the
> time.  That's what I thought the idea was.

Oh, sorry.  I must admit that I didn't completely understand what was
the proposal.  It just reminded me of the fact that auto-fill-mode doesn't
use fill-newline but that it would be good to make it use it.

AFAIU the proposal was only to add a hook to fill-newline such that code
like longlines-mode can tweak the behavior of filling functions rather than
reimplement their own filling code.  I probably misunderstood some part of
it, tho.


        Stefan

Re: Patch: Syntax and Hard Newlines

[Richard Stallman]

    AFAIU the proposal was only to add a hook to fill-newline such that code
    like longlines-mode can tweak the behavior of filling functions rather than
    reimplement their own filling code.

That method could work.

Re: Patch: Syntax and Hard Newlines

[Herbert Euler]

>Or even just:
>
>       (progn (insert-before-markers (propertize "\n" 'syntax-table '(0)))
>              ...)

Will this create a lot of temporary string object
with "\n" as content and '(syntax-table (0)) as
property?  This may cause problem when the
file is large.

Regards,
Guanpeng Xu

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

Re: Patch: Syntax and Hard Newlines

[Miles Bader]

"Herbert Euler" <herberteuler@hotmail.com> writes:
>>       (progn (insert-before-markers (propertize "\n" 'syntax-table '(0)))
>>              ...)
>
> Will this create a lot of temporary string object
> with "\n" as content and '(syntax-table (0)) as
> property?  This may cause problem when the
> file is large.

Why not put it in a defconst?

(defconst xxx-soft-newline (propertize "\n" 'syntax-table '(0)))

...

    (progn (insert-before-markers xxx-soft-newline)
           ...)

-Miles
-- 
"Most attacks seem to take place at night, during a rainstorm, uphill,
 where four map sheets join."   -- Anon. British Officer in WW I

Re: Patch: Syntax and Hard Newlines

[Stefan Monnier]

>> (progn (insert-before-markers (propertize "\n" 'syntax-table '(0)))
>> ...)

> Will this create a lot of temporary string object
> with "\n" as content and '(syntax-table (0)) as
> property?  This may cause problem when the
> file is large.

Then just do:

 (progn (insert-before-markers (eval-when-compile
                                (propertize "\n" 'syntax-table '(0))))
 ...)

or change byte-opt.el to mark the `propertize' function with
`byte-optimize-pure-func' rather than with `side-effect-free'.


        Stefan

Re: Patch: Syntax and Hard Newlines

[Johan Bockgård]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> or change byte-opt.el to mark the `propertize' function with
> `byte-optimize-pure-func' rather than with `side-effect-free'.

But then byte-compilation would change the semantics of `propertize'.

    -- Function: propertize string &rest properties
        This function returns a copy of STRING [...]
                                ^^^^

Actually, that is already the case for (at least) `concat':

    (progn
      (defun f () (concat "a" "b" "c"))
      (aset (f) 0 ?X)
      (f))

     =>  "abc"

    (progn
      (defun f () (concat "a" "b" "c"))
      (byte-compile 'f)
      (aset (f) 0 ?X)
      (f))

     =>  "Xbc"


    -- Function: concat &rest sequences
        This function returns a new string [...]

        The `concat' function always constructs a new string that
        is not `eq' to any existing string. [...]




While playing with this I found another bug.

This change 

    revision 2.143
    date: 2004-03-12 11:09:59 +0100;  author: rms;  state: Exp;  lines: +3 -2
    (byte-compile-get-constant): For strings, do compare text properties.

was made to fix a bug
(http://lists.gnu.org/archive/html/bug-gnu-emacs/2004-03/msg00080.html),
but it accidentally broke constant folding for strings entirely.

It replaced 

    (assoc ,const byte-compile-constants)

with

     (assoc-default ,const byte-compile-constants
                    'equal-including-properties nil)

However, unlike `assoc' which returns the matching cons,
`assoc-default' returns the cdr. In the code in question it is always
nil.

    (assoc-default "foo" '(("foo")) 'equal-including-properties nil)

    => nil

   (assoc "foo" '(("foo")))

    => ("foo")


(This should be trivial to fix.)

-- 
Johan Bockgård

Re: Patch: Syntax and Hard Newlines

[Stefan Monnier]

>> or change byte-opt.el to mark the `propertize' function with
>> `byte-optimize-pure-func' rather than with `side-effect-free'.
> But then byte-compilation would change the semantics of `propertize'.

Slightly, yes.

>     -- Function: propertize string &rest properties
>         This function returns a copy of STRING [...]

Yes, with byte-optimize-pure-func the result would still be a different
string than the argument, but the result would just always be the same
string rather than creating a new one each time.  As you've noticed this is
already done for `concat' (and this has been the case for a long time
already).  Elisp strings are almost never modified, so it's not
a big problem.


        Stefan

Re: Patch: Syntax and Hard Newlines

[Richard Stallman]

    Actually, that is already the case for (at least) `concat':

Stefan argues that modifying strings is not common; nonetheless I
worry that this optimization as currently implemented constitutes a
bug.  Maybe we should make the compiled code copy the string each
time.

Would someone like to implement that?

Re: Patch: Syntax and Hard Newlines

[Richard Stallman]

    However, unlike `assoc' which returns the matching cons,
    `assoc-default' returns the cdr. In the code in question it is always
    nil.

Does this code work right?

*** bytecomp.el	08 Jul 2006 16:59:45 -0400	2.186
--- bytecomp.el	16 Nov 2006 08:24:45 -0500	
***************
*** 2864,2871 ****
  
  (defmacro byte-compile-get-constant (const)
    `(or (if (stringp ,const)
! 	   (assoc-default ,const byte-compile-constants
! 			  'equal-including-properties nil)
  	 (assq ,const byte-compile-constants))
         (car (setq byte-compile-constants
  		  (cons (list ,const) byte-compile-constants)))))
--- 2864,2875 ----
  
  (defmacro byte-compile-get-constant (const)
    `(or (if (stringp ,const)
! 	   ;; In a string constant, treat properties as significant.
! 	   (let (result)
! 	     (dolist (elt byte-compile-constants)
! 	       (if (equal-including-properties (car elt) ,const)
! 		   (setq result elt)))
! 	     result)
  	 (assq ,const byte-compile-constants))
         (car (setq byte-compile-constants
  		  (cons (list ,const) byte-compile-constants)))))

Re: Patch: Syntax and Hard Newlines

[Johan Bockgård]

Richard Stallman <rms@gnu.org> writes:

>     However, unlike `assoc' which returns the matching cons,
>     `assoc-default' returns the cdr. In the code in question it is always
>     nil.
>
> Does this code work right?
>
> *** bytecomp.el	08 Jul 2006 16:59:45 -0400	2.186
> --- bytecomp.el	16 Nov 2006 08:24:45 -0500	
> ***************
> *** 2864,2871 ****
>   
>   (defmacro byte-compile-get-constant (const)
>     `(or (if (stringp ,const)
> ! 	   (assoc-default ,const byte-compile-constants
> ! 			  'equal-including-properties nil)
>   	 (assq ,const byte-compile-constants))
>          (car (setq byte-compile-constants
>   		  (cons (list ,const) byte-compile-constants)))))
> --- 2864,2875 ----
>   
>   (defmacro byte-compile-get-constant (const)
>     `(or (if (stringp ,const)
> ! 	   ;; In a string constant, treat properties as significant.
> ! 	   (let (result)
> ! 	     (dolist (elt byte-compile-constants)
> ! 	       (if (equal-including-properties (car elt) ,const)
> ! 		   (setq result elt)))
> ! 	     result)
>   	 (assq ,const byte-compile-constants))
>          (car (setq byte-compile-constants
>   		  (cons (list ,const) byte-compile-constants)))))

I think so.

Also, Chong removed the "Prevent byte compiler from causing spurious
string sharing when it optimizes away calls to functions such as
concat" note from FOR-RELEASE, but it doesn't seem like anything was
done about it.

-- 
Johan Bockgård

Re: Patch: Syntax and Hard Newlines

[Richard Stallman]

    It's not clear at all that it's a bug rather than a feature.
    I think if it's a bug, the bug is in longlines.el since it is longlines
    which decides that soft-newlines are basically "display only artifacts".

    In most other contexts, soft newlines are much more real (they're soft, but
    they're newlines nevertheless).

In which contexts is it undesirable for soft newlines to be treated
syntactically as spaces?  The idea of soft newlines is that they and
spaces are sort of interchangeable.

Soft newlines are only used in longlines mode and in enriched mode,
and enriched mode won't be used for programs very much.

Re: Patch: Syntax and Hard Newlines

[Stefan Monnier]

>     It's not clear at all that it's a bug rather than a feature.  I think
>     if it's a bug, the bug is in longlines.el since it is longlines which
>     decides that soft-newlines are basically "display only artifacts".

>     In most other contexts, soft newlines are much more real (they're
>     soft, but they're newlines nevertheless).

> In which contexts is it undesirable for soft newlines to be treated
> syntactically as spaces?

I think the determining factor is whether save-buffer saves those soft
newlines as newlines or as something else.

> Soft newlines are only used in longlines mode and in enriched mode,
> and enriched mode won't be used for programs very much.

It's also used by the `use-hard-newline' command.


        Stefan

Re: Patch: Syntax and Hard Newlines

[Richard Stallman]

    > In which contexts is it undesirable for soft newlines to be treated
    > syntactically as spaces?

    I think the determining factor is whether save-buffer saves those soft
    newlines as newlines or as something else.

Ok, I am convinced.

Re: Patch: Syntax and Hard Newlines

[Richard Stallman]

This seems like a necessary fix.  Does anyone see a problem with it?

Meanwhile, would you please send a change log entry?

Re: Patch: Syntax and Hard Newlines

[Herbert Euler]

>Meanwhile, would you please send a change log entry?

        * syntax.c (forw_comment): Don't stop at soft newlines, regard
        them as part of comment.

Regards,
Guanpeng Xu

_________________________________________________________________
Don't just search. Find. Check out the new MSN Search! 
http://search.msn.click-url.com/go/onm00200636ave/direct/01/

RE: Patch: Syntax and Hard Newlines

[Herbert Euler]

Please wait a moment, let me find if there are other
sulotions, just as Stefan said.

Regards,
Guanpeng Xu


>From: "Herbert Euler" <herberteuler@hotmail.com>
>To: emacs-devel@gnu.org
>Subject: Patch: Syntax and Hard Newlines
>Date: Mon, 13 Nov 2006 13:24:59 +0800
>
>Emacs provides a hard newline mode.  If you type M-x use-hard-newlines
>RET, you turn on the hard newline mode.  In this mode, all newlines in
>a buffer are re-scanned and marked as either a ``hard'' one or a
>``soft'' one.  The behavior of commands inserting a newline character
>is also altered: `newline' and `open-line' will add the text property
>`hard' (value set to t) to newlines that they insert.
>
>As documented in the doc string of the command `use-hard-newlines',
>newlines not marked hard are internal to paragraphs.  However,
>currently the syntax feature does not work well with this.  For
>example, if you turn on long-lines mode with M-x longlines-mode RET in
>the *scratch* buffer, you will see the word `evaluation' in the first
>line is wrapped into the second line, but not recognized as a part of
>the comment.
>
>To fix this, we can let the function `forw_comment' in syntax.c skip
>the soft newlines when possible.  Below is the patch.
>
>Regards,
>Guanpeng Xu
>
>Index: syntax.c
>===================================================================
>RCS file: /sources/emacs/emacs/src/syntax.c,v
>retrieving revision 1.166.4.20
>diff -r1.166.4.20 syntax.c
>2126a2127
>>   register int skip_soft_newlines;
>2129a2131,2137
>>   /* If `use-hard-newlines' is t, only newlines with the `hard'
>>      property set to t are real newlines.  Other newlines are soft and
>>      should be skipped. */
>>   skip_soft_newlines = !NILP (Fsymbol_value
>>                               (intern
>>                                ("use-hard-newlines")));
>>
>2145a2154,2163
>>       if (skip_soft_newlines
>>           && c == '\n'
>>           && NILP (Fget_text_property (make_number (from_byte),
>>                                        intern ("hard"),
>>                                        Qnil)))
>>         {
>>           INC_BOTH (from, from_byte);
>>           UPDATE_SYNTAX_TABLE_FORWARD (from);
>>           continue;
>>         }
>
>_________________________________________________________________
>Express yourself instantly with MSN Messenger! Download today it's FREE! 
>http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/
>
>
>
>_______________________________________________
>Emacs-devel mailing list
>Emacs-devel@gnu.org
>http://lists.gnu.org/mailman/listinfo/emacs-devel

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

RE: Patch: Syntax and Hard Newlines

[Herbert Euler]

Opps, I can't understand Emacs' output.  I added some code in syntax.c
and longlines.el:

--------------------------------------------------

*** syntax.c.orignal    Tue Nov 14 18:33:22 2006
--- syntax.c    Tue Nov 14 18:35:47 2006
*************** forw_comment (from, from_byte, stop, nes
*** 2147,2157 ****
        if (code == Sendcomment
          && SYNTAX_FLAGS_COMMENT_STYLE (syntax) == style
          && (SYNTAX_FLAGS_COMMENT_NESTED (syntax) ?
!             (nesting > 0 && --nesting == 0) : nesting < 0))
!       /* we have encountered a comment end of the same style
!          as the comment sequence which began this comment
!          section */
!       break;
        if (code == Scomment_fence
          && style == ST_COMMENT_STYLE)
        /* we have encountered a comment end of the same style
--- 2147,2164 ----
        if (code == Sendcomment
          && SYNTAX_FLAGS_COMMENT_STYLE (syntax) == style
          && (SYNTAX_FLAGS_COMMENT_NESTED (syntax) ?
!             (nesting > 0 && --nesting == 0) : nesting < 0))
!         {
!           message ("in `forw_comment': `syntax-table' property at point %d 
not nil: %d",
!                    from_byte,
!                    !NILP (Fget_text_property (make_number (from_byte),
!                                               intern ("syntax-table"),
!                                               Qnil)));
!           /* we have encountered a comment end of the same style
!              as the comment sequence which began this comment
!              section */
!           break;
!         }
        if (code == Scomment_fence
          && style == ST_COMMENT_STYLE)
        /* we have encountered a comment end of the same style
*************** forw_comment (from, from_byte, stop, nes
*** 2206,2211 ****
--- 2213,2219 ----
      }
    *charpos_ptr = from;
    *bytepos_ptr = from_byte;
+   message ("This comment stopped at %d", from_byte);
    return 1;
  }

*************** scan_sexps_forward (stateptr, from, from
*** 3120,3125 ****
--- 3128,3137 ----
        case Scomment:
          if (commentstop || boundary_stop) goto done;
        startincomment:
+           message ("in `scan_sexps_forward': `syntax-table' property at 
point 65 not nil: %d",
+                    !NILP (Fget_text_property (make_number (65),
+                                               intern ("syntax-table"),
+                                               Qnil)));
          /* The (from == BEGV) test was to enter the loop in the middle so
             that we find a 2-char comment ender even if we start in the
             middle of it.  We don't want to do that if we're just at the

--------------------------------------------------

*** longlines.el.orignal        Mon Nov 13 10:58:17 2006
--- longlines.el        Tue Nov 14 18:24:00 2006
***************
*** 243,248 ****
--- 243,255 ----
        (progn (insert-before-markers ?\n)
             (backward-char 1)
               (delete-char -1)
+            (add-text-properties (point)
+                                 (1+ (point))
+                                 '(syntax-table (14)))
+            (message (format "(in `longlines-wrap-line': char at point %d 
is `%c (%d)', `syntax-table' property: %s)"
+                             (point) (char-after (point))
+                             (char-after (point))
+                             (get-text-property (point) 'syntax-table)))
             (forward-char 1)
               nil)
      (if (longlines-merge-lines-p)

--------------------------------------------------

And I prepared another file:

--------------------------------------------------

$ cat a.el; echo
;; This buffer is for notes you don't want to save, and for Lisp evaluation.
;; If you want to create a file, visit that file with C-x C-f,
;; then enter the text in that file's own buffer.

(progn (longlines-mode)
       (sleep-for 1)
       (switch-to-buffer "*Messages*"))
$

--------------------------------------------------

Note that the position 65 in a.el is at the first line, after the word
`Lisp', i.e. before the space that before the word `evaluation'.  Then
I recompiled Emacs, and run `./emacs -Q a.el'.  I evaluated the
`progn' form, and read the output in the *Messages* buffer:

--------------------------------------------------

Loading /home/xgp/src/emacs/emacs/lisp/longlines.el (source)...done
(in `longlines-wrap-line': char at point 65 is `
(10)', `syntax-table' property: (14))
in `scan_sexps_forward': `syntax-table' property at point 65 not nil: 1
in `forw_comment': `syntax-table' property at point 65 not nil: 1
This comment stopped at 65
#<buffer *Messages*>

--------------------------------------------------

The syntax code 14 means "generic comment".  As documented in the
elisp manual, the parsing should use "generic comment", rather than
"comment stop" for the newline character at point 65.  Stefan's reply,

    Another way to fix it, local to longlines.el is to add a
    syntax-table property to soft-newlines so as to prevent them from
    being treated like an end-of-comment.

seems to mean the same thing.

But why "This comment stopped at 65"?  Does this indicate a bug in the
syntax implementation?

Regards,
Guanpeng Xu

_________________________________________________________________
Don't just search. Find. Check out the new MSN Search! 
http://search.msn.click-url.com/go/onm00200636ave/direct/01/

Re: Patch: Syntax and Hard Newlines

[Stefan Monnier]

> The syntax code 14 means "generic comment".  As documented in the
> elisp manual, the parsing should use "generic comment", rather than
> "comment stop" for the newline character at point 65.

I'm not sure I understand the above sentence, but you need to setq
parse-sexp-lookup-properties.


        Stefan

RE: Patch: Syntax and Hard Newlines

[Herbert Euler]

Summing up:

        * longlines.el (longlines-mode): Set
        `parse-sexp-lookup-properties' so that syntax parsing uses text
        properties.
        (longlines-soft-newline): New constant.
        (longlines-wrap-line): Use `longlines-soft-newline' to make syntax
        parsing regard soft newlines the same as spaces.

*** longlines.el.~1.2.2.16.~    Mon Nov 13 10:58:17 2006
--- longlines.el        Thu Nov 16 13:49:15 2006
***************
*** 80,85 ****
--- 80,91 ----
  (defvar longlines-wrap-point nil)
  (defvar longlines-showing nil)

+ (defconst longlines-soft-newline
+   (propertize "\n"
+             'syntax-table
+             '(0))
+   "The soft newline character for wrapping long lines.")
+
  (make-variable-buffer-local 'longlines-wrap-beg)
  (make-variable-buffer-local 'longlines-wrap-end)
  (make-variable-buffer-local 'longlines-wrap-point)
*************** (define-minor-mode longlines-mode
*** 112,117 ****
--- 118,124 ----
          (make-local-variable 'buffer-substring-filters)
        (set (make-local-variable 'isearch-search-fun-function)
             'longlines-search-function)
+       (set (make-local-variable 'parse-sexp-lookup-properties) t)
          (add-to-list 'buffer-substring-filters 'longlines-encode-string)
          (when longlines-wrap-follows-window-size
            (set (make-local-variable 'fill-column)
*************** (define-minor-mode longlines-mode
*** 156,161 ****
--- 163,169 ----
            (add-hook 'post-command-hook
                      'longlines-post-command-function nil t)))
      ;; Turn off longlines mode
+     (kill-local-variable 'parse-sexp-lookup-properties)
      (setq buffer-file-format (delete 'longlines buffer-file-format))
      (if longlines-showing
          (longlines-unshow-hard-newlines))
*************** (defun longlines-wrap-line ()
*** 240,246 ****
  If wrapping is performed, point remains on the line.  If the line does
  not need to be wrapped, move point to the next line and return t."
    (if (longlines-set-breakpoint)
!       (progn (insert-before-markers ?\n)
             (backward-char 1)
               (delete-char -1)
             (forward-char 1)
--- 248,254 ----
  If wrapping is performed, point remains on the line.  If the line does
  not need to be wrapped, move point to the next line and return t."
    (if (longlines-set-breakpoint)
!       (progn (insert-before-markers longlines-soft-newline)
             (backward-char 1)
               (delete-char -1)
             (forward-char 1)

I chose Miles' approach, because it

    [1] is limited in `longlines.el',

    [2] takes the same effect both in interpretation and compilation.

Regards,
Guanpeng Xu

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

Re: Patch: Syntax and Hard Newlines

[martin rudalics]

A silly question maybe: Wouldn't

+     (kill-local-variable 'parse-sexp-lookup-properties)

cause problems for major modes using that?

I suppose it would be cleaner to require minor modes that want to turn
on/off `parse-sexp-lookup-properties' in a buffer append/remove their
name from/to that variable and kill it iff that list gets empty.

Re: Patch: Syntax and Hard Newlines

[Herbert Euler]

>A silly question maybe: Wouldn't
>
>+     (kill-local-variable 'parse-sexp-lookup-properties)
>
>cause problems for major modes using that?
>
>I suppose it would be cleaner to require minor modes that want to turn
>on/off `parse-sexp-lookup-properties' in a buffer append/remove their
>name from/to that variable and kill it iff that list gets empty.

No, `kill-local-variable' only makes a variable no longer have a
separate value in the current buffer.  The same variable (perhaps
local) in other buffers is not affected, either does the global
version of the variable:

(setq abc 7)
==> 7
(setq b1 (get-buffer-create "b1"))
==> #<buffer b1>
(setq b2 (get-buffer-create "b2"))
==> #<buffer b2>
(save-window-excursion (switch-to-buffer b1)
                       abc)
==> 7
(save-window-excursion (switch-to-buffer b2)
                       abc)
==> 7
(save-window-excursion (switch-to-buffer b1)
                       (make-local-variable 'abc)
                       (setq abc 5)
                       abc)
==> 5
abc
==> 7
(save-window-excursion (switch-to-buffer b2)
                       (make-local-variable 'abc)
                       (setq abc 9)
                       abc)
==> 9
abc
==> 7
(save-window-excursion (switch-to-buffer b1)
                       (kill-local-variable 'abc)
                       abc)
==> 7
abc
==> 7
(save-window-excursion (switch-to-buffer b2)
                       abc)
==> 9
abc
==> 7

Regards,
Guanpeng Xu

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

Re: Patch: Syntax and Hard Newlines

[martin rudalics]

 > No, `kill-local-variable' only makes a variable no longer have a
 > separate value in the current buffer.

That's what I meant.  For example, in cc-mode.el you have

     (when (boundp 'parse-sexp-lookup-properties)
       (make-local-variable 'parse-sexp-lookup-properties)
       (setq parse-sexp-lookup-properties t))

that is cc-mode makes a "separate value in the current buffer" and sets
that locally to t.  When you turn off longlines in that buffer you turn
that off for cc-mode too.  Or am I missing something?

Re: Patch: Syntax and Hard Newlines

[Herbert Euler]

> > No, `kill-local-variable' only makes a variable no longer have a
> > separate value in the current buffer.
>
>That's what I meant.  For example, in cc-mode.el you have
>
>     (when (boundp 'parse-sexp-lookup-properties)
>       (make-local-variable 'parse-sexp-lookup-properties)
>       (setq parse-sexp-lookup-properties t))
>
>that is cc-mode makes a "separate value in the current buffer" and sets
>that locally to t.  When you turn off longlines in that buffer you turn
>that off for cc-mode too.  Or am I missing something?

No, `make-local-variable' makes a variable local to a buffer,
not to a mode:

(setq b1 (get-buffer-create "b1"))
==> #<buffer b1>
(setq b2 (get-buffer-create "b2"))
==> #<buffer b2>
(save-window-excursion (switch-to-buffer b1)
                       (c-mode)
                       major-mode)
==> c-mode
(save-window-excursion (switch-to-buffer b2)
                       (c-mode)
                       major-mode)
==> c-mode
parse-sexp-lookup-properties
==> nil
(save-window-excursion (switch-to-buffer b1)
                       parse-sexp-lookup-properties)
==> nil
(save-window-excursion (switch-to-buffer b2)
                       parse-sexp-lookup-properties)
==> nil
(save-window-excursion (switch-to-buffer b1)
                       (make-local-variable 'parse-sexp-lookup-properties)
                       (setq parse-sexp-lookup-properties t)
                       parse-sexp-lookup-properties)
==> t
(save-window-excursion (switch-to-buffer b2)
                       (make-local-variable 'parse-sexp-lookup-properties)
                       (setq parse-sexp-lookup-properties nil)
                       parse-sexp-lookup-properties)
==> nil
parse-sexp-lookup-properties
==> nil
(save-window-excursion (switch-to-buffer b1)
                       (kill-local-variable 'parse-sexp-lookup-properties)
                       parse-sexp-lookup-properties)
==> nil
(save-window-excursion (switch-to-buffer b2)
                       parse-sexp-lookup-properties)
==> nil
(save-window-excursion (switch-to-buffer b2)
                       (setq parse-sexp-lookup-properties t)
                       parse-sexp-lookup-properties)
==> t
parse-sexp-lookup-properties
==> nil

Regards,
Guanpeng Xu

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

Re: Patch: Syntax and Hard Newlines

[martin rudalics]

 > No, `make-local-variable' makes a variable local to a buffer,
 > not to a mode:

The major modes I've been talking about (including ada, cperl, python)
use `make-local-variable' to make `parse-sexp-lookup-properties' local
to the buffer they're operating on.  When you do `kill-local-variable'
you kill their local binding too and they will see the global binding
which is usually not what they want.

Re: Patch: Syntax and Hard Newlines

[Herbert Euler]

>The major modes I've been talking about (including ada, cperl, python)
>use `make-local-variable' to make `parse-sexp-lookup-properties' local
>to the buffer they're operating on.  When you do `kill-local-variable'
>you kill their local binding too and they will see the global binding
>which is usually not what they want.

What about this?

        * longlines.el (longlines-mode): Set
        `parse-sexp-lookup-properties' so that syntax parsing uses text
        properties.
        (longlines-parse-sexp-lookup-properties)
        (longlines-pslp-was-local): New variables.
        (longlines-soft-newline): New constant.
        (longlines-wrap-line): Use `longlines-soft-newline' to make syntax
        parsing regard soft newlines the same as spaces.

*** longlines.el.orignal        2006-11-16 23:00:49.000000000 +0000
--- longlines.el        2006-11-16 23:00:55.000000000 +0000
***************
*** 79,89 ****
--- 79,100 ----
  (defvar longlines-wrap-end nil)
  (defvar longlines-wrap-point nil)
  (defvar longlines-showing nil)
+ (defvar longlines-parse-sexp-lookup-properties nil)
+ ;; PSLP: Parse-Sexp-Lookup-Properties.
+ (defvar longlines-pslp-was-local nil)

  (make-variable-buffer-local 'longlines-wrap-beg)
  (make-variable-buffer-local 'longlines-wrap-end)
  (make-variable-buffer-local 'longlines-wrap-point)
  (make-variable-buffer-local 'longlines-showing)
+ (make-variable-buffer-local 'longlines-parse-sexp-lookup-properties)
+ (make-variable-buffer-local 'longlines-pslp-was-local)
+
+ (defconst longlines-soft-newline
+   (propertize "\n"
+             'syntax-table
+             '(0))
+   "The soft newline character for wrapping long lines.")

  ;; Mode

***************
*** 112,117 ****
--- 123,137 ----
          (make-local-variable 'buffer-substring-filters)
        (set (make-local-variable 'isearch-search-fun-function)
             'longlines-search-function)
+       (if (local-variable-p 'parse-sexp-lookup-properties)
+           (setq longlines-parse-sexp-lookup-properties
+                 parse-sexp-lookup-properties
+                 longlines-pslp-was-local
+                 t
+                 parse-sexp-lookup-properties
+                 t)
+         (setq longlines-pslp-was-local nil)
+         (set (make-local-variable 'parse-sexp-lookup-properties) t))
          (add-to-list 'buffer-substring-filters 'longlines-encode-string)
          (when longlines-wrap-follows-window-size
            (set (make-local-variable 'fill-column)
***************
*** 156,161 ****
--- 176,185 ----
            (add-hook 'post-command-hook
                      'longlines-post-command-function nil t)))
      ;; Turn off longlines mode
+     (if longlines-pslp-was-local
+       (setq parse-sexp-lookup-properties
+             longlines-parse-sexp-lookup-properties)
+       (kill-local-variable 'parse-sexp-lookup-properties))
      (setq buffer-file-format (delete 'longlines buffer-file-format))
      (if longlines-showing
          (longlines-unshow-hard-newlines))
***************
*** 240,246 ****
  If wrapping is performed, point remains on the line.  If the line does
  not need to be wrapped, move point to the next line and return t."
    (if (longlines-set-breakpoint)
!       (progn (insert-before-markers ?\n)
             (backward-char 1)
               (delete-char -1)
             (forward-char 1)
--- 264,270 ----
  If wrapping is performed, point remains on the line.  If the line does
  not need to be wrapped, move point to the next line and return t."
    (if (longlines-set-breakpoint)
!       (progn (insert-before-markers longlines-soft-newline)
             (backward-char 1)
               (delete-char -1)
             (forward-char 1)

Regards,
Guanpeng Xu

_________________________________________________________________
Don't just search. Find. Check out the new MSN Search! 
http://search.msn.click-url.com/go/onm00200636ave/direct/01/

Re: Patch: Syntax and Hard Newlines

[martin rudalics]

 > What about this?

 > +       (if (local-variable-p 'parse-sexp-lookup-properties)
 > +           (setq longlines-parse-sexp-lookup-properties
 > +                 parse-sexp-lookup-properties
 > +                 longlines-pslp-was-local
 > +                 t
 > +                 parse-sexp-lookup-properties
 > +                 t)
 > +         (setq longlines-pslp-was-local nil)
 > +         (set (make-local-variable 'parse-sexp-lookup-properties) t))

[...]

 > +     (if longlines-pslp-was-local
 > +       (setq parse-sexp-lookup-properties
 > +             longlines-parse-sexp-lookup-properties)
 > +       (kill-local-variable 'parse-sexp-lookup-properties))

It should work with major modes since their activation usually kills all
local variables.  It won't work if another minor mode eventually needs
this too.  Suppose you

(1) turn on longlines mode with `parse-sexp-lookup-properties' locally nil,

(2) turn on a minor mode that sets `parse-sexp-lookup-properties' to t,

(3) turn off longlines mode.

Hence I'd rather leave this alone when turning off longlines mode.

Re: Patch: Syntax and Hard Newlines

[Herbert Euler]

> > What about this?
>
> > +       (if (local-variable-p 'parse-sexp-lookup-properties)
> > +           (setq longlines-parse-sexp-lookup-properties
> > +                 parse-sexp-lookup-properties
> > +                 longlines-pslp-was-local
> > +                 t
> > +                 parse-sexp-lookup-properties
> > +                 t)
> > +         (setq longlines-pslp-was-local nil)
> > +         (set (make-local-variable 'parse-sexp-lookup-properties) t))
>
>[...]
>
> > +     (if longlines-pslp-was-local
> > +       (setq parse-sexp-lookup-properties
> > +             longlines-parse-sexp-lookup-properties)
> > +       (kill-local-variable 'parse-sexp-lookup-properties))
>
>It should work with major modes since their activation usually kills all
>local variables.  It won't work if another minor mode eventually needs
>this too.  Suppose you
>
>(1) turn on longlines mode with `parse-sexp-lookup-properties' locally nil,
>
>(2) turn on a minor mode that sets `parse-sexp-lookup-properties' to t,
>
>(3) turn off longlines mode.
>
>Hence I'd rather leave this alone when turning off longlines mode.

It seems this shows a conflict between minor modes.  `longlines' mode
requires `parse-sexp-lookup-properties' to be t, while the other minor
mode you refered requires it to be nil.  Or at least, the other minor
mode works even if it is t, but that is not desirable to the user.
This I think is not because Emacs lacks some mechanism to handle local
variables well, but rather that it is because the two minor modes are
not well-designed so that they conflicts with each other.  I cannot
imagine a reasonable value for `parse-sexp-lookup-properties' to get a
correct behavior when turning on both the two minor modes refered in
your example.

Regards,
Guanpeng Xu

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

Re: Patch: Syntax and Hard Newlines

[Stefan Monnier]

> It seems this shows a conflict between minor modes.  `longlines' mode
> requires `parse-sexp-lookup-properties' to be t, while the other minor
> mode you refered requires it to be nil.

That is not true: AFAIK there is no elisp code around which *requires*
parse-sexp-lookup-properties to be nil.  So just set it to t and leave it.


        Stefan

Re: Patch: Syntax and Hard Newlines

[Herbert Euler]

I'm sorry I misunderstood you again.  Let me try another fix....

Regards,
Guanpeng Xu

>From: martin rudalics <rudalics@gmx.at>
>To: Herbert Euler <herberteuler@hotmail.com>
>CC: emacs-devel@gnu.org,  rms@gnu.org,  monnier@iro.umontreal.ca
>Subject: Re: Patch: Syntax and Hard Newlines
>Date: Thu, 16 Nov 2006 17:03:33 +0100
>
> > What about this?
>
> > +       (if (local-variable-p 'parse-sexp-lookup-properties)
> > +           (setq longlines-parse-sexp-lookup-properties
> > +                 parse-sexp-lookup-properties
> > +                 longlines-pslp-was-local
> > +                 t
> > +                 parse-sexp-lookup-properties
> > +                 t)
> > +         (setq longlines-pslp-was-local nil)
> > +         (set (make-local-variable 'parse-sexp-lookup-properties) t))
>
>[...]
>
> > +     (if longlines-pslp-was-local
> > +       (setq parse-sexp-lookup-properties
> > +             longlines-parse-sexp-lookup-properties)
> > +       (kill-local-variable 'parse-sexp-lookup-properties))
>
>It should work with major modes since their activation usually kills all
>local variables.  It won't work if another minor mode eventually needs
>this too.  Suppose you
>
>(1) turn on longlines mode with `parse-sexp-lookup-properties' locally nil,
>
>(2) turn on a minor mode that sets `parse-sexp-lookup-properties' to t,
>
>(3) turn off longlines mode.
>
>Hence I'd rather leave this alone when turning off longlines mode.
>

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

Re: Patch: Syntax and Hard Newlines

[Stefan Monnier]

> A silly question maybe: Wouldn't

> +     (kill-local-variable 'parse-sexp-lookup-properties)

> cause problems for major modes using that?

Indeed.  AFAIK the only reason to set parse-sexp-lookup-properties to nil is
performance, so it shouldn't hurt to leave it set to t.

> I suppose it would be cleaner to require minor modes that want to turn
> on/off `parse-sexp-lookup-properties' in a buffer append/remove their
> name from/to that variable and kill it iff that list gets empty.

Maybe.  It seems overkill to me for this specific case, but it would make
sense to try and devise a general scheme for these kinds of problems,


        Stefan

Re: Patch: Syntax and Hard Newlines

[Herbert Euler]

>I suppose it would be cleaner to require minor modes that want to turn
>on/off `parse-sexp-lookup-properties' in a buffer append/remove their
>name from/to that variable and kill it iff that list gets empty.

All minor modes should use a same value for a variable, either local
or global, to avoid conflicting each other.  If the value of a global
variable is changed, all buffers will be affected.  Carefully designed
minor modes will not likely do this for a variable that does not
belong to itself, so only the behavior of killing local variables
matters here.

Because many minor modes may be activated or deactivated in any order,
your method seems to be the only way.  But it requires checking every
minor mode, which is a lot of work.  :(

Regards,
Guanpeng Xu

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

Re: Patch: Syntax and Hard Newlines

[martin rudalics]

 > All minor modes should use a same value for a variable, either local
 > or global, to avoid conflicting each other.  If the value of a global
 > variable is changed, all buffers will be affected.  Carefully designed
 > minor modes will not likely do this for a variable that does not
 > belong to itself, so only the behavior of killing local variables
 > matters here.

A minor mode is certainly incorrect if it doesn't work as intended when
`parse-sexp-lookup-properties' is t.  A minor is also incorrect if it
globally sets `parse-sexp-lookup-properties'.  Hence, as you note, only
the behavior when killing this variable locally matters.  If modes need
conflicting values for one and the same variable, they must use a `let'
binding.

 > Because many minor modes may be activated or deactivated in any order,
 > your method seems to be the only way.  But it requires checking every
 > minor mode, which is a lot of work.  :(

It would require a minor mode to add a symbol to a list when it is
turned on and remove that symbol when it is turned off.  I don't think
that's a lot of work.

Re: Patch: Syntax and Hard Newlines

[Herbert Euler]

>It would require a minor mode to add a symbol to a list when it is
>turned on and remove that symbol when it is turned off.  I don't think 
>that's a lot of work.

There are too many minor modes in existence to make this a lot of
work, I meant.

Regards,
Guanpeng Xu

BTW: How about this problem?  I'm not sure what I should do now.
Could somebody please give some suggestions?  Thanks.

A list of problems:

    [1] The orignal problem in longlines.el.

    [2] Whether adding a `syntax-table' property when inserting
        soft newlines.

    [3] The problem Johan refered.

    [4] The problem Martin refered.

_________________________________________________________________
FREE pop-up blocking with the new MSN Toolbar - get it now! 
http://toolbar.msn.click-url.com/go/onm00200415ave/direct/01/

Re: Patch: Syntax and Hard Newlines

[martin rudalics]

 > BTW: How about this problem?  I'm not sure what I should do now.
 > Could somebody please give some suggestions?  Thanks.

1. Define `longlines-wrap-region' somewhat like the following untested

(defun longlines-wrap-region (beg end)
   "Wrap lines between BEG and END."
   (save-excursion
     (save-restriction
       (goto-char end)
       (unless (eolp) (forward-line))
       (setq end (point))
       (goto-char beg)
       (forward-line -1)
       (setq beg (point))
       (narrow-to-region beg end)
       (remove-overlays nil nil 'longlines t)
       (while (not (eobp))
	(save-restriction
	  (narrow-to-region
	   (line-beginning-position) (line-end-position))
	  (while (and (move-to-column fill-column)
		      (save-match-data
			(not (looking-at "[ \t]*$")))
		      (or (longlines-find-break-backward)
			  (progn (move-to-column fill-column)
				 (longlines-find-break-forward))))
	    (let* ((from (save-excursion
			   (skip-chars-backward " \t") (point)))
		   (to (save-excursion
			 (skip-chars-forward " \t") (point)))
		   (overlay (make-overlay from to)))
	      ;; Overlay with display property.
	      (overlay-put overlay 'display "\n")
	      (overlay-put overlay 'longlines t)
	      (narrow-to-region (point) (point-max)))))
	(forward-line)))))

2. Modify the remainder of longlines.el accordingly (I'd wrap only
regions between `window-start' and `window-end').

3. Redefine `next-line' and `previous-line' to treat an overlay with a
display property of "\n" just as another newline (possibly guided by an
additional "soft" parameter).

4. Add two new commands `beginning-of-line-soft' and `end-of-line-soft'
that handle such overlays as well.

Re: Patch: Syntax and Hard Newlines

[Miles Bader]

martin rudalics <rudalics@gmx.at> writes:
> 3. Redefine `next-line' and `previous-line' to treat an overlay with a
> display property of "\n" just as another newline (possibly guided by an
> additional "soft" parameter).
>
> 4. Add two new commands `beginning-of-line-soft' and `end-of-line-soft'
> that handle such overlays as well.

That way lies madness...

-Miles

-- 
What the fuck do white people have to be blue about!?  Banana Republic ran
out of Khakis?  The Espresso Machine is jammed?  Hootie and The Blowfish are
breaking up??!  Shit, white people oughtta understand, their job is to GIVE
people the blues, not to get them!    -- George Carlin

Re: Patch: Syntax and Hard Newlines

[martin rudalics]

 > That way lies madness...

...  yet there is method in 't.  I nearly spent an hour analyzing
longlines-mode and writing my proposal.  Please spend a few minutes on
the following questions:

- longlines-mode substitutes newlines for spaces.  This means that any
   function based on scan_newline (like `goto-line') may get me results
   that are inconsistent with those of tools analyzing the file my buffer
   visits.  How can I treat compiler or grep output with longlines-mode?

- Matching against the regexp "." may get me different results for the
   same buffer with longlines-mode disabled and enabled.  Ignore that?

- longlines-mode may wrap a regexp like "[ \t]*" at the space character.
   How can I evaluate a wrapped regexp like that?

- How can I avoid that longlines-mode wraps a Lisp or Perl expression at
   a space preceding a left paren (with the left paren ending up at bol
   and messing up `beginning-of-defun-raw' and thus font-locking)?

- How can I avoid that longlines-mode wraps a C string and
   `c-font-lock-invalid-string' complains?

- I put an arbitrary text property on a space character.  longlines-mode
   wraps the line at that character.  How can I restore that property
   when turning off longlines-mode?

- I use two windows of different widths to simultaneously show one and
   the same buffer.  How can I adapt longlines-mode to wrap at the right
   borders of my windows?

Re: Patch: Syntax and Hard Newlines

[Stefan Monnier]

>> That way lies madness...
> ...  yet there is method in it.

Agreed.  I think either approach can be made to work, and they both have
their shortcomings.  But since the intent is to make the line wraps when
displayed, it makes sense to use display properties (at least until Kim's
patch to make the display support word-wrapping natively gets installed).

BTW, to support your proposed solution, you want to use screen-lines.el
(which may need some improvement: I've never tried it).


        Stefan

Re: Patch: Syntax and Hard Newlines

[Richard Stallman]

First, I note that in this message you're talking about user-level
behavor.  Your previous message, which Miles responded to, proposed an
implementation, and his response was about that implementation, not
the user-level issues.  These are two different (though closely
related) topics.

So let's look at the user-level issues you've raised, and how they
relate to implementation.

    - longlines-mode substitutes newlines for spaces.  This means that any
       function based on scan_newline (like `goto-line') may get me results
       that are inconsistent with those of tools analyzing the file my buffer
       visits.  How can I treat compiler or grep output with longlines-mode?

That is true, but I suspect it is not a problem since the broken lines
of compiler output normally won't match the templates for error
messages.  (And I am not sure it makes sense to use longlines mode in
such buffers.)

    - Matching against the regexp "." may get me different results for the
       same buffer with longlines-mode disabled and enabled.  Ignore that?

That is true.  Whether it is good or bad, I am not sure.  It depends on what
the program or the user is trying to achieve.

    - longlines-mode may wrap a regexp like "[ \t]*" at the space character.
       How can I evaluate a wrapped regexp like that?

The only way to avoid that consequence would be to switch from "soft newlines"
to "power spaces".  I am not sure that is a good idea, and do we really
care about making Longlines mode work for Lisp code?

    - How can I avoid that longlines-mode wraps a Lisp or Perl expression at
       a space preceding a left paren (with the left paren ending up at bol
       and messing up `beginning-of-defun-raw' and thus font-locking)?

Likewise.

    - How can I avoid that longlines-mode wraps a C string and
       `c-font-lock-invalid-string' complains?

Do we really care about making Longlines mode work for C code?

    - I put an arbitrary text property on a space character.  longlines-mode
       wraps the line at that character.  How can I restore that property
       when turning off longlines-mode?

Longlines mode should preserve other text properties when it converts
a space to a soft newline and vise versa.

    - I use two windows of different widths to simultaneously show one and
       the same buffer.  How can I adapt longlines-mode to wrap at the right
       borders of my windows?

It is impossible for anything like Longlines mode to do that.
That would require a feature implemented entirely within redisplay.

There would be no fundamental difficulty in implementing such a
feature inside Emacs redisplay, but it would have to be implemented
entirely in C.  It would be a variant of the current feature of line
continuation, and it would share the big inconvenience of line
continuation: all editing commands treat the line that has been
continued as one single line.

Re: Patch: Syntax and Hard Newlines

[Stefan Monnier]

>     - I use two windows of different widths to simultaneously show one and
>        the same buffer.  How can I adapt longlines-mode to wrap at the right
>        borders of my windows?

> It is impossible for anything like Longlines mode to do that.
> That would require a feature implemented entirely within redisplay.

Actually, applying a `display' property of "\n" on some spaces, using
overlays that also have a `window' property would make it possible.
Maybe impractical, tho ;-)

And since Kim had a patch to do it much more cleanly and efficiently at the
C level, it doesn't seem worth the trouble.


        Stefan

Re: Patch: Syntax and Hard Newlines

[Richard Stallman]

    > It is impossible for anything like Longlines mode to do that.
    > That would require a feature implemented entirely within redisplay.

    Actually, applying a `display' property of "\n" on some spaces, using
    overlays that also have a `window' property would make it possible.
    Maybe impractical, tho ;-)

This would have the disadvantages that I described before,
for doing this in redisplay: editing commands would treat
those apparent line-breaks like spaces.

Re: Patch: Syntax and Hard Newlines

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

>     > It is impossible for anything like Longlines mode to do that.
>     > That would require a feature implemented entirely within redisplay.
>
>     Actually, applying a `display' property of "\n" on some spaces, using
>     overlays that also have a `window' property would make it possible.
>     Maybe impractical, tho ;-)
>
> This would have the disadvantages that I described before,
> for doing this in redisplay: editing commands would treat
> those apparent line-breaks like spaces.

There is an Elisp library that switches the line-based movement key
sequences to appropriately modified commands.  I think that most other
stuff would not be minded by the user.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Patch: Syntax and Hard Newlines

[Richard Stallman]

    There is an Elisp library that switches the line-based movement key
    sequences to appropriately modified commands.  I think that most other
    stuff would not be minded by the user.

I'd like to see people report on how they like this.
(Which library is it?)

Re: Patch: Syntax and Hard Newlines

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

>     There is an Elisp library that switches the line-based movement key
>     sequences to appropriately modified commands.  I think that most other
>     stuff would not be minded by the user.
>
> I'd like to see people report on how they like this.
> (Which library is it?)

I think it is called screen-lines.el.  Looking for it turned up
<URL:http://homepage1.nifty.com/bmonkey/emacs/elisp/screen-lines.el>,
and this says:

;;; screen-lines.el --- a minor mode for screen-line-based point motion

;; Copyright (C) 2000, 2001, 2002, 2003, 2004
;; Yuji 'bmonkey' Minejima <ggb01164@nifty.ne.jp>

;; Filename:      screen-lines.el
;; Author:        Yuji Minejima <ggb01164@nifty.ne.jp>
;; Keywords:      convenience
;; Description:   a minor mode for screen-line-based point motion
;; Compatibility: GNU Emacs 20.7.2, XEmacs 21.1.12
;; URL: http://homepage1.nifty.com/bmonkey/emacs/elisp/screen-lines.el
;; $Revision: 0.55 $

;; This file is NOT part of GNU Emacs.

;; This program is free software; you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation; either version 2, or (at your option)
;; any later version.

[...]

;; This package contains part of the code and the documentation of the
;; following packages:
;;
;;   simple.el --- basic editing commands for Emacs
;;   Copyright (C) 1985, 86, 87, 93, 94, 95, 96, 97, 98, 1999
;;   Free Software Foundation, Inc.
;;
;;   window-lines.el --- basic cursor moving commands based on window lines
;;   Copyright (C) 1990 enami tsugutomo (enami@ptgd.sony.co.jp)
;;
;; All of these are distributed under GPL.

[...]

So it would appear that there is also window-lines.el (no idea whether
that works at all nowadays).

It is not too infrequent for people to ask for this functionality on
the Emacs help list and get referred to screen-lines.el IIRC.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Patch: Syntax and Hard Newlines

[Richard Stallman]

I think someone said Kim has already implemented the change at the
display level.  I would like to see as many users as possible report
on trying that feature together with screen-lines or window-lines.
This is for deciding what to do after the coming release.

Re: Patch: Syntax and Hard Newlines

[Kim F. Storm]

Richard Stallman <rms@gnu.org> writes:

> I think someone said Kim has already implemented the change at the
> display level.  I would like to see as many users as possible report
> on trying that feature together with screen-lines or window-lines.
> This is for deciding what to do after the coming release.

I'm not sure my patches will apply cleanly these days, and I don't
have time to look at them now.

If we install them (or something similar), I think we should modify
the primitives like next-line and previous-line to do the right thing
(whatever that is) in those cases.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: Patch: Syntax and Hard Newlines

[David Kastrup]

storm@cua.dk (Kim F. Storm) writes:

> Richard Stallman <rms@gnu.org> writes:
>
>> I think someone said Kim has already implemented the change at the
>> display level.  I would like to see as many users as possible report
>> on trying that feature together with screen-lines or window-lines.
>> This is for deciding what to do after the coming release.
>
> I'm not sure my patches will apply cleanly these days, and I don't
> have time to look at them now.
>
> If we install them (or something similar), I think we should modify
> the primitives like next-line and previous-line to do the right thing
> (whatever that is) in those cases.

I don't think we should modify those in order not to cause trouble
where they are used programmatically.  Either we add an optional
argument SCREEN-LINES to them which the interactive form sets to t, or
we create separate next-screen-line and previous-screen-line macros
that can be bound to the keys in question.  It is conceivable that we
would do this just in longlines-mode or similar modes.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Patch: Syntax and Hard Newlines

[Kim F. Storm]

David Kastrup <dak@gnu.org> writes:

> I don't think we should modify those in order not to cause trouble
> where they are used programmatically.  

In general, next-line and previous-line should not be used in Lisp
programs.

>                                        Either we add an optional
> argument SCREEN-LINES to them which the interactive form sets to t, or
> we create separate next-screen-line and previous-screen-line macros
> that can be bound to the keys in question.  It is conceivable that we
> would do this just in longlines-mode or similar modes.

Let's discuss the details after the release...

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: Patch: Syntax and Hard Newlines

[David Kastrup]

storm@cua.dk (Kim F. Storm) writes:

> David Kastrup <dak@gnu.org> writes:
>
>> I don't think we should modify those in order not to cause trouble
>> where they are used programmatically.  
>
> In general, next-line and previous-line should not be used in Lisp
> programs.

Well, they might make it into keyboard macros.  But that actually is
not addressed properly by my proposal either:

>> Either we add an optional argument SCREEN-LINES to them which the
>> interactive form sets to t, or we create separate next-screen-line
>> and previous-screen-line macros that can be bound to the keys in
>> question.  It is conceivable that we would do this just in
>> longlines-mode or similar modes.
>
> Let's discuss the details after the release...

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Patch: Syntax and Hard Newlines

[David Reitter]

On 22 Nov 2006, at 13:15, Richard Stallman wrote:

>     There is an Elisp library that switches the line-based movement  
> key
>     sequences to appropriately modified commands.  I think that  
> most other
>     stuff would not be minded by the user.
>
> I'd like to see people report on how they like this.
> (Which library is it?)

We've had a similar library installed and enabled by default in the  
Aquamacs distribution for about a year now (see URL below).

Discussions with users resulted in binding <down> and <up> to the  
appropriate functions, but not C-p and C-n, so that wrapped lines can  
still be navigated comfortably.
If a mode binds up/down to something else than next/prior, this new  
binding is preserved.

IIRC, the functions `visual-line-up' and friends  that I wrote differ  
from the other available library in that they work well with variable- 
width fonts.

`kill-visual-line' and `kill-whole-visual-line', `beginning/end-of- 
visual-line' are provided as well and do the obvious.

Inevitably, some users were annoyed with the functionality, so it  
makes sense to allow for customization (e.g. with a minor mode  
`visual-line-navigation' or so).


Code:

http://aquamacs.cvs.sourceforge.net/*checkout*/aquamacs/aquamacs/src/ 
site-lisp/macosx/osxkeys.el
(This library doesn't work as stand-alone, but I can easily extract  
the visual-line-* code.)

Re: Patch: Syntax and Hard Newlines

[Richard Stallman]

    Discussions with users resulted in binding <down> and <up> to the  
    appropriate functions, but not C-p and C-n, so that wrapped lines can  
    still be navigated comfortably.

Those words are vague and I cannot undersand what meanings
you have given to these four keys.

What behavior did you implement for the <down> and <up> keys?
What behavior did you implement for the C-n and C-p keys?

    IIRC, the functions `visual-line-up' and friends  that I wrote differ  
    from the other available library in that they work well with variable- 
    width fonts.

    `kill-visual-line' and `kill-whole-visual-line', `beginning/end-of- 
    visual-line' are provided as well and do the obvious.

It isn't obvious to me.  I could try to guess, but I should not,
because guessing is unreliable.  What do these commands do?

Re: Patch: Syntax and Hard Newlines

[David Reitter]

On 6 Dec 2006, at 14:24, Richard Stallman wrote:

> What behavior did you implement for the <down> and <up> keys?
> What behavior did you implement for the C-n and C-p keys?

<down> -> visual-line-down
<up> -> visual-line-up
<C-n> -> (default - normally next-line)
<C-p> -> (default - normally previous-line)


>     `kill-visual-line' and `kill-whole-visual-line', `beginning/end- 
> of-
>     visual-line' are provided as well and do the obvious.
>
> It isn't obvious to me.  I could try to guess, but I should not,
> because guessing is unreliable.  What do these commands do?

except that they will only affect the current screen line as shown on  
screen.

`end-of-visual-line' moves the point to the end of the current screen  
line, that is, to the right hand side boundary.
`beginning-of-visual-line' moves to the beginning of the current  
screen line, that is, to the left hand side boundary.

kill-visual-line and kill-whole-visual-line work just like their kill- 
line and kill-whole-line pendants, except that they will only kill  
the text to `end-of-visual-line', and in the case of kill-whole- 
visual-line, text beginning from `beginning-of-visual-line'.

visual-line-up is an interactive Lisp function in `osxkeys.el'.
(visual-line-up num-lines)

Move cursor vertically up num-lines lines.
Interactively, vscroll tall lines if `auto-window-vscroll' is
enabled.  If there is no character in the target line exactly
over the current horizontal pixel position, the cursor is
positioned close to the character in that line at the same position,
or at the end of the line if it is not long enough.

The command C-x C-n can be used to create
a semipermanent goal column for this command.
Then instead of trying to move exactly vertically (or as close as  
possible),
this command moves to the specified goal column (or as close as  
possible).
The goal column is stored in the variable `goal-column', which is nil
when there is no goal column.

This function differs from `previous-line' as it moves vertically
in the visual sense: the cursor will be positioned as horizontally  
close as
possible, and one visual line up. The result differs when variable- 
width font is
used or when characters of non-standard width (e.g. TABs) are used, and
when lines are wrapped.

If you are thinking of using this in a Lisp program, consider using
`forward-line' with a negative argument instead.  It is usually easier
to use and more reliable (no dependence on goal column, etc.).



visual-line-down is analogous.

Re: Patch: Syntax and Hard Newlines

[martin rudalics]

 >     - longlines-mode substitutes newlines for spaces.  This means that any
 >        function based on scan_newline (like `goto-line') may get me results
 >        that are inconsistent with those of tools analyzing the file my buffer
 >        visits.  How can I treat compiler or grep output with longlines-mode?
 >
 > That is true, but I suspect it is not a problem since the broken lines
 > of compiler output normally won't match the templates for error
 > messages.  (And I am not sure it makes sense to use longlines mode in
 > such buffers.)

I probably formulated that badly: I would use longlines-mode for the
source buffer and *not* for the output buffer.  Hence compiler output
would not contain any "broken lines" and would match the lines on the
file.  It would not match the lines in the source buffer though.

 > I am not sure that is a good idea, and do we really
 > care about making Longlines mode work for Lisp code?

Sorry, I completely lost you here.  This thread started as follows:

 >> As documented in the doc string of the command `use-hard-newlines',
 >> newlines not marked hard are internal to paragraphs.  However,
 >> currently the syntax feature does not work well with this.  For
 >> example, if you turn on long-lines mode with M-x longlines-mode RET in
 >> the *scratch* buffer, you will see the word `evaluation' in the first
 >> line is wrapped into the second line, but not recognized as a part of
 >> the comment.

 >> To fix this, we can let the function `forw_comment' in syntax.c skip
 >> the soft newlines when possible.  Below is the patch.

Since I never use longlines-mode I don't have any opinion on this.  But
if people do not "care about making Longlines mode work for Lisp code"
why discuss this at all?  Why reason about a syntax-table property on
newlines if their only purpose is to override a syntax property of lisp
mode?

Re: Patch: Syntax and Hard Newlines

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

> First, I note that in this message you're talking about user-level
> behavor.  Your previous message, which Miles responded to, proposed an
> implementation, and his response was about that implementation, not
> the user-level issues.  These are two different (though closely
> related) topics.
>
> So let's look at the user-level issues you've raised, and how they
> relate to implementation.
>
>     - longlines-mode substitutes newlines for spaces.  This means that any
>        function based on scan_newline (like `goto-line') may get me results
>        that are inconsistent with those of tools analyzing the file my buffer
>        visits.  How can I treat compiler or grep output with longlines-mode?
>
> That is true, but I suspect it is not a problem since the broken lines
> of compiler output normally won't match the templates for error
> messages.  (And I am not sure it makes sense to use longlines mode in
> such buffers.)

People use longlines-mode in connection with TeX source code (however
ill-conceived that may be since TeX has a hard line length limit,
nowadays set at something like 32k), and I would suspect that they'd
also use it for things like groff and HTML source, all of which might
be processed by systems giving out line numbers (and hopefully column
numbers for long lines...) for their errors.

>     - longlines-mode may wrap a regexp like "[ \t]*" at the space
>     character.  How can I evaluate a wrapped regexp like that?
>
> The only way to avoid that consequence would be to switch from "soft
> newlines" to "power spaces".

Yes, this might be the ticket.

> I am not sure that is a good idea,

For some modes, probably.

> Longlines mode should preserve other text properties when it
> converts a space to a soft newline and vise versa.
>
>     - I use two windows of different widths to simultaneously show
>     one and the same buffer.  How can I adapt longlines-mode to wrap
>     at the right borders of my windows?
>
> It is impossible for anything like Longlines mode to do that.  That
> would require a feature implemented entirely within redisplay.

window-dependent overlays would do the trick, or a display text
property could use (when...

However, I see in

(info "(elisp) Other Display Specs.")

       You can make any display specification conditional.  To do that,
    package it in another list of the form `(when CONDITION . SPEC)'.  Then
    the specification SPEC applies only when CONDITION evaluates to a
    non-`nil' value.  During the evaluation, `object' is bound to the
    string or buffer having the conditional `display' property.  `position'
    and `buffer-position' are bound to the position within `object' and the
    buffer position where the `display' property was found, respectively.
    Both positions can be different when `object' is a string.

Since object is bound to a buffer, not a window, this would not help.
Given that text properties are supposed to be a feature of the text,
not the window, I can understand that.

> There would be no fundamental difficulty in implementing such a
> feature inside Emacs redisplay, but it would have to be implemented
> entirely in C.  It would be a variant of the current feature of line
> continuation, and it would share the big inconvenience of line
> continuation: all editing commands treat the line that has been
> continued as one single line.

Well, there is linemove.el or something like that IIRC that alleviates
some of the navigation problems.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Patch: Syntax and Hard Newlines

[Herbert Euler]

>- I use two windows of different widths to simultaneously show one and
>   the same buffer.  How can I adapt longlines-mode to wrap at the right
>   borders of my windows?

Having read your new definition of `longlines-wrap-region',
how about making the value of the `display' property more
sophisticated?  I see in the ``Other Display Specs'' section
in the elisp manual that display specifications can be
conditional.  How about letting it take care of fill-column
and window width?

Regards,
Guanpeng Xu

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

Re: Patch: Syntax and Hard Newlines

[martin rudalics]

 > Having read your new definition of `longlines-wrap-region',
 > how about making the value of the `display' property more
 > sophisticated?  I see in the ``Other Display Specs'' section
 > in the elisp manual that display specifications can be
 > conditional.  How about letting it take care of fill-column
 > and window width?

That's been the idea behind my proposal.  In addition, the display
property could accommodate leading spaces to align continuation lines
with the original line and highlight such spaces in a different face.
However, if Kim plans to incorporate a longlines-like feature in the
redisplay engine we really shouldn't bother.

Re: Patch: Syntax and Hard Newlines

[Stefan Monnier]

> BTW: How about this problem?  I'm not sure what I should do now.
> Could somebody please give some suggestions?  Thanks.

> A list of problems:

>    [1] The orignal problem in longlines.el.

>    [2] Whether adding a `syntax-table' property when inserting
>        soft newlines.

>    [3] The problem Johan refered.

>    [4] The problem Martin refered.

I don't know those problems are.  But here's what I think you should do:

1 - add a syntax-table property (value (0), meaning "whitespace") on
    soft newlines.

2 - set parse-sexp-lookup-property to t when turning on the minor mode.

3 - don't unset parse-sexp-lookup-property because it's too much trouble
    with no known benefit.

4 - if you want to remove the syntax-table property when turning of the
    minor mode off, that's would be OK, although I'm not sure it's worth the
    trouble for now.


-- Stefan

Re: Patch: Syntax and Hard Newlines

[Richard Stallman]

    3 - don't unset parse-sexp-lookup-property because it's too much trouble
	with no known benefit.

The only benefit I can think of for turning this off is speedup.
But how much of a speedup does it give?  Would someone like to try
some real cases with large files and see if there is any case
where this really matters?

Re: Patch: Syntax and Hard Newlines

[Stefan Monnier]

>     3 - don't unset parse-sexp-lookup-property because it's too much trouble
> 	with no known benefit.

> The only benefit I can think of for turning this off is speedup.
> But how much of a speedup does it give?  Would someone like to try
> some real cases with large files and see if there is any case
> where this really matters?

I haven't bothered to test, but I do know that the slowdown can't be very
serious, other people using c-mode, c++-mode, java-mode, perl-mode,
scheme-mode, python-mode, latex-mode, haskell-mode, ada-mode, texinfo-mode,
... would have complained already.


        Stefan

Re: Patch: Syntax and Hard Newlines

[Richard Stallman]

    > But how much of a speedup does it give?  Would someone like to try
    > some real cases with large files and see if there is any case
    > where this really matters?

    I haven't bothered to test,

I would like someone to test this so I can see for sure.

Re: Patch: Syntax and Hard Newlines

[Herbert Euler]

>I would like someone to test this so I can see for sure.

I created a large file, which contains 4185181 lines that consist of
randomly generated characters as Emacs Lisp comment, and totally
204MB.  The following forms are evaluated:

(let (time-diff)
  (save-window-excursion
    (let (old-time)
      (switch-to-buffer "large_file")
      (fundamental-mode)
      (setq parse-sexp-lookup-properties nil
            old-time (current-time))
      (emacs-lisp-mode)
      (setq time-diff (time-subtract (current-time)
                                     old-time))))
  time-diff)
==> (0 6 748961)

(let (time-diff)
  (save-window-excursion
    (let (old-time)
      (switch-to-buffer "large_file")
      (fundamental-mode)
      (setq parse-sexp-lookup-properties t
            old-time (current-time))
      (emacs-lisp-mode)
      (setq time-diff (time-subtract (current-time)
                                     old-time))))
  time-diff)
==> (0 6 740149)

(let (time-diff)
  (save-window-excursion
    (let (old-time)
      (switch-to-buffer "large_file")
      (fundamental-mode)
      (setq parse-sexp-lookup-properties nil
            old-time (current-time))
      (emacs-lisp-mode)
      (setq time-diff (time-subtract (current-time)
                                     old-time))))
  time-diff)
==> (0 6 727099)

(let (time-diff)
  (save-window-excursion
    (let (old-time)
      (switch-to-buffer "large_file")
      (fundamental-mode)
      (setq parse-sexp-lookup-properties t
            old-time (current-time))
      (emacs-lisp-mode)
      (setq time-diff (time-subtract (current-time)
                                     old-time))))
  time-diff)
==> (0 6 740501)

I evaluated them twice.  It seems memory allocation took too long in
the first time.

Regards,
Guanpeng Xu

_________________________________________________________________
Don't just search. Find. Check out the new MSN Search! 
http://search.msn.com/

Re: Patch: Syntax and Hard Newlines

[Richard Stallman]

	  (setq parse-sexp-lookup-properties t
		old-time (current-time))
	  (emacs-lisp-mode)
	  (setq time-diff (time-subtract (current-time)
					 old-time))))

I don't think that is a real test, because merely enabling Emacs Lisp
mode does not parse anything.  You need to do some parsing
across the buffer.

Re: Patch: Syntax and Hard Newlines

[Herbert Euler]

>I don't think that is a real test, because merely enabling Emacs Lisp
>mode does not parse anything.  You need to do some parsing
>across the buffer.

What about this?  The same file with 4185181 lines, totally 204MB.

(let (time-diff
      state)
  (save-window-excursion
    (let (old-time)
      (switch-to-buffer "large_file")
      (emacs-lisp-mode)
      (setq parse-sexp-lookup-properties nil
            old-time (current-time))
      (goto-char (point-max))
      (setq state (syntax-ppss (point))
            time-diff (time-subtract (current-time)
                                     old-time))))
  (pp state)
  time-diff)
     -| (0 nil 213444184 nil nil nil 0 nil nil nil)
     => (0 25 882543)

(let (time-diff
      state)
  (save-window-excursion
    (let (old-time)
      (switch-to-buffer "large_file")
      (emacs-lisp-mode)
      (setq parse-sexp-lookup-properties t
            old-time (current-time))
      (goto-char (point-max))
      (setq state (syntax-ppss (point))
            time-diff (time-subtract (current-time)
                                     old-time))))
  (pp state)
  time-diff)
     -| (0 nil nil nil t nil 0 nil 1 nil)
     => (0 20 81466)

(let (time-diff
      state)
  (save-window-excursion
    (let (old-time)
      (switch-to-buffer "large_file")
      (emacs-lisp-mode)
      (setq parse-sexp-lookup-properties nil
            old-time (current-time))
      (goto-char (point-max))
      (setq state (syntax-ppss (point))
            time-diff (time-subtract (current-time)
                                     old-time))))
  (pp state)
  time-diff)
     -| (0 nil 213444184 nil nil nil 0 nil nil nil)
     => (0 26 100278)

(let (time-diff
      state)
  (save-window-excursion
    (let (old-time)
      (switch-to-buffer "large_file")
      (emacs-lisp-mode)
      (setq parse-sexp-lookup-properties t
            old-time (current-time))
      (goto-char (point-max))
      (setq state (syntax-ppss (point))
            time-diff (time-subtract (current-time)
                                     old-time))))
  (pp state)
  time-diff)
     -| (0 nil nil nil t nil 0 nil 1 nil)
     => (0 20 88486)

(let (time-diff
      state)
  (save-window-excursion
    (let (old-time)
      (switch-to-buffer "large_file")
      (emacs-lisp-mode)
      (setq parse-sexp-lookup-properties nil
            old-time (current-time))
      (goto-char (point-max))
      (setq state (syntax-ppss (point))
            time-diff (time-subtract (current-time)
                                     old-time))))
  (pp state)
  time-diff)
     -| (0 nil 213444184 nil nil nil 0 nil nil nil)
     => (0 25 897448)

(let (time-diff
      state)
  (save-window-excursion
    (let (old-time)
      (switch-to-buffer "large_file")
      (emacs-lisp-mode)
      (setq parse-sexp-lookup-properties t
            old-time (current-time))
      (goto-char (point-max))
      (setq state (syntax-ppss (point))
            time-diff (time-subtract (current-time)
                                     old-time))))
  (pp state)
  time-diff)
     -| (0 nil nil nil t nil 0 nil 1 nil)
     => (0 20 48766)

I surprisedly found that it is faster of setting
`parse-sexp-lookup-properties' to t than setting it to nil.

Regards,
Guanpeng Xu

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

Re: Patch: Syntax and Hard Newlines

[Richard Stallman]

    I surprisedly found that it is faster of setting
    `parse-sexp-lookup-properties' to t than setting it to nil.

It is a surprise for me too, but anyway, I am convinced it is
ok for `parse-sexp-lookup-properties' to be t by default.

Stefan, do you think there is any risk this change might break
anything?

Re: Patch: Syntax and Hard Newlines

[Stefan Monnier]

>     I surprisedly found that it is faster of setting
>     `parse-sexp-lookup-properties' to t than setting it to nil.

I'm not completely sure whether your measurements are fair.  Have you made
sure that syntax-ppss's caching doesn't give an unfair advantage to one or
the other?  I'd suggest to try with a real "micro-benchmark" and use
parse-partial-sexp rather than syntax-ppss.  You could also do it in a real
buffer (e.g. turn off jit-lock, load a big Perl file: font-lock should
place a fair bit of syntax-table properties in there, then call
parse-partial-sexp with and without parse-sexp-lookup-properties).

Try it also with other important affected commands such a forward-sexp.

That it goes faster sounds *very* surprising indeed.

> It is a surprise for me too, but anyway, I am convinced it is
> ok for `parse-sexp-lookup-properties' to be t by default.

> Stefan, do you think there is any risk this change might break
> anything?

I can't think of any case where it would cause a problem, no.  OTOH I'm not
sure it'd be worth taking the risk now,


        Stefan

Re: Patch: Syntax and Hard Newlines

[Herbert Euler]

>I'm not completely sure whether your measurements are fair.  Have you made
>sure that syntax-ppss's caching doesn't give an unfair advantage to one or
>the other?  I'd suggest to try with a real "micro-benchmark" and use
>parse-partial-sexp rather than syntax-ppss.  You could also do it in a real
>buffer (e.g. turn off jit-lock, load a big Perl file: font-lock should
>place a fair bit of syntax-table properties in there, then call
>parse-partial-sexp with and without parse-sexp-lookup-properties).

I copied the content of

    http://www.cpan.org/authors/id/W/WA/WAYNEM/betsie-1.5.12.pl

many times so that got a Perl file of 268274172 bytes (257 MB).  I
then evaluated the following forms:

(save-window-excursion (find-file "~/1.pl"))
     => #<buffer 1.pl>

(let (old-time
      time-diff
      (parse-sexp-lookup-properties nil))
  (save-window-excursion
    (switch-to-buffer "1.pl")
    (font-lock-mode -1)
    (goto-char 268273054)
    (jit-lock-mode nil)
    (font-lock-mode 1)
    (setq old-time (current-time))
    (parse-partial-sexp 1 (point))
    (setq time-diff (time-subtract (current-time)
				   old-time)))
  time-diff)
     => (0 12 952121)

(let (old-time
      time-diff
      (parse-sexp-lookup-properties t))
  (save-window-excursion
    (switch-to-buffer "1.pl")
    (font-lock-mode -1)
    (goto-char 268273054)
    (jit-lock-mode nil)
    (font-lock-mode 1)
    (setq old-time (current-time))
    (parse-partial-sexp 1 (point))
    (setq time-diff (time-subtract (current-time)
				   old-time)))
  time-diff)
     => (0 12 977639)

(let (old-time
      time-diff
      (parse-sexp-lookup-properties nil))
  (save-window-excursion
    (switch-to-buffer "1.pl")
    (font-lock-mode -1)
    (goto-char 268273054)
    (jit-lock-mode nil)
    (font-lock-mode 1)
    (setq old-time (current-time))
    (parse-partial-sexp 1 (point))
    (setq time-diff (time-subtract (current-time)
				   old-time)))
  time-diff)
     => (0 12 936413)

(let (old-time
      time-diff
      (parse-sexp-lookup-properties t))
  (save-window-excursion
    (switch-to-buffer "1.pl")
    (font-lock-mode -1)
    (goto-char 268273054)
    (jit-lock-mode nil)
    (font-lock-mode 1)
    (setq old-time (current-time))
    (parse-partial-sexp 1 (point))
    (setq time-diff (time-subtract (current-time)
				   old-time)))
  time-diff)
     => (0 12 941655)

(let (old-time
      time-diff
      (parse-sexp-lookup-properties nil))
  (save-window-excursion
    (switch-to-buffer "1.pl")
    (font-lock-mode -1)
    (goto-char 268273054)
    (jit-lock-mode nil)
    (font-lock-mode 1)
    (setq old-time (current-time))
    (parse-partial-sexp 1 (point))
    (setq time-diff (time-subtract (current-time)
				   old-time)))
  time-diff)
     => (0 12 936427)

(let (old-time
      time-diff
      (parse-sexp-lookup-properties t))
  (save-window-excursion
    (switch-to-buffer "1.pl")
    (font-lock-mode -1)
    (goto-char 268273054)
    (jit-lock-mode nil)
    (font-lock-mode 1)
    (setq old-time (current-time))
    (parse-partial-sexp 1 (point))
    (setq time-diff (time-subtract (current-time)
				   old-time)))
  time-diff)
     => (0 12 985199)

(let (old-time
      time-diff
      (parse-sexp-lookup-properties nil))
  (save-window-excursion
    (switch-to-buffer "1.pl")
    (font-lock-mode -1)
    (goto-char 268273054)
    (jit-lock-mode nil)
    (font-lock-mode 1)
    (setq old-time (current-time))
    (parse-partial-sexp 1 (point))
    (setq time-diff (time-subtract (current-time)
				   old-time)))
  time-diff)
     => (0 12 928313)

(let (old-time
      time-diff
      (parse-sexp-lookup-properties t))
  (save-window-excursion
    (switch-to-buffer "1.pl")
    (font-lock-mode -1)
    (goto-char 268273054)
    (jit-lock-mode nil)
    (font-lock-mode 1)
    (setq old-time (current-time))
    (parse-partial-sexp 1 (point))
    (setq time-diff (time-subtract (current-time)
				   old-time)))
  time-diff)
     => (0 12 928447)

(let (old-time
      time-diff
      (parse-sexp-lookup-properties nil))
  (save-window-excursion
    (switch-to-buffer "1.pl")
    (font-lock-mode -1)
    (goto-char 268273054)
    (jit-lock-mode nil)
    (font-lock-mode 1)
    (setq old-time (current-time))
    (parse-partial-sexp 1 (point))
    (setq time-diff (time-subtract (current-time)
				   old-time)))
  time-diff)
     => (0 13 70452)

(let (old-time
      time-diff
      (parse-sexp-lookup-properties t))
  (save-window-excursion
    (switch-to-buffer "1.pl")
    (font-lock-mode -1)
    (goto-char 268273054)
    (jit-lock-mode nil)
    (font-lock-mode 1)
    (setq old-time (current-time))
    (parse-partial-sexp 1 (point))
    (setq time-diff (time-subtract (current-time)
				   old-time)))
  time-diff)
     => (0 12 929632)

(let (old-time
      time-diff
      (parse-sexp-lookup-properties nil))
  (save-window-excursion
    (switch-to-buffer "1.pl")
    (font-lock-mode -1)
    (goto-char 268273054)
    (jit-lock-mode nil)
    (font-lock-mode 1)
    (setq old-time (current-time))
    (parse-partial-sexp 1 (point))
    (setq time-diff (time-subtract (current-time)
				   old-time)))
  time-diff)
     => (0 12 926650)

(let (old-time
      time-diff
      (parse-sexp-lookup-properties t))
  (save-window-excursion
    (switch-to-buffer "1.pl")
    (font-lock-mode -1)
    (goto-char 268273054)
    (jit-lock-mode nil)
    (font-lock-mode 1)
    (setq old-time (current-time))
    (parse-partial-sexp 1 (point))
    (setq time-diff (time-subtract (current-time)
				   old-time)))
  time-diff)
     => (0 12 929234)

(let (old-time
      time-diff
      (parse-sexp-lookup-properties nil))
  (save-window-excursion
    (switch-to-buffer "1.pl")
    (font-lock-mode -1)
    (goto-char 268273054)
    (jit-lock-mode nil)
    (font-lock-mode 1)
    (setq old-time (current-time))
    (parse-partial-sexp 1 (point))
    (setq time-diff (time-subtract (current-time)
				   old-time)))
  time-diff)
     => (0 12 978655)

(let (old-time
      time-diff
      (parse-sexp-lookup-properties t))
  (save-window-excursion
    (switch-to-buffer "1.pl")
    (font-lock-mode -1)
    (goto-char 268273054)
    (jit-lock-mode nil)
    (font-lock-mode 1)
    (setq old-time (current-time))
    (parse-partial-sexp 1 (point))
    (setq time-diff (time-subtract (current-time)
				   old-time)))
  time-diff)
     => (0 12 928371)

Can I conclude from this result that `parse-sexp-lookup-properties'
has no effect over syntax parsing?

Regards,
Guanpeng Xu

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

Re: Patch: Syntax and Hard Newlines

[Herbert Euler]

On timing, I meant.

Regards,
Guanpeng Xu


>From: "Herbert Euler" <herberteuler@hotmail.com>
>To: monnier@iro.umontreal.ca, rms@gnu.org
>CC: rudalics@gmx.at, emacs-devel@gnu.org
>Subject: Re: Patch: Syntax and Hard Newlines
>Date: Wed, 29 Nov 2006 12:06:12 +0800
>
>>I'm not completely sure whether your measurements are fair.  Have you made
>>sure that syntax-ppss's caching doesn't give an unfair advantage to one or
>>the other?  I'd suggest to try with a real "micro-benchmark" and use
>>parse-partial-sexp rather than syntax-ppss.  You could also do it in a 
>>real
>>buffer (e.g. turn off jit-lock, load a big Perl file: font-lock should
>>place a fair bit of syntax-table properties in there, then call
>>parse-partial-sexp with and without parse-sexp-lookup-properties).
>
>I copied the content of
>
>    http://www.cpan.org/authors/id/W/WA/WAYNEM/betsie-1.5.12.pl
>
>many times so that got a Perl file of 268274172 bytes (257 MB).  I
>then evaluated the following forms:
>
>(save-window-excursion (find-file "~/1.pl"))
>     => #<buffer 1.pl>
>
>(let (old-time
>      time-diff
>      (parse-sexp-lookup-properties nil))
>  (save-window-excursion
>    (switch-to-buffer "1.pl")
>    (font-lock-mode -1)
>    (goto-char 268273054)
>    (jit-lock-mode nil)
>    (font-lock-mode 1)
>    (setq old-time (current-time))
>    (parse-partial-sexp 1 (point))
>    (setq time-diff (time-subtract (current-time)
>				   old-time)))
>  time-diff)
>     => (0 12 952121)
>
>(let (old-time
>      time-diff
>      (parse-sexp-lookup-properties t))
>  (save-window-excursion
>    (switch-to-buffer "1.pl")
>    (font-lock-mode -1)
>    (goto-char 268273054)
>    (jit-lock-mode nil)
>    (font-lock-mode 1)
>    (setq old-time (current-time))
>    (parse-partial-sexp 1 (point))
>    (setq time-diff (time-subtract (current-time)
>				   old-time)))
>  time-diff)
>     => (0 12 977639)
>
>(let (old-time
>      time-diff
>      (parse-sexp-lookup-properties nil))
>  (save-window-excursion
>    (switch-to-buffer "1.pl")
>    (font-lock-mode -1)
>    (goto-char 268273054)
>    (jit-lock-mode nil)
>    (font-lock-mode 1)
>    (setq old-time (current-time))
>    (parse-partial-sexp 1 (point))
>    (setq time-diff (time-subtract (current-time)
>				   old-time)))
>  time-diff)
>     => (0 12 936413)
>
>(let (old-time
>      time-diff
>      (parse-sexp-lookup-properties t))
>  (save-window-excursion
>    (switch-to-buffer "1.pl")
>    (font-lock-mode -1)
>    (goto-char 268273054)
>    (jit-lock-mode nil)
>    (font-lock-mode 1)
>    (setq old-time (current-time))
>    (parse-partial-sexp 1 (point))
>    (setq time-diff (time-subtract (current-time)
>				   old-time)))
>  time-diff)
>     => (0 12 941655)
>
>(let (old-time
>      time-diff
>      (parse-sexp-lookup-properties nil))
>  (save-window-excursion
>    (switch-to-buffer "1.pl")
>    (font-lock-mode -1)
>    (goto-char 268273054)
>    (jit-lock-mode nil)
>    (font-lock-mode 1)
>    (setq old-time (current-time))
>    (parse-partial-sexp 1 (point))
>    (setq time-diff (time-subtract (current-time)
>				   old-time)))
>  time-diff)
>     => (0 12 936427)
>
>(let (old-time
>      time-diff
>      (parse-sexp-lookup-properties t))
>  (save-window-excursion
>    (switch-to-buffer "1.pl")
>    (font-lock-mode -1)
>    (goto-char 268273054)
>    (jit-lock-mode nil)
>    (font-lock-mode 1)
>    (setq old-time (current-time))
>    (parse-partial-sexp 1 (point))
>    (setq time-diff (time-subtract (current-time)
>				   old-time)))
>  time-diff)
>     => (0 12 985199)
>
>(let (old-time
>      time-diff
>      (parse-sexp-lookup-properties nil))
>  (save-window-excursion
>    (switch-to-buffer "1.pl")
>    (font-lock-mode -1)
>    (goto-char 268273054)
>    (jit-lock-mode nil)
>    (font-lock-mode 1)
>    (setq old-time (current-time))
>    (parse-partial-sexp 1 (point))
>    (setq time-diff (time-subtract (current-time)
>				   old-time)))
>  time-diff)
>     => (0 12 928313)
>
>(let (old-time
>      time-diff
>      (parse-sexp-lookup-properties t))
>  (save-window-excursion
>    (switch-to-buffer "1.pl")
>    (font-lock-mode -1)
>    (goto-char 268273054)
>    (jit-lock-mode nil)
>    (font-lock-mode 1)
>    (setq old-time (current-time))
>    (parse-partial-sexp 1 (point))
>    (setq time-diff (time-subtract (current-time)
>				   old-time)))
>  time-diff)
>     => (0 12 928447)
>
>(let (old-time
>      time-diff
>      (parse-sexp-lookup-properties nil))
>  (save-window-excursion
>    (switch-to-buffer "1.pl")
>    (font-lock-mode -1)
>    (goto-char 268273054)
>    (jit-lock-mode nil)
>    (font-lock-mode 1)
>    (setq old-time (current-time))
>    (parse-partial-sexp 1 (point))
>    (setq time-diff (time-subtract (current-time)
>				   old-time)))
>  time-diff)
>     => (0 13 70452)
>
>(let (old-time
>      time-diff
>      (parse-sexp-lookup-properties t))
>  (save-window-excursion
>    (switch-to-buffer "1.pl")
>    (font-lock-mode -1)
>    (goto-char 268273054)
>    (jit-lock-mode nil)
>    (font-lock-mode 1)
>    (setq old-time (current-time))
>    (parse-partial-sexp 1 (point))
>    (setq time-diff (time-subtract (current-time)
>				   old-time)))
>  time-diff)
>     => (0 12 929632)
>
>(let (old-time
>      time-diff
>      (parse-sexp-lookup-properties nil))
>  (save-window-excursion
>    (switch-to-buffer "1.pl")
>    (font-lock-mode -1)
>    (goto-char 268273054)
>    (jit-lock-mode nil)
>    (font-lock-mode 1)
>    (setq old-time (current-time))
>    (parse-partial-sexp 1 (point))
>    (setq time-diff (time-subtract (current-time)
>				   old-time)))
>  time-diff)
>     => (0 12 926650)
>
>(let (old-time
>      time-diff
>      (parse-sexp-lookup-properties t))
>  (save-window-excursion
>    (switch-to-buffer "1.pl")
>    (font-lock-mode -1)
>    (goto-char 268273054)
>    (jit-lock-mode nil)
>    (font-lock-mode 1)
>    (setq old-time (current-time))
>    (parse-partial-sexp 1 (point))
>    (setq time-diff (time-subtract (current-time)
>				   old-time)))
>  time-diff)
>     => (0 12 929234)
>
>(let (old-time
>      time-diff
>      (parse-sexp-lookup-properties nil))
>  (save-window-excursion
>    (switch-to-buffer "1.pl")
>    (font-lock-mode -1)
>    (goto-char 268273054)
>    (jit-lock-mode nil)
>    (font-lock-mode 1)
>    (setq old-time (current-time))
>    (parse-partial-sexp 1 (point))
>    (setq time-diff (time-subtract (current-time)
>				   old-time)))
>  time-diff)
>     => (0 12 978655)
>
>(let (old-time
>      time-diff
>      (parse-sexp-lookup-properties t))
>  (save-window-excursion
>    (switch-to-buffer "1.pl")
>    (font-lock-mode -1)
>    (goto-char 268273054)
>    (jit-lock-mode nil)
>    (font-lock-mode 1)
>    (setq old-time (current-time))
>    (parse-partial-sexp 1 (point))
>    (setq time-diff (time-subtract (current-time)
>				   old-time)))
>  time-diff)
>     => (0 12 928371)
>
>Can I conclude from this result that `parse-sexp-lookup-properties'
>has no effect over syntax parsing?
>
>Regards,
>Guanpeng Xu
>
>_________________________________________________________________
>Express yourself instantly with MSN Messenger! Download today it's FREE! 
>http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/
>
>
>
>_______________________________________________
>Emacs-devel mailing list
>Emacs-devel@gnu.org
>http://lists.gnu.org/mailman/listinfo/emacs-devel

_________________________________________________________________
FREE pop-up blocking with the new MSN Toolbar - get it now! 
http://toolbar.msn.click-url.com/go/onm00200415ave/direct/01/

Re: Patch: Syntax and Hard Newlines

[Stefan Monnier]

> many times so that got a Perl file of 268274172 bytes (257 MB).  I
> then evaluated the following forms:
[...]
> Can I conclude from this result that `parse-sexp-lookup-properties'
> has no effect over syntax parsing?

I believe so, yes.

Although I wonder why you did

   (let (old-time
         time-diff
         (parse-sexp-lookup-properties nil))
    (save-window-excursion
      (switch-to-buffer "1.pl")
      (font-lock-mode -1)
      (goto-char 268273054)
      (jit-lock-mode nil)
      (font-lock-mode 1)
      (setq old-time (current-time))
      (parse-partial-sexp 1 (point))
      (setq time-diff (time-subtract (current-time)
   				   old-time)))

rather than

    (with-current-buffer (find-file-noselect "1.pl")
      (font-lock-mode -1)
      (setq font-lock-support-mode nil)
      (font-lock-mode 1)
      (let ((old-time (current-time))
            (parse-sexp-lookup-properties nil))
        (parse-partial-sexp (point-min) (point-max))
        (time-subtract (current-time) old-time)))

The main difference is that:/ your code did:
- misused switch-to-buffer and generally fiddled unnecessarily with windows
- failed to disable jit-lock, so the buffer probably had no syntax-table
  (or face) property set anywhere while running parse-partial-sexp.
- did not bind parse-sexp-lookup-properties right before running
  parse-partial-sexp which is safer: perl-mode sets
  parse-sexp-lookup-properties via font-lock-defaults, so your let-binding
  of parse-sexp-lookup-properties to nil outside may be overridden when
  turning on font-lock-mode.
  But I believe that this did not affect your measurements because
  font-lock-mode had been turned ON before, so when you turned it on the
  second time, font-lock-set-defaults didn't do anything.


        Stefan

Re: Patch: Syntax and Hard Newlines

[Herbert Euler]

>Although I wonder why you did
>
>    (let (old-time
>          time-diff
>          (parse-sexp-lookup-properties nil))
>     (save-window-excursion
>       (switch-to-buffer "1.pl")
>       (font-lock-mode -1)
>       (goto-char 268273054)
>       (jit-lock-mode nil)
>       (font-lock-mode 1)
>       (setq old-time (current-time))
>       (parse-partial-sexp 1 (point))
>       (setq time-diff (time-subtract (current-time)
>    				   old-time)))
>
>rather than
>
>     (with-current-buffer (find-file-noselect "1.pl")
>       (font-lock-mode -1)
>       (setq font-lock-support-mode nil)
>       (font-lock-mode 1)
>       (let ((old-time (current-time))
>             (parse-sexp-lookup-properties nil))
>         (parse-partial-sexp (point-min) (point-max))
>         (time-subtract (current-time) old-time)))

These are the results of evaluating the forms defined by you:

(with-current-buffer (find-file-noselect "1.pl")
  (font-lock-mode -1)
  (setq font-lock-support-mode nil)
  (font-lock-mode 1)
  (let ((old-time (current-time))
	(parse-sexp-lookup-properties nil))
    (parse-partial-sexp (point-min) (point-max))
    (time-subtract (current-time) old-time)))
     => (0 12 944334)

(with-current-buffer (find-file-noselect "1.pl")
  (font-lock-mode -1)
  (setq font-lock-support-mode nil)
  (font-lock-mode 1)
  (let ((old-time (current-time))
	(parse-sexp-lookup-properties t))
    (parse-partial-sexp (point-min) (point-max))
    (time-subtract (current-time) old-time)))
     => (0 12 918259)

(with-current-buffer (find-file-noselect "1.pl")
  (font-lock-mode -1)
  (setq font-lock-support-mode nil)
  (font-lock-mode 1)
  (let ((old-time (current-time))
	(parse-sexp-lookup-properties nil))
    (parse-partial-sexp (point-min) (point-max))
    (time-subtract (current-time) old-time)))
     => (0 12 907874)

(with-current-buffer (find-file-noselect "1.pl")
  (font-lock-mode -1)
  (setq font-lock-support-mode nil)
  (font-lock-mode 1)
  (let ((old-time (current-time))
	(parse-sexp-lookup-properties t))
    (parse-partial-sexp (point-min) (point-max))
    (time-subtract (current-time) old-time)))
     => (0 12 916414)

(with-current-buffer (find-file-noselect "1.pl")
  (font-lock-mode -1)
  (setq font-lock-support-mode nil)
  (font-lock-mode 1)
  (let ((old-time (current-time))
	(parse-sexp-lookup-properties nil))
    (parse-partial-sexp (point-min) (point-max))
    (time-subtract (current-time) old-time)))
     => (0 12 935408)

(with-current-buffer (find-file-noselect "1.pl")
  (font-lock-mode -1)
  (setq font-lock-support-mode nil)
  (font-lock-mode 1)
  (let ((old-time (current-time))
	(parse-sexp-lookup-properties t))
    (parse-partial-sexp (point-min) (point-max))
    (time-subtract (current-time) old-time)))
     => (0 13 12071)

(with-current-buffer (find-file-noselect "1.pl")
  (font-lock-mode -1)
  (setq font-lock-support-mode nil)
  (font-lock-mode 1)
  (let ((old-time (current-time))
	(parse-sexp-lookup-properties nil))
    (parse-partial-sexp (point-min) (point-max))
    (time-subtract (current-time) old-time)))
     => (0 12 927589)

(with-current-buffer (find-file-noselect "1.pl")
  (font-lock-mode -1)
  (setq font-lock-support-mode nil)
  (font-lock-mode 1)
  (let ((old-time (current-time))
	(parse-sexp-lookup-properties t))
    (parse-partial-sexp (point-min) (point-max))
    (time-subtract (current-time) old-time)))
     => (0 12 920150)

(with-current-buffer (find-file-noselect "1.pl")
  (font-lock-mode -1)
  (setq font-lock-support-mode nil)
  (font-lock-mode 1)
  (let ((old-time (current-time))
	(parse-sexp-lookup-properties nil))
    (parse-partial-sexp (point-min) (point-max))
    (time-subtract (current-time) old-time)))
     => (0 12 819639)

(with-current-buffer (find-file-noselect "1.pl")
  (font-lock-mode -1)
  (setq font-lock-support-mode nil)
  (font-lock-mode 1)
  (let ((old-time (current-time))
	(parse-sexp-lookup-properties t))
    (parse-partial-sexp (point-min) (point-max))
    (time-subtract (current-time) old-time)))
     => (0 12 926456)

(with-current-buffer (find-file-noselect "1.pl")
  (font-lock-mode -1)
  (setq font-lock-support-mode nil)
  (font-lock-mode 1)
  (let ((old-time (current-time))
	(parse-sexp-lookup-properties nil))
    (parse-partial-sexp (point-min) (point-max))
    (time-subtract (current-time) old-time)))
     => (0 12 877235)

(with-current-buffer (find-file-noselect "1.pl")
  (font-lock-mode -1)
  (setq font-lock-support-mode nil)
  (font-lock-mode 1)
  (let ((old-time (current-time))
	(parse-sexp-lookup-properties t))
    (parse-partial-sexp (point-min) (point-max))
    (time-subtract (current-time) old-time)))
     => (0 12 915513)

(with-current-buffer (find-file-noselect "1.pl")
  (font-lock-mode -1)
  (setq font-lock-support-mode nil)
  (font-lock-mode 1)
  (let ((old-time (current-time))
	(parse-sexp-lookup-properties nil))
    (parse-partial-sexp (point-min) (point-max))
    (time-subtract (current-time) old-time)))
     => (0 12 822557)

(with-current-buffer (find-file-noselect "1.pl")
  (font-lock-mode -1)
  (setq font-lock-support-mode nil)
  (font-lock-mode 1)
  (let ((old-time (current-time))
	(parse-sexp-lookup-properties t))
    (parse-partial-sexp (point-min) (point-max))
    (time-subtract (current-time) old-time)))
     => (0 12 960385)

I got "Fontifying 1.pl...buffer size greater than
font-lock-maximum-size" when evaluating the forms.

>From this result I can see the delay caused by
`parse-sexp-lookup-properties'.

Regards,
Guanpeng Xu

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

Re: Patch: Syntax and Hard Newlines

[Stefan Monnier]

> I got "Fontifying 1.pl...buffer size greater than
> font-lock-maximum-size" when evaluating the forms.

Which again implies that the buffer had no syntax-table (or face)
properties applied.  Better set font-lock-maximum-size to nil.
And BTW, there's no need to redo the font-lock-mode OFF/ON all the time.
You just need to redo the call to parse-partial-sexp.

>> From this result I can see the delay caused by
> `parse-sexp-lookup-properties'.

To me it looked like it was lost in the noise.


        Stefan

Re: Patch: Syntax and Hard Newlines

[Richard Stallman]

It seems to be a 1% slowdown, which is small enough we
can ignore it.

The only way we can get more thorough testing is to make the change
(enabling parse-sexp-lookup-properties by default) and have people try
it for a while.  If we don't NEED to do that now, let's not.
However, if there is a bug which is hard to fix in any other way,
we can fix it this way now.

Re: Patch: Syntax and Hard Newlines

[David Kastrup]

"Herbert Euler" <herberteuler@hotmail.com> writes:

>>I'm not completely sure whether your measurements are fair.  Have you made
>>sure that syntax-ppss's caching doesn't give an unfair advantage to one or
>>the other?  I'd suggest to try with a real "micro-benchmark" and use
>>parse-partial-sexp rather than syntax-ppss.  You could also do it in a real
>>buffer (e.g. turn off jit-lock, load a big Perl file: font-lock should
>>place a fair bit of syntax-table properties in there, then call
>>parse-partial-sexp with and without parse-sexp-lookup-properties).
>
> I copied the content of
>
>    http://www.cpan.org/authors/id/W/WA/WAYNEM/betsie-1.5.12.pl
>
> many times so that got a Perl file of 268274172 bytes (257 MB).  I
> then evaluated the following forms:

Note that features like syntax highlighting that are enabled by
default need to avoid catastrophic slowdown in all cases (excepting at
most pathological ones), not just average large ones.

So a single example will not really prove that such a change is
without major consequences.  Of course, if such a single example fails
in that respect, this is not better...

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Patch: Syntax and Hard Newlines

[Richard Stallman]

The usual way to solve a problem where two different modes
want to enable a certain feature, and it should only be disabled
when both modes are disabled, is to create two variables
each of which enables the feature when non-nil.  Then each mode
can set its own variable.

This sort of method

    >I suppose it would be cleaner to require minor modes that want to turn
    >on/off `parse-sexp-lookup-properties' in a buffer append/remove their
    >name from/to that variable and kill it iff that list gets empty.

is ok if no simpler method will work.  However, using
`parse-sexp-lookup-properties' this way would require changing all
modes that set it.  We can't reliably do that, since some of them may
not be included in Emacs.

Therefore, it could be better to create a new variable to be used this
way, alongside `parse-sexp-lookup-properties'.  If either variable
is non-nil, then parse-sexp would check the properties.