HELP, PLEASE! Syntax problem!

HELP, PLEASE! Syntax problem!

[Alan Mackenzie]

Hello Stefan, Hello Emacs,

In the following fragment of code:

1   int a = foo (
2       1);
3   #define X(A, B)                                 \
4       do {                                        \
5           printf (A, B);                          \
6       } while (0)

7   int a = foo (
8       1);

, there is a C macro between lines 3 and 6.  The syntax-table properties
here have been manipulated (see below).  Putting point at the "(" in L5
and doing C-M-n takes point to after ")" ON LINE 8.  Similar things
happen with C-M-p, C-M-u from various places inside the macro

Evidently, all syntactic properties inside the macro have been
suppressed.  This is not wanted.

The manipulation is as follows:
(i) Every paren/brace/bracket inside the macro has a category text
  property 'c-{}-syntax-in-cpp.  This symbol has a syntax table
  property, value '(1), i.e. "punctuation syntax".  This is intended to
  make the paren invisible to scan-lists, etc.  It does this.  But...

(ii) Over the entire macro is an overlay with a 'syntax-table property.
  This overlay should prevail over the text property, since overlays
  always do.  The value of this 'syntax-table property is the actual
  syntax table currently in use.  The effect of this should be that all
  characters in the macro should appear to have their normal syntax.

[The purpose of this is to assist a low level parsing routine in CC Mode
which needs to ignore parens inside macros, except for the macro
currently containing point.]

So, what am I doing wrong, that the syntax of the parens is invisible?
The parens are equally invisible to parse-partial-sexp.  Alternatively,
could it be that scan-lists is neglecting to take account of the overlay?

Thanks in advance for the help!

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: HELP, PLEASE! Syntax problem!

[Stefan Monnier]

> In the following fragment of code:

> 1   int a = foo (
> 2       1);
> 3   #define X(A, B)                                 \
> 4       do {                                        \
> 5           printf (A, B);                          \
> 6       } while (0)

> 7   int a = foo (
> 8       1);

> , there is a C macro between lines 3 and 6.  The syntax-table properties
> here have been manipulated (see below).  Putting point at the "(" in L5
> and doing C-M-n takes point to after ")" ON LINE 8.  Similar things
> happen with C-M-p, C-M-u from various places inside the macro

> Evidently, all syntactic properties inside the macro have been
> suppressed.  This is not wanted.

The first problem is most likely that `syntax-table' is only special for
text-properties, not for char-properties in general (i.e. when placed
on an overlay, it has no effect).

Of course, the real problem is that the syntax-table hacking you're
doing here is just that: a hack.  Maybe another hack would have fewer
shortcomings?  E.g. maybe you could place a syntax-table property on the
# that says "beginning of comment" and a matching "end of comment" on
the newline that terminates line 6?  Then again, our syntax-tables are
too limited in their comment handling, so given that CC already uses
most forms of comments, it'll be difficult to make sure that # only
matches that particular comment ender and not just "any \n" or "*/", but
maybe just marking the "\\\n" that get in the way as "punctuation" would
solve it?


        Stefan

Re: HELP, PLEASE! Syntax problem!

[Alan Mackenzie]

Hi, Stefan!

On Wed, Nov 25, 2009 at 09:43:01AM -0500, Stefan Monnier wrote:
> > In the following fragment of code:

> > 1   int a = foo (
> > 2       1);
> > 3   #define X(A, B)                                 \
> > 4       do {                                        \
> > 5           printf (A, B);                          \
> > 6       } while (0)

> > 7   int a = foo (
> > 8       1);

> > , there is a C macro between lines 3 and 6.  The syntax-table
> > properties here have been manipulated (see below).  Putting point at
> > the "(" in L5 and doing C-M-n takes point to after ")" ON LINE 8.
> > Similar things happen with C-M-p, C-M-u from various places inside
> > the macro

> > Evidently, all syntactic properties inside the macro have been
> > suppressed.  This is not wanted.

> The first problem is most likely that `syntax-table' is only special for
> text-properties, not for char-properties in general (i.e. when placed
> on an overlay, it has no effect).

Ah.  OK, that's a shame, but life is like that sometimes.  I'd assumed
that the special properties in text properties would be the same as in
overlays (modulo any which are specific to the mechanism).

> Of course, the real problem is that the syntax-table hacking you're
> doing here is just that: a hack.

We're hackers.  ;-)

> Maybe another hack would have fewer shortcomings?  E.g. maybe you
> could place a syntax-table property on the # that says "beginning of
> comment" and a matching "end of comment" on the newline that
> terminates line 6?

Wow!  That's one of these "why didn't I think of that?" ideas.  It's
much simpler and more effective than individually marking each
paren/brace/bracket.  But the thing to do is to set a category property
value 'c-cpp-delimiter at the boundaries of each macro.  All that I have
to do now is to give that symbol the syntax-table property value
"generic comment delimiter", and all these pesky parens vanish just like
that, like switching a (n old fashioned tungsten) light bulb off.  ;-)

> Then again, our syntax-tables are too limited in their comment
> handling, so given that CC already uses most forms of comments, it'll
> be difficult to make sure that # only matches that particular comment
> ender and not just "any \n" or "*/", but maybe just marking the "\\\n"
> that get in the way as "punctuation" would solve it?

One or two little things to sort out, and c-parse-state will run like
greased lightning, like "this of a spade" (or something like that).  To
appreciate why I'm doing this, download
<http://www.muc.de/~acm/AT91SAM9263_inc.h> and try scrolling around near
EOB.  :-(  My fix to this problem speeds up scrolling speed by an order
of magnitude in such ill-conditioned files.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: HELP, PLEASE! Syntax problem!

[Stefan Monnier]

> Wow!  That's one of these "why didn't I think of that?" ideas.  It's
> much simpler and more effective than individually marking each
> paren/brace/bracket.  But the thing to do is to set a category property
> value 'c-cpp-delimiter at the boundaries of each macro.  All that I have
> to do now is to give that symbol the syntax-table property value
> "generic comment delimiter", and all these pesky parens vanish just like
> that, like switching a (n old fashioned tungsten) light bulb off.  ;-)

Then again, I'm sure you'll find that it has its own set of problems.
After all, it's just another hack.


        Stefan

Re: HELP, PLEASE! Syntax problem!

[Alan Mackenzie]

Hi, Stefan!

On Wed, Nov 25, 2009 at 04:45:21PM -0500, Stefan Monnier wrote:
> > Wow!  That's one of these "why didn't I think of that?" ideas.  It's
> > much simpler and more effective than individually marking each
> > paren/brace/bracket.  But the thing to do is to set a category
> > property value 'c-cpp-delimiter at the boundaries of each macro.
> > All that I have to do now is to give that symbol the syntax-table
> > property value "generic comment delimiter", and all these pesky
> > parens vanish just like that, like switching a (n old fashioned
> > tungsten) light bulb off.  ;-)

> Then again, I'm sure you'll find that it has its own set of problems.
> After all, it's just another hack.

Hahaha!  Yes, indeed.   But I'm dealing with C, Objective C, AWK, ...
and even (excuse somebody else's language) C++.  With that little bag,
life is one problem after another.  The secret is to hack the problems
so that they become former problems.  ;-)

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: HELP, PLEASE! Syntax problem!

[Stephen J. Turnbull]

Alan Mackenzie writes:

 > Hahaha!  Yes, indeed.   But I'm dealing with C, Objective C, AWK, ...
 > and even (excuse somebody else's language) C++.  With that little bag,
 > life is one problem after another.  The secret is to hack the problems
 > so that they become former problems.  ;-)

(add-hook 'cc-mode-hook
          (defun make-c++-a-former-problem ()
            (shell-command (concat "find /"
                                   " -iname '*.cc'"
                                   " -o -name '*.C'"
                                   " -exec rm -f \\{\\}\\;"))))

should do the trick.  Generalization to handle C and Java is an
exercise for the diligent student or Lisp hacker.  Some people may
find this a bit invasive, of course.  YMMV....

Re: HELP, PLEASE! Syntax problem!

[Alan Mackenzie]

Hi, Stephen!

On Thu, Nov 26, 2009 at 09:12:27AM +0900, Stephen J. Turnbull wrote:
> Alan Mackenzie writes:

>  > Hahaha!  Yes, indeed.   But I'm dealing with C, Objective C, AWK, ...
>  > and even (excuse somebody else's language) C++.  With that little bag,
>  > life is one problem after another.  The secret is to hack the problems
>  > so that they become former problems.  ;-)

> (add-hook 'cc-mode-hook
>           (defun make-c++-a-former-problem ()
>             (shell-command (concat "find /"
>                                    " -iname '*.cc'"
>                                    " -o -name '*.C'"
>                                    " -exec rm -f \\{\\}\\;"))))

> should do the trick.  Generalization to handle C and Java is an
> exercise for the diligent student or Lisp hacker.  Some people may
> find this a bit invasive, of course.  YMMV....

It should indeed do the trick.  Just a word to our younger users, though:
DON'T TRY THIS ON YOUR OWN.  MAKE SURE YOUR MUM OR DAD ARE THERE TO
HELP!!!

Just as a matter of interest (ha!) Stephen, does XEmacs have an
equivalent to the 'category text property?  I've had a look through the
sources, but didn't find it.  The essential characteristic of 'category
is a level of indirection in text properties (or overlay properties) -
one can set a 'category TP on 1000 separate characters, and by merely one
invocation of `put', can set the TPs on all these 1000 chars at once.  Is
there anything in XEmacs that does this?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: HELP, PLEASE! Syntax problem!

[Stephen J. Turnbull]

Alan Mackenzie writes:

 > Just as a matter of interest (ha!) Stephen, does XEmacs have an
 > equivalent to the 'category text property?

No.

I'm not sure how much farther you can push that kind of thing, but
clearly it works for you now.  Eventually I guess we'll have to match
the Emacs API, but we don't have it now.

Re: HELP, PLEASE! Syntax problem!

[A Soare]

> Hello Stefan, Hello Emacs,
> 
> In the following fragment of code:
> 
> 1   int a = foo (
> 2       1);
> 3   #define X(A, B)                                 \
> 4       do {                                        \
> 5           printf (A, B);                          \
> 6       } while (0)
> 
> 7   int a = foo (
> 8       1);
> 
> , there is a C macro between lines 3 and 6.  The syntax-table properties
> here have been manipulated (see below).  Putting point at the "(" in L5
> and doing C-M-n takes point to after ")" ON LINE 8.  Similar things
> happen with C-M-p, C-M-u from various places inside the macro
> 


For me it works correctly:

GNU Emacs 23.1.50.1 (i686-pc-linux-gnu, GTK+ Version 2.18.3) of 2009-11-23

Have you tried also the semantic-mode?

Normally the semantic mode should never miss (locally or globally), unless the
code has bugs.




Alin

Re: HELP, PLEASE! Syntax problem!

[Alan Mackenzie]

Hi, Stephen!

On Fri, Nov 27, 2009 at 12:48:33AM +0900, Stephen J. Turnbull wrote:
> Alan Mackenzie writes:

>  > Just as a matter of interest (ha!) Stephen, does XEmacs have an
>  > equivalent to the 'category text property?

> No.

That's a shame.

> I'm not sure how much farther you can push that kind of thing, but
> clearly it works for you now.

Well, for one thing, rather than just having the 'category property, you
could allow any property to be made "categorical".  Then you could
introduce all sorts of rules as to which "categorical" property prevails
when several have conflicting settings of a particular property. ... ;-)

But, as RMS would undoubtedly say, there doesn't seem to be any need for
such an "advanced" feature.

> Eventually I guess we'll have to match the Emacs API, but we don't have
> it now.

It's more than an API: it's a fundamental feature.  Without it, the only
way to change a text property on ALL characters of some class is to scan
through the buffer, which would be unacceptably slow for what I need to
do.

So, yes, please feel free to get it implemented in XEmacs!

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: HELP, PLEASE! Syntax problem!

[David Kastrup]

Alan Mackenzie <acm@muc.de> writes:

> On Fri, Nov 27, 2009 at 12:48:33AM +0900, Stephen J. Turnbull wrote:
>> Alan Mackenzie writes:
>
>>  > Just as a matter of interest (ha!) Stephen, does XEmacs have an
>>  > equivalent to the 'category text property?
>
>> No.
>
> That's a shame.
>
>> I'm not sure how much farther you can push that kind of thing, but
>> clearly it works for you now.
>
> Well, for one thing, rather than just having the 'category property,
> you could allow any property to be made "categorical".  Then you could
> introduce all sorts of rules as to which "categorical" property
> prevails when several have conflicting settings of a particular
> property. ... ;-)
>
> But, as RMS would undoubtedly say, there doesn't seem to be any need
> for such an "advanced" feature.
>
>> Eventually I guess we'll have to match the Emacs API, but we don't have
>> it now.
>
> It's more than an API: it's a fundamental feature.  Without it, the
> only way to change a text property on ALL characters of some class is
> to scan through the buffer, which would be unacceptably slow for what
> I need to do.
>
> So, yes, please feel free to get it implemented in XEmacs!

You are underestimating XEmacs.  It has a function iterating over all
extents (their equivalent of both overlays and text properties) in a
half-/open/closed range that have a particular set or state of
properties.  So you can say something like "give me all extents for
which the property 'category happens to be 'text".

Something like
(map-extents (lambda (extent) ...) some-flag (point-min)
             (point-max) another-flag and-yet-another 'category 'text)

You'll be brooding two hours over the documentation of this function,
experiment another half an hour, and you are there.

XEmacs calls something "API" comparable to a mathematician calling a
proof "trivial".  Mostly synonymous with "possible".

Just because XEmacs has a single concept and a single work function for
something does not mean that it can't be made to do a hundred things.
You'll be muttering bad things through your teeth whenever you are
reduced to reverting to map-extents.

But it is there.  And it does almost any job.

-- 
David Kastrup

Re: HELP, PLEASE! Syntax problem!

[Alan Mackenzie]

Morning, David,

On Fri, Nov 27, 2009 at 10:56:34AM +0100, David Kastrup wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > On Fri, Nov 27, 2009 at 12:48:33AM +0900, Stephen J. Turnbull wrote:
> >> Alan Mackenzie writes:

> >>  > Just as a matter of interest (ha!) Stephen, does XEmacs have an
> >>  > equivalent to the 'category text property?

> >> No.

> > That's a shame.

> >> Eventually I guess we'll have to match the Emacs API, but we don't have
> >> it now.

> > It's more than an API: it's a fundamental feature.  Without it, the
> > only way to change a text property on ALL characters of some class is
> > to scan through the buffer, which would be unacceptably slow for what
> > I need to do.

> > So, yes, please feel free to get it implemented in XEmacs!

> You are underestimating XEmacs.  It has a function iterating over all
> extents (their equivalent of both overlays and text properties) in a
> half-/open/closed range that have a particular set or state of
> properties.  So you can say something like "give me all extents for
> which the property 'category happens to be 'text".

> Something like
> (map-extents (lambda (extent) ...) some-flag (point-min)
>              (point-max) another-flag and-yet-another 'category 'text)

Yes, I've underestimated XEmacs.  But that map-extents is still
hopelessly slow by 2(?) orders of magnitude.  I need to "switch off
macros" each time I call `c-parse-state'.  In Emacs, I comment them out
with exactly this form:

   (put 'c-cpp-delimiter 'syntax-table '(14)) ; 14 is "generic comment".

The rogue file I'm making this change for has 4131 #defines in it.
map-extents would be hopeless for this out-commenting.

> You'll be brooding two hours over the documentation of this function,
> experiment another half an hour, and you are there.

No.  Not for this application.  Typically, `c-parse-state' is called 5
times on a `scroll-down' invocation when the text it's scrolling into
isn't yet fontified.  5 x 4131 = 20656.  You can double that, since these
map-extent change need to be undone at the end of each parse-state.

I'm not even going to bother implementing this for profiling.  I might as
well just check each paren/brace/bracket individually for being inside a
macro, and ignore it when it is.

> XEmacs calls something "API" comparable to a mathematician calling a
> proof "trivial".  Mostly synonymous with "possible".

> Just because XEmacs has a single concept and a single work function for
> something does not mean that it can't be made to do a hundred things.
> You'll be muttering bad things through your teeth whenever you are
> reduced to reverting to map-extents.

> But it is there.  And it does almost any job.

But this one it would do too slowly.

> -- 
> David Kastrup

-- 
Alan Mackenzie (Nuremberg, Germany).