Help etags parse lisp.j

Help etags parse lisp.j

[Eli Zaretskii]

Today I found that some inline functions defined in lisp.h are not in
src/TAGS.  For example, lispstpcpy is not there.  After some digging,
it turned out that lines like the one below confuse etags:

  extern _Noreturn void emacs_abort (void) NO_INLINE;

Specifically, the very next inline function doesn't wind up in TAGS.
The problem is that NO_INLINE part after the argument list.  These
macros expand either to __attribute__((SOMETHING)) or to nothing;
etags knows about __attribute__, but it cannot know about the macros
we use for that.  The problem happens with any attribute we hide
behind a macro, not just with NO_INLINE.

I could think about several possible solutions, but they are all ugly.
One is the patch shown below (if acceptable, it will have to be
augmented by a comment explaining the kludge).  Another one is to move
all the inline functions to a separate header, say inlines.h, and
include it in lisp.h near its end.  The disadvantage is that it will
be harder to understand the code of those inline functions because
their context (support declarations and macros) will be elsewhere.
Same thing if we just move them to the end of lisp.h.

Yet another idea would be to explicitly expand these macros in lisp.h,
so, for example, the line above becomes

  #if __GNUC__ >= 3
  extern _Noreturn void emacs_abort (void) __attribute__((noinline));
  #else
  extern _Noreturn void emacs_abort (void);
  #endif

Not too pretty, either.

We could also teach etags about the macros we use that expand into
__attribute__, but that would be fragile.

Running C files through cpp will solve this particular problem, but
all the cpp macros will disappear from TAGS, which I think is much
worse.

Any other ideas?

--- src/lisp.h~	2016-03-20 08:36:00.925638700 +0200
+++ src/lisp.h	2016-03-20 19:10:02.914122700 +0200
@@ -3428,6 +3428,7 @@
 /* Defined in fns.c.  */
 enum { NEXT_ALMOST_PRIME_LIMIT = 11 };
 extern EMACS_INT next_almost_prime (EMACS_INT) ATTRIBUTE_CONST;
+enum { dummy1 = 1 };
 extern Lisp_Object larger_vector (Lisp_Object, ptrdiff_t, ptrdiff_t);
 extern void sweep_weak_hash_tables (void);
 EMACS_UINT hash_string (char const *, ptrdiff_t);
@@ -3542,6 +3543,7 @@
 extern Lisp_Object current_message (void);
 extern void clear_message (bool, bool);
 extern void message (const char *, ...) ATTRIBUTE_FORMAT_PRINTF (1, 2);
+enum { dummy2 = 1 };
 extern void message1 (const char *);
 extern void message1_nolog (const char *);
 extern void message3 (Lisp_Object);
@@ -3632,6 +3634,7 @@
 extern Lisp_Object make_string (const char *, ptrdiff_t);
 extern Lisp_Object make_formatted_string (char *, const char *, ...)
   ATTRIBUTE_FORMAT_PRINTF (2, 3);
+enum { dummy3 = 1 };
 extern Lisp_Object make_unibyte_string (const char *, ptrdiff_t);
 
 /* Make unibyte string from C string when the length isn't known.  */
@@ -3760,6 +3763,7 @@
 extern void *r_alloc (void **, size_t) ATTRIBUTE_ALLOC_SIZE ((2));
 extern void r_alloc_free (void **);
 extern void *r_re_alloc (void **, size_t) ATTRIBUTE_ALLOC_SIZE ((2));
+enum { dummy4 = 1 };
 extern void r_alloc_reset_variable (void **, void **);
 extern void r_alloc_inhibit_buffer_relocation (int);
 #endif
@@ -3782,6 +3786,7 @@
 /* Defined in print.c.  */
 extern Lisp_Object Vprin1_to_string_buffer;
 extern void debug_print (Lisp_Object) EXTERNALLY_VISIBLE;
+enum { dummy5 = 1 };
 extern void temp_output_buffer_setup (const char *);
 extern int print_level;
 extern void write_string (const char *);
@@ -3805,6 +3810,7 @@
 extern ptrdiff_t evxprintf (char **, ptrdiff_t *, char const *, ptrdiff_t,
 			    char const *, va_list)
   ATTRIBUTE_FORMAT_PRINTF (5, 0);
+enum { dummy6 = 1 };
 
 /* Defined in lread.c.  */
 extern Lisp_Object check_obarray (Lisp_Object);
@@ -3900,6 +3906,7 @@
 extern _Noreturn void error (const char *, ...) ATTRIBUTE_FORMAT_PRINTF (1, 2);
 extern _Noreturn void verror (const char *, va_list)
   ATTRIBUTE_FORMAT_PRINTF (1, 0);
+enum { dummy7 = 1 };
 extern void un_autoload (Lisp_Object);
 extern Lisp_Object call_debugger (Lisp_Object arg);
 extern void *near_C_stack_top (void);
@@ -4255,6 +4262,7 @@
 extern void init_random (void);
 extern void emacs_backtrace (int);
 extern _Noreturn void emacs_abort (void) NO_INLINE;
+enum { dummy8 = 1 };
 extern int emacs_open (const char *, int, int);
 extern int emacs_pipe (int[2]);
 extern int emacs_close (int);
@@ -4292,6 +4300,7 @@
 extern void syms_of_term (void);
 extern _Noreturn void fatal (const char *msgid, ...)
   ATTRIBUTE_FORMAT_PRINTF (1, 2);
+enum { dummy9 = 1 };
 
 /* Defined in terminal.c.  */
 extern void syms_of_terminal (void);
@@ -4400,6 +4409,7 @@
 
 extern char *xstrdup (const char *) ATTRIBUTE_MALLOC;
 extern char *xlispstrdup (Lisp_Object) ATTRIBUTE_MALLOC;
+enum { dummy10 = 1 };
 extern void dupstring (char **, char const *);
 
 /* Make DEST a copy of STRING's data.  Return a pointer to DEST's terminating
@@ -4445,6 +4455,7 @@
 enum MAX_ALLOCA { MAX_ALLOCA = 16 * 1024 };
 
 extern void *record_xmalloc (size_t) ATTRIBUTE_ALLOC_SIZE ((1));
+enum { dummy11 = 1 };
 
 #define USE_SAFE_ALLOCA			\
   ptrdiff_t sa_avail = MAX_ALLOCA;	\

Re: Help etags parse lisp.j

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> Today I found that some inline functions defined in lisp.h are not in
> src/TAGS.  For example, lispstpcpy is not there.  After some digging,
> it turned out that lines like the one below confuse etags:
>
>   extern _Noreturn void emacs_abort (void) NO_INLINE;
>
> Specifically, the very next inline function doesn't wind up in TAGS.
> The problem is that NO_INLINE part after the argument list.  These
> macros expand either to __attribute__((SOMETHING)) or to nothing;
> etags knows about __attribute__, but it cannot know about the macros
> we use for that.  The problem happens with any attribute we hide
> behind a macro, not just with NO_INLINE.

Can't etags reset its parser state after a toplevel semicolon?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Help etags parse lisp.j

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 20 Mar 2016 18:34:05 +0100
> 
> >   extern _Noreturn void emacs_abort (void) NO_INLINE;
> >
> > Specifically, the very next inline function doesn't wind up in TAGS.
> > The problem is that NO_INLINE part after the argument list.  These
> > macros expand either to __attribute__((SOMETHING)) or to nothing;
> > etags knows about __attribute__, but it cannot know about the macros
> > we use for that.  The problem happens with any attribute we hide
> > behind a macro, not just with NO_INLINE.
> 
> Can't etags reset its parser state after a toplevel semicolon?

No, evidently because it wants to support K&R function definition:

  void foo () int arg; { bar = arg; }

That's why, whenever it finds lines like the one above, it resets its
state only after the closing brace.

Hmm... can a function definition use mixed K&R and ANSI style, i.e.
some arguments declared inside the parentheses and some outside, like
this:

  void foo (int arg1) int arg2; { bar = baz; }

Or is this invalid C?  If so, perhaps we could take note of nonempty
argument list.

Re: Help etags parse lisp.h

[Eli Zaretskii]

> Date: Sun, 20 Mar 2016 19:59:28 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> No, evidently because it wants to support K&R function definition:
> 
>   void foo () int arg; { bar = arg; }
> 
> That's why, whenever it finds lines like the one above, it resets its
> state only after the closing brace.
> 
> Hmm... can a function definition use mixed K&R and ANSI style, i.e.
> some arguments declared inside the parentheses and some outside, like
> this:
> 
>   void foo (int arg1) int arg2; { bar = baz; }
> 
> Or is this invalid C?  If so, perhaps we could take note of nonempty
> argument list.

Actually, the K&R style is this:

  void foo (arg1, arg2) TYPE1 arg1; TYPE2 arg2; { body... }

So the only way to discern between this and what we have in lisp.h is
to find data type keywords inside the parentheses (hoping they are not
hidden by some other macro).

Re: Help etags parse lisp.j

[Paul Eggert]

Eli Zaretskii wrote:
>> >Can't etags reset its parser state after a toplevel semicolon?
> No, evidently because it wants to support K&R function definition:
>
>    void foo () int arg; { bar = arg; }

OK, how about if it resets its state when it sees a toplevel semicolon not 
immediately followed by "{"?

Re: Help etags parse lisp.j

[Andreas Schwab]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Eli Zaretskii wrote:
>>> >Can't etags reset its parser state after a toplevel semicolon?
>> No, evidently because it wants to support K&R function definition:
>>
>>    void foo () int arg; { bar = arg; }
>
> OK, how about if it resets its state when it sees a toplevel semicolon not
> immediately followed by "{"?

void foo (a, b) int a; int b; {}

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Help etags parse lisp.j

[Stefan Monnier]

> No, evidently because it wants to support K&R function definition:

How 'bout we add an option to etags whether or not to support
K&R definitions?

In my experience, supporting K&R is a nightmare with terrible
consequences, and nowadays it's sufficiently rarely used that it's OK to
relegate it to some optional behavior.


        Stefan

Re: Help etags parse lisp.j

[Eli Zaretskii]

> From: Andreas Schwab <schwab@suse.de>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Mon, 21 Mar 2016 11:58:14 +0100
> 
> Paul Eggert <eggert@cs.ucla.edu> writes:
> 
> > Eli Zaretskii wrote:
> >>> >Can't etags reset its parser state after a toplevel semicolon?
> >> No, evidently because it wants to support K&R function definition:
> >>
> >>    void foo () int arg; { bar = arg; }
> >
> > OK, how about if it resets its state when it sees a toplevel semicolon not
> > immediately followed by "{"?
> 
> void foo (a, b) int a; int b; {}

Right.

To expand on that, a K&R function definition generally looks like
this:

  TYPE0 foo (arg1, arg2, arg3, ..., argN)
    TYPE1 arg1;
    TYPE2 arg2;
    TYPE3 arg3;
    ...
    TYPEN argN;
  {
    body...
  }

Compare this with what we have in lisp.h:

  extern Lisp_Object make_formatted_string (char *, const char *, ...)
    ATTRIBUTE_FORMAT_PRINTF (2, 3);
  extern Lisp_Object make_unibyte_string (const char *, ptrdiff_t);
  SOMETHING;
  SOMETHING-ELSE;
  ...
  INLINE Lisp_Object
  build_unibyte_string (const char *str)
  {
    return make_unibyte_string (str, strlen (str));
  }

A tool that doesn't really parse C will have hard time discerning
between these two.  In particular, by the time etags finds a top-level
semicolon not followed by a "{", it's too late, because the first
inline function (in this case, build_unibyte_string) was already
missed.

Hmm... can we use the fact that in a K&R definition, the last token
before the opening brace "{" of the body is always a semicolon?  So if
there's no semi-colon there, then it's a function that needs to be
tagged?  (Of course, the semicolon could be hidden behind some clever
macro, but I think we don't need to cater to such uses.)

Re: Help etags parse lisp.j

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 21 Mar 2016 09:40:43 -0400
> 
> > No, evidently because it wants to support K&R function definition:
> 
> How 'bout we add an option to etags whether or not to support
> K&R definitions?

Thanks.  Unfortunately, this is IMO not better than the other
alternatives:

 . in a large enough project, the user doesn't have an easy way of
   knowing whether she needs K&R support
 . implementing such an option is not trivial, to say the least, given
   the complex state machines used by parsing of C-like languages by
   etags

Re: Help etags parse lisp.j

[Paul Eggert]

Eli Zaretskii wrote:
>   . in a large enough project, the user doesn't have an easy way of
>     knowing whether she needs K&R support

Yes they do. Any current project won't bother with K&R. The handful of older 
projects that still have K&R and haven't bothered to rip it out (e.g., glibc is 
*still* ripping it out) should have a toolsmith that knows ancient history. 
Really, K&R support should be turned off by default.

>  . implementing such an option is not trivial, to say the least, given
>    the complex state machines used by parsing of C-like languages

Nontrivial yes, but still it's not terribly hard.

In the short term, perhaps we could work around the problem by replacing this:

extern int make_ctrl_char (int) ATTRIBUTE_CONST;

with this:

extern ATTRIBUTE_CONST int make_ctrl_char (int);

and similarly for the other affected declarations.

Re: Help etags parse lisp.j

[Stefan Monnier]

>  . in a large enough project, the user doesn't have an easy way of
>    knowing whether she needs K&R support

I wouldn't worry about that.

>  . implementing such an option is not trivial, to say the least, given
>    the complex state machines used by parsing of C-like languages by
>    etags

Oh well,


        Stefan

Re: Help etags parse lisp.j

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Mon, 21 Mar 2016 10:40:20 -0700
> 
> Eli Zaretskii wrote:
> >   . in a large enough project, the user doesn't have an easy way of
> >     knowing whether she needs K&R support
> 
> Yes they do. Any current project won't bother with K&R. The handful of older 
> projects that still have K&R and haven't bothered to rip it out (e.g., glibc is 
> *still* ripping it out) should have a toolsmith that knows ancient history. 
> Really, K&R support should be turned off by default.

I don't want to turn it off, sorry.  Etags is part of the set of tools
that are invaluable when you are assigned with a task of making some
change in a large project that lay dormant for years -- they allow you
to quickly find your way in unfamiliar code.  It should support old
code styles.

> >  . implementing such an option is not trivial, to say the least, given
> >    the complex state machines used by parsing of C-like languages
> 
> Nontrivial yes, but still it's not terribly hard.
> 
> In the short term, perhaps we could work around the problem by replacing this:
> 
> extern int make_ctrl_char (int) ATTRIBUTE_CONST;
> 
> with this:
> 
> extern ATTRIBUTE_CONST int make_ctrl_char (int);
> 
> and similarly for the other affected declarations.

Let me first see if I can find a better solution.

Thanks.

Re: Help etags parse lisp.j

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> I don't want to turn it off, sorry. Etags is part of the set of tools that
> are invaluable when you are assigned with a task of making some change in a
> large project that lay dormant for years -- they allow you to quickly find
> your way in unfamiliar code. It should support old code styles.

Eli is very right on this point. Often, when Etags helps best is when you know
the least about the code being examined.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Help etags parse lisp.j

[Stefan Monnier]

> I don't want to turn it off, sorry.  Etags is part of the set of tools
> that are invaluable when you are assigned with a task of making some
> change in a large project that lay dormant for years -- they allow you
> to quickly find your way in unfamiliar code.  It should support old
> code styles.

The suggestion is not to drop K&R completely, but to have it as an
option (which could have other consequences such as failing to notice
our "extern foo () NO_RETURN;").


        Stefan

Re: Help etags parse lisp.j

[Stefan Monnier]

>   TYPE0 foo (arg1, arg2, arg3, ..., argN)

The first potential difference here is that each arg is
a single identifier.  Of course, in non-K&R declarations, this can also
be the case, but at least if there's a *, a [..], or two identifiers
without a comma between them, then we know already it's not K&R.

> Hmm... can we use the fact that in a K&R definition, the last token
> before the opening brace "{" of the body is always a semicolon?  So if
> there's no semi-colon there, then it's a function that needs to be
> tagged?  (Of course, the semicolon could be hidden behind some clever
> macro, but I think we don't need to cater to such uses.)

Sounds fine, yes,


        Stefan

Re: Help etags parse lisp.j

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 22 Mar 2016 08:46:59 -0400
> 
> >   TYPE0 foo (arg1, arg2, arg3, ..., argN)
> 
> The first potential difference here is that each arg is
> a single identifier.  Of course, in non-K&R declarations, this can also
> be the case, but at least if there's a *, a [..], or two identifiers
> without a comma between them, then we know already it's not K&R.

I thought about this, but concluded that it's too unreliable (as you
point out).

> > Hmm... can we use the fact that in a K&R definition, the last token
> > before the opening brace "{" of the body is always a semicolon?  So if
> > there's no semi-colon there, then it's a function that needs to be
> > tagged?  (Of course, the semicolon could be hidden behind some clever
> > macro, but I think we don't need to cater to such uses.)
> 
> Sounds fine, yes,

I will try to do something useful with this.