* CEDET merge question
@ 2009-09-05 16:28 Chong Yidong
2009-09-05 17:22 ` David Engster
` (2 more replies)
0 siblings, 3 replies; 29+ messages in thread
From: Chong Yidong @ 2009-09-05 16:28 UTC (permalink / raw
To: emacs-devel
I have a question about CEDET that hopefully someone on this list, who
has more experience using CEDET than me, can help answer (I've been
corresponding with Eric Ludlam, but he's gone on vacation).
The Semantic parser appears to have two major "back-ends", bovine and
wisent, which are used to generate Semantic tags. Does anyone know how
crucial these packages are, and whether one or the other (or both) be
dropped or somehow trimmed down?
I ask because the CEDET merge already involves an uncomfortably large
amount of code, and it's rather dismaying to see these two big code
trees "embedded" in subdirectories of Semantic. (Wisent, for instance,
appears to be an entire Elisp reimplementation of Bison...)
(The CVS branch I'm using for the CEDET merge is not yet suitable for
general testing; I'll inform the list when it's ready.)
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: CEDET merge question
2009-09-05 16:28 CEDET merge question Chong Yidong
@ 2009-09-05 17:22 ` David Engster
2009-09-05 20:53 ` Chong Yidong
2009-09-06 15:37 ` Richard Stallman
2009-09-08 8:11 ` joakim
2 siblings, 1 reply; 29+ messages in thread
From: David Engster @ 2009-09-05 17:22 UTC (permalink / raw
To: Chong Yidong; +Cc: emacs-devel
Chong Yidong <cyd@stupidchicken.com> writes:
> The Semantic parser appears to have two major "back-ends", bovine and
> wisent, which are used to generate Semantic tags. Does anyone know how
> crucial these packages are, and whether one or the other (or both) be
> dropped or somehow trimmed down?
I think it depends on the question if people should be able to edit and
compile the grammars itself, only using Emacs proper.
The bovine/wisent parsers and major-modes are crucial for development,
but I think they are not necessarily needed for the resulting parser; I
may be wrong though, especially when it comes to the Wisent parser,
which I'm not familiar with at all.
For example, the file semantic/bovine/c.by is the Bison grammar for
C/C++ parsing. During CEDET's make process, the 'bovine' code generates
the file semantic/bovine/semantic-c-by.el, which is the resulting C(++)
lexer in Emacs Lisp. This file is then required by
semantic-c.el. Therefore, I would think that including the resulting
semantic-c-by.el should be enough for the C parser to be working.
As mentioned above, there are also the major-modes for bison/wisent in
CEDET (bovine-grammar.el, wisent-grammar.el) which are needed for
writing and debugging the grammar files. I think those would also not
necessarily be needed in Emacs. However, if people would like to extend
or fix grammar files (or write new ones), they would then have to get
CEDET from CVS.
> (Wisent, for instance, appears to be an entire Elisp reimplementation
> of Bison...)
Yes, it is exactly that. :-)
Regards,
David
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: CEDET merge question
2009-09-05 17:22 ` David Engster
@ 2009-09-05 20:53 ` Chong Yidong
2009-09-05 23:08 ` David Engster
0 siblings, 1 reply; 29+ messages in thread
From: Chong Yidong @ 2009-09-05 20:53 UTC (permalink / raw
To: David Engster; +Cc: emacs-devel
David Engster <deng@randomsample.de> writes:
> The bovine/wisent parsers and major-modes are crucial for development,
> but I think they are not necessarily needed for the resulting parser; I
> may be wrong though, especially when it comes to the Wisent parser,
> which I'm not familiar with at all.
>
> For example, the file semantic/bovine/c.by is the Bison grammar for
> C/C++ parsing. During CEDET's make process, the 'bovine' code generates
> the file semantic/bovine/semantic-c-by.el, which is the resulting C(++)
> lexer in Emacs Lisp. This file is then required by
> semantic-c.el. Therefore, I would think that including the resulting
> semantic-c-by.el should be enough for the C parser to be working.
I see. I think it's better for us to merge just the generated Lisp
grammar files, leaving the grammar development for upstream. It's an
awful lot of infrastructure to pull in, considering that CEDET
development won't be carried out in our repository anyway.
Do you know if the bovine and wisent parsers are mutually replacable?
For instance, the default parser seems to be bovine; would it be a big
deal if we included just the bovine parser?
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: CEDET merge question
2009-09-05 20:53 ` Chong Yidong
@ 2009-09-05 23:08 ` David Engster
0 siblings, 0 replies; 29+ messages in thread
From: David Engster @ 2009-09-05 23:08 UTC (permalink / raw
To: Chong Yidong; +Cc: emacs-devel
Chong Yidong <cyd@stupidchicken.com> writes:
> Do you know if the bovine and wisent parsers are mutually replacable?
> For instance, the default parser seems to be bovine; would it be a big
> deal if we included just the bovine parser?
I don't think it makes much sense to include just the Bovine parser. If
you look at
http://cedet.sourceforge.net/languagesupport.shtml
you'll see the currently supported languages in CEDET, together with
their current status regarding completion, project support etc.. The
grammar column shows the type of grammar, "LL" or "LALR". The former is
done with Bovine/Bison, the latter with Wisent. So Bison isn't really
the default, but it's the older one, and especially the C/C++ support is
pretty stable by now (there's also a Wisent parser for C, but it doesn't
support C++ and AFAIK is currently not used). Some of the Wisent
grammars are in the contrib directory, which probably means they
basically work, but lack further infrastructure in Semantic.
But I think the Wisent grammars work pretty much the same as the Bison
ones, i.e., during CEDET's compilation a file 'wisent-<LANG>-wy.el' file
is created, which contains the actual parser.
-David
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: CEDET merge question
2009-09-05 16:28 CEDET merge question Chong Yidong
2009-09-05 17:22 ` David Engster
@ 2009-09-06 15:37 ` Richard Stallman
2009-09-06 17:46 ` Ken Raeburn
2009-09-08 8:11 ` joakim
2 siblings, 1 reply; 29+ messages in thread
From: Richard Stallman @ 2009-09-06 15:37 UTC (permalink / raw
To: Chong Yidong; +Cc: emacs-devel
I ask because the CEDET merge already involves an uncomfortably large
amount of code, and it's rather dismaying to see these two big code
trees "embedded" in subdirectories of Semantic. (Wisent, for instance,
appears to be an entire Elisp reimplementation of Bison...)
Is it possible to use Bison itself rather than implement the
same functionality differently? Or perhaps add an option
to Bison to output its data in whatever format is convenient?
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: CEDET merge question
2009-09-06 15:37 ` Richard Stallman
@ 2009-09-06 17:46 ` Ken Raeburn
2009-09-06 21:11 ` David Engster
2009-09-07 13:34 ` Richard Stallman
0 siblings, 2 replies; 29+ messages in thread
From: Ken Raeburn @ 2009-09-06 17:46 UTC (permalink / raw
To: rms; +Cc: Chong Yidong, emacs-devel
On Sep 6, 2009, at 11:37, Richard Stallman wrote:
> I ask because the CEDET merge already involves an uncomfortably
> large
> amount of code, and it's rather dismaying to see these two big code
> trees "embedded" in subdirectories of Semantic. (Wisent, for
> instance,
> appears to be an entire Elisp reimplementation of Bison...)
>
> Is it possible to use Bison itself rather than implement the
> same functionality differently? Or perhaps add an option
> to Bison to output its data in whatever format is convenient?
Guile is also using a translation/reimplementation of Bison in
Scheme. I haven't looked at the CEDET code, but Guile's version wants
the grammar input using Scheme (s-expression) syntax.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: CEDET merge question
2009-09-06 17:46 ` Ken Raeburn
@ 2009-09-06 21:11 ` David Engster
2009-09-06 22:26 ` Ken Raeburn
2009-09-07 13:33 ` Richard Stallman
2009-09-07 13:34 ` Richard Stallman
1 sibling, 2 replies; 29+ messages in thread
From: David Engster @ 2009-09-06 21:11 UTC (permalink / raw
To: Ken Raeburn; +Cc: Chong Yidong, rms, emacs-devel
Ken Raeburn <raeburn@raeburn.org> writes:
> On Sep 6, 2009, at 11:37, Richard Stallman wrote:
>> Is it possible to use Bison itself rather than implement the
>> same functionality differently? Or perhaps add an option
>> to Bison to output its data in whatever format is convenient?
>
> Guile is also using a translation/reimplementation of Bison in Scheme.
> I haven't looked at the CEDET code, but Guile's version wants the
> grammar input using Scheme (s-expression) syntax.
CEDET uses Bison grammars which are extended through "Optional Lambda
Expressions" (OLE). They produce the actual tags, which are the basic
objects resulting from the parsing stage. I don't think this can be
easily replaced by Bison itself or Guile.
But there's really not that much additional framework associated with
Bison/Bovine. In the 'bovine' subdirectory, there are the actual grammar
files (like c.by, erlang.by, etc.), and the major- and debugging-modes
(bovine-grammar.el, bovine-debug.el). I think they are really only
needed for developing and testing grammars.
The file semantic-bovine.el contains the parsing core and is
crucial. Then, there are files which deal with language-specific issues,
for example semantic-c.el, semantic-erlang.el, semantic-java.el,
etc.. These files contain overrides and helper functions to deal with
stuff which usually differs between languages, like smart completion,
local variables, namespaces and scoping issues, special preprocessor
macros, etc. These files are only crucial for parsing the named
language.
The file semantic-gcc.el sets up stuff like system include paths for
C/C++ by looking at the local gcc installation; it's very helpful for
people using gcc.
The files semantic-skel.el and skeleton.by are just there to get people
started developing their own grammars and overrides; I think they can be
safely dropped.
As mentioned in my previous mail, the files semantic-<LANG>-by.el result
from the compilation of the *.by files and could probably just be
provided 'as is' in Emacs, without the additional grammar developing
framework (this also implies that files could not just get synced from
CEDET CVS to Emacs, but would need a compilation step in between). I
can't speak for Eric here, of course. Maybe there's some not-so-obvious
dependency, or another good reason to include the full grammar
framework.
Regards,
David
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: CEDET merge question
2009-09-06 21:11 ` David Engster
@ 2009-09-06 22:26 ` Ken Raeburn
2009-09-07 13:33 ` Richard Stallman
1 sibling, 0 replies; 29+ messages in thread
From: Ken Raeburn @ 2009-09-06 22:26 UTC (permalink / raw
To: David Engster; +Cc: Chong Yidong, rms, emacs-devel
On Sep 6, 2009, at 17:11, David Engster wrote:
> Ken Raeburn <raeburn@raeburn.org> writes:
>> On Sep 6, 2009, at 11:37, Richard Stallman wrote:
>>> Is it possible to use Bison itself rather than implement the
>>> same functionality differently? Or perhaps add an option
>>> to Bison to output its data in whatever format is convenient?
>>
>> Guile is also using a translation/reimplementation of Bison in
>> Scheme.
>> I haven't looked at the CEDET code, but Guile's version wants the
>> grammar input using Scheme (s-expression) syntax.
>
> CEDET uses Bison grammars which are extended through "Optional Lambda
> Expressions" (OLE). They produce the actual tags, which are the basic
> objects resulting from the parsing stage. I don't think this can be
> easily replaced by Bison itself or Guile.
Sorry, I didn't mean to suggest replacing it with Guile, more that, if
the requirements were similar enough, Bison extensions to support both
CEDET and Guile might be possible. But if you're extending the
grammar with Lisp code, that may not be feasible....
Ken
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: CEDET merge question
2009-09-06 21:11 ` David Engster
2009-09-06 22:26 ` Ken Raeburn
@ 2009-09-07 13:33 ` Richard Stallman
2009-09-12 12:49 ` Eric M. Ludlam
1 sibling, 1 reply; 29+ messages in thread
From: Richard Stallman @ 2009-09-07 13:33 UTC (permalink / raw
To: David Engster; +Cc: cyd, raeburn, emacs-devel
CEDET uses Bison grammars which are extended through "Optional Lambda
Expressions" (OLE). They produce the actual tags, which are the basic
objects resulting from the parsing stage. I don't think this can be
easily replaced by Bison itself or Guile.
Why is it hard to add these to Bison?
It can handle embedded C code, so why not embedded Lisp code?
It should be straightforward to make such changes.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: CEDET merge question
2009-09-06 17:46 ` Ken Raeburn
2009-09-06 21:11 ` David Engster
@ 2009-09-07 13:34 ` Richard Stallman
1 sibling, 0 replies; 29+ messages in thread
From: Richard Stallman @ 2009-09-07 13:34 UTC (permalink / raw
To: Ken Raeburn; +Cc: cyd, emacs-devel
> Is it possible to use Bison itself rather than implement the
> same functionality differently? Or perhaps add an option
> to Bison to output its data in whatever format is convenient?
Guile is also using a translation/reimplementation of Bison in
Scheme.
That may be wasteful too, but it is a separate issue.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: CEDET merge question
2009-09-05 16:28 CEDET merge question Chong Yidong
2009-09-05 17:22 ` David Engster
2009-09-06 15:37 ` Richard Stallman
@ 2009-09-08 8:11 ` joakim
2009-09-08 9:07 ` Lennart Borgman
2009-09-08 14:41 ` Chong Yidong
2 siblings, 2 replies; 29+ messages in thread
From: joakim @ 2009-09-08 8:11 UTC (permalink / raw
To: Chong Yidong; +Cc: Tom Tromey, emacs-devel
Chong Yidong <cyd@stupidchicken.com> writes:
> I have a question about CEDET that hopefully someone on this list, who
> has more experience using CEDET than me, can help answer (I've been
> corresponding with Eric Ludlam, but he's gone on vacation).
>
> The Semantic parser appears to have two major "back-ends", bovine and
> wisent, which are used to generate Semantic tags. Does anyone know how
> crucial these packages are, and whether one or the other (or both) be
> dropped or somehow trimmed down?
>
> I ask because the CEDET merge already involves an uncomfortably large
> amount of code, and it's rather dismaying to see these two big code
> trees "embedded" in subdirectories of Semantic. (Wisent, for instance,
> appears to be an entire Elisp reimplementation of Bison...)
>
Emacs hackers would still need easy access to these tools. Maybe this is
a further case for including something like Tom Tromeys ELPA in Emacs?
If we had something like that by default it wouldnt be a big deal to
distribute tools like these to Emacs hackers.
>
> (The CVS branch I'm using for the CEDET merge is not yet suitable for
> general testing; I'll inform the list when it's ready.)
>
--
Joakim Verona
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: CEDET merge question
2009-09-08 8:11 ` joakim
@ 2009-09-08 9:07 ` Lennart Borgman
2009-09-08 9:09 ` Lennart Borgman
2009-09-08 14:41 ` Chong Yidong
1 sibling, 1 reply; 29+ messages in thread
From: Lennart Borgman @ 2009-09-08 9:07 UTC (permalink / raw
To: joakim; +Cc: Tom Tromey, Chong Yidong, emacs-devel
On Tue, Sep 8, 2009 at 10:11 AM, <joakim@verona.se> wrote:
>
> Emacs hackers would still need easy access to these tools. Maybe this is
> a further case for including something like Tom Tromeys ELPA in Emacs?
> If we had something like that by default it wouldnt be a big deal to
> distribute tools like these to Emacs hackers.
But does ELPA have the necessary version info structure?
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: CEDET merge question
2009-09-08 9:07 ` Lennart Borgman
@ 2009-09-08 9:09 ` Lennart Borgman
0 siblings, 0 replies; 29+ messages in thread
From: Lennart Borgman @ 2009-09-08 9:09 UTC (permalink / raw
To: joakim; +Cc: Tom Tromey, Chong Yidong, emacs-devel
On Tue, Sep 8, 2009 at 11:07 AM, Lennart
Borgman<lennart.borgman@gmail.com> wrote:
> On Tue, Sep 8, 2009 at 10:11 AM, <joakim@verona.se> wrote:
>>
>> Emacs hackers would still need easy access to these tools. Maybe this is
>> a further case for including something like Tom Tromeys ELPA in Emacs?
>> If we had something like that by default it wouldnt be a big deal to
>> distribute tools like these to Emacs hackers.
>
>
> But does ELPA have the necessary version info structure?
Eh, sorry. I mean wouldn't it be better to have a tool to install
directly from the repository where this part of CEDET is? I think it
would be rather easy to right such a tool which access the web
interface of the repository just for downloading.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: CEDET merge question
2009-09-08 8:11 ` joakim
2009-09-08 9:07 ` Lennart Borgman
@ 2009-09-08 14:41 ` Chong Yidong
2009-09-08 15:10 ` joakim
2009-09-08 21:21 ` Romain Francoise
1 sibling, 2 replies; 29+ messages in thread
From: Chong Yidong @ 2009-09-08 14:41 UTC (permalink / raw
To: joakim; +Cc: Tom Tromey, emacs-devel
joakim@verona.se writes:
> Emacs hackers would still need easy access to these tools. Maybe this is
> a further case for including something like Tom Tromeys ELPA in Emacs?
Eric's still going to develop CEDET in his repository, so if you'll be
hacking on CEDET, I think you should use the version of CEDET he has
installed, instead of the version that will eventually be bundled with
Emacs.
This is a practical matter: since CEDET is such a large and complicated
package, we shouldn't make changes directly to our copy of it, apart
from those that are necessary to adapt it to Emacs' conventions and
build system (which is what I've been working on). Instead, changes
should be applied first to Eric's repository, then merged back into our
tree.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: CEDET merge question
2009-09-08 14:41 ` Chong Yidong
@ 2009-09-08 15:10 ` joakim
2009-09-08 17:18 ` Chong Yidong
2009-09-08 21:21 ` Romain Francoise
1 sibling, 1 reply; 29+ messages in thread
From: joakim @ 2009-09-08 15:10 UTC (permalink / raw
To: Chong Yidong; +Cc: Tom Tromey, emacs-devel
Chong Yidong <cyd@stupidchicken.com> writes:
> joakim@verona.se writes:
>
>> Emacs hackers would still need easy access to these tools. Maybe this is
>> a further case for including something like Tom Tromeys ELPA in Emacs?
>
> Eric's still going to develop CEDET in his repository, so if you'll be
> hacking on CEDET, I think you should use the version of CEDET he has
> installed, instead of the version that will eventually be bundled with
> Emacs.
>
> This is a practical matter: since CEDET is such a large and complicated
> package, we shouldn't make changes directly to our copy of it, apart
> from those that are necessary to adapt it to Emacs' conventions and
> build system (which is what I've been working on). Instead, changes
> should be applied first to Eric's repository, then merged back into our
> tree.
I will manage. I was more thinking of newcommers to the project that
would like to contribute grammars for instance.
--
Joakim Verona
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: CEDET merge question
2009-09-08 15:10 ` joakim
@ 2009-09-08 17:18 ` Chong Yidong
0 siblings, 0 replies; 29+ messages in thread
From: Chong Yidong @ 2009-09-08 17:18 UTC (permalink / raw
To: joakim; +Cc: Tom Tromey, emacs-devel
joakim@verona.se writes:
>> This is a practical matter: since CEDET is such a large and complicated
>> package, we shouldn't make changes directly to our copy of it, apart
>> from those that are necessary to adapt it to Emacs' conventions and
>> build system (which is what I've been working on). Instead, changes
>> should be applied first to Eric's repository, then merged back into our
>> tree.
>
> I will manage. I was more thinking of newcommers to the project that
> would like to contribute grammars for instance.
I agree that it would be good to make it easier for newcomers to hack on
the parsing infrastructure. The ELP suggestion is a good one, but I
think it's a bit ambitious to implement it in the 23.2 timeframe.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: CEDET merge question
2009-09-08 14:41 ` Chong Yidong
2009-09-08 15:10 ` joakim
@ 2009-09-08 21:21 ` Romain Francoise
2009-09-08 22:27 ` Chong Yidong
1 sibling, 1 reply; 29+ messages in thread
From: Romain Francoise @ 2009-09-08 21:21 UTC (permalink / raw
To: Chong Yidong; +Cc: emacs-devel
Chong Yidong <cyd@stupidchicken.com> writes:
> This is a practical matter: since CEDET is such a large and
> complicated package, we shouldn't make changes directly to our
> copy of it, apart from those that are necessary to adapt it to
> Emacs' conventions and build system (which is what I've been
> working on). Instead, changes should be applied first to Eric's
> repository, then merged back into our tree.
If it's so large and complicated that we can't handle it like the
rest of Emacs, is it really a good idea to merge it in? I don't
think we have any other packages where the rule is "make the change
upstream first". That sounds like a liability to me.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: CEDET merge question
2009-09-08 21:21 ` Romain Francoise
@ 2009-09-08 22:27 ` Chong Yidong
0 siblings, 0 replies; 29+ messages in thread
From: Chong Yidong @ 2009-09-08 22:27 UTC (permalink / raw
To: Romain Francoise; +Cc: emacs-devel
Romain Francoise <romain@orebokech.com> writes:
> I don't think we have any other packages where the rule is "make the
> change upstream first". That sounds like a liability to me.
Actually, that's the situation for Org mode. If you want to *develop*
Org mode, I would encourage you to work on the upstream version, not the
version in Emacs.
This refers to development, not bug fixes (I apologize if my prior post
caused confusion). In the preceding discussion, Joakim was talking
about writing new Semantic grammars, i.e. development. Bugfixes can of
course be applied to the version in the Emacs repository. (Though
bugfixes should also be pushed upstream too, in most cases.)
In the future, we will want to integrate CEDET more deeply into Emacs.
When that time comes, we'll need a new arrangement. But for the time
being, I'd prefer to treat CEDET more than (say) Org mode than (say)
Calendar.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: CEDET merge question
2009-09-07 13:33 ` Richard Stallman
@ 2009-09-12 12:49 ` Eric M. Ludlam
2009-09-12 13:37 ` Miles Bader
` (3 more replies)
0 siblings, 4 replies; 29+ messages in thread
From: Eric M. Ludlam @ 2009-09-12 12:49 UTC (permalink / raw
To: rms; +Cc: cyd, raeburn, David Engster, emacs-devel
On Mon, 2009-09-07 at 09:33 -0400, Richard Stallman wrote:
> CEDET uses Bison grammars which are extended through "Optional Lambda
> Expressions" (OLE). They produce the actual tags, which are the basic
> objects resulting from the parsing stage. I don't think this can be
> easily replaced by Bison itself or Guile.
>
> Why is it hard to add these to Bison?
> It can handle embedded C code, so why not embedded Lisp code?
> It should be straightforward to make such changes.
I don't know how bison works, but I would assume that bison parses basic
C code (thus replacing $1 with some other piece of code.) In the same
way, it would need to be taught about Emacs Lisp, Scheme, or any other
language someone might want.
Bison also outputs the code needed for traversing the generated parser
table. When creating more than one parser in one application (ie - any
scripting language case) this would be detrimental since it is basically
the same code for every parser, which is wasteful.
That said, I do think that it is possible, and maybe even desirable to
do such a thing. The end result, however, would involve rather extreme
changes to bison, and possibly flex if flex is also used.
As others have pointed out, there are newer parser technologies
available too such as PEG. How much of that is fad vs fabulous, I don't
really know. What I do know is that the CEDET tools don't care much
about the specifics of the parser. The parser tools it does have are to
make it easy to create new parsers so Emacs can support a large number
of languages.
A very similar question to "why not make bison support Emacs Lisp
output", is "why not have gcc support tagging output".
If gcc supported a tagging output format with the details needed for
CEDET to get its job done, it could just call out to gcc instead of
parsing it in Emacs. CEDET would then magically support a lot more
languages.
There are a huge number of tools out there trying to do what gcc does,
like ctags, etags, ectags, cscope, gnu global, doxygen, and idutils.
What's worse is that none of them work well.
Of course, an Emacs Lisp parser can do lots of other things besides
create tags. That's just what it is currently used for.
Eric
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: CEDET merge question
2009-09-12 12:49 ` Eric M. Ludlam
@ 2009-09-12 13:37 ` Miles Bader
2009-09-13 16:39 ` Richard Stallman
2009-09-12 16:34 ` David Engster
` (2 subsequent siblings)
3 siblings, 1 reply; 29+ messages in thread
From: Miles Bader @ 2009-09-12 13:37 UTC (permalink / raw
To: eric; +Cc: cyd, raeburn, rms, David Engster, emacs-devel
"Eric M. Ludlam" <eric@siege-engine.com> writes:
> As others have pointed out, there are newer parser technologies
> available too such as PEG. How much of that is fad vs fabulous, I don't
> really know.
It's only anecdotal, but my experience with Lpeg is that it's a lot more
convenient and approachable than olde-style stuff like bison/flex, and
not obviously any less powerful.
In any case, it seems clear that some thought should be given before
putting any significant effort into bison/flex.
-Miles
--
Fast, small, soon; pick any 2.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: CEDET merge question
2009-09-12 12:49 ` Eric M. Ludlam
2009-09-12 13:37 ` Miles Bader
@ 2009-09-12 16:34 ` David Engster
2009-09-13 16:39 ` Richard Stallman
2009-09-13 16:40 ` Richard Stallman
3 siblings, 0 replies; 29+ messages in thread
From: David Engster @ 2009-09-12 16:34 UTC (permalink / raw
To: eric; +Cc: cyd, raeburn, rms, emacs-devel
Eric M. Ludlam <eric@siege-engine.com> writes:
> On Mon, 2009-09-07 at 09:33 -0400, Richard Stallman wrote:
>> CEDET uses Bison grammars which are extended through "Optional Lambda
>> Expressions" (OLE). They produce the actual tags, which are the basic
>> objects resulting from the parsing stage. I don't think this can be
>> easily replaced by Bison itself or Guile.
>>
>> Why is it hard to add these to Bison?
>> It can handle embedded C code, so why not embedded Lisp code?
>> It should be straightforward to make such changes.
[...]
> A very similar question to "why not make bison support Emacs Lisp
> output", is "why not have gcc support tagging output".
>
> If gcc supported a tagging output format with the details needed for
> CEDET to get its job done, it could just call out to gcc instead of
> parsing it in Emacs. CEDET would then magically support a lot more
> languages.
Yes, I think that would be the way to go. Some time ago, I looked at a
way to add Fortran 90/95 parsing to CEDET. It seems there's no free
Bison grammar out there, but there is for example g95-xml [1], which
apparently reuses the g95 parser and produces a XML output file, which
could then be converted to Emacs Lisp tags. Also, in gfortran, there is
a debug option '-fdump-parse-tree', which seems to produce an output
almost usable by Semantic (most importantly, it's missing any source
code information like line numbers, etc.).
Similar to g95-xml, there's gcc-xml [2], which uses gcc's C++ parser to
output a XML file. But it seems its development has stalled, and it
currently can't parse templates, for example.
One problem with this approach is how the parser reacts to 'code in
progress', meaning syntactically incorrect code which is, for example,
lacking some closing statements. I think that g95-xml just aborted in
this case, which is why I never went further with this project.
-David
[1] http://sourceforge.net/projects/g95-xml/
[2] http://www.gccxml.org/HTML/Index.html
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: CEDET merge question
2009-09-12 12:49 ` Eric M. Ludlam
2009-09-12 13:37 ` Miles Bader
2009-09-12 16:34 ` David Engster
@ 2009-09-13 16:39 ` Richard Stallman
2009-09-13 17:38 ` Eric M. Ludlam
2009-09-13 16:40 ` Richard Stallman
3 siblings, 1 reply; 29+ messages in thread
From: Richard Stallman @ 2009-09-13 16:39 UTC (permalink / raw
To: eric; +Cc: cyd, raeburn, deng, emacs-devel
A very similar question to "why not make bison support Emacs Lisp
output", is "why not have gcc support tagging output".
Could you please explain what you mean by that?
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: CEDET merge question
2009-09-12 13:37 ` Miles Bader
@ 2009-09-13 16:39 ` Richard Stallman
2009-09-14 11:22 ` tomas
0 siblings, 1 reply; 29+ messages in thread
From: Richard Stallman @ 2009-09-13 16:39 UTC (permalink / raw
To: Miles Bader; +Cc: cyd, raeburn, emacs-devel, deng, eric
What is Lpeg, and what does it do?
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: CEDET merge question
2009-09-12 12:49 ` Eric M. Ludlam
` (2 preceding siblings ...)
2009-09-13 16:39 ` Richard Stallman
@ 2009-09-13 16:40 ` Richard Stallman
3 siblings, 0 replies; 29+ messages in thread
From: Richard Stallman @ 2009-09-13 16:40 UTC (permalink / raw
To: eric; +Cc: cyd, raeburn, deng, emacs-devel
I don't know how bison works, but I would assume that bison parses basic
C code (thus replacing $1 with some other piece of code.) In the same
way, it would need to be taught about Emacs Lisp, Scheme, or any other
language someone might want.
Bison parses grammar definition files, which can contain segments of code.
Normally the syntax for a segment of code is {...}.
Bison generates tables for a parser, and puts the segments of code
into a function to do the parsing. Normally that function is written
in C.
However, using a different language and different syntax is just a
superficial change.
The end result, however, would involve rather extreme
changes to bison, and possibly flex if flex is also used.
Oh no. The complex parts of Bison would not be changed at all.
Only some of the parser and the output code. These are the parts that
are easy to understand, without even minimal knowledge of parsing.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: CEDET merge question
2009-09-13 16:39 ` Richard Stallman
@ 2009-09-13 17:38 ` Eric M. Ludlam
2009-09-14 18:28 ` Richard Stallman
0 siblings, 1 reply; 29+ messages in thread
From: Eric M. Ludlam @ 2009-09-13 17:38 UTC (permalink / raw
To: rms; +Cc: cyd, raeburn, deng, emacs-devel
On Sun, 2009-09-13 at 12:39 -0400, Richard Stallman wrote:
> A very similar question to "why not make bison support Emacs Lisp
> output", is "why not have gcc support tagging output".
>
> Could you please explain what you mean by that?
Sure.
Etags, ctags, gnu global, idutils and cscope all have parsers of some
sort that parse C and C++ code. Some use regexp matchers. Others have
primitive parsers.
gcc, of course, has a full language compliant parser which it uses to
compile code. I'm not a gcc expert, but I assume that as it parses, it
keeps track of the various symbols (functions, variables, namespaces,
etc) and where they are. (ie - debug info for gdb).
As such, it should be possible for gcc to easily output text
representing a tags file. Etags style would be fairly simple. The
output of exuberant ctags is more complex. The data needed by CEDET is
more complex still, but is still a subset of everything that gcc needs
to know.
For CEDET, if gcc saw this file:
-------------
int main(int argc, char *argv[]) {
}
-------------
it would be handy (for my application) for it to output:
--------------
(("main" function
(:arguments
(("argc" variable (:type "int") [ 11 20 ])
("argv" variable (:pointer 1 :dereference 1 :type "char")
[ 21 34] ))
:type "int")
[2 38]))
---------------
though, to be honest, any text output that is very regular would be
fine.
The part that makes this imperfect is that in Emacs, a file that needs
parsing may be in the middle of an edit. Handling these cases can be a
bit tricky for my simplified parser, and gcc doesn't have that editing
information available.
To handle this, the CEDET tools have different ways to parse files, such
as "on save", and can track when a file is unparsable and take alternate
actions when that happens.
Eric
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: CEDET merge question
2009-09-13 16:39 ` Richard Stallman
@ 2009-09-14 11:22 ` tomas
2009-09-14 12:15 ` Miles Bader
0 siblings, 1 reply; 29+ messages in thread
From: tomas @ 2009-09-14 11:22 UTC (permalink / raw
To: Richard Stallman; +Cc: deng, cyd, emacs-devel, raeburn, eric, Miles Bader
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Sun, Sep 13, 2009 at 12:39:44PM -0400, Richard Stallman wrote:
> What is Lpeg, and what does it do?
PEG stands for "Parsing Expression Grammars" and it is a grammar
notation which basically represents formally a recursive descent parser.
They are said to be a bit more powerful than context free grammars and
(usually) more expressive. The most salient point for us "old-timers"
is probably that the choices are "ordered" -- this has some price, but
we get someething for that: the distinction between lexer and parser
becomes more flexible. The relevant paper seems to be [1].
It seems that they are very nice to bind to a languag.
LPEG is the implementation of PEGs to be used in Lua.
[1] <http://pdos.csail.mit.edu/~baford/packrat/popl04/peg-popl04.pdf>
Regards
- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQFKridbBcgs9XrR2kYRAr7mAJ4wFQQd1aKLujMnAvlNST/TlibSUQCfTeCI
qOWOujkLZVNLsv+I8/vUlbM=
=88yG
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: CEDET merge question
2009-09-14 11:22 ` tomas
@ 2009-09-14 12:15 ` Miles Bader
2009-09-14 20:04 ` tomas
0 siblings, 1 reply; 29+ messages in thread
From: Miles Bader @ 2009-09-14 12:15 UTC (permalink / raw
To: tomas; +Cc: Richard Stallman, deng, cyd, emacs-devel, raeburn, eric
tomas@tuxteam.de writes:
> PEG stands for "Parsing Expression Grammars" and it is a grammar
> notation which basically represents formally a recursive descent parser.
>
> They are said to be a bit more powerful than context free grammars and
> (usually) more expressive. The most salient point for us "old-timers"
> is probably that the choices are "ordered" -- this has some price, but
> we get someething for that: the distinction between lexer and parser
> becomes more flexible. The relevant paper seems to be [1].
>
> LPEG is the implementation of PEGs to be used in Lua.
>
> [1] <http://pdos.csail.mit.edu/~baford/packrat/popl04/peg-popl04.pdf>
Note that while LPEG is a PEG parser, it's _not_ a packrat parser (as in
[1]); the packrat algorithm is just an implementation technique.
I've appended a copy of an a message I sent a while ago to emacs-devel
on the same subject (LPEG vs. packrat).
Note that I think it's not just the implementation technique which is
interesting about LPEG, but also the very nice manner in which it's
integrated with the language and made available for use. It's just an
amazingly powerful and handy tool. I recommend reading the LPEG web
page, where it gives a quick overview of it.
Since Lua is in many ways feels quite similar to lisp, I think an elisp
version would be similarly very natural and powerful. One difference
though -- for Lua, LPEG uses overloaded operators for building up
grammars; in elisp, it would probably be better to just use
s-expressions to represent grammars, using backquotes to embed
non-literal values.
[earlier message:]
You also might be interested in Roberto Ierusalimschy's paper on the
implemenation of LPEG, which is a PEG implementation for Lua:
http://www.inf.puc-rio.br/~roberto/docs/peg.pdf
Note that LPEG does _not_ use the packrat algorithm, as apparently it
presents some serious practical problems for common uses of parsing
tools:
In 2002, Ford proposed Packrat [5], an adaptation of the original
algorithm that uses lazy evaluation to avoid that inefficiency.
Even with this improvement, however, the space complexity of the
algorithm is still linear on the subject size (with a somewhat big
constant), even in the best case. As its author himself recognizes,
this makes the algorithm not befitting for parsing “large amounts of
relatively flat” data ([5], p. 57). However, unlike parsing tools,
regular-expression tools aim exactly at large amounts of relatively
flat data.
To avoid these difficulties, we did not use the Packrat algorithm
for LPEG. To implement LPEG we created a virtual parsing machine, not
unlike Knuth’s parsing machine [15], where each pattern is
represented as a program for the machine. The program is somewhat
similar to a recursive-descendent parser (with limited backtracking)
for the pattern, but it uses an explicit stack instead of recursion.
The general LPEG page is here:
http://www.inf.puc-rio.br/~roberto/lpeg/lpeg.html
-Miles
--
Back, n. That part of your friend which it is your privilege to contemplate in
your adversity.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: CEDET merge question
2009-09-13 17:38 ` Eric M. Ludlam
@ 2009-09-14 18:28 ` Richard Stallman
0 siblings, 0 replies; 29+ messages in thread
From: Richard Stallman @ 2009-09-14 18:28 UTC (permalink / raw
To: eric; +Cc: cyd, raeburn, deng, emacs-devel
Etags, ctags, gnu global, idutils and cscope all have parsers of some
sort that parse C and C++ code. Some use regexp matchers. Others have
primitive parsers.
gcc, of course, has a full language compliant parser which it uses to
compile code. I'm not a gcc expert, but I assume that as it parses, it
keeps track of the various symbols (functions, variables, namespaces,
etc) and where they are. (ie - debug info for gdb).
Now I know what you are talking about. This idea seems very
appealing, but it has a grave flaw. The flaw comes from the way GCC
handles input: it does preprocessing first, and real parsing operates
only on the output of preprocessing. So the output that GCC can
easily make would describe only the output of preprocessing.
Definitions and calls which are not actually compiled won't be seen at
all. Macros and references to them won't be seen at all.
What etags does now is much better, because it avoids that problem.
It is true that output from GCC would give more details about types,
etc., and would avoid getting confused in a few strange situations.
So there is indeed an advantage to generating the output from GCC.
But the disadvantage is much more important.
I designed a way to make GCC analyze and report on macros and on the
code that's not compiled in. That would get the best of both aspects.
But this is not a small job. Please don't ask me to write more
details unless you're prepared to do a substantial amount of work
and study the GCC parsing code carefully.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: CEDET merge question
2009-09-14 12:15 ` Miles Bader
@ 2009-09-14 20:04 ` tomas
0 siblings, 0 replies; 29+ messages in thread
From: tomas @ 2009-09-14 20:04 UTC (permalink / raw
To: Miles Bader
Cc: Richard Stallman, deng, cyd, emacs-devel, tomas, eric, raeburn
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Mon, Sep 14, 2009 at 09:15:02PM +0900, Miles Bader wrote:
[...]
> Note that while LPEG is a PEG parser, it's _not_ a packrat parser
> [...]
Thanks, Miles for the (as usually clearly expounded) insights.
Regards
- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQFKrqHUBcgs9XrR2kYRAhQOAJoDlVSzNGIa0TTaPK0tThYoKSW1bgCdFefi
VVADVlO9geSG8Vonhv9d/Jk=
=gAgk
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 29+ messages in thread
end of thread, other threads:[~2009-09-14 20:04 UTC | newest]
Thread overview: 29+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2009-09-05 16:28 CEDET merge question Chong Yidong
2009-09-05 17:22 ` David Engster
2009-09-05 20:53 ` Chong Yidong
2009-09-05 23:08 ` David Engster
2009-09-06 15:37 ` Richard Stallman
2009-09-06 17:46 ` Ken Raeburn
2009-09-06 21:11 ` David Engster
2009-09-06 22:26 ` Ken Raeburn
2009-09-07 13:33 ` Richard Stallman
2009-09-12 12:49 ` Eric M. Ludlam
2009-09-12 13:37 ` Miles Bader
2009-09-13 16:39 ` Richard Stallman
2009-09-14 11:22 ` tomas
2009-09-14 12:15 ` Miles Bader
2009-09-14 20:04 ` tomas
2009-09-12 16:34 ` David Engster
2009-09-13 16:39 ` Richard Stallman
2009-09-13 17:38 ` Eric M. Ludlam
2009-09-14 18:28 ` Richard Stallman
2009-09-13 16:40 ` Richard Stallman
2009-09-07 13:34 ` Richard Stallman
2009-09-08 8:11 ` joakim
2009-09-08 9:07 ` Lennart Borgman
2009-09-08 9:09 ` Lennart Borgman
2009-09-08 14:41 ` Chong Yidong
2009-09-08 15:10 ` joakim
2009-09-08 17:18 ` Chong Yidong
2009-09-08 21:21 ` Romain Francoise
2009-09-08 22:27 ` Chong Yidong
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/emacs.git
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.