From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Eric Ludlam Newsgroups: gmane.emacs.devel Subject: Re: Emacs contributions, C and Lisp Date: Fri, 09 Jan 2015 23:06:35 -0500 Message-ID: <54B0A54B.5080206@siege-engine.com> References: <83bnxuzyl4.fsf@gnu.org> <8761nusb90.fsf@uwakimon.sk.tsukuba.ac.jp> <87vbkovhh7.fsf@engster.org> <87387rvobr.fsf@engster.org> <874ms2tsw5.fsf@engster.org> <87r3v5snza.fsf@engster.org> <87mw5ts1d6.fsf@engster.org> <87d26ornxd.fsf@engster.org> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Trace: ger.gmane.org 1420862825 19857 80.91.229.3 (10 Jan 2015 04:07:05 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Sat, 10 Jan 2015 04:07:05 +0000 (UTC) Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org To: David Engster , Richard Stallman Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Sat Jan 10 05:06:59 2015 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1Y9nKD-00064i-BT for ged-emacs-devel@m.gmane.org; Sat, 10 Jan 2015 05:06:57 +0100 Original-Received: from localhost ([::1]:53073 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y9nKB-0005vF-Va for ged-emacs-devel@m.gmane.org; Fri, 09 Jan 2015 23:06:55 -0500 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:48567) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y9nJy-0005u6-6G for emacs-devel@gnu.org; Fri, 09 Jan 2015 23:06:43 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Y9nJu-00066H-WB for emacs-devel@gnu.org; Fri, 09 Jan 2015 23:06:42 -0500 Original-Received: from mail-qg0-f43.google.com ([209.85.192.43]:47032) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y9nJu-00066D-Qz; Fri, 09 Jan 2015 23:06:38 -0500 Original-Received: by mail-qg0-f43.google.com with SMTP id z107so11852011qgd.2; Fri, 09 Jan 2015 20:06:38 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=S7GoHt5EFnlXj2h9vCaBtqMNDSP6xq1CziXqUq7ZhPM=; b=abcxeewifhz2hQa2ZpdKLMtHhJDVrOqIjrl8xrWV2E94bW2zFS+dMS8Oxr5RRjEICF CWrGo9+e24pio3bWUOUhYmdxgBl6QY8oDtQL2FWhF7+lnHg9H59urMTGGL0pLuF9w6AE gsvtNW2y8dSCN5XWlcqaOtcOhoFsFzpA9r5/H7VuiBqvOz0xVcCB3c81OON026Q5qXV5 785yHnxiiSbpYeWTAqAWoDDzGUUmDHK5Dzt0idWBfQWT50mNzlVRQ2211k3MnSnWUDQC C+SiwjaPc6XMODtTNUhDkfDaHs4QYZpojqn8SvM3gwlkun/ryxzhoirm0DQiQ0cP+gy9 Wf9A== X-Received: by 10.140.93.139 with SMTP id d11mr30732636qge.101.1420862797865; Fri, 09 Jan 2015 20:06:37 -0800 (PST) Original-Received: from [192.168.1.202] (pool-108-20-224-24.bstnma.fios.verizon.net. [108.20.224.24]) by mx.google.com with ESMTPSA id p9sm8862810qax.8.2015.01.09.20.06.36 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 09 Jan 2015 20:06:36 -0800 (PST) User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 In-Reply-To: <87d26ornxd.fsf@engster.org> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.85.192.43 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:181121 Archived-At: On 01/09/2015 10:09 AM, David Engster wrote: > Richard Stallman writes: >> You and several others are trying to pressure me to decide to make >> GCC output the full AST. I have seen insults and harassment. > > Not from me, and I haven't seen anything like it on this thread. > >> This is not the way to convince me. It is the way to make me resent >> your behavior. > > I've no idea what I've done to earn your resentment. I think my behavior > was entirely reasonable, given that I've started with this only because > you asked to base our tooling efforts on GCC. Anyway, you don't have to > worry that I'll continue with this. This conversation seems unnecessarily final. Richard has valid concerns that need details, but since the AST (which I know almost nothing about) is so huge (based on parser's I've written), no matter how many details we may think of that are needed, someone else can think of a bit that could indeed be unnecessary. I wrote the first run for the "smart completion" engine that currently ships in Emacs, the parts of CEDET that includes EDE and Semantic. While I personally think it is pretty awesome, it really isn't hard to fool it which is where a lot of this GCC interest comes from. It took many years of my part-time work (and contributions from others like David) to assemble what is there now into a robust well tested system. The basic pieces of the system which is implemented in Emacs Lisp consists of a parser generator plus some parsers written in a bison-like syntax including a C++ parser. Due to limitations of Emacs' performance, only the parts of the language that handle definitions are implemented. (ie - tags for functions, variables, structures, etc.) The parser outputs a tag table with lots of details. Having a full parser generator and the parser is what makes this convenient to do. Hacks like etags, GNU Global, etc can't produce enough information for the next step. The next step is the completion engine. This is where regexp hacks exist to "parse" a statement like: i = foo.bar.substring which peels it apart into a variable "i", a notion of assignment, and ("foo" "bar" "substring") via several assumptions, such as that users don't write code like this: i /* some variable */ = /* equals */ foo. /* mystruct */ bar.substring The engine then goes and looks up i in reverse to see what it is. It then looks up foo in various tables that get built of known symbols, derives the data type, and thus members of foo. It iterates down through the "." symbols dereferencing each symbol by data type to get to the next step. This depends on the fact that most projects compile all their headers "the same way" so that tables parsed from some header included in this C file will have the same symbols when included in a different C file. With that background, there are a couple options for a GCC plugin. One option would be to have one plugin that outputs tags compatible with some standard. Naturally I suggest the one already in Emacs. A second plugin could be used to figure out all the state I mentioned earlier when looking up symbols, and provide completions directly (ie - a list of text strings to offer as completions.) That plugin would ONLY be used for completion, and all the internal logic couldn't be reused for another purpose in Emacs. The alternative is to dump out the AST into an Emacs friendly form, and write the above logic in Emacs instead. This is convenient because Emacs is easy to hack, and gcc plugins (based on what I've been reading) are really complicated. In terms of "get up and running quickly", dumping a big scary data structure out of a scary environment into a friendly easy to hack environment is a desirable path for us, and as Richard points out, for non-free software. I personally think that if there were a good way to bridge the gap so that gcc could directly output tags for the existing Semantic engine, then there is an incremental benefit of nearly perfect tag generation for the existing tool AND a performance boost. It won't solve the whole problem though. To solve the rest of it, we'd need a gcc plugin to parse a file up to a chosen point. For a file with 1000 lines of code and included headers, gcc needs the WHOLE AST to make sense of "the last line", or the part that needs the completion. This is because we can't guess at what isn't needed until you've actually processed it all. This is where the boundary between gcc and Emacs comes in. In theory, the GCC plugin could process the AST and output ONLY the completions, or ONLY whatever was asked for (local types, scope information, refactoring data or what not.) An alternative might be to output a subset of the AST for processing in Emacs that is local to the the completion area, and depend on our old Emacs code to do the type lookup, etc. This would improve the current completion, but still could be fooled based on the quality of the Emacs data which is now, by definition, incomplete. In the past (call it year 2000) people thought my smart completion was lame (ie - inaccurate) and slow compared to dynamic abbreviation completion where claims of "dabbrevs is good enough" were stated. This proposal could be "good enough" for this single feature. So, I've laid out some scenarios that are "not full AST" friendly. There are some benefits (performance), and tradeoffs (difficulty). Even so, we've only touched on one feature. There are lots of other features in the existing Semantic tool already in Emacs derived from having a parser built right into Emacs, such as highlighting code with syntax errors (but only the code for definitions, not the logic.) I have a long list of other things I'd love to do to such as redo font-lock with the many "hints" about what your code is doing that only the compiler could know, but can't because writing a parser from scratch is actually pretty hard and error prone regardless of doing so in Emacs. Many folks have touched on those features in a myriad of other thread replies in this mailing list. I've taken my best stab at some of them that seemed attainable, but feel I've gone as far as I can aside from some incremental improvements, or just adding new languages. I've been very thankful for David's help with the CEDET project, and the many improvements in our existing smart completion engine he's made. For myself, and I imagine David, having gone through that and persevered simulating a compiler for so long to try and get these features only to have dabbrev people scoff on one side and clang users sneer on the other is disheartening. The hope of having a "real compiler" to lean on could open so many doors for us we just can't get to right now it is hard not to be discouraged by non technical issues. I would hope that David, who is looking into the gcc plugin route, and Richard can find a reasonable compromise that enables Emacs to have data from gcc that would enabled our existing tools to grow in its accuracy, and would encourage contributions from folks who do not have the skills to hack gcc plugins create new features. I suspect that isn't possible until someone learns more about gcc's AST and thinks about what a good abstraction model for Emacs is, and how it could be applied to the existing pretty good smart completion system. Eric