From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Stefan Monnier Newsgroups: gmane.emacs.devel Subject: Re: access to parser stack in SMIE Date: Sat, 06 Oct 2012 08:37:40 -0400 Message-ID: References: <85pq4wgrho.fsf@member.fsf.org> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain X-Trace: ger.gmane.org 1349527068 5277 80.91.229.3 (6 Oct 2012 12:37:48 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Sat, 6 Oct 2012 12:37:48 +0000 (UTC) Cc: emacs-devel To: Stephen Leake Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Sat Oct 06 14:37:53 2012 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1TKTdf-0000dD-Vl for ged-emacs-devel@m.gmane.org; Sat, 06 Oct 2012 14:37:52 +0200 Original-Received: from localhost ([::1]:43603 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TKTda-0001A4-0a for ged-emacs-devel@m.gmane.org; Sat, 06 Oct 2012 08:37:46 -0400 Original-Received: from eggs.gnu.org ([208.118.235.92]:41830) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TKTdX-00019y-ED for emacs-devel@gnu.org; Sat, 06 Oct 2012 08:37:44 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TKTdW-0002st-A4 for emacs-devel@gnu.org; Sat, 06 Oct 2012 08:37:43 -0400 Original-Received: from ironport2-out.teksavvy.com ([206.248.154.182]:19447) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TKTdW-0002so-5j for emacs-devel@gnu.org; Sat, 06 Oct 2012 08:37:42 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Av0EAG6Zu09FxLQG/2dsb2JhbABEtBGBCIIVAQEEAVYjBQsLNBIUGA0kLoV3gXcFugmQRAOjM4FYgwWBOgkR X-IronPort-AV: E=Sophos;i="4.75,637,1330923600"; d="scan'208";a="200791663" Original-Received: from 69-196-180-6.dsl.teksavvy.com (HELO pastel.home) ([69.196.180.6]) by ironport2-out.teksavvy.com with ESMTP/TLS/ADH-AES256-SHA; 06 Oct 2012 08:37:41 -0400 Original-Received: by pastel.home (Postfix, from userid 20848) id CF9A5594D1; Sat, 6 Oct 2012 08:37:40 -0400 (EDT) In-Reply-To: <85pq4wgrho.fsf@member.fsf.org> (Stephen Leake's message of "Sat, 06 Oct 2012 00:21:39 -0400") User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.2.50 (gnu/linux) X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 206.248.154.182 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:154121 Archived-At: Your problem is one I also bumped into for the Modula-2 and Pascal modes (can't remember how I "solved" them nor to what extent the "solution" works). > When all the tokens are properly disambiguated, SMIE can traverse > correctly from package "begin" to "package". But we can't do that while > disambiguating "begin"; that's circular. Actually, in some cases, it can be made to work: to disambiguate "begin" setup a loop that calls smie-backward-sexp repeatedly (starting from the position just before the "begin", of course) checking after each call whether the result lets us decide which of the two begins we're dealing with. Depending on the particular language's grammar this may not work because "checking whether the result lets us decide" may not be possible. But the recursion does terminate at least. > The solution I found is to deliberately call `smie-forward-sexp' > starting at the beginning of the buffer. Right: the only way the parsing-stack can be used reliably is if we always parse from the beginning. > It might also make sense to incorporate the refined keyword cache > mechanism into smie. Right, if we want to make the stack visible, then we also need to implement the cache. Note that such a "forward full-parse with cache" approach has several downsides: - potential performance impact on long buffers. - risk of the cache going out of sync. - parse errors far away in an unrelated (earlier) part of the buffer can prevent proper local indentation. Parse errors can occur for lots of reasons (e.g. temporarily incorrect code, incomplete parser, or use of "exotic" language extension, use of a preprocessor, ...). That doesn't mean it's not workable, but those downsides had better come with some significant upside. One significant upside is that by only parsing forward we can use any other parsing technology, such as LALR, GLR, PEG, ... I.e. I think it's an interesting direction but it would be another package. Stefan