From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: npostavs@users.sourceforge.net Newsgroups: gmane.emacs.bugs Subject: bug#24870: 26.0.50; parse-partial-sexp ignores comment-end Date: Thu, 29 Dec 2016 20:55:09 -0500 Message-ID: <87eg0q89ky.fsf@users.sourceforge.net> References: <87mvggh2hu.fsf@users.sourceforge.net> <87h967cj0f.fsf@users.sourceforge.net> <87eg1bcg0y.fsf@users.sourceforge.net> <20161214215834.GB3881@acm.fritz.box> <20161215164458.GA2437@acm.fritz.box> <8737hldce8.fsf@users.sourceforge.net> <20161229111428.GA12532@acm.fritz.box> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Trace: blaine.gmane.org 1483062922 15642 195.159.176.226 (30 Dec 2016 01:55:22 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Fri, 30 Dec 2016 01:55:22 +0000 (UTC) User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux) Cc: 24870@debbugs.gnu.org, Matt Armstrong To: Alan Mackenzie Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Fri Dec 30 02:55:15 2016 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cMmPb-0002pM-EJ for geb-bug-gnu-emacs@m.gmane.org; Fri, 30 Dec 2016 02:55:15 +0100 Original-Received: from localhost ([::1]:38029 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cMmPb-0003C1-K5 for geb-bug-gnu-emacs@m.gmane.org; Thu, 29 Dec 2016 20:55:15 -0500 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:58960) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cMmPS-0003Ad-8i for bug-gnu-emacs@gnu.org; Thu, 29 Dec 2016 20:55:08 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cMmPP-0006d9-09 for bug-gnu-emacs@gnu.org; Thu, 29 Dec 2016 20:55:06 -0500 Original-Received: from debbugs.gnu.org ([208.118.235.43]:44080) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1cMmPO-0006ck-Rz for bug-gnu-emacs@gnu.org; Thu, 29 Dec 2016 20:55:02 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1cMmPO-00057a-D7 for bug-gnu-emacs@gnu.org; Thu, 29 Dec 2016 20:55:02 -0500 X-Loop: help-debbugs@gnu.org Resent-From: npostavs@users.sourceforge.net Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Fri, 30 Dec 2016 01:55:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 24870 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: confirmed Original-Received: via spool by 24870-submit@debbugs.gnu.org id=B24870.148306286319626 (code B ref 24870); Fri, 30 Dec 2016 01:55:02 +0000 Original-Received: (at 24870) by debbugs.gnu.org; 30 Dec 2016 01:54:23 +0000 Original-Received: from localhost ([127.0.0.1]:59479 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cMmOj-00056T-1P for submit@debbugs.gnu.org; Thu, 29 Dec 2016 20:54:22 -0500 Original-Received: from mail-io0-f194.google.com ([209.85.223.194]:33490) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cMmOg-00056C-0v for 24870@debbugs.gnu.org; Thu, 29 Dec 2016 20:54:19 -0500 Original-Received: by mail-io0-f194.google.com with SMTP id j76so22801919ioe.0 for <24870@debbugs.gnu.org>; Thu, 29 Dec 2016 17:54:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:in-reply-to:references:user-agent:date :message-id:mime-version; bh=t9G2nUfHjXFE7MgHTu4+0L1dNH+z2/Vv1qAszOfHGdQ=; b=jUbZBXDMy+PN4+BkjIHOZSUlz51Brz+wX41kmMK9WBrqdIdYkXzmE1VZWb6hwCGMr5 A5U2e78ck/02Wic2ec07emrDPSm351L5i0xkyxh06Z6ddh771KNvYNIfLKwu873mKfrw ihTA90nZxuc8EcFtxBIaTWEqCWCg+nzAClqvaOx49ZesB8excY5lmHucu2vJcIRb8rb3 +dHUgrXptpphcYlGfd+ACWhleq81qnPLmetr4X4LZKGCbwYq6b0DIGWekg+GOHOnWZ4A pcWfdsVXqakVbWLMpoZCkaDsFyRobgvz9WvYyONZytC0yO78DvCKSeOve3JbRQi/vqiB cvFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:in-reply-to:references :user-agent:date:message-id:mime-version; bh=t9G2nUfHjXFE7MgHTu4+0L1dNH+z2/Vv1qAszOfHGdQ=; b=EkC4tLfMMj5M47K36ZEQ5kx58Qm7InR01E69fhBqkDen7QzGqtpkCbfOn85pzkT4Ll AwjnplLcayUqC78hWSjrlaxENS3WckKs4UJNbzOHn2DDgpUTnZyLMBfFdiuCNYFecdd/ rDVWQ8CP3VVi9ODnYB5alVMHbSLHIrw2pVeEJ/RPxFwGDrWeSdd8VcsS8wUJiyIP/jPr A01WC8SUx2wMtR2aPCu7Y3ap+JrMoIiFsIExzZ0Lku6cB0+pgh9ddPaGQ9+yY9vAZmpH lopLgJCC0WatbkG1EuYPFWpyXmLwA14Jd7vpjdS7J2IH47X6ixtN26/yy6TNIP6wok2V bsfQ== X-Gm-Message-State: AIkVDXKYP+2tprRKcOaYDEvyyNA8jwoprmyADNVKGuGw3scQiNiObK1m98cZBtIjcFiIkw== X-Received: by 10.107.62.69 with SMTP id l66mr40244011ioa.213.1483062852071; Thu, 29 Dec 2016 17:54:12 -0800 (PST) Original-Received: from zony ([45.2.7.65]) by smtp.googlemail.com with ESMTPSA id 9sm27043606itv.0.2016.12.29.17.54.10 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 29 Dec 2016 17:54:11 -0800 (PST) In-Reply-To: <20161229111428.GA12532@acm.fritz.box> (Alan Mackenzie's message of "Thu, 29 Dec 2016 11:14:28 +0000") X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 208.118.235.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.org gmane.emacs.bugs:127566 Archived-At: --=-=-= Content-Type: text/plain Alan Mackenzie writes: > > (i) The new function `check_comment_start' doesn't have a comment saying > what its return value means. Possibly you could instead rename it so > that the name implies what it returns. Maybe something like > `in_double_comment_opener'. Good point, I've updated the patch. > > I'll admit I haven't actually tried out the code, mainly because you've > written a test file. Oh dear, maybe I should have withheld the test file ;) > > (ii) In `parse-partial-sexp-continue-over-comment-marker', variable aftC > is the position in the middle of the comment closer "*/". I don't think > you are testing in any way that element 10 (nil, or the syntax of the > position just before the end point when that position might be the first > character of a two-character construct, i.e. an escape or first char of a > double-char comment delimiter) is correct. My idea was that its effect would be tested by using pps-preC as OLDSTATE, which avoids having to encode the specifics in the test. I added another clause which uses pps-aftC to cover parsing from the middle of a comment closer as well as opener. --=-=-= Content-Type: text/plain Content-Disposition: attachment; filename=v2-0001-Fix-comment-detection-on-open-parens.patch Content-Description: patch v2 >From 50773c5194591526797eddf70a3cce5f4ccf6f25 Mon Sep 17 00:00:00 2001 From: Noam Postavsky Date: Sun, 18 Dec 2016 00:00:30 -0500 Subject: [PATCH v2] Fix comment detection on open parens Characters having both open paren syntax and comment start syntax were being detected as open parens even when they should have been part a comment starter (Bug#24870). * src/syntax.c (in_2char_comment_start): New function, extracted from `scan_sexps_forward'. (scan_sexps_forward): Add check for a 2-char comment starter before the loop. Inside the loop, do that check after incrementing the 'from' character index. Move the single char comment syntax cases into the switch instead of special casing them before. * test/src/syntax-tests.el (parse-partial-sexp-paren-comments): (parse-partial-sexp-continue-over-comment-marker): New tests. --- src/syntax.c | 117 +++++++++++++++++++++++++---------------------- test/src/syntax-tests.el | 85 ++++++++++++++++++++++++++++++++++ 2 files changed, 147 insertions(+), 55 deletions(-) create mode 100644 test/src/syntax-tests.el diff --git a/src/syntax.c b/src/syntax.c index 7c15e77..d291890 100644 --- a/src/syntax.c +++ b/src/syntax.c @@ -3092,6 +3092,36 @@ DEFUN ("backward-prefix-chars", Fbackward_prefix_chars, Sbackward_prefix_chars, return Qnil; } + +/* If the character at FROM_BYTE is the second part of a 2-character + comment opener based on PREV_FROM_SYNTAX, update STATE and return + true. */ +static bool +in_2char_comment_start (struct lisp_parse_state *state, + int prev_from_syntax, + ptrdiff_t prev_from, + ptrdiff_t from_byte) +{ + int c1, syntax; + if (SYNTAX_FLAGS_COMSTART_FIRST (prev_from_syntax) + && (c1 = FETCH_CHAR_AS_MULTIBYTE (from_byte), + syntax = SYNTAX_WITH_FLAGS (c1), + SYNTAX_FLAGS_COMSTART_SECOND (syntax))) + { + /* Record the comment style we have entered so that only + the comment-end sequence of the same style actually + terminates the comment section. */ + state->comstyle + = SYNTAX_FLAGS_COMMENT_STYLE (syntax, prev_from_syntax); + bool comnested = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax) + | SYNTAX_FLAGS_COMMENT_NESTED (syntax)); + state->incomment = comnested ? 1 : -1; + state->comstr_start = prev_from; + return true; + } + return false; +} + /* Parse forward from FROM / FROM_BYTE to END, assuming that FROM has state STATE, and return a description of the state of the parse at END. @@ -3107,8 +3137,6 @@ scan_sexps_forward (struct lisp_parse_state *state, int commentstop) { enum syntaxcode code; - int c1; - bool comnested; struct level { ptrdiff_t last, prev; }; struct level levelstart[100]; struct level *curlevel = levelstart; @@ -3122,7 +3150,6 @@ scan_sexps_forward (struct lisp_parse_state *state, ptrdiff_t prev_from; /* Keep one character before FROM. */ ptrdiff_t prev_from_byte; int prev_from_syntax, prev_prev_from_syntax; - int syntax; bool boundary_stop = commentstop == -1; bool nofence; bool found; @@ -3187,53 +3214,31 @@ scan_sexps_forward (struct lisp_parse_state *state, } else if (start_quoted) goto startquoted; + else if ((from < end) + && (in_2char_comment_start (state, prev_from_syntax, + prev_from, from_byte))) + { + INC_FROM; + prev_from_syntax = Smax; /* the syntax has already been "used up". */ + goto atcomment; + } while (from < end) { - if (SYNTAX_FLAGS_COMSTART_FIRST (prev_from_syntax) - && (c1 = FETCH_CHAR (from_byte), - syntax = SYNTAX_WITH_FLAGS (c1), - SYNTAX_FLAGS_COMSTART_SECOND (syntax))) - { - /* Record the comment style we have entered so that only - the comment-end sequence of the same style actually - terminates the comment section. */ - state->comstyle - = SYNTAX_FLAGS_COMMENT_STYLE (syntax, prev_from_syntax); - comnested = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax) - | SYNTAX_FLAGS_COMMENT_NESTED (syntax)); - state->incomment = comnested ? 1 : -1; - state->comstr_start = prev_from; - INC_FROM; - prev_from_syntax = Smax; /* the syntax has already been - "used up". */ - code = Scomment; - } - else + INC_FROM; + + if ((from < end) + && (in_2char_comment_start (state, prev_from_syntax, + prev_from, from_byte))) { INC_FROM; - code = prev_from_syntax & 0xff; - if (code == Scomment_fence) - { - /* Record the comment style we have entered so that only - the comment-end sequence of the same style actually - terminates the comment section. */ - state->comstyle = ST_COMMENT_STYLE; - state->incomment = -1; - state->comstr_start = prev_from; - code = Scomment; - } - else if (code == Scomment) - { - state->comstyle = SYNTAX_FLAGS_COMMENT_STYLE (prev_from_syntax, 0); - state->incomment = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax) ? - 1 : -1); - state->comstr_start = prev_from; - } + prev_from_syntax = Smax; /* the syntax has already been "used up". */ + goto atcomment; } if (SYNTAX_FLAGS_PREFIX (prev_from_syntax)) continue; + code = prev_from_syntax & 0xff; switch (code) { case Sescape: @@ -3252,24 +3257,15 @@ scan_sexps_forward (struct lisp_parse_state *state, symstarted: while (from < end) { - int symchar = FETCH_CHAR_AS_MULTIBYTE (from_byte); - - if (SYNTAX_FLAGS_COMSTART_FIRST (prev_from_syntax) - && (syntax = SYNTAX_WITH_FLAGS (symchar), - SYNTAX_FLAGS_COMSTART_SECOND (syntax))) + if (in_2char_comment_start (state, prev_from_syntax, + prev_from, from_byte)) { - state->comstyle - = SYNTAX_FLAGS_COMMENT_STYLE (syntax, prev_from_syntax); - comnested = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax) - | SYNTAX_FLAGS_COMMENT_NESTED (syntax)); - state->incomment = comnested ? 1 : -1; - state->comstr_start = prev_from; INC_FROM; - prev_from_syntax = Smax; - code = Scomment; + prev_from_syntax = Smax; /* the syntax has already been "used up". */ goto atcomment; } + int symchar = FETCH_CHAR_AS_MULTIBYTE (from_byte); switch (SYNTAX (symchar)) { case Scharquote: @@ -3290,8 +3286,19 @@ scan_sexps_forward (struct lisp_parse_state *state, curlevel->prev = curlevel->last; break; - case Scomment_fence: /* Can't happen because it's handled above. */ + case Scomment_fence: + /* Record the comment style we have entered so that only + the comment-end sequence of the same style actually + terminates the comment section. */ + state->comstyle = ST_COMMENT_STYLE; + state->incomment = -1; + state->comstr_start = prev_from; + goto atcomment; case Scomment: + state->comstyle = SYNTAX_FLAGS_COMMENT_STYLE (prev_from_syntax, 0); + state->incomment = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax) ? + 1 : -1); + state->comstr_start = prev_from; atcomment: if (commentstop || boundary_stop) goto done; startincomment: diff --git a/test/src/syntax-tests.el b/test/src/syntax-tests.el new file mode 100644 index 0000000..776ce5b --- /dev/null +++ b/test/src/syntax-tests.el @@ -0,0 +1,85 @@ +;;; syntax-tests.el --- tests for syntax.c functions -*- lexical-binding: t -*- + +;; Copyright (C) 2016 Free Software Foundation, Inc. + +;; This file is part of GNU Emacs. + +;; GNU Emacs is free software: you can redistribute it and/or modify +;; it under the terms of the GNU General Public License as published by +;; the Free Software Foundation, either version 3 of the License, or +;; (at your option) any later version. + +;; GNU Emacs is distributed in the hope that it will be useful, +;; but WITHOUT ANY WARRANTY; without even the implied warranty of +;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;; GNU General Public License for more details. + +;; You should have received a copy of the GNU General Public License +;; along with GNU Emacs. If not, see . + +;;; Code: + +(require 'ert) + +(ert-deftest parse-partial-sexp-continue-over-comment-marker () + "Continue a parse that stopped in the middle of a comment marker." + (with-temp-buffer + (let ((table (make-syntax-table))) + (modify-syntax-entry ?/ ". 124") + (modify-syntax-entry ?* ". 23b") + (set-syntax-table table)) + (insert "/*C*/\nX") + (goto-char (point-min)) + (let* ((pointC (progn (search-forward "C") (1- (point)))) + (preC (1- pointC)) + (pointX (progn (search-forward "X") (1- (point)))) + (aftC (+ 2 pointC)) + (ppsC (parse-partial-sexp (point-min) pointC)) + (pps-preC (parse-partial-sexp (point-min) preC)) + (pps-aftC (parse-partial-sexp (point-min) aftC)) + (ppsX (parse-partial-sexp (point-min) pointX))) + ;; C should be inside comment. + (should (= (nth 0 ppsC) 0)) + (should (eq (nth 4 ppsC) t)) + (should (= (nth 8 ppsC) (- pointC 2))) + ;; X should not be in comment or list. + (should (= (nth 0 ppsX) 0)) + (should-not (nth 4 ppsX)) + ;; Try using OLDSTATE. + (should (equal (parse-partial-sexp preC pointC nil nil pps-preC) + ppsC)) + (should (equal (parse-partial-sexp pointC aftC nil nil ppsC) + pps-aftC)) + (should (equal (parse-partial-sexp preC aftC nil nil pps-preC) + pps-aftC)) + (should (equal (parse-partial-sexp aftC pointX nil nil pps-aftC) + ppsX))))) + +(ert-deftest parse-partial-sexp-paren-comments () + "Test syntax parsing with paren comment markers. +Specifically, where the first character of the comment marker is +also has open paren syntax (see Bug#24870)." + (with-temp-buffer + (let ((table (make-syntax-table))) + (modify-syntax-entry ?\{ "(}1nb" table) + (modify-syntax-entry ?\} "){4nb" table) + (modify-syntax-entry ?- ". 123" table) + (set-syntax-table table)) + (insert "{-C-}\nX") + (goto-char (point-min)) + (let* ((pointC (progn (search-forward "C") (1- (point)))) + (pointX (progn (search-forward "X") (1- (point)))) + (ppsC (parse-partial-sexp (point-min) pointC)) + (ppsX (parse-partial-sexp (point-min) pointX))) + ;; C should be inside nestable comment, not list. + (should (= (nth 0 ppsC) 0)) + (should (= (nth 4 ppsC) 1)) + (should (= (nth 8 ppsC) (- pointC 2))) + ;; X should not be in comment or list. + (should (= (nth 0 ppsX) 0)) + (should-not (nth 4 ppsX)) + ;; Try using OLDSTATE. + (should (equal (parse-partial-sexp pointC pointX nil nil ppsC) + ppsX))))) + +;;; syntax-tests.el ends here -- 2.9.3 --=-=-=--