From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED.blaine.gmane.org!not-for-mail From: arthur miller Newsgroups: gmane.emacs.devel Subject: Re: Using incremental parsing in Emacs Date: Sat, 4 Jan 2020 13:30:18 +0000 Message-ID: References: <83blrkj1o1.fsf@gnu.org> <41b3e9a0-2866-4692-a35c-6d9541bc3aaa@Spark> <83lfqnha8a.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Injection-Info: blaine.gmane.org; posting-host="blaine.gmane.org:195.159.176.226"; logging-data="56771"; mail-complaints-to="usenet@blaine.gmane.org" Cc: HaiJun Zhang , "emacs-devel@gnu.org" To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Sat Jan 04 14:30:33 2020 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([209.51.188.17]) by blaine.gmane.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1injVh-000EaP-4H for ged-emacs-devel@m.gmane.org; Sat, 04 Jan 2020 14:30:33 +0100 Original-Received: from localhost ([::1]:34034 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1injVf-0003bi-Sp for ged-emacs-devel@m.gmane.org; Sat, 04 Jan 2020 08:30:31 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:57478) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1injVZ-0003at-8G for emacs-devel@gnu.org; Sat, 04 Jan 2020 08:30:26 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1injVX-00043a-1s for emacs-devel@gnu.org; Sat, 04 Jan 2020 08:30:24 -0500 Original-Received: from mail-oln040092074099.outbound.protection.outlook.com ([40.92.74.99]:37192 helo=EUR04-DB3-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1injVV-0003zR-3O; Sat, 04 Jan 2020 08:30:21 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UuboLH2v5O8+kGUAla279ueAlOcSN7tzMI5CmtslH+Gyx7JXYfK4pLYayghLStkRFy5Ud0ksj6++2z3errzsVnRDC1xtyvBZYlDzkuj8NDxuCk7I5+FFQ+8lOdoc0eUwyBvlHCTSk8jIlDMZsTJIIjWhyJ1sARi3yQEA1+yARmM05NToXRGeW9du/ZD8zp2+KkLQEWhcA83s8j65uJsfOH6BxcH7L4IGD2amT7vaPbqTCxG910n2/eYI5Oa7WpCf3juOmxNs7mEoEF/aktKMoirL0gK1JXumKeZfN1tINr/Prja/S70w4WIOCfZ1dtUnmshFE+4jPU3crr0+iTk+Sw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LJTcNQ7OM/Au6Qp0vy4ltNjGEqjr94MGmqRCAnPhsJo=; b=fv5E5w8QtygnBO3a6o4wWFnsghD2JqBRnaY4qPem+RrbL9CbhDwlYLYVinzW6HAprQTw1zIWwxwIETsnGPYqOZWFmlj9gugkVecwXLcdSx27pcyYoZgWqNdc4+FYdyRflo8XSFtQPI0SDXYn5VAi0viDGUOGFK2umUxOto9Zfm6A3A5B2jVqiL92i6FeNoWFaJbBvpPU8pQ9goZ/XpD6QdZkm23635RsdRrEUiLQi22mFqoanZTgyreBxRpUiXTOAMxxj8Tqy3a3oAN1CkEVHuKf/7na8jEu9Tves9NGGEmD7S9EjfMT0mRDGuiWSbcbOwuv96OGkRGLmT3vfp61pg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=live.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LJTcNQ7OM/Au6Qp0vy4ltNjGEqjr94MGmqRCAnPhsJo=; b=J2xzD48JQyAP28nSjpqzMNOa3DP061MzcRMNZ374ESRSG1ZfHSs3V1jMAG29ApHAP5pc3wb/1iW+PFcZmPx2hOUTOzzEGzbREDOxNglQKHjwQH0sCxMJhMmN61cajdLDZ5lOIEXjB+5D9PSDJS2rw3FrFax4A11bJbB884s1rQVeYctsggcimsFqrJhXr7+RI3XUsID6v/8RsXLYghBeDcyXatUfqERYNHBAH+6nDQhQRHYnKmQnerinx2/YjSpd1U6P6GyBCNNwYXceOOafJ50B49++0xXGXi61ktFodSQkuN8lduFrzt40jzxiAAjLHXqGSBI52FIw+c1fv+53lw== Original-Received: from DB3EUR04FT011.eop-eur04.prod.protection.outlook.com (10.152.24.59) by DB3EUR04HT042.eop-eur04.prod.protection.outlook.com (10.152.24.162) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2602.11; Sat, 4 Jan 2020 13:30:18 +0000 Original-Received: from VI1P194MB0429.EURP194.PROD.OUTLOOK.COM (10.152.24.59) by DB3EUR04FT011.mail.protection.outlook.com (10.152.25.20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2602.11 via Frontend Transport; Sat, 4 Jan 2020 13:30:18 +0000 Original-Received: from VI1P194MB0429.EURP194.PROD.OUTLOOK.COM ([fe80::35f2:9ea2:efd6:1d46]) by VI1P194MB0429.EURP194.PROD.OUTLOOK.COM ([fe80::35f2:9ea2:efd6:1d46%5]) with mapi id 15.20.2602.015; Sat, 4 Jan 2020 13:30:18 +0000 Original-Received: from pascal.homepc (90.230.29.56) by AM7PR03CA0012.eurprd03.prod.outlook.com (2603:10a6:20b:130::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2602.11 via Frontend Transport; Sat, 4 Jan 2020 13:30:17 +0000 Thread-Topic: Using incremental parsing in Emacs Thread-Index: AQHVwh1Uy8tVxqoXLEaWXDDoUSKGMA== In-Reply-To: <83lfqnha8a.fsf@gnu.org> (Eli Zaretskii's message of "Sat, 04 Jan 2020 10:55:17 +0200") Accept-Language: sv-SE, en-US Content-Language: en-US x-clientproxiedby: AM7PR03CA0012.eurprd03.prod.outlook.com (2603:10a6:20b:130::22) To VI1P194MB0429.EURP194.PROD.OUTLOOK.COM (2603:10a6:803:41::33) x-incomingtopheadermarker: OriginalChecksum:CB419D19A291A12A934CDBE8429F503F09ABFECE3E84CEC95693611B8AD44509; UpperCasedChecksum:41DADA8F338535C60A3F6B42517B0D05A013F815E480505351B3FE6B79F528BD; SizeAsReceived:7675; Count:50 x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [qI5C9FBw8OhTQIcugkQvQFgG5PQgg1ng] x-microsoft-original-message-id: <87woa7mjrv.fsf@live.com> x-ms-publictraffictype: Email x-incomingheadercount: 50 x-eopattributedmessage: 0 x-ms-office365-filtering-correlation-id: c37af6a9-399f-43f2-76f0-08d7911a3d58 x-ms-exchange-slblob-mailprops: Zv/SX2iM+5XVJaXEUGm9jf4jcUs32Vjtr3Fl5WuEg+o6oc/jeoEHr2Bx4QemYUaX9SnNqWO3vP/GQXuvA2EslfDKAdXdbHoFTOxZHo/nxcWIljaErQ5WQ9lHb4W/2XxqYhbjNLSqk5nVufqll9Wro0ocY9MN1LKYdG+0RODqic8jnx0/7a/IE3RidQ+57SzOKgtvSv9qZaOwbOu+/uCUz7S8sT1kYu8wsjK3IINYSZfrLd4Diy5oN0lCXRRApzAEduKILgVTseAWVPWtu1loCQOqgfC3Bo4EbF/VS98+e0Ar32jHE1iE0xLh/wKQ32/0gumt1eUeKTTj58Kvx+14q6xa5ZUw0wvm23zqC/tj7WHIZTtzw9Na2iiQYLFFPI/oZ2Y4C1YgnzrxlMRGQVzRABrYG+qLXG/Px07P8xsFc4zxtiaJxmlnX2g6bC4un4Idng9G6zBmvAI2CzpeWgC8LaF1s2WGVZ8mmcDVJVJKtEpOjhradksL+gL+2x7YjMR85xrraHe7XmnKtLaFHxbxpuMIQvA4Oi/Q7zg2V6kLVDgT0xEM1Y754p5pcFuweP+1p2OpvzBN0nQwAQRhqVgDdPWNwr+csFfxRWuOO6Mnb4yHwUfQRVFkCuJMEZGiVCUS720hErLAuQ8UR6ItfR9NiyRXceNdWG147DHKW9JAvI5+Q8sCd9uU0ZqrB94sRg3eaQ7+xy9iyNU= x-ms-traffictypediagnostic: DB3EUR04HT042: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: OyBqYgQ14RvJ7BSozcMoz/fYdg1doN0SEQg5hPZb93tBdr0H9rIH3nwZJkBPTVjDEDQ+jlvVvX7Sgb1lZ5qlug9r4o0M6u9v3NvwNk+mDvdv6z3LN715+34eBK3t7wivO78jA8zluaNMEW1mvpA5VmD+johxQdDXkM4hTM7Tv7vulYhU0lkpwTuSkdl6xYZL x-ms-exchange-transport-forked: True X-OriginatorOrg: live.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: c37af6a9-399f-43f2-76f0-08d7911a3d58 X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Jan 2020 13:30:18.4707 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Internet X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3EUR04HT042 X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 [fuzzy] X-Received-From: 40.92.74.99 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.org gmane.emacs.devel:243919 Archived-At: Eli Zaretskii writes: >> Date: Sat, 4 Jan 2020 12:57:24 +0800 >> From: HaiJun Zhang >>=20 >> For font-lock, I think it may work with tree-sitter like this: >> 1. After openning a file, parse the whole buffer with tree-sitter. We ge= t a syntax tree from tree-sitter. >> 2. Get the syntax nodes with ts_node_descendant_for_point_range and font= ify the buffer text in the visible >> region or whole buffer. >> 3. After each modification of buffer text, make a copy of the syntax tre= e as the new one. Update the new one >> with the modification. >> 4. Get the changed range and changed nodes list by comparing the old and= new syntax trees. Then free the >> old syntax tree. >> 5. Update the text properties in the changed range. >> 6. Goto 3 > > I encourage you to study how JIT font lock works in Emacs, and in > particular how it plugs itself into the display engine. Because using > any incremental parsing technology for font-lock needs a good > understanding of how font-lock is typically used in Emacs, and any > practical suggestions for integration and interfaces must take that > into consideration. > > E.g., step 1 is anathema to JIT font-lock: it would produce a long > delay in displaying a file's buffer when the file is first visited. > For example, think about visiting a large and complex source file such > as xdisp.c: even if it takes tens of milliseconds to parse all of it, > as some tree-sitter presentation claims, waiting for that long before > we even start displaying the first window-full would be an annoyance. > And that's even before we consider the time to compute all the face > text properties from the syntax tree, something that will also take > time. > > Thanks. Do it in a thread and display file originally without syntax coloring, and then gradually display results as the tree-sitter thread work it's way? Maybe start with displayed portion of the file only. Then since tree-sitter accepts changes as fine-grained as on character level, send new lines for syntax colouring as they are scrolled upp or down. Or other strategy could be to have tree-sitter thread highlight the visible portion of the file and to continue to work with non-visible part of the buffer in background? Could something like that work?