From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Stefan Monnier Newsgroups: gmane.emacs.devel Subject: Re: Tree-sitter introduction documentation Date: Sun, 01 Jan 2023 19:40:18 -0500 Message-ID: References: <87cz8jxoat.fsf@ledu-giraud.fr> <83wn6ri7pn.fsf@gnu.org> <5e0a3185-de82-b339-0fa2-956779e63d6f@cornell.edu> <868rj6vfep.fsf@gmail.com> <4895891b-e5ea-9c37-f51b-df2e479ee758@yandex.ru> <83y1qt11xq.fsf@gnu.org> <9eb013da-d0fc-8e17-c6e3-1e8f913aebfa@yandex.ru> <83pmc50xxc.fsf@gnu.org> <71cfe4e8-3bb8-b0a6-9be5-8c0a6d92cfab@yandex.ru> <83h6xg29z3.fsf@gnu.org> <87wn6cyey5.fsf@posteo.net> <787B1EB4-1925-4679-8747-449DCD685432@gmail.com> <83y1qo6h7a.fsf@gnu.org> <837cy64v2f.fsf@gnu.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="17377"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Cc: Eli Zaretskii , Philip Kaludercic , Dmitry Gutov , theophilusx@gmail.com, emacs-devel@gnu.org To: Yuan Fu Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Mon Jan 02 01:40:58 2023 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pC8sr-0004Ln-8D for ged-emacs-devel@m.gmane-mx.org; Mon, 02 Jan 2023 01:40:57 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pC8sO-0000a8-Hl; Sun, 01 Jan 2023 19:40:28 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pC8sM-0000Zs-Mn for emacs-devel@gnu.org; Sun, 01 Jan 2023 19:40:26 -0500 Original-Received: from mailscanner.iro.umontreal.ca ([132.204.25.50]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pC8sK-0003aI-NN; Sun, 01 Jan 2023 19:40:26 -0500 Original-Received: from pmg3.iro.umontreal.ca (localhost [127.0.0.1]) by pmg3.iro.umontreal.ca (Proxmox) with ESMTP id EC2B744148F; Sun, 1 Jan 2023 19:40:21 -0500 (EST) Original-Received: from mail01.iro.umontreal.ca (unknown [172.31.2.1]) by pmg3.iro.umontreal.ca (Proxmox) with ESMTP id 5E87F44147F; Sun, 1 Jan 2023 19:40:20 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=iro.umontreal.ca; s=mail; t=1672620020; bh=nkDMoYHqy+jqrljNFdVXk2jwOyh/mtWHn2p0Ah194VU=; h=From:To:Cc:Subject:In-Reply-To:References:Date:From; b=fBh4f+qhatBsZElInLOqDGAtrcfEA2Ri4DxhxG1xl1DK7DZC7p9X34KDnSOkR9wmP OMhj8l2BQxhUXYlaQh18HqV/pCFs9n3KCFXKJvmNWsziNR+V30uSY7zaWk7iNDwRHy p43zeUJxFTZ6Cxntr4rS9RPAA8tKtlJ6CtRLXPfi4lQAH7nQo0F7i6fQ7GSsOVFnWH VI1TUOAckrk9n3o8eNYoiH9W7DEAcT80bVI50f+S0pyHihs3vIdN/jqKDRVsIk3ZRH 3OGJRLJXLp9pkSeNt4SMX8NGflaVujS+jrAg+LVXifap8Bh9OqeMeusU2u2jpHXRxE 05vZkIXPAcI3Q== Original-Received: from pastel (unknown [45.72.200.228]) by mail01.iro.umontreal.ca (Postfix) with ESMTPSA id 0F4591208FD; Sun, 1 Jan 2023 19:40:20 -0500 (EST) In-Reply-To: (Yuan Fu's message of "Sun, 1 Jan 2023 16:31:30 -0800") Received-SPF: pass client-ip=132.204.25.50; envelope-from=monnier@iro.umontreal.ca; helo=mailscanner.iro.umontreal.ca X-Spam_score_int: -42 X-Spam_score: -4.3 X-Spam_bar: ---- X-Spam_report: (-4.3 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.devel:302187 Archived-At: --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable >>>> I think we should default to asynchonous operations as much as possibl= e. >>> If you are talking about the whole command, maybe. Does call-process >>> automatically yield when ran in a make-thread? >> No, it doesn't. And it's threads that yield, not Emacs primitives. A >> thread will yield when it calls some API that invokes pselect. > Thanks. I guess what I=E2=80=99m asking is that if I run the command in m= ake-thread, > will the thread yield when it=E2=80=99s waiting for the subprocess? I tri= ed it and > it still blocks Emacs so I guess the answer is no. But maybe I=E2=80=99m = doing > it wrong. No, indeed by "asynchronous" I wasn't thinking of using threads, but rather using `start-process` and then putting the "rest" into its sentinel. It's rather cumbersome to do, admittedly. I have a work-in-progress library of "promises/futures" for Emacs which should (eventually) make it easier, but that's not an option for the `emacs-29` branch :-( Stefan --=-=-= Content-Type: application/emacs-lisp Content-Disposition: attachment; filename=futur.el Content-Transfer-Encoding: quoted-printable ;;; futur.el --- Future/promise-based async library -*- lexical-binding: t= ; -*- ;; Copyright (C) 2022 Stefan Monnier ;; Author: Stefan Monnier ;; Keywords: ;; This program is free software; you can redistribute it and/or modify ;; it under the terms of the GNU General Public License as published by ;; the Free Software Foundation, either version 3 of the License, or ;; (at your option) any later version. ;; This program is distributed in the hope that it will be useful, ;; but WITHOUT ANY WARRANTY; without even the implied warranty of ;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the ;; GNU General Public License for more details. ;; You should have received a copy of the GNU General Public License ;; along with this program. If not, see . ;;; Commentary: ;; A library to try and make async programming a bit easier. ;; This is inspired from Javscript's async/await, Haskell's monads, ;; and ConcurrentML's events. ;; You can create trivial futures with `futur-pure'. ;; You can create a "process future" with `futur-process-make'. ;; And the main way to use futures is to compose them with `futur-let*', ;; which can be used as follows: ;; (futur-let* ;; ((cmd (build-arg-list)) ;; (exitcode <- (futur-process-make :command cmd :buffer t)) ;; (out (buffer-string)) ;; Get the process's output. ;; (cmd2 (build-second-arg-list exitcode out)) ;; (otherexit <- (futur-process-make :command cmd :buffer t))) ;; (futur-pure (buffer-string))) ;; This example builds a future which runs two commands in sequence. ;; For those rare cases where you really do want to block everything ;; else and wait for a future to complete, you can use`futur-get'. ;; New kinds of futures can be constructed from: ;; - `futur-waiting' to create the actual future. ;; - `futur-deliver' to deliver the value to the future created earlier ;; with `futur-waiting'. ;;; Code: ;; TODO: ;; - Handle exceptions. (require 'cl-lib) ;;;; The `futur' data structure (cl-defstruct (futur (:conc-name futur--) (:predicate futur--p) (:constructor nil) (:constructor futur--pure (value &aux (clients 't))) (:constructor futur--waiting (&optional value))) "A promise/future. A future has 3 states: - Done: in that state, `clients' is `t', and `value' holds the result. - Error: in that state, `clients' is `error', and `value' holds the error. - Waiting: in that state, `clients' is a list of \"callbacks\" waiting for the value or the error, and `value' holds the object that will deliver the value (can be another future, a process, a thread, a list (of futures), or possibly other objects with a `futur-wait' method)." (clients nil) (value nil)) (defun futur--waiting-p (futur) (listp (futur--clients futur))) (defun futur-deliver (futur val) (cl-assert (futur--waiting-p futur)) (setf (futur--value futur) val) ;; Don't run the clients directly from here, so we don't nest. (let ((args (list val))) (dolist (client (prog1 (futur--clients futur) (setf (futur--clients futur) t))) (funcall-later client args)))) (defun futur-fail (futur error) (cl-assert (futur--waiting-p futur)) (setf (futur--value futur) error) ;; Don't run the clients directly from here, so we don't nest. ;; FIXME: Don't run them the same way we run them when we deliver! (let ((args (list error))) (dolist (client (prog1 (futur--clients futur) (setf (futur--clients futur) t))) (funcall-later client args)))) (defun futur-pure (val) "Build a trivial future which returns VAL." (futur--pure val)) (defun futur-new (builder) "Build a future. BUILDER is a function that will be called with one argument \(the new `futur' object, not yet fully initialized) and it should return the object on which the future is waiting. The code creating this future needs to call `futur-deliver' when the object has done the needed work. The object can be any object for which there is a `futur-wait' method." (let ((f (futur--waiting)) (x (funcall builder f))) (cl-assert (futur--waiting-p f)) (cl-assert (not (futur--value f))) (setf (futur--value f) x) f)) ;;;; Composing futures (defun futur--bind (futur fun) "Compose FUTUR with FUN. Calls FUN with FUTUR's value when it becomes available, and throws away its return value." (if (not (futur--waiting-p futur)) (funcall fun (futur--value futur)) (push fun (futur--clients futur))) nil) (defun futur--join (oldpro newpro) (cl-assert (futur--waiting-p oldpro)) (setf (futur--value oldpro) newpro) ;Update the object we're waiting for. (futur--bind newpro (lambda (val) (futur-deliver oldpro val)))) (defun futur-bind (futur fun) "Build a new future by composition. FUN will be called with the return value of FUTUR and should return a new future." (if (not (futur--waiting-p futur)) (funcall fun (futur--value futur)) (let ((new (futur--waiting futur))) (push (lambda (val) (if (futur--p val) ;; FIXME: During the execution of FUN, `new' says it's ;; waiting for `futur', yet `futur' is already done! (futur--join new (funcall fun val)) (futur-deliver new val))) (futur--clients futur)) new))) (defun futur-get (futur) "Wait for FUTUR to deliver and then return its value." (futur-wait futur) (cl-assert (not (futur--waiting-p futur))) (futur--value futur)) (cl-defgeneric futur-wait (object &optional futur) "Wait for OBJECT to deliver. OBJECT is an object for which FUTUR is waiting.") (cl-defmethod futur-wait ((futur futur) &optional _) (let (object) ;; Loops since the object may change, e.g. in `futur-bind' we first ;; wait for one future and then for another. (while (and (futur--waiting-p futur) (not (eq object (setq object (futur--value futur))))) (futur-wait object futur))) (cl-assert (not (futur--waiting-p futur)))) (cl-defmethod futur-wait :before (o &optional f) (cl-assert (futur--p (or f o)) nil "%S %S" o f) (cl-assert (futur--waiting-p (or f o)))) (cl-defmethod futur-wait :after (o &optional f) (cl-assert (not (futur--waiting-p (or f o))))) (define-error 'futur-aborted "Future aborted") (cl-defgeneric futur-abort (futur &optional error) "Abort processing of FUTUR and all of its clients. If it had not been computed yet, then make it fail with ERROR.") (cl-defmethod futur-abort ((futur futur) &optional error) (if (not (futur--waiting-p futur)) () ;; Do nothing. (futur-abort (futur--value futur) error))) (defmacro futur-let* (bindings &rest body) "Sequence asynchronous operations via futures. BINDINGS can contain the usual (VAR EXP) bindings of `let*' but also \(VAR <- EXP) bindings where EXP should return a future, in which case the rest of the code is executed only once the future terminates, binding the result in VAR. BODY is executed at the very end and should return a future." (declare (indent 1) (debug ((&rest (sexp . [&or ("<-" form) (form)])) bod= y))) (cl-assert lexical-binding) ;; FIXME: Should we allow pcase patterns in the bindings, like `pcase-let= *'? (pcase-exhaustive bindings ('() (macroexp-progn body)) (`((,var ,exp) . ,bindings) `(let ((,var ,exp)) (futur-let* ,bindings ,@body))) (`((,var <- ,exp) . ,bindings) `(futur-bind ,exp (lambda (,var) (futur-let* ,bindings ,@body)))))) ;;;; Processes (defun futur--process-sentinel (proc futur) (when (memq (process-status proc) '(exit signal closed failed)) (futur-deliver futur (process-exit-status proc)))) (defun futur-process-make (&rest args) "Create a process and return a future that delivers its exit code. The ARGS are like those of `make-process' except that they can't include `:sentinel' because that is used internally." (futur-new (lambda (f) (apply #'make-process :sentinel (lambda (proc _state) (futur--process-sentinel proc f)) args)))) (defun futur-process-exit-status (proc) "Create a future that returns the exit code of the process PROC." (if (memq (process-status proc) '(exit signal closed failed)) (futur-pure (process-exit-status proc)) (futur-new (lambda (f) ;; FIXME: If the process's sentinel signals an error, it won't run u= s :-( (add-function :after (process-sentinel proc) (lambda (proc _state) (futur--process-sentinel proc f))) proc)))) (cl-defmethod futur-wait ((proc process) &optional futur) (while (and (futur--waiting-p futur) (accept-process-output proc 1.0)) (sit-for 0))) ;; Just redisplay every 1s if needed. (cl-defmethod futur-abort ((proc process)) (let ((futur (process-get proc 'futur))) (delete-process proc) (setf (futur--clients futur) t))) (defun futur-process-send (proc string) ;; FIXME: This is quite inefficient. Our C code should instead provide ;; a non-blocking `(process-send-string PROC STRING CALLBACK)'. (futur-new (lambda (f) (make-thread (lambda () (futur-deliver f (process-send-string proc string))))= ))) (cl-defmethod futur-wait ((th thread) &optional _futur) (thread-join th)) (cl-defmethod futur-abort ((th thread) &optional futur) ;; FIXME: This doesn't guarantee that the thread is aborted. (thread-signal th 'error "Abort future") (setf (futur--clients futur) t)) ;;;; Multi futures: Futures that are waiting for several other futures. (defun futur-multi-bind (futurs fun) "Build a new future by composition. FUTURS is a list of `futur's. FUN will be called with the return values of FUTURS (one argument per future) and should return a new future." (if (null futurs) (funcall fun) (let* ((new (futur--waiting futurs)) (count (length futurs)) (args (make-list count nil)) (i 0)) (dolist (futur futurs) (futur--bind futur (let ((cell (nthcdr i args))) (lambda (val) (cl-assert (null (car cell))) (setf (car cell) val) (setq count (1- count)) (when (zerop count) (futur--join new (apply fun args)))))) (setq i (1+ i))) new))) (cl-defmethod futur-wait ((futurs cons) &optional _) (mapc #'futur-wait futurs)) (cl-defmethod futur-abort ((futurs cons)) (mapc #'futur-abort futurs)) (provide 'futur) ;;; futur.el ends here --=-=-=--