From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Akira Kyle Newsgroups: gmane.emacs.devel Subject: Rethinking the design of xwidgets Date: Mon, 12 Oct 2020 14:58:19 -0600 Message-ID: <864kmzupp0.fsf@akirakyle.com> Mime-Version: 1.0 Content-Type: text/plain; format=flowed Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="18625"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: mu4e 1.4.13; emacs 28.0.50 To: emacs-devel@gnu.org Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Mon Oct 12 23:12:16 2020 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kS57A-0004jn-9X for ged-emacs-devel@m.gmane-mx.org; Mon, 12 Oct 2020 23:12:16 +0200 Original-Received: from localhost ([::1]:40710 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kS579-0006Qh-Ak for ged-emacs-devel@m.gmane-mx.org; Mon, 12 Oct 2020 17:12:15 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:42856) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kS4tl-0001gh-CF for emacs-devel@gnu.org; Mon, 12 Oct 2020 16:58:25 -0400 Original-Received: from mail-io1-f67.google.com ([209.85.166.67]:44516) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kS4tj-0005OV-Au for emacs-devel@gnu.org; Mon, 12 Oct 2020 16:58:24 -0400 Original-Received: by mail-io1-f67.google.com with SMTP id l8so18979836ioh.11 for ; Mon, 12 Oct 2020 13:58:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:user-agent:from:to:subject:date:message-id :mime-version; bh=UocbGl+irRJqV5KZAO5vg8wV4L/dUbbrrUVIhF4qZ54=; b=UV/ug+D29Lfa0s7tk828S2YGbeKa8vdsX9fsUvJniQ1vICOz7xL7IWbGoCiPYlKufC bpYBrJK7VmSBmOdk7PUTPzmVNssoDpASWbCWXNVHHpQrMp3gq+03YbLUh2AGgJhsf3hT 8KfPBJPyv4TsuyuJgs1OIpfuJd3peGz0wO6T1ZL7+pJUcaL+vivlj8ejUoP3EDoraOaH WYLl/OfbX27JbFFxM9b4Sw3YM5XrYlhs8tx/7lwG53E0NkhrEWQtVnfX67gumZb41paZ HdLBZXVbtVbfjD2jb4CA0cbkaH53MwZwS4zNeMa9FiRN8Dc9QN6R5TW4/eqm6XPcG+yS EPhw== X-Gm-Message-State: AOAM530BcDVqcyUQxi+lXaXA26G9XVQhITKFQHb5x56qGERfmxKUVQGt oXXrloVbF3Y9pnl715lpA5bNgVisaJCGtw== X-Google-Smtp-Source: ABdhPJz6j4PH8HuzHEkAgXj0/0NAjLF278V2BSyivUx7VXwxMCEZrOuQ/CRccf8iLQE73MqF5neOgw== X-Received: by 2002:a5e:9b11:: with SMTP id j17mr17595130iok.176.1602536301553; Mon, 12 Oct 2020 13:58:21 -0700 (PDT) Original-Received: from lore ([2601:281:8080:45f0:392b:d69b:64fd:1cef]) by smtp.gmail.com with ESMTPSA id 12sm7641101ioz.33.2020.10.12.13.58.20 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Oct 2020 13:58:21 -0700 (PDT) Received-SPF: pass client-ip=209.85.166.67; envelope-from=aikokyle@gmail.com; helo=mail-io1-f67.google.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/12 16:58:22 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -13 X-Spam_score: -1.4 X-Spam_bar: - X-Spam_report: (-1.4 / 5.0 requ) BAYES_00=-1.9, FREEMAIL_FORGED_FROMDOMAIN=0.249, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.25, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:257499 Archived-At: Hi all, If you don't care about the motivation and background for this problem skip the next two paragraphs. One of my primary uses of emacs is as a scientific notebook utilizing org mode's support for literate programming to interweave prose, LaTeX, and code. I find this to be far more powerful and comfortable than using the Jupyter notebooks which have quickly become incredibly popular for such scientific computing. However the biggest pain point I have is due to Emacs' relatively poor capabilities for interfacing with objects other than text. Sure Emacs can display images ok, but I'm not aware of any way to get emacs to display more dynamic content. Furthermore I always want to be able to do more inside Emacs, and the one other application I spend as much time in as I spend in Emacs is a browser. Unfortunately I can't even really use eww for the simplest of my web browsing tasks such as looking up documentation as it usually fails to adequately render the variety of ways websites embed mathematics. This has all led me to be interested in ways to enhance Emacs' display capabilities. The two projects I've come across that do this are the experimental xwidgets configuration and the Emacs Application Framework (EAF) [1]. Unfortunately EAF uses Qt to reparent a window into emacs and Python to do two way RPC with elisp. IMO it isn't a very emacsy way to enhance Emacs' display capabilites and is limited to just taking over a whole window rather than being embedded as part of a buffer (key to supporting dynamic content such as an interactive inline plot inside an org mode document). That leaves xwidgets which seems to be somewhat incomplete, unused, and neglected. I've been poking around the xwidgets code and I really like the concept, however the implementation seems to need some more thought and work. I see two fundamental issues with the current xwidget code. The first is that in order to add a new type of widget, one must integrate it into the C portion of Emacs' codebase and expose relevant functionality to elisp. The difficulty of navigating how xwidgets are integrated into redisplay and how to expose the inherently event driven nature of GUI toolkits into elisp in a safe way is a big hurdle to hacking on xwidgets. In the last decade of this features existence no one has added any additional xwidgets besides the original webkit one (see make-xwidget). This despite the fact that xwidgets were originally designed to allow embedding of any gtk widgets such as buttons, sliders, image views, opengl contexts [2]. Recently support for webkit using the native cocoa framework was added for the ns toolkit demonstrating that this feature need not be tied to a specific GUI toolkit. The second issue is that most interesting applications of xwidgets will likely need to use some external C library. Historically, package maintainers have been hesitant to package emacs with xwidgets support due to the additional dependency on webkitgtk, which was often not kept current with security patches [3]. In the long run I don't think adding extra dependencies to emacs just for the sake of some specific xwidget is worth it. That leads me to my following proposal to rethink the design of xwidgets. Given that Emacs now has support for dynamic modules and it is now enabled by default, I propose exposing an interface for dynamic modules to define custom xwidgets. I believe this would be a good way to solve the two fundamental issues with the current xwidgets implementation that I described above. Having a well defined interface means one wouldn't have to deal with debugging subtle redisplay and lisp interpreter issues when creating a custom xwidget type. Furthermore any external library needed for a custom xwidget type will only be required if one chooses to use that xwidget type and would keep such dependencies out of Emacs itself. I've already done a proof of concept prototype to convince myself that this is possible. I first modified the xwidget display spec to include a field which is expected to contain a function to initialize an xwidget view. When redisplay acts on this spec, the xwidgets code calls this function with the xwidget display spec and expects a user_ptr Lisp_Object to be returned which wraps a pointer to a GtkWidget. The xwidgets code then adds this widget to an appropriate container that it uses to ensure the widget is moved when the position or clipping of its glyph changes. I then define this xwidget view initialization function in a dynamic module. Since every window containing the xwidget has its own view, and hence separate GtkWidget instance, it is then up to the dynamic module to define the how to handle the updating of each view based on some common model each is supposed to represent. This works around the limitation that in most GUI toolkits each widget can only have one view. Thus for something like the webkitgtk widget, one implementation might choose to do offscreen rendering (as is currently done) so that all windows displaying the widget are updated together when, say, one of them is scrolled. While another implementation might decide each view gets a separate view of the same webpage and scrolling one will not scroll the other (which is what the cocoa webkit implementation should probably do, but it is currently only limited to being displayed in a single window). I'd be interested in any thoughts anyone might have about this proposal and if some changes along this line might have some chance of being accepted in master someday. I've always wanted to dig into the nity grity c code of emacs and this has finally given me the excuse to do so! Akira [1] https://github.com/manateelazycat/emacs-application-framework [2] https://github.com/jave/xwidget-emacs [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=843462