From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Akira Kyle Newsgroups: gmane.emacs.devel Subject: Re: Rethinking the design of xwidgets Date: Tue, 13 Oct 2020 14:37:31 -0600 Message-ID: <86pn5luak4.fsf@akirakyle.com> References: <864kmzupp0.fsf@akirakyle.com> <835z7e2ouj.fsf@gnu.org> <86v9fet5sg.fsf@akirakyle.com> <83imbe1040.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; format=flowed Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="32259"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: mu4e 1.4.13; emacs 28.0.50 Cc: emacs-devel@gnu.org To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Tue Oct 13 22:39:34 2020 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kSR53-0008GE-Nu for ged-emacs-devel@m.gmane-mx.org; Tue, 13 Oct 2020 22:39:33 +0200 Original-Received: from localhost ([::1]:41168 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kSR52-0007JP-Kt for ged-emacs-devel@m.gmane-mx.org; Tue, 13 Oct 2020 16:39:32 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:57592) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kSR3A-000584-Kz for emacs-devel@gnu.org; Tue, 13 Oct 2020 16:37:36 -0400 Original-Received: from mail-io1-f45.google.com ([209.85.166.45]:44582) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kSR38-00018L-N6; Tue, 13 Oct 2020 16:37:36 -0400 Original-Received: by mail-io1-f45.google.com with SMTP id l8so1520218ioh.11; Tue, 13 Oct 2020 13:37:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:references:user-agent:from:to:cc:subject :in-reply-to:date:message-id:mime-version; bh=u1wtYfWZkKjHqFblhabbfyyomwEAK7OH9KkXy30Z7Hg=; b=LfwvLhNMEjeOFnh0C8UlSHS9WV5I8A1IUVSXzcc9riA8Bh0VWCp6cBILtiKXCSSXYJ NUW5iRNv0zJDHbuvWU4tUw2SSWpV9Z0bUJ2VwbAswR2deJ8nEqHyRs3aVW+RXZOWUsWt BVf2h89/nPOqGvtzQB0CC/wlBT8Z+yRmWS/ZFMyNb4aaUmnRNl46kMluFKYr76bgEqaz TMH9gzEtMGRYpVAYd0IUZ4g/BiPy+sCkGRDAO7/6MZ4PqqMDFXNCzi0AHrvqRF2DmdBk azmSmoN/7UdRVmA8xdOXUBnDZ/DIHaQFBdoEqkUJ5rOzHPcrESSQiNba8Z3x1lkvWvyo rNZg== X-Gm-Message-State: AOAM533eneBER2jqud/UC4oo4T6dL1a+s3D1vGJLdk/NKlhb+22TZZ+H FgjX9KYNykR6f/PHF0AtNGEZ1/1xAjTpE3Qo X-Google-Smtp-Source: ABdhPJwxw0GmAg3z8LEUmMyCDyTNgLr+JZCtOxByU5w+RNZi2OEK7gzqZS93sr3B0zsGrCXHIpMuXA== X-Received: by 2002:a05:6638:13cd:: with SMTP id i13mr545109jaj.55.1602621452702; Tue, 13 Oct 2020 13:37:32 -0700 (PDT) Original-Received: from lore ([2601:281:8080:45f0:31b3:d856:a75f:645]) by smtp.gmail.com with ESMTPSA id c9sm747616iow.1.2020.10.13.13.37.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Oct 2020 13:37:32 -0700 (PDT) In-reply-to: <83imbe1040.fsf@gnu.org> Received-SPF: pass client-ip=209.85.166.45; envelope-from=aikokyle@gmail.com; helo=mail-io1-f45.google.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/13 16:37:33 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -13 X-Spam_score: -1.4 X-Spam_bar: - X-Spam_report: (-1.4 / 5.0 requ) BAYES_00=-1.9, FREEMAIL_FORGED_FROMDOMAIN=0.249, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.25, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:257583 Archived-At: On Tue, Oct 13, 2020 at 11:55 AM, Eli Zaretskii wrote: > I'll need to read up on those two topics, but in general, as > someone > already pointed out, the Emacs display of images supports > interactivity by letting you call functions triggered by mouse > gestures on the varipous spots on the image. If such a function > produces a zoomed-in/out image and displays it instead of the > original > one, wouldn't that fulfill the need? E.g., we already support > animated GIF images and zoom-in/out by mouse scroll, so it is > inaccurate to say that we can only display static images. Can actions be triggered on a click and drag as one would do to pan within a plot? > This use case doesn't necessarily call for the same solution, > but I > could think of a differently designed pdf-viewer that didn't > need to > rasterize each page separately. Again, the mouse-sensitivity > feature > of our image display could perhaps be used for something like > that. I > reckon the hardest problem is to find an engine, be it a library > or a > program, that could convert PDF into something Emacs can > display. As far as I know there are really only two FLOSS PDF libraries out there: poppler and muPDF. Between them poppler supports more PDF features and seems to be more widely used. I agree this isn't really as compelling of a use case for xwidgets, but I seems like it would be easier to achieve better PDF rendering via xwidgets. I don't think it's about needing to rasterize each page separately, its more about directly rendering the rasterized image without having to copy it around a bunch or convert it to different formats. I suppose the image rendering from an elisp string could be made more performant or Emacs could handle pdf display natively but that would require adding poppler as a dependency. > E.g., the kludge in dispnew.c around line 4365. It disables one > of > the most important redisplay optimizations in Emacs, once you > build > with xwidgets enabled. Ah, I had briefly looked at that but I wasn't sure how important it was. I may be able to make some progress on that but it's hard to understand the context for that function. It seems like that function is responsible for an optimization that avoids redrawing glyphs if the window is being scrolled by copying them, however I couldn't find where or how the glyphs are actually copied? >> For example, consider svg support, which is my preferred image >> format. As far is I can tell, currently Emacs rasterizes svgs >> itself before ultimately drawing them on a cairo context. > > Do we actually do that? We use librsvg, don't we? > > Or maybe you are talking about the generic layer of image > handling, > which basically needs a bitmap/pixmap? Yes this is what I'm talking about. In svg_load_image librsvg is used to end up with the rasterized svg in an Emacs_Pix_Container which then goes through a song and dance of pixel level color and transparency transformations to all eventually end back up as a cairo surface. Unless I'm missing something it seems like all this extra complexity and copying around is just for the sake of maintaining Emacs's own image format which was established long before HiDPI, deep color depth displays and libraries like cairo that do the hard work of transforming and rendering onto the glass. > In any case, this seems to be a much more fundamental issue of > how we > display images in Emacs. Which is not necessarily the same as > the > issue you raised originally. It's at least related since the integration xwidgets into Emacs' display was originally modeled off of the way Emacs' displays images, however I see it as potentially being a simpler and more flexible alternative to the way Emacs currently displays images. > The problem with that is that handling such "widgets" in our > display > engine is entirely non-trivial, as the xwidgets code clearly > shows. > There are basic problems in displaying xwidgets that from my POV > are > left unsolved, and I'm not sure we will be able to find > satisfactory > solutions for them. The display engine has several requirements > that > aren't easy to support with widgets that want to manage > themselves. I think it would be good to try to either resolve such issues or decide that Emacs would be better off without xwidgets and instead focus on enhancing its existing rich display capabilities because as it is, the xwidgets code seems to have been lingering in this experimental state for a decade now. I think many like myself are drawn by its promise of being able to rendering complex dynamic content using existing optimized toolkits. There is of course the fact that in its current state on master the only thing it does do is embed a webkit browser in an Emacs window. A feature many would like to see enhanced while many others have voiced both security, privacy, and freedom concerns with such a feature. I'd just like to see a plan on where the community thinks xwidgets should go and work from there since it's been experimental for far too long. > I see where you are coming from with the idea of dynamic > modules, but > I'm not sure your assumption that it will make things easier for > implementing "display elements" that have their own behavior > independent of Emacs -- I'm not sure this assumption is indeed > true. > The Emacs display code has a deceptively simple main design idea > and > execution concept, and as long as one thinks this is all there > is to > it, it could indeed seem that adding such widgets should be > almost > trivial. But the reality is that there are a lot of details to > go > with the main idea, and it's the details that make implementing > this > stuff more difficult than one thinks first. Thank you for taking the time to talk through this with me. I'm seeing more and more that when it comes to the C portion of Emacs' code, nothing is ever quite so straightforward and it's difficult to understand all the intricacies due to the sheer amount of code. > I'm not trying to scare you, mind you, just point out that the > ideas > you have should probably be talked through in mode detail, to > see that > they are workable. For example, did you study the various > window-redisplay optimizations, and if so, did you think how the > widgets will adapt to them? I'm just starting to dig my teeth into some of that. However I'm wondering if anyone has more recently tried to quantify the improvements the various redisplay optimizations make to both actual and perceived render time of Emacs. For example the comment on the scrolling_window function in dispnew.c that you pointed me too references that it is implementing an algorithm from 1978, perhaps it would be worth exploring if there are any algorithmic advances? It strikes me that perhaps some optimizations may no longer be as necessary on modern hardware or on GUI displays as they were when running emacs in a VT on 20 year hardware. Conversely there may be further optimization that could be done by taking advantage of more recent hardware advances such as offloading to a GPU or utilizing vectorized instructions. If there's one think I've taken away from my CS education, its that indeed "premature optimization is the root of all evil". I know Emacs prides itself on its broad portability, however when it comes to xwidgets we already know such a feature is restricted to running on the hardware that gtk3 expects. Anyways I know these are mostly lofty thoughts but I'd like to continue to see Emacs evolve.