From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.ciao.gmane.io!not-for-mail From: Adam Porter Newsgroups: gmane.emacs.devel Subject: Re: On elisp running native Date: Thu, 05 Mar 2020 04:54:09 -0600 Message-ID: <87mu8vjcmm.fsf@alphapapa.net> References: <83tv5mp48l.fsf@gnu.org> <83sgl0lchm.fsf@gnu.org> <83imlwl9vm.fsf@gnu.org> <83o8uegykm.fsf@gnu.org> <74dd94a9-28cb-a5fd-dbc7-ab21009834ad@cs.ucla.edu> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="ciao.gmane.io:159.69.161.202"; logging-data="109695"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) To: emacs-devel@gnu.org Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Thu Mar 05 11:55:03 2020 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j9o9f-000SQe-4b for ged-emacs-devel@m.gmane-mx.org; Thu, 05 Mar 2020 11:55:03 +0100 Original-Received: from localhost ([::1]:47074 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j9o9e-0001vz-77 for ged-emacs-devel@m.gmane-mx.org; Thu, 05 Mar 2020 05:55:02 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:50763) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j9o93-0000y9-5b for emacs-devel@gnu.org; Thu, 05 Mar 2020 05:54:26 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j9o91-0002RM-7j for emacs-devel@gnu.org; Thu, 05 Mar 2020 05:54:25 -0500 Original-Received: from ciao.gmane.io ([159.69.161.202]:56154) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1j9o91-0002O6-1P for emacs-devel@gnu.org; Thu, 05 Mar 2020 05:54:23 -0500 Original-Received: from list by ciao.gmane.io with local (Exim 4.92) (envelope-from ) id 1j9o8w-000Rcu-Jj for emacs-devel@gnu.org; Thu, 05 Mar 2020 11:54:18 +0100 X-Injected-Via-Gmane: http://gmane.org/ X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 159.69.161.202 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:245259 Archived-At: Hi Andrea, Andrea Corallo writes: > - Finally I've set up a docker image for people willing to test it > without having to configure and compile Emacs and libgccjit. Now that > I've learned how it works I plan to use it to setup a small CI to keep > an eye that the latest GCC does not break with our build. Thank you for setting up the Docker image. I had tried to build the native-comp branch on my system but finally was unable to overcome a weird error related to libgccjit (probably because I need to upgrade my system). I even tried using Guix to build it, but libgccjit isn't packaged in it (and I'm completely new to Guix packaging, so I wasn't able to figure out how to adjust the GCC build for that in a reasonable amount of time). I managed to get the Docker image working, and I did a little testing with one of my packages, org-ql, which is for searching Org files. I installed it and its dependencies, then used native-comp-async to compile all of ~/.emacs.d/elpa. Then I restarted Emacs to ensure that the native libraries were loaded, and finally I ran the following code and observed a more than 2x improvement! #+BEGIN_SRC elisp ;; Reset the cache to force search to run again. (setf org-ql-cache (make-hash-table :test #'equal)) ;; Set my personal groups (omitted) and enable org-super-agenda-mode ;; to display in groups. (setf org-super-agenda-groups my-groups) (org-super-agenda-mode) ;; Load all my Org files and search them. (let* ((enable-local-variables nil) (files (directory-files org-directory 'full (rx ".org" eos)))) (dolist (file files) (find-file-noselect file 'nowarn)) (benchmark-run-compiled (org-ql-search files ;; The (tags) selector tends to be slow compared to other ;; predicates because of searching the hierarchy repeatedly ;; and tag inheritance, so it makes for an interesting test. '(and (todo) (tags "Emacs")) :super-groups org-super-agenda-groups))) #+END_SRC #+RESULTS: | 5.819892666 | 1 | 0.6101534569999956 | In Emacs 26.3 | 2.317804428 | 3 | 0.6223487619999997 | In Emacs 28 with native-comp Then I tested this more complex query and observed an even more impressive speedup of about 3.4x! '(and (not (done)) (or (habit) (deadline auto) (scheduled :to today) (ts-active :on today))) #+RESULTS: | 8.377843199 | 1 | 0.6277314330000081 | In Emacs 26.3 | 2.436048375 | 2 | 0.4278436059999997 | In Emacs 28 with native-comp This is very exciting! The difference between 8.4 seconds and 2.4 seconds is a huge usability improvement. And everything seems to work fine! No bugs, errors, or crashes of any kind. I did encounter a minor issue related to macro-expansion and loading that may be my fault. When I tried to (require 'org-ql) or call a function that is autoloaded from it, I got an error saying that the .eln file did not provide the org-ql feature. I surmised that meant that the file hadn't been compiled properly, so I looked at the async compilation log and saw an error saying that the variable org-ql-predicates was not defined. At this URL you can see a macro definition, org-ql--def-plain-query-fn, and the call to it that's wrapped in cl-eval-when to ensure it works properly with the byte-compiler: https://github.com/alphapapa/org-ql/blob/06481960764a55a36d886e1db79a58258d5d2ffb/org-ql.el#L1671 The macro uses the value of org-ql-predicates, which is defined here: https://github.com/alphapapa/org-ql/blob/06481960764a55a36d886e1db79a58258d5d2ffb/org-ql.el#L126 But when the org-ql--defpred macro: https://github.com/alphapapa/org-ql/blob/06481960764a55a36d886e1db79a58258d5d2ffb/org-ql.el#L140 ...is called, it adds to that variable. So all of that works with the byte-compiler, but apparently not with the native compilation. To work around it, I wrapped (defvar org-ql-predicates...) in eval-and-compile, and then wrapped all of the calls to org-ql--defpred in eval-and-compile as well. Then I deleted the org-ql.eln file and recompiled it, restarted Emacs, and then everything worked fine. I'm guessing that this issue is "native" to the native-compilation and might need to be documented in some way, but I wouldn't expect it to affect many packages, only ones that do tricky things with variables and macro-expansion like this. So, a few ideas for fixes or improvements: 1. I guess the .eln file should not have been produced or left behind in the directory, since there was a compilation error. 2. The compilation error should probably be shown as a warning or something, because it's easily lost in the async compilation log buffer. 3. It would probably be good if the loader fell back to .elc and .el files if the .eln file doesn't seem to work, and showed a warning when doing so. The library would remain usable in its byte-compiled form, and in the case of a package like this, the user could then report the native-compilation error to the developer. Other than that minor issue, everything works and feels very fast! This is great! > ~eln~ files are compiled in specific sub-folders taking in account > host architecture and current Emacs configuration to disambiguate > possible incompatibilities. > As example a file ~.../foo.el~ will compile into something like > ~.../x86_64-pc-linux-gnu-12383a81d4f33a87/foo.eln~. FYI, I did not see the .eln files being put in subdirectories like that in the Docker image. Maybe it didn't receive that update yet. This is all very exciting. I'm imagining an update to package.el that would byte-compile packages immediately, as usual, then start native-compile-async'ing them in the background, loading each file's native version as it completes, with the packages being immediately usable and seeming to get faster as each file is compiled and loaded. > I'm generally very satisfied (surprised) about the stability of the > toy. I'm looking forward going into compile time mitigation. I guess > I'll start this weekend with deferred compilation and GCC profiling. I think it isn't a toy anymore! :) Thanks very much for your work on this. Exciting times for Emacs, as this feature, in one form or another, has been eagerly awaited for years.