From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Andrea Corallo via "Emacs development discussions." Newsgroups: gmane.emacs.devel Subject: Re: Native compilation: the bird-eye view Date: Sun, 23 Aug 2020 16:31:46 +0000 Message-ID: References: <83o8qocd32.fsf@gnu.org> <83ftbzdewp.fsf@gnu.org> <83blmndbpo.fsf@gnu.org> <838shrdb1c.fsf@gnu.org> <837dxbd93b.fsf@gnu.org> Reply-To: Andrea Corallo Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="22629"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) Cc: Andrea Corallo via "Emacs development discussions." , Eli Zaretskii , Paul Eggert To: Stefan Monnier Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sun Aug 23 18:32:28 2020 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1k9sux-0005mB-CK for ged-emacs-devel@m.gmane-mx.org; Sun, 23 Aug 2020 18:32:27 +0200 Original-Received: from localhost ([::1]:57666 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k9suw-0000SM-El for ged-emacs-devel@m.gmane-mx.org; Sun, 23 Aug 2020 12:32:26 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:37704) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k9suP-0008Mr-Ew for emacs-devel@gnu.org; Sun, 23 Aug 2020 12:31:53 -0400 Original-Received: from mx.sdf.org ([205.166.94.24]:51339) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k9suM-00032d-TF; Sun, 23 Aug 2020 12:31:53 -0400 Original-Received: from mab (ma.sdf.org [205.166.94.33]) by mx.sdf.org (8.15.2/8.14.5) with ESMTP id 07NGVkcC016538; Sun, 23 Aug 2020 16:31:46 GMT In-Reply-To: (Stefan Monnier's message of "Sun, 23 Aug 2020 11:37:37 -0400") Received-SPF: pass client-ip=205.166.94.24; envelope-from=akrl@sdf.org; helo=mx.sdf.org X-detected-operating-system: by eggs.gnu.org: First seen = 2020/08/23 12:31:47 X-ACL-Warn: Detected OS = ??? X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:254145 Archived-At: Stefan Monnier writes: >> I've updated how we generate the eln filenames and now they are in the >> form: filename-path_hash-content_hash.eln >> >> where path_hash is computed using the absoulte path of the source file >> and content_hash using its content. > > [ Note: it's not a "path" but a "file name", according the GNU's > conventions. ] > > Hmm... this results in too-long file names IMO, and the absolute file > name can change "gratuitously", making those eln files suddenly unusable. I shorted also the hash length for this reason, I think should be very unlikely to get collisions anyway. This is a file I've here as an example: advice-505b05f39e8e7db0ef9610bf48b6298e-7c08dcc023e8926affda38e4071cba52.eln >> This should enable us for keepeng the eln-cache folders clean and solve >> the issue mentioned in my previous message. > > I don't have a clear understanding of the problem this is trying to solve. > Could you summarize it? Sure, we need a rule to go from the source file to the eln filename. /xxx/foo.el -> foo-something1.eln /yyy/foo.el -> foo-something2.eln Given we store all eln "flatted" in a single directory indeed we need to have "something1" different from "something2". As a first approach I went for using just the abs filename as input to the hashing algo (with some tweaking for the eln being installed by make install). Unfortunatelly dlopen may return the same handle on two subsequent loads if the filename is the same, even if the first file has been renamed or removed. This make recompiling and reloading a file uneffective. Here comes in the requirement of generating a different filename when recompiling. Facing that I came up with the idea of using the last modificaiton date as additional input to the hash, modifying the source then generates a different eln filename and everything works. Unfortunatelly make install is using tar to zip the source file and the last modificaiton date is not preserved precisely. Aside from that I perceived the last modification date is a bit weak for this scope. I then came-up with the current idea of using the source content as input of the hash, this should be robust. At the moment we are using 2 separate hashes to make the cleaning (easy) possible, this is a separate problem that's correct. The idea is that with the two hashes when recompiling is easy to just remove the old file as you can identify using the first hash the old one. Additionally to that, at any point in time, if you have two file sharing the first hash you know that all but the most recent can be removed. > If you're worried about how to delete old .eln files, I think there are > two problems: identifying which .elc files still exist and identifying > which Emacs binaries still exist. > > But the first problem can arguably be solved by scanning the `load-path`, > collecting the .elc files we find there (and kind of GC algorithm > where `load-path` acts as the roots). I think this may be problematic as we could have two different Emacs with two different load-path that are sharing the eln-cache folders. This may be a bit of patological case tho. If we want to solve the cleaning in some other way we could certanly remove the hash computed from the filename. Thanks Andrea -- akrl@sdf.org