From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Michael Albinus Newsgroups: gmane.emacs.devel Subject: Re: Upstreaming filenotify-recursive Date: Thu, 26 Aug 2021 17:34:02 +0200 Message-ID: <87zgt49q51.fsf@gmx.de> References: <871r6n8dw0.fsf@gmail.com> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="7554"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) Cc: emacs-devel@gnu.org To: Jethro Kuan Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Thu Aug 26 17:34:58 2021 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mJHP7-0001cw-OC for ged-emacs-devel@m.gmane-mx.org; Thu, 26 Aug 2021 17:34:57 +0200 Original-Received: from localhost ([::1]:37388 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mJHP6-0002mW-68 for ged-emacs-devel@m.gmane-mx.org; Thu, 26 Aug 2021 11:34:56 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:54194) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mJHOO-00027o-5R for emacs-devel@gnu.org; Thu, 26 Aug 2021 11:34:12 -0400 Original-Received: from mout.gmx.net ([212.227.15.18]:55077) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mJHOL-0005wc-Ow for emacs-devel@gnu.org; Thu, 26 Aug 2021 11:34:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1629992045; bh=NjWZffFIJeym8+ItWw8dEUbXFWEEd56KxpKoipkvOTY=; h=X-UI-Sender-Class:From:To:Cc:Subject:References:Date:In-Reply-To; b=Oxgb34hPlAOdXJNp/u5g38Tf/3dWJQpBD2RTAxzkfNFFZBrzXK2z4Nran3sdDQdiQ IEgpmXPVr/ENbY0Jqq6fmjcEOXyvcgCTLFP404f0ij3b5bRz6NwmcMsstKB3rnJWWx bcf8V0P3WtRuF3G7EmyBFPVik+StqdPJfUzQpjdE= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Original-Received: from gandalf.gmx.de ([213.220.159.145]) by mail.gmx.net (mrgmx005 [212.227.17.190]) with ESMTPSA (Nemesis) id 1M42nS-1mJHOH1BFw-00029v; Thu, 26 Aug 2021 17:34:05 +0200 In-Reply-To: <871r6n8dw0.fsf@gmail.com> (Jethro Kuan's message of "Sat, 21 Aug 2021 15:18:07 +0800") X-Provags-ID: V03:K1:Bq0hc3rQJ7PTqeMvxFeu4pKxKh9vK6EAz4VhcZ6eoNV53+8eGZk GtwFmlZK6M0j3uBD15MW84P7fv7LVm5+/LxiYF4bCNsbDFPnKDeL+KzifARrXoepxpGm0JY nvMXfxck2TF/XmNAbADc5wsBApz1lFk3q7m6u52Lx8PzXlAiMGYqCT02GAkWh14i7EJS6fm XDJTIQvIz+dNihBxKBiWw== X-UI-Out-Filterresults: notjunk:1;V03:K0:K/4u+dr2esQ=:6TuQ7jyCnlsjLD4D6Yl8e4 xCjitbtNLu6d53S3m04HKxlw9rxhO04+yricG8qVlyDhJlqIxJCp8dh9jO8VgijzJZ7/Bca0p B846X185Ee0wQmAU+voRtDS5q8fVYp6EJ//1iYAcNQ0h/sADp1BGzPgkutuoOe7dmPv/KXKyL YmDIaKhED6DT77ImBzA0bKST7uuvhJMNB8T8iTwJfCMd5awdfRZK0XseXv9iGEXo5rKP6t6SW fCDA3rABD6Uw/wS/JXb1ikGW3wbatABj7FnW9TTrYwFwyEDxp7eJhVHEW1wng++Oi4Qvm+PZT omsdKq5CGQ9htdXpBEcd7MSTkN6ju9llGm8MUJgHUPa6Jz6cSPNkUgRU/1btyR32q7+qqepqk Oo+E+6PV5/uCEsHgxDxgKMm6Asc51QVADy117A5hbqq8ruD+lWW2N8vv8P1fWLazfBVi9UsC6 Dul5sBZ2Af7Wq7PvnxjtKDUY8hNozk8ue0Y+JzXZPHEC8LXeI9CMrzgeWSMefkb82dt2WvNML vtplSK9N8PwIsEaDEWumoqUfTQn4zkDta8MaEC6el/gls2He9QLyczWpm/QmxVXWPiRW3jT17 IH0giLaeMswKjKpsIJOMdPZvdsaL74h+RbctbrUZ7z8MX40d/qwmaqsNJp2gtFYnFbrfasQ7z cCCYUfqAXlp223eOdiG04oMiWTR3jtKfxjhoPEWNqt0ABXHzwyZ5pjx/3aOSp2BXAvVkg2qPd GkyIA14LkUBM3vllNAELb3vxokomB8oBcQV8lLKphZqsjt9f7dWivK7+4BmvtYgTbs2gOTdj Received-SPF: pass client-ip=212.227.15.18; envelope-from=michael.albinus@gmx.de; helo=mout.gmx.net X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:273036 Archived-At: Jethro Kuan writes: > Hello! Hi Jethro, I now found the time to dig into your code. > I'd recently written a package that wraps filenotify somewhat > transparently, providing a similar API to the built-in filenotify but it > watches the directory recursively, spawning and destroying watchers as > appropriately. It was suggested to me that this could be upstreamed to > Emacs. The code is here: > > https://github.com/jethrokuan/filenotify-recursive Which filenotify libraries do you have tested with? There are inotify, kqueue, gfilenotify and w32notify as well as (for remote files) inotifywait and gio. And for gfilenotify/gio, there are different monitors like GFamFileMonitor, GFamDirectoryMonitor GInotifyFileMonitor, GKqueueFileMonitor and GPollFileMonitor. All of them behave slightly different. Some of the native libraries allow even recursive watching of directories, like inotifywait and (I believe) w32notify. This is not implemented in Emacs' library wrappers, but for performance reasons, it might be better to use this low level. > I don't mind doing the work in getting this upstreamed, but I have a few > concerns. Because of how filenotify works, filenotify-recursive uses own > its global variables to keep track of the descriptors. It's best to > think of a way to merge this functionality into filenotify.el itself, > but I can't think of a good way right now. I'm open to any thoughts and > suggestions here. filenotify.el uses file-notify-descriptors for book-keeping: --8<---------------cut here---------------start------------->8--- (file-notify-add-watch "~/.emacs.d" '(change) #'ignore) => (1 . 0) (pp file-notify-descriptors) => #s(hash-table size 65 test equal rehash-size 1.5 rehash-threshold 0.8125 data ((1 . 0) #s(file-notify--watch "/home/albinus/.emacs.d" nil ignore))) --8<---------------cut here---------------end--------------->8--- The hash key is the object returned by file-notify-add-watch, and the value is a struct of type file-notify--watch. In your library, you replace the callback by a closure for all watched directories: --8<---------------cut here---------------start------------->8--- (fnr-add-watch "~/.emacs.d" '(change) #'ignore) => "86fe4882-3d47-43a5-8841-4f5f2a85d558" (pp file-notify-descriptors) => #s(hash-table size 145 test equal rehash-size 1.5 rehash-threshold 0.8125 data ((1 . 0) #s(file-notify--watch "/home/albinus/.emacs.d" nil ignore) (1 . 1) #s(file-notify--watch "/home/albinus/.emacs.d" nil (closure ((callback . ignore) (uuid . "86fe4882-3d47-43a5-8841-4f5f2a85d558") cl-struct-fnr--watch-tags t) (event) (fnr--update-directory-watchers uuid event) (funcall callback event))) (2 . 0) #s(file-notify--watch "/home/albinus/.emacs.d/auto-save-list" nil (closure ((callback . ignore) (uuid . "86fe4882-3d47-43a5-8841-4f5f2a85d558") cl-struct-fnr--watch-tags t) (event) (fnr--update-directory-watchers uuid event) (funcall callback event))) ... (115 . 0) #s(file-notify--watch "/home/albinus/.emacs.d/url/cache/albinus/https/io/cloudimg/heise" nil (closure ((callback . ignore) (uuid . "86fe4882-3d47-43a5-8841-4f5f2a85d558") cl-struct-fnr--watch-tags t) (event) (fnr--update-directory-watchers uuid event) (funcall callback event))))) --8<---------------cut here---------------end--------------->8--- IIUC, this is for calling fnr--update-directory-watchers. And the uuid is used then in your own book-keeping variable fnr-descriptors. However, this is not needed I believe. You could extend the file-notify--watch struct by new slots parent and children, which refer to the watch which has triggered watching the directory, and all the watches which are used for subdirectories. Something like --8<---------------cut here---------------start------------->8--- (cl-defstruct (fnr--watch (:include file-notify--watch)) parent children) --8<---------------cut here---------------end--------------->8--- Such objects would look in file-notify-descriptors as --8<---------------cut here---------------start------------->8--- (2 . 0) #s(file-notify--watch "/home/albinus/.emacs.d/auto-save-list" nil (closure ((callback . ignore) (uuid . "86fe4882-3d47-43a5-8841-4f5f2a85d558") cl-struct-fnr--watch-tags t) (event) (fnr--update-directory-watchers uuid event) (funcall callback event)) ;; parent (1 . 1) ;; children ((10 . 0) (20 . 0) ...)) --8<---------------cut here---------------end--------------->8--- With this, you don't need uuids any longer, you could navigate in file-notify-descriptors directly. Of course, you could also add other slots as in the current definition of fnr--watch. Some few comments on the code: (defun fnr--subdirectories-recursively (dir &optional regexp predicate follow-symlinks) This gives errors: (fnr--subdirectories-recursively "/") => (file-error "Opening directory" "Permission denied" "//boot/efi") I believe "//boot/efi" is a wrong file name. And not readable subdirectories should be (silently) ignored. (unless (or (member file '("./" "../")) (and regexp (string-match regexp file))) There is the defconst directory-files-no-dot-files-regexp which could be used instead. > Jethro Best regards, Michael.