Autoloads and subdirectories

Autoloads and subdirectories

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 1240 bytes --]

Hi emacs-devel,

What is the proper way to organize the sources of a multi-file package into subdirectories, without breaking autoloads? As far as I can see `update-directory-autoloads` doesn't explore subdirectories recursively (in fact, it is documented not to), and package.el doesn't call it on subdirectories of a package's content directory. And yet, the manual says (emphasis mine):

> Do not include any file named name-autoloads.el. This file is 
> reserved for the package’s autoload definitions (see Packaging 
> Basics). It is created automatically when the package is installed, 
> by searching *all the Lisp files* in the package for autoload magic 
> comments.

This claims that all files in the package are searched (which sounds nice, but seems incorrect).  Presumably this is just imprecise wording, since Packaging Basics does say the following:

> Emacs then searches every Lisp file in the content directory for
> autoload magic comments (see Autoload). These autoload definitions
> are saved to a file named name-autoloads.el in the content directory.

Which brings me back to square one: how do I move parts of a package to a subdirectory, without breaking their autoloads?

Thanks!
Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Autoloads and subdirectories

[Robert Weiner]

[-- Attachment #1: Type: text/plain, Size: 3134 bytes --]

On Sat, May 28, 2016 at 12:25 AM, Clément Pit--Claudel <
clement.pit@gmail.com> wrote:

> Hi emacs-devel,
>
> What is the proper way to organize the sources of a multi-file package
> into subdirectories, without breaking autoloads? As far as I can see
> `update-directory-autoloads` doesn't explore subdirectories recursively
>

You missed my message and solution for this 5 days back, so I have included
the solution here.  Just replace this one function in package.el in the
lisp/emacs-lisp/ subdirectory of your emacs installation.
No one had any comments on it surprisingly.  I have added the patch file
here as well.  Let me know how it goes for you.


(defun package-generate-autoloads (name pkg-dir)
  (let* ((auto-name (format "%s-autoloads.el" name))
         ;;(ignore-name (concat name "-pkg.el"))
         (generated-autoload-file (expand-file-name auto-name pkg-dir))
         ;; Silence `autoload-generate-file-autoloads'.
         (noninteractive inhibit-message)
         (backup-inhibited t)
         (version-control 'never))
    (package-autoload-ensure-default-file generated-autoload-file)
    (apply #'update-directory-autoloads pkg-dir
  (delq nil (mapcar (lambda (f) (and (file-directory-p f)
     (not (file-symlink-p f))
     f))
    (directory-files pkg-dir t "[a-zA-Z].*" nil))))
    (let ((buf (find-buffer-visiting generated-autoload-file)))
      (when buf (kill-buffer buf)))
    auto-name))

-------
*** package-orig.el 2016-05-28 00:54:25.000000000 -0400
--- package.el 2016-05-28 00:54:25.000000000 -0400
***************
*** 916,928 ****
           (backup-inhibited t)
           (version-control 'never))
      (package-autoload-ensure-default-file generated-autoload-file)
!     (update-directory-autoloads pkg-dir)
      (let ((buf (find-buffer-visiting generated-autoload-file)))
        (when buf (kill-buffer buf)))
      auto-name))

  (defun package--make-autoloads-and-stuff (pkg-desc pkg-dir)
   "Generate autoloads, description file, etc.. for PKG-DESC installed at
PKG-DIR."
    (package-generate-autoloads (package-desc-name pkg-desc) pkg-dir)
    (let ((desc-file (expand-file-name (package--description-file pkg-dir)
                                       pkg-dir)))
--- 911,927 ----
           (backup-inhibited t)
           (version-control 'never))
      (package-autoload-ensure-default-file generated-autoload-file)
!     (apply #'update-directory-autoloads pkg-dir
!   (delq nil (mapcar (lambda (f) (and (file-directory-p f)
!      (not (file-symlink-p f))
!      f))
!     (directory-files pkg-dir t "[a-zA-Z].*" nil))))
      (let ((buf (find-buffer-visiting generated-autoload-file)))
        (when buf (kill-buffer buf)))
      auto-name))

  (defun package--make-autoloads-and-stuff (pkg-desc pkg-dir)
   "Generate autoloads, description file, etc. for PKG-DESC installed at
PKG-DIR."
    (package-generate-autoloads (package-desc-name pkg-desc) pkg-dir)
    (let ((desc-file (expand-file-name (package--description-file pkg-dir)
                                       pkg-dir)))
***************

[-- Attachment #2: Type: text/html, Size: 5941 bytes --]

Re: Autoloads and subdirectories

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 3898 bytes --]

On 2016-05-28 01:00, Robert Weiner wrote:
> You missed my message and solution for this 5 days back, so I have
> included the solution here. 

Indeed, I did miss it! Thanks for sharing it.

> Just replace this one function in package.el in the lisp/emacs-lisp/
> subdirectory of your emacs installation. No one had any comments on
> it surprisingly. I have added the patch file here as well. Let me
> know how it goes for you.

It looks like a neat patch. Small question: why do you only accept directories that start with a letter? The regular expression that you pass to directory files seems rather restrictive.

That being said, this is no really the kind of answers that I'm looking for: applying this patch would fix the problem for future version of Emacs, but I'd like to support current versions of Emacs too, and the upcoming release as well.

Ideally, I'd love a solution that works with current Emacsen, and that won't be broken by future upstream fixes. One such solution is to statically generate my autoloads (using code similar to the one you posted) and ship these as part of my package, while adding an autoloaded form to a new file in my content directory that just loads the additional autoloads file that I ship. This isn't very pretty, though.

Thanks for your help!
Clément.

> (defun package-generate-autoloads (name pkg-dir)
>   (let* ((auto-name (format "%s-autoloads.el" name))
>          ;;(ignore-name (concat name "-pkg.el"))
>          (generated-autoload-file (expand-file-name auto-name pkg-dir))
>          ;; Silence `autoload-generate-file-autoloads'.
>          (noninteractive inhibit-message)
>          (backup-inhibited t)
>          (version-control 'never))
>     (package-autoload-ensure-default-file generated-autoload-file)
>     (apply #'update-directory-autoloads pkg-dir
>   (delq nil (mapcar (lambda (f) (and (file-directory-p f)
>      (not (file-symlink-p f))
>      f))
>     (directory-files pkg-dir t "[a-zA-Z].*" nil))))
>     (let ((buf (find-buffer-visiting generated-autoload-file)))
>       (when buf (kill-buffer buf)))
>     auto-name))
> 
> -------
> *** package-orig.el2016-05-28 00:54:25.000000000 -0400
> --- package.el2016-05-28 00:54:25.000000000 -0400
> ***************
> *** 916,928 ****
>            (backup-inhibited t)
>            (version-control 'never))
>       (package-autoload-ensure-default-file generated-autoload-file)
> !     (update-directory-autoloads pkg-dir)
>       (let ((buf (find-buffer-visiting generated-autoload-file)))
>         (when buf (kill-buffer buf)))
>       auto-name))
>   
>   (defun package--make-autoloads-and-stuff (pkg-desc pkg-dir)
>    "Generate autoloads, description file, etc.. for PKG-DESC installed at PKG-DIR."
>     (package-generate-autoloads (package-desc-name pkg-desc) pkg-dir)
>     (let ((desc-file (expand-file-name (package--description-file pkg-dir)
>                                        pkg-dir)))
> --- 911,927 ----
>            (backup-inhibited t)
>            (version-control 'never))
>       (package-autoload-ensure-default-file generated-autoload-file)
> !     (apply #'update-directory-autoloads pkg-dir
> !   (delq nil (mapcar (lambda (f) (and (file-directory-p f)
> !      (not (file-symlink-p f))
> !      f))
> !     (directory-files pkg-dir t "[a-zA-Z].*" nil))))
>       (let ((buf (find-buffer-visiting generated-autoload-file)))
>         (when buf (kill-buffer buf)))
>       auto-name))
>   
>   (defun package--make-autoloads-and-stuff (pkg-desc pkg-dir)
>    "Generate autoloads, description file, etc. for PKG-DESC installed at PKG-DIR."
>     (package-generate-autoloads (package-desc-name pkg-desc) pkg-dir)
>     (let ((desc-file (expand-file-name (package--description-file pkg-dir)
>                                        pkg-dir)))
> ***************
> 
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Autoloads and subdirectories

[Stefan Monnier]

> be broken by future upstream fixes. One such solution is to statically
> generate my autoloads (using code similar to the one you posted) and ship
> these as part of my package, while adding an autoloaded form to a new file
> in my content directory that just loads the additional autoloads file that
> I ship. This isn't very pretty, though.

The other option is to split your package into several packages.


        Stefan

Re: Autoloads and subdirectories

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 2317 bytes --]

On 2016-05-28 14:18, Stefan Monnier wrote:
>> be broken by future upstream fixes. One such solution is to statically
>> generate my autoloads (using code similar to the one you posted) and ship
>> these as part of my package, while adding an autoloaded form to a new file
>> in my content directory that just loads the additional autoloads file that
>> I ship. This isn't very pretty, though.
> 
> The other option is to split your package into several packages.

Interesting, thanks Stefan! The context in which this comes up is ‘realgud’, a rewrite of gud that supports about 15 debuggers out of the box. I don't think it would be too great to split this into 15 packages.

I think I'll push an implementation of my suggestion above. I included a draft below; I'd be very happy to hear comments about it :) IOW, assuming I manually generate autoloads using the following snippet:

    (defun realgud-refresh-subdirectory-autoloads ()
      "Update subdirectory-autoloads.el.
    This is needed because the package.el infrastructure doesn't
    process autoloads in subdirectories; instead we create an
    additional autoloads file of our own, and we load it from an
    autoloaded form."
      (interactive)
      (let ((generated-autoload-file
             (expand-file-name realgud-subdir-autoloads-name realgud-content-dir)))
        (when (file-exists-p generated-autoload-file)
          (delete-file generated-autoload-file))
        (dolist (name (directory-files-recursively realgud-content-dir "" t))
          (when (file-directory-p name)
            (update-directory-autoloads name)))))

Then would it be enough to put the following in a file that lives in the package's root directory?

    ;;;###autoload
    (defconst realgud-subdir-autoloads-name "subdirectory-autoloads.el")

    ;;;###autoload
    (defconst realgud-content-dir
      (file-name-directory
       (or (and load-in-progress load-file-name)
           (bound-and-true-p byte-compile-current-file)
           buffer-file-name)))

    ;;;###autoload
    (with-demoted-errors "Error loading autoloads: %s"
      (load (expand-file-name realgud-subdir-autoloads-name realgud-content-dir) t t))

(btw, is there a more concise way of writing realgud-content-directory?)

Thanks!
Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Autoloads and subdirectories

[Stefan Monnier]

>     ;;;###autoload
>     (defconst realgud-content-dir
>       (file-name-directory
>        (or (and load-in-progress load-file-name)
>            (bound-and-true-p byte-compile-current-file)
>            buffer-file-name)))

Why do you need to check load-in-progress?
And why do you need to fallback on byte-compile-current-file or
buffer-file-name?


        Stefan

Re: Autoloads and subdirectories

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 794 bytes --]

On 2016-05-28 15:07, Stefan Monnier wrote:
>>     ;;;###autoload
>>     (defconst realgud-content-dir
>>       (file-name-directory
>>        (or (and load-in-progress load-file-name)
>>            (bound-and-true-p byte-compile-current-file)
>>            buffer-file-name)))
> 
> Why do you need to check load-in-progress?

The docs of load-file-name don't say what this variable becomes when outside of load. It is always nil when load-in-progress is nil?

> And why do you need to fallback on byte-compile-current-file or
> buffer-file-name?

I'd like this code to work when the form is evaluated with C-M-x in an Emacs buffer. Since this isn't in an eval-when-compile form, though, I guess it would work to remove the byte-compile-current-file part.

Thanks!
Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Autoloads and subdirectories

[Stefan Monnier]

> The docs of load-file-name don't say what this variable becomes when outside
> of load. It is always nil when load-in-progress is nil?

I believe so, yes.

>> And why do you need to fallback on byte-compile-current-file or
>> buffer-file-name?
> I'd like this code to work when the form is evaluated with C-M-x in an Emacs
> buffer.  Since this isn't in an eval-when-compile form, though, I guess it
> would work to remove the byte-compile-current-file part.

For C-M-x I guess you do need buffer-file-name.  For M-x eval-buffer
you could use

    ;;;###autoload (defconst realgud-subdir-autoloads-name "subdirectory-autoloads.el")
    ;;;###autoload (defconst realgud-content-dir)
    ;;;###autoload   (file-name-directory load-file-name))
    ;;;###autoload (with-demoted-errors "Error loading autoloads: %s"
    ;;;###autoload   (load (expand-file-name realgud-subdir-autoloads-name realgud-content-dir) t t))


-- Stefan

Re: Autoloads and subdirectories

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 1030 bytes --]

On 2016-05-29 11:01, Stefan Monnier wrote:
>> The docs of load-file-name don't say what this variable becomes when outside
>> of load. It is always nil when load-in-progress is nil?
> 
> I believe so, yes.
> 
>>> And why do you need to fallback on byte-compile-current-file or
>>> buffer-file-name?
>> I'd like this code to work when the form is evaluated with C-M-x in an Emacs
>> buffer.  Since this isn't in an eval-when-compile form, though, I guess it
>> would work to remove the byte-compile-current-file part.
> 
> For C-M-x I guess you do need buffer-file-name.  For M-x eval-buffer
> you could use
> 
>     ;;;###autoload (defconst realgud-subdir-autoloads-name "subdirectory-autoloads.el")
>     ;;;###autoload (defconst realgud-content-dir)
>     ;;;###autoload   (file-name-directory load-file-name))
>     ;;;###autoload (with-demoted-errors "Error loading autoloads: %s"
>     ;;;###autoload   (load (expand-file-name realgud-subdir-autoloads-name realgud-content-dir) t t))

Got it, thanks!


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]