bug#71817: 29.3; Sub-directory handling of ELPA package

[Stefan Monnier via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>]

> Currently as observed, ELPA packages only get their root path added to
> `load-path', but source code in sub-directories will still get
> byte-compiled.  That is, for an ELPA package elpafoo with a nested
> sub-directory of the following structure (installed through package.el):

The recursive compilation is somewhat of an "accident": it was the
easiest to implement (and seemed like a good idea anyway).

The `load-path` behavior is conscious: it's easy for a package to add
more subdirectories to the `load-path` but it would be much harder to
remove undesired ones.  [ And of course, the current behavior is also
the easiest one to implement.  ]

> If this is not yet a policy, I wonder whether this will be the path
> forward for `load-path' handling.

In `elpafoo.el`, include something like:

    ;;;###autoload
    (add-to-list 'load-path
                 (expand-file-name
                  "elpabar" (file-name-directory load-file-name)))

This assumes that you want `elpabar` to be in your `load-path` right
from the start (i.e. that an entry point to your package is in the
`elpabar` subdirectory).  If `elpabar` can only ever be used from code
that's in the `elpafoo` directory, there are other options (such as
`(require 'elpabar/elpabar)` or using an auxiliary `elpafoo-loaddefs.el`
which you load when `elpafoo.el` is loaded, ...

> I see some pros of adding sub-directories recursively,

I don't.  Most of the packages which use subdirectories have a complex
enough layout that some of those directories should not be in
`load-path`: it's better to let them add entries "manually" at the
appropriate time than to try and do it automatically.

The more real problem is that the way `elpafoo-autoloads.el` is created
does *not* scan ELisp files in subdirectories.  The way this is handled
typically in that the ELPA tarball comes with its own
`elpabar/elpabar-autoloads.el` file and `elpafoo.el` then needs to
contain something like

    ;;;###autoload
    (require 'elpabar/elpabar-autoloads)

The main downside here is that the current elpa.gnu.org scripts don't
know how to build such a `elpabar/elpabar-autoloads.el`, so you either
need to store it in the Git (which is ugly since it's a generated file),
or use an ad-hoc `:make` rule.

IMO we should change the ELPA protocol so that the
`elpafoo-autoloads.el` is not created during installation but is instead
included in the tarball, so it can be generated any way we like.


        Stefan