Dependency graph for Emacs Lisp files

Dependency graph for Emacs Lisp files

[Lars Ingebrigtsen]

Currently, saying "make" in Emacs doesn't recompile .el files that
depend on other files.  This is generally not a catastrophe, because it
only makes a difference when you change a macro/defsubst (and sometimes
a variable).

But when it does matter, it's annoying that you have to do a "make
bootstrap" to get to a working state.

Making a full dependency graph is difficult, of course, due to all the
various methods files depend on each other (require, load, autoload),
but for the things we're interested in to reduce breakages
(macros/defsubst/defvar), we get 97% of the way there by just looking at
`require'.

So would it make a sense to add something that, along the lines of the
stuff that gathers autoloads, creates a dependency.el file (by just
doing `re-search-forward "(require '"' on all the files), and that would
use this to delete .elc files before compiling the lisp directory, based
on whether something "up" in the dependency chain has changed?

(I'd guess there are some complications here because the resulting graph
can be cyclical, but I don't think that's something that can't be worked
around...)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Dependency graph for Emacs Lisp files

[Phillip Lord]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Making a full dependency graph is difficult, of course, due to all the
> various methods files depend on each other (require, load, autoload),
> but for the things we're interested in to reduce breakages
> (macros/defsubst/defvar), we get 97% of the way there by just looking at
> `require'.
>
> So would it make a sense to add something that, along the lines of the
> stuff that gathers autoloads, creates a dependency.el file (by just
> doing `re-search-forward "(require '"' on all the files), and that would
> use this to delete .elc files before compiling the lisp directory, based
> on whether something "up" in the dependency chain has changed?
>
> (I'd guess there are some complications here because the resulting graph
> can be cyclical, but I don't think that's something that can't be worked
> around...)


`make check` currently uses the a fudge based on name matching between
tests and source files, but it missed any other form of
dependency. Having something better would be nice. I think autoloads
dependencies might be more thatn 3% of the problem, though, but if you
add a mechanism based on `require` then an autoload dependency could be
added later.

Rather than creating "dependency.el", though, why not create an
includeable Makefile? That way you might need to delete anything, just
let Make to it's dependency thing.

Phil

Re: Dependency graph for Emacs Lisp files

[Lars Ingebrigtsen]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> `make check` currently uses the a fudge based on name matching between
> tests and source files, but it missed any other form of
> dependency. Having something better would be nice. I think autoloads
> dependencies might be more thatn 3% of the problem, though, but if you
> add a mechanism based on `require` then an autoload dependency could be
> added later.

autoloads generally don't lead to compilation breakages, do they?  You
can't autoload a macro meaningfully, or a variable...

> Rather than creating "dependency.el", though, why not create an
> includeable Makefile? That way you might need to delete anything, just
> let Make to it's dependency thing.

I am no make expert at all, so I don't quite know what that would look
like.

My thought was that if file A.el depended on file B.el, and B.el has
changed, then there would just be a step in the compilation process that
deleted both A.elc and B.elc, and then the rest of the process would
continue on as before.

(We need to delete B.elc because otherwise `(require B)' in A.el would
load the old B.elc, and A.el might be byte-compiled before B.el.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Dependency graph for Emacs Lisp files

[Noam Postavsky]

On Thu, 1 Aug 2019 at 08:16, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> can't autoload a macro meaningfully, or a variable...

You can definitely autoload a macro. See pcase.

Re: Dependency graph for Emacs Lisp files

[Lars Ingebrigtsen]

Noam Postavsky <npostavs@gmail.com> writes:

> On Thu, 1 Aug 2019 at 08:16, Lars Ingebrigtsen <larsi@gnus.org> wrote:
>> can't autoload a macro meaningfully, or a variable...
>
> You can definitely autoload a macro. See pcase.

Oh, you can?  And it's then loaded while compiling the files?  Why
aren't we slapping ;;;###autoload on stuff like `when-let' in subr-x to
avoid having (eval-when-compile (require 'subr-x)) in basically every
single file in the tree?  (I'm just exaggerating slightly.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Dependency graph for Emacs Lisp files

[Phillip Lord]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
>
>> `make check` currently uses the a fudge based on name matching between
>> tests and source files, but it missed any other form of
>> dependency. Having something better would be nice. I think autoloads
>> dependencies might be more thatn 3% of the problem, though, but if you
>> add a mechanism based on `require` then an autoload dependency could be
>> added later.
>
> autoloads generally don't lead to compilation breakages, do they?  You
> can't autoload a macro meaningfully, or a variable...

Lots of dependencies happen by side-effect, though.


(defun fred-function-1())
(defmacro fred-macro(&rest body))
(provide 'fred)


(autoload "fred-function-1" "fred")
(defun george-function-1()
   (fred-function-1)
   (fred-macro (george-function-2)))
(provide 'george)


Besides, if this is to work for more than compilation (i.e. testing)
it's not just about compilation breakage. Functional breakage also
counts.


>> Rather than creating "dependency.el", though, why not create an
>> includeable Makefile? That way you might need to delete anything, just
>> let Make to it's dependency thing.
>
> I am no make expert at all, so I don't quite know what that would look
> like.
>
> My thought was that if file A.el depended on file B.el, and B.el has
> changed, then there would just be a step in the compilation process that
> deleted both A.elc and B.elc, and then the rest of the process would
> continue on as before.
>
> (We need to delete B.elc because otherwise `(require B)' in A.el would
> load the old B.elc, and A.el might be byte-compiled before B.el.)


a.elc: a.el b.elc
b.elc: b.el


Changes to b.el would now force a.elc recompilation. If this were in
place then the current name based fudge for tests should work.

a-test.log: a.test.el a.elc


Of course, I have put this forward in simple terms -- Emacs makefile are
in multiple directories, with dependencies between directories, with a
complicated bootstrap, so the reality would be rather more brutal.

Phil

Re: Dependency graph for Emacs Lisp files

[Lars Ingebrigtsen]

phillip.lord@russet.org.uk (Phillip Lord) writes:

>> (We need to delete B.elc because otherwise `(require B)' in A.el would
>> load the old B.elc, and A.el might be byte-compiled before B.el.)
>
> a.elc: a.el b.elc
> b.elc: b.el
>
> Changes to b.el would now force a.elc recompilation. If this were in
> place then the current name based fudge for tests should work.

But triggering recompilation isn't enough: b.elc has to be deleted,
otherwise compiling a.el will just load the old b.elc.

> Of course, I have put this forward in simple terms -- Emacs makefile are
> in multiple directories, with dependencies between directories, with a
> complicated bootstrap, so the reality would be rather more brutal.

Right.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Dependency graph for Emacs Lisp files

[Andreas Schwab]

On Aug 01 2019, Lars Ingebrigtsen <larsi@gnus.org> wrote:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
>
>>> (We need to delete B.elc because otherwise `(require B)' in A.el would
>>> load the old B.elc, and A.el might be byte-compiled before B.el.)
>>
>> a.elc: a.el b.elc
>> b.elc: b.el
>>
>> Changes to b.el would now force a.elc recompilation. If this were in
>> place then the current name based fudge for tests should work.
>
> But triggering recompilation isn't enough: b.elc has to be deleted,
> otherwise compiling a.el will just load the old b.elc.

Unless load-prefer-newer is non-nil.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Dependency graph for Emacs Lisp files

[Stefan Monnier]

>> a.elc: a.el b.elc
>> b.elc: b.el
>>
>> Changes to b.el would now force a.elc recompilation. If this were in
>> place then the current name based fudge for tests should work.
>
> But triggering recompilation isn't enough: b.elc has to be deleted,
> otherwise compiling a.el will just load the old b.elc.

I don't think so: the dependencies mean that a.el will not be recompiled
until b.el has been recompiled.


        Stefan

Re: Dependency graph for Emacs Lisp files

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> But triggering recompilation isn't enough: b.elc has to be deleted,
>> otherwise compiling a.el will just load the old b.elc.
>
> I don't think so: the dependencies mean that a.el will not be recompiled
> until b.el has been recompiled.

But we have cyclical dependencies...

Just removing the .elc files seem like a pretty easy way to do what we
want to do...  Are there any drawbacks here?

(Or use `load-prefer-newer', as was suggested.  Same effect here,
really.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Dependency graph for Emacs Lisp files

[Stefan Monnier]

> So would it make a sense to add something that, along the lines of the
> stuff that gathers autoloads, creates a dependency.el file (by just
> doing `re-search-forward "(require '"' on all the files), and that would
> use this to delete .elc files before compiling the lisp directory, based
> on whether something "up" in the dependency chain has changed?

Many years ago, I wrote a quick scripts which added dependency rules to
the lisp/Makefile based on a quick search for "^(require ".

The result was rather disappointing:
1- lots of cycles
2- incomplete dependencies (mostly due to autoloads).
3- too many dependencies, resulting in excessive recompilation for
   my taste

I believe (1) has improved over the years, but I'm sure there's still
a fair bit of that.  Make didn't like them back then, and I don't think
it likes them more now.  So we'd need some way to fix that (I guess we
could add some annotations in the source code to "ignore" some
dependencies).

For (2), I think a better option is to generate the dependencies as
a side-effect of compilation using after-load-functions: we should be
able to collect reliably all the dependencies this way, regardless of
how the files get loaded.

Regarding (3), I'm not sure how to deal with that.  The problem is all
the (require 'foo) which are only really needed at run time.  There are
various ways we could try and tackle the problem, but they all seem to
be way too invasive.


        Stefan

Re: Dependency graph for Emacs Lisp files

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> The result was rather disappointing:
> 1- lots of cycles
> 2- incomplete dependencies (mostly due to autoloads).
> 3- too many dependencies, resulting in excessive recompilation for
>    my taste

Very excessive recompilation wouldn't be nice...

> I believe (1) has improved over the years, but I'm sure there's still
> a fair bit of that.  Make didn't like them back then, and I don't think
> it likes them more now.  So we'd need some way to fix that (I guess we
> could add some annotations in the source code to "ignore" some
> dependencies).

But are cycles a problem?  You just delete all the .elc files in the
cycle and then stop.

> For (2), I think a better option is to generate the dependencies as
> a side-effect of compilation using after-load-functions: we should be
> able to collect reliably all the dependencies this way, regardless of
> how the files get loaded.

Hm...  interesting.  That makes me wonder whether the same thing could
be used to collect the requires, too?

> Regarding (3), I'm not sure how to deal with that.  The problem is all
> the (require 'foo) which are only really needed at run time.  There are
> various ways we could try and tackle the problem, but they all seem to
> be way too invasive.

Speaking of excessive recompilation, it seems like all the .c files are
recompiled just about whenever anybody does something trivial in a .h
file, but I guess that's unavoidable.  (I get the feeling that every day
when I do a "git pull" I get a full recompilation of the C bits in
Emacs, and then a couple of .el files, but perhaps that's not completely
accurate.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Dependency graph for Emacs Lisp files

[Stefan Monnier]

> But are cycles a problem?  You just delete all the .elc files in the
> cycle and then stop.

That's only if you want to do it yourself: instead I want GNU Make to do
the job of traversing the dependencies.

>> For (2), I think a better option is to generate the dependencies as
>> a side-effect of compilation using after-load-functions: we should be
>> able to collect reliably all the dependencies this way, regardless of
>> how the files get loaded.
>
> Hm...  interesting.  That makes me wonder whether the same thing could
> be used to collect the requires, too?

This would work "regardless of how the files get loaded", so of course
it would handle `require`s as well.

> Speaking of excessive recompilation, it seems like all the .c files are
> recompiled just about whenever anybody does something trivial in a .h
> file, but I guess that's unavoidable.  (I get the feeling that every day
> when I do a "git pull" I get a full recompilation of the C bits in
> Emacs, and then a couple of .el files, but perhaps that's not completely
> accurate.)

That's about right: we have a few .h files that are often changed and that
are included in many/most other files.  E.g. globals.h, config.h, and
lisp.h are probably the main culprits.  Not sure how to improve
the situation.


        Stefan