How does Emacs load a non-existent .el file?

How does Emacs load a non-existent .el file?

[Konstantin Kharlamov]

I was just looking whether it's too hard to add an option for reading
from stdin (a very requested feature¹). Long story short, I presume
command line parsing happens in `lisp/startup.el`, function `(command-
line-1)`. That's where the weirdness starts. Since it is an ELisp file,
I figured to avoid recompiling I can move its `.elc` file out of the
way and just do edits to `startup.el` directly. Turned out though, not
only Emacs does not notice these edits, it somehow manages to load the
file when it does not exist!

So e.g. I did a `sudo rm /usr/share/emacs/30.0.50/lisp/startup*`, so no
.el or elc files. But starting up `emacs -Q` and asking it `C-h f
command-line-1` still results in Emacs answering that such function
exists, except the help buffer doesn't have a link to it.

I am thoroughly confused. Any idea what's going on here?

1:
https://superuser.com/questions/31404/how-to-make-emacs-read-buffer-from-stdin-on-start

Re: How does Emacs load a non-existent .el file?

[Eli Zaretskii]

> From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> Date: Fri, 01 Mar 2024 13:52:30 +0300
> 
> I was just looking whether it's too hard to add an option for reading
> from stdin (a very requested feature¹). Long story short, I presume
> command line parsing happens in `lisp/startup.el`, function `(command-
> line-1)`. That's where the weirdness starts. Since it is an ELisp file,
> I figured to avoid recompiling I can move its `.elc` file out of the
> way and just do edits to `startup.el` directly. Turned out though, not
> only Emacs does not notice these edits, it somehow manages to load the
> file when it does not exist!
> 
> So e.g. I did a `sudo rm /usr/share/emacs/30.0.50/lisp/startup*`, so no
> .el or elc files. But starting up `emacs -Q` and asking it `C-h f
> command-line-1` still results in Emacs answering that such function
> exists, except the help buffer doesn't have a link to it.
> 
> I am thoroughly confused. Any idea what's going on here?

Yes: startup.el is preloaded, see lisp/loadup.el.  So if you change
it, you need to rebuild Emacs to let the changes have their effect at
startup.  You can also load startup.el manually into a running
session, and then invoke functions you've changed, but that will only
be useful if what you are changing is not some special behavior that
happens only at startup.

> https://superuser.com/questions/31404/how-to-make-emacs-read-buffer-from-stdin-on-start

There are a few answers there that solve the problem, so I'm not sure
what else are you trying to do, and why.

Re: How does Emacs load a non-existent .el file?

[Konstantin Kharlamov]

On Fri, 2024-03-01 at 14:22 +0200, Eli Zaretskii wrote:
> > From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> > Date: Fri, 01 Mar 2024 13:52:30 +0300
> > 
> > I was just looking whether it's too hard to add an option for
> > reading
> > from stdin (a very requested feature¹). Long story short, I presume
> > command line parsing happens in `lisp/startup.el`, function
> > `(command-
> > line-1)`. That's where the weirdness starts. Since it is an ELisp
> > file,
> > I figured to avoid recompiling I can move its `.elc` file out of
> > the
> > way and just do edits to `startup.el` directly. Turned out though,
> > not
> > only Emacs does not notice these edits, it somehow manages to load
> > the
> > file when it does not exist!
> > 
> > So e.g. I did a `sudo rm /usr/share/emacs/30.0.50/lisp/startup*`,
> > so no
> > .el or elc files. But starting up `emacs -Q` and asking it `C-h f
> > command-line-1` still results in Emacs answering that such function
> > exists, except the help buffer doesn't have a link to it.
> > 
> > I am thoroughly confused. Any idea what's going on here?
> 
> Yes: startup.el is preloaded, see lisp/loadup.el.  So if you change
> it, you need to rebuild Emacs to let the changes have their effect at
> startup.  You can also load startup.el manually into a running
> session, and then invoke functions you've changed, but that will only
> be useful if what you are changing is not some special behavior that
> happens only at startup.

Okay, I see, so the `startup.el` is a dummy file that's not being read.
I wonder if it's too hard to add to `*Help*` buffer mention that
certain functions are not being read from the file where the buffer
claims that they reside.

> > https://superuser.com/questions/31404/how-to-make-emacs-read-buffer-from-stdin-on-start
> 
> There are a few answers there that solve the problem, so I'm not sure
> what else are you trying to do, and why.

I see only one answer that is more or less convenient, which is `emacs
--insert <(echo 123)`. The problem with this one is that `--insert` is
not supported in `emacsclient`; and in the context of the problem such
support would be useless anyway because it would make `emacsclient`
paste the text to a random buffer which is not at all something a user
expects.

The rest of answers require jumping through the hoops with either
writing separate scripts or using a temporary file.

I agree with the opinion of one of the commenters here: this is
absolutely basic feature that most editors have. Not having it in Emacs
is very strange.

Re: How does Emacs load a non-existent .el file?

[Eli Zaretskii]

> From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> Cc: emacs-devel@gnu.org
> Date: Fri, 01 Mar 2024 16:30:07 +0300
> 
> > Yes: startup.el is preloaded, see lisp/loadup.el.  So if you change
> > it, you need to rebuild Emacs to let the changes have their effect at
> > startup.  You can also load startup.el manually into a running
> > session, and then invoke functions you've changed, but that will only
> > be useful if what you are changing is not some special behavior that
> > happens only at startup.
> 
> Okay, I see, so the `startup.el` is a dummy file that's not being read.

It isn't a "dummy file".  It is not being read in your scenario, but
that's not all there is to it.

> I wonder if it's too hard to add to `*Help*` buffer mention that
> certain functions are not being read from the file where the buffer
> claims that they reside.

That claim is correct, so why change it?

> > There are a few answers there that solve the problem, so I'm not sure
> > what else are you trying to do, and why.
> 
> I see only one answer that is more or less convenient, which is `emacs
> --insert <(echo 123)`.

No, there's also an answer using --script, for example.

> I agree with the opinion of one of the commenters here: this is
> absolutely basic feature that most editors have. Not having it in Emacs
> is very strange.

I actually find it strange that someone wants it.  The need was never
explained.  The fact that we don't have it speaks volumes about the
need (or lack thereof).

Of course, if this is your itch, feel free to scratch it, regardless.
Don't mind me.

Re: How does Emacs load a non-existent .el file?

[Konstantin Kharlamov]

On Fri, 2024-03-01 at 15:45 +0200, Eli Zaretskii wrote:
> > From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> > Cc: emacs-devel@gnu.org
> > Date: Fri, 01 Mar 2024 16:30:07 +0300
> > 
> > > Yes: startup.el is preloaded, see lisp/loadup.el.  So if you
> > > change
> > > it, you need to rebuild Emacs to let the changes have their
> > > effect at
> > > startup.  You can also load startup.el manually into a running
> > > session, and then invoke functions you've changed, but that will
> > > only
> > > be useful if what you are changing is not some special behavior
> > > that
> > > happens only at startup.
> > 
> > Okay, I see, so the `startup.el` is a dummy file that's not being
> > read.
> 
> It isn't a "dummy file".  It is not being read in your scenario, but
> that's not all there is to it.
> 
> > I wonder if it's too hard to add to `*Help*` buffer mention that
> > certain functions are not being read from the file where the buffer
> > claims that they reside.
> 
> That claim is correct, so why change it?

What claim? I'm pointing out the lack of information in *Help* buffer
that certain functions are not read from the location where *Help*
claims them to be. That confuses, so I'm just wondering if it's too
hard to modify help function to add a note about such functions (ofc if
this information is globally available, which Idk if it is)

> > > There are a few answers there that solve the problem, so I'm not
> > > sure
> > > what else are you trying to do, and why.
> > 
> > I see only one answer that is more or less convenient, which is
> > `emacs
> > --insert <(echo 123)`.
> 
> No, there's also an answer using --script, for example.

This answer is part of those "rest of them" that I mentioned that
require writing a script 😊

> > I agree with the opinion of one of the commenters here: this is
> > absolutely basic feature that most editors have. Not having it in
> > Emacs
> > is very strange.
> 
> I actually find it strange that someone wants it.  The need was never
> explained.  The fact that we don't have it speaks volumes about the
> need (or lack thereof).

That's not how it works. Emacs is not an ordinary project (90% of open
source projects do not require signing docs with FSF and using a mail-
managed bugtracker for patches), so I'm sure many potential
contributors are getting lost on that part.

Another point: I use reading from stdin from time to time, mostly at
work, but I always used `vim -` for that purpose because with the
amount of plugins I have Emacs startup is not instant. It's only now
that I migrate to using `emacsclient` for quick edits from terminal I
found out that Emacs was never able to read from stdin. And I am an
Emacs contributor. So there you have it: a contributor who uses
"reading from stdin" but who did not even know Emacs can't do that.

Instead what matters is the amount of views and upvotes on the linked
Emacs question: 16k. As someone who often gets stackexchange sites in
search-engine results while figuring out how to do something in Emacs,
I can assure you that 16k views and 47 upvotes on an Emacs-related
question makes it one of the top ones.

> Of course, if this is your itch, feel free to scratch it, regardless.
> Don't mind me.

Thank you 😊 I'll look into that, but won't promise. I don't know yet
what needs to be done (btw, in case anybody knows, pointers are
appreciated), but if it's not too hard I might try adding that.

Re: How does Emacs load a non-existent .el file?

[Eli Zaretskii]

> From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> Cc: emacs-devel@gnu.org
> Date: Fri, 01 Mar 2024 17:15:21 +0300
> 
> > > I wonder if it's too hard to add to `*Help*` buffer mention that
> > > certain functions are not being read from the file where the buffer
> > > claims that they reside.
> > 
> > That claim is correct, so why change it?
> 
> What claim?

The claim you mention above.

> I'm pointing out the lack of information in *Help* buffer
> that certain functions are not read from the location where *Help*
> claims them to be.

The doc string doesn't say where the functions is read from, it says
where the function is defined in the source code.  And it's true that
command-line-1 is defined in startup.el.  Like it's true that, for
example, kill-buffer is defined in buffer.c, although it is built into
Emacs by way of compiling buffer.c into buffer.o, and then linking
buffer.o (together with other *.o files) into the Emacs executable.
the same happens with functions defined in startup.el: the code is
compiled into startup.elc or startup-xxx.eln, and then loaded into the
Emacs binary during dumping (actually, the *.eln files are only
recorded during dumping and then loaded anew at each startup).

Re: How does Emacs load a non-existent .el file?

[Andreas Schwab]

On Mär 01 2024, Konstantin Kharlamov wrote:

> Okay, I see, so the `startup.el` is a dummy file that's not being read.

Emacs won't read random files, only when instructed to do so.  That can
happen due to an autoload or an explict load.  Since startup.el does not
contain any autoloads, only an explit load will cause Emacs to read it.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: How does Emacs load a non-existent .el file?

[Konstantin Kharlamov]

On Fri, 2024-03-01 at 18:53 +0100, Andreas Schwab wrote:
> On Mär 01 2024, Konstantin Kharlamov wrote:
> 
> > Okay, I see, so the `startup.el` is a dummy file that's not being
> > read.
> 
> Emacs won't read random files, only when instructed to do so.  That
> can
> happen due to an autoload or an explict load.  Since startup.el does
> not
> contain any autoloads, only an explit load will cause Emacs to read
> it.

The file is far from random though, it's where the command line
processing is defined. As Eli explained elsewhere in the thread, it's
compiled into something else, so Emacs have to be rebuilt for the file
changes to have effect.

Re: How does Emacs load a non-existent .el file?

[Andreas Schwab]

On Mär 01 2024, Konstantin Kharlamov wrote:

> processing is defined. As Eli explained elsewhere in the thread, it's
> compiled into something else, so Emacs have to be rebuilt for the file
> changes to have effect.

No, you can just load it like any other lisp file.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: How does Emacs load a non-existent .el file?

[Konstantin Kharlamov]

On Fri, 2024-03-01 at 21:48 +0100, Andreas Schwab wrote:
> On Mär 01 2024, Konstantin Kharlamov wrote:
> 
> > processing is defined. As Eli explained elsewhere in the thread,
> > it's
> > compiled into something else, so Emacs have to be rebuilt for the
> > file
> > changes to have effect.
> 
> No, you can just load it like any other lisp file.

Well, it may be possible to override a few functions but certainly not
all of them from the file. The earliest I see it's possible to make
Emacs to load an el file would be done by `emacs -q -l my-override.el`,
but that implies that Emacs has processed command line args, whereas
command-line processing is exactly the part of `startup.el` file that I
want to override.