Canonical location for emacs-version string in source tree?

Canonical location for emacs-version string in source tree?

[Jeff Kowalczyk]

Gentoo GNU/Linux's packaging of emacs bzr trunk recently adapted to a bzr
changeset that removed one of the several copies of emacs-version (e.g.
"24.0.50") found in the source tree [1].

Can one of these locations in the source tree  (i.e. emacs.c or configure.in) be
designated as the canonical location for packagers to find the version string?

Thanks,
Jeff Kowalczyk

[1] http://bugs.gentoo.org/show_bug.cgi?id=320021

Re: Canonical location for emacs-version string in source tree?

[Ken Raeburn]

On May 20, 2010, at 12:57, Jeff Kowalczyk wrote:
> Gentoo GNU/Linux's packaging of emacs bzr trunk recently adapted to a bzr
> changeset that removed one of the several copies of emacs-version (e.g.
> "24.0.50") found in the source tree [1].

Well, it moved it, from version.el to emacs.c.

> Can one of these locations in the source tree  (i.e. emacs.c or configure.in) be
> designated as the canonical location for packagers to find the version string?

I thought about moving in that direction a bit when making my change, but the C code needs to get the value easily at compile time on both UNIX and Windows platforms, which rules out using autoconf's PACKAGE_VERSION (unless we can generate PACKAGE_VERSION on Windows) and using non-trivial scripts to extract it from configure.in or elsewhere (unless someone wants to write and debug both versions).

I'm not opposed to someone doing the work; I just couldn't effectively do it myself at the time, and didn't need it for my work.  And for Emacs maintenance itself, there are scripts and lisp packages (not distributed in the Emacs package) that know where the version strings live and how to update them consistently, so there hasn't been a big demand to reduce them all to one master source location.

If you need to pick one place, then I think for now emacs.c is the right place.

Ken

Re: Canonical location for emacs-version string in source tree?

[Eli Zaretskii]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Thu, 20 May 2010 13:12:29 -0400
> Cc: emacs-devel@gnu.org
> 
> If you need to pick one place, then I think for now emacs.c is the right place.

Until someone moves it again, that is...

Why was it moved, anyway?  I don't remember any discussions of that,
apologies if I missed something.  The string was in version.el since
forever.  The move subtly broke the MS-DOS port, btw, which I could
have prevented if I just knew about it coming, which I didn't until
now.

<Shrug> I guess it's a lost battle, anyway.

Re: Canonical location for emacs-version string in source tree?

[Stefan Monnier]

> I'm not opposed to someone doing the work; I just couldn't effectively
> do it myself at the time, and didn't need it for my work.  And for
> Emacs maintenance itself, there are scripts and lisp packages (not
> distributed in the Emacs package) that know where the version strings
> live and how to update them consistently, so there hasn't been a big
> demand to reduce them all to one master source location.

Actually, I'd be happy to see it reduced to just a single location,
because it's the second biggest source of conflicts when merging the
emacs-23 branch into trunk (tho it's only when the merge includes
a change of revision obviously, but that's pretty frequent during the
pretest period).


        Stefan

Re: Canonical location for emacs-version string in source tree?

[Ken Raeburn]

On May 20, 2010, at 14:01, Eli Zaretskii wrote:
>> From: Ken Raeburn <raeburn@raeburn.org>
>> Date: Thu, 20 May 2010 13:12:29 -0400
>> Cc: emacs-devel@gnu.org
>> 
>> If you need to pick one place, then I think for now emacs.c is the right place.
> 
> Until someone moves it again, that is...

Well, yeah.

> Why was it moved, anyway?  I don't remember any discussions of that,
> apologies if I missed something.  The string was in version.el since
> forever.  The move subtly broke the MS-DOS port, btw, which I could
> have prevented if I just knew about it coming, which I didn't until
> now.

"emacs --version" would print out the version if it was set, otherwise ignore the option; it does this early in main(), before Lisp code is loaded.  The "make install" rule always runs "emacs --version" to figure out the DOC file name.  If you build with CANNOT_DUMP, the emacs binary that gets run hasn't had the basic Lisp code pre-loaded into it.  So "emacs --version" pops up a window.

Making the C code aware of the version number seemed like the easiest way to deal with it.  I'm sorry about the problem with the MS-DOS build; I took this approach because I thought that it (unlike trying to generate all version-string appearances from one canonical source) would be a simple, relatively minor change with little impact on the DOS/Windows builds.  (Using PACKAGE_VERSION presumes you're using the autoconf-based configure script.  Extracting it from anywhere to make something visible to C requires running some external program like grep or sed.)  Apparently I underestimated the impact anyways.  I did discuss it a bit with Chong Yidong because he had made some comments a while back in response to my talking about the "--version" problem, but I didn't bring up my planned change on the list.

It seems like half the seemingly-minor tweaks I do relating to the build system wind up stomping on the DOS/Windows builds in some subtle way I didn't expect.  *sigh*  Next time I'll try to make sure you're in the loop....  I don't have any more such changes planned right now though.

Ken

Re: Canonical location for emacs-version string in source tree?

[Ulrich Mueller]

>>>>> On Thu, 20 May 2010, Eli Zaretskii wrote:

> Until someone moves it again, that is...

> Why was it moved, anyway? I don't remember any discussions of that,
> apologies if I missed something. The string was in version.el since
> forever.

It was there in Emacs 16.56 already. If the location is changing only
every 25 years, then I won't complain. ;-)

Ulrich

Re: Canonical location for emacs-version string in source tree?

[Eli Zaretskii]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Thu, 20 May 2010 21:39:53 -0400
> Cc: "Emacs Dev \[emacs-devel\]" <emacs-devel@gnu.org>
> 
> Making the C code aware of the version number seemed like the
> easiest way to deal with it.  I'm sorry about the problem with the
> MS-DOS build; I took this approach because I thought that it (unlike
> trying to generate all version-string appearances from one canonical
> source) would be a simple, relatively minor change with little
> impact on the DOS/Windows builds.

It _is_ with little impact, but the DOS build looks into version.el to
set $(version) in the top-level Makefile, which all the sub-Make's
then use.  You probably didn't think of grepping the files in the
msdos/ subdirectory when you looked for the effects of that change.  I
will fix this when I have time.

I don't expect anyone to know all those factoids, much less being an
expert on how the DOS port builds and works.  Just talking before
changes like that are made is good enough.

Re: Canonical location for emacs-version string in source tree?

[Stefan Monnier]

> "emacs --version" would print out the version if it was set, otherwise
> ignore the option; it does this early in main(), before Lisp code is
> loaded.  The "make install" rule always runs "emacs --version" to
> figure out the DOC file name.  If you build with CANNOT_DUMP, the
> emacs binary that gets run hasn't had the basic Lisp code pre-loaded
> into it.  So "emacs --version" pops up a window.

A simple fix is to get rid of the DOC-NN.NN.NN madness and just use
"DOC" as the file name.  Any objection?


        Stefan

Re: Canonical location for emacs-version string in source tree?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  "Emacs Dev \[emacs-devel\]" <emacs-devel@gnu.org>
> Date: Fri, 21 May 2010 16:19:32 -0400
> 
> > "emacs --version" would print out the version if it was set, otherwise
> > ignore the option; it does this early in main(), before Lisp code is
> > loaded.  The "make install" rule always runs "emacs --version" to
> > figure out the DOC file name.  If you build with CANNOT_DUMP, the
> > emacs binary that gets run hasn't had the basic Lisp code pre-loaded
> > into it.  So "emacs --version" pops up a window.
> 
> A simple fix is to get rid of the DOC-NN.NN.NN madness and just use
> "DOC" as the file name.  Any objection?

Would the emacs-NN.NN.NN madness go with it as well?

Re: Canonical location for emacs-version string in source tree?

[Stefan Monnier]

>> > "emacs --version" would print out the version if it was set, otherwise
>> > ignore the option; it does this early in main(), before Lisp code is
>> > loaded.  The "make install" rule always runs "emacs --version" to
>> > figure out the DOC file name.  If you build with CANNOT_DUMP, the
>> > emacs binary that gets run hasn't had the basic Lisp code pre-loaded
>> > into it.  So "emacs --version" pops up a window.
>> A simple fix is to get rid of the DOC-NN.NN.NN madness and just use
>> "DOC" as the file name.  Any objection?
> Would the emacs-NN.NN.NN madness go with it as well?

I'd be in favor of it, tho I don't really care.


        Stefan

Re: Canonical location for emacs-version string in source tree?

[Ken Raeburn]

On May 21, 2010, at 20:54, Stefan Monnier wrote:
>>> A simple fix is to get rid of the DOC-NN.NN.NN madness and just use
>>> "DOC" as the file name.  Any objection?
>> Would the emacs-NN.NN.NN madness go with it as well?
> 
> I'd be in favor of it, tho I don't really care.

I thought about that, and was trying to guess what reasons there might be for keeping it as is.  All I could come up with was using one emacs binary while installing a newly-compiled one into the same tree.  If you change the doc strings or order of functions in the emacs C sources, or the pre-loaded Lisp sources, the offsets in the DOC file may change, so the values loaded into one Emacs binary correspond to its matching DOC file.  (Though there are attempts to keep them consistent by running make-docfile on all the stuff you *might* be loading depending on the configuration, via ${SOME_MACHINE_OBJECTS}.)

Though, I've also been wondering if it would be worthwhile to compile those doc strings into read-only data (shared between processes) in the Emacs binary itself, and scrap the DOC file altogether.  Through the wonder of gcc 'section' attributes, we could (depending on the configuration) even lump all the documentation together on disk pages that don't even need to get loaded unless you actually reference them.  Without the emacs/DOC file coordination to worry about, simply installing a new binary after deleting the old one or renaming it to "emacs.old" should be safer on most systems, even if you're running the old one.

Ken

Re: Canonical location for emacs-version string in source tree?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 21 May 2010 20:54:00 -0400
> Cc: raeburn@raeburn.org, emacs-devel@gnu.org
> 
> >> > "emacs --version" would print out the version if it was set, otherwise
> >> > ignore the option; it does this early in main(), before Lisp code is
> >> > loaded.  The "make install" rule always runs "emacs --version" to
> >> > figure out the DOC file name.  If you build with CANNOT_DUMP, the
> >> > emacs binary that gets run hasn't had the basic Lisp code pre-loaded
> >> > into it.  So "emacs --version" pops up a window.
> >> A simple fix is to get rid of the DOC-NN.NN.NN madness and just use
> >> "DOC" as the file name.  Any objection?
> > Would the emacs-NN.NN.NN madness go with it as well?
> 
> I'd be in favor of it, tho I don't really care.

Without its companion DOC-NN.NN.NN, emacs-NN.NN.NN is not really fully
operational.

Re: Canonical location for emacs-version string in source tree?

[Stefan Monnier]

> while installing a newly-compiled one into the same tree.  If you change
> the doc strings or order of functions in the emacs C sources, or the
> pre-loaded Lisp sources, the offsets in the DOC file may change, so the
> values loaded into one Emacs binary correspond to its matching DOC file.

That's OK: there's code to detect such "out of date offsets" that then
calls Snarf-documentation again to reset the offsets.
It doesn't work in every possible case, but I've been using such a setup
for many years now and have seen a wrong docstring only maybe twice.


        Stefan

Re: Canonical location for emacs-version string in source tree?

[Richard Stallman]

    > A simple fix is to get rid of the DOC-NN.NN.NN madness and just use
    > "DOC" as the file name.  Any objection?

    Would the emacs-NN.NN.NN madness go with it as well?

Using version numbers in the DOC file name (and elsewhere) makes it
possible for multiple Emacs versions to coexist.  And putting the version
numbers in the executables is useful for this too.  You can install
the executable under whatever name you like.

Re: Canonical location for emacs-version string in source tree?

[Stefan Monnier]

>> A simple fix is to get rid of the DOC-NN.NN.NN madness and just use
>> "DOC" as the file name.  Any objection?
>     Would the emacs-NN.NN.NN madness go with it as well?

> Using version numbers in the DOC file name (and elsewhere) makes it
> possible for multiple Emacs versions to coexist.

I have several co-existing `emacs' executables here, and they coexist
just fine with a single DOC file.  They just all show the same
docstrings, which I wouldn't even consider as an issue.


        Stefan

Re: Canonical location for emacs-version string in source tree?

[Lennart Borgman]

On Sun, May 23, 2010 at 3:19 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>>> A simple fix is to get rid of the DOC-NN.NN.NN madness and just use
>>> "DOC" as the file name.  Any objection?
>>     Would the emacs-NN.NN.NN madness go with it as well?
>
>> Using version numbers in the DOC file name (and elsewhere) makes it
>> possible for multiple Emacs versions to coexist.
>
> I have several co-existing `emacs' executables here, and they coexist
> just fine with a single DOC file.  They just all show the same
> docstrings, which I wouldn't even consider as an issue.

If someone think that is problematic then perhaps Emacs could look for
DOC-NN.NN.NN first and use that if it happens to exist (i.e. is copied
by the user).

RE: Canonical location for emacs-version string in source tree?

[Drew Adams]

> >> A simple fix is to get rid of the DOC-NN.NN.NN madness and just use
> >> "DOC" as the file name.  Any objection?
> >     Would the emacs-NN.NN.NN madness go with it as well?
> 
> > Using version numbers in the DOC file name (and elsewhere) makes it
> > possible for multiple Emacs versions to coexist.
> 
> I have several co-existing `emacs' executables here, and they coexist
> just fine with a single DOC file.  They just all show the same
> docstrings, which I wouldn't even consider as an issue.

I haven't been following this thread, so excuse me if I misunderstand.

I have multiple Emacs versions installed (multiple binaries), and I expect each
of them to faithfully have its own set of doc strings, that is, specific to the
particular build.  Trying to deal with doc bugs and improvements would be
problematic without this distinction.

That concern might be irrelevant to having "a single DOC file" in the source
tree - dunno.  I'm not concerned about the build process, but I would not want
multiple binaries of Emacs on my machine to show the same set of doc strings.

Re: Canonical location for emacs-version string in source tree?

[Stefan Monnier]

> I have multiple Emacs versions installed (multiple binaries), and
> I expect each of them to faithfully have its own set of doc strings,
> that is, specific to the particular build.  Trying to deal with doc
> bugs and improvements would be problematic without this distinction.

Please explain more precisely your use case.

> That concern might be irrelevant to having "a single DOC file" in the
> source tree - dunno.  I'm not concerned about the build process, but
> I would not want multiple binaries of Emacs on my machine to show the
> same set of doc strings.

They already show the same docstrings for all non-preloaded files.


        Stefan

RE: Canonical location for emacs-version string in source tree?

[Drew Adams]

> > I have multiple Emacs versions installed (multiple binaries), and
> > I expect each of them to faithfully have its own set of doc strings,
> > that is, specific to the particular build.  Trying to deal with doc
> > bugs and improvements would be problematic without this distinction.
> 
> Please explain more precisely your use case.

? 

The doc a user sees for a given version should be up-to-date and reflect that
version faithfully. That's all.

> > That concern might be irrelevant to having "a single DOC 
> > file" in the source tree - dunno.  I'm not concerned about
> > the build process, but I would not want multiple binaries
> > of Emacs on my machine to show the same set of doc strings.
> 
> They already show the same docstrings for all non-preloaded files.

If a doc string has changed between versions then the versions should not show
the same doc string (which is the case currently). That's appropriate: the doc
strings for a given version should be up-to-date, reflecting that version.

As I say, perhaps the proposed change will not affect that. Just asking.

Re: Canonical location for emacs-version string in source tree?

[Ken Raeburn]

On May 23, 2010, at 10:16, Drew Adams wrote:
> I have multiple Emacs versions installed (multiple binaries), and I expect each
> of them to faithfully have its own set of doc strings, that is, specific to the
> particular build.  Trying to deal with doc bugs and improvements would be
> problematic without this distinction.

The release version number is coded into the directory name, so it's only different builds of the same release that would have the DOC files winding up in the same directory.  And the build process jumps through some hoops to ensure that the file is consistently built from the same input files, regardless of build options.  And as Stefan has pointed out, there's code to detect when the offsets are wrong (e.g., if functions got added, removed, or reordered).  So if we didn't use filenames varying for each build, the doc strings would actually have to change between builds for the result to be wrong, which I think pretty much confines it to developers' builds.  And even then, it probably doesn't matter much unless the semantics of a function change; having an old binary see typographical or grammatical fixes from a newer build may be less objectionable.

(The Windows build appears to be different on the naming score, with DOC and DOC-X instead of DOC-nn.nn.nn; I don't know about the other stuff.)

Still, I'm inclined to think it's better that each version uses the doc strings it was built and installed with.

Possibly more important is that multiple versions installed under the same prefix with the same release version number will share lisp files, even if the lisp files have changed.  So I'm a bit skeptical of the utility of keeping the binaries separate from one install to the next.

> That concern might be irrelevant to having "a single DOC file" in the source
> tree - dunno.  I'm not concerned about the build process, but I would not want
> multiple binaries of Emacs on my machine to show the same set of doc strings.

Ken

Re: Canonical location for emacs-version string in source tree?

[Stefan Monnier]

>> > I have multiple Emacs versions installed (multiple binaries), and
>> > I expect each of them to faithfully have its own set of doc strings,
>> > that is, specific to the particular build.  Trying to deal with doc
>> > bugs and improvements would be problematic without this distinction.
>> Please explain more precisely your use case.

> ?
> The doc a user sees for a given version should be up-to-date and reflect that
> version faithfully. That's all.

That's neither precise, nor is it a use case.

In any case, IIUC your use-case (which you fail to describe) is
unaffected because you don't use several binary executables that share
the same source tree, same DOC file and same lisp tree.


        Stefan

Re: Canonical location for emacs-version string in source tree?

[Stefan Monnier]

> Still, I'm inclined to think it's better that each version uses the
> doc strings it was built and installed with.

Even tho noone cares that it will still use "the latest .elc file
around" for the non-prebuilt packages?  And "the docstring from the
latest .elc file" for non-preloaded packages that have been modified
and recompiled since you loaded that package?


        Stefan

RE: Canonical location for emacs-version string in source tree?

[Drew Adams]

> > I have multiple Emacs versions installed (multiple 
> > binaries), and I expect each of them to faithfully have
> > its own set of doc strings, that is, specific to the
> > particular build.  Trying to deal with doc bugs and 
> > improvements would be problematic without this distinction.
> 
> The release version number is coded into the directory name, 
> so it's only different builds of the same release that would 
> have the DOC files winding up in the same directory.  And the 
> build process jumps through some hoops to ensure that the 
> file is consistently built from the same input files, 
> regardless of build options.  And as Stefan has pointed out, 
> there's code to detect when the offsets are wrong (e.g., if 
> functions got added, removed, or reordered).  So if we didn't 
> use filenames varying for each build, the doc strings would 
> actually have to change between builds for the result to be 
> wrong, which I think pretty much confines it to developers' 
> builds.  And even then, it probably doesn't matter much 
> unless the semantics of a function change; having an old 
> binary see typographical or grammatical fixes from a newer 
> build may be less objectionable.
> 
> (The Windows build appears to be different on the naming 
> score, with DOC and DOC-X instead of DOC-nn.nn.nn; I don't 
> know about the other stuff.)
> 
> Still, I'm inclined to think it's better that each version 
> uses the doc strings it was built and installed with.
> 
> Possibly more important is that multiple versions installed 
> under the same prefix with the same release version number 
> will share lisp files, even if the lisp files have changed.  
> So I'm a bit skeptical of the utility of keeping the binaries 
> separate from one install to the next.
> 
> > That concern might be irrelevant to having "a single DOC 
> > file" in the source tree - dunno.  I'm not concerned about
> > the build process, but I would not want multiple binaries
> > of Emacs on my machine to show the same set of doc strings.

Thanks for trying to clarify. My impression from your first paragraph was that
this only affected the build process, not the resulting binaries, so there would
be no problem for users of different binaries seeing the proper doc.

But your last paragraph gives me the opposite impression wrt Lisp code. I admit
I don't really understand.

What I would like is (as now) for each binary (development build) that I use to
have the doc and source code that belongs to it, so I can refer to that doc and
code accurately. If nothing else, that is important for investigating bugs and
filing bug reports.

If that were not the case, it would affect also bug reports that 3rd-party
libraries receive from users of Emacs development versions. Those users need to
have access to the latest (for their build) doc and source code. And so do the
3rd-party developers who try to investigate such bug reports.

Recently I reported an Emacs bug that I narrowed down to changes made between
two particular dates. (Unfortunately, those dates were not close together.) With
the proposed change I might still be able to do that in terms of binary
behavior, but if the source code did not actually differ between the two builds
then I would not be able to use the debugger on the source code or otherwise dig
deeper to find the change that introduced the bug.

I'm guessing that I am just not understanding the issue at all, however. I
cannot imagine that builds would be made available for testing that would not
contain the relevant (up-to-date) doc strings or source code.

RE: Canonical location for emacs-version string in source tree?

[Drew Adams]

> > The doc a user sees for a given version should be 
> > up-to-date and reflect that version faithfully. That's all.
> 
> In any case, IIUC your use-case (which you fail to describe) is
> unaffected because you don't use several binary executables that share
> the same source tree, same DOC file and same lisp tree.

No, I do not. I was afraid that the proposed change would make that the case, so
that different builds for the same release that I obtain as Windows binaries
would somehow have exactly the same doc (and Lisp source code?). That would be
bad.

As I said before, I did not understand the proposal (and still don't). But IIUC
now, it seems to affect only those who build Emacs and not those of us who use
prebuilt (e.g. by Sean) Windows binaries. The zip file for a given prebuilt
Windows binary will, IIUC, still contain the up-to-date doc and Lisp code.

If so, fine, and please ignore the noise of my question and lack of
understanding.

Re: Canonical location for emacs-version string in source tree?

[Ken Raeburn]

On May 24, 2010, at 10:16, Stefan Monnier wrote:
>> Still, I'm inclined to think it's better that each version uses the
>> doc strings it was built and installed with.
> 
> Even tho noone cares that it will still use "the latest .elc file
> around" for the non-prebuilt packages?  And "the docstring from the
> latest .elc file" for non-preloaded packages that have been modified
> and recompiled since you loaded that package?

Well, I think it would make sense for it *all* to stay consistent, once installed, or don't even try; the Lisp issue is why I said I'm skeptical of the utility of keeping separate binaries.  They get separate doc files but use the latest lisp files (and other stuff), and if the old binary is already running, or if there were changes in the pre-loaded lisp files, the old binary may run into inconsistencies in the interfaces in the Lisp code loaded.  On the other hand, if this use case is dependent on stuff not changing much, shouldn't that include the DOC file data?

Most other packages I've seen don't make any effort to keep the old installed binaries useable after a new installation of the same version (or if they do, it's by renaming the old installed binary to "foo.old", not by coming up with new names for each new install), and Emacs only does it part-way.  I don't think it's worth the effort.  If you really want separately useable versions, use --prefix or --program-transform-name or something.  Or we could make Emacs figure out where it lives at run time and compute relative paths to the Lisp and other support files so the whole tree can be relocated if the developer wants to save it, much like Emacs.app on the Mac.

Ken

Re: Canonical location for emacs-version string in source tree?

[Richard Stallman]

    I have several co-existing `emacs' executables here, and they coexist
    just fine with a single DOC file.  They just all show the same
    docstrings, which I wouldn't even consider as an issue.

That means only one of them gives the correct doc strings.

I set up a mechanism that has worked fine for decades
and enables each version to display its own doc string.
Please don't break it.

Re: Canonical location for emacs-version string in source tree?

[Stefan Monnier]

>     I have several co-existing `emacs' executables here, and they coexist
>     just fine with a single DOC file.  They just all show the same
>     docstrings, which I wouldn't even consider as an issue.
> That means only one of them gives the correct doc strings.

Depends what you mean by "correct docstrings".  It means they will all
show the latest docstrings, which are usually the more correct ones, in
my world.  As you know all too well, documentation is not quite like
code.  It's for that same reason that we do not need to document every
docstring change in the ChangeLog.

> I set up a mechanism that has worked fine for decades
> and enables each version to display its own doc string.

Actually, it stopped working when we started to leave docstrings (and
"worse": byte-code) in the .elc files.  Ever since this (good) change
was installed your mechanism has only worked for pre-loaded packages.
And noone complained about it, AFAICT, so this subtlety you're trying to
defend is really not worth the trouble.

My bet is that if you try it, you won't notice the difference.


        Stefan