package.el: bytecode portability across emacs versions

package.el: bytecode portability across emacs versions

[Trent Buck]

[-- Attachment #1.1: Type: text/plain, Size: 2269 bytes --]

Arbitrary emacs lisp cannot be byte-compiled in one version of Emacs
and loaded in another, as the following transcript shows:

| $ emacs-snapshot-nox --version
| GNU Emacs 22.1.50.1
| Copyright (C) 2007 Free Software Foundation, Inc.
| GNU Emacs comes with ABSOLUTELY NO WARRANTY.
| You may redistribute copies of Emacs
| under the terms of the GNU General Public License.
| For more information about these matters, see the file named COPYING.
| $ emacs-unicode-2 --version
| GNU Emacs 23.0.0.1
| Copyright (C) 2007 Free Software Foundation, Inc.
| GNU Emacs comes with ABSOLUTELY NO WARRANTY.
| You may redistribute copies of Emacs
| under the terms of the GNU General Public License.
| For more information about these matters, see the file named COPYING.
| $ cat >tmp.el
| (unless (= (eval-when-compile emacs-major-version)
|            emacs-major-version)
|   (error "bytecode is not portable!"))
| $ emacs-snapshot-nox -batch -Q -eval '(load (expand-file-name "tmp"))'
| Loading /home/twb/Desktop/tmp.el (source)...
| $ emacs-unicode-2-nox -batch -Q -eval '(load (expand-file-name "tmp"))'
| Loading /home/twb/Desktop/tmp.el (source)...
| $ emacs-snapshot-nox -batch -Q -f batch-byte-compile tmp.el
| Wrote /home/twb/Desktop/tmp.elc
| $ emacs-snapshot-nox -batch -Q -eval '(load (expand-file-name "tmp"))'
| Loading /home/twb/Desktop/tmp...
| $ emacs-unicode-2-nox -batch -Q -eval '(load (expand-file-name "tmp"))'
| Loading /home/twb/Desktop/tmp...
| bytecode is not portable!

This is most obvious in the wild in emacs-w3m, but I've had problems
with other packages also.

package.el appears to create bytecode in a common directory.  Unless
code is carefully audited, any time .emacs.d is shared between
multiple emacsen packages errors can be expected.

Recommendation: either 0) tell users not to run more than one emacs;
1) don't byte-compile; or 2) place bytecode in path(s) local to the
current emacs-version.

Debian's elisp package framework adopts (2), that might be a source of
inspiration.  Through rampant lethargy I have historically used (1)
for my .emacs.d/lisp/.  It may also be worth looking at how
Debian's/Gentoo's common-lisp-controller package handles "local" fasl
files.
-- 
Trent Buck

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: package.el: bytecode portability across emacs versions

[David Kastrup]

"Trent Buck" <trentbuck@gmail.com> writes:

> Arbitrary emacs lisp cannot be byte-compiled in one version of Emacs
> and loaded in another, as the following transcript shows:

[...]

> This is most obvious in the wild in emacs-w3m, but I've had problems
> with other packages also.
>
> package.el appears to create bytecode in a common directory.

It creates the bytecode in the directory where the source Elisp files
lie, I would presume.

> Unless code is carefully audited, any time .emacs.d is shared
> between multiple emacsen packages errors can be expected.

Sharing such a directory is a mistake, anyway.

> Recommendation: either 0) tell users not to run more than one emacs;
> 1) don't byte-compile; or 2) place bytecode in path(s) local to the
> current emacs-version.
>
> Debian's elisp package framework adopts (2), that might be a source of
> inspiration.

Debian's Elisp package framework socks boulders through straws.  Emacs
is designed to have Elisp and elc files in the same directory.  That's
how load-path orders can take effect.  Debian completely breaks this,
as witnessed by calling M-x list-load-path-shadows RET.  And, of
course, things like M-x byte-recompile-directory RET don't work in
Debian, either.

Sharing Elisp files between different Emacs variants is not a good
idea.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: package.el: bytecode portability across emacs versions

[Kevin Ryde]

David Kastrup <dak@gnu.org> writes:
>
> Debian's Elisp package framework socks boulders through straws.  Emacs
> is designed to have Elisp and elc files in the same directory.  That's
> how load-path orders can take effect.  Debian completely breaks this,
> as witnessed by calling M-x list-load-path-shadows RET.

emacs-goodies-el has recently taken to symlinking the source into the
per-flavour bytecomp directories.  It was motivated by wanting to
exclude selected files from some emacs versions (either because they're
already there, or because they don't run).  I don't think there's
anything fundamental about separate source/compiled, only like the
"streaker's excuse": it seemed like a good idea at the time.

(And yes, I've wrestled with load-path or find-function-source-path for
packages that don't make the .el's available in those already from
friendly browsing.  Very annoying.)

Re: package.el: bytecode portability across emacs versions

[Trent Buck]

[-- Attachment #1.1: Type: text/plain, Size: 1429 bytes --]

On Mon, May 21, 2007 at 08:10:47PM +0200, David Kastrup wrote:
> "Trent Buck" <trentbuck@gmail.com> writes:
> > Recommendation: either 0) tell users not to run more than one emacs;
> > 1) don't byte-compile; or 2) place bytecode in path(s) local to the
> > current emacs-version.
> >
> > Debian's elisp package framework adopts (2), that might be a source of
> > inspiration.
> 
> Debian's Elisp package framework socks boulders through straws.  Emacs
> is designed to have Elisp and elc files in the same directory.  That's
> how load-path orders can take effect.  Debian completely breaks this,
> as witnessed by calling M-x list-load-path-shadows RET.  And, of
> course, things like M-x byte-recompile-directory RET don't work in
> Debian, either.

Forgive me, I did not know.

> > package.el appears to create bytecode in a common directory.
> 
> It creates the bytecode in the directory where the source Elisp files
> lie, I would presume.

Correct.

> > Unless code is carefully audited, any time .emacs.d is shared
> > between multiple emacsen packages errors can be expected.
> 
> Sharing such a directory is a mistake, anyway.
>
> [...]
>
> Sharing Elisp files between different Emacs variants is not a good
> idea.

OK, so what file system do you think package.el should use?  Or is it
also a bad idea to share $HOME across hosts running different Emacs
releases?
-- 
Trent Buck

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: package.el: bytecode portability across emacs versions

[Tom Tromey]

>>>>> "Trent" == Trent Buck <trentbuck@gmail.com> writes:

Trent> OK, so what file system do you think package.el should use?  Or is it
Trent> also a bad idea to share $HOME across hosts running different Emacs
Trent> releases?

Sorry for the delay in my reply to this.

I know that sharing bytecode is bad.  I just didn't know what to do
about it.

It would be pretty easy to have separate package universes depending
on what Emacs you are running.  But that doesn't seem very
user-friendly.  Putting the bytecode elsewhere could be done but I
gather that this doesn't work too well in some situations.  package.el
could re-byte-compile everything when it detects that you switched
Emacs -- this would be fine for cases like "I upgraded my system and
got a new Emacs", but it would be very unfriendly for other cases.
And of course there's always "don't byte compile at all", but that
would probably rule out installing some of the larger and more
performance-sensitive packages.

Anyway, I'm open to implementing whatever the Emacs experts agree is
the best solution.

Tom

Re: package.el: bytecode portability across emacs versions

[Michael Olson]

[-- Attachment #1.1: Type: text/plain, Size: 3066 bytes --]

Tom Tromey <elpa@tromey.com> writes:

> I know that sharing bytecode is bad.  I just didn't know what to do
> about it.
>
> It would be pretty easy to have separate package universes depending
> on what Emacs you are running.  But that doesn't seem very
> user-friendly.  Putting the bytecode elsewhere could be done but I
> gather that this doesn't work too well in some situations.  package.el
> could re-byte-compile everything when it detects that you switched
> Emacs -- this would be fine for cases like "I upgraded my system and
> got a new Emacs", but it would be very unfriendly for other cases.
> And of course there's always "don't byte compile at all", but that
> would probably rule out installing some of the larger and more
> performance-sensitive packages.
>
> Anyway, I'm open to implementing whatever the Emacs experts agree is
> the best solution.

I would recommend doing things in the following way.

Store the byte code in subdirectories like
~/.emacs.d/elpa/emacs-22.0.98.1/.  Ensure that symlinks in that
directory exist that point to the source files in ~/.emacs/elpa/.  When
elpa is started, it should only add the proper version-specific
subdirectory to the load-path.  If under Windows, package.el would
simply copy every source file to the subdirectory rather than use a
symlink.

If package.el is run from a "new" version of Emacs/XEmacs/SXEmacs during
startup, it will display a message like "Byte-compiling ELPA packages
for <Emacs variant and version>", create the directory, copy or symlink
source files to it (also create subdirectories of symlinks/copies for
larger projects that have their own elpa subdirectory), byte compile
that directory, and add it to load-path.

Here's some code that could be used to detect the current Emacs variant
and version.  I'm just guessing about the (featurep 'sxemacs) part
because I have yet to use SXEmacs.

(defun package-get-emacs-version (&optional display-friendly)
  "Return a string that describes the current variant and version of Emacs.
If DISPLAY-FRIENDLY is non-nil, return a string that would be
useful as part of a displayed message.
Otherwise, return a string of the form \"<variant>-<version>.\""
  (let ((variant (cond ((featurep 'sxemacs) "SXEmacs")
                       ((featurep 'xemacs) "XEmacs")
                       (t "Emacs")))
        (version (cond ((featurep 'xemacs)
                        (concat (number-to-string emacs-major-version)
                                "." (number-to-string emacs-minor-version)
                                "." (number-to-string emacs-patch-level)))
                       (t emacs-version))))
    (if display-friendly
        (concat variant " " version)
      (concat (downcase variant) "-" version))))

-- 
       Michael Olson -- FSF Associate Member #652     |
 http://mwolson.org/ -- Jabber: mwolson_at_hcoop.net  |  /` |\ | | |
            Sysadmin -- Hobbies: Lisp, GP2X, HCoop    | |_] | \| |_|
Projects: Emacs, Muse, ERC, EMMS, ErBot, DVC, Planner |

[-- Attachment #1.2: Type: application/pgp-signature, Size: 188 bytes --]

[-- Attachment #2: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: package.el: bytecode portability across emacs versions

[David Kastrup]

Michael Olson <mwolson@gnu.org> writes:

> I would recommend doing things in the following way.
>
> Store the byte code in subdirectories like
> ~/.emacs.d/elpa/emacs-22.0.98.1/.  Ensure that symlinks in that
> directory exist that point to the source files in ~/.emacs/elpa/.

Windows has no symlinks.  That is no concern for Debian, but it would
be for a general package system.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: package.el: bytecode portability across emacs versions

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Sat, 26 May 2007 08:34:50 +0200
> Cc: emacs-devel@gnu.org
> 
> Windows has no symlinks.  That is no concern for Debian, but it would
> be for a general package system.

Latest versions of Windows do have symlinks.  (Which of course does
not change in any way the conclusion that symlinks are not the best
idea for a general package system.)

Re: package.el: bytecode portability across emacs versions

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Sat, 26 May 2007 08:34:50 +0200
>> Cc: emacs-devel@gnu.org
>> 
>> Windows has no symlinks.  That is no concern for Debian, but it would
>> be for a general package system.
>
> Latest versions of Windows do have symlinks.  (Which of course does
> not change in any way the conclusion that symlinks are not the best
> idea for a general package system.)

It might work to do "Emacs symlinks", single files only containing
(load "/absolute/path/something.el")

On the other hand, when thinking about a package system, we are not
restricted to maintaining the basic assumption in Emacs that demands
compiled and source Lisp files to be in the same place.  While I have
no good ideas about a different reliable scheme, if people can come up
with a good design that works when separating source and compiled
files, we could switch to using it.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: package.el: bytecode portability across emacs versions

[Trent Buck]

[-- Attachment #1.1: Type: text/plain, Size: 1017 bytes --]

On Fri, May 25, 2007 at 10:00:58AM -0600, Tom Tromey wrote:
> >>>>> "Trent" == Trent Buck <trentbuck@gmail.com> writes:
> 
> Trent> OK, so what file system do you think package.el should use?  Or is it
> Trent> also a bad idea to share $HOME across hosts running different Emacs
> Trent> releases?
> 
> I know that sharing bytecode is bad.  I just didn't know what to do
> about it.
> 
> Putting the bytecode elsewhere could be done but I gather that this
> doesn't work too well in some situations.  

On systems with links, you could do something like

    sources/foo-1/foo.el
    21.4/foo-1/foo.el -> ../../sources/foo-1/foo.el
    21.4/foo-1/foo.elc
    23.0.0.1/foo-1/foo.el -> ../../sources/foo-1/foo.el
    22.0.0.1/foo-1/foo.elc

I don't know how well this solves the
sources-should-be-in-the-same-place-as-bytecode issue, because I don't
fully understand the consequences of *not* doing it.  In any case, it
wouldn't work on at least some Windows systems (right?).
-- 
Trent Buck

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: package.el: bytecode portability across emacs versions

[Eli Zaretskii]

> From: "Trent Buck" <trentbuck@gmail.com>
> Date: Sat, 26 May 2007 15:33:09 +1000
> Cc: emacs-devel@gnu.org
> 
> In any case, [symlinks] wouldn't work on at least some Windows
> systems (right?).

Right.

Re: package.el: bytecode portability across emacs versions

[Richard Stallman]

    I know that sharing bytecode is bad.  I just didn't know what to do
    about it.

You could put the bytecode in
/usr/local/share/emacs/VERSION/site-lisp, which is a standard element
of load-path, version-specific.  The source code could go in
/usr/local/share/emacs/site-lisp.  If package.el finds the source
code and not the binary, it can compile for the current Emacs version.

Re: package.el: bytecode portability across emacs versions

[Trent Buck]

[-- Attachment #1.1: Type: text/plain, Size: 913 bytes --]

On Sat, May 26, 2007 at 09:00:32PM -0400, Richard Stallman wrote:
>     I know that sharing bytecode is bad.  I just didn't know what to do
>     about it.
> 
> You could put the bytecode in
> /usr/local/share/emacs/VERSION/site-lisp, which is a standard element
> of load-path, version-specific.  The source code could go in
> /usr/local/share/emacs/site-lisp.  If package.el finds the source
> code and not the binary, it can compile for the current Emacs version.

...except that package.el is (correct me if I'm wrong) intended to be
run by nonpriviledged users, and therefore only $HOME and $TMPDIR are
(typically) writable.

I earlier suggested something like 

    $HOME/.emacs.d/elpa/lisp

for sources, and

    $HOME/.emacs.d/elpa/VERSION

for byte-code, but David suggested this would cause problems.  I
confess I don't quite understand what those problems are.
-- 
Trent Buck

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: package.el: bytecode portability across emacs versions

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

>     I know that sharing bytecode is bad.  I just didn't know what to do
>     about it.
>
> You could put the bytecode in
> /usr/local/share/emacs/VERSION/site-lisp, which is a standard element
> of load-path, version-specific.  The source code could go in
> /usr/local/share/emacs/site-lisp.  If package.el finds the source
> code and not the binary, it can compile for the current Emacs version.

Emacs is not well-equipped to have source and byte code in different
directories.  For example, it messes with the load path order: Emacs
will prefer loading an .el file in an earlier place in the load-path
over a .elc file in a later place.

It also stops byte-recompile-directory from working and makes
list-load-path-shadow complain.

I don't say that such a system could not be supported, but it would
need code changes outside of package.el itself.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: package.el: bytecode portability across emacs versions

[Richard Stallman]

    Emacs is not well-equipped to have source and byte code in different
    directories.  For example, it messes with the load path order: Emacs
    will prefer loading an .el file in an earlier place in the load-path
    over a .elc file in a later place.

That is not a problem for this approach, because the directory for the
specific version always comes before the version-independent directory.

    It also stops byte-recompile-directory from working and makes
    list-load-path-shadow complain.

It would be a bit strange to use byte-recompile-directory 
on these directories, though.

Re: package.el: bytecode portability across emacs versions

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

>     Emacs is not well-equipped to have source and byte code in different
>     directories.  For example, it messes with the load path order: Emacs
>     will prefer loading an .el file in an earlier place in the load-path
>     over a .elc file in a later place.
>
> That is not a problem for this approach, because the directory for the
> specific version always comes before the version-independent directory.

You are talking about "the directory".  load-path establishes an
_order_, and we can have repetitive elements in this order when the
user installs his own version of some packages intended to override
the system version.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: package.el: bytecode portability across emacs versions

[Richard Stallman]

    You are talking about "the directory".

I'n talking about two specific directories that we would put into the
normal value of load-path, in the order that we choose.

					    load-path establishes an
    _order_, and we can have repetitive elements in this order when the
    user installs his own version of some packages intended to override
    the system version.

I don't understand the scenario you have in mind, but most people who
add to load-path are lilely to do so by adding directories.  Adding
directories which package.el doesn't use won't cause any problem.  So
I don't see a problewm here.

Re: package.el: bytecode portability across emacs versions

[Michael Olson]

[-- Attachment #1.1: Type: text/plain, Size: 1614 bytes --]

David Kastrup <dak@gnu.org> writes:

> Richard Stallman <rms@gnu.org> writes:
>
>>     I know that sharing bytecode is bad.  I just didn't know what to do
>>     about it.
>>
>> You could put the bytecode in
>> /usr/local/share/emacs/VERSION/site-lisp, which is a standard element
>> of load-path, version-specific.  The source code could go in
>> /usr/local/share/emacs/site-lisp.  If package.el finds the source
>> code and not the binary, it can compile for the current Emacs version.
>
> Emacs is not well-equipped to have source and byte code in different
> directories.  For example, it messes with the load path order: Emacs
> will prefer loading an .el file in an earlier place in the load-path
> over a .elc file in a later place.
>
> It also stops byte-recompile-directory from working and makes
> list-load-path-shadow complain.
>
> I don't say that such a system could not be supported, but it would
> need code changes outside of package.el itself.

If you put symlinks to the source files in the same directory as the
byte-compiled files (or just copy them there), and don't include the
version-independent directory in load-path, there is no problem (to the
best of my knowledge).  I'm trying to get something to this effect added
to the Debian Emacs packaging policy, but the policy-makers are rather
slow.

-- 
       Michael Olson -- FSF Associate Member #652     |
 http://mwolson.org/ -- Jabber: mwolson_at_hcoop.net  |  /` |\ | | |
            Sysadmin -- Hobbies: Lisp, GP2X, HCoop    | |_] | \| |_|
Projects: Emacs, Muse, ERC, EMMS, ErBot, DVC, Planner |

[-- Attachment #1.2: Type: application/pgp-signature, Size: 188 bytes --]

[-- Attachment #2: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: package.el: bytecode portability across emacs versions

[Richard Stallman]

    If you put symlinks to the source files in the same directory as the
    byte-compiled files (or just copy them there), and don't include the
    version-independent directory in load-path, there is no problem (to the
    best of my knowledge).

Yes, that would work.

Re: package.el: bytecode portability across emacs versions

[Stefan Monnier]

> I know that sharing bytecode is bad.  I just didn't know what to do
> about it.

I think the current approach is best: leave it to the user.


        Stefan