native-comp and resetting of search paths

native-comp and resetting of search paths

[Alex Bennée]

Hi Andrea, Jorgen,

I just thought I'd give you a heads up on a potential clash between
native-comp's background compilation and elpy. In essence native-comp
changes the value of elpy-rpc-pythonpath from:

  "/home/alex/.emacs.d/elpa/elpy-20200527.2021/"

To:

  "/home/alex/.emacs.d/elpa/elpy-20200527.2021/eln-x86_64-pc-linux-gnu-93d8fa3f77c8fe62/"

which is fine for the .elc -> .eln change but because elpy also uses
this to find the python server in .emacs.d/elpa/elpy-20200527.2021/elpy/
it gets very confused.

What was the rationale behind this change? I can see it makes sense to
tweak load-path but generally I would expect variables belonging to
packages to remain untouched.

Is this a recent change? it only seemed to break in the last week.

-- 
Alex Bennée

Re: native-comp and resetting of search paths

[Andrea Corallo via Emacs development discussions.]

Alex Bennée <alex.bennee@linaro.org> writes:

> Hi Andrea, Jorgen,
>
> I just thought I'd give you a heads up on a potential clash between
> native-comp's background compilation and elpy. In essence native-comp
> changes the value of elpy-rpc-pythonpath from:
>
>   "/home/alex/.emacs.d/elpa/elpy-20200527.2021/"
>
> To:
>
>   "/home/alex/.emacs.d/elpa/elpy-20200527.2021/eln-x86_64-pc-linux-gnu-93d8fa3f77c8fe62/"
>
> which is fine for the .elc -> .eln change but because elpy also uses
> this to find the python server in .emacs.d/elpa/elpy-20200527.2021/elpy/
> it gets very confused.
>
> What was the rationale behind this change? I can see it makes sense to
> tweak load-path but generally I would expect variables belonging to
> packages to remain untouched.
>
> Is this a recent change? it only seemed to break in the last week.

Hi Alex,

AFAIR nothing recently changed should cause a regression in this area.

If you cold look into how elpy-rpc-pythonpath is computed I guess should
be easy to undestand what is going on.

Thanks

  Andrea

-- 
akrl@sdf.org

Re: native-comp and resetting of search paths

[Alex Bennée]

Andrea Corallo <akrl@sdf.org> writes:

> Alex Bennée <alex.bennee@linaro.org> writes:
>
>> Hi Andrea, Jorgen,
>>
>> I just thought I'd give you a heads up on a potential clash between
>> native-comp's background compilation and elpy. In essence native-comp
>> changes the value of elpy-rpc-pythonpath from:
>>
>>   "/home/alex/.emacs.d/elpa/elpy-20200527.2021/"
>>
>> To:
>>
>>   "/home/alex/.emacs.d/elpa/elpy-20200527.2021/eln-x86_64-pc-linux-gnu-93d8fa3f77c8fe62/"
>>
>> which is fine for the .elc -> .eln change but because elpy also uses
>> this to find the python server in .emacs.d/elpa/elpy-20200527.2021/elpy/
>> it gets very confused.
>>
>> What was the rationale behind this change? I can see it makes sense to
>> tweak load-path but generally I would expect variables belonging to
>> packages to remain untouched.
>>
>> Is this a recent change? it only seemed to break in the last week.
>
> Hi Alex,
>
> AFAIR nothing recently changed should cause a regression in this area.
>
> If you cold look into how elpy-rpc-pythonpath is computed I guess should
> be easy to undestand what is going on.

Hmm odd. It's a custom variable:

  (defcustom elpy-rpc-pythonpath (file-name-directory (locate-library "elpy"))
    "A directory to add to the PYTHONPATH for the RPC process.

  This should be a directory where the elpy module can be found. If
  this is nil, it's assumed elpy can be found in the standard path.
  Usually, there is no need to change this."
    :type 'directory
    :safe #'file-directory-p
    :group 'elpy)

As I've not customised it it defaults to:

  (file-name-directory (locate-library "elpy"))

Which of course now resolves to:

  "/home/alex/.emacs.d/elpa/elpy-20200527.2021/eln-x86_64-pc-linux-gnu-93d8fa3f77c8fe62/"

So I guess this is a rare corner case because really we are using the
locate machinery as a side-effect to find the rest of the elpa install
directory.

There must be a mechanism to find the original source file because my
*helpful* buffer is able to navigate to source for all the functions
that are now native compiled so once I know what that is I can submit a
patch.

I guess the package is "wrong" for making the assumption based on the
locate-library information but it was probably a reasonable assumption
to make up to this point. I wonder if any other packages would get
broken by this changed assumption?

A quick grep of my ~/.emacs.d/elpa certainly shows a lot of packages
locate-library themselves. For example async-start has this comment:

   ;; Using `locate-library' ensure we use the right file
   ;; when the .elc have been deleted.
   (locate-library "async")

Not sure what the correct fix for those cases would be to correctly deal
with native-comp? Should we have a (locate-library-source)?

>
> Thanks
>
>   Andrea


-- 
Alex Bennée

Re: native-comp and resetting of search paths

[Stephen Leake]

Alex Bennée <alex.bennee@linaro.org> writes:

>
> Hmm odd. It's a custom variable:
>
>   (defcustom elpy-rpc-pythonpath (file-name-directory (locate-library "elpy"))
>     "A directory to add to the PYTHONPATH for the RPC process.
>

Use (locate-library "elpy.el" t); that will find the .el file, not one
of the compiled files.

-- 
-- Stephe

Re: native-comp and resetting of search paths

[Stefan Monnier]

>> Hmm odd. It's a custom variable:
>>
>>   (defcustom elpy-rpc-pythonpath (file-name-directory (locate-library "elpy"))
>>     "A directory to add to the PYTHONPATH for the RPC process.
>
> Use (locate-library "elpy.el" t); that will find the .el file, not one
> of the compiled files.

This will workaround the incompatibility introduced by the native-comp
branch, but we should probably try and see if we can change the way the
branch works such that such incompatibility is avoided.


        Stefan

Re: native-comp and resetting of search paths

[Alex Bennée]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Hmm odd. It's a custom variable:
>>>
>>>   (defcustom elpy-rpc-pythonpath (file-name-directory (locate-library "elpy"))
>>>     "A directory to add to the PYTHONPATH for the RPC process.
>>
>> Use (locate-library "elpy.el" t); that will find the .el file, not one
>> of the compiled files.
>
> This will workaround the incompatibility introduced by the native-comp
> branch, but we should probably try and see if we can change the way the
> branch works such that such incompatibility is avoided.

Is there a particular reason why .eln files are stored in a
sub-directory?

I would guess there is a concern about clashing between versions of the
compiler and different architectures on network file-systems?

The .eln files themselves are ELF files so we could just shove any extra
metadata in there although I don't know if that becomes expensive
compared to a simple file-path check?

I assume the byte-compiler potentially has similar problems if a user
runs two different versions of Emacs but we don't dump .elc files in
special locations. Is this just not a concern that hits often enough or
something that is detected and dealt with by Emacs itself?

I note the symbols in the eln files also have fairly unique identifiers
so maybe we are attempting to avoid a problem that's not really there?

I realise that was a lot of questions so maybe I should just propose the
eln stay in the directory of the source files and have a form like:

  <source dir of .el>/<basename>.<arch>.eln

-- 
Alex Bennée

Re: native-comp and resetting of search paths

[Andrea Corallo via Emacs development discussions.]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Hmm odd. It's a custom variable:
>>>
>>>   (defcustom elpy-rpc-pythonpath (file-name-directory (locate-library "elpy"))
>>>     "A directory to add to the PYTHONPATH for the RPC process.
>>
>> Use (locate-library "elpy.el" t); that will find the .el file, not one
>> of the compiled files.
>
> This will workaround the incompatibility introduced by the native-comp
> branch, but we should probably try and see if we can change the way the
> branch works such that such incompatibility is avoided.

Agree.  So far the only way I see to remove this directory dualism is
what you have proposed here:

https://lists.gnu.org/archive/html/emacs-devel/2020-06/msg00050.html

Given I think we should solve this source of problems once for all the
more time is passing the more I'm getting convinced that this is the way
to go.

Still I've to make up my mind on certain details but probably I should
start implementing to get a cleaner view.

This choice is key for making the native compiler project succeed or not
(and interestly enough has nothing to do with the compiler it-self), so
please if you have ideas on the topic don't esitate to share them :)

  Andrea
  
-- 
akrl@sdf.org

Re: native-comp and resetting of search paths

[Andrea Corallo via Emacs development discussions.]

Alex Bennée <alex.bennee@linaro.org> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>>> Hmm odd. It's a custom variable:
>>>>
>>>>   (defcustom elpy-rpc-pythonpath (file-name-directory (locate-library "elpy"))
>>>>     "A directory to add to the PYTHONPATH for the RPC process.
>>>
>>> Use (locate-library "elpy.el" t); that will find the .el file, not one
>>> of the compiled files.
>>
>> This will workaround the incompatibility introduced by the native-comp
>> branch, but we should probably try and see if we can change the way the
>> branch works such that such incompatibility is avoided.
>
> Is there a particular reason why .eln files are stored in a
> sub-directory?
>
> I would guess there is a concern about clashing between versions of the
> compiler and different architectures on network file-systems?

Elns are not only arch dependent but also Emacs version + configuration
dependent, so yeah this is to disambiguates these.

>
> The .eln files themselves are ELF files so we could just shove any extra
> metadata in there although I don't know if that becomes expensive
> compared to a simple file-path check?

I think the issue is that we need a disambiguation at file name level to
retain a reasonably fast loading system.

> I assume the byte-compiler potentially has similar problems if a user
> runs two different versions of Emacs but we don't dump .elc files in
> special locations. Is this just not a concern that hits often enough or
> something that is detected and dealt with by Emacs itself?
>
> I note the symbols in the eln files also have fairly unique identifiers
> so maybe we are attempting to avoid a problem that's not really there?

This is because lisp symbols are a superset of valid C ones, so this is
another level of disambiguation but here it does not play a role.

> I realise that was a lot of questions so maybe I should just propose the
> eln stay in the directory of the source files and have a form like:
>
>   <source dir of .el>/<basename>.<arch>.eln

I don't think we can use '/' in unix filenames IIUC your proposal.

Stefan idea is about making elns completely transparent, you'll find the
link my other mail.

Thanks

  Andrea

-- 
akrl@sdf.org

Re: native-comp and resetting of search paths

[Stefan Monnier]

> Agree.  So far the only way I see to remove this directory dualism is
> what you have proposed here:
>
> https://lists.gnu.org/archive/html/emacs-devel/2020-06/msg00050.html

I must say I really like what this guy proposes.
He must be bright.

> This choice is key for making the native compiler project succeed or not
> (and interestly enough has nothing to do with the compiler it-self), so
> please if you have ideas on the topic don't esitate to share them :)

Indeed, I think we should strive to make it as transparent as possible:
compiling .el to .elc isn't transparent, but that doesn't mean we can't
make the compilation from .elc to .eln transparent.


        Stefan