Project local variables.

Project local variables.

[Ergus]

Hi:

Looking at the evolution of project.el I want to know if there is any
sort of project local environment/namespace/scope. The idea is to
provide some variables that could be shared by all the files in the same
project but that could be modified by external-packages at the elisp
level. something like "setq-project"??

A use case is for example when working in tramp some operations are
expensive like looking for an executable in the remote system. The
search could be made only once; but then if a local file is open the
cached value will be wrong. The other case is to use the buffer-local
variables, but then it will need to be initialized once/per file every
time a file is open. Something that in my case affected performance
especially with lsp and elpy

Currently I am using a work-around with a global hash-table and a
variable definer in the dir-locals.el as a prefix, but maybe something
more elegant should/must be already implemented and I am not aware of??

Best,
Ergus

Re: Project local variables.

[Stephen Leake]

Ergus <spacibba@aol.com> writes:

> Looking at the evolution of project.el I want to know if there is any
> sort of project local environment/namespace/scope. The idea is to
> provide some variables that could be shared by all the files in the same
> project but that could be modified by external-packages at the elisp
> level. something like "setq-project"??

You can derive a new project type:

(cl-defstruct my-prj
  name     ;; A user-friendly string, used in menus and messages.

  variable-1
  variable-2
  ...
  )

add a project-find-functions for it:

(defun my-prj-find (dir)
...
)
(add-hook 'project-find-functions #'my-prj-find 90)

and then define new project ops that use the variable for it:

(cl-defgeneric my-method-1 ((_project wisi-prj))
 ...
 )

The GNU ELPA wisi package takes this approach.

But I suspect you are asking for something more lightweight.

> A use case is for example when working in tramp some operations are
> expensive like looking for an executable in the remote system. The
> search could be made only once; but then if a local file is open the
> cached value will be wrong. 

For this specific case, adding an optional cache to tramp seems like a
better approach. Perhaps that just means creating a new file handler
that wraps the tramp one with a cache?

You could use the above approach to define a project whose files are in
a remote directory accessed via tramp, but that doesn't work if you want
all the normal emacs functions that access files via file handlers to
use the cache. So I don't see how a project-local variable would help.

-- 
-- Stephe

Written from unceded ancestral homelands of the Huichiun, an Ohlone
people,for which I pay Shuumi Land Tax
(https://sogoreate-landtrust.com/shuumi-land-tax) as an inadequate token
of reparation.

Re: Project local variables.

[Dmitry Gutov]

Hi!

On 03.11.2020 21:41, Ergus wrote:

> Looking at the evolution of project.el I want to know if there is any
> sort of project local environment/namespace/scope. The idea is to
> provide some variables that could be shared by all the files in the same
> project but that could be modified by external-packages at the elisp
> level. something like "setq-project"??

Something like project-set and project-get could be arranged (or perhaps 
just a project-session hash table). But given that the values won't 
persist between Emacs sessions, we'd have to consider to use cases first.

> A use case is for example when working in tramp some operations are
> expensive like looking for an executable in the remote system. The
> search could be made only once; but then if a local file is open the
> cached value will be wrong. The other case is to use the buffer-local
> variables, but then it will need to be initialized once/per file every
> time a file is open. Something that in my case affected performance
> especially with lsp and elpy

Two things to consider: the performance of the operation you're trying 
to "cache", and the performance of 'project-current'.

The latter is not "free". And my near-term plan is to make 
project-try-vc slower by removing the vc-file-getprop/vc-file-setprop 
dance because it can lead to outdated information. Or at least try that 
and see which problems that brings.

In any case, what I'm saying is, 'project-current' on a remote host 
might not be fast either. Though it could be cached to at least only do 
the search once per user command.

> Currently I am using a work-around with a global hash-table and a
> variable definer in the dir-locals.el as a prefix, but maybe something
> more elegant should/must be already implemented and I am not aware of??

Have you tried using (file-remote-p buffer-file-name) as the hash key?

If the operation to be sped up is really (executable-find "cat"), the 
result is really project-independent and should only depend on the host.

If there are other, actually project-dependent examples, the 
project-local variables (or "session cache", rather) could be 
implemented as a hash of hashes, keyed by project instance. There 
implementation seems like it will be rather trivial, but first I would 
like to know the use cases.

Re: Project local variables.

[Ergus]

Hi:

Thanks for replying

On Sun, Nov 08, 2020 at 03:48:45AM +0200, Dmitry Gutov wrote:
>Hi!
>
>On 03.11.2020 21:41, Ergus wrote:
>
>
>Something like project-set and project-get could be arranged (or 
>perhaps just a project-session hash table). But given that the values 
>won't persist between Emacs sessions, we'd have to consider to use 
>cases first.
>
Yes, I was not thinking in variables that persist among sessions; but
only in the active session, so initialize them once. Let's say; when a
file is open Emacs now checks if it belongs to a project; such a project
is detected (AFAIK) looking up for a vc directory like .git or something
similar. Is exactly in this case where some environment could be
shared/scoped to the current project. In LSP I see that most of the
checks are made every time a file is opened; when using ctags it asks
every time for the tags file to use instead of reusing "what could be
inferred" from the current project information if there are other files
open. To persist among session in the worst case we can add some
declarations in dir-locals... but I was not thinking on that.

>
>Two things to consider: the performance of the operation you're trying 
>to "cache", and the performance of 'project-current'.
>
>The latter is not "free". And my near-term plan is to make 
>project-try-vc slower by removing the vc-file-getprop/vc-file-setprop 
>dance because it can lead to outdated information. Or at least try 
>that and see which problems that brings.
>
>In any case, what I'm saying is, 'project-current' on a remote host 
>might not be fast either. Though it could be cached to at least only 
>do the search once per user command.
>
I don't really know how both performance will compare indeed. In general
I would expect that looking into the local memory hash-table will be
faster that looking in a remote file system among the network for an
executable/file or path. Ex: parallel files-systems, slow networks, or
experimental boards with slow processors, big processes and so on.

I mean; after opening a file, of course a check in the file system will
be needed to know if that file is in a current opened project (I use
project-root). If it is not, then we need to initialize some things like
read the dir-locals, TAGS, look for ctags/gtags/etags executables, tag
files, clang files and so on. Else, we don't need to repeat anything. If
the project has 3000 files, that saves a lot of overhead opening files
and probably in some search engines for completions (I am not really
sure if this is actually possible). In my case; when I open many projects
at the same time I get completions in one from the other... which is
conceptually wrong too.

>
>Have you tried using (file-remote-p buffer-file-name) as the hash key?
>
I am using project-root and setting a buffer local variables to use as
the key. I set the variable after opening a file (in a hook); the rest
of my variables are set lazily... if-set->use else set-and-use. And also
added a function hook to check if other files of the same projects
remain open when I kill a buffer; else I delete the info from the hash table".

>If the operation to be sped up is really (executable-find "cat"), the 
>result is really project-independent and should only depend on the 
>host.
>
  Every file I open in one of the projects may share the project-root at
least. executable-find will be host dependent, but TAGS file, a build
directory, compile_commands.json, root to build or open a shell/vterm,
maybe some modes that I want to enable temporally for this project (ex:
lsp, company, langtool for Latex). 

>If there are other, actually project-dependent examples, the 
>project-local variables (or "session cache", rather) could be 
>implemented as a hash of hashes, keyed by project instance. There 
>implementation seems like it will be rather trivial, but first I would 
>like to know the use cases.
>
Se above; maybe this is not enough to justify the implementation of a
new level of scopes in emacs; I am just mentioning the use cases I had
in mind and facing very often.

Re: Project local variables.

[Dmitry Gutov]

On 08.11.2020 17:21, Ergus wrote:

>> Something like project-set and project-get could be arranged (or 
>> perhaps just a project-session hash table). But given that the values 
>> won't persist between Emacs sessions, we'd have to consider to use 
>> cases first.
>>
> Yes, I was not thinking in variables that persist among sessions; but
> only in the active session, so initialize them once.

Sounds like maybe "project session cache" might be an appropriate name.

> Let's say; when a
> file is open Emacs now checks if it belongs to a project; such a project
> is detected (AFAIK) looking up for a vc directory like .git or something
> similar. Is exactly in this case where some environment could be
> shared/scoped to the current project. In LSP I see that most of the
> checks are made every time a file is opened; when using ctags it asks
> every time for the tags file to use instead of reusing "what could be
> inferred" from the current project information if there are other files
> open. To persist among session in the worst case we can add some
> declarations in dir-locals... but I was not thinking on that.

lsp-mode doesn't always use project.el for projects, though. Eglot does, 
but then I'm not sure it's a good idea for it in general (after all, it 
might have to support more project markers purely for internal purposes).

In any case, both of them could use a hash table keyed on a project 
root, for instance. Which would be the simplest version of the "project 
session cache" I mentioned above.

>> Two things to consider: the performance of the operation you're trying 
>> to "cache", and the performance of 'project-current'.
>>
>> The latter is not "free". And my near-term plan is to make 
>> project-try-vc slower by removing the vc-file-getprop/vc-file-setprop 
>> dance because it can lead to outdated information. Or at least try 
>> that and see which problems that brings.
>>
>> In any case, what I'm saying is, 'project-current' on a remote host 
>> might not be fast either. Though it could be cached to at least only 
>> do the search once per user command.
>>
> I don't really know how both performance will compare indeed. In general
> I would expect that looking into the local memory hash-table will be
> faster that looking in a remote file system among the network for an
> executable/file or path. Ex: parallel files-systems, slow networks, or
> experimental boards with slow processors, big processes and so on.

Hash tables are fast. What I meant is, whether the original operation, 
which we hope to speed up with said cache, is going to slower than the 
'project-current' call. The latter is not free either (we currently 
cache the result indefinitely, but we should do less of that in the future).

I suppose whether it's feasible depends on how one would organize the 
code. I.e. whether one could call project-current once, and then do a 
bunch of lookups inside its session cache. If so, it should be an 
improvement.

> I mean; after opening a file, of course a check in the file system will
> be needed to know if that file is in a current opened project (I use
> project-root).

project.el doesn't have a global variable which tracks the "current 
opened project". It just knows how to map a file (or, usually, its 
parent directory) to a structure that knows how to describe the project 
that contains it.

Even if we do add something like that to serve as another kind of cache, 
a file-in-directory-p check wouldn't suffice because we want to support 
nested projects, too.

>> Have you tried using (file-remote-p buffer-file-name) as the hash key?
>>
> I am using project-root and setting a buffer local variables to use as
> the key.

The value of project-root is probably good enough. I was going to 
suggest to key by the project instance itself, but really, it's very 
unlikely to have two different project instances that have the same root 
(even from different backends). Though not impossible, of course.

> I set the variable after opening a file (in a hook); the rest
> of my variables are set lazily... if-set->use else set-and-use. And also
> added a function hook to check if other files of the same projects
> remain open when I kill a buffer; else I delete the info from the hash 
> table".

Unless the cached data structures are really big, I probably wouldn't 
bother with eviction/garbage collection.

>> If the operation to be sped up is really (executable-find "cat"), the 
>> result is really project-independent and should only depend on the host.
>>
>   Every file I open in one of the projects may share the project-root at
> least. executable-find will be host dependent, but TAGS file, a build
> directory, compile_commands.json, root to build or open a shell/vterm,
> maybe some modes that I want to enable temporally for this project (ex:
> lsp, company, langtool for Latex).
>> If there are other, actually project-dependent examples, the 
>> project-local variables (or "session cache", rather) could be 
>> implemented as a hash of hashes, keyed by project instance. There 
>> implementation seems like it will be rather trivial, but first I would 
>> like to know the use cases.
>>
> Se above; maybe this is not enough to justify the implementation of a
> new level of scopes in emacs; I am just mentioning the use cases I had
> in mind and facing very often.

As discussed, the implementation of that new feature seems pretty trivial.

The questions are whether it does save you on CPU time, whether it will 
fit the goals even with not-immediate project-current, and whether we do 
need an implementation of it in project.el itself.

If any project can add a hash table and key the data by project-root, 
that sounds trivial, maybe that's good enough? Unless we identify some 
common pitfalls which we could help people avoid by writing some extra 
code here.