How to use project.el

How to use project.el

[excalamus--- via Users list for the GNU Emacs text editor]

With 27.1 out, I see talk of project.el, how great it is, and how it's been around for years.  What I don't see is anything about how to use it.  There's one sparse page in the manual, C-h i m emacs m Projects.  I've run project-find-file and project-search.  I fail to see how they're fundamentally different than find-file or grep.

I happen to have ESS installed which appears to use project.el.  It defines some new methods.  I'll need to brush up on CL objects and generic methods, though, before I can decipher how those interact with project.el.  In the meantime, does anyone have a basic setup and usage example?

Re: How to use project.el

[Jean Louis]

* excalamus--- via Users list for the GNU Emacs text editor <help-gnu-emacs@gnu.org> [2020-11-20 17:53]:

> With 27.1 out, I see talk of project.el, how great it is, and how
> it's been around for years.  What I don't see is anything about how
> to use it.  There's one sparse page in the manual, C-h i m emacs m
> Projects.  I've run project-find-file and project-search.  I fail to
> see how they're fundamentally different than find-file or grep.

Even me I was reading about it before some days. The difference is
is this below. 

File: emacs.info,  Node: Projects,  Next: Change Log,  Prev: Version Control,  Up: Maintaining

28.2 Working with Projects
==========================

A “project” is a collection of files used for producing one or more
programs.  Files that belong to a project are typically stored in a
hierarchy of directories; the top-level directory of the hierarchy is
known as the “project root”.

Re: How to use project.el

[Dmitry Gutov]

On 20.11.2020 16:52, excalamus--- via Users list for the GNU Emacs text 
editor wrote:
> With 27.1 out, I see talk of project.el, how great it is, and how it's been around for years.  What I don't see is anything about how to use it.  There's one sparse page in the manual, C-h i m emacs m Projects.  I've run project-find-file and project-search.  I fail to see how they're fundamentally different than find-file or grep.
> 
> I happen to have ESS installed which appears to use project.el.  It defines some new methods.  I'll need to brush up on CL objects and generic methods, though, before I can decipher how those interact with project.el.  In the meantime, does anyone have a basic setup and usage example?

It's basically zero setup. If you don't have a Git repo for your 
project, the setup is 'git init'. The repo is the project, and the 
project files are all the non-ignored files in its root directory 
(registered or not). At least, that's with the built-in 'vc' backend.

To get the most benefit from the package, you should install the latest 
version from GNU ELPA ('M-x list-packages', then install 'project'). The 
latest version is 0.5.2.

To see the full list of commands defined in that package, type 'C-x p C-h'.

You can also read the full description of the package with 'C-h P 
project RET'.

Re: How to use project.el

[excalamus--- via Users list for the GNU Emacs text editor]

> To get the most benefit from the package, you should install the latest version from GNU ELPA ('M-x list-packages', then install 'project'). The latest version is 0.5.2.
>
> To see the full list of commands defined in that package, type 'C-x p C-h'.
>
> You can also read the full description of the package with 'C-h P project RET'.
>
Thank you, I was unfamiliar with 'C-h P'.

I have updated to 0.5.2 and it's more like what I would have expected.  Actually, several functions mirror my own project management code.  My use-case is, of course, specific to me, but I see overlap.  I wonder if any of my code, or its ideas, may be useful?  Who is project.el's point person?

 I develop Python and prefer to interact through shell directly by sending commands to the comint buffer.   At this point the main command is 'xc/sh-send-command'.  This sends something like 'python main.py' or 'pyinstaller main.spec' to the appropriate shell and steps.  I create a shell, either by name or on the fly, similar to 'project-shell' with 'xc/create-shell'.  Everything else is just a helper.  

It's all hacky stuff I've made in the moment over time.  This weekend, I hoped to create project profiles, a plist or something, with :root, :venv, :entry-point-file, etc., so that I can switch between shells and associated command sets easily, maybe also formalize it (hence my looking into project.el first).

Thoughts?

(defun xc/set-global-default-directory (new-default-directory)
  "Set xc/global-default-directory to NEW-DEFAULT-DIRECTORY."
  (interactive "DSet global default directory: ")
  (setq xc/global-default-directory new-default-directory))

(defvar xc/python nil
  "Python interpreter to be used in shell calls.")

(defvar xc/shell "*shell*"
  "Shell process buffer to be used in shell calls.")

(defun xc/set-python (exe)
  "Set python executable."
   (interactive
   (list (read-file-name "Python executable: " "C:/Users/excalamus/Anaconda3/envs/" nil t)))
  (setq xc/python (concat exe " "))
  (message "Set `xc/python' to: %s" xc/python))

(defun xc/set-shell (pbuff)
  "Set `xc/shell' to process associated with PBUFF buffer."
  (interactive
   (list (read-buffer "Process buffer: " nil t '(lambda (x) (processp (get-buffer-process (car x)))))))
  (setq xc/shell pbuff)
  (message "Set `xc/shell' to: %s" xc/shell))

(defun xc/create-shell (name)
    "Create shell with a given NAME.

NAME should have earmuffs (e.g. *NAME*) if it is to follow Emacs
naming conventions.  Earmuffs indicate that the buffer is special
use and not associated with a file.

Returns newly created shell process.

Adapted from URL `https://stackoverflow.com/a/36450889/5065796'"
    (interactive
     (let ((name (read-string "shell name: " nil)))
       (list name)))
    (let ((name (or name xc/shell)))
      (get-buffer-process (shell name))))

(defun xc/sh-send-command (command &optional pbuff)
  "Send COMMAND to shell process with buffer PBUFF.

PBUFF is the buffer name string of a process.  If the process
associated with PBUFF does not exist, it is created.  PBUFF is
then opened in the other window and control is returned to the
calling buffer.

See URL `https://stackoverflow.com/a/7053298/5065796'"
  (let* ((pbuff (or pbuff xc/shell))
         (proc (or (get-buffer-process pbuff)
                   ;; create new process
                   (let ((currbuff (current-buffer))
                         (new-proc (xc/create-shell pbuff)))  ; creates a buried pbuff
                     (switch-to-buffer-other-window pbuff)
                     (switch-to-buffer currbuff)
                     new-proc)))
         (command-and-go (concat command "\n")))
    (with-current-buffer pbuff
      (goto-char (process-mark proc))
      (insert command-and-go)
      (move-marker (process-mark proc) (point)))
    (process-send-string proc command-and-go)))

(defun xc/set-global-shell-command (new-command)
  "Set `xc/global-shell-command' to NEW-COMMAND."
  (interactive "sShell command: ")
  (setq xc/global-shell-command new-command))

;; todo make interactive, read envs dir for available envs, send to shell, etc.
(defun xc/conda-activate ()
  "Activate conda venv."
  (interactive)
  (insert "C:\\Users\\excalamus\\Anaconda3\\condabin\\conda.bat activate "))

(defun xc/set-global-shell-command-to-current-file ()
  "Set the global shell command to use the current file.

This is useful if, for instance, a project was started using one
file, but later in development another file needs to be called
frequently.  It is like a permanent version of
`xc/buffer-file-to-shell'."
  (interactive)
  (setq xc/global-shell-command (concat xc/python (buffer-file-name)))
  (message "Set `xc/global-shell-command' to \"%s\"" xc/global-shell-command))

(defun xc/buffer-file-to-shell ()
  "Send current buffer file to shell as python call."
  (interactive)
  (let ((script (buffer-file-name)))
    (if script
        (xc/sh-send-command (concat xc/python script))
      (error "Command not sent. Buffer not visiting file"))))

Re: How to use project.el

[Dmitry Gutov]

On 21.11.2020 03:03, excalamus@tutanota.com wrote:
> 
> 
>> To get the most benefit from the package, you should install the latest version from GNU ELPA ('M-x list-packages', then install 'project'). The latest version is 0.5.2.
>>
>> To see the full list of commands defined in that package, type 'C-x p C-h'.
>>
>> You can also read the full description of the package with 'C-h P project RET'.
>>
> Thank you, I was unfamiliar with 'C-h P'.
> 
> I have updated to 0.5.2 and it's more like what I would have expected.  Actually, several functions mirror my own project management code.  My use-case is, of course, specific to me, but I see overlap.  I wonder if any of my code, or its ideas, may be useful?  Who is project.el's point person?

That would be me.

>   I develop Python and prefer to interact through shell directly by sending commands to the comint buffer.   At this point the main command is 'xc/sh-send-command'.  This sends something like 'python main.py' or 'pyinstaller main.spec' to the appropriate shell and steps.  I create a shell, either by name or on the fly, similar to 'project-shell' with 'xc/create-shell'.  Everything else is just a helper.
> 
> It's all hacky stuff I've made in the moment over time.  This weekend, I hoped to create project profiles, a plist or something, with :root, :venv, :entry-point-file, etc., so that I can switch between shells and associated command sets easily, maybe also formalize it (hence my looking into project.el first).
> 
> Thoughts?

In general, I like things to happen automatically for the user. For 
instance, if you are working on a Python project, shouldn't there be 
some file in the project root directory which will indicate which 
virtualenv to use?

Then your workflow could be something like call (project-root 
(project-current)), then read that file, then set the venv for the 
current buffer to the appropriate value.

And :root, is that project root? The idea in project.el is that you can 
write a function which will determine it, again, programmatically, 
based, say, on the current directory (usually by traversal up the tree 
and looking for some directory markers).

Of course, you can whip up a project backend which will plug into 
project-find-functions, but allow you, as the user, to set up everything 
manually, with manual switching between "profiles", etc. I'm just not 
sure why you would prefer that.

As for switching, there is the project-switch-project command.

Re: How to use project.el

[excalamus--- via Users list for the GNU Emacs text editor]

21 nov. 2020 à 21:55 de dgutov@yandex.ru:

>  For instance, if you are working on a Python project, shouldn't there be some file in the project root directory which will indicate which virtualenv to use?
>
Maybe? I've never heard of such a file being used with the venv module or conda environments. 

Re: How to use project.el

[Dmitry Gutov]

On 23.11.2020 18:22, excalamus--- via Users list for the GNU Emacs text 
editor wrote:
> 
> 21 nov. 2020 à 21:55 de dgutov@yandex.ru:
> 
>>   For instance, if you are working on a Python project, shouldn't there be some file in the project root directory which will indicate which virtualenv to use?
>>
> Maybe? I've never heard of such a file being used with the venv module or conda environments.

OK, I'm not well-versed in Python ecosystem, but there is pyenv:

   https://github.com/pyenv/pyenv

which apparently supports reading the python version from a file called 
.python-version, and an Emacs package that provides some sort of 
integration with it:

   https://github.com/pythonic-emacs/pyenv-mode

But even if those packages don't work to your liking, or together with 
"conda" (whatever that is), I think the general idea could be 
implemented in Emacs Lisp: you call (project-root (project-current)) and 
see if there is a file with a predetermined name inside that root. That 
file could point to a specific virtualenv which you would then switch to.

Re: How to use project.el

[excalamus--- via Users list for the GNU Emacs text editor]

23 nov. 2020 à 17:19 de dgutov@yandex.ru:

> On 23.11.2020 18:22, excalamus--- via Users list for the GNU Emacs text editor wrote:
>
>>
>> 21 nov. 2020 à 21:55 de dgutov@yandex.ru:
>>
>>>  For instance, if you are working on a Python project, shouldn't there be some file in the project root directory which will indicate which virtualenv to use?
>>>
>> Maybe? I've never heard of such a file being used with the venv module or conda environments.
>>
>
> OK, I'm not well-versed in Python ecosystem
>
I sadly am (it's my day job). "Should" is a dangerous assumption for Python. As far as I'm aware there's no unit of project nor standard way to indicate which environment to use or what packages are in that environment. Some people put a bajillion packages into a giant catch all environment to "reduce duplication", others have a 1:1 project to virtual environment relationship. 
In practice, a project and virtual environment are separate, although each are determined by a directory (maybe the virtual environment is a subdirectory of the project directory; maybe virtual environments are all co-located in a separate envs/ directory). The user has to remember which virtual environment to activate and where that environment is located.

The virtual environment is activated by running some script which sets environment variables for the current shell instance. 
> I think the general idea could be implemented in Emacs Lisp: you call (project-root (project-current)) and see if there is a file with a predetermined name inside that root. That file could point to a specific virtualenv which you would then switch to.
>
I could create my own project indicator file, I suppose.  I'm not sure how that would help.  But I'm getting (myself) lost. I may be stuck in my own pattern of solving things. What problem are we trying to solve here?

My project workflow is:
1. Create a shell2. Activate a virtual environment
3. Assign some commonly used commands based on the project (e.g. run this entry point, build according to this spec, etc.)
The trouble is managing several of these simultaneously (e.g. "Hey can you quickly fix this thing in this other project?") while maintaining the state of each.

Re: How to use project.el

[Dmitry Gutov]

On 24.11.2020 05:54, excalamus--- via Users list for the GNU Emacs text 
editor wrote:
> 
> 23 nov. 2020 à 17:19 de dgutov@yandex.ru:
> 
>> On 23.11.2020 18:22, excalamus--- via Users list for the GNU Emacs text editor wrote:
>>
>>>
>>> 21 nov. 2020 à 21:55 de dgutov@yandex.ru:
>>>
>>>>   For instance, if you are working on a Python project, shouldn't there be some file in the project root directory which will indicate which virtualenv to use?
>>>>
>>> Maybe? I've never heard of such a file being used with the venv module or conda environments.
>>>
>>
>> OK, I'm not well-versed in Python ecosystem
>>
> I sadly am (it's my day job). "Should" is a dangerous assumption for Python. As far as I'm aware there's no unit of project nor standard way to indicate which environment to use or what packages are in that environment. Some people put a bajillion packages into a giant catch all environment to "reduce duplication", others have a 1:1 project to virtual environment relationship.
> In practice, a project and virtual environment are separate, although each are determined by a directory (maybe the virtual environment is a subdirectory of the project directory; maybe virtual environments are all co-located in a separate envs/ directory). The user has to remember which virtual environment to activate and where that environment is located.

pyenv seems to be one such standard. Though of course it might not be 
adopted by the projects you use (especially if they are collaborative, 
and you don't make that decision).

> The virtual environment is activated by running some script which sets environment variables for the current shell instance.
>> I think the general idea could be implemented in Emacs Lisp: you call (project-root (project-current)) and see if there is a file with a predetermined name inside that root. That file could point to a specific virtualenv which you would then switch to.
>>
> I could create my own project indicator file, I suppose.  I'm not sure how that would help.  But I'm getting (myself) lost. I may be stuck in my own pattern of solving things. What problem are we trying to solve here?
> 
> My project workflow is:
> 1. Create a shell2. Activate a virtual environment
> 3. Assign some commonly used commands based on the project (e.g. run this entry point, build according to this spec, etc.)
> The trouble is managing several of these simultaneously (e.g. "Hey can you quickly fix this thing in this other project?") while maintaining the state of each.

You asked how to maintain a number of "profiles" corresponding to 
projects, each containing a venv, entry point, and so on.

My response is that in my preferred paradigm most or all of that 
information is inferred from the files that reside in the project's root 
directory. So to get some of that info you first find the project root, 
and then read that info from disk. You can subsequently cache it, of course.

But if there is no existing standard for most of those things in your 
projects, you might as well write some registry in Lisp, in your init 
script, that would map, say, project roots to such profiles.

The benefit of having those auto-determined is that you could always 
have them up-to-date and correspond to the current file you have opened 
in Emacs (the current buffer). And when you switch to another project, 
with project-switch-project or even the plain find-file, your 
profile-finding logic would use the correct profile automatically, 
without you having to recall which profile corresponds to which project.

But that's how I prefer to approach this kind of problem. You might 
choose to do it differently.

Re: How to use project.el

[excalamus--- via Users list for the GNU Emacs text editor]

24 nov. 2020 à 20:54 de dgutov@yandex.ru:

> But that's how I prefer to approach this kind of problem. You might choose to do it differently.
>

That makes sense. Thank you so much for taking the time to talk with me.