Navigating Lisp data structures

Navigating Lisp data structures

[Matt]

I'm writing an Emacs package and struggling to figure out how to represent the data in a Lisp data structure.  I'm getting lost in whether it should be an alist of alists, a plist of plists, an alist of plists, etc.  Or maybe it would be better to use a cl-struct or a hash or a class or a vector or a record.  Should I use keywords or not, etc.  

Here is a detailed explanation of my issue:

My Emacs package manages my project workflows (e.g. Python or C).  Each project has a name, associated files, one or more shell processes (comints), as well as commands for a specific shell and commands used across the shells.  It also sets my window layout.

In Python, I might structure the data something like this:

projects = {
    "my_c_project": {
        'shells': {
            "*build*": {"root": "/path/to/build/directory/", "setup": ("export MYBUILDVAR1=1", "export MYBUILDVAR2=0")},
            "*run*": {"root": "/path/to/run/directory/", "setup": ("export MYRUNVAR=1")}
        },
        'commands': ("./build_my_project.sh", "./run_my_build"),
        'files': ("/path/to/entrypoint.c", "/path/to/other/file/I/care/about.c"),
        'window_split': ("quad", ),  # C-x 3 C-x 2 other-window other-window C-x 2
    },
    "my_python_project": {
        'shells': {"*my_python_project*":  {"root": "/path/to/my/python/project", "setup": ("source venv/bin/activate")}},
        'commands': ("python3 -m my_python_project",
                     "python3 -m unittest discover tests/ --failfast --quiet", ),
        'files': ("/path/to/my_python_project/entry_point.py", ),
        'window_split': ("half", ), # C-x 3
    }
}

I could then get the information I need through chaining look ups.  If we pretended that Emacs used Python, activating a project might look something like this:

(Pdb) # create shells
(Pdb) for name in projects['my_python_project']['shells']: pass  # call comint creation command
(Pdb) # setup shell environments
(Pdb) for command in projects['my_python_project']['shells']['*my_python_project*']['setup']: pass  # pass each command in the comint
(Pdb) # load files
(Pdb) for file in projects['my_python_project']['files']: pass  #  run emacs find-file
(Pdb) # and so on

Of course the data could be represented using classes instead of dicts.  However, I hope the problem is clear.

I'm obviously looking at the problem through the lens of Python key-value structures.  When I try to translate the data structure to something like nested plists, it feels messy.  Maybe that's me simply being uncomfortable with nested lists?  When I mix list types (e.g. an alist of plists), the lookup syntax becomes inconsistent (a mix of assocs and plist-gets, scattered with cars and cdrs and weird cases in which I have a list of strings I can't seem to access).  

Guidance would be sincerely appreciated.

Re: Navigating Lisp data structures

[Joost Kremers]

On Sun, Dec 04 2022, Matt wrote:

> I'm obviously looking at the problem through the lens of Python key-value
> structures. When I try to translate the data structure to something like
> nested plists, it feels messy. Maybe that's me simply being uncomfortable with
> nested lists? When I mix list types (e.g. an alist of plists), the lookup
> syntax becomes inconsistent (a mix of assocs and plist-gets, scattered with
> cars and cdrs

Note that there's also `alist-get`, which should at least help in reducing the
number of cars and cdrs.

You may also want to check out `let-alist`, which actually gives a nice way to
access the elements inside a nested alist.


-- 
Joost Kremers
Life has its moments

Re: Navigating Lisp data structures

[Eric Abrahamsen]

Matt <matt@excalamus.com> writes:

> I'm writing an Emacs package and struggling to figure out how to represent the data in a Lisp data structure.  I'm getting lost in whether it should be an alist of alists, a plist of plists, an alist of plists, etc.  Or maybe it would be better to use a cl-struct or a hash or a class or a vector or a record.  Should I use keywords or not, etc.  
>
> Here is a detailed explanation of my issue:
>
> My Emacs package manages my project workflows (e.g. Python or C).  Each project has a name, associated files, one or more shell processes (comints), as well as commands for a specific shell and commands used across the shells.  It also sets my window layout.
>
> In Python, I might structure the data something like this:
>
> projects = {
>     "my_c_project": {
>         'shells': {
>             "*build*": {"root": "/path/to/build/directory/", "setup": ("export MYBUILDVAR1=1", "export MYBUILDVAR2=0")},
>             "*run*": {"root": "/path/to/run/directory/", "setup": ("export MYRUNVAR=1")}
>         },
>         'commands': ("./build_my_project.sh", "./run_my_build"),
>         'files': ("/path/to/entrypoint.c", "/path/to/other/file/I/care/about.c"),
>         'window_split': ("quad", ),  # C-x 3 C-x 2 other-window other-window C-x 2
>     },
>     "my_python_project": {
>         'shells': {"*my_python_project*":  {"root": "/path/to/my/python/project", "setup": ("source venv/bin/activate")}},
>         'commands': ("python3 -m my_python_project",
>                      "python3 -m unittest discover tests/ --failfast --quiet", ),
>         'files': ("/path/to/my_python_project/entry_point.py", ),
>         'window_split': ("half", ), # C-x 3
>     }
> }

Wow, it looks like you're recreating complete project management
facilities from scratch! That's impressive, and also a bit terrifying.

Your adventurous spirit is to be commended, but you might _also_ look
into making use of more of Emacs' built-in facilities for this stuff.
Emacs has projects, and projects have `project-compile', which calls
`compile', and a bunch of the config above looks like it could be worked
into existing facilities.

There are/have been some truly monster discussion threads recently in
emacs.devel and the bug tracker about how to expand and make use of
projects, so that's very much in the process of evolution. I haven't
seen anyone suggest incorporating window layout into projects, and I'll
bet that some people would find that interesting.

You might not want to wade into the midst of a firefight, but now is the
right time to be asking these questions, and showing people how you use
projects, and would like to use them.

Just a suggestion!

Eric

Re: Navigating Lisp data structures

[Matt]

 ---- On Sun, 04 Dec 2022 19:46:20 -0500  Joost Kremers  wrote --- 

 > Note that there's also `alist-get`, which should at least help in reducing the
 > number of cars and cdrs.
 > 
 > You may also want to check out `let-alist`, which actually gives a nice way to
 > access the elements inside a nested alist.
 
Yes, I had over looked `alist-get` and was unaware of `let-alist`.  Thank you.

For posterity, should anyone have similar questions, here's a translation of my nested Python dict into an alist, along with how to access it using `alist-get` and `let-alist`:

(setq project-alist
       '((my_c_project .
                       ((shells .
                                ((build .
                                        ((root . "/path/to/build/directory/")
                                         (setup . ("export MYBUILDVAR1=1"
                                                   "export MYBUILDVAR2=0"))))
                                 (run .
                                      ((root . "/path/to/run/directory/")
                                       (setup . ("export MYRUNVAR=1"))))))
                        (commands . ("./build_my_project.sh"
                                     "./run_my_build"))
                        (files . ("/path/to/entrypoint.c"
                                  "/path/to/other/file/I/care/about.c"))
                        (window-split . :quad)))
         (my-python-project .
                            ((shells .
                                     ((my-python-project .
                                                         ((root . "/path/to/my/python/project")
                                                          (setup . ("source venv/bin/activate"))))))
                             (commands . ("python3 -m my_python_project"
                                          "python3 -m unittest discover tests/ --failfast --quiet"))
                             (files . ("/path/to/my_python_project/entry_point.py"))
                             (window-split . :half)))))

(alist-get 'my_c_project project-alist)

(let-alist project-alist
  .my-python-project.shells.my-python-project.root)

Re: Navigating Lisp data structures

[Matt]

 ---- On Sun, 04 Dec 2022 21:05:13 -0500  Eric Abrahamsen  wrote --- 
 > Wow, it looks like you're recreating complete project management
 > facilities from scratch! That's impressive, and also a bit terrifying.

Yeah, pretty much :) 

I've been refactoring a package I wrote and have been using for a few years now, peut-gerer (https://codeberg.org/excalamus/peut-gerer).   I found I'd written a handful of utilities sharing the loose theme of "project workflow management" and so packaged them up.  I recall at the time project.el being largely undocumented (it seems to have an info entry now) and was, at least to me, incoherent.  I'm glad to see it's been developed since then.  I also found myself reading the projectile documentation trying to figure out how to do what I wanted instead of actually coding.  Why spend 30 minutes reading when you could spend 30 hours programming, right?

 > Your adventurous spirit is to be commended, but you might _also_ look
 > into making use of more of Emacs' built-in facilities for this stuff.
 > Emacs has projects, and projects have `project-compile', which calls
 > `compile', and a bunch of the config above looks like it could be worked
 > into existing facilities.

I agree, it looks like what I'm doing could be fit to project.el.  I appreciate you mentioning it because I was able to steal some ideas >:)  Unfortunately, I still find the documentation for project.el lacking.  Specifically, there appears to be nothing about how to actually define a project.  All the commands assume one exists.  I'm on 28.2, though, so maybe the documentation and code is different on HEAD.  

 > There are/have been some truly monster discussion threads recently in
 > emacs.devel and the bug tracker about how to expand and make use of
 > projects, so that's very much in the process of evolution. I haven't
 > seen anyone suggest incorporating window layout into projects, and I'll
 > bet that some people would find that interesting.
 > 
 > Just a suggestion!
 
Thanks for the suggestion and for welcoming my opinion!  

Re: Navigating Lisp data structures

[Eric Abrahamsen]

Matt <matt@excalamus.com> writes:

>  ---- On Sun, 04 Dec 2022 21:05:13 -0500  Eric Abrahamsen  wrote --- 
>  > Wow, it looks like you're recreating complete project management
>  > facilities from scratch! That's impressive, and also a bit terrifying.
>
> Yeah, pretty much :) 
>
> I've been refactoring a package I wrote and have been using for a few
> years now, peut-gerer (https://codeberg.org/excalamus/peut-gerer). I
> found I'd written a handful of utilities sharing the loose theme of
> "project workflow management" and so packaged them up. I recall at the
> time project.el being largely undocumented (it seems to have an info
> entry now) and was, at least to me, incoherent. I'm glad to see it's
> been developed since then. I also found myself reading the projectile
> documentation trying to figure out how to do what I wanted instead of
> actually coding. Why spend 30 minutes reading when you could spend 30
> hours programming, right?

Ugh, I know that mindset...

>  > Your adventurous spirit is to be commended, but you might _also_ look
>  > into making use of more of Emacs' built-in facilities for this stuff.
>  > Emacs has projects, and projects have `project-compile', which calls
>  > `compile', and a bunch of the config above looks like it could be worked
>  > into existing facilities.
>
> I agree, it looks like what I'm doing could be fit to project.el. I
> appreciate you mentioning it because I was able to steal some ideas
>>:) Unfortunately, I still find the documentation for project.el
> lacking. Specifically, there appears to be nothing about how to
> actually define a project. All the commands assume one exists. I'm on
> 28.2, though, so maybe the documentation and code is different on
> HEAD.

And like I said, it's all under very active development, and I think
will end up looking much different (and much more feature-complete) in a
few months' time. Personally I'm making the minimal effective use of
project.el (and eglot), and waiting for the dust to clear to invest more
time in customization. But I do think they appreciate hearing feature
requests and use cases.