Dev workflow in Emacs with containers

Dev workflow in Emacs with containers

[Sébastien Gendre]

[-- Attachment #1: Type: text/plain, Size: 1237 bytes --]

Hello,

I got a few questions about how to integrate Emacs with a development
workflow who use containers.


First, a bit of context:

Emacs provide some great tools to integrate with the dev workflow:

- The `compile` function to compile or to run tests

- Integration with multiple debuggers (GDB, PDB, etc)

- Integration with multiple REPL (Python, Lua, etc)

- Shell and terminal to launch the newly developed software


These tools are great when the complied/tested/debugged/run software is
local. But I use more and more the containers. Even for micro-controller
firmware dev.

I use Podman, but the situation is similar with Docker. And my Emacs
process run outside of the containers.


My questions are:

What are your experience with using Emacs, the tools cited above and
containers ?

Do you use a `.dir-locals.el` to set the commands run by Emacs to
debugg/compile/etc ? Or do you use a Makefile with pre-defined targets
for test, debug, build, etc ?

How do you manage when Emacs highlight a link to a file, but the
obtained path correspond to the container file system while Emacs can
access this file from the host file system ?


Best regards

-------
Gendre Sébastien

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 849 bytes --]

Re: Dev workflow in Emacs with containers

[Yuri Khan]

On Fri, 29 Nov 2024 at 21:41, Sébastien Gendre <seb@k-7.ch> wrote:

> Emacs provide some great tools to integrate with the dev workflow:
>
> These tools are great when the complied/tested/debugged/run software is
> local. But I use more and more the containers. Even for micro-controller
> firmware dev.
>
> My questions are:
>
> What are your experience with using Emacs, the tools cited above and
> containers ?

Emacs integrates excellently with containerized tools.

> Do you use a `.dir-locals.el` to set the commands run by Emacs to
> debugg/compile/etc ? Or do you use a Makefile with pre-defined targets
> for test, debug, build, etc ?

1. I have a Dockerfile or several that describes the build
environment. If necessary, a different one that describes the
runtime/debugging environment.

2. I have a docker-compose.yml which utilizes the image(s) built from
Dockerfile(s) of step 1, and mounts the source tree, the build
directory, etc into service containers. The compose file also arranges
for the services to run as my user ID so that I don’t get root-owned
files in my build tree.

The purpose of the compose file is to remember all the docker run
options for me so that I don’t have to specify them every time. I
could alternatively use shell scripts.

3. I also have a Makefile that encapsulates the build logic. In some
targets, it arranges for itself to execute in a service container
(with a different target). The purpose of the Makefile is that I could
run it either from Emacs or from local shell if I so desire.

4. .dir-locals.el to tell Emacs to use ‘docker-compose run --rm
runtime gdb’ as my debugger and ‘docker-compose run --rm builder
clangd’ as my language server, and ‘make build test’ as my compile
command.

> How do you manage when Emacs highlight a link to a file, but the
> obtained path correspond to the container file system while Emacs can
> access this file from the host file system ?

Where practical, I prefer to mount my volumes at the same paths in the
container as locally, this simplifies the mapping a lot. But in some
cases I hook ‘compilation-filter’ to a function that replaces
prefixes.

Re: Dev workflow in Emacs with containers

[Leo Butler]

On Sat, Nov 30 2024, Yuri Khan <yuri.v.khan@gmail.com> wrote:

> On Fri, 29 Nov 2024 at 21:41, Sébastien Gendre <seb@k-7.ch> wrote:
>
>> Emacs provide some great tools to integrate with the dev workflow:
>>
>> These tools are great when the complied/tested/debugged/run software is
>> local. But I use more and more the containers. Even for micro-controller
>> firmware dev.
>>
>> My questions are:
>>
>> What are your experience with using Emacs, the tools cited above and
>> containers ?
>
> Emacs integrates excellently with containerized tools.
>
>> Do you use a `.dir-locals.el` to set the commands run by Emacs to
>> debugg/compile/etc ? Or do you use a Makefile with pre-defined targets
>> for test, debug, build, etc ?
>
> 1. I have a Dockerfile or several that describes the build
> environment. If necessary, a different one that describes the
> runtime/debugging environment.
>
> 2. I have a docker-compose.yml which utilizes the image(s) built from
> Dockerfile(s) of step 1, and mounts the source tree, the build
> directory, etc into service containers. The compose file also arranges
> for the services to run as my user ID so that I don’t get root-owned
> files in my build tree.
>
> The purpose of the compose file is to remember all the docker run
> options for me so that I don’t have to specify them every time. I
> could alternatively use shell scripts.
>
> 3. I also have a Makefile that encapsulates the build logic. In some
> targets, it arranges for itself to execute in a service container
> (with a different target). The purpose of the Makefile is that I could
> run it either from Emacs or from local shell if I so desire.
>
> 4. .dir-locals.el to tell Emacs to use ‘docker-compose run --rm
> runtime gdb’ as my debugger and ‘docker-compose run --rm builder
> clangd’ as my language server, and ‘make build test’ as my compile
> command.

Yuri, would you be willing to share the stuff you mention?

TIA,
Leo

>
>> How do you manage when Emacs highlight a link to a file, but the
>> obtained path correspond to the container file system while Emacs can
>> access this file from the host file system ?
>
> Where practical, I prefer to mount my volumes at the same paths in the
> container as locally, this simplifies the mapping a lot. But in some
> cases I hook ‘compilation-filter’ to a function that replaces
> prefixes.

Re: Dev workflow in Emacs with containers

[Yuri Khan]

On Sat, 30 Nov 2024 at 01:20, Leo Butler <Leo.Butler@umanitoba.ca> wrote:
>
> On Sat, Nov 30 2024, Yuri Khan <yuri.v.khan@gmail.com> wrote:
>
> > Emacs integrates excellently with containerized tools.
> >
> > 1. a Dockerfile
> > 2. a docker-compose.yml
> > 3. a Makefile
> > 4. .dir-locals.el

> Yuri, would you be willing to share the stuff you mention?

The stuff, as you aptly call it, is, on the one hand, always somewhat
project-specific, and on the other hand, reasonably trivial to write
(as in that joke where a math professor says “trivial” and then
disappears for 40 minutes to ascertain the triviality).

I feel the proper way to share that would be a tutorial article. In
fact, some of my coẅorkers would benefit from such a tutorial. So I
might try my hand at writing it, but I can’t promise any specific
timeframe.

Re: Dev workflow in Emacs with containers

[Sébastien Gendre]

[-- Attachment #1: Type: text/plain, Size: 1756 bytes --]

Thank you for your reply.

I added questions/remarks bellow.


Yuri Khan <yuri.v.khan@gmail.com> writes:
> 1. I have a Dockerfile or several that describes the build
> environment. If necessary, a different one that describes the
> runtime/debugging environment.
>
> 2. I have a docker-compose.yml which utilizes the image(s) built from
> Dockerfile(s) of step 1, and mounts the source tree, the build
> directory, etc into service containers. The compose file also arranges
> for the services to run as my user ID so that I don’t get root-owned
> files in my build tree.
>
> The purpose of the compose file is to remember all the docker run
> options for me so that I don’t have to specify them every time. I
> could alternatively use shell scripts.
>
> 3. I also have a Makefile that encapsulates the build logic. In some
> targets, it arranges for itself to execute in a service container
> (with a different target). The purpose of the Makefile is that I could
> run it either from Emacs or from local shell if I so desire.
>
> 4. .dir-locals.el to tell Emacs to use ‘docker-compose run --rm
> runtime gdb’ as my debugger and ‘docker-compose run --rm builder
> clangd’ as my language server, and ‘make build test’ as my compile
> command.

Seems a good way to do it.

I haven't think about the language server.
How do you configure the container ?


> Where practical, I prefer to mount my volumes at the same paths in the
> container as locally, this simplifies the mapping a lot. But in some
> cases I hook ‘compilation-filter’ to a function that replaces
> prefixes.

I'm very curious about your compilation filter. Can you share it ?



Best regards

-------
Gendre Sébastien

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 849 bytes --]

Re: Dev workflow in Emacs with containers

[Yuri Khan]

On Sun, 1 Dec 2024 at 01:21, Sébastien Gendre <seb@k-7.ch> wrote:

> Seems a good way to do it.
>
> I haven't think about the language server.
> How do you configure the container ?

I install the language server in the same image that is used for
building. This way, it has access to the same version of the standard
library as the compiler does.

If the project is set up to use system-packaged libraries, I install
them in the same image.

I bind-mount the sources and vendored third-party libraries. This way,
I do not need to rebuild the image whenever they change.

All this taken together ensures that (1) the language server sees the
same sources and libraries that the compiler sees, and (2) I don’t
need to install the same libraries on my host machine, which would
defeat the point of dockerization.

Another thing to look out for is where the language server caches its
data. This should also be bind-mounted so that it persists between
sessions.

> > Where practical, I prefer to mount my volumes at the same paths in the
> > container as locally, this simplifies the mapping a lot. But in some
> > cases I hook ‘compilation-filter’ to a function that replaces
> > prefixes.
>
> I'm very curious about your compilation filter. Can you share it ?

As I said, mounting in matching paths is a lot simpler.

Suppose I have the project root at /home/me/work/project on the host,
but I mount it at /mnt/project in the container.

The compiler will report its messages with the file names starting
with /mnt/project. Then when I instruct Emacs to go to the error
location, it will not find the file. It may ask me for assistance.

So I have this somewhere in my load-path:

    ;; yk-compile.el
    (defvar yk-compilation-filter-alist nil
      "List of (REGEXP REPLACEMENT) to apply to compilation messages.")

    (defun yk-compilation-filter ()
      "Apply replacements to compilation messages."
      (save-excursion
        (pcase-dolist (`(,regexp ,replacement) yk-compilation-filter-alist)
          (goto-char compilation-filter-start)
          (beginning-of-line)
          (while (re-search-forward regexp nil t)
            (replace-match replacement nil nil)))))

    (provide 'yk-compile)

(this is probably not the most efficient way to do it, but it works for me).

In the project root, I have a .dir-locals.el that sets up the
replacements for this project:

    ((nil . ((yk-compilation-filter-alist
              . (("/mnt/project/" "/home/me/work/project/"))))))

(Now that I think about it, I could probably achieve a similar effect
using the ‘compilation-transform-file-match-alist’ variable, except
that it only helps with the issue of *jumping* to error location and
not solving the *display* of file paths.)

Additionally, with sources mounted in a path different from the host,
the language server will also talk in terms of container paths. It
will expect paths starting with ‘/mnt/project’ in the
textDocument/didOpen notifications, and report that path in its
textDocument/references responses. clangd happens to have a path
mapping feature which is specified in its command line.

My only motivation for mounting the sources in a path different from
the host is that I want to have multiple working copies of the project
on the host, so that I could work with multiple feature branches at
the same time; and mounting them in the same place in the container
allows me to speed up compilation using ccache. If compilation time is
not an issue, I much prefer the ease of transparent paths.

Re: Dev workflow in Emacs with containers

[James Thomas]

Sébastien Gendre wrote:

> Hello,
>
> I got a few questions about how to integrate Emacs with a development
> workflow who use containers.
>
>
> First, a bit of context:
>
> Emacs provide some great tools to integrate with the dev workflow:
>
> - The `compile` function to compile or to run tests
>
> - Integration with multiple debuggers (GDB, PDB, etc)
>
> - Integration with multiple REPL (Python, Lua, etc)
>
> - Shell and terminal to launch the newly developed software
>
>
> These tools are great when the complied/tested/debugged/run software is
> local. But I use more and more the containers. Even for micro-controller
> firmware dev.
>
> I use Podman, but the situation is similar with Docker. And my Emacs
> process run outside of the containers.
>
>
> My questions are:
>
> What are your experience with using Emacs, the tools cited above and
> containers ?
>
> Do you use a `.dir-locals.el` to set the commands run by Emacs to
> debugg/compile/etc ? Or do you use a Makefile with pre-defined targets
> for test, debug, build, etc ?
>
> How do you manage when Emacs highlight a link to a file, but the
> obtained path correspond to the container file system while Emacs can
> access this file from the host file system ?

Have you tried simply with Tramp's docker/podman methods?

--