[ELPA] new package: tramp-docker

[ELPA] new package: tramp-docker

[Brian Cully via Emacs development discussions.]

This package allows Tramp to use Docker (and Podman) containers as its 
remote environment.

Note that there is an extant 'docker-tramp' module in MELPA, so there 
may be some confusion. I wrote this one in order to get it into ELPA, 
and, as such, I've assigned copyright to the FSF.

The repository lives at: https://git.spork.org/tramp-docker.git

Docker is a big enough target that it may be worth integrating this 
directly in Emacs rather than going through ELPA. I'm not currently set 
up to do this, but if anyone wants to integrate it and submit it 
upstream, you have my blessing.

-bjc

Re: [ELPA] new package: tramp-docker

[Philip Kaludercic]

Brian Cully via "Emacs development discussions." <emacs-devel@gnu.org>
writes:

> This package allows Tramp to use Docker (and Podman) containers as its
> remote environment.
>
> Note that there is an extant 'docker-tramp' module in MELPA, so there
> may be some confusion. I wrote this one in order to get it into ELPA,
> and, as such, I've assigned copyright to the FSF.
>
> The repository lives at: https://git.spork.org/tramp-docker.git
>
> Docker is a big enough target that it may be worth integrating this
> directly in Emacs rather than going through ELPA. I'm not currently
> set up to do this, but if anyone wants to integrate it and submit it
> upstream, you have my blessing.

I certainly think this would be a good addition, and the package appears
to be short enough for it to be added to an existing file.

> -bjc

Re: [ELPA] new package: tramp-docker

[Michael Albinus]

Brian Cully via "Emacs development discussions." <emacs-devel@gnu.org>
writes:

Hi Brian,

> This package allows Tramp to use Docker (and Podman) containers as its
> remote environment.

Thanks!

> Docker is a big enough target that it may be worth integrating this
> directly in Emacs rather than going through ELPA. I'm not currently
> set up to do this, but if anyone wants to integrate it and submit it
> upstream, you have my blessing.

I haven't reviewed the code yet (will do next days), but I believe we
could add this to the Tramp package. By this, it could be distributed
over ELPA as well, and it would profit from general Tramp improvements.

Well, you have specified a dependency from Emacs 23. Integrating it into
Tramp 2.5 (the recent ELPA version), would change this to Emacs 25. But
that shouldn't be a problem in practice I believe.

WDYT?

> -bjc

Best regards, Michael.

Re: [ELPA] new package: tramp-docker

[Michael Albinus]

Brian Cully <bjc@spork.org> writes:

Hi Brian,

> On 9/23/22 13:47, Michael Albinus wrote:
>> I haven't reviewed the code yet (will do next days), but I believe we
>> could add this to the Tramp package. By this, it could be distributed
>> over ELPA as well, and it would profit from general Tramp improvements.
>
> That sounds good to me. Let me know what I need to do that'd make it easier.

Let me read the code over the weekend. Likely, we can simply add your
(slighly modified) file to the Tramp repo. After that, we can think
about enhancements, like Tramp manual update, implementing host name
completion, etc. This stuff.

>> Well, you have specified a dependency from Emacs 23. Integrating it into
>> Tramp 2.5 (the recent ELPA version), would change this to Emacs 25. But
>> that shouldn't be a problem in practice I believe.
>
> Emacs 29 would be fine with me. ;)
>
> -bjc

Best regards, Michael.

Re: [ELPA] new package: tramp-docker

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

Hi Brian,

> Let me read the code over the weekend. Likely, we can simply add your
> (slighly modified) file to the Tramp repo. After that, we can think
> about enhancements, like Tramp manual update, implementing host name
> completion, etc. This stuff.

I've just seen there's already completion functionality. Hmm, let me
read (test) the code, before I make further suggestions.

Best regards, Michael.

Re: [ELPA] new package: tramp-docker

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Could you tell me a bit more about what it means for Tramp to "use
Docker containers" as the remote environment?

As I recall, Tramp connects to another computer using a standard
protocol and transmits files that way.  Doing this, Tramp would not
know how the other machine implements the protocol.  For instance, if
it used ftp (in the past), how that machine runs an ftp server was not
Tramp's concern.  Likewise if it uses http.

Thus, Tramp would neither know nor care whether the service is
implemented with a container.

Why then does Tramp need amy special code for the case where
that service is implemented with a container?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [ELPA] new package: tramp-docker

[Robin Tarsiger]

Richard Stallman wrote:
> Could you tell me a bit more about what it means for Tramp to "use
> Docker containers" as the remote environment?

This code is used to access files in Docker or Podman containers that are
running on the same system as Emacs. It calls the Docker or Podman program
to spawn a shell inside the container to communicate with. It is similar
to the su or sudo Tramp methods, in that the connection to the "remote"
system involves shared kernel resources (unless Docker or Podman itself
eventually chooses to do something else).

-RTT

Re: [ELPA] new package: tramp-docker

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

Hi Brian,

>> Let me read the code over the weekend. Likely, we can simply add your
>> (slighly modified) file to the Tramp repo. After that, we can think
>> about enhancements, like Tramp manual update, implementing host name
>> completion, etc. This stuff.
>
> I've just seen there's already completion functionality. Hmm, let me
> read (test) the code, before I make further suggestions.

It works almost in the Tramp environment. So I propose I add it (in your
name) to the Tramp and Emacs git repositories, master branch. After that
I'll apply some changes for better integration, like removing
tramp-docker-setup and friends, which aren't needed anymore.

There are some minor problems with tramp-docker--completion-function,
because I'm running Fedora 36 and using the "docker" command, which emits
an additional message when /etc/containers/nodocker doesn't exist. But
these things we can fix afterwards.

You have a comment ";; todo: check tramp-async-args and
tramp-direct-async" which I didn't follow yet, but this can be
investigated also afterwards. Also multi-hops I didn't check yet.

Do you agree? With this approach, tramp-docker.el would be part of Tramp
2.6 and Emacs 29. Later, when Tramp 2.6 is distributed via GNU ELPA, it
will be available also for older Emacsen down to Emacs 26. Roughly, I
expect this for the end of this year (but pls don't beat me when there
are delays).

Best regards, Michael.

Re: [ELPA] new package: tramp-docker

[Michael Albinus]

Robin Tarsiger <rtt@dasyatidae.com> writes:

Hi,

> Richard Stallman wrote:
>> Could you tell me a bit more about what it means for Tramp to "use
>> Docker containers" as the remote environment?
>
> This code is used to access files in Docker or Podman containers that are
> running on the same system as Emacs. It calls the Docker or Podman program
> to spawn a shell inside the container to communicate with. It is similar
> to the su or sudo Tramp methods, in that the connection to the "remote"
> system involves shared kernel resources (unless Docker or Podman itself
> eventually chooses to do something else).

When the virtual host a running container exist of offers services like
"ftpd" or "sshd", the container method isn't needed. But there are
containers w/o such services, like the famous "alpine" container. A
dedicated Tramp connection via the "docker" method will be useful.

Furthermore, this isn't restricted to a container running on a local
machine. You could always use Tramp's multi-hops, viewing a file like
"/ssh:rms@another-machine|docker:that-container:~/.history".

Furthermore, there are techniques to access containers on a remote host
like podman-remote(1). I'm not familiar with such techniques (yet), but
I guess we could exploit them as well.

> -RTT

Best regards, Michael.

Re: [ELPA] new package: tramp-docker

[Michael Albinus]

Brian Cully <bjc@spork.org> writes:

Hi Brian,

[pls keep the Cc, for the archives]

> On 9/24/22 06:34, Michael Albinus wrote:
>> It works almost in the Tramp environment. So I propose I add it (in your
>> name) to the Tramp and Emacs git repositories, master branch. After that
>> I'll apply some changes for better integration, like removing
>> tramp-docker-setup and friends, which aren't needed anymore.
>
> This sounds fine to me.

OK, done. tramp-docker.el exists now in both the Tramp and Emacs
repositories. Please check my changes, whether everything looks proper
to you.

>> You have a comment ";; todo: check tramp-async-args and
>> tramp-direct-async" which I didn't follow yet, but this can be
>> investigated also afterwards.
>
> That comment is there because I stumbled across the variable while
> looking at some re-entrant Tramp errors, and it seemed like it may
> solve them, but I couldn't find enough explanation to see if it would
> apply to the docker case (though it seems like it should?). Feel free
> to remove it if it suits you to do so.

The idea of these settings is to improve asynchronous processes
speed. It might also reduce the "forbidden reentrant call" errors in
process sentinels and filters, but they cannot be avoided in general,
because they happen also for timers. I will check tomorrow or so whether
we could apply these settings.

Another question: You have

--8<---------------cut here---------------start------------->8---
         (tramp-login-args (("exec")
                            ("-it")
                            ("-u" "%u")
                            ("%h")
			    ("/bin/sh")))
         (tramp-remote-shell "/bin/sh")
--8<---------------cut here---------------end--------------->8---

Are you sure, that /bin/sh will always exist in containers? And shouldn't
we rather use tramp-default-remote-shell instead of hard-coding
"/bin/sh", for the benefit of customization?

> -bjc

Best regards, Michael.

Re: [ELPA] new package: tramp-docker

[Brian Cully via Emacs development discussions.]

On 9/24/22 12:56, Michael Albinus wrote:
> OK, done. tramp-docker.el exists now in both the Tramp and Emacs
> repositories. Please check my changes, whether everything looks proper
> to you.

Looks good at first glance. I don't really have time to check 
functionality right now, but nothing stands out.


> Another question: You have
> 
> --8<---------------cut here---------------start------------->8---
>           (tramp-login-args (("exec")
>                              ("-it")
>                              ("-u" "%u")
>                              ("%h")
> 			    ("/bin/sh")))
>           (tramp-remote-shell "/bin/sh")
> --8<---------------cut here---------------end--------------->8---
> 
> Are you sure, that /bin/sh will always exist in containers?

No, I don't think it's possible to know that for sure. But I think 
Docker only supports *nix containers, in which case it's a reasonable 
assumption. Even Guix has /bin/sh and /usr/bin/env.

> And shouldn't
> we rather use tramp-default-remote-shell instead of hard-coding
> "/bin/sh", for the benefit of customization?

I wasn't aware of `tramp-default-remote-shell' at all, but it seems like 
that's what should be used rather than hardcoding the value.

-bjc

Re: new package: tramp-docker

[Michael Albinus]

Brian Cully <bjc@spork.org> writes:

Hi Brian,

>> Are you sure, that /bin/sh will always exist in containers?
>
> No, I don't think it's possible to know that for sure. But I think
> Docker only supports *nix containers, in which case it's a reasonable
> assumption. Even Guix has /bin/sh and /usr/bin/env.

It's almost true. But on Android, for example, it doesn't exist. There's
/system/bin/sh instead.

>> And shouldn't
>> we rather use tramp-default-remote-shell instead of hard-coding
>> "/bin/sh", for the benefit of customization?
>
> I wasn't aware of `tramp-default-remote-shell' at all, but it seems
> like that's what should be used rather than hardcoding the value.

I've changed it accordingly. And I've also changed
tramp-docker--completion-function in order to make it more robust.

I've also tested, whether tramp-direct-async works out-of-the box, it
doesn't. I'm pretty sure we could make it running, it just needs some
more love.

> -bjc

Best regards, Michael.

Re: [ELPA] new package: tramp-docker

[Philippe Vaucher]

[-- Attachment #1: Type: text/plain, Size: 308 bytes --]

>
> Note that there is an extant 'docker-tramp' module in MELPA, so there
> may be some confusion. I wrote this one in order to get it into ELPA,
> and, as such, I've assigned copyright to the FSF.


So, if I understand correctly tramp-docker is a rewrite of docker-tramp
just so it can be included in ELPA?

[-- Attachment #2: Type: text/html, Size: 524 bytes --]

Re: [ELPA] new package: tramp-docker

[Brian Cully via Emacs development discussions.]

[resending to emacs-devel, as I managed to drop it from the Cc list]


On 10/3/22 09:03, Philippe Vaucher wrote:
> So, if I understand correctly tramp-docker is a rewrite of docker-tramp 
> just so it can be included in ELPA?

This is correct. Although it has now been moved into Tramp directly so 
it will be included in Emacs 29.1.

-bjc

Re: [ELPA] new package: tramp-docker

[Michael Albinus]

Brian Cully via "Emacs development discussions." <emacs-devel@gnu.org>
writes:

> On 10/3/22 09:03, Philippe Vaucher wrote:
>> So, if I understand correctly tramp-docker is a rewrite of
>> docker-tramp just so it can be included in ELPA?
>
> This is correct. Although it has now been moved into Tramp directly so
> it will be included in Emacs 29.1.

... and back in GNU ELPA for the goodness of older Emacs versions, once
Tramp 2.6 has been released.

> -bjc

Best regards, Michael.

Re: [ELPA] new package: tramp-docker

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > This code is used to access files in Docker or Podman containers that are
  > running on the same system as Emacs. It calls the Docker or Podman program
  > to spawn a shell inside the container to communicate with. It is similar
  > to the su or sudo Tramp methods, in that the connection to the "remote"
  > system involves shared kernel resources (unless Docker or Podman itself
  > eventually chooses to do something else).

Thanks for explaining.  My overload is such that I just saw this today
-- because I recalled I hadn't seen a reply and decided to search for it.

Now I understand what this is does, and it will be a convenient
feature.  But it raises a couple of possible moral issues.

1. Is the Docker program free software?  Is the Podman program free
software?  If neither of them is free software, is this a feature that
promotes running nonfree software on GNU?

2. Supposing that one of them is free software, and there is no
problem of that kind, there's another problem that people have
reported to me: in making a container, there is a risk of including
nonfree programs and you can't easily tell if that has happened, let
alone make sure it won't happen.  The container-making process tends
to pull in dependencies without checking whether they are free.

That is not a reason to refuse to support this access-into-containers
feature, but we should take advantage of this feature and its
documentation to inform people about that problem.

3. Distributing free programs in containers tends to be bad for
the community's control over the program.  Because people
don't build the program on the GNU/Linux distros they use,
and don't package it for those distros.

This too we should use the opportunity to warn people about.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [ELPA] new package: tramp-docker

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > This code is used to access files in Docker or Podman containers that are
>   > running on the same system as Emacs. It calls the Docker or Podman program
>   > to spawn a shell inside the container to communicate with. It is similar
>   > to the su or sudo Tramp methods, in that the connection to the "remote"
>   > system involves shared kernel resources (unless Docker or Podman itself
>   > eventually chooses to do something else).
>
> Thanks for explaining.  My overload is such that I just saw this today
> -- because I recalled I hadn't seen a reply and decided to search for it.
>
> Now I understand what this is does, and it will be a convenient
> feature.  But it raises a couple of possible moral issues.
>
> 1. Is the Docker program free software?  Is the Podman program free
> software?  If neither of them is free software, is this a feature that
> promotes running nonfree software on GNU?

Yes, both are free software.

> 2. Supposing that one of them is free software, and there is no
> problem of that kind, there's another problem that people have
> reported to me: in making a container, there is a risk of including
> nonfree programs and you can't easily tell if that has happened, let
> alone make sure it won't happen.  The container-making process tends
> to pull in dependencies without checking whether they are free.n

To my knowledge there is the danger of either having a build-time or a
run-time dependency on a non-free container, though looking through a
container index like (https://hub.docker.com/search?q=), it appears that
the overwhelming majority of popular software is free software, if only
because distribution is easier.

That being said, TRAMP+Docker is a popular combination for developing
software, so what people often just do is use a distribution image
(Ubuntu, Debian, Alpine) as the foundation and then instruct the
container to install all the software they need using the distributions
package manager, while building their own image.  Seems backwards to me,
but it appears to be popular.  There is a saying that Docker is the
final stage of the "works on my machine"-mentality, as what you are
ultimately doing is shipping your entire "own" machine.

> That is not a reason to refuse to support this access-into-containers
> feature, but we should take advantage of this feature and its
> documentation to inform people about that problem.

I was looking around but couldn't find anything.  The "best" I could
find was license checking tools that made sure there was no GPL software
in a container (to avoid virality), maybe that could be turned on its
head for our needs.

> 3. Distributing free programs in containers tends to be bad for
> the community's control over the program.  Because people
> don't build the program on the GNU/Linux distros they use,
> and don't package it for those distros.
>
> This too we should use the opportunity to warn people about.

I think this could be added to the commentary section.

Re: [ELPA] new package: tramp-docker

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > 1. Is the Docker program free software?  Is the Podman program free
  > > software?  If neither of them is free software, is this a feature that
  > > promotes running nonfree software on GNU?

  > Yes, both are free software.

That is a relief -- at the first level, this is not a problem.

  > To my knowledge there is the danger of either having a build-time or a
  > run-time dependency on a non-free container,

That's what was reported to me.

Does Docker provide an easy way to verify that you have avoided such
dependencies?  A way to make sure to avoid including them?

                                                 though looking through a
  > container index like (https://hub.docker.com/search?q=)

I tried visiting http://hub.docker.com/ and got a blank window.  It depends
on nonfree software to see even the first page.  We must not refer anyone
to that site.

Likewise for https://hub.docker.com/search.

I surmise that the standard way to develop a container involves using
https://hub.docker.com/search.  Is that correct?

Is that the _only_ way to develop a container?  Is it possible,
practically speaking, to build a container without using that site at all?

Has anyone here had practical experience?

                                                           , it appears that
  > the overwhelming majority of popular software is free software, if only
  > because distribution is easier.

Alas, that does not by itself ensure that, supposing you build a container,
you won't consider including nonfree programs.

Is there an easy way you can ensure that _all_ the programs you put
into a new container are free?  Is there an easy way to verify that
the contents of a container are free?

After I get a little information here, I will ask on gnu-misc-discuss.

  > That being said, TRAMP+Docker is a popular combination for developing
  > software, so what people often just do is use a distribution image
  > (Ubuntu, Debian, Alpine) as the foundation and then instruct the
  > container to install all the software they need using the distributions
  > package manager, while building their own image.

I see how that is buzarre, but paradoxically it might work in
freedom's favor here.  If you use a free distro to build the
container, and put things in it with apt-get, you will get only free
software in it.  Maybe that is a reliable method we could recommend.

  > > 3. Distributing free programs in containers tends to be bad for
  > > the community's control over the program.  Because people
  > > don't build the program on the GNU/Linux distros they use,
  > > and don't package it for those distros.
  > >
  > > This too we should use the opportunity to warn people about.

  > I think this could be added to the commentary section.

Maybe so, but when you say "the commentary section", could you
be more precise?  The commentary section of what documentation?


After I get a little information here, I will move this to gnu-misc-discuss.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [ELPA] new package: tramp-docker

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

>   > To my knowledge there is the danger of either having a build-time or a
>   > run-time dependency on a non-free container,
>
> That's what was reported to me.
>
> Does Docker provide an easy way to verify that you have avoided such
> dependencies?  A way to make sure to avoid including them?

As I said, I don't see any such option, but I am not an Docker expert.

>                                                  though looking through a
>   > container index like (https://hub.docker.com/search?q=)
>
> I tried visiting http://hub.docker.com/ and got a blank window.  It depends
> on nonfree software to see even the first page.  We must not refer anyone
> to that site.
>
> Likewise for https://hub.docker.com/search.
>
> I surmise that the standard way to develop a container involves using
> https://hub.docker.com/search.  Is that correct?

Not necessarily, both docker and podman have a "search" subcommand:

        $ podman search gnu
        NAME                                        DESCRIPTION
        docker.io/library/bash                      Bash is the GNU Project's Bourne Again SHell
        docker.io/library/gcc                       The GNU Compiler Collection is a compiling s...
        docker.io/biocontainers/gnumed-server       
        docker.io/biocontainers/gnumed-common       
        docker.io/biocontainers/gnumed-client       
        docker.io/biocontainers/gnumed-client-de    
        docker.io/matpower/matpower                 A complete MATPOWER environment running on G...
        docker.io/matpower/octave                   A complete GNU Octave environment, with IPOP...
        docker.io/gnut3ll4/livy-k8s                 
        docker.io/gnubila/mhmd-documentation        
        ...

> Is that the _only_ way to develop a container?  Is it possible,
> practically speaking, to build a container without using that site at all?

It should be possible, first of all because it is possible to configure
what sites to use as an index for images.  But you also don't need to
use the site itself, the "docker"/"podman" commands take care of
fetching everything you need, just like "apt-get" would.

> Has anyone here had practical experience?
>
>                                                            , it appears that
>   > the overwhelming majority of popular software is free software, if only
>   > because distribution is easier.
>
> Alas, that does not by itself ensure that, supposing you build a container,
> you won't consider including nonfree programs.
>
> Is there an easy way you can ensure that _all_ the programs you put
> into a new container are free?  Is there an easy way to verify that
> the contents of a container are free?

Without an index that would only host free software, I don't see how
this would currently be possible.

> After I get a little information here, I will ask on gnu-misc-discuss.
>
>   > That being said, TRAMP+Docker is a popular combination for developing
>   > software, so what people often just do is use a distribution image
>   > (Ubuntu, Debian, Alpine) as the foundation and then instruct the
>   > container to install all the software they need using the distributions
>   > package manager, while building their own image.
>
> I see how that is buzarre, but paradoxically it might work in
> freedom's favor here.  If you use a free distro to build the
> container, and put things in it with apt-get, you will get only free
> software in it.  Maybe that is a reliable method we could recommend.

It also helps people that are stuck on non-free development platforms to
reliably use free software.

Though I should mention, that AFAIK the most popular package manager is
Alpine's "apk", not Debian's "apt-get".

>   > > 3. Distributing free programs in containers tends to be bad for
>   > > the community's control over the program.  Because people
>   > > don't build the program on the GNU/Linux distros they use,
>   > > and don't package it for those distros.
>   > >
>   > > This too we should use the opportunity to warn people about.
>
>   > I think this could be added to the commentary section.
>
> Maybe so, but when you say "the commentary section", could you
> be more precise?  The commentary section of what documentation?

The commentary section (;;; Commentary:) of an Emacs lisp file, as
described in (elisp) Simple Packages.

> After I get a little information here, I will move this to gnu-misc-discuss.

Re: [ELPA] new package: tramp-docker

[Brian Cully via Emacs development discussions.]

Richard Stallman <rms@gnu.org> writes:

>   > To my knowledge there is the danger of either having a build-time or a
>   > run-time dependency on a non-free container,
>
> That's what was reported to me.
>
> Does Docker provide an easy way to verify that you have avoided such
> dependencies?  A way to make sure to avoid including them?

No it does not, but I do not regard Docker or its ilk as special
here. They provide a framework for running an operating
system. *Inspecting* that operating system is just as out-of-scope for
those projects as it would be for Qemu.

> Is that the _only_ way to develop a container?  Is it possible,
> practically speaking, to build a container without using that site at all?

There are a host of ways of building containers, and there are open
specifications for how to do so (see the Open Container Initiative),
which are all APL-2.0 licensed, to my knowledge.

Additionally, GNU's own Guix project provides a way to create Docker
containers for Guix, from the ground up.

> Alas, that does not by itself ensure that, supposing you build a container,
> you won't consider including nonfree programs.
>
> Is there an easy way you can ensure that _all_ the programs you put
> into a new container are free?  Is there an easy way to verify that
> the contents of a container are free?

Is there an easy way to do that on any given operating system? If so,
then yes, there is a way, and it is the same tool. If there is no such
tool, that's not a Docker problem, and such a tool should be created, at
which point it will be applicable to containers or other virtualization
technologies. Because the entire aim of these technologies is to present
an interface that looks very close to a physical machine (within some
limitations).

>   > > 3. Distributing free programs in containers tends to be bad for
>   > > the community's control over the program.  Because people
>   > > don't build the program on the GNU/Linux distros they use,
>   > > and don't package it for those distros.
>   > >
>   > > This too we should use the opportunity to warn people about.
>
>   > I think this could be added to the commentary section.
>
> Maybe so, but when you say "the commentary section", could you
> be more precise?  The commentary section of what documentation?

Personally, I don't think adding these notes to the “Commentary” section
is appropriate, any more than it would be appropriate to add it to the
SSH support. I see no difference, in the abstract, end-user sense,
between accessing a remote server via SSH or Docker. They accomplish the
same goal, with the same drawbacks, in broadly similar ways.

-bjc

Re: [ELPA] new package: tramp-docker

[zimoun]

Hi,

On sam., 08 oct. 2022 at 18:34, Richard Stallman <rms@gnu.org> wrote:

> Does Docker provide an easy way to verify that you have avoided such
> dependencies?  A way to make sure to avoid including them?

Docker refers to many things, from the container format to the way to
build such format via Dockerfile or even the company behind many tools.

Somehow, Docker is just a format for packing and distributing – as
tarball or .deb packages.  And it is possible to build this pack using
only free software.

For instance, GNU Guix is able to directly produce Docker packs [1] or
images [2] where all is verified and transparent.

Even, it is possible to inspect the resulting binary Docker pack
produced by Guix and then extract the required information for checking
and/or rebuilding it.  See [2] for an example.

The issue is not Docker per-se but all the tools around; for what my
opinion is worth.


1: <https://guix.gnu.org/manual/devel/en/guix.html#Invoking-guix-pack>
2: <https://guix.gnu.org/manual/devel/en/guix.html#image_002dtype-Reference>
3: <https://hpc.guix.info/blog/2021/10/when-docker-images-become-fixed-point/>

Cheers,
simon

Re: [ELPA] new package: tramp-docker

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Does Docker provide an easy way to verify that you have avoided such
  > > dependencies?  A way to make sure to avoid including them?

  > As I said, I don't see any such option, but I am not an Docker expert.

We need to find out the situation before we make more than the tiniest
incidental mention of these containers.  Otherwise we risk sliding
into a trap that would require drastic measures to get out of.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [ELPA] new package: tramp-docker

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >   But you also don't need to
  > use the site itself, the "docker"/"podman" commands take care of
  > fetching everything you need, just like "apt-get" would.

apt-get fetches lists of packages from a web site, and then fetched
the packages themselves from it too.  Is that what the `docker' and
`podman' commands do?  It looks that way.  If so, then running them
is a way of using the respective sites, not an alternative to doing so.

Accessing the site that way has an advantage: if `docker' and `podman'
are free programs, and assuming they don't silently run any software
fetched from the site, this avoids the danger that browsing the site
would run nonfree JS code.

So far, so much the better.  But that leaves this problem:

  > > Is there an easy way you can ensure that _all_ the programs you put
  > > into a new container are free?  Is there an easy way to verify that
  > > the contents of a container are free?

  > Without an index that would only host free software, I don't see how
  > this would currently be possible.

That's what I expected.  Alas, the natural consequence is that
building containers implies a risk of including nonfree software.  The
more packages, the more risk.

As long as that is the case, we should warn people off of distributing
containers.

GNU Emacs is not the place to publish that general point, but where we
mention support for containers, let's include this.

  Containers pose problems for software freedom.  If you are careful,
  you can make and then use a container with only free packages.  But
  when you make a container with more than a few packages, there is
  nothing to help you make sure each and every one is free/libre, and
  no easy way to verify this for an existing container.  We recommend
  staying away from containers made by others unless they explicitly
  commit to carefully ensure the whole contents are free/libre.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [ELPA] new package: tramp-docker

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Can anyone find a container expert to advise me about these issues
and containers?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [ELPA] new package: tramp-docker

[Payas Relekar]

Richard Stallman <rms@gnu.org> writes:

>   > > Does Docker provide an easy way to verify that you have avoided such
>   > > dependencies?  A way to make sure to avoid including them?
>
>   > As I said, I don't see any such option, but I am not an Docker expert.
>
> We need to find out the situation before we make more than the tiniest
> incidental mention of these containers.  Otherwise we risk sliding
> into a trap that would require drastic measures to get out of.

Docker is a platform that provides few services:
1. A uniform container format (Now separated into OCI standard)
2. A way to build and run these containers (docker daemon)
3. A global (and optionally local) container registry (akin to apt-get)

Out of these, #1 is already standardised and fully open.
#2 is not exclusive to docker, and there are multiple ways to acheive
this thanks to #1 being open. e.g. I use Nix (Similar to GNU Guix) to
build containers without relying on any proprietory software.

#3 is where things start getting fuzzy as docker the platform does not
provide any way to ensure containers only include free software. That
being said, as mentioned earlier, docker itself is not necessary to
build these containers.

Coming back to the package at hand (tramp-docker), it provides a way to
access these containers from within Emacs via TRAMP. It is no different
than using regular TRAMP where SSHing into another proprietory system is
exactly same as SSHing into free system, and there is quite possibly no
way to ensure otherwise.

As such, the name docker only signifies what has become de-facto calling
convention for OCI containers, but otherwise as long as we avoid linking
to docker the service, we are in the clear.

--

Re: [ELPA] new package: tramp-docker

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   >   But you also don't need to
>   > use the site itself, the "docker"/"podman" commands take care of
>   > fetching everything you need, just like "apt-get" would.
>
> apt-get fetches lists of packages from a web site, and then fetched
> the packages themselves from it too.  Is that what the `docker' and
> `podman' commands do?  It looks that way.  If so, then running them
> is a way of using the respective sites, not an alternative to doing so.
>
> Accessing the site that way has an advantage: if `docker' and `podman'
> are free programs, and assuming they don't silently run any software
> fetched from the site, this avoids the danger that browsing the site
> would run nonfree JS code.

This is my understanding as well, albeit as someone who has never taken
the time to take a look at the internal details of how this is
implemented.

> So far, so much the better.  But that leaves this problem:
>
>   > > Is there an easy way you can ensure that _all_ the programs you put
>   > > into a new container are free?  Is there an easy way to verify that
>   > > the contents of a container are free?
>
>   > Without an index that would only host free software, I don't see how
>   > this would currently be possible.
>
> That's what I expected.  Alas, the natural consequence is that
> building containers implies a risk of including nonfree software.  The
> more packages, the more risk.
>
> As long as that is the case, we should warn people off of distributing
> containers.
>
> GNU Emacs is not the place to publish that general point, but where we
> mention support for containers, let's include this.
>
>   Containers pose problems for software freedom.  If you are careful,
>   you can make and then use a container with only free packages.  But
>   when you make a container with more than a few packages, there is
>   nothing to help you make sure each and every one is free/libre, and
>   no easy way to verify this for an existing container.  We recommend
>   staying away from containers made by others unless they explicitly
>   commit to carefully ensure the whole contents are free/libre.

I think this sounds good.

Re: [ELPA] new package: tramp-docker

[zimoun]

Hi,

On sam., 15 oct. 2022 at 16:43, Richard Stallman <rms@gnu.org> wrote:

> That's what I expected.  Alas, the natural consequence is that
> building containers implies a risk of including nonfree software.  The
> more packages, the more risk.

This is inaccurate.

> As long as that is the case, we should warn people off of distributing
> containers.

Container is not the issue – it is just a format for packing as tar is.

GNU Guix produces container images including only free software and not
relying on Docker binary or infrastructure.  Please see this message [1].

1: <https://lists.gnu.org/archive/html/emacs-devel/2022-10/msg01078.html>


All the best,
simon

Re: [ELPA] new package: tramp-docker

[Payas Relekar]

Richard Stallman <rms@gnu.org> writes:

>   > Docker is a platform that provides few services:
>   > 1. A uniform container format (Now separated into OCI standard)
>   > 2. A way to build and run these containers (docker daemon)
>   ...
>
>   > Out of these, #1 is already standardised and fully open.
>   > #2 is not exclusive to docker, and there are multiple ways to acheive
>   > this thanks to #1 being open.
>
> Sad to say, that doesn't mean that #2 is no problem for software
> freedom.  Rather, it means that some methods are a problem and others
> are not.

Correct. Please see below for more thoughts.

> I can't be certain with my current partial knowledge, but I think that
> the methods which are a problem are a very grave problem, and we need
> to speak to the community about avoiding them.  I think we need to
> identify any frequently used methods for building Docker containers
> that makes it easy to fail to think about whether the contents are free.

While I am still in doubt whether it is possible to ensure a container
can only be built using free software, similar to .deb packages, short
of not using anything beyond official repos, the discussion is valuable IMO.

>   > 3. A global (and optionally local) container registry (akin to apt-get)
>
>   > #3 is where things start getting fuzzy as docker the platform does not
>   > provide any way to ensure containers only include free software. That
>   > being said, as mentioned earlier, docker itself is not necessary to
>   > build these containers.
>
> I think this is a crucial part of the reason why some ways of
> building docker containers are a grave problem.  Would you disagree?

Partially agree. What we need to separate out is a mechanism to
build/package/deploy (e.g. dpkg, containers) from mechanism to
distribution (debian repos, docker repository).

To continue the debian analogy, it is already possible to package up
proprietory software via .deb file, in face, Google Chrome is
distributed exactly like this. That does not make apt or dpkg inherently
a problem does it? It doesn't classify GNU Make as harmful, and IMO same
rule should be applied to containers. Calling containers harmful means
we are communicating false information to potential users and this may
implore them to question the whole thing rather than just the
questionable parts.

>   > As such, the name docker only signifies what has become de-facto calling
>   > convention for OCI containers, but otherwise as long as we avoid linking
>   > to docker the service, we are in the clear.
>
> That seems plausible to me.
>
> emacs-devel is not the right place to have the discssion about building
> containers.  I should launch it in some other place or way.
>
> Do you know enough about containers to help in that discussion?

I wouldn't call myself an expert, but will be happy provide any
links/study on matter as needed.

--

Re: [ELPA] new package: tramp-docker

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Docker is a platform that provides few services:
  > 1. A uniform container format (Now separated into OCI standard)
  > 2. A way to build and run these containers (docker daemon)
  ...

  > Out of these, #1 is already standardised and fully open.
  > #2 is not exclusive to docker, and there are multiple ways to acheive
  > this thanks to #1 being open.

Sad to say, that doesn't mean that #2 is no problem for software
freedom.  Rather, it means that some methods are a problem and others
are not.

I can't be certain with my current partial knowledge, but I think that
the methods which are a problem are a very grave problem, and we need
to speak to the community about avoiding them.  I think we need to
identify any frequently used methods for building Docker containers
that makes it easy to fail to think about whether the contents are free.

  > 3. A global (and optionally local) container registry (akin to apt-get)

  > #3 is where things start getting fuzzy as docker the platform does not
  > provide any way to ensure containers only include free software. That
  > being said, as mentioned earlier, docker itself is not necessary to
  > build these containers.

I think this is a crucial part of the reason why some ways of
building docker containers are a grave problem.  Would you disagree?

  > As such, the name docker only signifies what has become de-facto calling
  > convention for OCI containers, but otherwise as long as we avoid linking
  > to docker the service, we are in the clear.

That seems plausible to me.

emacs-devel is not the right place to have the discssion about building
containers.  I should launch it in some other place or way.

Do you know enough about containers to help in that discussion?


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [ELPA] new package: tramp-docker

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >   Alas, the natural consequence is that
  > > building containers implies a risk of including nonfree software.  The
  > > more packages, the more risk.

  > > As long as that is the case, we should warn people off of distributing
  > > containers.

  > Container is not the issue – it is just a format for packing as tar is.

That is true, in a sense, but it is only one abstract part of the truth.
I stand by what I said.

  > GNU Guix produces container images including only free software and not
  > relying on Docker binary or infrastructure.

That is good -- as far as it goes.  But most people who build
containers to release a free program don't do it with GNU Guix.  And I
don't think most of them are free software activists.  When they
release free program this way, the risk is real.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [ELPA] new package: tramp-docker

[zimoun]

Hi,

On mer., 19 oct. 2022 at 13:02, Richard Stallman <rms@gnu.org> wrote:

>   >   Alas, the natural consequence is that
>   > > building containers implies a risk of including nonfree software.  The
>   > > more packages, the more risk.
>
>   > > As long as that is the case, we should warn people off of distributing
>   > > containers.
>
>   > Container is not the issue – it is just a format for packing as tar is.
>
> That is true, in a sense, but it is only one abstract part of the truth.
> I stand by what I said.
>
>   > GNU Guix produces container images including only free software and not
>   > relying on Docker binary or infrastructure.
>
> That is good -- as far as it goes.  But most people who build
> containers to release a free program don't do it with GNU Guix.  And I
> don't think most of them are free software activists.  When they
> release free program this way, the risk is real.

Applying this logic and argument, you have to put the same
consideration and/or warning around the usage of GNU tar inside GNU
Emacs.  Because your concern applies equally.

Which additional criteria do you apply for “container“ that you do not
apply for the usage of GNU tar inside GNU Emacs?


Cheers,
simon

Re: [ELPA] new package: tramp-docker

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > While I am still in doubt whether it is possible to ensure a container
  > can only be built using free software, similar to .deb packages, short
  > of not using anything beyond official repos, 

If that is the only reliable method, we should publicly say so.
First we should see what we can find.

  > Partially agree. What we need to separate out is a mechanism to
  > build/package/deploy (e.g. dpkg, containers) from mechanism to
  > distribution (debian repos, docker repository).

In the abstract, I think your logic is correct.

Nonetheless, there seems to be a difference in practice between what
tends to happen when people release programs in other ways and when
people release them in containers.  When developers release tarballs,
various users build them and may report, "Hay, this depends on the
nonfree library libdropdead."  But when developers release containers,
users tend not to notice that one of the 50 packages included is
libdropdead and that that is nonfree.

This isn't the place to continue the discussion.  I should move
it somewhere else.  Would you like to be part of it?



-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [ELPA] new package: tramp-docker

[Payas Relekar]

Richard Stallman <rms@gnu.org> writes:

> This isn't the place to continue the discussion.  I should move
> it somewhere else.  Would you like to be part of it?

Sure.
--

Re: [ELPA] new package: tramp-docker

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > That is good -- as far as it goes.  But most people who build
  > > containers to release a free program don't do it with GNU Guix.  And I
  > > don't think most of them are free software activists.  When they
  > > release free program this way, the risk is real.

  > Applying this logic and argument, you have to put the same
  > consideration and/or warning around the usage of GNU tar inside GNU
  > Emacs.

The problem with containers (Docker, etc) is that commonly used
ways of creating the container choose the contents for you,
and might include nonfree programs without your knowing.

Running GNU tar cannot do that.  To create a tar archive, you have
to specify the precise list of files to include in it.
That is the crucial difference between containers and tar.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)