Proxy settings wrt guix daemon

Proxy settings wrt guix daemon

[Vincent Legoll]

Hello,

following the packaging of squid, I have been
searching how to configure the guix daemon
to use a proxy, and this looks problematic.

I found very sparse info in the documentation,
at least something explaining that this area is
WIP would have helped. [1]

I found the following config.scm snippet on the
ML [2]:

      (modify-services %desktop-services
        (guix-service-type config =>
                           (guix-configuration
                            (inherit config)
                            (http-proxy
                             "http://<IP>:<PORT>"))))

but as mentionned in [3], this will not fly in the
case of an intermittently available proxy, as guix
system-reconfigure will depends on this working
properly to reset it back to no proxy usage.

There is an issue about the problem [4], but it
does not looks like it is making progress.

Could something be done from shepherd PoV,
like easily setting environment vars for specific
services, in /etc/... config files ? So that can be
easily modified with a simple text editor in case
of failure.

[1] https://lists.gnu.org/archive/html/help-guix/2017-02/msg00090.html
[2] https://lists.gnu.org/archive/html/help-guix/2019-07/msg00031.html
[3] https://lists.gnu.org/archive/html/guix-devel/2017-10/msg00206.html
[4] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25569

Solving this would be awesome for the imminent
release. I'd like to help as much as I can.

WDYT ?

-- 
Vincent Legoll

Re: Proxy settings wrt guix daemon

[Ludovic Courtès]

Hi Vincent,

Vincent Legoll <vincent.legoll@gmail.com> skribis:

> but as mentionned in [3], this will not fly in the
> case of an intermittently available proxy, as guix
> system-reconfigure will depends on this working
> properly to reset it back to no proxy usage.
>
> There is an issue about the problem [4], but it
> does not looks like it is making progress.
>
> Could something be done from shepherd PoV,
> like easily setting environment vars for specific
> services, in /etc/... config files ? So that can be
> easily modified with a simple text editor in case
> of failure.

In general, /etc on Guix System is meant to be immutable (with the
exception of “state” files like /etc/resolv.conf and /etc/passwd).

So my recommendation would be to fix [4] specifically for ‘guix-daemon’
by adding a ‘set-proxy’ action or something.

> [1] https://lists.gnu.org/archive/html/help-guix/2017-02/msg00090.html
> [2] https://lists.gnu.org/archive/html/help-guix/2019-07/msg00031.html
> [3] https://lists.gnu.org/archive/html/guix-devel/2017-10/msg00206.html
> [4] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25569
>
> Solving this would be awesome for the imminent
> release. I'd like to help as much as I can.

Yup.

Thanks,
Ludo’.

Re: Proxy settings wrt guix daemon

[Vincent Legoll]

Hello

On Sun, Mar 29, 2020 at 5:00 PM Ludovic Courtès <ludo@gnu.org> wrote:
> So my recommendation would be to fix [4] specifically for ‘guix-daemon’
> by adding a ‘set-proxy’ action or something.

Trying to understand what that would imply, I stumbled upon my
ignorance of how to test attempts at implementing that.

Would the following be useful :
make + pre-inst guix system container in a git checkout / branch
with modifications to nix/libstore/builtins.cc::builtinDownload (for
the server part)

Then how to implement the "asking the server to change its
proxy setting" ? Where should I look ?

What UI (client-side) ?

`guix daemon set-proxy` or something like that ?

I.e. I'm willing to try, but will need guidance...

-- 
Vincent Legoll

Re: Proxy settings wrt guix daemon

[Ludovic Courtès]

Hi,

Vincent Legoll <vincent.legoll@gmail.com> skribis:

> On Sun, Mar 29, 2020 at 5:00 PM Ludovic Courtès <ludo@gnu.org> wrote:
>> So my recommendation would be to fix [4] specifically for ‘guix-daemon’
>> by adding a ‘set-proxy’ action or something.
>
> Trying to understand what that would imply, I stumbled upon my
> ignorance of how to test attempts at implementing that.
>
> Would the following be useful :
> make + pre-inst guix system container in a git checkout / branch
> with modifications to nix/libstore/builtins.cc::builtinDownload (for
> the server part)
>
> Then how to implement the "asking the server to change its
> proxy setting" ? Where should I look ?
>
> What UI (client-side) ?
>
> `guix daemon set-proxy` or something like that ?
>
> I.e. I'm willing to try, but will need guidance...

I was proposing a custom action for the Shepherd service, just like the
mcron Shepherd service has a custom ‘schedule’ action that one can
invoke with “herd schedule mcron”.

Hope that’s clearer!

Ludo’.

Re: Proxy settings wrt guix daemon

[Vincent Legoll]

Hello,

thanks for the help, but...

On Tue, Mar 31, 2020 at 5:26 PM Ludovic Courtès <ludo@gnu.org> wrote:
> I was proposing a custom action for the Shepherd service, just like the
> mcron Shepherd service has a custom ‘schedule’ action that one can
> invoke with “herd schedule mcron”.
>
> Hope that’s clearer!

I've found the shepherd-schedule-action code, but that does not tell me
how to implement the whole thing.

Let me try harder, the following are all questions.
I do not know if they make any sense at all...

How do I make the running daemon use a proxy ?
By doing setenv() calls in builtins.cc builtinDownload()
before the execv(guix perform-download) ?

How do I make the running daemon know which proxy to use ?
By getting it from the settings object from globals.hh ?

How do I make the running daemon change the value stored
in the settings object ?
By sending it an order to do so via the listening socket ?

What will send this order through that socket ?
Another, one-shot, ephemeral, guix-daemon process with
the --set-proxy command-line option ?

What is the UI to run that guix-daemon --set-proxy ?
Use herd set-proxy guix-daemon "https://proxy:3128" ?
(Like what is done for mcron)

I really need a few "make this thing here do that" hints...

And then also a bit of a hint on how I would test the
modifications without risking my running system.

I hope that's not too demanding...

Thanks

-- 
Vincent Legoll

Re: Proxy settings wrt guix daemon

[Mathieu Othacehe]

Hello Vicent,

>
> What is the UI to run that guix-daemon --set-proxy ?
> Use herd set-proxy guix-daemon "https://proxy:3128" ?
> (Like what is done for mcron)
>
> I really need a few "make this thing here do that" hints...
>
> And then also a bit of a hint on how I would test the
> modifications without risking my running system.

From what I understand, the guix-daemon process is able to propagate
some of its environments variable to the builder. It seems that
http_proxy variable (and friends), are set as impureEnvVars by Guix. You
can search for '#:leaked-env-vars' over the Guix sources.

So it seems to indicate that if the guix-daemon process has the
http_proxy environment variable set, it will handle it correctly. So the
question is how to be able to set this variable dynamically?

I don't think there's a way to change the environment variables of a
process from the outside. So unless you come up with a new mechanism
inside guix-daemon, able to read proxy parameters from a file or so (not
desirable), the variable needs to be set before guix-daemon is launched.

Implementing what's proposed by Ludo would then make the following
command: 'herd set-proxy guix-daemon "https://proxy:3128"' do the
following things:

--8<---------------cut here---------------start------------->8---
(setenv "HTTP_PROXY" "https://proxy:3128")
(restart guix-daemon) ; not the appropriated syntax
--8<---------------cut here---------------end--------------->8---

I don't know if that's what Ludo has in mind but I can't see any other
ways to do it.

Now Ludo has also proposed another solution here[1]. This solution
would require that (guix script perform-download) and (guix script
substitute) honor what would look like a 'build-option' (see
set-build-options in (guix store)). But this seems harder to implement.

Mathieu

[1]: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25569

Re: Proxy settings wrt guix daemon

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 586 bytes --]

Hi,

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

> Implementing what's proposed by Ludo would then make the following
> command: 'herd set-proxy guix-daemon "https://proxy:3128"' do the
> following things:
>
> (setenv "HTTP_PROXY" "https://proxy:3128")
> (restart guix-daemon) ; not the appropriated syntax
>
> I don't know if that's what Ludo has in mind but I can't see any other
> ways to do it.

That’s roughly what I had in mind.  I had to give it a go to see if it
made sense :-) and I came up with this patch.

Let me know what you think!

Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 6531 bytes --]

diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index 8d9a563e2b..9c7f1395aa 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -1633,6 +1633,29 @@ archive' public keys, with GUIX."
 (define %default-guix-configuration
   (guix-configuration))
 
+(define shepherd-set-http-proxy-action
+  ;; Shepherd action to change the HTTP(S) proxy.
+  (shepherd-action
+   (name 'set-http-proxy)
+   (documentation
+    "Change the HTTP(S) proxy used by 'guix-daemon' and restart it.")
+   (procedure #~(lambda* (_ #:optional proxy)
+                  (let ((environment (environ)))
+                    ;; A bit of a hack: communicate PROXY to the 'start'
+                    ;; method via environment variables.
+                    (if proxy
+                        (begin
+                          (format #t "changing HTTP/HTTPS \
+proxy of 'guix-daemon' to ~s...~%"
+                                  proxy)
+                          (setenv "http_proxy" proxy))
+                        (begin
+                          (format #t "unsetting HTTP/HTTPS \
+proxy of 'guix-daemon'...~%")
+                          (unsetenv "http_proxy")))
+                    (action 'guix-daemon 'restart)
+                    (environ environment))))))
+
 (define (guix-shepherd-service config)
   "Return a <shepherd-service> for the Guix daemon service with CONFIG."
   (match-record config <guix-configuration>
@@ -1644,47 +1667,58 @@ archive' public keys, with GUIX."
            (documentation "Run the Guix daemon.")
            (provision '(guix-daemon))
            (requirement '(user-processes))
+           (actions (list shepherd-set-http-proxy-action))
            (modules '((srfi srfi-1)))
            (start
-            #~(make-forkexec-constructor
-               (cons* #$(file-append guix "/bin/guix-daemon")
-                      "--build-users-group" #$build-group
-                      "--max-silent-time" #$(number->string max-silent-time)
-                      "--timeout" #$(number->string timeout)
-                      "--log-compression" #$(symbol->string log-compression)
-                      #$@(if use-substitutes?
-                             '()
-                             '("--no-substitutes"))
-                      "--substitute-urls" #$(string-join substitute-urls)
-                      #$@extra-options
+            #~(lambda _
+                (define proxy
+                  ;; HTTP/HTTPS proxy.  The 'http_proxy' variable is set by
+                  ;; the 'set-http-proxy' action.
+                  (or (getenv "http_proxy") #$http-proxy))
 
-                      ;; Add CHROOT-DIRECTORIES and all their dependencies (if
-                      ;; these are store items) to the chroot.
-                      (append-map (lambda (file)
-                                    (append-map (lambda (directory)
-                                                  (list "--chroot-directory"
-                                                        directory))
-                                                (call-with-input-file file
-                                                  read)))
-                                  '#$(map references-file chroot-directories)))
+                (fork+exec-command
+                 (cons* #$(file-append guix "/bin/guix-daemon")
+                        "--build-users-group" #$build-group
+                        "--max-silent-time" #$(number->string max-silent-time)
+                        "--timeout" #$(number->string timeout)
+                        "--log-compression" #$(symbol->string log-compression)
+                        #$@(if use-substitutes?
+                               '()
+                               '("--no-substitutes"))
+                        "--substitute-urls" #$(string-join substitute-urls)
+                        #$@extra-options
 
-               #:environment-variables
-               (list #$@(if http-proxy
-                            (list (string-append "http_proxy=" http-proxy))
-                            '())
-                     #$@(if tmpdir
-                            (list (string-append "TMPDIR=" tmpdir))
-                            '())
+                        ;; Add CHROOT-DIRECTORIES and all their dependencies
+                        ;; (if these are store items) to the chroot.
+                        (append-map (lambda (file)
+                                      (append-map (lambda (directory)
+                                                    (list "--chroot-directory"
+                                                          directory))
+                                                  (call-with-input-file file
+                                                    read)))
+                                    '#$(map references-file
+                                            chroot-directories)))
 
-                     ;; Make sure we run in a UTF-8 locale so that 'guix
-                     ;; offload' correctly restores nars that contain UTF-8
-                     ;; file names such as 'nss-certs'.  See
-                     ;; <https://bugs.gnu.org/32942>.
-                     (string-append "GUIX_LOCPATH="
-                                    #$glibc-utf8-locales "/lib/locale")
-                     "LC_ALL=en_US.utf8")
+                 #:environment-variables
+                 (append (list #$@(if tmpdir
+                                      (list (string-append "TMPDIR=" tmpdir))
+                                      '())
 
-               #:log-file #$log-file))
+                               ;; Make sure we run in a UTF-8 locale so that
+                               ;; 'guix offload' correctly restores nars that
+                               ;; contain UTF-8 file names such as
+                               ;; 'nss-certs'.  See
+                               ;; <https://bugs.gnu.org/32942>.
+                               (string-append "GUIX_LOCPATH="
+                                              #$glibc-utf8-locales
+                                              "/lib/locale")
+                               "LC_ALL=en_US.utf8")
+                         (if proxy
+                             (list (string-append "http_proxy=" proxy)
+                                   (string-append "https_proxy=" proxy))
+                             '()))
+
+                 #:log-file #$log-file)))
            (stop #~(make-kill-destructor))))))
 
 (define (guix-accounts config)

Re: Proxy settings wrt guix daemon

[Vincent Legoll]

I'm not sure I understand, but isn't the setenv in
shepherd-set-http-proxy-action done for the whole
running shepherd process ?

-- 
Vincent Legoll

Re: Proxy settings wrt guix daemon

[Ludovic Courtès]

Hi Vincent,

Vincent Legoll <vincent.legoll@gmail.com> skribis:

> I'm not sure I understand, but isn't the setenv in
> shepherd-set-http-proxy-action done for the whole
> running shepherd process ?

It is, but only temporarily: the environment is restored once
‘guix-daemon’ has been started.

Also, since Shepherd 0.7.0, processes started by ‘fork+exec-command’ &
co. do not inherit environment variables of PID 1.

Thanks,
Ludo’.

Re: Proxy settings wrt guix daemon

[Vincent Legoll]

Hello,

On Tue, Apr 7, 2020 at 11:39 AM Ludovic Courtès <ludo@gnu.org> wrote:
> Vincent Legoll <vincent.legoll@gmail.com> skribis:
>
> > I'm not sure I understand, but isn't the setenv in
> > shepherd-set-http-proxy-action done for the whole
> > running shepherd process ?
>
> It is, but only temporarily: the environment is restored once
> ‘guix-daemon’ has been started.
>
> Also, since Shepherd 0.7.0, processes started by ‘fork+exec-command’ &
> co. do not inherit environment variables of PID 1.

OK, thanks, LGTM then

-- 
Vincent Legoll

Re: Proxy settings wrt guix daemon

[Ludovic Courtès]

Hello,

Ludovic Courtès <ludo@gnu.org> skribis:

> That’s roughly what I had in mind.  I had to give it a go to see if it
> made sense :-) and I came up with this patch.

I went ahead and pushed it as 3302e03ba0edca49347c6a2b215e56bd53a6b113.
Another 2017 bug closed!  \o/

Ideally, the installer would have a dialog box to select a proxy.
Do we want to do that?  Or leave it for the next release?

Thanks,
Ludo’.

Re: Proxy settings wrt guix daemon

[Mathieu Othacehe]

Hey,

> I went ahead and pushed it as 3302e03ba0edca49347c6a2b215e56bd53a6b113.
> Another 2017 bug closed!  \o/

Well done!

>
> Ideally, the installer would have a dialog box to select a proxy.
> Do we want to do that?  Or leave it for the next release?

Yes I could do that. I wonder what's the best place for such a dialog
box, because I suspect the vast majority of people is not concerned by
network proxy settings.

Maybe from the new help menu? But then the "help" term is maybe not the
best fit. We could make it a "parameter" menu, this way:

--8<---------------cut here---------------start------------->8---
Press <F1> to access installation parameters.

This is the installer parameters menu, please make a choice.

  * Change keyboard layout
  * Configure network proxy
  ...
--8<---------------cut here---------------end--------------->8---

WDYT?

Mathieu

Re: Proxy settings wrt guix daemon

[Mathieu Othacehe]

Another issue could be that, to make sure that we are connected to the
internet, we check connman status (see connman-state in (gnu installer
connman)) during the install.

If there's a network HTTP proxy, I guess this may fail. So we would need
to also call "connmanctl proxy auto PROXY_URL" or something alike.

Is there someone here that is behind a HTTP proxy and could test that?

Thanks,

Mathieu

Re: Proxy settings wrt guix daemon

[Ludovic Courtès]

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

> Another issue could be that, to make sure that we are connected to the
> internet, we check connman status (see connman-state in (gnu installer
> connman)) during the install.
>
> If there's a network HTTP proxy, I guess this may fail. So we would need
> to also call "connmanctl proxy auto PROXY_URL" or something alike.

Hmm OK.

> Is there someone here that is behind a HTTP proxy and could test that?

One could test with Privoxy as the proxy.  Do people know of other easy
to set up proxy software?

Ludo’.

Re: Proxy settings wrt guix daemon

[Ludovic Courtès]

Hello,

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

>> I went ahead and pushed it as 3302e03ba0edca49347c6a2b215e56bd53a6b113.
>> Another 2017 bug closed!  \o/
>
> Well done!
>
>>
>> Ideally, the installer would have a dialog box to select a proxy.
>> Do we want to do that?  Or leave it for the next release?
>
> Yes I could do that. I wonder what's the best place for such a dialog
> box, because I suspect the vast majority of people is not concerned by
> network proxy settings.
>
> Maybe from the new help menu? But then the "help" term is maybe not the
> best fit. We could make it a "parameter" menu, this way:
>
> Press <F1> to access installation parameters.

Yes, “parameters” seems better suited than “help” in any case.

> This is the installer parameters menu, please make a choice.
>
>   * Change keyboard layout
>   * Configure network proxy
>   ...
>
> WDYT?

Sounds good!

Thanks,
Ludo’.

Re: Proxy settings wrt guix daemon

[Ludovic Courtès]

Hi,

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

> Press <F1> to access installation parameters.
>
> This is the installer parameters menu, please make a choice.

Maybe just: “Please choose one of the follow parameters or press ‘Back’
to go back to the installation process.”

>   * Change keyboard layout
>   * Configure network proxy
>   ...

BTW, if we agree, could you change these strings so we can do a last
round through the TP and release Thursday/Friday?

Ludo’.

Re: Proxy settings wrt guix daemon

[Mathieu Othacehe]

Hello,

> BTW, if we agree, could you change these strings so we can do a last
> round through the TP and release Thursday/Friday?

Yes, I applied those changes and added proxy support. I'll try to see
how connman is behaving now.

Thanks,

Mathieu

Re: Proxy settings wrt guix daemon

[Ludovic Courtès]

Hi Mathieu,

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

>> BTW, if we agree, could you change these strings so we can do a last
>> round through the TP and release Thursday/Friday?
>
> Yes, I applied those changes and added proxy support. I'll try to see
> how connman is behaving now.

That was fast!  It looks really good, this new parameters menu comes in
handy!

Ludo’.

Re: Proxy settings wrt guix daemon

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 526 bytes --]

Hi,

could we mention an example value for the HTTP proxy setting?

When I use it it's never quite clear to me whether there's a slash at the end
or not, and what to do with https etc.

Currently it says "Please enter the HTTP proxy URL.  If you enter an empty
string, proxy usage will be disabled.".

I'd add a note: "(Example: http://10.0.0.1:8000)".

Also, if authentication is possible, maybe add that to the example.

What happens to https?  Will it go via the HTTP proxy?  Maybe mention that
there, too.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Proxy settings wrt guix daemon

[Ludovic Courtès]

Hi Danny,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> could we mention an example value for the HTTP proxy setting?
>
> When I use it it's never quite clear to me whether there's a slash at the end
> or not, and what to do with https etc.
>
> Currently it says "Please enter the HTTP proxy URL.  If you enter an empty
> string, proxy usage will be disabled.".
>
> I'd add a note: "(Example: http://10.0.0.1:8000)".
>
> Also, if authentication is possible, maybe add that to the example.
>
> What happens to https?  Will it go via the HTTP proxy?  Maybe mention that
> there, too.

Yes.

So we could write “HTTP and HTTPS” and also include the example.

Downside is that translations will become stale.  Oh well, maybe we’ll
have to live with that.

Ludo’.

Re: Proxy settings wrt guix daemon

[Vincent Legoll]

Hello,

On Fri, Apr 10, 2020 at 3:26 PM Danny Milosavljevic
<dannym@scratchpost.org> wrote:
> What happens to https?  Will it go via the HTTP proxy?  Maybe mention that
> there, too.

It does go through the proxy, I've seen the hits in squid
log files here.

-- 
Vincent Legoll

Re: Proxy settings wrt guix daemon

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 426 bytes --]

On Fri, 10 Apr 2020 15:26:32 +0200
Danny Milosavljevic <dannym@scratchpost.org> wrote:

> What happens to https?  Will it go via the HTTP proxy?  Maybe mention that
> there, too.

In https://lists.gnu.org/archive/html/guix-devel/2020-04/msg00084.html
there's only http_proxy honored.

According to ./guix/build/download.scm:
                   ;; For HTTPS URIs, honor 'https_proxy', not 'http_proxy'.

Hmmm...


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Proxy settings wrt guix daemon

[Ludovic Courtès]

Hi,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> On Fri, 10 Apr 2020 15:26:32 +0200
> Danny Milosavljevic <dannym@scratchpost.org> wrote:
>
>> What happens to https?  Will it go via the HTTP proxy?  Maybe mention that
>> there, too.
>
> In https://lists.gnu.org/archive/html/guix-devel/2020-04/msg00084.html
> there's only http_proxy honored.

It’s a bit misleading because the ‘http_proxy’ env. var. is used to
communicate the info to the ‘start’ method, but no:

               (if proxy
                   (list (string-append "http_proxy=" proxy)
                         (string-append "https_proxy=" proxy))
                   '())

Ludo’.