Bug? Importing (gnu rest ...) fails due to lack of patch files on build side.

Bug? Importing (gnu rest ...) fails due to lack of patch files on build side.

[David Dashyan]

Hello Guix!

I've been having lots of fun with derivations and gexps lately :)

There are couple of quirks I stumbled upon and I'm not sure whether It's
a bug or I'm doing weird things.

Take a look at this snippet:

(define select?
  (match-lambda
    (('guix  'config)   #f)
    (('guix  rest ...)  #t)
    (('gnu   rest ...)  #t)
    (_ #f)))

(with-store store
  (let* ((exp (with-extensions (list guile-gcrypt)
                (with-imported-modules
                    `(((guix config) => ,(make-config.scm))
                      ,@(source-module-closure
                         '((gnu packages base)) #:select? select?))
                  #~(pk 'hello))))
         (drv (run-with-store store
                (mlet %store-monad ((drv (gexp->script "error" exp)))
                  (return drv)))))
    (build-derivations store (list drv))))

Here I building derivation producing dummy script that pk-ing 'hello.
The point really is to try to include source module closure containing
(gnu packages base), or really almost any (gnu rest ..) module that
requires loading package patches.  As I found looking at .drv file
dependencies patch files are not included there.

Here is the tail of error trace-back:

222:17 11 (map1 (((gnu packages gnuzilla)) ((gnu packages #)) # ?))
  3297:17 10 (resolve-interface (gnu packages gnuzilla) #:select _ # ?)
In ice-9/threads.scm:
    390:8  9 (_ _)
In ice-9/boot-9.scm:
  3223:13  8 (_)
In ice-9/threads.scm:
    390:8  7 (_ _)
In ice-9/boot-9.scm:
  3507:20  6 (_)
   2806:4  5 (save-module-excursion _)
  3527:26  4 (_)
In unknown file:
           3 (primitive-load-path "gnu/packages/gnuzilla" #<procedur?>)
In ice-9/eval.scm:
   626:19  2 (_ #<directory (gnu packages gnuzilla) 7fffee6206e0>)
   293:34  1 (_ #(#(#(#(#(#(#(#(#(#<direc?> ?) ?) ?) ?) ?) ?) ?) ?) ?))
    159:9  0 (_ #(#(#(#(#(#(#(#(#(#<direc?> ?) ?) ?) ?) ?) ?) ?) ?) ?))

ice-9/eval.scm:159:9: ERROR:
  1. &formatted-message:
      format: "~a: patch not found\n"
      arguments: ("icecat-use-older-reveal-hidden-html.patch")
builder for `/gnu/store/7g4kk1ncyh2s3yb55rhpblks4z3bf27h-module-import-compiled.drv' failed with exit code 1
cannot build derivation `/gnu/store/ksjs8j167mgj1bpqvqnf0rci9pv7ib3k-error.drv': 1 dependencies couldn't be built
ice-9/boot-9.scm:1669:16: In procedure raise-exception:
ERROR:
  1. &store-protocol-error:
      message: "build of `/gnu/store/ksjs8j167mgj1bpqvqnf0rci9pv7ib3k-error.drv' failed"
      status: 100

So again.  Is it supposed to work?  I think it could have gone unnoticed
because after searching through guix repo I found that (gnu rest ...)
are generally not included in with-imported-modules.

Some context to why am I doing it in the first place.
I found this out while I was implementing "guix-cloud-init" service.
The service to simplify guix deployment to digialocean, aws and others.
Cloud-init is widely used way of configuring cloud vm instances and it
works by providing metadata via mounted storage or link-local address
http endpoint.  This includes ip addresses, devices, ssh keys etc...  So
given you have a guix image with guix-cloud-init service you can upload
to your cloud and it will "just work". :)

So my thinking was that I will write shepherd-root-service extension
that will query metadata import (gnu services ...) modules and run them
with values gotten from metadata.  To be honest it feels a bit hackish
to me, but I didn't find any other ways to make such service and reuse
existing code.

Comments are much appreciated!

P.S. Side note to macrologysts out there...
If I embed select? into with-imported-modules form like so:

(with-imported-modules
    `(((guix config) => ,(make-config.scm))
      ,@(source-module-closure
         '((gnu packages base))
         #:select? (match-lambda
                     (('guix  'config)   #f)
                     (('guix  rest ...)  #t)
                     (('gnu   rest ...)  #t)
                     (_ #f))))
  #~(pk 'hello))

I get this:

While compiling expression:
Syntax error:
unknown file:139:0: syntax: extra ellipsis in form (syntax (quasiquote
(((guix config) => (unquote (make-config.scm))) (unquote-splicing
(source-module-closure (quote ((gnu packages base))) #:select?
(match-lambda (((quote guix) (quote config)) #f) (((quote guix) rest
...) #t) (((quote gnu) rest ...) #t) (_ #f)))))))

Any ideas why is that?

-- 
David aka zzappie

Re: Bug? Importing (gnu rest ...) fails due to lack of patch files on build side.

[David Dashyan]

Hello Guix!

My update on this.

So I though of a dirty hack...

(define guix-patches
  (package
    (inherit guix)
    (name "guix-patches")
    (build-system copy-build-system)
    (arguments
     '(#:install-plan
       '(("gnu/packages/patches/"
          "share/guile/site/3.0/gnu/packages/patches/"))))
    (native-inputs '())
    (inputs '())
    (propagated-inputs '())
    (native-search-paths '())
    (home-page "")
    (synopsis "")
    (description "")
    (properties '())))

The reasoning was that I can wrap everithing it in 
(with-extensions (list guix-patches ...) ...)

But still loading patches fails.  Even though they should be now in
%load-path...

-- 
David aka zzappie

Re: Bug? Importing (gnu rest ...) fails due to lack of patch files on build side.

[Joshua Branson]

David Dashyan <mail@davie.li> writes:

I'm just responding so that you get a response.  :)

Your last email had some pretty cool code bro!  It certainly looks like
you're having fun with guix!

> Hello Guix!
>
> My update on this.
>
> So I though of a dirty hack...
>
> (define guix-patches
>   (package
>     (inherit guix)
>     (name "guix-patches")
>     (build-system copy-build-system)
>     (arguments
>      '(#:install-plan
>        '(("gnu/packages/patches/"
>           "share/guile/site/3.0/gnu/packages/patches/"))))
>     (native-inputs '())
>     (inputs '())
>     (propagated-inputs '())
>     (native-search-paths '())
>     (home-page "")
>     (synopsis "")
>     (description "")
>     (properties '())))
>
> The reasoning was that I can wrap everithing it in 
> (with-extensions (list guix-patches ...) ...)
>
> But still loading patches fails.  Even though they should be now in
> %load-path...

-- 
Joshua Branson (joshuaBPMan in #guix)
Sent from Emacs and Gnus
  https://gnucode.me
  https://video.hardlimit.com/accounts/joshua_branson/video-channels
  https://propernaming.org
  "You can have whatever you want, as long as you help
enough other people get what they want." - Zig Ziglar
  

Re: Bug? Importing (gnu rest ...) fails due to lack of patch files on build side.

[David Dashyan]

Oh you so nice :)

Joshua Branson writes:

> David Dashyan <mail@davie.li> writes:
>
> I'm just responding so that you get a response.  :)
>
> Your last email had some pretty cool code bro!  It certainly looks like
> you're having fun with guix!

-- 
David aka zzappie

Re: Bug? Importing (gnu rest ...) fails due to lack of patch files on build side.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 5599 bytes --]

David Dashyan schreef op ma 12-04-2021 om 12:48 [+0300]:
> Hello Guix!
> 
> I've been having lots of fun with derivations and gexps lately :)
> 
> There are couple of quirks I stumbled upon and I'm not sure whether It's
> a bug or I'm doing weird things.
> 
> Take a look at this snippet:
> 
> (define select?
>   (match-lambda
>     (('guix  'config)   #f)
>     (('guix  rest ...)  #t)
>     (('gnu   rest ...)  #t)
>     (_ #f)))
> 
> (with-store store
>   (let* ((exp (with-extensions (list guile-gcrypt)
>                 (with-imported-modules
>                     `(((guix config) => ,(make-config.scm))
>                       ,@(source-module-closure
>                          '((gnu packages base)) #:select? select?))
>                   #~(pk 'hello))))
>          (drv (run-with-store store
>                 (mlet %store-monad ((drv (gexp->script "error" exp)))
>                   (return drv)))))
>     (build-derivations store (list drv))))
> 
> Here I building derivation producing dummy script that pk-ing 'hello.
> The point really is to try to include source module closure containing
> (gnu packages base), or really almost any (gnu rest ..) module that
> requires loading package patches.  As I found looking at .drv file
> dependencies patch files are not included there.
> 
> Here is the tail of error trace-back:
> 
> 222:17 11 (map1 (((gnu packages gnuzilla)) ((gnu packages #)) # ?))
>   3297:17 10 (resolve-interface (gnu packages gnuzilla) #:select _ # ?)
> In ice-9/threads.scm:
>     390:8  9 (_ _)
> In ice-9/boot-9.scm:
>   3223:13  8 (_)
> In ice-9/threads.scm:
>     390:8  7 (_ _)
> In ice-9/boot-9.scm:
>   3507:20  6 (_)
>    2806:4  5 (save-module-excursion _)
>   3527:26  4 (_)
> In unknown file:
>            3 (primitive-load-path "gnu/packages/gnuzilla" #<procedur?>)
> In ice-9/eval.scm:
>    626:19  2 (_ #<directory (gnu packages gnuzilla) 7fffee6206e0>)
>    293:34  1 (_ #(#(#(#(#(#(#(#(#(#<direc?> ?) ?) ?) ?) ?) ?) ?) ?) ?))
>     159:9  0 (_ #(#(#(#(#(#(#(#(#(#<direc?> ?) ?) ?) ?) ?) ?) ?) ?) ?))
> 
> ice-9/eval.scm:159:9: ERROR:
>   1. &formatted-message:
>       format: "~a: patch not found\n"
>       arguments: ("icecat-use-older-reveal-hidden-html.patch")
> builder for `/gnu/store/7g4kk1ncyh2s3yb55rhpblks4z3bf27h-module-import-compiled.drv' failed with exit code 1
> cannot build derivation `/gnu/store/ksjs8j167mgj1bpqvqnf0rci9pv7ib3k-error.drv': 1 dependencies couldn't be built
> ice-9/boot-9.scm:1669:16: In procedure raise-exception:
> ERROR:
>   1. &store-protocol-error:
>       message: "build of `/gnu/store/ksjs8j167mgj1bpqvqnf0rci9pv7ib3k-error.drv' failed"
>       status: 100
> 
> So again.  Is it supposed to work? I think it could have gone unnoticed
> because after searching through guix repo I found that (gnu rest ...)
> are generally not included in with-imported-modules.

No, it is not supposed to work!  Only guix/build/... and gnu/build/utils/...
(and maybe others I forgot) are supposed to be imported, so source-module-closure
excludes other modules.  The reason is that gnu/packages/*.scm are supposed to be
changeable without causing any change in derivation hash (and therefore resulting
in rebuilds).

> Some context to why am I doing it in the first place.
> I found this out while I was implementing "guix-cloud-init" service.

Instead of importing (gnu packages ...), you could use
(with-extensions (list guix) ...), where guix is the guix package.
Then there's no need for with-imported-modules IIUC.
That package isn't always the newest though (it is primarily used
for the guix-daemon, for cuirass, emacs-guix and some others),
so consider using an inferior or something.

> Some context to why am I doing it in the first place.
> I found this out while I was implementing "guix-cloud-init" service.
> The service to simplify guix deployment to digialocean, aws and others.
> Cloud-init is widely used way of configuring cloud vm instances and it
> works by providing metadata via mounted storage or link-local address
> http endpoint.  This includes ip addresses, devices, ssh keys etc...  So
> given you have a guix image with guix-cloud-init service you can upload
> to your cloud and it will "just work". :)

Do you know about "guix deploy"?

> So my thinking was that I will write shepherd-root-service extension
> that will query metadata import (gnu services ...) modules and run them
> with values gotten from metadata.
Not sure what you're trying to do there. ‘query metadata import’ and
‘run them with values from metadata’ is a bit vague.

> To be honest it feels a bit hackish
> to me, but I didn't find any other ways to make such service and reuse
> existing code.
> 
> Comments are much appreciated!
> 
> P.S. Side note to macrologysts out there...
> If I embed select? into with-imported-modules form like so:
> 
> (with-imported-modules
>     `(((guix config) => ,(make-config.scm))
>       ,@(source-module-closure
>          '((gnu packages base))
>          #:select? (match-lambda
>                      (('guix  'config)   #f)
>                      (('guix  rest ...)  #t)
>                      (('gnu   rest ...)  #t)
>                      (_ #f))))
>   #~(pk 'hello))

I wonder what would happen if you replace #t with
(and (list rest ...) #t)?  

> Any ideas why is that?

There seems to be some kind of hygiene problem here.
Perhaps take a look at the definition of with-imported-modules
and match-lambda.

Greetings,
Maxime

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Bug? Importing (gnu rest ...) fails due to lack of patch files on build side.

[David Dashyan]

Hello, Maxime!

> No, it is not supposed to work!  Only guix/build/... and
> gnu/build/utils/...  (and maybe others I forgot) are supposed to be
> imported, so source-module-closure excludes other modules.  The reason
> is that gnu/packages/*.scm are supposed to be changeable without
> causing any change in derivation hash (and therefore resulting in
> rebuilds).
Ah, great! I'm glad I got definitive answer to this question.  But I
don't get the last sentence.  Changeable when?  You mean that by
including (gnu package ...) in inputs of a derivation change in
gnu/package/*scm files will cause change of derivation hash?  But why it
is a problem for service derivations?  As I understand it won't cause
change in other package derivations hashes propagating down the graph.
Service derivation sits on the leaf of dependency graph.  Or I missing
something?

>> Some context to why am I doing it in the first place.  I found this
>> out while I was implementing "guix-cloud-init" service.
>
> Instead of importing (gnu packages ...), you could use
> (with-extensions (list guix) ...), where guix is the guix package.
> Then there's no need for with-imported-modules IIUC.  That package
> isn't always the newest though (it is primarily used for the
> guix-daemon, for cuirass, emacs-guix and some others), so consider
> using an inferior or something.
Thanks! I'll try it.  I havent thought of that.


>> Some context to why am I doing it in the first place.  I found this
>> out while I was implementing "guix-cloud-init" service.  The service
>> to simplify guix deployment to digialocean, aws and others.
>> Cloud-init is widely used way of configuring cloud vm instances and
>> it works by providing metadata via mounted storage or link-local
>> address http endpoint.  This includes ip addresses, devices, ssh keys
>> etc...  So given you have a guix image with guix-cloud-init service
>> you can upload to your cloud and it will "just work". :)
>
> Do you know about "guix deploy"?
>
Yes.  I think ssh deploy is great, but I don't really like how
digitalocean deployment works, particularly the idea of mutating debian
instance by running big chunk of shell code and leaving all the debian
stuff laying around.  Especially when you have all the info needed for
system configuration provided to you.  I think creating with `guix
system image`, uploading it to digitalocean/aws/etc and only then use
`guix deploy` is much cleaner and robust way.  There are also use cases
when guix deploy just won't work.  E.g. vm instance might be teleported
to other location with other network, ssh keys, network devices.  By
using metadata services instance can automatically reconfigure itself
after migration.

example metadata structure:

(define %test-metadata-json-string
  "{
  \"droplet_id\": 213129999,
  \"hostname\": \"test-test\",
  \"vendor_data\": \"Content-Type: loadsof/randomstuff\"
  \"public_keys\": [
    \"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCUnbO+g3Gc8zaxCLfpVegABaCyrKQwq3l7D\
asMqcVXdFFdDkaR26RfZkS13T0XQuxeM+2o7HGwFebkts56vWe/FLDMdNjAvViNo8hlX6bUaR+IPRf3\
KJSyy0aS7Ht7qtWHdDcuxtaDS8YebOw3GE9AG1P+nQMVRR12TesJnuq632gLdv/UlzIH32Qk7rMR3/9\
oDIbO24kFx9dTU/5Mdj2b+GQzfstux3uLMvlNxztRLTqFNSwQcc09vMAS0WZUkB4trT/WlkoDka/L46\
7k5gA0KjHtIprnP73oKlMBLu7NFq+UANUqncUzGbb99XbXQDFQYRiDdYCMX6wrW8mRL+Cz\"
  ],
  \"auth_key\": \"d873f96dbd087d05a4c5490379a925b5\",
  \"region\": \"fra1\",
  \"interfaces\": {
    \"private\": [
      {
        \"ipv4\": {
          \"ip_address\": \"10.114.0.2\",
          \"netmask\": \"255.255.240.0\",
          \"gateway\": \"0.0.0.0\"
        },
        \"mac\": \"de:83:28:c9:fe:92\",
        \"type\": \"private\"
      }
    ],
    \"public\": [
      {
        \"ipv4\": {
          \"ip_address\": \"207.188.191.111\",
          \"netmask\": \"255.255.240.0\",
          \"gateway\": \"207.155.240.1\"
        },
        \"ipv6\": {
          \"ip_address\": \"2848:F0F:5171:2F45:58B5:3A88:6062:AB3F\",
          \"CIDR\": 64,
          \"gateway\": \"2A03:B0C0:0003:00D0:0000:0000:0000:0001\"
        },
        \"anchor_ipv4\": {
          \"ip_address\": \"10.19.0.5\",
          \"netmask\": \"255.255.0.0\",
          \"gateway\": \"10.19.0.1\"
        },
        \"mac\": \"12:BE:48:43:D9:0D\",
        \"type\": \"public\"
      }
    ]
  },
  \"floating_ip\": {
    \"ipv4\": {
      \"active\": false
    }
  },
  \"dns\": {
    \"nameservers\": [
        \"67.207.67.2\",
        \"67.207.67.3\"
    ]
  },
  \"tags\": [
    \"some\",
    \"test\",
    \"tags\"
  ],
  \"features\": {
    \"dhcp_enabled\": false
  },
  \"modify_index\": 88992338
}")

It usually either provided on 169.244.169.244/metadata/v1.json lladdr
endpoint and/or inside mounted /dev/vdb storage.

Plus cloud-init is kind of standard nowadays. So it will provide
integration with other "clouds" for free.

>> So my thinking was that I will write shepherd-root-service extension
>> that will query metadata import (gnu services ...) modules and run them
>> with values gotten from metadata.
> Not sure what you're trying to do there. ‘query metadata import’ and
> ‘run them with values from metadata’ is a bit vague.

Sorry forgot the comma here. It should have been 
>> "..will query metadata#{,}# import (gnu services ...) modules and..."

>> To be honest it feels a bit hackish
>> to me, but I didn't find any other ways to make such service and reuse
>> existing code.
>> 
>> Comments are much appreciated!
>> 
>> P.S. Side note to macrologysts out there...
>> If I embed select? into with-imported-modules form like so:
>> 
>> (with-imported-modules
>>     `(((guix config) => ,(make-config.scm))
>>       ,@(source-module-closure
>>          '((gnu packages base))
>>          #:select? (match-lambda
>>                      (('guix  'config)   #f)
>>                      (('guix  rest ...)  #t)
>>                      (('gnu   rest ...)  #t)
>>                      (_ #f))))
>>   #~(pk 'hello))
>
> I wonder what would happen if you replace #t with
> (and (list rest ...) #t)?  
Same result...

-- 
David aka zzappie

Re: Bug? Importing (gnu rest ...) fails due to lack of patch files on build side.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 2561 bytes --]

David Dashyan schreef op di 27-04-2021 om 20:03 [+0300]:
> 
> Hello, Maxime!
> 
> > No, it is not supposed to work!  Only guix/build/... and
> > gnu/build/utils/...  (and maybe others I forgot) are supposed to be
> > imported, so source-module-closure excludes other modules.  The reason
> > is that gnu/packages/*.scm are supposed to be changeable without
> > causing any change in derivation hash (and therefore resulting in
> > rebuilds).
> Ah, great! I'm glad I got definitive answer to this question.  But I
> don't get the last sentence.  Changeable when?  You mean that by
> including (gnu package ...) in inputs of a derivation change in
> gnu/package/*scm files will cause change of derivation hash?

If you mean:
All package, services, etc ... derivations that include the (gnu packages ...)
module (and its dependencies) would have a change in derivation hash,
then yes.

> But why it is a problem for service derivations?

It guess for service derivations (FWIW, that's not an ‘official’ term, I think),
this isn't much of a problem.  I still recommend using
(with-extensions (list guix) ...) or similar though (with the caveat that
the 'guix' package isn't always the latest guix, so you may want to consider
inferiors or something).

> As I understand it won't cause
> change in other package derivations hashes propagating down the graph.
> Service derivation sits on the leaf of dependency graph.  Or I missing
> something?
No, that's correct.

> > > Some context to why am I doing it in the first place.  I found this
> > > out while I was implementing "guix-cloud-init" service.  The service
> > > to simplify guix deployment to digialocean, aws and others.
> > > Cloud-init is widely used way of configuring cloud vm instances and
> > > it works by providing metadata via mounted storage or link-local
> > > address http endpoint.  This includes ip addresses, devices, ssh keys
> > > etc...  So given you have a guix image with guix-cloud-init service
> > > you can upload to your cloud and it will "just work". :)
>
> Do you know about "guix deploy"?
> [Plenty of text]

I'm not familiar with clouds that aren't

 aerosol consisting of a visible mass of minute liquid droplets, frozen crystals,
 or other particles suspended in the atmosphere of a planetary body or similar space.
 (Source: <https://en.wikipedia.org/wiki/Cloud>, accessed 2021)

so I'm afraid I don't have much to say here, but your idea of skipping Debian
seems better than is done currently!

Greetings,
Maxime

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]