Newbie packagers

Newbie packagers

[Vincent Legoll]

I put this in a separate email, as it's becoming long...

The packaging doc:

https://www.gnu.org/software/guix/manual/guix.html#Defining-Packages
is good, but still not enough for beginners.

Essentially it's missing what is hidden behind "Without being a Scheme ..."

It wouldn't hurt to have the scheme non-obvious parts explained:

- the comma operator
- the backquote operator
- the quote operator
- the arobase operator (is it for list unpacking ?)
- the percent operator
- the #: operator
- the different module import things (#:use-module vs (use-modules) vs ...)

(Forgive my probably-not-appropriate terms.)

No need to explain function calls, string quoting and simple stuff, though.

I tried to find a good tutorial explaining all of those, but couldn't. I found
snippets that helped me understand some of those, but they were scattered,
and it's still blurry.

Specific explanations will be more useful that a general scheme tutorial, the
hello.scm is good as an example :

(inputs `(("gawk" ,gawk)))

here we use the backquote because ...
the comma is there for ...

(arguments `(#:configure-flags '("--enable-silent-rules")))

here the #: means ...
we use the simple quote because ...

Obviously I cannot write that myself, but if someone does the explanation,
I volunteer the .texi translation...

-- 
Vincent Legoll

Re: Newbie packagers

[Ricardo Wurmus]

Hi Vincent,

> - the comma operator
> - the backquote operator
> - the quote operator
> - the arobase operator (is it for list unpacking ?)

These are all about list quoting.
This is code:

    (+ 1 2 3)

It evaluates to 6.

This is quoted code (aka data):

    '(+ 1 2 3)

It doesn’t get evaluated any further.
This is pretty much the same as:

    `(+ 1 2 3)

But with the backtick (called “quasiquote”) you are permitted to
temporarily switch from “data mode” to “code mode”.  That’s what the
comma (“unquote”) does:

    `(+ 1 2 3 4 ,(+ 3 2) 6)
    ^           ^
    data mode   code mode

The result is the same as

    '(+ 1 2 3 4 5 6)

What’s nice about this is that we can use the same syntax for code that
is to be evaluated right now and for code that we want to pass somewhere
else as inert data which will be evaluated at a later point.

This allows for “staging”.  When you look at a typical package
expression you see that the value of the “arguments” field is quoted.
It is not evaluated right away but in a second stage.

The value of the inputs field is also quoted.  You see that we unquote
the values of package variables there.  Package expressions in Guix are
just Scheme values.  The inputs field does not merely hold a list of
symbols that somehow represent the packages — it actually contains the
very values themselves!

“,@” is for splicing lists:

    (let ((moo '(1 2 3)))
      `(foo bar ,@moo meh))

This binds the list '(1 2 3) to the name “moo” and then splices it into
another list.  This results in

    '(foo bar 1 2 3 meh)

Without the “@” and just a regular unquote comma it would be a nested
list:

    (let ((moo '(1 2 3)))
      `(foo bar ,moo meh))

=>  '(foo bar (1 2 3) meh)

Quoting and unquoting is a very useful feature.  I hope my explanations
above are easy to understand.

> - the percent operator

That’s no operator.  It’s part of the name.  We use the percent as a
prefix for variables that are “special”, e.g. global variables or
values that appear without having to be explicitly let-bound.

> - the #: operator

These are called keyword arguments.  They are no operators either.
Using keyword arguments allows you to specify arguments by name, not in
some fixed order.

> - the different module import things (#:use-module vs (use-modules) vs ...)

That’s probably explained in the Guile reference manual.  We use
“#:use-module” only in module definitions.  “(use-modules …)” can be
used anywhere else.

> I tried to find a good tutorial explaining all of those, but couldn't. I found
> snippets that helped me understand some of those, but they were scattered,
> and it's still blurry.
>
> Specific explanations will be more useful that a general scheme tutorial, the
> hello.scm is good as an example :
>
> (inputs `(("gawk" ,gawk)))
>
> here we use the backquote because ...
> the comma is there for ...
>
> (arguments `(#:configure-flags '("--enable-silent-rules")))
>
> here the #: means ...
> we use the simple quote because ...

Yeah, I agree.  There should be a quick crash course.  I spent half the
day yesterday to introduce all these things to a colleague who will
create a few Guix packages in the future.  It would be cool to have this
as part of the manual.

~~ Ricardo

Re: Newbie packagers

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1988 bytes --]

Hi Vincent,

Vincent Legoll <vincent.legoll@gmail.com> skribis:

> I put this in a separate email, as it's becoming long...
>
> The packaging doc:
>
> https://www.gnu.org/software/guix/manual/guix.html#Defining-Packages
> is good, but still not enough for beginners.
>
> Essentially it's missing what is hidden behind "Without being a Scheme ..."
>
> It wouldn't hurt to have the scheme non-obvious parts explained:
>
> - the comma operator
> - the backquote operator
> - the quote operator
> - the arobase operator (is it for list unpacking ?)
> - the percent operator
> - the #: operator
> - the different module import things (#:use-module vs (use-modules) vs ...)
>
> (Forgive my probably-not-appropriate terms.)
>
> No need to explain function calls, string quoting and simple stuff, though.
>
> I tried to find a good tutorial explaining all of those, but couldn't. I found
> snippets that helped me understand some of those, but they were scattered,
> and it's still blurry.
>
> Specific explanations will be more useful that a general scheme tutorial, the
> hello.scm is good as an example :
>
> (inputs `(("gawk" ,gawk)))
>
> here we use the backquote because ...
> the comma is there for ...
>
> (arguments `(#:configure-flags '("--enable-silent-rules")))
>
> here the #: means ...
> we use the simple quote because ...

I would propose the patch below as a start.  While it does not give
detailed explanations, it provides terminology and pointers, in
particular to this section of the Guile manual:

  https://www.gnu.org/software/guile/manual/html_node/Expression-Syntax.html

Unfortunately, this section of Guile’s manual is a reference and it
lacks examples and a good overview.  I think we should fix it on Guile’s
side instead of palliating this deficiency in Guix’s manual.

If nobody beats me at it, I can try and improve this is in Guile’s
manual afterwards.

What do you think?

Thanks,
Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1882 bytes --]

diff --git a/doc/guix.texi b/doc/guix.texi
index e7b233d..8d332f6 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -2452,12 +2452,35 @@ The @code{arguments} field specifies options for the build system
 @var{gnu-build-system} as a request run @file{configure} with the
 @code{--enable-silent-rules} flag.
 
+@cindex quote
+@cindex quoting
+@cindex backquote (quasiquote)
+What about these backquote (@code{`}) and quote (@code{'}) characters?
+They are Scheme syntax to introduce a literal data structure; @code{'}
+is synonymous with @code{quote}, and @code{`} is synonymous with
+@code{quasiquote}.  @xref{Expression Syntax, quoting,, guile, GNU Guile
+Reference Manual}, for details.  Here the value of the @code{arguments}
+field is a list of arguments passed to the build system.
+
+The hash-colon (@code{#:}) sequence defines a Scheme @dfn{keyword}
+(@pxref{Keywords,,, guile, GNU Guile Reference Manual}), and
+@code{#:configure-flags} is a keyword used to pass a keyword argument
+to the build system (@pxref{Coding With Keywords,,, guile, GNU Guile
+Reference Manual}).
+
 @item
 The @code{inputs} field specifies inputs to the build process---i.e.,
 build-time or run-time dependencies of the package.  Here, we define an
 input called @code{"gawk"} whose value is that of the @var{gawk}
 variable; @var{gawk} is itself bound to a @code{<package>} object.
 
+@cindex comma (unquote)
+Again, @code{`} (@code{quasiquote}) allows us to introduce a literal
+list in the @code{inputs} field, while @code{,} (a comma, synonymous
+with @code{unquote}) allows us to insert a value in the list
+(@pxref{Expression Syntax, unquote,, guile, GNU Guile Reference
+Manual}).
+
 Note that GCC, Coreutils, Bash, and other essential tools do not need to
 be specified as inputs here.  Instead, @var{gnu-build-system} takes care
 of ensuring that they are present (@pxref{Build Systems}).

Re: Newbie packagers

[Vincent Legoll]

Hello,

Thanks a lot, this is really helpful !

On Thu, Jul 21, 2016 at 5:52 PM, Ricardo Wurmus <rekado@elephly.net> wrote:
>> - the comma operator
>> - the backquote operator
>> - the quote operator
>> - the arobase operator (is it for list unpacking ?)
>
> These are all about list quoting.
> This is code:
>
>     (+ 1 2 3)
>
> It evaluates to 6.
>
> This is quoted code (aka data):
>
>     '(+ 1 2 3)

Those I knew/remembered from my CS student days, it's the other
2 I didn't grok...

> It doesn’t get evaluated any further.
> This is pretty much the same as:
>
>     `(+ 1 2 3)
>
> But with the backtick (called “quasiquote”) you are permitted to
> temporarily switch from “data mode” to “code mode”.  That’s what the
> comma (“unquote”) does:
>
>     `(+ 1 2 3 4 ,(+ 3 2) 6)
>     ^           ^
>     data mode   code mode
>
> The result is the same as
>
>     '(+ 1 2 3 4 5 6)
>
> What’s nice about this is that we can use the same syntax for code that
> is to be evaluated right now and for code that we want to pass somewhere
> else as inert data which will be evaluated at a later point.
>
> This allows for “staging”.  When you look at a typical package
> expression you see that the value of the “arguments” field is quoted.
> It is not evaluated right away but in a second stage.
>
> The value of the inputs field is also quoted.  You see that we unquote
> the values of package variables there.  Package expressions in Guix are
> just Scheme values.  The inputs field does not merely hold a list of
> symbols that somehow represent the packages — it actually contains the
> very values themselves!
>
> “,@” is for splicing lists:

That one I guessed, but wasn't sure because of the still unknown comma...
Python has something similar, but it does not look as useful as this, especially
with that quasiquote thing...

>     (let ((moo '(1 2 3)))
>       `(foo bar ,@moo meh))
>
> This binds the list '(1 2 3) to the name “moo” and then splices it into
> another list.  This results in
>
>     '(foo bar 1 2 3 meh)
>
> Without the “@” and just a regular unquote comma it would be a nested
> list:
>
>     (let ((moo '(1 2 3)))
>       `(foo bar ,moo meh))
>
> =>  '(foo bar (1 2 3) meh)
>
> Quoting and unquoting is a very useful feature.  I hope my explanations
> above are easy to understand.

Perfect, clear, that was just what I needed. I will not immediately
be used to those concepts, but I understand what they do, that's
sufficient to read code...

The big problem with the abbreviated forms is that they're not easily
googlable, if I knew their long names, I could've found them in online
doc...

>> - the percent operator
>
> That’s no operator.  It’s part of the name.  We use the percent as a
> prefix for variables that are “special”, e.g. global variables or

OK, only a naming convention then ?

> values that appear without having to be explicitly let-bound.

That could use a little pointer, because the idea is strange, you
mean implicit variables (bindings), kind of like perl's $_ ?

>> - the #: operator
>
> These are called keyword arguments.  They are no operators either.
> Using keyword arguments allows you to specify arguments by name, not in
> some fixed order.

That I also guessed (I'm coming from **kwargs-land...), but they also look
more powerful...

>> - the different module import things (#:use-module vs (use-modules) vs ...)
>
> That’s probably explained in the Guile reference manual.

Yes, I read a bit about that in there, but the following is what I
wanted to know :

> We use “#:use-module” only in module definitions.
>  “(use-modules …)” can be used anywhere else.

> Yeah, I agree.  There should be a quick crash course.  I spent half the
> day yesterday to introduce all these things to a colleague who will
> create a few Guix packages in the future.  It would be cool to have this
> as part of the manual.

Thanks a lot, this is just what's missing, and looks like Ludo's covered the
.texi part !

-- 
Vincent Legoll

Re: Newbie packagers

[Vincent Legoll]

Thanks Ludo,

> I would propose the patch below as a start.  While it does not give
> detailed explanations, it provides terminology and pointers, in
> particular to this section of the Guile manual:
>
>   https://www.gnu.org/software/guile/manual/html_node/Expression-Syntax.html

this is nice, the pointer to the reference is actually the nicest
part because of the ungooglability of the various quoting characters.
And the small local explanation is good too.

> Unfortunately, this section of Guile’s manual is a reference and it
> lacks examples and a good overview.  I think we should fix it on Guile’s
> side instead of palliating this deficiency in Guix’s manual.
>
> If nobody beats me at it, I can try and improve this is in Guile’s
> manual afterwards.

I'll go read it and see what I find missing

-- 
Vincent Legoll

Re: Newbie packagers

[Vincent Legoll]

> I'll go read it and see what I find missing

quick ideas:

1 - the concept index has those: ! # ( . / maybe add ` ' , ,@ with
their aliases / long names alongside (quote, quasiquote, unquote,
unquote-splicing)

2 - they are well hidden in :

The Guile Reference Manual
API Reference
Reading and Evaluating Scheme Code
Scheme Syntax: Standard and Guile Extensions
Expression Syntax

That's far away, and the title don't help find them...

In the mean time I did a search "guile scheme quote", and indeed the
right doc page is there (16th on duckduckgo), and google is way
better (1st place)...

I forgot that sometimes google gives better answers than ddg...

3 - I'd add the following to (unquote-splicing expr):

`(1 ,x 4)  ⇒ (1 (2 3) 4)

to better demonstrate the effect of splicing

and to (unquote expr) :

`(1 2 (* 9 9) 3 4)      ⇒ (1 2 (* 9 9) 3 4)

-- 
Vincent Legoll

Re: Newbie packagers

[Vincent Legoll]

Just to see if I understood properly...

In the hello.scm example, there is :

(arguments `(#:configure-flags '("--enable-silent-rules")))

1 - can the quasiquote be replaced by a regular quote, because
unquote is not used ?

I'd guess yes, now I'll try it... <later>

So, it seems like we can change the quasiquote to a simple quote.

A question remains, why do the configure flags need to be a quoted list ?

Is it because it could be a function call evaluated by the gnu-build-system
which has to return a list of strings ?

That would mean the following should work:

(arguments `(#:configure-flags (lambda () (list "--enable-silent-rules"))))

-- 
Vincent Legoll

Re: Newbie packagers

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1047 bytes --]

Vincent Legoll <vincent.legoll@gmail.com> writes:

> The big problem with the abbreviated forms is that they're not easily
> googlable, if I knew their long names, I could've found them in online
> doc...

FYI, one of the best ways to find that information is to look it up in
the manual by index or by searching the entire manual for the string.

For example, in both the standard Info reader and the Emacs info reader,
you can press "i", type ",@" (without quotes), hit enter, and you will
magically arrive at the right section.  If that doesn't work, you
can also do C-s to incrementally search the entire manual for the string
in question.

This works in any well-written manual authored with GNU Texinfo.  For
example, if you want to know about the "perror" function from the GNU C
Library, just pull up the manual and type "i", then "perror", the hit
enter.

This is often faster and more accurate than doing an Internet search for
the symbols.  It's a great feature of any manual written in GNU Texinfo!

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: Newbie packagers

[Ludovic Courtès]

Vincent Legoll <vincent.legoll@gmail.com> skribis:

> Just to see if I understood properly...
>
> In the hello.scm example, there is :
>
> (arguments `(#:configure-flags '("--enable-silent-rules")))
>
> 1 - can the quasiquote be replaced by a regular quote, because
> unquote is not used ?

Yes.

> A question remains, why do the configure flags need to be a quoted list ?

It’s a list of arguments that gets passed to a function down the road.

Thanks for your feedback!

Ludo’.

Re: Newbie packagers

[Ludovic Courtès]

Vincent Legoll <vincent.legoll@gmail.com> skribis:

> quick ideas:
>
> 1 - the concept index has those: ! # ( . / maybe add ` ' , ,@ with
> their aliases / long names alongside (quote, quasiquote, unquote,
> unquote-splicing)

Good idea.

> 2 - they are well hidden in :
>
> The Guile Reference Manual
> API Reference
> Reading and Evaluating Scheme Code
> Scheme Syntax: Standard and Guile Extensions
> Expression Syntax
>
> That's far away, and the title don't help find them...
>
> In the mean time I did a search "guile scheme quote", and indeed the
> right doc page is there (16th on duckduckgo), and google is way
> better (1st place)...
>
> I forgot that sometimes google gives better answers than ddg...

I have no control over search engines.  :-)

> 3 - I'd add the following to (unquote-splicing expr):
>
> `(1 ,x 4)  ⇒ (1 (2 3) 4)
>
> to better demonstrate the effect of splicing
>
> and to (unquote expr) :
>
> `(1 2 (* 9 9) 3 4)      ⇒ (1 2 (* 9 9) 3 4)

I’d prefer to avoid duplication with the Guile manual.

I pushed a variant of what I had posted as commit
654c0d97c9de4bf25b9facda1278835883555ae0.

Thanks!

Ludo’.

Re: Newbie packagers

[Vincent Legoll]

>> A question remains, why do the configure flags need to be a quoted list ?
>
> It’s a list of arguments that gets passed to a function down the road.

but as the list of arguments is already in a quoted list, it is quoted and not
interpreted here.

'(toto (titi)) => (toto (titi))

and not an error telling titi is unbound variable.

'(toto '(titi)) => (toto (quote (titi)))

there's an additional quote level...

-- 
Vincent Legoll

Re: Newbie packagers

[Vincent Legoll]

> I have no control over search engines.  :-)

Yep, that was just a lame excuse, sorry...
;-)

>> 3 - I'd add the following to (unquote-splicing expr):
>>
>> `(1 ,x 4)  ⇒ (1 (2 3) 4)
>>
>> to better demonstrate the effect of splicing
>>
>> and to (unquote expr) :
>>
>> `(1 2 (* 9 9) 3 4)      ⇒ (1 2 (* 9 9) 3 4)
>
> I’d prefer to avoid duplication with the Guile manual.

Those additional examples would have been for:
https://www.gnu.org/software/guile/manual/guile.html#Expression-Syntax
and not for the guix manual...

Do you want me to submit a patch for those ?

-- 
Vincent Legoll

Re: Newbie packagers

[Ludovic Courtès]

Vincent Legoll <vincent.legoll@gmail.com> skribis:

>>> 3 - I'd add the following to (unquote-splicing expr):
>>>
>>> `(1 ,x 4)  ⇒ (1 (2 3) 4)
>>>
>>> to better demonstrate the effect of splicing
>>>
>>> and to (unquote expr) :
>>>
>>> `(1 2 (* 9 9) 3 4)      ⇒ (1 2 (* 9 9) 3 4)
>>
>> I’d prefer to avoid duplication with the Guile manual.
>
> Those additional examples would have been for:
> https://www.gnu.org/software/guile/manual/guile.html#Expression-Syntax
> and not for the guix manual...

Ah OK, done:

  http://git.savannah.gnu.org/cgit/guile.git/commit/?h=stable-2.0&id=141e390b596bfb6464f4d692381232e974de8783

Ludo’.

Re: Newbie packagers

[Vincent Legoll]

>> Those additional examples would have been for:
>> https://www.gnu.org/software/guile/manual/guile.html#Expression-Syntax
>> and not for the guix manual...
>
> Ah OK, done:
>
>   http://git.savannah.gnu.org/cgit/guile.git/commit/?h=stable-2.0&id=141e390b596bfb6464f4d692381232e974de8783

Great, this should help

-- 
Vincent Legoll

Re: Newbie packagers

[Andreas Enge]

Hello Ricardo,

On Thu, Jul 21, 2016 at 05:52:43PM +0200, Ricardo Wurmus wrote:
> > - the comma operator
> > - the backquote operator
> > - the quote operator
> > - the arobase operator (is it for list unpacking ?)
> These are all about list quoting.

very nice explanations! I wonder whether this should not appear in the manual
or in some blog post that discusses how to contribute to Guix.

Andreas

Re: Newbie packagers

[Vincent Legoll]

Hello,

> On Thu, Jul 21, 2016 at 05:52:43PM +0200, Ricardo Wurmus wrote:
>> > - the comma operator
>> > - the backquote operator
>> > - the quote operator
>> > - the arobase operator (is it for list unpacking ?)
>> These are all about list quoting.
>
> very nice explanations!

Yep, I understood them... ;-)

> I wonder whether this should not appear in the manual
> or in some blog post that discusses how to contribute to Guix.

Parts of that has already percolated into guile & guix manuals...

The most important one IMHO, being the reference link from
guix -> guile...

I don't know when they'll be updated on the web, but the modifs
are committed...

-- 
Vincent Legoll