* Re: G expressions
2014-04-28 7:43 ` Ludovic Courtès
@ 2014-04-28 14:03 ` Amirouche Boubekki
2014-04-28 15:58 ` Ludovic Courtès
2014-04-28 21:45 ` Ludovic Courtès
1 sibling, 1 reply; 5+ messages in thread
From: Amirouche Boubekki @ 2014-04-28 14:03 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: guix-devel
[-- Attachment #1: Type: text/plain, Size: 13048 bytes --]
2014-04-28 9:43 GMT+02:00 Ludovic Courtès <ludo@gnu.org>:
# Hello!
# 1. The Problem
# ===============
Héllo!
I was going to learn a bit of emacs lisp then... a duck got in my way.
# There’s a recurrent problem that we need to communicate the file name of
# store items to Scheme code that is going to live in another process:
# expressions to build a derivation, Scheme files that are to be loaded by
# other processes, etc.
This seems to be a referential transparency problem. Probably the
Funarg problem.
Quoting http://en.wikipedia.org/wiki/Funarg_problem:
:w: The difficulty only arises if the body of a [[nested function]]
refers directly
:w: (i.e., not via argument passing) to identifiers defined in the environment
:w: in which the function is defined, but not in the environment of the function
:w: call.
:w: <ref>[http://portal.acm.org/citation.cfm?id=1093411
:w: ''The function of FUNCTION in LISP or why the FUNARG problem should be
:w: called the environment problem'',
:w: </ref>
:w: by Joel Moses, in: ACM SIGSAM Bulletin 17 (July 1970), pp. 13-27]</ref>
:w: [Two] standard resolutions are to either forbid such references or to
:w: create [[closure (computer science)|closure]]s.
:w: <ref>[http://portal.acm.org/citation.cfm?id=1093420.1093422
:w: ''A proposed solution to the FUNARG problem'',
:w: by Erik Sandewall, in: ACM SIGSAM Bulletin 17 (Jan. 1971), pp. 29-42]
:w: </ref>
It's a scoping issue due to the context of where the call is really done. In
Guile arguments are passed as value. So, the procedure loose the context of
the value. It's just a "thunk" or log.
# This had been partly addressed by having ‘build-expression->derivation’
# install two global variables in the build code: ‘%build-inputs’ and
# ‘%outputs’.
I don't like globals but I have no good reasons, except that it "pollutes"
the environment. IRL, it's a problem when you want to move to "higher
level"s.
# However, for generated Scheme files (as in (gnu system) and related
# modules), there’s no such mechanism. Thus, either we use something
# like:
# (mlet %store-monad ((ls (package-file coreutils "bin/ls")))
# (text-file "foo" (object->string `(system* ,ls))))
cf. https://www.gnu.org/software/guix/manual/guix.html#The-Store-Monad
# but then the problem is that the resulting text file doesn’t hold a
# reference to Coreutils, which is wrong. Or, we do something like:
# (text-file* "foo" "(system* \"" coreutils "/bin/ls\")")
# The resulting file does have a reference to Coreutils, but the approach
# obviously sucks.
# Besides, ‘%build-inputs’ is not particularly elegant either, and it’s
# error-prone (if you refer to an input by the wrong name, you only notice
# when building, with an obscure wrong-type-arg error.) That’s been OK as
# it’s only used occasionally in package recipes.
I can not make justice to the style (it's ugly!). I don't have enough experience
in scheme/fp/guile to distinguish between a good solution and workaround
-- something. AFAIK, both process don't share the same memory, so it's even
more difficult to share data, in the broad sens. In this case, writring sexp
directly allows to keep some of the information, but not all of it, because
the values are evaluated.
This evaluation thing troubles me a lot. That's one of the reasons I started
scheme: macros. I did not write, consciously, a single macro, yet.
.. python:: https://github.com/lihaoyi/macropy
In python, there is a *run* *time*. That is all. Some time you take care of
*import* *time*. But then you forget about it. So far, my understanding
of scheme led me to think _that anything anywhere could be something else in
reality_. In particular, it's possible to provide procedures to the parser
called macros that are evaluated when sexps are parsed.
In the particular case of guix why not send to just data? So that
context can not be
/overriden/.
Like I said, I don't understand fully macros nor monads and I stumbled upon
http://adit.io/posts/2013-04-17-functors,_applicatives,_and_monads_in_pictures.html
I understand that the article explains what does it mean that Haskell is typed.
I understand that haskell has "higher order" procedures, that are not readily
available in imperative languages plus some extras for being self aware of that
fact.
In fact, what /functors/, /applicatives/ and /monads/ do, is possible in all
languages. If they are not used often so far, it's because people were missing
something... Those /higher level/ procedures that can be found in Haskell
are *forbidden* in imperative languages: GOTO, monkey-patching, metaclasses or
grossly ineficient like callbacks for dealing with asynchronous behaviors and
otherwise.
.. seealso:: (url [Callbacks as our Generations' Go To Statement]
http://tirania.org/blog/archive/2013/Aug-15.html)
IMO the article explains different *patterns* that dynamically change
the runtime.
It's like a bestiary of some sort, see
http://en.wikibooks.org/wiki/Haskell/Understanding_arrows
In the end it's just a procedure call, a combinatory logic, to make something
that looked like http://community.schemewiki.org/?amb, more discrete
(and more human?).
I don't say there is no interest in this approach(!).
.. seealso:: (url [Combinatorial Generation Using Coroutines With
Examples in Python]
https://news.ycombinator.com/item?id=7655488)
At least the above, lead me to (quote gexp).
# 2. G Expressions
# =================
# “G-expressions”, or gexps, are meant to address these two issues
# mentioned above:
Passing parameters by *intent* instead of by value?
# 1. Replacing a reference to a package or derivation by its output file
# name;
cf. https://www.gnu.org/software/guix/manual/guix.html#Derivations
# 2. Keeping track of the derivations it refers to.
I interpret this as:
«It must be possible to (back)trace anywhere, anytime... anything»
Out of the context of guix, that could be better or worse misunderstanding :)
# In addition, the substitution in (1) must be done lazily, so that you
# get the output file name for the ‘%current-system’ value when the gexp
# is used.
# The ‘gexp’ form is essentially like ‘quasiquote’, with ‘ungexp’ as the
# counterpart of ‘unquote’:
# (gexp (system* (string-append (ungexp coreutils) "/bin/ls")))
What about the following:
;; (quote (system* (string-append (coreutils "/bin/ls"))))
I don't understand the difference with classic macro. Is it a problem of
nesting? Something like that isn't possible:
;; (quote (system* (string-append (coreutils
(:somekind-of-sexp-parser-hook-eval: program-path))))
# That gexp can then be passed to ‘gexp->file’, which returns a derivation
# that builds a file containing:
It is passed to the server ie. the guix builder?
# (system* (string-append "/gnu/store/…" "/bin/ls"))
# And it can also be used to describe derivation builds:
# (gexp->derivation "foo"
# (gexp
# (begin
# (mkdir (ungexp output))
# (chdir (ungexp output))
# (symlink (ungexp coreutils) "cu"))))
# Note that we don’t need #:inputs here, because the gexp embeds that
# info. So eventually, we could even get rid of the label in the ‘inputs’
# field of packages (not a priority, though.)
# We could use some sugar to make it more concise (suggestions welcome):
# (gexp->derivation "foo"
# #~(begin
# (mkdir #$output)
# (chdir #$output)
# (symlink #$coreutils "cu")))
- It's possible to parse variable names in macros?
- I don't like #~ or #$ I prefer an infix notation with *words* as
it is more readable:
;; (gexp (gexp.begin (make.directory ungexp.output)
(change.directory ungexp.output)
(symlink ungexp.coreutils "cu")))
I added some fioritures that you will probably find pointless. IMHO
it's significant.
# 3. Conclusion
# ==============
# I think gexps can be helpful to serialize Scheme code that refers to
# packages/derivations. Preliminary work is in ‘wip-gexp’.
# What do you think?
It's seems to me that it's yet another /upward & downard funarg/.
# 4. Related Work :-)
# ====================
# HOP has a quotation mechanism to introduce client-side code in server
# code and vice-versa:
# (<BUTTON>
# :onclick ~(with-hop ($foo)))
# Nix uses string interpolation to keep achieve what we do:
# script = ''${coreutils}/bin/ls'';
# Here ${coreutils} is replaced by the output file name, and the resulting
# string internally has additional info about the reference to that
# derivation.
- This can be the job of guile reader?
- It exists also in native javascript framework like react
http://facebook.github.io/react/#helloExample
.. seealso:: http://www.quora.com/What-does-Facebook-use-OCaml-for
Quoting your last message:
# I’ve been experimenting with gexps in all the contexts with it seemed
# “obviously” useful and in some other places under gnu/{system,services}
# (see the commits that say “Rewrite using gexps” in wip-gexp.)
cf. (url [Add (guix gexp).]
http://git.savannah.gnu.org/cgit/guix.git/commit/?id=20ed77c71e41b41911cf152f8feae7e9ac5f9f6b)
- I need to learn macros. The decyphering route is langweileich.
- I did not read all the code, it seems like ``ungexp`` is
``:sexp-parser-hook-eval:``.
=> if that's it, that was the exact problem I have (had?). What I'd like to do
is "describe an intent" more precisly an intent that is bound to an unknown
context, a lot of /Maybe/ in you prefer. This reflexion grew out of:
- an api to traverse graphs in Python based on Gremlin DSL
https://bitbucket.org/amirouche/omak-unmaintained
- declarative programming in general (cf. Prolog)
- the reactive programming culprit (cf. reactconf)
guile-2d does reactive programming, I'm quoting David Thompson the creator:
// Functional reactive programming (FRP) provides a way to describe
// time-varying values and their relationships using a functional
// and declarative programming style. To understand what this means,
// let's investigate a typical variable definition in Scheme [...]
Later there is an example of guile-2d FRP ie. Functional Reactive Programming:
// ;;
// ;; (define-signal position
// ;; (signal-fold v+ (vector2 320 240)
// ;; (signal-map (lambda (v)
// ;; (vscale v 4))
// ;; (signal-sample game-agenda 1 key-arrows))))
// ;;
In the project I'm working on, using guilen I would have written in as:
;; (voxel player position signal shift with (* 4 (voxel agenda sample
(voxel keyboard arrows)) 1))
or with somekind of macro
;; (player.position.signal.shift.with (4 * (agenda.sample keyboard.arrows 1)))
I'm calling the thing voxel in reference to "scalar field"s used in some 3D
engines like the one described in [1] and [2]. Current s/code/draft is
attached to
this mail.
# that carries all the necessary dependency info.
Regarding the commentary in guix/gexp.scm, It can be improved, by describing
what it does in general and then explain the handy procedure useful only in the
context of (guix). I commenting the commentary below:
# ;;; This module implements "G-expressions", or "gexps". Gexps are like
# ;;; S-expressions (sexps), with two differences:
# ;;;
# ;;; 1. References (un-quotations) to derivations or packages in a gexp are
# ;;; replaced by the corresponding output file name;
In the point of view of the caller, the derivation will refer to itself,
the locally bound object.
# ;;;
# ;;; 2. Gexps embed information about the derivations they refer to.
# ;;;
In "derivation" word, there the notion of palpable future (cf.
https://en.wiktionary.org/wiki/palpable),
things that will happen. Indeed the derivation declares the expected outcome
of the build. (ungexp) kind of fights this global movement induced by the
Guile/Scheme environment...
# ;;; Gexps make it easy to write to files Scheme code that refers to store
# ;;; items, or to write Scheme code to build derivations.
Outside the context of guix, I wouldn't understand.
;;;
;;; Code:
If I'm not clear (gexp) is very useful to me, I will use it. I am curious to
know how it relates to http://wingolog.org/archives/2014/03/17/stack-overflow
[1] Jump to AI http://www.voxelquest.com/1/post/2014/03/march-mega-update.html
(via https://hn.algolia.com/#!/story/sort_by_date/0/voxel)
[2] http://www.simonsfoundation.org/quanta/20140416-times-arrow-traced-to-quantum-source/
[-- Attachment #2: voxels-and-the-arrow-of-time.scm --]
[-- Type: text/x-scheme, Size: 5843 bytes --]
;;
;; goal:
;; - write
;; - compute
;; - make the computer do its best
;; - (voxel zero? end-of-time resolve unexec timeout end-of-time)
;;
;; ~/src/epsilon/
;;
;; (url "http://openvibe.inria.fr/features/")
;; (url "http://freeimage.sourceforge.net/features.html")
;; (url "http://splashcon.org/2013/")
;;
(use-modules (ice-9 optargs))
(use-modules (ice-9 pretty-print))
(define* (zero? . others)
zero?)
(define* (print . others)
(write others)
(newline))
(define* (comment . others) (print others))
(define* (reference-needed . others) (print others))
(define* (debug . others) (print others))
(define* (info . others) (print others))
(define* (warning . others) (print others))
(define* (error . others) (print others))
(define* (critical . others) (print others))
(define* (hack . others) (print others))
;; time! time! time!
(define time '())
;; oops!
(define (reset)
(set! time '()))
;; tango color palette from (url "/usr/share/emacs/24.3/etc/themes/tango-dark-theme.el")
;;
;; (butter-1 "#fce94f") (butter-2 "#edd400") (butter-3 "#c4a000")
;; (orange-1 "#fcaf3e") (orange-2 "#f57900") (orange-3 "#ce5c00")
;; (choc-1 "#e9b96e") (choc-2 "#c17d11") (choc-3 "#8f5902")
;; (cham-1 "#8ae234") (cham-2 "#73d216") (cham-3 "#4e9a06")
;; (blue-1 "#729fcf") (blue-2 "#3465a4") (blue-3 "#204a87")
;; (plum-1 "#ad7fa8") (plum-2 "#75507b") (plum-3 "#5c3566")
;; (red-1 "#ef2929") (red-2 "#cc0000") (red-3 "#a40000")
;; (alum-1 "#eeeeec") (alum-2 "#d3d7cf") (alum-3 "#babdb6")
;; (alum-4 "#888a85") (alum-5 "#555753") (alum-6 "#2e3436")
;; ;; Not in Tango palette; used for better contrast.
;; (cham-0 "#b4fa70") (blue-0 "#8cc4ff") (plum-0 "#e6a8df")
;; (red-0 "#ff4b4b") (alum-5.5 "#41423f") (alum-7 "#212526"))
(define (time:set! key value)
(show "settings" key "to" value)
(set! time (assoc-set! time key value))
value)
(define (time:get key)
(assoc-ref time key))
(define (time:show-rec map)
(if (not (empty-list? map))
(and (show "time:" (object->string (car map))) (time:show-rec (cdr map)))))
(define (time:show)
(time:show-rec time))
(define (voxel-string depth string)
(if (eq? depth 0)
string
(voxel-string (- depth 1) (string-append string " "))))
(define (object->string object)
(if (list? object) (objects->string object)
(cond ((string? object) object)
((pair? object) (string-append (object->string (car object)) " => " (object->string (cdr object))))
((symbol? object) (string-append "'" (symbol->string object)))
((procedure? object) (string-append "(procedure " (symbol->string (procedure-name object)) ")"))
((boolean? object) (if object "#t" "#f"))
(else "[oops]"))))
(define (objects->string-rec objects strings)
(if (empty-list? objects)
(string-append "(list " (string-join (reverse strings) " ") ")")
(objects->string-rec (cdr objects) (cons (object->string (car objects)) strings))))
(define (empty-list? l)
(if (pair? l)
#f
(or (not l) (eq? (length l) 0) (eq? l '()) (eq? l #nil))))
(define (objects->string objects)
(if (empty-list? objects)
"[empty]"
(objects->string-rec objects '())))
(define (voxel-depth->string)
(voxel-string voxel-depth ""))
(define (show-rec objects string)
(if (empty-list? objects)
(display string)
(show-rec (cdr objects) (string-append string " " (object->string (car objects))))))
(define (show . args)
(show-rec args "")
(newline))
(define (show-past args)
(show "past is" (past->string args)))
(define (voxel-name x)
(string->symbol (symbol->string x)))
(define (make-voxel attribute)
(letrec ((new-voxel (lambda (past future)
(voxel (cons voxel (cons new-voxel past)) future))))
(set-procedure-property! new-voxel 'name (voxel-name attribute))
new-voxel))
(define (resolve-symbol attribute)
(if (time:get attribute) (time:get attribute)
(let ((time-voxel (make-voxel attribute)))
(show "fuuu" time-voxel)
(time:set! attribute time-voxel)
time-voxel)))
(define (voxel-resolve-procedure procedure)
;; if it's voxel ie. in the time
;; else convert it -- to a root voxel
(let ((name (procedure-name procedure)))
(if (time:get name)
procedure
(time:set! name procedure))))
(define (voxel-resolve-next next)
(cond ((symbol? next) (resolve-symbol next))
((procedure? next) (voxel-resolve-procedure next))
(else (string-append "voxel-resolve-next failed with " (object->string next)))))
(define (voxel-resolve-future past future)
(letrec ((next (car future))
(future-tail (cdr future))
(next-voxel (voxel-resolve-next next)))
(time:set! 'current next-voxel)
(next-voxel past future-tail)))
(define voxel-depth 0)
(define (voxel-depth-inc)
(set! voxel-depth (+ 1 voxel-depth))
voxel-depth)
(define (voxel-depth-reset)
(set! voxel-depth 0))
(define (entering-voxel past future)
(show "entering voxel")
(show "past is" (objects->string past))
(show "future is" (objects->string future))
(show "end of showing voxel"))
(define (empty-past? past)
(empty-list? past))
(define (empty-future? future)
(empty-list? future))
(define (voxel past future . args)
(entering-voxel past future)
(if (empty-past? past)
(if (empty-future? future)
voxel ;; case voxel is called alone with no future and no past
(voxel-resolve-future past future))
;; handle voxels with history
;; resolve next voxel if any
;;
(if (empty-future? future)
(and (time:set! (reverse past) #t) voxel)
(voxel-resolve-future past future))))
(voxel 'key define)
;; example override of a voxel
(define (end-of-time past future)
(voxel (cons end-of-time past) future)
(show "> end of time")
(time:show)
(time:reset)
(show "> end of time"))
(define (testing past future)
(voxel (cons testing past) future))
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: G expressions
2014-04-28 7:43 ` Ludovic Courtès
2014-04-28 14:03 ` Amirouche Boubekki
@ 2014-04-28 21:45 ` Ludovic Courtès
1 sibling, 0 replies; 5+ messages in thread
From: Ludovic Courtès @ 2014-04-28 21:45 UTC (permalink / raw)
To: guix-devel
Merged!
Here’s the documentation.
Comments and bug reports welcome!
Ludo’.
4.6 G-Expressions
=================
So we have “derivations”, which represent a sequence of build actions to
be performed to produce an item in the store (*note Derivations::).
Those build actions are performed when asking the daemon to actually
build the derivations; they are run by the daemon in a container (*note
Invoking guix-daemon::).
It should come as no surprise that we like to write those build
actions in Scheme. When we do that, we end up with two "strata" of
Scheme code(1): the “host code”—code that defines packages, talks to the
daemon, etc.—and the “build code”—code that actually performs build
actions, such as making directories, invoking ‘make’, etc.
To describe a derivation and its build actions, one typically needs
to embed build code inside host code. It boils down to manipulating
build code as data, and Scheme’s homoiconicity—code has a direct
representation as data—comes in handy for that. But we need more than
Scheme’s normal ‘quasiquote’ mechanism to construct build expressions.
The ‘(guix gexp)’ module implements "G-expressions", a form of
S-expressions adapted to build expressions. G-expressions, or "gexps",
consist essentially in three syntactic forms: ‘gexp’, ‘ungexp’, and
‘ungexp-splicing’ (or simply: ‘#~’, ‘#$’, and ‘#$@’), which are
comparable respectively to ‘quasiquote’, ‘unquote’, and
‘unquote-splicing’ (*note ‘quasiquote’: (guile)Expression Syntax.).
However, there are major differences:
• Gexps are meant to be written to a file and run or manipulated by
other processes.
• When a package or derivation is unquoted inside a gexp, the result
is as if its output file name had been introduced.
• Gexps carry information about the packages or derivations they
refer to, and these dependencies are automatically added as inputs
to the build processes that use them.
To illustrate the idea, here is an example of a gexp:
(define build-exp
#~(begin
(mkdir #$output)
(chdir #$output)
(symlink (string-append #$coreutils "/bin/ls")
"list-files")))
This gexp can be passed to ‘gexp->derivation’; we obtain a derivation
that builds a directory containing exactly one symlink to
‘/gnu/store/…-coreutils-8.22/bin/ls’:
(gexp->derivation "the-thing" build-exp)
As one would expect, the ‘"/gnu/store/…-coreutils"’ string is
substituted to the reference to the COREUTILS package in the actual
build code, and COREUTILS is automatically made an input to the
derivation. Likewise, ‘#$output’ (equivalent to ‘(ungexp output)’) is
replaced by a string containing the derivation’s output directory name.
The syntactic form to construct gexps is summarized below.
-- Scheme Syntax: #~ EXP
-- Scheme Syntax: (gexp EXP)
Return a G-expression containing EXP. EXP may contain one or more
of the following forms:
‘#$OBJ’
‘(ungexp OBJ)’
Introduce a reference to OBJ. OBJ may be a package or a
derivation, in which case the ‘ungexp’ form is replaced by its
output file name—e.g., ‘"/gnu/store/…-coreutils-8.22’.
If OBJ is a list, it is traversed and any package or
derivation references are substituted similarly.
If OBJ is another gexp, its contents are inserted and its
dependencies are added to those of the containing gexp.
If OBJ is another kind of object, it is inserted as is.
‘#$PACKAGE-OR-DERIVATION:OUTPUT’
‘(ungexp PACKAGE-OR-DERIVATION OUTPUT)’
This is like the form above, but referring explicitly to the
OUTPUT of PACKAGE-OR-DERIVATION—this is useful when
PACKAGE-OR-DERIVATION produces multiple outputs (*note
Packages with Multiple Outputs::).
‘#$output[:OUTPUT]’
‘(ungexp output [OUTPUT])’
Insert a reference to derivation output OUTPUT, or to the main
output when OUTPUT is omitted.
This only makes sense for gexps passed to ‘gexp->derivation’.
‘#$@LST’
‘(ungexp-splicing LST)’
Like the above, but splices the contents of LST inside the
containing list.
G-expressions created by ‘gexp’ or ‘#~’ are run-time objects of the
‘gexp?’ type (see below.)
-- Scheme Procedure: gexp? OBJ
Return ‘#t’ if OBJ is a G-expression.
G-expressions are meant to be written to disk, either as code
building some derivation, or as plain files in the store. The monadic
procedures below allow you to do that (*note The Store Monad::, for more
information about monads.)
-- Monadic Procedure: gexp->derivation NAME EXP [#:system
(%current-system)] [#:inputs '()] [#:hash #f] [#:hash-algo #f]
[#:recursive? #f] [#:env-vars '()] [#:modules '()]
[#:references-graphs #f] [#:local-build? #f]
[#:guile-for-build #f]
Return a derivation NAME that runs EXP (a gexp) with
GUILE-FOR-BUILD (a derivation) on SYSTEM.
Make MODULES available in the evaluation context of EXP; MODULES is
a list of names of Guile modules from the current search path to be
copied in the store, compiled, and made available in the load path
during the execution of EXP—e.g., ‘((guix build utils) (guix build
gnu-build-system))’.
The other arguments are as for ‘derivation’.
-- Monadic Procedure: gexp->script NAME EXP
Return an executable script NAME that runs EXP using GUILE with
MODULES in its search path.
The example below builds a script that simply invokes the ‘ls’
command:
(use-modules (guix gexp) (gnu packages base))
(gexp->script "list-files"
#~(execl (string-append #$coreutils "/bin/ls")
"ls"))
When “running” it through the store (*note ‘run-with-store’: The
Store Monad.), we obtain a derivation that procedures an executable
file ‘/gnu/store/…-list-files’ along these lines:
#!/gnu/store/…-guile-2.0.11/bin/guile -ds
!#
(execl (string-append "/gnu/store/…-coreutils-8.22"/bin/ls")
"ls")
-- Monadic Procedure: gexp->file NAME EXP
Return a derivation that builds a file NAME containing EXP.
The resulting file holds references to all the dependencies of EXP
or a subset thereof.
Of course, in addition to gexps embedded in “host” code, there are
also modules containing build tools. To make it clear that they are
meant to be used in the build stratum, these modules are kept in the
‘(guix build …)’ name space.
---------- Footnotes ----------
(1) The term "stratum" in this context was coined by Manuel Serrano
et al. in the context of their work on Hop.
^ permalink raw reply [flat|nested] 5+ messages in thread