hpcguix-web, channels, and inferiors

hpcguix-web, channels, and inferiors

[Ludovic Courtès]

Hello Guix!

There’s a little-known but neat program that Roel Janssen wrote called
hpcguix-web.  It’s a web UI (with JavaScript) that allows you to browse
packages; the UI can be customized, and Roel’s institution, for
instance, customizes it to provide instructions for their clusters.  You
can see it in action here:

  https://guix-hpc.bordeaux.inria.fr/browse

Until now hpcguix-web would just allow you to browse the list of
packages of the Guix it was built against (it used ’fold-packages’,
etc.)

With channels and inferiors now in the house, I’ve tweaked it so that it
would periodically build the latest Guix, run it as an inferior, and
build the package list from there:

  https://github.com/UMCUGenetics/hpcguix-web/commit/e0a9a1ae35a422ba46e55f5e63441599ee53100e

That’s all I wanted to share.  :-)

Ludo’.

Re: hpcguix-web, channels, and inferiors

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 495 bytes --]

Hi!

This is fantastic news, thanks!

In fact, I think it's much better than
https://www.gnu.org/software/guix/packages/. Could we use hpcguix-web on the
official page too?

As far as I remember, it's problematic because it's hosted on gnu.org.  And
there were talks about moving the website, right?

In the meantime, what about adding a link to the hpcguix-web page, something
like "Find an up-to-date package list here"?

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: hpcguix-web, channels, and inferiors

[Ricardo Wurmus]

Hi Pierre,

> In fact, I think it's much better than
> https://www.gnu.org/software/guix/packages/. Could we use hpcguix-web on the
> official page too?
>
> As far as I remember, it's problematic because it's hosted on gnu.org.  And
> there were talks about moving the website, right?

By default everything hosted on gnu.org is static and has to be checked
into version control.  We asked for getting control over a subdomain
such as guix.gnu.org, but there are still open questions involving DNS
setup.

> In the meantime, what about adding a link to the hpcguix-web page, something
> like "Find an up-to-date package list here"?

We could add it to the website hosted on guix.info.  Currently that’s
almost identical to the site at gnu.org/s/guix.

--
Ricardo

Re: hpcguix-web, channels, and inferiors

[Clément Lassieur]

Hello,

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

> Hello Guix!
>
> There’s a little-known but neat program that Roel Janssen wrote called
> hpcguix-web.  It’s a web UI (with JavaScript) that allows you to browse
> packages; the UI can be customized, and Roel’s institution, for
> instance, customizes it to provide instructions for their clusters.  You
> can see it in action here:
>
>   https://guix-hpc.bordeaux.inria.fr/browse
>
> Until now hpcguix-web would just allow you to browse the list of
> packages of the Guix it was built against (it used ’fold-packages’,
> etc.)

Would it make sense to integrate it into Cuirass?  We could browse
packages on Cuirass and to each package would be associated a link to
its build status on different architectures, etc.

What do you think?

Clément

Re: hpcguix-web, channels, and inferiors

[Ludovic Courtès]

Hello,

Clément Lassieur <clement@lassieur.org> skribis:

> Ludovic Courtès <ludovic.courtes@inria.fr> writes:
>
>> Hello Guix!
>>
>> There’s a little-known but neat program that Roel Janssen wrote called
>> hpcguix-web.  It’s a web UI (with JavaScript) that allows you to browse
>> packages; the UI can be customized, and Roel’s institution, for
>> instance, customizes it to provide instructions for their clusters.  You
>> can see it in action here:
>>
>>   https://guix-hpc.bordeaux.inria.fr/browse
>>
>> Until now hpcguix-web would just allow you to browse the list of
>> packages of the Guix it was built against (it used ’fold-packages’,
>> etc.)
>
> Would it make sense to integrate it into Cuirass?  We could browse
> packages on Cuirass and to each package would be associated a link to
> its build status on different architectures, etc.
>
> What do you think?

Good question.  Some sort of integration is desirable, with links as you
mention.  Now Cuirass is not specifically about packages, conceptually
at least, so IMO having both features in the same tool would “break the
abstraction.”

What we could do, though, is run hpcguix-web and Cuirass on the same
machine, and customize hpcguix-web such that it includes links to
Cuirass builds, for example.  hpcguix-web already provides a level of
customization that should make it easy.

WDYT?

Thanks,
Ludo’.

Re: hpcguix-web, channels, and inferiors

[Clément Lassieur]

Hello Ludovic,

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

> Hello,
>
> Clément Lassieur <clement@lassieur.org> skribis:
>
>> Ludovic Courtès <ludovic.courtes@inria.fr> writes:
>>
>>> Hello Guix!
>>>
>>> There’s a little-known but neat program that Roel Janssen wrote called
>>> hpcguix-web.  It’s a web UI (with JavaScript) that allows you to browse
>>> packages; the UI can be customized, and Roel’s institution, for
>>> instance, customizes it to provide instructions for their clusters.  You
>>> can see it in action here:
>>>
>>>   https://guix-hpc.bordeaux.inria.fr/browse
>>>
>>> Until now hpcguix-web would just allow you to browse the list of
>>> packages of the Guix it was built against (it used ’fold-packages’,
>>> etc.)
>>
>> Would it make sense to integrate it into Cuirass?  We could browse
>> packages on Cuirass and to each package would be associated a link to
>> its build status on different architectures, etc.
>>
>> What do you think?
>
> Good question.  Some sort of integration is desirable, with links as you
> mention.  Now Cuirass is not specifically about packages, conceptually
> at least, so IMO having both features in the same tool would “break the
> abstraction.”

Yes I agree.  I wonder though: is there a way to know if a specific
derivation is a package?

> What we could do, though, is run hpcguix-web and Cuirass on the same
> machine, and customize hpcguix-web such that it includes links to
> Cuirass builds, for example.  hpcguix-web already provides a level of
> customization that should make it easy.

Sounds good!

Clément

Re: hpcguix-web, channels, and inferiors

[Ludovic Courtès]

Hi,

Clément Lassieur <clement@lassieur.org> skribis:

> Ludovic Courtès <ludovic.courtes@inria.fr> writes:
>
>> Hello,
>>
>> Clément Lassieur <clement@lassieur.org> skribis:

[...]

>>> Would it make sense to integrate it into Cuirass?  We could browse
>>> packages on Cuirass and to each package would be associated a link to
>>> its build status on different architectures, etc.
>>>
>>> What do you think?
>>
>> Good question.  Some sort of integration is desirable, with links as you
>> mention.  Now Cuirass is not specifically about packages, conceptually
>> at least, so IMO having both features in the same tool would “break the
>> abstraction.”
>
> Yes I agree.  I wonder though: is there a way to know if a specific
> derivation is a package?

In general no.  Of course one could play trick to try and guess whether
it corresponds to a package, but that should be avoided IMO.

Thanks,
Ludo’.

Re: hpcguix-web, channels, and inferiors

[Ricardo Wurmus]

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

>> Yes I agree.  I wonder though: is there a way to know if a specific
>> derivation is a package?
>
> In general no.  Of course one could play trick to try and guess whether
> it corresponds to a package, but that should be avoided IMO.

I wonder if we should work towards specifying a type for derivations.
This could be useful for the user interface to be able to clearly
distinguish grafts from full package builds.

Currently, there’s no way to tell if the derivations listed under “These
derivations will be built” are expensive package builds or just simple
graft derivations.

--
Ricardo

Re: hpcguix-web, channels, and inferiors

[Ludovic Courtès]

Ricardo Wurmus <rekado@elephly.net> skribis:

> Ludovic Courtès <ludovic.courtes@inria.fr> writes:
>
>>> Yes I agree.  I wonder though: is there a way to know if a specific
>>> derivation is a package?
>>
>> In general no.  Of course one could play trick to try and guess whether
>> it corresponds to a package, but that should be avoided IMO.
>
> I wonder if we should work towards specifying a type for derivations.
> This could be useful for the user interface to be able to clearly
> distinguish grafts from full package builds.
>
> Currently, there’s no way to tell if the derivations listed under “These
> derivations will be built” are expensive package builds or just simple
> graft derivations.

Indeed.  A simple trick would be to (ab)use the environment variable
part of derivations as a property list, the way Nix has traditionally
done it (see ‘user+system-env-vars’ in (guix derivations)).

So we could have, say, a ‘hint’ environment variable, and the UI would
use that to determine if it’s a graft.

Thoughts?

Ludo’.

Re: hpcguix-web, channels, and inferiors

[Ricardo Wurmus]

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> Ludovic Courtès <ludovic.courtes@inria.fr> writes:
>>
>>>> Yes I agree.  I wonder though: is there a way to know if a specific
>>>> derivation is a package?
>>>
>>> In general no.  Of course one could play trick to try and guess whether
>>> it corresponds to a package, but that should be avoided IMO.
>>
>> I wonder if we should work towards specifying a type for derivations.
>> This could be useful for the user interface to be able to clearly
>> distinguish grafts from full package builds.
>>
>> Currently, there’s no way to tell if the derivations listed under “These
>> derivations will be built” are expensive package builds or just simple
>> graft derivations.
>
> Indeed.  A simple trick would be to (ab)use the environment variable
> part of derivations as a property list, the way Nix has traditionally
> done it (see ‘user+system-env-vars’ in (guix derivations)).
>
> So we could have, say, a ‘hint’ environment variable, and the UI would
> use that to determine if it’s a graft.

This sounds like a good trick to me.  I think it would be great to give
more hints to the UI and make it clearer to users what work they can
expect Guix to perform.

--
Ricardo

Estimating build time

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 3490 bytes --]

Hello!

Ricardo Wurmus <rekado@elephly.net> skribis:

> Ludovic Courtès <ludovic.courtes@inria.fr> writes:
>
>> Ricardo Wurmus <rekado@elephly.net> skribis:

[...]

>>> Currently, there’s no way to tell if the derivations listed under “These
>>> derivations will be built” are expensive package builds or just simple
>>> graft derivations.
>>
>> Indeed.  A simple trick would be to (ab)use the environment variable
>> part of derivations as a property list, the way Nix has traditionally
>> done it (see ‘user+system-env-vars’ in (guix derivations)).
>>
>> So we could have, say, a ‘hint’ environment variable, and the UI would
>> use that to determine if it’s a graft.
>
> This sounds like a good trick to me.  I think it would be great to give
> more hints to the UI and make it clearer to users what work they can
> expect Guix to perform.

In a similar vein, the attached module provides code to estimate package
build time based on locally-available build logs.  It can be used to
show hints like this:

--8<---------------cut here---------------start------------->8---
The following derivations would be built (estimated time: 54 mn):
   /gnu/store/3627svyhih9cfss8gnxllp9nmxqp23cq-gcc-4.9.4.drv
   /gnu/store/3didvp9c3sfqwmb9kkdr211vg5myygsf-gcc-4.9.4.tar.xz.drv
--8<---------------cut here---------------end--------------->8---

or:

--8<---------------cut here---------------start------------->8---
The following derivations would be built (estimated time: 22 hr):
   /gnu/store/safjgjqhxlf59rknygqdfq175cl5wvks-rust-1.27.2.drv
   /gnu/store/v154ah7f8wqcga104df9ldb25bjk2pm8-rustc-1.27.2-src.tar.gz.drv
   /gnu/store/nz2xzl1yizcwcxhnw2w5mdqnp3q1gggx-rustc-1.25.0-src.tar.gz.drv
   /gnu/store/wn422brymn6ysxq07090ijb4wx78dc1l-rustc-1.24.1-src.tar.gz.drv
   /gnu/store/3cj5083j5aq7az7n5dmkds77g84xqc33-rust-bootstrap-1.22.1.drv
   /gnu/store/rm1nghqrvw43wlbwws49rw921qh58m35-rustc-1.23.0-src.tar.xz.drv
   /gnu/store/a9gzrq0bsc3ynfj4cnrsxxd2jwjgf4zj-rust-1.23.0.drv
   /gnu/store/bmbpbhgby84yrj0mvkv279cy2ira4xqf-rustc-1.24.1-src.tar.xz.drv
   /gnu/store/bxxyzp6kzq88ii4aindgnggbb2m193rk-rust-1.24.1.drv
   /gnu/store/r26s0y5fi055pclhnivvf63llbrj54yw-rustc-1.25.0-src.tar.xz.drv
   /gnu/store/x4l0rsqni9lglfzazkjyxqjp432yr33s-rustc-1.26.2-src.tar.gz.drv
   /gnu/store/03y9zf5imbm0ni1llcmxixb8c78nmxdd-rustc-1.26.2-src.tar.xz.drv
   /gnu/store/ici0m0bia0f6f4wh0ykn12x6wg1ckck0-rust-1.25.0.drv
   /gnu/store/2l6fn1nxs2sfl93khki5jzz6dh7gfqpr-rust-1.26.2.drv
   /gnu/store/9bn1gxnsc59zi8bdpvfgqcjpczmk3ch0-rustc-1.27.2-src.tar.xz.drv
--8<---------------cut here---------------end--------------->8---

(That’s from my x86_64 laptop.)

The obvious downside is that it works by first retrieving the names of
the files under /var/log/guix/drvs, and then opening, decompressing, and
parsing the candidate log files.  That typically takes a few seconds on
a recent SSD laptop, but clearly we don’t want to do that every time.
We could maintain a cache, but even then, it might still be too
expensive.

Perhaps we should keep build times in the database somehow; the daemon
can keep it up-to-date.

Thoughts?

There’s the obvious downside that both approaches rely on having
previously built the package, but I think that’s a necessary limitation,
unless we are to resort to external services (which could hardly provide
estimates that make sense for the user’s machine anyway.)

Ludo’.


[-- Attachment #2: the code --]
[-- Type: text/plain, Size: 6073 bytes --]

;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2018 Ludovic Courtès <ludo@gnu.org>
;;;
;;; This file is part of GNU Guix.
;;;
;;; GNU Guix is free software; you can redistribute it and/or modify it
;;; under the terms of the GNU General Public License as published by
;;; the Free Software Foundation; either version 3 of the License, or (at
;;; your option) any later version.
;;;
;;; GNU Guix is distributed in the hope that it will be useful, but
;;; WITHOUT ANY WARRANTY; without even the implied warranty of
;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;;; GNU General Public License for more details.
;;;
;;; You should have received a copy of the GNU General Public License
;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.

(define-module (guix build-logs)
  #:use-module (guix config)
  #:use-module (guix store)
  #:use-module (srfi srfi-1)
  #:use-module (guix utils)
  #:use-module (ice-9 match)
  #:use-module (ice-9 ftw)
  #:use-module (ice-9 regex)
  #:use-module (ice-9 rdelim)
  #:export (%log-directory
            log-file-build-phases
            log-file-build-time
            estimated-build-time))

(define %log-directory
  (string-append (dirname %state-directory) ; XXX
                 "/log/guix/drvs"))

(define %end-of-phase-rx
  (make-regexp "^phase [`']([[:graph:]]+)' succeeded after ([0-9.]+) seconds$"))

(define (log-file-build-phases file)
  "Interpret the build log in FILE and return an alist of name/duration pairs
for each build phase, such as:

  ((unpack . 1.3) (configure . 4.2) (build . 383.8) …)

Duration is expressed in seconds.  Return the empty list if no build phase
information could be extracted from FILE."
  (define compression
    (cond ((string-suffix? ".gz" file) 'gzip)
          ((string-suffix? ".bz2" file) 'bzip2)
          ((string-suffix? ".xz" file) 'xz)
          (else 'none)))

  (call-with-input-file file
    (lambda (input)
      (call-with-decompressed-port compression input
        (lambda (port)
          (set-port-conversion-strategy! port 'substitute)
          (let loop ((result '()))
            (match (read-line port)
              ((? eof-object?)
               (reverse result))
              (line
               (match (regexp-exec %end-of-phase-rx line)
                 (#f
                  (loop result))
                 (hit
                  (loop (alist-cons (string->symbol
                                     (match:substring hit 1))
                                    (string->number
                                     (match:substring hit 2))
                                    result))))))))))))

(define (log-file-build-time file)
  "Return the total build time described by FILE, a build log, or zero if
build phase information was not found."
  (match (log-file-build-phases file)
    (((names . durations) ...)
     (if (memq 'install names)
         (reduce + 0 durations)
         0))))

(define (matching-log-files package)
  (define noop
    (lambda (file stat result) result))

  (file-system-fold (const #t)
                    (lambda (file stat result)    ;leaf
                      (let* ((base (basename
                                    (file-sans-extension
                                     (file-sans-extension file))))
                             (dash (string-index base #\-))
                             (full (string-drop base (+ dash 1))))
                        (call-with-values
                            (lambda ()
                              (package-name->name+version full #\-))
                          (lambda (p v)
                            (if (and (string=? p package)
                                     (not (string-suffix? ".gz" v))
                                     (not (string-suffix? ".bz2" v))
                                     (not (string-suffix? ".xz" v))
                                     (not (string-suffix? ".lz" v))
                                     (not (string-suffix? ".zip" v)))
                                (cons (list file p v) result)
                                result)))))
                    noop                          ;down
                    noop                          ;up
                    noop                          ;skip
                    (lambda (file stat error result) ;error
                      result)
                    '()
                    %log-directory))



(define %not-dot
  (char-set-complement (char-set #\.)))

(define (version-distance version reference)
  "Compute a super rough estimate of the distance of VERSION to REFERENCE,
both of which being version strings."
  (let* ((reference (string-tokenize reference %not-dot))
         (version   (string-tokenize version %not-dot))
         (len       (length reference)))
    (let loop ((i len)
               (reference reference)
               (version version)
               (distance 0))
      (match version
        (() distance)
        ((head . tail)
         (match reference
           (()
            distance)
           ((ref-head . ref-tail)
            (loop (- i 1) ref-tail tail
                  (if (string=? ref-head head)
                      distance
                      (+ distance i))))))))))

(define (estimated-build-time package version)
  "Return the estimate time it takes to build PACKAGE at VERSION, or #f if no
such estimate is available."
  (let ((logs (sort (matching-log-files package)
                    (match-lambda*
                      (((file1 _ version1) (file2 _ version2))
                       (< (version-distance version1 version)
                          (version-distance version2 version)))))))
    (any (match-lambda
           ((log package version)
            (let ((duration (log-file-build-time log)))
              (and (not (zero? duration))
                   duration))))
         logs)))

Re: Estimating build time

[Clément Lassieur]

Ludovic Courtès <ludo@gnu.org> writes:

> In a similar vein, the attached module provides code to estimate package
> build time based on locally-available build logs.  It can be used to
> show hints like this:

[...]

> The obvious downside is that it works by first retrieving the names of
> the files under /var/log/guix/drvs, and then opening, decompressing, and
> parsing the candidate log files.  That typically takes a few seconds on
> a recent SSD laptop, but clearly we don’t want to do that every time.
> We could maintain a cache, but even then, it might still be too
> expensive.
>
> Perhaps we should keep build times in the database somehow; the daemon
> can keep it up-to-date.
>
> Thoughts?
>
> There’s the obvious downside that both approaches rely on having
> previously built the package, but I think that’s a necessary limitation,
> unless we are to resort to external services (which could hardly provide
> estimates that make sense for the user’s machine anyway.)

I think it's an excellent idea!  I agree that the downside is necessary,
and yes, keeping the build times in the database sounds good!

Clément

Re: Estimating build time

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 71 bytes --]

Same here, I love it!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Estimating build time

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1713 bytes --]

On Sat, Oct 13, 2018 at 01:13:17AM +0200, Clément Lassieur wrote:
> Ludovic Courtès <ludo@gnu.org> writes:
> 
> > In a similar vein, the attached module provides code to estimate package
> > build time based on locally-available build logs.  It can be used to
> > show hints like this:
> 
> [...]
> 
> > The obvious downside is that it works by first retrieving the names of
> > the files under /var/log/guix/drvs, and then opening, decompressing, and
> > parsing the candidate log files.  That typically takes a few seconds on
> > a recent SSD laptop, but clearly we don’t want to do that every time.
> > We could maintain a cache, but even then, it might still be too
> > expensive.
> >
> > Perhaps we should keep build times in the database somehow; the daemon
> > can keep it up-to-date.
> >
> > Thoughts?
> >
> > There’s the obvious downside that both approaches rely on having
> > previously built the package, but I think that’s a necessary limitation,
> > unless we are to resort to external services (which could hardly provide
> > estimates that make sense for the user’s machine anyway.)
> 
> I think it's an excellent idea!  I agree that the downside is necessary,
> and yes, keeping the build times in the database sounds good!
> 

One option would be, if it were a daemon-handled thing, to have it
configurable with a flag.

Off the top of my head, the linux from scratch documentation lists the
build times in units of binutils build time.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]