Looking to contribute

Looking to contribute

[cinder88]

Hello. I recently learned about GuixSD, and it seems like a distro
I would actually _enjoy_ using. Essentially, my questions are:

What is the current state of GuixSD? Is it currently too buggy or
feature-deficient to use as a day-to-day os?

How can I best contribute?

-Andrew

Re: Looking to contribute

[Danny Milosavljevic]

Hi and welcome,

On Tue, 27 Jun 2017 18:16:08 -0400
cinder88@hushmail.com wrote:

> What is the current state of GuixSD? Is it currently too buggy or
> feature-deficient to use as a day-to-day os?

I'm using it every day, also for professional programming.

It's quite okay to use.  There are occassional problems - which can be mitigated by the feature below:

If there are bugs on system update, you can roll back by selecting another entry in the bootloader menu (a new one is created everytime you reconfigure the system).  This includes all the installed packages (!!!!).  So it's pretty much indestructible - I think that this is the killer feature of GuixSD.

Check the package list https://www.gnu.org/software/guix/packages/ for information on which packages are available.

Note that a normal GuixSD system takes at least 80 GB of disk space (it will suck), better 160 GB (that will work very nicely).  That mostly because of the rollback feature.

> How can I best contribute?

If you want to install GuixSD there's a "wip-installer" (WIP means "work in progress") branch on git <https://git.savannah.gnu.org/git/guix.git>.  If you just want to test the finished installer please download http://web.fdn.fr/~lcourtes/software/guix/guixsd-graphical-installer-x86-64-linux.lz and unpack it by invoking "lzip -d guixsd*.lz".  This results in a disk image that you can flash onto a USB flash drive (don't copy it as a file unto the filesystem on the flash USB drive - rather replace the entire drive by it).

It has quite a few rough edges so don't expect Ubuntu-style usability (yet).

If you do try the installer, it would be good if you noted which areas suck :)

I think you should contribute in an area that you care about.  There's plenty to do.  For example:

Installer:
- Currently, the installer invokes the parted executable for partitioning which is quite jarring (it looks very different).  It would be nice if it just used the parted library.  But there's no good guile-parted yet.  I've started hacking on one and it's starting to look OK but it's not done.  If you are interested in that I can upload it somewhere (github, gitlab etc).
- No mouse support yet.
- We have ISO9660 CD support code so it would be quite easy to make it so that the image file for the graphical installer (see link above) also worked if you burned it to a CD.  See threads mentioning "grub-mkrescue" on the mainling list.

Bootloader:
- The bootloader support code doesn't support booting Windows systems etcetc.
- There's no u-boot installer yet.  (There's support for using existing u-boot but it doesn't install the u-boot bootloader itself yet.  That's because doing that safely is very difficult)

Packages:
- Texlive for a long time was a 2 GB download which was done every time one of the dependencies changed.  It's better now but there are still situations where it does that.  It should be more modular so the downloads aren't that large (and not done as often).
- Rust is not the newest version because they changed the build system they use (sigh...) and I didn't get it to work yet.
- Our packaging is such that each package goes into its own directory /gnu/store/3523523643262hashvaluehere WHICH IS READ-ONLY, you can't possibly write there.  But dub, the D package manager, sometimes tries to do that and fails.  This is a design limitation in the D package manager (it has so-called "configurations" and if you switch configurations it will try to switch symlinks) but it means you can't run many of the D tests.  Fixing that in dub would be great.
- The GUI hooks for desktop files and mime types suck.  Usually, the desktop file database is updated much too late (mostly by accident).  I've looked into fixing it but it would entail rebuilding almost all the graphical packages we have - and I think KDE would still not do it correctly.

Some of the areas I work in are mostly bootloader, filesystem and partitioning stuff - which happen to be the areas where I have to be quite careful about not breaking the killer feature.  If you like that anyway, please help us with it :)

What are your interests?

Re: Looking to contribute

[Alex Vong]

[-- Attachment #1: Type: text/plain, Size: 455 bytes --]

Hi Cinder,

Welcome to Guix!

cinder88@hushmail.com writes:

> Hello. I recently learned about GuixSD, and it seems like a distro
> I would actually _enjoy_ using. Essentially, my questions are:
>
> What is the current state of GuixSD? Is it currently too buggy or
> feature-deficient to use as a day-to-day os?
>
> How can I best contribute?
>
This webpage contains a lot of useful information:
<https://www.gnu.org/software/guix/contribute/>

> -Andrew

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Looking to contribute

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 476 bytes --]

On Wed, Jun 28, 2017 at 03:49:43PM +0200, Danny Milosavljevic wrote:
> Note that a normal GuixSD system takes at least 80 GB of disk space
> (it will suck), better 160 GB (that will work very nicely).  That
> mostly because of the rollback feature.

This seems very high to me. I had >100 system generations over 6 months
with less than 100 GB, although that was not a graphical system. So, it
depends on how "big" you make each generation, and how many generations
you keep.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Looking to contribute

[James Richardson]

cinder88@hushmail.com writes:

> Hello. I recently learned about GuixSD, and it seems like a distro
> I would actually _enjoy_ using. Essentially, my questions are:
>
> What is the current state of GuixSD? Is it currently too buggy or
> feature-deficient to use as a day-to-day os?
>
I use it on my laptop and my wife's laptop. feature-deficient is a
rather subjective term. I have xfce on my wife's laptop. Occasionally
she'll complain about lack of flash or that a website doesn't like
icecat. I have a few boxen that I haven't converted (from Debian) yes as
I rely heavily on lvm and don't want to sort out how to do without
lvm. On those boxen I use guix as a foreign package manager which
actually works out quite well.


> How can I best contribute?
>
I've found the people on the #guix channel and on this list to be very
helpful. The website has a section on how to contribute. I've started by
just packaging programs that I use that were net yet in Guix. Knowing
guile is not (IHMO) a prerequisite, although willingness to learn
probably is ;).

HTH, James.
-- 
I prefer encrypted email.
GPG Fingerprint = 8FD2 7619 C19A 2201 CB1D  E131 EA1C F14B D846 7AFB
https://jamestechnotes.com/gpgkey
https://jamestechnotes.com

Re: Looking to contribute

[cinder88]

On 6/28/2017 at 9:49 AM, "Danny Milosavljevic" <dannym@scratchpost.org> wrote:
>
>Hi and welcome,

Thanks for for the excellent post. It looks like there's a good community here.

>If there are bugs on system update, you can roll back by selecting 
>another entry in the bootloader menu (a new one is created 
>everytime you reconfigure the system).  This includes all the 
>installed packages (!!!!).  So it's pretty much indestructible - I 
>think that this is the killer feature of GuixSD.

Exactly. Currently I run Gentoo with systemd. To manage packages
and configuration, I wrote a script which tracks config files throughout
the system. Then a system-wide recompile with Gentoo updates any
reconfigured software to meet the new config specifications. Effectively
what I'm doing is specifying the state of the system with the config, and
telling Portage (the package manager) to meet the specs. Then I
remembered using NixOS a while back, which had exactly solved that
problem! And when I heard of a Nix-based, libre system using Scheme,
I knew I was home.

>> How can I best contribute?
>
>If you want to install GuixSD there's a "wip-installer" (WIP means 
>"work in progress") branch on git 
><https://git.savannah.gnu.org/git/guix.git>.  If you just want to 
>test the finished installer please download 
>http://web.fdn.fr/~lcourtes/software/guix/guixsd-graphical-
>installer-x86-64-linux.lz and unpack it by invoking "lzip -d 
>guixsd*.lz".  This results in a disk image that you can flash onto 
>a USB flash drive (don't copy it as a file unto the filesystem on 
>the flash USB drive - rather replace the entire drive by it).
>
>It has quite a few rough edges so don't expect Ubuntu-style 
>usability (yet).
>
>If you do try the installer, it would be good if you noted which 
>areas suck :)

Alright, I will try that and let you all know how it goes.

>I think you should contribute in an area that you care about.  
>There's plenty to do.  For example:
>
>Installer:
>- Currently, the installer invokes the parted executable for 
>partitioning which is quite jarring (it looks very different).  It 
>would be nice if it just used the parted library.  But there's no 
>good guile-parted yet.  I've started hacking on one and it's 
>starting to look OK but it's not done.  If you are interested in 
>that I can upload it somewhere (github, gitlab etc).

I would be happy to work on this.

>Packages:
>- Texlive for a long time was a 2 GB download which was done every 
>time one of the dependencies changed.  It's better now but there 
>are still situations where it does that.  It should be more 
>modular so the downloads aren't that large (and not done as often).

I'd be happy to work on this as well, as I use TeX extensively.

>Some of the areas I work in are mostly bootloader, filesystem and 
>partitioning stuff - which happen to be the areas where I have to 
>be quite careful about not breaking the killer feature.  If you 
>like that anyway, please help us with it :)
>

> What are your interests?

Professionally I do higher-level software design and implementation.
I'm not too used to lower level system details, but by using Gentoo
I've learned a decent amount about how GNU/Linux operates.

I'm a mathematician, which means I'm lazy and want to do things the
right way once so I don't have to do them again. The way that modern
operating systems work, with their one huge environment and no
tracking of state, is insane and unsustainable. (How many times have
I wasted a Saturday morning trying to get audio work on a new Gentoo
install?) Stateless, declarative systems seem to be the obvious way forward.

I'm a big fan of Alan Kay, and one thing he always made clear is that
there should be no separation between the language and the operating
system. I've found that many of the problems which plague modern OSes
are the same which plague C. Which is why a system which sits on top
of Scheme, one of the most powerful programming paradigms we have,
is so appealing.

More concretely, I can be of the most help with:
- Texlive and dealing with pdf stuff (cairo, etc)
- Emacs
- Networking
But realy I'll try to help wherever it's needed, because this project
should be a reality.

Re: Looking to contribute

[Katherine Cox-Buday]

cinder88@hushmail.com writes:

> I'm a mathematician, which means I'm lazy and want to do things the
> right way once so I don't have to do them again. The way that modern
> operating systems work, with their one huge environment and no
> tracking of state, is insane and unsustainable. (How many times have
> I wasted a Saturday morning trying to get audio work on a new Gentoo
> install?) Stateless, declarative systems seem to be the obvious way
> forward.
>
> I'm a big fan of Alan Kay, and one thing he always made clear is that
> there should be no separation between the language and the operating
> system. I've found that many of the problems which plague modern OSes
> are the same which plague C. Which is why a system which sits on top
> of Scheme, one of the most powerful programming paradigms we have,
> is so appealing.
>
> More concretely, I can be of the most help with:
> - Texlive and dealing with pdf stuff (cairo, etc)
> - Emacs
> - Networking
> But realy I'll try to help wherever it's needed, because this project
> should be a reality.

I have nothing meaningful to contribute except to say that I appreciate
and agree with this line of thinking.

I have also found this to be a wonderful, warm, and welcoming community
doing great work. The only thing I don't like is that I haven't been
able to find time to contribute :)

Welcome!

-- 
Katherine

Re: Looking to contribute

[Alex Vong]

[-- Attachment #1: Type: text/plain, Size: 991 bytes --]

cinder88@hushmail.com writes:
[...]
> I'm a big fan of Alan Kay, and one thing he always made clear is that
> there should be no separation between the language and the operating
> system. I've found that many of the problems which plague modern OSes
> are the same which plague C. Which is why a system which sits on top
> of Scheme, one of the most powerful programming paradigms we have,
> is so appealing.
>
Since you mentioned you see analogy between language and OS (they face
similar problems), I think you will find this[0] interesting. It
discusses how delimted continuations are often used in OS without
explicit mentioning it.

Also, this reminds me of the joke: Emacs is an operating system

> More concretely, I can be of the most help with:
> - Texlive and dealing with pdf stuff (cairo, etc)
> - Emacs
> - Networking
> But realy I'll try to help wherever it's needed, because this project
> should be a reality.

[0]: http://okmij.org/ftp/continuations/zipper.html#context-OS

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Looking to contribute

[Danny Milosavljevic]

Hi Leo,

On Wed, 28 Jun 2017 11:01:45 -0400
Leo Famulari <leo@famulari.name> wrote:

> This seems very high to me. I had >100 system generations over 6 months
> with less than 100 GB, although that was not a graphical system. 

I have a graphical system with fluxbox, some electrical engineering & FPGA programs, D, Rust, Haskell, gimp, a (Guix) cross compilation environment for armhf - and a lot of Python packages.  It's basically what I need for work and some play.  That's reeeeally annoying to use with 80 GB (I should know, my old system partition has that...).

I have 568 system generations, I win ;-)  But it's mostly because of excessive bootloader testing - no normal person should have that many system generations... so I guess it's an overestimate.

>So, it depends on how "big" you make each generation, and how many generations you keep.

Wait.  AFAIK one cannot delete system generations [yet].  Has that changed?

But deleting user [package] generations works, yes.

Re: Looking to contribute

[Pjotr Prins]

On Thu, Jun 29, 2017 at 09:22:46AM -0400, cinder88@hushmail.com wrote:
> telling Portage (the package manager) to meet the specs. Then I
> remembered using NixOS a while back, which had exactly solved that
> problem! And when I heard of a Nix-based, libre system using Scheme,
> I knew I was home.

Yep, that is me too. Nix is great; Guix is even better.

Pj.

Re: Looking to contribute

[Mekeor Melire]

Danny Milosavljevic <dannym@scratchpost.org> writes:

> Wait.  AFAIK one cannot delete system generations [yet].  Has that changed?
>
> But deleting user [package] generations works, yes.

Somewhere on the mailing-list I found out that you can delete a system
generation like this:

    $ rm /var/guix/profiles/system-N-link
    $ guix gc

(as root?)

Deleting system generations

[Ludovic Courtès]

Mekeor Melire <mekeor.melire@gmail.com> skribis:

> Danny Milosavljevic <dannym@scratchpost.org> writes:
>
>> Wait.  AFAIK one cannot delete system generations [yet].  Has that changed?
>>
>> But deleting user [package] generations works, yes.
>
> Somewhere on the mailing-list I found out that you can delete a system
> generation like this:
>
>     $ rm /var/guix/profiles/system-N-link
>     $ guix gc
>
> (as root?)

Yes, that’s all it takes.

Thomas: I think you (was it you?) started working on a ‘guix system
delete-generations’ sub-command so make it more accessible.  Any update?
:-)

We should definitely add that command.

Thanks,
Ludo’.

Re: Deleting system generations

[Thomas Danckaert]

From: ludo@gnu.org (Ludovic Courtès)
Subject: Deleting system generations
Date: Sat, 01 Jul 2017 15:15:23 +0200

> Thomas: I think you (was it you?) started working on a ‘guix system
> delete-generations’ sub-command so make it more accessible.  Any 
> update?
> :-)

Yes, that's correct :-)  No update, unfortunately, though it's still 
on my todo-list ...  If anyone wants to take over, here's the last 
message about it:

https://lists.gnu.org/archive/html/guix-devel/2017-03/msg00194.html

(the point is to add extra boot configurations to <boot-parameters> 
so we can handle them properly in guix system switch-generation or 
delete-generations)

Thomas

Re: Deleting system generations

[Ludovic Courtès]

Thomas Danckaert <post@thomasdanckaert.be> skribis:

> From: ludo@gnu.org (Ludovic Courtès)
> Subject: Deleting system generations
> Date: Sat, 01 Jul 2017 15:15:23 +0200
>
>> Thomas: I think you (was it you?) started working on a ‘guix system
>> delete-generations’ sub-command so make it more accessible.  Any
>> update?
>> :-)
>
> Yes, that's correct :-)  No update, unfortunately, though it's still
> on my todo-list ...  If anyone wants to take over, here's the last
> message about it:
>
> https://lists.gnu.org/archive/html/guix-devel/2017-03/msg00194.html
>
> (the point is to add extra boot configurations to <boot-parameters> so
> we can handle them properly in guix system switch-generation or
> delete-generations)

Oh right, thanks for the link!

Ludo’.

Binding generator, for Guile, to C libraries

[Danny Milosavljevic]

Hi,

> >- Currently, the installer invokes the parted executable for 
> >partitioning which is quite jarring (it looks very different).  It 
> >would be nice if it just used the parted library.  But there's no 
> >good guile-parted yet.  I've started hacking on one and it's 
> >starting to look OK but it's not done.  If you are interested in 
> >that I can upload it somewhere (github, gitlab etc).  

> I would be happy to work on this.

Okay, I've cleaned up what I have and put it on github, under:

https://github.com/daym/guile-gcc-unit

What this does is it allows you to read gcc output files which specify all the prototypes - in our case in order to extract the prototypes and generate Guile bindings (of parted, or whatever).

The next steps:
- Provide the actual binding generator (I started a version of it - I'll commit a cleaned-up version in a few days I guess).
- Get it to work with non-toy examples (see file "tests/huge/input" for the parted non-toy example - the goal is to read this file, find all the record decls and all the function decls in it and generate the Guile pendants of them).  If you happen to know LALR parsing theory, I have a question :)
- Find out whether that can be integrated as a Guile "language".  Just being able to do something like ',load "foo.h"' and have it register all the functions from it would be way cool.  On the other hand, on non-Guix systems the header files aren't usually installed when the library is installed.  So maybe an offline version would be good as well.

Example Makefile:

---------------------------------------------------
.PHONY: clean all distclean

CC = gcc

all: parted.lu
	guile -L . main.scm $^

a.c:
	echo "#include <parted/parted.h>" > a.c.new && mv a.c.new a.c

parted.lu: a.c
	$(CC) -fsyntax-only -fdump-tree-all-graph=$@ $^

clean:
	rm -f a.c

distclean: clean
	rm -f parted.lu
--------------------------------------------------

To test it, put this file into guile-gcc-unit (as "Makefile") and then invoke "guix package -i parted" and then invoke "make".

(I didn't commit it to the repo yet because maybe we can put guile-parted into an extra repo?  Or even just have a helper script that does it all without any extra repo?)

For the bindings generator, if you want to play with it, what I already can share is:

(use-modules (system foreign))

(define-public %parted-library (dynamic-link "/gnu/store/ddpg6rlr5f3xv8fjh8812ll9g584x51z-parted-3.2/lib/libparted"))

(define-public enum-_PedPartitionType
  (make-enumeration '(
    PED_PARTITION_NORMAL ; 0
    PED_PARTITION_LOGICAL ; 1
    PED_PARTITION_EXTENDED ; 2
    reserved3 ; 3
    PED_PARTITION_FREESPACE ; 4
    reserved5 ; 5
    reserved6 ; 6
    reserved7 ; 7
    PED_PARTITION_METADATA ; 8
    reserved9
    reserved10 ; 10
    reserved11 ; 11
    reserved12 ; 12
    reserved13 ; 13
    reserved14 ; 14
    reserved15 ; 15
    PED_PARTITION_PROTECTED ; 16
)))

(define-public long-long uint64)
(define-public PedSector long-long)

(define ped_disk_set_partition_geom (pointer->procedure int (dynamic-func "ped_disk_set_partition_geom" %parted-library) '* '* '* PedSector PedSector))

So that's the kind of code that should be generated in the end (not sure whether in Tree-IL or whether to go via the Scheme intermediary).

For the binding generator, I'm thinking of having a global hashtable containing what symbols are known from C header files - and what is known about them.

So (hypothetical) usage would be like:

; Magically registers all the external function declarations in some global hashtable (only the declarations, no shared object)
,load "parted/parted.h"

(define-public %parted-library (dynamic-link "/gnu/store/ddpg6rlr5f3xv8fjh8812ll9g584x51z-parted-3.2/lib/libparted"))
(define ped_disk_set_partition_geom (dynamic-callable-func "ped_disk_set_partition_geom" %parted-library))
(ped_disk_set_partition_geom ...)

Wouldn't that be cool?

If that's too lowlevel, check out guix wip-installer and try using gurses, John's ncurses frontend.  The installer is under "gnu/system/installer/"  :)

We eventually would like to have a nice partitioning interface in the installer.  So if you'd like to you can also add some pages there.

In order to do that, see gnu/system/installer/guixsd-installer.scm . It contains the list of pages that are used in the installer.  You can add your own pages there.

See gnu/system/installer/mount-point.scm for a simple page.

Installer development

[Danny Milosavljevic]

Hi,

if you want, in order to start developing the installer, the easiest way is to run as NORMAL USER (not root):

$ mkdir guix-installer
$ cd guix-installer
$ git clone -b wip-installer-2 https://git.savannah.gnu.org/git/guix.git
$ cd guix
$ ./pre-inst-env guix environment guix --fallback --pure --ad-hoc guile-ncurses
[env] $ ./pre-inst-env guix system installer

The "environment --pure" makes sure that parted, mkfs etc is not available and you don't kill your existing system by accident.

In order to cancel the installer, press Ctrl-z and then invoke:

guix [env] $ kill -9 %1

I pushed a wip-installer-2 branch, starting from master.  Into the former I cherry-picked all the wip-installer-only commits and massaged them until the installer worked.  I hope that's OK with everyone.

Re: Looking to contribute

[Danny Milosavljevic]

Here's an update :)

>> Installer:
>> - Currently, the installer invokes the parted executable for partitioning which is quite jarring (it looks very different).  It would be nice if it just used the parted library.

>>But there's no good guile-parted yet.  I've started hacking on one and it's starting to look OK but it's not done.  If you are interested in that I can upload it somewhere (github, gitlab etc).

>I would be happy to work on this.

guile-gcc-unit is 80% done now (it also works on the real parted headers by now).  See <https://github.com/daym/guile-gcc-unit>.  (I think I'll not use records after all and just keep the result as huge trees that you can (ice-9 match) on).

What it allows you is to:
- invoke gcc (with a special option) on a tiny source file which only "#include"s stuff and then
- feed the output to guile-gcc-unit and then
- get out a Scheme list of all the C prototypes, records, enums etc that are available.

The next step would be to map the generated declaration tree (output of main.scm) to the wrapper forms I posted - to have actual callable parted procedures.

> - No mouse support yet.

Now the installer does have mouse support :)

Updated installer dev usage (branch "wip-installer-2" on <https://git.savannah.gnu.org/git/guix.git>):

$ ./pre-inst-env guix environment guix --fallback --pure --ad-hoc guile-ncurses kbd pciutils util-linux wireless-tools
[env]$ ./pre-inst-env guix system installer
... press Ctrl-z when you are fed up with it :)

Be careful that it doesn't nuke your system (it does ask before formatting or installing anything - so it's not that bad).

Also, don't run it as root...

> - We have ISO9660 CD support code so it would be quite easy to make it so that the image file for the graphical installer (see link above) also worked if you burned it to a CD.  See threads mentioning "grub-mkrescue" on the mainling list.

95% done.  If would actually work if we came to a consensus about the volume label (it must be uppercase; see bug# 27520 in guix-patches).  Also, UUID boot support is still mostly missing - same as in the non-iso9660 case.

> Bootloader:
> - The bootloader support code doesn't support booting Windows systems etcetc.

Started to provide scaffolding on this - let's see what we decide about using <menu-entry> (see bug# 27529 in guix-patches).  I don't really have a use for it so I won't go much further than providing the scaffolding.

> - There's no u-boot installer yet.  (There's support for using existing u-boot but it doesn't install the u-boot bootloader itself yet.  That's because doing that safely is very difficult)

Depends on the parted bindings.

> Packages:
> - The GUI hooks for desktop files and mime types suck.  Usually, the desktop file database is updated much too late (mostly by accident).  I've looked into fixing it but it would entail rebuilding almost all the graphical packages we have - and I think KDE would still not do it correctly.

宋文武 fixed it :)

Re: Binding generator, for Guile, to C libraries

[Ludovic Courtès]

Hello,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> Okay, I've cleaned up what I have and put it on github, under:
>
> https://github.com/daym/guile-gcc-unit
>
> What this does is it allows you to read gcc output files which specify all the prototypes - in our case in order to extract the prototypes and generate Guile bindings (of parted, or whatever).
>
> The next steps:
> - Provide the actual binding generator (I started a version of it - I'll commit a cleaned-up version in a few days I guess).
> - Get it to work with non-toy examples (see file "tests/huge/input" for the parted non-toy example - the goal is to read this file, find all the record decls and all the function decls in it and generate the Guile pendants of them).  If you happen to know LALR parsing theory, I have a question :)
> - Find out whether that can be integrated as a Guile "language".  Just being able to do something like ',load "foo.h"' and have it register all the functions from it would be way cool.  On the other hand, on non-Guix systems the header files aren't usually installed when the library is installed.  So maybe an offline version would be good as well.

Matt Wette, the author of nyacc, was apparently in the process of
writing a similar tool using nyacc’s C99 parser.  Might be worth looking
at.

Besides and FWIW, I’m skeptical of binding generators.  Things like
SWIG, for instance, generate interfaces that are often not quite what
you’d like, so you end up writing a layer on top of the generated glue
anyway, not to mention annotation in headers.  (Bindings generated from
high-level descriptions like GIR and XCB are a different story; those
seem to work quite well.)

That said, you might want to share on guile-user@gnu.org to get feedback!

Ludo’.

Re: Binding generator, for Guile, to C libraries

[Danny Milosavljevic]

Hi Ludo,

On Wed, 05 Jul 2017 23:53:36 +0200
ludo@gnu.org (Ludovic Courtès) wrote:

> Danny Milosavljevic <dannym@scratchpost.org> skribis:
> 
> > Okay, I've cleaned up what I have and put it on github, under:
> >
> > https://github.com/daym/guile-gcc-unit
> >
> > What this does is it allows you to read gcc output files which specify all the prototypes - in our case in order to extract the prototypes and generate Guile bindings (of parted, or whatever).
> >
> > The next steps:
> > - Provide the actual binding generator (I started a version of it - I'll commit a cleaned-up version in a few days I guess).

> > - Get it to work with non-toy examples (see file "tests/huge/input" for the parted non-toy example - the goal is to read this file, find all the record decls and all the function decls in it

This part is working now, also for parted.

> and generate the Guile pendants of them).  If you happen to know LALR parsing theory, I have a question :)
> > - Find out whether that can be integrated as a Guile "language".  Just being able to do something like ',load "foo.h"' and have it register all the functions from it would be way cool.  On the other hand, on non-Guix systems the header files aren't usually installed when the library is installed.  So maybe an offline version would be good as well.  
> 
> Matt Wette, the author of nyacc, was apparently in the process of
> writing a similar tool using nyacc’s C99 parser.  Might be worth looking
> at.

Nice!  I've actually thought about using nyacc and mes for that - but since we use gcc to compile everything, we can also use gcc to find out what it compiled - it's more probable that that actually matches up.  What I like is that gcc actually dumps the declarations even when you *don't* call the function.  So you can actually create a mostly-empty source file with #include in it and gcc will give you everything that is in scope.

But it's definitely worth taking a look at Matt Wette's actual Guile dynamic-* generator, maybe that can be shared between the projects.

> Besides and FWIW, I’m skeptical of binding generators.  Things like
> SWIG, for instance, generate interfaces that are often not quite what
> you’d like, so you end up writing a layer on top of the generated glue
> anyway, not to mention annotation in headers. 

Yeah, probably the bindings will suck a bit.  But one can always write a wrapper and not export all the original functions from there.  Rust for example has a "...-sys" convention where the package "foo-sys" contains the easily-generated bindings and the package "foo" contains the actually simple high-level bindings.

What I don't want is to manually maintain all the bindings.  It's a computer, it should automate it :P

Writing "override" declarations (or maybe even just literally overwriting the definition in Scheme) once every few months I'm fine with.  Adapting the bindings every time upstream adds a function?  No.

If I plan to use something like the "...-sys" convention in Guile, can I somehow re-export all the same names as the "-sys" package did, in a non-sys package?

For example:

(define-module (foo-sys)
  #:export (a b c))

...
------
(define-module (foo)
  #:re-export (same-names-as-in foo-sys, but use newer versions from below if applicable))

(define (b ...)
  fix it up)
-----
Hmm, I guess I can use (include-from-path "foo-sys") in the second module before I overwrite anything.

> (Bindings generated from
> high-level descriptions like GIR and XCB are a different story; those
> seem to work quite well.)

Yeah, I like glib's approach with the def files which specify also the object lifetimes etc.

Installer, ISO9660, etc.

[Ludovic Courtès]

Hi!

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> 95% done.  If would actually work if we came to a consensus about the volume label (it must be uppercase; see bug# 27520 in guix-patches).  Also, UUID boot support is still mostly missing - same as in the non-iso9660 case.

Dose the ‘string-upcase’ suggestion work for you?  I thought we had come
up with a solution, I hope I’m not holding anything back in this area!

>> - There's no u-boot installer yet.  (There's support for using existing u-boot but it doesn't install the u-boot bootloader itself yet.  That's because doing that safely is very difficult)
>
> Depends on the parted bindings.

FWIW GNU fdisk has (outdated) Parted bindings in C, not sure if that’s
worth reusing; perhaps we’d be better off rolling our own with the FFI.

Ludo’.

Re: Binding generator, for Guile, to C libraries

[Ludovic Courtès]

Hi Danny,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

>>  Matt Wette, the author of nyacc, was apparently in the process of
>> writing a similar tool using nyacc’s C99 parser.  Might be worth
>> looking at.
>
> Nice!  I've actually thought about using nyacc and mes for that - but
> since we use gcc to compile everything, we can also use gcc to find
> out what it compiled

Sure; I think Matt’s idea is just to use nyacc’s cpp/C99 parser to
extract function declarations, nothing more.

>> Besides and FWIW, I’m skeptical of binding generators.  Things like
>> SWIG, for instance, generate interfaces that are often not quite
>> what you’d like, so you end up writing a layer on top of the
>> generated glue anyway, not to mention annotation in headers.
>
> Yeah, probably the bindings will suck a bit.  But one can always write
> a wrapper and not export all the original functions from there.  Rust
> for example has a "...-sys" convention where the package "foo-sys"
> contains the easily-generated bindings and the package "foo" contains
> the actually simple high-level bindings.
>
> What I don't want is to manually maintain all the bindings.  It's a
> computer, it should automate it :P
>
> Writing "override" declarations (or maybe even just literally
> overwriting the definition in Scheme) once every few months I'm fine
> with.  Adapting the bindings every time upstream adds a function?  No.
>
> If I plan to use something like the "...-sys" convention in Guile, can
> I somehow re-export all the same names as the "-sys" package did, in a
> non-sys package?
>
> For example:
>
> (define-module (foo-sys) #:export (a b c))

Yes, that’s a workable plan.  I’m unsure this approach saves a lot of
time over “hand-written” FFI bindings (I write “hand-written” with
quotes because usually you’d come up with macros to do the heavy lifting
for the library you’re wrapping), but maybe you’re right!

Ludo’.

Re: Installer, ISO9660, etc.

[Danny Milosavljevic]

Hi Ludo,

On Fri, 07 Jul 2017 13:34:52 +0200
ludo@gnu.org (Ludovic Courtès) wrote:

> Danny Milosavljevic <dannym@scratchpost.org> skribis:
> 
> > 95% done.  If would actually work if we came to a consensus about the volume label (it must be uppercase; see bug# 27520 in guix-patches).  Also, UUID boot support is still mostly missing - same as in the non-iso9660 case.  

>I hope I’m not holding anything back in this area!

Oh, not at all.  I'm just not clear on which way we chose (if any?).

What was the string-upcase solution?  Even if it's created with (string-upcase "GuixSD") (and it is - if you don't override it) the boot code as it is now will still fail to find the root - because it matches labels case-sensitively.  The string-upcase is buried deep within the image creation procedure.

Or do you mean I should put (string-upcase "GuixSD") in system-disk-image in gnu/system/vm.scm as well ? That would work, I guess... although the finished image (iso9660 or not!) would still have "GUIXSD" then.  I don't see that as a big deal, though :)

There's still the following places:

./gnu/build/vm.scm:                    search --set=root --label gnu-disk-image~@
./gnu/system/install.scm:              (device "gnu-disk-image")
./gnu/system/vm.scm:    "gnu-disk-image")

Or do you mean we should just match case-insensitively in gnu/build/file-systems.scm ?

I.e. use

 (define partition-label-predicate
  (partition-predicate read-partition-label string-ci=?))

That would mean match case-insensitively for both iso9660 and non-iso9660.  I would very much prefer this fix.

Whatever it is, we should just pick a way, any way :)

Now Hydra is building a 1 GB iso9660 image that won't boot.

I've pushed bug# 27521 so "iso9660-image" will now heed the root-label (from system-disk-image) just like the other filesystems do - and then uppercase it.  That finishes the creation part.  Now the booting part... how? :)

Re: Installer, ISO9660, etc.

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1760 bytes --]

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> On Fri, 07 Jul 2017 13:34:52 +0200
> ludo@gnu.org (Ludovic Courtès) wrote:
>
>> Danny Milosavljevic <dannym@scratchpost.org> skribis:
>> 
>> > 95% done.  If would actually work if we came to a consensus about the volume label (it must be uppercase; see bug# 27520 in guix-patches).  Also, UUID boot support is still mostly missing - same as in the non-iso9660 case.  
>
>>I hope I’m not holding anything back in this area!
>
> Oh, not at all.  I'm just not clear on which way we chose (if any?).
>
> What was the string-upcase solution?  Even if it's created with (string-upcase "GuixSD") (and it is - if you don't override it) the boot code as it is now will still fail to find the root - because it matches labels case-sensitively.  The string-upcase is buried deep within the image creation procedure.

Right.

> Or do you mean I should put (string-upcase "GuixSD") in system-disk-image in gnu/system/vm.scm as well ? That would work, I guess... although the finished image (iso9660 or not!) would still have "GUIXSD" then.  I don't see that as a big deal, though :)
>
> There's still the following places:
>
> ./gnu/build/vm.scm:                    search --set=root --label gnu-disk-image~@
> ./gnu/system/install.scm:              (device "gnu-disk-image")
> ./gnu/system/vm.scm:    "gnu-disk-image")
>
> Or do you mean we should just match case-insensitively in gnu/build/file-systems.scm ?
>
> I.e. use
>
>  (define partition-label-predicate
>   (partition-predicate read-partition-label string-ci=?))
>
> That would mean match case-insensitively for both iso9660 and non-iso9660.  I would very much prefer this fix.

What about this not-so-bad solution:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 974 bytes --]

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 3e722d081..662fd0085 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -335,11 +335,18 @@ the image."
 system described by OS.  Said image can be copied on a USB stick as is.  When
 VOLATILE? is true, the root file system is made volatile; this is useful
 to USB sticks meant to be read-only."
+  (define normalize-label
+    ;; ISO labels are all-caps (case-insensitive), but since
+    ;; 'find-partition-by-label' is case-sensitive, make it all-caps here.
+    (if (string=? "iso9660" file-system-type)
+        string-upcase
+        identity))
+
   (define root-label
     ;; Volume name of the root file system.  Since we don't know which device
     ;; will hold it, we use the volume name to find it (using the UUID would
     ;; be even better, but somewhat less convenient.)
-    "gnu-disk-image")
+    (normalize-label "gnu-disk-image"))
 
   (define file-systems-to-keep
     (remove (lambda (fs)

[-- Attachment #3: Type: text/plain, Size: 396 bytes --]


That way we preserve case-sensitivity for the other file systems.

If that’s fine with you, let’s do that!

Longer-term we should probably create <file-system> records for each
file system in (gnu build file-system); this would include, for each
file system, the ‘superblock?’ procedure, the ‘label’ procedure, and,
probably, a ‘label=?’ procedure.

Thanks,
Ludo’.

Re: Installer, ISO9660, etc.

[Danny Milosavljevic]

Aha!

I finally found why I couldn't use the iso9660 image on an usb stick.

Fix:

diff --git a/gnu/build/file-systems.scm b/gnu/build/file-systems.scm
index b6930497d..7f2c5dcb0 100644
--- a/gnu/build/file-systems.scm
+++ b/gnu/build/file-systems.scm
@@ -398,7 +398,7 @@ not valid header was found."
               (match (string-tokenize line)
                 (((= string->number major) (= string->number minor)
                   blocks name)
-                 (if (partition? name major minor)
+                 (if #t ;(partition? name major minor)
                      (loop (cons name parts))
                      (loop parts))))))))))

Then it works fine on CD, on USB stick, whereever.

Is there a reason why (disk-partitions) second-guesses the Linux kernel and filters out some entries from /proc/partitions ?

Re: Installer, ISO9660, etc.

[Ludovic Courtès]

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> Aha!
>
> I finally found why I couldn't use the iso9660 image on an usb stick.
>
> Fix:
>
> diff --git a/gnu/build/file-systems.scm b/gnu/build/file-systems.scm
> index b6930497d..7f2c5dcb0 100644
> --- a/gnu/build/file-systems.scm
> +++ b/gnu/build/file-systems.scm
> @@ -398,7 +398,7 @@ not valid header was found."
>                (match (string-tokenize line)
>                  (((= string->number major) (= string->number minor)
>                    blocks name)
> -                 (if (partition? name major minor)
> +                 (if #t ;(partition? name major minor)
>                       (loop (cons name parts))
>                       (loop parts))))))))))
>
> Then it works fine on CD, on USB stick, whereever.
>
> Is there a reason why (disk-partitions) second-guesses the Linux kernel and filters out some entries from /proc/partitions ?

It has this:

  (define (partition? name major minor)
    ;; Select device names that end in a digit, like libblkid's 'probe_all'
    ;; function does.  Checking for "/sys/dev/block/MAJOR:MINOR/partition"
    ;; doesn't work for partitions coming from mapped devices.
    (and (char-set-contains? char-set:digit (last-character name))
         (> major 2)))                      ;ignore RAM disks and floppy disks

What’s the /dev name in your case?

Ludo’.

Re: Installer, ISO9660, etc.

[Danny Milosavljevic]

> What’s the /dev name in your case?

In the working case as a CD, it's /dev/sr0.
In the non-working case on a USB flash drive, it's /dev/sda and /dev/sda1.

There's a dummy partition in the iso image as well, so I'm not sure why /dev/sda1 doesn't work as well... let's see...

When fdisk says that the starting sector is "1", does that mean the first sector (i.e. there's no sector before it) or does that mean the sector after the MBR?

On the guix rescue console, I tried comparing the first few bytes of /dev/sda and /dev/sda1 and it seems they are different.

That would mean that the ISO filesystem support code can't find the primary volume descriptor any more because it's reading it from the wrong position because it reads from the device file "/dev/sda1" and that file shifts it from where it would have been if it read "/dev/sda".

But this "partition?" thing would mean that having a partition table is now mandatory, right?  Is that really supposed to be the case?  The partition table has a magic mark (0x55 0xAA) in order for it *not* to be mandatory (i.e. if you want to use the whole disk without partitioning anything, go right ahead) and I can still remember using (floppy) disks which had no partition table.

Re: Installer, ISO9660, etc.

[Ludovic Courtès]

Heya,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> But this "partition?" thing would mean that having a partition table is now mandatory, right?  Is that really supposed to be the case?  The partition table has a magic mark (0x55 0xAA) in order for it *not* to be mandatory (i.e. if you want to use the whole disk without partitioning anything, go right ahead) and I can still remember using (floppy) disks which had no partition table.

‘partition?’ is meant to return true when given a partition, so yes, it
assumes there’s a partition table somewhere.  :-)

In general, I think that’s a reasonable assumption.  Is it really a
problem to always have a partition table?

Ludo’.

Re: Installer, ISO9660, etc.

[Danny Milosavljevic]

Hi Ludo,

On Tue, 11 Jul 2017 11:23:46 +0200
ludo@gnu.org (Ludovic Courtès) wrote:

> Is it really a problem to always have a partition table?

The ISO9660 filesystem code doesn't find its primary volume descriptor when it's looking for it on the (fake) partition.  (I'm not sure what the fake partition is for in the first place).

But /proc/partitions would have contained "sda" itself, Guix just filters it out.

Of course we could special-case ISO9660 - but even easier (and less surprising) would be to just use /proc/partitions as-is and not fiddle with it (ok, maybe filter out the ram disks - not sure what that's about).

WDYT?