unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / code / Atom feed
From: Christopher Lemmer Webber <cwebber@dustycloud.org>
To: "Bonface M. K." <bonfacemunyoki@gmail.com>
Cc: guix-devel@gnu.org, Dimos Dimakakos <me@bendersteed.tech>
Subject: Re: Racket packages / build system
Date: Mon, 19 Oct 2020 13:04:03 -0400	[thread overview]
Message-ID: <87v9f6w3jw.fsf@dustycloud.org> (raw)
In-Reply-To: <868sc27cix.fsf@gmail.com>

[-- Attachment #1: Type: text/plain, Size: 2591 bytes --]

Bonface M. K. writes:

> Christopher Lemmer Webber <cwebber@dustycloud.org>
> writes:
>
>> Bonface M. K. writes:
>>
>>> Christopher Lemmer Webber <cwebber@dustycloud.org> writes:
>>>
>>>> Just a heads up that right now you *can* run Racket-on-Chez, but soon
>>>> Racket-on-Chez will be the "default"... maybe it's a good idea to see
>>>> how hard it will be to make a racket-on-chez package variant.
>>>>
>>>> I'm interested in looking at that but it'll probably have to be a while
>>>> before I can do so... if someone does so before me, I'll be interested
>>>> in beta testing at least.
>>>>
>>>> But also leaving this here as a self-reminder so we aren't surprised
>>>> when it becomes a more important thing to deal with :)
>>>>
>>>>
>>>
>>> Also, since we are talking about Racket, what
>>> happened to having a racket build system?
>>
>> There's interest in it, and Dimos made interesting progress towards
>> figuring out some of the key problems... there's been interest beyond
>> that but sadly it seems like organizing the energy to work on it hasn't
>> happened for whatever reason yet.
>>
>
> Can I volunteer on this task? There's some work in
> my team done in Racket; and it would be of great
> interest to us to have a working Racket build
> system. I can set apart some time to work on
> this. I'd ask for alot of guidance/ review on this
> though :)

YES!  Please do.  Let's talk.  You can ping me on IRC also, dustyweb on
freenode.

>> I've offered to throw money at the problem if someone's willing to take
>> it on btw... not much, but some money.  I've talked to a couple of
>> people about that but it stalled in each iteration.... I don't think
>> it's impossible but I guess it's one of those tasks that for whatever
>> reason seems difficult to get going on because there are some
>> complexities around the way Racket builds "collections" that eg don't
>> seem to show up in Python.  Anyway that's not a judgement because
>> despite wanting it fairly badly clearly I haven't made progress on it.
>>
>> I have the notes that Dimos wrote up not long ago in case anyone is
>> interested.  Dimos, do you mind if I post them to the list?
>>
> Please do share the notes! I can try to hack
> something up \m/\m/.

I looked at the email that Dimos sent to me (also the email I had was
apparently not the most recent email that they are using, corrected in
the addressing this time), and they had asked me if they should post it
to the mailing list, so I think that's consent to post it since they
expressed consideration in doing so in our exchange... so I'm attaching
it.


[-- Attachment #2: Dimos' notes on Racket on Guix stuff --]
[-- Type: text/plain, Size: 5943 bytes --]

* Basics of Racket packaging system

Racket provides three abstractions of how to reuse and move around modules of code. These are:
** Libraries

A library in Racket is a single file module that can be used in other files. Libraries that are
serving some higher purpose are organised together in Collections.

Documentation can be found [[https://docs.racket-lang.org/reference/collects.html][here]].

** Collections

Collections are a number of libraries bundled together. They can be added to the system through
packages. Racket gets informed for presence of collections through collection link files.

In the filesystem collections are directories that include library files. The default path they are
stored is #<path:/usr/share/racket/collects>. The corresponding link file is located at
#<path:/usr/share/racket/links.rktd> and includes a list as illustrated below:

#+begin_src racket
((root "pkgs/racket-lib")
 (root "pkgs/main-distribution")
 (root "pkgs/2d"))
#+end_src

More specifics about link files and their structure can be found [[https://docs.racket-lang.org/reference/collects.html#%28tech._collection._links._file%29][here]]. More links can be added to
arbitary directories through 'raco link'. This file informs racket where a collection resides in the
#<path:/usr/share/racket> directory.

** Packages

Packages in Racket are the abstractions used to share and move around modules of code. The include a
number of libraries in a collection, or more collections. They are the means through which
dependencies are defined. The main entry for managing them is 'raco pkg'.

The tool for installing packages is 'raco pkg install' that takes as an argument a package source
(name of package in a catalog, directory, tar or zip file etc). How it will act is defined by the
"info.rkt" file included in the package source. 

Documentation for [[https://docs.racket-lang.org/pkg/cmdline.html#%28part._raco-pkg-install%29]['raco pkg install']] and [[https://docs.racket-lang.org/pkg/metadata.html]["info.rkt"]].

Racket can be configured for where to install and search for packages through a confing file
#<path:/etc/usr/racket/config.rktd>. The documentation for the configurations can be found [[https://docs.racket-lang.org/raco/config-file.html?q=raco][here]].

* Racket build system

I will get into the thoughts that I have for the various phases of the build procedure:

** unpack phase
Here normal procedures form 'gnu-build-system' work just fine. The only exception is the handling of
.plt files, which can be done as the .gem files are handled in 'ruby-build-system'.

** bootstrap/build/install phase
Since building and installing is done with 'raco pkg install', it makes sense to have a single phase
dealing with this. The problems that exist here are:
1. <#path:/etc/racket/config.rktd> needs to be updated with all the the places it needs to look for
   the packages. These include the following:
   1. lib-search-dirs
   2. lib-dir
   3. bin-dir
   4. links-search-files
   5. pkgs-search-dirs
   6. collects-search-dirs
   7. doc-search-dirs
      
   This needs to occur two times. First time to build the package, including just its inputs.  Then
   after/while installing the package, we need to create a new config.rktd that includes the proper
   places for all formerly and newly installed packages.

2. Racket packages have circular dependencies.

3. Racket tries to rebuild the dependencies for the package it installs, even when not needed, based
   on timestamps. There exists an environmental variable "PLT_COMPILED_FILE_CHECK", that in
   documentation says if it's set on exist this won't happen. This doesn't seem to work as intended.

4. Documentation is created by mutating the docs structure and adding new links to new nodes of documentation.

For the *first issue*, Claes Wallin that worked on racket2nix, in fact recreates the racket structure
in the outputs folder by copying in the /usr/share/collects, and /lib/racket folders, and creating a
symlink of /usr/share/racket/include. He then creates uses the store provided binaries of
racket(racket, raco, gracket) with -G flag to set the config directory in the outputs and -X flat to
set the collects directory. This way the packages are built and installed in the racket
configuration we created at the outputs. Then the config.rktd needs to be generated. Claes Wallin
does this with a racket script, which we could use, but could also just use a guile function.
I have implemented this, and works as expected, in most cases that don't meet the following issues.

The *second issue* is resolved by the importer in racket2nix. The derivations are expanded to
include circular dependencies and while building the offending packages, dummy inputs are
created. We can do the same using some DFS guile package. Are there any standard graph libraries in
guile? 

The *third issue* is more complicated. Racket will try to to recompile dependencies based on
timestamps. This creates issues during the build. In racket2nix there is created a separate
environment where all dependencies are writable. It is an ugly workaround, but seems to work.

As for the *fourth issue* I didn't research the racket documentation system a lot, especially since
it's packages are the of the main offenders in circulars dependencies.

Binaries of racket packages are placed in "launchers" in the racket configuration folder, so
symlinks to '/bin' shall be created.

** testing phase

It's mostly trivial with 'raco test'.

* Racket importer
Racket packages are defined with "info.rkt" files. Since it's sexp-based it's trivial to parse them
and create package-definitons in guile. There exists some edge-cases like 'implies' that change the
way the packages are upgraded, but in general it pretty simple.

Here is also the place where maybe we could solve the thing with circular dependencies. Is it
possible to create custom fields for inputs in guix derivations?


  reply	other threads:[~2020-10-19 17:05 UTC|newest]

Thread overview: 19+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-10-17 18:42 Racket will move on top of Chez soon Christopher Lemmer Webber
2020-10-17 19:14 ` Pierre Neidhardt
2020-10-18 13:06 ` Bonface M. K.
2020-10-19  4:53   ` Racket packages / build system Christopher Lemmer Webber
2020-10-19 10:09     ` Bonface M. K.
2020-10-19 17:04       ` Christopher Lemmer Webber [this message]
2020-10-19 17:13         ` Dimos Dimakakos
2020-10-19 18:08         ` Bonface M. K.
2020-10-21  0:49           ` Christopher Lemmer Webber
2020-10-21 10:33           ` Ludovic Courtès
2020-10-21 12:59             ` Bonface M. K.
2021-01-28  3:03           ` Stephen Paul Weber
2020-11-09 20:54     ` Bonface M. K.
2020-11-09 21:21       ` Dimos Dimakakos
2020-11-09 22:51         ` Christopher Lemmer Webber
2020-11-10 12:07           ` Bonface M. K.
2020-11-10 17:37             ` Christopher Lemmer Webber
2020-11-11 10:53               ` Bonface M. K.
2020-11-10 12:30           ` Bonface M. K.

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://guix.gnu.org/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87v9f6w3jw.fsf@dustycloud.org \
    --to=cwebber@dustycloud.org \
    --cc=bonfacemunyoki@gmail.com \
    --cc=guix-devel@gnu.org \
    --cc=me@bendersteed.tech \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).