[GSoC 2020] [Proposal draft #0] Initial JVM support for Guix

[GSoC 2020] [Proposal draft #0] Initial JVM support for Guix

[Leandro Doctors]

On Fri, 20 Mar 2020 at 21:41, Leandro Doctors <ldoctors@gmail.com> wrote:
>  On Tue, 3 Mar 2020 at 19:32, zimoun <zimon.toutoune@gmail.com> wrote:
> > Based on your interests (Clojure, Leiningen, etc.), you should
> > consider something around a Clojure "importer".
>
> I am preparing my proposal. I will send it in the next few days.

Hello, everybody!

This is my first draft (at the bottom of this message). Please note
that the document is in a very early stage, so at this point there are
still many questions unanswered (to which I have added some notes
below).

Your feedback will be crucial in answering questions and evolving the
attached draft into a full proposal.

Best,
Leandro


****** Notes ******
- Whereas I may switch to org-mode, I am currently using LyX for
writing the proposal.
- I thought plain text was the best for getting feedback. If you think
another format is better (Markdown, LaTeX, PDF...) please let me know.
Pandoc is my friend :-)
- I watched the whole Guix-Jupyter Scicloj video session from last
January 9th, 2020.
https://scicloj.github.io/posts/2020-03-07-guix-jupyter/
- From what I see, in Guix there are compilers and importers (I'm in
the process of getting familiar to this terminology).
- There is already a compiler for Clojure, and Clojure has been
packaged into Guix. However, there is no importer for Clojure
packages...
- I agree with some comments from the talk: given that Clojure is a
hosted language, merely importing "Clojure packages" is impossible. In
this case, as Clojure is hosted in the JVM, we should aim to importing
Maven (an eventually, also Clojars) packages. So, adding "Clojure
support" would mean "adding JVM support".
- I guess that supporting tools/deps.edn would enable supporting Maven
dependencies?
- Packaging clojupyter would be a potential task to consider during
the first coding period (maybe even before?)




******************** Draft **********************


Initial JVM support for Guix

Leandro Doctors
<ldoctors@gmail.com>

1 Overview<sec:Overview>

Add support for importing JVM packages (jar files) into Guix.


2 Status Quo<sec:Problem-Statement>

Guix supports importing package metadata from multiple sources.
Currently, these sources are as diverse as GNU packages,
repositories such as the Python Package Index and the
Comprehensive R Archive Network, and JSON files. However, there
is another source not yet supported by Guix: JVM-based languages.
Currently, Guix does not support importing from any JVM-based
language, such as Java, Clojure. Considering Java is the most
used programming language, this would be a very valuable addition
for Guix.


3 Status Desideravit<sec:Solution-Overview>


3.1 Objectives

1. Add a JVM importer.

2. Also support Clojure jars.


3.2 Benefits

• Gain access to the JVM environment.


4 Implementation Plan<sec:Implementation-Plan>


4.1 Stages & Deliverables

<TBD>

4.2 Timeline & Milestones<sec:Timeline-&-Milestones>


Note: I have considered 5-days weeks for all periods, so there
can be slack time if needed.

1. Student Application Period (March 16th - 31st) (2 weeks)

  • Start flicking through Guix's code. [done]
  • Set up a development environment. [done]
  • Start learning the basics about Guix's internal processes
    (release management, developer interactions, codes of
    conduct...).
  • Start reading Guix documentation. [in progress]
  • Start exploring possible approaches to implement proposed
    features. [in progress]

2. Application Review Period (March 31st - April 27th) (4 weeks)

  • Open PRs with small code and/or documentation glitches
    discovered during the first step of this list.
  • Finish reading introductory material.
  • Start experimenting with possible approaches to implement
    proposed features.
  • Finish learning the basics about Guix internal processes
    (release management, developer interactions, codes of
    conduct...).
  • Continue hacking into Guix's codebase to get to know it
    better.
  • Engage with the Community and develop possible features not
initially considered.


3. Student Projects Announced (April 27th) (1 day)

4. Community Bonding (April 27th - May 18th) (3 weeks)

  • Continue hacking into Guix's codebase to get to know it
    better.
  • Finish experimenting with possible approaches found during
    the Application Review period with which to implement
    proposed features.
  • Explore options to implement proposed features.
  • Re-assessment of implementation difficulty proposed features.

5. Coding #1 (May 18th - June 12th) (4 working weeks)

  Implement Stage #1:

  (a) Weeks 1 & 2: <Focus for the Week Here>
    i. <Task here>

    Milestones:
    i. M1.1: <Milestone here>

  (b) Weeks 2 & 3

  (c) Week 4

6. Partial Evaluation #1 (June 15th - 19th) (1 buffer week)

  (a) Week 5: (Buffer)

    Milestones:
    i. (finish unfinished milestones)

7. Coding #2 (June 23rd - July 10th) (3 working weeks)

  Implement Stage #2:

  (a) Week 6

  (b) Week 7

  (c) Week 8

8. Partial Evaluation #2 (July 13th - 17th) (1 buffer week)

  (a) Week 9: (Buffer)

    Milestones:
    i. (finish unfinished milestones)

9. Coding #3 (July 20th - August 7th) (3 working weeks)

  Implement Stage #3:

  (a) Weeks 10 & 11

  (b) Week 12

10. Students Submit Code and Final Evaluations (August 10th -
  17th) (1 buffer week)

  (a) Week 13: (Buffer)

    Milestones:
    i. (finish unfinished milestones)

11. Mentors Submit Final Evaluations (August 17th - 24th) (1
  week)

12. Results Announced (August 25th) (1 day)


A About the Applicant<sec:Personal-background>

<Annex withheld for now>

Re: [GSoC 2020] [Proposal draft #0] Initial JVM support for Guix

[Julien Lepiller]

Le 23 mars 2020 11:21:06 GMT-04:00, Leandro Doctors <ldoctors@gmail.com> a écrit :
>On Fri, 20 Mar 2020 at 21:41, Leandro Doctors <ldoctors@gmail.com>
>wrote:
>>  On Tue, 3 Mar 2020 at 19:32, zimoun <zimon.toutoune@gmail.com>
>wrote:
>> > Based on your interests (Clojure, Leiningen, etc.), you should
>> > consider something around a Clojure "importer".
>>
>> I am preparing my proposal. I will send it in the next few days.
>
>Hello, everybody!
>
>This is my first draft (at the bottom of this message). Please note
>that the document is in a very early stage, so at this point there are
>still many questions unanswered (to which I have added some notes
>below).
>
>Your feedback will be crucial in answering questions and evolving the
>attached draft into a full proposal.
>
>Best,
>Leandro
>
>
>****** Notes ******
>- Whereas I may switch to org-mode, I am currently using LyX for
>writing the proposal.
>- I thought plain text was the best for getting feedback. If you think
>another format is better (Markdown, LaTeX, PDF...) please let me know.
>Pandoc is my friend :-)
>- I watched the whole Guix-Jupyter Scicloj video session from last
>January 9th, 2020.
>https://scicloj.github.io/posts/2020-03-07-guix-jupyter/
>- From what I see, in Guix there are compilers and importers (I'm in
>the process of getting familiar to this terminology).
>- There is already a compiler for Clojure, and Clojure has been
>packaged into Guix. However, there is no importer for Clojure
>packages...
>- I agree with some comments from the talk: given that Clojure is a
>hosted language, merely importing "Clojure packages" is impossible. In
>this case, as Clojure is hosted in the JVM, we should aim to importing
>Maven (an eventually, also Clojars) packages. So, adding "Clojure
>support" would mean "adding JVM support".
>- I guess that supporting tools/deps.edn would enable supporting Maven
>dependencies?
>- Packaging clojupyter would be a potential task to consider during
>the first coding period (maybe even before?)
>
>
>
>
>******************** Draft **********************
>
>
>Initial JVM support for Guix
>
>Leandro Doctors
><ldoctors@gmail.com>
>
>1 Overview<sec:Overview>
>
>Add support for importing JVM packages (jar files) into Guix.
>
>
>2 Status Quo<sec:Problem-Statement>
>
>Guix supports importing package metadata from multiple sources.
>Currently, these sources are as diverse as GNU packages,
>repositories such as the Python Package Index and the
>Comprehensive R Archive Network, and JSON files. However, there
>is another source not yet supported by Guix: JVM-based languages.
>Currently, Guix does not support importing from any JVM-based
>language, such as Java, Clojure. Considering Java is the most
>used programming language, this would be a very valuable addition
>for Guix.
>
>
>3 Status Desideravit<sec:Solution-Overview>
>
>
>3.1 Objectives
>
>1. Add a JVM importer.
>
>2. Also support Clojure jars.
>
>
>3.2 Benefits
>
>• Gain access to the JVM environment.
>
>
>4 Implementation Plan<sec:Implementation-Plan>
>
>
>4.1 Stages & Deliverables
>
><TBD>
>
>4.2 Timeline & Milestones<sec:Timeline-&-Milestones>
>
>
>Note: I have considered 5-days weeks for all periods, so there
>can be slack time if needed.
>
>1. Student Application Period (March 16th - 31st) (2 weeks)
>
>  • Start flicking through Guix's code. [done]
>  • Set up a development environment. [done]
>  • Start learning the basics about Guix's internal processes
>    (release management, developer interactions, codes of
>    conduct...).
>  • Start reading Guix documentation. [in progress]
>  • Start exploring possible approaches to implement proposed
>    features. [in progress]
>
>2. Application Review Period (March 31st - April 27th) (4 weeks)
>
>  • Open PRs with small code and/or documentation glitches
>    discovered during the first step of this list.
>  • Finish reading introductory material.
>  • Start experimenting with possible approaches to implement
>    proposed features.
>  • Finish learning the basics about Guix internal processes
>    (release management, developer interactions, codes of
>    conduct...).
>  • Continue hacking into Guix's codebase to get to know it
>    better.
>  • Engage with the Community and develop possible features not
>initially considered.
>
>
>3. Student Projects Announced (April 27th) (1 day)
>
>4. Community Bonding (April 27th - May 18th) (3 weeks)
>
>  • Continue hacking into Guix's codebase to get to know it
>    better.
>  • Finish experimenting with possible approaches found during
>    the Application Review period with which to implement
>    proposed features.
>  • Explore options to implement proposed features.
>  • Re-assessment of implementation difficulty proposed features.
>
>5. Coding #1 (May 18th - June 12th) (4 working weeks)
>
>  Implement Stage #1:
>
>  (a) Weeks 1 & 2: <Focus for the Week Here>
>    i. <Task here>
>
>    Milestones:
>    i. M1.1: <Milestone here>
>
>  (b) Weeks 2 & 3
>
>  (c) Week 4
>
>6. Partial Evaluation #1 (June 15th - 19th) (1 buffer week)
>
>  (a) Week 5: (Buffer)
>
>    Milestones:
>    i. (finish unfinished milestones)
>
>7. Coding #2 (June 23rd - July 10th) (3 working weeks)
>
>  Implement Stage #2:
>
>  (a) Week 6
>
>  (b) Week 7
>
>  (c) Week 8
>
>8. Partial Evaluation #2 (July 13th - 17th) (1 buffer week)
>
>  (a) Week 9: (Buffer)
>
>    Milestones:
>    i. (finish unfinished milestones)
>
>9. Coding #3 (July 20th - August 7th) (3 working weeks)
>
>  Implement Stage #3:
>
>  (a) Weeks 10 & 11
>
>  (b) Week 12
>
>10. Students Submit Code and Final Evaluations (August 10th -
>  17th) (1 buffer week)
>
>  (a) Week 13: (Buffer)
>
>    Milestones:
>    i. (finish unfinished milestones)
>
>11. Mentors Submit Final Evaluations (August 17th - 24th) (1
>  week)
>
>12. Results Announced (August 25th) (1 day)
>
>
>A About the Applicant<sec:Personal-background>
>
><Annex withheld for now>

Hi Leandro!

I'm glad you're interested in JVM languages. I am currently trying to build a proper maven-build-system (it's a matter of days now). Some of the issues we have with JVM languages and that you should take into account  for your proposal:

- a jar file is a binary file that needs to be built from source. Exceptions are compilers when there is no alternative (gcc used to be an example, but there is nim, ocaml, maybe closure itself) and some tools that need themselves to build (such as a parser, like javacc).
- maven allows programmers to upload a "source" jar, but it is meant for debugging, contains generated sources and might be incomplete.
- maven doesn't link to source repositories
- how to find a source repository from a groupId and an artifactId is not clear to me.

Considering that, I don't know if your project is doable. At least, all other importers are able to give you a working source, but if it's really impossible, having a list of dependencies would already be great :)

We have a lot of java packages, but none of them are usable with maven yet. I've worked on those that are directly required to run maven and its common plugins already, and will send patches as soon as everything works. There will need to be a huge work on these packages to rebuild them, probably with the maven-build-system.

Re: [GSoC 2020] [Proposal draft #0] Initial JVM support for Guix

[Leandro Doctors]

On Mon, 23 Mar 2020 at 12:39, Julien Lepiller <julien@lepiller.eu> wrote:

> Hi Leandro!

Hi, Julien,

Thanks a lot for your feedback! I add my comments below.


> I'm glad you're interested in JVM languages.

In fact, I am only interested in Clojure... However, given that
Clojure is hosted on the JVM, it seems very difficult to me not to
think in JVM terms when describing a "Clojure importer". This is the
main reason behind the proposal name change from my previous emails.


> I cannot find a way to I am currently trying to build a proper maven-build-system (it's a matter of days now).

Again, I am only speaking in JVM terms because of Clojure's hosted nature.


> Some of the issues we have with JVM languages and that you should take into account  for your proposal:
[...]

Thanks for this!


> - maven allows programmers to upload a "source" jar, but it is meant for debugging, contains generated sources and might be incomplete.

Even though I don't have data to back up this claim, in the
Guix-Jupyter Scicloj video session [1] they mention that, in the case
of Clojure, most packages usually include sources.

[1] Guix-Jupyter Scicloj video session from January 9th, 2020.
https://scicloj.github.io/posts/2020-03-07-guix-jupyter/


> - maven doesn't link to source repositories

You're right.

However, Clojars (the primary hub for Clojure packages) does. Maybe
implementing Clojars support could be a first step?
(I still have to assess this, as there are many Clojure packages that
are published primarily in Maven Central...)


> - how to find a source repository from a groupId and an artifactId is not clear to me.

Considering the previous points, this seems mostly a Java-specific problem?


> Considering that, I don't know if your project is doable. At least, all other importers are able to give you a working source, but if it's
> really impossible, having a list of dependencies would already be great :)

I agree with you in that this does seem non-trivial...
Perhaps, this could be a research-oriented project? I mean, to see
whether it is feasible to (eventually) support the JVM, and the major
milestone being a minimalistic POC implementation for Clojars support?


> We have a lot of java packages, but none of them are usable with maven yet. I've worked on those that are directly required to run
> maven and its common plugins already, and will send patches as soon as everything works. There will need to be a huge work on
> these packages to rebuild them, probably with the maven-build-system.

I guess this means I should focus on Clojars?


Best,
Leandro

Re: [GSoC 2020] [Proposal draft #0] Initial JVM support for Guix

[Julien Lepiller]

Le 23 mars 2020 12:06:12 GMT-04:00, Leandro Doctors <ldoctors@gmail.com> a écrit :
>On Mon, 23 Mar 2020 at 12:39, Julien Lepiller <julien@lepiller.eu>
>wrote:
>
>> Hi Leandro!
>
>Hi, Julien,
>
>Thanks a lot for your feedback! I add my comments below.
>
>
>> I'm glad you're interested in JVM languages.
>
>In fact, I am only interested in Clojure... However, given that
>Clojure is hosted on the JVM, it seems very difficult to me not to
>think in JVM terms when describing a "Clojure importer". This is the
>main reason behind the proposal name change from my previous emails.
>
>
>> I cannot find a way to I am currently trying to build a proper
>maven-build-system (it's a matter of days now).
>
>Again, I am only speaking in JVM terms because of Clojure's hosted
>nature.

Actually, I'm interested in android apps, but maven ended up being a requirement ^^"

>
>
>> Some of the issues we have with JVM languages and that you should
>take into account  for your proposal:
>[...]
>
>Thanks for this!
>
>
>> - maven allows programmers to upload a "source" jar, but it is meant
>for debugging, contains generated sources and might be incomplete.
>
>Even though I don't have data to back up this claim, in the
>Guix-Jupyter Scicloj video session [1] they mention that, in the case
>of Clojure, most packages usually include sources.
>
>[1] Guix-Jupyter Scicloj video session from January 9th, 2020.
>https://scicloj.github.io/posts/2020-03-07-guix-jupyter/

I'll check, thanks :)

>
>
>> - maven doesn't link to source repositories
>
>You're right.
>
>However, Clojars (the primary hub for Clojure packages) does. Maybe
>implementing Clojars support could be a first step?
>(I still have to assess this, as there are many Clojure packages that
>are published primarily in Maven Central...)
>
>
>> - how to find a source repository from a groupId and an artifactId is
>not clear to me.
>
>Considering the previous points, this seems mostly a Java-specific
>problem?
>
>
>> Considering that, I don't know if your project is doable. At least,
>all other importers are able to give you a working source, but if it's
>> really impossible, having a list of dependencies would already be
>great :)
>
>I agree with you in that this does seem non-trivial...
>Perhaps, this could be a research-oriented project? I mean, to see
>whether it is feasible to (eventually) support the JVM, and the major
>milestone being a minimalistic POC implementation for Clojars support?
>
>
>> We have a lot of java packages, but none of them are usable with
>maven yet. I've worked on those that are directly required to run
>> maven and its common plugins already, and will send patches as soon
>as everything works. There will need to be a huge work on
>> these packages to rebuild them, probably with the maven-build-system.
>
>I guess this means I should focus on Clojars?

From what you say, yes. A clojars importer would be great. If/when we find other ways to be able to import other JVM packages, we'll be able to improve upon that work :). Maybe the importer could work in that way:

Import from Clojars, if it fails, import from central, without source information. Basically, try multiple repos, from the most specific (that have source info) to more generic (that at least haveqversion and dependency information).

If you leave some space to be able to plug in other sources (eg. when we have scala, for sbt repos), I think you'll have a substantial contribution :). Does it sound doable?

>
>
>Best,
>Leandro

Re: [GSoC 2020] [Proposal draft #0] Initial JVM support for Guix

[Leandro Doctors]

On Mon, 23 Mar 2020 at 14:00, Julien Lepiller <julien@lepiller.eu> wrote:
> Le 23 mars 2020 12:06:12 GMT-04:00, Leandro Doctors <ldoctors@gmail.com> a écrit :
> >Even though I don't have data to back up this claim, in the
> >Guix-Jupyter Scicloj video session [1] they mention that, in the case
> >of Clojure, most packages usually include sources.
> I'll check, thanks :)

If you only are interested in the Clojure-related comments, only watch
the last 25' or so from the video.


> >I guess this means I should focus on Clojars?
> From what you say, yes.

> Does it sound doable?

I think it does. It will be a Clojars importer, then :-)
We'll see how far can I take the idea...


Best,
Leandro