How to make GNU Guile more successful

How to make GNU Guile more successful

[Amirouche]

I don't know why Racket is successful probably because
it has a very good documentation and also a documentation
generator written in scheme.

  => This is a long debate and core principle of GNU project
     so I don't want to enter that debate

And it's easy to packages.

  => Another pain topic. In reality it's very easy to package
     all sort of guile programs for guix. But guix itself needs
     to be more stable.

Python is successful because it has a massive number
of libraries. In particular it has dozens of:

- web framework
- library to generate diagrams and graphical stuff
- scientific libraries (machine learning and others)
- books which teach python (the good enough way)
- 2D and 3D libraries for games

   => We (all the people reading this mail) can not just
      create all those projects. It requires to many domain
      knowledge for a handful of people to be able to compete.
      But what we can do is *share*, *share* and *share*
      again *all* what we do (on mailing list, blogs, on gitlab
      or whatever) whether it is a full blown project with a
      website made by a professional designer with tests suite,
      a tutorial documentation reference doc and a community or
      a simple *snippet* or broken experiments. I have lost
      (read deleteted) at least the code of 5 or 6 experiments
      only to recognize later that it could be useful for future
      me.

   => In terms of software I really think, we need something like
      ActiveState Python Cookbook [0]. I still go there sometime even
      I could simply ask the question on StackOverflow and get an
      answer in minutes.

      [0] http://code.activestate.com/recipes/langs/python/

During a long time I was said that Pythonist were
among the smartest developer, because simply they had
the guts to learn a language that was not learned in
school.

  => this can be true for schemers;

Python *was* easy to learn.

  => Scheme is easy to learn if you stick in the "garden"
     (garden must be defined but it's basically what I
     introduce in my tutorial [1]) Which is totally the
     same for the Python of today (read Python 3.5+).

     [1] http://hyperdev.fr

PHP was successful back in the days because it was easy
to install,

    => A scheme web application is much easier to run!

Now PHP is successful because there is a few successful
software written in it. By successful I mean they GTD.
They aren't perfect, you get bitten etc. But the basic
are there. I think about Drupal, Wordpress and Prestashop.

    => While you might not be interested to build something
       like prestashop. You very much be interested to build
       something like Drupal or Wordpress to allow mom and
       dad and others create a personnal experience on the
       web.

Another thing, Python has GNU Media Goblin

    => We must do better!

WDYT?

What block you from contributing to the wide ecosystem of GNU Guile?

Best regards,

Amirouche

Re: How to make GNU Guile more successful

[Amirouche]

Le 13/02/2017 à 00:56, Amirouche a écrit :

> WDYT?
> What block you from contributing to the wide ecosystem of GNU Guile?

I heared a lot around me about the projects I was doing before in my
previous life in as Pythonista. That I had NIH syndrom. Yeah, I rebuild
stuff that already exists hence according to the "haters" that was not
useful.

=> THIS IS A FALSE. Fully FUD! Two ways you can learn a programming
a) coding b) reading code. Basically in Guile, if there is no code to read
you only have the other option.

=> Remember they were dozen wikis before wikipedia with similar intents,
dozen search engines before Google with similar intents; You never reinvent
the wheel. You update it little by little, it might not be revolution.
Look at paintings for instance, it's seems some painters do not know proper
painting but they are still successful in their niche. Why? Because they tried.
My advice is to aim big or small whatever, but don't be shy to share.

I personnaly don't believe that code pollution exists. My theory is that
this concept was invented by techno oligarcs to limit the creativity of
those that can make. Basicly techno overlords supply the tools and you provide
the finished product that they are interested in. This is not in the interest
of creativity or liberty. Code what you want to code and think you need to code.
Don't be shy. Javascript fatigue is another name for code pollution.

Another thing i heard a lot is: It's impossible.

=> I was said that about Python->Javascript translator dozen of times. About
    my project to create a new web framework too. Basically Django is too
    mainstream. I don't want to be bigger than Django I want to have the
    *correct* solution. In many situations, for me, django was not up to the
    mark.

Another thing I hear a lot: nothing. Yeah. Nothing.

=> It take a lot of courage to do free and open source software. First,
    you might find it difficult to find people that have the same interest
    as you. Second they probably perfer to fork and do it from scratch by
    themself. Maybe they are shy and do it secretly aka. they don't want to
    disturb the matrix.

    You will also maybe meet people that hate open source or worst that hate
    GNU. Leave if possible.

That said I don't say HACK ALL DAY until something BAD happens. I say love
the code and the community share what you did.

Re: How to make GNU Guile more successful

[Arne Babenhauserheide]

Hi Amirouche,

Thank you for your nice writeup!

Amirouche writes:

> I don't know why Racket is successful probably because
> it has a very good documentation and also a documentation
> generator written in scheme.
>
>   => This is a long debate and core principle of GNU project
>      so I don't want to enter that debate

But I want to add something: I’m getting the hang of just checking the
info-page to look something up. And I realize that with well-written
info-pages this is faster than googling for stack overflow.

> And it's easy to packages.
>
>   => Another pain topic. In reality it's very easy to package
>      all sort of guile programs for guix. But guix itself needs
>      to be more stable.

I wish I had guildhall ready. Got hit by
time-eaten-by-other-project-because-guildhall-was-priority-three.

It needs to be easy to not just package some code, but also to share
that package without hitting a bottleneck.

> Python is successful because it has a massive number
> of libraries.
…
>    => We (all the people reading this mail) can not just
>       create all those projects. It requires to many domain
>       knowledge for a handful of people to be able to compete.
>       But what we can do is *share*, *share* and *share*
>       again *all* what we do (on mailing list, blogs, on gitlab
>       or whatever) whether it is a full blown project with a
>       website made by a professional designer with tests suite,
>       a tutorial documentation reference doc and a community or
>       a simple *snippet* or broken experiments.

A really simple way to share a project would be nice here. Nowadays many
people just push something with a README to github, but github is
unfree.

>    => In terms of software I really think, we need something like
>       ActiveState Python Cookbook [0]. I still go there sometime even
>       I could simply ask the question on StackOverflow and get an
>       answer in minutes.
>
>       [0] http://code.activestate.com/recipes/langs/python/
>
> During a long time I was said that Pythonist were
> among the smartest developer, because simply they had
> the guts to learn a language that was not learned in
> school.

I never heard that one :)

> Python *was* easy to learn.
>
>   => Scheme is easy to learn if you stick in the "garden"
>      (garden must be defined but it's basically what I
>      introduce in my tutorial [1]) Which is totally the
>      same for the Python of today (read Python 3.5+).

Or look at py2guile: http://draketo.de/py2guile

Python is no longer easy due to integrating additional syntactic forms,
each of which makes a specific kind of code much nicer but complicates
learning.

We have something similar in foof-loop:

(define (count-matching-items list predicate)
  (loop ((for item (in-list list))    
         (with count 0   
               (if (predicate item)
                   (+ count 1)
                   count)))
        => count))
;       ^^ here

This looks really simple in this place, but it introduces special syntax
which will be unintelligible for someone who does did not explicitly
learn foof-loop.

And guildhall is filled with special forms I had to understand just to
be able to do small changes.

This is a social problem. For a deeper discussion why it is a problem,
see http://winestockwebdesign.com/Essays/Lisp_Curse.html
"Lisp is so powerful that problems which are technical issues in other
programming languages are social issues in Lisp."

We can counter this with easy tutorials and with writing something like
canonical Scheme. But for this, we need to define a canonical Scheme
which is hits the performance and readability sweet-spot for
Guile. Canonical code must be close to the fastest code.

Practically put: We need Andy Wingo to nitpick the tutorial about things
which will cause overheads the compiler cannot fix easily — including
expensive use of macros.

>      [1] http://hyperdev.fr
>
> PHP was successful back in the days because it was easy
> to install,

Or because it was the only thing which existed. Didn’t it make it easy
for the hoster to sandbox it — both in terms of access and in terms of
resource requirements?

>     => A scheme web application is much easier to run!

Sadly not on my cheap webhoster which provides unlimited bandwidth for
a fixed cost.

>     => While you might not be interested to build something
>        like prestashop. You very much be interested to build
>        something like Drupal or Wordpress to allow mom and
>        dad and others create a personnal experience on the
>        web.
>
> Another thing, Python has GNU Media Goblin
>
>     => We must do better!

Or, maybe collaborate with it? I’m sure that there’s lots of stuff which
could be shared, because it is not language-specific (more exactly: it’s
HTML, CSS, Javascript and images).

> What block you from contributing to the wide ecosystem of GNU Guile?

- publishing a program for non-Linux platforms (well, Windows, OSX and
  mobile phones) so most of my friends can use it.

- an alternative to `python setup.py register sdist upload`

Best wishes,
Arne

Re: How to make GNU Guile more successful

[Arne Babenhauserheide]

Arne Babenhauserheide writes:
> - an alternative to `python setup.py register sdist upload`

Since we are GNUfolks, we can leverage autotools for that. I have a
pet-project which automates the setup of autotools-projects in the style
of Rust Cargo (but without the NIH). I should share it … 

Best wishes,
Arne

Re: How to make GNU Guile more successful

[Amirouche]

Le 13/02/2017 à 12:06, Arne Babenhauserheide a écrit :

> Hi Amirouche,
>
> Thank you for your nice writeup!
>
> Amirouche writes:
>
>> I don't know why Racket is successful probably because
>> it has a very good documentation and also a documentation
>> generator written in scheme.
>>
>>    => This is a long debate and core principle of GNU project
>>       so I don't want to enter that debate
> But I want to add something: I’m getting the hang of just checking the
> info-page to look something up. And I realize that with well-written
> info-pages this is faster than googling for stack overflow.

My peers have this habit during system programming looking at man
pages and other stuff. I don't know much command line-fu. My experience,
about programming:

- looking up the API referennce (procedure index)
- looking up ansers in stack overflow

In the case of Guile programming I also look general scheme solution like
in racket. Last time I checked they had a Racket->JS translator, but not
much graphdb stuff.

The ability to generate multiple format is what makes TEXINFO powerful.

>> And it's easy to packages.
>>
>>    => Another pain topic. In reality it's very easy to package
>>       all sort of guile programs for guix. But guix itself needs
>>       to be more stable.
> I wish I had guildhall ready. Got hit by
> time-eaten-by-other-project-because-guildhall-was-priority-three.
>
> It needs to be easy to not just package some code, but also to share
> that package without hitting a bottleneck.

I got stuck with guile-git; because of new way of bindings C. I use
guile-bytestructures.


>> Python is successful because it has a massive number
>> of libraries.
> …
>>     => We (all the people reading this mail) can not just
>>        create all those projects. It requires to many domain
>>        knowledge for a handful of people to be able to compete.
>>        But what we can do is *share*, *share* and *share*
>>        again *all* what we do (on mailing list, blogs, on gitlab
>>        or whatever) whether it is a full blown project with a
>>        website made by a professional designer with tests suite,
>>        a tutorial documentation reference doc and a community or
>>        a simple *snippet* or broken experiments.
> A really simple way to share a project would be nice here. Nowadays many
> people just push something with a README to github, but github is
> unfree.

I try to move to framagit. but I still use github sometimes.


>>     => In terms of software I really think, we need something like
>>        ActiveState Python Cookbook [0]. I still go there sometime even
>>        I could simply ask the question on StackOverflow and get an
>>        answer in minutes.
>>
>>        [0] http://code.activestate.com/recipes/langs/python/
>>
>> During a long time I was said that Pythonist were
>> among the smartest developer, because simply they had
>> the guts to learn a language that was not learned in
>> school.
> I never heard that one :)
>
>> Python *was* easy to learn.
>>
>>    => Scheme is easy to learn if you stick in the "garden"
>>       (garden must be defined but it's basically what I
>>       introduce in my tutorial [1]) Which is totally the
>>       same for the Python of today (read Python 3.5+).
> Or look at py2guile: http://draketo.de/py2guile
>
> Python is no longer easy due to integrating additional syntactic forms,
> each of which makes a specific kind of code much nicer but complicates
> learning.
>
> We have something similar in foof-loop:
>
> (define (count-matching-items list predicate)
>    (loop ((for item (in-list list))
>           (with count 0
>                 (if (predicate item)
>                     (+ count 1)
>                     count)))
>          => count))
> ;       ^^ here
>
> This looks really simple in this place, but it introduces special syntax
> which will be unintelligible for someone who does did not explicitly
> learn foof-loop.
>
> And guildhall is filled with special forms I had to understand just to
> be able to do small changes.
>
> This is a social problem. For a deeper discussion why it is a problem,
> see http://winestockwebdesign.com/Essays/Lisp_Curse.html
> "Lisp is so powerful that problems which are technical issues in other
> programming languages are social issues in Lisp."

I didn't know that one

>
> We can counter this with easy tutorials and with writing something like
> canonical Scheme. But for this, we need to define a canonical Scheme
> which is hits the performance and readability sweet-spot for
> Guile. Canonical code must be close to the fastest code.
>
> Practically put: We need Andy Wingo to nitpick the tutorial about things
> which will cause overheads the compiler cannot fix easily — including
> expensive use of macros.

Right now, I don't have performance issues. Except maybe with the crawler code.

I improved stream srfi-41 performance, if you want the code.

>
>>       [1] http://hyperdev.fr
>>
>> PHP was successful back in the days because it was easy
>> to install,
> Or because it was the only thing which existed. Didn’t it make it easy
> for the hoster to sandbox it — both in terms of access and in terms of
> resource requirements?
>
>>      => A scheme web application is much easier to run!
> Sadly not on my cheap webhoster which provides unlimited bandwidth for
> a fixed cost.
Ok, I have a vps, I simply spawn the guile application in a screen and
point nginx to it. It's like configuring nginx for gunicorn. nginx forward
the traffic from :80 to :8080 or whatever based on some rules. It's like
a router for the ports being smart about http protocol.

>>      => While you might not be interested to build something
>>         like prestashop. You very much be interested to build
>>         something like Drupal or Wordpress to allow mom and
>>         dad and others create a personnal experience on the
>>         web.
>>
>> Another thing, Python has GNU Media Goblin
>>
>>      => We must do better!
> Or, maybe collaborate with it?

There is the protocol Chris is working at w3c which is interesting.
I forgot the name :p but it looked cool. At some point, I'd like to talk
some kind of federation protocol. The issue is that they require some
identification, so that people can name you unlike 4chan. Yes my current
project is like a text 4chan with a search engine.

>   I’m sure that there’s lots of stuff which
> could be shared, because it is not language-specific (more exactly: it’s
> HTML, CSS, Javascript and images).

I love their website and logo!

>
>> What block you from contributing to the wide ecosystem of GNU Guile?
> - publishing a program for non-Linux platforms (well, Windows, OSX and
>    mobile phones) so most of my friends can use it.
>
> - an alternative to `python setup.py register sdist upload`
>
> Best wishes,
> Arne
>

Re: How to make GNU Guile more successful

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 3544 bytes --]


Amirouche <amirouche@hypermove.net> writes:

> Le 13/02/2017 à 12:06, Arne Babenhauserheide a écrit :
> My peers have this habit during system programming looking at man
> pages and other stuff. I don't know much command line-fu. My experience,
> about programming:
>
> - looking up the API referennce (procedure index)
> - looking up ansers in stack overflow

I used to do that. But Emacs C-h i C-s <search phrase> is becoming more
and more efficient for me (since I know more and more keywords).

I still google stuff, but less than in Python. It feels like the info
documentation is actually better.

> In the case of Guile programming I also look general scheme solution like
> in racket. Last time I checked they had a Racket->JS translator, but not
> much graphdb stuff.
>
> The ability to generate multiple format is what makes TEXINFO powerful.

Yes.

>> It needs to be easy to not just package some code, but also to share
>> that package without hitting a bottleneck.
>
> I got stuck with guile-git; because of new way of bindings C. I use
> guile-bytestructures.

It would be great if you could simply run "make binary" and get binaries
for multiple platforms which you can provide for different people.

But that’s just what blocks me the most.

>> A really simple way to share a project would be nice here. Nowadays many
>> people just push something with a README to github, but github is
>> unfree.
>
> I try to move to framagit. but I still use github sometimes.

I did not hear from that until today. I prefer Mercurial with my
self-written site extension[1] for static http repositories — at least
as a backup.

[1]: https://www.mercurial-scm.org/wiki/SiteExtension

>> Practically put: We need Andy Wingo to nitpick the tutorial about things
>> which will cause overheads the compiler cannot fix easily — including
>> expensive use of macros.
>
> Right now, I don't have performance issues. Except maybe with the crawler code.
>
> I improved stream srfi-41 performance, if you want the code.

Sounds good. Can you get that into Guile? Having the default
implementation more efficient would help me the most (in case I need
streams at some point).

>>>      => A scheme web application is much easier to run!
>> Sadly not on my cheap webhoster which provides unlimited bandwidth for
>> a fixed cost.
> Ok, I have a vps, I simply spawn the guile application in a screen and
> point nginx to it. It's like configuring nginx for gunicorn. nginx forward
> the traffic from :80 to :8080 or whatever based on some rules. It's like
> a router for the ports being smart about http protocol.

That’s like a remote server with full access. What I have is FTP
access. But in exchange my sites can cope with serving several hundreds
of GiB of data per month.

> There is the protocol Chris is working at w3c which is interesting.
> I forgot the name :p but it looked cool. At some point, I'd like to talk
> some kind of federation protocol. The issue is that they require some
> identification, so that people can name you unlike 4chan. Yes my current
> project is like a text 4chan with a search engine.

In GNU social you can simply use name@instance.tld

But this is again standard proliferation. I already moaned when
identi.ca turned away from status.net. The new pump.io interface never
worked well for me :(

> I love their website and logo!

:)

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

Re: How to make GNU Guile more successful

[Panicz Maciej Godek]

2017-02-13 12:06 GMT+01:00 Arne Babenhauserheide <arne_bab@web.de>:

> Hi Amirouche,
>
> Thank you for your nice writeup!
>
> Amirouche writes:
>
> > I don't know why Racket is successful probably because
> > it has a very good documentation and also a documentation
> > generator written in scheme.
> >
> >   => This is a long debate and core principle of GNU project
> >      so I don't want to enter that debate
>
> But I want to add something: I’m getting the hang of just checking the
> info-page to look something up. And I realize that with well-written
> info-pages this is faster than googling for stack overflow.
>

Once I was strucked by this:
http://www.wilfred.me.uk/blog/2016/07/30/example-driven-development/

There's also a similar project by William Byrd that could make the
difference
https://github.com/webyrd/Barliman

I think that Python is popualr because you don't need Emacs to work with it.
And that the milestone to the success of Racket, apart from intensive
propaganda, is Dr Racket.
(also, an important factor is that you can install both Python and Racket
on Windows, Linux
and OSX, and it Just Works)

The problem with Emacs is that it is that, although you probably can do
everything imaginable with it,
it is difficult to approach. The problem with the problem wth Emacs is that
once you get it, you usually
 have better ideas in mind than making Emacs more approachable.

There's also this problem with Scheme that it is a very diverse community
with plethora
of implementations. And while I use Guile, I don't mind using Chicken or
Chez or Gambit
for various purposes. Or even Stalin sometimes. Maybe, instead of getting a
better compiler,
we should focus on integrating existing compilers?

> And it's easy to packages.
> >
> >   => Another pain topic. In reality it's very easy to package
> >      all sort of guile programs for guix. But guix itself needs
> >      to be more stable.
>
> I wish I had guildhall ready. Got hit by
> time-eaten-by-other-project-because-guildhall-was-priority-three.
>
> It needs to be easy to not just package some code, but also to share
> that package without hitting a bottleneck.
>
>
What happened to initiatives like Scheme NOW, that tried to integrate
various groups of developers.
Or why can't we just "steal" eggs from Chicken? (I mean it. They already
have that problem solved)


> > Python is successful because it has a massive number
> > of libraries.
>

> This is a social problem. For a deeper discussion why it is a problem,
> see http://winestockwebdesign.com/Essays/Lisp_Curse.html
> "Lisp is so powerful that problems which are technical issues in other
> programming languages are social issues in Lisp."
>
> We can counter this with easy tutorials and with writing something like
> canonical Scheme. But for this, we need to define a canonical Scheme
> which is hits the performance and readability sweet-spot for
> Guile. Canonical code must be close to the fastest code.
>
Practically put: We need Andy Wingo to nitpick the tutorial about things
> which will cause overheads the compiler cannot fix easily — including
> expensive use of macros.
>
>
I definitely oppose. If Chez has something solved better, why not use Chez?
The ultimate goal is not to optimize programs, but programmers.
But I agree on the point of the canonical code. I think that the "zen of
python"
is one of important factors of making it popular. I think it could be the
same
with Scheme, but here the problem is that people have various ideas
with regard to how the "ideal" program should look like.

For example, I have a very strong opinion with regard to the use of pattern
matching
and restructuring -- I try to use destructuring almost everywhere (from
what I've seen,
Clojure programmers are alike with this regard). However, I am aware that
there are
people who don't like it that much, like Taylan or Barry Margolin.

Also, I think that the astounding success of R provides hints with regard
what is needed
for a programming language to gain popularity. R has a large repository of
packages
that fills a certain niche. When I advertised my Pamphlet on Hacker News,
someone
responded critically: "are there out of the box libraries to estimate a
zero inflated negative
binomial regression model in guile". Of course, if I knew what a
zero-inflated
negative binomial regression model, I could deliver an implementation by
just explaining
the notions used in that phrase. (But then again, I try to write functional
programs whenever
possible, often consciously sacrificing time complexity of my solutions,
and I'm aware
that not everybody will like it.)

Thanks,
Panicz

Re: How to make GNU Guile more successful

[Amirouche]

Le 13/02/2017 à 21:28, Panicz Maciej Godek a écrit :
>
>
> 2017-02-13 12:06 GMT+01:00 Arne Babenhauserheide <arne_bab@web.de 
> <mailto:arne_bab@web.de>>:
>
>     Hi Amirouche,
>
>     Thank you for your nice writeup!
>
>     Amirouche writes:
>
>     > I don't know why Racket is successful probably because
>     > it has a very good documentation and also a documentation
>     > generator written in scheme.
>     >
>     >   => This is a long debate and core principle of GNU project
>     >      so I don't want to enter that debate
>
>     But I want to add something: I’m getting the hang of just checking the
>     info-page to look something up. And I realize that with well-written
>     info-pages this is faster than googling for stack overflow.
>
>
> Once I was strucked by this:
> http://www.wilfred.me.uk/blog/2016/07/30/example-driven-development/
>
> There's also a similar project by William Byrd that could make the 
> difference
> https://github.com/webyrd/Barliman
This is awesome!

Re: How to make GNU Guile more successful

[Marko Rauhamaa]

Panicz Maciej Godek <godek.maciek@gmail.com>:

> I think that Python is popualr because you don't need Emacs to work
> with it.

You do need a special editor to work with Python, and emacs is perfect
for it.

Python is popular because it is a very traditional, dynamic, high-level
programming language. It is conceptually very similar to Lisp
(especially coupled with JSON). Definitely a gateway drug to Lisp.

> The problem with Emacs is that it is that, although you probably can
> do everything imaginable with it, it is difficult to approach.

I never noticed that problem with emacs. Emacs is really the only
reasonable way to type text. For example, when I enter text of any
length to a web app, I first type it in emacs and then copy it to the
web form.

I have typed this message in emacs.

> There's also this problem with Scheme that it is a very diverse
> community with plethora of implementations.

Scheme might have a chance as people's tastes evolve. Python is an
example of that evolution.

In my opinion one of the worst problems with Scheme is the Schemers:
Scheme lovers are often far too enthusiastic with defining new, esoteric
syntax instead of solving practical problems.

Then, there's GOOPS, which in my opinion is simply an unnatural way to
go about object-oriented programming. It does violence both to ordinary
OO way of thinking and classic Lisp idioms.

Continuations and multiple values are obstacles instead of enablers.

BTW, Python seems to be suffering from quite many self-inflicted wounds:
Python2/Python3 incompatibility, asyncio, type annotation, decorators,
dunder jungle, meta-object programming, Unicode illusion. (Guile has
fallen into that last trap as well, unfortunately.) In its eagerness to
please everyone, Python might well be jumping the shark.

There's one thing Python has over Guile, still: Python's system call
support is outstanding. For example, I managed to implement a full-blown
shell using Python. That was the first time I ever ran into
terminal-related system calls, and Python had them all.


Marko

Re: How to make GNU Guile more successful

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 2940 bytes --]


Marko Rauhamaa <marko@pacujo.net> writes:
> I have typed this message in emacs.

Same for me, but getting people to use Emacs is harder. It might not
*be* that complicated, but it *feels* different.

> In my opinion one of the worst problems with Scheme is the Schemers:
> Scheme lovers are often far too enthusiastic with defining new, esoteric
> syntax instead of solving practical problems.
>
> Then, there's GOOPS, which in my opinion is simply an unnatural way to
> go about object-oriented programming. It does violence both to ordinary
> OO way of thinking and classic Lisp idioms.

GOOPS works pretty well for me where I use it (for dispatch by
type). Could you clarify your criticism: Do you think it is bad or is it
just different?

> Continuations and multiple values are obstacles instead of enablers.

I think multiple values are nice. But they are not well-integrated (I
have to import something to use them). Why do you think them enablers?

Why do you think that continuations are an obstacle (do you mean general
continuations or do you mean prompts)?

> BTW, Python seems to be suffering from quite many self-inflicted wounds:
> Python2/Python3 incompatibility,

Py2 vs. Py3 really hurts, yes.

It still really hurts and will likely hurt for the next 10 years. I
think that should be a lesson for all other language-implementors: Once
you start to become popular, you cannot do any backwards-incompatible
changes without endangering your position. Once you start to get
traction, you have to stick to your horse.

Many other wounds I do not see as much:

> asyncio, type annotation,

type annotations should provide nice ways to optimize for pypy and other
more optimizing Python implementations.

> decorators,

Decorators are really cool to use:

@with_progress
def something():
    time.sleep(10)

That’s almost like

(with-progress
  (define (something)
          (sleep 10)))

But decorators are actually definition time, so they bring the tower of
meta-levels to Python …

> dunder jungle,

What’s that?

> meta-object programming,

That’s needed to make some things elegant. "Different 20%"-rule again.

> Unicode illusion. (Guile has fallen into that last trap as well,
> unfortunately.)

I’m using Python and Guile for physics and math (to some degree), and
having good Unicode-support is great for that. What’s your practical
criticism here?

> In its eagerness to please everyone, Python might well
> be jumping the shark.
>
> There's one thing Python has over Guile, still: Python's system call
> support is outstanding. For example, I managed to implement a full-blown
> shell using Python. That was the first time I ever ran into
> terminal-related system calls, and Python had them all.

Which ones are missing in Guile?

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

Re: How to make GNU Guile more successful

[David Kastrup]

Arne Babenhauserheide <arne_bab@web.de> writes:

> Marko Rauhamaa <marko@pacujo.net> writes:
>> I have typed this message in emacs.
>
> Same for me, but getting people to use Emacs is harder. It might not
> *be* that complicated, but it *feels* different.
>
>> In my opinion one of the worst problems with Scheme is the Schemers:
>> Scheme lovers are often far too enthusiastic with defining new, esoteric
>> syntax instead of solving practical problems.
>>
>> Then, there's GOOPS, which in my opinion is simply an unnatural way to
>> go about object-oriented programming. It does violence both to ordinary
>> OO way of thinking and classic Lisp idioms.
>
> GOOPS works pretty well for me where I use it (for dispatch by
> type). Could you clarify your criticism: Do you think it is bad or is it
> just different?

My main beef with GOOPS is that it does not help with narrowing down on
a solution but rather with extending the problem space.  It is too
generic to provide guidance and a cohesive framework: if two different
people solve problems using GOOPS, the likelihood that those solutions
can be connected in interface or design better than non-GOOPS solutions
is slim.

-- 
David Kastrup

Re: How to make GNU Guile more successful

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1562 bytes --]


David Kastrup <dak@gnu.org> writes:

> Arne Babenhauserheide <arne_bab@web.de> writes:
>
>> Marko Rauhamaa <marko@pacujo.net> writes:
>>> I have typed this message in emacs.
>>
>> Same for me, but getting people to use Emacs is harder. It might not
>> *be* that complicated, but it *feels* different.
>>
>>> In my opinion one of the worst problems with Scheme is the Schemers:
>>> Scheme lovers are often far too enthusiastic with defining new, esoteric
>>> syntax instead of solving practical problems.
>>>
>>> Then, there's GOOPS, which in my opinion is simply an unnatural way to
>>> go about object-oriented programming. It does violence both to ordinary
>>> OO way of thinking and classic Lisp idioms.
>>
>> GOOPS works pretty well for me where I use it (for dispatch by
>> type). Could you clarify your criticism: Do you think it is bad or is it
>> just different?
>
> My main beef with GOOPS is that it does not help with narrowing down on
> a solution but rather with extending the problem space.  It is too
> generic to provide guidance and a cohesive framework: if two different
> people solve problems using GOOPS, the likelihood that those solutions
> can be connected in interface or design better than non-GOOPS solutions
> is slim.

That’s a good point.

A framework which provides that guidance could be built on top of GOOPS,
though. Or would that carry around too much burden from the generic
approach it’s based on?

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

Re: How to make GNU Guile more successful

[Christopher Allan Webber]

David Kastrup writes:

> Arne Babenhauserheide <arne_bab@web.de> writes:
>
>> Marko Rauhamaa <marko@pacujo.net> writes:
>>> I have typed this message in emacs.
>>
>> Same for me, but getting people to use Emacs is harder. It might not
>> *be* that complicated, but it *feels* different.
>>
>>> In my opinion one of the worst problems with Scheme is the Schemers:
>>> Scheme lovers are often far too enthusiastic with defining new, esoteric
>>> syntax instead of solving practical problems.
>>>
>>> Then, there's GOOPS, which in my opinion is simply an unnatural way to
>>> go about object-oriented programming. It does violence both to ordinary
>>> OO way of thinking and classic Lisp idioms.
>>
>> GOOPS works pretty well for me where I use it (for dispatch by
>> type). Could you clarify your criticism: Do you think it is bad or is it
>> just different?
>
> My main beef with GOOPS is that it does not help with narrowing down on
> a solution but rather with extending the problem space.  It is too
> generic to provide guidance and a cohesive framework: if two different
> people solve problems using GOOPS, the likelihood that those solutions
> can be connected in interface or design better than non-GOOPS solutions
> is slim.

Generic methods help somewhat with this though right?  That's part of
the goal of generic methods even; the portion of SICP where they are
introduced is even about getting two programmers who don't necessarily
get along to get their code to work together.

Re: How to make GNU Guile more successful

[sirgazil]

Amirouche says:
> What block you from contributing to the wide ecosystem of GNU Guile?

I think about Guile (and GuixSD) almost every day, but I don't contribute as much as I'd like mostly because of lack of resources to do so. Sadly, it's hard to find the opportunity to immerse yourself in libre projects.

As for how to make GNU Guile more successful, I'd like to see what I call a "Hello Guile" series of guides targeted at people who just want to start programming. The series would include:

+ Hello Guile: Getting Started (An overview of programming, and how to start with Guile)
+ Hello Guile: Let's develop a console application
+ Hello Guile: Let's develop a desktop application
+ Hello Guile: Let's develop a web application
+ Hello Guile: Let's develop a 2D game
+ Hello Guile: Let's package your software for distribution

Additionally, I'd like the following:

+ More high level libraries [1]
+ Add Guile to the GNOME Developer Center [2][3][4]
+ Package more guile libraries in Guix

Finally, we need to clone a bunch of people to do all this work :)


[1]: For example, I'm following the Systematic Program Design course with
Gregor Kiczales. The course uses DrRacket for interactive
programming and a language called BSL (Beginning Student
Language).

When I started, my plan was to follow the course in Guile instead,
since the course is about the design method, and not the tools, but
I had to stop very early when I had to do some projects involving
graphics and animation.

Personally, I don't feel the need for something like DrRacket,
but it would have been nice to be able to find a Guile "canvas" or a
2D game library to play easily with basic graphics and animations.
[2]: https://developer.gnome.org/
[3]: https://developer.gnome.org/gnome-devel-demos/stable/
[4]: https://developer.gnome.org/references (see Language Bindings section)



---
https://sirgazil.bitbucket.io/

Re: How to make GNU Guile more successful

[Amirouche]

Le 16/02/2017 à 20:18, sirgazil a écrit :
> Amirouche says:
>> What block you from contributing to the wide ecosystem of GNU Guile?
> I think about Guile (and GuixSD) almost every day, but I don't contribute as much as I'd like mostly because of lack of resources to do so. Sadly, it's hard to find the opportunity to immerse yourself in libre projects.
>
> As for how to make GNU Guile more successful, I'd like to see what I call a "Hello Guile" series of guides targeted at people who just want to start programming. The series would include:
>
> + Hello Guile: Getting Started (An overview of programming, and how to start with Guile)
> + Hello Guile: Let's develop a console application
> + Hello Guile: Let's develop a desktop application
> + Hello Guile: Let's develop a web application
> + Hello Guile: Let's develop a 2D game
> + Hello Guile: Let's package your software for distribution
>
> Additionally, I'd like the following:
>
> + More high level libraries [1]
> + Add Guile to the GNOME Developer Center [2][3][4]
> + Package more guile libraries in Guix
>
> Finally, we need to clone a bunch of people to do all this work :)
>
>
> [1]: For example, I'm following the Systematic Program Design course with
> Gregor Kiczales. The course uses DrRacket for interactive
> programming and a language called BSL (Beginning Student
> Language).
>
> When I started, my plan was to follow the course in Guile instead,
> since the course is about the design method, and not the tools, but
> I had to stop very early when I had to do some projects involving
> graphics and animation.
>
> Personally, I don't feel the need for something like DrRacket,
> but it would have been nice to be able to find a Guile "canvas" or a
> 2D game library to play easily with basic graphics and animations.
> [2]: https://developer.gnome.org/
> [3]: https://developer.gnome.org/gnome-devel-demos/stable/
> [4]: https://developer.gnome.org/references (see Language Bindings section)
>
>
>
> ---
> https://sirgazil.bitbucket.io/
I have to +1 all of this post.

Re: How to make GNU Guile more successful

[Marko Rauhamaa]

Arne Babenhauserheide <arne_bab@web.de>:

> Marko Rauhamaa <marko@pacujo.net> writes:
>> Then, there's GOOPS, which in my opinion is simply an unnatural way
>> to go about object-oriented programming. It does violence both to
>> ordinary OO way of thinking and classic Lisp idioms.
>
> GOOPS works pretty well for me where I use it (for dispatch by type).
> Could you clarify your criticism: Do you think it is bad or is it just
> different?

GOOPS starts by defining slots. Slots are an objects internal business
and should not be visible to the outside. Instead, objects should be
black boxes that interact with methods. For example, the port interface
is a nice, classic OO API. You don't know anything about the slots of a
port.

Furthermore, I think an extensive type system is un-Lisp-like. OO is not
about things belonging to a type but objects interacting through
methods. In fact, the concept of a "class" could be done away with.

>> Continuations and multiple values are obstacles instead of enablers.
>
> I think multiple values are nice. But they are not well-integrated (I
> have to import something to use them). Why do you think them enablers?

Yes, the multiple values API is a problem in and of itself, but more
generally, what does:

    (values a b c)

give you on top of, say:

    (list a b c)

> Why do you think that continuations are an obstacle (do you mean
> general continuations or do you mean prompts)?

Continuations prevent you from implementing Python's try/finally and
emacs' (unwind-protect).

On the other hand, I don't think any application developer would come to
a point of thinking, "Hey, I know, I'll use a continuation!"

>> asyncio, type annotation,
>
> type annotations should provide nice ways to optimize for pypy and
> other more optimizing Python implementations.

You can't have it bothways. A dynamic language liberates you from
shackles and boilerplate even if it destroys your performance. The
moment you bring in the boilerplate, you are back to Java.

> Decorators are really cool to use:

Have yet to find a need for one.

>> dunder jungle,
>
> What’s that?

<URL: https://docs.python.org/3/genindex-_.html>

>> meta-object programming,
>
> That’s needed to make some things elegant. "Different 20%"-rule again.

Again, I haven't yet found a need to deviate from the regular object
semantics.

>> Unicode illusion. (Guile has fallen into that last trap as well,
>> unfortunately.)
>
> I’m using Python and Guile for physics and math (to some degree), and
> having good Unicode-support is great for that. What’s your practical
> criticism here?

In Linux, all pathnames, files, sockets, environment variables and
command-line arguments are, or deal with, bytes. The bytes are often
encoded with UTF-8, but a programming language cannot assume that, even
if LOCALE promises otherwise.

It would be better to leave Unicode out of Guile's system interface and
have the application encode and decode explicitly where needed.

>> There's one thing Python has over Guile, still: Python's system call
>> support is outstanding. For example, I managed to implement a
>> full-blown shell using Python. That was the first time I ever ran
>> into terminal-related system calls, and Python had them all.
>
> Which ones are missing in Guile?

For example tcgetattr(3). (Ok, not a system call.)

Missing epoll(2) is inconvenient. Not being able to transfer an open
file descriptor via sendmsg(2) is really bad.


Marko

Re: How to make GNU Guile more successful

[Linas Vepstas]

Hey,

On Mon, Feb 13, 2017 at 11:59 PM, Marko Rauhamaa <marko@pacujo.net> wrote:
>
> Arne Babenhauserheide <arne_bab@web.de>:
>
> > Marko Rauhamaa <marko@pacujo.net> writes:
> >> Then, there's GOOPS, which in my opinion is simply an unnatural way
> >> to go about object-oriented programming. It does violence both to
> >> ordinary OO way of thinking and classic Lisp idioms.
> >
> > GOOPS works pretty well for me where I use it (for dispatch by type).
> > Could you clarify your criticism: Do you think it is bad or is it just
> > different?
>
> GOOPS starts by defining slots. Slots are an objects internal business
> and should not be visible to the outside. Instead, objects should be
> black boxes that interact with methods. For example, the port interface
> is a nice, classic OO API. You don't know anything about the slots of a
> port.


Yes, but ... maybe goops should be renamed guijs ? Javascript has this
... wonderful ... ability to just attach new data and new methods to any
"object" at runtime.   Now, I've never used goops in any serious way, so
maybe I'm quite off the mark, but it seems to me that maybe this is what
slots are trying to accomplish.

That is, when you say "classic OO API", it seems you really have C++ in
the back of your mind.  But the early days of OO had all sorts of weird ways
of saying things and thinking about things (e.g. "sending messages to objects")
that was a bigger, more inclusive tent.  Perhaps GOOPS is just OO in this
older, and now deprecated sense.

My naive impression is that some javascrtipt-ish extension to guile would
be very cool.  Perhaps goops could be pushed further in that direction.


> >> Continuations and multiple values are obstacles instead of enablers.
> >
> > I think multiple values are nice. But they are not well-integrated (I
> > have to import something to use them). Why do you think them enablers?
>
> Yes, the multiple values API is a problem in and of itself, but more
> generally, what does:
>
>     (values a b c)
>
> give you on top of, say:
>
>     (list a b c)

+1 every time I think I need multiple values, I try it, and then back
away from it.
Its a cool-sounding idea that is a consistent obstacle.


> > Why do you think that continuations are an obstacle (do you mean
> > general continuations or do you mean prompts)?
>
> Continuations prevent you from implementing Python's try/finally and
> emacs' (unwind-protect).
>
> On the other hand, I don't think any application developer would come to
> a point of thinking, "Hey, I know, I'll use a continuation!"

Continuations are stunningly-badly explained, almost everywhere.  More
or less every example I've ever seen is pointlessly obtuse and opaque,
seemingly written to demonstrate how smart the author is, and how stupid
the reader is.

There is one place where an application programmer could say "Hey, I know,
I'll use a continuation!" -- whenever they need a single "global"  "OO object"
with a "global variable" in it. I use them for implementing statistics
printing and
profiling. The idea is simple, the solution is elegant:

-- I want to have a single "global" counter which I will increment by one from
now until forever.
-- I want to have a function to increment it, and I want to call that function
at any time, from anywhere at all.
-- every hundreth time that it is called, it will print something like
"xyz operations per second"

Turns out that continuations offer a marvelously simple, easy-to-understand
easy-to-use solution to this problem.   You'd think that there would be some
tutorial or some common "here's the cookbook way of implementing this"
but I sure haven't seen one.  But once you get the hang of the above example,
one can see how it could be fleshed out into a full-fledged OO-like
message-passing system, complete with frames and slots.

Continuations do not need to be a swear word; its just that they're badly
explained.  They are a stumbling block, but that's in part because there's
no analogous concept in proceedural languages; so most programmers
never have to learn it.

> > Decorators are really cool to use:
>
> Have yet to find a need for one.

Sounds like gain-saying.  Just because you don't need it, doesn't mean its
not useful.  "A typical user uses 20% of an application. The problem is,
different users use a different 20%".  "recall there were dozens of wikis before
wikipedia, dozens of search engines before google, dozens of social media
sites before facebook"

I conclude that the difference between a successful(popular) and a failed
(unpopular) tech is often very subtle.  For all I know, decorators "done right"
in guile could skyrocket it to the top of the popularity ranking.  Or not. Just
be careful in dismissing some of these ideas.

> >> Unicode illusion. (Guile has fallen into that last trap as well,
> >> unfortunately.)
> >
> > I’m using Python and Guile for physics and math (to some degree), and
> > having good Unicode-support is great for that. What’s your practical
> > criticism here?

Unicode support in guile remains rocky and buggy, as I once again
rediscovered recently when pumping Chinese through it.

> It would be better to leave Unicode out of Guile's system interface and
> have the application encode and decode explicitly where needed.

Huh?  That would suck.  Unless I misunderstand, in which case it would
be great.   Unicode is just a bunch of bytes that are null-terminated.  Guile
does very stupidly try to  convert unicode strings into arrays of wide chars
which is absolutely idiotic, and a total bottleneck for what I do.
Its a useless,
pointless processing step.  I keep planning on submitting a patch to fix this
but never have time.

I regularly have strings that are 10 Mbytes or more, that contain mixtures of
scheme code and chinese characters (in utf8), that I have to send to guile
for eval. Anything that would make that faster would be awesome.

--linas

Re: How to make GNU Guile more successful

[Marko Rauhamaa]

Linas Vepstas <linasvepstas@gmail.com>:

> Unicode is just a bunch of bytes that are null-terminated.

That is not what Unicode is. Maybe you are thinking of UTF-8, an
ingenious way to map a Unicode sequence onto a byte sequence. Trouble
is, there are byte sequences that are illegal UTF-8. While they are
illegal for Unicode use, they are perfectly possible and valid as far as
the (Linux) operating system is concerned.


Marko

Re: How to make GNU Guile more successful

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 6436 bytes --]


Marko Rauhamaa <marko@pacujo.net> writes:

> Arne Babenhauserheide <arne_bab@web.de>:
>
>> Marko Rauhamaa <marko@pacujo.net> writes:
>>> Then, there's GOOPS, which in my opinion is simply an unnatural way
>>> to go about object-oriented programming. It does violence both to
>>> ordinary OO way of thinking and classic Lisp idioms.
>>
>> GOOPS works pretty well for me where I use it (for dispatch by type).
>> Could you clarify your criticism: Do you think it is bad or is it just
>> different?
>
> GOOPS starts by defining slots. Slots are an objects internal business
> and should not be visible to the outside. Instead, objects should be
> black boxes that interact with methods. For example, the port interface
> is a nice, classic OO API. You don't know anything about the slots of a
> port.
>
> Furthermore, I think an extensive type system is un-Lisp-like. OO is not
> about things belonging to a type but objects interacting through
> methods. In fact, the concept of a "class" could be done away with.
>
>>> Continuations and multiple values are obstacles instead of enablers.
>>
>> I think multiple values are nice. But they are not well-integrated (I
>> have to import something to use them). Why do you think them enablers?
>
> Yes, the multiple values API is a problem in and of itself, but more
> generally, what does:
>
>     (values a b c)
>
> give you on top of, say:
>
>     (list a b c)

Practically it gives me (let-values (((a b c) (values 1 2 3))) a)

I could get the same by using match on a list, but then I have to use
match on a list instead of just assigning the values. let-values could
be designed to work with lists, though.

So conceptually I assume that the real difference is in implementation:
avoiding the construction of a list.

>> Why do you think that continuations are an obstacle (do you mean
>> general continuations or do you mean prompts)?
>
> Continuations prevent you from implementing Python's try/finally and
> emacs' (unwind-protect).

If I understand it correctly, this is only true, if there are resources
in use which do not get created in the try. I hit that problem when
implementing 'with':
https://bitbucket.org/ArneBab/wisp/src/3a654cfe6632af4b0002ce98c753c07c400aac55/examples/with.w#with.w-8

with (open-file "with.w" "r") as port
      format #t "~a\n" (read port)

^ working Guile code which ensures that the port is closed after it is
  no longer used. But it can break if a continuation is used. To fix
  that, I’d have to add guards which save the state of the port when
  code creates a continuation and reconstructs that state when the
  continuation is used.

> On the other hand, I don't think any application developer would come to
> a point of thinking, "Hey, I know, I'll use a continuation!"

Error-handling is simply a specific case of that. In Python people use
exceptions for flow control, which can be efficient when the
non-exception case is much, much more common than the exception case
(because in the first case the overhead of that is almost zero).

>>> asyncio, type annotation,
>>
>> type annotations should provide nice ways to optimize for pypy and
>> other more optimizing Python implementations.
>
> You can't have it bothways. A dynamic language liberates you from
> shackles and boilerplate even if it destroys your performance. The
> moment you bring in the boilerplate, you are back to Java.

It’s optional boilerplate which you only add where you really need it.

>> Decorators are really cool to use:
>
> Have yet to find a need for one.

I used them quite a bit. The most efficient one was a caching decorator:

@cached
def fun(a):
    # something expensive
    return a

In Python 3.2 you can use the standard lru-cache for that:
https://docs.python.org/3/library/functools.html#functools.lru_cache

>>> dunder jungle,
>>
>> What’s that?
>
> <URL: https://docs.python.org/3/genindex-_.html>

Ah, yes :)

I think these are a good way to show that I as developer am venturing
into regions I should only enter when I am really, really sure I want
to, because they can cause problems I may not see right away.

Sadly this rule is broken with if __name__ == "__main__": ...
and for def __init__(self)

So I think it’s good to have that, but not good that programmers need to
use that during typical programming tasks.

>>> meta-object programming,
>>
>> That’s needed to make some things elegant. "Different 20%"-rule again.
>
> Again, I haven't yet found a need to deviate from the regular object
> semantics.

I saw some, like actually accessing and changing the bytecode at runtime
to add tracing commands.

>>> Unicode illusion. (Guile has fallen into that last trap as well,
>>> unfortunately.)
>>
>> I’m using Python and Guile for physics and math (to some degree), and
>> having good Unicode-support is great for that. What’s your practical
>> criticism here?
>
> In Linux, all pathnames, files, sockets, environment variables and
> command-line arguments are, or deal with, bytes. The bytes are often
> encoded with UTF-8, but a programming language cannot assume that, even
> if LOCALE promises otherwise.
>
> It would be better to leave Unicode out of Guile's system interface and
> have the application encode and decode explicitly where needed.

I’m not sure about that. Python3 requires explicit encode/decode, and
it’s a lot of hassle to port from Py2 to Py3 if your application does
lots of socket-operations. I have firsthand experience with that, and
the debugging is gruesome.

But it might have been much better if the program had been created with
Python3 from the start.

>>> There's one thing Python has over Guile, still: Python's system call
>>> support is outstanding. For example, I managed to implement a
>>> full-blown shell using Python. That was the first time I ever ran
>>> into terminal-related system calls, and Python had them all.
>>
>> Which ones are missing in Guile?
>
> For example tcgetattr(3). (Ok, not a system call.)
>
> Missing epoll(2) is inconvenient. Not being able to transfer an open
> file descriptor via sendmsg(2) is really bad.

Do you know whether there is a specific reason for that or whether these
could be added?

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

Re: How to make GNU Guile more successful

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 8011 bytes --]


Panicz Maciej Godek <godek.maciek@gmail.com> writes:

> 2017-02-13 12:06 GMT+01:00 Arne Babenhauserheide <arne_bab@web.de>:
> There's also this problem with Scheme that it is a very diverse
> community with plethora of implementations. And while I use Guile, I
> don't mind using Chicken or Chez or Gambit for various purposes. Or
> even Stalin sometimes. Maybe, instead of getting a better compiler, we
> should focus on integrating existing compilers?

You’ll have to ask Andy Wingo about all the hidden complexity in just
wiring in a compilier. I assume it’s pretty big.

I think the r7rs folks are working in that regard, to make it possible
to write code which you can simply run through another Scheme
implementation if you need those strengths.

I started to use (import (...)) to get closer to that: Getting used to
the portable stuff (also it’s shorter than (use-modules (...)). But not
everything is portable, and performance critical stuff most definitely
isn’t.

And for the stuff I do, most things get performance critical at some
point. Even looking for null-bytes in a file becomes a bottleneck if you
have a few hundred TiB to check.

Though I mainly need to write Python at work (Scheme isn’t widespread
enough for the folks there — Python is everywhere in science
nowadays). It’s still efficient to do that and I enjoy it, but not as
much as writing Scheme.

>> And it's easy to packages.
>> >
>> >   => Another pain topic. In reality it's very easy to package
>> >      all sort of guile programs for guix. But guix itself needs
>> >      to be more stable.
>>
>> I wish I had guildhall ready. Got hit by
>> time-eaten-by-other-project-because-guildhall-was-priority-three.
>>
>> It needs to be easy to not just package some code, but also to share
>> that package without hitting a bottleneck.
>>
> What happened to initiatives like Scheme NOW, that tried to integrate
> various groups of developers.

I did not hear from that.

> Or why can't we just "steal" eggs from Chicken? (I mean it. They already
> have that problem solved)

Do you mean, integrating eggs into Guile so we could just use them?
Conceptually what would be needed for that is a metadata field: "Works
with Guile". And practically someone to write the integration.

We could also have guildhall working, and add a tool for converting
chicken eggs into guildhall packages (and ideally vice versa). Then
people could do the required Guile integration steps once and provide
the packages to all who might want them.

>> > Python is successful because it has a massive number
>> > of libraries.
>>
>
>> This is a social problem. For a deeper discussion why it is a problem,
>> see http://winestockwebdesign.com/Essays/Lisp_Curse.html
>> "Lisp is so powerful that problems which are technical issues in other
>> programming languages are social issues in Lisp."
>>
>> We can counter this with easy tutorials and with writing something like
>> canonical Scheme. But for this, we need to define a canonical Scheme
>> which is hits the performance and readability sweet-spot for
>> Guile. Canonical code must be close to the fastest code.
>>
> Practically put: We need Andy Wingo to nitpick the tutorial about things
>> which will cause overheads the compiler cannot fix easily — including
>> expensive use of macros.

> I definitely oppose. If Chez has something solved better, why not use
> Chez?

It’s not "why not use Chez", but rather "what should I teach new
people?"

They are already learning a new language. When I now go and point them
towards many different implementations which differ in details of stuff
I teach them, I’m throwing more complexity at them. More things to
understand before even starting to write working code.

Maybe it would already help to mark the code which will work the same in
all (r7rs) Schemes. Is there stuff which is well optimized in all
Schemes? Who knows that? The r7rs benchmarks look like this is very much
not the case: http://ecraven.github.io/r7rs-benchmarks/benchmark.html

(Or rather, the benchmarks would ask: "why should I *not* use Chez or
stalin for everything I do?" Because those are the fastest for most
tasks.)

But that might mean to write less elegant or efficient code to keep it
cross-implementation. Should I rather teach new people to solve a
problem as well as possible, or teach them to solve a problem in a
portable way?

In my case, I decided to use Guile by checking Scheme
implementations. There were three with roughly equal activity and
features. I chose Guile, because it’s a GNU project and it has a long
history of surviving changing maintainers, so my skills are most likely
to stay useful.

> The ultimate goal is not to optimize programs, but programmers.

I’m not sure that I want to optimize them, I want to teach them tools to
be more efficient and enjoy their work more (and teach myself the same).

> But I agree on the point of the canonical code. I think that the "zen
> of python" is one of important factors of making it popular. I think
> it could be the same with Scheme, but here the problem is that people
> have various ideas with regard to how the "ideal" program should look
> like.

I think one important point for Scheme would be to gather some consensus
points. The Zen of Python is part of what made that one community
strong. There is no reason why there should not be a Zen of Scheme,
along with implementation-specific Koans which extend it.

Do you have ideas for principles which could be shared by most Schemers?

> For example, I have a very strong opinion with regard to the use of
> pattern matching and restructuring -- I try to use destructuring
> almost everywhere (from what I've seen, Clojure programmers are alike
> with this regard). However, I am aware that there are people who don't
> like it that much, like Taylan or Barry Margolin.
>
> Also, I think that the astounding success of R provides hints with
> regard what is needed for a programming language to gain popularity. R
> has a large repository of packages that fills a certain niche. When I
> advertised my Pamphlet on Hacker News, someone responded critically:
> "are there out of the box libraries to estimate a zero inflated
> negative binomial regression model in guile". Of course, if I knew
> what a zero-inflated negative binomial regression model, I could
> deliver an implementation by just explaining the notions used in that
> phrase.

I think there’s a quote which applies here: "Most people only use 20% of
the capabilities of any program, but they do not use the same 20%."
— Joel Spolksy, if I remember correctly.

And to make people really dig into it, a programming language must work
well for all tasks they might encounter. That’s why a comprehensive
standard library is important: It gives the assurance that the skills
you learn will stay useful wherever you go.

There’s a nice quote from Rust folks: "people started to feel safe when
we told them about unsafe" — someone on GNU social.
This essentially says: they then saw that Rust could work in the
uncomfortable way some of their tasks might require. That they could
take it into harsher territory, if needed.

> (But then again, I try to write functional programs whenever
> possible, often consciously sacrificing time complexity of my
> solutions, and I'm aware that not everybody will like it.)

I look for ease of understanding, with functional programming techniques
being a useful tool to reduce complexity and to simplify refactoring.

But then, this is a point where we actually move into a similar
direction out of different motives. I think the only problem here is the
danger of assuming contradiction where there does not need to be
any. Just different nuances of Zen :)

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

Re: How to make GNU Guile more successful

[Panicz Maciej Godek]

2017-02-13 23:54 GMT+01:00 Arne Babenhauserheide <arne_bab@web.de>:

>
> Panicz Maciej Godek <godek.maciek@gmail.com> writes:
>
> > 2017-02-13 12:06 GMT+01:00 Arne Babenhauserheide <arne_bab@web.de>:
> > There's also this problem with Scheme that it is a very diverse
> > community with plethora of implementations. And while I use Guile, I
> > don't mind using Chicken or Chez or Gambit for various purposes. Or
> > even Stalin sometimes. Maybe, instead of getting a better compiler, we
> > should focus on integrating existing compilers?
>
> You’ll have to ask Andy Wingo about all the hidden complexity in just
> wiring in a compilier. I assume it’s pretty big.
>
> I think the r7rs folks are working in that regard, to make it possible
> to write code which you can simply run through another Scheme
> implementation if you need those strengths.
>
> I started to use (import (...)) to get closer to that: Getting used to
> the portable stuff (also it’s shorter than (use-modules (...)). But not
> everything is portable, and performance critical stuff most definitely
> isn’t.
>
> And for the stuff I do, most things get performance critical at some
> point. Even looking for null-bytes in a file becomes a bottleneck if you
> have a few hundred TiB to check.
>
> Though I mainly need to write Python at work (Scheme isn’t widespread
> enough for the folks there — Python is everywhere in science
> nowadays). It’s still efficient to do that and I enjoy it, but not as
> much as writing Scheme.
>
> >> And it's easy to packages.
> >> >
> >> >   => Another pain topic. In reality it's very easy to package
> >> >      all sort of guile programs for guix. But guix itself needs
> >> >      to be more stable.
> >>
> >> I wish I had guildhall ready. Got hit by
> >> time-eaten-by-other-project-because-guildhall-was-priority-three.
> >>
> >> It needs to be easy to not just package some code, but also to share
> >> that package without hitting a bottleneck.
> >>
> > What happened to initiatives like Scheme NOW, that tried to integrate
> > various groups of developers.
>
> I did not hear from that.
>
> > Or why can't we just "steal" eggs from Chicken? (I mean it. They already
> > have that problem solved)
>
> Do you mean, integrating eggs into Guile so we could just use them?
> Conceptually what would be needed for that is a metadata field: "Works
> with Guile". And practically someone to write the integration.
>
>
There's surely many ways to approach that issue. The point is that for some
reason, Schemers from various tribes prefer to reinvent the wheel,
rather than use an existing one. (Not that I am any different)
However, I also think that these CPAN-alike solutions are far from optimal:
ideally, programming could be made an experience similar the game
"The Journey" by thatgamecompany (where you travel to your goal and
sometimes
encounter other players heading the same direction), and the repository
itself could look
more like Wikipedia (but first we'd need to start perceiveing programming as
a compact way of representing knowledge)

> Practically put: We need Andy Wingo to nitpick the tutorial about things
> >> which will cause overheads the compiler cannot fix easily — including
> >> expensive use of macros.
>
> > I definitely oppose. If Chez has something solved better, why not use
> > Chez?
>
> It’s not "why not use Chez", but rather "what should I teach new
> people?"
>
> They are already learning a new language. When I now go and point them
> towards many different implementations which differ in details of stuff
> I teach them, I’m throwing more complexity at them. More things to
> understand before even starting to write working code.
>
> I totally agree. Programmers shouldn't be concerned whether they're
using Chez or Guile or Gambit or Stalin or whatever else. Ideally, they
should do well without even knowing that such things exist.
There are Schemes, such as Kawa or Biwa or IronScheme, that are tied
to their environment, but most of them just run on the PC


> Maybe it would already help to mark the code which will work the same in
> all (r7rs) Schemes. Is there stuff which is well optimized in all
> Schemes? Who knows that? The r7rs benchmarks look like this is very much
> not the case: http://ecraven.github.io/r7rs-benchmarks/benchmark.html
>
> (Or rather, the benchmarks would ask: "why should I *not* use Chez or
> stalin for everything I do?" Because those are the fastest for most
> tasks.)
>
> But that might mean to write less elegant or efficient code to keep it
> cross-implementation. Should I rather teach new people to solve a
> problem as well as possible, or teach them to solve a problem in a
> portable way?
>
>
I think that in principle, the programmer should only focus on writing
most elegant code


> In my case, I decided to use Guile by checking Scheme
> implementations. There were three with roughly equal activity and
> features. I chose Guile, because it’s a GNU project and it has a long
> history of surviving changing maintainers, so my skills are most likely
> to stay useful.
>
>
That's interesting. I chose it by trying to add a scripting language to my
game engine written in C++. But I guess that if I was starting now, I'd
write
the whole engine in Chez (which wasn't opensource until last year)


> > The ultimate goal is not to optimize programs, but programmers.
>
> I’m not sure that I want to optimize them, I want to teach them tools to
> be more efficient and enjoy their work more (and teach myself the same).
>
>
Yeah, maybe I used a harsh word, but I meant exactly optimizing
programmers' efficiency (or if it sounds too bad, "giving them tools
to be more efficient")


> I think one important point for Scheme would be to gather some consensus
> points. The Zen of Python is part of what made that one community
> strong. There is no reason why there should not be a Zen of Scheme,
> along with implementation-specific Koans which extend it.
>
> Do you have ideas for principles which could be shared by most Schemers?
>
>
It depends on the most Schemers, not me :)
However, you can have a look at the (grand scheme) glossary that I've been
maintaining for some time:

https://github.com/plande/grand-scheme

of course, the intent is to make the glossary usable for someone else than
me,
and I can boast that I have at least one user distinct from the maintainer.
You can have a look and let me know what you think. (for now it only works
with Guile). In particular, are there any factors that could convince you to
adapt it as a base for your library, or what would you want changed
(I assume you won't stop at the README file).
Anyways, the rules (of thumb) are:


   - functions should be pure
   - language constructs that cause side-effects should be encapsulated
   with macros that limit the extent of those side-effects
   - if possible, use the quasiquote macro, rather than cons, append or list
   - prefer the use of match <http://synthcode.com/scheme/match.scm> to cond,
   if the patterns are simple
   - never use abbreviations
   - use commented prepositions to separate arguments

Also, I recently added infix syntax to Scheme, but (unlike in Racket or
David Wheeler's solutions) in the way it is still a prefix syntax. So for
example,
instead of writing (< a b), one ought to write (is a < b). The rationale is
that
prefix notation is terrible for non-symmetrical binary predicates, because
it
obscures the roles of respective arguments (in addition, (is a < b <= c)
expands
to (and (< a b) (<= b c))). I'm still not sure whether it is a good idea,
but I have
no reasons to think that it is bad either :]
[Curiously enough, I think that prefix notation in arithmetic is just fine:
(+ a b c) reads as "sum of a, b and c"]

> Also, I think that the astounding success of R provides hints with
> > regard what is needed for a programming language to gain popularity. R
> > has a large repository of packages that fills a certain niche. When I
> > advertised my Pamphlet on Hacker News, someone responded critically:
> > "are there out of the box libraries to estimate a zero inflated
> > negative binomial regression model in guile". Of course, if I knew
> > what a zero-inflated negative binomial regression model, I could
> > deliver an implementation by just explaining the notions used in that
> > phrase.
>
> I think there’s a quote which applies here: "Most people only use 20% of
> the capabilities of any program, but they do not use the same 20%."
> — Joel Spolksy, if I remember correctly.
>
> And to make people really dig into it, a programming language must work
> well for all tasks they might encounter. That’s why a comprehensive
> standard library is important: It gives the assurance that the skills
> you learn will stay useful wherever you go.
>
> There’s a nice quote from Rust folks: "people started to feel safe when
> we told them about unsafe" — someone on GNU social.
> This essentially says: they then saw that Rust could work in the
> uncomfortable way some of their tasks might require. That they could
> take it into harsher territory, if needed.
>
> I like it that you and Amirouche look a lot at the sociocultural
perspective
of programming


> > (But then again, I try to write functional programs whenever
> > possible, often consciously sacrificing time complexity of my
> > solutions, and I'm aware that not everybody will like it.)
>
> I look for ease of understanding, with functional programming techniques
> being a useful tool to reduce complexity and to simplify refactoring.
>
> But then, this is a point where we actually move into a similar
> direction out of different motives. I think the only problem here is the
> danger of assuming contradiction where there does not need to be
> any. Just different nuances of Zen :)
>
> Yes, to me the ease of understanding is also crucial. (I'd even go
further and say that undestranding is the very point of programming)

Best regards,
Panicz

Re: How to make GNU Guile more successful

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 8750 bytes --]


Panicz Maciej Godek <godek.maciek@gmail.com> writes:
> There's surely many ways to approach that issue. The point is that for
> some reason, Schemers from various tribes prefer to reinvent the
> wheel, rather than use an existing one. (Not that I am any different)
> However, I also think that these CPAN-alike solutions are far from
> optimal: ideally, programming could be made an experience similar the
> game "The Journey" by thatgamecompany (where you travel to your goal
> and sometimes encounter other players heading the same direction), and
> the repository itself could look more like Wikipedia (but first we'd
> need to start perceiveing programming as a compact way of representing
> knowledge)

That sounds somehow like stackoverflow. We might already be going there :)

>> Practically put: We need Andy Wingo to nitpick the tutorial about things
>> >> which will cause overheads the compiler cannot fix easily — including
>> >> expensive use of macros.
>>
>> > I definitely oppose. If Chez has something solved better, why not use
>> > Chez?
>>
>> It’s not "why not use Chez", but rather "what should I teach new
>> people?"
>>
>> They are already learning a new language. When I now go and point them
>> towards many different implementations which differ in details of stuff
>> I teach them, I’m throwing more complexity at them. More things to
>> understand before even starting to write working code.
>>
> I totally agree. Programmers shouldn't be concerned whether they're
> using Chez or Guile or Gambit or Stalin or whatever else. Ideally, they
> should do well without even knowing that such things exist.
> There are Schemes, such as Kawa or Biwa or IronScheme, that are tied
> to their environment, but most of them just run on the PC

I think this is a point which cannot be reached, because the design of
the Scheme system itself must make certain tradeoffs — especially for
the default environment (what to present a new user). If that is too
large, then users cannot start quickly. If it is too small, then users
cannot solve problems easily. But different problem spaces require
different base tools.

Another points are the basic datastructures: If they are too complex,
then the system cannot be used easily on tiny computers.

Essentially you have to find reasons for decisions like this:
https://docs.python.org/3/faq/design.html#how-are-lists-implemented
and this:
https://wiki.python.org/moin/TimeComplexity

>> Maybe it would already help to mark the code which will work the same in
>> all (r7rs) Schemes. Is there stuff which is well optimized in all
>> Schemes? Who knows that? The r7rs benchmarks look like this is very much
>> not the case: http://ecraven.github.io/r7rs-benchmarks/benchmark.html
>>
>> (Or rather, the benchmarks would ask: "why should I *not* use Chez or
>> stalin for everything I do?" Because those are the fastest for most
>> tasks.)
>>
>> But that might mean to write less elegant or efficient code to keep it
>> cross-implementation. Should I rather teach new people to solve a
>> problem as well as possible, or teach them to solve a problem in a
>> portable way?

> I think that in principle, the programmer should only focus on writing
> most elegant code

I think there are several different kinds of elegance. There’s technical
elegance (combining existing things to build something unexpectedly
powerful with minimal effort), readability (being easy to understand for
newcomers), minimal code (solving a problem with the minimal amount of
code — often close to mathematics), uniformity (using the same basic
structure everywhere), efficiency (solving a problem with minimal
resource-consumption), ...
I’m sure we’ll find more.

Sadly these elegances contradict each other in some points.

>> In my case, I decided to use Guile by checking Scheme
>> implementations. There were three with roughly equal activity and
>> features. I chose Guile, because it’s a GNU project and it has a long
>> history of surviving changing maintainers, so my skills are most likely
>> to stay useful.

> That's interesting. I chose it by trying to add a scripting language
> to my game engine written in C++. But I guess that if I was starting
> now, I'd write the whole engine in Chez (which wasn't opensource until
> last year)

Chez becoming free software is an interesting development.

>> > The ultimate goal is not to optimize programs, but programmers.
>>
>> I’m not sure that I want to optimize them, I want to teach them tools to
>> be more efficient and enjoy their work more (and teach myself the same).
> Yeah, maybe I used a harsh word, but I meant exactly optimizing
> programmers' efficiency (or if it sounds too bad, "giving them tools
> to be more efficient")

Sounds good :)

>> I think one important point for Scheme would be to gather some consensus
>> points. The Zen of Python is part of what made that one community
>> strong. There is no reason why there should not be a Zen of Scheme,
>> along with implementation-specific Koans which extend it.
>>
>> Do you have ideas for principles which could be shared by most Schemers?
>>
>>
> It depends on the most Schemers, not me :)

That’s why its a social problem :)

Finding a set of basic koans which most Schemers agree with requires
talking to and knowing a lot of Schemers.

> However, you can have a look at the (grand scheme) glossary that I've been
> maintaining for some time:
>
> https://github.com/plande/grand-scheme

This looks interesting.

> of course, the intent is to make the glossary usable for someone else than
> me,
> and I can boast that I have at least one user distinct from the maintainer.
> You can have a look and let me know what you think. (for now it only works
> with Guile). In particular, are there any factors that could convince you to
> adapt it as a base for your library, or what would you want changed
> (I assume you won't stop at the README file).

I’d mainly need practical examples. Tell me how I’d solve a problem
with (grand scheme) and how it would be solved without it — showing me
why using it is better.

That means: Documentation.

Also I’d need to know whether some of these methods have performance
problems: Will this scale as well as the method without (grand scheme)?

>    - functions should be pure
>    - language constructs that cause side-effects should be encapsulated
>    with macros that limit the extent of those side-effects
>    - if possible, use the quasiquote macro, rather than cons, append or list
>    - prefer the use of match <http://synthcode.com/scheme/match.scm> to cond,
>    if the patterns are simple
>    - never use abbreviations
>    - use commented prepositions to separate arguments

As guidelines for easy-to-use libraries, that sounds good.

> Also, I recently added infix syntax to Scheme, but (unlike in Racket
> or David Wheeler's solutions) in the way it is still a prefix
> syntax. So for example, instead of writing (< a b), one ought to write
> (is a < b). The rationale is that prefix notation is terrible for
> non-symmetrical binary predicates, because it obscures the roles of
> respective arguments (in addition, (is a < b <= c) expands to (and (<
> a b) (<= b c))). I'm still not sure whether it is a good idea, but I
> have no reasons to think that it is bad either :] [Curiously enough, I
> think that prefix notation in arithmetic is just fine: (+ a b c) reads
> as "sum of a, b and c"]

This is an old idea, but using (is) as prefix looks
interesting. Typically the prefix was something like (nfx), which
doesn’t say anything semantic (other than that what follows uses infix
syntax). It’s like an expanded equal?

(if (is a = 5)
    #t
    #f)

(cond
    ((is q = p)
     q)
    ((is p > 5)
     p)
    (else
     q))

This seems pretty close to natural language. Can I also use (is A 5)
instead of (equal? A 5)?

Though (is? a = 5) might be closer to canonical Scheme.

> I like it that you and Amirouche look a lot at the sociocultural
> perspective of programming

I’m glad to hear that :)

> Yes, to me the ease of understanding is also crucial. (I'd even go
> further and say that undestranding is the very point of programming)

I would not go that far: We’re programming both for the future
maintainers of the code and for the computer. This restricts us in
expressing things, because we always have to keep two targets audiences
in mind (many different systems *and* many different people).

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

Re: How to make GNU Guile more successful

[Amirouche]

Le 14/02/2017 à 22:35, Arne Babenhauserheide a écrit :
> Panicz Maciej Godek <godek.maciek@gmail.com> writes:
>> There's surely many ways to approach that issue. The point is that for
>> some reason, Schemers from various tribes prefer to reinvent the
>> wheel, rather than use an existing one. (Not that I am any different)
>> However, I also think that these CPAN-alike solutions are far from
>> optimal: ideally, programming could be made an experience similar the
>> game "The Journey" by thatgamecompany (where you travel to your goal
>> and sometimes encounter other players heading the same direction), and
>> the repository itself could look more like Wikipedia (but first we'd
>> need to start perceiveing programming as a compact way of representing
>> knowledge)
> That sounds somehow like stackoverflow. We might already be going there :)

That's a very good idea!

>
>>> Practically put: We need Andy Wingo to nitpick the tutorial about things
>>>>> which will cause overheads the compiler cannot fix easily — including
>>>>> expensive use of macros.
>>>> I definitely oppose. If Chez has something solved better, why not use
>>>> Chez?
>>> It’s not "why not use Chez", but rather "what should I teach new
>>> people?"
>>>
>>> They are already learning a new language. When I now go and point them
>>> towards many different implementations which differ in details of stuff
>>> I teach them, I’m throwing more complexity at them. More things to
>>> understand before even starting to write working code.
>>>
>> I totally agree. Programmers shouldn't be concerned whether they're
>> using Chez or Guile or Gambit or Stalin or whatever else. Ideally, they
>> should do well without even knowing that such things exist.
>> There are Schemes, such as Kawa or Biwa or IronScheme, that are tied
>> to their environment, but most of them just run on the PC
> I think this is a point which cannot be reached, because the design of
> the Scheme system itself must make certain tradeoffs — especially for
> the default environment (what to present a new user). If that is too
> large, then users cannot start quickly. If it is too small, then users
> cannot solve problems easily. But different problem spaces require
> different base tools.
>
> Another points are the basic datastructures: If they are too complex,
> then the system cannot be used easily on tiny computers.
>
> Essentially you have to find reasons for decisions like this:
> https://docs.python.org/3/faq/design.html#how-are-lists-implemented
> and this:
> https://wiki.python.org/moin/TimeComplexity
>
>>> Maybe it would already help to mark the code which will work the same in
>>> all (r7rs) Schemes. Is there stuff which is well optimized in all
>>> Schemes? Who knows that? The r7rs benchmarks look like this is very much
>>> not the case: http://ecraven.github.io/r7rs-benchmarks/benchmark.html
>>>
>>> (Or rather, the benchmarks would ask: "why should I *not* use Chez or
>>> stalin for everything I do?" Because those are the fastest for most
>>> tasks.)
>>>
>>> But that might mean to write less elegant or efficient code to keep it
>>> cross-implementation. Should I rather teach new people to solve a
>>> problem as well as possible, or teach them to solve a problem in a
>>> portable way?
>> I think that in principle, the programmer should only focus on writing
>> most elegant code
> I think there are several different kinds of elegance. There’s technical
> elegance (combining existing things to build something unexpectedly
> powerful with minimal effort), readability (being easy to understand for
> newcomers), minimal code (solving a problem with the minimal amount of
> code — often close to mathematics), uniformity (using the same basic
> structure everywhere), efficiency (solving a problem with minimal
> resource-consumption), ...
> I’m sure we’ll find more.
>
> Sadly these elegances contradict each other in some points.
>
>>> In my case, I decided to use Guile by checking Scheme
>>> implementations. There were three with roughly equal activity and
>>> features. I chose Guile, because it’s a GNU project and it has a long
>>> history of surviving changing maintainers, so my skills are most likely
>>> to stay useful.
>> That's interesting. I chose it by trying to add a scripting language
>> to my game engine written in C++. But I guess that if I was starting
>> now, I'd write the whole engine in Chez (which wasn't opensource until
>> last year)
> Chez becoming free software is an interesting development.
>
>>>> The ultimate goal is not to optimize programs, but programmers.
>>> I’m not sure that I want to optimize them, I want to teach them tools to
>>> be more efficient and enjoy their work more (and teach myself the same).
>> Yeah, maybe I used a harsh word, but I meant exactly optimizing
>> programmers' efficiency (or if it sounds too bad, "giving them tools
>> to be more efficient")
> Sounds good :)
>
>>> I think one important point for Scheme would be to gather some consensus
>>> points. The Zen of Python is part of what made that one community
>>> strong. There is no reason why there should not be a Zen of Scheme,
>>> along with implementation-specific Koans which extend it.
>>>
>>> Do you have ideas for principles which could be shared by most Schemers?
>>>
>>>
>> It depends on the most Schemers, not me :)
> That’s why its a social problem :)
>
> Finding a set of basic koans which most Schemers agree with requires
> talking to and knowing a lot of Schemers.
>
>> However, you can have a look at the (grand scheme) glossary that I've been
>> maintaining for some time:
>>
>> https://github.com/plande/grand-scheme
> This looks interesting.
>
>> of course, the intent is to make the glossary usable for someone else than
>> me,
>> and I can boast that I have at least one user distinct from the maintainer.
>> You can have a look and let me know what you think. (for now it only works
>> with Guile). In particular, are there any factors that could convince you to
>> adapt it as a base for your library, or what would you want changed
>> (I assume you won't stop at the README file).
> I’d mainly need practical examples. Tell me how I’d solve a problem
> with (grand scheme) and how it would be solved without it — showing me
> why using it is better.
>
> That means: Documentation.
>
> Also I’d need to know whether some of these methods have performance
> problems: Will this scale as well as the method without (grand scheme)?
>
>>     - functions should be pure
>>     - language constructs that cause side-effects should be encapsulated
>>     with macros that limit the extent of those side-effects
>>     - if possible, use the quasiquote macro, rather than cons, append or list
>>     - prefer the use of match <http://synthcode.com/scheme/match.scm> to cond,
>>     if the patterns are simple
>>     - never use abbreviations
>>     - use commented prepositions to separate arguments
> As guidelines for easy-to-use libraries, that sounds good.
>
>> Also, I recently added infix syntax to Scheme, but (unlike in Racket
>> or David Wheeler's solutions) in the way it is still a prefix
>> syntax. So for example, instead of writing (< a b), one ought to write
>> (is a < b). The rationale is that prefix notation is terrible for
>> non-symmetrical binary predicates, because it obscures the roles of
>> respective arguments (in addition, (is a < b <= c) expands to (and (<
>> a b) (<= b c))). I'm still not sure whether it is a good idea, but I
>> have no reasons to think that it is bad either :] [Curiously enough, I
>> think that prefix notation in arithmetic is just fine: (+ a b c) reads
>> as "sum of a, b and c"]
> This is an old idea, but using (is) as prefix looks
> interesting. Typically the prefix was something like (nfx), which
> doesn’t say anything semantic (other than that what follows uses infix
> syntax). It’s like an expanded equal?
>
> (if (is a = 5)
>      #t
>      #f)
>
> (cond
>      ((is q = p)
>       q)
>      ((is p > 5)
>       p)
>      (else
>       q))
>
> This seems pretty close to natural language. Can I also use (is A 5)
> instead of (equal? A 5)?
>
> Though (is? a = 5) might be closer to canonical Scheme.
>
>> I like it that you and Amirouche look a lot at the sociocultural
>> perspective of programming
> I’m glad to hear that :)
>
>> Yes, to me the ease of understanding is also crucial. (I'd even go
>> further and say that undestranding is the very point of programming)
> I would not go that far: We’re programming both for the future
> maintainers of the code and for the computer. This restricts us in
> expressing things, because we always have to keep two targets audiences
> in mind (many different systems *and* many different people).
>
> Best wishes,
> Arne

Re: How to make GNU Guile more successful

[Amirouche]

Le 01/03/2017 à 20:21, Amirouche a écrit :
>
>
> Le 14/02/2017 à 22:35, Arne Babenhauserheide a écrit :
>> Panicz Maciej Godek <godek.maciek@gmail.com> writes:
>>> There's surely many ways to approach that issue. The point is that for
>>> some reason, Schemers from various tribes prefer to reinvent the
>>> wheel, rather than use an existing one. (Not that I am any different)
>>> However, I also think that these CPAN-alike solutions are far from
>>> optimal: ideally, programming could be made an experience similar the
>>> game "The Journey" by thatgamecompany (where you travel to your goal
>>> and sometimes encounter other players heading the same direction), and
>>> the repository itself could look more like Wikipedia (but first we'd
>>> need to start perceiveing programming as a compact way of representing
>>> knowledge)
>> That sounds somehow like stackoverflow. We might already be going 
>> there :)
>
> That's a very good idea!

There is trusting issue tho. Do you trust all what is written in 
wikipedia right? You
can not simply execute code from the programming cloud like that. I try 
to review
at least once the code I use. But I trust most of it to be safe.

Re: How to make GNU Guile more successful

[Linas Vepstas]

On Mon, Feb 13, 2017 at 2:28 PM, Panicz Maciej Godek <godek.maciek@gmail.com
> wrote:

>
> someone
> responded critically: "are there out of the box libraries to estimate a
> zero inflated negative
> binomial regression model in guile". Of course, if I knew what a
> zero-inflated
> negative binomial regression model, I could deliver an implementation by
> just explaining
> the notions used in that phrase.


Caution: the message below sounds negative.  Sorry, I use guile daily and
almost exclusively now. So there ...

Lack of decent science libraries for scheme is a major stumbling block, for
me. Simply having sine and cosine is not enough.   I got excited (a decade
ago) when I realized that guile supported GnuMP, and then rapidly deflated
when I realized it only supported integers and rationals in GnuMP .. I work
with arbitrary-precision floats.  Or, I did back then.

Maybe more important is making guile work well with large-RAM setups.
Currently, I do data analysis, every day, in guile, on datasets that take
20GB or 40GB -- my current one is 110GB when loaded in RAM, and guile
starts getting buggy, crashy and slow when working at that size.
Sometimes, it starts calling GC half-a-dozen times per second, for no
apparent reason, eating up 6 cores (or more!) doing nothing but GC. Why?
Who knows? Who can tell?

Yes, I have a machine with 256 GB RAM and a few dozen cores, and SSD's that
hold the data, but every time guile crashes, I have to wait an hour for the
data to reload.  I can live with it, but its a dirty secret I would not
share with guile wannabe users.

String handling in guile is a disaster area: If I give it a
10-megabyte-long string in utf8, it promptly tries to convert all of that
string in utf32, for utterly pointless reasons. This just makes it slow.

There are still bugs between GC and the compiler: if call (eval "(some
stuff) (other stuff)")  the compiler will try to compile that string (after
it was converted ti utf32!) and if GC happens to run at just that moment,
guile crashes or hangs.  These bugs need to be fixed.

So although its a good start, there's a lot of work left until it can get
to "the next level". And that work can't happen until guile is more
popular. So it's very much chicken-and-egg scenario.

--linas

Re: How to make GNU Guile more successful

[Marko Rauhamaa]

Linas Vepstas <linasvepstas@gmail.com>:

> String handling in guile is a disaster area: If I give it a
> 10-megabyte-long string in utf8, it promptly tries to convert all of
> that string in utf32, for utterly pointless reasons. This just makes
> it slow.

It's not only the conversion. Strings should be bytes. Python3's example
shouldn't have been followed.


Marko

Re: How to make GNU Guile more successful

[Jan Wedekind]

One could implement something like Theano+NumPy with GNU Guile. I am trying to do something like that (github.com/wedesoft/aiscm) but I am doing it in my spare time only. Theoretically GNU Guile is better suited for this than Python because of macros.

On July 14, 2017 10:54:45 PM GMT+01:00, Linas Vepstas <linasvepstas@gmail.com> wrote:
>On Mon, Feb 13, 2017 at 2:28 PM, Panicz Maciej Godek
><godek.maciek@gmail.com
>> wrote:
>
>>
>> someone
>> responded critically: "are there out of the box libraries to estimate
>a
>> zero inflated negative
>> binomial regression model in guile". Of course, if I knew what a
>> zero-inflated
>> negative binomial regression model, I could deliver an implementation
>by
>> just explaining
>> the notions used in that phrase.
>
>
>Caution: the message below sounds negative.  Sorry, I use guile daily
>and
>almost exclusively now. So there ...
>
>Lack of decent science libraries for scheme is a major stumbling block,
>for
>me. Simply having sine and cosine is not enough.   I got excited (a
>decade
>ago) when I realized that guile supported GnuMP, and then rapidly
>deflated
>when I realized it only supported integers and rationals in GnuMP .. I
>work
>with arbitrary-precision floats.  Or, I did back then.
>
>Maybe more important is making guile work well with large-RAM setups.
>Currently, I do data analysis, every day, in guile, on datasets that
>take
>20GB or 40GB -- my current one is 110GB when loaded in RAM, and guile
>starts getting buggy, crashy and slow when working at that size.
>Sometimes, it starts calling GC half-a-dozen times per second, for no
>apparent reason, eating up 6 cores (or more!) doing nothing but GC.
>Why?
>Who knows? Who can tell?
>
>Yes, I have a machine with 256 GB RAM and a few dozen cores, and SSD's
>that
>hold the data, but every time guile crashes, I have to wait an hour for
>the
>data to reload.  I can live with it, but its a dirty secret I would not
>share with guile wannabe users.
>
>String handling in guile is a disaster area: If I give it a
>10-megabyte-long string in utf8, it promptly tries to convert all of
>that
>string in utf32, for utterly pointless reasons. This just makes it
>slow.
>
>There are still bugs between GC and the compiler: if call (eval "(some
>stuff) (other stuff)")  the compiler will try to compile that string
>(after
>it was converted ti utf32!) and if GC happens to run at just that
>moment,
>guile crashes or hangs.  These bugs need to be fixed.
>
>So although its a good start, there's a lot of work left until it can
>get
>to "the next level". And that work can't happen until guile is more
>popular. So it's very much chicken-and-egg scenario.
>
>--linas

-- 
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

Re: How to make GNU Guile more successful

[Nala Ginrut]

@Linas Thanks for sharing your experiences on large data processing
using Guile! I'm developing a framework to do the similar work for our
product too. But what I'm planning is to do map/reduce for large data
in a distributed system. Anyway, I don't think it's a good way to
analysis massive data in just one very-strong machine. It requires too
much for the language compiler.

I'm still very confident to use Guile on machine learning work. Please
don't forget that no any dynamic language do the heavy computation
directly, they wrapped BLAS or MKL, just like what numpy does. And I
have chance to get a very strong Nvidia GPU cluster server for the
work on the next stage. I don't think the computing speed will be the
issue for me. We're using Python and C++14 at present, but I don't
think Python is the idea glue language, no one wants to use Cython,
trust me. The tail-recursive and delimited-continuation for
lightweight task scheduling is preferred too. The key issue for me is
the expressiveness, which could let us type less code to do more work.
Actually, we use Guile to generate many C++ code in compiling time to
save lot work coding work. This does matter, especially when you have
few people but much work to finish in a tight deadline.

There should be better paradigm to use Guile to glue C/C++ code.
Implement algorithm in Guile directly is not the good way. And wrap
C++ interface with FFI simply is not good too. Sometimes we need to do
more abstract work to get both advantages of Guile and C/C++.
This is about how we use Guile, not the compiler issue.

Of course, I confess there're many problems in our current Guile
compiler. The problem is that we have to use Guile a lot to get to
know what is the problem exactly, and give compelling reasons for the
improvement.

@Jan Yes, that should be a way to go. And I have a new idea which is
just an idea at present. Many we could find a way to read PyObject to
Gulie, and call Python module directly (say, numpy). There should be a
type-compatible abstract level between Guile and PyObject. If it
works, we may implement Python3 on Guile. Although it seems a large
work to implement complete Python3 frontend, we may save lot of work
to write alternative Python modules for Guile.
Julia language does in this idea, but it's backend is compatible with
Python. My idea is not to convert all Python types to Guile, just wrap
some types to a special object like <pyobject> is enough, then Guile
could be glue language for Python too.

Maybe people ask, why bother to glue Python? Python rocks!
Rocks? I need complete lambda calculus, proper tail call, static
scoping, and delimited-continuations (not the generators).
In our product code, we use lot of lambdas in C++ code, so I expect
the programming mind could be consistent between glue language and
main language. But Python can't provide multi-lines lambda expression
only because it uses whitespaces for delimiters, there's no any chance
to delimit multi expression within a lambda unless introducing more
complexity to the parser.

Back to the topic, Guile lacks sufficient practical usage in industry
to reveal its disadvantages. But its advantages are apparent enough
for me so that it worth for me to go further with it.


Best regards.




On Sat, Jul 15, 2017 at 6:10 PM, Jan Wedekind <jan@wedesoft.de> wrote:
> One could implement something like Theano+NumPy with GNU Guile. I am trying to do something like that (github.com/wedesoft/aiscm) but I am doing it in my spare time only. Theoretically GNU Guile is better suited for this than Python because of macros.
>
> On July 14, 2017 10:54:45 PM GMT+01:00, Linas Vepstas <linasvepstas@gmail.com> wrote:
>>On Mon, Feb 13, 2017 at 2:28 PM, Panicz Maciej Godek
>><godek.maciek@gmail.com
>>> wrote:
>>
>>>
>>> someone
>>> responded critically: "are there out of the box libraries to estimate
>>a
>>> zero inflated negative
>>> binomial regression model in guile". Of course, if I knew what a
>>> zero-inflated
>>> negative binomial regression model, I could deliver an implementation
>>by
>>> just explaining
>>> the notions used in that phrase.
>>
>>
>>Caution: the message below sounds negative.  Sorry, I use guile daily
>>and
>>almost exclusively now. So there ...
>>
>>Lack of decent science libraries for scheme is a major stumbling block,
>>for
>>me. Simply having sine and cosine is not enough.   I got excited (a
>>decade
>>ago) when I realized that guile supported GnuMP, and then rapidly
>>deflated
>>when I realized it only supported integers and rationals in GnuMP .. I
>>work
>>with arbitrary-precision floats.  Or, I did back then.
>>
>>Maybe more important is making guile work well with large-RAM setups.
>>Currently, I do data analysis, every day, in guile, on datasets that
>>take
>>20GB or 40GB -- my current one is 110GB when loaded in RAM, and guile
>>starts getting buggy, crashy and slow when working at that size.
>>Sometimes, it starts calling GC half-a-dozen times per second, for no
>>apparent reason, eating up 6 cores (or more!) doing nothing but GC.
>>Why?
>>Who knows? Who can tell?
>>
>>Yes, I have a machine with 256 GB RAM and a few dozen cores, and SSD's
>>that
>>hold the data, but every time guile crashes, I have to wait an hour for
>>the
>>data to reload.  I can live with it, but its a dirty secret I would not
>>share with guile wannabe users.
>>
>>String handling in guile is a disaster area: If I give it a
>>10-megabyte-long string in utf8, it promptly tries to convert all of
>>that
>>string in utf32, for utterly pointless reasons. This just makes it
>>slow.
>>
>>There are still bugs between GC and the compiler: if call (eval "(some
>>stuff) (other stuff)")  the compiler will try to compile that string
>>(after
>>it was converted ti utf32!) and if GC happens to run at just that
>>moment,
>>guile crashes or hangs.  These bugs need to be fixed.
>>
>>So although its a good start, there's a lot of work left until it can
>>get
>>to "the next level". And that work can't happen until guile is more
>>popular. So it's very much chicken-and-egg scenario.
>>
>>--linas
>
> --
> Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

Re: How to make GNU Guile more successful

[Nala Ginrut]

s/Many we could find a way/Maybe we could find a way
Sorry

On Sat, Jul 15, 2017 at 8:55 PM, Nala Ginrut <nalaginrut@gmail.com> wrote:
> @Linas Thanks for sharing your experiences on large data processing
> using Guile! I'm developing a framework to do the similar work for our
> product too. But what I'm planning is to do map/reduce for large data
> in a distributed system. Anyway, I don't think it's a good way to
> analysis massive data in just one very-strong machine. It requires too
> much for the language compiler.
>
> I'm still very confident to use Guile on machine learning work. Please
> don't forget that no any dynamic language do the heavy computation
> directly, they wrapped BLAS or MKL, just like what numpy does. And I
> have chance to get a very strong Nvidia GPU cluster server for the
> work on the next stage. I don't think the computing speed will be the
> issue for me. We're using Python and C++14 at present, but I don't
> think Python is the idea glue language, no one wants to use Cython,
> trust me. The tail-recursive and delimited-continuation for
> lightweight task scheduling is preferred too. The key issue for me is
> the expressiveness, which could let us type less code to do more work.
> Actually, we use Guile to generate many C++ code in compiling time to
> save lot work coding work. This does matter, especially when you have
> few people but much work to finish in a tight deadline.
>
> There should be better paradigm to use Guile to glue C/C++ code.
> Implement algorithm in Guile directly is not the good way. And wrap
> C++ interface with FFI simply is not good too. Sometimes we need to do
> more abstract work to get both advantages of Guile and C/C++.
> This is about how we use Guile, not the compiler issue.
>
> Of course, I confess there're many problems in our current Guile
> compiler. The problem is that we have to use Guile a lot to get to
> know what is the problem exactly, and give compelling reasons for the
> improvement.
>
> @Jan Yes, that should be a way to go. And I have a new idea which is
> just an idea at present. Many we could find a way to read PyObject to
> Gulie, and call Python module directly (say, numpy). There should be a
> type-compatible abstract level between Guile and PyObject. If it
> works, we may implement Python3 on Guile. Although it seems a large
> work to implement complete Python3 frontend, we may save lot of work
> to write alternative Python modules for Guile.
> Julia language does in this idea, but it's backend is compatible with
> Python. My idea is not to convert all Python types to Guile, just wrap
> some types to a special object like <pyobject> is enough, then Guile
> could be glue language for Python too.
>
> Maybe people ask, why bother to glue Python? Python rocks!
> Rocks? I need complete lambda calculus, proper tail call, static
> scoping, and delimited-continuations (not the generators).
> In our product code, we use lot of lambdas in C++ code, so I expect
> the programming mind could be consistent between glue language and
> main language. But Python can't provide multi-lines lambda expression
> only because it uses whitespaces for delimiters, there's no any chance
> to delimit multi expression within a lambda unless introducing more
> complexity to the parser.
>
> Back to the topic, Guile lacks sufficient practical usage in industry
> to reveal its disadvantages. But its advantages are apparent enough
> for me so that it worth for me to go further with it.
>
>
> Best regards.
>
>
>
>
> On Sat, Jul 15, 2017 at 6:10 PM, Jan Wedekind <jan@wedesoft.de> wrote:
>> One could implement something like Theano+NumPy with GNU Guile. I am trying to do something like that (github.com/wedesoft/aiscm) but I am doing it in my spare time only. Theoretically GNU Guile is better suited for this than Python because of macros.
>>
>> On July 14, 2017 10:54:45 PM GMT+01:00, Linas Vepstas <linasvepstas@gmail.com> wrote:
>>>On Mon, Feb 13, 2017 at 2:28 PM, Panicz Maciej Godek
>>><godek.maciek@gmail.com
>>>> wrote:
>>>
>>>>
>>>> someone
>>>> responded critically: "are there out of the box libraries to estimate
>>>a
>>>> zero inflated negative
>>>> binomial regression model in guile". Of course, if I knew what a
>>>> zero-inflated
>>>> negative binomial regression model, I could deliver an implementation
>>>by
>>>> just explaining
>>>> the notions used in that phrase.
>>>
>>>
>>>Caution: the message below sounds negative.  Sorry, I use guile daily
>>>and
>>>almost exclusively now. So there ...
>>>
>>>Lack of decent science libraries for scheme is a major stumbling block,
>>>for
>>>me. Simply having sine and cosine is not enough.   I got excited (a
>>>decade
>>>ago) when I realized that guile supported GnuMP, and then rapidly
>>>deflated
>>>when I realized it only supported integers and rationals in GnuMP .. I
>>>work
>>>with arbitrary-precision floats.  Or, I did back then.
>>>
>>>Maybe more important is making guile work well with large-RAM setups.
>>>Currently, I do data analysis, every day, in guile, on datasets that
>>>take
>>>20GB or 40GB -- my current one is 110GB when loaded in RAM, and guile
>>>starts getting buggy, crashy and slow when working at that size.
>>>Sometimes, it starts calling GC half-a-dozen times per second, for no
>>>apparent reason, eating up 6 cores (or more!) doing nothing but GC.
>>>Why?
>>>Who knows? Who can tell?
>>>
>>>Yes, I have a machine with 256 GB RAM and a few dozen cores, and SSD's
>>>that
>>>hold the data, but every time guile crashes, I have to wait an hour for
>>>the
>>>data to reload.  I can live with it, but its a dirty secret I would not
>>>share with guile wannabe users.
>>>
>>>String handling in guile is a disaster area: If I give it a
>>>10-megabyte-long string in utf8, it promptly tries to convert all of
>>>that
>>>string in utf32, for utterly pointless reasons. This just makes it
>>>slow.
>>>
>>>There are still bugs between GC and the compiler: if call (eval "(some
>>>stuff) (other stuff)")  the compiler will try to compile that string
>>>(after
>>>it was converted ti utf32!) and if GC happens to run at just that
>>>moment,
>>>guile crashes or hangs.  These bugs need to be fixed.
>>>
>>>So although its a good start, there's a lot of work left until it can
>>>get
>>>to "the next level". And that work can't happen until guile is more
>>>popular. So it's very much chicken-and-egg scenario.
>>>
>>>--linas
>>
>> --
>> Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

Re: How to make GNU Guile more successful

[Jan Wedekind]

On Sat, 15 Jul 2017, Nala Ginrut wrote:

> @Jan Yes, that should be a way to go. And I have a new idea which is
> just an idea at present. Many we could find a way to read PyObject to
> Gulie, and call Python module directly (say, numpy). There should be a
> type-compatible abstract level between Guile and PyObject. If it
> works, we may implement Python3 on Guile. Although it seems a large
> work to implement complete Python3 frontend, we may save lot of work
> to write alternative Python modules for Guile.
> Julia language does in this idea, but it's backend is compatible with
> Python. My idea is not to convert all Python types to Guile, just wrap
> some types to a special object like <pyobject> is enough, then Guile

Sure, a Guile Python bridge would be nice for using NumPy and SciPy. 
However while NumPy is quite mature, it cannot do compose array 
operations and avoid intermediate results as Theano can.
One could write bindings to Theano. However I think that the bridging code 
would get in the way at some point.

From the Theano documentation [1]:

>>> import numpy
>>> import theano.tensor as T
>>> from theano import function
>>> x = T.dscalar('x')
>>> y = T.dscalar('y')
>>> f = function([x, y], x + y)
>>> f(2, 3)

This is about having "2 + 3" being computed with fast compiled code.
With Scheme macros most of that can be implemented transparently [2]:

>>> (use-modules (aiscm tensor) (aiscm int))
>>> (tensor (+ 2 3))

[1] http://www.deeplearning.net/software/theano/tutorial/adding.html#adding-two-matrices
[2] http://wedesoft.github.io/aiscm/operation.html#tensor-operations

Re: How to make GNU Guile more successful

[Nala Ginrut]

I've taken a look at Python bridge for other languages, there're several:
For Lua
https://labix.org/lunatic-python

For ObjC
https://pythonhosted.org/pyobjc/

For JS
https://github.com/ipython/ipython/wiki/IPEP-26:-Full-Featured-python-js-object-bridge

I'm glad that the idea is not just my imagination, it exists and
there're something to read.
I'll start this work after release Artanis-0.2.2.

Best regards.


On Sun, Jul 16, 2017 at 6:17 AM, Jan Wedekind <jan@wedesoft.de> wrote:
> On Sat, 15 Jul 2017, Nala Ginrut wrote:
>
>> @Jan Yes, that should be a way to go. And I have a new idea which is
>> just an idea at present. Many we could find a way to read PyObject to
>> Gulie, and call Python module directly (say, numpy). There should be a
>> type-compatible abstract level between Guile and PyObject. If it
>> works, we may implement Python3 on Guile. Although it seems a large
>> work to implement complete Python3 frontend, we may save lot of work
>> to write alternative Python modules for Guile.
>> Julia language does in this idea, but it's backend is compatible with
>> Python. My idea is not to convert all Python types to Guile, just wrap
>> some types to a special object like <pyobject> is enough, then Guile
>
>
> Sure, a Guile Python bridge would be nice for using NumPy and SciPy. However
> while NumPy is quite mature, it cannot do compose array operations and avoid
> intermediate results as Theano can.
> One could write bindings to Theano. However I think that the bridging code
> would get in the way at some point.
>
> From the Theano documentation [1]:
>
>>>> import numpy
>>>> import theano.tensor as T
>>>> from theano import function
>>>> x = T.dscalar('x')
>>>> y = T.dscalar('y')
>>>> f = function([x, y], x + y)
>>>> f(2, 3)
>
>
> This is about having "2 + 3" being computed with fast compiled code.
> With Scheme macros most of that can be implemented transparently [2]:
>
>>>> (use-modules (aiscm tensor) (aiscm int))
>>>> (tensor (+ 2 3))
>
>
> [1]
> http://www.deeplearning.net/software/theano/tutorial/adding.html#adding-two-matrices
> [2] http://wedesoft.github.io/aiscm/operation.html#tensor-operations

Re: How to make GNU Guile more successful

[Arun Isaac]

Jan Wedekind writes:

> I am trying to do something like that (github.com/wedesoft/aiscm) but
> I am doing it in my spare time only.

Just hearing about aiscm. I was looking for something like this. Thank
you!

Re: How to make GNU Guile more successful

[Ernest Adrogué]

2017-07-15, 11:10 (+0100); Jan Wedekind escriu:
> One could implement something like Theano+NumPy with GNU Guile. I am
> trying to do something like that (github.com/wedesoft/aiscm) but I am
> doing it in my spare time only. Theoretically GNU Guile is better
> suited for this than Python because of macros.

This is interesting as well, a Lisp interface to R.

https://github.com/dirkschumacher/llr

Re: How to make GNU Guile more successful

[Freja Nordsiek]

If I was to hazard a reason for why Guile gets very slow when loading
20 GB or more (may or may not be related to it being buggy and
crashy), my guesses would be a lot of the data when loaded into Guile
was allocated such that the GC scans it for pointers (using
scm_gc_malloc instead of scm_gc_malloc_pointerless) which would vastly
increase the amount of memory the GC needs to scan every time it runs.

Depending on the data types and what is in them, it may be needless
for the GC to run through the bulk of the data looking for pointers
and this might be a fixable problem. For example, it generally isn't
necessary to scan inside strings for pointers so if that is being
done, there is something in Guile to fix.

If there are really pointers in it (say it is a lot of and/or big
lists, vectors, hash tables, etc.) then the GC really does need to
scan them, which suggests a different kind of data structure would
work around the problem. This is not always doable, and even if doable
could take a lot of programmer time. It seems that Go programmers have
run into this with very large maps already (see
https://github.com/golang/go/issues/9477 and
https://groups.google.com/forum/#!topic/golang-nuts/pHYverdFcLc ).

No idea how this relates to being buggy or crashy.


Freja Nordsiek

On Fri, Jul 14, 2017 at 11:54 PM, Linas Vepstas <linasvepstas@gmail.com> wrote:
> On Mon, Feb 13, 2017 at 2:28 PM, Panicz Maciej Godek <godek.maciek@gmail.com
>> wrote:
>
>>
>> someone
>> responded critically: "are there out of the box libraries to estimate a
>> zero inflated negative
>> binomial regression model in guile". Of course, if I knew what a
>> zero-inflated
>> negative binomial regression model, I could deliver an implementation by
>> just explaining
>> the notions used in that phrase.
>
>
> Caution: the message below sounds negative.  Sorry, I use guile daily and
> almost exclusively now. So there ...
>
> Lack of decent science libraries for scheme is a major stumbling block, for
> me. Simply having sine and cosine is not enough.   I got excited (a decade
> ago) when I realized that guile supported GnuMP, and then rapidly deflated
> when I realized it only supported integers and rationals in GnuMP .. I work
> with arbitrary-precision floats.  Or, I did back then.
>
> Maybe more important is making guile work well with large-RAM setups.
> Currently, I do data analysis, every day, in guile, on datasets that take
> 20GB or 40GB -- my current one is 110GB when loaded in RAM, and guile
> starts getting buggy, crashy and slow when working at that size.
> Sometimes, it starts calling GC half-a-dozen times per second, for no
> apparent reason, eating up 6 cores (or more!) doing nothing but GC. Why?
> Who knows? Who can tell?
>
> Yes, I have a machine with 256 GB RAM and a few dozen cores, and SSD's that
> hold the data, but every time guile crashes, I have to wait an hour for the
> data to reload.  I can live with it, but its a dirty secret I would not
> share with guile wannabe users.
>
> String handling in guile is a disaster area: If I give it a
> 10-megabyte-long string in utf8, it promptly tries to convert all of that
> string in utf32, for utterly pointless reasons. This just makes it slow.
>
> There are still bugs between GC and the compiler: if call (eval "(some
> stuff) (other stuff)")  the compiler will try to compile that string (after
> it was converted ti utf32!) and if GC happens to run at just that moment,
> guile crashes or hangs.  These bugs need to be fixed.
>
> So although its a good start, there's a lot of work left until it can get
> to "the next level". And that work can't happen until guile is more
> popular. So it's very much chicken-and-egg scenario.
>
> --linas

Re: How to make GNU Guile more successful

[Marko Rauhamaa]

Freja Nordsiek <fnordsie@gmail.com>:

> If I was to hazard a reason for why Guile gets very slow when loading
> 20 GB or more (may or may not be related to it being buggy and
> crashy), my guesses would be a lot of the data when loaded into Guile
> was allocated such that the GC scans it for pointers (using
> scm_gc_malloc instead of scm_gc_malloc_pointerless) which would vastly
> increase the amount of memory the GC needs to scan every time it runs.

Good point!

If you didn't to any C programming, what kind of native Guile data
structures are good for such large random-access storage? At least
arrays haven't specifically been documented for such GC optimization:
<URL: https://www.gnu.org/software/guile/manual/html_node/Arrays.htm
l#Arrays>.

Maybe bytevectors would do: <URL: https://www.gnu.org/software/guile/m
anual/html_node/Bytevectors.html#Bytevectors>.

Of course constantly encoding to and decoding from a bytevector using
scheme code might be very slow without the help of some binary bulk
formatting facilities for the data records.


Marko

Re: How to make GNU Guile more successful

[Freja Nordsiek]

I checked the implementation of bytecectors and SRFI-4 in Guile and they are definitely not scanned for pointers. But I would say hacking them is not a good general solution for this problem. They are good and natural data structures for large arrays of numerical data that are standard signed/unsigned integers of various fixed sizes and IEEE floating point numbers, or structures/unions of these types. Using them for things other than that or strings/byte-arrays could be error prone, messy, and performance poor.

Freja Nordsiek

On July 16, 2017 11:18:18 AM GMT+02:00, Marko Rauhamaa <marko@pacujo.net> wrote:
>Freja Nordsiek <fnordsie@gmail.com>:
>
>> If I was to hazard a reason for why Guile gets very slow when loading
>> 20 GB or more (may or may not be related to it being buggy and
>> crashy), my guesses would be a lot of the data when loaded into Guile
>> was allocated such that the GC scans it for pointers (using
>> scm_gc_malloc instead of scm_gc_malloc_pointerless) which would
>vastly
>> increase the amount of memory the GC needs to scan every time it
>runs.
>
>Good point!
>
>If you didn't to any C programming, what kind of native Guile data
>structures are good for such large random-access storage? At least
>arrays haven't specifically been documented for such GC optimization:
><URL: https://www.gnu.org/software/guile/manual/html_node/Arrays.htm
>l#Arrays>.
>
>Maybe bytevectors would do: <URL: https://www.gnu.org/software/guile/m
>anual/html_node/Bytevectors.html#Bytevectors>.
>
>Of course constantly encoding to and decoding from a bytevector using
>scheme code might be very slow without the help of some binary bulk
>formatting facilities for the data records.
>
>
>Marko

Re: How to make GNU Guile more successful

[Marko Rauhamaa]

Freja Nordsiek <fnordsie@gmail.com>:

> I checked the implementation of bytecectors and SRFI-4 in Guile and
> they are definitely not scanned for pointers. But I would say hacking
> them is not a good general solution for this problem. They are good
> and natural data structures for large arrays of numerical data that
> are standard signed/unsigned integers of various fixed sizes and IEEE
> floating point numbers, or structures/unions of these types. Using
> them for things other than that or strings/byte-arrays could be error
> prone, messy, and performance poor.

So no Scheme solution recommended then?


Marko

Re: How to make GNU Guile more successful

[Freja Nordsiek]

Not necessarily. Using bytevectors and SRFI-4 vectors works very well for some kinds of data. Just, they aren't the best for other kinds (in principle they can work with all data if one is one implements a heap or other custom memory management inside one). They can be made to work but are hard for some types of data, and if you are working with pointers inside them very easy to get a segfault (even python numpy, which has a lot of work done on it over many years, still has a fairly easy way to segfault it if one uses object dtype).


Freja Nordsiek

On July 16, 2017 12:31:37 PM GMT+02:00, Marko Rauhamaa <marko@pacujo.net> wrote:
>Freja Nordsiek <fnordsie@gmail.com>:
>
>> I checked the implementation of bytecectors and SRFI-4 in Guile and
>> they are definitely not scanned for pointers. But I would say hacking
>> them is not a good general solution for this problem. They are good
>> and natural data structures for large arrays of numerical data that
>> are standard signed/unsigned integers of various fixed sizes and IEEE
>> floating point numbers, or structures/unions of these types. Using
>> them for things other than that or strings/byte-arrays could be error
>> prone, messy, and performance poor.
>
>So no Scheme solution recommended then?
>
>
>Marko

Re: How to make GNU Guile more successful

[Freja Nordsiek]

Oh, sorry, just realized I skipped the main question.

There are various data type substitutions that can help or hurt. Replacing lists with vectors or vice versa can significantly help or hurt performance depending on the use case. Same for charsets vs strings. In situations where the data is preserved from collection by other means, weak references to that data might reduce the GC load since there is just less data to go through (not sure how the GC implements weak references so could be very worng about that).


Freja Nordsiek

On July 16, 2017 12:39:20 PM GMT+02:00, Freja Nordsiek <fnordsie@gmail.com> wrote:
>Not necessarily. Using bytevectors and SRFI-4 vectors works very well
>for some kinds of data. Just, they aren't the best for other kinds (in
>principle they can work with all data if one is one implements a heap
>or other custom memory management inside one). They can be made to work
>but are hard for some types of data, and if you are working with
>pointers inside them very easy to get a segfault (even python numpy,
>which has a lot of work done on it over many years, still has a fairly
>easy way to segfault it if one uses object dtype).
>
>
>Freja Nordsiek
>
>On July 16, 2017 12:31:37 PM GMT+02:00, Marko Rauhamaa
><marko@pacujo.net> wrote:
>>Freja Nordsiek <fnordsie@gmail.com>:
>>
>>> I checked the implementation of bytecectors and SRFI-4 in Guile and
>>> they are definitely not scanned for pointers. But I would say
>hacking
>>> them is not a good general solution for this problem. They are good
>>> and natural data structures for large arrays of numerical data that
>>> are standard signed/unsigned integers of various fixed sizes and
>IEEE
>>> floating point numbers, or structures/unions of these types. Using
>>> them for things other than that or strings/byte-arrays could be
>error
>>> prone, messy, and performance poor.
>>
>>So no Scheme solution recommended then?
>>
>>
>>Marko

Guile bugs

[Ludovic Courtès]

Hi Linas,

Linas Vepstas <linasvepstas@gmail.com> skribis:

> Lack of decent science libraries for scheme is a major stumbling block, for
> me. Simply having sine and cosine is not enough.   I got excited (a decade
> ago) when I realized that guile supported GnuMP, and then rapidly deflated
> when I realized it only supported integers and rationals in GnuMP .. I work
> with arbitrary-precision floats.  Or, I did back then.

I think Someone should write MPFR bindings…

> Maybe more important is making guile work well with large-RAM setups.
> Currently, I do data analysis, every day, in guile, on datasets that take
> 20GB or 40GB -- my current one is 110GB when loaded in RAM, and guile
> starts getting buggy, crashy and slow when working at that size.
> Sometimes, it starts calling GC half-a-dozen times per second, for no
> apparent reason, eating up 6 cores (or more!) doing nothing but GC. Why?
> Who knows? Who can tell?
>
> Yes, I have a machine with 256 GB RAM and a few dozen cores, and SSD's that
> hold the data, but every time guile crashes, I have to wait an hour for the
> data to reload.  I can live with it, but its a dirty secret I would not
> share with guile wannabe users.

I think these are genuine serious bugs that should be reported.  Of
course the user-to-developer ratio for Guile is pretty bad (but maybe
not worse than that of CPython after all!), so it will help a lot if you
(1) provide a reduced test case and anyone can use to reproduce the
problem, and (2) do some investigation of your own.

At any rate, if you don’t report it bugs that make your life difficult,
your life is definitely not going to be easier.  ;-)

> String handling in guile is a disaster area: If I give it a
> 10-megabyte-long string in utf8, it promptly tries to convert all of that
> string in utf32, for utterly pointless reasons. This just makes it slow.

We’ve discussed it before, and while I agree that there’s much room for
improvement, I’m very skeptical about this use case (I’d use ‘mmap’ and
get a bytevector.)

> There are still bugs between GC and the compiler: if call (eval "(some
> stuff) (other stuff)")  the compiler will try to compile that string (after
> it was converted ti utf32!) and if GC happens to run at just that moment,
> guile crashes or hangs.  These bugs need to be fixed.

Please report!

> So although its a good start, there's a lot of work left until it can get
> to "the next level". And that work can't happen until guile is more
> popular. So it's very much chicken-and-egg scenario.

Definitely.  I think one has to get ready to get their hands dirty when
pushing Guile to its limits.  That’s probably true of any piece of
software, but that’s even more acute for software with a smaller user
base.

My 2¢!

Ludo’.

Re: Guile bugs

[Marko Rauhamaa]

ludo@gnu.org (Ludovic Courtès):

> I’m very skeptical about this use case (I’d use ‘mmap’ and get a
> bytevector.)

Please elaborate.

 1. Does Guile offer mmap to Scheme code?

 2. What would prevent Guile's GC from scanning the mmapped area for
    pointers?

 3. How do I efficiently encode information in a bytevector in Scheme
    code?


Marko

Re: Guile bugs

[Taylan Ulrich Bayırlı/Kammer]

Marko Rauhamaa <marko@pacujo.net> writes:

> ludo@gnu.org (Ludovic Courtès):
>
>> I’m very skeptical about this use case (I’d use ‘mmap’ and get a
>> bytevector.)
>
> Please elaborate.
>
>  1. Does Guile offer mmap to Scheme code?

It's pretty easy to call C functions from Scheme.  Here's a trivial
example.

$ cat map_file.c
#include <stddef.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/mman.h>

void *map_file(char *file) {
  int fd = open(file, O_RDONLY);
  off_t size = lseek(fd, 0, SEEK_END);
  return mmap(NULL, size, PROT_READ, MAP_PRIVATE, fd, 0);
}
$ gcc -shared -o map_file.so -fPIC map_file.c
$ guile
)> ,use (system foreign)
)> (define map-file
     (pointer->procedure
      '*
      (dynamic-func "map_file"
                    (dynamic-link "./map_file"))
      (list '*)))
)> (define file-ptr (map-file (string->pointer "/home/taylan/todo")))
)> (define file-bv (pointer->bytevector file-ptr 100))  ;see note below
)> file-bv
$3 = #vu8(45 42 45 32 111 114 103 32 45 42 45 10 10 ...)

That's ASCII for "-*- org -*-\n\n", which is what my todo file begins
with. :-)

(It's a file mode marker for Emacs, for those who don't know.)

Note: The second argument to pointer->bytevector defines the size of the
bytevector.  I passed 100 because I was too lazy to get the size of the
file.  You should pass the size of the file in bytes.

>  2. What would prevent Guile's GC from scanning the mmapped area for
>     pointers?

I don't know the details but AFAIK this is no problem with Boehm GC.

>  3. How do I efficiently encode information in a bytevector in Scheme
>     code?

What sort of data?

I have a library called bytestructures that imitates the C type system
within Scheme, to be used on bytevectors that contain data structures
generated by C code, though the library is built upon a generic core
with which other structures can be declared as well.  Not sure if this
helps you:

https://github.com/TaylanUB/scheme-bytestructures/

> Marko

Hope I could help,
Taylan

Re: Guile bugs

[Marko Rauhamaa]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer"):

> Marko Rauhamaa <marko@pacujo.net> writes:
>
>> ludo@gnu.org (Ludovic Courtès):
>>
>>> I’m very skeptical about this use case (I’d use ‘mmap’ and get a
>>> bytevector.)
>>
>> Please elaborate.
>>
>>  1. Does Guile offer mmap to Scheme code?
>
> It's pretty easy to call C functions from Scheme.

Not good enough. I would have to precompile libraries for different
target platforms (or require the target environment to have a C
compiler). Besides, the need is quite generic and shouldn't require an
extension.

>>  2. What would prevent Guile's GC from scanning the mmapped area for
>>     pointers?
>
> I don't know the details but AFAIK this is no problem with Boehm GC.

I don't understand. It can be quite time-consuming to scan gigabytes of
RAM for pointers.

>>  3. How do I efficiently encode information in a bytevector in Scheme
>>     code?
>
> What sort of data?
>
> I have a library called bytestructures that imitates the C type system
> within Scheme, to be used on bytevectors that contain data structures
> generated by C code, though the library is built upon a generic core
> with which other structures can be declared as well.  Not sure if this
> helps you:
>
> https://github.com/TaylanUB/scheme-bytestructures/

That's precisely it. It would be nice to have it as part of standard
Guile.

I wonder, though, if doing that is fast enough in Scheme code.


Marko

Re: Guile bugs

[Ludovic Courtès]

Hi Marko,

Marko Rauhamaa <marko@pacujo.net> skribis:

> taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer"):

[...]

>>>  2. What would prevent Guile's GC from scanning the mmapped area for
>>>     pointers?
>>
>> I don't know the details but AFAIK this is no problem with Boehm GC.
>
> I don't understand. It can be quite time-consuming to scan gigabytes of
> RAM for pointers.

libgc knows which regions it must scan and mmap’d regions like this are
not among them.

For the record, the loader in Guile 2.2 mmaps .go files in memory just
like this (see ‘load-thunk-from-file’.)

>>>  3. How do I efficiently encode information in a bytevector in Scheme
>>>     code?
>>
>> What sort of data?
>>
>> I have a library called bytestructures that imitates the C type system
>> within Scheme, to be used on bytevectors that contain data structures
>> generated by C code, though the library is built upon a generic core
>> with which other structures can be declared as well.  Not sure if this
>> helps you:
>>
>> https://github.com/TaylanUB/scheme-bytestructures/
>
> That's precisely it. It would be nice to have it as part of standard
> Guile.

Even though I agree that something like bytestructures should eventually
land in Guile, I think Taylan just pointed out that this is something
you can use if you’re dealing with structured binary data.

But again, that really depends on the kind of data you’re dealing with.
Maybe sometimes the tools that (system foreign) provides are good
enough, maybe sometimes you’d rather do something different like Guile’s
ELF parser does.

HTH,
Ludo’.

Re: Guile bugs

[Marko Rauhamaa]

ludo@gnu.org (Ludovic Courtès):

> libgc knows which regions it must scan and mmap’d regions like this are
> not among them.

Wow, where is that documented? I would have imagined it scanned all
writable RAM and CPU registers.


Marko

Re: Guile bugs

[Mark H Weaver]

Marko Rauhamaa <marko@pacujo.net> writes:

> ludo@gnu.org (Ludovic Courtès):
>
>> libgc knows which regions it must scan and mmap’d regions like this are
>> not among them.
>
> Wow, where is that documented? I would have imagined it scanned all
> writable RAM and CPU registers.

It's documented here:  http://www.hboehm.info/gc/gcdescr.html

      Mark

Re: Guile bugs

[Marko Rauhamaa]

Mark H Weaver <mhw@netris.org>:

> Marko Rauhamaa <marko@pacujo.net> writes:
>
>> ludo@gnu.org (Ludovic Courtès):
>>
>>> libgc knows which regions it must scan and mmap’d regions like this
>>> are not among them.
>>
>> Wow, where is that documented? I would have imagined it scanned all
>> writable RAM and CPU registers.
>
> It's documented here:  http://www.hboehm.info/gc/gcdescr.html

Please point out the sentence.

The closest I could find is:

   * Static data region(s). In the simplest case, this is the region
     between DATASTART and DATAEND, as defined in gcconfig.h. However,
     in most cases, this will also involve static data regions
     associated with dynamic libraries. These are identified by the
     mostly platform-specific code in dyn_load.c.

But that doesn't clearly state if mmapped regions are in or out. I would
have assumed they were in.


Marko

Re: Guile bugs

[Chris Vine]

On Fri, 21 Jul 2017 09:15:28 +0300
Marko Rauhamaa <marko@pacujo.net> wrote:
> Mark H Weaver <mhw@netris.org>:
> 
> > Marko Rauhamaa <marko@pacujo.net> writes:
> >  
> >> ludo@gnu.org (Ludovic Courtès):
> >>  
> >>> libgc knows which regions it must scan and mmap’d regions like
> >>> this are not among them.  
> >>
> >> Wow, where is that documented? I would have imagined it scanned all
> >> writable RAM and CPU registers.  
> >
> > It's documented here:  http://www.hboehm.info/gc/gcdescr.html  
> 
> Please point out the sentence.
> 
> The closest I could find is:
> 
>    * Static data region(s). In the simplest case, this is the region
>      between DATASTART and DATAEND, as defined in gcconfig.h. However,
>      in most cases, this will also involve static data regions
>      associated with dynamic libraries. These are identified by the
>      mostly platform-specific code in dyn_load.c.
> 
> But that doesn't clearly state if mmapped regions are in or out. I
> would have assumed they were in.

Since the C runtime does not store static variables in mmapped memory, I
would have assumed the opposite.

There is also some information at
https://www.gnu.org/software/guile/manual/html_node/Garbage-Collection.html#Garbage-Collection
That states that the areas that the garbage collector scans for live
objects are the areas where global and other static variables are
placed, local variables (including function arguments and registers),
and heap memory allocated by scm_gc_malloc() (but not memory allocated
by scm_gc_malloc_pointerless() or by malloc() and cognates).

If a SCM object keeps references to sub-objects residing in unscanned
memory, this applies:
https://www.gnu.org/software/guile/manual/html_node/Foreign-Object-Memory-Management.html#Foreign-Object-Memory-Management
(and the equivalent for smobs in guile-2.0).

Chris

Re: Guile bugs

[Marko Rauhamaa]

Chris Vine <vine35792468@gmail.com>:

> On Fri, 21 Jul 2017 09:15:28 +0300
> Marko Rauhamaa <marko@pacujo.net> wrote:
>> The closest I could find is:
>> 
>>    * Static data region(s). In the simplest case, this is the region
>>      between DATASTART and DATAEND, as defined in gcconfig.h. However,
>>      in most cases, this will also involve static data regions
>>      associated with dynamic libraries. These are identified by the
>>      mostly platform-specific code in dyn_load.c.
>> 
>> But that doesn't clearly state if mmapped regions are in or out. I
>> would have assumed they were in.
>
> Since the C runtime does not store static variables in mmapped memory,
> I would have assumed the opposite.

I use libgc in non-Guile code. If mmapped areas are not considered for
GC, I'll have to take special care.

I had assumed libgc simply introspects the process's memory map and
traverses anything reachable. After all, the process might be also using
a secondary memory allocator based on mmap. What if you do place
pointers in a mmapped region?

> There is also some information at https://www.gnu.org/softwa
> re/guile/manual/html_node/Garbage-Collection.html#Garbage-Collection
> That states that the areas that the garbage collector scans for live
> objects are the areas where global and other static variables are
> placed, local variables (including function arguments and registers),
> and heap memory allocated by scm_gc_malloc() (but not memory allocated
> by scm_gc_malloc_pointerless() or by malloc() and cognates).

Which doesn't say anything explicit about mmap.

Anyway, I suspect a bytearray is guaranteed to be pointerless and could
be used for the desired kind of mass storage in RAM. It all hinges on
the efficiency of encoding and decoding objects. As much as possible, I
would like to stay in Scheme-land.


Marko

Re: Guile bugs

[Mark H Weaver]

Marko Rauhamaa <marko@pacujo.net> writes:

> Mark H Weaver <mhw@netris.org>:
>
>> Marko Rauhamaa <marko@pacujo.net> writes:
>>
>>> ludo@gnu.org (Ludovic Courtès):
>>>
>>>> libgc knows which regions it must scan and mmap’d regions like this
>>>> are not among them.
>>>
>>> Wow, where is that documented? I would have imagined it scanned all
>>> writable RAM and CPU registers.
>>
>> It's documented here:  http://www.hboehm.info/gc/gcdescr.html
>
> Please point out the sentence.

Read the "Mark phase" section.  In brief, roots are scanned from the
registers, stack(s), and static data region(s).  The only other areas
scanned are heap areas specifically managed by libgc.  No pointer is
considered valid unless it points to a libgc-managed heap block.

      Mark

Re: Guile bugs

[Marko Rauhamaa]

Mark H Weaver <mhw@netris.org>:

> Marko Rauhamaa <marko@pacujo.net> writes:
>
>> Mark H Weaver <mhw@netris.org>:
>>
>>> Marko Rauhamaa <marko@pacujo.net> writes:
>>>
>>>> ludo@gnu.org (Ludovic Courtès):
>>>>
>>>>> libgc knows which regions it must scan and mmap’d regions like
>>>>> this are not among them.
>>>>
>>>> Wow, where is that documented? I would have imagined it scanned all
>>>> writable RAM and CPU registers.
>>>
>>> It's documented here:  http://www.hboehm.info/gc/gcdescr.html
>>
>> Please point out the sentence.
>
> Read the "Mark phase" section.  In brief, roots are scanned from the
> registers, stack(s), and static data region(s).  The only other areas
> scanned are heap areas specifically managed by libgc.  No pointer is
> considered valid unless it points to a libgc-managed heap block.

If you say so.

Both libgc and Guile would benefit from explicitly mentioning (example)
areas of RAM that aren't considered by GC. This page is quite
dismissive:

   The BDW-GC “just works”, for the most part. [...]

   Thus, the BDW-GC uses a technique called conservative garbage
   collection, to make the local variable list unnecessary. [...]

   Obviously, such a system will occasionally retain objects that are
   actually garbage, and should be freed. In practice, this is not a
   problem. The alternative, an explicitly maintained list of local
   variable addresses, is effectively much less reliable, due to
   programmer error. Interested readers should see the BDW-GC web page
   at <URL: http://www.hboehm.info/gc/>, for more information.

   <URL: https://www.gnu.org/software/guile/manual/html_node/Conservativ
   e-GC.html>


What you're saying points out a secondary problem that applies to Guile
C programming. If you have stored Guile smob references in, say,
third-party library objects, you'll need to carefully protect and
unprotect them.


Marko

Re: Guile bugs

[David Kastrup]

Marko Rauhamaa <marko@pacujo.net> writes:

> What you're saying points out a secondary problem that applies to
> Guile C programming. If you have stored Guile smob references in, say,
> third-party library objects, you'll need to carefully protect and
> unprotect them.

It may be worth mentioning that this includes _all_ C++ STL container
classes (like std::vector).  If you want to employ them in connection
with Guile2's automatic memory scanning, you need to use them with
custom allocators.

-- 
David Kastrup

Re: Guile bugs

[Linas Vepstas]

Stale reply to old message, but maybe its still helpful:

On Fri, Jul 21, 2017 at 1:15 AM, Marko Rauhamaa <marko@pacujo.net> wrote:

> Mark H Weaver <mhw@netris.org>:
>
> > Marko Rauhamaa <marko@pacujo.net> writes:
> >
> >> ludo@gnu.org (Ludovic Courtès):
> >>
> >>> libgc knows which regions it must scan and mmap’d regions like this
> >>> are not among them.
> >>
> >> Wow, where is that documented? I would have imagined it scanned all
> >> writable RAM and CPU registers.
> >
> > It's documented here:  http://www.hboehm.info/gc/gcdescr.html
>
> Please point out the sentence.


 https://github.com/ivmai/bdwgc/blob/master/include/gc.h
lines 448-450:

/* General purpose allocation routines, with roughly malloc calling     */
/* conv.  The atomic versions promise that no relevant pointers are     */
/* contained in the object.

So if you used GC_malloc_atomic() in your code, then gc will NOT scan that
region for pointers. guile-2.2 does this correctly for strings (I checked)
because strings will never contain pointers.  I have not checked other uses.

--linas


-- 
*"The problem is not that artificial intelligence will get too smart and
take over the world," computer scientist Pedro Domingos writes, "the
problem is that it's too stupid and already has." *

Re: Guile bugs

[Marko Rauhamaa]

Linas Vepstas <linasvepstas@gmail.com>:
> So if you used GC_malloc_atomic() in your code, then gc will NOT scan
> that region for pointers. guile-2.2 does this correctly for strings (I
> checked) because strings will never contain pointers. I have not
> checked other uses.

The question about mmap(2) is addressed more directly by the README
under <URL: https://github.com/ivmai/bdwgc>:

   Any objects not intended to be collected must be pointed to either
   from other such accessible objects, or from the registers, stack,
   data, or statically allocated bss segments. [...]

   WARNING: pointers inside memory allocated by the standard malloc are
   not seen by the garbage collector. Thus objects pointed to only from
   such a region may be prematurely deallocated. It is thus suggested
   that the standard malloc be used only for memory regions, such as I/O
   buffers, that are guaranteed not to contain pointers to garbage
   collectible memory.

   [...]

   WARNING: the collector does not guarantee to scan thread-local
   storage (e.g. of the kind accessed with pthread_getspecific). The
   collector does scan thread stacks

Now, the question is, can you make your multigigabyte allocation go into
these excluded memory segments? Are you still hit by the pathological GC
behavior you reported?


Marko

Re: Guile bugs

[Linas Vepstas]

Hi Marko,

On Sat, Sep 9, 2017 at 5:31 PM, Marko Rauhamaa <marko@pacujo.net> wrote:

> Linas Vepstas <linasvepstas@gmail.com>:
> > So if you used GC_malloc_atomic() in your code, then gc will NOT scan
> > that region for pointers. guile-2.2 does this correctly for strings (I
> > checked) because strings will never contain pointers. I have not
> > checked other uses.
>
> The question about mmap(2) is addressed more directly by the README
> under <URL: https://github.com/ivmai/bdwgc>:
>
>    Any objects not intended to be collected must be pointed to either
>    from other such accessible objects, or from the registers, stack,
>    data, or statically allocated bss segments. [...]
>
>    WARNING: pointers inside memory allocated by the standard malloc are
>    not seen by the garbage collector. Thus objects pointed to only from
>    such a region may be prematurely deallocated. It is thus suggested
>    that the standard malloc be used only for memory regions, such as I/O
>    buffers, that are guaranteed not to contain pointers to garbage
>    collectible memory.
>
>    [...]
>
>    WARNING: the collector does not guarantee to scan thread-local
>    storage (e.g. of the kind accessed with pthread_getspecific). The
>    collector does scan thread stacks
>
> Now, the question is, can you make your multigigabyte allocation go into
> these excluded memory segments? Are you still hit by the pathological GC
> behavior you reported?
>

What, my app, specifically? Uh yeah. All of my c++ objects go through
plain-old malloc and should be 100% invisible to guile and to bdwgc and
that's perfect.

I do have a layer where guile interacts with the C++ code, and all the
various bits and pieces are correctly declared everywhere.  That code is
about 8-10 years old, its been maintained all that time, its been in use
all that time, and works without issues.

"works without issues" means this: A typical run for me uses some 50-100
threads, maybe 1/2 of these in guile. Due to lock contention, in practice
only 3 to 10 of these threads ever get scheduled, and this is fine, and not
unexpected.  My algos are the opposite of "embrassingly parallel", so low
parallelism is expected. Each processing run goes for a few days to weeks
or over a month, uses 2 to 50GB RAM, does not leak memory, does not crash,
its stable, predictable, does what its supposed to do, terminates cleanly.

It does sometimes run much slower than expected (about 1/2x or 1/4x) and I
don't know why.  I don't usually monitor GC, but when I do, it seems that a
lot of cpu time gets wasted there. Sometimes other people also notice this,
bitch at me, and propose silly solutions. I've been ignoring that as
lower-priority.

Only now it's risen up and is biting hard, and I don't understand how or
why gc gets triggered in guile.  For my app, it seems that it is running
far, far too often, and is severely slowing/preventing forward progress.

--linas

>
>
> Marko
>



-- 
*"The problem is not that artificial intelligence will get too smart and
take over the world," computer scientist Pedro Domingos writes, "the
problem is that it's too stupid and already has." *

Re: Guile bugs

[Taylan Ulrich Bayırlı/Kammer]

Marko Rauhamaa <marko@pacujo.net> writes:

>>>  1. Does Guile offer mmap to Scheme code?
>>
>> It's pretty easy to call C functions from Scheme.
>
> Not good enough. I would have to precompile libraries for different
> target platforms (or require the target environment to have a C
> compiler). Besides, the need is quite generic and shouldn't require an
> extension.

One can also call standard C functions through the same Guile interface
that I used in my example, so one can e.g. turn mmap() into a Scheme
procedure.  open() already happens to exist in the form of 'open-fdes',
and to get the file size one can use (stat:size (stat "filepath")).

The only remaining problem is that mmap's flag constants like PROT_READ,
MAP_SHARED, etc. don't exist in Guile, and can't be reached through FFI
since they're #define constants.  Sans that problem, this works:

)> ,use (system foreign)
)> ,use (rnrs bytevectors)
)> (define mmap
     (pointer->procedure
      '*
      (dynamic-func "mmap" (dynamic-link))                   ;[1]
      (list '* size_t int int int size_t)))
)> (define fd (open-fdes "/home/taylan/todo" O_RDONLY))
)> (define size (stat:size (stat fd)))
)> (define file-ptr (mmap (make-pointer 0) size 1 2 fd 0))   ;[2]
)> (define file-bv (pointer->bytevector file-ptr size))
)> (integer->char (bytevector-u8-ref file-bv 0))
$1 = #\-

[1] When dynamic-link is called with no argument, it returns a "global
    symbol handle" that contains the lib bindings of the Guile process.
    A Guile process is sure to have "mmap" linked in.

[2] The numbers 1 and 2 happen to correspond to PROT_READ and
    MAP_PRIVATE on my system.

As a temporary workaround to the problem that mmap flag constants don't
exist in Guile, one could define them manually for the systems one
intends to support in the foreseeable future.

>>>  3. How do I efficiently encode information in a bytevector in Scheme
>>>     code?
>>
>> What sort of data?
>>
>> I have a library called bytestructures that imitates the C type system
>> within Scheme, to be used on bytevectors that contain data structures
>> generated by C code, though the library is built upon a generic core
>> with which other structures can be declared as well.  Not sure if this
>> helps you:
>>
>> https://github.com/TaylanUB/scheme-bytestructures/
>
> That's precisely it. It would be nice to have it as part of standard
> Guile.

Maybe one day.  These days I don't have much time to work on it.  Maybe
I'll see that I provide a GNU-compliant build system first so people can
install it easily.

In case you use Guix, someone made a package for it, so you could
install it through there too.

> I wonder, though, if doing that is fast enough in Scheme code.

Since Guile supports syntax-case, the "macro API" of the library can be
used so the library itself adds absolutely zero runtime overhead.

(This API puts a few limits on the full feature set, but the standard
modules that mimic C types don't use those features.)


Taylan

Re: Guile bugs

[Marko Rauhamaa]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer"):

> Marko Rauhamaa <marko@pacujo.net> writes:
>> I wonder, though, if doing that is fast enough in Scheme code.
>
> Since Guile supports syntax-case, the "macro API" of the library can
> be used so the library itself adds absolutely zero runtime overhead.

No doubt it's as fast as Guile can be for the API. However, you'll need
to have the encoding/decoding machinery ultimately executed in Scheme.

Compare this with Python's struct.(un)pack(), for example, which lets
the "machine code" handle the bulk encoding/decoding of a record.

Instead of binary, one could use S-expressions, of course, but then
you'd need to know the maximum length of the encoding up front.

Heck, maybe your bulk RAM store should be implemented with straight
non-GC malloc...


Marko

Re: Guile bugs

[Matt Wette]

> On Jul 20, 2017, at 9:22 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
> 
> ludo@gnu.org (Ludovic Courtès):
> 
>> I’m very skeptical about this use case (I’d use ‘mmap’ and get a
>> bytevector.)
> 
> Please elaborate.
> 
> 1. Does Guile offer mmap to Scheme code?
> 
> 2. What would prevent Guile's GC from scanning the mmapped area for
>    pointers?
> 
> 3. How do I efficiently encode information in a bytevector in Scheme
>    code?
> 
> 
> Marko
> 

I have submitted mmap as a Guile wishlist item.  See https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27782

Re: Guile bugs

[Linas Vepstas]

Hi Ludo, a very late reply...

On Thu, Jul 20, 2017 at 10:28 AM, Ludovic Courtès <ludo@gnu.org> wrote:

>
> > String handling in guile is a disaster area: If I give it a
> > 10-megabyte-long string in utf8, it promptly tries to convert all of that
> > string in utf32, for utterly pointless reasons. This just makes it slow.
>
> We’ve discussed it before, and while I agree that there’s much room for
> improvement, I’m very skeptical about this use case


It's OK to be skeptical.  Let me briefly sketch a not-uncommon thought
process.

1) There is a need to pass messages between subsystems running on different
machines.

2) Solutions include ROS, ZeroMQ,  google protocol buffers... whatever. All
require lots of work, a learning curve, complexity, etc.

3) Wait!  I know! I will just send around ascii (utf8) strings that are
guile programs!  Just use the guile repl server, connect to it, and send
some guile string!  No muss, no fuss, no coding, no learning curve, simple
easy ...  You can use netcat to drive things!  echo (display "hello
world\n") | nc 1.2.3.4 37146

4) Solutions 3 works great, so lets scale it up!  Send more messages,
larger messages, more quickly.

5) unhappiness.  I reported one here, but it needs follow-up on my part.
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27234

Other issues turn out to be that the REPL server is slow; I replaced it
with a hand-coded blob that is wayyy faster.  On a related note, there is
no login/authentication for the repl server, but this is not a guile issue,
its a generic issue.  Amazingly, no one has ever created a generic login
server, aside from ssh... and there's no easy way to glue ssh to the guile
repl.

--linas

-- 
*"The problem is not that artificial intelligence will get too smart and
take over the world," computer scientist Pedro Domingos writes, "the
problem is that it's too stupid and already has." *

Re: Guile bugs

[Ludovic Courtès]

Hi,

Linas Vepstas <linasvepstas@gmail.com> skribis:

> 1) There is a need to pass messages between subsystems running on different
> machines.
>
> 2) Solutions include ROS, ZeroMQ,  google protocol buffers... whatever. All
> require lots of work, a learning curve, complexity, etc.
>
> 3) Wait!  I know! I will just send around ascii (utf8) strings that are
> guile programs!  Just use the guile repl server, connect to it, and send
> some guile string!  No muss, no fuss, no coding, no learning curve, simple
> easy ...  You can use netcat to drive things!  echo (display "hello
> world\n") | nc 1.2.3.4 37146

Then you’re sending sexps, not strings, and you’re reading them with
‘read’ from a port, as opposed to loading whole strings in memory.

Also, the REPL server is not necessarily the right thing here.  You
conceptually want a REPL, but not Guile’s full-blown REPL, which can be
hard to deal with (you need to parse the prompts, determine if you’re in
a recursive REPL, whether an exception occurred, etc.)

In the Shepherd, I implemented a very simple REPL that is just enough to
communicate evaluation results to the client, and to distinguish return
values from exceptions.  This does not use Guile’s REPL server, only
‘read’, ‘eval’, ‘write’.

HTH,
Ludo’.

Re: Guile bugs

[Linas Vepstas]

Hi Ludo,

Perhaps I was too brief in my remarks. By a networked REPL, I meant this:

https://www.gnu.org/software/guile/manual/html_node/REPL-Servers.html

For accidental, historical reasons, (i.e. I did not know the above existed)
I wrote my own variant, which accepts network connections, handles guile in
full, including the printing of exception frames, handles async printing to
stdio (i.e. to the network connection) so that long-running functions can
print, and the user can see that output, unbuffered.   Its full
multi-threaded, uses a pool of threads pre-initialized in guile, hot and
ready to go, to avoid the overhead of initializing and entering guile.  Its
fast, robust, it works.

The stuff coming over the network sockets are bytes, not s-exps. Since none
of the bytes are ever zero, they are effectively C/C++ strings, and are
handled as such. These C strings are sent to  scm_eval_string() wrapped
by scm_c_catch().

As mentioned, this customized blob of code is easily 3x or 5x faster
than (use-modules (system repl server)), and also more  robust ...
sometimes (system repl server) seems to lock up or hang or something, I
have not delved into it.

One of my complaints is that the C strings sent to scm_eval_string() go
through some pointless layers that convert utf8 to utf32, and then back
again. I do not recall the details. There are some obvious optimizations
that can be performed. Its more than an afternoon or two but less than a
week or two.  I am not motivated to submit patches, as I fear these will
just provoke arguments and be rejected.

The custom guile network repl server can be viewed at the URL's below. To
preserve your sanity, you probably don't want to study it.  But in the
spirit of open source, here it is:

https://github.com/opencog/atomspace/blob/master/opencog/guile/SchemeEval.cc

The network REPL shell is here:
https://github.com/opencog/opencog/blob/master/opencog/cogserver/shell/GenericShell.cc

we've layered both guile and python on it.  Python is annoying because
executing python from repl behaves differently than executing python from a
file. Not my fault; its built into the python language definition. Idiots.

--linas





On Sun, Sep 10, 2017 at 8:11 AM, Ludovic Courtès <ludo@gnu.org> wrote:

> Hi,
>
> Linas Vepstas <linasvepstas@gmail.com> skribis:
>
> > 1) There is a need to pass messages between subsystems running on
> different
> > machines.
> >
> > 2) Solutions include ROS, ZeroMQ,  google protocol buffers... whatever.
> All
> > require lots of work, a learning curve, complexity, etc.
> >
> > 3) Wait!  I know! I will just send around ascii (utf8) strings that are
> > guile programs!  Just use the guile repl server, connect to it, and send
> > some guile string!  No muss, no fuss, no coding, no learning curve,
> simple
> > easy ...  You can use netcat to drive things!  echo (display "hello
> > world\n") | nc 1.2.3.4 37146
>
> Then you’re sending sexps, not strings, and you’re reading them with
> ‘read’ from a port, as opposed to loading whole strings in memory.
>
> Also, the REPL server is not necessarily the right thing here.  You
> conceptually want a REPL, but not Guile’s full-blown REPL, which can be
> hard to deal with (you need to parse the prompts, determine if you’re in
> a recursive REPL, whether an exception occurred, etc.)
>
> In the Shepherd, I implemented a very simple REPL that is just enough to
> communicate evaluation results to the client, and to distinguish return
> values from exceptions.  This does not use Guile’s REPL server, only
> ‘read’, ‘eval’, ‘write’.
>
> HTH,
> Ludo’.
>



-- 
*"The problem is not that artificial intelligence will get too smart and
take over the world," computer scientist Pedro Domingos writes, "the
problem is that it's too stupid and already has." *

Re: Guile bugs

[Ludovic Courtès]

Hello,

Linas Vepstas <linasvepstas@gmail.com> skribis:

> The stuff coming over the network sockets are bytes, not s-exps. Since none
> of the bytes are ever zero, they are effectively C/C++ strings, and are
> handled as such. These C strings are sent to  scm_eval_string() wrapped
> by scm_c_catch().

I don’t know to what extent that is applicable to your software, but my
recommendation would be to treat that network socket as a Scheme port,
pass it to ‘read’, and pass the result to ‘eval’ (as opposed to reading
the whole string from C++ and passing it to ‘scm_eval_string’.)

HTH,
Ludo’.

Re: Guile bugs

[Marko Rauhamaa]

ludo@gnu.org (Ludovic Courtès):

> I don’t know to what extent that is applicable to your software, but my
> recommendation would be to treat that network socket as a Scheme port,
> pass it to ‘read’, and pass the result to ‘eval’ (as opposed to reading
> the whole string from C++ and passing it to ‘scm_eval_string’.)

Octet stream sockets break the stream at arbitrary locations. Eval isn't
going to like it if you blindly hand bytes over to it. You first need to
break the stream into complete S-expressions. Note also that the socket
might not even respect UTF-8 boundaries so you can't assume you can read
the bytes as text.

Secondly, this lucrative pattern can be very dangerous as it allows
arbitrary code to be executed at the peer end. In most applications, it
is better to interpret the S-expressions explicitly.


Marko

Re: Guile bugs

[Ludovic Courtès]

Hi Marko,

Marko Rauhamaa <marko@pacujo.net> skribis:

> ludo@gnu.org (Ludovic Courtès):
>
>> I don’t know to what extent that is applicable to your software, but my
>> recommendation would be to treat that network socket as a Scheme port,
>> pass it to ‘read’, and pass the result to ‘eval’ (as opposed to reading
>> the whole string from C++ and passing it to ‘scm_eval_string’.)
>
> Octet stream sockets break the stream at arbitrary locations. Eval isn't
> going to like it if you blindly hand bytes over to it. You first need to
> break the stream into complete S-expressions.

Sure, that’s what the ‘read’ part above does.

> Secondly, this lucrative pattern can be very dangerous as it allows
> arbitrary code to be executed at the peer end. In most applications, it
> is better to interpret the S-expressions explicitly.

Definitely.  I’m not saying eval-over-the-network is a good thing to do
in general, mind you!  :-)

Ludo’.

Re: Guile bugs

[Linas Vepstas]

On Mon, Sep 11, 2017 at 2:26 AM, Ludovic Courtès <ludo@gnu.org> wrote:

> Hello,
>
> Linas Vepstas <linasvepstas@gmail.com> skribis:
>
> > The stuff coming over the network sockets are bytes, not s-exps. Since
> none
> > of the bytes are ever zero, they are effectively C/C++ strings, and are
> > handled as such. These C strings are sent to  scm_eval_string() wrapped
> > by scm_c_catch().
>
> I don’t know to what extent that is applicable to your software, but my
> recommendation would be to treat that network socket as a Scheme port,
> pass it to ‘read’, and pass the result to ‘eval’ (as opposed to reading
> the whole string from C++ and passing it to ‘scm_eval_string’.)
>

Why?  What advantage does this offer?

Its not clear that guile eval is smart enough to manage a network socket --
if the user starts a long-running process with intermittent prints, will it
send that to the socket?  What if the user hits cntrl-C in the middle of it
all? What if the code that came over the socket happened to throw an
exception?

I've had to deal with all of these issues in the past, and have a stable
code base; but if I had to start all over again, its not clear that these
issues have gone away.  I mean, eval was designed to eval -- it was not
designed to support multi-threaded, concurrent network operations, right?

To support my point: the default guile network REPL server is painfully
slow, and frequently crashes/hangs. It works well enough to do some demos
but is not stable enough for production use ... if its just read+eval, that
might explain why its unstable.

--linas

-- 
*"The problem is not that artificial intelligence will get too smart and
take over the world," computer scientist Pedro Domingos writes, "the
problem is that it's too stupid and already has." *

Re: Guile bugs

[Ludovic Courtès]

Linas Vepstas <linasvepstas@gmail.com> skribis:

> On Mon, Sep 11, 2017 at 2:26 AM, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> Hello,
>>
>> Linas Vepstas <linasvepstas@gmail.com> skribis:
>>
>> > The stuff coming over the network sockets are bytes, not s-exps. Since
>> none
>> > of the bytes are ever zero, they are effectively C/C++ strings, and are
>> > handled as such. These C strings are sent to  scm_eval_string() wrapped
>> > by scm_c_catch().
>>
>> I don’t know to what extent that is applicable to your software, but my
>> recommendation would be to treat that network socket as a Scheme port,
>> pass it to ‘read’, and pass the result to ‘eval’ (as opposed to reading
>> the whole string from C++ and passing it to ‘scm_eval_string’.)
>>
>
> Why?  What advantage does this offer?

It avoids copies and conversions, which is big deal if you deal with
very big strings.

> Its not clear that guile eval is smart enough to manage a network socket --
> if the user starts a long-running process with intermittent prints, will it
> send that to the socket?  What if the user hits cntrl-C in the middle of it
> all? What if the code that came over the socket happened to throw an
> exception?

These are important considerations, but it’s not eval’s business IMO.
Instead, I suggest building your own protocol around it, and having a
way in that protocol to report both exceptions and normal returns.

> I've had to deal with all of these issues in the past, and have a stable
> code base; but if I had to start all over again, its not clear that these
> issues have gone away.  I mean, eval was designed to eval -- it was not
> designed to support multi-threaded, concurrent network operations, right?

Right.

> To support my point: the default guile network REPL server is painfully
> slow, and frequently crashes/hangs. It works well enough to do some demos
> but is not stable enough for production use ... if its just read+eval, that
> might explain why its unstable.

I’ve never noticed slowness of the REPL server, nor crashes.

That said, if you run a REPL server in a separate thread and mutate the
global state of the program, you could possibly crash it—no wonders
here.

Likewise, the REPL server is meant to be used for debugging on
localhost.  If you talk to a REPL server over the network with high
latency, it’s going to be slow, not surprisingly.

So yes, I find the REPL server to be a really pleasant tool when
debugging an application locally, but that’s all it is—it’s not a remote
procedure call framework or anything like that.

Thanks,
Ludo’.

Re: Guile bugs

[Linas Vepstas]

Hi Ludo,

On Fri, Sep 15, 2017 at 3:56 PM, Ludovic Courtès <ludo@gnu.org> wrote:

> Linas Vepstas <linasvepstas@gmail.com> skribis:
>
> > On Mon, Sep 11, 2017 at 2:26 AM, Ludovic Courtès <ludo@gnu.org> wrote:
> >
> >> Hello,
> >>
> >> Linas Vepstas <linasvepstas@gmail.com> skribis:
> >>
> >> > The stuff coming over the network sockets are bytes, not s-exps. Since
> >> none
> >> > of the bytes are ever zero, they are effectively C/C++ strings, and
> are
> >> > handled as such. These C strings are sent to  scm_eval_string()
> wrapped
> >> > by scm_c_catch().
> >>
> >> I don’t know to what extent that is applicable to your software, but my
> >> recommendation would be to treat that network socket as a Scheme port,
> >> pass it to ‘read’, and pass the result to ‘eval’ (as opposed to reading
> >> the whole string from C++ and passing it to ‘scm_eval_string’.)
> >>
> >
> > Why?  What advantage does this offer?
>
> It avoids copies and conversions, which is big deal if you deal with
> very big strings.
>
> > Its not clear that guile eval is smart enough to manage a network socket
> --
> > if the user starts a long-running process with intermittent prints, will
> it
> > send that to the socket?  What if the user hits cntrl-C in the middle of
> it
> > all? What if the code that came over the socket happened to throw an
> > exception?
>
> These are important considerations, but it’s not eval’s business IMO.
> Instead, I suggest building your own protocol around it, and having a
> way in that protocol to report both exceptions and normal returns.
>

Well, yes, this is exactly what I've done.

This conversation is frustrating: either piping read to eval is the right
thing to do, in which case eval must handle network connections correctly,
or else piping read to eval is the wrong thing to do.  You can't have it
both ways.


> > I've had to deal with all of these issues in the past, and have a stable
> > code base; but if I had to start all over again, its not clear that these
> > issues have gone away.  I mean, eval was designed to eval -- it was not
> > designed to support multi-threaded, concurrent network operations, right?
>
> Right.
>
> > To support my point: the default guile network REPL server is painfully
> > slow, and frequently crashes/hangs. It works well enough to do some demos
> > but is not stable enough for production use ... if its just read+eval,
> that
> > might explain why its unstable.
>
> I’ve never noticed slowness of the REPL server, nor crashes.
>

You are probably using it only very lightly, and not in a high-load systems
environment. It runs maybe 5x slower than my current guile shell server,
and it is very definitely unstable and crashy.

In my environment, I am sending it approximately from one up to twenty
scheme expressions every second, with a new socket opened for each scheme
expression. This goes on for days or weeks. I am using a custom guile
server written in C++, which accepts network connections, reads bytes from
the network, and sends them to scm_eval_string(). It mostly works fine,
with a couple of problems: there seems to be a pointless utf8-utf32
conversion, which started this email chain.

There also seems to be some sort of very rare race condition in the
compiler that leads to corruption inside of guile. I believe that this can
be triggered by starting twenty threads (for example) and then compiling
and running fairly short programs in each thread. By "fairly short" I mean
"less than 5-10 lines of code", and which compute and return answers in
less than a tenth of a second. Doing this for a few hours eventually causes
guile to hang in a spinloop, trying to read some guile-internal structure
that has invalid data in it. I opened a bug report for this a month or two
ago, but did not supply an easy-to-trigger test case.

I tried replacing my guile network server with the REPL shell, and
discovered that the REPL server is much much slower; I don't recall exactly
how I measured the 5x number, but that was from an actual measurement.   I
don't think the REPL server can handle 20 network connections per second.

I remember hypothesizing that guile was being re-initialized for every
network connection. Obviously, this is wasteful and slow.

Entering guile is a large bottleneck.  I once measured this, and I think it
takes approximately 200 microseconds to enter guile, which implies a
maximum limit of about 5K guile evaluations per second, when using the
simple-minded design of having the C code enter guile each time before
evaluation an expression.  By contrast, python (cython) can be entered in
10 or 20 microseconds.

The test case here is how many times per second can one eval some simple
expression, e.g. (+ 2 2) or the equivalent of that in python.

The solution for the heavy cost of entering guile is to create a pool for a
few dozen threads, enter guile in each, and then never exit -- just return
threads to the thread pool, when the eval is completed, and the thread is
no longer needed.  This cuts the  200 microseconds overhead to zero, and
what one is then left with is the cost of calling scm_eval_string().  I did
measure that too, but I don't recall the numbers.


> That said, if you run a REPL server in a separate thread and mutate the
> global state of the program, you could possibly crash it—no wonders
> here.
>

Yes, well, I would call that a bug! It feels like you are trying to blame
me for a guile bug -- its not my fault that it crashes!

I did not look very carefully, and don't recall what the stack traces
looked like, but I got the impression that there were race conditions in
guile init, and how it interacted with the sockets.

Likewise, the REPL server is meant to be used for debugging on
> localhost.  If you talk to a REPL server over the network with high
> latency, it’s going to be slow, not surprisingly.
>

The performance problem was not the  latency, it was the number of
connections it could accept.

I'll say it again: I have a different network server that is 5x faster than
the REPL server, and it works, it is stable.

For reasons completely unrelated to guile, I would like to declare my
network server deprecated and obsolete.  However, I cannot do this, because
the guile REPL server is not yet good enough to be an adequate replacement.

--linas

>
> So yes, I find the REPL server to be a really pleasant tool when
> debugging an application locally, but that’s all it is—it’s not a remote
> procedure call framework or anything like that.
>

> Thanks,
> Ludo’.
>



-- 
*"The problem is not that artificial intelligence will get too smart and
take over the world," computer scientist Pedro Domingos writes, "the
problem is that it's too stupid and already has." *

Re: Guile bugs

[Chris Vine]

On Tue, 19 Sep 2017 19:04:21 +0800
Linas Vepstas <linasvepstas@gmail.com> wrote:
> On Fri, Sep 15, 2017 at 3:56 PM, Ludovic Courtès <ludo@gnu.org> wrote:
[snip]
> > That said, if you run a REPL server in a separate thread and mutate
> > the global state of the program, you could possibly crash it—no
> > wonders here.
> >  
> 
> Yes, well, I would call that a bug! It feels like you are trying to
> blame me for a guile bug -- its not my fault that it crashes!
> 
> I did not look very carefully, and don't recall what the stack traces
> looked like, but I got the impression that there were race conditions
> in guile init, and how it interacted with the sockets.

Your problems are clearly at least partly at a deeper level, and you
may already know what I say below on thread safety, but:

(i) you cannot auto-compile the same file concurrently in two different
threads, for obvious reasons: the generated 'go' file will be a mash up
to two different compilations.  I have not had a problem with
user-compiling files concurrently to different locations, although I
accept that you may have (although you say this is rare).

(ii) prior to guile-2.0.10, guile initialization was not thread safe -
you needed to have entered scm_with_guile() at least once in a single
thread, and completed initialization (say by having scm_with_guile()
return), before starting any new threads which might invoke
scm_with_guile() or evaluate any scheme expressions.

(iii) modifying most system non-fluid/non-parameter globals is thread
unsafe.  In particular, loading modules concurrently for the first time
will lead to problems (you have to load all required modules up front
in a single thread before starting any new threads which might
independently try to load the modules).  Ports are also not thread safe
(rightly so in my view), and I believe that extends to the global stdout
and stdin ports.

Chris

Re: Guile bugs

[Chris Vine]

On Tue, 19 Sep 2017 21:18:40 +0100
Chris Vine <vine35792468@gmail.com> wrote:
> ... Ports are also not thread safe (rightly so in my view), and I
> believe that extends to the global stdout and stdin ports.

By that I mean that any one port object is not thread safe against
concurrent use by two or more different threads, without external
synchronization.

Re: Guile bugs

[Nala Ginrut]

Hi folks!
Now that we all have such need, why bother to polish the nasty REPL way for
remote execution?
Let's discuss serializable-continuations which I always expect very much!


2017年9月20日 上午4:38，"Chris Vine" <vine35792468@gmail.com>写道：

> On Tue, 19 Sep 2017 21:18:40 +0100
> Chris Vine <vine35792468@gmail.com> wrote:
> > ... Ports are also not thread safe (rightly so in my view), and I
> > believe that extends to the global stdout and stdin ports.
>
> By that I mean that any one port object is not thread safe against
> concurrent use by two or more different threads, without external
> synchronization.
>
>

Re: How to make GNU Guile more successful

[sirgazil]

sirgazil says:
> When I started, my plan was to follow the course in Guile instead,
> since the course is about the design method, and not the tools, but
> I had to stop very early when I had to do some projects involving
> graphics and animation.
>
> Personally, I don't feel the need for something like DrRacket,
> but it would have been nice to be able to find a Guile "canvas" or a
> 2D game library to play easily with basic graphics and animations.

You might want to try these:
https://dthompson.us/projects/chickadee.html
https://dthompson.us/projects/sly.html

Hi, Rodger.

Yes, I know those. I tried Sly, but then realized it requires OpenGL, which is broken in the system I use (nvidia seems incompatible with free software). I haven't tried chickadee, but, if I remember correctly, it has the same requirement.

So I still need something that works without OpenGL, like Racket's universe module or Pygame :)

Re: How to make GNU Guile more successful

[sirgazil]

-------- Original Message --------
Subject: Re: How to make GNU Guile more successful
Local Time: 17 de febrero de 2017 9:18 PM
UTC Time: 18 de febrero de 2017 2:18
From: thylakoid@openmailbox.org
To: sirgazil <lizagris@protonmail.com>

On 2017-02-17 5:04 pm, sirgazil wrote:
>> sirgazil says:
>>
>>> When I started, my plan was to follow the course in Guile
>> instead,
>>
>>> since the course is about the design method, and not the tools,
>> but
>>
>>> I had to stop very early when I had to do some projects involving
>>
>>> graphics and animation.
>>
>>>
>>
>>> Personally, I don't feel the need for something like DrRacket,
>>
>>> but it would have been nice to be able to find a Guile "canvas"
>> or a
>>
>>> 2D game library to play easily with basic graphics and
>> animations.
>>
>> You might want to try these:
>>
>> https://dthompson.us/projects/chickadee.html
>>
>> https://dthompson.us/projects/sly.html
>
> Hi, Rodger.
>
> Yes, I know those. I tried Sly, but then realized it requires OpenGL,
> which is broken in the system I use (nvidia seems incompatible with
> free software). I haven't tried chickadee, but, if I remember
> correctly, it has the same requirement.
>
> So I still need something that works without OpenGL, like Racket's
> universe module or Pygame :)

I'm not familiar with Racket's universe or Pygame, but I have a couple
more suggestions that you might consider.

There is Guile-Cairo if you just want a basic canvas.
http://www.nongnu.org/guile-cairo/
See also: https://www.cairographics.org/

Maybe you can use Guile-SDL or Guile-SDL2, which is what Sly and
Chickadee are built on, but like normal SDL, you can use it with or
without OpenGL.
And it will also provides more of the API needed for simple games.
https://www.gnu.org/software/guile-sdl/
https://dthompson.us/projects/guile-sdl2.html
See also: http://libsdl.org/


Well, I considered both when I started the course :)

The problem is their level is not as high as I'd like. The course I'm following is for beginner programmers, so they use higher level libraries for drawing and animating. For example, you can write a checker pattern function like this:

(define (checker-pattern color-a color-b)
(above (beside (square 20 "solid" color-a) (square 20 "solid" color-b))
(beside (square 20 "solid" color-b) (square 20 "solid" color-a))))

The functions "above", "beside", and "square" come with a Racket's image library. Guile cairo's interface is lower level, it feels like reading the SVG specification, but without the geometric shapes.

I think the situation with SDL is similar.

Maybe when I finish the course I can start designing a very high level library using the existing guile-cairo and guile-sdl2 :)

Re: How to make GNU Guile more successful

[David Pirotte]

[-- Attachment #1: Type: text/plain, Size: 1608 bytes --]

Heya,

> There is Guile-Cairo if you just want a basic canvas.
> http://www.nongnu.org/guile-cairo/
> See also: https://www.cairographics.org/

Which you can use from Guile-Clutter [1], a must. I maintain a personal toolbox,
called grip,  which as some guile-clutter examples [2], some using cairo (the
spline, the clock ...)

	https://www.gnu.org/software/guile-gnome/clutter/	[1]
	http://www.nongnu.org/grip/examples.html			[2]

> The problem is their level is not as high as I'd like. The course I'm following is
> for beginner programmers, so they use higher level libraries for drawing and
> animating. For example, you can write a checker pattern function like this:

> (define (checker-pattern color-a color-b)
> (above (beside (square 20 "solid" color-a) (square 20 "solid" color-b))
> (beside (square 20 "solid" color-b) (square 20 "solid" color-a))))

> The functions "above", "beside", and "square" come with a Racket's image library.
> Guile cairo's interface is lower level, it feels like reading the SVG
> specification, but without the geometric shapes.

I'm working on Guile-CV, a computer vision functional programming library for Guile,
which provides a high level API.  I uploaded a tarball here (for GNU evaluators),
here:

	http://download.savannah.gnu.org/releases/grip/guile-cv/guile-cv-0.1.0.tar.gz
	http://download.savannah.gnu.org/releases/grip/guile-cv/guile-cv-0.1.0.tar.gz.sig

	[ GPG Key: A3057AD7
	[ gpg --keyserver keys.gnupg.net --recv-keys A3057AD7

	[ I will upload a new version within the next few days, stay tuned...

Chers,
David

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: How to make GNU Guile more successful

[sirgazil]

-------- Original Message --------
Subject: Re: How to make GNU Guile more successful
Local Time: 18 de febrero de 2017 11:55 AM
UTC Time: 18 de febrero de 2017 16:55
From: david@altosw.be
To: sirgazil <lizagris@protonmail.com>
Rodger Fox <thylakoid@openmailbox.org>, guile-user@gnu.org <guile-user@gnu.org>

Heya,

> There is Guile-Cairo if you just want a basic canvas.
> http://www.nongnu.org/guile-cairo/
> See also: https://www.cairographics.org/

Which you can use from Guile-Clutter [1], a must. I maintain a personal toolbox,
called grip, which as some guile-clutter examples [2], some using cairo (the
spline, the clock ...)

https://www.gnu.org/software/guile-gnome/clutter/ [1]
http://www.nongnu.org/grip/examples.html [2]

> The problem is their level is not as high as I'd like. The course I'm following is
> for beginner programmers, so they use higher level libraries for drawing and
> animating. For example, you can write a checker pattern function like this:

> (define (checker-pattern color-a color-b)
> (above (beside (square 20 "solid" color-a) (square 20 "solid" color-b))
> (beside (square 20 "solid" color-b) (square 20 "solid" color-a))))

> The functions "above", "beside", and "square" come with a Racket's image library.
> Guile cairo's interface is lower level, it feels like reading the SVG
> specification, but without the geometric shapes.

I'm working on Guile-CV, a computer vision functional programming library for Guile,
which provides a high level API. I uploaded a tarball here (for GNU evaluators),
here:

http://download.savannah.gnu.org/releases/grip/guile-cv/guile-cv-0.1.0.tar.gz
http://download.savannah.gnu.org/releases/grip/guile-cv/guile-cv-0.1.0.tar.gz.sig

[ GPG Key: A3057AD7
[ gpg --keyserver keys.gnupg.net --recv-keys A3057AD7

[ I will upload a new version within the next few days, stay tuned...

Chers,
David

Thanks, David, I'll take a look at those :)

Re: How to make GNU Guile more successful

[David Pirotte]

[-- Attachment #1: Type: text/plain, Size: 349 bytes --]

Hello,

> Thanks, David, I'll take a look at those :)

Welcome, for info, I've uploaded a new version:

	http://download.savannah.gnu.org/releases/grip/guile-cv/
		guile-cv-0.1.1.tar.gz
		guile-cv-0.1.1.tar.gz.sig

	[ GPG Key: A3057AD7
	[ gpg --keyserver keys.gnupg.net --recv-keys A3057AD7

	See NEWS and INSTALL...

Cheers,
David

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: How to make GNU Guile more successful

[Michael Vehrs]

As a late-comer to this discussion, here are my two cents. The thing I 
miss most is a central package repository like Eggs Unlimited 
(http://wiki.call-cc.org/chicken-projects/egg-index-4.html), or the 
Racket Package List (http://pkgs.racket-lang.org/), or CPAN, of course. 
Sure, a bespoke package manager might be nifty, but a single curated 
list of packages would be a game-changer.


Regards


Michael

Re: How to make GNU Guile more successful

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1113 bytes --]


Michael Vehrs <Michael.Burschik@gmx.de> writes:

> As a late-comer to this discussion, here are my two cents. The thing I 
> miss most is a central package repository like Eggs Unlimited 
> (http://wiki.call-cc.org/chicken-projects/egg-index-4.html), or the 
> Racket Package List (http://pkgs.racket-lang.org/), or CPAN, of course. 
> Sure, a bespoke package manager might be nifty, but a single curated 
> list of packages would be a game-changer.

In theory we have guildhall for this: https://github.com/ijp/guildhall

In practice it does not provide a web interface for uploading packages.

If you want to do something truly exciting, you could take wingos fibers
and build a high performance web interface for guildhall with them.

This is something I’d love to do but I fear that it’s not high enough in
my todo list that I’ll actually get it done.¹

Best wishes,
Arne

¹: I became Freenet Release Manager a few months ago. That took away the
   last free time I could allocate to new challenging projects.
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

Re: How to make GNU Guile more successful

[Michael Vehrs]

On 02/20/2017 09:41 PM, Arne Babenhauserheide wrote:
> Michael Vehrs <Michael.Burschik@gmx.de> writes:
>
>> As a late-comer to this discussion, here are my two cents. The thing I
>> miss most is a central package repository like Eggs Unlimited
>> (http://wiki.call-cc.org/chicken-projects/egg-index-4.html), or the
>> Racket Package List (http://pkgs.racket-lang.org/), or CPAN, of course.
>> Sure, a bespoke package manager might be nifty, but a single curated
>> list of packages would be a game-changer.
> In theory we have guildhall for this: https://github.com/ijp/guildhall

In theory, yes. But there isn't actually very much available in the 
repository.

>
> In practice it does not provide a web interface for uploading packages.
>
> If you want to do something truly exciting, you could take wingos fibers
> and build a high performance web interface for guildhall with them.

High performance is not really important in this case. We are not 
talking about gazillions of npm packages.

>
> This is something I’d love to do but I fear that it’s not high enough in
> my todo list that I’ll actually get it done.¹
>
> Best wishes,
> Arne
>
> ¹: I became Freenet Release Manager a few months ago. That took away the
>     last free time I could allocate to new challenging projects.

If someone had a realistic plan of building something like that, I might 
be able to contribute. I am not going to tackle it alone.


Regards

Michael

Re: How to make GNU Guile more successful

[Arne Babenhauserheide]

Michael Vehrs writes:

> On 02/20/2017 09:41 PM, Arne Babenhauserheide wrote:
>> Michael Vehrs <Michael.Burschik@gmx.de> writes:
>>
>>> As a late-comer to this discussion, here are my two cents. The thing I
>>> miss most is a central package repository like Eggs Unlimited
>>> (http://wiki.call-cc.org/chicken-projects/egg-index-4.html), or the
>>> Racket Package List (http://pkgs.racket-lang.org/), or CPAN, of course.
>>> Sure, a bespoke package manager might be nifty, but a single curated
>>> list of packages would be a game-changer.
>> In theory we have guildhall for this: https://github.com/ijp/guildhall
>
> In theory, yes. But there isn't actually very much available in the 
> repository.

I think the main reasons for that are (a) that it’s unmaintained (you
can’t just get your package added), (b) that it’s underdocumented (it
doesn’t say how to add a package or create a repo in less than 5 steps),
and (c) that there is no web interface for uploading a package.

>> In practice it does not provide a web interface for uploading packages.
>>
>> If you want to do something truly exciting, you could take wingos fibers
>> and build a high performance web interface for guildhall with them.
>
> High performance is not really important in this case. We are not 
> talking about gazillions of npm packages.

That’s true. That’s why I wrote exciting :)

You could also use the existing (web server) tools to build such a site.

Best wishes,
Arne

Re: How to make GNU Guile more successful

[Amirouche]

Le 21/02/2017 à 18:18, Arne Babenhauserheide a écrit :
> Michael Vehrs writes:
>
>> On 02/20/2017 09:41 PM, Arne Babenhauserheide wrote:
>>> Michael Vehrs <Michael.Burschik@gmx.de> writes:
>>>> As a late-comer to this discussion, here are my two ce
>>> In practice it does not provide a web interface for uploading packages.
>>>
>>> If you want to do something truly exciting, you could take wingos fibers
>>> and build a high performance web interface for guildhall with them.
>> High performance is not really important in this case. We are not
>> talking about gazillions of npm packages.
> That’s true. That’s why I wrote exciting :)
>
> You could also use the existing (web server) tools to build such a site.

We can make it exciting and use fibers web server nonetheless.

The interface of the *fibers* web server is the same as guile web
server interface.

Here is it [0]:

```

(use-modules (fibers web server))

(define (handler request body)
   (values '((content-type . (text/plain)))
           "Hello, World!"))

(run-server handler)

```

Basically you just need to change the use-modules line in your current
project.

[0]

Re: How to make GNU Guile more successful

[Mike Gran]

>>> If you want to do something truly exciting, you could take wingos fibers

>>> and build a high performance web interface for guildhall with them.
>> High performance is not really important in this case. We are not
>> talking about gazillions of npm packages.
> That’s true. That’s why I wrote exciting :)
>
> You could also use the existing (web server) tools to build such a site.



The fastest thing would be to create a git project with a
{meta}/{maintainer}/{project} type structure and let people hack away unsupervised
in their {maintainer} subdirectories.  Then add a thin libraries that
allows one to pick a specific Git commit UUID as a version number.

It would be chaos, but, we don't need professionalism when there is nothing
about which to be professional.


Or, to be fancier, clone metacpan. It exists.


-Mike

Re: How to make GNU Guile more successful

[Amirouche]

Le 21/02/2017 à 19:31, Mike Gran a écrit :
>
>>>> If you want to do something truly exciting, you could take wingos fibers
>>>> and build a high performance web interface for guildhall with them.
>>> High performance is not really important in this case. We are not
>>> talking about gazillions of npm packages.
>> That’s true. That’s why I wrote exciting :)
>>
>> You could also use the existing (web server) tools to build such a site.
>
>
> The fastest thing would be to create a git project with a
> {meta}/{maintainer}/{project} type structure and let people hack away unsupervised
> in their {maintainer} subdirectories.  Then add a thin libraries that
> allows one to pick a specific Git commit UUID as a version number.
>
> It would be chaos, but, we don't need professionalism when there is nothing
> about which to be professional.
>
>
> Or, to be fancier, clone metacpan. It exists.

Sorry, I don't understand you want to use git as the package index?

Re: How to make GNU Guile more successful

[Mike Gran]

On Tuesday, February 21, 2017 10:33 AM, Amirouche <amirouche@hypermove.net> wrote:


> Sorry, I don't understand you want to use git as the package index?


Just have a top-level README.md that everyone edits as the package index.
It would suffice until so long as the number of packages remains in the low
hundreds.

Re: How to make GNU Guile more successful

[Amirouche]

Le 21/02/2017 à 07:01, Michael Vehrs a écrit :
> On 02/20/2017 09:41 PM, Arne Babenhauserheide wrote:
>> Michael Vehrs <Michael.Burschik@gmx.de> writes:
>>
>>> As a late-comer to this discussion, here are my two cents. The thing I
>>> miss most is a central package repository like Eggs Unlimited
>>> (http://wiki.call-cc.org/chicken-projects/egg-index-4.html), or the
>>> Racket Package List (http://pkgs.racket-lang.org/), or CPAN, of course.
>>> Sure, a bespoke package manager might be nifty, but a single curated
>>> list of packages would be a game-changer.
>> In theory we have guildhall for this: https://github.com/ijp/guildhall
>
> In theory, yes. But there isn't actually very much available in the 
> repository.
>
>>
>> In practice it does not provide a web interface for uploading packages.
>>
>> If you want to do something truly exciting, you could take wingos fibers
>> and build a high performance web interface for guildhall with them.
>
> High performance is not really important in this case. We are not 
> talking about gazillions of npm packages.
>
>>
>> This is something I’d love to do but I fear that it’s not high enough in
>> my todo list that I’ll actually get it done.¹
>>
>> Best wishes,
>> Arne
>>
>> ¹: I became Freenet Release Manager a few months ago. That took away the
>>     last free time I could allocate to new challenging projects.
>
> If someone had a realistic plan of building something like that, I 
> might be able to contribute. I am not going to tackle it alone.
>

What's the status of guildhall, does it work? where are the latest changes?

amirouche ~ amz3

Re: How to make GNU Guile more successful

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 2098 bytes --]


Amirouche <amirouche@hypermove.net> writes:
>> If someone had a realistic plan of building something like that, I 
>> might be able to contribute. I am not going to tackle it alone.
>
> What's the status of guildhall, does it work? where are the latest changes?

What we need is someone who regularly puts the package folder onto a
static webserver (starnge that I did not think of just using a repo
instead of implementing a full web interface with access rights and
such).

Guildhall works with guile-2.0, and there are no changes (though I have
a local version in which `guild hall` outputs the default installation
locations).

However I get an error with guile 2.1.5:

$ make
./env /home/arne/.local/bin//guile-tools compile -Wunbound-variable -Warity-mismatch -Wformat -o "guildhall/cli/config.go" "guildhall/cli/config.scm"
Backtrace:
In ice-9/boot-9.scm:
  2860:10 19 (define-module* _ #:filename _ #:pure _ #:version _ # _ …)
  2799:17 18 (resolve-interface (guildhall config) #:select _ #:hide …)
  2724:10 17 (_ (guildhall config) _ _ #:ensure _)
  3000:16 16 (try-module-autoload _ _)
   2336:4 15 (save-module-excursion #<procedure 6d5d80 at ice-9/boot…>)
  3020:22 14 (_)
In unknown file:
          13 (primitive-load-path "guildhall/config" #<procedure d47…>)
In guildhall/config.scm:
     25:0 12 (_)
In ice-9/boot-9.scm:
  2799:17 11 (resolve-interface (guildhall destination fhs) #:select …)
  2724:10 10 (_ (guildhall destination fhs) _ _ #:ensure _)
  3000:16  9 (try-module-autoload _ _)
   2336:4  8 (save-module-excursion #<procedure da8930 at ice-9/boot…>)
  3020:22  7 (_)
In unknown file:
           6 (primitive-load-path "guildhall/destination/fhs" #<proc…>)
In guildhall/destination/fhs.scm:
    152:0  5 (_)
In guildhall/ext/inc/irregex-r6rs.scm:
  1505:17  4 (sre->irregex _ . _)
  2448:15  3 (sre->nfa _ 0)
  2391:29  2 (lp ((* any) (: "." ($ (+ (~ numeric))))) 1 0 0)
  2301:52  1 (lp ("." ($ (+ (~ numeric)))) 1 0 _)
In ice-9/boot-9.scm:
   762:26  0 (dispatch-exception _ _ _)

Best wishes,
Arne

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

Re: How to make GNU Guile more successful

[Amirouche]

Le 21/02/2017 à 20:25, Arne Babenhauserheide a écrit :
> Amirouche <amirouche@hypermove.net> writes:
>>> If someone had a realistic plan of building something like that, I
>>> might be able to contribute. I am not going to tackle it alone.
>> What's the status of guildhall, does it work? where are the latest changes?
> What we need is someone who regularly puts the package folder onto a
> static webserver

What does it mean?

> (starnge that I did not think of just using a repo
> instead of implementing a full web interface with access rights and
> such).

The issue I have with the web repository (I assume you meant git repo),
is that it requires for the maintainer of the repository to push a button,
accept new packages (at the very least). Which would mean that I offer
some kind of guarantees over the content of the software that you
download. Which maybe right now I can assume, but if there is as much
as patch in the repo as there is in guix I will need help...

>
> Guildhall works with guile-2.0, and there are no changes (though I have
> a local version in which `guild hall` outputs the default installation
> locations).
>
> However I get an error with guile 2.1.5:
>
> $ make
> ./env /home/arne/.local/bin//guile-tools compile -Wunbound-variable -Warity-mismatch -Wformat -o "guildhall/cli/config.go" "guildhall/cli/config.scm"
> Backtrace:
> In ice-9/boot-9.scm:
>    2860:10 19 (define-module* _ #:filename _ #:pure _ #:version _ # _ …)
>    2799:17 18 (resolve-interface (guildhall config) #:select _ #:hide …)
>    2724:10 17 (_ (guildhall config) _ _ #:ensure _)
>    3000:16 16 (try-module-autoload _ _)
>     2336:4 15 (save-module-excursion #<procedure 6d5d80 at ice-9/boot…>)
>    3020:22 14 (_)
> In unknown file:
>            13 (primitive-load-path "guildhall/config" #<procedure d47…>)
> In guildhall/config.scm:
>       25:0 12 (_)
> In ice-9/boot-9.scm:
>    2799:17 11 (resolve-interface (guildhall destination fhs) #:select …)
>    2724:10 10 (_ (guildhall destination fhs) _ _ #:ensure _)
>    3000:16  9 (try-module-autoload _ _)
>     2336:4  8 (save-module-excursion #<procedure da8930 at ice-9/boot…>)
>    3020:22  7 (_)
> In unknown file:
>             6 (primitive-load-path "guildhall/destination/fhs" #<proc…>)
> In guildhall/destination/fhs.scm:
>      152:0  5 (_)
> In guildhall/ext/inc/irregex-r6rs.scm:
>    1505:17  4 (sre->irregex _ . _)
>    2448:15  3 (sre->nfa _ 0)
>    2391:29  2 (lp ((* any) (: "." ($ (+ (~ numeric))))) 1 0 0)
>    2301:52  1 (lp ("." ($ (+ (~ numeric)))) 1 0 _)
> In ice-9/boot-9.scm:
>     762:26  0 (dispatch-exception _ _ _)
>
> Best wishes,
> Arne

Re: How to make GNU Guile more successful

[Nala Ginrut]

I think we have to elaborate the question clearer.

1. How to make guile-scheme more successful?
I think this is similar to ask "how to make scheme more successful".
This is the big question to the whole scheme community.

2. How to make guile platform more successful?
I this case, let me ask a question, if we have guile-python3
(compatible with most of Python3 features), and provide more
convenient FFI helper function to help bind more libraries in short
time, is it possible to attract more people from Python land? Given
we'll have good JIT compiler in the future.


On Thu, Mar 2, 2017 at 3:25 AM, Amirouche <amirouche@hypermove.net> wrote:
>
>
> Le 21/02/2017 à 20:25, Arne Babenhauserheide a écrit :
>>
>> Amirouche <amirouche@hypermove.net> writes:
>>>>
>>>> If someone had a realistic plan of building something like that, I
>>>> might be able to contribute. I am not going to tackle it alone.
>>>
>>> What's the status of guildhall, does it work? where are the latest
>>> changes?
>>
>> What we need is someone who regularly puts the package folder onto a
>> static webserver
>
>
> What does it mean?
>
>> (starnge that I did not think of just using a repo
>> instead of implementing a full web interface with access rights and
>> such).
>
>
> The issue I have with the web repository (I assume you meant git repo),
> is that it requires for the maintainer of the repository to push a button,
> accept new packages (at the very least). Which would mean that I offer
> some kind of guarantees over the content of the software that you
> download. Which maybe right now I can assume, but if there is as much
> as patch in the repo as there is in guix I will need help...
>
>
>>
>> Guildhall works with guile-2.0, and there are no changes (though I have
>> a local version in which `guild hall` outputs the default installation
>> locations).
>>
>> However I get an error with guile 2.1.5:
>>
>> $ make
>> ./env /home/arne/.local/bin//guile-tools compile -Wunbound-variable
>> -Warity-mismatch -Wformat -o "guildhall/cli/config.go"
>> "guildhall/cli/config.scm"
>> Backtrace:
>> In ice-9/boot-9.scm:
>>    2860:10 19 (define-module* _ #:filename _ #:pure _ #:version _ # _ …)
>>    2799:17 18 (resolve-interface (guildhall config) #:select _ #:hide …)
>>    2724:10 17 (_ (guildhall config) _ _ #:ensure _)
>>    3000:16 16 (try-module-autoload _ _)
>>     2336:4 15 (save-module-excursion #<procedure 6d5d80 at ice-9/boot…>)
>>    3020:22 14 (_)
>> In unknown file:
>>            13 (primitive-load-path "guildhall/config" #<procedure d47…>)
>> In guildhall/config.scm:
>>       25:0 12 (_)
>> In ice-9/boot-9.scm:
>>    2799:17 11 (resolve-interface (guildhall destination fhs) #:select …)
>>    2724:10 10 (_ (guildhall destination fhs) _ _ #:ensure _)
>>    3000:16  9 (try-module-autoload _ _)
>>     2336:4  8 (save-module-excursion #<procedure da8930 at ice-9/boot…>)
>>    3020:22  7 (_)
>> In unknown file:
>>             6 (primitive-load-path "guildhall/destination/fhs" #<proc…>)
>> In guildhall/destination/fhs.scm:
>>      152:0  5 (_)
>> In guildhall/ext/inc/irregex-r6rs.scm:
>>    1505:17  4 (sre->irregex _ . _)
>>    2448:15  3 (sre->nfa _ 0)
>>    2391:29  2 (lp ((* any) (: "." ($ (+ (~ numeric))))) 1 0 0)
>>    2301:52  1 (lp ("." ($ (+ (~ numeric)))) 1 0 _)
>> In ice-9/boot-9.scm:
>>     762:26  0 (dispatch-exception _ _ _)
>>
>> Best wishes,
>> Arne
>
>
>

Re: How to make GNU Guile more successful

[David Kastrup]

Nala Ginrut <nalaginrut@gmail.com> writes:

> I think we have to elaborate the question clearer.
>
> 1. How to make guile-scheme more successful?
> I think this is similar to ask "how to make scheme more successful".
> This is the big question to the whole scheme community.
>
> 2. How to make guile platform more successful?
> I this case, let me ask a question, if we have guile-python3
> (compatible with most of Python3 features), and provide more
> convenient FFI helper function to help bind more libraries in short
> time, is it possible to attract more people from Python land? Given
> we'll have good JIT compiler in the future.

Frankly, I doubt that migration of large Python-based applications is
going to be a thing when nobody can even be bothered with immersing
himself in the problems with migrating LilyPond from Guile-1.8 to
Guile-2.

-- 
David Kastrup

Re: How to make GNU Guile more successful

[Nala Ginrut]

On Fri, Mar 3, 2017 at 5:18 PM, David Kastrup <dak@gnu.org> wrote:
> Frankly, I doubt that migration of large Python-based applications is
> going to be a thing when nobody can even be bothered with immersing
> himself in the problems with migrating LilyPond from Guile-1.8 to
> Guile-2.

No, I don't think so.
If we have guile-python3, the migration work becomes attractive to
Guile community. Because each time you migrate a library, it can be
used in all languages implemented on Guile platform. Write only once
for all supported languages. Even if we don't care about Python (do
we?), we have to write many libraries for Guile to make it more
useful. It's the work Guile community has to do anyway.

And LilyPond is not a good case here, since not everybody needs it. I
think the best way to push a community is to provide convenient way to
let users who care certain library to contribute it. But we don't have
it now. For example, the documentation or tools to help 1.8->2.0.
Python has tools for Python2->Python3 and documents for it. It is the
management of Guile community, not technical problem.
Fortunately, it's just management problem, and it's easier to improve
than technical one, only if we found a persistent way to push and
there's enough contributors.

Re: How to make GNU Guile more successful

[David Kastrup]

Nala Ginrut <nalaginrut@gmail.com> writes:

> On Fri, Mar 3, 2017 at 5:18 PM, David Kastrup <dak@gnu.org> wrote:
>
>> Frankly, I doubt that migration of large Python-based applications is
>> going to be a thing when nobody can even be bothered with immersing
>> himself in the problems with migrating LilyPond from Guile-1.8 to
>> Guile-2.
>
> No, I don't think so.
> If we have guile-python3, the migration work becomes attractive to
> Guile community. Because each time you migrate a library, it can be
> used in all languages implemented on Guile platform.

The .go organization and call gate costs (for example constant string
conversions) and memory organization and foreign string hardiness issues
bogging down LilyPond will affect interfacing to every external library
with a high call rate for processing.

> I think the best way to push a community is to provide convenient way
> to let users who care certain library to contribute it. But we don't
> have it now. For example, the documentation or tools to help 1.8->2.0.
> Python has tools for Python2->Python3 and documents for it. It is the
> management of Guile community, not technical problem.  Fortunately,
> it's just management problem, and it's easier to improve than
> technical one, only if we found a persistent way to push and there's
> enough contributors.

The technical problems won't go away by themselves.

So which migration of a large Python-based application do you expect to
become a thing without addressing significant amounts of technical
problems first?  Or how do I have to interpret your "No, I don't think
so."?

-- 
David Kastrup

Re: How to make GNU Guile more successful

[Nala Ginrut]

On Fri, Mar 3, 2017 at 8:19 PM, David Kastrup <dak@gnu.org> wrote:
> The .go organization and call gate costs (for example constant string
> conversions) and memory organization and foreign string hardiness issues
> bogging down LilyPond will affect interfacing to every external library
> with a high call rate for processing.

I respect LilyPond contributors and people who put effort on it. But
it's not on my hack-line.
If someone really care about it, it'll be done someday definitely.

> The technical problems won't go away by themselves.
>
> So which migration of a large Python-based application do you expect to
> become a thing without addressing significant amounts of technical
> problems first?  Or how do I have to interpret your "No, I don't think
> so."?
>

If I don't say "no, I don't think so", should I say "well, yes, Guile
has no chance, let's remove it".? ;-)
I think most of the people in Guile community knows the problems and
difficulties, it's unnecessary to repeat them again, what we need is
to find the solution.
I gave part of solution, and I'm enjoying to go with it.
If you're afraid of "large python based apps", how about get a seat
and see how others solve it?
Anyway, talking the problems and difficulties people already know
again and again is a contribution too. Then we don't need to write
TODO.

Do you think I know nothing about the complexity? I'll tell you that I
love that complexity so I'm in.
When I came to Guile community 7 years ago, I know nothing as a pure newbie.
But now I know I can learn many things each time when I challenge the
difficulties.
If you feel depressed when you face LilyPond, please don't forget I
feel frustrated when I'm debugging the Artanis server-core and our JIT
compiler too.
No one is facing the easy job.

Frankly, I have better idea than migrating all Python pacakges: just
migrate when you need it, or write a better one.
I'm trying to use Guile things in real products, so I'm serious.

I still want to ask "why do we have to compare to Python?"
I'm going to write guile-python3 because some young people in my team
has difficult to use Scheme directly, and we don't want them to
rewrite whole things in Scheme again.
But Guile will bring many potential advantages rather than our current
Python and C++14 combination.

Be more confident, folks.
That's why the answer from me is "no, I don't think so".

Best regards.

Re: How to make GNU Guile more successful

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 17190 bytes --]


David Kastrup <dak@gnu.org> writes:

> Nala Ginrut <nalaginrut@gmail.com> writes:
>
>> I think we have to elaborate the question clearer.
>>
>> 1. How to make guile-scheme more successful?
>> I think this is similar to ask "how to make scheme more successful".
>> This is the big question to the whole scheme community.
>>
>> 2. How to make guile platform more successful?
>> I this case, let me ask a question, if we have guile-python3
>> (compatible with most of Python3 features), and provide more
>> convenient FFI helper function to help bind more libraries in short
>> time, is it possible to attract more people from Python land? Given
>> we'll have good JIT compiler in the future.
>
> Frankly, I doubt that migration of large Python-based applications is
> going to be a thing when nobody can even be bothered with immersing
> himself in the problems with migrating LilyPond from Guile-1.8 to
> Guile-2.

I worked on testing Lilypond with user installed Guile 2.x, does that
count?

Just to have it recorded somewhere, here’s a patch to lilypond along
with a copy of the bash history of the setup (cleaned up, it was many
times as long):

## patch

From bd2ffea6f4c4c1ede13f5ac82d0a8ce31ccfe3c7 Mon Sep 17 00:00:00 2001
Subject: [PATCH] Build fixes for Guile 2.1.x (not yet functional)

---
 configure.ac         | 7 ++++++-
 lily/pdf-scheme.cc   | 4 ++++
 scm/memory-trace.scm | 3 ++-
 3 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/configure.ac b/configure.ac
index d77ea15..393976b 100644
--- a/configure.ac
+++ b/configure.ac
@@ -267,7 +267,12 @@ STEPMAKE_FREETYPE2(freetype2, REQUIRED, 2.1.10)
 STEPMAKE_WINDOWS
 
 # guile executable for some scripts
-STEPMAKE_GUILE(OPTIONAL, 1.8.2, 1.9.0)
+if test "$GUILEv2" = "yes"
+then
+    STEPMAKE_GUILE(OPTIONAL, 2.0.7, 2.2.0)
+else
+    STEPMAKE_GUILE(OPTIONAL, 1.8.2, 1.9.0)
+fi
 
 # perl for help2man and for mf2pt1.pl
 STEPMAKE_PERL(REQUIRED)
diff --git a/lily/pdf-scheme.cc b/lily/pdf-scheme.cc
index da2ce2c..f5ae70c 100644
--- a/lily/pdf-scheme.cc
+++ b/lily/pdf-scheme.cc
@@ -91,7 +91,11 @@ LY_DEFINE (ly_encode_string_for_pdf, "ly:encode-string-for-pdf",
    *                  (string->utf16 str 'big)
    */
   if (g)
+#if GUILEV2
+    return scm_take_locale_stringn (g, bytes_written); // scm_take_str eventually frees g!
+#else
     return scm_take_str (g, bytes_written); // scm_take_str eventually frees g!
+#endif
   else
     return str;
 }
diff --git a/scm/memory-trace.scm b/scm/memory-trace.scm
index d8ffeb9..9ebd722 100644
--- a/scm/memory-trace.scm
+++ b/scm/memory-trace.scm
@@ -2,7 +2,8 @@
 
 (define-module (scm memory-trace))
 (use-modules (lily)
-             (ice-9 format))
+             (ice-9 format)
+             (ice-9 threads))
 
 (define-public (mtrace:start-trace freq)
   (set! usecond-interval (inexact->exact (/ 1000000 freq)))
-- 
2.10.2


## bash history

sudo emerge media-fonts/tex-gyre
sudo nano /etc/portage/package.keywords/sonstiges 
sudo emerge media-fonts/tex-gyre
sudo pmerge dblatex
git clone git://git.sv.gnu.org/lilypond.git
cd lilypond/
./autogen.sh --prefix ~/.local --enable-guile2
mkdir -p ~/texmf/lh
cp ~/Downloads/lhfnt35g-source.zip ~/texmf/lh/
unzip lhfnt35g-source.zip 
cd ~/texmf/source/latex/lh
latex lcyfonts.ins 
latex ot2fonts.ins; latex t2ccfonts.ins
ls ~/texmf/examples/
mkdir ~/texmf/lh
cp *sty ~/texmf/lh/
./autogen.sh --prefix ~/.local --enable-guile2
sudo emerge dev-texlive/texlive-langcyrillic
sudo pmerge dev-texlive/texlive-langcyrillic
make out/lilypond-invoke-editor
cd scripts/
/home/arne/lilypond/scripts/build/out/help2man out/lilypond-invoke-editor --no-discard-stderr
/home/arne/lilypond/scripts/build/out/help2man out/lilypond-invoke-editor 
./autogen.sh --prefix ~/.local --enable-guile2
make
LD_LIBRARY_PATH=/home/arne/.local/lib/guile/2.2/ g++ -o out/lilypond  ./out/modified-font-metric.o ./out/sequential-iterator.o ./out/hara-kiri-group-spanner.o ./out/completion-note-heads-engraver.o ./out/slur-score-parameters.o ./out/flag.o ./out/span-arpeggio-engraver.o ./out/beam-quanting.o ./out/dynamic-performer.o ./out/parse-scm.o ./out/tie.o ./out/bend-engraver.o ./out/staff-spacing.o ./out/rest-collision-engraver.o ./out/simple-spacer-scheme.o ./out/audio-item.o ./out/pdf-scheme.o ./out/paper-score.o ./out/slur-configuration.o ./out/constrained-breaking.o ./out/score-engraver.o ./out/unpure-pure-container.o ./out/multi-measure-rest-engraver.o ./out/system-start-delimiter-engraver.o ./out/context-property.o ./out/freetype.o ./out/interval-minefield.o ./out/kievan-ligature.o ./out/completion-rest-engraver.o ./out/pure-from-neighbor-engraver.o ./out/horizontal-bracket-engraver.o ./out/grob-closure.o ./out/default-bar-line-engraver.o ./out/control-track-performer.o ./out/file-name-map.o ./out/audio-staff.o ./out/line-spanner.o ./out/grob-info.o ./out/spring.o ./out/slur.o ./out/music-scheme.o ./out/program-option.o ./out/fretboard-engraver.o ./out/hyphen-engraver.o ./out/page-marker.o ./out/global-vars.o ./out/stream-event-scheme.o ./out/script-column-engraver.o ./out/bezier-bow.o ./out/page-breaking-scheme.o ./out/lily-version.o ./out/repeat-tie-engraver.o ./out/tweak-engraver.o ./out/staff-symbol-referencer.o ./out/moment-scheme.o ./out/note-heads-engraver.o ./out/paper-outputter.o ./out/music-wrapper.o ./out/grob.o ./out/performance-scheme.o ./out/bar-line.o ./out/page-turn-engraver.o ./out/dimensions-scheme.o ./out/tab-note-heads-engraver.o ./out/piano-pedal-bracket.o ./out/spaceable-grob.o ./out/event-iterator.o ./out/lyric-combine-music-iterator.o ./out/percent-repeat-iterator.o ./out/audio-element-info.o ./out/midi-cc-announcer.o ./out/font-metric.o ./out/note-spacing-engraver.o ./out/duration-scheme.o ./out/accidental-placement.o ./out/music-iterator.o ./out/rhythmic-head.o ./out/grob-array.o ./out/tab-tie-follow-engraver.o ./out/context.o ./out/dispatcher.o ./out/scale.o ./out/chord-name-engraver.o ./out/auto-change-iterator.o ./out/tie-engraver.o ./out/grid-line-interface.o ./out/forbid-break-engraver.o ./out/note-column.o ./out/lookup.o ./out/global-ctor.o ./out/custos-engraver.o ./out/piano-pedal-engraver.o ./out/simple-music-iterator.o ./out/pitch-scheme.o ./out/mensural-ligature-engraver.o ./out/articulations.o ./out/horizontal-bracket.o ./out/script-column.o ./out/global-context.o ./out/slur-performer.o ./out/guile-init.o ./out/stencil-scheme.o ./out/font-size-engraver.o ./out/rest-engraver.o ./out/skyline.o ./out/percent-repeat-item.o ./out/enclosing-bracket.o ./out/module-scheme.o ./out/input.o ./out/paper-outputter-scheme.o ./out/spacing-interface.o ./out/break-align-engraver.o ./out/chord-tremolo-iterator.o ./out/general-scheme.o ./out/least-squares.o ./out/audio-element.o ./out/paper-system.o ./out/lily-lexer-scheme.o ./out/beam.o ./out/music-sequence.o ./out/music.o ./out/stem.o ./out/relative-octave-check.o ./out/keyword.o ./out/translator-ctors.o ./out/stencil-integral.o ./out/arpeggio-engraver.o ./out/extender-engraver.o ./out/prob-scheme.o ./out/ligature-engraver.o ./out/lyric-hyphen.o ./out/simple-spacer.o ./out/output-property-engraver.o ./out/drum-note-performer.o ./out/fingering-column-engraver.o ./out/midi-item.o ./out/undead.o ./out/repeat-acknowledge-engraver.o ./out/grace-engraver.o ./out/figured-bass-position-engraver.o ./out/tie-formatting-problem.o ./out/context-mod-scheme.o ./out/pfb.o ./out/fingering-engraver.o ./out/profile.o ./out/break-alignment-interface.o ./out/trill-spanner-engraver.o ./out/multi-measure-rest.o ./out/collision-engraver.o ./out/system-start-delimiter.o ./out/one-line-auto-height-breaking.o ./out/minimal-page-breaking.o ./out/dot-column-engraver.o ./out/note-name-engraver.o ./out/breathing-sign.o ./out/paper-book.o ./out/sustain-pedal.o ./out/tie-configuration.o ./out/staff-grouper-interface.o ./out/output-def.o ./out/nested-property.o ./out/timing-translator.o ./out/protected-scm.o ./out/volta-bracket.o ./out/beam-performer.o ./out/beaming-pattern.o ./out/prob.o ./out/part-combine-engraver.o ./out/smobs.o ./out/system.o ./out/figured-bass-engraver.o ./out/paper-def.o ./out/performer-group.o ./out/rhythmic-column-engraver.o ./out/volta-engraver.o ./out/paper-column-engraver.o ./out/scm-hash.o ./out/beam-collision-engraver.o ./out/apply-context-iterator.o ./out/skyline-pair.o ./out/gdb.o ./out/spacing-spanner.o ./out/stanza-number-align-engraver.o ./out/line-interface.o ./out/measure-grouping-spanner.o ./out/staff-symbol-referencer-scheme.o ./out/stem-engraver.o ./out/font-metric-scheme.o ./out/beam-engraver.o ./out/freetype-error.o ./out/instrument-name-engraver.o ./out/partial-iterator.o ./out/lily-lexer.o ./out/metronome-engraver.o ./out/warn-scheme.o ./out/includable-lexer.o ./out/item.o ./out/spacing-engraver.o ./out/relocate.o ./out/pointer-group-interface.o ./out/measure-grouping-engraver.o ./out/grace-music.o ./out/staff-performer.o ./out/slur-engraver.o ./out/function-documentation.o ./out/dots.o ./out/performer.o ./out/page-spacing.o ./out/separating-line-group-engraver.o ./out/axis-group-interface-scheme.o ./out/box.o ./out/spacing-determine-loose-columns.o ./out/duration.o ./out/stencil-expression.o ./out/phrasing-slur-engraver.o ./out/side-position-interface.o ./out/pitch-interval.o ./out/accidental.o ./out/pure-from-neighbor-interface.o ./out/optimal-page-breaking.o ./out/spanner-break-forbid-engraver.o ./out/drum-note-engraver.o ./out/stanza-number-engraver.o ./out/midi-walker.o ./out/ly-module.o ./out/span-bar-stub-engraver.o ./out/book-scheme.o ./out/grid-point-engraver.o ./out/scheme-engraver.o ./out/dot-column.o ./out/note-head-scheme.o ./out/lily-imports.o ./out/font-config-scheme.o ./out/open-type-font-scheme.o ./out/hairpin.o ./out/quote-iterator.o ./out/ledger-line-engraver.o ./out/auto-beam-engraver.o ./out/pango-font.o ./out/misc.o ./out/repeated-music.o ./out/figured-bass-continuation.o ./out/grace-iterator.o ./out/cue-clef-engraver.o ./out/pango-select.o ./out/main.o ./out/music-wrapper-iterator.o ./out/pitch-squash-engraver.o ./out/tuplet-iterator.o ./out/stencil-interpret.o ./out/key-signature-interface.o ./out/pitched-trill-engraver.o ./out/score.o ./out/context-mod.o ./out/fingering-column.o ./out/music-function-scheme.o ./out/bar-number-engraver.o ./out/break-substitution.o ./out/sources.o ./out/piano-pedal-performer.o ./out/performance.o ./out/volta-repeat-iterator.o ./out/slash-repeat-engraver.o ./out/font-config.o ./out/lily-guile.o ./out/balloon.o ./out/text-spanner-engraver.o ./out/bezier.o ./out/stencil.o ./out/clef.o ./out/spacing-basic.o ./out/dynamic-engraver.o ./out/time-signature-engraver.o ./out/ottava-engraver.o ./out/music-output.o ./out/lily-parser.o ./out/ligature-bracket-engraver.o ./out/translator-dispatch-list.o ./out/lily-parser-scheme.o ./out/engraver-scheme.o ./out/dynamic-align-engraver.o ./out/part-combine-iterator.o ./out/custos.o ./out/translator-scheme.o ./out/slur-scoring.o ./out/accidental-engraver.o ./out/rest-collision.o ./out/staff-symbol-engraver.o ./out/spanner.o ./out/note-column-scheme.o ./out/piano-pedal-align-engraver.o ./out/clef-modifier.o ./out/note-performer.o ./out/pango-font-scheme.o ./out/melody-spanner.o ./out/tie-column.o ./out/tuplet-bracket.o ./out/episema-engraver.o ./out/lyric-extender.o ./out/grob-property.o ./out/lyric-combine-music.o ./out/vaticana-ligature.o ./out/cluster-engraver.o ./out/instrument-switch-engraver.o ./out/part-combine-part-iterator.o ./out/audio-column.o ./out/key-performer.o ./out/gregorian-ligature-engraver.o ./out/context-specced-music-iterator.o ./out/pango-select-scheme.o ./out/translator-group.o ./out/book.o ./out/stream-event.o ./out/staff-symbol.o ./out/font-interface.o ./out/lyric-engraver.o ./out/page-breaking.o ./out/engraver-group.o ./out/grob-smob.o ./out/grob-array-scheme.o ./out/rod.o ./out/moment.o ./out/dispatcher-scheme.o ./out/lilypond-version.o ./out/tab-staff-symbol-engraver.o ./out/font-select.o ./out/grob-interface-scheme.o ./out/tuplet-engraver.o ./out/balloon-engraver.o ./out/template5.o ./out/paper-column.o ./out/script-row-engraver.o ./out/clef-engraver.o ./out/one-page-breaking.o ./out/midi-chunk.o ./out/context-def.o ./out/paper-score-scheme.o ./out/axis-group-engraver.o ./out/cluster.o ./out/translator.o ./out/item-scheme.o ./out/double-percent-repeat-engraver.o ./out/separation-item.o ./out/key-engraver.o ./out/keep-alive-together-engraver.o ./out/source-file.o ./out/all-font-metrics-scheme.o ./out/self-alignment-interface.o ./out/axis-group-interface.o ./out/midi-cc-performer.o ./out/input-scheme.o ./out/lyric-performer.o ./out/grid-line-span-engraver.o ./out/glissando-engraver.o ./out/event-chord-iterator.o ./out/chord-name.o ./out/spring-smob.o ./out/score-performer.o ./out/paper-system-scheme.o ./out/coherent-ligature-engraver.o ./out/context-scheme.o ./out/tie-performer.o ./out/semi-tie.o ./out/chord-tremolo-engraver.o ./out/pitch.o ./out/listener.o ./out/percent-repeat-engraver.o ./out/note-head.o ./out/relative-octave-music.o ./out/note-spacing.o ./out/dimension-cache.o ./out/spacing-loose-columns.o ./out/melody-engraver.o ./out/page-spacing-result.o ./out/script-engraver.o ./out/ttf.o ./out/dot-formatting-problem.o ./out/all-font-metrics.o ./out/ledger-line-spanner.o ./out/staff-collecting-engraver.o ./out/spacing-options.o ./out/time-signature-performer.o ./out/pfb-scheme.o ./out/property-iterator.o ./out/vaticana-ligature-engraver.o ./out/script-interface.o ./out/score-scheme.o ./out/one-line-page-breaking.o ./out/page-marker-scheme.o ./out/spanner-scheme.o ./out/music-function.o ./out/ambitus-engraver.o ./out/new-fingering-engraver.o ./out/bar-check-iterator.o ./out/change-iterator.o ./out/context-handle.o ./out/parenthesis-engraver.o ./out/align-interface.o ./out/change-sequence-iterator.o ./out/column-x-positions.o ./out/output-def-scheme.o ./out/global-context-scheme.o ./out/tie-details.o ./out/rest.o ./out/note-collision.o ./out/dot-configuration.o ./out/tempo-performer.o ./out/translator-group-ctors.o ./out/ottava-bracket.o ./out/stem-tremolo.o ./out/mensural-ligature.o ./out/grob-pq-engraver.o ./out/lily-modules.o ./out/page-turn-page-breaking.o ./out/gregorian-ligature.o ./out/engraver.o ./out/rhythmic-music-iterator.o ./out/mark-engraver.o ./out/page-layout-problem-scheme.o ./out/vertical-align-engraver.o ./out/kievan-ligature-engraver.o ./out/tie-specification.o ./out/tuplet-number.o ./out/laissez-vibrer-engraver.o ./out/dots-engraver.o ./out/directional-element-interface.o ./out/concurrent-hairpin-engraver.o ./out/line-interface-scheme.o ./out/paper-book-scheme.o ./out/pointer-group-interface-scheme.o ./out/grob-interface.o ./out/open-type-font.o ./out/span-bar-engraver.o ./out/midi-stream.o ./out/page-layout-problem.o ./out/input-smob.o ./out/note-head-line-engraver.o ./out/bar-engraver.o ./out/text-interface.o ./out/text-engraver.o ./out/grob-scheme.o ./out/arpeggio.o ./out/semi-tie-column.o ./out/program-option-scheme.o ./out/footnote-engraver.o ./out/simultaneous-music-iterator.o ./out/breathing-sign-engraver.o ./out/grace-spacing-engraver.o ./out/lexer.o ./out/parser.o ./out/../../flower/out/library.a  -ldl   -L/home/arne/.local/lib -L/home/arne/.local/lib/guile/2.2 -lguile-2.2 -lgc  -lpangoft2-1.0 -lpango-1.0 -lgobject-2.0 -lglib-2.0 -lfontconfig -lfreetype  -lfontconfig -lfreetype  -lfreetype -v 2>&1 | grep -i lib
nm out/lilypond 
LD_LIBRARY_PATH=~/.local/lib/  out/lilypond 
GUILE_LOAD_PATH=$GUILE_LOAD_PATH:../:../scm LD_LIBRARY_PATH=~/.local/lib/  out/lilypond 
wget http://infinite-hands.draketo.de/infinite_hands_sheet.ly
wget -O mf/out/emmentaler-20.otf https://github.com/saebekassebil/subito/raw/master/resources/gonville/lilyfonts/otf/emmentaler-20.otf
GUILE_LOAD_PATH=$GUILE_LOAD_PATH:.:scm LD_LIBRARY_PATH=~/.local/lib/  lily/out/lilypond -I ly/ -I mf/out/ -I ps/ test.ly
cp lilypond-data_2.18.2-4.1_all.deb /tmp
scp kav:/usr/share/lilypond/2.16.2/fonts/otf/*otf mf/out/
GUILE_LOAD_PATH=$GUILE_LOAD_PATH:.:scm LD_LIBRARY_PATH=~/.local/lib/  lily/out/lilypond -I ly/ -I mf/out/ -I ps/ test.ly
git add configure.ac lily/general-scheme.cc lily/pdf-scheme.cc scm/memory-trace.scm
time GUILE_LOAD_PATH=$GUILE_LOAD_PATH:.:scm LD_LIBRARY_PATH=~/.local/lib/  lily/out/lilypond -I ly/ -I mf/out/ -I ps/ infinite_hands_sheet.ly 
cat 0001-Build-fixes-for-Guile-2.1.x-not-yet-functional.patch
cat > 1 <<EOF
diff --git a/lily/general-scheme.cc b/lily/general-scheme.cc
index 1168ee9..2df63fc 100644
--- a/lily/general-scheme.cc
+++ b/lily/general-scheme.cc
@@ -275,7 +275,8 @@ LY_DEFINE (ly_protects, "ly:protects",
 #if SCM_MAJOR_VERSION < 2 || SCM_MAJOR_VERSION == 2 && SCM_MINOR_VERSION < 1
   return scm_protects;
 #else
-  return programming_error ("ly:protects is not supported in Guile 2.1");
+  // return programming_error ("ly:protects is not supported in Guile 2.1");
+  return 0; // FIXME: Evil hack just to get this to build.
 #endif
 }
EOF

Best wishes,
Arne

PS: If this provides at least a small step towards guile 2 in lilypond,
    it’s worth its while.
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

Re: How to make GNU Guile more successful

[Thomas Morley]

2017-03-05 0:44 GMT+01:00 Arne Babenhauserheide <arne_bab@web.de>:
>
> David Kastrup <dak@gnu.org> writes:
>
>> Nala Ginrut <nalaginrut@gmail.com> writes:
>>
>>> I think we have to elaborate the question clearer.
>>>
>>> 1. How to make guile-scheme more successful?
>>> I think this is similar to ask "how to make scheme more successful".
>>> This is the big question to the whole scheme community.
>>>
>>> 2. How to make guile platform more successful?
>>> I this case, let me ask a question, if we have guile-python3
>>> (compatible with most of Python3 features), and provide more
>>> convenient FFI helper function to help bind more libraries in short
>>> time, is it possible to attract more people from Python land? Given
>>> we'll have good JIT compiler in the future.
>>
>> Frankly, I doubt that migration of large Python-based applications is
>> going to be a thing when nobody can even be bothered with immersing
>> himself in the problems with migrating LilyPond from Guile-1.8 to
>> Guile-2.
>
> I worked on testing Lilypond with user installed Guile 2.x, does that
> count?
>
> Just to have it recorded somewhere, here’s a patch to lilypond along
> with a copy of the bash history of the setup (cleaned up, it was many
> times as long):
>
> ## patch
>
> From bd2ffea6f4c4c1ede13f5ac82d0a8ce31ccfe3c7 Mon Sep 17 00:00:00 2001
> Subject: [PATCH] Build fixes for Guile 2.1.x (not yet functional)
>
> ---
>  configure.ac         | 7 ++++++-
>  lily/pdf-scheme.cc   | 4 ++++
>  scm/memory-trace.scm | 3 ++-
>  3 files changed, 12 insertions(+), 2 deletions(-)
>
> diff --git a/configure.ac b/configure.ac
> index d77ea15..393976b 100644
> --- a/configure.ac
> +++ b/configure.ac
> @@ -267,7 +267,12 @@ STEPMAKE_FREETYPE2(freetype2, REQUIRED, 2.1.10)
>  STEPMAKE_WINDOWS
>
>  # guile executable for some scripts
> -STEPMAKE_GUILE(OPTIONAL, 1.8.2, 1.9.0)
> +if test "$GUILEv2" = "yes"
> +then
> +    STEPMAKE_GUILE(OPTIONAL, 2.0.7, 2.2.0)
> +else
> +    STEPMAKE_GUILE(OPTIONAL, 1.8.2, 1.9.0)
> +fi
>
>  # perl for help2man and for mf2pt1.pl
>  STEPMAKE_PERL(REQUIRED)
> diff --git a/lily/pdf-scheme.cc b/lily/pdf-scheme.cc
> index da2ce2c..f5ae70c 100644
> --- a/lily/pdf-scheme.cc
> +++ b/lily/pdf-scheme.cc
> @@ -91,7 +91,11 @@ LY_DEFINE (ly_encode_string_for_pdf, "ly:encode-string-for-pdf",
>     *                  (string->utf16 str 'big)
>     */
>    if (g)
> +#if GUILEV2
> +    return scm_take_locale_stringn (g, bytes_written); // scm_take_str eventually frees g!
> +#else
>      return scm_take_str (g, bytes_written); // scm_take_str eventually frees g!
> +#endif
>    else
>      return str;
>  }
> diff --git a/scm/memory-trace.scm b/scm/memory-trace.scm
> index d8ffeb9..9ebd722 100644
> --- a/scm/memory-trace.scm
> +++ b/scm/memory-trace.scm
> @@ -2,7 +2,8 @@
>
>  (define-module (scm memory-trace))
>  (use-modules (lily)
> -             (ice-9 format))
> +             (ice-9 format)
> +             (ice-9 threads))
>
>  (define-public (mtrace:start-trace freq)
>    (set! usecond-interval (inexact->exact (/ 1000000 freq)))
> --
> 2.10.2
>
>
> ## bash history
>
> sudo emerge media-fonts/tex-gyre
> sudo nano /etc/portage/package.keywords/sonstiges
> sudo emerge media-fonts/tex-gyre
> sudo pmerge dblatex
> git clone git://git.sv.gnu.org/lilypond.git
> cd lilypond/
> ./autogen.sh --prefix ~/.local --enable-guile2
> mkdir -p ~/texmf/lh
> cp ~/Downloads/lhfnt35g-source.zip ~/texmf/lh/
> unzip lhfnt35g-source.zip
> cd ~/texmf/source/latex/lh
> latex lcyfonts.ins
> latex ot2fonts.ins; latex t2ccfonts.ins
> ls ~/texmf/examples/
> mkdir ~/texmf/lh
> cp *sty ~/texmf/lh/
> ./autogen.sh --prefix ~/.local --enable-guile2
> sudo emerge dev-texlive/texlive-langcyrillic
> sudo pmerge dev-texlive/texlive-langcyrillic
> make out/lilypond-invoke-editor
> cd scripts/
> /home/arne/lilypond/scripts/build/out/help2man out/lilypond-invoke-editor --no-discard-stderr
> /home/arne/lilypond/scripts/build/out/help2man out/lilypond-invoke-editor
> ./autogen.sh --prefix ~/.local --enable-guile2
> make
> LD_LIBRARY_PATH=/home/arne/.local/lib/guile/2.2/ g++ -o out/lilypond  ./out/modified-font-metric.o ./out/sequential-iterator.o ./out/hara-kiri-group-spanner.o ./out/completion-note-heads-engraver.o ./out/slur-score-parameters.o ./out/flag.o ./out/span-arpeggio-engraver.o ./out/beam-quanting.o ./out/dynamic-performer.o ./out/parse-scm.o ./out/tie.o ./out/bend-engraver.o ./out/staff-spacing.o ./out/rest-collision-engraver.o ./out/simple-spacer-scheme.o ./out/audio-item.o ./out/pdf-scheme.o ./out/paper-score.o ./out/slur-configuration.o ./out/constrained-breaking.o ./out/score-engraver.o ./out/unpure-pure-container.o ./out/multi-measure-rest-engraver.o ./out/system-start-delimiter-engraver.o ./out/context-property.o ./out/freetype.o ./out/interval-minefield.o ./out/kievan-ligature.o ./out/completion-rest-engraver.o ./out/pure-from-neighbor-engraver.o ./out/horizontal-bracket-engraver.o ./out/grob-closure.o ./out/default-bar-line-engraver.o ./out/control-track-performer.o ./out/file-name-map.o ./out/audio-staff.o ./out/line-spanner.o ./out/grob-info.o ./out/spring.o ./out/slur.o ./out/music-scheme.o ./out/program-option.o ./out/fretboard-engraver.o ./out/hyphen-engraver.o ./out/page-marker.o ./out/global-vars.o ./out/stream-event-scheme.o ./out/script-column-engraver.o ./out/bezier-bow.o ./out/page-breaking-scheme.o ./out/lily-version.o ./out/repeat-tie-engraver.o ./out/tweak-engraver.o ./out/staff-symbol-referencer.o ./out/moment-scheme.o ./out/note-heads-engraver.o ./out/paper-outputter.o ./out/music-wrapper.o ./out/grob.o ./out/performance-scheme.o ./out/bar-line.o ./out/page-turn-engraver.o ./out/dimensions-scheme.o ./out/tab-note-heads-engraver.o ./out/piano-pedal-bracket.o ./out/spaceable-grob.o ./out/event-iterator.o ./out/lyric-combine-music-iterator.o ./out/percent-repeat-iterator.o ./out/audio-element-info.o ./out/midi-cc-announcer.o ./out/font-metric.o ./out/note-spacing-engraver.o ./out/duration-scheme.o ./out/accidental-placement.o ./out/music-iterator.o ./out/rhythmic-head.o ./out/grob-array.o ./out/tab-tie-follow-engraver.o ./out/context.o ./out/dispatcher.o ./out/scale.o ./out/chord-name-engraver.o ./out/auto-change-iterator.o ./out/tie-engraver.o ./out/grid-line-interface.o ./out/forbid-break-engraver.o ./out/note-column.o ./out/lookup.o ./out/global-ctor.o ./out/custos-engraver.o ./out/piano-pedal-engraver.o ./out/simple-music-iterator.o ./out/pitch-scheme.o ./out/mensural-ligature-engraver.o ./out/articulations.o ./out/horizontal-bracket.o ./out/script-column.o ./out/global-context.o ./out/slur-performer.o ./out/guile-init.o ./out/stencil-scheme.o ./out/font-size-engraver.o ./out/rest-engraver.o ./out/skyline.o ./out/percent-repeat-item.o ./out/enclosing-bracket.o ./out/module-scheme.o ./out/input.o ./out/paper-outputter-scheme.o ./out/spacing-interface.o ./out/break-align-engraver.o ./out/chord-tremolo-iterator.o ./out/general-scheme.o ./out/least-squares.o ./out/audio-element.o ./out/paper-system.o ./out/lily-lexer-scheme.o ./out/beam.o ./out/music-sequence.o ./out/music.o ./out/stem.o ./out/relative-octave-check.o ./out/keyword.o ./out/translator-ctors.o ./out/stencil-integral.o ./out/arpeggio-engraver.o ./out/extender-engraver.o ./out/prob-scheme.o ./out/ligature-engraver.o ./out/lyric-hyphen.o ./out/simple-spacer.o ./out/output-property-engraver.o ./out/drum-note-performer.o ./out/fingering-column-engraver.o ./out/midi-item.o ./out/undead.o ./out/repeat-acknowledge-engraver.o ./out/grace-engraver.o ./out/figured-bass-position-engraver.o ./out/tie-formatting-problem.o ./out/context-mod-scheme.o ./out/pfb.o ./out/fingering-engraver.o ./out/profile.o ./out/break-alignment-interface.o ./out/trill-spanner-engraver.o ./out/multi-measure-rest.o ./out/collision-engraver.o ./out/system-start-delimiter.o ./out/one-line-auto-height-breaking.o ./out/minimal-page-breaking.o ./out/dot-column-engraver.o ./out/note-name-engraver.o ./out/breathing-sign.o ./out/paper-book.o ./out/sustain-pedal.o ./out/tie-configuration.o ./out/staff-grouper-interface.o ./out/output-def.o ./out/nested-property.o ./out/timing-translator.o ./out/protected-scm.o ./out/volta-bracket.o ./out/beam-performer.o ./out/beaming-pattern.o ./out/prob.o ./out/part-combine-engraver.o ./out/smobs.o ./out/system.o ./out/figured-bass-engraver.o ./out/paper-def.o ./out/performer-group.o ./out/rhythmic-column-engraver.o ./out/volta-engraver.o ./out/paper-column-engraver.o ./out/scm-hash.o ./out/beam-collision-engraver.o ./out/apply-context-iterator.o ./out/skyline-pair.o ./out/gdb.o ./out/spacing-spanner.o ./out/stanza-number-align-engraver.o ./out/line-interface.o ./out/measure-grouping-spanner.o ./out/staff-symbol-referencer-scheme.o ./out/stem-engraver.o ./out/font-metric-scheme.o ./out/beam-engraver.o ./out/freetype-error.o ./out/instrument-name-engraver.o ./out/partial-iterator.o ./out/lily-lexer.o ./out/metronome-engraver.o ./out/warn-scheme.o ./out/includable-lexer.o ./out/item.o ./out/spacing-engraver.o ./out/relocate.o ./out/pointer-group-interface.o ./out/measure-grouping-engraver.o ./out/grace-music.o ./out/staff-performer.o ./out/slur-engraver.o ./out/function-documentation.o ./out/dots.o ./out/performer.o ./out/page-spacing.o ./out/separating-line-group-engraver.o ./out/axis-group-interface-scheme.o ./out/box.o ./out/spacing-determine-loose-columns.o ./out/duration.o ./out/stencil-expression.o ./out/phrasing-slur-engraver.o ./out/side-position-interface.o ./out/pitch-interval.o ./out/accidental.o ./out/pure-from-neighbor-interface.o ./out/optimal-page-breaking.o ./out/spanner-break-forbid-engraver.o ./out/drum-note-engraver.o ./out/stanza-number-engraver.o ./out/midi-walker.o ./out/ly-module.o ./out/span-bar-stub-engraver.o ./out/book-scheme.o ./out/grid-point-engraver.o ./out/scheme-engraver.o ./out/dot-column.o ./out/note-head-scheme.o ./out/lily-imports.o ./out/font-config-scheme.o ./out/open-type-font-scheme.o ./out/hairpin.o ./out/quote-iterator.o ./out/ledger-line-engraver.o ./out/auto-beam-engraver.o ./out/pango-font.o ./out/misc.o ./out/repeated-music.o ./out/figured-bass-continuation.o ./out/grace-iterator.o ./out/cue-clef-engraver.o ./out/pango-select.o ./out/main.o ./out/music-wrapper-iterator.o ./out/pitch-squash-engraver.o ./out/tuplet-iterator.o ./out/stencil-interpret.o ./out/key-signature-interface.o ./out/pitched-trill-engraver.o ./out/score.o ./out/context-mod.o ./out/fingering-column.o ./out/music-function-scheme.o ./out/bar-number-engraver.o ./out/break-substitution.o ./out/sources.o ./out/piano-pedal-performer.o ./out/performance.o ./out/volta-repeat-iterator.o ./out/slash-repeat-engraver.o ./out/font-config.o ./out/lily-guile.o ./out/balloon.o ./out/text-spanner-engraver.o ./out/bezier.o ./out/stencil.o ./out/clef.o ./out/spacing-basic.o ./out/dynamic-engraver.o ./out/time-signature-engraver.o ./out/ottava-engraver.o ./out/music-output.o ./out/lily-parser.o ./out/ligature-bracket-engraver.o ./out/translator-dispatch-list.o ./out/lily-parser-scheme.o ./out/engraver-scheme.o ./out/dynamic-align-engraver.o ./out/part-combine-iterator.o ./out/custos.o ./out/translator-scheme.o ./out/slur-scoring.o ./out/accidental-engraver.o ./out/rest-collision.o ./out/staff-symbol-engraver.o ./out/spanner.o ./out/note-column-scheme.o ./out/piano-pedal-align-engraver.o ./out/clef-modifier.o ./out/note-performer.o ./out/pango-font-scheme.o ./out/melody-spanner.o ./out/tie-column.o ./out/tuplet-bracket.o ./out/episema-engraver.o ./out/lyric-extender.o ./out/grob-property.o ./out/lyric-combine-music.o ./out/vaticana-ligature.o ./out/cluster-engraver.o ./out/instrument-switch-engraver.o ./out/part-combine-part-iterator.o ./out/audio-column.o ./out/key-performer.o ./out/gregorian-ligature-engraver.o ./out/context-specced-music-iterator.o ./out/pango-select-scheme.o ./out/translator-group.o ./out/book.o ./out/stream-event.o ./out/staff-symbol.o ./out/font-interface.o ./out/lyric-engraver.o ./out/page-breaking.o ./out/engraver-group.o ./out/grob-smob.o ./out/grob-array-scheme.o ./out/rod.o ./out/moment.o ./out/dispatcher-scheme.o ./out/lilypond-version.o ./out/tab-staff-symbol-engraver.o ./out/font-select.o ./out/grob-interface-scheme.o ./out/tuplet-engraver.o ./out/balloon-engraver.o ./out/template5.o ./out/paper-column.o ./out/script-row-engraver.o ./out/clef-engraver.o ./out/one-page-breaking.o ./out/midi-chunk.o ./out/context-def.o ./out/paper-score-scheme.o ./out/axis-group-engraver.o ./out/cluster.o ./out/translator.o ./out/item-scheme.o ./out/double-percent-repeat-engraver.o ./out/separation-item.o ./out/key-engraver.o ./out/keep-alive-together-engraver.o ./out/source-file.o ./out/all-font-metrics-scheme.o ./out/self-alignment-interface.o ./out/axis-group-interface.o ./out/midi-cc-performer.o ./out/input-scheme.o ./out/lyric-performer.o ./out/grid-line-span-engraver.o ./out/glissando-engraver.o ./out/event-chord-iterator.o ./out/chord-name.o ./out/spring-smob.o ./out/score-performer.o ./out/paper-system-scheme.o ./out/coherent-ligature-engraver.o ./out/context-scheme.o ./out/tie-performer.o ./out/semi-tie.o ./out/chord-tremolo-engraver.o ./out/pitch.o ./out/listener.o ./out/percent-repeat-engraver.o ./out/note-head.o ./out/relative-octave-music.o ./out/note-spacing.o ./out/dimension-cache.o ./out/spacing-loose-columns.o ./out/melody-engraver.o ./out/page-spacing-result.o ./out/script-engraver.o ./out/ttf.o ./out/dot-formatting-problem.o ./out/all-font-metrics.o ./out/ledger-line-spanner.o ./out/staff-collecting-engraver.o ./out/spacing-options.o ./out/time-signature-performer.o ./out/pfb-scheme.o ./out/property-iterator.o ./out/vaticana-ligature-engraver.o ./out/script-interface.o ./out/score-scheme.o ./out/one-line-page-breaking.o ./out/page-marker-scheme.o ./out/spanner-scheme.o ./out/music-function.o ./out/ambitus-engraver.o ./out/new-fingering-engraver.o ./out/bar-check-iterator.o ./out/change-iterator.o ./out/context-handle.o ./out/parenthesis-engraver.o ./out/align-interface.o ./out/change-sequence-iterator.o ./out/column-x-positions.o ./out/output-def-scheme.o ./out/global-context-scheme.o ./out/tie-details.o ./out/rest.o ./out/note-collision.o ./out/dot-configuration.o ./out/tempo-performer.o ./out/translator-group-ctors.o ./out/ottava-bracket.o ./out/stem-tremolo.o ./out/mensural-ligature.o ./out/grob-pq-engraver.o ./out/lily-modules.o ./out/page-turn-page-breaking.o ./out/gregorian-ligature.o ./out/engraver.o ./out/rhythmic-music-iterator.o ./out/mark-engraver.o ./out/page-layout-problem-scheme.o ./out/vertical-align-engraver.o ./out/kievan-ligature-engraver.o ./out/tie-specification.o ./out/tuplet-number.o ./out/laissez-vibrer-engraver.o ./out/dots-engraver.o ./out/directional-element-interface.o ./out/concurrent-hairpin-engraver.o ./out/line-interface-scheme.o ./out/paper-book-scheme.o ./out/pointer-group-interface-scheme.o ./out/grob-interface.o ./out/open-type-font.o ./out/span-bar-engraver.o ./out/midi-stream.o ./out/page-layout-problem.o ./out/input-smob.o ./out/note-head-line-engraver.o ./out/bar-engraver.o ./out/text-interface.o ./out/text-engraver.o ./out/grob-scheme.o ./out/arpeggio.o ./out/semi-tie-column.o ./out/program-option-scheme.o ./out/footnote-engraver.o ./out/simultaneous-music-iterator.o ./out/breathing-sign-engraver.o ./out/grace-spacing-engraver.o ./out/lexer.o ./out/parser.o ./out/../../flower/out/library.a  -ldl   -L/home/arne/.local/lib -L/home/arne/.local/lib/guile/2.2 -lguile-2.2 -lgc  -lpangoft2-1.0 -lpango-1.0 -lgobject-2.0 -lglib-2.0 -lfontconfig -lfreetype  -lfontconfig -lfreetype  -lfreetype -v 2>&1 | grep -i lib
> nm out/lilypond
> LD_LIBRARY_PATH=~/.local/lib/  out/lilypond
> GUILE_LOAD_PATH=$GUILE_LOAD_PATH:../:../scm LD_LIBRARY_PATH=~/.local/lib/  out/lilypond
> wget http://infinite-hands.draketo.de/infinite_hands_sheet.ly
> wget -O mf/out/emmentaler-20.otf https://github.com/saebekassebil/subito/raw/master/resources/gonville/lilyfonts/otf/emmentaler-20.otf
> GUILE_LOAD_PATH=$GUILE_LOAD_PATH:.:scm LD_LIBRARY_PATH=~/.local/lib/  lily/out/lilypond -I ly/ -I mf/out/ -I ps/ test.ly
> cp lilypond-data_2.18.2-4.1_all.deb /tmp
> scp kav:/usr/share/lilypond/2.16.2/fonts/otf/*otf mf/out/
> GUILE_LOAD_PATH=$GUILE_LOAD_PATH:.:scm LD_LIBRARY_PATH=~/.local/lib/  lily/out/lilypond -I ly/ -I mf/out/ -I ps/ test.ly
> git add configure.ac lily/general-scheme.cc lily/pdf-scheme.cc scm/memory-trace.scm
> time GUILE_LOAD_PATH=$GUILE_LOAD_PATH:.:scm LD_LIBRARY_PATH=~/.local/lib/  lily/out/lilypond -I ly/ -I mf/out/ -I ps/ infinite_hands_sheet.ly
> cat 0001-Build-fixes-for-Guile-2.1.x-not-yet-functional.patch
> cat > 1 <<EOF
> diff --git a/lily/general-scheme.cc b/lily/general-scheme.cc
> index 1168ee9..2df63fc 100644
> --- a/lily/general-scheme.cc
> +++ b/lily/general-scheme.cc
> @@ -275,7 +275,8 @@ LY_DEFINE (ly_protects, "ly:protects",
>  #if SCM_MAJOR_VERSION < 2 || SCM_MAJOR_VERSION == 2 && SCM_MINOR_VERSION < 1
>    return scm_protects;
>  #else
> -  return programming_error ("ly:protects is not supported in Guile 2.1");
> +  // return programming_error ("ly:protects is not supported in Guile 2.1");
> +  return 0; // FIXME: Evil hack just to get this to build.
>  #endif
>  }
> EOF
>
> Best wishes,
> Arne
>
> PS: If this provides at least a small step towards guile 2 in lilypond,
>     it’s worth its while.
> --
> Unpolitisch sein
> heißt politisch sein
> ohne es zu merken

Hi Arne,

many thanks for your work!
I'll have a closer look tomorrow (it's in the middle of the night here)...

The main problem is not that we can't build lilypond with 2.0.14 or 2.1.7
Checkout the branch dev/guile-v2-work in the lilypond-repository (you
will need to rebase it against current master) and you'll be able to
build LilyPond with guile-2.0.13, so I'm pretty confident it will work
with 2.0.14. For building with guile 2.1.7 an additional patch is
needed, currently I've simply deleted all about ly:protects
(brute-force, I know...).

Though the perfomance of LilyPond is dramatically slowed down. Making
it effectively unusable for any longer input.
I'm currently testing this for builds with guile 1.8.8, 2.0.14 and
2.1.7 and will report tomorrow.

Cheers,
  Harm

Re: How to make GNU Guile more successful

[Thomas Morley]

Hi Arne,

as promised here my thoughts/findings.
All tested with a LilyPond build from the rebased branch
dev/guile-v2-work, with some additional patches on top of it:

diff --git a/lily/general-scheme.cc b/lily/general-scheme.cc
index 1168ee9..db376d11 100644
--- a/lily/general-scheme.cc
+++ b/lily/general-scheme.cc
@@ -267,6 +267,7 @@ LY_DEFINE (ly_dimension_p, "ly:dimension?", 1, 0,
0, (SCM d),
 /*
   Debugging mem leaks:
 */
+/*
 LY_DEFINE (ly_protects, "ly:protects",
            0, 0, 0, (),
            "Return hash of protected objects.")
@@ -278,6 +279,7 @@ LY_DEFINE (ly_protects, "ly:protects",
   return programming_error ("ly:protects is not supported in Guile 2.1");
 #endif
 }
+*/

 LY_DEFINE (ly_gettext, "ly:gettext",
            1, 0, 0, (SCM original),
diff --git a/scm/lily.scm b/scm/lily.scm
index d3164e4..23b1647 100644
--- a/scm/lily.scm
+++ b/scm/lily.scm
@@ -836,10 +836,11 @@ messages into errors.")

 (define-public (dump-gc-protects)
   (set! gc-protect-stat-count (1+ gc-protect-stat-count))
-  (let* ((protects (sort (hash-table->alist (ly:protects))
-                         (lambda (a b)
-                           (< (object-address (car a))
-                              (object-address (car b))))))
+  (let* (;(protects (sort (hash-table->alist (ly:protects))
+         ;                (lambda (a b)
+         ;                  (< (object-address (car a))
+         ;                     (object-address (car b))))))
+         (protects '())
          (out-file-name (string-append
                          "gcstat-" (number->string gc-protect-stat-count)
                          ".scm"))


diff --git a/ly/scheme-sandbox.ly b/ly/scheme-sandbox.ly
index 7dec0dc..1f233c0 100644
--- a/ly/scheme-sandbox.ly
+++ b/ly/scheme-sandbox.ly
@@ -13,4 +13,8 @@
 % requirements may be different.

 #(newline)
-#(scm-style-repl)
+#(if (guile-v2)
+     (begin
+       (use-modules (system repl repl))
+       (start-repl))
+     (scm-style-repl))



The ly-file used for testing can be found at
http://lists.gnu.org/archive/html/lilypond-user/2016-11/msg00948.html
As the author says, this file was not finished, so several errors and
warnings because of suboptimal user-input happen.
Quite typical for ongoing work and good for testings.
Compiling the file Urtext.ly returns a 159-pages-pdf.

2017-03-05 3:05 GMT+01:00 Thomas Morley <thomasmorley65@gmail.com>:
> 2017-03-05 0:44 GMT+01:00 Arne Babenhauserheide <arne_bab@web.de>:

>> I worked on testing Lilypond with user installed Guile 2.x, does that
>> count?
>>
>> Just to have it recorded somewhere, here’s a patch to lilypond along
>> with a copy of the bash history of the setup (cleaned up, it was many
>> times as long):
>>
>> ## patch
>>
>> From bd2ffea6f4c4c1ede13f5ac82d0a8ce31ccfe3c7 Mon Sep 17 00:00:00 2001
>> Subject: [PATCH] Build fixes for Guile 2.1.x (not yet functional)
>>
>> ---
>>  configure.ac         | 7 ++++++-
>>  lily/pdf-scheme.cc   | 4 ++++
>>  scm/memory-trace.scm | 3 ++-
>>  3 files changed, 12 insertions(+), 2 deletions(-)
>>
>> diff --git a/configure.ac b/configure.ac
>> index d77ea15..393976b 100644
>> --- a/configure.ac
>> +++ b/configure.ac
>> @@ -267,7 +267,12 @@ STEPMAKE_FREETYPE2(freetype2, REQUIRED, 2.1.10)
>>  STEPMAKE_WINDOWS
>>
>>  # guile executable for some scripts
>> -STEPMAKE_GUILE(OPTIONAL, 1.8.2, 1.9.0)
>> +if test "$GUILEv2" = "yes"
>> +then
>> +    STEPMAKE_GUILE(OPTIONAL, 2.0.7, 2.2.0)
>> +else
>> +    STEPMAKE_GUILE(OPTIONAL, 1.8.2, 1.9.0)
>> +fi

The above is already in, see
$ git log -p 91ff9563ebe1c1cd720ad1a44890e7375fd83da8


>>  # perl for help2man and for mf2pt1.pl
>>  STEPMAKE_PERL(REQUIRED)
>> diff --git a/lily/pdf-scheme.cc b/lily/pdf-scheme.cc
>> index da2ce2c..f5ae70c 100644
>> --- a/lily/pdf-scheme.cc
>> +++ b/lily/pdf-scheme.cc
>> @@ -91,7 +91,11 @@ LY_DEFINE (ly_encode_string_for_pdf, "ly:encode-string-for-pdf",
>>     *                  (string->utf16 str 'big)
>>     */
>>    if (g)
>> +#if GUILEV2
>> +    return scm_take_locale_stringn (g, bytes_written); // scm_take_str eventually frees g!
>> +#else
>>      return scm_take_str (g, bytes_written); // scm_take_str eventually frees g!
>> +#endif
>>    else
>>      return str;
>>  }

Instead of the above, currently this code is in:

diff --git a/lily/pdf-scheme.cc b/lily/pdf-scheme.cc
index da2ce2c..61cf382 100644
--- a/lily/pdf-scheme.cc
+++ b/lily/pdf-scheme.cc
@@ -84,14 +84,16 @@ LY_DEFINE (ly_encode_string_for_pdf,
"ly:encode-string-for-pdf",
   free (p);

   /* Convert back to SCM object and return it */
-  /* FIXME guile-2.0: With guile 2.0 the internal representation of a string
-   *                  has changed (char vector rather than binary bytes in
-   *                  UTF-8). However, with guile 2.0, ly:encode-string-for-pdf
-   *                  is no longer needed and can be replaced by the new
-   *                  (string->utf16 str 'big)
-   */
   if (g)
-    return scm_take_str (g, bytes_written); // scm_take_str eventually frees g!
+    {
+      /*
+       * Return the raw byte representation of the UTF-16BE encoded string,
+       * in a locale independent way.
+       */
+      SCM string = scm_from_latin1_stringn (g, bytes_written);
+      free(g);
+      return string;
+    }
   else
     return str;
 }
See:
git log -p d15c38c0ddd4c04edcf82cda50ca30f6dc4941fa

Though, I tested your suggestion as well.
It changes one of our regtests, utf-8.ly. But I have to admit I tested
only this one (not all regtests) and did not try to build the docs.
(All this stuff is pretty heavy for my admittedly weak laptop, it
lasts ages...)
Compiling the above mentioned Urtext.ly-file with your proposal shows
no significant difference for timings (see below)


>> diff --git a/scm/memory-trace.scm b/scm/memory-trace.scm
>> index d8ffeb9..9ebd722 100644
>> --- a/scm/memory-trace.scm
>> +++ b/scm/memory-trace.scm
>> @@ -2,7 +2,8 @@
>>
>>  (define-module (scm memory-trace))
>>  (use-modules (lily)
>> -             (ice-9 format))
>> +             (ice-9 format)
>> +             (ice-9 threads))
>>
>>  (define-public (mtrace:start-trace freq)
>>    (set! usecond-interval (inexact->exact (/ 1000000 freq)))
>> --
>> 2.10.2

Thanks for the above.
It fixes an annoying warning when LilyPond uses guile2.
I'll have to test, but I guess it's compatible with guile-1.
If so I'll likely check it in, ofcourse giving proper credits ;)

[...]
>>
>> PS: If this provides at least a small step towards guile 2 in lilypond,
>>     it’s worth its while.
>> --
>> Unpolitisch sein
>> heißt politisch sein
>> ohne es zu merken
>
> Hi Arne,
>
> many thanks for your work!
> I'll have a closer look tomorrow (it's in the middle of the night here)...
>
> The main problem is not that we can't build lilypond with 2.0.14 or 2.1.7
> Checkout the branch dev/guile-v2-work in the lilypond-repository (you
> will need to rebase it against current master) and you'll be able to
> build LilyPond with guile-2.0.13, so I'm pretty confident it will work
> with 2.0.14. For building with guile 2.1.7 an additional patch is
> needed, currently I've simply deleted all about ly:protects
> (brute-force, I know...).
>
> Though the perfomance of LilyPond is dramatically slowed down. Making
> it effectively unusable for any longer input.
> I'm currently testing this for builds with guile 1.8.8, 2.0.14 and
> 2.1.7 and will report tomorrow.

Here some timing values

(1)
lilypond-2.19.52 using guile 1.8.7
(I would have prefered to build lilypond with a guile-1.8.8 build from
the guile-repository. Though my try to build it from the
branch_release-1-8 failed. Instead attempting to fix it, I then used a
released lilypond-version)

real    8m16.191s
user    6m39.864s
sys    0m10.860s

(2)
guile-2.0.14 build from guile-git-repository, branch remotes/origin/stable-2.0
lilypond-2.19.56, build from local branch dev/guile-v2.2-work

real    34m11.762s
user    45m11.316s
sys    0m5.604s

(3)
guile-2.1.7 build from guile-git-repository, branch master
(I've got this warning:
configure: WARNING: *** GNU Readline is too old on your system.
configure: WARNING: *** You need readline version 2.1 or later.
No idea whether this may have an impact on lilyponds compiling-time
I'll have to test.)
lilypond-2.19.56, build from local branch dev/guile-v2.2-work

real    67m29.132s
user    93m14.812s
sys    0m7.332s


Thanks,
  Harm

Re: How to make GNU Guile more successful

[David Kastrup]

Thomas Morley <thomasmorley65@gmail.com> writes:

> Here some timing values
>
> (1)
> lilypond-2.19.52 using guile 1.8.7
> (I would have prefered to build lilypond with a guile-1.8.8 build from
> the guile-repository. Though my try to build it from the
> branch_release-1-8 failed. Instead attempting to fix it, I then used a
> released lilypond-version)
>
> real    8m16.191s
> user    6m39.864s
> sys    0m10.860s
>
> (2)
> guile-2.0.14 build from guile-git-repository, branch remotes/origin/stable-2.0
> lilypond-2.19.56, build from local branch dev/guile-v2.2-work
>
> real    34m11.762s
> user    45m11.316s
> sys    0m5.604s
>
> (3)
> guile-2.1.7 build from guile-git-repository, branch master
> (I've got this warning:
> configure: WARNING: *** GNU Readline is too old on your system.
> configure: WARNING: *** You need readline version 2.1 or later.
> No idea whether this may have an impact on lilyponds compiling-time
> I'll have to test.)
> lilypond-2.19.56, build from local branch dev/guile-v2.2-work
>
> real    67m29.132s
> user    93m14.812s
> sys    0m7.332s

Same compilation options?  This looks like a surprisingly serious
regression compared to 2.0.

-- 
David Kastrup

Re: How to make GNU Guile more successful

[Thomas Morley]

2017-03-05 15:09 GMT+01:00 David Kastrup <dak@gnu.org>:
> Thomas Morley <thomasmorley65@gmail.com> writes:
>
>> Here some timing values
>>
>> (1)
>> lilypond-2.19.52 using guile 1.8.7
>> (I would have prefered to build lilypond with a guile-1.8.8 build from
>> the guile-repository. Though my try to build it from the
>> branch_release-1-8 failed. Instead attempting to fix it, I then used a
>> released lilypond-version)
>>
>> real    8m16.191s
>> user    6m39.864s
>> sys    0m10.860s
>>
>> (2)
>> guile-2.0.14 build from guile-git-repository, branch remotes/origin/stable-2.0
>> lilypond-2.19.56, build from local branch dev/guile-v2.2-work
>>
>> real    34m11.762s
>> user    45m11.316s
>> sys    0m5.604s
>>
>> (3)
>> guile-2.1.7 build from guile-git-repository, branch master
>> (I've got this warning:
>> configure: WARNING: *** GNU Readline is too old on your system.
>> configure: WARNING: *** You need readline version 2.1 or later.
>> No idea whether this may have an impact on lilyponds compiling-time
>> I'll have to test.)
>> lilypond-2.19.56, build from local branch dev/guile-v2.2-work
>>
>> real    67m29.132s
>> user    93m14.812s
>> sys    0m7.332s
>
> Same compilation options?

Yep.
To get comparable results I always did exactly the same, for building
the guile-versions as well as for building lilypond.
Doing all tests with a fresh restarted computer.


Cheers,
  Harm

> This looks like a surprisingly serious
> regression compared to 2.0.
>
> --
> David Kastrup

Re: How to make GNU Guile more successful

[Thomas Morley]

2017-03-05 15:01 GMT+01:00 Thomas Morley <thomasmorley65@gmail.com>:

> The above is already in, see
> $ git log -p 91ff9563ebe1c1cd720ad1a44890e7375fd83da8
[...}
> See:
> git log -p d15c38c0ddd4c04edcf82cda50ca30f6dc4941fa

Aargh, those are from my local _rebased_ branch.
Try instead:
e23d7b1763fdc2e7815c7069d5d1702f1a132383
59197ffbf370b12986ef3f3ee225322048110477

Sorry,
  Harm

Re: How to make GNU Guile more successful

[Matt Wette]

> On Mar 2, 2017, at 9:28 PM, Nala Ginrut <nalaginrut@gmail.com> wrote:

> 2. How to make guile platform more successful?
> I this case, let me ask a question, if we have guile-python3
> (compatible with most of Python3 features), and provide more
> convenient FFI helper function to help bind more libraries in short
> time, is it possible to attract more people from Python land? Given
> we'll have good JIT compiler in the future.

Now that the C preprocessor in NYACC/C99 is more robust I could start looking at generating some sort of  “FFI helper” functionality.  I have been thinking about this problem on and off for a while.  Completely automating things might be tough, but a “helper” would probably go a long way.

Matt

Re: How to make GNU Guile more successful

[Amirouche]

Le 03/03/2017 à 18:21, Matt Wette a écrit :
>> On Mar 2, 2017, at 9:28 PM, Nala Ginrut <nalaginrut@gmail.com> wrote:
>> 2. How to make guile platform more successful?
>> I this case, let me ask a question, if we have guile-python3
>> (compatible with most of Python3 features), and provide more
>> convenient FFI helper function to help bind more libraries in short
>> time, is it possible to attract more people from Python land? Given
>> we'll have good JIT compiler in the future.
> Now that the C preprocessor in NYACC/C99 is more robust I could start looking at generating some sort of  “FFI helper” functionality.  I have been thinking about this problem on and off for a while.  Completely automating things might be tough, but a “helper” would probably go a long way.
>
> Matt
Tx for working on that!

Re: How to make GNU Guile more successful

[Amirouche]

Le 03/03/2017 à 18:21, Matt Wette a écrit :
>> On Mar 2, 2017, at 9:28 PM, Nala Ginrut <nalaginrut@gmail.com> wrote:
>> 2. How to make guile platform more successful?
>> I this case, let me ask a question, if we have guile-python3
>> (compatible with most of Python3 features), and provide more
>> convenient FFI helper function to help bind more libraries in short
>> time, is it possible to attract more people from Python land? Given
>> we'll have good JIT compiler in the future.
> Now that the C preprocessor in NYACC/C99 is more robust I could start looking at generating some sort of  “FFI helper” functionality.  I have been thinking about this problem on and off for a while.  Completely automating things might be tough, but a “helper” would probably go a long way.
>
> Matt
>
>
Are you aware of scheme-bytestructures [0] that's what we are using
in guile-git. If you target that API to wrap C structs, it would be helpful.

[0] https://github.com/TaylanUB/scheme-bytestructures

Re: How to make GNU Guile more successful

[Mike Gran]

> On Friday, March 3, 2017 11:18 AM, Amirouche <amirouche@hypermove.net> wrote:

>> Now that the C preprocessor in NYACC/C99 is more robust I could start looking at generating some sort of  “FFI helper” functionality.  I have been thinking about this problem on and off for a while.  Completely automating things might be tough, but a “helper” would probably go a long way.


> Are you aware of scheme-bytestructures [0] that's what we are using> in guile-git. If you target that API to wrap C structs, it would be helpful.

SWIG used to work for Guile.  It was a rather complete solution to
the wrapping problem.

Re: How to make GNU Guile more successful

[Matt Wette]

> On Mar 3, 2017, at 11:24 AM, Mike Gran <spk121@yahoo.com> wrote:
> 
>> On Friday, March 3, 2017 11:18 AM, Amirouche <amirouche@hypermove.net> wrote:
> 
>>> Now that the C preprocessor in NYACC/C99 is more robust I could start looking at generating some sort of  “FFI helper” functionality.  I have been thinking about this problem on and off for a while.  Completely automating things might be tough, but a “helper” would probably go a long way.
> 
> 
>> Are you aware of scheme-bytestructures [0] that's what we are using> in guile-git. If you target that API to wrap C structs, it would be helpful.
> 
> SWIG used to work for Guile.  It was a rather complete solution to
> the wrapping problem.

I had forgotten about that.  Seems targeted to 1.X but might be worth trying to autogenerate swig template or equivalent to start.

Re: How to make GNU Guile more successful

[Matt Wette]

> On Mar 3, 2017, at 11:16 AM, Amirouche <amirouche@hypermove.net> wrote:
>> Now that the C preprocessor in NYACC/C99 is more robust I could start looking at generating some sort of  “FFI helper” functionality.  I have been thinking about this problem on and off for a while.  Completely automating things might be tough, but a “helper” would probably go a long way.
> Are you aware of scheme-bytestructures [0] that's what we are using
> in guile-git. If you target that API to wrap C structs, it would be helpful.
> 
> [0] https://github.com/TaylanUB/scheme-bytestructures <https://github.com/TaylanUB/scheme-bytestructures>
Cloned it.  Thanks for the reference. — Matt

Re: How to make GNU Guile more successful

[Michael Vehrs]

On 02/21/2017 07:01 AM, Michael Vehrs wrote:
> On 02/20/2017 09:41 PM, Arne Babenhauserheide wrote:
>> Michael Vehrs <Michael.Burschik@gmx.de> writes:
>>
>>> As a late-comer to this discussion, here are my two cents. The thing I
>>> miss most is a central package repository like Eggs Unlimited
>>> (http://wiki.call-cc.org/chicken-projects/egg-index-4.html), or the
>>> Racket Package List (http://pkgs.racket-lang.org/), or CPAN, of course.
>>> Sure, a bespoke package manager might be nifty, but a single curated
>>> list of packages would be a game-changer.
>> In theory we have guildhall for this: https://github.com/ijp/guildhall
>
> In theory, yes. But there isn't actually very much available in the 
> repository.
>
>>
>> In practice it does not provide a web interface for uploading packages.
>>
>> If you want to do something truly exciting, you could take wingos fibers
>> and build a high performance web interface for guildhall with them.
>
> High performance is not really important in this case. We are not 
> talking about gazillions of npm packages.
>
>>
>> This is something I’d love to do but I fear that it’s not high enough in
>> my todo list that I’ll actually get it done.¹
>>
>> Best wishes,
>> Arne
>>
>> ¹: I became Freenet Release Manager a few months ago. That took away the
>>     last free time I could allocate to new challenging projects.
>
> If someone had a realistic plan of building something like that, I 
> might be able to contribute. I am not going to tackle it alone.
>
>
> Regards
>
> Michael
>
>

By the way, this (http://sph.mn/content/3e73) is the most complete list 
of guile projects I have seen so far.


Regards

Michael

Re: How to make GNU Guile more successful

[Alejandro Sanchez]

If I may add my two cents as a Scheme layman: the question is not so much about making Guile more popular, but about making Scheme itself more popular.

One big reason for Python’s popularity is something I haven’t seen mentioned in this thread so far: if you know pseudocode you basically know Python already. Of course this is hyperbolic, there are a lot of finer details to Python, but the superficial simplicity of Python makes you think you already know the language and that you can get started right away. By the time you encounter the more arcane aspects of Python you have already invested enough time into it that you will put up with learning something new.

Scheme on the other hand is weird; weird in a good way, but still weird. For me the reason for picking up Scheme was working my way through SICP, but even before that I had heard of the Lisp family. Every time I tried to find out what the fuss was about Lisp all I could find was nebulous concepts like “it’s a programmable programming language” without explaining what that even meant and why I would ever want to use it.

Of course once I got over the weird parentheses and I understood why they are actually a beneficial aspect I liked it. Yet, even after getting into Scheme there are problems. The r5rs standard is pretty limited, and so every implementation does its own thing. There was not even a standard module system until r6rs. Unfortunately r6rs never got fully adopted by any of the major implementations, so we are back to where we were with r5rs more or less. r7rs-small looks like it could finally fill in the most glaring gaps, and r7rs-large is still nowhere done.

All this is very frustrating, and getting into a new language is a long-term commitment to learn something new. When faced with Python, Ruby, JavaScript, Lua and Scheme, it’s easy to go with the flow and just use what everyone else is using.

I think someone here mentioned the lack of a proper Scheme IDE, other than Dr. Racket for Racket. I don’t use IDEs anymore, but I can see how that can be a problem for other people who only want to do intermediate scripting rather than write entire applications. Writing a full IDE would be quite a lot of work, so perhaps integrating with existing IDEs would be better. Think something like Geiser for programs other than Emacs. There is the “language server protocol" by Microsoft, the idea is to have a standardised protocol that can be supported by a language server and an editor application.
https://github.com/Microsoft/language-server-protocol

I know it’s Microsoft, but this is just a specification. With this protocol one would write a Scheme server that receives instructions form the editor via RPC and sends the result back to the server via RPC as well. This would allow using the same Scheme server regardless of the editor (Emacs, Vim, Neovim, Eclipse, …). I had wanted to write something like this for Neovim myself, but I don’t know enough of Scheme yet.

I hope these ramblings of mine offer something useful from an outsider perspective to you guys.

Re: How to make GNU Guile more successful

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1827 bytes --]


Alejandro Sanchez <hiphish@openmailbox.org> writes:

> If I may add my two cents as a Scheme layman: the question is not so much about making Guile more popular, but about making Scheme itself more popular.
>
> One big reason for Python’s popularity is something I haven’t seen
> mentioned in this thread so far: if you know pseudocode you basically
> know Python already. Of course this is hyperbolic, there are a lot of
> finer details to Python, but the superficial simplicity of Python
> makes you think you already know the language and that you can get
> started right away. By the time you encounter the more arcane aspects
> of Python you have already invested enough time into it that you will
> put up with learning something new.

This is basically what wisp¹ is intended to address:

define : hello who
    format #t "Hello ~a!\n" who

hello "World"

And it’s still full scheme. Just looking like pseudocode.

I documented that in py2guile: http://draketo.de/py2guile

¹: http://draketo.de/english/wisp

> I think someone here mentioned the lack of a proper Scheme IDE, other
> than Dr. Racket for Racket. I don’t use IDEs anymore, but I can see
> how that can be a problem for other people who only want to do
> intermediate scripting rather than write entire applications.

I also see lots of people at work using Python IDEs, which all bring
their own problems but provide the kind of information I get from the
commandline.

Just having a geiser setup for Emacs properly documented — or maybe an
Emacs customized for Scheme development — would help a lot, I think.

"This is our canonical interface for writing Scheme" ← That is what
people need to be able to find.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

Re: How to make GNU Guile more successful

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 631 bytes --]


() Arne Babenhauserheide <arne_bab@web.de>
() Sun, 05 Mar 2017 01:23:59 +0100

   Just having a geiser setup for Emacs properly documented — or
   maybe an Emacs customized for Scheme development — would help
   a lot, I think.

Could you please summarize (or point to a summary of) Geiser
documentation deficiencies?

-- 
Thien-Thi Nguyen -----------------------------------------------
 (defun responsep (query)
   (pcase (context query)
     (`(technical ,ml) (correctp ml))
     ...))                              748E A0E8 1CB8 A748 9BFA
--------------------------------------- 6CE4 6703 2224 4C80 7502

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: How to make GNU Guile more successful

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1940 bytes --]


Thien-Thi Nguyen <ttn@gnu.org> writes:

> () Arne Babenhauserheide <arne_bab@web.de>
> () Sun, 05 Mar 2017 01:23:59 +0100
>
>    Just having a geiser setup for Emacs properly documented — or
>    maybe an Emacs customized for Scheme development — would help
>    a lot, I think.
>
> Could you please summarize (or point to a summary of) Geiser
> documentation deficiencies?

I can’t just say "give me the canonical Guile development environment"
and have that work without any additional setup needed from me. This is
not about the documentation of Geiser itself but rather about the
documentation of "how to make Emacs the best tool for hacking Scheme".

The following contains user experience questions of coming to
geiser. These will sound unfair, because they assume a user who did not
read any documentation and just wants to work on a Scheme file given to
him/her by a co-worker, likely with a sentence like "here’s the file,
you’ll need to adapt it for your usecase".

This is how I come to geiser:


First I use M-x package-list-packages to find and install geiser (Why is
geiser not shipped with Emacs (like org-mode)?).

Then I open a Scheme file, then I type M-x geiser followed by guile (why
doesn’t geiser start by default? Why do I have to know that the best way
to write scheme is called geiser?).

Finally I add M-x geiser-mode in the file (why does this not happen
automatically? — or: why is there no information in the REPL that I can
do so?).


Now I have a Geiser repl and a function tool tip in the echo area.

Is this the canonical setup people use? What else is there I do not get
right away? From the IDEs co-workers use I know display of local
variables (and their values). And warnings and errors shown in the
fringe (I get that in Python with flycheck mode). Is this also provided
by geiser (maybe harnessing some other tool)?


Best wishes,
Arne

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

Re: How to make GNU Guile more successful

[Erik Edrosa]

On 03/04/2017 06:41 PM, Alejandro Sanchez wrote:
> If I may add my two cents as a Scheme layman: the question is not so much about making Guile more popular, but about making Scheme itself more popular.
> 
> One big reason for Python’s popularity is something I haven’t seen mentioned in this thread so far: if you know pseudocode you basically know Python already. Of course this is hyperbolic, there are a lot of finer details to Python, but the superficial simplicity of Python makes you think you already know the language and that you can get started right away. By the time you encounter the more arcane aspects of Python you have already invested enough time into it that you will put up with learning something new.
> 
> Scheme on the other hand is weird; weird in a good way, but still weird. For me the reason for picking up Scheme was working my way through SICP, but even before that I had heard of the Lisp family. Every time I tried to find out what the fuss was about Lisp all I could find was nebulous concepts like “it’s a programmable programming language” without explaining what that even meant and why I would ever want to use it.
> 
> Of course once I got over the weird parentheses and I understood why they are actually a beneficial aspect I liked it. Yet, even after getting into Scheme there are problems. The r5rs standard is pretty limited, and so every implementation does its own thing. There was not even a standard module system until r6rs. Unfortunately r6rs never got fully adopted by any of the major implementations, so we are back to where we were with r5rs more or less. r7rs-small looks like it could finally fill in the most glaring gaps, and r7rs-large is still nowhere done.
> 
> All this is very frustrating, and getting into a new language is a long-term commitment to learn something new. When faced with Python, Ruby, JavaScript, Lua and Scheme, it’s easy to go with the flow and just use what everyone else is using.
> 
> I think someone here mentioned the lack of a proper Scheme IDE, other than Dr. Racket for Racket. I don’t use IDEs anymore, but I can see how that can be a problem for other people who only want to do intermediate scripting rather than write entire applications. Writing a full IDE would be quite a lot of work, so perhaps integrating with existing IDEs would be better. Think something like Geiser for programs other than Emacs. There is the “language server protocol" by Microsoft, the idea is to have a standardised protocol that can be supported by a language server and an editor application.
> https://github.com/Microsoft/language-server-protocol
> 
> I know it’s Microsoft, but this is just a specification. With this protocol one would write a Scheme server that receives instructions form the editor via RPC and sends the result back to the server via RPC as well. This would allow using the same Scheme server regardless of the editor (Emacs, Vim, Neovim, Eclipse, …). I had wanted to write something like this for Neovim myself, but I don’t know enough of Scheme yet.
> 
> I hope these ramblings of mine offer something useful from an outsider perspective to you guys.
> 

I think one issue when you want to compare languages like Python to
Scheme is that these languages are dominated by a single implementation.
Python, Ruby, Go, Rust, Clojure, and many others have a single dominate
implementation. The compatibility of libraries isn't something devs
usually have to worry about (except for breaking changes in the language
like Python 2 to Python 3). I agree with how r5rs standard is pretty
limited, languages with multiple implementations tend to have pretty
large standards so that these implementations are somewhat compatible. I
am liking the direction of r7rs and I hope it will bring scheme
implementations closer together. Probably the other option for scheme
would be for everyone to get behind a single implementation like Chez
Scheme.

Of course that isn't the only thing needed to make Scheme or Guile more
popular. I believe a programming language needs a somewhat unique and
powerful library or framework, an awesome application to show off the
power of the language, good tooling, and some luck.

The language server protocol is pretty cool and I would like to see a
lisp implement it. An IDE for a lisp obviously needs to implement more
features to be useful, I believe being able to easily interact with a
REPL is one of those.

Re: How to make GNU Guile more successful

[Jan Wedekind]

According to Alan Kay's research, the future might be a lot of domain specific mini languages. I.e. languages tomorrow are going to be like libraries today. Guile supports custom languages which makes it future proof at least to some extent.

Myself, I am using Scheme because of it's unique metaprogramming facilities (Scheme macros) and GNU Guile because it has multiple-dispatch object oriented programming (unlike Racket). Also Guild's native interface with the Boehm Garbage Collector is quite easy to use.

On 5. März 2017 03:09:54 GMT+00:00, Erik Edrosa <erik.edrosa@gmail.com> wrote:
>On 03/04/2017 06:41 PM, Alejandro Sanchez wrote:
>> If I may add my two cents as a Scheme layman: the question is not so
>much about making Guile more popular, but about making Scheme itself
>more popular.
>> 
>> One big reason for Python’s popularity is something I haven’t seen
>mentioned in this thread so far: if you know pseudocode you basically
>know Python already. Of course this is hyperbolic, there are a lot of
>finer details to Python, but the superficial simplicity of Python makes
>you think you already know the language and that you can get started
>right away. By the time you encounter the more arcane aspects of Python
>you have already invested enough time into it that you will put up with
>learning something new.
>> 
>> Scheme on the other hand is weird; weird in a good way, but still
>weird. For me the reason for picking up Scheme was working my way
>through SICP, but even before that I had heard of the Lisp family.
>Every time I tried to find out what the fuss was about Lisp all I could
>find was nebulous concepts like “it’s a programmable programming
>language” without explaining what that even meant and why I would ever
>want to use it.
>> 
>> Of course once I got over the weird parentheses and I understood why
>they are actually a beneficial aspect I liked it. Yet, even after
>getting into Scheme there are problems. The r5rs standard is pretty
>limited, and so every implementation does its own thing. There was not
>even a standard module system until r6rs. Unfortunately r6rs never got
>fully adopted by any of the major implementations, so we are back to
>where we were with r5rs more or less. r7rs-small looks like it could
>finally fill in the most glaring gaps, and r7rs-large is still nowhere
>done.
>> 
>> All this is very frustrating, and getting into a new language is a
>long-term commitment to learn something new. When faced with Python,
>Ruby, JavaScript, Lua and Scheme, it’s easy to go with the flow and
>just use what everyone else is using.
>> 
>> I think someone here mentioned the lack of a proper Scheme IDE, other
>than Dr. Racket for Racket. I don’t use IDEs anymore, but I can see how
>that can be a problem for other people who only want to do intermediate
>scripting rather than write entire applications. Writing a full IDE
>would be quite a lot of work, so perhaps integrating with existing IDEs
>would be better. Think something like Geiser for programs other than
>Emacs. There is the “language server protocol" by Microsoft, the idea
>is to have a standardised protocol that can be supported by a language
>server and an editor application.
>> https://github.com/Microsoft/language-server-protocol
>> 
>> I know it’s Microsoft, but this is just a specification. With this
>protocol one would write a Scheme server that receives instructions
>form the editor via RPC and sends the result back to the server via RPC
>as well. This would allow using the same Scheme server regardless of
>the editor (Emacs, Vim, Neovim, Eclipse, …). I had wanted to write
>something like this for Neovim myself, but I don’t know enough of
>Scheme yet.
>> 
>> I hope these ramblings of mine offer something useful from an
>outsider perspective to you guys.
>> 
>
>I think one issue when you want to compare languages like Python to
>Scheme is that these languages are dominated by a single
>implementation.
>Python, Ruby, Go, Rust, Clojure, and many others have a single dominate
>implementation. The compatibility of libraries isn't something devs
>usually have to worry about (except for breaking changes in the
>language
>like Python 2 to Python 3). I agree with how r5rs standard is pretty
>limited, languages with multiple implementations tend to have pretty
>large standards so that these implementations are somewhat compatible.
>I
>am liking the direction of r7rs and I hope it will bring scheme
>implementations closer together. Probably the other option for scheme
>would be for everyone to get behind a single implementation like Chez
>Scheme.
>
>Of course that isn't the only thing needed to make Scheme or Guile more
>popular. I believe a programming language needs a somewhat unique and
>powerful library or framework, an awesome application to show off the
>power of the language, good tooling, and some luck.
>
>The language server protocol is pretty cool and I would like to see a
>lisp implement it. An IDE for a lisp obviously needs to implement more
>features to be useful, I believe being able to easily interact with a
>REPL is one of those.

-- 
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

Re: How to make GNU Guile more successful

[Erik Edrosa]

On 03/05/2017 06:57 AM, Jan Wedekind wrote:
> According to Alan Kay's research, the future might be a lot of domain
> specific mini languages. I.e. languages tomorrow are going to be like
> libraries today. Guile supports custom languages which makes it future
> proof at least to some extent.
> 

I think I can see how domain specific mini languages might be the
future. You can look at what has happened in web development to see this
explosion of JavaScript transpilers, hoping to either extend or provide
new ways to design web applications.

> Myself, I am using Scheme because of it's unique metaprogramming
> facilities (Scheme macros) and GNU Guile because it has
> multiple-dispatch object oriented programming (unlike Racket). Also
> Guild's native interface with the Boehm Garbage Collector is quite easy
> to use.
> 
> 
> -- 
> Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail
> gesendet.

I started using GNU Guile after reading about rms, emacs, and GNU Guile
where GNU Guile would be used as a similar way to extend software. Then
I fell in love with scheme.

Re: How to make GNU Guile more successful

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1360 bytes --]


Erik Edrosa <erik.edrosa@gmail.com> writes:

> I think one issue when you want to compare languages like Python to
> Scheme is that these languages are dominated by a single
> implementation.

I read this argument repeatedly, but it does not apply to C and C++, two
of the most widely used languages. Therefore that argument might just
interpret too much into a coincidence — or rather: into a situation
which happens in the beginning of language development.

Even Java nowadays has several different implementations.

> Of course that isn't the only thing needed to make Scheme or Guile more
> popular. I believe a programming language needs a somewhat unique and
> powerful library or framework

From what I know, that’s what ruby had: many people wanted to use rails
and after they already knew ruby well, they also used it for other
tasks.

For Python I’m not sure.

Java had that big company behind it, which marketed it as the best
choice for companies.

> an awesome application to show off the
> power of the language, good tooling, and some luck.

And zealots, never forget zealots. I’d dare to say that people, who
believe that the language or tool is the best thing in existence and
give lots of talks about it, are a far too easily underestimated factor
in the success of any tool.

Best wishes,
Arne

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

Re: How to make GNU Guile more successful

[Nala Ginrut]

Folks, I have to claim that life is not a popularity contest. That's why I
always dislike to compare to Python on popularity topic. Of course, it's
good to make Guile more successful, but please don't forget the popularity
is the result, and the productive and convenient (both in technics and
community management) is the reason.
For me, I have already started projects to solve part of productive issues.
And I think a more usable Guildhall is very important on the other hand.
I'll appreciate if someone could put effort on it. :-)

Best regards.

Re: How to make GNU Guile more successful

[Alejandro Sanchez]

Thank you for you responses, I will try to respond to them all in one email.


> About Emacs + Geiser as the default development environment
Emacs is a power tool. Giving someone Emacs to write a simple script is like handing someone a jackhammer to nail plywood together. OK, maybe I’m stretching the analogy a bit, but when I think IDE I imagine something that handholds people, for the better or worse. Sure, you could offer some sort of pre-configured Emacs setup, but what’s the point of that?

This is why I had brought up the Language Server protocol, integrating Scheme with an existing IDE allows people to jump right in with whatever tool they are familiar with. It should also be less work to write a server or modify Geiser to follow the Language Server protocol than to maintain a separate distribution of Emacs.

> About the lack of a standard implementation
As Arne Babenhauserheide pointed out, there is no standard implementation of C or C++ either, and yet those languages manage just fine. The difference is in *what* is undefined. Undefined in C/C++ usually means “do not rely on this, demons might fly out of your nose” and you are not supposed to touch it unless you really know what your compiler will do. Most of the undefined parts of C/C++ can be avoided in a portable way.

Undefined in Scheme before r6rs basically meant “we couldn’t all agree on one thing, so whatever”. So you have to pick one implementation and roll with it. My understanding is that’s how PLT-Scheme turned into Racket and became its own thing. This makes my code completely dependent on the implementation’s lifespan, and that’s a commitment that’s hard to sell.

From what I understand, with r6rs and r7rs the voting has been changed from unanimous to majory-vote, so this should become less of an issue over time.

> About the syntax
I know about wisp, but that looks like an awkward cross-breed between Python and Lisp. Lisp syntax is simply something one needs to get over. For me it really clicked when I understood that Lisp syntax is not really a syntax, it’s the AST (sort of). Wisp reminds me of those people who use the arrow keys Vim and never leave insert-mode, in the end they just have a convoluted Nano. Maybe Wisp will help people, I don’t know, but it didn’t do anything for me.

> Closing remark
The question for someone shopping around for languages is: what is the selling point for Guile (or Scheme)? Being a programmable programming language sounds cool, but what does it let me do? Nothing on Guile's website tells me “look at how easy this is to solve in Guile, you cannot do this in other languages”. What would that be? I don’t know, I’m just a layman. The topic of the thread is “how to make Guile more successful”, and I most of those are technical points, so I’m afraid that discussing popularity is all I can contribute.

Re: How to make GNU Guile more successful

[Arne Babenhauserheide]

Alejandro Sanchez writes:

> Thank you for you responses, I will try to respond to them all in one email.
>
>
>> About Emacs + Geiser as the default development environment
> Emacs is a power tool. Giving someone Emacs to write a simple script is like handing someone a jackhammer to nail plywood together. OK, maybe I’m stretching the analogy a bit, but when I think IDE I imagine something that handholds people, for the better or worse. Sure, you could offer some sort of pre-configured Emacs setup, but what’s the point of that?

The point of that is that every developer starts by doing small
things. Therefore the small startup things should work well.

I know many people at work who’ve been using Emacs for years but mostly
limit their use to what it provides by default.

What works well from the start is where most people will stay. And
that’s not unreasonable: If it doesn’t work now, how are the chances
that it will be better than something which works well right away?

>> About the syntax
> I know about wisp, but that looks like an awkward cross-breed between Python and Lisp. Lisp syntax is simply something one needs to get over. For me it really clicked when I understood that Lisp syntax is not really a syntax, it’s the AST (sort of). Wisp reminds me of those people who use the arrow keys Vim and never leave insert-mode, in the end they just have a convoluted Nano. Maybe Wisp will help people, I don’t know, but it didn’t do anything for me.

The difference to vim in insert mode is that wisp provides all features
of scheme. The opening and closing parens just look different :)

(or rather: they are merged into the existing indentation, so this
reduces duplication of information)

>> Closing remark
> The question for someone shopping around for languages is: what is the selling point for Guile (or Scheme)? Being a programmable programming language sounds cool, but what does it let me do? Nothing on Guile's website tells me “look at how easy this is to solve in Guile, you cannot do this in other languages”. What would that be? I don’t know, I’m just a layman. The topic of the thread is “how to make Guile more successful”, and I most of those are technical points, so I’m afraid that discussing popularity is all I can contribute.

You can for example build wisp ;)

And "natural script writing" would have been much harder (and much more
mechanical work) in other languages. In most languages it would have
required parsing the text by hand and explicitly extracting code and
handing it to some function to execute it.

Best wishes,
Arne

Re: How to make GNU Guile more successful

[Matt Wette]

Scheme is not for everyone.  But getting more people to use it is going to mean people seeing what it offers over other languages.  I do a fair amount of Python programming.  Python code is fairly portable: my Mac comes with it installed and so I can send code to anyone with a Mac and it will probably run.  Python also has a *huge* set of libraries.  Writing extension code (in C) for Python sucks, though.  And Python `lambda’ is weak.

And Python does not work well on tasks that Scheme excels at: tree structures, little-languages, list processing, etc.  If you want more people to use Scheme/Guile you need to write applications that demonstrate it’s strengths over other languages.  I know of nothing that beats Scheme macros for building your own language extensions: that capability is so awesome.   And Scheme looks pretty promising to me for handling XML-based tree structures (e.g., SXML with foldts and friends).

Matt

Re: How to make GNU Guile more successful

[Erik Edrosa]

On 03/05/2017 09:27 AM, Arne Babenhauserheide wrote:
> 
> Erik Edrosa <erik.edrosa@gmail.com> writes:
> 
>> I think one issue when you want to compare languages like Python to
>> Scheme is that these languages are dominated by a single
>> implementation.
> 
> I read this argument repeatedly, but it does not apply to C and C++, two
> of the most widely used languages. Therefore that argument might just
> interpret too much into a coincidence — or rather: into a situation
> which happens in the beginning of language development.
> 
> Even Java nowadays has several different implementations.
> 

C, C++, and Java have very large specifications and standards which are
meant to keep the various implementations as compatible as possible.
Although implementations do provide their own extensions and there is
issues between OSes. I agree this argument isn't too strong, but I see
it brought up once in awhile. One issue is it does fracture the
community a bit. If implementations have greater compatibility you see
developers use them in interesting ways. For example in Common Lisp,
I've heard some developers will use CCL in development because it
compiles faster and produces easier to debug code and use SBCL for
building releases because it produces faster code.


>> an awesome application to show off the
>> power of the language, good tooling, and some luck.
> 
> And zealots, never forget zealots. I’d dare to say that people, who
> believe that the language or tool is the best thing in existence and
> give lots of talks about it, are a far too easily underestimated factor
> in the success of any tool.
> 
> Best wishes,
> Arne
> 

I think we have many of those. ;)

- Erik

Re: How to make GNU Guile more successful

[David Kastrup]

Alejandro Sanchez <hiphish@openmailbox.org> writes:

> Scheme on the other hand is weird; weird in a good way, but still
> weird. For me the reason for picking up Scheme was working my way
> through SICP, but even before that I had heard of the Lisp
> family. Every time I tried to find out what the fuss was about Lisp
> all I could find was nebulous concepts like “it’s a programmable
> programming language” without explaining what that even meant and why
> I would ever want to use it.
>
> Of course once I got over the weird parentheses and I understood why
> they are actually a beneficial aspect I liked it.

Everything falls into place once you understand that Scheme does not
offer a programming language as user interface.  Instead, you directly
write down the parse trees in Scheme's compact list representation.

That's not exactly readable but makes structure-preserving code
manipulation a breeze.

-- 
David Kastrup

Re: How to make GNU Guile more successful

[Vítor De Araújo]

Hi,

I'm relatively new to Guile and new to this list, but I saw this
thread in the archives and I'd like to make some comments.

First, I think it would be nice to have a Guile wiki, in the likes of
cliki.net. A publicly-editable reference place where people can share
knowledge about Guile and Guile libraries and projects. Sure, having a
centralized package manager/repository would be nice, but just having
a common place where people can look up what libraries exist and where
they can be downloaded would be nice. (I don't know how easy would it
be to set up a wiki at the Guile website.)

On the topic of package management, I think that, more important than
a central repository, is a standard format in which to distribute
packages for easy installation. I've never used guildhall, and I guess
it defines something like this, but I think just having an
installation command where you can pass the URL of a package to
download and install (say, guild install http://some.url/somepackage.zip,
or even a git URL) would already go a long way towards making it
easier for people to use third-party libraries in Guile.

(Or maybe package _names_ could be URLs pointing to an index in a
well-defined format, which tells how to download each version of a
package; then we could have even dependency management without a
central repository. I'm pipe-dreaming here, though.)

Best regards,

Vítor De Araújo

-- 
Vítor De Araújo
http://inf.ufrgs.br/~vbuaraujo

Re: How to make GNU Guile more successful

[Arne Babenhauserheide]

Vítor De Araújo writes:

> (Or maybe package _names_ could be URLs pointing to an index in a
> well-defined format, which tells how to download each version of a
> package; then we could have even dependency management without a
> central repository. I'm pipe-dreaming here, though.)

You’re not just dreaming:

https://github.com/plande/compan/blob/master/compan.scm

    ;; clone the repository and load (lib) module from the top level:
    (load-modules ("https://bitbucket.org/panicz/dogma" (lib)))

    ;; as above, but load modules from "libraries" directory:
    (load-modules (("https://github.com/panicz/pamphlet" "libraries") (pamphlet)))

    ;; one can also fix on a particular tag/branch/commit:
    (load-modules (("https://bitbucket.org/panicz/dogma" "." "3884445191c2") (lib)))

We’d just have to decide to use this.

Best wishes,
Arne
-- 
Sent with my mu4e

Re: How to make GNU Guile more successful

[Panicz Maciej Godek]

2017-03-10 10:55 GMT+01:00 Arne Babenhauserheide <arne_bab@web.de>:

>
> Vítor De Araújo writes:
>
> > (Or maybe package _names_ could be URLs pointing to an index in a
> > well-defined format, which tells how to download each version of a
> > package; then we could have even dependency management without a
> > central repository. I'm pipe-dreaming here, though.)
>
> You’re not just dreaming:
>
> https://github.com/plande/compan/blob/master/compan.scm
>
>     ;; clone the repository and load (lib) module from the top level:
>     (load-modules ("https://bitbucket.org/panicz/dogma" (lib)))
>
>     ;; as above, but load modules from "libraries" directory:
>     (load-modules (("https://github.com/panicz/pamphlet" "libraries")
> (pamphlet)))
>
>     ;; one can also fix on a particular tag/branch/commit:
>     (load-modules (("https://bitbucket.org/panicz/dogma" "."
> "3884445191c2") (lib)))
>
> We’d just have to decide to use this.
>
> I think that the situation isn't quite as good as one could think. The
code works fine from
the interpreter, but if fails to work for compiled code, and I think that
we'd need to come up
with some solution of delaing module imports in compiled code until they're
available.

(Note also that the source code of Compan is extremely straightforward, as
it boils down
to a few calls of the "system" function which in turns calls some "hg"
commands)

So while I think that the author's idea was very nice, it clearly needs
some elaboration :)

Re: How to make GNU Guile more successful

[vbuaraujo]

Citando Panicz Maciej Godek <godek.maciek@gmail.com>:

> 2017-03-10 10:55 GMT+01:00 Arne Babenhauserheide <arne_bab@web.de>:
>
>>
>> Vítor De Araújo writes:
>>
>> > (Or maybe package _names_ could be URLs pointing to an index in a
>> > well-defined format, which tells how to download each version of a
>> > package; then we could have even dependency management without a
>> > central repository. I'm pipe-dreaming here, though.)
>>
>> You’re not just dreaming:
>>
>> https://github.com/plande/compan/blob/master/compan.scm
>>
>>     ;; clone the repository and load (lib) module from the top level:
>>     (load-modules ("https://bitbucket.org/panicz/dogma" (lib)))
>>
>>     ;; as above, but load modules from "libraries" directory:
>>     (load-modules (("https://github.com/panicz/pamphlet" "libraries")
>> (pamphlet)))
>>
>>     ;; one can also fix on a particular tag/branch/commit:
>>     (load-modules (("https://bitbucket.org/panicz/dogma" "."
>> "3884445191c2") (lib)))
>>
>> We’d just have to decide to use this.

That's very cool :) And also supports my point that there should be something
like a wiki for publishing information about Guile packages. It'd be hard
for me to find this on my own.

>> I think that the situation isn't quite as good as one could think. The
> code works fine from
> the interpreter, but if fails to work for compiled code, and I think that
> we'd need to come up
> with some solution of delaing module imports in compiled code until they're
> available.
>
> (Note also that the source code of Compan is extremely straightforward, as
> it boils down
> to a few calls of the "system" function which in turns calls some "hg"
> commands)
>
> So while I think that the author's idea was very nice, it clearly needs
> some elaboration :)

Yeah, for one I don't think the URLs should go directly into the code, but
rather in a package metadata file. The reason is I don't think we should
hard-code the package manager in the code. People should be able to use
the code without having the package manager, if they already have the
dependencies installed by some other means. But the basic idea is cool. :)

--
Vítor De Araújo

Re: How to make GNU Guile more successful

[Panicz Maciej Godek]

2017-03-10 15:27 GMT+01:00 <vbuaraujo@sapo.pt>:


> That's very cool :) And also supports my point that there should be
> something
> like a wiki for publishing information about Guile packages. It'd be hard
> for me to find this on my own.
>
> [...]
>>
>
> Yeah, for one I don't think the URLs should go directly into the code, but
> rather in a package metadata file. The reason is I don't think we should
> hard-code the package manager in the code. People should be able to use
> the code without having the package manager, if they already have the
> dependencies installed by some other means. But the basic idea is cool. :)
>

There is a reason why I like the idea of having URLs directly in the code,
namely that it makes you independent from some big institutions/particular
maintainers. Of course it is risky if your code refers to repositories that
you do not control, but you can always make your private forks.

Essentially the power of this idea is that it suddenly turns the whole web
into a Guile Incomprehensive Archive Network (clearly, a word beginning
with a letter T is missing to get a nice acronym).

Of course, it should be very easy to build a translation layer that
would map package names to addresses, but it requires a middleman
which always makes the costs higher (but perhaps having a proxy that
would make the necessary forks automatically would be a nice idea.
Plus if it would be able to check whether the changes between
some commits break compatibility then, well, that would really be
something. Which brings to mind the recent talk by Rich Hickey:
https://www.youtube.com/watch?v=oyLBGkS5ICk )

I do agree that something wiki-like would be a lovely thing.

Re: How to make GNU Guile more successful

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 948 bytes --]


() Panicz Maciej Godek <godek.maciek@gmail.com>
() Fri, 10 Mar 2017 16:08:23 +0100

   Essentially the power of this idea is that it suddenly turns
   the whole web into a Guile Incomprehensive Archive Network
   (clearly, a word beginning with a letter T is missing to get
   a nice acronym).

Cool.  Some suggestions:

tactic
tap
tapestry
tatter
tattoo
taunt
teacup
teardrop
tech
tee
temperament
tendency
tenet
thanks
thicket
thing
thrall
thwack
tine
tofu
toke
torpor
torrent
toy
tracks
traipse
travail
trawl
tree
tremor
trick
trinkets
triumph
troth
truffle
tsunami
tub
tufts
twig
twists
typo

-- 
Thien-Thi Nguyen -----------------------------------------------
 (defun responsep (query)
   (pcase (context query)
     (`(technical ,ml) (correctp ml))
     ...))                              748E A0E8 1CB8 A748 9BFA
--------------------------------------- 6CE4 6703 2224 4C80 7502


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: How to make GNU Guile more successful

[Nala Ginrut]

Well, I like tofu :-)

Thien-Thi Nguyen <ttn@gnu.org>于2017年3月11日周六 16:48写道：

>
> () Panicz Maciej Godek <godek.maciek@gmail.com>
> () Fri, 10 Mar 2017 16:08:23 +0100
>
>    Essentially the power of this idea is that it suddenly turns
>    the whole web into a Guile Incomprehensive Archive Network
>    (clearly, a word beginning with a letter T is missing to get
>    a nice acronym).
>
> Cool.  Some suggestions:
>
> tactic
> tap
> tapestry
> tatter
> tattoo
> taunt
> teacup
> teardrop
> tech
> tee
> temperament
> tendency
> tenet
> thanks
> thicket
> thing
> thrall
> thwack
> tine
> tofu
> toke
> torpor
> torrent
> toy
> tracks
> traipse
> travail
> trawl
> tree
> tremor
> trick
> trinkets
> triumph
> troth
> truffle
> tsunami
> tub
> tufts
> twig
> twists
> typo
>
> --
> Thien-Thi Nguyen -----------------------------------------------
>  (defun responsep (query)
>    (pcase (context query)
>      (`(technical ,ml) (correctp ml))
>      ...))                              748E A0E8 1CB8 A748 9BFA
> --------------------------------------- 6CE4 6703 2224 4C80 7502
>
>

Re: How to make GNU Guile more successful

[Panicz Maciej Godek]

My mind has actually been oscillating between "trick", "toy", "thing" and
"torrent".
The last one would go particularly nice with the concept of "GitTorrent"

http://blog.printf.net/articles/2015/05/29/announcing-gittorrent-a-decentralized-github/


2017-03-13 16:55 GMT+01:00 Nala Ginrut <nalaginrut@gmail.com>:

> Well, I like tofu :-)
>
> Thien-Thi Nguyen <ttn@gnu.org>于2017年3月11日周六 16:48写道：
>
> >
> > () Panicz Maciej Godek <godek.maciek@gmail.com>
> > () Fri, 10 Mar 2017 16:08:23 +0100
> >
> >    Essentially the power of this idea is that it suddenly turns
> >    the whole web into a Guile Incomprehensive Archive Network
> >    (clearly, a word beginning with a letter T is missing to get
> >    a nice acronym).
> >
> > Cool.  Some suggestions:
> >
> > tactic
> > tap
> > tapestry
> > tatter
> > tattoo
> > taunt
> > teacup
> > teardrop
> > tech
> > tee
> > temperament
> > tendency
> > tenet
> > thanks
> > thicket
> > thing
> > thrall
> > thwack
> > tine
> > tofu
> > toke
> > torpor
> > torrent
> > toy
> > tracks
> > traipse
> > travail
> > trawl
> > tree
> > tremor
> > trick
> > trinkets
> > triumph
> > troth
> > truffle
> > tsunami
> > tub
> > tufts
> > twig
> > twists
> > typo
> >
> > --
> > Thien-Thi Nguyen -----------------------------------------------
> >  (defun responsep (query)
> >    (pcase (context query)
> >      (`(technical ,ml) (correctp ml))
> >      ...))                              748E A0E8 1CB8 A748 9BFA
> > --------------------------------------- 6CE4 6703 2224 4C80 7502
> >
> >
>

Re: How to make GNU Guile more successful

[Amirouche]

Héllo!


Le 10/03/2017 à 03:08, Vítor De Araújo a écrit :
> Hi,
>
> I'm relatively new to Guile and new to this list, but I saw this
> thread in the archives and I'd like to make some comments.

Welcome!

>
> First, I think it would be nice to have a Guile wiki, in the likes of
> cliki.net. A publicly-editable reference place where people can share
> knowledge about Guile and Guile libraries and projects.
I agree.
> Sure, having a
> centralized package manager/repository would be nice, but just having
> a common place where people can look up what libraries exist and where
> they can be downloaded would be nice. (I don't know how easy would it
> be to set up a wiki at the Guile website.)

The current most comprehensive list of GNU Guile package is
available at http://sph.mn/content/3e73

There is also the github org:

https://github.com/guildhall

> On the topic of package management, I think that, more important than
> a central repository, is a standard format in which to distribute
> packages for easy installation.
I agree. The standard could be to have a tarball named using
semantic versionning, from which one can extract the PACKAGE-NAME.
Inside the tarball, the package manager will expect a PACKAGE-NAME.scm
or a PACKAGE-NAME directory which can be installed into ~/.local/lib/guile

WDYT?

> I've never used guildhall, and I guess
> it defines something like this,
Probably but it's not documented.

> but I think just having an
> installation command where you can pass the URL of a package to
> download and install (say, guild install http://some.url/somepackage.zip,
> or even a git URL)
This will still require the user to set GUILE_LOAD_PATH. Maybe we can
add ~/.local/lib/guile/ to the default GUILE_LOAD_PATH to make things
easier.

A simple guild install https://url/to/patch can be done in mater of hours
(if not minutes).

WDYT?

> would already go a long way towards making it
> easier for people to use third-party libraries in Guile.
I agree.
> (Or maybe package _names_ could be URLs pointing to an index in a
> well-defined format, which tells how to download each version of a
> package; then we could have even dependency management without a
> central repository. I'm pipe-dreaming here, though.)
This seems fun, but at the same time odd.

FWIW, I've started a new effort to build a package manager for GNU
Guile. You can find a demo at http://guildhall.hypermove.net/

It started as a guildhall web frontend but I noticed that guildhall:

1) doesn't work with guile 2.2 and I can not make it work
2) The solver is too complicated, even if it's based on aptitude algorithm
   it's still complicated.
3) use a lot of other scheme libraries [0] (like foof-loop) which
   doesn't help noobs like me to dive into the code. Maybe those
   libraries are *very* neat but why not include them in Guile proper
   then and make them default.

[0] https://github.com/ijp/guildhall/tree/master/guildhall/ext

For all this reasons I forked the effort.

I address each previous point as follow:

1) I only use guile 2.2, right now guile 2.1 is not supported. I think it's
bad this should be addressed at some point. AFAIR the issue is in
guile-wiredtiger (or it's an issue with setlocale (anyway this must be
addressed)))

2) I use a logic programming library called minikanren to solve the
dependency problem. This is the first real problem I can use minikanren
to solve a problem that is craft. This is the logic programming
library in scheme. While it's not very advanced compared to core.logic,
the version I use called microkaren has a straight forward 
implementation that can
hold all at once in a single head. Logic programming, like probabilistic
programming (cf. ) are two research areas in programming languages
that could improve the way we craft algorithms today. The logic approach
being more useful to me (but someone doing machine learning stuff,
might find the probabilistic scheme very useful. Ask me for a link!). 
OpenCog
does have a logic language embedded in it's language in their hyper 
graph database
(AFAIU it works using backward / forward inference but implements on top
of it higher logic constructs, like abduction, anaphora resolution...). It's
something they call PLN as Probabilistic Logic Network. AFAIK it's some kind
of probabilistic Datalog (which is one of the API Datomic support). 
There is
also probabilistic minikanren (albeit not useable). So, to sum up (because
I can (because I am not a robot)):

programming language research + guile + logic = minikanren.

Mind the fact that there is also a guile-log. But I still don't get what
guile-log does (hints: http://c-lambda.se/) whereas I understand how
microkanren does its stuff (I still need to benchmark it).

3) I don't use foof-loop but I bring my own dependencies. I use my
own database library that is based on wiredtiger. I've been working
on this database library for 3 years, it's well documented and various
example codes. That said, AFAIK, I am the only one to use it. I've built
a clone of Datomic (with the persistence part, patch welcome) which
use a query engine similar to Datomic I guess based on minikanren.

It's performance on a middle end laptop run Guile 2.2 are the followings:

- 1500 document reads per seconds
- 1000 to 500 document write per second

Documents are scheme assoc, and those are inserted 1000 per 1000 and
read 1000 per 1000 until it reach 50Go of data. At that point writes 
take 1/500
seconds or 0.002 seconds.

The biggest dataset I *loaded* into that database is wikidata which is 
50Go, I
don't remember how much time it takes to load it.

I had issues with wiredtiger using the Python bindings but only during reads
over a gigantic dataset of 60Giga. I say gigantic for a blog  not for Google
scale. Also upstream can solve issues if we can have a way to reproduce the
issues (which I plan to do once guile 2.2 is out (which means I will 
redo the
benchmarks against wikidata and read/write)). I understand the problem
that wiredtiger being part of MongoDB is problematic as MongoDB might not
care much for the same problem as ours. They will always be interested by
bigger free software database, tho. As it make free publicity of how 
successful
is their software.

Also there is much documentation about this library. I created several 
Guile projects
using it (albeit not big) they document several layers of the library 
and one
Guile user reported using it for doing human/social science research.

Search for guile wiredtiger in you favorite duckduckgo search engine.

Some people claim that PostgreSQL has all the required feature that 
someone wants
to store data and that this mongodb/wiredtiger is a bad. I recognize 
that wiredtiger
can be poor man's database right now. I don't have the required 
expertise to verify
whether it's good enough for your usecase. PostgreSQL is used in 
"production" in
all the world using multiple workload and stuff. PostgreSQL is good.

1) There is no dynamic ffi bindings for PostgreSQL yet, otherwise said you
     can't use pgsql from Guile RIGHT NOW.
2) wiredtiger is GOOD enough, benchmark it before saying it's not good 
ENOUGH.
3) it's not SQL, it's Schemey! It's inspired from datomic which the goto 
database
     for clojure with multiple implementation in the browser. This is 
killer feature
     of the clojure ecosystem that is closed source.
4) I find it more funny to fiddle with logic programming that set pseudo 
theory of SQL.
5) The first thing I do when pgsql bindings will be out, is to port 
feature space to PostgreSQL
and compare both pgsql and wiredtiger.
6) wiredtiger will always be simpler to use that Postgresql it's like 
comparing sqlite
and Datomic.

What happens next? You choose to hack on this project and find something 
interesting in this
TODO list:

- fix guile-wiredtiger to work on guile 2.0

- make it possible in guile-wiredtiger to stream the results of a look 
up using traversi using multiple cursors

- implement disequality in microkanren (for implementing the package 
dependency resolver
   and for fun because logic is awesome).

- implement a pastebin service using GNU Guile, yes because I think this 
a good bad reason to start
   another simple project that is not a static blog generator.

- Port the wiki of http://culturia.one to use feature space library.

- implement or find a scheme library that does the diff two lists (to 
finish wiki implementation
   and for the package dependency resolver).

- Help Matt Wette to complete his ffi helper 
http://lists.gnu.org/archive/html/guile-user/2017-03/msg00085.html
   (because we will need it for guix).

- add guile-fibers as a submodule and use it

- there is also a simple boolean keyword search engine that must be 
ported to feature space

- find the code that implements trigram transderivational search and put 
it in the repository. Bridging
   the gap with tsearch2.

- index the wiki

- index the packages

- index the pastebin

- index the web

And add gnunet to the mix.

There is not a lot of get-together pure guile project out there. There 
is guix. I hope this new project
can be the occasion for new guilers to submit patches or ideas.

The code is currently hosted at 
https://framagit.org/a-guile-mind/culturia.one/tree/guildhall/

The codename is "primary".

Also I'd like to point out, that I don't need primary. I do it for the 
community only. The
road is fascinating, tho. I could make mistakes so please input your 
feedback if you want
to use a tool like that.

You can:

guix package -i wiredtiger
git clone https://framagit.org/a-guile-mind/culturia.one.git
git checkout guildhall
emacs culturia.one/src/webui.scm

or

cd culturia.one/src/ && guile -L . webui.scm

Re: How to make GNU Guile more successful

[Vítor De Araújo]

On 10/03/2017 17:17, Amirouche wrote:
> Héllo!
> 
> 
> Le 10/03/2017 à 03:08, Vítor De Araújo a écrit :
>> Hi,
>>
>> I'm relatively new to Guile and new to this list, but I saw this
>> thread in the archives and I'd like to make some comments.
> 
> Welcome!

Thanks :)

[...]

>> On the topic of package management, I think that, more important than
>> a central repository, is a standard format in which to distribute
>> packages for easy installation.
> I agree. The standard could be to have a tarball named using
> semantic versionning, from which one can extract the PACKAGE-NAME.
> Inside the tarball, the package manager will expect a PACKAGE-NAME.scm
> or a PACKAGE-NAME directory which can be installed into ~/.local/lib/guile
> 
> WDYT?

That's good, but it'd also have to contain a metadata file containing at
least the package's dependencies and what commands should be run to
install (for example, ./configure && make && make install, for packages
which need some compilation step before copying to ~/.local/lib/guile).

>> I've never used guildhall, and I guess
>> it defines something like this,
> Probably but it's not documented.
> 
>> but I think just having an
>> installation command where you can pass the URL of a package to
>> download and install (say, guild install http://some.url/somepackage.zip,
>> or even a git URL)
> This will still require the user to set GUILE_LOAD_PATH. Maybe we can
> add ~/.local/lib/guile/ to the default GUILE_LOAD_PATH to make things
> easier.

Yes, ~/.local/lib/guile/ (or other user-level directory) definitely
should be in the default GUILE_LOAD_PATH.

> A simple guild install https://url/to/patch can be done in mater of hours
> (if not minutes).
> 
> WDYT?

Sure. Adding dependency handling would make things somewhat more
complex, though.

I thought a little bit about the packages-as-urls idea and I had some
ideas I'll just throw out here.

First, I realized using the URL as the package _name_ is problematic,
because it hard-codes the source of the package; that complicates things
if someone wants to make a fork of the package and install it in place
of the original. However, packages can still have a normal name and a
"canonical URL" for fetching it.

The package would come with a metadata file (named something like
metadata.foopkg, where 'foopkg' is a placeholder for the name of the
package manager (and I'd rather have the name of the package manager as
an extension, because I'd rather not assume I'm the only package manager
in the world)). This metadata would contain the package name, version,
and dependencies. For each dependency, it would specify the package
name, the version, *and the canonical URL* for that package. Upoin
installation, the package manager would go through each dependency and
check if a package of that name and compatible version (assuming
semantic versioning) is already installed; if it's not, it would fetch
it from the canonical URL.

In this way, we can have decentralized package management which can
handle dependencies.

And commenting on something I said previously:

On 10/03/2017 12:08, Panicz Maciej Godek wrote:
> 2017-03-10 15:27 GMT+01:00 <vbuaraujo@sapo.pt>:
>
>     Yeah, for one I don't think the URLs should go directly into the
>     code, but
>     rather in a package metadata file. The reason is I don't think we
should
>     hard-code the package manager in the code. People should be able
to use
>     the code without having the package manager, if they already have the
>     dependencies installed by some other means. But the basic idea is
>     cool. :)
>
>
> There is a reason why I like the idea of having URLs directly in the code,
> namely that it makes you independent from some big institutions/particular
> maintainers. Of course it is risky if your code refers to repositories
that
> you do not control, but you can always make your private forks.

I'm totally in favor of independence from particular institutions or
maintainers, but I think this can be achieved by putting the URLs in the
metadata file rather than directly in the code. Code would just use
(use-modules ...) as usual to load code, and the business of dowloading
and installing would be solely the task of the package manager. I think
that enables a friendlier co-existence with other package managers, and
allows the same library to be packaged for multiple package managers at
the same time (e.g., guix, compan, guildhall, etc.). It also allows
people to just unpack and install the code manually if they want to.

(A problem that still remains if packages have names separate from the
URLs is avoiding name clashes. The wiki would go some way towards that
(people could check more easily if the name they want is already in
use), but maybe some conventions for globally unique names (e.g.,
prepending a domain prefix like Java people do) could ameliorate this. I
don't think this is really important at this point.)

[...]
> FWIW, I've started a new effort to build a package manager for GNU
> Guile. You can find a demo at http://guildhall.hypermove.net/
> 
> It started as a guildhall web frontend but I noticed that guildhall:
> 
> 1) doesn't work with guile 2.2 and I can not make it work
> 2) The solver is too complicated, even if it's based on aptitude algorithm
>   it's still complicated.
> 3) use a lot of other scheme libraries [0] (like foof-loop) which
>   doesn't help noobs like me to dive into the code. Maybe those
>   libraries are *very* neat but why not include them in Guile proper
>   then and make them default.
> 
> [0] https://github.com/ijp/guildhall/tree/master/guildhall/ext
> 
> For all this reasons I forked the effort.
> 
> I address each previous point as follow:
> 
> 1) I only use guile 2.2, right now guile 2.1 is not supported. I think it's
> bad this should be addressed at some point. AFAIR the issue is in
> guile-wiredtiger (or it's an issue with setlocale (anyway this must be
> addressed)))
> 
> 2) I use a logic programming library called minikanren to solve the
> dependency problem. This is the first real problem I can use minikanren
> to solve a problem that is craft. This is the logic programming
> library in scheme. While it's not very advanced compared to core.logic,
> the version I use called microkaren has a straight forward
> implementation that can
> hold all at once in a single head. Logic programming, like probabilistic
> programming (cf. ) are two research areas in programming languages
> that could improve the way we craft algorithms today. The logic approach
> being more useful to me (but someone doing machine learning stuff,
> might find the probabilistic scheme very useful. Ask me for a link!).
> OpenCog
> does have a logic language embedded in it's language in their hyper
> graph database
> (AFAIU it works using backward / forward inference but implements on top
> of it higher logic constructs, like abduction, anaphora resolution...).
> It's
> something they call PLN as Probabilistic Logic Network. AFAIK it's some
> kind
> of probabilistic Datalog (which is one of the API Datomic support).
> There is
> also probabilistic minikanren (albeit not useable). So, to sum up (because
> I can (because I am not a robot)):
> 
> programming language research + guile + logic = minikanren.
> 
> Mind the fact that there is also a guile-log. But I still don't get what
> guile-log does (hints: http://c-lambda.se/) whereas I understand how
> microkanren does its stuff (I still need to benchmark it).
> 
> 3) I don't use foof-loop but I bring my own dependencies. I use my
> own database library that is based on wiredtiger. I've been working
> on this database library for 3 years, it's well documented and various
> example codes. That said, AFAIK, I am the only one to use it. I've built
> a clone of Datomic (with the persistence part, patch welcome) which
> use a query engine similar to Datomic I guess based on minikanren.
> 
> It's performance on a middle end laptop run Guile 2.2 are the followings:
> 
> - 1500 document reads per seconds
> - 1000 to 500 document write per second
> 
> Documents are scheme assoc, and those are inserted 1000 per 1000 and
> read 1000 per 1000 until it reach 50Go of data. At that point writes
> take 1/500
> seconds or 0.002 seconds.
> 
> The biggest dataset I *loaded* into that database is wikidata which is
> 50Go, I
> don't remember how much time it takes to load it.
> 
> I had issues with wiredtiger using the Python bindings but only during
> reads
> over a gigantic dataset of 60Giga. I say gigantic for a blog  not for
> Google
> scale. Also upstream can solve issues if we can have a way to reproduce the
> issues (which I plan to do once guile 2.2 is out (which means I will
> redo the
> benchmarks against wikidata and read/write)). I understand the problem
> that wiredtiger being part of MongoDB is problematic as MongoDB might not
> care much for the same problem as ours. They will always be interested by
> bigger free software database, tho. As it make free publicity of how
> successful
> is their software.
> 
> Also there is much documentation about this library. I created several
> Guile projects
> using it (albeit not big) they document several layers of the library
> and one
> Guile user reported using it for doing human/social science research.
> 
> Search for guile wiredtiger in you favorite duckduckgo search engine.
> 
> Some people claim that PostgreSQL has all the required feature that
> someone wants
> to store data and that this mongodb/wiredtiger is a bad. I recognize
> that wiredtiger
> can be poor man's database right now. I don't have the required
> expertise to verify
> whether it's good enough for your usecase. PostgreSQL is used in
> "production" in
> all the world using multiple workload and stuff. PostgreSQL is good.
> 
> 1) There is no dynamic ffi bindings for PostgreSQL yet, otherwise said you
>     can't use pgsql from Guile RIGHT NOW.
> 2) wiredtiger is GOOD enough, benchmark it before saying it's not good
> ENOUGH.
> 3) it's not SQL, it's Schemey! It's inspired from datomic which the goto
> database
>     for clojure with multiple implementation in the browser. This is
> killer feature
>     of the clojure ecosystem that is closed source.
> 4) I find it more funny to fiddle with logic programming that set pseudo
> theory of SQL.
> 5) The first thing I do when pgsql bindings will be out, is to port
> feature space to PostgreSQL
> and compare both pgsql and wiredtiger.
> 6) wiredtiger will always be simpler to use that Postgresql it's like
> comparing sqlite
> and Datomic.
> 
> What happens next? You choose to hack on this project and find something
> interesting in this
> TODO list:
> 
> - fix guile-wiredtiger to work on guile 2.0
> 
> - make it possible in guile-wiredtiger to stream the results of a look
> up using traversi using multiple cursors
> 
> - implement disequality in microkanren (for implementing the package
> dependency resolver
>   and for fun because logic is awesome).
> 
> - implement a pastebin service using GNU Guile, yes because I think this
> a good bad reason to start
>   another simple project that is not a static blog generator.
> 
> - Port the wiki of http://culturia.one to use feature space library.
> 
> - implement or find a scheme library that does the diff two lists (to
> finish wiki implementation
>   and for the package dependency resolver).
> 
> - Help Matt Wette to complete his ffi helper
> http://lists.gnu.org/archive/html/guile-user/2017-03/msg00085.html
>   (because we will need it for guix).
> 
> - add guile-fibers as a submodule and use it
> 
> - there is also a simple boolean keyword search engine that must be
> ported to feature space
> 
> - find the code that implements trigram transderivational search and put
> it in the repository. Bridging
>   the gap with tsearch2.
> 
> - index the wiki
> 
> - index the packages
> 
> - index the pastebin
> 
> - index the web
> 
> And add gnunet to the mix.
> 
> There is not a lot of get-together pure guile project out there. There
> is guix. I hope this new project
> can be the occasion for new guilers to submit patches or ideas.
> 
> The code is currently hosted at
> https://framagit.org/a-guile-mind/culturia.one/tree/guildhall/
> 
> The codename is "primary".
> 
> Also I'd like to point out, that I don't need primary. I do it for the
> community only. The
> road is fascinating, tho. I could make mistakes so please input your
> feedback if you want
> to use a tool like that.
> 
> You can:
> 
> guix package -i wiredtiger
> git clone https://framagit.org/a-guile-mind/culturia.one.git
> git checkout guildhall
> emacs culturia.one/src/webui.scm
> 
> or
> 
> cd culturia.one/src/ && guile -L . webui.scm

There are certainly lots of fun subprojects to tackle here, but I'm
afraid it would take really long to get ready. I'd rather have a really
simple package manager, preferably with zero dependencies other than
Guile itself. But then again I think multiple package managers should be
able to co-exist, so there's no problem for different people to tackle
multiple approaches to the problem.

-- 
Vítor De Araújo

Re: How to make GNU Guile more successful

[Vítor De Araújo]

On 10/03/2017 21:50, Vítor De Araújo wrote:
[...]
> I thought a little bit about the packages-as-urls idea and I had some
> ideas I'll just throw out here.
> 
> First, I realized using the URL as the package _name_ is problematic,
> because it hard-codes the source of the package; that complicates things
> if someone wants to make a fork of the package and install it in place
> of the original. However, packages can still have a normal name and a
> "canonical URL" for fetching it.
> 
> The package would come with a metadata file (named something like
> metadata.foopkg, where 'foopkg' is a placeholder for the name of the
> package manager (and I'd rather have the name of the package manager as
> an extension, because I'd rather not assume I'm the only package manager
> in the world)). This metadata would contain the package name, version,
> and dependencies. For each dependency, it would specify the package
> name, the version, *and the canonical URL* for that package. Upoin
> installation, the package manager would go through each dependency and
> check if a package of that name and compatible version (assuming
> semantic versioning) is already installed; if it's not, it would fetch
> it from the canonical URL.
> 
> In this way, we can have decentralized package management which can
> handle dependencies.
[...]

Just one more little thought on package names vs. URLs. Guile already
has a standard module system, and modules already have names which are
supposed to be globally unique, and even they even have version numbers.
So module names can double as package names. The advantage is that then
you can check whether a dependency is installed *whether or not it was
installed via the package manager*. Basically you'd describe your
dependencies in terms of modules, and specify where the package that
implements that module can be fetched from.

I have to think this through a bit more, though.

-- 
Vítor De Araújo

Re: How to make GNU Guile more successful

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 929 bytes --]


() Amirouche <amirouche@hypermove.net>
() Fri, 10 Mar 2017 21:17:37 +0100

   1) There is no dynamic ffi bindings for PostgreSQL yet,
      otherwise said you can't use pgsql from Guile RIGHT NOW.

Plug: Guile-PG[0] does not provide ffi bindings for PostgreSQL,
but it does work w/ Guile 2.0 today (last i checked: 2.0.11).
Paren lovers might like the query construction DSL[1], too.

[0] http://www.nongnu.org/guile-pg/
[1] http://www.nongnu.org/guile-pg/doc/Query-Construction.html

Please report any problems you find w/ it to this mailing list.
I'm especially interested in experiences w/ Guile 2.x (x > 0).

-- 
Thien-Thi Nguyen -----------------------------------------------
 (defun responsep (query)
   (pcase (context query)
     (`(technical ,ml) (correctp ml))
     ...))                              748E A0E8 1CB8 A748 9BFA
--------------------------------------- 6CE4 6703 2224 4C80 7502


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: How to make GNU Guile more successful

[Christopher Allan Webber]

Vítor De Araújo writes:

> First, I think it would be nice to have a Guile wiki, in the likes of
> cliki.net. A publicly-editable reference place where people can share
> knowledge about Guile and Guile libraries and projects. Sure, having a
> centralized package manager/repository would be nice, but just having
> a common place where people can look up what libraries exist and where
> they can be downloaded would be nice. (I don't know how easy would it
> be to set up a wiki at the Guile website.)

A Guile wiki would be pretty nice, but my experience is that running a
wiki is a maintenance drag, and nobody really wants to keep it going for
a project of Guile's size (I wish this weren't the case).  Of course, if
someone wants to run one, great!  But IMO it would be better to "team
up" with something/someone that's already running a wiki.  I would
suggest using a namespace under the LibrePlanet wiki?  That one is
already maintained by the FSF sysadmins and/or GNU sysadmin
volunteers (I don't quite know the setup).