How to make GNU Guile more successful

How to make GNU Guile more successful

[Amirouche]

I don't know why Racket is successful probably because
it has a very good documentation and also a documentation
generator written in scheme.

  => This is a long debate and core principle of GNU project
     so I don't want to enter that debate

And it's easy to packages.

  => Another pain topic. In reality it's very easy to package
     all sort of guile programs for guix. But guix itself needs
     to be more stable.

Python is successful because it has a massive number
of libraries. In particular it has dozens of:

- web framework
- library to generate diagrams and graphical stuff
- scientific libraries (machine learning and others)
- books which teach python (the good enough way)
- 2D and 3D libraries for games

   => We (all the people reading this mail) can not just
      create all those projects. It requires to many domain
      knowledge for a handful of people to be able to compete.
      But what we can do is *share*, *share* and *share*
      again *all* what we do (on mailing list, blogs, on gitlab
      or whatever) whether it is a full blown project with a
      website made by a professional designer with tests suite,
      a tutorial documentation reference doc and a community or
      a simple *snippet* or broken experiments. I have lost
      (read deleteted) at least the code of 5 or 6 experiments
      only to recognize later that it could be useful for future
      me.

   => In terms of software I really think, we need something like
      ActiveState Python Cookbook [0]. I still go there sometime even
      I could simply ask the question on StackOverflow and get an
      answer in minutes.

      [0] http://code.activestate.com/recipes/langs/python/

During a long time I was said that Pythonist were
among the smartest developer, because simply they had
the guts to learn a language that was not learned in
school.

  => this can be true for schemers;

Python *was* easy to learn.

  => Scheme is easy to learn if you stick in the "garden"
     (garden must be defined but it's basically what I
     introduce in my tutorial [1]) Which is totally the
     same for the Python of today (read Python 3.5+).

     [1] http://hyperdev.fr

PHP was successful back in the days because it was easy
to install,

    => A scheme web application is much easier to run!

Now PHP is successful because there is a few successful
software written in it. By successful I mean they GTD.
They aren't perfect, you get bitten etc. But the basic
are there. I think about Drupal, Wordpress and Prestashop.

    => While you might not be interested to build something
       like prestashop. You very much be interested to build
       something like Drupal or Wordpress to allow mom and
       dad and others create a personnal experience on the
       web.

Another thing, Python has GNU Media Goblin

    => We must do better!

WDYT?

What block you from contributing to the wide ecosystem of GNU Guile?

Best regards,

Amirouche

Re: How to make GNU Guile more successful

[Amirouche]

Le 13/02/2017 à 00:56, Amirouche a écrit :

> WDYT?
> What block you from contributing to the wide ecosystem of GNU Guile?

I heared a lot around me about the projects I was doing before in my
previous life in as Pythonista. That I had NIH syndrom. Yeah, I rebuild
stuff that already exists hence according to the "haters" that was not
useful.

=> THIS IS A FALSE. Fully FUD! Two ways you can learn a programming
a) coding b) reading code. Basically in Guile, if there is no code to read
you only have the other option.

=> Remember they were dozen wikis before wikipedia with similar intents,
dozen search engines before Google with similar intents; You never reinvent
the wheel. You update it little by little, it might not be revolution.
Look at paintings for instance, it's seems some painters do not know proper
painting but they are still successful in their niche. Why? Because they tried.
My advice is to aim big or small whatever, but don't be shy to share.

I personnaly don't believe that code pollution exists. My theory is that
this concept was invented by techno oligarcs to limit the creativity of
those that can make. Basicly techno overlords supply the tools and you provide
the finished product that they are interested in. This is not in the interest
of creativity or liberty. Code what you want to code and think you need to code.
Don't be shy. Javascript fatigue is another name for code pollution.

Another thing i heard a lot is: It's impossible.

=> I was said that about Python->Javascript translator dozen of times. About
    my project to create a new web framework too. Basically Django is too
    mainstream. I don't want to be bigger than Django I want to have the
    *correct* solution. In many situations, for me, django was not up to the
    mark.

Another thing I hear a lot: nothing. Yeah. Nothing.

=> It take a lot of courage to do free and open source software. First,
    you might find it difficult to find people that have the same interest
    as you. Second they probably perfer to fork and do it from scratch by
    themself. Maybe they are shy and do it secretly aka. they don't want to
    disturb the matrix.

    You will also maybe meet people that hate open source or worst that hate
    GNU. Leave if possible.

That said I don't say HACK ALL DAY until something BAD happens. I say love
the code and the community share what you did.

Re: How to make GNU Guile more successful

[Arne Babenhauserheide]

Hi Amirouche,

Thank you for your nice writeup!

Amirouche writes:

> I don't know why Racket is successful probably because
> it has a very good documentation and also a documentation
> generator written in scheme.
>
>   => This is a long debate and core principle of GNU project
>      so I don't want to enter that debate

But I want to add something: I’m getting the hang of just checking the
info-page to look something up. And I realize that with well-written
info-pages this is faster than googling for stack overflow.

> And it's easy to packages.
>
>   => Another pain topic. In reality it's very easy to package
>      all sort of guile programs for guix. But guix itself needs
>      to be more stable.

I wish I had guildhall ready. Got hit by
time-eaten-by-other-project-because-guildhall-was-priority-three.

It needs to be easy to not just package some code, but also to share
that package without hitting a bottleneck.

> Python is successful because it has a massive number
> of libraries.
…
>    => We (all the people reading this mail) can not just
>       create all those projects. It requires to many domain
>       knowledge for a handful of people to be able to compete.
>       But what we can do is *share*, *share* and *share*
>       again *all* what we do (on mailing list, blogs, on gitlab
>       or whatever) whether it is a full blown project with a
>       website made by a professional designer with tests suite,
>       a tutorial documentation reference doc and a community or
>       a simple *snippet* or broken experiments.

A really simple way to share a project would be nice here. Nowadays many
people just push something with a README to github, but github is
unfree.

>    => In terms of software I really think, we need something like
>       ActiveState Python Cookbook [0]. I still go there sometime even
>       I could simply ask the question on StackOverflow and get an
>       answer in minutes.
>
>       [0] http://code.activestate.com/recipes/langs/python/
>
> During a long time I was said that Pythonist were
> among the smartest developer, because simply they had
> the guts to learn a language that was not learned in
> school.

I never heard that one :)

> Python *was* easy to learn.
>
>   => Scheme is easy to learn if you stick in the "garden"
>      (garden must be defined but it's basically what I
>      introduce in my tutorial [1]) Which is totally the
>      same for the Python of today (read Python 3.5+).

Or look at py2guile: http://draketo.de/py2guile

Python is no longer easy due to integrating additional syntactic forms,
each of which makes a specific kind of code much nicer but complicates
learning.

We have something similar in foof-loop:

(define (count-matching-items list predicate)
  (loop ((for item (in-list list))    
         (with count 0   
               (if (predicate item)
                   (+ count 1)
                   count)))
        => count))
;       ^^ here

This looks really simple in this place, but it introduces special syntax
which will be unintelligible for someone who does did not explicitly
learn foof-loop.

And guildhall is filled with special forms I had to understand just to
be able to do small changes.

This is a social problem. For a deeper discussion why it is a problem,
see http://winestockwebdesign.com/Essays/Lisp_Curse.html
"Lisp is so powerful that problems which are technical issues in other
programming languages are social issues in Lisp."

We can counter this with easy tutorials and with writing something like
canonical Scheme. But for this, we need to define a canonical Scheme
which is hits the performance and readability sweet-spot for
Guile. Canonical code must be close to the fastest code.

Practically put: We need Andy Wingo to nitpick the tutorial about things
which will cause overheads the compiler cannot fix easily — including
expensive use of macros.

>      [1] http://hyperdev.fr
>
> PHP was successful back in the days because it was easy
> to install,

Or because it was the only thing which existed. Didn’t it make it easy
for the hoster to sandbox it — both in terms of access and in terms of
resource requirements?

>     => A scheme web application is much easier to run!

Sadly not on my cheap webhoster which provides unlimited bandwidth for
a fixed cost.

>     => While you might not be interested to build something
>        like prestashop. You very much be interested to build
>        something like Drupal or Wordpress to allow mom and
>        dad and others create a personnal experience on the
>        web.
>
> Another thing, Python has GNU Media Goblin
>
>     => We must do better!

Or, maybe collaborate with it? I’m sure that there’s lots of stuff which
could be shared, because it is not language-specific (more exactly: it’s
HTML, CSS, Javascript and images).

> What block you from contributing to the wide ecosystem of GNU Guile?

- publishing a program for non-Linux platforms (well, Windows, OSX and
  mobile phones) so most of my friends can use it.

- an alternative to `python setup.py register sdist upload`

Best wishes,
Arne

Re: How to make GNU Guile more successful

[Arne Babenhauserheide]

Arne Babenhauserheide writes:
> - an alternative to `python setup.py register sdist upload`

Since we are GNUfolks, we can leverage autotools for that. I have a
pet-project which automates the setup of autotools-projects in the style
of Rust Cargo (but without the NIH). I should share it … 

Best wishes,
Arne

Re: How to make GNU Guile more successful

[Amirouche]

Le 13/02/2017 à 12:06, Arne Babenhauserheide a écrit :

> Hi Amirouche,
>
> Thank you for your nice writeup!
>
> Amirouche writes:
>
>> I don't know why Racket is successful probably because
>> it has a very good documentation and also a documentation
>> generator written in scheme.
>>
>>    => This is a long debate and core principle of GNU project
>>       so I don't want to enter that debate
> But I want to add something: I’m getting the hang of just checking the
> info-page to look something up. And I realize that with well-written
> info-pages this is faster than googling for stack overflow.

My peers have this habit during system programming looking at man
pages and other stuff. I don't know much command line-fu. My experience,
about programming:

- looking up the API referennce (procedure index)
- looking up ansers in stack overflow

In the case of Guile programming I also look general scheme solution like
in racket. Last time I checked they had a Racket->JS translator, but not
much graphdb stuff.

The ability to generate multiple format is what makes TEXINFO powerful.

>> And it's easy to packages.
>>
>>    => Another pain topic. In reality it's very easy to package
>>       all sort of guile programs for guix. But guix itself needs
>>       to be more stable.
> I wish I had guildhall ready. Got hit by
> time-eaten-by-other-project-because-guildhall-was-priority-three.
>
> It needs to be easy to not just package some code, but also to share
> that package without hitting a bottleneck.

I got stuck with guile-git; because of new way of bindings C. I use
guile-bytestructures.


>> Python is successful because it has a massive number
>> of libraries.
> …
>>     => We (all the people reading this mail) can not just
>>        create all those projects. It requires to many domain
>>        knowledge for a handful of people to be able to compete.
>>        But what we can do is *share*, *share* and *share*
>>        again *all* what we do (on mailing list, blogs, on gitlab
>>        or whatever) whether it is a full blown project with a
>>        website made by a professional designer with tests suite,
>>        a tutorial documentation reference doc and a community or
>>        a simple *snippet* or broken experiments.
> A really simple way to share a project would be nice here. Nowadays many
> people just push something with a README to github, but github is
> unfree.

I try to move to framagit. but I still use github sometimes.


>>     => In terms of software I really think, we need something like
>>        ActiveState Python Cookbook [0]. I still go there sometime even
>>        I could simply ask the question on StackOverflow and get an
>>        answer in minutes.
>>
>>        [0] http://code.activestate.com/recipes/langs/python/
>>
>> During a long time I was said that Pythonist were
>> among the smartest developer, because simply they had
>> the guts to learn a language that was not learned in
>> school.
> I never heard that one :)
>
>> Python *was* easy to learn.
>>
>>    => Scheme is easy to learn if you stick in the "garden"
>>       (garden must be defined but it's basically what I
>>       introduce in my tutorial [1]) Which is totally the
>>       same for the Python of today (read Python 3.5+).
> Or look at py2guile: http://draketo.de/py2guile
>
> Python is no longer easy due to integrating additional syntactic forms,
> each of which makes a specific kind of code much nicer but complicates
> learning.
>
> We have something similar in foof-loop:
>
> (define (count-matching-items list predicate)
>    (loop ((for item (in-list list))
>           (with count 0
>                 (if (predicate item)
>                     (+ count 1)
>                     count)))
>          => count))
> ;       ^^ here
>
> This looks really simple in this place, but it introduces special syntax
> which will be unintelligible for someone who does did not explicitly
> learn foof-loop.
>
> And guildhall is filled with special forms I had to understand just to
> be able to do small changes.
>
> This is a social problem. For a deeper discussion why it is a problem,
> see http://winestockwebdesign.com/Essays/Lisp_Curse.html
> "Lisp is so powerful that problems which are technical issues in other
> programming languages are social issues in Lisp."

I didn't know that one

>
> We can counter this with easy tutorials and with writing something like
> canonical Scheme. But for this, we need to define a canonical Scheme
> which is hits the performance and readability sweet-spot for
> Guile. Canonical code must be close to the fastest code.
>
> Practically put: We need Andy Wingo to nitpick the tutorial about things
> which will cause overheads the compiler cannot fix easily — including
> expensive use of macros.

Right now, I don't have performance issues. Except maybe with the crawler code.

I improved stream srfi-41 performance, if you want the code.

>
>>       [1] http://hyperdev.fr
>>
>> PHP was successful back in the days because it was easy
>> to install,
> Or because it was the only thing which existed. Didn’t it make it easy
> for the hoster to sandbox it — both in terms of access and in terms of
> resource requirements?
>
>>      => A scheme web application is much easier to run!
> Sadly not on my cheap webhoster which provides unlimited bandwidth for
> a fixed cost.
Ok, I have a vps, I simply spawn the guile application in a screen and
point nginx to it. It's like configuring nginx for gunicorn. nginx forward
the traffic from :80 to :8080 or whatever based on some rules. It's like
a router for the ports being smart about http protocol.

>>      => While you might not be interested to build something
>>         like prestashop. You very much be interested to build
>>         something like Drupal or Wordpress to allow mom and
>>         dad and others create a personnal experience on the
>>         web.
>>
>> Another thing, Python has GNU Media Goblin
>>
>>      => We must do better!
> Or, maybe collaborate with it?

There is the protocol Chris is working at w3c which is interesting.
I forgot the name :p but it looked cool. At some point, I'd like to talk
some kind of federation protocol. The issue is that they require some
identification, so that people can name you unlike 4chan. Yes my current
project is like a text 4chan with a search engine.

>   I’m sure that there’s lots of stuff which
> could be shared, because it is not language-specific (more exactly: it’s
> HTML, CSS, Javascript and images).

I love their website and logo!

>
>> What block you from contributing to the wide ecosystem of GNU Guile?
> - publishing a program for non-Linux platforms (well, Windows, OSX and
>    mobile phones) so most of my friends can use it.
>
> - an alternative to `python setup.py register sdist upload`
>
> Best wishes,
> Arne
>

Re: How to make GNU Guile more successful

[Panicz Maciej Godek]

2017-02-13 12:06 GMT+01:00 Arne Babenhauserheide <arne_bab@web.de>:

> Hi Amirouche,
>
> Thank you for your nice writeup!
>
> Amirouche writes:
>
> > I don't know why Racket is successful probably because
> > it has a very good documentation and also a documentation
> > generator written in scheme.
> >
> >   => This is a long debate and core principle of GNU project
> >      so I don't want to enter that debate
>
> But I want to add something: I’m getting the hang of just checking the
> info-page to look something up. And I realize that with well-written
> info-pages this is faster than googling for stack overflow.
>

Once I was strucked by this:
http://www.wilfred.me.uk/blog/2016/07/30/example-driven-development/

There's also a similar project by William Byrd that could make the
difference
https://github.com/webyrd/Barliman

I think that Python is popualr because you don't need Emacs to work with it.
And that the milestone to the success of Racket, apart from intensive
propaganda, is Dr Racket.
(also, an important factor is that you can install both Python and Racket
on Windows, Linux
and OSX, and it Just Works)

The problem with Emacs is that it is that, although you probably can do
everything imaginable with it,
it is difficult to approach. The problem with the problem wth Emacs is that
once you get it, you usually
 have better ideas in mind than making Emacs more approachable.

There's also this problem with Scheme that it is a very diverse community
with plethora
of implementations. And while I use Guile, I don't mind using Chicken or
Chez or Gambit
for various purposes. Or even Stalin sometimes. Maybe, instead of getting a
better compiler,
we should focus on integrating existing compilers?

> And it's easy to packages.
> >
> >   => Another pain topic. In reality it's very easy to package
> >      all sort of guile programs for guix. But guix itself needs
> >      to be more stable.
>
> I wish I had guildhall ready. Got hit by
> time-eaten-by-other-project-because-guildhall-was-priority-three.
>
> It needs to be easy to not just package some code, but also to share
> that package without hitting a bottleneck.
>
>
What happened to initiatives like Scheme NOW, that tried to integrate
various groups of developers.
Or why can't we just "steal" eggs from Chicken? (I mean it. They already
have that problem solved)


> > Python is successful because it has a massive number
> > of libraries.
>

> This is a social problem. For a deeper discussion why it is a problem,
> see http://winestockwebdesign.com/Essays/Lisp_Curse.html
> "Lisp is so powerful that problems which are technical issues in other
> programming languages are social issues in Lisp."
>
> We can counter this with easy tutorials and with writing something like
> canonical Scheme. But for this, we need to define a canonical Scheme
> which is hits the performance and readability sweet-spot for
> Guile. Canonical code must be close to the fastest code.
>
Practically put: We need Andy Wingo to nitpick the tutorial about things
> which will cause overheads the compiler cannot fix easily — including
> expensive use of macros.
>
>
I definitely oppose. If Chez has something solved better, why not use Chez?
The ultimate goal is not to optimize programs, but programmers.
But I agree on the point of the canonical code. I think that the "zen of
python"
is one of important factors of making it popular. I think it could be the
same
with Scheme, but here the problem is that people have various ideas
with regard to how the "ideal" program should look like.

For example, I have a very strong opinion with regard to the use of pattern
matching
and restructuring -- I try to use destructuring almost everywhere (from
what I've seen,
Clojure programmers are alike with this regard). However, I am aware that
there are
people who don't like it that much, like Taylan or Barry Margolin.

Also, I think that the astounding success of R provides hints with regard
what is needed
for a programming language to gain popularity. R has a large repository of
packages
that fills a certain niche. When I advertised my Pamphlet on Hacker News,
someone
responded critically: "are there out of the box libraries to estimate a
zero inflated negative
binomial regression model in guile". Of course, if I knew what a
zero-inflated
negative binomial regression model, I could deliver an implementation by
just explaining
the notions used in that phrase. (But then again, I try to write functional
programs whenever
possible, often consciously sacrificing time complexity of my solutions,
and I'm aware
that not everybody will like it.)

Thanks,
Panicz

Re: How to make GNU Guile more successful

[Amirouche]

Le 13/02/2017 à 21:28, Panicz Maciej Godek a écrit :
>
>
> 2017-02-13 12:06 GMT+01:00 Arne Babenhauserheide <arne_bab@web.de 
> <mailto:arne_bab@web.de>>:
>
>     Hi Amirouche,
>
>     Thank you for your nice writeup!
>
>     Amirouche writes:
>
>     > I don't know why Racket is successful probably because
>     > it has a very good documentation and also a documentation
>     > generator written in scheme.
>     >
>     >   => This is a long debate and core principle of GNU project
>     >      so I don't want to enter that debate
>
>     But I want to add something: I’m getting the hang of just checking the
>     info-page to look something up. And I realize that with well-written
>     info-pages this is faster than googling for stack overflow.
>
>
> Once I was strucked by this:
> http://www.wilfred.me.uk/blog/2016/07/30/example-driven-development/
>
> There's also a similar project by William Byrd that could make the 
> difference
> https://github.com/webyrd/Barliman
This is awesome!

Re: How to make GNU Guile more successful

[Marko Rauhamaa]

Panicz Maciej Godek <godek.maciek@gmail.com>:

> I think that Python is popualr because you don't need Emacs to work
> with it.

You do need a special editor to work with Python, and emacs is perfect
for it.

Python is popular because it is a very traditional, dynamic, high-level
programming language. It is conceptually very similar to Lisp
(especially coupled with JSON). Definitely a gateway drug to Lisp.

> The problem with Emacs is that it is that, although you probably can
> do everything imaginable with it, it is difficult to approach.

I never noticed that problem with emacs. Emacs is really the only
reasonable way to type text. For example, when I enter text of any
length to a web app, I first type it in emacs and then copy it to the
web form.

I have typed this message in emacs.

> There's also this problem with Scheme that it is a very diverse
> community with plethora of implementations.

Scheme might have a chance as people's tastes evolve. Python is an
example of that evolution.

In my opinion one of the worst problems with Scheme is the Schemers:
Scheme lovers are often far too enthusiastic with defining new, esoteric
syntax instead of solving practical problems.

Then, there's GOOPS, which in my opinion is simply an unnatural way to
go about object-oriented programming. It does violence both to ordinary
OO way of thinking and classic Lisp idioms.

Continuations and multiple values are obstacles instead of enablers.

BTW, Python seems to be suffering from quite many self-inflicted wounds:
Python2/Python3 incompatibility, asyncio, type annotation, decorators,
dunder jungle, meta-object programming, Unicode illusion. (Guile has
fallen into that last trap as well, unfortunately.) In its eagerness to
please everyone, Python might well be jumping the shark.

There's one thing Python has over Guile, still: Python's system call
support is outstanding. For example, I managed to implement a full-blown
shell using Python. That was the first time I ever ran into
terminal-related system calls, and Python had them all.


Marko

Re: How to make GNU Guile more successful

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 8011 bytes --]


Panicz Maciej Godek <godek.maciek@gmail.com> writes:

> 2017-02-13 12:06 GMT+01:00 Arne Babenhauserheide <arne_bab@web.de>:
> There's also this problem with Scheme that it is a very diverse
> community with plethora of implementations. And while I use Guile, I
> don't mind using Chicken or Chez or Gambit for various purposes. Or
> even Stalin sometimes. Maybe, instead of getting a better compiler, we
> should focus on integrating existing compilers?

You’ll have to ask Andy Wingo about all the hidden complexity in just
wiring in a compilier. I assume it’s pretty big.

I think the r7rs folks are working in that regard, to make it possible
to write code which you can simply run through another Scheme
implementation if you need those strengths.

I started to use (import (...)) to get closer to that: Getting used to
the portable stuff (also it’s shorter than (use-modules (...)). But not
everything is portable, and performance critical stuff most definitely
isn’t.

And for the stuff I do, most things get performance critical at some
point. Even looking for null-bytes in a file becomes a bottleneck if you
have a few hundred TiB to check.

Though I mainly need to write Python at work (Scheme isn’t widespread
enough for the folks there — Python is everywhere in science
nowadays). It’s still efficient to do that and I enjoy it, but not as
much as writing Scheme.

>> And it's easy to packages.
>> >
>> >   => Another pain topic. In reality it's very easy to package
>> >      all sort of guile programs for guix. But guix itself needs
>> >      to be more stable.
>>
>> I wish I had guildhall ready. Got hit by
>> time-eaten-by-other-project-because-guildhall-was-priority-three.
>>
>> It needs to be easy to not just package some code, but also to share
>> that package without hitting a bottleneck.
>>
> What happened to initiatives like Scheme NOW, that tried to integrate
> various groups of developers.

I did not hear from that.

> Or why can't we just "steal" eggs from Chicken? (I mean it. They already
> have that problem solved)

Do you mean, integrating eggs into Guile so we could just use them?
Conceptually what would be needed for that is a metadata field: "Works
with Guile". And practically someone to write the integration.

We could also have guildhall working, and add a tool for converting
chicken eggs into guildhall packages (and ideally vice versa). Then
people could do the required Guile integration steps once and provide
the packages to all who might want them.

>> > Python is successful because it has a massive number
>> > of libraries.
>>
>
>> This is a social problem. For a deeper discussion why it is a problem,
>> see http://winestockwebdesign.com/Essays/Lisp_Curse.html
>> "Lisp is so powerful that problems which are technical issues in other
>> programming languages are social issues in Lisp."
>>
>> We can counter this with easy tutorials and with writing something like
>> canonical Scheme. But for this, we need to define a canonical Scheme
>> which is hits the performance and readability sweet-spot for
>> Guile. Canonical code must be close to the fastest code.
>>
> Practically put: We need Andy Wingo to nitpick the tutorial about things
>> which will cause overheads the compiler cannot fix easily — including
>> expensive use of macros.

> I definitely oppose. If Chez has something solved better, why not use
> Chez?

It’s not "why not use Chez", but rather "what should I teach new
people?"

They are already learning a new language. When I now go and point them
towards many different implementations which differ in details of stuff
I teach them, I’m throwing more complexity at them. More things to
understand before even starting to write working code.

Maybe it would already help to mark the code which will work the same in
all (r7rs) Schemes. Is there stuff which is well optimized in all
Schemes? Who knows that? The r7rs benchmarks look like this is very much
not the case: http://ecraven.github.io/r7rs-benchmarks/benchmark.html

(Or rather, the benchmarks would ask: "why should I *not* use Chez or
stalin for everything I do?" Because those are the fastest for most
tasks.)

But that might mean to write less elegant or efficient code to keep it
cross-implementation. Should I rather teach new people to solve a
problem as well as possible, or teach them to solve a problem in a
portable way?

In my case, I decided to use Guile by checking Scheme
implementations. There were three with roughly equal activity and
features. I chose Guile, because it’s a GNU project and it has a long
history of surviving changing maintainers, so my skills are most likely
to stay useful.

> The ultimate goal is not to optimize programs, but programmers.

I’m not sure that I want to optimize them, I want to teach them tools to
be more efficient and enjoy their work more (and teach myself the same).

> But I agree on the point of the canonical code. I think that the "zen
> of python" is one of important factors of making it popular. I think
> it could be the same with Scheme, but here the problem is that people
> have various ideas with regard to how the "ideal" program should look
> like.

I think one important point for Scheme would be to gather some consensus
points. The Zen of Python is part of what made that one community
strong. There is no reason why there should not be a Zen of Scheme,
along with implementation-specific Koans which extend it.

Do you have ideas for principles which could be shared by most Schemers?

> For example, I have a very strong opinion with regard to the use of
> pattern matching and restructuring -- I try to use destructuring
> almost everywhere (from what I've seen, Clojure programmers are alike
> with this regard). However, I am aware that there are people who don't
> like it that much, like Taylan or Barry Margolin.
>
> Also, I think that the astounding success of R provides hints with
> regard what is needed for a programming language to gain popularity. R
> has a large repository of packages that fills a certain niche. When I
> advertised my Pamphlet on Hacker News, someone responded critically:
> "are there out of the box libraries to estimate a zero inflated
> negative binomial regression model in guile". Of course, if I knew
> what a zero-inflated negative binomial regression model, I could
> deliver an implementation by just explaining the notions used in that
> phrase.

I think there’s a quote which applies here: "Most people only use 20% of
the capabilities of any program, but they do not use the same 20%."
— Joel Spolksy, if I remember correctly.

And to make people really dig into it, a programming language must work
well for all tasks they might encounter. That’s why a comprehensive
standard library is important: It gives the assurance that the skills
you learn will stay useful wherever you go.

There’s a nice quote from Rust folks: "people started to feel safe when
we told them about unsafe" — someone on GNU social.
This essentially says: they then saw that Rust could work in the
uncomfortable way some of their tasks might require. That they could
take it into harsher territory, if needed.

> (But then again, I try to write functional programs whenever
> possible, often consciously sacrificing time complexity of my
> solutions, and I'm aware that not everybody will like it.)

I look for ease of understanding, with functional programming techniques
being a useful tool to reduce complexity and to simplify refactoring.

But then, this is a point where we actually move into a similar
direction out of different motives. I think the only problem here is the
danger of assuming contradiction where there does not need to be
any. Just different nuances of Zen :)

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

Re: How to make GNU Guile more successful

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 3544 bytes --]


Amirouche <amirouche@hypermove.net> writes:

> Le 13/02/2017 à 12:06, Arne Babenhauserheide a écrit :
> My peers have this habit during system programming looking at man
> pages and other stuff. I don't know much command line-fu. My experience,
> about programming:
>
> - looking up the API referennce (procedure index)
> - looking up ansers in stack overflow

I used to do that. But Emacs C-h i C-s <search phrase> is becoming more
and more efficient for me (since I know more and more keywords).

I still google stuff, but less than in Python. It feels like the info
documentation is actually better.

> In the case of Guile programming I also look general scheme solution like
> in racket. Last time I checked they had a Racket->JS translator, but not
> much graphdb stuff.
>
> The ability to generate multiple format is what makes TEXINFO powerful.

Yes.

>> It needs to be easy to not just package some code, but also to share
>> that package without hitting a bottleneck.
>
> I got stuck with guile-git; because of new way of bindings C. I use
> guile-bytestructures.

It would be great if you could simply run "make binary" and get binaries
for multiple platforms which you can provide for different people.

But that’s just what blocks me the most.

>> A really simple way to share a project would be nice here. Nowadays many
>> people just push something with a README to github, but github is
>> unfree.
>
> I try to move to framagit. but I still use github sometimes.

I did not hear from that until today. I prefer Mercurial with my
self-written site extension[1] for static http repositories — at least
as a backup.

[1]: https://www.mercurial-scm.org/wiki/SiteExtension

>> Practically put: We need Andy Wingo to nitpick the tutorial about things
>> which will cause overheads the compiler cannot fix easily — including
>> expensive use of macros.
>
> Right now, I don't have performance issues. Except maybe with the crawler code.
>
> I improved stream srfi-41 performance, if you want the code.

Sounds good. Can you get that into Guile? Having the default
implementation more efficient would help me the most (in case I need
streams at some point).

>>>      => A scheme web application is much easier to run!
>> Sadly not on my cheap webhoster which provides unlimited bandwidth for
>> a fixed cost.
> Ok, I have a vps, I simply spawn the guile application in a screen and
> point nginx to it. It's like configuring nginx for gunicorn. nginx forward
> the traffic from :80 to :8080 or whatever based on some rules. It's like
> a router for the ports being smart about http protocol.

That’s like a remote server with full access. What I have is FTP
access. But in exchange my sites can cope with serving several hundreds
of GiB of data per month.

> There is the protocol Chris is working at w3c which is interesting.
> I forgot the name :p but it looked cool. At some point, I'd like to talk
> some kind of federation protocol. The issue is that they require some
> identification, so that people can name you unlike 4chan. Yes my current
> project is like a text 4chan with a search engine.

In GNU social you can simply use name@instance.tld

But this is again standard proliferation. I already moaned when
identi.ca turned away from status.net. The new pump.io interface never
worked well for me :(

> I love their website and logo!

:)

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

Re: How to make GNU Guile more successful

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 2940 bytes --]


Marko Rauhamaa <marko@pacujo.net> writes:
> I have typed this message in emacs.

Same for me, but getting people to use Emacs is harder. It might not
*be* that complicated, but it *feels* different.

> In my opinion one of the worst problems with Scheme is the Schemers:
> Scheme lovers are often far too enthusiastic with defining new, esoteric
> syntax instead of solving practical problems.
>
> Then, there's GOOPS, which in my opinion is simply an unnatural way to
> go about object-oriented programming. It does violence both to ordinary
> OO way of thinking and classic Lisp idioms.

GOOPS works pretty well for me where I use it (for dispatch by
type). Could you clarify your criticism: Do you think it is bad or is it
just different?

> Continuations and multiple values are obstacles instead of enablers.

I think multiple values are nice. But they are not well-integrated (I
have to import something to use them). Why do you think them enablers?

Why do you think that continuations are an obstacle (do you mean general
continuations or do you mean prompts)?

> BTW, Python seems to be suffering from quite many self-inflicted wounds:
> Python2/Python3 incompatibility,

Py2 vs. Py3 really hurts, yes.

It still really hurts and will likely hurt for the next 10 years. I
think that should be a lesson for all other language-implementors: Once
you start to become popular, you cannot do any backwards-incompatible
changes without endangering your position. Once you start to get
traction, you have to stick to your horse.

Many other wounds I do not see as much:

> asyncio, type annotation,

type annotations should provide nice ways to optimize for pypy and other
more optimizing Python implementations.

> decorators,

Decorators are really cool to use:

@with_progress
def something():
    time.sleep(10)

That’s almost like

(with-progress
  (define (something)
          (sleep 10)))

But decorators are actually definition time, so they bring the tower of
meta-levels to Python …

> dunder jungle,

What’s that?

> meta-object programming,

That’s needed to make some things elegant. "Different 20%"-rule again.

> Unicode illusion. (Guile has fallen into that last trap as well,
> unfortunately.)

I’m using Python and Guile for physics and math (to some degree), and
having good Unicode-support is great for that. What’s your practical
criticism here?

> In its eagerness to please everyone, Python might well
> be jumping the shark.
>
> There's one thing Python has over Guile, still: Python's system call
> support is outstanding. For example, I managed to implement a full-blown
> shell using Python. That was the first time I ever ran into
> terminal-related system calls, and Python had them all.

Which ones are missing in Guile?

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

Re: How to make GNU Guile more successful

[David Kastrup]

Arne Babenhauserheide <arne_bab@web.de> writes:

> Marko Rauhamaa <marko@pacujo.net> writes:
>> I have typed this message in emacs.
>
> Same for me, but getting people to use Emacs is harder. It might not
> *be* that complicated, but it *feels* different.
>
>> In my opinion one of the worst problems with Scheme is the Schemers:
>> Scheme lovers are often far too enthusiastic with defining new, esoteric
>> syntax instead of solving practical problems.
>>
>> Then, there's GOOPS, which in my opinion is simply an unnatural way to
>> go about object-oriented programming. It does violence both to ordinary
>> OO way of thinking and classic Lisp idioms.
>
> GOOPS works pretty well for me where I use it (for dispatch by
> type). Could you clarify your criticism: Do you think it is bad or is it
> just different?

My main beef with GOOPS is that it does not help with narrowing down on
a solution but rather with extending the problem space.  It is too
generic to provide guidance and a cohesive framework: if two different
people solve problems using GOOPS, the likelihood that those solutions
can be connected in interface or design better than non-GOOPS solutions
is slim.

-- 
David Kastrup

Re: How to make GNU Guile more successful

[Marko Rauhamaa]

Arne Babenhauserheide <arne_bab@web.de>:

> Marko Rauhamaa <marko@pacujo.net> writes:
>> Then, there's GOOPS, which in my opinion is simply an unnatural way
>> to go about object-oriented programming. It does violence both to
>> ordinary OO way of thinking and classic Lisp idioms.
>
> GOOPS works pretty well for me where I use it (for dispatch by type).
> Could you clarify your criticism: Do you think it is bad or is it just
> different?

GOOPS starts by defining slots. Slots are an objects internal business
and should not be visible to the outside. Instead, objects should be
black boxes that interact with methods. For example, the port interface
is a nice, classic OO API. You don't know anything about the slots of a
port.

Furthermore, I think an extensive type system is un-Lisp-like. OO is not
about things belonging to a type but objects interacting through
methods. In fact, the concept of a "class" could be done away with.

>> Continuations and multiple values are obstacles instead of enablers.
>
> I think multiple values are nice. But they are not well-integrated (I
> have to import something to use them). Why do you think them enablers?

Yes, the multiple values API is a problem in and of itself, but more
generally, what does:

    (values a b c)

give you on top of, say:

    (list a b c)

> Why do you think that continuations are an obstacle (do you mean
> general continuations or do you mean prompts)?

Continuations prevent you from implementing Python's try/finally and
emacs' (unwind-protect).

On the other hand, I don't think any application developer would come to
a point of thinking, "Hey, I know, I'll use a continuation!"

>> asyncio, type annotation,
>
> type annotations should provide nice ways to optimize for pypy and
> other more optimizing Python implementations.

You can't have it bothways. A dynamic language liberates you from
shackles and boilerplate even if it destroys your performance. The
moment you bring in the boilerplate, you are back to Java.

> Decorators are really cool to use:

Have yet to find a need for one.

>> dunder jungle,
>
> What’s that?

<URL: https://docs.python.org/3/genindex-_.html>

>> meta-object programming,
>
> That’s needed to make some things elegant. "Different 20%"-rule again.

Again, I haven't yet found a need to deviate from the regular object
semantics.

>> Unicode illusion. (Guile has fallen into that last trap as well,
>> unfortunately.)
>
> I’m using Python and Guile for physics and math (to some degree), and
> having good Unicode-support is great for that. What’s your practical
> criticism here?

In Linux, all pathnames, files, sockets, environment variables and
command-line arguments are, or deal with, bytes. The bytes are often
encoded with UTF-8, but a programming language cannot assume that, even
if LOCALE promises otherwise.

It would be better to leave Unicode out of Guile's system interface and
have the application encode and decode explicitly where needed.

>> There's one thing Python has over Guile, still: Python's system call
>> support is outstanding. For example, I managed to implement a
>> full-blown shell using Python. That was the first time I ever ran
>> into terminal-related system calls, and Python had them all.
>
> Which ones are missing in Guile?

For example tcgetattr(3). (Ok, not a system call.)

Missing epoll(2) is inconvenient. Not being able to transfer an open
file descriptor via sendmsg(2) is really bad.


Marko

Re: How to make GNU Guile more successful

[Panicz Maciej Godek]

2017-02-13 23:54 GMT+01:00 Arne Babenhauserheide <arne_bab@web.de>:

>
> Panicz Maciej Godek <godek.maciek@gmail.com> writes:
>
> > 2017-02-13 12:06 GMT+01:00 Arne Babenhauserheide <arne_bab@web.de>:
> > There's also this problem with Scheme that it is a very diverse
> > community with plethora of implementations. And while I use Guile, I
> > don't mind using Chicken or Chez or Gambit for various purposes. Or
> > even Stalin sometimes. Maybe, instead of getting a better compiler, we
> > should focus on integrating existing compilers?
>
> You’ll have to ask Andy Wingo about all the hidden complexity in just
> wiring in a compilier. I assume it’s pretty big.
>
> I think the r7rs folks are working in that regard, to make it possible
> to write code which you can simply run through another Scheme
> implementation if you need those strengths.
>
> I started to use (import (...)) to get closer to that: Getting used to
> the portable stuff (also it’s shorter than (use-modules (...)). But not
> everything is portable, and performance critical stuff most definitely
> isn’t.
>
> And for the stuff I do, most things get performance critical at some
> point. Even looking for null-bytes in a file becomes a bottleneck if you
> have a few hundred TiB to check.
>
> Though I mainly need to write Python at work (Scheme isn’t widespread
> enough for the folks there — Python is everywhere in science
> nowadays). It’s still efficient to do that and I enjoy it, but not as
> much as writing Scheme.
>
> >> And it's easy to packages.
> >> >
> >> >   => Another pain topic. In reality it's very easy to package
> >> >      all sort of guile programs for guix. But guix itself needs
> >> >      to be more stable.
> >>
> >> I wish I had guildhall ready. Got hit by
> >> time-eaten-by-other-project-because-guildhall-was-priority-three.
> >>
> >> It needs to be easy to not just package some code, but also to share
> >> that package without hitting a bottleneck.
> >>
> > What happened to initiatives like Scheme NOW, that tried to integrate
> > various groups of developers.
>
> I did not hear from that.
>
> > Or why can't we just "steal" eggs from Chicken? (I mean it. They already
> > have that problem solved)
>
> Do you mean, integrating eggs into Guile so we could just use them?
> Conceptually what would be needed for that is a metadata field: "Works
> with Guile". And practically someone to write the integration.
>
>
There's surely many ways to approach that issue. The point is that for some
reason, Schemers from various tribes prefer to reinvent the wheel,
rather than use an existing one. (Not that I am any different)
However, I also think that these CPAN-alike solutions are far from optimal:
ideally, programming could be made an experience similar the game
"The Journey" by thatgamecompany (where you travel to your goal and
sometimes
encounter other players heading the same direction), and the repository
itself could look
more like Wikipedia (but first we'd need to start perceiveing programming as
a compact way of representing knowledge)

> Practically put: We need Andy Wingo to nitpick the tutorial about things
> >> which will cause overheads the compiler cannot fix easily — including
> >> expensive use of macros.
>
> > I definitely oppose. If Chez has something solved better, why not use
> > Chez?
>
> It’s not "why not use Chez", but rather "what should I teach new
> people?"
>
> They are already learning a new language. When I now go and point them
> towards many different implementations which differ in details of stuff
> I teach them, I’m throwing more complexity at them. More things to
> understand before even starting to write working code.
>
> I totally agree. Programmers shouldn't be concerned whether they're
using Chez or Guile or Gambit or Stalin or whatever else. Ideally, they
should do well without even knowing that such things exist.
There are Schemes, such as Kawa or Biwa or IronScheme, that are tied
to their environment, but most of them just run on the PC


> Maybe it would already help to mark the code which will work the same in
> all (r7rs) Schemes. Is there stuff which is well optimized in all
> Schemes? Who knows that? The r7rs benchmarks look like this is very much
> not the case: http://ecraven.github.io/r7rs-benchmarks/benchmark.html
>
> (Or rather, the benchmarks would ask: "why should I *not* use Chez or
> stalin for everything I do?" Because those are the fastest for most
> tasks.)
>
> But that might mean to write less elegant or efficient code to keep it
> cross-implementation. Should I rather teach new people to solve a
> problem as well as possible, or teach them to solve a problem in a
> portable way?
>
>
I think that in principle, the programmer should only focus on writing
most elegant code


> In my case, I decided to use Guile by checking Scheme
> implementations. There were three with roughly equal activity and
> features. I chose Guile, because it’s a GNU project and it has a long
> history of surviving changing maintainers, so my skills are most likely
> to stay useful.
>
>
That's interesting. I chose it by trying to add a scripting language to my
game engine written in C++. But I guess that if I was starting now, I'd
write
the whole engine in Chez (which wasn't opensource until last year)


> > The ultimate goal is not to optimize programs, but programmers.
>
> I’m not sure that I want to optimize them, I want to teach them tools to
> be more efficient and enjoy their work more (and teach myself the same).
>
>
Yeah, maybe I used a harsh word, but I meant exactly optimizing
programmers' efficiency (or if it sounds too bad, "giving them tools
to be more efficient")


> I think one important point for Scheme would be to gather some consensus
> points. The Zen of Python is part of what made that one community
> strong. There is no reason why there should not be a Zen of Scheme,
> along with implementation-specific Koans which extend it.
>
> Do you have ideas for principles which could be shared by most Schemers?
>
>
It depends on the most Schemers, not me :)
However, you can have a look at the (grand scheme) glossary that I've been
maintaining for some time:

https://github.com/plande/grand-scheme

of course, the intent is to make the glossary usable for someone else than
me,
and I can boast that I have at least one user distinct from the maintainer.
You can have a look and let me know what you think. (for now it only works
with Guile). In particular, are there any factors that could convince you to
adapt it as a base for your library, or what would you want changed
(I assume you won't stop at the README file).
Anyways, the rules (of thumb) are:


   - functions should be pure
   - language constructs that cause side-effects should be encapsulated
   with macros that limit the extent of those side-effects
   - if possible, use the quasiquote macro, rather than cons, append or list
   - prefer the use of match <http://synthcode.com/scheme/match.scm> to cond,
   if the patterns are simple
   - never use abbreviations
   - use commented prepositions to separate arguments

Also, I recently added infix syntax to Scheme, but (unlike in Racket or
David Wheeler's solutions) in the way it is still a prefix syntax. So for
example,
instead of writing (< a b), one ought to write (is a < b). The rationale is
that
prefix notation is terrible for non-symmetrical binary predicates, because
it
obscures the roles of respective arguments (in addition, (is a < b <= c)
expands
to (and (< a b) (<= b c))). I'm still not sure whether it is a good idea,
but I have
no reasons to think that it is bad either :]
[Curiously enough, I think that prefix notation in arithmetic is just fine:
(+ a b c) reads as "sum of a, b and c"]

> Also, I think that the astounding success of R provides hints with
> > regard what is needed for a programming language to gain popularity. R
> > has a large repository of packages that fills a certain niche. When I
> > advertised my Pamphlet on Hacker News, someone responded critically:
> > "are there out of the box libraries to estimate a zero inflated
> > negative binomial regression model in guile". Of course, if I knew
> > what a zero-inflated negative binomial regression model, I could
> > deliver an implementation by just explaining the notions used in that
> > phrase.
>
> I think there’s a quote which applies here: "Most people only use 20% of
> the capabilities of any program, but they do not use the same 20%."
> — Joel Spolksy, if I remember correctly.
>
> And to make people really dig into it, a programming language must work
> well for all tasks they might encounter. That’s why a comprehensive
> standard library is important: It gives the assurance that the skills
> you learn will stay useful wherever you go.
>
> There’s a nice quote from Rust folks: "people started to feel safe when
> we told them about unsafe" — someone on GNU social.
> This essentially says: they then saw that Rust could work in the
> uncomfortable way some of their tasks might require. That they could
> take it into harsher territory, if needed.
>
> I like it that you and Amirouche look a lot at the sociocultural
perspective
of programming


> > (But then again, I try to write functional programs whenever
> > possible, often consciously sacrificing time complexity of my
> > solutions, and I'm aware that not everybody will like it.)
>
> I look for ease of understanding, with functional programming techniques
> being a useful tool to reduce complexity and to simplify refactoring.
>
> But then, this is a point where we actually move into a similar
> direction out of different motives. I think the only problem here is the
> danger of assuming contradiction where there does not need to be
> any. Just different nuances of Zen :)
>
> Yes, to me the ease of understanding is also crucial. (I'd even go
further and say that undestranding is the very point of programming)

Best regards,
Panicz

Re: How to make GNU Guile more successful

[Linas Vepstas]

Hey,

On Mon, Feb 13, 2017 at 11:59 PM, Marko Rauhamaa <marko@pacujo.net> wrote:
>
> Arne Babenhauserheide <arne_bab@web.de>:
>
> > Marko Rauhamaa <marko@pacujo.net> writes:
> >> Then, there's GOOPS, which in my opinion is simply an unnatural way
> >> to go about object-oriented programming. It does violence both to
> >> ordinary OO way of thinking and classic Lisp idioms.
> >
> > GOOPS works pretty well for me where I use it (for dispatch by type).
> > Could you clarify your criticism: Do you think it is bad or is it just
> > different?
>
> GOOPS starts by defining slots. Slots are an objects internal business
> and should not be visible to the outside. Instead, objects should be
> black boxes that interact with methods. For example, the port interface
> is a nice, classic OO API. You don't know anything about the slots of a
> port.


Yes, but ... maybe goops should be renamed guijs ? Javascript has this
... wonderful ... ability to just attach new data and new methods to any
"object" at runtime.   Now, I've never used goops in any serious way, so
maybe I'm quite off the mark, but it seems to me that maybe this is what
slots are trying to accomplish.

That is, when you say "classic OO API", it seems you really have C++ in
the back of your mind.  But the early days of OO had all sorts of weird ways
of saying things and thinking about things (e.g. "sending messages to objects")
that was a bigger, more inclusive tent.  Perhaps GOOPS is just OO in this
older, and now deprecated sense.

My naive impression is that some javascrtipt-ish extension to guile would
be very cool.  Perhaps goops could be pushed further in that direction.


> >> Continuations and multiple values are obstacles instead of enablers.
> >
> > I think multiple values are nice. But they are not well-integrated (I
> > have to import something to use them). Why do you think them enablers?
>
> Yes, the multiple values API is a problem in and of itself, but more
> generally, what does:
>
>     (values a b c)
>
> give you on top of, say:
>
>     (list a b c)

+1 every time I think I need multiple values, I try it, and then back
away from it.
Its a cool-sounding idea that is a consistent obstacle.


> > Why do you think that continuations are an obstacle (do you mean
> > general continuations or do you mean prompts)?
>
> Continuations prevent you from implementing Python's try/finally and
> emacs' (unwind-protect).
>
> On the other hand, I don't think any application developer would come to
> a point of thinking, "Hey, I know, I'll use a continuation!"

Continuations are stunningly-badly explained, almost everywhere.  More
or less every example I've ever seen is pointlessly obtuse and opaque,
seemingly written to demonstrate how smart the author is, and how stupid
the reader is.

There is one place where an application programmer could say "Hey, I know,
I'll use a continuation!" -- whenever they need a single "global"  "OO object"
with a "global variable" in it. I use them for implementing statistics
printing and
profiling. The idea is simple, the solution is elegant:

-- I want to have a single "global" counter which I will increment by one from
now until forever.
-- I want to have a function to increment it, and I want to call that function
at any time, from anywhere at all.
-- every hundreth time that it is called, it will print something like
"xyz operations per second"

Turns out that continuations offer a marvelously simple, easy-to-understand
easy-to-use solution to this problem.   You'd think that there would be some
tutorial or some common "here's the cookbook way of implementing this"
but I sure haven't seen one.  But once you get the hang of the above example,
one can see how it could be fleshed out into a full-fledged OO-like
message-passing system, complete with frames and slots.

Continuations do not need to be a swear word; its just that they're badly
explained.  They are a stumbling block, but that's in part because there's
no analogous concept in proceedural languages; so most programmers
never have to learn it.

> > Decorators are really cool to use:
>
> Have yet to find a need for one.

Sounds like gain-saying.  Just because you don't need it, doesn't mean its
not useful.  "A typical user uses 20% of an application. The problem is,
different users use a different 20%".  "recall there were dozens of wikis before
wikipedia, dozens of search engines before google, dozens of social media
sites before facebook"

I conclude that the difference between a successful(popular) and a failed
(unpopular) tech is often very subtle.  For all I know, decorators "done right"
in guile could skyrocket it to the top of the popularity ranking.  Or not. Just
be careful in dismissing some of these ideas.

> >> Unicode illusion. (Guile has fallen into that last trap as well,
> >> unfortunately.)
> >
> > I’m using Python and Guile for physics and math (to some degree), and
> > having good Unicode-support is great for that. What’s your practical
> > criticism here?

Unicode support in guile remains rocky and buggy, as I once again
rediscovered recently when pumping Chinese through it.

> It would be better to leave Unicode out of Guile's system interface and
> have the application encode and decode explicitly where needed.

Huh?  That would suck.  Unless I misunderstand, in which case it would
be great.   Unicode is just a bunch of bytes that are null-terminated.  Guile
does very stupidly try to  convert unicode strings into arrays of wide chars
which is absolutely idiotic, and a total bottleneck for what I do.
Its a useless,
pointless processing step.  I keep planning on submitting a patch to fix this
but never have time.

I regularly have strings that are 10 Mbytes or more, that contain mixtures of
scheme code and chinese characters (in utf8), that I have to send to guile
for eval. Anything that would make that faster would be awesome.

--linas

Re: How to make GNU Guile more successful

[Marko Rauhamaa]

Linas Vepstas <linasvepstas@gmail.com>:

> Unicode is just a bunch of bytes that are null-terminated.

That is not what Unicode is. Maybe you are thinking of UTF-8, an
ingenious way to map a Unicode sequence onto a byte sequence. Trouble
is, there are byte sequences that are illegal UTF-8. While they are
illegal for Unicode use, they are perfectly possible and valid as far as
the (Linux) operating system is concerned.


Marko

Re: How to make GNU Guile more successful

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 8750 bytes --]


Panicz Maciej Godek <godek.maciek@gmail.com> writes:
> There's surely many ways to approach that issue. The point is that for
> some reason, Schemers from various tribes prefer to reinvent the
> wheel, rather than use an existing one. (Not that I am any different)
> However, I also think that these CPAN-alike solutions are far from
> optimal: ideally, programming could be made an experience similar the
> game "The Journey" by thatgamecompany (where you travel to your goal
> and sometimes encounter other players heading the same direction), and
> the repository itself could look more like Wikipedia (but first we'd
> need to start perceiveing programming as a compact way of representing
> knowledge)

That sounds somehow like stackoverflow. We might already be going there :)

>> Practically put: We need Andy Wingo to nitpick the tutorial about things
>> >> which will cause overheads the compiler cannot fix easily — including
>> >> expensive use of macros.
>>
>> > I definitely oppose. If Chez has something solved better, why not use
>> > Chez?
>>
>> It’s not "why not use Chez", but rather "what should I teach new
>> people?"
>>
>> They are already learning a new language. When I now go and point them
>> towards many different implementations which differ in details of stuff
>> I teach them, I’m throwing more complexity at them. More things to
>> understand before even starting to write working code.
>>
> I totally agree. Programmers shouldn't be concerned whether they're
> using Chez or Guile or Gambit or Stalin or whatever else. Ideally, they
> should do well without even knowing that such things exist.
> There are Schemes, such as Kawa or Biwa or IronScheme, that are tied
> to their environment, but most of them just run on the PC

I think this is a point which cannot be reached, because the design of
the Scheme system itself must make certain tradeoffs — especially for
the default environment (what to present a new user). If that is too
large, then users cannot start quickly. If it is too small, then users
cannot solve problems easily. But different problem spaces require
different base tools.

Another points are the basic datastructures: If they are too complex,
then the system cannot be used easily on tiny computers.

Essentially you have to find reasons for decisions like this:
https://docs.python.org/3/faq/design.html#how-are-lists-implemented
and this:
https://wiki.python.org/moin/TimeComplexity

>> Maybe it would already help to mark the code which will work the same in
>> all (r7rs) Schemes. Is there stuff which is well optimized in all
>> Schemes? Who knows that? The r7rs benchmarks look like this is very much
>> not the case: http://ecraven.github.io/r7rs-benchmarks/benchmark.html
>>
>> (Or rather, the benchmarks would ask: "why should I *not* use Chez or
>> stalin for everything I do?" Because those are the fastest for most
>> tasks.)
>>
>> But that might mean to write less elegant or efficient code to keep it
>> cross-implementation. Should I rather teach new people to solve a
>> problem as well as possible, or teach them to solve a problem in a
>> portable way?

> I think that in principle, the programmer should only focus on writing
> most elegant code

I think there are several different kinds of elegance. There’s technical
elegance (combining existing things to build something unexpectedly
powerful with minimal effort), readability (being easy to understand for
newcomers), minimal code (solving a problem with the minimal amount of
code — often close to mathematics), uniformity (using the same basic
structure everywhere), efficiency (solving a problem with minimal
resource-consumption), ...
I’m sure we’ll find more.

Sadly these elegances contradict each other in some points.

>> In my case, I decided to use Guile by checking Scheme
>> implementations. There were three with roughly equal activity and
>> features. I chose Guile, because it’s a GNU project and it has a long
>> history of surviving changing maintainers, so my skills are most likely
>> to stay useful.

> That's interesting. I chose it by trying to add a scripting language
> to my game engine written in C++. But I guess that if I was starting
> now, I'd write the whole engine in Chez (which wasn't opensource until
> last year)

Chez becoming free software is an interesting development.

>> > The ultimate goal is not to optimize programs, but programmers.
>>
>> I’m not sure that I want to optimize them, I want to teach them tools to
>> be more efficient and enjoy their work more (and teach myself the same).
> Yeah, maybe I used a harsh word, but I meant exactly optimizing
> programmers' efficiency (or if it sounds too bad, "giving them tools
> to be more efficient")

Sounds good :)

>> I think one important point for Scheme would be to gather some consensus
>> points. The Zen of Python is part of what made that one community
>> strong. There is no reason why there should not be a Zen of Scheme,
>> along with implementation-specific Koans which extend it.
>>
>> Do you have ideas for principles which could be shared by most Schemers?
>>
>>
> It depends on the most Schemers, not me :)

That’s why its a social problem :)

Finding a set of basic koans which most Schemers agree with requires
talking to and knowing a lot of Schemers.

> However, you can have a look at the (grand scheme) glossary that I've been
> maintaining for some time:
>
> https://github.com/plande/grand-scheme

This looks interesting.

> of course, the intent is to make the glossary usable for someone else than
> me,
> and I can boast that I have at least one user distinct from the maintainer.
> You can have a look and let me know what you think. (for now it only works
> with Guile). In particular, are there any factors that could convince you to
> adapt it as a base for your library, or what would you want changed
> (I assume you won't stop at the README file).

I’d mainly need practical examples. Tell me how I’d solve a problem
with (grand scheme) and how it would be solved without it — showing me
why using it is better.

That means: Documentation.

Also I’d need to know whether some of these methods have performance
problems: Will this scale as well as the method without (grand scheme)?

>    - functions should be pure
>    - language constructs that cause side-effects should be encapsulated
>    with macros that limit the extent of those side-effects
>    - if possible, use the quasiquote macro, rather than cons, append or list
>    - prefer the use of match <http://synthcode.com/scheme/match.scm> to cond,
>    if the patterns are simple
>    - never use abbreviations
>    - use commented prepositions to separate arguments

As guidelines for easy-to-use libraries, that sounds good.

> Also, I recently added infix syntax to Scheme, but (unlike in Racket
> or David Wheeler's solutions) in the way it is still a prefix
> syntax. So for example, instead of writing (< a b), one ought to write
> (is a < b). The rationale is that prefix notation is terrible for
> non-symmetrical binary predicates, because it obscures the roles of
> respective arguments (in addition, (is a < b <= c) expands to (and (<
> a b) (<= b c))). I'm still not sure whether it is a good idea, but I
> have no reasons to think that it is bad either :] [Curiously enough, I
> think that prefix notation in arithmetic is just fine: (+ a b c) reads
> as "sum of a, b and c"]

This is an old idea, but using (is) as prefix looks
interesting. Typically the prefix was something like (nfx), which
doesn’t say anything semantic (other than that what follows uses infix
syntax). It’s like an expanded equal?

(if (is a = 5)
    #t
    #f)

(cond
    ((is q = p)
     q)
    ((is p > 5)
     p)
    (else
     q))

This seems pretty close to natural language. Can I also use (is A 5)
instead of (equal? A 5)?

Though (is? a = 5) might be closer to canonical Scheme.

> I like it that you and Amirouche look a lot at the sociocultural
> perspective of programming

I’m glad to hear that :)

> Yes, to me the ease of understanding is also crucial. (I'd even go
> further and say that undestranding is the very point of programming)

I would not go that far: We’re programming both for the future
maintainers of the code and for the computer. This restricts us in
expressing things, because we always have to keep two targets audiences
in mind (many different systems *and* many different people).

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

Re: How to make GNU Guile more successful

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 6436 bytes --]


Marko Rauhamaa <marko@pacujo.net> writes:

> Arne Babenhauserheide <arne_bab@web.de>:
>
>> Marko Rauhamaa <marko@pacujo.net> writes:
>>> Then, there's GOOPS, which in my opinion is simply an unnatural way
>>> to go about object-oriented programming. It does violence both to
>>> ordinary OO way of thinking and classic Lisp idioms.
>>
>> GOOPS works pretty well for me where I use it (for dispatch by type).
>> Could you clarify your criticism: Do you think it is bad or is it just
>> different?
>
> GOOPS starts by defining slots. Slots are an objects internal business
> and should not be visible to the outside. Instead, objects should be
> black boxes that interact with methods. For example, the port interface
> is a nice, classic OO API. You don't know anything about the slots of a
> port.
>
> Furthermore, I think an extensive type system is un-Lisp-like. OO is not
> about things belonging to a type but objects interacting through
> methods. In fact, the concept of a "class" could be done away with.
>
>>> Continuations and multiple values are obstacles instead of enablers.
>>
>> I think multiple values are nice. But they are not well-integrated (I
>> have to import something to use them). Why do you think them enablers?
>
> Yes, the multiple values API is a problem in and of itself, but more
> generally, what does:
>
>     (values a b c)
>
> give you on top of, say:
>
>     (list a b c)

Practically it gives me (let-values (((a b c) (values 1 2 3))) a)

I could get the same by using match on a list, but then I have to use
match on a list instead of just assigning the values. let-values could
be designed to work with lists, though.

So conceptually I assume that the real difference is in implementation:
avoiding the construction of a list.

>> Why do you think that continuations are an obstacle (do you mean
>> general continuations or do you mean prompts)?
>
> Continuations prevent you from implementing Python's try/finally and
> emacs' (unwind-protect).

If I understand it correctly, this is only true, if there are resources
in use which do not get created in the try. I hit that problem when
implementing 'with':
https://bitbucket.org/ArneBab/wisp/src/3a654cfe6632af4b0002ce98c753c07c400aac55/examples/with.w#with.w-8

with (open-file "with.w" "r") as port
      format #t "~a\n" (read port)

^ working Guile code which ensures that the port is closed after it is
  no longer used. But it can break if a continuation is used. To fix
  that, I’d have to add guards which save the state of the port when
  code creates a continuation and reconstructs that state when the
  continuation is used.

> On the other hand, I don't think any application developer would come to
> a point of thinking, "Hey, I know, I'll use a continuation!"

Error-handling is simply a specific case of that. In Python people use
exceptions for flow control, which can be efficient when the
non-exception case is much, much more common than the exception case
(because in the first case the overhead of that is almost zero).

>>> asyncio, type annotation,
>>
>> type annotations should provide nice ways to optimize for pypy and
>> other more optimizing Python implementations.
>
> You can't have it bothways. A dynamic language liberates you from
> shackles and boilerplate even if it destroys your performance. The
> moment you bring in the boilerplate, you are back to Java.

It’s optional boilerplate which you only add where you really need it.

>> Decorators are really cool to use:
>
> Have yet to find a need for one.

I used them quite a bit. The most efficient one was a caching decorator:

@cached
def fun(a):
    # something expensive
    return a

In Python 3.2 you can use the standard lru-cache for that:
https://docs.python.org/3/library/functools.html#functools.lru_cache

>>> dunder jungle,
>>
>> What’s that?
>
> <URL: https://docs.python.org/3/genindex-_.html>

Ah, yes :)

I think these are a good way to show that I as developer am venturing
into regions I should only enter when I am really, really sure I want
to, because they can cause problems I may not see right away.

Sadly this rule is broken with if __name__ == "__main__": ...
and for def __init__(self)

So I think it’s good to have that, but not good that programmers need to
use that during typical programming tasks.

>>> meta-object programming,
>>
>> That’s needed to make some things elegant. "Different 20%"-rule again.
>
> Again, I haven't yet found a need to deviate from the regular object
> semantics.

I saw some, like actually accessing and changing the bytecode at runtime
to add tracing commands.

>>> Unicode illusion. (Guile has fallen into that last trap as well,
>>> unfortunately.)
>>
>> I’m using Python and Guile for physics and math (to some degree), and
>> having good Unicode-support is great for that. What’s your practical
>> criticism here?
>
> In Linux, all pathnames, files, sockets, environment variables and
> command-line arguments are, or deal with, bytes. The bytes are often
> encoded with UTF-8, but a programming language cannot assume that, even
> if LOCALE promises otherwise.
>
> It would be better to leave Unicode out of Guile's system interface and
> have the application encode and decode explicitly where needed.

I’m not sure about that. Python3 requires explicit encode/decode, and
it’s a lot of hassle to port from Py2 to Py3 if your application does
lots of socket-operations. I have firsthand experience with that, and
the debugging is gruesome.

But it might have been much better if the program had been created with
Python3 from the start.

>>> There's one thing Python has over Guile, still: Python's system call
>>> support is outstanding. For example, I managed to implement a
>>> full-blown shell using Python. That was the first time I ever ran
>>> into terminal-related system calls, and Python had them all.
>>
>> Which ones are missing in Guile?
>
> For example tcgetattr(3). (Ok, not a system call.)
>
> Missing epoll(2) is inconvenient. Not being able to transfer an open
> file descriptor via sendmsg(2) is really bad.

Do you know whether there is a specific reason for that or whether these
could be added?

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

Re: How to make GNU Guile more successful

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1562 bytes --]


David Kastrup <dak@gnu.org> writes:

> Arne Babenhauserheide <arne_bab@web.de> writes:
>
>> Marko Rauhamaa <marko@pacujo.net> writes:
>>> I have typed this message in emacs.
>>
>> Same for me, but getting people to use Emacs is harder. It might not
>> *be* that complicated, but it *feels* different.
>>
>>> In my opinion one of the worst problems with Scheme is the Schemers:
>>> Scheme lovers are often far too enthusiastic with defining new, esoteric
>>> syntax instead of solving practical problems.
>>>
>>> Then, there's GOOPS, which in my opinion is simply an unnatural way to
>>> go about object-oriented programming. It does violence both to ordinary
>>> OO way of thinking and classic Lisp idioms.
>>
>> GOOPS works pretty well for me where I use it (for dispatch by
>> type). Could you clarify your criticism: Do you think it is bad or is it
>> just different?
>
> My main beef with GOOPS is that it does not help with narrowing down on
> a solution but rather with extending the problem space.  It is too
> generic to provide guidance and a cohesive framework: if two different
> people solve problems using GOOPS, the likelihood that those solutions
> can be connected in interface or design better than non-GOOPS solutions
> is slim.

That’s a good point.

A framework which provides that guidance could be built on top of GOOPS,
though. Or would that carry around too much burden from the generic
approach it’s based on?

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

Re: How to make GNU Guile more successful

[Christopher Allan Webber]

David Kastrup writes:

> Arne Babenhauserheide <arne_bab@web.de> writes:
>
>> Marko Rauhamaa <marko@pacujo.net> writes:
>>> I have typed this message in emacs.
>>
>> Same for me, but getting people to use Emacs is harder. It might not
>> *be* that complicated, but it *feels* different.
>>
>>> In my opinion one of the worst problems with Scheme is the Schemers:
>>> Scheme lovers are often far too enthusiastic with defining new, esoteric
>>> syntax instead of solving practical problems.
>>>
>>> Then, there's GOOPS, which in my opinion is simply an unnatural way to
>>> go about object-oriented programming. It does violence both to ordinary
>>> OO way of thinking and classic Lisp idioms.
>>
>> GOOPS works pretty well for me where I use it (for dispatch by
>> type). Could you clarify your criticism: Do you think it is bad or is it
>> just different?
>
> My main beef with GOOPS is that it does not help with narrowing down on
> a solution but rather with extending the problem space.  It is too
> generic to provide guidance and a cohesive framework: if two different
> people solve problems using GOOPS, the likelihood that those solutions
> can be connected in interface or design better than non-GOOPS solutions
> is slim.

Generic methods help somewhat with this though right?  That's part of
the goal of generic methods even; the portion of SICP where they are
introduced is even about getting two programmers who don't necessarily
get along to get their code to work together.

Re: How to make GNU Guile more successful

[sirgazil]

Amirouche says:
> What block you from contributing to the wide ecosystem of GNU Guile?

I think about Guile (and GuixSD) almost every day, but I don't contribute as much as I'd like mostly because of lack of resources to do so. Sadly, it's hard to find the opportunity to immerse yourself in libre projects.

As for how to make GNU Guile more successful, I'd like to see what I call a "Hello Guile" series of guides targeted at people who just want to start programming. The series would include:

+ Hello Guile: Getting Started (An overview of programming, and how to start with Guile)
+ Hello Guile: Let's develop a console application
+ Hello Guile: Let's develop a desktop application
+ Hello Guile: Let's develop a web application
+ Hello Guile: Let's develop a 2D game
+ Hello Guile: Let's package your software for distribution

Additionally, I'd like the following:

+ More high level libraries [1]
+ Add Guile to the GNOME Developer Center [2][3][4]
+ Package more guile libraries in Guix

Finally, we need to clone a bunch of people to do all this work :)


[1]: For example, I'm following the Systematic Program Design course with
Gregor Kiczales. The course uses DrRacket for interactive
programming and a language called BSL (Beginning Student
Language).

When I started, my plan was to follow the course in Guile instead,
since the course is about the design method, and not the tools, but
I had to stop very early when I had to do some projects involving
graphics and animation.

Personally, I don't feel the need for something like DrRacket,
but it would have been nice to be able to find a Guile "canvas" or a
2D game library to play easily with basic graphics and animations.
[2]: https://developer.gnome.org/
[3]: https://developer.gnome.org/gnome-devel-demos/stable/
[4]: https://developer.gnome.org/references (see Language Bindings section)



---
https://sirgazil.bitbucket.io/

Re: How to make GNU Guile more successful

[Amirouche]

Le 16/02/2017 à 20:18, sirgazil a écrit :
> Amirouche says:
>> What block you from contributing to the wide ecosystem of GNU Guile?
> I think about Guile (and GuixSD) almost every day, but I don't contribute as much as I'd like mostly because of lack of resources to do so. Sadly, it's hard to find the opportunity to immerse yourself in libre projects.
>
> As for how to make GNU Guile more successful, I'd like to see what I call a "Hello Guile" series of guides targeted at people who just want to start programming. The series would include:
>
> + Hello Guile: Getting Started (An overview of programming, and how to start with Guile)
> + Hello Guile: Let's develop a console application
> + Hello Guile: Let's develop a desktop application
> + Hello Guile: Let's develop a web application
> + Hello Guile: Let's develop a 2D game
> + Hello Guile: Let's package your software for distribution
>
> Additionally, I'd like the following:
>
> + More high level libraries [1]
> + Add Guile to the GNOME Developer Center [2][3][4]
> + Package more guile libraries in Guix
>
> Finally, we need to clone a bunch of people to do all this work :)
>
>
> [1]: For example, I'm following the Systematic Program Design course with
> Gregor Kiczales. The course uses DrRacket for interactive
> programming and a language called BSL (Beginning Student
> Language).
>
> When I started, my plan was to follow the course in Guile instead,
> since the course is about the design method, and not the tools, but
> I had to stop very early when I had to do some projects involving
> graphics and animation.
>
> Personally, I don't feel the need for something like DrRacket,
> but it would have been nice to be able to find a Guile "canvas" or a
> 2D game library to play easily with basic graphics and animations.
> [2]: https://developer.gnome.org/
> [3]: https://developer.gnome.org/gnome-devel-demos/stable/
> [4]: https://developer.gnome.org/references (see Language Bindings section)
>
>
>
> ---
> https://sirgazil.bitbucket.io/
I have to +1 all of this post.

Re: How to make GNU Guile more successful

[Amirouche]

Le 14/02/2017 à 22:35, Arne Babenhauserheide a écrit :
> Panicz Maciej Godek <godek.maciek@gmail.com> writes:
>> There's surely many ways to approach that issue. The point is that for
>> some reason, Schemers from various tribes prefer to reinvent the
>> wheel, rather than use an existing one. (Not that I am any different)
>> However, I also think that these CPAN-alike solutions are far from
>> optimal: ideally, programming could be made an experience similar the
>> game "The Journey" by thatgamecompany (where you travel to your goal
>> and sometimes encounter other players heading the same direction), and
>> the repository itself could look more like Wikipedia (but first we'd
>> need to start perceiveing programming as a compact way of representing
>> knowledge)
> That sounds somehow like stackoverflow. We might already be going there :)

That's a very good idea!

>
>>> Practically put: We need Andy Wingo to nitpick the tutorial about things
>>>>> which will cause overheads the compiler cannot fix easily — including
>>>>> expensive use of macros.
>>>> I definitely oppose. If Chez has something solved better, why not use
>>>> Chez?
>>> It’s not "why not use Chez", but rather "what should I teach new
>>> people?"
>>>
>>> They are already learning a new language. When I now go and point them
>>> towards many different implementations which differ in details of stuff
>>> I teach them, I’m throwing more complexity at them. More things to
>>> understand before even starting to write working code.
>>>
>> I totally agree. Programmers shouldn't be concerned whether they're
>> using Chez or Guile or Gambit or Stalin or whatever else. Ideally, they
>> should do well without even knowing that such things exist.
>> There are Schemes, such as Kawa or Biwa or IronScheme, that are tied
>> to their environment, but most of them just run on the PC
> I think this is a point which cannot be reached, because the design of
> the Scheme system itself must make certain tradeoffs — especially for
> the default environment (what to present a new user). If that is too
> large, then users cannot start quickly. If it is too small, then users
> cannot solve problems easily. But different problem spaces require
> different base tools.
>
> Another points are the basic datastructures: If they are too complex,
> then the system cannot be used easily on tiny computers.
>
> Essentially you have to find reasons for decisions like this:
> https://docs.python.org/3/faq/design.html#how-are-lists-implemented
> and this:
> https://wiki.python.org/moin/TimeComplexity
>
>>> Maybe it would already help to mark the code which will work the same in
>>> all (r7rs) Schemes. Is there stuff which is well optimized in all
>>> Schemes? Who knows that? The r7rs benchmarks look like this is very much
>>> not the case: http://ecraven.github.io/r7rs-benchmarks/benchmark.html
>>>
>>> (Or rather, the benchmarks would ask: "why should I *not* use Chez or
>>> stalin for everything I do?" Because those are the fastest for most
>>> tasks.)
>>>
>>> But that might mean to write less elegant or efficient code to keep it
>>> cross-implementation. Should I rather teach new people to solve a
>>> problem as well as possible, or teach them to solve a problem in a
>>> portable way?
>> I think that in principle, the programmer should only focus on writing
>> most elegant code
> I think there are several different kinds of elegance. There’s technical
> elegance (combining existing things to build something unexpectedly
> powerful with minimal effort), readability (being easy to understand for
> newcomers), minimal code (solving a problem with the minimal amount of
> code — often close to mathematics), uniformity (using the same basic
> structure everywhere), efficiency (solving a problem with minimal
> resource-consumption), ...
> I’m sure we’ll find more.
>
> Sadly these elegances contradict each other in some points.
>
>>> In my case, I decided to use Guile by checking Scheme
>>> implementations. There were three with roughly equal activity and
>>> features. I chose Guile, because it’s a GNU project and it has a long
>>> history of surviving changing maintainers, so my skills are most likely
>>> to stay useful.
>> That's interesting. I chose it by trying to add a scripting language
>> to my game engine written in C++. But I guess that if I was starting
>> now, I'd write the whole engine in Chez (which wasn't opensource until
>> last year)
> Chez becoming free software is an interesting development.
>
>>>> The ultimate goal is not to optimize programs, but programmers.
>>> I’m not sure that I want to optimize them, I want to teach them tools to
>>> be more efficient and enjoy their work more (and teach myself the same).
>> Yeah, maybe I used a harsh word, but I meant exactly optimizing
>> programmers' efficiency (or if it sounds too bad, "giving them tools
>> to be more efficient")
> Sounds good :)
>
>>> I think one important point for Scheme would be to gather some consensus
>>> points. The Zen of Python is part of what made that one community
>>> strong. There is no reason why there should not be a Zen of Scheme,
>>> along with implementation-specific Koans which extend it.
>>>
>>> Do you have ideas for principles which could be shared by most Schemers?
>>>
>>>
>> It depends on the most Schemers, not me :)
> That’s why its a social problem :)
>
> Finding a set of basic koans which most Schemers agree with requires
> talking to and knowing a lot of Schemers.
>
>> However, you can have a look at the (grand scheme) glossary that I've been
>> maintaining for some time:
>>
>> https://github.com/plande/grand-scheme
> This looks interesting.
>
>> of course, the intent is to make the glossary usable for someone else than
>> me,
>> and I can boast that I have at least one user distinct from the maintainer.
>> You can have a look and let me know what you think. (for now it only works
>> with Guile). In particular, are there any factors that could convince you to
>> adapt it as a base for your library, or what would you want changed
>> (I assume you won't stop at the README file).
> I’d mainly need practical examples. Tell me how I’d solve a problem
> with (grand scheme) and how it would be solved without it — showing me
> why using it is better.
>
> That means: Documentation.
>
> Also I’d need to know whether some of these methods have performance
> problems: Will this scale as well as the method without (grand scheme)?
>
>>     - functions should be pure
>>     - language constructs that cause side-effects should be encapsulated
>>     with macros that limit the extent of those side-effects
>>     - if possible, use the quasiquote macro, rather than cons, append or list
>>     - prefer the use of match <http://synthcode.com/scheme/match.scm> to cond,
>>     if the patterns are simple
>>     - never use abbreviations
>>     - use commented prepositions to separate arguments
> As guidelines for easy-to-use libraries, that sounds good.
>
>> Also, I recently added infix syntax to Scheme, but (unlike in Racket
>> or David Wheeler's solutions) in the way it is still a prefix
>> syntax. So for example, instead of writing (< a b), one ought to write
>> (is a < b). The rationale is that prefix notation is terrible for
>> non-symmetrical binary predicates, because it obscures the roles of
>> respective arguments (in addition, (is a < b <= c) expands to (and (<
>> a b) (<= b c))). I'm still not sure whether it is a good idea, but I
>> have no reasons to think that it is bad either :] [Curiously enough, I
>> think that prefix notation in arithmetic is just fine: (+ a b c) reads
>> as "sum of a, b and c"]
> This is an old idea, but using (is) as prefix looks
> interesting. Typically the prefix was something like (nfx), which
> doesn’t say anything semantic (other than that what follows uses infix
> syntax). It’s like an expanded equal?
>
> (if (is a = 5)
>      #t
>      #f)
>
> (cond
>      ((is q = p)
>       q)
>      ((is p > 5)
>       p)
>      (else
>       q))
>
> This seems pretty close to natural language. Can I also use (is A 5)
> instead of (equal? A 5)?
>
> Though (is? a = 5) might be closer to canonical Scheme.
>
>> I like it that you and Amirouche look a lot at the sociocultural
>> perspective of programming
> I’m glad to hear that :)
>
>> Yes, to me the ease of understanding is also crucial. (I'd even go
>> further and say that undestranding is the very point of programming)
> I would not go that far: We’re programming both for the future
> maintainers of the code and for the computer. This restricts us in
> expressing things, because we always have to keep two targets audiences
> in mind (many different systems *and* many different people).
>
> Best wishes,
> Arne

Re: How to make GNU Guile more successful

[Amirouche]

Le 01/03/2017 à 20:21, Amirouche a écrit :
>
>
> Le 14/02/2017 à 22:35, Arne Babenhauserheide a écrit :
>> Panicz Maciej Godek <godek.maciek@gmail.com> writes:
>>> There's surely many ways to approach that issue. The point is that for
>>> some reason, Schemers from various tribes prefer to reinvent the
>>> wheel, rather than use an existing one. (Not that I am any different)
>>> However, I also think that these CPAN-alike solutions are far from
>>> optimal: ideally, programming could be made an experience similar the
>>> game "The Journey" by thatgamecompany (where you travel to your goal
>>> and sometimes encounter other players heading the same direction), and
>>> the repository itself could look more like Wikipedia (but first we'd
>>> need to start perceiveing programming as a compact way of representing
>>> knowledge)
>> That sounds somehow like stackoverflow. We might already be going 
>> there :)
>
> That's a very good idea!

There is trusting issue tho. Do you trust all what is written in 
wikipedia right? You
can not simply execute code from the programming cloud like that. I try 
to review
at least once the code I use. But I trust most of it to be safe.

Re: How to make GNU Guile more successful

[Linas Vepstas]

On Mon, Feb 13, 2017 at 2:28 PM, Panicz Maciej Godek <godek.maciek@gmail.com
> wrote:

>
> someone
> responded critically: "are there out of the box libraries to estimate a
> zero inflated negative
> binomial regression model in guile". Of course, if I knew what a
> zero-inflated
> negative binomial regression model, I could deliver an implementation by
> just explaining
> the notions used in that phrase.


Caution: the message below sounds negative.  Sorry, I use guile daily and
almost exclusively now. So there ...

Lack of decent science libraries for scheme is a major stumbling block, for
me. Simply having sine and cosine is not enough.   I got excited (a decade
ago) when I realized that guile supported GnuMP, and then rapidly deflated
when I realized it only supported integers and rationals in GnuMP .. I work
with arbitrary-precision floats.  Or, I did back then.

Maybe more important is making guile work well with large-RAM setups.
Currently, I do data analysis, every day, in guile, on datasets that take
20GB or 40GB -- my current one is 110GB when loaded in RAM, and guile
starts getting buggy, crashy and slow when working at that size.
Sometimes, it starts calling GC half-a-dozen times per second, for no
apparent reason, eating up 6 cores (or more!) doing nothing but GC. Why?
Who knows? Who can tell?

Yes, I have a machine with 256 GB RAM and a few dozen cores, and SSD's that
hold the data, but every time guile crashes, I have to wait an hour for the
data to reload.  I can live with it, but its a dirty secret I would not
share with guile wannabe users.

String handling in guile is a disaster area: If I give it a
10-megabyte-long string in utf8, it promptly tries to convert all of that
string in utf32, for utterly pointless reasons. This just makes it slow.

There are still bugs between GC and the compiler: if call (eval "(some
stuff) (other stuff)")  the compiler will try to compile that string (after
it was converted ti utf32!) and if GC happens to run at just that moment,
guile crashes or hangs.  These bugs need to be fixed.

So although its a good start, there's a lot of work left until it can get
to "the next level". And that work can't happen until guile is more
popular. So it's very much chicken-and-egg scenario.

--linas

Re: How to make GNU Guile more successful

[Marko Rauhamaa]

Linas Vepstas <linasvepstas@gmail.com>:

> String handling in guile is a disaster area: If I give it a
> 10-megabyte-long string in utf8, it promptly tries to convert all of
> that string in utf32, for utterly pointless reasons. This just makes
> it slow.

It's not only the conversion. Strings should be bytes. Python3's example
shouldn't have been followed.


Marko

Re: How to make GNU Guile more successful

[Jan Wedekind]

One could implement something like Theano+NumPy with GNU Guile. I am trying to do something like that (github.com/wedesoft/aiscm) but I am doing it in my spare time only. Theoretically GNU Guile is better suited for this than Python because of macros.

On July 14, 2017 10:54:45 PM GMT+01:00, Linas Vepstas <linasvepstas@gmail.com> wrote:
>On Mon, Feb 13, 2017 at 2:28 PM, Panicz Maciej Godek
><godek.maciek@gmail.com
>> wrote:
>
>>
>> someone
>> responded critically: "are there out of the box libraries to estimate
>a
>> zero inflated negative
>> binomial regression model in guile". Of course, if I knew what a
>> zero-inflated
>> negative binomial regression model, I could deliver an implementation
>by
>> just explaining
>> the notions used in that phrase.
>
>
>Caution: the message below sounds negative.  Sorry, I use guile daily
>and
>almost exclusively now. So there ...
>
>Lack of decent science libraries for scheme is a major stumbling block,
>for
>me. Simply having sine and cosine is not enough.   I got excited (a
>decade
>ago) when I realized that guile supported GnuMP, and then rapidly
>deflated
>when I realized it only supported integers and rationals in GnuMP .. I
>work
>with arbitrary-precision floats.  Or, I did back then.
>
>Maybe more important is making guile work well with large-RAM setups.
>Currently, I do data analysis, every day, in guile, on datasets that
>take
>20GB or 40GB -- my current one is 110GB when loaded in RAM, and guile
>starts getting buggy, crashy and slow when working at that size.
>Sometimes, it starts calling GC half-a-dozen times per second, for no
>apparent reason, eating up 6 cores (or more!) doing nothing but GC.
>Why?
>Who knows? Who can tell?
>
>Yes, I have a machine with 256 GB RAM and a few dozen cores, and SSD's
>that
>hold the data, but every time guile crashes, I have to wait an hour for
>the
>data to reload.  I can live with it, but its a dirty secret I would not
>share with guile wannabe users.
>
>String handling in guile is a disaster area: If I give it a
>10-megabyte-long string in utf8, it promptly tries to convert all of
>that
>string in utf32, for utterly pointless reasons. This just makes it
>slow.
>
>There are still bugs between GC and the compiler: if call (eval "(some
>stuff) (other stuff)")  the compiler will try to compile that string
>(after
>it was converted ti utf32!) and if GC happens to run at just that
>moment,
>guile crashes or hangs.  These bugs need to be fixed.
>
>So although its a good start, there's a lot of work left until it can
>get
>to "the next level". And that work can't happen until guile is more
>popular. So it's very much chicken-and-egg scenario.
>
>--linas

-- 
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

Re: How to make GNU Guile more successful

[Nala Ginrut]

@Linas Thanks for sharing your experiences on large data processing
using Guile! I'm developing a framework to do the similar work for our
product too. But what I'm planning is to do map/reduce for large data
in a distributed system. Anyway, I don't think it's a good way to
analysis massive data in just one very-strong machine. It requires too
much for the language compiler.

I'm still very confident to use Guile on machine learning work. Please
don't forget that no any dynamic language do the heavy computation
directly, they wrapped BLAS or MKL, just like what numpy does. And I
have chance to get a very strong Nvidia GPU cluster server for the
work on the next stage. I don't think the computing speed will be the
issue for me. We're using Python and C++14 at present, but I don't
think Python is the idea glue language, no one wants to use Cython,
trust me. The tail-recursive and delimited-continuation for
lightweight task scheduling is preferred too. The key issue for me is
the expressiveness, which could let us type less code to do more work.
Actually, we use Guile to generate many C++ code in compiling time to
save lot work coding work. This does matter, especially when you have
few people but much work to finish in a tight deadline.

There should be better paradigm to use Guile to glue C/C++ code.
Implement algorithm in Guile directly is not the good way. And wrap
C++ interface with FFI simply is not good too. Sometimes we need to do
more abstract work to get both advantages of Guile and C/C++.
This is about how we use Guile, not the compiler issue.

Of course, I confess there're many problems in our current Guile
compiler. The problem is that we have to use Guile a lot to get to
know what is the problem exactly, and give compelling reasons for the
improvement.

@Jan Yes, that should be a way to go. And I have a new idea which is
just an idea at present. Many we could find a way to read PyObject to
Gulie, and call Python module directly (say, numpy). There should be a
type-compatible abstract level between Guile and PyObject. If it
works, we may implement Python3 on Guile. Although it seems a large
work to implement complete Python3 frontend, we may save lot of work
to write alternative Python modules for Guile.
Julia language does in this idea, but it's backend is compatible with
Python. My idea is not to convert all Python types to Guile, just wrap
some types to a special object like <pyobject> is enough, then Guile
could be glue language for Python too.

Maybe people ask, why bother to glue Python? Python rocks!
Rocks? I need complete lambda calculus, proper tail call, static
scoping, and delimited-continuations (not the generators).
In our product code, we use lot of lambdas in C++ code, so I expect
the programming mind could be consistent between glue language and
main language. But Python can't provide multi-lines lambda expression
only because it uses whitespaces for delimiters, there's no any chance
to delimit multi expression within a lambda unless introducing more
complexity to the parser.

Back to the topic, Guile lacks sufficient practical usage in industry
to reveal its disadvantages. But its advantages are apparent enough
for me so that it worth for me to go further with it.


Best regards.




On Sat, Jul 15, 2017 at 6:10 PM, Jan Wedekind <jan@wedesoft.de> wrote:
> One could implement something like Theano+NumPy with GNU Guile. I am trying to do something like that (github.com/wedesoft/aiscm) but I am doing it in my spare time only. Theoretically GNU Guile is better suited for this than Python because of macros.
>
> On July 14, 2017 10:54:45 PM GMT+01:00, Linas Vepstas <linasvepstas@gmail.com> wrote:
>>On Mon, Feb 13, 2017 at 2:28 PM, Panicz Maciej Godek
>><godek.maciek@gmail.com
>>> wrote:
>>
>>>
>>> someone
>>> responded critically: "are there out of the box libraries to estimate
>>a
>>> zero inflated negative
>>> binomial regression model in guile". Of course, if I knew what a
>>> zero-inflated
>>> negative binomial regression model, I could deliver an implementation
>>by
>>> just explaining
>>> the notions used in that phrase.
>>
>>
>>Caution: the message below sounds negative.  Sorry, I use guile daily
>>and
>>almost exclusively now. So there ...
>>
>>Lack of decent science libraries for scheme is a major stumbling block,
>>for
>>me. Simply having sine and cosine is not enough.   I got excited (a
>>decade
>>ago) when I realized that guile supported GnuMP, and then rapidly
>>deflated
>>when I realized it only supported integers and rationals in GnuMP .. I
>>work
>>with arbitrary-precision floats.  Or, I did back then.
>>
>>Maybe more important is making guile work well with large-RAM setups.
>>Currently, I do data analysis, every day, in guile, on datasets that
>>take
>>20GB or 40GB -- my current one is 110GB when loaded in RAM, and guile
>>starts getting buggy, crashy and slow when working at that size.
>>Sometimes, it starts calling GC half-a-dozen times per second, for no
>>apparent reason, eating up 6 cores (or more!) doing nothing but GC.
>>Why?
>>Who knows? Who can tell?
>>
>>Yes, I have a machine with 256 GB RAM and a few dozen cores, and SSD's
>>that
>>hold the data, but every time guile crashes, I have to wait an hour for
>>the
>>data to reload.  I can live with it, but its a dirty secret I would not
>>share with guile wannabe users.
>>
>>String handling in guile is a disaster area: If I give it a
>>10-megabyte-long string in utf8, it promptly tries to convert all of
>>that
>>string in utf32, for utterly pointless reasons. This just makes it
>>slow.
>>
>>There are still bugs between GC and the compiler: if call (eval "(some
>>stuff) (other stuff)")  the compiler will try to compile that string
>>(after
>>it was converted ti utf32!) and if GC happens to run at just that
>>moment,
>>guile crashes or hangs.  These bugs need to be fixed.
>>
>>So although its a good start, there's a lot of work left until it can
>>get
>>to "the next level". And that work can't happen until guile is more
>>popular. So it's very much chicken-and-egg scenario.
>>
>>--linas
>
> --
> Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

Re: How to make GNU Guile more successful

[Nala Ginrut]

s/Many we could find a way/Maybe we could find a way
Sorry

On Sat, Jul 15, 2017 at 8:55 PM, Nala Ginrut <nalaginrut@gmail.com> wrote:
> @Linas Thanks for sharing your experiences on large data processing
> using Guile! I'm developing a framework to do the similar work for our
> product too. But what I'm planning is to do map/reduce for large data
> in a distributed system. Anyway, I don't think it's a good way to
> analysis massive data in just one very-strong machine. It requires too
> much for the language compiler.
>
> I'm still very confident to use Guile on machine learning work. Please
> don't forget that no any dynamic language do the heavy computation
> directly, they wrapped BLAS or MKL, just like what numpy does. And I
> have chance to get a very strong Nvidia GPU cluster server for the
> work on the next stage. I don't think the computing speed will be the
> issue for me. We're using Python and C++14 at present, but I don't
> think Python is the idea glue language, no one wants to use Cython,
> trust me. The tail-recursive and delimited-continuation for
> lightweight task scheduling is preferred too. The key issue for me is
> the expressiveness, which could let us type less code to do more work.
> Actually, we use Guile to generate many C++ code in compiling time to
> save lot work coding work. This does matter, especially when you have
> few people but much work to finish in a tight deadline.
>
> There should be better paradigm to use Guile to glue C/C++ code.
> Implement algorithm in Guile directly is not the good way. And wrap
> C++ interface with FFI simply is not good too. Sometimes we need to do
> more abstract work to get both advantages of Guile and C/C++.
> This is about how we use Guile, not the compiler issue.
>
> Of course, I confess there're many problems in our current Guile
> compiler. The problem is that we have to use Guile a lot to get to
> know what is the problem exactly, and give compelling reasons for the
> improvement.
>
> @Jan Yes, that should be a way to go. And I have a new idea which is
> just an idea at present. Many we could find a way to read PyObject to
> Gulie, and call Python module directly (say, numpy). There should be a
> type-compatible abstract level between Guile and PyObject. If it
> works, we may implement Python3 on Guile. Although it seems a large
> work to implement complete Python3 frontend, we may save lot of work
> to write alternative Python modules for Guile.
> Julia language does in this idea, but it's backend is compatible with
> Python. My idea is not to convert all Python types to Guile, just wrap
> some types to a special object like <pyobject> is enough, then Guile
> could be glue language for Python too.
>
> Maybe people ask, why bother to glue Python? Python rocks!
> Rocks? I need complete lambda calculus, proper tail call, static
> scoping, and delimited-continuations (not the generators).
> In our product code, we use lot of lambdas in C++ code, so I expect
> the programming mind could be consistent between glue language and
> main language. But Python can't provide multi-lines lambda expression
> only because it uses whitespaces for delimiters, there's no any chance
> to delimit multi expression within a lambda unless introducing more
> complexity to the parser.
>
> Back to the topic, Guile lacks sufficient practical usage in industry
> to reveal its disadvantages. But its advantages are apparent enough
> for me so that it worth for me to go further with it.
>
>
> Best regards.
>
>
>
>
> On Sat, Jul 15, 2017 at 6:10 PM, Jan Wedekind <jan@wedesoft.de> wrote:
>> One could implement something like Theano+NumPy with GNU Guile. I am trying to do something like that (github.com/wedesoft/aiscm) but I am doing it in my spare time only. Theoretically GNU Guile is better suited for this than Python because of macros.
>>
>> On July 14, 2017 10:54:45 PM GMT+01:00, Linas Vepstas <linasvepstas@gmail.com> wrote:
>>>On Mon, Feb 13, 2017 at 2:28 PM, Panicz Maciej Godek
>>><godek.maciek@gmail.com
>>>> wrote:
>>>
>>>>
>>>> someone
>>>> responded critically: "are there out of the box libraries to estimate
>>>a
>>>> zero inflated negative
>>>> binomial regression model in guile". Of course, if I knew what a
>>>> zero-inflated
>>>> negative binomial regression model, I could deliver an implementation
>>>by
>>>> just explaining
>>>> the notions used in that phrase.
>>>
>>>
>>>Caution: the message below sounds negative.  Sorry, I use guile daily
>>>and
>>>almost exclusively now. So there ...
>>>
>>>Lack of decent science libraries for scheme is a major stumbling block,
>>>for
>>>me. Simply having sine and cosine is not enough.   I got excited (a
>>>decade
>>>ago) when I realized that guile supported GnuMP, and then rapidly
>>>deflated
>>>when I realized it only supported integers and rationals in GnuMP .. I
>>>work
>>>with arbitrary-precision floats.  Or, I did back then.
>>>
>>>Maybe more important is making guile work well with large-RAM setups.
>>>Currently, I do data analysis, every day, in guile, on datasets that
>>>take
>>>20GB or 40GB -- my current one is 110GB when loaded in RAM, and guile
>>>starts getting buggy, crashy and slow when working at that size.
>>>Sometimes, it starts calling GC half-a-dozen times per second, for no
>>>apparent reason, eating up 6 cores (or more!) doing nothing but GC.
>>>Why?
>>>Who knows? Who can tell?
>>>
>>>Yes, I have a machine with 256 GB RAM and a few dozen cores, and SSD's
>>>that
>>>hold the data, but every time guile crashes, I have to wait an hour for
>>>the
>>>data to reload.  I can live with it, but its a dirty secret I would not
>>>share with guile wannabe users.
>>>
>>>String handling in guile is a disaster area: If I give it a
>>>10-megabyte-long string in utf8, it promptly tries to convert all of
>>>that
>>>string in utf32, for utterly pointless reasons. This just makes it
>>>slow.
>>>
>>>There are still bugs between GC and the compiler: if call (eval "(some
>>>stuff) (other stuff)")  the compiler will try to compile that string
>>>(after
>>>it was converted ti utf32!) and if GC happens to run at just that
>>>moment,
>>>guile crashes or hangs.  These bugs need to be fixed.
>>>
>>>So although its a good start, there's a lot of work left until it can
>>>get
>>>to "the next level". And that work can't happen until guile is more
>>>popular. So it's very much chicken-and-egg scenario.
>>>
>>>--linas
>>
>> --
>> Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

Re: How to make GNU Guile more successful

[Jan Wedekind]

On Sat, 15 Jul 2017, Nala Ginrut wrote:

> @Jan Yes, that should be a way to go. And I have a new idea which is
> just an idea at present. Many we could find a way to read PyObject to
> Gulie, and call Python module directly (say, numpy). There should be a
> type-compatible abstract level between Guile and PyObject. If it
> works, we may implement Python3 on Guile. Although it seems a large
> work to implement complete Python3 frontend, we may save lot of work
> to write alternative Python modules for Guile.
> Julia language does in this idea, but it's backend is compatible with
> Python. My idea is not to convert all Python types to Guile, just wrap
> some types to a special object like <pyobject> is enough, then Guile

Sure, a Guile Python bridge would be nice for using NumPy and SciPy. 
However while NumPy is quite mature, it cannot do compose array 
operations and avoid intermediate results as Theano can.
One could write bindings to Theano. However I think that the bridging code 
would get in the way at some point.

From the Theano documentation [1]:

>>> import numpy
>>> import theano.tensor as T
>>> from theano import function
>>> x = T.dscalar('x')
>>> y = T.dscalar('y')
>>> f = function([x, y], x + y)
>>> f(2, 3)

This is about having "2 + 3" being computed with fast compiled code.
With Scheme macros most of that can be implemented transparently [2]:

>>> (use-modules (aiscm tensor) (aiscm int))
>>> (tensor (+ 2 3))

[1] http://www.deeplearning.net/software/theano/tutorial/adding.html#adding-two-matrices
[2] http://wedesoft.github.io/aiscm/operation.html#tensor-operations

Re: How to make GNU Guile more successful

[Freja Nordsiek]

If I was to hazard a reason for why Guile gets very slow when loading
20 GB or more (may or may not be related to it being buggy and
crashy), my guesses would be a lot of the data when loaded into Guile
was allocated such that the GC scans it for pointers (using
scm_gc_malloc instead of scm_gc_malloc_pointerless) which would vastly
increase the amount of memory the GC needs to scan every time it runs.

Depending on the data types and what is in them, it may be needless
for the GC to run through the bulk of the data looking for pointers
and this might be a fixable problem. For example, it generally isn't
necessary to scan inside strings for pointers so if that is being
done, there is something in Guile to fix.

If there are really pointers in it (say it is a lot of and/or big
lists, vectors, hash tables, etc.) then the GC really does need to
scan them, which suggests a different kind of data structure would
work around the problem. This is not always doable, and even if doable
could take a lot of programmer time. It seems that Go programmers have
run into this with very large maps already (see
https://github.com/golang/go/issues/9477 and
https://groups.google.com/forum/#!topic/golang-nuts/pHYverdFcLc ).

No idea how this relates to being buggy or crashy.


Freja Nordsiek

On Fri, Jul 14, 2017 at 11:54 PM, Linas Vepstas <linasvepstas@gmail.com> wrote:
> On Mon, Feb 13, 2017 at 2:28 PM, Panicz Maciej Godek <godek.maciek@gmail.com
>> wrote:
>
>>
>> someone
>> responded critically: "are there out of the box libraries to estimate a
>> zero inflated negative
>> binomial regression model in guile". Of course, if I knew what a
>> zero-inflated
>> negative binomial regression model, I could deliver an implementation by
>> just explaining
>> the notions used in that phrase.
>
>
> Caution: the message below sounds negative.  Sorry, I use guile daily and
> almost exclusively now. So there ...
>
> Lack of decent science libraries for scheme is a major stumbling block, for
> me. Simply having sine and cosine is not enough.   I got excited (a decade
> ago) when I realized that guile supported GnuMP, and then rapidly deflated
> when I realized it only supported integers and rationals in GnuMP .. I work
> with arbitrary-precision floats.  Or, I did back then.
>
> Maybe more important is making guile work well with large-RAM setups.
> Currently, I do data analysis, every day, in guile, on datasets that take
> 20GB or 40GB -- my current one is 110GB when loaded in RAM, and guile
> starts getting buggy, crashy and slow when working at that size.
> Sometimes, it starts calling GC half-a-dozen times per second, for no
> apparent reason, eating up 6 cores (or more!) doing nothing but GC. Why?
> Who knows? Who can tell?
>
> Yes, I have a machine with 256 GB RAM and a few dozen cores, and SSD's that
> hold the data, but every time guile crashes, I have to wait an hour for the
> data to reload.  I can live with it, but its a dirty secret I would not
> share with guile wannabe users.
>
> String handling in guile is a disaster area: If I give it a
> 10-megabyte-long string in utf8, it promptly tries to convert all of that
> string in utf32, for utterly pointless reasons. This just makes it slow.
>
> There are still bugs between GC and the compiler: if call (eval "(some
> stuff) (other stuff)")  the compiler will try to compile that string (after
> it was converted ti utf32!) and if GC happens to run at just that moment,
> guile crashes or hangs.  These bugs need to be fixed.
>
> So although its a good start, there's a lot of work left until it can get
> to "the next level". And that work can't happen until guile is more
> popular. So it's very much chicken-and-egg scenario.
>
> --linas

Re: How to make GNU Guile more successful

[Marko Rauhamaa]

Freja Nordsiek <fnordsie@gmail.com>:

> If I was to hazard a reason for why Guile gets very slow when loading
> 20 GB or more (may or may not be related to it being buggy and
> crashy), my guesses would be a lot of the data when loaded into Guile
> was allocated such that the GC scans it for pointers (using
> scm_gc_malloc instead of scm_gc_malloc_pointerless) which would vastly
> increase the amount of memory the GC needs to scan every time it runs.

Good point!

If you didn't to any C programming, what kind of native Guile data
structures are good for such large random-access storage? At least
arrays haven't specifically been documented for such GC optimization:
<URL: https://www.gnu.org/software/guile/manual/html_node/Arrays.htm
l#Arrays>.

Maybe bytevectors would do: <URL: https://www.gnu.org/software/guile/m
anual/html_node/Bytevectors.html#Bytevectors>.

Of course constantly encoding to and decoding from a bytevector using
scheme code might be very slow without the help of some binary bulk
formatting facilities for the data records.


Marko

Re: How to make GNU Guile more successful

[Nala Ginrut]

I've taken a look at Python bridge for other languages, there're several:
For Lua
https://labix.org/lunatic-python

For ObjC
https://pythonhosted.org/pyobjc/

For JS
https://github.com/ipython/ipython/wiki/IPEP-26:-Full-Featured-python-js-object-bridge

I'm glad that the idea is not just my imagination, it exists and
there're something to read.
I'll start this work after release Artanis-0.2.2.

Best regards.


On Sun, Jul 16, 2017 at 6:17 AM, Jan Wedekind <jan@wedesoft.de> wrote:
> On Sat, 15 Jul 2017, Nala Ginrut wrote:
>
>> @Jan Yes, that should be a way to go. And I have a new idea which is
>> just an idea at present. Many we could find a way to read PyObject to
>> Gulie, and call Python module directly (say, numpy). There should be a
>> type-compatible abstract level between Guile and PyObject. If it
>> works, we may implement Python3 on Guile. Although it seems a large
>> work to implement complete Python3 frontend, we may save lot of work
>> to write alternative Python modules for Guile.
>> Julia language does in this idea, but it's backend is compatible with
>> Python. My idea is not to convert all Python types to Guile, just wrap
>> some types to a special object like <pyobject> is enough, then Guile
>
>
> Sure, a Guile Python bridge would be nice for using NumPy and SciPy. However
> while NumPy is quite mature, it cannot do compose array operations and avoid
> intermediate results as Theano can.
> One could write bindings to Theano. However I think that the bridging code
> would get in the way at some point.
>
> From the Theano documentation [1]:
>
>>>> import numpy
>>>> import theano.tensor as T
>>>> from theano import function
>>>> x = T.dscalar('x')
>>>> y = T.dscalar('y')
>>>> f = function([x, y], x + y)
>>>> f(2, 3)
>
>
> This is about having "2 + 3" being computed with fast compiled code.
> With Scheme macros most of that can be implemented transparently [2]:
>
>>>> (use-modules (aiscm tensor) (aiscm int))
>>>> (tensor (+ 2 3))
>
>
> [1]
> http://www.deeplearning.net/software/theano/tutorial/adding.html#adding-two-matrices
> [2] http://wedesoft.github.io/aiscm/operation.html#tensor-operations

Re: How to make GNU Guile more successful

[Freja Nordsiek]

I checked the implementation of bytecectors and SRFI-4 in Guile and they are definitely not scanned for pointers. But I would say hacking them is not a good general solution for this problem. They are good and natural data structures for large arrays of numerical data that are standard signed/unsigned integers of various fixed sizes and IEEE floating point numbers, or structures/unions of these types. Using them for things other than that or strings/byte-arrays could be error prone, messy, and performance poor.

Freja Nordsiek

On July 16, 2017 11:18:18 AM GMT+02:00, Marko Rauhamaa <marko@pacujo.net> wrote:
>Freja Nordsiek <fnordsie@gmail.com>:
>
>> If I was to hazard a reason for why Guile gets very slow when loading
>> 20 GB or more (may or may not be related to it being buggy and
>> crashy), my guesses would be a lot of the data when loaded into Guile
>> was allocated such that the GC scans it for pointers (using
>> scm_gc_malloc instead of scm_gc_malloc_pointerless) which would
>vastly
>> increase the amount of memory the GC needs to scan every time it
>runs.
>
>Good point!
>
>If you didn't to any C programming, what kind of native Guile data
>structures are good for such large random-access storage? At least
>arrays haven't specifically been documented for such GC optimization:
><URL: https://www.gnu.org/software/guile/manual/html_node/Arrays.htm
>l#Arrays>.
>
>Maybe bytevectors would do: <URL: https://www.gnu.org/software/guile/m
>anual/html_node/Bytevectors.html#Bytevectors>.
>
>Of course constantly encoding to and decoding from a bytevector using
>scheme code might be very slow without the help of some binary bulk
>formatting facilities for the data records.
>
>
>Marko

Re: How to make GNU Guile more successful

[Marko Rauhamaa]

Freja Nordsiek <fnordsie@gmail.com>:

> I checked the implementation of bytecectors and SRFI-4 in Guile and
> they are definitely not scanned for pointers. But I would say hacking
> them is not a good general solution for this problem. They are good
> and natural data structures for large arrays of numerical data that
> are standard signed/unsigned integers of various fixed sizes and IEEE
> floating point numbers, or structures/unions of these types. Using
> them for things other than that or strings/byte-arrays could be error
> prone, messy, and performance poor.

So no Scheme solution recommended then?


Marko

Re: How to make GNU Guile more successful

[Freja Nordsiek]

Not necessarily. Using bytevectors and SRFI-4 vectors works very well for some kinds of data. Just, they aren't the best for other kinds (in principle they can work with all data if one is one implements a heap or other custom memory management inside one). They can be made to work but are hard for some types of data, and if you are working with pointers inside them very easy to get a segfault (even python numpy, which has a lot of work done on it over many years, still has a fairly easy way to segfault it if one uses object dtype).


Freja Nordsiek

On July 16, 2017 12:31:37 PM GMT+02:00, Marko Rauhamaa <marko@pacujo.net> wrote:
>Freja Nordsiek <fnordsie@gmail.com>:
>
>> I checked the implementation of bytecectors and SRFI-4 in Guile and
>> they are definitely not scanned for pointers. But I would say hacking
>> them is not a good general solution for this problem. They are good
>> and natural data structures for large arrays of numerical data that
>> are standard signed/unsigned integers of various fixed sizes and IEEE
>> floating point numbers, or structures/unions of these types. Using
>> them for things other than that or strings/byte-arrays could be error
>> prone, messy, and performance poor.
>
>So no Scheme solution recommended then?
>
>
>Marko

Re: How to make GNU Guile more successful

[Freja Nordsiek]

Oh, sorry, just realized I skipped the main question.

There are various data type substitutions that can help or hurt. Replacing lists with vectors or vice versa can significantly help or hurt performance depending on the use case. Same for charsets vs strings. In situations where the data is preserved from collection by other means, weak references to that data might reduce the GC load since there is just less data to go through (not sure how the GC implements weak references so could be very worng about that).


Freja Nordsiek

On July 16, 2017 12:39:20 PM GMT+02:00, Freja Nordsiek <fnordsie@gmail.com> wrote:
>Not necessarily. Using bytevectors and SRFI-4 vectors works very well
>for some kinds of data. Just, they aren't the best for other kinds (in
>principle they can work with all data if one is one implements a heap
>or other custom memory management inside one). They can be made to work
>but are hard for some types of data, and if you are working with
>pointers inside them very easy to get a segfault (even python numpy,
>which has a lot of work done on it over many years, still has a fairly
>easy way to segfault it if one uses object dtype).
>
>
>Freja Nordsiek
>
>On July 16, 2017 12:31:37 PM GMT+02:00, Marko Rauhamaa
><marko@pacujo.net> wrote:
>>Freja Nordsiek <fnordsie@gmail.com>:
>>
>>> I checked the implementation of bytecectors and SRFI-4 in Guile and
>>> they are definitely not scanned for pointers. But I would say
>hacking
>>> them is not a good general solution for this problem. They are good
>>> and natural data structures for large arrays of numerical data that
>>> are standard signed/unsigned integers of various fixed sizes and
>IEEE
>>> floating point numbers, or structures/unions of these types. Using
>>> them for things other than that or strings/byte-arrays could be
>error
>>> prone, messy, and performance poor.
>>
>>So no Scheme solution recommended then?
>>
>>
>>Marko

Re: How to make GNU Guile more successful

[Arun Isaac]

Jan Wedekind writes:

> I am trying to do something like that (github.com/wedesoft/aiscm) but
> I am doing it in my spare time only.

Just hearing about aiscm. I was looking for something like this. Thank
you!

Re: How to make GNU Guile more successful

[Ernest Adrogué]

2017-07-15, 11:10 (+0100); Jan Wedekind escriu:
> One could implement something like Theano+NumPy with GNU Guile. I am
> trying to do something like that (github.com/wedesoft/aiscm) but I am
> doing it in my spare time only. Theoretically GNU Guile is better
> suited for this than Python because of macros.

This is interesting as well, a Lisp interface to R.

https://github.com/dirkschumacher/llr

Guile bugs

[Ludovic Courtès]

Hi Linas,

Linas Vepstas <linasvepstas@gmail.com> skribis:

> Lack of decent science libraries for scheme is a major stumbling block, for
> me. Simply having sine and cosine is not enough.   I got excited (a decade
> ago) when I realized that guile supported GnuMP, and then rapidly deflated
> when I realized it only supported integers and rationals in GnuMP .. I work
> with arbitrary-precision floats.  Or, I did back then.

I think Someone should write MPFR bindings…

> Maybe more important is making guile work well with large-RAM setups.
> Currently, I do data analysis, every day, in guile, on datasets that take
> 20GB or 40GB -- my current one is 110GB when loaded in RAM, and guile
> starts getting buggy, crashy and slow when working at that size.
> Sometimes, it starts calling GC half-a-dozen times per second, for no
> apparent reason, eating up 6 cores (or more!) doing nothing but GC. Why?
> Who knows? Who can tell?
>
> Yes, I have a machine with 256 GB RAM and a few dozen cores, and SSD's that
> hold the data, but every time guile crashes, I have to wait an hour for the
> data to reload.  I can live with it, but its a dirty secret I would not
> share with guile wannabe users.

I think these are genuine serious bugs that should be reported.  Of
course the user-to-developer ratio for Guile is pretty bad (but maybe
not worse than that of CPython after all!), so it will help a lot if you
(1) provide a reduced test case and anyone can use to reproduce the
problem, and (2) do some investigation of your own.

At any rate, if you don’t report it bugs that make your life difficult,
your life is definitely not going to be easier.  ;-)

> String handling in guile is a disaster area: If I give it a
> 10-megabyte-long string in utf8, it promptly tries to convert all of that
> string in utf32, for utterly pointless reasons. This just makes it slow.

We’ve discussed it before, and while I agree that there’s much room for
improvement, I’m very skeptical about this use case (I’d use ‘mmap’ and
get a bytevector.)

> There are still bugs between GC and the compiler: if call (eval "(some
> stuff) (other stuff)")  the compiler will try to compile that string (after
> it was converted ti utf32!) and if GC happens to run at just that moment,
> guile crashes or hangs.  These bugs need to be fixed.

Please report!

> So although its a good start, there's a lot of work left until it can get
> to "the next level". And that work can't happen until guile is more
> popular. So it's very much chicken-and-egg scenario.

Definitely.  I think one has to get ready to get their hands dirty when
pushing Guile to its limits.  That’s probably true of any piece of
software, but that’s even more acute for software with a smaller user
base.

My 2¢!

Ludo’.

Re: Guile bugs

[Marko Rauhamaa]

ludo@gnu.org (Ludovic Courtès):

> I’m very skeptical about this use case (I’d use ‘mmap’ and get a
> bytevector.)

Please elaborate.

 1. Does Guile offer mmap to Scheme code?

 2. What would prevent Guile's GC from scanning the mmapped area for
    pointers?

 3. How do I efficiently encode information in a bytevector in Scheme
    code?


Marko

Re: Guile bugs

[Taylan Ulrich Bayırlı/Kammer]

Marko Rauhamaa <marko@pacujo.net> writes:

> ludo@gnu.org (Ludovic Courtès):
>
>> I’m very skeptical about this use case (I’d use ‘mmap’ and get a
>> bytevector.)
>
> Please elaborate.
>
>  1. Does Guile offer mmap to Scheme code?

It's pretty easy to call C functions from Scheme.  Here's a trivial
example.

$ cat map_file.c
#include <stddef.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/mman.h>

void *map_file(char *file) {
  int fd = open(file, O_RDONLY);
  off_t size = lseek(fd, 0, SEEK_END);
  return mmap(NULL, size, PROT_READ, MAP_PRIVATE, fd, 0);
}
$ gcc -shared -o map_file.so -fPIC map_file.c
$ guile
)> ,use (system foreign)
)> (define map-file
     (pointer->procedure
      '*
      (dynamic-func "map_file"
                    (dynamic-link "./map_file"))
      (list '*)))
)> (define file-ptr (map-file (string->pointer "/home/taylan/todo")))
)> (define file-bv (pointer->bytevector file-ptr 100))  ;see note below
)> file-bv
$3 = #vu8(45 42 45 32 111 114 103 32 45 42 45 10 10 ...)

That's ASCII for "-*- org -*-\n\n", which is what my todo file begins
with. :-)

(It's a file mode marker for Emacs, for those who don't know.)

Note: The second argument to pointer->bytevector defines the size of the
bytevector.  I passed 100 because I was too lazy to get the size of the
file.  You should pass the size of the file in bytes.

>  2. What would prevent Guile's GC from scanning the mmapped area for
>     pointers?

I don't know the details but AFAIK this is no problem with Boehm GC.

>  3. How do I efficiently encode information in a bytevector in Scheme
>     code?

What sort of data?

I have a library called bytestructures that imitates the C type system
within Scheme, to be used on bytevectors that contain data structures
generated by C code, though the library is built upon a generic core
with which other structures can be declared as well.  Not sure if this
helps you:

https://github.com/TaylanUB/scheme-bytestructures/

> Marko

Hope I could help,
Taylan

Re: Guile bugs

[Marko Rauhamaa]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer"):

> Marko Rauhamaa <marko@pacujo.net> writes:
>
>> ludo@gnu.org (Ludovic Courtès):
>>
>>> I’m very skeptical about this use case (I’d use ‘mmap’ and get a
>>> bytevector.)
>>
>> Please elaborate.
>>
>>  1. Does Guile offer mmap to Scheme code?
>
> It's pretty easy to call C functions from Scheme.

Not good enough. I would have to precompile libraries for different
target platforms (or require the target environment to have a C
compiler). Besides, the need is quite generic and shouldn't require an
extension.

>>  2. What would prevent Guile's GC from scanning the mmapped area for
>>     pointers?
>
> I don't know the details but AFAIK this is no problem with Boehm GC.

I don't understand. It can be quite time-consuming to scan gigabytes of
RAM for pointers.

>>  3. How do I efficiently encode information in a bytevector in Scheme
>>     code?
>
> What sort of data?
>
> I have a library called bytestructures that imitates the C type system
> within Scheme, to be used on bytevectors that contain data structures
> generated by C code, though the library is built upon a generic core
> with which other structures can be declared as well.  Not sure if this
> helps you:
>
> https://github.com/TaylanUB/scheme-bytestructures/

That's precisely it. It would be nice to have it as part of standard
Guile.

I wonder, though, if doing that is fast enough in Scheme code.


Marko

Re: Guile bugs

[Ludovic Courtès]

Hi Marko,

Marko Rauhamaa <marko@pacujo.net> skribis:

> taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer"):

[...]

>>>  2. What would prevent Guile's GC from scanning the mmapped area for
>>>     pointers?
>>
>> I don't know the details but AFAIK this is no problem with Boehm GC.
>
> I don't understand. It can be quite time-consuming to scan gigabytes of
> RAM for pointers.

libgc knows which regions it must scan and mmap’d regions like this are
not among them.

For the record, the loader in Guile 2.2 mmaps .go files in memory just
like this (see ‘load-thunk-from-file’.)

>>>  3. How do I efficiently encode information in a bytevector in Scheme
>>>     code?
>>
>> What sort of data?
>>
>> I have a library called bytestructures that imitates the C type system
>> within Scheme, to be used on bytevectors that contain data structures
>> generated by C code, though the library is built upon a generic core
>> with which other structures can be declared as well.  Not sure if this
>> helps you:
>>
>> https://github.com/TaylanUB/scheme-bytestructures/
>
> That's precisely it. It would be nice to have it as part of standard
> Guile.

Even though I agree that something like bytestructures should eventually
land in Guile, I think Taylan just pointed out that this is something
you can use if you’re dealing with structured binary data.

But again, that really depends on the kind of data you’re dealing with.
Maybe sometimes the tools that (system foreign) provides are good
enough, maybe sometimes you’d rather do something different like Guile’s
ELF parser does.

HTH,
Ludo’.

Re: Guile bugs

[Marko Rauhamaa]

ludo@gnu.org (Ludovic Courtès):

> libgc knows which regions it must scan and mmap’d regions like this are
> not among them.

Wow, where is that documented? I would have imagined it scanned all
writable RAM and CPU registers.


Marko

Re: Guile bugs

[Mark H Weaver]

Marko Rauhamaa <marko@pacujo.net> writes:

> ludo@gnu.org (Ludovic Courtès):
>
>> libgc knows which regions it must scan and mmap’d regions like this are
>> not among them.
>
> Wow, where is that documented? I would have imagined it scanned all
> writable RAM and CPU registers.

It's documented here:  http://www.hboehm.info/gc/gcdescr.html

      Mark

Re: Guile bugs

[Marko Rauhamaa]

Mark H Weaver <mhw@netris.org>:

> Marko Rauhamaa <marko@pacujo.net> writes:
>
>> ludo@gnu.org (Ludovic Courtès):
>>
>>> libgc knows which regions it must scan and mmap’d regions like this
>>> are not among them.
>>
>> Wow, where is that documented? I would have imagined it scanned all
>> writable RAM and CPU registers.
>
> It's documented here:  http://www.hboehm.info/gc/gcdescr.html

Please point out the sentence.

The closest I could find is:

   * Static data region(s). In the simplest case, this is the region
     between DATASTART and DATAEND, as defined in gcconfig.h. However,
     in most cases, this will also involve static data regions
     associated with dynamic libraries. These are identified by the
     mostly platform-specific code in dyn_load.c.

But that doesn't clearly state if mmapped regions are in or out. I would
have assumed they were in.


Marko

Re: Guile bugs

[Chris Vine]

On Fri, 21 Jul 2017 09:15:28 +0300
Marko Rauhamaa <marko@pacujo.net> wrote:
> Mark H Weaver <mhw@netris.org>:
> 
> > Marko Rauhamaa <marko@pacujo.net> writes:
> >  
> >> ludo@gnu.org (Ludovic Courtès):
> >>  
> >>> libgc knows which regions it must scan and mmap’d regions like
> >>> this are not among them.  
> >>
> >> Wow, where is that documented? I would have imagined it scanned all
> >> writable RAM and CPU registers.  
> >
> > It's documented here:  http://www.hboehm.info/gc/gcdescr.html  
> 
> Please point out the sentence.
> 
> The closest I could find is:
> 
>    * Static data region(s). In the simplest case, this is the region
>      between DATASTART and DATAEND, as defined in gcconfig.h. However,
>      in most cases, this will also involve static data regions
>      associated with dynamic libraries. These are identified by the
>      mostly platform-specific code in dyn_load.c.
> 
> But that doesn't clearly state if mmapped regions are in or out. I
> would have assumed they were in.

Since the C runtime does not store static variables in mmapped memory, I
would have assumed the opposite.

There is also some information at
https://www.gnu.org/software/guile/manual/html_node/Garbage-Collection.html#Garbage-Collection
That states that the areas that the garbage collector scans for live
objects are the areas where global and other static variables are
placed, local variables (including function arguments and registers),
and heap memory allocated by scm_gc_malloc() (but not memory allocated
by scm_gc_malloc_pointerless() or by malloc() and cognates).

If a SCM object keeps references to sub-objects residing in unscanned
memory, this applies:
https://www.gnu.org/software/guile/manual/html_node/Foreign-Object-Memory-Management.html#Foreign-Object-Memory-Management
(and the equivalent for smobs in guile-2.0).

Chris

Re: Guile bugs

[Marko Rauhamaa]

Chris Vine <vine35792468@gmail.com>:

> On Fri, 21 Jul 2017 09:15:28 +0300
> Marko Rauhamaa <marko@pacujo.net> wrote:
>> The closest I could find is:
>> 
>>    * Static data region(s). In the simplest case, this is the region
>>      between DATASTART and DATAEND, as defined in gcconfig.h. However,
>>      in most cases, this will also involve static data regions
>>      associated with dynamic libraries. These are identified by the
>>      mostly platform-specific code in dyn_load.c.
>> 
>> But that doesn't clearly state if mmapped regions are in or out. I
>> would have assumed they were in.
>
> Since the C runtime does not store static variables in mmapped memory,
> I would have assumed the opposite.

I use libgc in non-Guile code. If mmapped areas are not considered for
GC, I'll have to take special care.

I had assumed libgc simply introspects the process's memory map and
traverses anything reachable. After all, the process might be also using
a secondary memory allocator based on mmap. What if you do place
pointers in a mmapped region?

> There is also some information at https://www.gnu.org/softwa
> re/guile/manual/html_node/Garbage-Collection.html#Garbage-Collection
> That states that the areas that the garbage collector scans for live
> objects are the areas where global and other static variables are
> placed, local variables (including function arguments and registers),
> and heap memory allocated by scm_gc_malloc() (but not memory allocated
> by scm_gc_malloc_pointerless() or by malloc() and cognates).

Which doesn't say anything explicit about mmap.

Anyway, I suspect a bytearray is guaranteed to be pointerless and could
be used for the desired kind of mass storage in RAM. It all hinges on
the efficiency of encoding and decoding objects. As much as possible, I
would like to stay in Scheme-land.


Marko

Re: Guile bugs

[Mark H Weaver]

Marko Rauhamaa <marko@pacujo.net> writes:

> Mark H Weaver <mhw@netris.org>:
>
>> Marko Rauhamaa <marko@pacujo.net> writes:
>>
>>> ludo@gnu.org (Ludovic Courtès):
>>>
>>>> libgc knows which regions it must scan and mmap’d regions like this
>>>> are not among them.
>>>
>>> Wow, where is that documented? I would have imagined it scanned all
>>> writable RAM and CPU registers.
>>
>> It's documented here:  http://www.hboehm.info/gc/gcdescr.html
>
> Please point out the sentence.

Read the "Mark phase" section.  In brief, roots are scanned from the
registers, stack(s), and static data region(s).  The only other areas
scanned are heap areas specifically managed by libgc.  No pointer is
considered valid unless it points to a libgc-managed heap block.

      Mark

Re: Guile bugs

[Marko Rauhamaa]

Mark H Weaver <mhw@netris.org>:

> Marko Rauhamaa <marko@pacujo.net> writes:
>
>> Mark H Weaver <mhw@netris.org>:
>>
>>> Marko Rauhamaa <marko@pacujo.net> writes:
>>>
>>>> ludo@gnu.org (Ludovic Courtès):
>>>>
>>>>> libgc knows which regions it must scan and mmap’d regions like
>>>>> this are not among them.
>>>>
>>>> Wow, where is that documented? I would have imagined it scanned all
>>>> writable RAM and CPU registers.
>>>
>>> It's documented here:  http://www.hboehm.info/gc/gcdescr.html
>>
>> Please point out the sentence.
>
> Read the "Mark phase" section.  In brief, roots are scanned from the
> registers, stack(s), and static data region(s).  The only other areas
> scanned are heap areas specifically managed by libgc.  No pointer is
> considered valid unless it points to a libgc-managed heap block.

If you say so.

Both libgc and Guile would benefit from explicitly mentioning (example)
areas of RAM that aren't considered by GC. This page is quite
dismissive:

   The BDW-GC “just works”, for the most part. [...]

   Thus, the BDW-GC uses a technique called conservative garbage
   collection, to make the local variable list unnecessary. [...]

   Obviously, such a system will occasionally retain objects that are
   actually garbage, and should be freed. In practice, this is not a
   problem. The alternative, an explicitly maintained list of local
   variable addresses, is effectively much less reliable, due to
   programmer error. Interested readers should see the BDW-GC web page
   at <URL: http://www.hboehm.info/gc/>, for more information.

   <URL: https://www.gnu.org/software/guile/manual/html_node/Conservativ
   e-GC.html>


What you're saying points out a secondary problem that applies to Guile
C programming. If you have stored Guile smob references in, say,
third-party library objects, you'll need to carefully protect and
unprotect them.


Marko

Re: Guile bugs

[David Kastrup]

Marko Rauhamaa <marko@pacujo.net> writes:

> What you're saying points out a secondary problem that applies to
> Guile C programming. If you have stored Guile smob references in, say,
> third-party library objects, you'll need to carefully protect and
> unprotect them.

It may be worth mentioning that this includes _all_ C++ STL container
classes (like std::vector).  If you want to employ them in connection
with Guile2's automatic memory scanning, you need to use them with
custom allocators.

-- 
David Kastrup

Re: Guile bugs

[Matt Wette]

> On Jul 20, 2017, at 9:22 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
> 
> ludo@gnu.org (Ludovic Courtès):
> 
>> I’m very skeptical about this use case (I’d use ‘mmap’ and get a
>> bytevector.)
> 
> Please elaborate.
> 
> 1. Does Guile offer mmap to Scheme code?
> 
> 2. What would prevent Guile's GC from scanning the mmapped area for
>    pointers?
> 
> 3. How do I efficiently encode information in a bytevector in Scheme
>    code?
> 
> 
> Marko
> 

I have submitted mmap as a Guile wishlist item.  See https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27782

Re: Guile bugs

[Taylan Ulrich Bayırlı/Kammer]

Marko Rauhamaa <marko@pacujo.net> writes:

>>>  1. Does Guile offer mmap to Scheme code?
>>
>> It's pretty easy to call C functions from Scheme.
>
> Not good enough. I would have to precompile libraries for different
> target platforms (or require the target environment to have a C
> compiler). Besides, the need is quite generic and shouldn't require an
> extension.

One can also call standard C functions through the same Guile interface
that I used in my example, so one can e.g. turn mmap() into a Scheme
procedure.  open() already happens to exist in the form of 'open-fdes',
and to get the file size one can use (stat:size (stat "filepath")).

The only remaining problem is that mmap's flag constants like PROT_READ,
MAP_SHARED, etc. don't exist in Guile, and can't be reached through FFI
since they're #define constants.  Sans that problem, this works:

)> ,use (system foreign)
)> ,use (rnrs bytevectors)
)> (define mmap
     (pointer->procedure
      '*
      (dynamic-func "mmap" (dynamic-link))                   ;[1]
      (list '* size_t int int int size_t)))
)> (define fd (open-fdes "/home/taylan/todo" O_RDONLY))
)> (define size (stat:size (stat fd)))
)> (define file-ptr (mmap (make-pointer 0) size 1 2 fd 0))   ;[2]
)> (define file-bv (pointer->bytevector file-ptr size))
)> (integer->char (bytevector-u8-ref file-bv 0))
$1 = #\-

[1] When dynamic-link is called with no argument, it returns a "global
    symbol handle" that contains the lib bindings of the Guile process.
    A Guile process is sure to have "mmap" linked in.

[2] The numbers 1 and 2 happen to correspond to PROT_READ and
    MAP_PRIVATE on my system.

As a temporary workaround to the problem that mmap flag constants don't
exist in Guile, one could define them manually for the systems one
intends to support in the foreseeable future.

>>>  3. How do I efficiently encode information in a bytevector in Scheme
>>>     code?
>>
>> What sort of data?
>>
>> I have a library called bytestructures that imitates the C type system
>> within Scheme, to be used on bytevectors that contain data structures
>> generated by C code, though the library is built upon a generic core
>> with which other structures can be declared as well.  Not sure if this
>> helps you:
>>
>> https://github.com/TaylanUB/scheme-bytestructures/
>
> That's precisely it. It would be nice to have it as part of standard
> Guile.

Maybe one day.  These days I don't have much time to work on it.  Maybe
I'll see that I provide a GNU-compliant build system first so people can
install it easily.

In case you use Guix, someone made a package for it, so you could
install it through there too.

> I wonder, though, if doing that is fast enough in Scheme code.

Since Guile supports syntax-case, the "macro API" of the library can be
used so the library itself adds absolutely zero runtime overhead.

(This API puts a few limits on the full feature set, but the standard
modules that mimic C types don't use those features.)


Taylan

Re: Guile bugs

[Marko Rauhamaa]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer"):

> Marko Rauhamaa <marko@pacujo.net> writes:
>> I wonder, though, if doing that is fast enough in Scheme code.
>
> Since Guile supports syntax-case, the "macro API" of the library can
> be used so the library itself adds absolutely zero runtime overhead.

No doubt it's as fast as Guile can be for the API. However, you'll need
to have the encoding/decoding machinery ultimately executed in Scheme.

Compare this with Python's struct.(un)pack(), for example, which lets
the "machine code" handle the bulk encoding/decoding of a record.

Instead of binary, one could use S-expressions, of course, but then
you'd need to know the maximum length of the encoding up front.

Heck, maybe your bulk RAM store should be implemented with straight
non-GC malloc...


Marko

Re: Guile bugs

[Linas Vepstas]

Hi Ludo, a very late reply...

On Thu, Jul 20, 2017 at 10:28 AM, Ludovic Courtès <ludo@gnu.org> wrote:

>
> > String handling in guile is a disaster area: If I give it a
> > 10-megabyte-long string in utf8, it promptly tries to convert all of that
> > string in utf32, for utterly pointless reasons. This just makes it slow.
>
> We’ve discussed it before, and while I agree that there’s much room for
> improvement, I’m very skeptical about this use case


It's OK to be skeptical.  Let me briefly sketch a not-uncommon thought
process.

1) There is a need to pass messages between subsystems running on different
machines.

2) Solutions include ROS, ZeroMQ,  google protocol buffers... whatever. All
require lots of work, a learning curve, complexity, etc.

3) Wait!  I know! I will just send around ascii (utf8) strings that are
guile programs!  Just use the guile repl server, connect to it, and send
some guile string!  No muss, no fuss, no coding, no learning curve, simple
easy ...  You can use netcat to drive things!  echo (display "hello
world\n") | nc 1.2.3.4 37146

4) Solutions 3 works great, so lets scale it up!  Send more messages,
larger messages, more quickly.

5) unhappiness.  I reported one here, but it needs follow-up on my part.
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27234

Other issues turn out to be that the REPL server is slow; I replaced it
with a hand-coded blob that is wayyy faster.  On a related note, there is
no login/authentication for the repl server, but this is not a guile issue,
its a generic issue.  Amazingly, no one has ever created a generic login
server, aside from ssh... and there's no easy way to glue ssh to the guile
repl.

--linas

-- 
*"The problem is not that artificial intelligence will get too smart and
take over the world," computer scientist Pedro Domingos writes, "the
problem is that it's too stupid and already has." *

Re: Guile bugs

[Linas Vepstas]

Stale reply to old message, but maybe its still helpful:

On Fri, Jul 21, 2017 at 1:15 AM, Marko Rauhamaa <marko@pacujo.net> wrote:

> Mark H Weaver <mhw@netris.org>:
>
> > Marko Rauhamaa <marko@pacujo.net> writes:
> >
> >> ludo@gnu.org (Ludovic Courtès):
> >>
> >>> libgc knows which regions it must scan and mmap’d regions like this
> >>> are not among them.
> >>
> >> Wow, where is that documented? I would have imagined it scanned all
> >> writable RAM and CPU registers.
> >
> > It's documented here:  http://www.hboehm.info/gc/gcdescr.html
>
> Please point out the sentence.


 https://github.com/ivmai/bdwgc/blob/master/include/gc.h
lines 448-450:

/* General purpose allocation routines, with roughly malloc calling     */
/* conv.  The atomic versions promise that no relevant pointers are     */
/* contained in the object.

So if you used GC_malloc_atomic() in your code, then gc will NOT scan that
region for pointers. guile-2.2 does this correctly for strings (I checked)
because strings will never contain pointers.  I have not checked other uses.

--linas


-- 
*"The problem is not that artificial intelligence will get too smart and
take over the world," computer scientist Pedro Domingos writes, "the
problem is that it's too stupid and already has." *

Re: Guile bugs

[Marko Rauhamaa]

Linas Vepstas <linasvepstas@gmail.com>:
> So if you used GC_malloc_atomic() in your code, then gc will NOT scan
> that region for pointers. guile-2.2 does this correctly for strings (I
> checked) because strings will never contain pointers. I have not
> checked other uses.

The question about mmap(2) is addressed more directly by the README
under <URL: https://github.com/ivmai/bdwgc>:

   Any objects not intended to be collected must be pointed to either
   from other such accessible objects, or from the registers, stack,
   data, or statically allocated bss segments. [...]

   WARNING: pointers inside memory allocated by the standard malloc are
   not seen by the garbage collector. Thus objects pointed to only from
   such a region may be prematurely deallocated. It is thus suggested
   that the standard malloc be used only for memory regions, such as I/O
   buffers, that are guaranteed not to contain pointers to garbage
   collectible memory.

   [...]

   WARNING: the collector does not guarantee to scan thread-local
   storage (e.g. of the kind accessed with pthread_getspecific). The
   collector does scan thread stacks

Now, the question is, can you make your multigigabyte allocation go into
these excluded memory segments? Are you still hit by the pathological GC
behavior you reported?


Marko

Re: Guile bugs

[Linas Vepstas]

Hi Marko,

On Sat, Sep 9, 2017 at 5:31 PM, Marko Rauhamaa <marko@pacujo.net> wrote:

> Linas Vepstas <linasvepstas@gmail.com>:
> > So if you used GC_malloc_atomic() in your code, then gc will NOT scan
> > that region for pointers. guile-2.2 does this correctly for strings (I
> > checked) because strings will never contain pointers. I have not
> > checked other uses.
>
> The question about mmap(2) is addressed more directly by the README
> under <URL: https://github.com/ivmai/bdwgc>:
>
>    Any objects not intended to be collected must be pointed to either
>    from other such accessible objects, or from the registers, stack,
>    data, or statically allocated bss segments. [...]
>
>    WARNING: pointers inside memory allocated by the standard malloc are
>    not seen by the garbage collector. Thus objects pointed to only from
>    such a region may be prematurely deallocated. It is thus suggested
>    that the standard malloc be used only for memory regions, such as I/O
>    buffers, that are guaranteed not to contain pointers to garbage
>    collectible memory.
>
>    [...]
>
>    WARNING: the collector does not guarantee to scan thread-local
>    storage (e.g. of the kind accessed with pthread_getspecific). The
>    collector does scan thread stacks
>
> Now, the question is, can you make your multigigabyte allocation go into
> these excluded memory segments? Are you still hit by the pathological GC
> behavior you reported?
>

What, my app, specifically? Uh yeah. All of my c++ objects go through
plain-old malloc and should be 100% invisible to guile and to bdwgc and
that's perfect.

I do have a layer where guile interacts with the C++ code, and all the
various bits and pieces are correctly declared everywhere.  That code is
about 8-10 years old, its been maintained all that time, its been in use
all that time, and works without issues.

"works without issues" means this: A typical run for me uses some 50-100
threads, maybe 1/2 of these in guile. Due to lock contention, in practice
only 3 to 10 of these threads ever get scheduled, and this is fine, and not
unexpected.  My algos are the opposite of "embrassingly parallel", so low
parallelism is expected. Each processing run goes for a few days to weeks
or over a month, uses 2 to 50GB RAM, does not leak memory, does not crash,
its stable, predictable, does what its supposed to do, terminates cleanly.

It does sometimes run much slower than expected (about 1/2x or 1/4x) and I
don't know why.  I don't usually monitor GC, but when I do, it seems that a
lot of cpu time gets wasted there. Sometimes other people also notice this,
bitch at me, and propose silly solutions. I've been ignoring that as
lower-priority.

Only now it's risen up and is biting hard, and I don't understand how or
why gc gets triggered in guile.  For my app, it seems that it is running
far, far too often, and is severely slowing/preventing forward progress.

--linas

>
>
> Marko
>



-- 
*"The problem is not that artificial intelligence will get too smart and
take over the world," computer scientist Pedro Domingos writes, "the
problem is that it's too stupid and already has." *

Re: Guile bugs

[Ludovic Courtès]

Hi,

Linas Vepstas <linasvepstas@gmail.com> skribis:

> 1) There is a need to pass messages between subsystems running on different
> machines.
>
> 2) Solutions include ROS, ZeroMQ,  google protocol buffers... whatever. All
> require lots of work, a learning curve, complexity, etc.
>
> 3) Wait!  I know! I will just send around ascii (utf8) strings that are
> guile programs!  Just use the guile repl server, connect to it, and send
> some guile string!  No muss, no fuss, no coding, no learning curve, simple
> easy ...  You can use netcat to drive things!  echo (display "hello
> world\n") | nc 1.2.3.4 37146

Then you’re sending sexps, not strings, and you’re reading them with
‘read’ from a port, as opposed to loading whole strings in memory.

Also, the REPL server is not necessarily the right thing here.  You
conceptually want a REPL, but not Guile’s full-blown REPL, which can be
hard to deal with (you need to parse the prompts, determine if you’re in
a recursive REPL, whether an exception occurred, etc.)

In the Shepherd, I implemented a very simple REPL that is just enough to
communicate evaluation results to the client, and to distinguish return
values from exceptions.  This does not use Guile’s REPL server, only
‘read’, ‘eval’, ‘write’.

HTH,
Ludo’.

Re: Guile bugs

[Linas Vepstas]

Hi Ludo,

Perhaps I was too brief in my remarks. By a networked REPL, I meant this:

https://www.gnu.org/software/guile/manual/html_node/REPL-Servers.html

For accidental, historical reasons, (i.e. I did not know the above existed)
I wrote my own variant, which accepts network connections, handles guile in
full, including the printing of exception frames, handles async printing to
stdio (i.e. to the network connection) so that long-running functions can
print, and the user can see that output, unbuffered.   Its full
multi-threaded, uses a pool of threads pre-initialized in guile, hot and
ready to go, to avoid the overhead of initializing and entering guile.  Its
fast, robust, it works.

The stuff coming over the network sockets are bytes, not s-exps. Since none
of the bytes are ever zero, they are effectively C/C++ strings, and are
handled as such. These C strings are sent to  scm_eval_string() wrapped
by scm_c_catch().

As mentioned, this customized blob of code is easily 3x or 5x faster
than (use-modules (system repl server)), and also more  robust ...
sometimes (system repl server) seems to lock up or hang or something, I
have not delved into it.

One of my complaints is that the C strings sent to scm_eval_string() go
through some pointless layers that convert utf8 to utf32, and then back
again. I do not recall the details. There are some obvious optimizations
that can be performed. Its more than an afternoon or two but less than a
week or two.  I am not motivated to submit patches, as I fear these will
just provoke arguments and be rejected.

The custom guile network repl server can be viewed at the URL's below. To
preserve your sanity, you probably don't want to study it.  But in the
spirit of open source, here it is:

https://github.com/opencog/atomspace/blob/master/opencog/guile/SchemeEval.cc

The network REPL shell is here:
https://github.com/opencog/opencog/blob/master/opencog/cogserver/shell/GenericShell.cc

we've layered both guile and python on it.  Python is annoying because
executing python from repl behaves differently than executing python from a
file. Not my fault; its built into the python language definition. Idiots.

--linas





On Sun, Sep 10, 2017 at 8:11 AM, Ludovic Courtès <ludo@gnu.org> wrote:

> Hi,
>
> Linas Vepstas <linasvepstas@gmail.com> skribis:
>
> > 1) There is a need to pass messages between subsystems running on
> different
> > machines.
> >
> > 2) Solutions include ROS, ZeroMQ,  google protocol buffers... whatever.
> All
> > require lots of work, a learning curve, complexity, etc.
> >
> > 3) Wait!  I know! I will just send around ascii (utf8) strings that are
> > guile programs!  Just use the guile repl server, connect to it, and send
> > some guile string!  No muss, no fuss, no coding, no learning curve,
> simple
> > easy ...  You can use netcat to drive things!  echo (display "hello
> > world\n") | nc 1.2.3.4 37146
>
> Then you’re sending sexps, not strings, and you’re reading them with
> ‘read’ from a port, as opposed to loading whole strings in memory.
>
> Also, the REPL server is not necessarily the right thing here.  You
> conceptually want a REPL, but not Guile’s full-blown REPL, which can be
> hard to deal with (you need to parse the prompts, determine if you’re in
> a recursive REPL, whether an exception occurred, etc.)
>
> In the Shepherd, I implemented a very simple REPL that is just enough to
> communicate evaluation results to the client, and to distinguish return
> values from exceptions.  This does not use Guile’s REPL server, only
> ‘read’, ‘eval’, ‘write’.
>
> HTH,
> Ludo’.
>



-- 
*"The problem is not that artificial intelligence will get too smart and
take over the world," computer scientist Pedro Domingos writes, "the
problem is that it's too stupid and already has." *

Re: Guile bugs

[Ludovic Courtès]

Hello,

Linas Vepstas <linasvepstas@gmail.com> skribis:

> The stuff coming over the network sockets are bytes, not s-exps. Since none
> of the bytes are ever zero, they are effectively C/C++ strings, and are
> handled as such. These C strings are sent to  scm_eval_string() wrapped
> by scm_c_catch().

I don’t know to what extent that is applicable to your software, but my
recommendation would be to treat that network socket as a Scheme port,
pass it to ‘read’, and pass the result to ‘eval’ (as opposed to reading
the whole string from C++ and passing it to ‘scm_eval_string’.)

HTH,
Ludo’.

Re: Guile bugs

[Marko Rauhamaa]

ludo@gnu.org (Ludovic Courtès):

> I don’t know to what extent that is applicable to your software, but my
> recommendation would be to treat that network socket as a Scheme port,
> pass it to ‘read’, and pass the result to ‘eval’ (as opposed to reading
> the whole string from C++ and passing it to ‘scm_eval_string’.)

Octet stream sockets break the stream at arbitrary locations. Eval isn't
going to like it if you blindly hand bytes over to it. You first need to
break the stream into complete S-expressions. Note also that the socket
might not even respect UTF-8 boundaries so you can't assume you can read
the bytes as text.

Secondly, this lucrative pattern can be very dangerous as it allows
arbitrary code to be executed at the peer end. In most applications, it
is better to interpret the S-expressions explicitly.


Marko

Re: Guile bugs

[Ludovic Courtès]

Hi Marko,

Marko Rauhamaa <marko@pacujo.net> skribis:

> ludo@gnu.org (Ludovic Courtès):
>
>> I don’t know to what extent that is applicable to your software, but my
>> recommendation would be to treat that network socket as a Scheme port,
>> pass it to ‘read’, and pass the result to ‘eval’ (as opposed to reading
>> the whole string from C++ and passing it to ‘scm_eval_string’.)
>
> Octet stream sockets break the stream at arbitrary locations. Eval isn't
> going to like it if you blindly hand bytes over to it. You first need to
> break the stream into complete S-expressions.

Sure, that’s what the ‘read’ part above does.

> Secondly, this lucrative pattern can be very dangerous as it allows
> arbitrary code to be executed at the peer end. In most applications, it
> is better to interpret the S-expressions explicitly.

Definitely.  I’m not saying eval-over-the-network is a good thing to do
in general, mind you!  :-)

Ludo’.

Re: Guile bugs

[Linas Vepstas]

On Mon, Sep 11, 2017 at 2:26 AM, Ludovic Courtès <ludo@gnu.org> wrote:

> Hello,
>
> Linas Vepstas <linasvepstas@gmail.com> skribis:
>
> > The stuff coming over the network sockets are bytes, not s-exps. Since
> none
> > of the bytes are ever zero, they are effectively C/C++ strings, and are
> > handled as such. These C strings are sent to  scm_eval_string() wrapped
> > by scm_c_catch().
>
> I don’t know to what extent that is applicable to your software, but my
> recommendation would be to treat that network socket as a Scheme port,
> pass it to ‘read’, and pass the result to ‘eval’ (as opposed to reading
> the whole string from C++ and passing it to ‘scm_eval_string’.)
>

Why?  What advantage does this offer?

Its not clear that guile eval is smart enough to manage a network socket --
if the user starts a long-running process with intermittent prints, will it
send that to the socket?  What if the user hits cntrl-C in the middle of it
all? What if the code that came over the socket happened to throw an
exception?

I've had to deal with all of these issues in the past, and have a stable
code base; but if I had to start all over again, its not clear that these
issues have gone away.  I mean, eval was designed to eval -- it was not
designed to support multi-threaded, concurrent network operations, right?

To support my point: the default guile network REPL server is painfully
slow, and frequently crashes/hangs. It works well enough to do some demos
but is not stable enough for production use ... if its just read+eval, that
might explain why its unstable.

--linas

-- 
*"The problem is not that artificial intelligence will get too smart and
take over the world," computer scientist Pedro Domingos writes, "the
problem is that it's too stupid and already has." *

Re: Guile bugs

[Ludovic Courtès]

Linas Vepstas <linasvepstas@gmail.com> skribis:

> On Mon, Sep 11, 2017 at 2:26 AM, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> Hello,
>>
>> Linas Vepstas <linasvepstas@gmail.com> skribis:
>>
>> > The stuff coming over the network sockets are bytes, not s-exps. Since
>> none
>> > of the bytes are ever zero, they are effectively C/C++ strings, and are
>> > handled as such. These C strings are sent to  scm_eval_string() wrapped
>> > by scm_c_catch().
>>
>> I don’t know to what extent that is applicable to your software, but my
>> recommendation would be to treat that network socket as a Scheme port,
>> pass it to ‘read’, and pass the result to ‘eval’ (as opposed to reading
>> the whole string from C++ and passing it to ‘scm_eval_string’.)
>>
>
> Why?  What advantage does this offer?

It avoids copies and conversions, which is big deal if you deal with
very big strings.

> Its not clear that guile eval is smart enough to manage a network socket --
> if the user starts a long-running process with intermittent prints, will it
> send that to the socket?  What if the user hits cntrl-C in the middle of it
> all? What if the code that came over the socket happened to throw an
> exception?

These are important considerations, but it’s not eval’s business IMO.
Instead, I suggest building your own protocol around it, and having a
way in that protocol to report both exceptions and normal returns.

> I've had to deal with all of these issues in the past, and have a stable
> code base; but if I had to start all over again, its not clear that these
> issues have gone away.  I mean, eval was designed to eval -- it was not
> designed to support multi-threaded, concurrent network operations, right?

Right.

> To support my point: the default guile network REPL server is painfully
> slow, and frequently crashes/hangs. It works well enough to do some demos
> but is not stable enough for production use ... if its just read+eval, that
> might explain why its unstable.

I’ve never noticed slowness of the REPL server, nor crashes.

That said, if you run a REPL server in a separate thread and mutate the
global state of the program, you could possibly crash it—no wonders
here.

Likewise, the REPL server is meant to be used for debugging on
localhost.  If you talk to a REPL server over the network with high
latency, it’s going to be slow, not surprisingly.

So yes, I find the REPL server to be a really pleasant tool when
debugging an application locally, but that’s all it is—it’s not a remote
procedure call framework or anything like that.

Thanks,
Ludo’.

Re: Guile bugs

[Linas Vepstas]

Hi Ludo,

On Fri, Sep 15, 2017 at 3:56 PM, Ludovic Courtès <ludo@gnu.org> wrote:

> Linas Vepstas <linasvepstas@gmail.com> skribis:
>
> > On Mon, Sep 11, 2017 at 2:26 AM, Ludovic Courtès <ludo@gnu.org> wrote:
> >
> >> Hello,
> >>
> >> Linas Vepstas <linasvepstas@gmail.com> skribis:
> >>
> >> > The stuff coming over the network sockets are bytes, not s-exps. Since
> >> none
> >> > of the bytes are ever zero, they are effectively C/C++ strings, and
> are
> >> > handled as such. These C strings are sent to  scm_eval_string()
> wrapped
> >> > by scm_c_catch().
> >>
> >> I don’t know to what extent that is applicable to your software, but my
> >> recommendation would be to treat that network socket as a Scheme port,
> >> pass it to ‘read’, and pass the result to ‘eval’ (as opposed to reading
> >> the whole string from C++ and passing it to ‘scm_eval_string’.)
> >>
> >
> > Why?  What advantage does this offer?
>
> It avoids copies and conversions, which is big deal if you deal with
> very big strings.
>
> > Its not clear that guile eval is smart enough to manage a network socket
> --
> > if the user starts a long-running process with intermittent prints, will
> it
> > send that to the socket?  What if the user hits cntrl-C in the middle of
> it
> > all? What if the code that came over the socket happened to throw an
> > exception?
>
> These are important considerations, but it’s not eval’s business IMO.
> Instead, I suggest building your own protocol around it, and having a
> way in that protocol to report both exceptions and normal returns.
>

Well, yes, this is exactly what I've done.

This conversation is frustrating: either piping read to eval is the right
thing to do, in which case eval must handle network connections correctly,
or else piping read to eval is the wrong thing to do.  You can't have it
both ways.


> > I've had to deal with all of these issues in the past, and have a stable
> > code base; but if I had to start all over again, its not clear that these
> > issues have gone away.  I mean, eval was designed to eval -- it was not
> > designed to support multi-threaded, concurrent network operations, right?
>
> Right.
>
> > To support my point: the default guile network REPL server is painfully
> > slow, and frequently crashes/hangs. It works well enough to do some demos
> > but is not stable enough for production use ... if its just read+eval,
> that
> > might explain why its unstable.
>
> I’ve never noticed slowness of the REPL server, nor crashes.
>

You are probably using it only very lightly, and not in a high-load systems
environment. It runs maybe 5x slower than my current guile shell server,
and it is very definitely unstable and crashy.

In my environment, I am sending it approximately from one up to twenty
scheme expressions every second, with a new socket opened for each scheme
expression. This goes on for days or weeks. I am using a custom guile
server written in C++, which accepts network connections, reads bytes from
the network, and sends them to scm_eval_string(). It mostly works fine,
with a couple of problems: there seems to be a pointless utf8-utf32
conversion, which started this email chain.

There also seems to be some sort of very rare race condition in the
compiler that leads to corruption inside of guile. I believe that this can
be triggered by starting twenty threads (for example) and then compiling
and running fairly short programs in each thread. By "fairly short" I mean
"less than 5-10 lines of code", and which compute and return answers in
less than a tenth of a second. Doing this for a few hours eventually causes
guile to hang in a spinloop, trying to read some guile-internal structure
that has invalid data in it. I opened a bug report for this a month or two
ago, but did not supply an easy-to-trigger test case.

I tried replacing my guile network server with the REPL shell, and
discovered that the REPL server is much much slower; I don't recall exactly
how I measured the 5x number, but that was from an actual measurement.   I
don't think the REPL server can handle 20 network connections per second.

I remember hypothesizing that guile was being re-initialized for every
network connection. Obviously, this is wasteful and slow.

Entering guile is a large bottleneck.  I once measured this, and I think it
takes approximately 200 microseconds to enter guile, which implies a
maximum limit of about 5K guile evaluations per second, when using the
simple-minded design of having the C code enter guile each time before
evaluation an expression.  By contrast, python (cython) can be entered in
10 or 20 microseconds.

The test case here is how many times per second can one eval some simple
expression, e.g. (+ 2 2) or the equivalent of that in python.

The solution for the heavy cost of entering guile is to create a pool for a
few dozen threads, enter guile in each, and then never exit -- just return
threads to the thread pool, when the eval is completed, and the thread is
no longer needed.  This cuts the  200 microseconds overhead to zero, and
what one is then left with is the cost of calling scm_eval_string().  I did
measure that too, but I don't recall the numbers.


> That said, if you run a REPL server in a separate thread and mutate the
> global state of the program, you could possibly crash it—no wonders
> here.
>

Yes, well, I would call that a bug! It feels like you are trying to blame
me for a guile bug -- its not my fault that it crashes!

I did not look very carefully, and don't recall what the stack traces
looked like, but I got the impression that there were race conditions in
guile init, and how it interacted with the sockets.

Likewise, the REPL server is meant to be used for debugging on
> localhost.  If you talk to a REPL server over the network with high
> latency, it’s going to be slow, not surprisingly.
>

The performance problem was not the  latency, it was the number of
connections it could accept.

I'll say it again: I have a different network server that is 5x faster than
the REPL server, and it works, it is stable.

For reasons completely unrelated to guile, I would like to declare my
network server deprecated and obsolete.  However, I cannot do this, because
the guile REPL server is not yet good enough to be an adequate replacement.

--linas

>
> So yes, I find the REPL server to be a really pleasant tool when
> debugging an application locally, but that’s all it is—it’s not a remote
> procedure call framework or anything like that.
>

> Thanks,
> Ludo’.
>



-- 
*"The problem is not that artificial intelligence will get too smart and
take over the world," computer scientist Pedro Domingos writes, "the
problem is that it's too stupid and already has." *

Re: Guile bugs

[Chris Vine]

On Tue, 19 Sep 2017 19:04:21 +0800
Linas Vepstas <linasvepstas@gmail.com> wrote:
> On Fri, Sep 15, 2017 at 3:56 PM, Ludovic Courtès <ludo@gnu.org> wrote:
[snip]
> > That said, if you run a REPL server in a separate thread and mutate
> > the global state of the program, you could possibly crash it—no
> > wonders here.
> >  
> 
> Yes, well, I would call that a bug! It feels like you are trying to
> blame me for a guile bug -- its not my fault that it crashes!
> 
> I did not look very carefully, and don't recall what the stack traces
> looked like, but I got the impression that there were race conditions
> in guile init, and how it interacted with the sockets.

Your problems are clearly at least partly at a deeper level, and you
may already know what I say below on thread safety, but:

(i) you cannot auto-compile the same file concurrently in two different
threads, for obvious reasons: the generated 'go' file will be a mash up
to two different compilations.  I have not had a problem with
user-compiling files concurrently to different locations, although I
accept that you may have (although you say this is rare).

(ii) prior to guile-2.0.10, guile initialization was not thread safe -
you needed to have entered scm_with_guile() at least once in a single
thread, and completed initialization (say by having scm_with_guile()
return), before starting any new threads which might invoke
scm_with_guile() or evaluate any scheme expressions.

(iii) modifying most system non-fluid/non-parameter globals is thread
unsafe.  In particular, loading modules concurrently for the first time
will lead to problems (you have to load all required modules up front
in a single thread before starting any new threads which might
independently try to load the modules).  Ports are also not thread safe
(rightly so in my view), and I believe that extends to the global stdout
and stdin ports.

Chris

Re: Guile bugs

[Chris Vine]

On Tue, 19 Sep 2017 21:18:40 +0100
Chris Vine <vine35792468@gmail.com> wrote:
> ... Ports are also not thread safe (rightly so in my view), and I
> believe that extends to the global stdout and stdin ports.

By that I mean that any one port object is not thread safe against
concurrent use by two or more different threads, without external
synchronization.

Re: Guile bugs

[Nala Ginrut]

Hi folks!
Now that we all have such need, why bother to polish the nasty REPL way for
remote execution?
Let's discuss serializable-continuations which I always expect very much!


2017年9月20日 上午4:38，"Chris Vine" <vine35792468@gmail.com>写道：

> On Tue, 19 Sep 2017 21:18:40 +0100
> Chris Vine <vine35792468@gmail.com> wrote:
> > ... Ports are also not thread safe (rightly so in my view), and I
> > believe that extends to the global stdout and stdin ports.
>
> By that I mean that any one port object is not thread safe against
> concurrent use by two or more different threads, without external
> synchronization.
>
>