Re: How to make GNU Guile more successful

[Nala Ginrut <nalaginrut@gmail.com>]

@Linas Thanks for sharing your experiences on large data processing
using Guile! I'm developing a framework to do the similar work for our
product too. But what I'm planning is to do map/reduce for large data
in a distributed system. Anyway, I don't think it's a good way to
analysis massive data in just one very-strong machine. It requires too
much for the language compiler.

I'm still very confident to use Guile on machine learning work. Please
don't forget that no any dynamic language do the heavy computation
directly, they wrapped BLAS or MKL, just like what numpy does. And I
have chance to get a very strong Nvidia GPU cluster server for the
work on the next stage. I don't think the computing speed will be the
issue for me. We're using Python and C++14 at present, but I don't
think Python is the idea glue language, no one wants to use Cython,
trust me. The tail-recursive and delimited-continuation for
lightweight task scheduling is preferred too. The key issue for me is
the expressiveness, which could let us type less code to do more work.
Actually, we use Guile to generate many C++ code in compiling time to
save lot work coding work. This does matter, especially when you have
few people but much work to finish in a tight deadline.

There should be better paradigm to use Guile to glue C/C++ code.
Implement algorithm in Guile directly is not the good way. And wrap
C++ interface with FFI simply is not good too. Sometimes we need to do
more abstract work to get both advantages of Guile and C/C++.
This is about how we use Guile, not the compiler issue.

Of course, I confess there're many problems in our current Guile
compiler. The problem is that we have to use Guile a lot to get to
know what is the problem exactly, and give compelling reasons for the
improvement.

@Jan Yes, that should be a way to go. And I have a new idea which is
just an idea at present. Many we could find a way to read PyObject to
Gulie, and call Python module directly (say, numpy). There should be a
type-compatible abstract level between Guile and PyObject. If it
works, we may implement Python3 on Guile. Although it seems a large
work to implement complete Python3 frontend, we may save lot of work
to write alternative Python modules for Guile.
Julia language does in this idea, but it's backend is compatible with
Python. My idea is not to convert all Python types to Guile, just wrap
some types to a special object like <pyobject> is enough, then Guile
could be glue language for Python too.

Maybe people ask, why bother to glue Python? Python rocks!
Rocks? I need complete lambda calculus, proper tail call, static
scoping, and delimited-continuations (not the generators).
In our product code, we use lot of lambdas in C++ code, so I expect
the programming mind could be consistent between glue language and
main language. But Python can't provide multi-lines lambda expression
only because it uses whitespaces for delimiters, there's no any chance
to delimit multi expression within a lambda unless introducing more
complexity to the parser.

Back to the topic, Guile lacks sufficient practical usage in industry
to reveal its disadvantages. But its advantages are apparent enough
for me so that it worth for me to go further with it.


Best regards.




On Sat, Jul 15, 2017 at 6:10 PM, Jan Wedekind <jan@wedesoft.de> wrote:
> One could implement something like Theano+NumPy with GNU Guile. I am trying to do something like that (github.com/wedesoft/aiscm) but I am doing it in my spare time only. Theoretically GNU Guile is better suited for this than Python because of macros.
>
> On July 14, 2017 10:54:45 PM GMT+01:00, Linas Vepstas <linasvepstas@gmail.com> wrote:
>>On Mon, Feb 13, 2017 at 2:28 PM, Panicz Maciej Godek
>><godek.maciek@gmail.com
>>> wrote:
>>
>>>
>>> someone
>>> responded critically: "are there out of the box libraries to estimate
>>a
>>> zero inflated negative
>>> binomial regression model in guile". Of course, if I knew what a
>>> zero-inflated
>>> negative binomial regression model, I could deliver an implementation
>>by
>>> just explaining
>>> the notions used in that phrase.
>>
>>
>>Caution: the message below sounds negative.  Sorry, I use guile daily
>>and
>>almost exclusively now. So there ...
>>
>>Lack of decent science libraries for scheme is a major stumbling block,
>>for
>>me. Simply having sine and cosine is not enough.   I got excited (a
>>decade
>>ago) when I realized that guile supported GnuMP, and then rapidly
>>deflated
>>when I realized it only supported integers and rationals in GnuMP .. I
>>work
>>with arbitrary-precision floats.  Or, I did back then.
>>
>>Maybe more important is making guile work well with large-RAM setups.
>>Currently, I do data analysis, every day, in guile, on datasets that
>>take
>>20GB or 40GB -- my current one is 110GB when loaded in RAM, and guile
>>starts getting buggy, crashy and slow when working at that size.
>>Sometimes, it starts calling GC half-a-dozen times per second, for no
>>apparent reason, eating up 6 cores (or more!) doing nothing but GC.
>>Why?
>>Who knows? Who can tell?
>>
>>Yes, I have a machine with 256 GB RAM and a few dozen cores, and SSD's
>>that
>>hold the data, but every time guile crashes, I have to wait an hour for
>>the
>>data to reload.  I can live with it, but its a dirty secret I would not
>>share with guile wannabe users.
>>
>>String handling in guile is a disaster area: If I give it a
>>10-megabyte-long string in utf8, it promptly tries to convert all of
>>that
>>string in utf32, for utterly pointless reasons. This just makes it
>>slow.
>>
>>There are still bugs between GC and the compiler: if call (eval "(some
>>stuff) (other stuff)")  the compiler will try to compile that string
>>(after
>>it was converted ti utf32!) and if GC happens to run at just that
>>moment,
>>guile crashes or hangs.  These bugs need to be fixed.
>>
>>So although its a good start, there's a lot of work left until it can
>>get
>>to "the next level". And that work can't happen until guile is more
>>popular. So it's very much chicken-and-egg scenario.
>>
>>--linas
>
> --
> Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.