GNU Guile 2.1.7 released (beta)

GNU Guile 2.1.7 released (beta)

[Andy Wingo]

We are pleased to announce GNU Guile release 2.1.7.

Guile 2.1.7 is the seventh pre-release in what will eventually become
the 2.2 release series.  We encourage you to test this release and
provide feedback to guile-devel@gnu.org.

This is a bug-fix release, mostly fixing bugs related to suspending
partial continuations on one thread and resuming them on another.  See
the full NEWS below for details.

At this point you might ask yourself when 2.2.0 is coming.  The answer
is, very soon!  A followup mail will propose a timeline and a list of
blocker bugs.  I would like to aim for final prerelease in another 4
weeks and a final release a week after that.

                             *  *  *

The Guile web page is located at http://gnu.org/software/guile/, and
among other things, it contains a copy of the Guile manual and pointers
to more resources.

Guile is an implementation of the Scheme programming language, with
support for many SRFIs, packaged for use in a wide variety of
environments.  In addition to implementing the R5RS Scheme standard,
Guile includes a module system, full access to POSIX system calls,
networking support, multiple threads, dynamic linking, a foreign
function call interface, and powerful string processing.

Guile can run interactively, as a script interpreter, and as a Scheme
compiler to VM bytecode.  It is also packaged as a library so that
applications can easily incorporate a complete Scheme interpreter/VM.
An application can use Guile as an extension language, a clean and
powerful configuration language, or as multi-purpose "glue" to connect
primitives provided by the application.  It is easy to call Scheme code
From C code and vice versa.  Applications can add new functions, data
types, control structures, and even syntax to Guile, to create a
domain-specific language tailored to the task at hand.

Guile 2.1.7 can be installed in parallel with Guile 2.0.x; see
http://www.gnu.org/software/guile/manual/html_node/Parallel-Installations.html.

A more detailed NEWS summary follows these details on how to get the
Guile sources.

Here are the compressed sources:
  http://alpha.gnu.org/gnu/guile/guile-2.1.7.tar.gz   (17MB)
  http://alpha.gnu.org/gnu/guile/guile-2.1.7.tar.xz   (10MB)

Here are the GPG detached signatures[*]:
  http://alpha.gnu.org/gnu/guile/guile-2.1.7.tar.gz.sig
  http://alpha.gnu.org/gnu/guile/guile-2.1.7.tar.xz.sig

Use a mirror for higher download bandwidth:
  http://www.gnu.org/order/ftp.html

Here are the SHA256 checksums:

  917f2b06eb07383f092920a6e93ead78a0710afcdaabb66863aeb411b4cb3965  guile-2.1.7.tar.gz
  aae1ca162ff5a1f8d173effb8635e8691f9fdf612c254664ce65d3b78831c261  guile-2.1.7.tar.xz

[*] Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify guile-2.1.7.tar.gz.sig

If that command fails because you don't have the required public key,
then run this command to import it:

  gpg --keyserver keys.gnupg.net --recv-keys FF478FB264DE32EC296725A3DDC0F5358812F8F2

and rerun the 'gpg --verify' command.

This release was bootstrapped with the following tools:
  Autoconf 2.69
  Automake 1.15
  Libtool 2.4.6
  Gnulib v0.1-1157-gb03f418
  Makeinfo 6.3

                             *  *  *

Changes in 2.1.7 (changes since the 2.1.6 alpha release):

* Notable changes

** Web server now suspendable

The web server's implementation has been slightly modified in order to
allow coroutines to suspend and resume around it when it would block on
input or output.  See "Non-Blocking IO" in the manual for more.

** Add support for arrays in `truncated-print'.

See "Pretty Printing" in the manual.  Thanks to Daniel Llorens.

** Gnulib update

Gnulib has been updated to v0.1-1157-gb03f418.

* Performance improvements

** Stringbufs immutable by default

Stringbufs are backing buffers for strings, and are not user-visible.
Calling "substring" on a base string will result in a new string that
shares state with the base string's stringbuf.  A subsequent attempt to
mutate the substring will first copy a fresh stringbuf; that is, Guile's
strings are copy-on-write.  There is also "substring/shared" which
allows mutations to be shared between substring and base string; in that
case the stringbuf is modified directly.

It used to be that mutating a string would have to take a global lock,
to ensure that no one was concurrently taking a copy-on-write substring
of that string.  That is, stringbufs were mutable by default and
transitioning to immutable could happen at any time.

This situation has been reversed: stringbufs are now immutable by
default and attempts to mutate an immutable stringbuf will copy a fresh
stringbuf and mark it as mutable.  This way we can avoid the global
lock.  This change likely speeds up common "substring" workloads, though
it make make the first in-place mutation on an immutable string take
more time because it has to copy a fresh backing stringbuf.

** Speed up number->string

** `accept' now takes optional flags argument

These flags can include `SOCK_NONBLOCK' and `SOCK_CLOEXEC', indicating
options to apply to the returned socket, potentially removing the need
for additional system calls to set these options.  See "Network Sockets
and Communication" in the manual, for more.

* New deprecations

** `SCM_FDES_RANDOM_P'

Instead, use `lseek (fd, 0, SEEK_CUR)' directly.

* Bug fixes

** Fix too-broad capture of dynamic stack by delimited continuations

Guile was using explicit stacks to represent, for example, the chain of
current exception handlers.  This means that a delimited continuation
that captured a "catch" expression would capture the whole stack of
exception handlers, not just the exception handler added by the "catch".
This led to strangeness when resuming the continuation in some other
context like other threads; "throw" could see an invalid stack of
exception handlers.  This has been fixed by the addition of the new
"fluid-ref*" procedure that can access older values of fluids; in this
way the exception handler stack is now implicit.  See "Fluids and
Dynamic States" in the manual, for more on fluid-ref*.

** Fix bug comparing unboxed floating-point values (#25492)

Thanks to Daniel Llorens.

** Fix crasher bugs for multiple threads writing to same port

** Fix bug resuming partial continuations that contain prompts

Re: GNU Guile 2.1.7 released (beta)

[Andy Wingo]

On Sat 18 Feb 2017 11:31, Andy Wingo <wingo@pobox.com> writes:

> At this point you might ask yourself when 2.2.0 is coming.  The answer
> is, very soon!  A followup mail will propose a timeline and a list of
> blocker bugs.  I would like to aim for final prerelease in another 4
> weeks and a final release a week after that.

Ahem :)

I think that the 2.1.x series is ready to go.  There are bugs which we
should fix of course.  Some of those bugs can be fixed within the stable
series; some we can't.  Of the bugs that we can't, some we can punt to
the next stable series, but some are _release blockers_ -- bugs that we
need to fix before 2.2.0.

Release blocker bugs principally concern ABI.  For example the foreign
objects facility which was applied to 2.0 and 2.2 and then rolled out
from 2.0 because maybe it's not an interface that we want to support in
the future -- well that is still part of 2.2 and if we released today we
would indeed have to support it.  It's that kind of question.

So!  Release blockers.

 * Foreign objects.  In or out?  Let's start a subthread.

 * The approach to macro-introduced top-level identifiers: OK or no?
   Again a subthread would be the thing.

 * GOOPS: are there incompatible changes that we think are bad?
   Subthread :)

There are a number of other bugs that are also important
(reproducibility, broken REPL server) that aren't strictly blockers in
the sense that we can release a 2.2.x that fixes them.  Those are
interesting but unless you want to schedule _your_ work/resources to get
them done, they are not blockers, as far as I undersand things.

I propose to release a 2.1.8 on 9 March.  At that point I would like to
have all release blockers resolved one way or the other.  Then we can
release a 2.2.0 whenever, provided that 2.1.8 release actually compiles
:)

Thoughts?

Cheers,

Andy

Re: GNU Guile 2.1.7 released (beta)

[Mike Gran]

On Thursday, February 23, 2017 11:15 AM, Andy Wingo <wingo@pobox.com> wrote:


> So!  Release blockers.


I know you're asking about how to wrap up 2.2, and here
I am to suggest new features.  So rude. ;-)

If I had to come up with my own personal lists of features
I'd love to claim for a future release.

* A string encoding that can handle raw bytes
  to make Guile more Emacs friendly, as per many e-mails in this
  thread. 

  http://lists.gnu.org/archive/html/guile-user/2017-02/msg00118.html

* Some helper funcs for debugging.  For example, I just found
  out (after a decade) that Guile comes with its own pk.  Also,
  a "break REPL here when condition is true" procedure would
  be great.  I've sometimes rolled my own with ",break acos" and
  adding (acos 0.0) for the location in question.

* Fix GDB support.


From my understanding of Guile internals, the former is
tractable.
-Mike

Re: GNU Guile 2.1.7 released (beta)

[Andy Wingo]

Hi :)

On Thu 23 Feb 2017 21:04, Mike Gran <spk121@yahoo.com> writes:

> I know you're asking about how to wrap up 2.2, and here
> I am to suggest new features.  So rude. ;-)

Hehe :)  The suggestions are still welcome.  If they make 2.2.0, super!
If they make 2.2.1, also super :)

> * A string encoding that can handle raw bytes
>   to make Guile more Emacs friendly, as per many e-mails in this
>   thread.

I haven't digested the thread so I defer to you for the moment :)

> * Some helper funcs for debugging.  For example, I just found
>   out (after a decade) that Guile comes with its own pk.  Also,
>   a "break REPL here when condition is true" procedure would
>   be great.  I've sometimes rolled my own with ",break acos" and
>   adding (acos 0.0) for the location in question.

Yeah pk is nice!!!!  It is how I do a lot of my debugging.  Silly.

The "break when condition is true" thing is totally doable.

> * Fix GDB support.

I wasn't aware that it was broken but I don't doubt it :)

Cheers,

Andy

Re: GNU Guile 2.1.7 released (beta)

[Thien-Thi Nguyen]

[-- Attachment #1.1: Type: text/plain, Size: 162 bytes --]


() Andy Wingo <wingo@pobox.com>
() Thu, 23 Feb 2017 22:06:30 +0100

   Yeah pk is nice!!!!

Thumbs up for ‘pk’.  Here's the pair of commands i use:


[-- Attachment #1.2: pk-unpk.el --]
[-- Type: application/emacs-lisp, Size: 2012 bytes --]

[-- Attachment #1.3: Type: text/plain, Size: 405 bytes --]


(This file is in Unofficial Guile 1.4.x, in subdir emacs/ -- any
objections if i add it to GNU Guile?)

-- 
Thien-Thi Nguyen -----------------------------------------------
 (defun responsep (query)
   (pcase (context query)
     (`(technical ,ml) (correctp ml))
     ...))                              748E A0E8 1CB8 A748 9BFA
--------------------------------------- 6CE4 6703 2224 4C80 7502

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: GNU Guile 2.1.7 released (beta)

[Andy Wingo]

On Wed 01 Mar 2017 14:01, Thien-Thi Nguyen <ttn@gnu.org> writes:

> () Andy Wingo <wingo@pobox.com>
> () Thu, 23 Feb 2017 22:06:30 +0100
>
>    Yeah pk is nice!!!!
>
> Thumbs up for ‘pk’.  Here's the pair of commands i use:
>
> ;;; pk-unpk.el --- Commands to instrument Scheme code

Neat :)  I guess you are not a paredit person?  For me in that
situation:

> ;;              (let ((answer (compute-answer))) ...)
> ;;                            ^
> ;;                            point is here
> ;;
> ;; Try `M-x pk' then type
> ;;
> ;;              'answer 'is
> ;;
> ;; The code then looks like:
> ;;
> ;;              (let ((answer (pk 'answer 'is (compute-answer)))) ...)

I would do

  M-( pk 'answer is

and it "just works" (if you have a paredit-infected brain like me).

> ;;
> ;; `M-x unpk' searches backwards and upwards for a `pk' form and removes
> ;; it, or signals error if one cannot be found.

Here with point on the open paren in (compute-answer), in paredit M-<Up>
does a similar thing.

FWIW :)

Andy

Re: GNU Guile 2.1.7 released (beta)

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 465 bytes --]


() Andy Wingo <wingo@pobox.com>
() Wed, 01 Mar 2017 18:36:20 +0100

   [paredit-fu]
   FWIW :)

Cool.  I'll add this to the manual.  Any suggestions where?

-- 
Thien-Thi Nguyen -----------------------------------------------
 (defun responsep (query)
   (pcase (context query)
     (`(technical ,ml) (correctp ml))
     ...))                              748E A0E8 1CB8 A748 9BFA
--------------------------------------- 6CE4 6703 2224 4C80 7502


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Break-when [was GNU Guile 2.1.7 released (beta)]

[Matt Wette]

[-- Attachment #1: Type: text/plain, Size: 1154 bytes --]


> On Feb 23, 2017, at 12:04 PM, Mike Gran <spk121@yahoo.com> wrote:
> 
> * Some helper funcs for debugging.  For example, I just found
>  out (after a decade) that Guile comes with its own pk.  Also,
>  a "break REPL here when condition is true" procedure would
>  be great.  I've sometimes rolled my own with ",break acos" and
>  adding (acos 0.0) for the location in question.
> 

I am playing with this.  Now reading through system/repl/* and system/vm/* to understand the debugger.

Try

(use-modules (system repl repl))
(use-modules (system repl debug))

(define-syntax-rule (trap-here-0)
  (start-repl
   #:debug (make-debug (stack->vector (make-stack #t)) 0 "trap!" #t)))

(define (foo)
  (let iter ((sum 0) (vals '(1 2 3 5 8 2)))
    (trap-here-0)
    (cond
     ((null? vals) sum)
     (else (iter (+ sum (car vals)) (cdr vals))))))

(foo)

I have added some debugger commands: 
,loc - show current location
,qq - really quit (calls primitive-exit)

scheme@(guile-user) [1]> ,loc
      (let iter ((sum 0) (vals '(1 2 3 5 8 2)))
*       (trap-here-0)
        (cond
scheme@(guile-user) [1]> ,qq
mwette$ 


[-- Attachment #2: Type: text/html, Size: 7432 bytes --]

Re: GNU Guile 2.1.7 released (beta)

[Andy Wingo]

On Thu 23 Feb 2017 19:54, Andy Wingo <wingo@pobox.com> writes:

>  * Foreign objects.  In or out?  Let's start a subthread.

SMOBs have many problems:

 * they are limited in number (255 including guile's smobs)

 * SMOB types are only creatable from C (in a time where anything done
   in C should be doable in Scheme)

 * SMOBs can only be constructed from C

 * SMOB fields are only accessible from C

 * SMOB markers are ~~~~~~gnarly~~~~~~~

 * SMOB finalizers are tricky.  Well finalizers are tricky in general
   but for SMOBs specifically they can see other objects that have
   already been finalized (or which are being concurrently finalized)
   and thus have called their own free() functions, which can corrupt
   memory.

The situation with SMOBs is good if you don't specify mark procedures
(practically deprecated since Guile 1.8) and your finalizer doesn't
touch additional GC-managed objects that themselves have finalizers
(hard to determine) and if your finalizers are threadsafe (ha ha ha ha
ha ha).  The manual didn't really discuss any of these problems.  And
then there's the issue of the very 1980s interface of SMOBs.  Let's not
mention markers!

So you look at this whole situation and, this is an interface headed for
deprecation, right?  I mean surely we can do better -- or at the very
least we can do the same but in a more comprehensible way.  So I made
the "foreign object" interface to do just that.  See the docs here:

  https://www.gnu.org/software/guile/docs/master/guile.html/Defining-New-Foreign-Object-Types.html
  https://www.gnu.org/software/guile/docs/master/guile.html/Foreign-Objects.html

The former is in the introductory "using Guile from C" section and the
latter is the reference.  Compare to 2.0:

  https://www.gnu.org/software/guile/manual/html_node/Defining-New-Types-_0028Smobs_0029.html
    - it tells users to use mark functions (this is bad advice in
      general)
    - it doesn't mention much about threads when it talks
      about finalization
  https://www.gnu.org/software/guile/manual/html_node/Smobs.html#Smobs
    - not really linked to from the previous section??  (This one is in
      "API" and the other is in "Programming in C")
    - recommending scm_gc_free
    - scm_markcdr????

So foreign objects fixes the problems mentioned above and also makes the
whole facility more explainable.  It effectively replaces SMOBs; it's a
nicer interface in all regards.

I think it would be fine to continue having foreign objects going
forwards.  It's supportable just fine and strictly better than smobs
(unless you need a mark function, in which case SMOBs are still there
for you).

It could be that these foreign objects don't fulfill all use cases, such
as packed structs with C layouts.  Oh well; can't win them all.  We can
work on this more in the future I think.

Thoughts? :)

Andy

Re: GNU Guile 2.1.7 released (beta)

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 807 bytes --]


Andy Wingo <wingo@pobox.com> writes:

> I think that the 2.1.x series is ready to go.

Nice!

> Release blocker bugs principally concern ABI.  For example the foreign
> objects facility which was applied to 2.0 and 2.2 and then rolled out
> from 2.0 because maybe it's not an interface that we want to support in
> the future -- well that is still part of 2.2 and if we released today we
> would indeed have to support it.  It's that kind of question.

The main strategical question I see for that is: Does anything make it
harder to complete or improve the lilypond transition to Guile 2?

Is there something which would need to be done before 2.2 which could
make it easier for lilypond developers?

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

Re: GNU Guile 2.1.7 released (beta)

[Andy Wingo]

On Fri 24 Feb 2017 18:46, Arne Babenhauserheide <arne_bab@web.de> writes:

> The main strategical question I see for that is: Does anything make it
> harder to complete or improve the lilypond transition to Guile 2?
>
> Is there something which would need to be done before 2.2 which could
> make it easier for lilypond developers?

I don't know of any concrete points here.  2.2 is not dissimilar to 2.0.
If you someone like to try to determine exactly what specific bits of
Guile 2.2 impact Lilypond (and indeed any big application using the C
interface), that would be welcome :)  I guess I'd start with NEWS to see
what's up.

Andy

Re: GNU Guile 2.1.7 released (beta)

[Mike Gran]

On Sunday, February 26, 2017 9:58 AM, Andy Wingo <wingo@pobox.com> wrote:
On Fri 24 Feb 2017 18:46, Arne Babenhauserheide <arne_bab@web.de> writes:

>> The main strategical question I see for that is: Does anything make it
>> harder to complete or improve the lilypond transition to Guile 2?
>>
>> Is there something which would need to be done before 2.2 which could
>> make it easier for lilypond developers?

>I don't know of any concrete points here.  2.2 is not dissimilar to 2.0.
>If you someone like to try to determine exactly what specific bits of
>Guile 2.2 impact Lilypond (and indeed any big application using the C
>interface), that would be welcome :)  I guess I'd start with NEWS to see
>what's up.


As far as I ever understood it, the Lilypond SCM problem went
something like the following. Is this something that the new
foreign objects can be adapted to handle?

A C++ STL container holds a set of STL-allocated (non-GC allocated)
structs.

Those STL-allocated structs are also used as the payloads of
SCM foreign objects.

STL destruction can free those objects.  That free
should not allow SCM foreign objects that continue to exist but
contain junk payloads.


Also should SCM GC free the SCM foreign objects, this GC
should not cause the non-GC-allocated payload to be freed.


But I only ever had a distant view of the Lilypond problems,
so this summary might be invalid.


-Mike Gran

Re: GNU Guile 2.1.7 released (beta)

[Andy Wingo]

On Mon 27 Feb 2017 20:32, Mike Gran <spk121@yahoo.com> writes:

> A C++ STL container holds a set of STL-allocated (non-GC allocated)
> structs.
>
> Those STL-allocated structs are also used as the payloads of
> SCM foreign objects.
>
> STL destruction can free those objects.  That free
> should not allow SCM foreign objects that continue to exist but
> contain junk payloads.
>
> Also should SCM GC free the SCM foreign objects, this GC
> should not cause the non-GC-allocated payload to be freed.

I believe this issue was fixed in
8dff3af087c6eaa83ae0d72aa8b22aef5c65d65d in Guile 2.0 and in a related
commit in 2.2.

Andy

Re: GNU Guile 2.1.7 released (beta)

[Thomas Morley]

2017-02-26 18:57 GMT+01:00 Andy Wingo <wingo@pobox.com>:
> On Fri 24 Feb 2017 18:46, Arne Babenhauserheide <arne_bab@web.de> writes:
>
>> The main strategical question I see for that is: Does anything make it
>> harder to complete or improve the lilypond transition to Guile 2?
>>
>> Is there something which would need to be done before 2.2 which could
>> make it easier for lilypond developers?
>
> I don't know of any concrete points here.  2.2 is not dissimilar to 2.0.
> If you someone like to try to determine exactly what specific bits of
> Guile 2.2 impact Lilypond (and indeed any big application using the C
> interface), that would be welcome :)  I guess I'd start with NEWS to see
> what's up.
>
> Andy
>

Hi,

I'm more a musician and LilyPond power-user. As a developer I'm
focused on creating new, directly usable functionality.
So I feel pretty much out of my league here...
Though, I hope a view from a LilyPond user may find some interest.


Due to bug-fixes guile-2.0.12 was the first version which could be
used to build LilyPond, afaik.
We have now LilyDev-5.1 for use in a VB (with guile-2.10.13), for
everyone who wants to work on guilev2 migration.

Meanwhile I have a LilyPond-build with guile-2.1.7 on my machine.
(With a brute-force fix, simply deleting all functionality related to
scm_protects.
Replacing it with scm_gc_protect_object as the IRC-log
https://gnunet.org/bot/log/guile/2017-01-05
seems to suggest didn't work)

The main problems/TODOs are listed here (same for guile-2.0.13 and 2.1.7):
https://ao2.it/tmp/lilypond-guile2/TODO
With no warranty for completeness.

Let me pick some of them:
(1)
lilypond filename_名字.ly
returns
fatal error: failed files: "filename_??????.ly"

(2)
Floating point numbers are different in some decimal digits.
Possible impact on spacing in a regression-test for utf-8.

(3)
Most imortant for users:
LilyPond slowed down dramatically. Today I tested a huge file:

lilypond 2.19.52 with guile-1.8

real    9m8.229s
user    6m41.156s
sys     0m11.940s

lilypond 2.19.56 with guile-2.1.7

real    48m45.225s
user    65m43.252s
sys     0m6.320s



Investigating and fixing those is beyond my current knowledge.
Though, that's the current state, afaik.

Best,
  Harm

Re: GNU Guile 2.1.7 released (beta)

[Andy Wingo]

On Tue 28 Feb 2017 00:00, Thomas Morley <thomasmorley65@gmail.com> writes:

> The main problems/TODOs are listed here (same for guile-2.0.13 and 2.1.7):
> https://ao2.it/tmp/lilypond-guile2/TODO
> With no warranty for completeness.
>
> Let me pick some of them:
> (1)
> lilypond filename_名字.ly
> returns
> fatal error: failed files: "filename_??????.ly"

Interesting, I would have thought that there would be a difference
between 2.0.13 and 2.1.7 due to GUILE_INSTALL_LOCALE; I assume you are
in a UTF-8 locale and that file name is UTF-8?

> (2)
> Floating point numbers are different in some decimal digits.
> Possible impact on spacing in a regression-test for utf-8.

If you have more details on the floating-point issue, they are very
welcome :)

> (3)
> Most imortant for users:
> LilyPond slowed down dramatically. Today I tested a huge file:
>
> lilypond 2.19.52 with guile-1.8
>
> real    9m8.229s
> user    6m41.156s
> sys     0m11.940s
>
> lilypond 2.19.56 with guile-2.1.7
>
> real    48m45.225s
> user    65m43.252s
> sys     0m6.320s

Do you have Guile 2.0 numbers as well?

I understand that Lilypond uses the "local-eval" facility a lot for
embedded Scheme.  This is a facility that was indeed faster in 1.8.  I
would expect that 2.1.7 would be faster than 2.0, if that were the case,
as 2.1.7's evaluator is faster.  Hard to say, though.  That interface
does not get a lot of speed attention.  We could take a look and see
what we can do.  I guess we need some profiling first.

Can you run lilypond under callgrind under 1.8 and 2.1.7 and attach the
generated callgrind.out.PID for each run?  Run like this:

  valgrind --tool=callgrind --num-callers=20 lilypond foo.ly

Thanks,

Andy

Re: GNU Guile 2.1.7 released (beta)

[David Kastrup]

Andy Wingo <wingo@pobox.com> writes:

> On Tue 28 Feb 2017 00:00, Thomas Morley <thomasmorley65@gmail.com> writes:
>
>> The main problems/TODOs are listed here (same for guile-2.0.13 and 2.1.7):
>> https://ao2.it/tmp/lilypond-guile2/TODO
>> With no warranty for completeness.
>>
>> Let me pick some of them:
>> (1)
>> lilypond filename_名字.ly
>> returns
>> fatal error: failed files: "filename_??????.ly"
>
> Interesting, I would have thought that there would be a difference
> between 2.0.13 and 2.1.7 due to GUILE_INSTALL_LOCALE; I assume you are
> in a UTF-8 locale and that file name is UTF-8?
>
>> (2)
>> Floating point numbers are different in some decimal digits.
>> Possible impact on spacing in a regression-test for utf-8.
>
> If you have more details on the floating-point issue, they are very
> welcome :)
>
>> (3)
>> Most imortant for users:
>> LilyPond slowed down dramatically. Today I tested a huge file:
>>
>> lilypond 2.19.52 with guile-1.8
>>
>> real    9m8.229s
>> user    6m41.156s
>> sys     0m11.940s
>>
>> lilypond 2.19.56 with guile-2.1.7
>>
>> real    48m45.225s
>> user    65m43.252s
>> sys     0m6.320s
>
> Do you have Guile 2.0 numbers as well?
>
> I understand that Lilypond uses the "local-eval" facility a lot for
> embedded Scheme.

It doesn't use it at all.

> This is a facility that was indeed faster in 1.8.  I would expect that
> 2.1.7 would be faster than 2.0, if that were the case, as 2.1.7's
> evaluator is faster.  Hard to say, though.  That interface does not
> get a lot of speed attention.  We could take a look and see what we
> can do.  I guess we need some profiling first.

Moot.  What used to be done with local-eval before we had to come up
with a replacement (at that time, it did not appear like the facility
would be available until the migration to Guile 2.0 was completed) is a
lambda capture: the _forms_ that would have been used in local-eval are
already with some reliability determinable at the time the environment
is captured, so instead of capturing the local environment, the forms to
be evaluated later are captured as individual lambda functions in that
environment.

The feature implemented in this manner is the #{ ... #} construct for
embedding LilyPond.  It is used extensively but not likely in
performance-critical contexts.

Regular read and eval, however, is used a lot during the parsing of
files and startup of LilyPond.  But at least under Guile-1.8, the
parsing and preprocessing took up a rather small part of the overall
runtime (in the order of 15% or so), so it is unlikely to be responsible
for the bulk of the slowdown.

My personal guess is that the largest performance impact at the moment
will be due to an absence of generation and installation of .go files.
Since .go files are target-dependent (if I am not mistaken) and LilyPond
is cross-compiled for a number of architectures with different byte
orders and type sizes, it seems tricky to get this under wraps.

The next largest performance impact will be redecoding issues.

> Can you run lilypond under callgrind under 1.8 and 2.1.7 and attach the
> generated callgrind.out.PID for each run?  Run like this:
>
>   valgrind --tool=callgrind --num-callers=20 lilypond foo.ly
>
> Thanks,
>
> Andy


-- 
David Kastrup

Re: GNU Guile 2.1.7 released (beta)

[Andy Wingo]

On Tue 28 Feb 2017 10:38, David Kastrup <dak@gnu.org> writes:

> Andy Wingo <wingo@pobox.com> writes:
>> I understand that Lilypond uses the "local-eval" facility a lot for
>> embedded Scheme.
>
> It doesn't use it at all.

Ah, tx for the info (here and below).

> Regular read and eval, however, is used a lot during the parsing of
> files and startup of LilyPond.  But at least under Guile-1.8, the
> parsing and preprocessing took up a rather small part of the overall
> runtime (in the order of 15% or so), so it is unlikely to be responsible
> for the bulk of the slowdown.
>
> My personal guess is that the largest performance impact at the moment
> will be due to an absence of generation and installation of .go files.

This makes sense to me.  As you know, using .go files allows you to
avoid read and expansion costs at run-time.  I guess Lilypond disables
auto-compilation for its Scheme files also, and doesn't manually compile
files / expressions?  In that case you're also suffering the slower
"eval" in 2.0 / 2.2 compared to 1.8 and not benefitting from the
compiler.  I agree that fixing that would probably be the first step for
someone working on Lilypond perf.

> Since .go files are target-dependent (if I am not mistaken) and LilyPond
> is cross-compiled for a number of architectures with different byte
> orders and type sizes, it seems tricky to get this under wraps.

Yeah.  In both 2.0 and 2.2 there are only four "targets" really (32-bit
and 64-bit, big- and little-endian), so it's somewhat manageable.
"guild compile" does support cross-compilation, and I think there are
some projects that do so; but yep, wiring that up can be tricky like you
say.

Andy

Re: GNU Guile 2.1.7 released (beta)

[Thomas Morley]

Hi Andy,
sorry for the late reply.
My regular job eats too much time ....

2017-02-28 9:31 GMT+01:00 Andy Wingo <wingo@pobox.com>:
> On Tue 28 Feb 2017 00:00, Thomas Morley <thomasmorley65@gmail.com> writes:
>
>> The main problems/TODOs are listed here (same for guile-2.0.13 and 2.1.7):
>> https://ao2.it/tmp/lilypond-guile2/TODO
>> With no warranty for completeness.
>>
>> Let me pick some of them:
>> (1)
>> lilypond filename_名字.ly
>> returns
>> fatal error: failed files: "filename_??????.ly"
>
> Interesting, I would have thought that there would be a difference
> between 2.0.13 and 2.1.7 due to GUILE_INSTALL_LOCALE; I assume you are
> in a UTF-8 locale and that file name is UTF-8?

~$ locale
LANG=en_US.UTF-8
LANGUAGE=en
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="en_US.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="en_US.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=


>
>> (2)
>> Floating point numbers are different in some decimal digits.
>> Possible impact on spacing in a regression-test for utf-8.
>
> If you have more details on the floating-point issue, they are very
> welcome :)

Not really my topic.
Though first reported here:
http://lists.gnu.org/archive/html/lilypond-devel/2016-11/msg00156.html


>
>> (3)
>> Most imortant for users:
>> LilyPond slowed down dramatically. Today I tested a huge file:
>>
>> lilypond 2.19.52 with guile-1.8
>>
>> real    9m8.229s
>> user    6m41.156s
>> sys     0m11.940s
>>
>> lilypond 2.19.56 with guile-2.1.7
>>
>> real    48m45.225s
>> user    65m43.252s
>> sys     0m6.320s
>
> Do you have Guile 2.0 numbers as well?
>
> I understand that Lilypond uses the "local-eval" facility a lot for
> embedded Scheme.  This is a facility that was indeed faster in 1.8.  I
> would expect that 2.1.7 would be faster than 2.0, if that were the case,
> as 2.1.7's evaluator is faster.  Hard to say, though.  That interface
> does not get a lot of speed attention.  We could take a look and see
> what we can do.  I guess we need some profiling first.

Here some findings:

(1)
released lilypond-2.19.52 with guile-1.8.7

real    8m16.191s
user    6m39.864s
sys    0m10.860s

(2)
lilypond with guile-2.0.14 build from guile-git-repository, branch
remotes/origin/stable-2.0

real    34m11.762s
user    45m11.316s
sys    0m5.604s

(3)
lilypond with guile-2.1.7 build from guile-git-repository, branch master

real    67m29.132s
user    93m14.812s
sys    0m7.332s

More info in my reply to Arne on the user-list:
http://lists.gnu.org/archive/html/guile-user/2017-03/msg00042.html

> Can you run lilypond under callgrind under 1.8 and 2.1.7 and attach the
> generated callgrind.out.PID for each run?  Run like this:
>
>   valgrind --tool=callgrind --num-callers=20 lilypond foo.ly

Running valgrind on the same .ly-file as for my tests above is insane,
I aborted it after several (far too many) hours.
Let me try to find some ly-code/file of medium size to do so.

Thanks,
  Harm
>
> Thanks,
>
> Andy

Re: GNU Guile 2.1.7 released (beta)

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 854 bytes --]


Andy Wingo <wingo@pobox.com> writes:

> On Fri 24 Feb 2017 18:46, Arne Babenhauserheide <arne_bab@web.de> writes:
>
>> The main strategical question I see for that is: Does anything make it
>> harder to complete or improve the lilypond transition to Guile 2?
>>
>> Is there something which would need to be done before 2.2 which could
>> make it easier for lilypond developers?
>
> I don't know of any concrete points here.  2.2 is not dissimilar to 2.0.
> If you someone like to try to determine exactly what specific bits of
> Guile 2.2 impact Lilypond (and indeed any big application using the C
> interface), that would be welcome :)  I guess I'd start with NEWS to see
> what's up.

I’ll have to write up my test setup for lilypond, too …

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

Re: GNU Guile 2.1.7 released (beta)

[David Pirotte]

[-- Attachment #1: Type: text/plain, Size: 935 bytes --]

Hi Andy,

> So!  Release blockers.
> ...

Not a blocker, at all, but I was thinking to this, wrt manipulating (very) large
vectors, arrays, lists ...

-]	repl - truncated-print

Right now I edit the installed (system repl common), and wrote a tip in Guile-CV's
manual so users can do that as well:  less then optimal :).  It would be nice to
provide an option, so users could set it 'just like that', in the repl, or as a
global config in their .guile (I did see lloda does that in his for arrays, but it's
not an obvious option to set, it is a 'sophisticated' little piece of code (for an
end-user at least)).

-]	error(s) while manipulating (very) large vectors or arrays

Unlike the above, it appears there is currently no way to have error (the procedure)
and raised exceptions in general, to use truncated-print, it would be cool to 'link'
the above option so error reports use it as well.


WDYT?
David

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: GNU Guile 2.1.7 released (beta)

[Daniel Llorens]

On 28 Feb 2017, at 01:04, David Pirotte <david@altosw.be> wrote:

> Hi Andy,
> 
>> So!  Release blockers.
>> ...
> 
> Not a blocker, at all, but I was thinking to this, wrt manipulating (very) large
> vectors, arrays, lists ...
> 
> -]	repl - truncated-print
> 
> Right now I edit the installed (system repl common), and wrote a tip in Guile-CV's
> manual so users can do that as well:  less then optimal :).  It would be nice to
> provide an option, so users could set it 'just like that', in the repl, or as a
> global config in their .guile (I did see lloda does that in his for arrays, but it's
> not an obvious option to set, it is a 'sophisticated' little piece of code (for an
> end-user at least)).

I think this is the minimum for .guile:

(import (system repl common) (ice-9 format))
(repl-default-option-set! 'print (lambda (repl val) (format #t "~200@y" val)))

That doesn't seem so bad. For the current repl you can do:

(repl-option-set! (car (fluid-ref *repl-stack*)) 'print (lambda (repl val) (format #t "~200@y" val)))

We could have shortcuts, something like:

; not in current Guile
(repl-default-print-truncate! 200)
(repl-print-truncate! (current-repl) 200)

What do you think?

I'm guilty of finding these things by googling mailing lists and browsing the Guile source... repl-default-option-set! is documented, but repl-option-set!/ref isn't. There seems to be a chunk of text missing here:

https://www.gnu.org/software/guile/manual/html_node/System-Commands.html

There's one thing that I realized recently, that something like #1:5(a b c d e) or #@0:5(a b c d e) is actually valid read syntax, although #:5(a b c d e) is invalid, not sure if it should be. (Of course the truncated-print output doesn't *have* to be readable.) So it may be a good idea to print truncated arrays/vectors/etc that way, maybe even by default.

> -]	error(s) while manipulating (very) large vectors or arrays
> 
> Unlike the above, it appears there is currently no way to have error (the procedure)
> and raised exceptions in general, to use truncated-print, it would be cool to 'link'
> the above option so error reports use it as well.

Agreed, this is a serious issue for me. Unfortunately the current API lets the error reporter decide how to print the arguments using a format string (cf scm_error) so you have to work around that.

The hack I posted recently to guile-devel was buggy, so I'll repeat it here without the bug. It needs this patch:

http://git.savannah.gnu.org/gitweb/?p=guile.git;a=commitdiff;h=6118d9dd0cc7733ca71c6b803a942a15f463663a

and this code in .guile:

; Truncate output on exceptions. Requires exception-format to be used in ice-9/boot.scm.
; FIXME doesn't handle e.g. "x~~~s" -> "x~~~@y"
(define (rewrite-fmt fmt)
  (let loop ((f "") (b 0))
    (let ((next (string-contains-ci fmt "~s" b)))
      (if next
        (loop
         (if (or (zero? next) (not (char=? #\~ (string-ref fmt (- next 1)))))
           (string-append f (substring fmt b next) "~200@y")
           f)
         (+ next 2))
        (string-append f (substring fmt b))))))

(define (truncate-format port fmt . args)
  (apply format port (rewrite-fmt fmt) args))
(set! exception-format truncate-format)

This can't work in general, since the call to scm_error may decide to use ~a instead of ~s, etc. I'd limit further what can go into that format string.

But I would welcome a proper solution.

Regards

	—lloda

Re: GNU Guile 2.1.7 released (beta)

[David Pirotte]

[-- Attachment #1: Type: text/plain, Size: 824 bytes --]

Hi Daniel,
Andy,

> I think this is the minimum for .guile:
> 
> (import (system repl common) (ice-9 format))
> (repl-default-option-set! 'print (lambda (repl val) (format #t "~200@y" val)))
> 
> That doesn't seem so bad. For the current repl you can do:
> 
> (repl-option-set! (car (fluid-ref *repl-stack*)) 'print (lambda (repl val) (format
> #t "~200@y" val)))

Veeeeeeery complicated :)

Howe about this:

	(set! %repl-printer truncated-print)
	(set! %truncated-print-max-chars 72)

In the repl

	,repl-printer truncated-print
	,truncated-print-max-chars 72

Once set, the above would be used by error and raised exception 'printers', per
default imo, otherwise we could also have:

	(set! %error-printer truncated-print)

In the repl

	,error-printer truncated-print

Regards,
David

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]