bug#18520: string ports should not have an encoding

[David Kastrup <dak@gnu.org>]

ludo@gnu.org (Ludovic Courtès) writes:

> David Kastrup <dak@gnu.org> skribis:
>
>> I'm currently migrating LilyPond over to GUILE 2.0.  LilyPond has its
>> own UTF-8 verification, error flagging, processing and indexing.
>
> Do I understand correctly that LilyPond expects Guile strings to be byte
> vectors, which it can feed with UTF-8 byte sequences that it built by
> itself?

Not really.  LilyPond reads and parses its own files but it does divert
parts through GUILE occasionally in the process.  Some stuff is passed
through GUILE with time delays and parts wrapped into closures and
flagged with machine-identifiable source locations.

>> If you take a look at
>> <URL:http://git.savannah.gnu.org/cgit/lilypond.git/tree/scm/parser-ly-from-scheme.scm>,
>> ftell on a string port is here used for correlating the positions of
>> parsed subexpressions with the original data.  Reencoding strings in
>> utf-8 is not going to make this work with string indexing since ftell
>> does not bear a useful relation to string positions.
>
> AIUI the result of ‘ftell’ is used in only one place, while
> ‘port-line’ and ‘port-column’ are used in other places.

The ftell information is wrapped into an alist together with a closure
corresponding to the source location.  At a later point of time, the
surrounding string may be interpreted, and the source location is
correlated with the closure and the closure used instead of a call to
local-eval (which does not have the same power of evaluating materials
in a preserved lexical environment as a closure has).

> The latter seems more appropriate to me when it comes to tracking
> source location.

For error messages, yes.  For associating a position in a string with a
previously parsed closure, no.

> How is the result of ‘ftell’ used by callers of
> ‘read-lily-expression’?

See above.

>> I have more than enough crashes and obscure errors to contend with as
>> it stands,
>
> Could you open a separate bug with the backtrace of such crashes, if you
> think it may be Guile’s fault?

The backtraces are usually quite useless for diagnosing the crashes.
For example, there are crashes in scm_sloppy_assq.  If you look at the
code, it is clear that they can only happen for pairs that have already
been collected by garbage collection.  So the bug has occured quite a
bit previously to the crash.

So one has to figure out how the collection could possibly have happened
(naturally, it didn't with GUILE 1.8).  You can try doing that with the
rather expensive process of "reverse execution" (which basically traces
and keeps a history you can then explore backwards from the crash), but
that requires that the bugs are reproducible, and with collection in a
separate thread, that is not really the case.  Sometimes a crash
segfaults, more often you get std::exception triggered.  All with the
same input and executable.

-- 
David Kastrup