Worrying development

Worrying development

[Roland Orre]

At the moment I'm somewhat in a crisis how to continue with guile.

When such a fundamental function as shared substrings were removed from
guile 1.6 to guile 1.7, without being replaced, I see a serious risk
here for a split and do as Thien-Thi Nguyen.

OK, guile 1.7 is a development version so I should not rely on it,
on the other hand, as we are using guile as our main tool for analysis
we do of course also want to adapt to the development version to be
able to utilize facilities which are only available there.

I see this as an important policy issue and I quickly need to decide
how to go further. One way is of course that I patch specific parts
of the code, as in the shared substring case tags.h, strings.c and
gc.c. This is, however, also a risky thing to do, and it means more
maintainance work for every new version.

I suggest that shared substrings are moved back to guile. It is quite
serious when something which has been promoted in the reference manual
and which can not trivially be fixed in user code, is removed.

	Roland Orre




_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: Worrying development

[Thien-Thi Nguyen]

   From: Roland Orre <orre@nada.kth.se>
   Date: Fri, 16 Jan 2004 10:41:05 +0100

   to be able to utilize facilities which are only available there.

which facilities are these?  (could you be more specific?)

thi


_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: Worrying development

[Roland Orre]

On Fri, 2004-01-16 at 12:19, Thien-Thi Nguyen wrote:
>    From: Roland Orre <orre@nada.kth.se>
>    Date: Fri, 16 Jan 2004 10:41:05 +0100
> 
>    to be able to utilize facilities which are only available there.
> 
> which facilities are these?  (could you be more specific?)

The statement was not very specific and what I was thinking about was
not very specific, more based upon the general expectation that a new
version contains new functions (usually... now it was the opposite).

The main reason, however, for me to continue with 1.7 after 1.6 is that
as I've understood from Mikael Djurfeldt goops and the corresponding
matrix handling are only fully supported in 1.7. There is also real
pthreads support, which I've however not utilized yet, this is more
like a nice future potential. The only thing which I'll be dependent
on is goops support.

Another nice thing with guile 1.7, as I've understood, is that the
garbage collector now is utilizing Robert Strand's (Bobby Beach)
old idea to have the garbage collection bits separated from the
actual tags. This gives potentials for e.g. background garbage
collection as a parallell process. (I was by the way working with
Robert Strand at ASEA in the beginning of the 80ies when he invented
this idea).

Of course we also have the GMP support in 1.7, even though this is
nice, I won't use it myself very much. Mostly I'm doing number
crunching, statistical analysis and data base handling.

	Best regards
	Roland




_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: Worrying development

[tomas]

On Fri, Jan 16, 2004 at 10:41:05AM +0100, Roland Orre wrote:
> At the moment I'm somewhat in a crisis how to continue with guile.

[about the disappearing of shared substrings in Guile]

Hrm. I wasn't very happy with this decision either. But since
Dirk et al were doing the grunt work and I had nothing to
contribute, I decided to shut up.

As far as I remember, the main reasons for this step were
simplicity (with shared substrings you get to choose between
changes propagating from substrings to the containing string
(you don't always want that, although sometimes), a
copy-on-write implementation. Which is right?).

Besides, with non-shared substrings, you may guarantee
null-termination which helps naïve C interfacing (not a
major advantage to my eyes, but...).

I'd propose to implement a higher-level library (perhaps
using GOOPS) in which strings (or more generally arrays)
are implemented as (lists/trees/arrays) of arrays. That'd
help designing an interface, and when done (and if necessary),
re-implementing as smobs in C (or steal a datatype ;-)

Opinions?

-- tomás


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: Worrying development

[Marius Vollmer]

Roland Orre <orre@nada.kth.se> writes:

> I suggest that shared substrings are moved back to guile.

Agreed.

I'm sorry for previously giving the impression that shared substrings
wont come back.

There is no problem on the Scheme side of things: we can just add
shared substrings and make it a proper subtype of 'string'.

The problem lies with C code and there only with the low level API
consisting of SCM_STRINGP, SCM_STRING_CHARS etc.  Functions like
scm_c_string2str can be updated to just continue to work.

Shared substrings also touch on the issues of using Unicode in Guile
and on making sure we have a nice type conversion API that can replace
gh_ in all respects.

I'd like to do it in this order:

 - type conversion API (which allows for different encodings of
   strings, but doesn't need it immediately) (the first part of this
   was the 'frame' stuff for handling unwinds in C).

 - Unicode (with shared substrings in mind).

 - shared substrings

Of course, we shouldn't do too much lest 1.8 wont happen...

I'll try to put forth a proposal in the next days for the string part
of the type conversion API that allows Unicode and shared substrings.

-- 
GPG: D5D4E405 - 2F9B BCCC 8527 692A 04E3  331E FAF8 226A D5D4 E405


_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: Worrying development

[Tom Lord]

    > From: Marius Vollmer <mvo@zagadka.de>

    > I'll try to put forth a proposal in the next days for the string part
    > of the type conversion API that allows Unicode and shared substrings.

If you don't mind, I'd like to do that too -- independently.   But in
a screwy way.

I need to write a spec for the string API of Pika.  So far, the Pika
FFI is such that it could be implemented for Guile very easily.  And,
imo, it's also pretty good as an internal API for almost everything.
If used (gradually migrated to, in the case of Guile) as an internal
API -- I think it's pretty liberating (permitting lots of freedom for
different object representations, thread support, GC strategies,
etc.).

Maybe in the longer term -- unifying over the Pika APIs would be a
general win.

-t




_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: Worrying development

[Dirk Herrmann]

Hi folks,

first, sorry for the late answer...

Marius Vollmer wrote:

>Roland Orre <orre@nada.kth.se> writes:
>  
>
>>I suggest that shared substrings are moved back to guile.
>>    
>>
>
>Agreed.
>
>I'm sorry for previously giving the impression that shared substrings
>wont come back.
>
>There is no problem on the Scheme side of things: we can just add
>shared substrings and make it a proper subtype of 'string'.
>
>The problem lies with C code and there only with the low level API
>consisting of SCM_STRINGP, SCM_STRING_CHARS etc.  Functions like
>scm_c_string2str can be updated to just continue to work.
>
>Shared substrings also touch on the issues of using Unicode in Guile
>and on making sure we have a nice type conversion API that can replace
>gh_ in all respects.
>
>I'd like to do it in this order:
>
> - type conversion API (which allows for different encodings of
>   strings, but doesn't need it immediately) (the first part of this
>   was the 'frame' stuff for handling unwinds in C).
>
> - Unicode (with shared substrings in mind).
>
> - shared substrings
>
>Of course, we shouldn't do too much lest 1.8 wont happen...
>
>I'll try to put forth a proposal in the next days for the string part
>of the type conversion API that allows Unicode and shared substrings.
>
I am not quite sure, everybody is talking about the same issues here: 
When talking about the re-introduction of shared substrings, Marius, do 
you think of implicitly shared copy-on-write substrings, or guile's 
explicitly shared substrings?

Shared substrings as they have been provided by guile could have served 
two purposes:

1) saving resources (run time and memory)

2) communicating changes via something like a shared memory interface

The first purpose is just a matter of performance and should not change 
the functional behaviour of strings. However, guile's former 
implementation was only imperfectly suited to this kind of usage: 
Whoever used shared substrings for this purpose needed to be well aware 
of which strings were actually shared and which were not, because 
modifications on the strings would cause side effects. Thus, for this 
purpose the mechanism of implicitly shared copy-on-write substrings is 
safer. And, this is what we had intended to implement for guile. It 
would not require the user to perform any explicit action to have 
substrings to be shared.

The second purpose is about a change in behaviour, and to me it is not 
quite sure that it should be brought back in the old way. Providing 
shared substrings in that way changes the semantics of strings: Two 
strings s1 and s2, which are not eq? would become connected in a 
user-visible way such that modifications to s1 influence s2 and vice 
versa. What may users of a string data type expect? Shall it be granted 
that the following expression will always evaluate to true?
  (if (and
        (not (eq? s1 s2))
        (equal? s1 s3))
     (begin
       (string-set! s2 0 #\x)
       (equal? s1 s3))
     #t)
My assumption is that most users will assume the above expression to 
evaluate to true. If that was not the case, we would require users to 
perform aliasing checks in their code. Do we really want that? In which 
way would we extend the string API such that users are able to perform 
the necessary aliasing checks? IIRC, the old shared substring API did 
not provide a means for such aliasing checks.

The shared substring feature was deprecated since we had considered that 
feature as a bug in guile's design. I propose not to officially 
re-introduce it in its former way. The best thing was to have code 
changed that used the old behaviour. To allow applications to be 
migrated incrementally, we have provided the feature as deprecated since 
guile-1.6. If that is not possible for some applications, then a 
workaround like the one that Mikael and Roland have developed can be 
used. With that solution, the feature may even remain part of guile - 
but deprecated, only provided for backwards compatibility! Whoever uses 
it, should be aware of the fact that due to the aliasing it may lead to 
problems with other string libraries.

Best regards
Dirk Herrmann



_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: Shared Substrings [was: Worrying development]

[Robert Uhl]

Dirk Herrmann <dirk@dirk-herrmanns-seiten.de> writes:
> 
> Shared substrings as they have been provided by guile could have
> served two purposes:
> 
> 1) saving resources (run time and memory)
> 
> 2) communicating changes via something like a shared memory interface

It seems to me that the second case would be better served via a
more formally designed shared memory API.

As for the first case, that's just a performance hack.  True, some very
interesting examples have been given in which that performance hack has
been useful--but wouldn't it be better to work on improving the
performance of strings in general?  If an application really does need
the hack, wouldn't it be better for the author to write his own
extensions in C rather the cluttering up guile proper with what the
uninitiated might perceive as integral functionality?

-- 
Robert Uhl <ruhl@4dv.net>
Politicians and nappies should both be changed at regular intervals, and
for exactly the same reason.


_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: Worrying development

[Tom Lord]

    > From: Dirk Herrmann <dirk@dirk-herrmanns-seiten.de>

    > What may users of a string data type expect? Shall it be granted 
    > that the following expression will always evaluate to true?

    >   (if (and
    >         (not (eq? s1 s2))
    >         (equal? s1 s3))
    >      (begin
    >        (string-set! s2 0 #\x)
    >        (equal? s1 s3))
    >      #t)

What about:

    (if (and (not (eq? l1 l2))
             (equal? l1 l3))
        (begin
          (set-car! l2 'x)
          (equal? l1 l3))
        #t)


    > My assumption is that most users will assume the above expression to 
    > evaluate to true. 

When do they need to make such assumptions?   Why is it different from
the case with lists?

    > If that was not the case, we would require users to 
    > perform aliasing checks in their code. Do we really want that? 

I've never seen list-mutating code make such checks.   Why would
strings be different?


    > The shared substring feature was deprecated since we had
    > considered that feature as a bug in guile's design. I propose
    > not to officially re-introduce it in its former way. The best
    > thing was to have code changed that used the old behaviour. To
    > allow applications to be migrated incrementally, we have
    > provided the feature as deprecated since guile-1.6. If that is
    > not possible for some applications, then a workaround like the
    > one that Mikael and Roland have developed can be used. With that
    > solution, the feature may even remain part of guile - but
    > deprecated, only provided for backwards compatibility! Whoever
    > uses it, should be aware of the fact that due to the aliasing it
    > may lead to problems with other string libraries.

I am having trouble imagining any libraries that would break.

Let's suppose that, eventually, Guile has _both_ COW shared substrings
and shared-mutation shared substrings.

The only reason I would ever create a shared-mutation shared substring
in the first place is if I know that I want to mutate it (or it's
parent or some other shared-mutation string) and have the effect on
all of these strings.

Now what if I have shared-mutation substrings but not COW?

You say that the old implementation was flawed because it created
_only_ shared mutation substrings.  I don't think that that's a very
serious flaw.  In general, no procedure should mutate _any_ of its
arguments unless it is advertised as doing so.  Consequently, I simply
shouldn't hand a shared-mutation substring to a mutating procedure
unless I intend for that mutation to effect all sharing strings.

And on the other hand, if I have a mutating procedure -- almost
invariably the mutation is _unconditional_.  The "copy" of a COW
substring is guaranteed to take place.  If I'm going to pass a
substring to a mutating procedure and _don't_ want the mutations to
propogate, then I may as well do the "copy" eagerly in the first
place.

That decision to share mutations or not is one I can make locally --
at the point where I create the substring in the first place.
Libraries don't have to worry that I might have made that decision for
some parameters at all.  It's not their business.  There's no need for
aliasing checks.  Even if a library _wanted_ to worry about aliasing
it couldn't: it doesn't know what other strings to check for aliasing.

So, no, sorry -- the old implementation (shared-mutation-only) was
very good.

Adding a COW behavior to SUBSTRING is an upwards-compatible
improvement to the old way -- since many uses of SUBSTRING in portable
Scheme programs will never need to perform the copy -- but if you have
only one of the two kinds of shared substring, shared-mutation gives
you the greater functionality at essentially no cost to correct
standard programs.

As I vaguely recall, the only reason COW didn't become the behavior of
SUBSTRING "back then" was because of a tag-bit shortage (at the time).
It was something I had planned to eventually squeeze in.

-t




_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: Worrying development

[Dirk Herrmann]

Tom Lord wrote:

>    > From: Dirk Herrmann <dirk@dirk-herrmanns-seiten.de>
>
>    > What may users of a string data type expect? Shall it be granted 
>    > that the following expression will always evaluate to true?
>
>    >   (if (and
>    >         (not (eq? s1 s2))
>    >         (equal? s1 s3))
>    >      (begin
>    >        (string-set! s2 0 #\x)
>    >        (equal? s1 s3))
>    >      #t)
>
>What about:
>
>    (if (and (not (eq? l1 l2))
>             (equal? l1 l3))
>        (begin
>          (set-car! l2 'x)
>          (equal? l1 l3))
>        #t)
>
>
>    > My assumption is that most users will assume the above expression to 
>    > evaluate to true. 
>
>When do they need to make such assumptions?   Why is it different from
>the case with lists?
>
First: It's not a matter of whether users *need* to make certain 
assumptions: It's a matter of interface definition. Scheme defines the 
string data type and together with it, it defines the semantics of 
operations on it. This gives users a set of properties they *can* rely 
on. Certainly, not all of the properties are *needed* in every piece of 
code.

The difference with lists is, that lists are not _one_ object, but are 
made of several objects. That is, there is no statement saying that two 
different lists are independent of each other in the same way that two 
strings are independent of each other.

>You say that the old implementation was flawed because it created
>_only_ shared mutation substrings.  I don't think that that's a very
>serious flaw.  In general, no procedure should mutate _any_ of its
>arguments unless it is advertised as doing so.  Consequently, I simply
>shouldn't hand a shared-mutation substring to a mutating procedure
>unless I intend for that mutation to effect all sharing strings.
>
No, I said that the old implementation was flawed because it changed the 
semantics of the standard scheme string type. It is certainly all right 
if someone wants to use something like shared mutation substrings, but 
IMO this should be achieved with a different data type. It may be a good 
idea to provide a library for this kind of feature, but it should not 
modify scheme's standard string type. Who operates on strings should be 
allowed to make all assumptions about string behaviour that belong to 
the definition of the string data type.

Best regards
Dirk Herrmann



_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: Worrying development

[Tom Lord]

    > From: Dirk Herrmann <dirk@dirk-herrmanns-seiten.de>

    > First: It's not a matter of whether users *need* to make certain 
    > assumptions: It's a matter of interface definition. Scheme defines the 
    > string data type and together with it, it defines the semantics of 
    > operations on it. 

Please show me what existing lines of the Scheme standard will have to
change if mutation-sharing shared substrings are added.

The standard (not very formally but clearly enough) says that the
standard procedures which construct strings allocate fresh locations
for the contents of those strings.

That means that none of those procedures create mutation-sharing
shared substrings -- nobody has proposed anything different.

I think you are imagining that there is an additional requirement in
the standard: that any procedure at all which creates a new string
must allocate fresh locations for its contents.  But that additional
requirement isn't there.  Scheme programmers can not assume that that
requirement is part of Scheme.

Mutation-sharing shared substrings are an upwards compatible extension
to the Scheme standard.  They break no correct programs.  They enable
new kinds of programs.


-t



_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: Worrying development

[Marius Vollmer]

Tom Lord <lord@emf.net> writes:

> Mutation-sharing shared substrings are an upwards compatible extension
> to the Scheme standard.  They break no correct programs.  They enable
> new kinds of programs.

I'd say that the real 'trouble' is that strings are mutable at all.
Mutation-sharing substrings are only a minor additional semantical
annoyance.  They do enable new kinds of programs, and that makes them
valuable.

Also, there is the possibility on the horizon that we turn string-ref
etc into 'primitive generics' which means that people could implement
new kinds of strings using GOOPS.

Also, I still like the idea of using mutation-sharing substrings as
markers that allow O(1) access into variable-width encoded strings.

-- 
GPG: D5D4E405 - 2F9B BCCC 8527 692A 04E3  331E FAF8 226A D5D4 E405


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: Worrying development

[Tom Lord]

    > From: Marius Vollmer <mvo@zagadka.de>
    > Tom Lord <lord@emf.net> writes:
    > > Mutation-sharing shared substrings are an upwards compatible extension
    > > to the Scheme standard.  They break no correct programs.  They enable
    > > new kinds of programs.

    > I'd say that the real 'trouble' is that strings are mutable at
    > all.

Worried mostly about variable-length character encodings in string?
Or you'd just rather be programming in an ML-family language? :-)

If it's variable-length encodings that irk you:  if strings were
read-only you'd want to optimize the heck out of STRING-APPEND and
SUBSTRING and, once you did that, you'd have essentially enough
machinery to do mutations efficiently.


    > Also, I still like the idea of using mutation-sharing substrings as
    > markers that allow O(1) access into variable-width encoded strings.

Interesting.  The interaction with STRING-SET! will be tricky.  I
think you'll either have to "timestamp" strings (one tick per mutation
-- and you'll likely have to use a GC'ed value rather than an inline
integer for timestamps) or wind up with O(K) for mutations where K is
the number of shared substrings.

The same problem comes up if you add STRING-RESIZE!.  I keep going
back and forth on whether or not strings should be the same things as
or a subset buffers vs. making buffers a completely separate type.
(The latter certainly seems to be easier to implement.)


    > Also, there is the possibility on the horizon that we turn
    > string-ref etc into 'primitive generics' which means that people
    > could implement new kinds of strings using GOOPS.

Well, heck.  In that case, maybe consider what I'm planning for Pika
(at least initially).  Purely ASCII strings are stored 1-byte per
character.  Most other strings 2-bytes per character.   Strings using
characters outside the Basic Multilingual Plane, 4 bytes per
character.

You want some fancier-than-libc string functions in C for that -- but
it gives you an expected-case O(1) for STRING-REF and STRING-SET! and
pretty good space efficiency.  It also gives you some performance
glitches as when you store a U+0100 character in an otherwise purely
ASCII 10MB string.  (We're working on providing such fancier-than-libc
functions in libhackerlab -- so they'd be available independently of
Pika if you went this route.)
  
-t



_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: Worrying development

[Paul Jarc]

Marius Vollmer <mvo@zagadka.de> wrote:
> Also, there is the possibility on the horizon that we turn string-ref
> etc into 'primitive generics' which means that people could implement
> new kinds of strings using GOOPS.

Neat.  Has/might that also be done for car & cdr?  Then we could have
Python-like generators.  It would make SCM_C[AD]R less speedy, though
they could still be pretty fast for actual pairs.


paul


_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: Worrying development

[Dirk Herrmann]

Tom Lord wrote:

>    > From: Dirk Herrmann <dirk@dirk-herrmanns-seiten.de>
>
>    > First: It's not a matter of whether users *need* to make certain 
>    > assumptions: It's a matter of interface definition. Scheme defines the 
>    > string data type and together with it, it defines the semantics of 
>    > operations on it. 
>
>Please show me what existing lines of the Scheme standard will have to
>change if mutation-sharing shared substrings are added.
>
>The standard (not very formally but clearly enough) says that the
>standard procedures which construct strings allocate fresh locations
>for the contents of those strings.
>
>That means that none of those procedures create mutation-sharing
>shared substrings -- nobody has proposed anything different.
>
>I think you are imagining that there is an additional requirement in
>the standard: that any procedure at all which creates a new string
>must allocate fresh locations for its contents.  But that additional
>requirement isn't there.  Scheme programmers can not assume that that
>requirement is part of Scheme.
>
As you say, the standard only describes functions that, on creation of 
strings, requires to allocate fresh locations for the contents of the 
strings. That is, someone who only uses the functions that are described 
by the standard is not able to create any mutation-sharing substring. 
And this, implicitly, indicates that different (in the sense of eq?) 
strings use different locations for their contents. To me this seems 
like a valid assumption. Standard-conforming scheme programs may IMO 
rely on this fact. However, I may be wrong in my interpretation of the 
standard, which is why I suggest the srfi approach (see below).

>Mutation-sharing shared substrings are an upwards compatible extension
>to the Scheme standard.  They break no correct programs.  They enable
>new kinds of programs.
>
Introducing a separate data type for mutation-sharing character arrays 
also enables new kinds of programs. The difference between a separate 
data type and the former implementation is, that mutation-sharing 
substrings could be used everywhere where an ordinary string had been 
used before. That is, the difference is a matter of being able to re-use 
existing string-handling code rather than enabling new kinds of 
programs. However, it is exactly this re-using of existing 
string-handling code issue which becomes problematic when the semantics 
of the string objects change.

Marius, would it be an acceptable compromise to require that the 
mutation-sharing substring issue be submitted and discussed as an srfi 
before it becomes an official part of guile's core? The discussion of 
the topic in that forum would reduce the risk that the change introduces 
problems. I would then ask those who are interested to have it as a part 
of guile to submit a srfi proposal.

(Please note that, as I have said before, I have nothing against 
providing mutation-sharing substrings as a deprecated feature for some 
period - but not as an official part of guile's core.)

Best regards
Dirk



_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: Worrying development

[Tom Lord]

    > From: Dirk Herrmann <dirk@dirk-herrmanns-seiten.de>

    > As you say, the standard only describes functions that, on creation of 
    > strings, requires to allocate fresh locations for the contents of the 
    > strings. That is, someone who only uses the functions that are described 
    > by the standard is not able to create any mutation-sharing substring. 
    > And this, implicitly, indicates that different (in the sense of eq?) 
    > strings use different locations for their contents. To me this seems 
    > like a valid assumption. 

It's subtly wrong.

A program (or, more importantly, a module) can assume that non-eq?
strings are disjoint in their storage _if_and_only_if_ either (1) it
knows how both strings were created or (2) it's
assumption-of-disjointness is shared with the caller who passes the
strings to the module.

(1) is trivially obvious.

(2) may sound onerous but it really isn't.  It falls out of the
    existing practice of procedures advertising clearly what arguments
    they mutate.  A caller passing around mutation-sharing strings has
    to be careful to pass them to mutating procedures only when they
    are certain that they want mutations to be shared.   Ultimately,
    that obligation propogates right back to the point of creation of
    the mutation-sharing substring --- using mutation-sharing
    substrings is, every step of the way, a perfectly localized (a
    perfectly "modular") decision.

    You keep worrying about what happens if a procedure mutates a
    string and then is "surprised" when that mutation effects some
    other string.   No procedure need worry about that -- it is
    entirely up to code which _creates_ a mutation-sharing string to
    only pass it to a mutating procedure when it knows what it's doing.

    > Standard-conforming scheme programs may IMO 
    > rely on this fact [that non-eq? strings can't possibly share
    > mutations]

Freestanding portable standard programs can rely on that fact -- and
adding mutation-sharing substrings doesn't change that in the
slightest.

Modules, intended to be combined with other modules that may be using
non-standard feature can _not_ rely on that fact and have never been able
to rely on that fact.   Mutation-sharing substrings drive that point
home but they aren't the only reason.

But that property of modules is not really a problem in practice.
You have to have a pretty twisted Scheme programming style to run into
a case where it will make a difference.

    > >Mutation-sharing shared substrings are an upwards compatible extension
    > >to the Scheme standard.  They break no correct programs.  They enable
    > >new kinds of programs.

    > Introducing a separate data type for mutation-sharing character arrays 
    > also enables new kinds of programs. The difference between a separate 
    > data type and the former implementation is, that mutation-sharing 
    > substrings could be used everywhere where an ordinary string had been 
    > used before. That is, the difference is a matter of being able to re-use 
    > existing string-handling code rather than enabling new kinds of 
    > programs. However, it is exactly this re-using of existing 
    > string-handling code issue which becomes problematic when the semantics 
    > of the string objects change.

I don't believe that it's problematic in the slightest.   Certainly no
more so than re-using a standard Scheme module in a multi-threaded Scheme.

    > Marius, would it be an acceptable compromise to require that the 
    > mutation-sharing substring issue be submitted and discussed as an srfi 
    > before it becomes an official part of guile's core? The discussion of 
    > the topic in that forum would reduce the risk that the change introduces 
    > problems. I would then ask those who are interested to have it as a part 
    > of guile to submit a srfi proposal.

    > (Please note that, as I have said before, I have nothing against 
    > providing mutation-sharing substrings as a deprecated feature for some 
    > period - but not as an official part of guile's core.)

I guess what really bugs me about the issue is that the feature was
(semi-) removed based on pure speculation, opinion, and apparently by
imperfect interpretation of the standard.   I don't think that there
was a record of the feature causing any serious and sustained
problems.   It's removal seemed rather a rather gratuitous snub.

-t



_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: Worrying development

[Marius Vollmer]

Tom Lord <lord@emf.net> writes:

>     > I'd say that the real 'trouble' is that strings are mutable at
>     > all.
>
> Worried mostly about variable-length character encodings in string?
> Or you'd just rather be programming in an ML-family language? :-)

Heh, no, I'm not really worried, I was actually trying to comment
Dirk's concerns.

>     > Also, I still like the idea of using mutation-sharing substrings as
>     > markers that allow O(1) access into variable-width encoded strings.
>
> Interesting.  The interaction with STRING-SET! will be tricky.  I
> think you'll either have to "timestamp" strings (one tick per mutation
> -- and you'll likely have to use a GC'ed value rather than an inline
> integer for timestamps) or wind up with O(K) for mutations where K is
> the number of shared substrings.

Yes.  What I have in mind is that accessing strings is efficient as
long as no mutations are performed.  I.e., instead of indication
positions in a string with an integer index, you create a shared
substring that starts at the desired position.  (This could be done
with COW substrings, tho.)

>     > Also, there is the possibility on the horizon that we turn
>     > string-ref etc into 'primitive generics' which means that people
>     > could implement new kinds of strings using GOOPS.
>
> Well, heck.  In that case, maybe consider what I'm planning for Pika
> (at least initially).  Purely ASCII strings are stored 1-byte per
> character.  Most other strings 2-bytes per character.   Strings using
> characters outside the Basic Multilingual Plane, 4 bytes per
> character.

Yes, that's an attractive approach.  But I also find simply using
UTF-8 exclusively very attractive.  It might fit better with what
other people are doing and we might need fewer conversions when
wrapping external libraries.  Or maybe not.

-- 
GPG: D5D4E405 - 2F9B BCCC 8527 692A 04E3  331E FAF8 226A D5D4 E405


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: Worrying development

[Tom Lord]

    > From: Marius Vollmer <mvo@zagadka.de>

    > > Well, heck.  In that case, maybe consider what I'm planning for Pika
    > > (at least initially).  Purely ASCII strings are stored 1-byte per
    > > character.  Most other strings 2-bytes per character.   Strings using
    > > characters outside the Basic Multilingual Plane, 4 bytes per
    > > character.

    > Yes, that's an attractive approach.  But I also find simply using
    > UTF-8 exclusively very attractive.  It might fit better with what
    > other people are doing and we might need fewer conversions when
    > wrapping external libraries.  Or maybe not.

In case it helps seduce you to the dark side of the force just a
little more:  

Having wrappings of external libraries mostly rely on
copying/converting strings is a win for thread support.   Having
FFI-using routines directly access or munge string data is, in
general, pretty touchy.   It is, I admit, a total pain in the butt
that so much existing code already does access string data directly --
but for the most part, that code is unlikely to be expecting UTF-8
anyway so......

-t



_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: Worrying development

[rm]

On Fri, Jan 23, 2004 at 05:28:03PM -0500, Paul Jarc wrote:
> Marius Vollmer <mvo@zagadka.de> wrote:
> > Also, there is the possibility on the horizon that we turn string-ref
> > etc into 'primitive generics' which means that people could implement
> > new kinds of strings using GOOPS.
> 
> Neat.  Has/might that also be done for car & cdr?  Then we could have
> Python-like generators.  It would make SCM_C[AD]R less speedy, though
> they could still be pretty fast for actual pairs.

Wow, _that_ would be a real help! I have to build bindings for some 
heavy C++ libs that make excessive use of iterators. A generic 'sequence'
type with the right generics would help a lot -- the bindings would look
much more scheme-ish.

 Ralf Mattes
> 
> paul
> 
> 
> _______________________________________________
> Guile-devel mailing list
> Guile-devel@gnu.org
> http://mail.gnu.org/mailman/listinfo/guile-devel


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: Worrying development

[Marius Vollmer]

rm@fabula.de writes:

> On Fri, Jan 23, 2004 at 05:28:03PM -0500, Paul Jarc wrote:
>> Marius Vollmer <mvo@zagadka.de> wrote:
>> > Also, there is the possibility on the horizon that we turn string-ref
>> > etc into 'primitive generics' which means that people could implement
>> > new kinds of strings using GOOPS.
>> 
>> Neat.  Has/might that also be done for car & cdr?  Then we could have
>> Python-like generators.  It would make SCM_C[AD]R less speedy, though
>> they could still be pretty fast for actual pairs.
>
> Wow, _that_ would be a real help! I have to build bindings for some 
> heavy C++ libs that make excessive use of iterators. A generic 'sequence'
> type with the right generics would help a lot -- the bindings would look
> much more scheme-ish.

Hmm, my immediate reaction is that car/cdr are too low-level for
making them overrideable, but map and for-each and other operations
that work on whole sequences look like good targets...

-- 
GPG: D5D4E405 - 2F9B BCCC 8527 692A 04E3  331E FAF8 226A D5D4 E405


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: Worrying development

[Thien-Thi Nguyen]

   From: Roland Orre <orre@nada.kth.se>
   Date: Fri, 16 Jan 2004 12:34:25 +0100

   [matrix stuff, pthreads] The only thing which
   I'll be dependent on is goops support.

   [gc bits separate from tag bits]

   [gmp]

thanks for elaborating.  i think the matrix stuff and gmp will never
make it into guile 1.4.x core (perhaps it will be accessible as a module
at runtime).  i wonder if simply using unintegrated goops works.

as for pthreads and gc bits separation, i don't know enough (yet) about
the guile internals to say whether or not these changes can be done in a
backward-compatible manner.  we will see...

thi


_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

overriding car/cdr (was: Worrying development)

[Paul Jarc]

Marius Vollmer <mvo@zagadka.de> wrote:
> Hmm, my immediate reaction is that car/cdr are too low-level for
> making them overrideable, but map and for-each and other operations
> that work on whole sequences look like good targets...

Going that route, there will always be one more function that someone
wants to be converted.  Third-party libraries also often won't be able
to handle generated lists without modification.  OTOH, by modifying
SCM_CAR/SCM_CDR, everything that handles lists automatically becomes
able to handle generated lists, and the cost for normal lists is only
the same cost as when compiling with -DSCM_DEBUG_PAIR_ACCESSES=1.


paul


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: overriding car/cdr

[Dirk Herrmann]

Paul Jarc wrote:

>Marius Vollmer <mvo@zagadka.de> wrote:
>  
>
>>Hmm, my immediate reaction is that car/cdr are too low-level for
>>making them overrideable, but map and for-each and other operations
>>that work on whole sequences look like good targets...
>>    
>>
>
>Going that route, there will always be one more function that someone
>wants to be converted.  Third-party libraries also often won't be able
>to handle generated lists without modification.  OTOH, by modifying
>SCM_CAR/SCM_CDR, everything that handles lists automatically becomes
>able to handle generated lists, and the cost for normal lists is only
>the same cost as when compiling with -DSCM_DEBUG_PAIR_ACCESSES=1.
>
The macros SCM_CAR and SCM_CDR should IMO not changed to handle generic 
pairs. In the guile kernel we need a layer to deal with the built-in low 
level data types.

Best regards
Dirk



_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: overriding car/cdr

[Marius Vollmer]

prj@po.cwru.edu (Paul Jarc) writes:

> Marius Vollmer <mvo@zagadka.de> wrote:
>> Hmm, my immediate reaction is that car/cdr are too low-level for
>> making them overrideable, but map and for-each and other operations
>> that work on whole sequences look like good targets...
>
> Going that route, there will always be one more function that someone
> wants to be converted.

But car/cdr are not a good way to work with general sequences.  Think
of vectors.  We should not try hard to turn car/cdr into something
abstract (which they are not, even their names come from the lowest
level).  Somehthing like the 'sequence' concept of Common Lisp is a
better approach, I'd say.

-- 
GPG: D5D4E405 - 2F9B BCCC 8527 692A 04E3  331E FAF8 226A D5D4 E405


_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: overriding car/cdr

[Paul Jarc]

Marius Vollmer <mvo@zagadka.de> wrote:
> But car/cdr are not a good way to work with general sequences.  Think
> of vectors.

car/cdr certainly can be made to work with vectors.  Making them work
with arbitrary user-defined structures would give us a lot of
flexibility without too much work.

If there is a significant amount of C code using SCM_CAR/SCM_CDR with
non-pair objects, then it might be best to leave them unchanged.  But
I think the Scheme-level cxr functions would be a good place to make
the change.

> We should not try hard to turn car/cdr into something abstract
> (which they are not, even their names come from the lowest level).

Of course they currently are not, but that doesn't (nor do the names)
tell us whether it would be good if they were more abstract.

> Somehthing like the 'sequence' concept of Common Lisp is a better
> approach, I'd say.

AFAICT, that would involve a whole new set of procedures to work with
this new data type.  I'm suggesting modifying the existing procedures
so that most existing code would automatically be able to take
advantage of the new flexibility, with no further changes needed.


paul


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: overriding car/cdr

[Marius Vollmer]

prj@po.cwru.edu (Paul Jarc) writes:

> Marius Vollmer <mvo@zagadka.de> wrote:
>> But car/cdr are not a good way to work with general sequences.  Think
>> of vectors.
>
> car/cdr certainly can be made to work with vectors.  [...]

Yes, but would that be a _good_ way to work with them?  It would be,
in my view, only a kluge to make routines work with vectors that were
originally written for lists.  Such automatic code reuse or rather
code-napping doesn't sound like a good idea to me.

>> Somehthing like the 'sequence' concept of Common Lisp is a better
>> approach, I'd say.
>
> AFAICT, that would involve a whole new set of procedures to work with
> this new data type.  I'm suggesting modifying the existing procedures
> so that most existing code would automatically be able to take
> advantage of the new flexibility, with no further changes needed.

What would the advantage be?  Some of the existing list routines will
sort of work with vectors, but vectors are not lists, and the results
will be strange at best.  Lists in Scheme and Lisp are not merely
sequences, they are able to form general trees with all kinds of
intentional structure sharing.  Vectors are not at all like this.

What would is (cons (car vec) (cdr vec))?  A pair with a vector as its
second argument?  A vector?

Some lists are uses as sequences and it would indeed make sense to
formalize this by introducing an abstract 'sequence' type for this,
I'd say.  Maybe by going over SRFI-1 and picking out the procedures
that treat lists as sequences.  Maybe such a thing already exists.

-- 
GPG: D5D4E405 - 2F9B BCCC 8527 692A 04E3  331E FAF8 226A D5D4 E405


_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: overriding car/cdr

[David Van Horn]

Marius Vollmer wrote:
> Some lists are uses as sequences and it would indeed make sense to
> formalize this by introducing an abstract 'sequence' type for this,
> I'd say.  Maybe by going over SRFI-1 and picking out the procedures
> that treat lists as sequences.  Maybe such a thing already exists.

You might have a look at SRFI 44: Collections.

    http://srfi.schemers.org/srfi-44/

David


_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: overriding car/cdr

[Paul Jarc]

Marius Vollmer <mvo@zagadka.de> wrote:
> Lists in Scheme and Lisp are not merely sequences, they are able to
> form general trees with all kinds of intentional structure sharing.
> Vectors are not at all like this.

Hmm, good point.

> Some lists are uses as sequences and it would indeed make sense to
> formalize this by introducing an abstract 'sequence' type for this,
> I'd say.

Yes, that might be the best way.

Ideally, I think, programmers shouldn't have to worry about list
vs. vector representation of sequence objects any more than they have
to worry about memory management.  We ought to be able to say "give me
an object that supports these operations, and favors these certain
operations for performance", and let the computer figure out what
representation is best.


paul


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: overriding car/cdr

[Marius Vollmer]

David Van Horn <dvanhorn@cs.uvm.edu> writes:

> You might have a look at SRFI 44: Collections.
>
>     http://srfi.schemers.org/srfi-44/

Yes, exactly!

-- 
GPG: D5D4E405 - 2F9B BCCC 8527 692A 04E3  331E FAF8 226A D5D4 E405


_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user