New module system option :duplicates

New module system option :duplicates

[Mikael Djurfeldt]

Hi,

I've just committed the following experimental features.  Please try
them out and comment on if you they are a good idea.

The major weakness is of course the dynamicity of the merge-generics
handler, but note that such a handler could very well operate during
compile-time if module interfaces have typed signatures.

I'd like to add that even though automatic merging of generic
functions seems strange, my own experience is that it leads to very
nice, easily read, and intuitive code.

* Changes to Scheme functions and syntax

** Checking for duplicate bindings in module system

The module system now can check for duplicate imported bindings.
The syntax to enable this feature is:

(define-module (foo)
  :use-module (bar)
  :use-module (baz)
  :duplicates check)

This will report an error if both (bar) and (baz) exports a binding
with the same name.

The syntax for the :duplicates option is:

  :duplicates HANDLER-NAME | (HANDLER1-NAME HANDLER2-NAME ...)

Specifying multiple handlers is useful since some handlers (such as
merge-generics) can defer conflict resolution to others.

Currently available duplicates handlers are:

  check		  report an error for bindings with a common name
  first		  select the first encountered binding (override)
  last		  select the last encountered binding (override)
  merge-generics  merge generic functions with a common name
		  into an <extended-generic>

** Merging generic functions

It is sometimes tempting to use GOOPS accessors with short names.
For example, it is tempting to use the name `x' for the x-coordinate
in vector packages.

Assume that we work with a graphical package which needs to use two
independent vector packages for 2D and 3D vectors respectively.  If
both packages export `x' we will encounter a name collision.

This can now be resolved with the duplicates handler `merge-generics'
which merges all generic functions with a common name:

(define-module (math 2D-vectors)
  :use-module (oop goops)
  :export (x y ...))
		  
(define-module (math 3D-vectors)
  :use-module (oop goops)
  :export (x y z ...))

(define-module (my-module)
  :use-module (math 2D-vectors)
  :use-module (math 3D-vectors)
  :duplicates merge-generics)

x in (my-module) will now share methods with x in both imported
modules.

The detailed rule for method visibility is this:

Let's call the imported generic functions the "ancestor functions".
x in (my-module) is, in turn, a "descendant function" of the imported
functions.  For any generic function gf, the applicable methods are
selected from the union of the methods of the descendant functions,
the methods of gf and the methods of the ancestor functions.

This implies that x in (math 2D-vectors) can see the methods of x in
(my-module) and vice versa, while x in (math 2D-vectors) doesn't see
the methods of x in (math 3D-vectors), thus preserving modularity.

If duplicates checking is desired in the above example, the following
form of the :duplicates option can be used instead:

  :duplicates (merge-generics check)



_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: New module system option :duplicates

[tomas]

On Fri, Mar 07, 2003 at 02:19:19PM +0100, Mikael Djurfeldt wrote:
> Hi,
> 
> I've just committed the following experimental features.  Please try
> them out and comment on if you they are a good idea.

[:duplicates]

YES!

thanks
-- tomas


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: New module system option :duplicates

[Marius Vollmer]

Mikael Djurfeldt <mdj@kvast.blakulla.net> writes:

> I've just committed the following experimental features.  Please try
> them out and comment on if you they are a good idea.

I can understand the motivation for merging generics very, very well,
but I still am unhappy about its conflation with the module system.  I
think the merging and in general the resolution of duplicates should
be controllable more precisly.  (See below for what I mean by this.)


It is unfortunate that our current module system silently ignores
duplicates, or "collisions".  Sooner or later, we should move to a
more 'disciplined' system.

The core model of such a system (but not its UI) is in the text
workbook/compilation/new-model.text.  Mikael, could you try to
consolidate your duplicate checking with the model laid out in that
text?  I.e., can your :duplicate option be implemented on top of that
model?

I think we should not add features to the module system that can not
be implemented 'naturally' in the idealized compilation framework in
new-model.text.  Maybe the model needs to be changed, or maybe the
:duplicate option needs to be done differently, ut they both should
fit together conceptually.

Thinking superficially about it, I would like the following
alternatives better, since they are more precise.  They are also more
cumbersome, but I do think that it pays to be more long-winded at
times in order to be more robust with regard to a changing external
environment.

For real name collisions that should be resolved, I would prefer
renaming or selective imports over just specifying to take the first.
I don't think we have an elegant way to import all-but-a-selected-few
bindings from a module, but adding it should not be a problem.

So instead of

   (define-module (foo)
     :use-module (blarg)
     :use-module (bar)     ; exports mumble
     :use-module (baz)     ; also exports mumble
     :duplicates first)    ; but we choose the first one

I would prefer

   (define-module (foo)
     :use-module (blarg)
     :use-module (bar)                          ; exports mumble
     :use-module ((baz) :dont-select (mumble))  ; also exports mumble
                                                ; but we don't want it
     :duplicates check)                         ; and we expect no duplicates

where the ":duplicate check" option is ideally the default.

The former variant with ":duplicate first" has the 'race condition'
that (blarg) might at some time start to export mumble as well and we
will silently pick that instead of the one from (bar).  The second
variant will also fail since there are now duplicates, but it will
give an error.

Also (and I think new-model.text will have to change for this), we
might want to have the concept of "potentially imported" bindings:
when importing a set of bindings with unspecified names (such as "all
of (bar)" as opposed to "the binding with name mumble of (bar)"),
these bindings are only potentially visible.  Only when a binding is
looked up in the current module will these bindings be considered and
only then wehn none of the "actually imported" bindings matches.  When
a potentially imported binding is found in this way, it becomes
actually imported.  When more than one potentially imported binding
matches, an error is reported (or the first one is selected in 'legacy
mode').

Such a model can easily support an interface like

   (define-module (foo)
     :use-module (blarg)
     :use-module ((bar) :select (mumble))  ; explicitely import mumble
     :use-module (baz)                     ; also imports mumble
                                           ; but only implicitely and
                                           ; since we have an explicit
                                           ; mumble, we won't see it
     :duplicates check)                    ; we expect no explicit duplicates


So in my view only ":duplicates check" should be added, and ideally be
made the default.  How to orchestrate the transition is another
issue...


For merging generics, I would like to split the task in two: first,
getting the generics into the current module without conflicts; and
second, merging multiple generics into one.

The first part is indepenent from the fact that generics are involved
and deals only with the module system, while the second part has
nothing to do with colliding names, and deals only with generics.

I think this distinction is important, at least conceptually.  Your
:duplicates option respects this distinction to a largish degree, but
it still hooks into the module system at quite a deep level where the
separation of the two issues is not easy to see.  Also, the user has
the same low amount of precision when dealing with it as with the
general collision avoidance.  Just giving the license to merge any and
all generics that have colliding names is probably not what one wants.
One usualy has a specific few in mind that need to be merged, I think.

Given that we have a way to get multiple conflicting symbols into the
current module (via renaming), what about leaving generic merging to a
separate facility like 'merge-prefix-generics'.  Actual code might
look like:

  (define-module (my-module)
    :use-module ((math 2D-vectors) :prefixed 2d)
    :use-module ((math 3D-vectors) :prefixed 3d)
    :duplicates check)

  ;; merge 2d-x and 3d-x into x.  Likewise for y and z.
  (merge-prefix-generics (2d 3d) x y z)

This is not much more tedious than ":duplicates merge-generics" but
much more precise.  No unexpected mergings are done and the remaining
collisions of non-generics are also dealt with in a useful way.

Also, merging of generic functions might be a useful thing to do, even
without using the module system explicitely.

> The major weakness is of course the dynamicity of the merge-generics
> handler, but note that such a handler could very well operate during
> compile-time if module interfaces have typed signatures.

The compiler should likewise be able to handle the
merge-prefix-generics macro.

> * Changes to Scheme functions and syntax
> 
> ** Checking for duplicate bindings in module system
> The module system now can check for duplicate imported bindings.
> The syntax to enable this feature is:
> 
> (define-module (foo)
>   :use-module (bar)
>   :use-module (baz)
>   :duplicates check)
> 
> This will report an error if both (bar) and (baz) exports a binding
> with the same name.

What about adding:

  Eventually, we want to make ":duplicates check" the default.


In summary, I think that ":duplicates check" is a very good thing, but
I'm unconvinced about the rest of the handlers.  I'd rather see
improvements to the selection/renaming mechanism.

Does this make sense, Mikael?



(one more minor comment)

> The detailed rule for method visibility is this:

I think it is wrong to talk about "method visibility".  The visibility
of methods is not an important feature, one might even argue that
methods don't have names at all.

The thing that matters is the set of methods belonging to a certain
generic function.  This set is unaffected by visibility issues.

(I'm not saying that you don't understand methods and generic
functions, Mikael, of course you do, more than anyone else here.  But
still the use of terminology doesn't fit my mental model of generic
functions and their methods.)

-- 
GPG: D5D4E405 - 2F9B BCCC 8527 692A 04E3  331E FAF8 226A D5D4 E405


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: New module system option :duplicates

[Mikael Djurfeldt]

Thank you very much for your detailed reply!

Unfortunately, I'm very, very busy right now, so you'll have to excuse
me answering for real a little later.

Marius Vollmer <mvo@zagadka.de> writes:

> Just giving the license to merge any and all generics that have
> colliding names is probably not what one wants.

Actually, I'd like to have the option to do exactly that.  In my view,
we can reasonably safely regard the merging as an administrative
non-issue for most programs.

> One usualy has a specific few in mind that need to be merged, I think.

I'd very much like to be able to avoid explicitly listing these.
While I have a reasonable grasp of the meaning of object accessors
taken one by one, I usually have a very poor grasp of the intersection
between the sets of exported accessors from two modules.

Other than that, I very much understand your objections and agree with
them to a large extent.

I'll look through your letter carefully later and see if it's possible
to provide for both alternatives (license to merge + explicit
listing), while still being compatible with your workbook module
system proposal.  (I should have read that before implementing
duplicates, but I needed this working quickly! :)

Thanks,
Mikael


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: New module system option :duplicates

[Marius Vollmer]

Mikael Djurfeldt <djurfeldt@nada.kth.se> writes:

> Marius Vollmer <mvo@zagadka.de> writes:
> 
> > Just giving the license to merge any and all generics that have
> > colliding names is probably not what one wants.
> 
> Actually, I'd like to have the option to do exactly that.  In my view,
> we can reasonably safely regard the merging as an administrative
> non-issue for most programs.

I see, yes.  Unasked merging of generics has usually very few if any
negative consequences.

But still, you can not really savely use a generic that has been
merged from two generics without your explicit consent.

The typical situation would be that one part of a module intents to
use the one generic, while another part (written later or by a
different person) intents to use a second generic that happens to have
the same name.  The second part will likely add another ":use-module"
clause and ":duplicates merge-generics" will silently merge the two
generics.

When the two generics have non-overlapping type signatures for their
signatures, all is well.  But you can't be sure about this.  You need
to be made aware that there is a collision/merging going on and then
you can declare that this specific merge is OK.  

I guess it is next to impossible to automatically check whether
overlapping type signatures can be merged the Right Way.
  
-- 
GPG: D5D4E405 - 2F9B BCCC 8527 692A 04E3  331E FAF8 226A D5D4 E405


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: New module system option :duplicates

[Rob Browning]

Mikael Djurfeldt <mdj@kvast.blakulla.net> writes:

> The module system now can check for duplicate imported bindings.
> The syntax to enable this feature is:
>
> (define-module (foo)
>   :use-module (bar)
>   :use-module (baz)
>   :duplicates check)
>
> This will report an error if both (bar) and (baz) exports a binding
> with the same name.

Interesting.  Though I had thought that we were already heading in the
direction of being less tolerant of duplicate bindings as a default.

In any case, I would definitely be interested in thinking about
special treatment for generic functions.  Does anyone know offhand
what other systems do in this regard, say CLOS for example?  Naively,
it doesn't seem like (open <file>) should conflict with (open
<gripper>), but I haven't thought about the issue enough yet to feel
confident about that.

Also, doesn't ttn already have an augmented module system syntax for
renaming, etc.?  If so, and if people are using it much, then it might
be worth considering that syntax as well, and I seem to recall that
Marius may have been planning something on this front too...

> This implies that x in (math 2D-vectors) can see the methods of x in
> (my-module) and vice versa, while x in (math 2D-vectors) doesn't see
> the methods of x in (math 3D-vectors), thus preserving modularity.

Why would x in (math 2D-vectors) be able to see the methods of x in
(my-module)?  Isn't that a violation of modularity?  The reverse I
understand.

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG starting 2002-11-03 = 14DD 432F AE39 534D B592  F9A0 25C8 D377 8C7E 73A4


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: New module system option :duplicates

[Rob Browning]

Marius Vollmer <mvo@zagadka.de> writes:

> When the two generics have non-overlapping type signatures for their
> signatures, all is well.  But you can't be sure about this.  You
> need to be made aware that there is a collision/merging going on and
> then you can declare that this specific merge is OK.

That's what I just thought about too.  i.e. I'd guess that merging
(open <file>) and (open <gripper>) would be just fine, but what about
merging (open <file>) from module-1 with (open <file>) from module-2?
Personally, I'd probably want to see an error in the latter case, but
the question is in general, how would a "conflict" be defined?

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG starting 2002-11-03 = 14DD 432F AE39 534D B592  F9A0 25C8 D377 8C7E 73A4


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: New module system option :duplicates

[Greg Troxel]

   That's what I just thought about too.  i.e. I'd guess that merging
   (open <file>) and (open <gripper>) would be just fine, but what about
   merging (open <file>) from module-1 with (open <file>) from module-2?
   Personally, I'd probably want to see an error in the latter case, but
   the question is in general, how would a "conflict" be defined?

If one thinks of all procedures as generics, then adding a method with
specializers that does not conflict is different from one that does
conflict.  But, the definition of conflict has two interesting
wrinkles:

  If one defines two methods, and one is more specialized than the
  other, but they are not disjoint, then the second definition will
  modify the result of using the first definitions generic function.
  For example:
    guile> (use-modules (oop goops))
    guile> (define-method (f (a <number>)) (+ a 1))
    guile> (f 1)
    2
    guile> (f 1.0)
    2.0
    guile> (define-method (f (a <integer>)) (+ a 2))
    guile> (f 1)
    3
    guile> (f 1.0)
    2.0
  Here, the methods do not 'conflict' in the sense of specifying
  different behaviors for the same provided specializers, but they are
  not disjoint either.

The proposal of an extended generic which finds methods from several
generics seems nice.  In the case that Mikael talked about, it seemed
the methods were all disjoint, so this works cleanly.  With
non-disjoint methods, perhaps an error should be signalled, or this
should be treated the same way as an export-export conflict.

Mikael's proposal of merging methods but not modifying values also
avoids another nasty issue:

  define-generic overwrites previous generic function values and
  non-procedure values, but not previous simple procedures.  So if
  either module uses define-generic, there is no way to merge the
  values into a single generic function.

All that said, it troubles me somewhat to have a new extended-generic
type.  Is it possible to just

  construct a new generic function

  add all the methods from each of the generics in the modules to the
  new generic function

and be done with special treatment?

Finally, I did not understand

  This implies that x in (math 2D-vectors) can see the methods of x in
  (my-module) and vice versa, while x in (math 2D-vectors) doesn't see
  the methods of x in (math 3D-vectors), thus preserving modularity.

Why would any method in (math 2D-vectors), or use of gf x, ever see
methods defined in my-module?  This seems like a clear modularity
violation, but I suspect I'm confused.

I think a core underlying issue is that define-method is a mutator on
the gf (perhaps it needs a ! :-):

  guile> f
  #<<generic> f (2)>
  guile> (generic-function-methods f)
  (#<<method> (<integer>) 80902b0> #<<method> (<number>) 8090310>)
  guile> (define-method (f (a <complex>)) (+ a 3))
  guile> (generic-function-methods f)             
  (#<<method> (<complex>) 80905a0> #<<method> (<integer>) 80902b0>
  #<<method> (<number>) 8090310>)

So, perhaps _always_ exporting generic functions by creating a new
generic and adding the methods is the proper path.  Normally modules
only export procedures, and they are immutable - I would expect
calling set! on the exported name to rebind the name in the using
module, and not affect the used module.  Exporting variables of course
can lead to mutation, but this is far clearer and less likely to lead
to unexpected trouble.

        Greg Troxel <gdt@ir.bbn.com>


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: New module system option :duplicates

[Mikael Djurfeldt]

Marius Vollmer <mvo@zagadka.de> writes:

> The core model of such a system (but not its UI) is in the text
> workbook/compilation/new-model.text.  Mikael, could you try to
> consolidate your duplicate checking with the model laid out in that
> text?  I.e., can your :duplicate option be implemented on top of that
> model?

I've now read new-model.text and think it is very good.  I have some
smaller comments, though, which I'll give in a separate letter.

If I haven't missed something, there are no problems with implementing
the current :duplicate option within the framework of new-model.text.
In fact, while I had expected to need to use type information supplied
by the module system, your suggested framework allows the current
processing of the :duplicates option to continue to work in the same
way as currently.

The processing of the :duplicates option will be done in
process-define-module-for-compiler or process-define-module (as is
currently the case).  If I've understood your model correctly, this is
also where the partially constructed module processes its list of
imported modules.  When duplicate imported bindings are found, a set
of new bindings (an extra interface) is inserted at the front of the
use-list.  In the case of merging generic functions, these bindings
will have their own variables containing their own
<extended-generic>:s.  Thus, the compiler will never notice anything
unusual.  It will look just as if the generic function is imported
normally from a single module on the use-list.  (And this is, in fact,
exactly how merging is done in the current implementation.)

> Also (and I think new-model.text will have to change for this), we
> might want to have the concept of "potentially imported" bindings:
> when importing a set of bindings with unspecified names (such as "all
> of (bar)" as opposed to "the binding with name mumble of (bar)"),
> these bindings are only potentially visible.  Only when a binding is
> looked up in the current module will these bindings be considered and
> only then wehn none of the "actually imported" bindings matches.  When
> a potentially imported binding is found in this way, it becomes
> actually imported.  When more than one potentially imported binding
> matches, an error is reported (or the first one is selected in 'legacy
> mode').
>
> Such a model can easily support an interface like
>
>    (define-module (foo)
>      :use-module (blarg)
>      :use-module ((bar) :select (mumble))  ; explicitely import mumble
>      :use-module (baz)                     ; also imports mumble
>                                            ; but only implicitely and
>                                            ; since we have an explicit
>                                            ; mumble, we won't see it
>      :duplicates check)                    ; we expect no explicit duplicates

Do I understand this correctly in that you suggest having bindings
with two "priority levels": "explicitly imported" = priority 1,
"implicitly imported" = priority 2.  Only duplicate bindings with
priority 1 causes an error (:duplicates check) while other duplicate
pairs are resolved (at lookup-time) according to :duplicates first
with precedence for the binding of higher priority.

Hmm... Isn't this a little conceptually complex?  What is the gain of
differing between "potentially imported" and "actually imported"?
It seems like a nice thing to be able to determine the set of imported
bindings when the module is constructed.

Is the aim to support the current "lazy binding" technique used in
guile-tcltk?

> So in my view only ":duplicates check" should be added, and ideally be
> made the default.  How to orchestrate the transition is another
> issue...

My thought in adding "first" and "last" was to have a way of capturing
the current behavior of the module system explicitly, and to satisfy
the group of people who might like the "override" behavior.

Personally, I prefer "check", although the recently added "warn" might
give a more smooth behavior for users who like it.  E.g.

  :duplicates (warn last)

would behave as the current default, but, in addition, emit a warning.

Regarding the transition: What about making "check" the default
duplicates handler but provide a form which the user can put in
his/her .guile if he/she want to be backward compatible?

  (default-module-duplicates-handler 'last)

I've actually experimentally done and committed this.  (See further
below in the included NEWS entry.)

> Given that we have a way to get multiple conflicting symbols into the
> current module (via renaming), what about leaving generic merging to a
> separate facility like 'merge-prefix-generics'.  Actual code might
> look like:
>
>   (define-module (my-module)
>     :use-module ((math 2D-vectors) :prefixed 2d)
>     :use-module ((math 3D-vectors) :prefixed 3d)
>     :duplicates check)
>
>   ;; merge 2d-x and 3d-x into x.  Likewise for y and z.
>   (merge-prefix-generics (2d 3d) x y z)
>
> This is not much more tedious than ":duplicates merge-generics" but
> much more precise.  No unexpected mergings are done and the remaining
> collisions of non-generics are also dealt with in a useful way.

I like this.  However, I'd like to provide this facility *in addition*
to giving the license to merge.

Given that we already had `define-extended-generic' I've now added

  (define-extended-generics (x y z) #:prefix (2d: 3d:))

I chose this form because I have the feeling that we might want to
have other additional ways of selecting the generics in the future.

Please tell me if you like your `merge-prefix-generics' better or if
you have a third suggestion.

> Also, merging of generic functions might be a useful thing to do, even
> without using the module system explicitely.

Yes.  We've had `define-extended-generic' for a while, but I have not
come to documenting it in NEWS.  I'll fix my old sins ASAP.

> What about adding:
>
>   Eventually, we want to make ":duplicates check" the default.
>

Good idea.  As you can see below, the current default is

  (replace warn-override-core check)

> In summary, I think that ":duplicates check" is a very good thing, but
> I'm unconvinced about the rest of the handlers.  I'd rather see
> improvements to the selection/renaming mechanism.
>
> Does this make sense, Mikael?

Given my recent commits, I guess I have to return the question.  Do
you think my recent additions make sense?

>> The detailed rule for method visibility is this:
>
> I think it is wrong to talk about "method visibility".  The visibility
> of methods is not an important feature, one might even argue that
> methods don't have names at all.
>
> The thing that matters is the set of methods belonging to a certain
> generic function.  This set is unaffected by visibility issues.
>
> (I'm not saying that you don't understand methods and generic
> functions, Mikael, of course you do, more than anyone else here.  But
> still the use of terminology doesn't fit my mental model of generic
> functions and their methods.)

Agreed.  I'll change the terminology.

Best regards,
Mikael

Excerpt of new NEWS:

** Checking for duplicate bindings in module system

The module system now checks for duplicate imported bindings.

The behavior can be controlled by specifying one or more duplicates
handlers.  For example, to get back the old behavior (which was to use
the last imported binding of a certain name), write:

(define-module (foo)
  :use-module (bar)
  :use-module (baz)
  :duplicates last)

If you want the old behavior without changing your module headers, put
the line:

  (default-module-duplicates-handler 'last)

in your .guile init file.

The syntax for the :duplicates option is:

  :duplicates HANDLER-NAME | (HANDLER1-NAME HANDLER2-NAME ...)

Specifying multiple handlers is useful since some handlers (such as
replace) can defer conflict resolution to others.  Each handler is
tried until a binding is selected.

Currently available duplicates handlers are:

  check		     report an error for bindings with a common name
  warn		     issue a warning for bindings with a common name
  replace	     replace bindings which have an imported replacement
  warn-override-core issue a warning for imports which override core bindings
  first		     select the first encountered binding (override)
  last		     select the last encountered binding (override)
  merge-generics     merge generic functions with a common name
		     into an <extended-generic>

The default duplicates handler is:

  (replace warn-override-core check)

** New define-module option: :replace

:replace works as :export, but, in addition, marks the binding as a
replacement.

A typical example is `format' in (ice-9 format) which is a replacement
for the core binding `format'.

** Merging generic functions

It is sometimes tempting to use GOOPS accessors with short names.
For example, it is tempting to use the name `x' for the x-coordinate
in vector packages.

Assume that we work with a graphical package which needs to use two
independent vector packages for 2D and 3D vectors respectively.  If
both packages export `x' we will encounter a name collision.

This can now be resolved automagically with the duplicates handler
`merge-generics' which gives the module system license to merge all
generic functions sharing a common name:

(define-module (math 2D-vectors)
  :use-module (oop goops)
  :export (x y ...))
		  
(define-module (math 3D-vectors)
  :use-module (oop goops)
  :export (x y z ...))

(define-module (my-module)
  :use-module (math 2D-vectors)
  :use-module (math 3D-vectors)
  :duplicates merge-generics)

x in (my-module) will now share methods with x in both imported
modules.

There will, in fact, now be three distinct generic functions named
`x': x in (2D-vectors), x in (3D-vectors), and x in (my-module).  The
last function will be an <extended-generic>, extending the previous
two functions.

Let's call the imported generic functions the "ancestor functions".  x
in (my-module) is, in turn, a "descendant function" of the imported
functions, extending its ancestors.

For any generic function G, the applicable methods are selected from
the union of the methods of the descendant functions, the methods of G
itself and the methods of the ancestor functions.

This, ancestor functions share methods with their descendants and vice
versa.  This implies that x in (math 2D-vectors) can will share the
methods of x in (my-module) and vice versa, while x in (math 2D-vectors)
doesn't share the methods of x in (math 3D-vectors), thus preserving
modularity.

Sharing is dynamic, so that adding new methods to a descendant implies
adding it to the ancestor.

If duplicates checking is desired in the above example, the following
form of the :duplicates option can be used instead:

  :duplicates (merge-generics check)



_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: New module system option :duplicates

[Mikael Djurfeldt]

Rob Browning <rlb@defaultvalue.org> writes:

> Interesting.  Though I had thought that we were already heading in the
> direction of being less tolerant of duplicate bindings as a default.

Yeah.  Default now.

> Also, doesn't ttn already have an augmented module system syntax for
> renaming, etc.?  If so, and if people are using it much, then it might
> be worth considering that syntax as well, and I seem to recall that
> Marius may have been planning something on this front too...

Marius has implemented renaming.  I just augmented it so you can say:

  :use-module ((foo) :prefix foo:)
  
I'll document that in NEWS tomorrow.

>> This implies that x in (math 2D-vectors) can see the methods of x in
>> (my-module) and vice versa, while x in (math 2D-vectors) doesn't see
>> the methods of x in (math 3D-vectors), thus preserving modularity.
>
> Why would x in (math 2D-vectors) be able to see the methods of x in
> (my-module)?  Isn't that a violation of modularity?  The reverse I
> understand.

Think about dynamic method dispatch.  A `paint' method in a generic
graphics package need to see subclasses provided by modules importing
the generic package.

M


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: New module system option :duplicates

[Mikael Djurfeldt]

I promise a reply tomorrow.  Good night!


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: New module system option :duplicates

[Rob Browning]

Mikael Djurfeldt <djurfeldt@nada.kth.se> writes:

>> Why would x in (math 2D-vectors) be able to see the methods of x in
>> (my-module)?  Isn't that a violation of modularity?  The reverse I
>> understand.
>
> Think about dynamic method dispatch.  A `paint' method in a generic
> graphics package need to see subclasses provided by modules importing
> the generic package.

Hmm.  Though I haven't thought carefully about it yet, that still
feels like somewhat a modularity violation to me.

Say you have a working module called (math-frob) that defines/uses
some generic-function, say "do-fancy-+", that at the time (foo random)
was designed only had a method for ((x <number>) (y <number>))
arguments.

Then say someone writes module (bar) that defines do-fancy-+ for ((x
<integer>) (y <integer>)) arguments, but does it in a way that's not
compatibile with the do-fancy-+ usages in (foo random).  If someone
does

  (use-modules (foo random))
  (use-modules (bar))

then is (foo random) now broken, and if so, is that to be expected/OK?
i.e. does this just fall under the "well don't do that then" category?

(I still need to go look and see how CLOS handles such a situation...)

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG starting 2002-11-03 = 14DD 432F AE39 534D B592  F9A0 25C8 D377 8C7E 73A4


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: New module system option :duplicates

[Mikael Djurfeldt]

Rob Browning <rlb@defaultvalue.org> writes:

> Mikael Djurfeldt <djurfeldt@nada.kth.se> writes:
>
>>> Why would x in (math 2D-vectors) be able to see the methods of x in
>>> (my-module)?  Isn't that a violation of modularity?  The reverse I
>>> understand.
>>
>> Think about dynamic method dispatch.  A `paint' method in a generic
>> graphics package need to see subclasses provided by modules importing
>> the generic package.
>
> Hmm.  Though I haven't thought carefully about it yet, that still
> feels like somewhat a modularity violation to me.
>
> Say you have a working module called (math-frob) that defines/uses
> some generic-function, say "do-fancy-+", that at the time (foo random)
> was designed only had a method for ((x <number>) (y <number>))
> arguments.
>
> Then say someone writes module (bar) that defines do-fancy-+ for ((x
> <integer>) (y <integer>)) arguments, but does it in a way that's not
> compatibile with the do-fancy-+ usages in (foo random).  If someone
> does
>
>   (use-modules (foo random))
>   (use-modules (bar))
>
> then is (foo random) now broken

It is not.  Since do-fancy-+ in (bar) is neither descendant nor
ancestor of do-fancy-+ in (foo random), do-fancy-+ in (foo random)
won't share methods with do-fancy-+ in (bar), so both functions will
continue to work unaffectedly.

> i.e. does this just fall under the "well don't do that then" category?

The driving need behind my suggested automatic merging of generics is
to enable different imported modules to tell how to do the "x"-thing
on their provided new type of object.  In particular, it enables using
shorter names for the multitude of object accessors exported from
modules.

It is probably an error to import two generic functions which do
different things on the same type of object, and we may choose to
detect this error during the merge.  One could, for example, have
certain requirements on the disjointedness of the inheritance
hierarchies of the classes which the merged functions are specialized
to.

Still, this is a good reason to use the programming style suggested by
Marius: to explicitly state which functions should be merged so that
we have the guarantee that the merge is intentional.  I have now
provided the facilities for using Marius' style:
`define-extended-generics' and the :use-modules option :prefix.

However, I'd very much like to try out this "automatic merge"
programming style to see how it works out in practise in large
projects.  This is why I have provided it as a user option---the
default behavior is not to merge generic functions automatically.

If it turns out that implicit merge of generics causes lots of bugs,
one possible alternativ is to provide :merge-accessors which only
automatically merges object accessors.

> (I still need to go look and see how CLOS handles such a situation...)

The following paper is a good read:

http://kvast.blakulla.net/mdj/kiczales92.pdf

M


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: New module system option :duplicates

[Mikael Djurfeldt]

Rob Browning <rlb@defaultvalue.org> writes:

> Hmm.  Though I haven't thought carefully about it yet, that still
> feels like somewhat a modularity violation to me.
>
> Say you have a working module called (math-frob) that defines/uses
> some generic-function, say "do-fancy-+", that at the time (foo random)
> was designed only had a method for ((x <number>) (y <number>))
> arguments.
>
> Then say someone writes module (bar) that defines do-fancy-+ for ((x
> <integer>) (y <integer>)) arguments, but does it in a way that's not
> compatibile with the do-fancy-+ usages in (foo random).  If someone
> does
>
>   (use-modules (foo random))
>   (use-modules (bar))
>
> then is (foo random) now broken, and if so, is that to be expected/OK?
> i.e. does this just fall under the "well don't do that then" category?

I'd like to add to what I wrote in my previous reply that what you
describe above is not a new kind of problem introduced by the
:merge-generics possibility.  Consider the following code which is
realizable in old version of Guile and in CLOS:

(define-module (baz)
  :export (do-fancy-+))

(define-generic do-fancy-+)

(define-module (foo random)
  :use-module (baz))

(define-method (do-fancy-+ (x <integer>) (y <integer>)) ...)

(define-module (bar)
  :use-module (baz))

(define-method (do-fancy-+ (x <integer>) (y <integer>)) ...)

The Kiczales paper suggest design principles and a certain disciplin
which can be used to avoid these kind of problems.


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: New module system option :duplicates

[Mikael Djurfeldt]

Greg Troxel <gdt@ir.bbn.com> writes:

> All that said, it troubles me somewhat to have a new extended-generic
> type.  Is it possible to just
>
>   construct a new generic function
>
>   add all the methods from each of the generics in the modules to the
>   new generic function
>
> and be done with special treatment?
>
> Finally, I did not understand
>
>   This implies that x in (math 2D-vectors) can see the methods of x in
>   (my-module) and vice versa, while x in (math 2D-vectors) doesn't see
>   the methods of x in (math 3D-vectors), thus preserving modularity.
>
> Why would any method in (math 2D-vectors), or use of gf x, ever see
> methods defined in my-module?

These two things have a common answer: The GF represents some
generalized action and the methods specializes this action to certain
types.

Example:

An object-oriented graphics package (graphics object) exports the GF
`draw'.  So does the pixel-oriented graphics package (graphics paint).
These two packages are developed independently by different developers
but use a common representation of the output device, provided by a
third module.

Now we're developing a module (graphics object picture) which provides
the class <picture> which is a subclass of <graphics-object> exported
by (graphics object).  Internally, <picture> objects store a <bitmap>
from the (graphics paint) module.  Therefore (draw picture) internally
calls (draw bitmap).  This works, because `draw' in (graphics object
picture) is an <extended-generic> which shares methods with draw in
(graphics object) and draw in (graphics paint).

However, when the X window is redisplayed and all objects need to be
redrawn, the update function in the (graphics object) package calls
`draw' on each <graphics-object> on the list of visible objects.  This
may include our <picture> object, so it is essential that draw in
(graphics object) shares the (draw <picture>) method with draw in
(graphics object picture).

Does this answer your questions?

M


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: New module system option :duplicates

[Mikael Djurfeldt]

Mikael Djurfeldt <djurfeldt@nada.kth.se> writes:

> Example:
>
> An object-oriented graphics package (graphics object) exports the GF
> `draw'.  So does the pixel-oriented graphics package (graphics paint).
> These two packages are developed independently by different developers
> but use a common representation of the output device, provided by a
> third module.
>
> Now we're developing a module (graphics object picture) which provides
> the class <picture> which is a subclass of <graphics-object> exported
> by (graphics object).  Internally, <picture> objects store a <bitmap>
> from the (graphics paint) module.  Therefore (draw picture) internally
> calls (draw bitmap).  This works, because `draw' in (graphics object
> picture) is an <extended-generic> which shares methods with draw in
> (graphics object) and draw in (graphics paint).
>
> However, when the X window is redisplayed and all objects need to be
> redrawn, the update function in the (graphics object) package calls
> `draw' on each <graphics-object> on the list of visible objects.  This
> may include our <picture> object, so it is essential that draw in
> (graphics object) shares the (draw <picture>) method with draw in
> (graphics object picture).

Let me add that this is a quite bad example.  In the case above, the
natural design would be to have the module representing the output
device export `draw' and let both (graphics object) and (graphics
paint) add their methods on it.

Note though, that there are many cases where we can't do that: We are
often in the position that we simply want to re-use eisting packages.


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: New module system option :duplicates

[Greg Troxel]

Thanks for the example.

  An object-oriented graphics package (graphics object) exports the GF
  `draw'.  So does the pixel-oriented graphics package (graphics paint).
  These two packages are developed independently by different developers
  but use a common representation of the output device, provided by a
  third module.

OK - this is a sensible and realistic scenario.

  Now we're developing a module (graphics object picture) which provides
  the class <picture> which is a subclass of <graphics-object> exported
  by (graphics object).  Internally, <picture> objects store a <bitmap>
  from the (graphics paint) module.  Therefore (draw picture) internally
  calls (draw bitmap).  This works, because `draw' in (graphics object
  picture) is an <extended-generic> which shares methods with draw in
  (graphics object) and draw in (graphics paint).

The two draw methods were developed independently, and are thus in
general not safe to merge.  (graphics object picture) imports
(graphics object) and subclasses it, and imports (graphics paint) and
merely _uses_ it.  So the draw method from (graphics paint) should be
renamed to avoid the conflict.  That merging independent
implementations works is either an accident, or the result of the
special case of non-overlapping specifiers.

  However, when the X window is redisplayed and all objects need to be
  redrawn, the update function in the (graphics object) package calls
  `draw' on each <graphics-object> on the list of visible objects.  This
  may include our <picture> object, so it is essential that draw in
  (graphics object) shares the (draw <picture>) method with draw in
  (graphics object picture).

The draw GF is invoked on a <picture>, which finds the method
specialized to (graphics object picture).  This method was defined in
(graphics object picture), and thus there is no need for the (g o)
draw definition to find the draw of (graphics picture).  This method
is in the new module.  If that draw method calls the renamed draw
method that is the (graphics paint) GF, finding the specialized
method, all is well.

So, I don't see how the draw generic in (graphics object) needs to
modified/extended at all; this would be the modularity violation that
bothered me.  Having draw be extended in (graphics object picture)
is a bit bothersome, but much less so.  A problem would arise if each
defined draw on, say vectors or objects - really any specializers that
are not disjoint.

In multiple inheritance cases where implementation is shared, then the
GFs do probably need to be merged.  But then the author of the module
that inherits from both rightfully needs to grok the details of the
underlying, so declaring that GF draw should be merged is a reasonable
thing to put in the module declaration.  And MI with interface only,
the using module needs merged GFs.

One issue is whether the functions to be merged are the 'same
function' semantically or just have the same name.   But in any case,
I don't see why a module's GF should get any methods added to it when
another module uses it.

Sorry if I'm seeming difficult/dense - I'm trying hard to keep the
module/goops orthogonality while not violating Schemely sensibilities.
One of the things that makes me thing GFs should not be merged without
explicit instruction is that for all other kinds of values merging
can't happen.

  Let me add that this is a quite bad example.  In the case above, the
  natural design would be to have the module representing the output
  device export `draw' and let both (graphics object) and (graphics
  paint) add their methods on it.

This raises an interesting issue.  There is a difference between
extending a defined interface and using a modules provided
abstractions.  In your restructured example, the output device exports
draw, and presumably the low level functions that actualy let you draw
bits on the screen.
It would be nice for that export of draw to be marked in such a way
that the exported GF may be modified by adding methods.

        Greg Troxel <gdt@ir.bbn.com>


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: New module system option :duplicates

[Mikael Djurfeldt]

Greg Troxel <gdt@ir.bbn.com> writes:

> That merging independent implementations works is either an
> accident, or the result of the special case of non-overlapping
> specifiers.

Still, I'm very curious how this plays out in practise.

Your argument can be made regarding the old goops implementation when
two different modules specializes the same imported GF.  Yet, with
disciplined use (see Kiczales paper; I've tried to follow those rules
in my own implementations), it appears to me that this works well in
practise.

And, again, :merge-accessors should be completely safe to use.

>   However, when the X window is redisplayed and all objects need to be
>   redrawn, the update function in the (graphics object) package calls
>   `draw' on each <graphics-object> on the list of visible objects.  This
>   may include our <picture> object, so it is essential that draw in
>   (graphics object) shares the (draw <picture>) method with draw in
>   (graphics object picture).
>
> The draw GF is invoked on a <picture>, which finds the method
> specialized to (graphics object picture).  This method was defined in
> (graphics object picture), and thus there is no need for the (g o)
> draw definition to find the draw of (graphics picture).  This method
> is in the new module.

I think you're missing something here.  If the draw method for
<picture>:s hasn't been added to the generic function draw in
(graphics object), that module can't redraw the display.

> This raises an interesting issue.  There is a difference between
> extending a defined interface and using a modules provided
> abstractions.  In your restructured example, the output device exports
> draw, and presumably the low level functions that actualy let you draw
> bits on the screen.
> It would be nice for that export of draw to be marked in such a way
> that the exported GF may be modified by adding methods.

Yes.  I think there is such a facility in Dylan.  We could maybe use
something similar.

M


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: New module system option :duplicates

[Marius Vollmer]

Mikael Djurfeldt <djurfeldt@nada.kth.se> writes:

> If I haven't missed something, there are no problems with implementing
> the current :duplicate option within the framework of new-model.text.

Ok, that is the important bit as far as I am concerned.  I'm still a
bit wary about merge-generics from a language design point of view,
but since it is an optional feature and we can support it in the
future, I don't see a problem with offering it.

> > [...]
> > Such a model can easily support an interface like
> >
> >    (define-module (foo)
> >      :use-module (blarg)
> >      :use-module ((bar) :select (mumble))  ; explicitely import mumble
> >      :use-module (baz)                   ; also imports mumble
> >                                          ; but only implicitely and
> >                                          ; since we have an explicit
> >                                          ; mumble, we won't see it
> >      :duplicates check)                  ; we expect no explicit duplicates
> 
> Do I understand this correctly in that you suggest having bindings
> with two "priority levels": "explicitly imported" = priority 1,
> "implicitly imported" = priority 2.  Only duplicate bindings with
> priority 1 causes an error (:duplicates check) while other duplicate
> pairs are resolved (at lookup-time) according to :duplicates first
> with precedence for the binding of higher priority.

Yes.

> Hmm... Isn't this a little conceptually complex?

Yeah, perhaps.  ADA and VHDL do a similar thing with their "potential
visibility" and their rules are indeed complex (I only reall about
know VHDL, tho).  After getting my head around the concept of
potential visibility, however, I find it quite intuitive.

When there are no conflicts, the rules are still very simple: you get
to see every imported binding.  However, when there are conflicts and
you have asked explicitely for one of the conflicting bindings, you
get the one you asked for.

But we can probably implement this as an option in the module system
as well, if we ever want to.  That's the basic motivation behind
new-model.text: modules should be able to implement a very rich and
hackable behavior without the compiler having to know about it.

> What is the gain of differing between "potentially imported" and
> "actually imported"?  It seems like a nice thing to be able to
> determine the set of imported bindings when the module is
> constructed.

Yes, the effect will mostly be as if the module has been fixed after
the 'define-module' statement, but while 'define-module' does its
thing, incrementally importing one set of bindings after the other,
not all information is available to resolve conflicts.  The
distinction between potentially imported and actually imported
bindings in there to support such a incremental construction of a
module in a natural way.  Also, it defines what happens when you
interactively change a module with 'use-modules', say.

> Is the aim to support the current "lazy binding" technique used in
> guile-tcltk?

No, only to support lazy resolution of naming conflicts.  In effect, a
conflict is only then reported when it actually matters and when no
resultion has been specified yet by the user.


One fundamental concept in new-model.text is that a module never
shrinks: once a binding has been found in a module, it can not be
removed or changed.  The potentially imported bindings would not be
known by the compiler and as such, could still be removed and changed.
Once the compiler asks for a binding and the module finds it among the
potentially imported bindings, it must become 'actually' imported and
can no longer change.

Imagine an interactive session like

  guile> (use-modules (graphics 2d))   ; exports make-point
  guile> (use-modules (graphics 3d))   ; also exports make-point
  guile> (define p (make-point 0 0))
  ERROR: 'make-point' is ambigous (in (graphics 2d) and (graphics 3d)).
  guile> (use-modules ((graphics 2d) :select (make-point)))
  guile> (define p (make-point 0 0))
  #<point 0 0>

Only when make-point is actually used, we check for conflicts.  Once a
resolution is provided by the user, the session can continue.

> Regarding the transition: What about making "check" the default
> duplicates handler but provide a form which the user can put in
> his/her .guile if he/she want to be backward compatible?
> 
>   (default-module-duplicates-handler 'last)

I'd rather have this the other way around.  Making '(warn last)' the
default is OK, I'd say, since it does not stop code from working.

> >   ;; merge 2d-x and 3d-x into x.  Likewise for y and z.
> >   (merge-prefix-generics (2d 3d) x y z)
> >
> > This is not much more tedious than ":duplicates merge-generics" but
> > much more precise.  No unexpected mergings are done and the remaining
> > collisions of non-generics are also dealt with in a useful way.
> 
> I like this.  However, I'd like to provide this facility *in addition*
> to giving the license to merge.

Yes, very good.

> Given that we already had `define-extended-generic' I've now added
> 
>   (define-extended-generics (x y z) #:prefix (2d: 3d:))
> 
> I chose this form because I have the feeling that we might want to
> have other additional ways of selecting the generics in the future.
> 
> Please tell me if you like your `merge-prefix-generics' better or if
> you have a third suggestion.

'define-extended-generics' is better.

> > Also, merging of generic functions might be a useful thing to do, even
> > without using the module system explicitely.
> 
> Yes.  We've had `define-extended-generic' for a while, but I have not
> come to documenting it in NEWS.  I'll fix my old sins ASAP.

Ahh! :-)  I didn't know about it indeed...

> > Does this make sense, Mikael?
> 
> Given my recent commits, I guess I have to return the question.  Do
> you think my recent additions make sense?

Yep, very much.

> The default duplicates handler is:
> 
>   (replace warn-override-core check)

I would like to have a default that works exactly like what we have
now, with additional warnings.  That way, code does not suddenly stop
working, but people get warnings and are 'encouraged' to fix their
code anyway.  Would

    (warn last)

work?  People can still use (default-module-duplicates-handler ...).

-- 
GPG: D5D4E405 - 2F9B BCCC 8527 692A 04E3  331E FAF8 226A D5D4 E405


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: New module system option :duplicates

[Mikael Djurfeldt]

Marius Vollmer <mvo@zagadka.de> writes:

> Would
>
>     (warn last)
>
> work?

Certainly, but

  (replace warn last)

is probably better, since it is highly likely to be a bug if for
example (srfi srfi-1) doesn't get to provide its extended versions of
map, for-each etc.

Placing `replace' first also avoids most of the warning messages that
we'd otherwise have for a large number of Guile modules.

M


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: New module system option :duplicates

[Marius Vollmer]

Mikael Djurfeldt <djurfeldt@nada.kth.se> writes:

> Marius Vollmer <mvo@zagadka.de> writes:
> 
> > Would
> >
> >     (warn last)
> >
> > work?
> 
> Certainly, but
> 
>   (replace warn last)
> 
> is probably better, since it is highly likely to be a bug if for
> example (srfi srfi-1) doesn't get to provide its extended versions of
> map, for-each etc.
> 
> Placing `replace' first also avoids most of the warning messages that
> we'd otherwise have for a large number of Guile modules.

Ok!  Please make (replace warn last) the default, then.


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel