Questions about the compiler et al

Questions about the compiler et al

[Mark H Weaver]

Hello all,

While writing the more complex variant of the compiler implementation of
`the-environment' and `local-eval', I learned a great deal about the
compiler, and accumulated the following list of questions along the way.

If someone with sufficient knowledge could answer at least some of these
questions, I would be grateful.

* Why do compilers return two copies of the same environment?
  What is the intended meaning of these two return values?

* What is the logic behind the handling of returned environments by
  compile-fold and read-and-compile?

* Should we move to support compiler environments that are not simply
  modules?

* Should we support distinct environment types for different languages,
  or use a simple uniform type for all, e.g. an alist?

* Why does code in analyze.scm check (module? env) in so many places?
  Is it expected that sometimes env will not actually be a module?
  Is it important to support this?

* Is there any way to embed references to non-serializable objects in
  compiled code?  I think this is important.  Not all code needs to be
  serialized, and we shouldn't limit ourselves to serializable code, for
  the same reasons that we shouldn't limit ourselves to serializable
  data.

* Do modules created by (make-fresh-user-module) get garbage collected?

* Why is `procedure-environment' still documented as "Very deprecated"?
  Is it still available at all?

    Thanks!
      Mark

Re: Questions about the compiler et al

[Andy Wingo]

Hi Mark,

On Thu 05 Jan 2012 01:03, Mark H Weaver <mhw@netris.org> writes:

> * Why do compilers return two copies of the same environment?
>   What is the intended meaning of these two return values?

From "The Scheme Compiler", in the manual:

       Compiler procedures take three arguments: an expression, an
    environment, and a keyword list of options. They return three values:
    the compiled expression, the corresponding environment for the target
    language, and a "continuation environment". The compiled expression and
    environment will serve as input to the next language's compiler.  The
    "continuation environment" can be used to compile another expression
    from the same source language within the same module.

       For example, you might compile the expression, `(define-module
    (foo))'. This will result in a Tree-IL expression and environment. But
    if you compiled a second expression, you would want to take into
    account the compile-time effect of compiling the previous expression,
    which puts the user in the `(foo)' module. That is purpose of the
    "continuation environment"; you would pass it as the environment when
    compiling the subsequent expression.

> * What is the logic behind the handling of returned environments by
>   compile-fold and read-and-compile?

To preserve compile-time changes of the current module, while compiling
a sequence of forms, while not allowing those side effects to leak out
to the caller of `read-and-compile' (I think?).

> * Should we move to support compiler environments that are not simply
>   modules?

Perhaps.  It used to be the case that things were different: see
f95f82f8e183f2744740bdc950dba9c856e09094.  But it turned out that having
environments that could only be interpreted by specific languages made
it difficult to have a generic language tower, so I changed them all to
have one representation, namely, as modules.

> * Should we support distinct environment types for different languages,
>   or use a simple uniform type for all, e.g. an alist?

Uniform, please!

> * Why does code in analyze.scm check (module? env) in so many places?
>   Is it expected that sometimes env will not actually be a module?
>   Is it important to support this?

Probably something legacy-related.

> * Is there any way to embed references to non-serializable objects in
>   compiled code?

No.  Can you give an example of when you would want this?

> * Do modules created by (make-fresh-user-module) get garbage collected?

In theory, yes.

> * Why is `procedure-environment' still documented as "Very deprecated"?
>   Is it still available at all?

It is removed in master.

Andy
-- 
http://wingolog.org/

Re: Questions about the compiler et al

[Mark H Weaver]

Hi Andy, thanks for the quick response!

Andy Wingo <wingo@pobox.com> writes:
>> * Is there any way to embed references to non-serializable objects in
>>   compiled code?
>
> No.  Can you give an example of when you would want this?

My more complex `the-environment' patch fully supports captured local
syntax definitions in the evaluator, and would also do so for the
compiler if not for this limitation.  Code generated by
(the-environment) needs to embed references to the captured local syntax
transformers.  I can imagine some hacks to work around this limitation,
but they're not nice.

My more complex patch adds support for attaching an objtable and
free-vars to the compiler environment for use by `objcode->value', and I
successfully used this to attach the free-vars to the compiled closure.

I also did some work on providing a compiler option for
`compile-assembly' to attach the objtable to the target-environment
instead of serializing it as code, but didn't finish that job.  I still
think it might be useful.  Any suggestions (or objections)?

Furthermore, I'd suggest that _not_ serializing the objtable should be
the default behavior, and that `compile-file' should instead add a
compiler option to force objtable serialization.  The rationale is that
I expect most uses of `compile' from within user code to compile to
`value' anyway, and forcing serialization needlessly adds overhead, and
more importantly, limitations on what might be found within (quote _)
forms of generated code.  What do you think?

>> * Should we move to support compiler environments that are not simply
>>   modules?
>
> Perhaps.  It used to be the case that things were different: see
> f95f82f8e183f2744740bdc950dba9c856e09094.  But it turned out that having
> environments that could only be interpreted by specific languages made
> it difficult to have a generic language tower, so I changed them all to
> have one representation, namely, as modules.

I agree that a single extensible representation (like alists) for all
languages is probably best, but I think it's important to make this
representation distinct from modules ASAP, before the assumption that
they're modules becomes too deeply entrenched.

Although it's easy enough for compilers to accept a new representation
while still accepting modules, I'm more concerned about their return
values.  What happens if a bunch of external code is written that
assumes compilers will always return modules as environments?

There's a danger that we might become unable to use environments for
anything else without breaking compatibility.

>> * Do modules created by (make-fresh-user-module) get garbage collected?
>
> In theory, yes.

Is this practical?  Do these fresh modules have names?  If so, can't
references exist to them purely in source code using @@?  The only
reference to them might be from the template of a syntax transformer.

>> * Why is `procedure-environment' still documented as "Very deprecated"?
>>   Is it still available at all?
>
> It is removed in master.

I cannot find its implementation anywhere in stable-2.0, but it is still
documented.  I guess it should be removed from the doc, or the doc
should at least mention that it has already been removed.  Or no?

    Thanks,
      Mark

Re: Questions about the compiler et al

[Andy Wingo]

Hi Mark,

On Thu 05 Jan 2012 18:20, Mark H Weaver <mhw@netris.org> writes:

> Andy Wingo <wingo@pobox.com> writes:
>>> * Is there any way to embed references to non-serializable objects in
>>>   compiled code?
>>
>> No.  Can you give an example of when you would want this?
>
> My more complex `the-environment' patch fully supports captured local
> syntax definitions in the evaluator, and would also do so for the
> compiler if not for this limitation.  Code generated by
> (the-environment) needs to embed references to the captured local syntax
> transformers.  I can imagine some hacks to work around this limitation,
> but they're not nice.
>
> My more complex patch adds support for attaching an objtable and
> free-vars to the compiler environment for use by `objcode->value', and I
> successfully used this to attach the free-vars to the compiled closure.
>
> I also did some work on providing a compiler option for
> `compile-assembly' to attach the objtable to the target-environment
> instead of serializing it as code, but didn't finish that job.  I still
> think it might be useful.  Any suggestions (or objections)?

Hummmm.  Yeah, maybe.  I guess there is room for this -- see for example
module/language/objcode/spec.scm:76.  It would be nice to unify the
representations of environments between the compiler and the
decompiler.  Tricky, though.

> Furthermore, I'd suggest that _not_ serializing the objtable should be
> the default behavior, and that `compile-file' should instead add a
> compiler option to force objtable serialization.  The rationale is that
> I expect most uses of `compile' from within user code to compile to
> `value' anyway, and forcing serialization needlessly adds overhead, and
> more importantly, limitations on what might be found within (quote _)
> forms of generated code.  What do you think?

Makes sense to me.

>>> * Should we move to support compiler environments that are not simply
>>>   modules?
>>
>> Perhaps.  It used to be the case that things were different: see
>> f95f82f8e183f2744740bdc950dba9c856e09094.  But it turned out that having
>> environments that could only be interpreted by specific languages made
>> it difficult to have a generic language tower, so I changed them all to
>> have one representation, namely, as modules.
>
> I agree that a single extensible representation (like alists) for all
> languages is probably best, but I think it's important to make this
> representation distinct from modules ASAP, before the assumption that
> they're modules becomes too deeply entrenched.
>
> Although it's easy enough for compilers to accept a new representation
> while still accepting modules, I'm more concerned about their return
> values.  What happens if a bunch of external code is written that
> assumes compilers will always return modules as environments?
>
> There's a danger that we might become unable to use environments for
> anything else without breaking compatibility.

ACK.  Was it the right thing, in the end, to have environments specific
to the language?  Maybe so; the Scheme compiler shouldn't know anything
about object tables (or elisp DECLAIM settings), and each compiler knows
its source and target language, so that it can consume and produce
environments of the appropriate type...

If I can strain my memory enough, I think that the reason that I made
the change was so that I could document the interfaces in some way that
made coherent sense.  If you are able to envision a change that can be
explained coherently, then let's look at it.

>>> * Do modules created by (make-fresh-user-module) get garbage collected?
>>
>> In theory, yes.
>
> Is this practical?  Do these fresh modules have names?

They get names only on demand; see boot-9.scm:2405.  I think that once
they have names, there are no longer collectable.

>>> * Why is `procedure-environment' still documented as "Very deprecated"?
>>>   Is it still available at all?
>>
>> It is removed in master.
>
> I cannot find its implementation anywhere in stable-2.0, but it is still
> documented.  I guess it should be removed from the doc, or the doc
> should at least mention that it has already been removed.  Or no?

Probably removed from the documentation, I would think.

Thanks for taking a look at this piece of code!

Cheers,

Andy
-- 
http://wingolog.org/

Re: Questions about the compiler et al

[BT Templeton]

Mark H Weaver <mhw@netris.org> writes:

> Andy Wingo <wingo@pobox.com> writes:
>>> * Is there any way to embed references to non-serializable objects in
>>>   compiled code?
>>
>> No.  Can you give an example of when you would want this?
[...]
> Furthermore, I'd suggest that _not_ serializing the objtable should be
> the default behavior, and that `compile-file' should instead add a
> compiler option to force objtable serialization.  The rationale is that
> I expect most uses of `compile' from within user code to compile to
> `value' anyway, and forcing serialization needlessly adds overhead, and
> more importantly, limitations on what might be found within (quote _)
> forms of generated code.  What do you think?

This change would make it possible to use uninterned symbols as gensyms,
which would be useful for Elisp and CL support. Simply adding support
for serializing uninterned symbols isn't enough, because then (eq x
(eval `',x)) is sometimes false.

-- 
Inteligenta persono lernas la lingvon Esperanton rapide kaj facile.
Esperanto estas moderna, kultura lingvo por la mondo. Simpla, fleksebla,
belsona, Esperanto estas la praktika solvo de la problemo de universala
interkompreno. Lernu la interlingvon Esperanton!