Emacs Lisp and Guile

Emacs Lisp and Guile

[Richard Stallman]

Can you tell us the status of your work making Emacs Lisp work on Guile?

Re: Emacs Lisp and Guile

[Neil Jerram]

>>>>> "Richard" == Richard Stallman <rms@gnu.org> writes:

    Richard> Can you tell us the status of your work making Emacs Lisp
    Richard> work on Guile?

I've written a prototype translator, mostly in Scheme, which is
available in Guile CVS (HEAD branch).  In this prototype, things that
are `primitives' in Emacs are almost all defined in _Scheme_, not C.

The best demonstration of what it can do is obtained by telling it to
`(load "loadup.el")' and seeing how far it gets before hitting an
error.  Last time I tried, it processed 3279 lines of Elisp before
stopping in mule.el (because I haven't defined the make-char-table
primitive):

guile> (load-emacs)
Calling loadup.el to clothe the bare Emacs...
Loading /usr/share/emacs/20.7/lisp/loadup.el...
Using load-path ("/usr/share/emacs/20.7/lisp/" "/usr/share/emacs/20.7/lisp/emacs-lisp/")
Loading /usr/share/emacs/20.7/lisp/byte-run.el...
Loading /usr/share/emacs/20.7/lisp/byte-run.el...done
Loading /usr/share/emacs/20.7/lisp/subr.el...
Loading /usr/share/emacs/20.7/lisp/subr.el...done
Loading /usr/share/emacs/20.7/lisp/version.el...
Loading /usr/share/emacs/20.7/lisp/version.el...done
Loading /usr/share/emacs/20.7/lisp/map-ynp.el...
Loading /usr/share/emacs/20.7/lisp/map-ynp.el...done
Loading /usr/share/emacs/20.7/lisp/widget.el...
Loading /usr/share/emacs/20.7/lisp/emacs-lisp/cl.el...
Loading /usr/share/emacs/20.7/lisp/emacs-lisp/cl.el...done
Loading /usr/share/emacs/20.7/lisp/widget.el...done
Loading /usr/share/emacs/20.7/lisp/custom.el...
Loading /usr/share/emacs/20.7/lisp/custom.el...done
Loading /usr/share/emacs/20.7/lisp/cus-start.el...
Note, built-in variable `abbrev-all-caps' not bound
  ... [many other variable not bound messages] ...
Loading /usr/share/emacs/20.7/lisp/cus-start.el...done
Loading /usr/share/emacs/20.7/lisp/international/mule.el...
<unnamed port>: In procedure make-char-table in expression (@fop make-char-table (# #)):
<unnamed port>: Symbol's function definition is void
ABORT: (misc-error)

Type "(backtrace)" to get more information or "(debug)" to enter the debugger.
guile> 


There are two serious restrictions:

- Most Emacs Lisp primitives are not yet implemented.  In particular,
  there are no buffer-related primitives.

- Performance compares badly with Emacs.  Using a handful of
  completely unscientific tests, I found that Guile was between 2 and
  20 times slower than Emacs.

I think that both these restrictions point in the same direction: the
way forward is to define the primitives by compiling a preprocessed
version of the Emacs source code, not by trying to implement them in
Scheme.  This, of course, is what Ken Raeburn's project is already
trying to do, so there is little point in me trying to do the same
thing independently.

Unless this picture changes, I don't plan to do any further
significant work on the prototype translator.

I expect that most of the translator's Scheme code will eventually
become obsolete, replaced by bits of Emacs C code.  Until then,
though, it should have a role:

- as a guide to the Guile Emacs project on how to interface to the
  Elisp support in libguile (notably, usage of `@fop' and `@bind')

- as a proof of concept and fun thing to experiment with

- as a working translator that could help us develop our picture of
  how we want to integrate translator usage in general with the rest
  of Guile.

Best regards,
        Neil

Re: Emacs Lisp and Guile

[Richard Stallman]

    I think that both these restrictions point in the same direction: the
    way forward is to define the primitives by compiling a preprocessed
    version of the Emacs source code, not by trying to implement them in
    Scheme.

What precisely is "a preprocessed version"?  What I think we should do
is modify the code in Emacs so that it works with Scheme.

    Unless this picture changes, I don't plan to do any further
    significant work on the prototype translator.

Putting aside the issue of the Emacs primitives, which we are going to
handle with C code and should not be implemented in Scheme, does it
need any more work?  Not counting those primitives, are there Emacs
Lisp features it does not handle?  Or is it adequate as it stands?
(If so, why call it a "prototype"?  Why not call it "finished"?)

Re: Emacs Lisp and Guile

[Neil Jerram]

[-- Attachment #1: Type: text/plain, Size: 2726 bytes --]

>>>>> "Richard" == Richard Stallman <rms@gnu.org> writes:

         I think that both these restrictions point in the same direction: the
         way forward is to define the primitives by compiling a preprocessed
         version of the Emacs source code, not by trying to implement them in
         Scheme.

    Richard> What precisely is "a preprocessed version"?  What I think
    Richard> we should do is modify the code in Emacs so that it works
    Richard> with Scheme.

What you suggest would be ideal.  I only talked about a "preprocessed
version" because I was assuming that it might not be feasible to start
modifying the primary Emacs codebase immediately.  If this assumption
is wrong, so much the better.  (Would you agree with this, Ken?)

         Unless this picture changes, I don't plan to do any further
         significant work on the prototype translator.

    Richard> Putting aside the issue of the Emacs primitives, which we
    Richard> are going to handle with C code and should not be
    Richard> implemented in Scheme, does it need any more work?  Not
    Richard> counting those primitives, are there Emacs Lisp features
    Richard> it does not handle?  Or is it adequate as it stands?  (If
    Richard> so, why call it a "prototype"?  Why not call it
    Richard> "finished"?)

1. The Guile reader doesn't directly handle all Elisp syntax -- e.g. [
] for vectors and ? for character/integers.  The translator retrofixes
some of these discrepancies -- e.g. it turns (what it sees as) symbols
for the obvious alphabet characters (?a, ?b etc.) into Guile integers
-- but not all of them.

2. The representation of Elisp variables doesn't allow for buffer- and
frame-local variables.

3. There's no byte code interpreter.

4. There are special forms that I haven't implemented
(e.g. condition-case) and several data types (e.g. buffers, char
tables); but these are all really in the same category as primitives,
so no big deal.

It's adequate as it stands for the programmer who wants to write
general purpose code using (mostly) Emacs Lisp syntax and language
constructs.  (For an example, see the attached file, which is included
in Guile CVS for this purpose.)  It's not adequate for the programmer
who wants easily to build an editor application by taking advantage of
a large set of relevant primitives and data types.

If we assume that primitives, data types, special forms, the reader
and the byte code interpreter are all dealt with by using the Emacs C
code, I think the only design issue remaining is the representation of
Elisp variables so that buffer- and frame-local values are supported.

Does this sufficiently answer your questions?  Please let me know if
not.

        Neil


[-- Attachment #2: Example of Elisp code that translator can handle --]
[-- Type: application/emacs-lisp, Size: 1224 bytes --]

Re: Emacs Lisp and Guile

[Ken Raeburn]

Neil Jerram <neil@ossau.uklinux.net> writes:
> - as a guide to the Guile Emacs project on how to interface to the
>   Elisp support in libguile (notably, usage of `@fop' and `@bind')

So, why were @fop and @bind needed in libguile anyways?  I was never
clear on that.  Is it for performance, or is Scheme not up to the
task?

> - as a working translator that could help us develop our picture of
>   how we want to integrate translator usage in general with the rest
>   of Guile.

I wonder if it's the best choice.  I don't think encouraging people to
support translation by starting with writing more C code for libguile
is wise; we'll wind up with a libguile with random primitives for
supporting translated elisp and perl and tcl and python
and....  Personally, I'd rather see that support in Scheme modules.

That issue aside, yes, I think we definitely need work in this area.
Probably another language or two as well.  Real, useful languages;
inventing another programming language so that we have another
translator doesn't do us much good if no one uses that language.

Well, that's the way it used to be.  I've been focussed enough on
Emacs in what free time I've had lately that I haven't even been
following the Guile lists, just filing them away for later.  If there
are other real translators available now, then "yay!" :-)


Neil Jerram <neil@ossau.uklinux.net> writes:
>>>>>> "Richard" == Richard Stallman <rms@gnu.org> writes:
>          I think that both these restrictions point in the same direction: the
>          way forward is to define the primitives by compiling a preprocessed
>          version of the Emacs source code, not by trying to implement them in
>          Scheme.
>
>     Richard> What precisely is "a preprocessed version"?  What I think
>     Richard> we should do is modify the code in Emacs so that it works
>     Richard> with Scheme.
>
> What you suggest would be ideal.  I only talked about a "preprocessed
> version" because I was assuming that it might not be feasible to start
> modifying the primary Emacs codebase immediately.  If this assumption
> is wrong, so much the better.  (Would you agree with this, Ken?)

I don't think I'd use the term "preprocessed", but I suppose you could
look at it that way.

I haven't looked at how easy or hard it'll be to make Emacs C DEFUNs
produce SCM primitive procedures.  If it's easy, great.  If it's hard,
and we don't want to break the existing Lisp representations right
away, we could look for a simple, regular substitution that could be
done on the C code, or write Scheme wrappers that invoke a "call this
Lisp function" Scheme primitive procedure.

One of my concerns in that area is with mixing Lisp dynamic bindings
and Guile thread support.  If we want multi-threaded Lisp, do we pass
around some sort of environment pointer, or call thread-aware routines
to fetch current environment info?  Neil, does your code try to
support multi-threaded Lisp?


> 1. The Guile reader doesn't directly handle all Elisp syntax -- e.g. [
> ] for vectors and ? for character/integers.  The translator retrofixes
> some of these discrepancies -- e.g. it turns (what it sees as) symbols
> for the obvious alphabet characters (?a, ?b etc.) into Guile integers
> -- but not all of them.

Keeping the Elisp reader may be simplest.  I vaguely recall concluding
that cases could be constructed where the Guile reader wouldn't
distinguish things in some cases where Elisp would, but I don't
remember the details.  I think it had to do with character or number
data, or maybe "?0" type constructs.

> 2. The representation of Elisp variables doesn't allow for buffer- and
> frame-local variables.

Still an open issue in my book too.

> 3. There's no byte code interpreter.

Translating to Scheme (or Guile byte codes) is another possibility.

Ken

Re: Emacs Lisp and Guile

[Richard Stallman]

    What you suggest would be ideal.  I only talked about a "preprocessed
    version" because I was assuming that it might not be feasible to start
    modifying the primary Emacs codebase immediately.  If this assumption
    is wrong, so much the better.  (Would you agree with this, Ken?)

We may be miscommunicating.  I was talking about the right way to do
this eventually.  We certainly are not at the point of doing it now.

    1. The Guile reader doesn't directly handle all Elisp syntax -- e.g. [
    ] for vectors and ? for character/integers.

Since Emacs Lisp syntax is not the same as Guile syntax, we will need
a reader that handles Emacs Lisp syntax--unless it happens that the
Guile reader could be customized to support Emacs Lisp syntax.  Is
that feasible?

    2. The representation of Elisp variables doesn't allow for buffer- and
    frame-local variables.

How to handle these in a Guile-based system is not obvious, but they are
such important features that we must make them work.

    3. There's no byte code interpreter.

I don't think we need to support the Emacs Lisp byte code interpreter
if we have some equally efficient or more efficient way to run an
Emacs Lisp program.  Source compatibility is the important kind of
compatibility here.

    4. There are special forms that I haven't implemented
    (e.g. condition-case) and several data types (e.g. buffers, char
    tables); but these are all really in the same category as primitives,
    so no big deal.

condition-case needs to be handled; that is a language construct,
not an editing primitive.

The functions that operate on buffers would be implemented by C code,
the same as now, so they are not part of the scope of implementing the
Emacs Lisp language.  However, the issue of the data representation to
use for them is a significant one.

Re: Emacs Lisp and Guile

[Marius Vollmer]

Richard Stallman <rms@gnu.org> writes:

> Since Emacs Lisp syntax is not the same as Guile syntax, we will need
> a reader that handles Emacs Lisp syntax--unless it happens that the
> Guile reader could be customized to support Emacs Lisp syntax.  Is
> that feasible?

It is not possible to configure the Guile reader enough right now, but
it would certainly be a good thing to add that configurability.

>     2. The representation of Elisp variables doesn't allow for buffer- and
>     frame-local variables.
> 
> How to handle these in a Guile-based system is not obvious, but they are
> such important features that we must make them work.

A Elisp variable can start its existence as a normal variable and then
be later turned into a buffer- or frame-local variable and when such a
change occurs, all code (possibly already compiled) must start to
treat the variable as buffer local, right?

When that is the case, we need to treat Elisp variable references
differently from Scheme variable references.  In Scheme, we only look
up the storage location of a variable once and then each reference is
only a simple memory access.

However, we are also considering an automatic re-lookup machine that
would update the old lookups whenever the state of the module system
changes in such a way that these old lookups become outdated.  (That
is, when some code refers to a variable with the name "foo" and that
name is at first provided by the module mod1, and then mod1 is
changed to no longer export "foo" but mod2 now does, we will
automatically modify the code to refer to the new variable.)

We could use this (as of now non-existing) machinery to also modify
code that has been translated from Elisp.

I'm not sure that I like this approach, tho.  Right now, I'd say that
requiring the user to simply reload his files when he wants a new
name->variable mapping to take effect is good enough or maybe even
better.

That said, I think we should have a new primitive procedure, say
@elisp-ref, that does the right thing for Elisp variables.

-- 
GPG: 7CB2 927E F53F BD3C 5DC4  CD06 CF32 AB3E 1FC6 BAE7

Re: Emacs Lisp and Guile

[Richard Stallman]

    A Elisp variable can start its existence as a normal variable and then
    be later turned into a buffer- or frame-local variable and when such a
    change occurs, all code (possibly already compiled) must start to
    treat the variable as buffer local, right?

Right.

    When that is the case, we need to treat Elisp variable references
    differently from Scheme variable references.  In Scheme, we only look
    up the storage location of a variable once and then each reference is
    only a simple memory access.

I guess so.  But one question is, what would a reference to a "Lisp"
variable look like in Scheme?  Would you have to call a special
function to get or set the value?

Scheme variables are normally lexical.  What do people normally do in
Scheme when you want a dynamically scoped value?

Re: Emacs Lisp and Guile

[Neil Jerram]

>>>>> "Richard" == Richard Stallman <rms@gnu.org> writes:

    Richard> We may be miscommunicating.  I was talking about the
    Richard> right way to do this eventually.  We certainly are not at
    Richard> the point of doing it now.

Sorry, I got the wrong impression from what you said.  Can you
describe how and when you see the Emacs/Guile integration proceeding?
I think this would provide useful context for the more technical
details.

        Neil

Re: Emacs Lisp and Guile

[Richard Stallman]

    Sorry, I got the wrong impression from what you said.  Can you
    describe how and when you see the Emacs/Guile integration proceeding?

I have not tried to make much in the way of advance plans--it depends
on finding people who want to do the work, etc.

Re: Emacs Lisp and Guile

[Marius Vollmer]

Richard Stallman <rms@gnu.org> writes:

>     When that is the case, we need to treat Elisp variable references
>     differently from Scheme variable references.  In Scheme, we only look
>     up the storage location of a variable once and then each reference is
>     only a simple memory access.
> 
> I guess so.  But one question is, what would a reference to a "Lisp"
> variable look like in Scheme?  Would you have to call a special
> function to get or set the value?

Yes, probably, as far as I can see.  I don't think this is a problem,
tho.  A call to that special function can be very fast when needed.
For example, when we should get serious about compilation to machine
code, we will want to inline certain operations, like arithmetic on
fixnums, and the call to the special Elisp-variable-accessor-function
can be one of these operations.  For a bytecode interpreter (or
similar), we will likewise want to have special opcodes for these kind
of things.

> Scheme variables are normally lexical.  What do people normally do in
> Scheme when you want a dynamically scoped value?

In plain Scheme, you use 'dynamic-wind' to establish a dynamic
context.  Entering and leaving that context will run specified
procedures that can swap values in and out of variables that should
have dynamically scoped values.

We also have '@bind' which does just this but more efficiently.

In Guile, we also have 'fluids'.  A fluid is a normal object like a
cons pair that holds one object per thread.  That is, fluids are our
mechanism for thread local variables.  There is also support for
efficiently creating dynamic contexts that will swap the value of a
fluid with a backing store when entered/left.

    > (define var (make-fluid))
    > (define (peek-var)
        (peek (fluid-ref var)))
    > (fluid-set! var 12)
    > var
    => #<fluid 5>
    > (peek-var)
    ;;; 12
    > (with-fluids ((var 23))
        (peek-var))
    ;;; 23
    > (peek-var)
    ;;; 12

The "5" in #<fluid 5> is just an index, not the value of the fluid.

Re: Emacs Lisp and Guile

[Richard Stallman]

    > Scheme variables are normally lexical.  What do people normally do in
    > Scheme when you want a dynamically scoped value?

    In plain Scheme, you use 'dynamic-wind' to establish a dynamic
    context.  Entering and leaving that context will run specified
    procedures that can swap values in and out of variables that should
    have dynamically scoped values.

It is very undesirable for Lisp dynamic variables to be different
from Scheme dynamic variables.  It makes the system specs incoherent.

Is it possible to extend dynamic-wind so that it can handle
buffer-local, frame-local, etc. contexts?  Then a Lisp variable
could be handled with dynamic-wind as usual, and there would be
no divergence between Scheme variables and Lisp variables.

    In Guile, we also have 'fluids'.  A fluid is a normal object like a
    cons pair that holds one object per thread.  That is, fluids are our
    mechanism for thread local variables.

Could this be unified with dynamic-wind also?

Re: Emacs Lisp and Guile

[Marius Vollmer]

Richard Stallman <rms@gnu.org> writes:

>     > Scheme variables are normally lexical.  What do people normally do in
>     > Scheme when you want a dynamically scoped value?
> 
>     In plain Scheme, you use 'dynamic-wind' to establish a dynamic
>     context.  Entering and leaving that context will run specified
>     procedures that can swap values in and out of variables that should
>     have dynamically scoped values.
> 
> It is very undesirable for Lisp dynamic variables to be different
> from Scheme dynamic variables.  It makes the system specs incoherent.

There really aren't dynamic Scheme variables (in the sense that there
are dynamic 'special' variables in Common Lisp).  The effect where a
variable takes on a certain value during an dynamic extent and
reverting to its previous value when the context is left is not
implemented with a special kind of variable, but with a special
control flow construct that allows you to run hooks when a certain
dynamic context is entered or left.  These hooks can be used to swap
the values of a variable.

You normally hide these things behind some functions.  'dynamic-wind'
is the fundament upon which these functions are built, but it is too
clumsy to explicitly use it for dynamically scoped values.  For
example, there is no visible variable that holds the 'current output
port' (the Scheme analogon of C's stdout), but there is a function
'current-output-port' that returns the current output port, and there
is a function 'with-output-to-port' that established a new dynamic
context in which 'current-output-port' returns the specified port.
For example

    (with-output-to-port (current-error-port)
      (lambda ()
        (display "Hi\n")))

will output "Hi\n" on the current error port by temporarily switching
the current output port to the error port.

I don't think this has to be inconsistent with Lisp.

What is a typical part of the system specification that makes use of
Elisp's dynamic scoping?


Maybe I understand you wrong.  If you are saying that we should
implement Elisp's variables the way Scheme normally mimics dynamic
scoping, as opposed to saying that we should add dynamic scoping to
Scheme in order to handle the existing Emacs spec, then we have no
problem.  We already have '@bind' which can be used to implement Elisp
'let' and argument binding for function calls.  It will use
'dynamic-wind' as outlined above.

> Is it possible to extend dynamic-wind so that it can handle
> buffer-local, frame-local, etc. contexts?

No, dynamic-wind is only for control flow.  Think of it like an
extended unwind-protect (where the extension is only useful in the
presence of captured continuations).  However, we could have something
like it for buffer-local contexts, say 'buffer-wind'.  That is, we
could have a way to specify hooks that are run whenever the value of
(current-buffer) changes.  The hooks would then install the right
values in the variables that are supposed to behave buffer-locally.

I'm afraid this might turn out to be expensive and it is not a good
solution in a multi-threaded system.  I don't know whether you
consider making Emacs multi-threaded, but we should keep that option.
With something like 'buffer-wind', the values would be swapped at
every thread switch.  It wouldn't work at all for true concurrent
threads.  (We are far away from true concurrent thread support in
Guile, but we have nice cooperative threading.)


What about exporting all Elisp variables to Scheme as settable
functions?  For example

    (set! (require-final-newline) 'ask)

This would lead to a divergence between Scheme variables and Lisp
variables, but I don't think this is bad.

>     In Guile, we also have 'fluids'.  A fluid is a normal object like a
>     cons pair that holds one object per thread.  That is, fluids are our
>     mechanism for thread local variables.
> 
> Could this be unified with dynamic-wind also?

It complements dynamic-wind and works together with it.  Fluids
provide thread-local storage and dynamic-wind is used to swap values
in and out of these locations to establish dynamic contexts.  Fluids
don't incur an overhead on every thread switch, but on every access.
They are meant to be used to implement things like
'current-output-port' in a thread-safe manner.  For example,
'current-buffer' could be implemented with a fluid.

Re: Emacs Lisp and Guile

[Kai Großjohann]

Marius Vollmer <marius.vollmer@uni-dortmund.de> writes:

> What is a typical part of the system specification that makes use of
> Elisp's dynamic scoping?

I don't know what you mean by "system specification", but I guess
that a lot of code assumes that variables declared with defvar (or
defcustom) can be bound fluidly.  For example, to write something to
a file with a specific coding system, one would say

    (let ((coding-system-for-write 'foo))
      (write-region ...))

where coding-system-for-write is defvar'd.  Constructs like this must
be ubiquitous in Emacs, I don't think you can "isolate" them somehow.
And since there are hundreds of variables in Emacs, what kind of usage
would be "typical"?

Does this help at all?

kai
-- 
A large number of young women don't trust men with beards.  (BFBS Radio)

Re: Emacs Lisp and Guile

[Marius Vollmer]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Marius Vollmer <marius.vollmer@uni-dortmund.de> writes:
> 
> > What is a typical part of the system specification that makes use of
> > Elisp's dynamic scoping?
> 
> I don't know what you mean by "system specification", but I guess
> that a lot of code assumes that variables declared with defvar (or
> defcustom) can be bound fluidly.  For example, to write something to
> a file with a specific coding system, one would say
> 
>     (let ((coding-system-for-write 'foo))
>       (write-region ...))

Hmm, yes.  This is a thing that must be done right when implementing
Elisp on top of Guile.  Of course we must respect the dynamic scoping
rules of Elisp for Elisp itself.  I was asking about the spec of the
whole system where one can write in both Elisp and Scheme and access
a common global state.

Re: Emacs Lisp and Guile

[Kai Großjohann]

Marius Vollmer <marius.vollmer@uni-dortmund.de> writes:

> Hmm, yes.  This is a thing that must be done right when implementing
> Elisp on top of Guile.  Of course we must respect the dynamic scoping
> rules of Elisp for Elisp itself.  I was asking about the spec of the
> whole system where one can write in both Elisp and Scheme and access
> a common global state.

What would be another way to achieve the same effect that's now
achieved by these variables?  Take the example of
coding-system-for-write: how would you do it in Scheme?

kai
-- 
A large number of young women don't trust men with beards.  (BFBS Radio)

Re: Emacs Lisp and Guile

[Marius Vollmer]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> What would be another way to achieve the same effect that's now
> achieved by these variables?  Take the example of
> coding-system-for-write: how would you do it in Scheme?

When coding-system-for-write would have been designed for Scheme, I
would expect it to hidden behind an abstraction; it would not be a
variable that you can set.  It could be offered as

    (with-coding-system-for-write 'foo
      (lambda () ...))

I'm not clear about how to best do these things in Scheme; I don't
have enough experience with large systems.

Re: Emacs Lisp and Guile

[Kai Großjohann]

Marius Vollmer <marius.vollmer@uni-dortmund.de> writes:

> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
>
>> What would be another way to achieve the same effect that's now
>> achieved by these variables?  Take the example of
>> coding-system-for-write: how would you do it in Scheme?
>
> When coding-system-for-write would have been designed for Scheme, I
> would expect it to hidden behind an abstraction; it would not be a
> variable that you can set.  It could be offered as
>
>     (with-coding-system-for-write 'foo
>       (lambda () ...))

I see.  What would with-coding-system-for-write look like?  What
would functions look like that want to find out which coding system
to use for writing?

Maybe something can be abstracted so that

(with-variable 'coding-system-for-write 'foo
  ...statements...)

works for all variables.  Of course, you might as well call it `let',
then :-)

kai
-- 
A large number of young women don't trust men with beards.  (BFBS Radio)

Re: Emacs Lisp and Guile

[Richard Stallman]

    >     In plain Scheme, you use 'dynamic-wind' to establish a dynamic
    >     context.  Entering and leaving that context will run specified
    >     procedures that can swap values in and out of variables that should
    >     have dynamically scoped values.
    > 
    > It is very undesirable for Lisp dynamic variables to be different
    > from Scheme dynamic variables.  It makes the system specs incoherent.

    There really aren't dynamic Scheme variables (in the sense that there
    are dynamic 'special' variables in Common Lisp).

I know that.  Variables with dynamic bindings are dynamic variables
even if they are not a special kind of variable.  Emacs Lisp also has
only one kind of variable.  It is not like Common Lisp.

This is why I propose extending the dynamic-wind mechanism to handle
various kinds of context-local bindings.

    > Is it possible to extend dynamic-wind so that it can handle
    > buffer-local, frame-local, etc. contexts?

    No, dynamic-wind is only for control flow.

At present, the dynamic-wind mechanism only works with control flow.
I am proposing to generalize the mechanism so that it implements
various selectable "local" contexts (current buffer, selected frame,
etc.).

What exactly the API should look like, I am not venturing to say.

    You normally hide these things behind some functions.

That won't work here.

Emacs Lisp has hundreds of dynamic variables, whose names are
documented, that user programs examine, set, and bind.  It would be
clumsy and impractical to hide each of these variables behind a
separate function in Scheme.  It would be practical but clumsy to have
a few Scheme functions for accessing and binding Lisp variables.  That would
make Scheme very inconvenient to use, compared with Emacs Lisp.

The clean approach is that each of these Lisp variables is simply a
Scheme variable.

    What about exporting all Elisp variables to Scheme as settable
    functions?  For example

	(set! (require-final-newline) 'ask)

    This would lead to a divergence between Scheme variables and Lisp
    variables, but I don't think this is bad.

I think it is ugly--variables should remain variables.

      However, we could have something
    like it for buffer-local contexts, say 'buffer-wind'.  That is, we
    could have a way to specify hooks that are run whenever the value of
    (current-buffer) changes.  The hooks would then install the right
    values in the variables that are supposed to behave buffer-locally.

Perhaps this is what we should do.  However, these local contexts
should not be associated specifically with buffers.  There should be a
list of current local contexts, which the user program can use for
different purposes.  Emacs would use one for the current buffer's
bindings, and one for the current frame's bindings.

It is necessary for the mechanism that handles let-style dynamic scope
to know something about these contexts as well.  If they don't know
about each other, they will get confused in certain cases, with
bindings being swapped out into the wrong place.  Those bugs used
to exist in Emacs but we fixed them.

Essentially, it is necessary to label each variable by saying
which local context its current binding belongs to.

    I'm afraid this might turn out to be expensive and it is not a good
    solution in a multi-threaded system.

It does not have to be expensive.  Although Emacs is not
multi-threaded, it has an optimization for this: a variable's
buffer-local bindings are not swapped until you try to access the
variable.  Runnning another thread for a short time, if it does not
access many of these variables, won't have to swap many of them.

      Fluids
    provide thread-local storage and dynamic-wind is used to swap values
    in and out of these locations to establish dynamic contexts.

Are you saying that dynamic-wind can't be used directly to swap values
in and out of a variable, that you have to set the variable to a fluid
and use dynamic-wind to swap values in and out of the fluid?

Or are you saying that dynamic-wind can be used either to swap values
in and out of a variable, or to swap values in and out of a fluid?

Do fluids have an optimized mechanism like the one I described above
for Emacs buffer-local variables?  Or does each access to a fluid look
up the binding for the current thread?

Given that optimized mechanism, I think threa-local variable values
(as distinct from use of fluids) would be a natural and useful
feature.  There might be no need any more for fluids.

Re: Emacs Lisp and Guile

[Richard Stallman]

    When coding-system-for-write would have been designed for Scheme, I
    would expect it to hidden behind an abstraction; it would not be a
    variable that you can set.  It could be offered as

	(with-coding-system-for-write 'foo
	  (lambda () ...))

This approach is unacceptable.  There are hundreds of such variables.
If we had to create several macros-and-functions for each one,
and document them, that would be a gigantic amount of work (which we
cannot afford to do) and would result in a bloated and cumbersome system.

These variables in Lisp must be variables in Scheme as well.

    (with-variable 'coding-system-for-write 'foo
      ...statements...)

That would be feasible, but it is much less clean that handling
coding-system-for-write as a variable.

    works for all variables.  Of course, you might as well call it `let',
    then :-)

Exactly.  We should extend Guile so that its mechanism for handling
variables dynamically is up to the job; then we should use that mechanism
for these variables.

Re: Emacs Lisp and Guile

[Sam Steingold]

> * In message <200208022214.g72MELZ01879@aztec.santafe.edu>
> * On the subject of "Re: Emacs Lisp and Guile"
> * Sent on Fri, 2 Aug 2002 16:14:21 -0600 (MDT)
> * Honorable Richard Stallman <rms@gnu.org> writes:
> ...
> This approach is unacceptable.  There are hundreds of such variables.
> If we had to create several macros-and-functions for each one, and
> document them, that would be a gigantic amount of work (which we
> cannot afford to do) and would result in a bloated and cumbersome
> system.
> 
> These variables in Lisp must be variables in Scheme as well.
> 
> We should extend Guile so that its mechanism for handling variables
> dynamically is up to the job; then we should use that mechanism for
> these variables.

yeah - let's extend Guile to be a kind of Common Lisp.

Loading Emacs Lisp into CLISP was a weekend exercise for me a couple of
years ago.  It works perfectly (tested on Emacs calendar).

Loading Emacs Lisp into Guile requires major extensions.
It might be more fun this way, but if you want to get the job done,
this is the sure way to fail.

A lot of _really_ smart people contributed to the design of Common Lisp.
It is a shame that you want to throw away all that good work and
redesign everything from scratch.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.3 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
There's always free cheese in a mouse trap.

Re: Emacs Lisp and Guile

[Simon Josefsson]

Sam Steingold <sds@gnu.org> writes:

> Loading Emacs Lisp into CLISP was a weekend exercise for me a couple of
> years ago.  It works perfectly (tested on Emacs calendar).
>
> Loading Emacs Lisp into Guile requires major extensions.
> It might be more fun this way, but if you want to get the job done,
> this is the sure way to fail.
>
> A lot of _really_ smart people contributed to the design of Common Lisp.
> It is a shame that you want to throw away all that good work and
> redesign everything from scratch.

Since visibility of the work on Guile Emacs seem to be pretty much
nil, one way to prove that CLISP would be a better way would be to
simply get a barely working emacs clone running on it and document
some design considerations and thoughts.  I'm sure it would attract
people that want to work on it.

Can CLISP take advantage of the non-ASCII work that has gone into
GNOME?  This seems to be one major work item for a Guile port.
Leveraging on the GNOME work seem to give many modern features easily
(Unicode, bidi).

Re: Emacs Lisp and Guile

[Sam Steingold]

> * In message <ilur8hgt0jq.fsf@h133n1c1o299.bredband.skanova.com>
> * On the subject of "Re: Emacs Lisp and Guile"
> * Sent on Sat, 03 Aug 2002 17:38:33 +0200
> * Honorable Simon Josefsson <jas@extundo.com> writes:
>
> Since visibility of the work on Guile Emacs seem to be pretty much
> nil, one way to prove that CLISP would be a better way would be to
> simply get a barely working emacs clone running on it and document
> some design considerations and thoughts.  I'm sure it would attract
> people that want to work on it.

I am not so sure about it.

> Can CLISP take advantage of the non-ASCII work that has gone into
> GNOME?  This seems to be one major work item for a Guile port.
> Leveraging on the GNOME work seem to give many modern features easily
> (Unicode, bidi).

CLISP uses Unicode.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.3 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
Linux - find out what you've been missing while you've been rebooting Windows

Re: Emacs Lisp and Guile

[Lars Magne Ingebrigtsen]

Sam Steingold <sds@gnu.org> writes:

>> Since visibility of the work on Guile Emacs seem to be pretty much
>> nil, one way to prove that CLISP would be a better way would be to
>> simply get a barely working emacs clone running on it and document
>> some design considerations and thoughts.  I'm sure it would attract
>> people that want to work on it.
>
> I am not so sure about it.

I, for one, would be quite interested in working on a CL Emacs, if
there was any hope of there being a chance of it succeeding.

My day job is writing Common Lisp stuff, and there are quite a few
times I've yearned for the power of Emacs when doing stuff in Common
Lisp.  (Mostly text-based stuff, but there are just so many libraries
in Emacs for doing, well, absolutely anything, that it would be so
handy...)

And the reverse as well -- when writing Emacs Lisp, I'm crying out
for the power of a Common Lisp underneath the hood.  

A combination of a powerful Lisp with Emacs is extremely alluring.

(It's not that Scheme is a bad idea; it's just that there's a lot of
thought gone into Common Lisp on how to do, er, a *lot* of things.
Scheme is a lot more stripped down, and when you add new stuff to fill
in the blanks, you usually end up with something that doesn't cohere
as well as a mature implementation.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Emacs Lisp and Guile

[Kai Großjohann]

Simon Josefsson <jas@extundo.com> writes:

> Can CLISP take advantage of the non-ASCII work that has gone into
> GNOME?  This seems to be one major work item for a Guile port.
> Leveraging on the GNOME work seem to give many modern features easily
> (Unicode, bidi).

I thought the idea was to embed a Lisp in Emacs to replace Emacs
Lisp.  So all the coding system stuff can be taken from Mule.

And then, if desired, more and more of the C stuff can be moved to
the new Lisp, if the new Lisp can do that.

kai
-- 
A large number of young women don't trust men with beards.  (BFBS Radio)

Re: Emacs Lisp and Guile

[Simon Josefsson]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Can CLISP take advantage of the non-ASCII work that has gone into
>> GNOME?  This seems to be one major work item for a Guile port.
>> Leveraging on the GNOME work seem to give many modern features easily
>> (Unicode, bidi).
>
> I thought the idea was to embed a Lisp in Emacs to replace Emacs
> Lisp.  So all the coding system stuff can be taken from Mule.
>
> And then, if desired, more and more of the C stuff can be moved to
> the new Lisp, if the new Lisp can do that.

Yes, you are right, altough I think it would be nice if Emacs could
take advantage of the work that went into libc for low-level non-ASCII
manipulation, and into GNOME (GTK) for high-level non-ASCII display
(e.g., bidirectional text), instead of re-implementing it all just for
Emacs.  It doesn't seem very re-usable.  Ideally all non-ASCII stuff
should go into the lisp (CLISP or Guile) used, so other applications
than Emacs can take advantage of it.  All IMHO, of course.

Re: Emacs Lisp and Guile

[Sam Steingold]

> * In message <iluu1mb65hg.fsf@h133n1c1o299.bredband.skanova.com>
> * On the subject of "Re: Emacs Lisp and Guile"
> * Sent on Sat, 03 Aug 2002 22:40:27 +0200
> * Honorable Simon Josefsson <jas@extundo.com> writes:
>
> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
> 
> > Simon Josefsson <jas@extundo.com> writes:
> >
> >> Can CLISP take advantage of the non-ASCII work that has gone into
> >> GNOME?  This seems to be one major work item for a Guile port.
> >> Leveraging on the GNOME work seem to give many modern features easily
> >> (Unicode, bidi).
> >
> > I thought the idea was to embed a Lisp in Emacs to replace Emacs
> > Lisp.  So all the coding system stuff can be taken from Mule.
> >
> > And then, if desired, more and more of the C stuff can be moved to
> > the new Lisp, if the new Lisp can do that.
> 
> Yes, you are right, altough I think it would be nice if Emacs could
> take advantage of the work that went into libc for low-level non-ASCII
> manipulation, and into GNOME (GTK) for high-level non-ASCII display
> (e.g., bidirectional text), instead of re-implementing it all just for
> Emacs.  It doesn't seem very re-usable.  Ideally all non-ASCII stuff
> should go into the lisp (CLISP or Guile) used, so other applications
> than Emacs can take advantage of it.  All IMHO, of course.

CLISP i18n/Unicode functionality is head & shoulders above that of any
other software I have seen.  It was done by an expert in the field -
Bruno Haible.  See <http://clisp.cons.org/impnotes.html#i18n> and
<http://clisp.cons.org/impnotes.html#encoding>.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.3 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
I may be getting older, but I refuse to grow up!

Re: Emacs Lisp and Guile

[Eli Zaretskii]

On Sat, 3 Aug 2002, Simon Josefsson wrote:

>                               I think it would be nice if Emacs could
> take advantage of the work that went into libc for low-level non-ASCII
> manipulation, and into GNOME (GTK) for high-level non-ASCII display
> (e.g., bidirectional text), instead of re-implementing it all just for
> Emacs.  It doesn't seem very re-usable.

The display part of the bidirectional editing support must be 
reimplemented for Emacs because the Emacs display engine examines the 
buffer text one character at a time.  For each character examined, 
Emacs decides how to display it and also whether to examine more of the 
buffer.  By contrast, all the bidi libraries out there accept a buffer 
of text and return that buffer reordered for display.  Making the Emacs 
display work in such a batch-style manner means a total rewrite of the 
display code, and also loses over slow connections, which AFAIK we still 
want to support.

That's the main reason why Emacs development didn't use fribidi or some 
similar library, although people who (slowly) work on bidi Emacs are well 
aware of fribidi and similar efforts.

Re: Emacs Lisp and Guile

[Richard Stallman]

    > We should extend Guile so that its mechanism for handling variables
    > dynamically is up to the job; then we should use that mechanism for
    > these variables.

    yeah - let's extend Guile to be a kind of Common Lisp.

I am talking about a small change in features that already exist.  The
issue at hand is for features Common Lisp does not have.

    A lot of _really_ smart people contributed to the design of Common Lisp.
    It is a shame that you want to throw away all that good work and
    redesign everything from scratch.

You mean smart people like Guy Steele, who is also co-designer of Scheme?

Re: Emacs Lisp and Guile

[Richard Stallman]

I am not interested in hosting Emacs on Common Lisp
because that would require adding far too much to it.

Re: Emacs Lisp and Guile

[Marius Vollmer]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Maybe something can be abstracted so that
> 
> (with-variable 'coding-system-for-write 'foo
>   ...statements...)
> 
> works for all variables.

Yes.

> Of course, you might as well call it `let', then :-)

That wouldn't be right, for Scheme.

Re: Emacs Lisp and Guile

[Sam Steingold]

RMS = Richard Stallman <rms@gnu.org>
LMI = Lars Magne Ingebrigtsen <larsi@gnus.org>

RMS> I am not interested in hosting Emacs on Common Lisp

LMI> I, for one, would be quite interested in working on a CL Emacs, if
LMI> there was any hope of there being a chance of it succeeding.

I would rather not create yet another Emacs fork.
If you (we?) can convince either XEmacs people or RMS that Common Lisp
is the way to go, I will contribute.

RMS> because that would require adding far too much to it.

Actually, this is a red herring: the amount to be added to Emacs is
minuscule compared to what is already there, and Emacs will not grow
noticeably because of that.

I am pretty sure that Guile+"3 widely used Scheme object systems" is
larger than Common Lisp.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.3 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
"Complete Idiots Guide to Running LINUX Unleashed in a Nutshell for Dummies"

Re: Emacs Lisp and Guile

[Marius Vollmer]

Richard Stallman <rms@gnu.org> writes:

> Emacs Lisp has hundreds of dynamic variables, whose names are
> documented, that user programs examine, set, and bind.  It would be
> clumsy and impractical to hide each of these variables behind a
> separate function in Scheme.  It would be practical but clumsy to
> have a few Scheme functions for accessing and binding Lisp
> variables.  That would make Scheme very inconvenient to use,
> compared with Emacs Lisp.

Agreed.

>     What about exporting all Elisp variables to Scheme as settable
>     functions?  For example
> 
> 	(set! (require-final-newline) 'ask)
> 
>     This would lead to a divergence between Scheme variables and Lisp
>     variables, but I don't think this is bad.
> 
> I think it is ugly--variables should remain variables.

I do like it.  It would also make it trivial to include the
optimizations that you talk about below.  When Elisp variables should
appear as Scheme variables we would need to implement a new kind of
variable in the core of Guile (which is no argument against doing it).

Right now, a variable is a strictly passive data container.  You can't
intercept accesses to it.  A variable reference is a direct reference
to memory, not a function call.  We would have to change this model to
also allow 'active' variables that can run arbitrary code when they
are accessed.

Whenever some code uses a variable, it can either be a passive
variable or an active variable.  It needs to treat the two cases
differently since we want to make use of the extra performance that
passive variables allow.  We would need to make sure that the
distinction between 'passive' and 'active' variables can be made at
compile time, but that should be doable as well.

This makes 'active' variables pretty similar to symbol macros, so
maybe we should just use them.

> There should be a list of current local contexts, which the user
> program can use for different purposes.  Emacs would use one for the
> current buffer's bindings, and one for the current frame's bindings.

Yes, that should be no problem.

>     I'm afraid this might turn out to be expensive and it is not a good
>     solution in a multi-threaded system.
> 
> It does not have to be expensive.  Although Emacs is not
> multi-threaded, it has an optimization for this: a variable's
> buffer-local bindings are not swapped until you try to access the
> variable.  Runnning another thread for a short time, if it does not
> access many of these variables, won't have to swap many of them.

This assumes that only one thread is running at a time.  It wouldn't
work for a machine with multiple CPUs where threads can run at the
same time and the concept of a "thread switch" does not exist.  Maybe
we don't want to care about this, which I would be very happy with.

>       Fluids
>     provide thread-local storage and dynamic-wind is used to swap values
>     in and out of these locations to establish dynamic contexts.
> 
> [...]
> Or are you saying that dynamic-wind can be used either to swap values
> in and out of a variable, or to swap values in and out of a fluid?

That's what I wanted to say.  A fluid can have one value per thread.
Dynamic-wind is then used for the dynamic contexts within a single
thread.  The fluids implement the per-thread storage without having to
rely thread switching points.

> Do fluids have an optimized mechanism like the one I described above
> for Emacs buffer-local variables?  Or does each access to a fluid look
> up the binding for the current thread?

The latter. Each thread has a vector and a fluid is implemented as an
index into that vector.  Accessing a fluid consists of finding the
current thread, finding the vector of that thread and indexing it.
(Indices that are outside the vector yield #f on read and enlarge the
vector on write.)

Re: Emacs Lisp and Guile

[Sam Steingold]

> * In message <200208042325.g74NPtM03391@aztec.santafe.edu>
> * On the subject of "Re: Emacs Lisp and Guile"
> * Sent on Sun, 4 Aug 2002 17:25:55 -0600 (MDT)
> * Honorable Richard Stallman <rms@gnu.org> writes:
>
>     > We should extend Guile so that its mechanism for handling variables
>     > dynamically is up to the job; then we should use that mechanism for
>     > these variables.
> 
>     yeah - let's extend Guile to be a kind of Common Lisp.
> 
> I am talking about a small change in features that already exist.  The
> issue at hand is for features Common Lisp does not have.

such as?  dynamic (special) variables?

>     A lot of _really_ smart people contributed to the design of Common Lisp.
>     It is a shame that you want to throw away all that good work and
>     redesign everything from scratch.
> 
> You mean smart people like Guy Steele, who is also co-designer of Scheme?

You are missing the point, I am afraid.

Scheme _is_ a very nice language.  It is very well designed for the
purpose for which it was designed - CS education.

The problem with Scheme is that, without significant extensions (object
system, condition system, dynamic variables, multiple values, SETF &c),
it is not suitable for large projects like Emacs.

Yes, all such extensions can be done, and there are _several_ different
incompatible object systems for Scheme, i.e., Scheme is at the stage
where Common Lisp was 20 years ago, when there were several CL object
systems.

The whole point is that lots of thought went into things that CL _has_
and R5RS does _not_ have.
It is a shame that you want to discard all these design assets and
re-do all the great work that has already been done.

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.3 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
Bus error -- please leave by the rear door.

Re: Emacs Lisp and Guile

[Richard Stallman]

    Right now, a variable is a strictly passive data container.  You can't
    intercept accesses to it.  A variable reference is a direct reference
    to memory, not a function call.  We would have to change this model to
    also allow 'active' variables that can run arbitrary code when they
    are accessed.

That is not what I am suggesting.  Variables in Emacs do not run
arbitrary code when accessed; I think that would be a misguided
feature.  However, Emacs enables the value to reside in various places
besides the symbol itself.

For instance, a variable can be forwarded to a C variable.  There are
three ways of doing this: the C variable holds an integer, the C
variable holds a Lisp object, or the C variable holds a boolean value
(0 means nil, non-0 means t).

A similar feature in Guile would be useful for many sorts of
application programs.

This is independent of the feature of buffer-local and frame-local
variable bindings.  Their implementations in Emacs are not totally
independent, but perhaps they could be implemented independently in
Guile.

Re: Emacs Lisp and Guile

[Richard Stallman]

    This assumes that only one thread is running at a time.  It wouldn't
    work for a machine with multiple CPUs where threads can run at the
    same time and the concept of a "thread switch" does not exist.  Maybe
    we don't want to care about this, which I would be very happy with.

It is useful for Guile to support real parallel execution of threads,
but it is not important for Emacs.  Having multiple threads in Emacs
will be very useful, but it is perfectly adequate if only one thread
can run at a time.  It is not worth accepting any inconvenience in
Emacs to support parallel execution.  Instead, Guile should provide a
way that Emacs can turn it off.

In fact, it would simplify matters greatly if Emacs can specify where
a thread switch may occur.  Is that implemented in Guile?

Re: Emacs Lisp and Guile

[Simon Josefsson]

Richard Stallman <rms@gnu.org> writes:

>     This assumes that only one thread is running at a time.  It wouldn't
>     work for a machine with multiple CPUs where threads can run at the
>     same time and the concept of a "thread switch" does not exist.  Maybe
>     we don't want to care about this, which I would be very happy with.
>
> It is useful for Guile to support real parallel execution of threads,
> but it is not important for Emacs.  Having multiple threads in Emacs
> will be very useful, but it is perfectly adequate if only one thread
> can run at a time.

I beg to differ -- one fundamental design principle in GUI design is
that the user interface should never "hang" due to processing in the
application.  The user should be able to see e.g. the toolbar and the
buttons and tooltips etc should be updated even when Emacs is
processing something.  Ideally long standing operations should have a
"Stop" button in the toolbar that aborts the operation.

Perhaps this isn't in direct conflict with your statement though, if
the thread switching is fast enough so that users doesn't notice it.

Re: Emacs Lisp and Guile

[Richard Stallman]

It is not feasible to make Emacs handle truly asynchronous
thread-switching.  It would require too much change in too many
places.

Re: Emacs Lisp and Guile

[Marius Vollmer]

Richard Stallman <rms@gnu.org> writes:

> In fact, it would simplify matters greatly if Emacs can specify where
> a thread switch may occur.  Is that implemented in Guile?

Yes.  We only support cooperative threads right now and thread
switches can only occur during calls to some specific functions like
'yield', 'lock-mutex', ' wait-condition-variable', etc.  You also need
to call SCM_TICK occasionally in long runnning computations to allow
other threads to run.  However, there is no way to prevent 'yield' or
SCM_TICK from switching during critical sections.  You have to use
mutex for this.

Re: Emacs Lisp and Guile

[Marius Vollmer]

Richard Stallman <rms@gnu.org> writes:

>     Right now, a variable is a strictly passive data container.  You can't
>     intercept accesses to it.  A variable reference is a direct reference
>     to memory, not a function call.  We would have to change this model to
>     also allow 'active' variables that can run arbitrary code when they
>     are accessed.
> 
> That is not what I am suggesting.  Variables in Emacs do not run
> arbitrary code when accessed; I think that would be a misguided
> feature.  However, Emacs enables the value to reside in various places
> besides the symbol itself.

I think we might agree anyway: from a general, low-level Scheme
implementation point of view, Elisp variables require 'arbitrary' code
to be run when they are accessed.  From the Elisp point of view this
code isn't arbitrary at all, it is just the code that is needed to
implement the desired behavior of Elisp variables.  To the Elisp
programmer, variables are passive data containers, but to the Scheme
implementation, they are 'active'.

Thus, allowing 'active' variables in Guile would be a way to implement
the behavior of Elisp variables.  The active variables would not need
to be completely general; they could be restricted to just the set of
features that are needed for Elisp variables (like forwarding to C
variables).  But I don't think that would be an advantage.  They would
still be different from ordinary Scheme variables, but not as useful
as they might be.  If I remember correctly, TCL can sent out
notifications when the value of a variable changes.  This feature
could be implemented on top of the general active variables without
having to change the low-level Guile execution model.  It might not be
implementable with just the Elisp behavior of variables.

But there is also the question of how to present active variables to
Scheme code, once we have them.  Since passive variables are a special
case of active variables, we can just let all variables be active
variables and get a very consistent system.

But I think it is highly desirable to be able to distinguish between
passive variables and active ones at compile time.  We do have to
distinguish between functions and macros already at compile time, so
this would not be an entirely new concept.  We also will want to
identify calls to certain functions at compile time so that they can
be optimized.  Since telling passive variables from active variables
is merely an optimization, we can put it in the latter category.

> A similar feature in Guile would be useful for many sorts of
> application programs.

Yes.

Re: Emacs Lisp and Guile

[Stefan Monnier]

>     Right now, a variable is a strictly passive data container.  You can't
>     intercept accesses to it.  A variable reference is a direct reference
>     to memory, not a function call.  We would have to change this model to
>     also allow 'active' variables that can run arbitrary code when they
>     are accessed.
> 
> That is not what I am suggesting.  Variables in Emacs do not run
> arbitrary code when accessed; I think that would be a misguided
> feature.

In Scheme we can use fluid-let to implement Elisp-style dynamic binding.
The only difference with Elisp is that you use `fluid-let' instead
of `let' when binding a variable dynamically.

I think we should aim to move in the direction where most let-bound
variables are lexically scoped and the rest (i.e. `defvar'd variables)
is accessed/modified via special functions/macros.

In most cases (in Elisp practice) lexical or dynamic scoping of let-bound
(and not defvar'd) variables has exactly the same semantics.  Miles'
`lexbind' branch is trying to do just that as is someone on the XEmacs
side (the means are different, tho).

If we can get high enough compatibility (maybe full compatibility is
even possible), that would mean that running "arbitrary code" when
accessing a dynamic variable is at least not a performance problem
(and it can be very convenient when debugging).  It could also be
useful for meta-variables (such that setting them modifies all
those that depend on it: it currently can be done in very limited
ways via the :set parameter to defcustom).


	Stefan

Re: Emacs Lisp and Guile

[Richard Stallman]

      However, there is no way to prevent 'yield' or
    SCM_TICK from switching during critical sections.

I'm not sure I understand that properly.  If we avoid calling yield
or SCM_TICK within critical sections, won't that prevent thread
switches within critical sections?

Re: Emacs Lisp and Guile

[Richard Stallman]

    I think we might agree anyway: from a general, low-level Scheme
    implementation point of view, Elisp variables require 'arbitrary' code
    to be run when they are accessed.

They don't require this--there are other ways to implement this
feature in and on Guile.

It is true that if Guile supports running arbitrary code for variable
access, that would provide a basis on which to implement the Emacs
Lisp variable features.  But I think that running arbitrary code is not
good design--variables should be variables, not act like functions.

Can you look for some way to implement forwarding facilities in Guile?

      If I remember correctly, TCL can sent out
    notifications when the value of a variable changes.

That feature can be useful, but it is not sufficient for this job.
When a variable is forwarded to C code, the idea is that C code can
change the value at any time.  Each reference to the Scheme variable
needs to look in the C variable.

Re: Emacs Lisp and Guile

[Alex Schroeder]

Richard Stallman <rms@gnu.org> writes:

> I am not interested in hosting Emacs on Common Lisp
> because that would require adding far too much to it.

I do not understand the statement -- what does "it" refer to, and why
is there stuff to add -- and what does is added, size or functionality
or complexity?

After all, the Guile discussion seemed to indicate a lot of things
that have to be added to Guile.  So in my simplistic model, we either
add lots of stuff to Guile and and get Emacs + Scheme (a simple
language without many libraries), or we make some changes to Emacs to
incorporate CLISP and get Emacs + Common Lisp (a powerful and complex
language without many libraries).  In either case, Emacs is powerful
and complex and has many libraries.  Using Common Lisp will make
things more complex, but not increase the workload as far as I can
tell -- and the complexity of CL buys us the power of CL, something
many people have yearned for.  Using Scheme on the other hand does not
save us workload, and eventhough complexity seems manageable, the
power increase is not obvious either.  Faced with this, it seems to me
that I'd rather have Emacs + CL than Emacs + Scheme.

Alex.
-- 
http://www.electronicintifada.net/diaries/index.html
http://www.us-israel.org/jsource/US-Israel/hr2506c.html

Re: Emacs Lisp and Guile

[Noah Friedman]

>Yes, all such extensions can be done, and there are _several_ different
>incompatible object systems for Scheme, i.e., Scheme is at the stage
>where Common Lisp was 20 years ago, when there were several CL object
>systems.

I don't expect that the decision is going to be made based on popularity,
but for what it's worth I agree with Sam.  I hate writing Scheme programs
because every implementation is incompatible and usually incomplete.  On
the other hand I've been able to take CL programs and run them on a
dozen systems.  

Every few decades (or less) someone gets it in their head that the large,
complex but feature-complete languages they've been using are too big for
some project, and they start all over again with some toy language that is
more suitably scoped for their project.  And then it starts accreting
features until it's just as big and hairy as the language everyone ran away
from in the first place.  Scheme has had roughly 20 years to catch up and
has not succeeded.  As a research language it seems less compelling to me
than ML and as an applications language it's less compelling than CL or
Python.

Emacs is already a big program, but program size is not the biggest concern
in software development anymore.  Lack of code sharing IS.  Too many people
have spent too many years writing libraries and interfaces in one language
or another that cannot be used by other applications that have some other
foundation, and so more people have to waste time reimplementing them.  I
for one am tired of (e.g.) being unable to reuse Perl modules in Python, or
being unable to reuse CL libraries in Emacs or sawfish.

But at this point I am not that keen on replacing Emacs' interpreter if the
idiosyncrasies of maclisp (e.g. dynamic scope) and editor-specific hacks
like buffer and frame local variables are so deeply embedded in it that the
replacement language will have to mimic all of its behavior AND try to
remain faithful to whatever semantics the replacement language
(e.g. Scheme) is supposed to have.  If you use Scheme, the resulting
language is going to be incomprehensible to anyone who knows just R5RS.

What exactly is anyone gaining by this?  Could someone explain to me why
effort should be expended on it at all?

Re: Emacs Lisp and Guile

[Alex Schroeder]

Noah Friedman <friedman@splode.com> writes:

> What exactly is anyone gaining by this?  Could someone explain to me why
> effort should be expended on it at all?

I like the approch take by pymac -- those who want to write extensions
in Python can now do this without patching Emacs.  Emacs communicates
with an external process to do it.  Would such a solution -- perhaps
with even better integration into Emacs not involve the least amount
of work and a useful subset of features.  Writing Emacs extensions in
such languages will be not be like writing native programs, because
communication with Emacs has to be cared about, but at least it is a
starting point.

Alex.
-- 
http://www.electronicintifada.net/diaries/index.html
http://www.us-israel.org/jsource/US-Israel/hr2506c.html

Re: Emacs Lisp and Guile

[Michael Sperber [Mr. Preprocessor]]

>>>>> "RMS" == Richard Stallman <rms@gnu.org> writes:

RMS> The clean approach is that each of these Lisp variables is simply a
RMS> Scheme variable.

RMS> [...]

RMS> I think it is ugly--variables should remain variables.

Except the variable concepts in Emacs Lisp and Scheme are very
different.  Forcing the Emacs Lisp concept upon Scheme would be pretty
ugly and unclean.

Furthermore, I don't see why you think this is necessary.  I've seen a
lot of verbiage on this thread, but you haven't been very specific on
this issue.  (Maybe I've missed something; if so, I'd ask some kind
soul to rub my nose in that place.)

What exactly is the problem of marking parameter-like dynamic
variables specially and, in Scheme, using different constructs from
Emacs Lisp to bind, set, and access them?

-- 
Cheers =8-} Mike
Friede, Völkerverständigung und überhaupt blabla

Re: Emacs Lisp and Guile

[Richard Stallman]

    In Scheme we can use fluid-let to implement Elisp-style dynamic binding.
    The only difference with Elisp is that you use `fluid-let' instead
    of `let' when binding a variable dynamically.

Could you show me precisely what this looks like, and explain it?
It might be a good approach, but I don't know enough now to judge.

Re: Emacs Lisp and Guile

[Richard Stallman]

    > I am not interested in hosting Emacs on Common Lisp
    > because that would require adding far too much to it.

    I do not understand the statement -- what does "it" refer to, and why
    is there stuff to add -- and what does is added, size or functionality
    or complexity?

I am sorry.  My point was that Common Lisp would be an undesirably
large addition to Emacs.  I have never greatly admired Common Lisp
having implemented it once.

(Also, Scheme is the standard GNU extensibility language and has been
for many years.  The decision was made long ago, and I am not interested
in reconsidering it.)

Re: Emacs Lisp and Guile

[Richard Stallman]

    Except the variable concepts in Emacs Lisp and Scheme are very
    different.  Forcing the Emacs Lisp concept upon Scheme would be pretty
    ugly and unclean.

It appears they are not so different, and I think it can be done cleanly.

Can someone tell me the precise specs of dynamic-wind?

Re: Emacs Lisp and Guile

[Michael Sperber [Mr. Preprocessor]]

>>>>> "RMS" == Richard Stallman <rms@gnu.org> writes:

RMS>     Except the variable concepts in Emacs Lisp and Scheme are very
RMS>     different.  Forcing the Emacs Lisp concept upon Scheme would be pretty
RMS>     ugly and unclean.

RMS> It appears they are not so different, and I think it can be done cleanly.

How and why?  Things like buffer-local variables are concepts broken
in principle in Emacs Lisp.  Moving to Scheme is a chance to correct
these historic mistakes.

RMS> Can someone tell me the precise specs of dynamic-wind?

Check out R5RS.  For example, at:

http://www.schemers.org/Documents/Standards/R5RS/HTML/r5rs-Z-H-9.html#%_idx_576

-- 
Cheers =8-} Mike
Friede, Völkerverständigung und überhaupt blabla

Re: Emacs Lisp and Guile

[Sam Steingold]

> * In message <20020810001749.695883.FMU31823@piglet.prv.splode.com>
> * On the subject of "Re: Emacs Lisp and Guile"
> * Sent on Sat, 10 Aug 2002 00:17:49 -0700 (PDT)
> * Honorable Noah Friedman <friedman@splode.com> writes:
>
> >Yes, all such extensions can be done, and there are _several_ different
> >incompatible object systems for Scheme, i.e., Scheme is at the stage
> >where Common Lisp was 20 years ago, when there were several CL object
> >systems.
> 
> I don't expect that the decision is going to be made based on
> popularity, but for what it's worth I agree with Sam.  I hate writing
> Scheme programs because every implementation is incompatible and
> usually incomplete.  On the other hand I've been able to take CL
> programs and run them on a dozen systems.

Precisely!

Okay, RMS is not listening.

Maybe XEmacs maintainers are more receptive to the voice of reason?

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat7.3 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
If your VCR is still blinking 12:00, you don't want Linux.

Re: Emacs Lisp and Guile

[Marius Vollmer]

Richard Stallman <rms@gnu.org> writes:

>       However, there is no way to prevent 'yield' or
>     SCM_TICK from switching during critical sections.
> 
> I'm not sure I understand that properly.  If we avoid calling yield
> or SCM_TICK within critical sections, won't that prevent thread
> switches within critical sections?

Yes, but you might not be able to avoid to call a function during a
critical section that calls SCM_TICK without you wanting it.  Having
to keep track of which functions are guaranteed to not to call
SCM_TICK or yield is not very reliable, I think.

Re: Emacs Lisp and Guile

[Marius Vollmer]

Richard Stallman <rms@gnu.org> writes:

> But I think that running arbitrary code is not good
> design--variables should be variables, not act like functions.

But they could be used as the basis for all kind of funny
variable-like behavior.  I don't think we want to support all
possibilities directly in the Guile core.  Having variables that act
like functions is enough and we can build the rest on top of them,
without having to change the core.  The latter is important since
people might want to replace the execution engine of Guile (with a
compiler to machine code, say).

The TCL notification example was meant to show that there are also
other behaviors of variables that could be implemented on top of
variables-that-act-like-functions.  We could do that without having to
add an explicit notification feature to the Guile core.

Variables-that-act-like-functions can also be abused for things that
are better done by real functions, but I think it is wrong to not
offer powerful features only because they can be abused.

> Can you look for some way to implement forwarding facilities in
> Guile?

Ok.

Re: Emacs Lisp and Guile

[Richard Stallman]

    How and why?  Things like buffer-local variables are concepts broken
    in principle in Emacs Lisp.  Moving to Scheme is a chance to correct
    these historic mistakes.

We seem to have very basic disagreements.  It is not likely I would
agree with recommendations that you make on this issue.

    Check out R5RS.  For example, at:

    http://www.schemers.org/Documents/Standards/R5RS/HTML/r5rs-Z-H-9.html#%_idx_576

I will try to send email to fetch this.

Re: Emacs Lisp and Guile

[Richard Stallman]

    Yes, but you might not be able to avoid to call a function during a
    critical section that calls SCM_TICK without you wanting it.

Perhaps critical sections could set a flag that tells SCM_TICK not
to really do anything.

Re: Emacs Lisp and Guile

[Richard Stallman]

    > But I think that running arbitrary code is not good
    > design--variables should be variables, not act like functions.

    But they could be used as the basis for all kind of funny
    variable-like behavior.

It could be.  I wonder if there is a way to implement this that
discourages misuse of the feature.

Perhaps, instead of providing a way to say "run this code when there
is an access to variable FOO", it could have to be done in two steps:
first, define a feature for something special to do when accessing
certain variables; second, turn on one of these features for a certain
variable.

Of course, it is possible to write a higher level feature that would
package up and combine those two steps.  It would not be difficult.
But many people would get the point and realize that was the wrong
thing to do.

Re: Emacs Lisp and Guile

[Michael Sperber [Mr. Preprocessor]]

[-- Attachment #1: Type: text/plain, Size: 641 bytes --]

>>>>> "RMS" == Richard Stallman <rms@gnu.org> writes:

RMS>     How and why?  Things like buffer-local variables are concepts broken
RMS>     in principle in Emacs Lisp.  Moving to Scheme is a chance to correct
RMS>     these historic mistakes.

RMS> We seem to have very basic disagreements.  It is not likely I would
RMS> agree with recommendations that you make on this issue.

Is it possible to have a discussion on this?

RMS>     Check out R5RS.  For example, at:

RMS>     http://www.schemers.org/Documents/Standards/R5RS/HTML/r5rs-Z-H-9.html#%_idx_576

RMS> I will try to send email to fetch this.

I've included the text verbatim.


[-- Attachment #2: Type: text/plain, Size: 2680 bytes --]

procedure:  (dynamic-wind before thunk after) 

Calls thunk without arguments, returning the result(s) of this
call. Before and after are called, also without arguments, as required
by the following rules (note that in the absence of calls to
continuations captured using call-with-current-continuation the three
arguments are called once each, in order). Before is called whenever
execution enters the dynamic extent of the call to thunk and after is
called whenever it exits that dynamic extent. The dynamic extent of a
procedure call is the period between when the call is initiated and
when it returns. In Scheme, because of call-with-current-continuation,
the dynamic extent of a call may not be a single, connected time
period. It is defined as follows:

    * The dynamic extent is entered when execution of the body of the
      called procedure begins.

    * The dynamic extent is also entered when execution is not within
      the dynamic extent and a continuation is invoked that was
      captured (using call-with-current-continuation) during the
      dynamic extent.

    * It is exited when the called procedure returns.

* It is also exited when execution is within the dynamic extent and a
  continuation is invoked that was captured while not within the
  dynamic extent.

If a second call to dynamic-wind occurs within the dynamic extent of
the call to thunk and then a continuation is invoked in such a way
that the afters from these two invocations of dynamic-wind are both to
be called, then the after associated with the second (inner) call to
dynamic-wind is called first.

If a second call to dynamic-wind occurs within the dynamic extent of
the call to thunk and then a continuation is invoked in such a way
that the befores from these two invocations of dynamic-wind are both
to be called, then the before associated with the first (outer) call
to dynamic-wind is called first.

If invoking a continuation requires calling the before from one call
to dynamic-wind and the after from another, then the after is called
first.

The effect of using a captured continuation to enter or exit the
dynamic extent of a call to before or after is undefined.

(let ((path '())
      (c #f))
  (let ((add (lambda (s)
               (set! path (cons s path)))))
    (dynamic-wind
      (lambda () (add 'connect))
      (lambda ()
        (add (call-with-current-continuation
               (lambda (c0)
                 (set! c c0)
                 'talk1))))
      (lambda () (add 'disconnect)))
    (if (< (length path) 4)
        (c 'talk2)
        (reverse path))))
    
                ===> (connect talk1 disconnect
               connect talk2 disconnect)

[-- Attachment #3: Type: text/plain, Size: 72 bytes --]



-- 
Cheers =8-} Mike
Friede, Völkerverständigung und überhaupt blabla

Re: Emacs Lisp and Guile

[Marius Vollmer]

Richard Stallman <rms@gnu.org> writes:

>     > But I think that running arbitrary code is not good
>     > design--variables should be variables, not act like functions.
> 
>     But they could be used as the basis for all kind of funny
>     variable-like behavior.
> 
> It could be.  I wonder if there is a way to implement this that
> discourages misuse of the feature.

If we would implement this, we could first make it only available to
C.  I expect people to want to use it from Scheme immediately, and
yell at us for not giving it to them.  We could then answer that there
are better ways to achieve the same thing, like settable functions.

> Perhaps, instead of providing a way to say "run this code when there
> is an access to variable FOO", it could have to be done in two steps:
> first, define a feature for something special to do when accessing
> certain variables; second, turn on one of these features for a certain
> variable.

Yes.  I don't understand what you are proposing in detail, but I'm
thinking about adding flags to our existing variables: one would be
for read-only variables, one would be for 'dumb' variables that you
are allowed to access directly (which I think is important for
performance), and maybe others.  When a variable is not dumb, you need
to call scm_variable_ref, etc to access it, and these functions would
then be the single place where new behavior can be added.

If you don't care about performance too much, you can just use
scm_variable_ref, etc without looking at the flags.  Otherwise, the
flags tell you what shortcuts you are allowed to take.

The feature for doing something special would be implemented inside
scm_variabe_ref, etc.  To activate it, you would need to construct a
variable with the right flags.  You wouldn't be able to change the
flags of existing variables.

> Of course, it is possible to write a higher level feature that would
> package up and combine those two steps.  It would not be difficult.
> But many people would get the point and realize that was the wrong
> thing to do.

Yes, the variables that can run code on access need to be specially
constructed.  You wont be able to specify code to be run for an
arbitrary variable.  Sorry if I wasn't clear about this.

-- 
GPG: D5D4E405 - 2F9B BCCC 8527 692A 04E3  331E FAF8 226A D5D4 E405

Re: Emacs Lisp and Guile

[Marius Vollmer]

Richard Stallman <rms@gnu.org> writes:

>     Yes, but you might not be able to avoid to call a function during a
>     critical section that calls SCM_TICK without you wanting it.
> 
> Perhaps critical sections could set a flag that tells SCM_TICK not
> to really do anything.

Yes, but then we would have to admit that we don't support true
concurrent execution.  In a true concurrent system, you can't stop
other threads by setting a global flag.

An althernative would be to use a mutex for critical sections.

-- 
GPG: D5D4E405 - 2F9B BCCC 8527 692A 04E3  331E FAF8 226A D5D4 E405

Re: Emacs Lisp and Guile

[Richard Stallman]

Having now seen the R5RS description of dynamic-wind, I see it is a
very low-level mechanism.  For the sake of using dynamic-wind for
dynamic variable bindings, it would be convenient to have a function
to swap in bindings and a function to swap out bindings.  Then
when you write calls to dynamic-wind, you would call these functions,
not manipulate bindings directly.

These functions could also have code for correct interaction with
buffer-local bindings and frame-local bindings.  No special
primitive features would be needed for those, if they are implemented
in an unoptimized fashion and always swap the values in and out
immediately.

The hardest part of the mechanism that Emacs uses now to implement in
Guile is the optimization of not finding and swapping in a variable's
current binding until a reference occurs.  If Guile provided a
mechanism to run arbitrary code for a variable reference, we could use
that mechanism to implement this optimization.

Suppose that Guile could run arbitrary code before and after the
variable reference, for effect only, and the variable reference itself
would always occur normally.  This would not encourage people to use
variables for jobs that ought to be done by functions, but would
permit implementation of this optimization, and various sorts of
forwarding.  What do you think of this idea?

Re: Emacs Lisp and Guile

[Richard Stallman]

    Yes, the variables that can run code on access need to be specially
    constructed.

In Emacs, there is a fixed set of Lisp variables that forward to C
variables.  It would be fine to construct them specially.

However, any Lisp variable could have buffer-local and frame-local
bindings.  So if we are going to support the optimization of
not swapping in these bindings until a reference occurs,
it would be necessary to support it for variables that have
not been constructed specially.

Re: Emacs Lisp and Guile

[Richard Stallman]

    > Perhaps critical sections could set a flag that tells SCM_TICK not
    > to really do anything.

    Yes, but then we would have to admit that we don't support true
    concurrent execution.

In Emacs we don't want to support true concurrent execution.  It is
useful for Guile to provide convenient constructs for use in
applications that do not support true concurrent execution.

Applications that want true concurrent execution should not use
those constructs; perhaps you would get an error if you use them
in an application that allows concurrent execution.

Re: Emacs Lisp and Guile

[Marius Vollmer]

Richard Stallman <rms@gnu.org> writes:

> In Emacs we don't want to support true concurrent execution.  It is
> useful for Guile to provide convenient constructs for use in
> applications that do not support true concurrent execution.

Yes.  I agree that being able to rule out true concurrency is a good
thing.

> Applications that want true concurrent execution should not use
> those constructs; perhaps you would get an error if you use them
> in an application that allows concurrent execution.

Yes.  We can do that.

-- 
GPG: D5D4E405 - 2F9B BCCC 8527 692A 04E3  331E FAF8 226A D5D4 E405

Re: Emacs Lisp and Guile

[Marius Vollmer]

Richard Stallman <rms@gnu.org> writes:

> However, any Lisp variable could have buffer-local and frame-local
> bindings.  So if we are going to support the optimization of
> not swapping in these bindings until a reference occurs,
> it would be necessary to support it for variables that have
> not been constructed specially.

All Elisp variables would be constructed specially, even if they
didn't make use of any of the available features initially.  That
would mean that variables declared in Scheme can not be later made
buffer local since normal Scheme variabled would not be constructed in
the special way.  Scheme would have to declare from the start that a
variable is buffer-local (or frame-local, etc).  Would that be
acceptable?

-- 
GPG: D5D4E405 - 2F9B BCCC 8527 692A 04E3  331E FAF8 226A D5D4 E405

Re: Emacs Lisp and Guile

[Marius Vollmer]

Richard Stallman <rms@gnu.org> writes:

> Having now seen the R5RS description of dynamic-wind, I see it is a
> very low-level mechanism.  For the sake of using dynamic-wind for
> dynamic variable bindings, it would be convenient to have a function
> to swap in bindings and a function to swap out bindings.  Then
> when you write calls to dynamic-wind, you would call these functions,
> not manipulate bindings directly.

There is a nearly standard macro out there called 'fluid-let' that
encapsulates the process completely.  For example

  (fluid-let ((case-fold-search #f))
    (search-forward ...))

would expand to

  (let ((body (lambda () (search-forward ...)))
        (outer #f))
    (define (swap)
      (let ((t outer))
        (set! outer case-fold-search)
        (set! case-fold-search t)))
    (dynamic-wind swap body swap))

> These functions could also have code for correct interaction with
> buffer-local bindings and frame-local bindings.

Does this refer to the fact that buffer-localness and frame-localness
are not completely independent from dynamic scoping?  What would need
to change in the above example when case-fold-search would be a
buffer-local variable?

Or do you intend to implement buffer-localness (etc) itself by using
the functions?  These functions would then need to run when the
current buffer changes.  But dynamic-wind does not react to changes to
the current buffer so we can't use it to run the swapping functions at
the right times, I would say.  Can you elaborate?

> Suppose that Guile could run arbitrary code before and after the
> variable reference, for effect only, and the variable reference
> itself would always occur normally.  This would not encourage people
> to use variables for jobs that ought to be done by functions, but
> would permit implementation of this optimization, and various sorts
> of forwarding.  What do you think of this idea?

I think it's very nice!  I have sent a smallish proposal to the
guile-devel list about preparing us for specially constructed
'non-dumb' variables that need to be accessed with scm_variable_ref
and scm_variable_set_x.  We can then add whatever code is needed to
these two functions and be sure that they wont be side-stepped.

-- 
GPG: D5D4E405 - 2F9B BCCC 8527 692A 04E3  331E FAF8 226A D5D4 E405

Re: Emacs Lisp and Guile

[Richard Stallman]

    All Elisp variables would be constructed specially, even if they
    didn't make use of any of the available features initially.  That
    would mean that variables declared in Scheme can not be later made
    buffer local since normal Scheme variabled would not be constructed in
    the special way.

This would be a big inconvenience in practice.  It would mean that
mentioning a variable in Scheme code once prevents you from using it
in other ways.  It needs to be possible to make a variable "special"
(in this sense) after having used it in other non-lexical ways.

Re: Emacs Lisp and Guile

[Richard Stallman]

    There is a nearly standard macro out there called 'fluid-let' that
    encapsulates the process completely.  For example

It looks like the right idea, though the name is confusing given that
it does not use fluids.  Since the name fluid-let seems to be widely
known, perhaps fluids should be given a different name.  They could
be called per-threads or thread cells.

    > These functions could also have code for correct interaction with
    > buffer-local bindings and frame-local bindings.

    Does this refer to the fact that buffer-localness and frame-localness
    are not completely independent from dynamic scoping?

The point is, if the value that was saved by fluid-let came from a
buffer-local binding, it needs to restore that value into the same
buffer-local binding.  If the value that was saved was from the
buffer-global binding, fluid-let needs to restore that value into the
buffer-global binding.  The expansion of fluid-let needs extra code
both when saving and when restoring, to do this.

							  What would need
    to change in the above example when case-fold-search would be a
    buffer-local variable?

I am too overloaded to try to write the code (and I'd have to restudy
Scheme too).  Is my explanation above clear?

This needs to be done for all variables, since any might perhaps be
made buffer-local.  fluid-let could expand into the proper code for
this.

It might be useful to define a function to get the value and status of
a variable, and a function to restore based on that value and status.
(The status would identify the current buffer-local binding or
whatever.)  Then the expansion of fluid-let would call these
functions.

Re: Emacs Lisp and Guile

[Marius Vollmer]

Richard Stallman <rms@gnu.org> writes:

>     There is a nearly standard macro out there called 'fluid-let' that
>     encapsulates the process completely.  For example
> 
> It looks like the right idea, though the name is confusing given that
> it does not use fluids.  Since the name fluid-let seems to be widely
> known, perhaps fluids should be given a different name.

Yes, we could do that.  We could also try to generalize the fluids so
that they can be 'local' to other contexts besides threads, like
buffers or frames.  Or, since we might use variables for buffer-local
things, we could remove fluids and use the new features of variables
to have real thread-local variables.

>     > These functions could also have code for correct interaction with
>     > buffer-local bindings and frame-local bindings.
> 
>     Does this refer to the fact that buffer-localness and frame-localness
>     are not completely independent from dynamic scoping?
> 
> The point is, if the value that was saved by fluid-let came from a
> buffer-local binding, it needs to restore that value into the same
> buffer-local binding.  If the value that was saved was from the
> buffer-global binding, fluid-let needs to restore that value into the
> buffer-global binding.  The expansion of fluid-let needs extra code
> both when saving and when restoring, to do this.

I see, thanks.

> 							  What would need
>     to change in the above example when case-fold-search would be a
>     buffer-local variable?
> 
> I am too overloaded to try to write the code (and I'd have to restudy
> Scheme too).  Is my explanation above clear?

Yes.  (I thought you might enjoy a little bit of programming here and
there... :-)

> This needs to be done for all variables, since any might perhaps be
> made buffer-local.  fluid-let could expand into the proper code for
> this.

We should probably tackle this as the larger, more general problem of
working with arbitrary 'contexts' and variable-like 'entities'
(variables, or fluids, or settable functions, or...) that can take on
context-local values.  Let's call these entities "locations".

A particular context could then be specified by naming a location that
will hold the 'current context'.  For example, the location that
points to the current buffer is used to specify the context for
'buffer-local' locations.  The location that points to the current
thread is used for thread-local variables.

Thus, a location is characterized by its identity, and a reference to
another location that holds its context.  That reference is #f (or
nil) when the location is currently global.

The values of locations could be (conceptually) stored in a data base
that is indexed with keys that are the combination of a location and
another arbitrary value.

'Setting' a location LOC to the value VAL goes like this: when the
context reference is #f, store VAL with the key (LOC . #f).  When the
reference is not #f, 'get' the value CTXT of the referenced location
and store VAL with the key (LOC . CTXT).

'Getting' a location LOC means looking up the key (LOC . #f) when the
context reference is #f, or first 'getting' the value CTXT of the
referenced location and looking up the key (LOC . CTXT).

There would also be a way to set the context reference of a location,
of course.

fluid-let would then remember the (LOC . CTXT) key and would swap
according to that key.

With the above model, we can have buffer-local variables where the
curren-buffer variable is itself thread-local, for example.  We could
not have the situation where a variable is buffer-local in one thread
and frame-local in another (that is, where the context reference of a
location is itself stored in a location).  Maybe we can generalize the
model, or maybe we don't want to.

The real implementation can use the optimizations that Emacs currently
uses for its buffer-local etc variables.

(I like the name 'with' better instead of 'fluid-let'.  It goes with
'with-input-from-file', which sets the current-input-port in a dynamic
extent, for example.  But fluid-let is better known.)

-- 
GPG: D5D4E405 - 2F9B BCCC 8527 692A 04E3  331E FAF8 226A D5D4 E405

Re: Emacs Lisp and Guile

[Marius Vollmer]

Richard Stallman <rms@gnu.org> writes:

>     All Elisp variables would be constructed specially, even if they
>     didn't make use of any of the available features initially.  That
>     would mean that variables declared in Scheme can not be later made
>     buffer local since normal Scheme variabled would not be constructed in
>     the special way.
> 
> This would be a big inconvenience in practice.  It would mean that
> mentioning a variable in Scheme code once prevents you from using it
> in other ways.  It needs to be possible to make a variable "special"
> (in this sense) after having used it in other non-lexical ways.

Hmm, I think that maybe Scheme people will prefer to explicitely
declare that a variable is buffer local at the point of its
definition.  But maybe I'm wrong.

In any case, we can reverse the logic and say that variables need to
be specially constructed when the user does not want them to be used
"specially" (i.e., buffer-locally, etc).

That would work just as well, I think.

Variables that hold functions would be a candidate, for example.  That is,

  (define foo #f)

would construct a normal variable that can be made thread-local,
buffer-local, etc., while

  (define (frob-foo)
    ...)

would construct a 'dumb' variable that can't these things, but that
would allow faster access.

-- 
GPG: D5D4E405 - 2F9B BCCC 8527 692A 04E3  331E FAF8 226A D5D4 E405

Re: Emacs Lisp and Guile

[Richard Stallman]

    Yes, we could do that.  We could also try to generalize the fluids so
    that they can be 'local' to other contexts besides threads, like
    buffers or frames.

That would be much more inconvenient to use, so it is not a good idea.
Let's make these variables accessible simply *as variables*.

    > This needs to be done for all variables, since any might perhaps be
    > made buffer-local.  fluid-let could expand into the proper code for
    > this.

    We should probably tackle this as the larger, more general problem of
    working with arbitrary 'contexts' and variable-like 'entities'
    (variables, or fluids, or settable functions, or...) that can take on
    context-local values.  Let's call these entities "locations".

I think it is better if we do not generalize that far.  That extra
generalization makes things much more complicated and gives little
extra power.

I think we should start by handling just variables, and see how
convenient we can make that limited feature,

    The values of locations could be (conceptually) stored in a data base
    that is indexed with keys that are the combination of a location and
    another arbitrary value.

That is possible but it would make things much slower.  It makes no
sense to consider such a big slowdown for an unimportant
generalization.  The speed is more important than the extra
generality.  So this is not a good idea at all.

    'Setting' a location LOC to the value VAL goes like this: when the
    context reference is #f, store VAL with the key (LOC . #f).  When the
    reference is not #f, 'get' the value CTXT of the referenced location
    and store VAL with the key (LOC . CTXT).

This is totally unnatural for variables.  We want the ordinary Scheme
value of the variable `foo' to be the proper value from whichever
binding is current, even when `foo' has buffer-local bindings.

Please focus on implementing this feature for variables in the most
natural and efficient way you can.

Re: Emacs Lisp and Guile

[Neil Jerram]

Hello again - sorry for dropping this thread for so long.

[CC'ers - please say if you prefer not to be copied further]

>>>>> "Ken" == Ken Raeburn <raeburn@raeburn.org> writes:

    Ken> So, why were @fop and @bind needed in libguile anyways?  I was never
    Ken> clear on that.  Is it for performance, or is Scheme not up to the
    Ken> task?

Performance.  @fop (scm_m_atfop in C) does some pretty odd stuff, but
I think it could all be done in Scheme too.  @bind (scm_m_atbind) has
some special evaluator support, but again I think it could be done in
Scheme with set! and dynamic-wind.

    Ken> I wonder if it's the best choice.  I don't think encouraging people to
    Ken> support translation by starting with writing more C code for libguile
    Ken> is wise; we'll wind up with a libguile with random primitives for
    Ken> supporting translated elisp and perl and tcl and python
    Ken> and....  Personally, I'd rather see that support in Scheme modules.

Me too, as far as possible.  But I think it's acceptable to have a few
new primitives for convenience/performance, preferably abstracted
above the level of any single language (e.g. a general dynamic binding
primitive).

    Ken> One of my concerns in that area is with mixing Lisp dynamic bindings
    Ken> and Guile thread support.  If we want multi-threaded Lisp, do we pass
    Ken> around some sort of environment pointer, or call thread-aware routines
    Ken> to fetch current environment info?  Neil, does your code try to
    Ken> support multi-threaded Lisp?

No, it doesn't.

        Neil

Re: Emacs Lisp and Guile

[Richard Stallman]

      But I think it's acceptable to have a few
    new primitives for convenience/performance, preferably abstracted
    above the level of any single language (e.g. a general dynamic binding
    primitive).

I agree with that principle.  It is useful to add general purpose
facilities to Guile to make it better for translating other languages
into.

	Ken> One of my concerns in that area is with mixing Lisp dynamic bindings
	Ken> and Guile thread support.  If we want multi-threaded Lisp, do we pass
	Ken> around some sort of environment pointer, or call thread-aware routines
	Ken> to fetch current environment info?  Neil, does your code try to
	Ken> support multi-threaded Lisp?

    No, it doesn't.

This is a serious issue.  Multi-thread support is a usefu thing, and
when we add primitives, we should preserve multi-thread support if at
all possible.

Can you and Guile developers work on a way to do this with
multi-thread support?

Re: Emacs Lisp and Guile

[Neil Jerram]

>>>>> "Richard" == Richard Stallman <rms@gnu.org> writes:

    Ken> One of my concerns in that area is with mixing Lisp dynamic bindings
    Ken> and Guile thread support.  If we want multi-threaded Lisp, do we pass
    Ken> around some sort of environment pointer, or call thread-aware routines
    Ken> to fetch current environment info?  Neil, does your code try to
    Ken> support multi-threaded Lisp?

    Neil>     No, it doesn't.

    Richard> This is a serious issue.  Multi-thread support is a usefu thing, and
    Richard> when we add primitives, we should preserve multi-thread support if at
    Richard> all possible.

    Richard> Can you and Guile developers work on a way to do this with
    Richard> multi-thread support?

I'm not sure I understand.  Previously you said that you didn't think
it would be possible to implement a preemptively multithreaded Emacs,
because there is too much existing Elisp code that is not thread-safe.

When I said "No" above, I meant that my code doesn't do anything
special to support multithreading.  I thought this would be OK in the
light of your opinion just described.

If you mean that new primitives should be thread-safe in the sense of,
e.g., avoiding accessing global data without a mutex, it may be that
most of my code already has this property.  I will check.

        Neil

Re: Emacs Lisp and Guile

[Richard Stallman]

    I'm not sure I understand.  Previously you said that you didn't think
    it would be possible to implement a preemptively multithreaded Emacs,
    because there is too much existing Elisp code that is not thread-safe.

To do multi-threading in Emacs we would have to somehow specify in the
C code the places where a thread-switch is allowed.

    When I said "No" above, I meant that my code doesn't do anything
    special to support multithreading.

Multithreading won't be easy, but we don't want to add to the difficulty.
Please don't take a step away from that goal.