elisp glimpses

elisp glimpses

[Andy Wingo]

Hello!

Sorry for the delay, I got swamped with the GHM. Which was cool! But
distracting :) I'm copying the list on my answer; hope you don't mind.

On Thu 15 Jul 2010 07:48, Brian Templeton <bpt@hcoop.net> writes:

> I've started pushing stable patches into the 'bpt/elisp' branch of
> <git://git.hcoop.net/git/bpt/guile.git>

Cool. I like it. I like defspecial.

A couple things:

 * If pmatch doesn't match, it's not an error. It's more like cond not
   matching. You need to add an error case. Probably best to make an
   amatch macro that uses pmatch or something, and ensures there is an
   error case.

 * The quasiquote stuff is incredibly ugly. Why is ` not a reader macro?
   If it has to be like it is, OK, but why not expand to the more
   schemely quasiquote, unquote, and unquote-splicing?

> At the moment I'm rebasing to master frequently

OK, cool.

> * Multi-cell symbols and module system integration

Hmmm, very good questions. Currently of course there is no module system
in Elisp, so to a large degree we can punt on this, but it would be good
to have an answer for the future.

Other questions that would have to be answered at the same time:

  * What is the default elisp module called?

  * Do Guile modules and Elisp modules share a namespace?

  * What does it mean to import a module in Elisp? Is the import
    restricted to the importing file, or does it import into the global
    namespace? Or, can you only import a module when defining a new
    module?
  
  * Is it even a good idea to give Elisp a module system, or should that
    only be available from Scheme?

> Currently Elisp symbols are Scheme symbols, and their value and function
> cell contents are looked up in submodules of (language elisp runtime).
> This works well enough for now, but it will get quite ugly if Elisp is
> ever integrated with Guile's module system to any degree. Every module
> containing Elisp function bindings might need to have a 'function-cell'
> submodule, for example; an uninterned symbol could be used to avoid name
> clashes, but it would still pollute the namespace. It would be cleaner
> to maintain multiple hash tables in every module, and then e.g. (@
> (emacs) set-buffer function) could refer to the function binding.
>
> But here's another alternative: Guile already has function and plist
> slots in symbols, so clearly it's okay to waste a few bytes for features
> nobody uses. (: Why not keep the extra cells in variable objects, or in
> a new type of variable object? (I call this option "big variables" or
> "bigvars".) Guile's double cells are just the right size; there's enough
> room for a function cell as well as a pointer to a properties object,
> which would contain the plist cell and implementation-internal
> annotations. I've implemented the (trivial) libguile side of this, but
> at least Tree-IL, the module system, and the VM would also need to be
> modified to support this, if it's even a good idea.

Hmmm. Yes, I think this is a good idea. Does it make sense to be a
subtype of variables -- that is, with the same tc7, but with a flag in
the first word? Then we could make Guile's variables all be treated as
having a value binding already. Perhaps the new variable-function-ref /
toplevel-function-ref / etc opcodes (you are adding them, yes? :) should
return the value if it's a narrow variable holding a procedure. We would
need toplevel-value-define / toplevel-function-define as well, to ensure
that the binding is a wide variable.

But also... we should think about threads here. Elisp doesn't have flet,
right? So perhaps the default binding of a narrow var should be a
function binding, or something; and then we can guarantee that the third
word of a wide var is a fluid. Then a toplevel-value-ref can cache the
fluid instead of the var, and we get thread-local bindings for free. I'm
being brief here; let me know if I'm not being clear.

When you add this, please make your patches small, correct, and
orthogonal :)

> Also, Elisp really needs a package system, not a module system -- in the
> context of Guile Emacs Lisp that probably means having a "packaged
> symbol" type that refers to an (unpackaged) symbol in the context of a
> specific module. (This is only necessary if Elisp is going to have some
> level of integration into the module system, of course.) Guile appears
> to handle imported bindings in some way less direct than storing the
> other module's variable object in the obarray, so it can even work as
> required for CL's `symbol-package', etc. (although who knows what
> `symbol-name' on an imported and renamed symbol would actually mean)

Hm. Well, Guile's symbols do have four words, and space for some
flags. It's a bit ridiculous really, looking at the code now -- every
time you make a symbol you cons an extra pair too.

I would prefer to have symbols only be two words, and the stringbuf in
the second word should have a hash inside it. This would be advantageous
for hash maps with string keys too. Then symbol properties could be
implemented with a weak map. That way the deprecated features pay their
own way, and symbols are lighter.

But, I digress. I don't really know what a packaged symbol is. Is it a
separate data type from a "normal" symbol? Could we pull similar
narrow-vs-wide subtyping tricks, perhaps? Maybe that is the best
option.

> * Aliases
>
> Elisp has both defalias and defvaralias. My first thought was to assign
> the same variable object to multiple names in an obarray; that works for
> the submodule-based implementation of symbol cells currently used, but
> not for the bigvars implementation. It would be useful to have a special
> object like SCM_UNDEFINED to mark cells to be redirected, but if that's
> not practical, redirected cells can remain SCM_UNBOUND, redirection
> information information can be stored in the property cell, references
> to aliases can be processed in an error handler, and Elisp setq can
> check the property cell for every assignment. I think I'll implement a
> proper def(var)alias next; subr.el and other basic libraries load with
> defalias = fset, but Emacs implements buffer- and frame-local variables
> using a similar technique so this will make it easier to implement that
> when it becomes necessary.

Yes, I don't know how to do this nicely. Redirection is not so
nice. Necessary perhaps, though...

> * Performance
>
> The only real optimization I've added so far has been to introduce
> 'unbound fluids,' by analogy with unbound variables; with this change,
> TAK is 3.7x slower in Guile Elisp than in Emacs, vs. 4.8x slower
> without. (And Guile Scheme is about twice as fast as GNU Emacs Elisp.)

Well, good to have these numbers. I think caching the fluids in
functions' object tables for lexical variable access will be the real
"equalizer", so to speak; but that's just a guess.

> think Guile Elisp could get away with using lexical function bindings
> while remaining technically compatible with GNU Emacs Elisp, but cl.el's
> CL-incompatible `flet' simulates dynamically scoped function bindings --
> which don't even exist in standard CL! I may be missing something
> obvious, but I can't think of an efficient way to dynamically shadow the
> value of a globally lexical variable.

What does "globally lexical" mean? There are certainly a number of cases
in which we can prove that a variable reference may be lexical, and not
dynamic. E.g. in 

  (defun foo (bar)
    bar)

`bar' may be lexically accessed, and no fluid need be set. But in:

  (defun foo (bar)
    (baz bar))

We may access the `bar' binding lexically, but we still have to include
a dynlet in the generated code. Actually I don't see a make-dynlet or
the like in the compiler code; surely an oversight?

> * Nil
>
> My initial inclination was to remove #nil entirely and to simply use the
> symbol `nil' as Elisp's EOL and false. I now think it's probably worth
> keeping it since it would be useful for a Common Lisp implementation.
> Note that (symbol? #nil) => #f in Scheme and (symbolp nil) => nil in
> Elisp, when it will eventually need to be true in at least the latter
> case. But I would guess that, for symbols, only the type and equality
> predicates need to be especially fast, so I think it's okay to just
> special-case these in the Elisp symbol functions, even though it's an
> ugly hack.

The fact that t and nil are symbols for the purpose of elisp is really
going to bite us at some point, I think... We can get it right but there
will be a number of bugs along the way I think.

> * Where should I put Elisp files to be packaged with Guile? (I'm writing
>   most new subrs in Elisp -- ideally all subrs should be in Elisp and
>   all macros should be loaded from the Emacs source.) Just under
>   "module/language/elisp"? It seems odd to put them under the module
>   directory since, well, they're not modules, but that's what gets
>   installed to /usr/share/guile...

Sure, that sounds fine to me, for now at least.

> * The Tree-IL program `(begin)' does not compile. Is it supposed to be a
>   valid program?

No, this is not a valid program. Tree-IL's begin is a sequencing
operator, not a splicing operator -- you need 1 or more expressions to
sequence.

Happy hacking,

Andy
-- 
http://wingolog.org/

Re: hey dude

[Andy Wingo]

Heya,

On Fri 16 Jul 2010 00:27, Brian Templeton <bpt@tunes.org> writes:

> BTW, as far as real-world Elisp compatibility goes, subr.el loads in one
> of my branches:
>
> scheme@(guile-user)> ,L elisp
> Happy hacking with Emacs Lisp!  To switch back, type `,L scheme'.
> elisp@(guile-user)> (load "/home/bpt/el/subr.el")
> $1 = #t
> elisp@(guile-user)> (let ((x nil)) (push t x) x)
> $2 = (#t)
> elisp@(guile-user)> (when t 42)
> $3 = 42
> elisp@(guile-user)> (let ((x nil)) (dotimes (n 5) (push n x)) x)
> $4 = (4 3 2 1 0)
> elisp@(guile-user)> (nbutlast (list 1 2 3))
> $5 = (1 2)
> elisp@(guile-user)> (delete-dups (list 3 2 1 1 2 3))
> $6 = (3 2 1)
> elisp@(guile-user)> (number-sequence 1 10)
> $7 = (1 2 3 4 5 6 7 8 9 10)
>
> This is Emacs's subr.el with no changes except removing the `add-hook'
> definition (which really requires support for buffer-local variables).

Very awesome!!

> I've attached a partially-complete patch for unbound fluids

Looks mostly good to me. Please separate out the parts that patch
libguile/ from the parts that patch the elisp compiler, and send the
libguile/ part to guile-devel, along with additions to fluids.test.

> I've been reading old CL proposals regarding global lexical bindings,
> and I think I now understand why this can't work in Guile. Previously I
> wasn't considering the existence of a *global* environment, only lexical
> and dynamic bindings, and hence I wasn't understanding the bindings
> lookup order required for this. Scheme only allows LG lookup -- it looks
> for variables in the lexical environment, then falls back to the global
> environment.
>
> Globally-bound Elisp and CL variables are pervasively special, and
> cannot be lexically bound. Globally-bound CL functions are pervasively
> lexical (IIUC), and cannot be dynamically bound. Elisp bindings have
> only global bindings, and cannot be lexically or dynamically bound. But
> for Elisp function bindings, we need a globally-bound variable which can
> be bound both lexically and dynamically. To allow dynamic shadowing of a
> global lexical (but not pervasively lexical!) variable, the
> implementation must do LDG lookup -- checking in the dynamic environment
> after the lexical environment but before the global environment, which
> is distinct from both the lexical and dynamic environments. But Scheme
> doesn't *have* a true dynamic environment; it only simulates it by
> storing fluids (i.e., indices into local dynamic state objects) in the
> global environment, which of course means Guile Elisp can't have a
> dynamic binding for a variable while retaining the global binding.
> Therefore, it appears to me that Guile needs a real dynamic environment
> if Elisp function bindings are to be non-special without breaking cl.el
> FLET.

I have to admit that I don't understand this fully. Cl.el's flet is a
hack on top of emacs internals; while I do think we need to be very very
compatible, this might be an instance where we might choose an alternate
implementation. Surely it's possible to implement cl.el's flet via
dynamic-wind and toplevel-set!?

Cheers,

Andy
-- 
http://wingolog.org/