adding namespaces to emacs-lisp (better elisp?)

adding namespaces to emacs-lisp (better elisp?)

[Nic Ferrier]

We had a lot of discussion recently about making EmacsLisp

Here http://nic.ferrier.me.uk/blog/2013_06/adding-namespaces-to-elisp is
a proposal to add namespaces to emacs-lisp.

I'd be really interested in what people think about this, whether it
would be worth my time trying to do this or not.


Nic Ferrier

RE: adding namespaces to emacs-lisp (better elisp?)

[Drew Adams]

> http://nic.ferrier.me.uk/blog/2013_06/adding-namespaces-to-elisp is
> a proposal to add namespaces to emacs-lisp.
> 
> I'd be really interested in what people think about this, whether it
> would be worth my time trying to do this or not.

OK, I'll start.  I am in favor of the Common Lisp spec - IOW, Common
Lisp "packages".  I am in favor of such a namespace system for Emacs
Lisp.

I read your proposal overview, Nic.  It's not clear to me just what
the differences would be from the Common Lisp package system.
Perhaps you could spell the differences out in more detail somewhere.

But the closer we can get to the CL spec the better, IMO.  If we
could conform to it completely, that would be great.

Even keeping the same terminology, symbol names etc. as CL would
help.  It would help users who are coming from Common Lisp or who
happen to read Common Lisp doc.

And it would help the reuse/transfer of existing code from CL to
Elisp.  (Yes, such reuse/transfer might require some massaging, but
similar syntax and semantics would help minimize that operation.)

Of course, adopting CL terminology in this regard should mean that
we would drop the terminology used so far for Emacs "packages".
An argument can be made that both uses of the word "package" are
somewhat unfortunate.

At this point, I think conforming to the terminology that has been
used in CL for 30 years is the right approach, regardless of
whether CL "packages" are really, in effect, namespaces.

So +1 for adding CL-style namespaces to Emacs Lisp.  One opinion.

Re: adding namespaces to emacs-lisp (better elisp?)

[Stefan Monnier]

> We had a lot of discussion recently about making EmacsLisp

I'm not sure I understand: do you mean Emacs does not Lisp enough yet?

> Here http://nic.ferrier.me.uk/blog/2013_06/adding-namespaces-to-elisp is
> a proposal to add namespaces to emacs-lisp.

Comments:
- "Interning a symbol with "::" in it should no longer be possible.
  It should raise an error."  Why not simply intern it in the
  corresponding namespace?  It's probably a bad practice, but Emacs is
  usually not in the business of preventing bad practice.

- I'm not sure how well it will cope with shadowing and non-namespaced
  symbols (e.g. symbols like `face' that aren't used as variables).
  The rule "global obarray is inspected first and if a symbol is found
  there that's what is used" means that we have to be vary careful about
  interning things in the global obarray since it can affect the way
  other code is handled.  Currently, we're very liberal about interning
  in the global obarray.
  Basically I think this shadowing rule makes things a bit too automatic
  for something where we need more control.
  I think that, for backward compatibility reasons we can't easily mark
  "non-prefixed symbols that should be global" (such as `face'), so the
  safer alternative is to force local symbols to be specially marked
  (like ::foo, maybe, to mean "foo in the current local obarray").
  Having to prefix every single local symbol that way probably sounds
  rather unpalatable, tho.

- I'm not sure exactly how you expect importing namespaces
  should/will work.


-- Stefan

Re: adding namespaces to emacs-lisp (better elisp?)

[Nic Ferrier]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> We had a lot of discussion recently about making EmacsLisp
>
> I'm not sure I understand: do you mean Emacs does not Lisp enough yet?

Heh. Yeah. I dropped a word. I am going to work on blaming someone or
something for that and get back to you.

What I meant to say was:

  We had a lot of discussion recently about making EmacsLisp better.


> Comments:
> - "Interning a symbol with "::" in it should no longer be possible.
>   It should raise an error."  Why not simply intern it in the
>   corresponding namespace?  It's probably a bad practice, but Emacs is
>   usually not in the business of preventing bad practice.

That is a good idea. I will update the document with that. I agree emacs
is better off with it's laissez faire attitude.


> - I'm not sure how well it will cope with shadowing and non-namespaced
>   symbols (e.g. symbols like `face' that aren't used as variables).
>   The rule "global obarray is inspected first and if a symbol is found
>   there that's what is used" means that we have to be vary careful about
>   interning things in the global obarray since it can affect the way
>   other code is handled.

Give an example of a potential problem?

I *think* you mean that adding new global symbols could affect
namespaced code. But I don't think that's any different than right now.


>   Currently, we're very liberal about interning in the global obarray.
>   Basically I think this shadowing rule makes things a bit too
>   automatic for something where we need more control.

I disagree that's a reason not to try it. Yes, it could be a
problem... but the presumption has to be that new code would use this
way to namespace itself and that global pollution would therefore slow
down.

*Eventually* I'd expect things like the face use-case to be dealt with
in some sort of namespace system.


> - I'm not sure exactly how you expect importing namespaces
>   should/will work.

I'll try and add some examples I guess. It's quite simple though, if I
have a package "nic" with 2 functions:

  foo
  bar

and you have a package "stefan", then you should be able to make aliases
to nic::foo and nic::bar in "stefan" namespace.

Ideally that should work with some simple statement:

 ;; Package: stefan

 (import 'nic)
 (foo)

in this example nic::foo and nic::bar would both be imported directly
into "stefan" and after that was done the following would be true:

  (eq (symbol-function 'nic::foo)
      (symbol-function 'stefan::foo))

or:

 (import 'nic :as 'blah)
 (blah::foo)

in this example nic::foo and nic::bar would be imported under the
namespace 'blah in the package "stefan".

I guess this could then be a use case for trees of namespaces, so that:

 (stefan::blah::foo)

was possible.

I'm not sure about that last bit.


Nic

Re: adding namespaces to emacs-lisp (better elisp?)

[Pascal J. Bourguignon]

Drew Adams <drew.adams@oracle.com> writes:

>> http://nic.ferrier.me.uk/blog/2013_06/adding-namespaces-to-elis pis
>> a proposal to add namespaces to emacs-lisp.
>> 
>> I'd be really interested in what people think about this, whether it
>> would be worth my time trying to do this or not.
>
> OK, I'll start.  I am in favor of the Common Lisp spec - IOW, Common
> Lisp "packages".  I am in favor of such a namespace system for Emacs
> Lisp.
>
> I read your proposal overview, Nic.  It's not clear to me just what
> the differences would be from the Common Lisp package system.
> Perhaps you could spell the differences out in more detail somewhere.
>
> But the closer we can get to the CL spec the better, IMO.  If we
> could conform to it completely, that would be great.
>
> Even keeping the same terminology, symbol names etc. as CL would
> help.  It would help users who are coming from Common Lisp or who
> happen to read Common Lisp doc.
>
> And it would help the reuse/transfer of existing code from CL to
> Elisp.  (Yes, such reuse/transfer might require some massaging, but
> similar syntax and semantics would help minimize that operation.)
>
> Of course, adopting CL terminology in this regard should mean that
> we would drop the terminology used so far for Emacs "packages".
> An argument can be made that both uses of the word "package" are
> somewhat unfortunate.
>
> At this point, I think conforming to the terminology that has been
> used in CL for 30 years is the right approach, regardless of
> whether CL "packages" are really, in effect, namespaces.
>
> So +1 for adding CL-style namespaces to Emacs Lisp.  One opinion.

+1

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.  
You know you've been lisping too long when you see a recent picture of George 
Lucas and think "Wait, I thought John McCarthy was dead!" -- Dalek_Baldwin

CommonLisp namespace system (was Re: adding namespaces to emacs-lisp (better elisp?))

[Nic Ferrier]

Drew Adams <drew.adams@oracle.com> writes:

> I read your proposal overview, Nic.  It's not clear to me just what
> the differences would be from the Common Lisp package system.
> Perhaps you could spell the differences out in more detail somewhere.
>
> But the closer we can get to the CL spec the better, IMO.  If we
> could conform to it completely, that would be great.

I disagree. Emacs isn't CommonLisp, never has been CommonLisp and
very likely, never will be CommonLisp.


> Even keeping the same terminology, symbol names etc. as CL would
> help.  It would help users who are coming from Common Lisp or who
> happen to read Common Lisp doc.

But at the expense of muddying the waters for people who are not from
that world.

In Emacs world, we use "package" to mean something different from what
CL "package" means. So right at the start that effort is doomed without
a major change to Emacs 24.


> Of course, adopting CL terminology in this regard should mean that
> we would drop the terminology used so far for Emacs "packages".
> An argument can be made that both uses of the word "package" are
> somewhat unfortunate.

I am trying to make a namespace system that would be backwards
compatible with Emacs and yet encourage future good behaviour.

CommonLisp terminology or compatibility is not a major aim of mine.


Nic

RE: CommonLisp namespace system (was Re: adding namespaces to emacs-lisp (better elisp?))

[Drew Adams]

> > I read your proposal overview, Nic.  It's not clear to me just what
> > the differences would be from the Common Lisp package system.
> > Perhaps you could spell the differences out in more detail somewhere.
> >
> > But the closer we can get to the CL spec the better, IMO.  If we
> > could conform to it completely, that would be great.
> 
> I disagree. Emacs isn't CommonLisp, never has been CommonLisp and
> very likely, never will be CommonLisp.

Some of us nevertheless hope it will become closer in many respects.

> > Even keeping the same terminology, symbol names etc. as CL would
> > help.  It would help users who are coming from Common Lisp or who
> > happen to read Common Lisp doc.
> 
> But at the expense of muddying the waters for people who are not from
> that world.

"That world" is a longstanding one, with a rigorous, if informal, spec.

Emacs Lisp is younger (even if Emacs, in one form or another, is older
than Common Lisp itself, though not older than its ancestors).  And
Emacs Lisp has no spec - it is defined rigorously only by its (ever
moving) implementation.

And of course if Emacs adopts Common Lisp packages or similar then the
two worlds approach, and the waters become clearer with time and closeness.

> In Emacs world, we use "package" to mean something different from what
> CL "package" means.

That's a very recent introduction to the "Emacs world".  Hardly much of
a precedent.  "In [the] Emacs world" is a pretty bold way of describing
something we just introduced, as if it were essential to what Emacs Lisp
has always been.  It is a recent add-on - a welcome one, but hardly core.

And the question here is not about abandoning package.el etc.  It is
about the terminology: "package".  Who heard of Emacs "packages" a few
years ago?  Contrast that with who had heard of Common Lisp "packages".

> So right at the start that effort is doomed without
> a major change to Emacs 24.

Introducing proper packages (a la Common Lisp) would likely be a major
change to Emacs, yes.  And a welcome one, IMO.

> > Of course, adopting CL terminology in this regard should mean that
> > we would drop the terminology used so far for Emacs "packages".
> > An argument can be made that both uses of the word "package" are
> > somewhat unfortunate.
> 
> I am trying to make a namespace system that would be backwards
> compatible with Emacs and yet encourage future good behaviour.
> 
> CommonLisp terminology or compatibility is not a major aim of mine.

Too bad.  Emacs Lisp is already farther from Common Lisp than it should
be after 30-some years.

Re: adding namespaces to emacs-lisp (better elisp?)

[Davis Herring]

> - I'm not sure how well it will cope with shadowing and non-namespaced
>   symbols (e.g. symbols like `face' that aren't used as variables).
>   The rule "global obarray is inspected first and if a symbol is found
>   there that's what is used" means that we have to be vary careful about
>   interning things in the global obarray since it can affect the way
>   other code is handled.  Currently, we're very liberal about interning
>   in the global obarray.
>   Basically I think this shadowing rule makes things a bit too automatic
>   for something where we need more control.
>   I think that, for backward compatibility reasons we can't easily mark
>   "non-prefixed symbols that should be global" (such as `face'), so the
>   safer alternative is to force local symbols to be specially marked
>   (like ::foo, maybe, to mean "foo in the current local obarray").
>   Having to prefix every single local symbol that way probably sounds
>   rather unpalatable, tho.

I think Mathematica's solution here works pretty well (see
http://reference.wolfram.com/mathematica/tutorial/Contexts.html).  The
part I think relevant here would be to define a "built-in" namespace
that would be searched first (like the global obarray mentioned above),
but not have it be the default for namespace-oblivious code.  Then the
up-front cost would be merely that symbols like 'face (as well as more
obvious things like 'car) would need to be explicitly placed in that
special namespace, rather than that all references to them needing
modification.

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: adding namespaces to emacs-lisp (better elisp?)

[Stefan Monnier]

>> - I'm not sure how well it will cope with shadowing and non-namespaced
>> symbols (e.g. symbols like `face' that aren't used as variables).
>> The rule "global obarray is inspected first and if a symbol is found
>> there that's what is used" means that we have to be vary careful about
>> interning things in the global obarray since it can affect the way
>> other code is handled.
> Give an example of a potential problem?
> I *think* you mean that adding new global symbols could affect
> namespaced code. But I don't think that's any different than right now.

Here's a scenario:
- namespaced packages A and B both locally define a function `toto'.
- non-namespaced package C comes along with a symbol `toto' somewhere in
  its code, suddenly causing A and B's `toto' to be global rather
  than local.

Note that instead of "non-namespaced package C", we could have some
package which uses symbols as "uniquified strings" and which uses the
global obarray for it and might occasionally intern `toto' in the course
of its normal execution.

IOW I think we should instead first look in the local obarray (over
which the coder does have control) and if that fails then look in the
global obarray.

> I disagree that's a reason not to try it. Yes, it could be a
> problem... but the presumption has to be that new code would use this
> way to namespace itself and that global pollution would therefore slow
> down.

Old non-namespaced, hideous, dynamically scoped is Emacs's bread and
butter and we can't hope for it to disappear soon.

> Ideally that should work with some simple statement:
>  ;; Package: stefan
>  (import 'nic)
>  (foo)
> in this example nic::foo and nic::bar would both be imported directly
> into "stefan" and after that was done the following would be true:
>   (eq (symbol-function 'nic::foo)
>       (symbol-function 'stefan::foo))

Should this equality still stand if I (fset 'nic::foo 'blabla)?
I.e. is it one and the same symbol?

> or:
>  (import 'nic :as 'blah)
>  (blah::foo)
> in this example nic::foo and nic::bar would be imported under the
> namespace 'blah in the package "stefan".
> I guess this could then be a use case for trees of namespaces, so that:
>  (stefan::blah::foo)
> was possible.

Indeed, my impression is that you inevitably get to this kind
of situation, which you seemed to dislike.  I personally don't find it
problematic, not even if we generalize it to some arbitrary graph, with
cycles and all.


        Stefan

Re: CommonLisp namespace system (was Re: adding namespaces to emacs-lisp (better elisp?))

[Stefan Monnier]

> I disagree. Emacs isn't CommonLisp, never has been CommonLisp and
> very likely, never will be CommonLisp.

Agreed, but all else being equal, it's better to do the same as
Common-Lisp (or as Scheme) than to invent our own.


        Stefan

Re: CommonLisp namespace system (was Re: adding namespaces to emacs-lisp (better elisp?))

[Sebastian Wiesner]

2013/7/26 Drew Adams <drew.adams@oracle.com>:
>> In Emacs world, we use "package" to mean something different from what
>> CL "package" means.
>
> That's a very recent introduction to the "Emacs world".  Hardly much of
> a precedent.  "In [the] Emacs world" is a pretty bold way of describing
> something we just introduced, as if it were essential to what Emacs Lisp
> has always been.  It is a recent add-on - a welcome one, but hardly core.

As a developer of Emacs Lisp extensions, I consider package.el a core
addon, no matter how old it is.  I think it's the only more or less
sane and comfortable way to distribute Emacs Lisp code, and I see it
being used by many Emacs Lisp developers as primary distribution
channel for their libraries.

Imho, adding package.el to Emacs has boosted the productivity of the
Emacs community more than any past attempt to make Emacs Lisp more
Common Lisp.

> And the question here is not about abandoning package.el etc.  It is
> about the terminology: "package".  Who heard of Emacs "packages" a few
> years ago?  Contrast that with who had heard of Common Lisp "packages".

That's a bold saying, too.  I doubt that even Common Lisp itself has
much relevance to many Emacs users.  I doubt even that it's known to
many.

Re: CommonLisp namespace system (was Re: adding namespaces to emacs-lisp (better elisp?))

[Nic Ferrier]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I disagree. Emacs isn't CommonLisp, never has been CommonLisp and
>> very likely, never will be CommonLisp.
>
> Agreed, but all else being equal, it's better to do the same as
> Common-Lisp (or as Scheme) than to invent our own.

But they aren't equal.

I have looked at CL's system and Scheme's. I'll try and add a summary to
the blog post and also describe how it differs from CL and Scheme.

Re: CommonLisp namespace system (was Re: adding namespaces to emacs-lisp (better elisp?))

[Tom Tromey]

>>>>> "Nic" == Nic Ferrier <nferrier@ferrier.me.uk> writes:

Nic> I disagree. Emacs isn't CommonLisp, never has been CommonLisp and
Nic> very likely, never will be CommonLisp.

But it is like Common Lisp's little brother.
With EIEIO now it is even growing up a bit :)

Nic> In Emacs world, we use "package" to mean something different from what
Nic> CL "package" means. So right at the start that effort is doomed without
Nic> a major change to Emacs 24.

I think the name clash is unfortunate.  Bad self.  However, in practice
I don't think there's any difficulty of using the name to mean both
things, just a small overlap during discussions.  In particular I don't
think package.el has any symbols that conflict with things from CL.

Tom

RE: CommonLisp namespace system (was Re: adding namespaces to emacs-lisp (better elisp?))

[Drew Adams]

> >> In Emacs world, we use "package" to mean something different from what
> >> CL "package" means.
> >
> > That's a very recent introduction to the "Emacs world".  Hardly much of
> > a precedent.  "In [the] Emacs world" is a pretty bold way of describing
> > something we just introduced, as if it were essential to what Emacs Lisp
> > has always been.  It is a recent add-on - a welcome one, but hardly core.
> 
> As a developer of Emacs Lisp extensions, I consider package.el a core
> addon, no matter how old it is.  I think it's the only more or less
> sane and comfortable way to distribute Emacs Lisp code, and I see it
> being used by many Emacs Lisp developers as primary distribution
> channel for their libraries.
> 
> Imho, adding package.el to Emacs has boosted the productivity of the
> Emacs community more than any past attempt to make Emacs Lisp more
> Common Lisp.

Don't waste your breath lauding package.el.  No one said the slightest
thing against the addition of package.el to Emacs.  Quite the contrary.
I was clear that it is a welcome addition.

And it would be just as welcome if what it calls "package" were called
something else.  The question raised was only wrt the terminology to use,
if we add namespace support to Emacs Lisp.  IMO, we should use the same
terminology that Common Lisp uses, especially if what we implement in
this regard is reasonably close to the Common Lisp package system.

Just as we call an obarray an obarray, we should call something close to
a Common Lisp package a package.  Yes, that would mean changing the
terminology we introduced recently with package.el.  Too bad.

> > And the question here is not about abandoning package.el etc.  It is
> > about the terminology: "package".  Who heard of Emacs "packages" a few
> > years ago?  Contrast that with who had heard of Common Lisp "packages".
> 
> That's a bold saying, too.  I doubt that even Common Lisp itself has
> much relevance to many Emacs users.  I doubt even that it's known to
> many.

Counting among "Emacs users" for such things would be quite misleading
indeed.  Counting among Emacs Lisp users would be more relevant.

And if some Emacs Lisp users have no knowledge of the existence of Common
Lisp then it is truly too bad for them.  IMO, other things being equal,
Common Lisp should be a model toward which Emacs Lisp aims.

As Stefan put it: "all else being equal, it's better to do the same as
Common-Lisp (or as Scheme) than to invent our own."

(I would drop the "(or as Scheme)" for cases where that would conflict
with Common Lisp.  But otherwise, sure: or as Scheme.)

RE: CommonLisp namespace system (was Re: adding namespaces to emacs-lisp (better elisp?))

[Drew Adams]

> I think the name clash is unfortunate.  Bad self.  

;-)

> However, in practice I don't think there's any difficulty of using the
> name to mean both things, just a small overlap during discussions.

I guess I would have to agree with that.  It would proabably be workable.

That said, it would be better to change the terminology for package.el -
e.g., when we introduce CL-style "packages".

Re: adding namespaces to emacs-lisp (better elisp?)

[Nic Ferrier]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

[resolution]
>>> - I'm not sure how well it will cope with shadowing and non-namespaced
>>> symbols (e.g. symbols like `face' that aren't used as variables).
>>> The rule "global obarray is inspected first and if a symbol is found
>>> there that's what is used" means that we have to be vary careful about
>>> interning things in the global obarray since it can affect the way
>>> other code is handled.
>> Give an example of a potential problem?
>> I *think* you mean that adding new global symbols could affect
>> namespaced code. But I don't think that's any different than right now.
>
> Here's a scenario:
> - namespaced packages A and B both locally define a function `toto'.
> - non-namespaced package C comes along with a symbol `toto' somewhere in
>   its code, suddenly causing A and B's `toto' to be global rather
>   than local.

I don't think this is a serious problem personally. But I'm also not
wedded to global-obarray-first.

> Note that instead of "non-namespaced package C", we could have some
> package which uses symbols as "uniquified strings" and which uses the
> global obarray for it and might occasionally intern `toto' in the course
> of its normal execution.

Again, it only matters if it's a non-namespaced package that does it.

> IOW I think we should instead first look in the local obarray (over
> which the coder does have control) and if that fails then look in the
> global obarray.

I am not wedded to the proposal of using the global obarray first. The
rules for interning are slightly more complicated in that case:

- given a string X
- lookup X in the local obarray
- if it exists return the symbol
- else
-  lookup X in the global obarray
-  if it exists return the symbol
-  else
-    add the symbol to the local obarray

The only problem I see here is the possibility of problems with
concurrency. The whole operation above would have to be atomic and it
involves lookups in two separate data structures.

But since Emacs doesn't have concurrency yet and it would be a very bad
idea at this stage to add unfettered concurrency of the sort that would
cause a problem here (if there were a GIL-less threads implementation
for example) and the existing concurrency branch is tied to a GIL then I
really don't think that is actually a real problem we need to worry
about.

Although I bet that's what both Guido and Matz said when they were
designing the namespace bits of Python and Ruby.


[import]
>> Ideally that should work with some simple statement:
>>  ;; Package: stefan
>>  (import 'nic)
>>  (foo)
>> in this example nic::foo and nic::bar would both be imported directly
>> into "stefan" and after that was done the following would be true:
>>   (eq (symbol-function 'nic::foo)
>>       (symbol-function 'stefan::foo))
>
> Should this equality still stand if I (fset 'nic::foo 'blabla)?
> I.e. is it one and the same symbol?

I guess this needs more careful specification because that would not be
true of aliases.

My feeling is that an import should be like the creation of an alias.


>> or:
>>  (import 'nic :as 'blah)
>>  (blah::foo)
>> in this example nic::foo and nic::bar would be imported under the
>> namespace 'blah in the package "stefan".
>> I guess this could then be a use case for trees of namespaces, so that:
>>  (stefan::blah::foo)
>> was possible.
>
> Indeed, my impression is that you inevitably get to this kind
> of situation, which you seemed to dislike.  I personally don't find it
> problematic, not even if we generalize it to some arbitrary graph, with
> cycles and all.

It's not that I don't like it per-se. I just want this to be easy to
implement in the first instance. If the implementation gets more
difficult later I have no problem with that. But initial low cost is a
good thing.


Nic

Re: CommonLisp namespace system (was Re: adding namespaces to emacs-lisp (better elisp?))

[Nic Ferrier]

Drew Adams <drew.adams@oracle.com> writes:

>> However, in practice I don't think there's any difficulty of using the
>> name to mean both things, just a small overlap during discussions.
>
> I guess I would have to agree with that.  It would proabably be workable.
>
> That said, it would be better to change the terminology for package.el -
> e.g., when we introduce CL-style "packages".

Sadly, no one is offering to do that.

I am offering to do something based on what I proposed. I can see how
that can be implemented.

I am explicitly not offering to implement the CommonLisp name spacing
system in Emacs.

I can promise that I won't needlessly differ from it. Where I do differ
from it there will be a good reason.


Of course, I would be very happy if you, or someone else, built a proper
CommonLisp namespacing system in Emacs. Then I wouldn't even have to
implement my lesser proposal.


Nic

RE: CommonLisp namespace system (was Re: adding namespaces to emacs-lisp (better elisp?))

[Drew Adams]

> > However, in practice I don't think there's any difficulty of using the
> > name to mean both things, just a small overlap during discussions.
> 
> I guess I would have to agree with that.  It would proabably be workable.
> 
> That said, it would be better to change the terminology for package.el -
> e.g., when we introduce CL-style "packages".

Probably quite workable, actually.

If we don't change the "package" terminology for package.el, here is one proposal for referring to package.el packages vs Common Lisp packages:

* Use "library package" for packages of libraries.

* Use "symbol package" for packages of symbols.

Of course, if function and variable names included the qualifiers then
that would aid discovery etc.  But we could still try to have aliases
to the Common Lisp functions & variables (which do not include "symbol"
in the name) for any Emacs Lisp versions that are close enough.

Re: CommonLisp namespace system

[Lars Brinkhoff]

Nic Ferrier <nferrier@ferrier.me.uk> writes:
> Drew Adams <drew.adams@oracle.com> writes:
>> That said, it would be better to change the terminology for
>> package.el - e.g., when we introduce CL-style "packages".
> Sadly, no one is offering to do that. [...]  Of course, I would be
> very happy if you, or someone else, built a proper CommonLisp
> namespacing system in Emacs. Then I wouldn't even have to implement
> my lesser proposal.

I have a rather complete implementation of CL packages written in
Emacs Lisp.

It's not intended for use with Emacs Lisp programs, but for CL
programs running in Emacs.  Also its own definitions lacks customary
prefixes.  And it somewhat crudely uses of a hash table to store the
symbol-package mapping.  Etc, etc.

I would be intersting in bringing it up to shape for inclusion in
Emacs, if there is any interest.

Most of the code can be seen here:
http://github.com/larsbrinkhoff/emacs-cl/blob/master/src/cl-packages.el
http://github.com/larsbrinkhoff/emacs-cl/blob/master/src/cl-symbols.el

RE: CommonLisp namespace system (was Re: adding namespaces to emacs-lisp (better elisp?))

[Drew Adams]

> I am explicitly not offering to implement the CommonLisp name spacing
> system in Emacs.
> 
> I can promise that I won't needlessly differ from it. Where I do differ
> from it there will be a good reason.

Again, could you please "spell [out] the differences out in more detail
somewhere"?  How does what you propose differ from the Common Lisp
package system?  What are the differences and the reasons for them?

RE: CommonLisp namespace system

[Drew Adams]

> I have a rather complete implementation of CL packages written in
> Emacs Lisp.
> 
> It's not intended for use with Emacs Lisp programs, but for CL
> programs running in Emacs.  Also its own definitions lacks customary
> prefixes.  And it somewhat crudely uses of a hash table to store the
> symbol-package mapping.  Etc, etc.
> 
> I would be intersting in bringing it up to shape for inclusion in
> Emacs, if there is any interest.

Sounds good to me, based on the description.

Re: adding namespaces to emacs-lisp (better elisp?)

[Stefan Monnier]

>> Here's a scenario:
>> - namespaced packages A and B both locally define a function `toto'.
>> - non-namespaced package C comes along with a symbol `toto' somewhere in
>> its code, suddenly causing A and B's `toto' to be global rather
>> than local.
> I don't think this is a serious problem personally. But I'm also not
> wedded to global-obarray-first.

I think it's a very serious problem because it means packages A and
B are at the mercy of some random wholly unrelated package C.

>> Note that instead of "non-namespaced package C", we could have some
>> package which uses symbols as "uniquified strings" and which uses the
>> global obarray for it and might occasionally intern `toto' in the course
>> of its normal execution.
> Again, it only matters if it's a non-namespaced package that does it.

No, a namespaced package can just as easily call `intern'.
But even if you can hope the a namespaced package wouldn't do it, the
non-namespaced packages are very numerous and do all kinds of nasty
stuff and we have very little control over them (e.g. they're not
bundled with Emacs).

>> IOW I think we should instead first look in the local obarray (over
>> which the coder does have control) and if that fails then look in the
>> global obarray.
> I am not wedded to the proposal of using the global obarray first. The
> rules for interning are slightly more complicated in that case:
> - given a string X
> - lookup X in the local obarray
> - if it exists return the symbol
> - else
> -  lookup X in the global obarray
> -  if it exists return the symbol
> -  else
> -    add the symbol to the local obarray

Exactly.  In Mathematica, they have a list of obarrays to check in
sequence and a "current" obarray to which things are added if the
lookup fails.  Sounds clean and simple to me.

> The only problem I see here is the possibility of problems with
> concurrency. The whole operation above would have to be atomic and it
> involves lookups in two separate data structures.

That sounds like a very remote problem to me.  And if/when concurrency
is added it doesn't seem like it should be difficult to make it
work reliably.

> My feeling is that an import should be like the creation of an alias.

Function alias?  Variable alias?
I don't much like the sounds of it.  I'd much rather make sure they are
simply one and the same symbol (I guess "symbol alias").

> It's not that I don't like it per-se. I just want this to be easy to
> implement in the first instance. If the implementation gets more
> difficult later I have no problem with that. But initial low cost is a
> good thing.

I'm not sure why the implementation should be difficult.  `intern' would
"simply" need to parse the string into a list of elements (separated by
"::" or whatever), then lookup the first element in the obarray, which
should contain another obarray, then lookup the second element in that
obarray, etc... until the last element which is handled "in the old
way".

Then (import 'nic as 'foo) would amount to (setq foo nic)


        Stefan

Re: CommonLisp namespace system (was Re: adding namespaces to emacs-lisp (better elisp?))

[Pascal J. Bourguignon]

Nic Ferrier <nferrier@ferrier.me.uk> writes:

> Drew Adams <drew.adams@oracle.com> writes:
>
>> I read your proposal overview, Nic.  It's not clear to me just what
>> the differences would be from the Common Lisp package system.
>> Perhaps you could spell the differences out in more detail somewhere.
>>
>> But the closer we can get to the CL spec the better, IMO.  If we
>> could conform to it completely, that would be great.
>
> I disagree. Emacs isn't CommonLisp, never has been CommonLisp and
> very likely, never will be CommonLisp.
>
>
>> Even keeping the same terminology, symbol names etc. as CL would
>> help.  It would help users who are coming from Common Lisp or who
>> happen to read Common Lisp doc.
>
> But at the expense of muddying the waters for people who are not from
> that world.
>
> In Emacs world, we use "package" to mean something different from what
> CL "package" means. So right at the start that effort is doomed without
> a major change to Emacs 24.

Absolutely NOT.

That's the point of packages!

emacs:package can be different from common-lisp:package.

(cl:defpackage "my-program" 
  (:use "emacs")
  (:export "package"))

(cl:in-package "my-program")

(package-install "ups")

(defstruct package
   id
   destination-address
   expeditor-address
   weight)


There, you have three different packages.  common-lisp:package is the
namespace type, emacs:package is the system.  my-program:package is an
enterprise object representing some parcel being sent thru UPS.



>> Of course, adopting CL terminology in this regard should mean that
>> we would drop the terminology used so far for Emacs "packages".

It would be easier on the random user, but as I've shown above, once you
have packages, there's no confusion possible amongst the different kind
of packages.


>> An argument can be made that both uses of the word "package" are
>> somewhat unfortunate.

Definitely.


> I am trying to make a namespace system that would be backwards
> compatible with Emacs and yet encourage future good behaviour.
>
> CommonLisp terminology or compatibility is not a major aim of mine.

That's unfortunate.  This leads to a well known route, where ten or
twenty years, hundreds of thousands of users curse you and have to
proceed with a new round of standardization of a common common lisp. 


In four words: it's plain dumb.


-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.  
You know you've been lisping too long when you see a recent picture of George 
Lucas and think "Wait, I thought John McCarthy was dead!" -- Dalek_Baldwin

Re: CommonLisp namespace system (was Re: adding namespaces to emacs-lisp (better elisp?))

[Pascal J. Bourguignon]

Sebastian Wiesner <lunaryorn@gmail.com> writes:

> 2013/7/26 Drew Adams <drew.adams@oracle.com>:
>>> In Emacs world, we use "package" to mean something different from what
>>> CL "package" means.
>>
>> That's a very recent introduction to the "Emacs world".  Hardly much of
>> a precedent.  "In [the] Emacs world" is a pretty bold way of describing
>> something we just introduced, as if it were essential to what Emacs Lisp
>> has always been.  It is a recent add-on - a welcome one, but hardly core.
>
> As a developer of Emacs Lisp extensions, I consider package.el a core
> addon, no matter how old it is.  I think it's the only more or less
> sane and comfortable way to distribute Emacs Lisp code, and I see it
> being used by many Emacs Lisp developers as primary distribution
> channel for their libraries.
>
> Imho, adding package.el to Emacs has boosted the productivity of the
> Emacs community more than any past attempt to make Emacs Lisp more
> Common Lisp.

Yes, just like quicklisp in CL.  If emacs had been closer to CL,
quicklisp could have run on emacs, and you could have spared the effort
of developping package.el.  Duh.


>> And the question here is not about abandoning package.el etc.  It is
>> about the terminology: "package".  Who heard of Emacs "packages" a few
>> years ago?  Contrast that with who had heard of Common Lisp "packages".
>
> That's a bold saying, too.  I doubt that even Common Lisp itself has
> much relevance to many Emacs users.  I doubt even that it's known to
> many.

A lot of people ask emacs lisp questions on the Common Lisp forums.  Too
bad often even pure programming questions cannot be answered because of
gratuituous differences in emacs.


-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.  
You know you've been lisping too long when you see a recent picture of George 
Lucas and think "Wait, I thought John McCarthy was dead!" -- Dalek_Baldwin

Re: CommonLisp namespace system (was Re: adding namespaces to emacs-lisp (better elisp?))

[Juanma Barranquero]

On Fri, Jul 26, 2013 at 8:45 PM, Tom Tromey <tromey@redhat.com> wrote:

> I think the name clash is unfortunate.  Bad self.  However, in practice
> I don't think there's any difficulty of using the name to mean both
> things, just a small overlap during discussions.

Currently, we use "display":

 - as a verb, to mean show, as in `display-warning',
`display-time-mode', etc., and of course all the `display-buffer-*'
things.
 - as reference to X displays and the equivalent W32 and NS
"displays": `display-screens', `display-pixel-width', etc.
 - for the `display-table' structures.

In languages, context is everything. Ambiguity is the norm. As prof.
Pullum likes to say: "Languages love multiple meanings. They lust
after them. They roll around in them like a dog in fresh grass."
(http://languagelog.ldc.upenn.edu/nll/?p=3083)

(Consider this a +1 for as-Common-Lispy-as-possible, package-ly-named packages.)

   J

Re: adding namespaces to emacs-lisp (better elisp?)

[Nic Ferrier]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Note that instead of "non-namespaced package C", we could have some
>>> package which uses symbols as "uniquified strings" and which uses the
>>> global obarray for it and might occasionally intern `toto' in the course
>>> of its normal execution.
>> Again, it only matters if it's a non-namespaced package that does it.
>
> No, a namespaced package can just as easily call `intern'.
> But even if you can hope the a namespaced package wouldn't do it, the
> non-namespaced packages are very numerous and do all kinds of nasty
> stuff and we have very little control over them (e.g. they're not
> bundled with Emacs).

If a namespace package calls intern the symbol is interned in the
private obarray.

>> The only problem I see here is the possibility of problems with
>> concurrency. The whole operation above would have to be atomic and it
>> involves lookups in two separate data structures.
>
> That sounds like a very remote problem to me.  And if/when concurrency
> is added it doesn't seem like it should be difficult to make it
> work reliably.

Agreed.

>> My feeling is that an import should be like the creation of an alias.
>
> Function alias?  Variable alias?
> I don't much like the sounds of it.  I'd much rather make sure they are
> simply one and the same symbol (I guess "symbol alias").

Ok. Well I think I'm going to go through that in much more detail and
update the proposal.

>> It's not that I don't like it per-se. I just want this to be easy to
>> implement in the first instance. If the implementation gets more
>> difficult later I have no problem with that. But initial low cost is a
>> good thing.
>
> I'm not sure why the implementation should be difficult.  `intern' would
> "simply" need to parse the string into a list of elements (separated by
> "::" or whatever), then lookup the first element in the obarray, which
> should contain another obarray, then lookup the second element in that
> obarray, etc... until the last element which is handled "in the old
> way".

It would clearly be more work to add the extra indirection to allow
namespaced namespaces than to have one level.
 
But I'll spec it out more closely.


Nic

Re: CommonLisp namespace system (was Re: adding namespaces to emacs-lisp (better elisp?))

[Nic Ferrier]

"Pascal J. Bourguignon" <pjb@informatimago.com> writes:

> In four words: it's plain dumb.

You are welcome to do things you think are smart.

I can only do the same.


Nic

Re: CommonLisp namespace system

[Nic Ferrier]

Drew Adams <drew.adams@oracle.com> writes:

>> I have a rather complete implementation of CL packages written in
>> Emacs Lisp.
>> 
>> It's not intended for use with Emacs Lisp programs, but for CL
>> programs running in Emacs.  Also its own definitions lacks customary
>> prefixes.  And it somewhat crudely uses of a hash table to store the
>> symbol-package mapping.  Etc, etc.
>> 
>> I would be intersting in bringing it up to shape for inclusion in
>> Emacs, if there is any interest.
>
> Sounds good to me, based on the description.

Go to it! I hope it works with all the Emacs tooling, edebug, eldoc,
etc...


Nic

RE: adding namespaces to emacs-lisp (better elisp?)

[Drew Adams]

> It would clearly be more work to add the extra indirection to allow
> namespaced namespaces than to have one level.

Namespaced namespaces?  I haven't followed that part of the discussion.

FWIW, Common Lisp has no namespaced namespaces.  There is a single
namespace for all package names.

RE: adding namespaces to emacs-lisp (better elisp?)

[Drew Adams]

> > I am not wedded to the proposal of using the global obarray first. The
> > rules for interning are slightly more complicated in that case:
> > - given a string X
> > - lookup X in the local obarray
> > - if it exists return the symbol
> > - else
> > -  lookup X in the global obarray
> > -  if it exists return the symbol
> > -  else
> > -    add the symbol to the local obarray
> 
> Exactly.  In Mathematica, they have a list of obarrays to check in
> sequence and a "current" obarray to which things are added if the
> lookup fails.  Sounds clean and simple to me.

And in Common Lisp (from CLTL, 1984 - didn't check CLTL2):

"When the Lisp reader has, by parsing, obtained a string of characters
 thought to name a symbol, that name is looked up in the current package.
 This lookup may involve looking in other packages whose external symbols
 are inherited by the current package.  If the name is found, the
 corresponding symbol is returned.  If the name is not found (that is,
 there is no symbol of that name accessible in the current package), a
 new symbol is created for it and is placed in the current package as an
 internal symbol.  Moreover, the current package becomes the owner (home
 package) of the symbol, and so the symbol becomes interned in the current
 package.  If the name is later read again while this same package is
 current, the same symbol will then be found and returned."

> > My feeling is that an import should be like the creation of an alias.
> 
> Function alias?  Variable alias?
> I don't much like the sounds of it.  I'd much rather make sure they are
> simply one and the same symbol (I guess "symbol alias").

Yes.

In CL there are two different things, BTW:

* `import': Importing a symbol into a package.  Importing makes the
symbol *present*, not just *accessible* in the importing package.  If a different symbol with the same name is already accessible in the
importing package then a user-correctable error is raised: `import'
avoids letting one symbol shadow another.

* `use-package': Inheriting a symbol, by using another package in which
it is external.  The symbol becomes *accessible* as an internal symbol
in the using package.  (There is no way to inherit the internal symbols
of a package.)

`use-package', unlike `import', does not cause any new symbols to be
present in the using package.  It just makes them accessible by
inheritance.

A symbol is *accessible* in a package if it can be referred to without
a package qualifier when that package is current.  A symbol is *present*
in a package if the name-to-symbol mapping (obarray entry) is in the
package itself and is not inherited.  A symbol is interned in a given
package if (a) it is accessible there and (b) it is owned by some package
(i.e., it is interned somewhere). 

`intern' causes a symbol to be interned in a package, like this:

"[I]t first looks for the symbol among the external and internal symbols
 of the package itself [i.e., present there]; then it looks through the
 external symbols of the used packages in some unspecified order.  The
 order does not matter; according to the rules for handling name conflicts
 ..., if conflicting symbols appear in two or more packages inherited by
 package X, a symbol of this name must also appear [i.e., be present] in
 X itself as a shadowing symbol." 

"If the symbol was previously unowned, then the package it is interned
 in becomes its owner (home package); but if the symbol was previously
 owned by another package, that other package continues to own the
 symbol."

FWIW, perhaps the most important part of the CL packages design are
these consistency rules:

"
* Read-read consistency: Reading the same print name always results in
  the same (`eq') symbol.

* Print-read consistency: An interned symbol always prints as a sequence
  of characters that, when read back in, yields the same (`eq') symbol.

* Print-print consistency: If two interned symbols are not `eq', then
  their printed representations will be different sequences of characters.
"

Re: adding namespaces to emacs-lisp (better elisp?)

[Stefan Monnier]

> If a namespace package calls intern the symbol is interned in the
> private obarray.

I'm not sure how you could do that and neither am I sure that would
preserve backward compatibility.  So without more details of how that
would work, it sounds like wishful thinking to me.

>> I'm not sure why the implementation should be difficult.  `intern' would
>> "simply" need to parse the string into a list of elements (separated by
>> "::" or whatever), then lookup the first element in the obarray, which
>> should contain another obarray, then lookup the second element in that
>> obarray, etc... until the last element which is handled "in the old
>> way".
> It would clearly be more work to add the extra indirection to allow
> namespaced namespaces than to have one level.

Could be, but not necessarily.  It could amount to not much more than
replacing a `when' with a `while'.


        Stefan

Re: adding namespaces to emacs-lisp (better elisp?)

[Nic Ferrier]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> If a namespace package calls intern the symbol is interned in the
>> private obarray.
>
> I'm not sure how you could do that and neither am I sure that would
> preserve backward compatibility.  So without more details of how that
> would work, it sounds like wishful thinking to me.

This is the core of my proposal.

The guts of intern need to be altered to be aware of when it is being
called in a packaged context. I am currently expecting to be able to do
this with a file local variable.

intern can then do it's lookup/add thing with the global/specific lookup
instead of just the global.


Nic

Re: adding namespaces to emacs-lisp (better elisp?)

[Stefan Monnier]

> * `import': Importing a symbol into a package.  Importing makes the
> symbol *present*, not just *accessible* in the importing package.
> If a different symbol with the same name is already accessible in the
> importing package then a user-correctable error is raised: `import'
> avoids letting one symbol shadow another.

Sounds like CL's approach requires symbols to be present in several
packages, which might require more changes than I'd like in the way
obarrays and symbols work.

AFAIK in Mathematica, the list of obarrays to search is just part of the
current reader state, not a property of obarrays, whereas it seems that
in CL the list of obarrays to search is stored in obarrays as a list of
"parent" obarrays (so the list of obarrays to search is changed when we
change the current obarray, whereas in Mathematica the two are unrelated).

I get the impression that Mathematica's design might be fairly easy to
reproduce with the current Emacs code, whereas CL's design would
probably require more changes.


        Stefan

Re: adding namespaces to emacs-lisp (better elisp?)

[Stefan Monnier]

>>> If a namespace package calls intern the symbol is interned in the
>>> private obarray.
>> I'm not sure how you could do that and neither am I sure that would
>> preserve backward compatibility.  So without more details of how that
>> would work, it sounds like wishful thinking to me.
> This is the core of my proposal.
> The guts of intern need to be altered to be aware of when it is being
> called in a packaged context. I am currently expecting to be able to do
> this with a file local variable.

That's easy when the reader calls `intern', but I'm talking about when
the Elisp code itself makes calls to `intern'.


        Stefan

RE: adding namespaces to emacs-lisp (better elisp?)

[Drew Adams]

> > * `import': Importing a symbol into a package.  Importing makes the
> > symbol *present*, not just *accessible* in the importing package.
> > If a different symbol with the same name is already accessible in the
> > importing package then a user-correctable error is raised: `import'
> > avoids letting one symbol shadow another.
> 
> Sounds like CL's approach requires symbols to be present in several
> packages, which might require more changes than I'd like in the way
> obarrays and symbols work.
> 
> AFAIK in Mathematica, the list of obarrays to search is just part of the
> current reader state, not a property of obarrays, whereas it seems that
> in CL the list of obarrays to search is stored in obarrays as a list of
> "parent" obarrays (so the list of obarrays to search is changed when we
> change the current obarray, whereas in Mathematica the two are unrelated).
> 
> I get the impression that Mathematica's design might be fairly easy to
> reproduce with the current Emacs code, whereas CL's design would
> probably require more changes.

Please do not judge only by my paraphase.  Please check CLTL2 yourself
before deciding.  A lot of thought - and years of practice - went into
the definition of the CL package system.  Wrt the consistency rules,
for example, CLTL says this:

"Package-related bugs can be very subtle and confusing: things are not
 what they appear to be.  The Common Lisp package system is designed
 with a number of safety features to prevent most of the common bugs
 that would otherwise occur in normal use.  This may seem over-protective,
 but experience with earlier package systems has shown that such safety
 features are needed."

Re: adding namespaces to emacs-lisp (better elisp?)

[Lars Brinkhoff]

Stefan Monnier wrote:
> Sounds like CL's approach requires symbols to be present in several
> packages, which might require more changes than I'd like in the way
> obarrays and symbols work.

When I did my implementation of CL packages, I did consider using
obarrays for the symbol table.  But as you say, they are not quite
suitable for that because a symbol may have to be interned in more
than one package.  Whereas an Emacs symbol may only be intered in one
obarray.  Or so I vaguely recall.

Like I wrote before, I'm ready to "port" my code to a be a proper part
of Emacs, but not if it's considered a no-go from the start.

Re: adding namespaces to emacs-lisp (better elisp?)

[Richard Stallman]

        [ To any NSA and FBI agents reading my email: please consider
        [ whether defending the US Constitution against all enemies,
        [ foreign or domestic, requires you to follow Snowden's example.

I used the Common Lisp package system on the Lisp machine and found it
completely unhelpful.  Using name prefixes is better.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: CommonLisp namespace system (was Re: adding namespaces to emacs-lisp (better elisp?))

[Richard Stallman]

        [ To any NSA and FBI agents reading my email: please consider
        [ whether defending the US Constitution against all enemies,
        [ foreign or domestic, requires you to follow Snowden's example.

The dispute about the best order of obarrays does not surprise me.
What I recall, from when we designed the CL package system, is that
no order of search gave good results.  This is because it tries to
use multiple symbols for a job where what's really wanted is
some sort of scoping for the meanings of symbols.

There are cases where this can't work cleanly, mainly because the
obarray has to be chosen when the symbol is read, and what you really
want is to choose later on.

If you want to use symbols of the form foo:bar, I think it
is better to write them as foo:bar all the time.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: CommonLisp namespace system (was Re: adding namespaces to emacs-lisp (better elisp?))

[Nic Ferrier]

Richard Stallman <rms@gnu.org> writes:

> If you want to use symbols of the form foo:bar, I think it
> is better to write them as foo:bar all the time.

People don't want that though.

They want to avoid having long names for symbols or name clashes for
symbols.

The only way to achieve that is to separate the global name for the
symbol from the name used within the code.


Nic

Re: adding namespaces to emacs-lisp (better elisp?)

[Nic Ferrier]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> If a namespace package calls intern the symbol is interned in the
>>>> private obarray.
>>> I'm not sure how you could do that and neither am I sure that would
>>> preserve backward compatibility.  So without more details of how that
>>> would work, it sounds like wishful thinking to me.
>> This is the core of my proposal.
>> The guts of intern need to be altered to be aware of when it is being
>> called in a packaged context. I am currently expecting to be able to do
>> this with a file local variable.
>
> That's easy when the reader calls `intern', but I'm talking about when
> the Elisp code itself makes calls to `intern'.

I don't understand what would be different for my implementation
strategy. When intern is called directly as a function the evaluator
still knows about file local variables and so can detect whether it is
in a package or not.

What am I missing?

This is the crucial part really. It's why I thought it would be
achievable, because a package could just be a special file local
variable (that *is* similar to CL's system).


Nic

Re: CommonLisp namespace system

[Lars Brinkhoff]

Richard Stallman wrote:
> There are cases where this can't work cleanly, mainly because the
> obarray has to be chosen when the symbol is read, and what you really
> want is to choose later on.

There is another Lisp namespace implementation called "lexicons",
where the namespace resolution is done at compile time rather than at
read time as with CL packages.  Maybe some people would find that more
their taste.

An introduction is here:
http://blog.rongarret.info/2010/02/new-and-improved-lexicons-now-50-lexier.html

Longer article:
http://www.flownet.com/ron/lisp/lexicons.pdf

Source code:
http://www.flownet.com/ron/lisp/lexicons.lisp

Re: adding namespaces to emacs-lisp (better elisp?)

[Pascal J. Bourguignon]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> * `import': Importing a symbol into a package.  Importing makes the
>> symbol *present*, not just *accessible* in the importing package.
>> If a different symbol with the same name is already accessible in the
>> importing package then a user-correctable error is raised: `import'
>> avoids letting one symbol shadow another.
>
> Sounds like CL's approach requires symbols to be present in several
> packages, which might require more changes than I'd like in the way
> obarrays and symbols work.

Depends on what you mean by "present".  

Symbols cannot be interned inseveral packages.  They can be interned
only once, and in a single package.


cl-user> (defpackage :a (:use))
#<Package "A">
cl-user> (defpackage :b (:use))
#<Package "B">
cl-user> (intern "AA" :a)
a::aa
nil
cl-user> (import 'a::aa :b)
t
cl-user> (find-symbol "AA" :a)
a::aa
:internal
cl-user> (find-symbol "AA" :b)
a::aa
:internal
cl-user> (intern "AA" :a)
a::aa
:internal
cl-user> (intern "AA" :b)
a::aa
:internal
cl-user> 


The symbol named "AA" with which I played there, was interned only in
the package named "A", and only once (in the first call to INTERN).
Once I've imported this symbol in the package named "B', it is visible
there: FIND-SYMBOL and INTERN find it.  But it's still the symbol named
"AA" interned in the package named "A".

Notice that import may also intern a uninterned symbol into the give
package:

cl-user> (let ((s (make-symbol "S")))
           (import s :a))
t
cl-user> (find-symbol "S" :a)
a::s
:internal
cl-user> 

But that's not the usual case.


You can find a "reference" implementation of a CL package system in:
https://gitorious.org/com-informatimago/com-informatimago/trees/master/common-lisp/lisp-reader

Notice in find-symbol
https://gitorious.org/com-informatimago/com-informatimago/blobs/master/common-lisp/lisp-reader/package-fun.lisp#line1318

how the notion of "present" symbol only comes in the last place before
three other ways to find symbols in packages, some of them that may not
be interned in any way in that package (:inherited).


-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.  
You know you've been lisping too long when you see a recent picture of George 
Lucas and think "Wait, I thought John McCarthy was dead!" -- Dalek_Baldwin

Re: adding namespaces to emacs-lisp (better elisp?)

[Pascal J. Bourguignon]

Lars Brinkhoff <lars@nocrew.org> writes:

> Stefan Monnier wrote:
>> Sounds like CL's approach requires symbols to be present in several
>> packages, which might require more changes than I'd like in the way
>> obarrays and symbols work.
>
> When I did my implementation of CL packages, I did consider using
> obarrays for the symbol table.  But as you say, they are not quite
> suitable for that because a symbol may have to be interned in more
> than one package.  Whereas an Emacs symbol may only be intered in one
> obarray.  Or so I vaguely recall.

That must be another reason, because it's not the case.  See my previous
message.

> Like I wrote before, I'm ready to "port" my code to a be a proper part
> of Emacs, but not if it's considered a no-go from the start.

Your code has the advantage over mine of being written in emacs lisp
already :-)

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.  
You know you've been lisping too long when you see a recent picture of George 
Lucas and think "Wait, I thought John McCarthy was dead!" -- Dalek_Baldwin

Re: adding namespaces to emacs-lisp (better elisp?)

[Pascal J. Bourguignon]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> If a namespace package calls intern the symbol is interned in the
>>>> private obarray.
>>> I'm not sure how you could do that and neither am I sure that would
>>> preserve backward compatibility.  So without more details of how that
>>> would work, it sounds like wishful thinking to me.
>> This is the core of my proposal.
>> The guts of intern need to be altered to be aware of when it is being
>> called in a packaged context. I am currently expecting to be able to do
>> this with a file local variable.
>
> That's easy when the reader calls `intern', but I'm talking about when
> the Elisp code itself makes calls to `intern'.

Well, (emacs:intern name) is equivalent to (cl:intern name :emacs).
The difficult point to deal with is (emacs:intern ":keyword").  You want
it to be equivalent to (cl:intern "keyword" :keyword) instead, but this
would break some emacs lisp code.  You can leave it as (cl:intern name
:emacs) but then :keyword and emacs:\:keyword would have to be unified.


-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.  
You know you've been lisping too long when you see a recent picture of George 
Lucas and think "Wait, I thought John McCarthy was dead!" -- Dalek_Baldwin

Re: CommonLisp namespace system (was Re: adding namespaces to emacs-lisp (better elisp?))

[Bastien]

Hi Nic and all,

Nic Ferrier <nferrier@ferrier.me.uk> writes:

> They want to avoid having long names for symbols or name clashes for
> symbols.

You can have namespaces clashes too, it's purely conventional.

So to me the only complaint is about symbols' length.

But is that really such a big problem?

The common practice for Emacs Lisp seems to favor explicit and long
names over terse and hard-to-decipher ones.  I guess the length of
the symbols is more due to this (good) practice than to the length
of prefixes.

Personally, what I really like in Emacs Lisp is precisely the fact
that variables and functions names are available from the same and
unique global space.  When I read "org-x", it immediately tells me
where the variable is from, and I can use it as such from anywhere.

This is an advantage in terms of both discoverability and readability.

Maybe C-h a and friends can be clever enough to look up by taking a
namespace into account, but human-reading of the name of a variable
will not always be able to do this.

To sum up: I like the visual contextual clue you get with the prefix
and I'm not sure the length of symbols in *Elisp* is such a problem.

2 cts of course,

-- 
 Bastien

Re: CommonLisp namespace system (was Re: adding namespaces to emacs-lisp (better elisp?))

[David Engster]

Bastien writes:
> You can have namespaces clashes too, it's purely conventional.
>
> So to me the only complaint is about symbols' length.
>
> But is that really such a big problem?
>
> The common practice for Emacs Lisp seems to favor explicit and long
> names over terse and hard-to-decipher ones.  I guess the length of
> the symbols is more due to this (good) practice than to the length
> of prefixes.

For me, the main problem with the length of symbols is that I
intuitively tend to avoid writing them. For instance, I usually avoid
using defstructs for nicely wrapping global state, because always
writing things like

(setf (my-prefix-struct-somevalue my-prefix--struct) 'foobar)

is just verbose and tiresome.

-David

Re: adding namespaces to emacs-lisp (better elisp?)

[Stefan Monnier]

> I don't understand what would be different for my implementation
> strategy. When intern is called directly as a function the evaluator
> still knows about file local variables and so can detect whether it is
> in a package or not.

> What am I missing?

When intern is called directly, the buffer/file from which the code was
read probably doesn't even exist any more.


        Stefan

Re: adding namespaces to emacs-lisp (better elisp?)

[Stefan Monnier]

> The symbol named "AA" with which I played there, was interned only in
> the package named "A", and only once (in the first call to INTERN).
> Once I've imported this symbol in the package named "B', it is visible
> there: FIND-SYMBOL and INTERN find it.  But it's still the symbol named

Right, so there is a new notion of a symbol being "present" in an
obarray, whereas in Emacs there's only the notion of a symbol being
"interned" in an obarray.


        Stefan

Re: adding namespaces to emacs-lisp (better elisp?)

[Nic Ferrier]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I don't understand what would be different for my implementation
>> strategy. When intern is called directly as a function the evaluator
>> still knows about file local variables and so can detect whether it is
>> in a package or not.
>
>> What am I missing?
>
> When intern is called directly, the buffer/file from which the code was
> read probably doesn't even exist any more.

I do see your point... but that information will have to be persisted
such that intern knows whether it is part of a package or not. That's
the whole basis of the system I think, that intern have that
information.


Nic

RE: adding namespaces to emacs-lisp (better elisp?)

[Drew Adams]

> > The symbol named "AA" with which I played there, was interned only in
> > the package named "A", and only once (in the first call to INTERN).
> > Once I've imported this symbol in the package named "B', it is visible
> > there: FIND-SYMBOL and INTERN find it.  But it's still the symbol named
> 
> Right, so there is a new notion of a symbol being "present" in an
> obarray, whereas in Emacs there's only the notion of a symbol being
> "interned" in an obarray.

Yes, but it's not new ;-), though it would be new to Emacs Lisp.  It is
30 years old.  And it was designed after years of experimentation with
different Lisp namespace ideas and implementations.

On the question of "presence" vs "accessibility" (or "appearance") in a
package, see my previous msg and CLTL(2), chapter Packages [*],
particularly the parts about a symbol being "owned" by at most one package,
its "home package".  (An uninterned symbol has no home package - no package
owns it.)

Pascal's example makes things clear.  See also the consistency rules
I quoted earlier.

[*] http://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node111.html#XPACK

Re: CommonLisp namespace system (was Re: adding namespaces to emacs-lisp (better elisp?))

[Nic Ferrier]

David Engster <deng@randomsample.de> writes:

> Bastien writes:
>> You can have namespaces clashes too, it's purely conventional.
>>
>> So to me the only complaint is about symbols' length.
>>
>> But is that really such a big problem?
>>
>> The common practice for Emacs Lisp seems to favor explicit and long
>> names over terse and hard-to-decipher ones.  I guess the length of
>> the symbols is more due to this (good) practice than to the length
>> of prefixes.
>
> For me, the main problem with the length of symbols is that I
> intuitively tend to avoid writing them. For instance, I usually avoid
> using defstructs for nicely wrapping global state, because always
> writing things like
>
> (setf (my-prefix-struct-somevalue my-prefix--struct) 'foobar)
>
> is just verbose and tiresome.

This is half the reason I want to do something about namespaces. When
names get very long indeed even the spaces start to disappear.

Re: adding namespaces to emacs-lisp (better elisp?)

[Stefan Monnier]

> I do see your point... but that information will have to be persisted
> such that intern knows whether it is part of a package or not.  That's
> the whole basis of the system I think, that intern have that
> information.

I don't see how to do it.  When intern is called by the reader, the
reader can provide that info as extra info, but intern itself can't know.


        Stefan

Re: CommonLisp namespace system (was Re: adding namespaces to emacs-lisp (better elisp?))

[Richard Stallman]

        [ To any NSA and FBI agents reading my email: please consider
        [ whether defending the US Constitution against all enemies,
        [ foreign or domestic, requires you to follow Snowden's example.

    They want to avoid having long names for symbols or name clashes for
    symbols.

Emacs provides lots of ways to type the long names with fewer
characters.  That is a better solution because it doesn't complicate
the meaning of code.

If people don't want to have shorter names appear in the code,
then I suggest a system of read-time aliases:

  (def-read-alias 'foo 'foo:bar)

would tell the reader to replace `foo' with `foo:bar':

  'foo => foo:bar

`:::' could inhibit the alias processing for the symbol that follows.

  ':::foo => foo

prin1 could detect the cases where aliasing occurs
and replace the expansions with their aliases, or use `:::'
to protect against aliasing.

A file could load read aliases by calling `load-read-aliases'.
For instance,

   (load-read-aliases "cl-read-aliases")

Eval from files and buffers would have to handle this form specially.
Compilation too.

In the case of reading from a buffer, Emacs would set up the list of
aliases and pass that as an argument to `eval-region'.  Emacs could
use that list of aliases to make various Emacs features DTRT with them.


-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: CommonLisp namespace system (was Re: adding namespaces to emacs-lisp (better elisp?))

[Nic Ferrier]

Richard Stallman <rms@gnu.org> writes:

>         [ To any NSA and FBI agents reading my email: please consider
>         [ whether defending the US Constitution against all enemies,
>         [ foreign or domestic, requires you to follow Snowden's example.
>
>     They want to avoid having long names for symbols or name clashes for
>     symbols.
>
> Emacs provides lots of ways to type the long names with fewer
> characters.  That is a better solution because it doesn't complicate
> the meaning of code.

It does. It's very good like that.

But it doesn't provide any ways for reading the long names.


> If people don't want to have shorter names appear in the code,
> then I suggest a system of read-time aliases:
>
>   (def-read-alias 'foo 'foo:bar)
>
> would tell the reader to replace `foo' with `foo:bar':
>
>   'foo => foo:bar
>
> `:::' could inhibit the alias processing for the symbol that follows.
>
>   ':::foo => foo

Is this meant to be buffer-local/file-local somehow? Because otherwise
there is a name space pollution problem.


Nic

Re: CommonLisp namespace system (was Re: adding namespaces to emacs-lisp (better elisp?))

[Jambunathan K]

Nic Ferrier <nferrier@ferrier.me.uk> writes:

> But it doesn't provide any ways for reading the long names.

Give Emacspeak a try.

Re: CommonLisp namespace system (was Re: adding namespaces to emacs-lisp (better elisp?))

[Richard Stallman]

        [ To any NSA and FBI agents reading my email: please consider
        [ whether defending the US Constitution against all enemies,
        [ foreign or domestic, requires you to follow Snowden's example.

    Is this meant to be buffer-local/file-local somehow?

Yes, of course.  That is the point of this:

    A file could load read aliases by calling `load-read-aliases'.

If the aliases were global, no such thing would be needed;
you could use `load' to read a file of alias definitions.

I'm sorry I did not say this explicitly.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: CommonLisp namespace system (was Re: adding namespaces to emacs-lisp (better elisp?))

[Nic Ferrier]

Richard Stallman <rms@gnu.org> writes:

>         [ To any NSA and FBI agents reading my email: please consider
>         [ whether defending the US Constitution against all enemies,
>         [ foreign or domestic, requires you to follow Snowden's example.
>
>     Is this meant to be buffer-local/file-local somehow?
>
> Yes, of course.  That is the point of this:
>
>     A file could load read aliases by calling `load-read-aliases'.

Ok. But all my namespace proposal is, is basically this, with automatic
'load-read-alias'. 

I don't understand why it's a good thing to have this as a separate
operation.  Programmers will have to be explicitly encouraged to use it,
as opposed to implicitly encouraged to use it.


Nic

Re: CommonLisp namespace system (was Re: adding namespaces to emacs-lisp (better elisp?))

[Richard Stallman]

        [ To any NSA and FBI agents reading my email: please consider
        [ whether defending the US Constitution against all enemies,
        [ foreign or domestic, requires you to follow Snowden's example.

    Ok. But all my namespace proposal is, is basically this, with automatic
    'load-read-alias'. 

The virtue of my proposal is that it does not change the meaning of
symbols, obarrays, or any built-in functions.  It has no effect on any
file that doesn't use it, so it can't break anything, and if you don't
choose to use this facility, you don't need to know about it.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: CommonLisp namespace system (was Re: adding namespaces to emacs-lisp (better elisp?))

[Nic Ferrier]

Richard Stallman <rms@gnu.org> writes:

>         [ To any NSA and FBI agents reading my email: please consider
>         [ whether defending the US Constitution against all enemies,
>         [ foreign or domestic, requires you to follow Snowden's example.
>
>     Ok. But all my namespace proposal is, is basically this, with automatic
>     'load-read-alias'. 
>
> The virtue of my proposal is that it does not change the meaning of
> symbols, obarrays, or any built-in functions.  It has no effect on any
> file that doesn't use it, so it can't break anything, and if you don't
> choose to use this facility, you don't need to know about it.

I don't think there's anything in my proposal that causes any of those
effects either.

I am aiming for absolutely the same thing you are.

This is why I have been saying that those asking for a CommonLisp
namespace system should look to someone other than me to build it.



Nic

Re: CommonLisp namespace system

[Lars Brinkhoff]

Nic Ferrier wrote:
> Drew Adams wrote:
>> I wrote:
>>> I have a rather complete implementation of CL packages written in
>>> Emacs Lisp.  I would be interst[ed] in bringing it up to shape for
>>> inclusion in Emacs, if there is any interest.
>> Sounds good to me, based on the description.
> Go to it!

It now seems likely that it would be a waste of time, so I'm not
inclined to work on it at this time.

Re: CommonLisp namespace system (was Re: adding namespaces to emacs-lisp (better elisp?))

[Richard Stallman]

        [ To any NSA and FBI agents reading my email: please consider
        [ whether defending the US Constitution against all enemies,
        [ foreign or domestic, requires you to follow Snowden's example.

    I don't think there's anything in my proposal that causes any of those
    effects either.

The proposals I read involved prefixes with special significance, and
searching obarrays.  Maybe yours was not one of them.  Could you email
me your proposal?


-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: adding namespaces to emacs-lisp (better elisp?)

[Lars Brinkhoff]

Pascal J. Bourguignon wrote:
>> When I did my implementation of CL packages, I did consider using
>> obarrays for the symbol table.  But as you say, they are not quite
>> suitable for that because a symbol may have to be interned in more
>> than one package.  Whereas an Emacs symbol may only be intered in
>> one obarray.  Or so I vaguely recall.
> That must be another reason, because it's not the case.

Right, I was vague on the details.  How about this?

In Common Lisp, it may be the case that one particular symbol can be
found (e.g. by FIND-SYMBOL) in multiple Lisp packages.  But in Emacs
Lisp, a symbol can only be found (e.g. by intern-soft) in one single
obarray.  This made me hesitate to use obarrays to implement packages.
However, it can be made to work, and may actually be the best solution.