Proper namespaces in Elisp

Proper namespaces in Elisp

[João Távora]

Around 4 years ago, the "future of Emacs Lisp" came up, and
we discussed namespaces in Elisp. Well the future is now,
sort of.

Here's a sample of that thread:

https://lists.gnu.org/archive/html/emacs-devel/2016-10/msg00145.html

An here's the idea: if the compiler uses the lexical-binding file-local var
when compiling specific files, or functions inside those files, why can't
it use a local-namespaces var when reading symbols?  I think this
very idea was floated at the time, but I can't find it.

It would solve part of the problem people have here, one could
use that variable to shadow/rename symbols locally.  A modern-string.el
library with a long discriminating "modern-string-" prefix could have
its symbols locally accessible by "ms-" or even the "" empty string,
if the the using file's author so desired.

Same for modern-regexp.el library. And whoever likes those
libraries (and maybe I would be one of them), wouldn't be constrained
by this aliasing proposal which is very limitative, since it has to
keep the order of arguments and other parts of the contract.

thing-at-point would also be taught about these local-namespaces
so xref and C-h f stuff would work  well.  Grep wouldn't know about it,
sure, that's true, but that's something that already afflicts other
languages with namespace qualifiers and isn't a terrible thing.
Any other great difficulties you can think ot?

João Távora

PS: There's also this, but it's not based on file-local variables.

https://github.com/ellerh/namespace.el

Re: Proper namespaces in Elisp

[Adam Porter]

See also Artur Malabarba's https://github.com/Malabarba/names.  It's in
MELPA and is used by a few MELPA packages.

Re: Proper namespaces in Elisp

[Clément Pit-Claudel]

On 04/05/2020 06.52, João Távora wrote:
> An here's the idea: if the compiler uses the lexical-binding file-local var
> when compiling specific files, or functions inside those files, why can't
> it use a local-namespaces var when reading symbols?  I think this
> very idea was floated at the time, but I can't find it.

https://lists.gnu.org/archive/html/emacs-devel/2013-07/msg00738.html maybe?  I think your proposal is quite close to Nic's

Re: Proper namespaces in Elisp

[Clément Pit-Claudel]

On 04/05/2020 11.11, Adam Porter wrote:
> See also Artur Malabarba's https://github.com/Malabarba/names.  It's in
> MELPA and is used by a few MELPA packages.

And https://github.com/Malabarba/nameless/.  I'm surprised that this package hasn't seen more adoption.  It's basically solved the namespacing problem for me.
It uses font-lock to shorten package prefixes: my-package-xyz gets displayed as :xyz, my-package--abc gets displayed as ::abc, and other prefixes can be added (by default font-lock-xyz gets displayed as fl:xyz).
The ':' prefix has a special face that makes it clear that it's an abbreviation.

Re: Proper namespaces in Elisp

[Stefan Monnier]

> PS: There's also this, but it's not based on file-local variables.
> https://github.com/ellerh/namespace.el

In http://www.iro.umontreal.ca/~monnier/hopl-4-emacs-lisp.pdf
we list a few other attempts.
BTW, we have two attempts in GNU ELPA already (`names` and `nameless`).


        Stefan

Re: Proper namespaces in Elisp

[João Távora]

On Mon, May 4, 2020 at 4:45 PM Clément Pit-Claudel
<cpitclaudel@gmail.com> wrote:
>
> On 04/05/2020 11.11, Adam Porter wrote:
> > See also Artur Malabarba's https://github.com/Malabarba/names.  It's in
> > MELPA and is used by a few MELPA packages.
>
> And https://github.com/Malabarba/nameless/.  I'm surprised that this package hasn't seen more adoption.

I think the obvious objection would be: what do I type?
What about 'git diff'?  And github/gitlab? I mean, it's a clever
hack but it would drive me nuts, eventually.

João

Re: Proper namespaces in Elisp

[João Távora]

On Mon, May 4, 2020 at 4:29 PM Clément Pit-Claudel
<cpitclaudel@gmail.com> wrote:
>
> On 04/05/2020 06.52, João Távora wrote:
> > An here's the idea: if the compiler uses the lexical-binding file-local var
> > when compiling specific files, or functions inside those files, why can't
> > it use a local-namespaces var when reading symbols?  I think this
> > very idea was floated at the time, but I can't find it.
>
> https://lists.gnu.org/archive/html/emacs-devel/2013-07/msg00738.html maybe?  I think your proposal is quite close to Nic's

Yes it is.  Thanks you very much!  And his blog post describing the
is very good:

http://nic.ferrier.me.uk/blog/2013_06/adding-namespaces-to-elisp

Can't tell, but  it seems the thing fizzled partially because some people
wanted other systems.  I would also prefer CL packages but if a lower
effort thing can solve _some_ problems, and be backward compatible
and not slow down compilation, maybe we should give it a shot.

Anyway, are there any contemporary objections to Nic's plan, in
particular I am interested in knowing if there any any major objections
to Nic's reader logic:

- given a string X
- lookup X in the local obarray
- if it exists return the symbol
- else
-  lookup X in the global obarray
-  if it exists return the symbol
-  else
-    add the symbol to the local obarray


João

Re: Proper namespaces in Elisp

[Adam Porter]

João Távora <joaotavora@gmail.com> writes:

>> And https://github.com/Malabarba/nameless/.  I'm surprised that this
>> package hasn't seen more adoption.
>
> I think the obvious objection would be: what do I type?  What about
> 'git diff'?  And github/gitlab? I mean, it's a clever hack but it
> would drive me nuts, eventually.

IME it caused issues with indentation: either indentation looked wrong
when the transformations were applied to the display, or indentation
looked wrong in the saved code.  I tried to hack up a
before/after-save-hook to work around it, but it didn't work well for
me.

Since the Names package seems to be mature now, I'm planning to try it.
But it would be preferable to have built-in namespaces in Elisp.  I'd be
glad if an idea like the one João mentioned were implemented.

Re: Proper namespaces in Elisp

[João Távora]

On Mon, May 4, 2020 at 5:42 PM Adam Porter <adam@alphapapa.net> wrote:

> Since the Names package seems to be mature now, I'm planning to try it.
> But it would be preferable to have built-in namespaces in Elisp.  I'd be
> glad if an idea like the one João mentioned were implemented.

Let me just say that "my idea" was not really thought through.
As far as I know, the only thing that I can more or less guarantee
that would work, because it's been implemented over and
over in different ways and different places, is the CL spec for
packages.  All other ideas seem vapourware or lighted tested
at best, so don't hold your breath.

João

Re: Proper namespaces in Elisp

[Clément Pit-Claudel]

On 04/05/2020 11.49, João Távora wrote:
> On Mon, May 4, 2020 at 4:45 PM Clément Pit-Claudel
> <cpitclaudel@gmail.com> wrote:
>>
>> On 04/05/2020 11.11, Adam Porter wrote:
>>> See also Artur Malabarba's https://github.com/Malabarba/names.  It's in
>>> MELPA and is used by a few MELPA packages.
>>
>> And https://github.com/Malabarba/nameless/.  I'm surprised that this package hasn't seen more adoption.
> 
> I think the obvious objection would be: what do I type?

Not sure what you mean? You type _, or C-c C--.

> What about 'git diff'?  

That one is easy: nameless is applicable there too, if you run it in Emacs.

Re: Proper namespaces in Elisp

[Helmut Eller]

> Anyway, are there any contemporary objections to Nic's plan, in
> particular I am interested in knowing if there any any major objections
> to Nic's reader logic:
>
> - given a string X
> - lookup X in the local obarray
> - if it exists return the symbol
> - else
> -  lookup X in the global obarray
> -  if it exists return the symbol
> -  else
> -    add the symbol to the local obarray

The problem with this approach is that the namespace management is done
by the Lisp reader, while many poeple (RMS in particular) think it
should be done by the compiler (like lexical scoping).  In other words,
those people would like a mechanism to manage the names of variable
bindings not a mechanism to manage symbols.

Helmut

Re: Proper namespaces in Elisp

[Stefan Monnier]

> particular I am interested in knowing if there any any major objections
> to Nic's reader logic:

Richard mentioned his profound dislike of solutions that operate in
the reader.  I'm not sufficiently familiar with Common Lisp's package
system to have a strong opinion on that, but Richard's criticism does
appeal to me (I've been brought up on namespace systems more like
Standard ML's structures).


        Stefan

Re: Proper namespaces in Elisp

[João Távora]

On Mon, May 4, 2020 at 7:30 PM Helmut Eller <eller.helmut@gmail.com> wrote:
> should be done by the compiler (like lexical scoping).  In other words,
> those people would like a mechanism to manage the names of variable
> bindings not a mechanism to manage symbols.

That last sentence sounds clever but I can't fully grok it. Do you
really mean "variable bindings" or just "bindings"? (including function's).

On Mon, May 4, 2020 at 7:39 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> Richard mentioned his profound dislike of solutions that operate in
> the reader.  I'm not sufficiently familiar with Common Lisp's package
> system to have a strong opinion on that, but Richard's criticism does
> appeal to me (I've been brought up on namespace systems more like
> Standard ML's structures).

I can't even imagine what a solution that _doesn't_ operate on the reader
looks like. Are you talking about form walking macros that walk the
forms and switch the short version of names into the long versions
and intern everything into the same obarray?  Hmmm.

João

Re: Proper namespaces in Elisp

[Stefan Monnier]

> I can't even imagine what a solution that _doesn't_ operate on the reader
> looks like. Are you talking about form walking macros that walk the
> forms and switch the short version of names into the long versions
> and intern everything into the same obarray?  Hmmm.

Something like it, yes.

E.g. in `macroexpand` we'd change the symbol case so that when a symbol
has the shape `s:bar` we treat it a bit as if we were looking
at `(: s bar)` so we pass it to the `:` macro which might replace it
with `(gethash s-obarray 'bar)` or with `string-bar` or ...

It comes with its own hurdles, obviously: e.g. making it work not just
on variables but on function names, face names, etc...

You can look at Scheme and Clojure for yet more ways to skin this cat.

AFAICT, adding namespaces to Elisp will only happen if someone comes up
with a working implementation and Emacs's head maintainer is strongly
supportive.  I don't think Eli is more strongly supportive of the idea
than I was, and yet it didn't happen during my tenure, so I won't hold
my breath.


        Stefan

Re: Proper namespaces in Elisp

[João Távora]

On Mon, May 4, 2020 at 8:20 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> > I can't even imagine what a solution that _doesn't_ operate on the reader
> > looks like. Are you talking about form walking macros that walk the
> > forms and switch the short version of names into the long versions
> > and intern everything into the same obarray?  Hmmm.
>
> Something like it, yes.
>
> E.g. in `macroexpand` we'd change the symbol case so that when a symbol
> has the shape `s:bar` we treat it a bit as if we were looking
> at `(: s bar)` so we pass it to the `:` macro which might replace it
> with `(gethash s-obarray 'bar)` or with `string-bar` or ...

Curious, I could live without the ":", TBH.  If in a file where
local-namespaces is the alist (("^s-" . "modern-string-")), I'd just
like to have an implicit macro walk the tree, naively replacing atom's
names with the listed transformations, using replace-regexp-in-string,
then intern and return that. No special obarray.  In other words, one
would find both "s-bar" (since that had been read already) and
"modern-string-bar" in the obarray, but only "modern-string-bar" would
have any meaning attached.  Then one would type either version in the
source file.  The advantage is that we could compile all that code
that
relies on s.el with minimal (if any) changes and a proper
modern-string.el.

This is easier done at the reader level, I think (then no polluting
"s-bar" would be found in obarray).  But I think it would work in the
macroexpand phase too, I think.  Some vapourware to consider.  And if
this doesn't count as namespaces, we could call it a
"compiler-assisted
shorthand system", I don't care.

João Távora

Re: Proper namespaces in Elisp

[Vladimir Sedach]

João Távora <joaotavora@gmail.com> writes:
> I think the obvious objection would be: what do I type?
> What about 'git diff'?  And github/gitlab? I mean, it's a clever
> hack but it would drive me nuts, eventually.

This is not that different from the venerable glasses.el. You get
used to it

--
Vladimir Sedach
Software engineering services in Los Angeles https://oneofus.la

Re: Proper namespaces in Elisp

[Vladimir Sedach]

João Távora <joaotavora@gmail.com> writes:
> I would also prefer CL packages but if a lower
> effort thing can solve _some_ problems, and be backward compatible
> and not slow down compilation, maybe we should give it a shot.

The one big problem with the Common Lisp package system that should
not be copied over is the package :use mechanism:

--8<---------------cut here---------------start------------->8---
CL-USER> (defpackage "ABC" (:use "CL") (:export "FUNC1"))
#<PACKAGE "ABC">
CL-USER> (defun abc:func1 (x y) (list x y))
ABC:FUNC1
CL-USER> (abc:func1 1 2)
(1 2)
CL-USER> (defpackage "XYZ" (:use "CL" "ABC"))
#<PACKAGE "XYZ">
CL-USER> (defun xyz::func1 (x y) (+ x y))
WARNING: redefining ABC:FUNC1 in DEFUN
ABC:FUNC1
CL-USER> (abc:func1 1 2)
3
--8<---------------cut here---------------end--------------->8---

This causes problems when a new version of a library introduces a new
function FUNC1, which definition gets unknowingly overridden for
everybody by another package that uses the library. Explicitly using
:import avoids this problem, and makes figuring out who-uses-what
easier.

There has been lots of talk about the reader here, but no one has
brought up the printer. Any namespace system is going to have to
print symbols in a way such that they can be read back in EQually.

Common Lisp does not do a good job of this, because whether or not a
symbol's package prefix is printed depends on the current package you
are in when printing.

Any namespace system is going to have to be first-class itself, with
first-class identifiers (otherwise see Python, where code cannot be
re-loaded correctly depending on how import was done), which implies
full introspection support. Any system that makes it a chore to
re-define functions outside of the files they were defined in should
be rejected.

Namespaces are helpful for organizing code in a project, but they are
not going to stop people from naming their packages with one or two
or three letter names (happens with Common Lisp libraries all the
time), or stop people from naming their packages in a way that
conflicts with older libraries they were not aware of (also happens).
Those two arguments are not valid ones for introducing a namespace
system.

> Anyway, are there any contemporary objections to Nic's plan

I think a namespace system could be useful for:

1. Keeping people from accidentally defining and dynamically binding
   symbols they did not mean to in their packages.

2. Describing autoloads in a convenient first-class way. "Magic
   comments" and decorators are a terrible hack for defective
   languages; there is never a real need for them in Lisp. Same
   reason why any proposal that suggests declaring namespaces in
   comments should be rejected (my major objection to Nic's
   proposal).

3. Make it easier for people to see what commands/functions/variables
   a package provides, because they are all declared to be exported
   in a single place, and can also be queried from code because the
   namespace system is first-class and introspective.

--
Vladimir Sedach
Software engineering services in Los Angeles https://oneofus.la

Re: Proper namespaces in Elisp

[Andrea Corallo]

João Távora <joaotavora@gmail.com> writes:

> On Mon, May 4, 2020 at 7:30 PM Helmut Eller <eller.helmut@gmail.com> wrote:
>> should be done by the compiler (like lexical scoping).  In other words,
>> those people would like a mechanism to manage the names of variable
>> bindings not a mechanism to manage symbols.
>
> That last sentence sounds clever but I can't fully grok it. Do you
> really mean "variable bindings" or just "bindings"? (including function's).
>
> On Mon, May 4, 2020 at 7:39 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> Richard mentioned his profound dislike of solutions that operate in
>> the reader.  I'm not sufficiently familiar with Common Lisp's package
>> system to have a strong opinion on that, but Richard's criticism does
>> appeal to me (I've been brought up on namespace systems more like
>> Standard ML's structures).
>
> I can't even imagine what a solution that _doesn't_ operate on the reader
> looks like. Are you talking about form walking macros that walk the
> forms and switch the short version of names into the long versions
> and intern everything into the same obarray?  Hmmm.

Namespaces in the read time is a bad idea because it does not affect
only bindings but effectively all symbols.  I, as many others, do not
like CL package system too for this reason and all its implications.

For having a namespace system that works only on bindings a new
indirection must be added.  That implies that implementation-wise you
cannot keep the symbol-value directly in the symbol.  In this way you
can have more consumers of the same binding (that is what happend when
you import from another namespace).

One could argue that this will cause a slow down in code execution but
this is non true given this indirection has to be solved in the
compile-time for compiled or byte-compiled code.

Effectively namespace resolution is then moved from the read-time to the
compile-time.  And with that you solve also the problem of the code
walker because the compiler it is already *the* code walker.

I think Emacs has most of the infrastructure to implement this already
in the codebase but right now I've not time to work on a prototype (I'm
not sure either how much this feature is really desired).

  Andrea

--
akrl@sdf.org

Re: Proper namespaces in Elisp

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 2091 bytes --]

On Mon, May 4, 2020 at 10:51 PM Vladimir Sedach <vas@oneofus.la> wrote:
> --8<---------------cut here---------------start------------->8---
> CL-USER> (defpackage "ABC" (:use "CL") (:export "FUNC1"))
> #<PACKAGE "ABC">
> CL-USER> (defun abc:func1 (x y) (list x y))
> ABC:FUNC1
> CL-USER> (abc:func1 1 2)
> (1 2)
> CL-USER> (defpackage "XYZ" (:use "CL" "ABC"))
> #<PACKAGE "XYZ">
> CL-USER> (defun xyz::func1 (x y) (+ x y))
> WARNING: redefining ABC:FUNC1 in DEFUN
> ABC:FUNC1
> CL-USER> (abc:func1 1 2)
> 3
> --8<---------------cut here---------------end--------------->8---
>
> This causes problems when a new version of a library introduces a new
> function FUNC1, which definition gets unknowingly overridden for
> everybody by another package that uses the library. Explicitly using
> :import avoids this problem, and makes figuring out who-uses-what
> easier.

You're confused. With both :USE or :IMPORT there
is only ever one symbol named "FUNC1" in the package
named "ABC".  If someones changes the function
definition attached to it, then everybody sees it. When you
write XYZ::FUNC1 you're referring to the same symbol.

CL-USER> (defpackage "ABC" (:use "CL") (:export "FUNC1"))
#<The ABC package>
CL-USER> (defun abc:func1 (x y) (list x y))
ABC:FUNC1
CL-USER> (defpackage "XYZ" (:import-from "ABC" "FUNC1"))
#<The XYZ package>
CL-USER> (defun xyz::func1 (x y) (+ x y))
Warning: ABC:FUNC1 is defined more than once as `operator' in file NIL.
ABC:FUNC1
CL-USER> (abc:func1 1 2)
3 (2 bits, #x3, #o3, #b11)
CL-USER> (describe 'xyz::func1)
ABC:FUNC1 is a NEW SYMBOL.
  It is unbound.
  It is EXTERNAL in the ABC package and accessible in the XYZ package.
  Its function binding is #<Interpreted Function FUNC1>
    which function takes arguments (X Y)
; No values

That's the way it's supposed to work.

I don't understand how this is different from having a rogue
EL package define over a symbol in the same obarray.
I liked that your example tried to bring a real-life problem
to the table, but I can't make sense of it.

João

[-- Attachment #2: Type: text/html, Size: 2790 bytes --]

Re: Proper namespaces in Elisp

[João Távora]

On Mon, May 4, 2020 at 10:59 PM Andrea Corallo <akrl@sdf.org> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > On Mon, May 4, 2020 at 7:30 PM Helmut Eller <eller.helmut@gmail.com> wrote:
> >> should be done by the compiler (like lexical scoping).  In other words,
> >> those people would like a mechanism to manage the names of variable
> >> bindings not a mechanism to manage symbols.
> >
> > That last sentence sounds clever but I can't fully grok it. Do you
> > really mean "variable bindings" or just "bindings"? (including function's).
> >
> > On Mon, May 4, 2020 at 7:39 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> >> Richard mentioned his profound dislike of solutions that operate in
> >> the reader.  I'm not sufficiently familiar with Common Lisp's package
> >> system to have a strong opinion on that, but Richard's criticism does
> >> appeal to me (I've been brought up on namespace systems more like
> >> Standard ML's structures).
> >
> > I can't even imagine what a solution that _doesn't_ operate on the reader
> > looks like. Are you talking about form walking macros that walk the
> > forms and switch the short version of names into the long versions
> > and intern everything into the same obarray?  Hmmm.
>
> Namespaces in the read time is a bad idea because it does not affect
> only bindings but effectively all symbols.  I, as many others, do not
> like CL package system too for this reason and all its implications.

Well, let's have "all its implications" then. Because you've done
nothing more than describe in simplified fashion what it does,
and that's not an argument to support the contention that
"it is a bad idea".

I think it's a great idea, but I admit I've become so accustomed
to it I never have problems.  I've recently started working with
some CL newbies that caught on pretty fast, too.

What are concrete cases where having a CL-like package
system would cause bugs, confusion, time wasted, etc. Can
you present a hypothetical "horror story"? Oh, but to be fair
you have to present it against a system that also has
namespaces, otherwise it's not fair.

Actually, I can give you an argument against CL packages.
Because Emacs is also missing restarts, a symbol conflict is
hard to fix interactively. Most Lisps will ask the user what do
to when a symbol conflict is found and a good debugger
interface with proper restarts is very useful.

But let's not pretend that any namespacing system is
exempt from symbol conflicts. Indeed in Elisp we deal
with them every day, by inventing contortionist prefixes
and dealing with fun bugs at run-time.

> I think Emacs has most of the infrastructure to implement this already
> in the codebase but right now I've not time to work on a prototype (

I too think a dumb man's namespacing can be implemented,
just to alleviate this alias-all-the-functions pressure.  Did you
read my shorthand idea, I think it could fly.

> I'm not sure either how much this feature is really desired).

Neither am I but If it's not desired, it should be, because it's
hampering Elisp. Did you notice that the request that started
this all, to integrate s.el in Emacs is all bug forgotten and
people are thinking about giving extra names to longstanding
functions?  I for one think that users that like s.el should
use it, I just don't want it polluting and crashing with
everything else. If we had packages or dumb namespacing
that could happen and no-one would get hurt.

João

Re: Proper namespaces in Elisp

[João Távora]

On Mon, May 4, 2020 at 10:51 PM Vladimir Sedach <vas@oneofus.la> wrote:

> Common Lisp does not do a good job of this, because whether or not a
> symbol's package prefix is printed depends on the current package you
> are in when printing.

How is that a flaw? The point is that WRITE and READ are
complementary and they will produce the same symbols
in textual and symbolic representation.

If you want to always print/read consistently, print everything
from a package that doesn't use any other package, even :CL
SLIME/SLY do that.

Anyway, this is getting slightly off topic, I wasn't directly
arguing for CL packages, anyway.

João

Re: Proper namespaces in Elisp

[Vladimir Sedach]

João Távora <joaotavora@gmail.com> writes:
> I liked that your example tried to bring a real-life problem
> to the table, but I can't make sense of it.

Let me try a different way:

1. Library ABC version 1 does not define or export FUNC1.
2. Package XYZ :uses ABC, importing all of ABC's exported symbols.
3. Package XYZ defines FUNC1 for internal use. There is no problem.
4. Library ABC version 1.1 defines and exports FUNC1.
5. When XYZ is loaded with ABC version 1.1, XYZ unintentionally
   overrides the definition of FUNC1 for all users of ABC.

Perhaps that makes it more clear.

This is a real problem for a lot of Common Lisp packages right now,
that prevents popular libraries from adding more functionality.

--
Vladimir Sedach
Software engineering services in Los Angeles https://oneofus.la

Re: Proper namespaces in Elisp

[Vladimir Sedach]

João Távora <joaotavora@gmail.com> writes:
>> Common Lisp does not do a good job of this, because whether or not a
>> symbol's package prefix is printed depends on the current package you
>> are in when printing.
>
> How is that a flaw?

You just pointed it out:

> If you want to always print/read consistently, print everything
> from a package that doesn't use any other package, even :CL
> SLIME/SLY do that.

To get reading of printed s-expressions to work reliably in Common
Lisp requires a lot of coordination (not just packages). Any Elisp
namespace system should not add extra burden to reading of printed
s-expressions.

This is very on-topic, and also a good argument to avoid "magic
comments" and per-file settings. For example, if file A is
read/loaded, then file B is read/loaded, the effect should be the
same as reading/loading the concatenation of A and B. It is not clear
if this would be the case with Nic's proposal.

--
Vladimir Sedach
Software engineering services in Los Angeles https://oneofus.la

Re: Proper namespaces in Elisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > And https://github.com/Malabarba/nameless/.  I'm surprised that this package hasn't seen more adoption.  It's basically solved the namespacing problem for me.
  > It uses font-lock to shorten package prefixes: my-package-xyz gets displayed as :xyz, my-package--abc gets displayed as ::abc, and other prefixes can be added (by default font-lock-xyz gets displayed as fl:xyz).

Are you saying that this package implements _abbreviation_ for prefixes
in a totally determined way, with no searching of multiple namespaces?
It seems that way.

What causes problems with namespaces in Lisp is searching multiple
namespaces in series until one of them contains the desired symbol.
If the system does not do that, it might not cause a problem -- but it
won't do the same job that people expect.

I think this will cause the problems:

  > Anyway, are there any contemporary objections to Nic's plan, in
  > particular I am interested in knowing if there any any major objections
  > to Nic's reader logic:

  > - given a string X
  > - lookup X in the local obarray
  > - if it exists return the symbol
  > - else
  > -  lookup X in the global obarray
  > -  if it exists return the symbol
  > -  else
  > -    add the symbol to the local obarray

Any scheme for searching namespaces at read time will work out badly.
Good results from namespaces require doing the search based on how the
symbol will be used in each occurrence.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proper namespaces in Elisp

[Helmut Eller]

On Mon, May 04 2020, Andrea Corallo wrote:

> I think Emacs has most of the infrastructure to implement this already
> in the codebase

Except for macros.  Traditional Lisp macros do in fact work on symbols
and not on bindings.  If macros should work on bindings, then a rather
different macro system is needed, like the one of Scheme or Rust.  That
kinda works, but the simplicity and elegance of traditional macros is
lost.

Helmut

Re: Proper namespaces in Elisp

[João Távora]

On Tue, May 5, 2020 at 2:22 AM Vladimir Sedach <vas@oneofus.la> wrote:
> João Távora <joaotavora@gmail.com> writes:
> > I liked that your example tried to bring a real-life problem
> > to the table, but I can't make sense of it.
>
> Let me try a different way:
>
> 1. Library ABC version 1 does not define or export FUNC1.
> 2. Package XYZ :uses ABC, importing all of ABC's exported symbols.
> 3. Package XYZ defines FUNC1 for internal use. There is no problem.
> 4. Library ABC version 1.1 defines and exports FUNC1.
> 5. When XYZ is loaded with ABC version 1.1, XYZ unintentionally
>    overrides the definition of FUNC1 for all users of ABC.

In which implementation of CL? In the ones I know, step 5
will get you an error, exactly describing the conflict,
which you handle interactively or automhatically.
You can choose to shadow import the symbol, unintern
the conflicting symbol, or give up loading XYZ in its current
form.  You can reconsider :use for that new version of the
library or rename FUNC1 (which is for internal use anyway)
That's quite different of saying that :use is dangerous or
anything.  It's what it is if you know what it does, which is
saying "I want all of that API's symbols accessible here
without special qualification".  It's like in primary school:
remember those times when they merged two classes
and some kid in the other class had the same first name as
you? Same deal, basically.

BTW, what you described as dangerous is exactly what
happens in Elisp, _not_ Common Lisp.

> Perhaps that makes it more clear.

Nope.

> >> Common Lisp does not do a good job of this, because whether or not a
> >> symbol's package prefix is printed depends on the current package you
> >> are in when printing.
> > How is that a flaw?
> You just pointed it out:

I fail to see where. I think you'll agree that a  Vlamidir in emacs-devel
is not the same as a Vladimir somewhere else.  To communicate to
someone that I really do mean you, I wither print the full qualifying name
or I have to agree beforehand on a common frame of reference.  In CL,
that's *PACKAGE*. Again, if you only have one namespace you
don't have that problem.  But then you don't have namespaces, right?   :-)

João

Re: Proper namespaces in Elisp

[Andrea Corallo]

João Távora <joaotavora@gmail.com> writes:

> On Mon, May 4, 2020 at 10:59 PM Andrea Corallo <akrl@sdf.org> wrote:
>>
>> João Távora <joaotavora@gmail.com> writes:
>>
>> > On Mon, May 4, 2020 at 7:30 PM Helmut Eller <eller.helmut@gmail.com> wrote:
>> >> should be done by the compiler (like lexical scoping).  In other words,
>> >> those people would like a mechanism to manage the names of variable
>> >> bindings not a mechanism to manage symbols.
>> >
>> > That last sentence sounds clever but I can't fully grok it. Do you
>> > really mean "variable bindings" or just "bindings"? (including function's).
>> >
>> > On Mon, May 4, 2020 at 7:39 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> >> Richard mentioned his profound dislike of solutions that operate in
>> >> the reader.  I'm not sufficiently familiar with Common Lisp's package
>> >> system to have a strong opinion on that, but Richard's criticism does
>> >> appeal to me (I've been brought up on namespace systems more like
>> >> Standard ML's structures).
>> >
>> > I can't even imagine what a solution that _doesn't_ operate on the reader
>> > looks like. Are you talking about form walking macros that walk the
>> > forms and switch the short version of names into the long versions
>> > and intern everything into the same obarray?  Hmmm.
>>
>> Namespaces in the read time is a bad idea because it does not affect
>> only bindings but effectively all symbols.  I, as many others, do not
>> like CL package system too for this reason and all its implications.
>
> Well, let's have "all its implications" then. Because you've done
> nothing more than describe in simplified fashion what it does,
> and that's not an argument to support the contention that
> "it is a bad idea".

Here is a first simple implication of namespace at read-time:

#+SRC_BEGIN lisp

(make-package :xxx)
(in-package :xxx)
(defun foo () 'aaa)
(export 'foo)

(make-package :yyy)
(in-package :yyy)
(use-package :xxx)

(if (not (eq (foo) 'aaa))
    (print "boom"))

#+END_BEGIN

This is because foo is not returnig 'aaa' but 'xxx::aaa'.

It will be told that you have to export also 'aaa' from the package xxx
because "leaking symbols" is bad.  But in reality this implies you have
to identify all execution paths that can leak a symbol and remember to
export these syms otherwise you'll get subtle bugs for any symbol leaked
passed in any kind of object.

Sure can be done (it is done) and we all got used to that but is just a
source of bugs due to bad design, not a feature.

One should import from a package *bindings* not symbols.  Unfortunatelly
the implementation can't distinguish those at read time because those do
not exists already.

With all the goodies of CL packages are really not the best example to
look at :)

>
>> I think Emacs has most of the infrastructure to implement this already
>> in the codebase but right now I've not time to work on a prototype (
>
> I too think a dumb man's namespacing can be implemented,
> just to alleviate this alias-all-the-functions pressure.  Did you
> read my shorthand idea, I think it could fly.

I don't think the soulution should be dumb, and I think we should aim
for the correct solution if we decide to go for it.  IMO a good part
of the code is already present to support this but as usual the bring-up
would be considerably slower that the original optimistic expectation.

That said without a prototype all of this is just blablabla :)

Andrea

-- 
akrl@sdf.org

Re: Proper namespaces in Elisp

[Andrea Corallo]

Andrea Corallo <akrl@sdf.org> writes:

>> Well, let's have "all its implications" then. Because you've done
>> nothing more than describe in simplified fashion what it does,
>> and that's not an argument to support the contention that
>> "it is a bad idea".
>
> Here is a first simple implication of namespace at read-time:
>
> #+SRC_BEGIN lisp
>
> (make-package :xxx)
> (in-package :xxx)
> (defun foo () 'aaa)
> (export 'foo)
>
> (make-package :yyy)
> (in-package :yyy)
> (use-package :xxx)
>
> (if (not (eq (foo) 'aaa))
>     (print "boom"))
>
> #+END_BEGIN
>
> This is because foo is not returnig 'aaa' but 'xxx::aaa'.
>
> It will be told that you have to export also 'aaa' from the package xxx
> because "leaking symbols" is bad.  But in reality this implies you have
> to identify all execution paths that can leak a symbol and remember to
> export these syms otherwise you'll get subtle bugs for any symbol leaked
> passed in any kind of object.
>
> Sure can be done (it is done) and we all got used to that but is just a
> source of bugs due to bad design, not a feature.

OTOH in elisp we would write:

(defun xxx-foo ()
  'aaa)

and not

(defun xxx-foo ()
  'xxx-aaa)

And this because we are smarter then the reader :)

> One should import from a package *bindings* not symbols.  Unfortunatelly
> the implementation can't distinguish those at read time because those do
> not exists already.
>
> With all the goodies of CL packages are really not the best example to
> look at :)
>
>>
>>> I think Emacs has most of the infrastructure to implement this already
>>> in the codebase but right now I've not time to work on a prototype (
>>
>> I too think a dumb man's namespacing can be implemented,
>> just to alleviate this alias-all-the-functions pressure.  Did you
>> read my shorthand idea, I think it could fly.
>
> I don't think the soulution should be dumb, and I think we should aim
> for the correct solution if we decide to go for it.  IMO a good part
> of the code is already present to support this but as usual the bring-up
> would be considerably slower that the original optimistic expectation.
>
> That said without a prototype all of this is just blablabla :)
>
> Andrea

-- 
akrl@sdf.org

Re: Proper namespaces in Elisp

[João Távora]

On Tue, May 5, 2020 at 11:34 AM Andrea Corallo <akrl@sdf.org> wrote

> Here is a first simple implication of namespace at read-time:
>
> #+SRC_BEGIN lisp
>
> (make-package :xxx)
> (in-package :xxx)
> (defun foo () 'aaa)
> (export 'foo)
>
> (make-package :yyy)
> (in-package :yyy)
> (use-package :xxx)
>
> (if (not (eq (foo) 'aaa))
>     (print "boom"))
>
> #+END_BEGIN
>
> This is because foo is not returnig 'aaa' but 'xxx::aaa'.

Andrea, this makes no sense.  How is that different
from Elisp where foo returns 'xxx--aaa and yyy.el checks
for 'yyy-aaa? Because _that's_ what that program means,
except, because it is written in a properly namespaced
Lisp, it does it with fewer chars, is DRY, not WET.

Or don't you think packages should be allowed internal
symbols at all. Maybe you don't, but that's a question of
making a package system with the -- IMO quite ill-advised --
option to auto-exporting everything. It's _not_ an implication
of the reader-based approach in any way.

Or did you mean you want an auto-exported,
self-evaluating,  quick-to-type, one-big-namespace
qualifier? The keyword package is there for that, as
you probably know, and that's what everybody uses.
  _When_  they want those semantics.  So, for me,
being able to determine the semantics of my program
(also called "programming") with little typing is a very
good thing, a very good "implication",  it's exactly the
power I crave.

João

Re: Proper namespaces in Elisp

[Andrea Corallo]

João Távora <joaotavora@gmail.com> writes:

> On Tue, May 5, 2020 at 11:34 AM Andrea Corallo <akrl@sdf.org> wrote
>
>> Here is a first simple implication of namespace at read-time:
>>
>> #+SRC_BEGIN lisp
>>
>> (make-package :xxx)
>> (in-package :xxx)
>> (defun foo () 'aaa)
>> (export 'foo)
>>
>> (make-package :yyy)
>> (in-package :yyy)
>> (use-package :xxx)
>>
>> (if (not (eq (foo) 'aaa))
>>     (print "boom"))
>>
>> #+END_BEGIN
>>
>> This is because foo is not returnig 'aaa' but 'xxx::aaa'.
>
> Andrea, this makes no sense.

Hi João, I'm sorry it does make sense deeply to me.

> How is that different
> from Elisp where foo returns 'xxx--aaa and yyy.el checks
> for 'yyy-aaa? Because _that's_ what that program means,
> except, because it is written in a properly namespaced
> Lisp, it does it with fewer chars, is DRY, not WET.

I believe the program wants just to return the symbol 'aaa' and does not
want to bother about other 1000 complications of the CL package system.
And the reason for that is that returning a symbol does *not* create any
name clashing. The name clashing is dangerous only when you resolve the
binding.

> Or don't you think packages should be allowed internal
> symbols at all. Maybe you don't, but that's a question of
> making a package system with the -- IMO quite ill-advised --
> option to auto-exporting everything. It's _not_ an implication
> of the reader-based approach in any way.

I suggest to try to abstrac one second from the implementation point of
view.  This is a low level detail.

> Or did you mean you want an auto-exported,
> self-evaluating,  quick-to-type, one-big-namespace
> qualifier? The keyword package is there for that, as
> you probably know, and that's what everybody uses.
>   _When_  they want those semantics.  So, for me,
> being able to determine the semantics of my program
> (also called "programming") with little typing is a very
> good thing, a very good "implication",  it's exactly the
> power I crave.

I've no doubt you can make it working with CL, guess what they have made
full operating systems with it... and it worked.

I'm just saying that if you look to it from another perspective is very
complex and error prone system and this is an example.

Certainly an experienced CL users can use it because he understands how
this is implemented underneath and has all the knobs.  But is very hard
to understand why it was chosen to behave this way because the only
reason for that is the technical choice and the design was not driven by
ergonomic principles.

That said is just my opinion (arguably shared by a number of people) so
please take it just for what it is.

Andrea

-- 
akrl@sdf.org

Re: Proper namespaces in Elisp

[Stefan Monnier]

> Sure can be done (it is done) and we all got used to that but is just a
> source of bugs due to bad design, not a feature.

While I largely agree that working at the level of symbols is not fully
satisfactory, I don't think it deserves to be called "bad design".

If you work at the level of bindings only you will have trouble with all
the uses of symbols that are expected to be namespaced but appear quoted
in code (e.g. face names, variable and function names inside macros,
hooks, args to `add-to-list`, command names passed to `define-key`,
command names within menus, ...).

It's just a hard problem with no perfect solution, AFAIK.


        Stefan

Re: Proper namespaces in Elisp

[João Távora]

On Tue, May 5, 2020 at 2:34 PM Andrea Corallo <akrl@sdf.org> wrote:

> I believe the program wants just to return the symbol 'aaa' and does not
> want to bother about other 1000 complications of the CL package system.

But that's not the program you wrote.  If you want to return
a symbol that is universally univocally  independent of
context, you use a keyword, not an internal symbol.  That
is substituting a " ' " for a " : ", a one-char change. That
different program does what you want (which I suppose
is not print the string "boom")

You're thinking about One Big Namespace. Fine. But that's
Elisp. And it has the problems this thread is proposing to
solve.  But you can go full Elisp in Common Lisp, just use
a single package always.

> I suggest to try to abstract one second from the implementation point of
> view.  This is a low level detail.

I agree, but it's you who said the reader-based approach
had implications.

> > Or did you mean you want an auto-exported,
> > self-evaluating,  quick-to-type, one-big-namespace
> > qualifier? The keyword package is there for that, as
> > you probably know, and that's what everybody uses.
> >   _When_  they want those semantics.  So, for me,
> > being able to determine the semantics of my program
> > (also called "programming") with little typing is a very
> > good thing, a very good "implication",  it's exactly the
> > power I crave.
>
> I've no doubt you can make it working with CL, guess what they have made
> full operating systems with it... and it worked.

Even if they didn't make OS's out of it.  The point is I don't need
those to refute your particular arguments.

> I'm just saying that if you look to it from another perspective is very
> complex and error prone system and this is an example.

It's not complex. It's a powerful system.  Don't use the parts you
don't understand yet, then.  Look, if we had them, people would
happily hack on the "Emacs" package all the time and noone
would notice it.  But people you want to use modern-string.el
could (use-package :modern-string) and find that s:thingamabob
does that sweet thing they like.

And when you wanted to write a library, you would learn it.
It's not hard, believe me.

> That said is just my opinion (arguably shared by a number of people) so
> please take it just for what it is.

Oh I take it, sure.  I just think your particular one, based on the
examples you gave me, it's based on very light (mis)usage of
the package system and misunderstanding of its features.

You _could_ have said you don't like the ocasional erroring
and nagging by the debugger about symbol conflicts, but
I would  have answered, hey conflicts exists like when some
other Andrea shows up and we have to disambiguate.
Or you could have said that it is strange to mistype "foo" in
the REPL and then find FOO has been interned. Or how
it's tricky to resist DOS attacks in because of malicious
interning through the reader (if you don't take precautions).
But now that I think of it, these problems happen in Elisp too.

João

Re: Proper namespaces in Elisp

[João Távora]

On Tue, May 5, 2020 at 2:45 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> While I largely agree that working at the level of symbols is not fully
> satisfactory, I don't think it deserves to be called "bad design".
>
> If you work at the level of bindings only you will have trouble with all
> the uses of symbols that are expected to be namespaced but appear quoted
> in code (e.g. face names, variable and function names inside macros,
> hooks, args to `add-to-list`, command names passed to `define-key`,
> command names within menus, ...).

Sure, there's that. A symbol can have many types of meaning
attached, not just "variable" and "function". This is a wonderful
thing IMO, a transparent Singleton pattern (avant la lettre btw)
We already do that in symbol plists in Elisp. But, yes, no
namespacing. That sucks.

> It's just a hard problem with no perfect solution, AFAIK.

What's your take on "why CL packages are bad", Stefan?

João

Re: Proper namespaces in Elisp

[Andrea Corallo]

João Távora <joaotavora@gmail.com> writes:

> On Tue, May 5, 2020 at 2:34 PM Andrea Corallo <akrl@sdf.org> wrote:
>
>> I believe the program wants just to return the symbol 'aaa' and does not
>> want to bother about other 1000 complications of the CL package system.
>
> But that's not the program you wrote.  If you want to return
> a symbol that is universally univocally  independent of
> context, you use a keyword, not an internal symbol.  That
> is substituting a " ' " for a " : ", a one-char change. That
> different program does what you want (which I suppose
> is not print the string "boom")

Yes is how this is worked around that in CL. It works, I'm just saying
is error prone and unnecessary complex.

> You're thinking about One Big Namespace. Fine. But that's
> Elisp. And it has the problems this thread is proposing to
> solve.  But you can go full Elisp in Common Lisp, just use
> a single package always.

No, I'm thinking to one namespace only for symbols *not* for bindings.
This is not CL or Elisp either.

>>
>> I've no doubt you can make it working with CL, guess what they have made
>> full operating systems with it... and it worked.
>
> Even if they didn't make OS's out of it.  The point is I don't need
> those to refute your particular arguments.
>
>> I'm just saying that if you look to it from another perspective is very
>> complex and error prone system and this is an example.
>
> It's not complex. It's a powerful system.  Don't use the parts you
> don't understand yet, then.  Look, if we had them, people would
> happily hack on the "Emacs" package all the time and noone
> would notice it.  But people you want to use modern-string.el
> could (use-package :modern-string) and find that s:thingamabob
> does that sweet thing they like.
>
> And when you wanted to write a library, you would learn it.
> It's not hard, believe me.

Thanks for telling me I could learn it.  I'm trying to say that I think
is unnecessary complex being too low level and that there could be
another way.  But okay if you prefer I just haven't learned how it
works.

>> That said is just my opinion (arguably shared by a number of people) so
>> please take it just for what it is.
>
> Oh I take it, sure.  I just think your particular one, based on the
> examples you gave me, it's based on very light (mis)usage of
> the package system and misunderstanding of its features.

I've just understood how CL packages works now!  Thanks I'm now finally
convinced.  No way it can work in a way that is not already this, it
would just implies the system is not understood.

BTW I leave these two papers, sad to see that they have misunderstood
the system too:

https://zenodo.org/record/2648195
http://www.flownet.com/gat/locales.pdf

Andrea

-- 
akrl@sdf.org

Re: Proper namespaces in Elisp

[Tom Tromey]

>>>>> "João" == João Távora <joaotavora@gmail.com> writes:

João> An here's the idea: if the compiler uses the lexical-binding file-local var
João> when compiling specific files, or functions inside those files, why can't
João> it use a local-namespaces var when reading symbols?  I think this
João> very idea was floated at the time, but I can't find it.

I did something like this

https://github.com/tromey/emacs-module

It lets you write short names for things in your elisp; and lets you
import names with a shorter form.  However, the underlying symbols
actually use prefixes, in the traditional elisp way.

Unlike the "Names" package, it doesn't require wrapping your entire file
in a macro.  Instead it works using advice, in particular on
internal-macroexpand-for-load.

João> thing-at-point would also be taught about these local-namespaces
João> so xref and C-h f stuff would work  well.  Grep wouldn't know about it,
João> sure, that's true, but that's something that already afflicts other
João> languages with namespace qualifiers and isn't a terrible thing.
João> Any other great difficulties you can think ot?

These were the main problems I had when using it.

Tom

Re: Proper namespaces in Elisp

[Vladimir Sedach]

João Távora <joaotavora@gmail.com> writes:
>> 1. Library ABC version 1 does not define or export FUNC1.
>> 2. Package XYZ :uses ABC, importing all of ABC's exported symbols.
>> 3. Package XYZ defines FUNC1 for internal use. There is no problem.
>> 4. Library ABC version 1.1 defines and exports FUNC1.
>> 5. When XYZ is loaded with ABC version 1.1, XYZ unintentionally
>>    overrides the definition of FUNC1 for all users of ABC.
>
> In which implementation of CL? In the ones I know, step 5
> will get you an error, exactly describing the conflict,
> which you handle interactively or automhatically.

That (NAME-CONFLICT error) only happens at run-time if you have ABC
version 1 already loaded, load XYZ, then load ABC version 1.1.

If you install XYZ, and ABC version 1.1 is pulled in as a dependency,
the only thing you might see is a warning that FUNC1 is being
re-defined.

The compiler cannot magically tell if you are re-defining FUNC1 by
accident or intentionally.

--
Vladimir Sedach
Software engineering services in Los Angeles https://oneofus.la

Re: Proper namespaces in Elisp

[João Távora]

On Tue, May 5, 2020 at 3:27 PM Andrea Corallo <akrl@sdf.org> wrote:
> Yes is how this is worked around that in CL. It works, I'm just saying
> is error prone and unnecessary complex.

What is complex? Understanding how quoting works?
Replacing the quote with a colon?

> No, I'm thinking to one namespace only for symbols *not* for bindings.
> This is not CL or Elisp either.

I don't understand, I admit.  I thought this conversation was about
you telling me why a reader-based system is a bad idea.

(It seems Tom and Stefan have explained that to a point BTW,
it's that internal Emacs features rely on certain names being
interned in the main package. Not necessarily a big problem
IMO, we can discuss that if you want.)

> Thanks for telling me I could learn it.  I'm trying to say that I think
> is unnecessary complex being too low level and that there could be
> another way.  But okay if you prefer I just haven't learned how it
> works.

It's that you insist it should do A when you ask it to do B.
It's like expecting printf("Hello World") to print "Goodbye".

>  https://zenodo.org/record/2648195
>  http://www.flownet.com/gat/locales.pdf

Normally, I'd ignore a lazy paper-drop, but I happened
to know one of them and investigated the other. They are
both short.

So, I'll take the bait:

[1] https://zenodo.org/record/2648195
[2] http://www.flownet.com/gat/locales.pdf

Surely you are aware that [1] is proposing a reader-based system
where symbols are like packages themselves, so that you can
have basically packages inside packages, recursively. It's very
interesting, but incomplete and fraught with difficulties and
shortcomings as admitted by the author himself. But to quote
_that_ as support that plain CL packages are "very complex"?

Also, in [1]  in its section "Problems with CL packages", a very short
section in a very short paper, starts off by admitting that it's
"CL packages are a misunderstood part of CL" [1, p2]. Sounds
familiar, indeed.

Then, the main criticism it levels against them is read-time
side-effects, and points to [2] for justification. We check it out.
[2] show us the unintentional interning of a symbols in the REPL
caused an typo and an unusual way to :USE packages.  Forgets
to mention a REPL is interactive, by definition and that most Lisps
will contain restarts that explain the error and allow you to choose
a solution and continue cleanly after the error. The author also
presents the package version in an unnecessarily verbose way.
Finally that author admits, at the end of the section [2, page 4]
that "This is a matter of taste."  De Gustibus Non Est Disputandum.

Not exactly categorical, i would say.

So you could have just said "it's not my taste".

João




João

Re: Proper namespaces in Elisp

[João Távora]

On Tue, May 5, 2020 at 5:41 PM Vladimir Sedach <vas@oneofus.la> wrote:
> João Távora <joaotavora@gmail.com> writes:
> >> 1. Library ABC version 1 does not define or export FUNC1.
> >> 2. Package XYZ :uses ABC, importing all of ABC's exported symbols.
> >> 3. Package XYZ defines FUNC1 for internal use. There is no problem.
> >> 4. Library ABC version 1.1 defines and exports FUNC1.
> >> 5. When XYZ is loaded with ABC version 1.1, XYZ unintentionally
> >>    overrides the definition of FUNC1 for all users of ABC.
> >
> > In which implementation of CL? In the ones I know, step 5
> > will get you an error, exactly describing the conflict,
> > which you handle interactively or automhatically.
>
> That (NAME-CONFLICT error) only happens at run-time if you have ABC
> version 1 already loaded, load XYZ, then load ABC version 1.1.
>
> If you install XYZ, and ABC version 1.1 is pulled in as a dependency,
> the only thing you might see is a warning that FUNC1 is being
> re-defined.

Ah, but that wasn't your recipe 1-5 step, was it?  Anyway, it
sounds like you're describing dependencies failure. The
author of XYZ should probably say that his package
requires ABC 1.0. Or start paying attention to warnings. Or
stop `:USE` of packages of systems whose code he doesn't
control. This is a problem that happens in other languages.

Also note: I don't know what you use to "pull in dependencies",
but quicklisp in particular won't let this happen, since it works
in distributions. Packages within a dist are compatible between
them, that is checked by automated tests. Much like GNU
ELPA.

So the things you think happen, really just don't happen IRL.

João

Re: Proper namespaces in Elisp

[João Távora]

On Tue, May 5, 2020 at 4:10 PM Tom Tromey <tom@tromey.com> wrote:
> >>>>> "João" == João Távora <joaotavora@gmail.com> writes:
>
> João> An here's the idea: if the compiler uses the lexical-binding file-local var
> João> when compiling specific files, or functions inside those files, why can't
> João> it use a local-namespaces var when reading symbols?  I think this
> João> very idea was floated at the time, but I can't find it.
>
> I did something like this
>
> https://github.com/tromey/emacs-module
>
> It lets you write short names for things in your elisp; and lets you
> import names with a shorter form.  However, the underlying symbols
> actually use prefixes, in the traditional elisp way.

So you're saying one just types "s-format", but the actual symbol
that elisp sees is modern-string-format? That's great! Are both
symbols interned?  Only for vars&functions? I could live with that.

It seems similar my "shorthand" idea
(https://lists.gnu.org/archive/html/emacs-devel/2020-05/msg00617.html)
but that idea would fall into the pitfall of:

    "the Emacs core sometimes requires certain symbol names in APIs."

unless, that is, you added syntax to escape renaming in _those_
situations, like `::`

> Unlike the "Names" package, it doesn't require wrapping your entire file
> in a macro.

That's good.  With some more tweaking maybe you could even
use a fancy ";;; Directive: " instead of the the `define-module` and
the `import-module`, and have modules automatically defined from
file names and existing Elisp internal/external naming conventions.
This is so that those top-level forms don't remind people too much
of the CL packages they love to hate :-)

>  Instead it works using advice, in particular on
> internal-macroexpand-for-load.

That's good!

> João> thing-at-point would also be taught about these local-namespaces
> João> so xref and C-h f stuff would work  well.  Grep wouldn't know about it,
> João> sure, that's true, but that's something that already afflicts other
> João> languages with namespace qualifiers and isn't a terrible thing.
> João> Any other great difficulties you can think ot?
> These were the main problems I had when using it.

I think I could fix the thing-at-point thing reasonably easily. And, as
I wrote, the grep thing is just what you already get anyway when
you start using namespaces, in any language.


--
João Távora

Re: Proper namespaces in Elisp

[Andrea Corallo]

João Távora <joaotavora@gmail.com> writes:

> On Tue, May 5, 2020 at 3:27 PM Andrea Corallo <akrl@sdf.org> wrote:
>> Yes is how this is worked around that in CL. It works, I'm just saying
>> is error prone and unnecessary complex.
>
> What is complex? Understanding how quoting works?
> Replacing the quote with a colon?

It is complex what I've explained in the mail with the example, that is
keeping track of every execution paths that can lead to a symbol being
leaked.

The fix is obviously trivial as was (on purpose) the example.  BTW I
don't think the number of characters of a fix tells much about it.

After that would come all the topic of read-time side effects but is a
different one and IMO [2] does a good job on that.

>> No, I'm thinking to one namespace only for symbols *not* for bindings.
>> This is not CL or Elisp either.
>
> I don't understand, I admit.  I thought this conversation was about
> you telling me why a reader-based system is a bad idea.

What can I say, I tried to give my opinions, but if you are not
satisfied with them I just apologize.  You have probably over estimated
me.

> (It seems Tom and Stefan have explained that to a point BTW,
> it's that internal Emacs features rely on certain names being
> interned in the main package. Not necessarily a big problem
> IMO, we can discuss that if you want.)
>
>> Thanks for telling me I could learn it.  I'm trying to say that I think
>> is unnecessary complex being too low level and that there could be
>> another way.  But okay if you prefer I just haven't learned how it
>> works.
>
> It's that you insist it should do A when you ask it to do B.
> It's like expecting printf("Hello World") to print "Goodbye".

_If_ you allow, I can still think would be more convenient doing A even
if Hyperspec says must do B.

But if you like examples:

You know petrol cars, there is no reason to have electric ones.  If you
read the manual is explained you need to put petrol into them.  We can
then just suggest any proponent of electric cars to just read the manual
of petrol ones to understand how they works.

> [1] https://zenodo.org/record/2648195
> [2] http://www.flownet.com/gat/locales.pdf
>
>
> Also, in [1]  in its section "Problems with CL packages", a very short
> section in a very short paper, starts off by admitting that it's
> "CL packages are a misunderstood part of CL" [1, p2]. Sounds
> familiar, indeed.

I must say this is one the points where your writing sounds is a bit
arrogant.  I'm possibly wrong but I've to say I don't enjoy this
approach and the resulting tone of the conversation.  I don't think it
serves your points.

> Then, the main criticism it levels against them is read-time
> side-effects, and points to [2] for justification. We check it out.
> [2] show us the unintentional interning of a symbols in the REPL
> caused an typo and an unusual way to :USE packages.  Forgets
> to mention a REPL is interactive, by definition and that most Lisps
> will contain restarts that explain the error and allow you to choose
> a solution and continue cleanly after the error. The author also
> presents the package version in an unnecessarily verbose way.
> Finally that author admits, at the end of the section [2, page 4]
> that "This is a matter of taste."  De Gustibus Non Est Disputandum.

Yes he's trying to show his point of view, that is the system can be
confusing non convenient and error prone, for reasons he's convinced are
valid and he's exposing.

Obviously trying to prove that he is doing on purpose errors, because
every programming language used with no errors just works.  But here
he's *not* trying to prove CL does not work.

> Not exactly categorical, i would say.
>
> So you could have just said "it's not my taste".

Papers just reflect the vision of the authors, this is how it works.
These are not papers describing scientific experiments.

Anyway I don't think we are progressing much so I suggest we just cope
with our disagreement.

Andrea

-- 
akrl@sdf.org

Re: Proper namespaces in Elisp

[João Távora]

On Wed, May 6, 2020 at 12:37 AM Andrea Corallo <akrl@sdf.org> wrote:

> _If_ you allow, I can still think would be more convenient doing
> A even if Hyperspec says must do B.

So you want to also change the meaning of QUOTE because
typing a colon is less convenient than typing a quote?
Is it that you have to press the shift key to type the colon?
I find that odd, but in that case, never use (defpackage foo :use),
you can still make pretty good use of CL packages.

> I must say this is one the points where your writing sounds is a bit
> arrogant.  I'm possibly wrong but I've to say I don't enjoy this
> approach and the resulting tone of the conversation.  I don't think it
> serves your points.

You dropped a paper casually, presumably to support your case,
didn't give any framing, just left it there. I could say I found that a
bit arrogant, too. So I went and actually read it and quoted it back
to you.

> > Not exactly categorical, i would say.
> > So you could have just said "it's not my taste".
> Papers just reflect the vision of the authors, this is how it works.
> These are not papers describing scientific experiments.

No, that's not how it works. I had a paper in the recent ELS (as
did you, I understand) and I had to justify every assertion,
particularly the ones concerning other people's work. So they are
not mere opinions.  There is an effort to justify positions, or when
can't, do like the author of [2] did and at least frame them as a
question of taste.

But I'm spending my time in this list looking for hard facts to
solve a real problem and I thought you had some to back up
your strong statement of opposition. I must have misunderstood.

João

Re: Proper namespaces in Elisp

[Vladimir Sedach]

João Távora <joaotavora@gmail.com> writes:
> Anyway, it sounds like you're describing dependencies failure. The
> author of XYZ should probably say that his package requires ABC
> 1.0.

This is not a dependency failure. It should be possible to
automatically pull in minor version upgrades that add functionality
in a backwards-compatible manner (the MINOR part of MAJOR.MINOR.PATCH
in Semantic Versioning). ABC 1.1 adding a new function does not break
backward compatibility.

> Or start paying attention to warnings.

There are no warnings. The developer of XYZ developed against ABC
1.0, which did not have any warnings. In Elisp there would be no
warnings period. And there would be no warnings if you assign
variables instead of defining functions.

> Or stop `:USE` of packages of systems whose code he doesn't
> control.

That is the current recommended practice in Common Lisp-land.

> This is a problem that happens in other languages.

I was out of the loop on R6RS, and spent the afternoon reading up on
the R6RS library system to see the Scheme take on the issue.
Obviously the R6RS committee recognized this problem, because it does
not happen in R6RS. Their solution was to make exported bindings
immutable. This is not going to work for Elisp.

In contrast, Guile's module system has the same behavior and problem
as Common Lisp.

> Also note: I don't know what you use to "pull in dependencies",
> but quicklisp in particular won't let this happen, since it works
> in distributions.

AFAIK MELPA does not.

The real reason Quicklisp seems to avoid this problem is because it
requires all libraries to build without any warnings, and redefining
a function causes a warning in some implementations. Re-assigning a
variable will not give any warnings and may not show up in unit
tests, so this problem is still possible.

That does not address the problem of an Elisp package that wants to
re-define/patch some function in another package on purpose. That
happens in Elisp a lot more often than it does in Common Lisp.

> So the things you think happen, really just don't happen IRL.

I don't know why you keep insisting this.

If you don't believe me, you can ask in
ircs://chat.freenode.net/#lisp

This is a very real problem that has to be addressed in any Elisp
namespace system. Importing all exported symbols seems like a useful
and harmless operation, but counter-intuitively it can lead to worse
namespace conflicts. You think you are safe to use short names
because you are in your own private namespace, but then you end up
unknowingly and unintentionally redefining something.

--
Vladimir Sedach
Software engineering services in Los Angeles https://oneofus.la

Re: Proper namespaces in Elisp

[Andrea Corallo]

João Távora <joaotavora@gmail.com> writes:

> On Wed, May 6, 2020 at 12:37 AM Andrea Corallo <akrl@sdf.org> wrote:
>
>> _If_ you allow, I can still think would be more convenient doing
>> A even if Hyperspec says must do B.
>
> So you want to also change the meaning of QUOTE because
> typing a colon is less convenient than typing a quote?

No, what I suggested is to split symbols from bindings because only
these seconds are the ones causing name clashes.  I believe only the
bindings should be partinioned in different spaces.  I think I wrote
this already, but I've been told repeatedly that this is not how CL
works.  Please let us not restart on this again beacause I do understand
this is not how CL works.

> Is it that you have to press the shift key to type the colon?
> I find that odd, but in that case, never use (defpackage foo :use),
> you can still make pretty good use of CL packages.
>
>> I must say this is one the points where your writing sounds is a bit
>> arrogant.  I'm possibly wrong but I've to say I don't enjoy this
>> approach and the resulting tone of the conversation.  I don't think it
>> serves your points.
>
> You dropped a paper casually, presumably to support your case,
> didn't give any framing, just left it there. I could say I found that a
> bit arrogant, too. So I went and actually read it and quoted it back
> to you.

I fail to see how signaling two papers that touch on this topic can be
arrogant.  [2] in particular is entirely dedicated on the subject of
working at the binding level, that is my suggestion since the beginning.

But I suggest instead you to look at your mails before mine citing the
papers, where you suggest multiple times that I do not understand how CL
works and even encourage me studing it.  Do you think assuming others
are ignorant and suggesting it is a constructive way of having a
discussion?  Do you think this increases the probability that the
conversation will be fruitful?

>> > Not exactly categorical, i would say.
>> > So you could have just said "it's not my taste".
>> Papers just reflect the vision of the authors, this is how it works.
>> These are not papers describing scientific experiments.
>
> No, that's not how it works. I had a paper in the recent ELS (as
> did you, I understand) and I had to justify every assertion,
> particularly the ones concerning other people's work. So they are
> not mere opinions.  There is an effort to justify positions, or when
> can't, do like the author of [2] did and at least frame them as a
> question of taste.

Paper goes through peer review to ensure thereir quality, is up to the
publisher to decide if enough justification is given for a document to
be published.  This by no means implies scientific method is applied to
them or that they are factual distilled truth.

This is especially true while touching subjects as this, where we are
debating conveniency and not if CL is working or not.  This was out of
discussion since the beginning.

> But I'm spending my time in this list looking for hard facts to
> solve a real problem and I thought you had some to back up
> your strong statement of opposition.

I suggested my arguments more then once, you are not expected to share
them (this is perfectly fine) but please do not state I haven't.

I think read-time side effects would have been a following interesting
topic to discuss afterwards but unfortunatelly we got stuck considerably
earlier.

  Andrea

-- 
akrl@sdf.org

Re: Proper namespaces in Elisp

[João Távora]

On Wed, May 6, 2020 at 4:38 AM Vladimir Sedach <vas@oneofus.la> wrote:

> This is not a dependency failure. It should be possible to
> automatically pull in minor version upgrades that add functionality

Let me start with a fact from another non-Lisp language. If in
C++ (where you people that have the the very same
misunderstanding BTW) you abuse the "using" directive correctly
you run into the the very same situation.

So this is a fundamental characteristic of the problem, not any
language. If you want to have names from a different namespace
X addressed without qualification along with other names of
namespace Y also without qualification, either you control X and Y
forever or you expose yourself to these problems.  By definition, if
either X or Y is in a library that you don't  control, then you don't
control them. So _don't_ merge namespaces unless you know
what you are doing. Like anything in programming. Or anywhere.

> There are no warnings. The developer of XYZ developed against ABC
> 1.0, which did not have any warnings.

Yes, and because she decided to use :USE for a package that is
outside her control, she is obligated to register somewhere that her
library only supports <=1.0.  It's very simple. It's not related to
CL at all.

> In Elisp there would be no warnings period.

How do you know this? Have you seen the future?

> And there would be no warnings if you assign
> variables instead of defining functions.

That's why you `defvar` before assigning, and any decent lisp
(including Elisp) will tell you at compile time if you don't.

> > Or stop `:USE` of packages of systems whose code he doesn't
> > control.
> That is the current recommended practice in Common Lisp-land.

Where, pray tell, is this Village of Dubious Recommendations? :-)

> not happen in R6RS. Their solution was to make exported bindings
> immutable. This is not going to work for Elisp.

So it errors when symbols clash? A possible strategy, I guess, but
not really a "solution" because the problem has no solution.
I asked you to show me the problems with CL packages. If you said
:USE is mandatory always, then I would agree with you immediately.
That would be a pretty useless package system, though.

It seems you want a namespacing system that _doesn't_ have
namespace merging. Fine. Let's get one with local nicknames
and you'll use only that.  But full merging is useful, too.  It's the
usual "with great power".

> The real reason Quicklisp seems to avoid this problem is because it
> requires all libraries to build without any warnings, and redefining
> a function causes a warning in some implementations. Re-assigning a
> variable will not give any warnings and may not show up in unit
> tests, so this problem is still possible.

Again, you can't assign a variable without having it defined
first, or you get a warning, so there's a contradiction there
somewhere.

> If you don't believe me, you can ask in
> ircs://chat.freenode.net/#lisp

What should I ask? "has anyone here ever misused the package
system? or written bad code"? Doesn't sound very interesting. And
if those people have something to add, they can do it here, no?

To wrap this up:

- I understand your :USE situation, thank you very much;
- Don't merge namespaces that you don't control;
- For development, the compiler warns on redefinition. You
  can choose not to load a compilation unit that warned.

João

Re: Proper namespaces in Elisp

[João Távora]

On Wed, May 6, 2020 at 8:07 AM Andrea Corallo <akrl@sdf.org> wrote:

> No, what I suggested is to split symbols from bindings because only
> these seconds are the ones causing name clashes.

AFAIK your argument goes like this:  CL packages are a bad
idea because, when you use them like this other system that is
not CL packages, they don't work like you want. This is circular.
It will infloop this thread.

This is why I encouraged you to study it, because you presented
an odd example that seemed to indicate you didn't know
understand how quoting works inside package.  I apologize.

You _could_ have said you that some other namespacing system
X is also _sufficient_ for the problems we're trying to solve right
now and then proceeded to justify that.  E.g. you could say X is
easier to implement, is faster at x-time (where x = run/compile/
read), has a less verbose interface, fries omelets, etc. That would
be a productive discussion, quite different from sidelining a valid
alternative with a circular argument based loosely on taste and
no demonstration of technical downsides.

João

Re: Proper namespaces in Elisp

[Vladimir Sedach]

João Távora <joaotavora@gmail.com> writes:

> On Wed, May 6, 2020 at 4:38 AM Vladimir Sedach <vas@oneofus.la> wrote:
> Let me start with a fact from another non-Lisp language. If in
> C++ (where you people that have the the very same
> misunderstanding BTW) you abuse the "using" directive correctly
> you run into the the very same situation.

"C++ does it wrong" is not an argument for anything. C++ does
whole-program static compilation, and users of C++ programs do not
have the need to load and re-define code at run-time, and do not
regularly pull updates for libraries from MELPA.

Since Scheme is also mostly used for whole-program static
compilation, the R6RS approach is appropriate there. Elisp is not
used that way.

> Yes, and because she decided to use :USE for a package that is
> outside her control, she is obligated to register somewhere that her
> library only supports <=1.0.  It's very simple. It's not related to
> CL at all.

Trying to solve this as a dependency management problem is not going
to work.

> Where, pray tell, is this Village of Dubious Recommendations? :-)

I first learned about this in ircs://chat.freenode.net/#lisp from the
many experienced Common Lisp programmers there.

> So it errors when symbols clash?

When they clash, and when you try to re-define an exported symbol:

--8<---------------cut here---------------start------------->8---
Chez Scheme Version 9.5
Copyright 1984-2017 Cisco Systems, Inc.

> (library (ABC) (export FOO) (import (rnrs)) (define FOO 111))
> (library (XYZ) (export) (import (rnrs) (ABC)) (define FOO 222))

Exception: multiple definitions for FOO in body (library (XYZ) (export) (import (rnrs) (ABC)) (define FOO 222))
Type (debug) to enter the debugger.
> (library (XYZ) (export) (import (rnrs) (ABC)))
> (import (ABC))
> (set! FOO 222)

Exception: attempt to assign immutable variable FOO
Type (debug) to enter the debugger.
>
--8<---------------cut here---------------end--------------->8---


> A possible strategy, I guess, but not really a "solution" because
> the problem has no solution.

There are many different solutions, some better than others.

> Fine. Let's get one with local nicknames and you'll use only that.

That is one possible solution I had not even thought of. Relevant
reading:

https://gist.github.com/phoe/2b63f33a2a4727a437403eceb7a6b4a3

In Andrea's scheme (which I am in favor of), one possible solution
would be assign a new binding in the local namespace when an imported
symbol is re-defined. I think this could work for dynamic scoping if
you use the binding instead of the symbol for the lookup, but I have
not thought the problem fully through yet. Then, when you really want
to re-define something in another package, you can use something like
the Guile @ and @@ module access special forms (@ is for exported
symbols, @@ for internal; see section 6.20.2 Using Guile Modules of
the Guile manual):

(set! (@ (GNUS ABC) FOO) 456)

There are two things to note from the above:

1. The Common Lisp PACKAGE:SYMBOL syntax is infix in disguise. As I
   noted previously, it also complicates printing and reading
   s-expressions. The Guile approach does not have this problem and
   does not need changes to the reader.

2. The Scheme convention of naming namespaces with a list of symbols
   is a good way to organize namespace, and avoids the complications
   of hierarchical namespace systems and the problem of trying to
   express structure in a string or symbol name.

The Scheme approach would be very inconvenient when in an IELM REPL
or *scratch* buffer (it is annoying enough in Scheme), and it is not
going to work for M-x execute-extended-command.

How namespacing is going to work with commands is another important
consideration that has not been brought up.

>> If you don't believe me, you can ask in
>> ircs://chat.freenode.net/#lisp
>
> What should I ask? "has anyone here ever misused the package
> system? or written bad code"? Doesn't sound very interesting. And
> if those people have something to add, they can do it here, no?

You can ask other experienced Common Lisp programmers about defining
packages and the pitfalls of :use. A lot of people there have strong
opinions and useful recommendations.

--
Vladimir Sedach
Software engineering services in Los Angeles https://oneofus.la

Re: Proper namespaces in Elisp

[Tom Tromey]

>>>>> "João" == João Távora <joaotavora@gmail.com> writes:

>> https://github.com/tromey/emacs-module

João> So you're saying one just types "s-format", but the actual symbol
João> that elisp sees is modern-string-format?

Yes.  The test program there shows how it is used.

From testmodule.el:

    (import-module testm2 :symbols (zzzq))
    (defun somefunction ()
       [...stuff...]
         (zzzq))

From testm2.el:

    (define-module testm2 :export zzzq)
    (defun zzzq ()
      23)

Here, the actual symbol names are "testmodule-somefunction" and
"testm2-zzzq".

The only things that are renamed are either (1) symbols you import, and
(2) things in your .el file that (after macro expansion) are a defvar,
defconst, defalias, defvaralias, or fset (this handles defuns).

João> Are both symbols interned?  Only for vars&functions? I could live
João> with that.

Not sure I totally understand, but the renaming happens after macro
expansion, so everything is interned.  How exactly the symbol is used is
up to you -- basically the renamer is intentionally simplistic, it just
blindly renames all symbols that were imported.

I guess it is a little too simplistic since looking at
module--rewrite-form, it seems I didn't make it recurse into vectors.
Maybe it would struggle with other things, too, like if macro expansion
resulted in a string with properties applied.

João> It seems similar my "shorthand" idea
João> (https://lists.gnu.org/archive/html/emacs-devel/2020-05/msg00617.html)
João> but that idea would fall into the pitfall of:
João>     "the Emacs core sometimes requires certain symbol names in APIs."
João> unless, that is, you added syntax to escape renaming in _those_
João> situations, like `::`

I like the idea of doing it in the reader (it has some nice advantages),
but the downside is it is then harder to special-case defining forms.

Also, in my approach the issue of special symbols that are important to
the elisp core doesn't really arise, because only explicit imports (and
exports) are renamed.  This is essentially the same problem.

Probably functions taking a special symbol should be updated to also
take a special keyword.  That would eliminate this part of the problem
entirely.

João> I think I could fix the thing-at-point thing reasonably easily.

There's also whatever M-. uses under the hood when in a .el file.
Maybe that's thing-at-point, I haven't looked.


I'm not totally sold on my module approach, but I'm happy to assign it
to the FSF if that's useful.  Emacs could include it or someone could
just lift whatever code they like from it.

One thing I've occasionally wished for in a module system is the ability
to load two different versions of a library simultaneously.  That's not
readily done in this system.  Modules aren't really first class -- the
implementation just reuses the global obarray and the full symbol names
are baked into the .elc.  I tend to think Emacs could do better with
something in the core than what I implemented.

thanks,
Tom

Re: Proper namespaces in Elisp

[Stefan Monnier]

> I like the idea of doing it in the reader (it has some nice advantages),
> but the downside is it is then harder to special-case defining forms.

The upside is that you don't need to recognize defining forms ;-)


        Stefan

Re: Proper namespaces in Elisp

[Andrea Corallo]

João Távora <joaotavora@gmail.com> writes:

> On Wed, May 6, 2020 at 8:07 AM Andrea Corallo <akrl@sdf.org> wrote:
>
>> No, what I suggested is to split symbols from bindings because only
>> these seconds are the ones causing name clashes.
>
> AFAIK your argument goes like this:  CL packages are a bad
> idea because, when you use them like this other system that is
> not CL packages, they don't work like you want. This is circular.
> It will infloop this thread.

No my argument is: I think behavior B (the CL one) is not optimal
because X, I'd like more behavior A.

I don't see any circularity into that from my side.

> This is why I encouraged you to study it, because you presented
> an odd example that seemed to indicate you didn't know
> understand how quoting works inside package.  I apologize.

Thanks.

> You _could_ have said you that some other namespacing system
> X is also _sufficient_ for the problems we're trying to solve right
> now and then proceeded to justify that.  E.g. you could say X is
> easier to implement, is faster at x-time (where x = run/compile/
> read), has a less verbose interface, fries omelets, etc.

I think that's what I did decribing behavior A.

I may come-up one day with a prototype to play and discuss with if I
find the time.  I guess would be more interesting and effective in
highlighting pros or flaws of this.

  Andrea

-- 
akrl@sdf.org

Re: Proper namespaces in Elisp

[João Távora]

On Thu, May 7, 2020 at 1:52 AM Vladimir Sedach <vas@oneofus.la> wrote:
> "C++ does it wrong" is not an argument for anything. C++ does
> whole-program static compilation, and users of C++ programs do not
> have the need to load and re-define code at run-time, and do not
> regularly pull updates for libraries from MELPA.

But I never said C++ does it wrong, did I? You're saying it.
And I must say I don't understand this common trope about "Language
X does it wrong".  You can google for "is using C++ good or bad" and
you will see the same discussion" (and misunderstandings), complete
with examples about upgrading libraries and such.

Interesting point tho, about C++ programs not using libraries from
MELPA. :-)

> Trying to solve this as a dependency management problem is not going
> to work.

It's you who presented a dependency problem, remember? First
you talked about packages, then you moved on to talking about
libraries, and upgrading some dependency ABC to v1.1. You're
inverting the problem. Yes, packages when used wrong, can make
dependency management suck. But so can any other programming
construct, right?

> > Where, pray tell, is this Village of Dubious Recommendations? :-)
>
> I first learned about this in ircs://chat.freenode.net/#lisp from the
> many experienced Common Lisp programmers there.

Let me get this straight: people there recommended you
should use :USE for library packages you depend on?

> There are many different solutions, some better than others.

No there aren't, short of _not_ providing namespace merging.
There's no point in explaining this again, I think.

> You can ask other experienced Common Lisp programmers about defining
> packages and the pitfalls of :use. A lot of people there have strong
> opinions and useful recommendations.

I've worked for 15 years day to day with experienced Common Lisp
programmers. You can add my (and I think their) recommendation to
the list: Don't use :USE for packages outside your control, else feel
free to use it. Think of :USE as :MERGE.

João

Re: Proper namespaces in Elisp

[Tom Tromey]

>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I like the idea of doing it in the reader (it has some nice advantages),
>> but the downside is it is then harder to special-case defining forms.

Stefan> The upside is that you don't need to recognize defining forms ;-)

Haha, yeah, I suppose so.

The advantages I was thinking of are that it means that it's basically
impossible to forget a case (like I did with vectors), and that it means
the local "short" form of the symbol doesn't need to be interned.

Tom

Re: Proper namespaces in Elisp

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1912 bytes --]

On Thu, May 7, 2020 at 2:02 PM Tom Tromey <tom@tromey.com> wrote:
>
> >>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
> >> I like the idea of doing it in the reader (it has some nice
advantages),
> >> but the downside is it is then harder to special-case defining forms.
>
> Stefan> The upside is that you don't need to recognize defining forms ;-)
>
> Haha, yeah, I suppose so.
>
> The advantages I was thinking of are that it means that it's basically
> impossible to forget a case (like I did with vectors), and that it means
> the local "short" form of the symbol doesn't need to be interned.

Exactly. And that answers my question from before: it seems your current
approach _does_ intern both the short and long form (not a terribly bad
thing, mind you).

So Tom if we combine your "shorthand" approach with a reader-based one
we get a straightfoward, hopefully useful, system.  We could have this
directive in some files:

;;; Shorthand: (("vl-" . "very-long-prefix-")
;;;             ("yalp-" . "yet-another-long-prefix-"))

which would influence the reader.  That way we could have our
very-long-prefix.el library do this:

;; this defines very-long-prefix-foo which calls
;; yet-another-long-prefix-bar.  vl-foo and yalp-bar
;; are never interned.
(defun vl-foo () (substring (yalp-bar) 42 84))

What if later on someone adds an actual symbol vl-foo to the Emacs main
namespace?  Well, I wouldn't be able to see it in my file, but maybe I
don't need to.  If I do need to see it I use some special syntax to tell
the reader to escape the ;; Shorthand: directive, like #vl-foo, which
does mean the symbol named "vl-foo".

What about the printer?  We could decide either way.  I wouldn't
mind seeing my automatically PRINTed code in the long form, but
we could have it be influenced by the shorthand alist and do a
rassoc there.

João

[-- Attachment #2: Type: text/html, Size: 2425 bytes --]

Re: Proper namespaces in Elisp

[Stefan Monnier]

> ;;; Shorthand: (("vl-" . "very-long-prefix-")
> ;;;             ("yalp-" . "yet-another-long-prefix-"))

Adding such "local, read-time prefix remapping" does sound to me like
the best approach in the context of Elisp, indeed.

It avoids the problem of non-prefixed symbols being implicitly prefixed
with the name of the current package (which introduces the problem of
unintended addition of that prefix to symbols which were really meant to
stay without prefix), it should interact fairly well with existing
Elisp code, it doesn't need to know what's a definition or a use or
anything else of the sort, and it's hopefully simple and regular enough
that non-expert users wouldn't be endlessly confused by corner cases.


        Stefan

Re: Proper namespaces in Elisp

[Vladimir Sedach]

João Távora <joaotavora@gmail.com> writes:
> It's you who presented a dependency problem, remember?

This is a namespace problem that shows up in, and breaks, dependency
management. Trying to solve it at the step of dependency management
is too late.

> Let me get this straight: people there recommended you
> should use :USE for library packages you depend on?

They recommend the opposite, because a lot of Common Lisp libraries
do in fact :USE all of their dependencies, and this has resulted in
real problems. This was not a widely recognized problem 10 years ago.
I would appreciate it if you stop replying to my suggestion with
sarcastic misinterpretations, and discuss the issue with other
programmers in #lisp

> No there aren't, short of _not_ providing namespace merging.
> There's no point in explaining this again, I think.

I don't know why you are claiming this, considering that you
previously suggested an excellent solution.

If you look at what a best-practices Common Lisp package definition
with package-local nicknames would look like, it would :USE only the
"COMMON-LISP" package, and references other packages with local
nicknames.

Restricting the :USE list to 0 or 1 packages, and references to
symbols in other packages to package local nicknames, would be a
radical simplification of the Common Lisp package system without loss
of generality, and would solve many problems. In addition to avoiding
the unintended re-definition problem, this avoids the problem of
clashes in exported symbol names between packages - a problem which
R6RS also suffers from, but which is far less likely to occur than
unintended re-definition. You no longer need any of the import,
shadow, global nicknames, or conflict resolution facilities.

So you see, by engaging in constructive exploration of the design
space, you helped to make what I think is a major contribution to
namespace system design and to Common Lisp. I am going to start using
this pattern in my own code, and I hope any future Lisp that wants to
adopt a Common Lisp-style package system uses this modified design.

Unfortunately, this still relies on the reader to resolve namespaces,
so might not work for Elisp.

Which brings me to an important realization that is obvious in
hindsight:

In fact, most Common Lisp implementations already follow the R6RS
solution to the unintended re-definition problem by special-casing
the COMMON-LISP package to be locked, exactly to address the
unintended re-definition problem. And the COMMON-LISP package has not
changed since 1994.

This is not the case for Elisp.

The unintended re-definition problem is not a problem only for
third-party packages as I had previously thought. If left unaddressed
this is going to be a major problem for all Elisp code. You think you
would be safe defining a short name for something in your local
namespace, then a new function with the same name is added to Elisp
in the next Emacs release, and now you have broken
backwards-compatibility with what should be a backwards-compatible
change. This is a lot worse than the current status quo of not having
a namespace system.

One way to address this issue is to version Elisp namespaces to major
Emacs versions: elisp29, elisp30, etc. This would enhance both
backward compatibility (you can fix problematic behavior in Elisp
functions without breaking packages that depend on that behavior;
package maintainers would have more time to make changes for new
Emacs versions), and ongoing Emacs development (besides fixing
problematic functions, aliases and deprecated functions can be
dropped from the current version namespace so that new Elisp user
code does not unintentionally continue to use them, and moved to
separate files to reduce clutter).

I think this is a much better reason for having a namespace system
than "I don't want to use autocomplete to type long symbol names."

> I've worked for 15 years day to day with experienced Common Lisp
> programmers. You can add my (and I think their) recommendation to
> the list: Don't use :USE for packages outside your control, else feel
> free to use it. Think of :USE as :MERGE.

I am not here to tell Common Lisp programmers what to do. What I
would like is for any Elisp namespacing system to avoid known
problems. At the very least, if whole-namespace importing is
provided, it should come with a very loud disclaimer not to use it
with third-party packages. But that would mean the facility would be
very seldom used in good Elisp code, and abused in bad Elisp code (of
which there seems to be no shortage written every day), so why bother
having it and its many downsides when better alternatives are
available?

--
Vladimir Sedach
Software engineering services in Los Angeles https://oneofus.la

Re: Proper namespaces in Elisp

[Vladimir Sedach]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> ;;; Shorthand: (("vl-" . "very-long-prefix-")
>> ;;;             ("yalp-" . "yet-another-long-prefix-"))
>
> Adding such "local, read-time prefix remapping" does sound to me like
> the best approach in the context of Elisp, indeed.

You can do the same thing without breaking grep/TAGS/etc or changing the
reader with overlays and a modified abbrev facility.

--
Vladimir Sedach
Software engineering services in Los Angeles https://oneofus.la

Re: Proper namespaces in Elisp

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 2671 bytes --]

On Thu, May 7, 2020 at 7:41 PM Vladimir Sedach <vas@oneofus.la> wrote:

> I would appreciate it if you stop replying to my suggestion with
> sarcastic misinterpretations, and discuss the issue with other
> programmers in #lisp

I was not being sarcastic now. I didn't understand  what the
recommendation was.  From reading our exchange I thought
it was the exact opposite.

> > No there aren't, short of _not_ providing namespace merging.
> > There's no point in explaining this again, I think.
> I don't know why you are claiming this, considering that you
> previously suggested an excellent solution.

Because but it's not a "solution" to the problem.  A solution
to the potential would make sure it never happens. The only
way to do that is to make sure is to never merge.  But OK,
this is semantics: you call "good practices" a solution, that's
reasonable.

> So you see, by engaging in constructive exploration of the design
> space, you helped to make what I think is a major contribution to
> namespace system design and to Common Lisp.

Hmmm, i'd wait a bit before setting doign the "major contribution"
victory dance :-)

> I am going to start using
> this pattern in my own code, and I hope any future Lisp that wants to
> adopt a Common Lisp-style package system uses this modified design.

No nicknames, no :USE or little on. Seem decent. One fine day
you'll see a use for :USE, I'm sure.

> And the COMMON-LISP package has not changed since 1994.
> This is not the case for Elisp.

That's true.

> One way to address this issue is to version Elisp namespaces to major
> Emacs versions: elisp29, elisp30, etc.

That's a clever solution, indeed.

> I think this is a much better reason for having a namespace system
> than "I don't want to use autocomplete to type long symbol names."

Whatever works for you. But I have my reasons not to want to
want a prefix repeated ad aeternum (and do use autocomplete).

> which there seems to be no shortage written every day), so why bother
> having it and its many downsides when better alternatives are
> available?

Larger programs have many packages. :USE really is useful
for internal packages, for example.  One common example is the
FOO-TEST that has the tests for the FOO package. Or when the
FOO  package uses the FOO-UTILS package which has internal
and external stuff.  Those are two examples from the top of my
head where I most heartily recommend :USE.

But there are more, when one realizes packages are collections
of symbols, not necessarily definitions.

Also last time I checked package-local nicknames weren't widely
available in impls, so I don't use them.  Maybe that has changed
in the meantime?

[-- Attachment #2: Type: text/html, Size: 3899 bytes --]

Re: Proper namespaces in Elisp

[Daniel Colascione]

On 5/6/20 8:12 PM, Stefan Monnier wrote:
>> I like the idea of doing it in the reader (it has some nice advantages),
>> but the downside is it is then harder to special-case defining forms.
> 
> The upside is that you don't need to recognize defining forms ;-)

I don't like reader magic. It complicates code logic. I know we talked 
about symbol-macrolet and macrolet as possible options and know why 
neither is quite the right thing, but I still think that a macro of the 
same general kind would work fine. Macros don't need to recognize 
defining forms if they macroexpand their bodies before walking them.

Re: Proper namespaces in Elisp

[Stefan Monnier]

> I don't like reader magic. It complicates code logic. I know we talked about
> symbol-macrolet and macrolet as possible options and know why neither is
> quite the right thing, but I still think that a macro of the same general
> kind would work fine. Macros don't need to recognize defining forms if they
> macroexpand their bodies before walking them.

You need not only discover the defining forms but also the "using"
forms.  So you need to know that the arguments to `make-local-variable`,
`add-hook`, `symbol-value`, `facep`, ... (and all function that pass
their arg to one of those) are not just random symbols (that you should
leave untouched) but symbols that obey the namespace rules.

In Common Lisp, the distinction is made by using keywords instead of
normal symbols when the namespace rules shouldn't apply, but Elisp
doesn't have such a tradition.


        Stefan

Re: Proper namespaces in Elisp

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1473 bytes --]

On Thu, May 7, 2020 at 7:48 PM Vladimir Sedach <vas@oneofus.la> wrote:

>
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
> >> ;;; Shorthand: (("vl-" . "very-long-prefix-")
> >> ;;;             ("yalp-" . "yet-another-long-prefix-"))
> >
> > Adding such "local, read-time prefix remapping" does sound to me like
> > the best approach in the context of Elisp, indeed.
>
> You can do the same thing without breaking grep/TAGS/etc or changing the
> reader with overlays and a modified abbrev facility.


All namespacing in all languages systems "break" grep if
you give grep a volatile namespace qualifier.  This just comes
with the namespacing.

And in your solution the problem then becomes teaches isearch,
query-replace and its various variants, a (probably very large)
number of tools in the wild about this these overlays. You see
"yalp-" then C-s yalp- and find nothing!.  Another medium-size
annoyance would be alignment and keeping to column limits,
like the ones Emacs has.

But most crucially: All this started because there are a zillion
packages in the wild using s.el.  They all invoke its functions
with "s-foo". You would  ALL need to be massively and
intrusively changed and the authors convinced to use your system
instead. With my system we can mostly (if not entirely) keep their
code untouched, so these files that (require 's) will not know they
are actually working with modern-string.el instead of s.el.

João

[-- Attachment #2: Type: text/html, Size: 2290 bytes --]

Re: Proper namespaces in Elisp

[Daniel Colascione]

On 5/7/20 1:28 PM, Stefan Monnier wrote:
>> I don't like reader magic. It complicates code logic. I know we talked about
>> symbol-macrolet and macrolet as possible options and know why neither is
>> quite the right thing, but I still think that a macro of the same general
>> kind would work fine. Macros don't need to recognize defining forms if they
>> macroexpand their bodies before walking them.
> 
> You need not only discover the defining forms but also the "using"
> forms.  So you need to know that the arguments to `make-local-variable`,
> `add-hook`, `symbol-value`, `facep`, ... (and all function that pass
> their arg to one of those) are not just random symbols (that you should
> leave untouched) but symbols that obey the namespace rules.

I think these situations are rare enough that we can ask people to pass 
fully-qualified symbols.

> In Common Lisp, the distinction is made by using keywords instead of
> normal symbols when the namespace rules shouldn't apply, but Elisp
> doesn't have such a tradition.

Common Lisp does deal with this problem in some cases, e.g., in LOOP. We 
should be moving forwards keywords anyway.

FWIW, I think elisp should just literally copy Common Lisp's approach. 
It's tried and tested, and the pitfalls are well-understood. It'll take 
a while to work the kinks out of a bespoke approach, and the CL approach 
seems adequate for addressing the problem that people are trying to solve.

Re: Proper namespaces in Elisp

[Stefan Monnier]

>> You need not only discover the defining forms but also the "using"
>> forms.  So you need to know that the arguments to `make-local-variable`,
>> `add-hook`, `symbol-value`, `facep`, ... (and all function that pass
>> their arg to one of those) are not just random symbols (that you should
>> leave untouched) but symbols that obey the namespace rules.
> I think these situations are rare enough that we can ask people to pass
> fully-qualified symbols.

Experts would have no trouble dealing with that, indeed, but it would
introduce extra complexity which will negatively affect the average
user, I feel.

> FWIW, I think elisp should just literally copy Common Lisp's approach.

I thought your previous message was saying you don't like this approach
(when you said "I don't like reader magic").  Is it that I misunderstood
or that you don't much like the solution but think that from a pragmatic
point of view it's still the better option?


        Stefan

Re: Proper namespaces in Elisp

[Daniel Colascione]

On 5/7/20 2:06 PM, Stefan Monnier wrote:
>>> You need not only discover the defining forms but also the "using"
>>> forms.  So you need to know that the arguments to `make-local-variable`,
>>> `add-hook`, `symbol-value`, `facep`, ... (and all function that pass
>>> their arg to one of those) are not just random symbols (that you should
>>> leave untouched) but symbols that obey the namespace rules.
>> I think these situations are rare enough that we can ask people to pass
>> fully-qualified symbols.
> 
> Experts would have no trouble dealing with that, indeed, but it would
> introduce extra complexity which will negatively affect the average
> user, I feel.
> 
>> FWIW, I think elisp should just literally copy Common Lisp's approach.
> 
> I thought your previous message was saying you don't like this approach
> (when you said "I don't like reader magic").  Is it that I misunderstood
> or that you don't much like the solution but think that from a pragmatic
> point of view it's still the better option?

I don't like reader magic in general, but I don't think of the CL 
approach as being all that magical: it has uniform rules and a long 
history. CL namespaces *are* reader magic all right, but it's reader 
magic with which a lot of people are familiar and that has proven over a 
few decades now to be workable. I think the CL solution is the most 
pragmatic one. I'm much more opposed to ad-hoc, Emacs-specific reader magic.

Re: Proper namespaces in Elisp

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 2371 bytes --]

On Thu, May 7, 2020 at 10:10 PM Daniel Colascione <dancol@dancol.org> wrote:

> On 5/7/20 2:06 PM, Stefan Monnier wrote:
>
> > I thought your previous message was saying you don't like this approach
> > (when you said "I don't like reader magic").  Is it that I misunderstood
> > or that you don't much like the solution but think that from a pragmatic
> > point of view it's still the better option?
>
> I don't like reader magic in general, but I don't think of the CL
> approach as being all that magical: it has uniform rules and a long
> history. CL namespaces *are* reader magic all right, but it's reader
> magic with which a lot of people are familiar
>

Not only that, but CL the reader itself is programmable. So it's really
_not_ "magic", it's all (hyper)spec'ed!

http://www.lispworks.com/documentation/HyperSpec/Body/23_a.htm

However, Daniel, my proposal doesn't have much magic.  Is is really
dumb ;-).  Here's the gist of it (a pure elisp solution using advice,
very lightly tested)

(require 'cl-lib)

(eval-when-compile
  (advice-add 'read :around #'read-aware-of-shorthands)
  (defvar shorthand-shorthands '(("^vlp-" . "very-long-prefix-"))))

(defun shorthand-expand (form)
  (cond ((consp form)
         (setcar form (shorthand-expand (car form)))
         (setcdr form (shorthand-expand (cdr form))))
        ((arrayp form)
         (cl-loop for i from 0 for e across form
                  do (aset form i (shorthand-expand e))))
        ((symbolp form)
         (let ((name (symbol-name form)))
           (cl-loop for (short-pat . long-pat) in shorthand-shorthands
                    do (setq name
                             (replace-regexp-in-string short-pat long-pat
                                                       name))
                    finally (setq form (intern name))))))
  form)


(defun read-aware-of-shorthands (oldread &rest stuff)
  (let ((form (let ((obarray (obarray-make)))
                (apply oldread stuff))))
    (shorthand-expand form)))

(defun vlp-foo () 42) ;; => very-long-prefix-foo
(vlp-foo) ;; => 42

(defun vlp-aref1 (array) (aref array 1))
(vlp-aref1 [2 vlp-shiiz 4]) ;; => very-long-prefix-shiiz

(let ((shorthand-shorthands nil))
  ;; pretend this was some other file
  (eval (car (read-from-string "(vlp-foo)"))) ;; => errors
  (eval (car (read-from-string "(very-long-prefix-foo)"))) ;; => 42
  )

[-- Attachment #2: Type: text/html, Size: 3319 bytes --]

Re: Proper namespaces in Elisp

[Daniel Colascione]

On 5/7/20 2:46 PM, João Távora wrote:
> On Thu, May 7, 2020 at 10:10 PM Daniel Colascione <dancol@dancol.org 
> <mailto:dancol@dancol.org>> wrote:
> 
>     On 5/7/20 2:06 PM, Stefan Monnier wrote:
> 
>      > I thought your previous message was saying you don't like this
>     approach
>      > (when you said "I don't like reader magic").  Is it that I
>     misunderstood
>      > or that you don't much like the solution but think that from a
>     pragmatic
>      > point of view it's still the better option?
> 
>     I don't like reader magic in general, but I don't think of the CL
>     approach as being all that magical: it has uniform rules and a long
>     history. CL namespaces *are* reader magic all right, but it's reader
>     magic with which a lot of people are familiar
> 
> 
> Not only that, but CL the reader itself is programmable. So it's really
> _not_ "magic", it's all (hyper)spec'ed!

Sure. The subject of reader macros has come up before; I used to be in 
favor. Regardless, symbol namespaces don't require general reader macros.

> http://www.lispworks.com/documentation/HyperSpec/Body/23_a.htm
> 
> However, Daniel, my proposal doesn't have much magic.  Is is really
> dumb ;-).  Here's the gist of it (a pure elisp solution using advice,
> very lightly tested)

It's not that your solution is bad per se --- it's that it's a bespoke 
solution that solves fewer problems than the general CL mechanism. Sure, 
the implementation is simple, but the CL approach isn't really that hard 
to implement either if we decide to do it.

We should prefer familiar solutions to unfamiliar ones unless the new 
solution comes with some compelling advantage that compensates for its 
novelty, and I don't see the scheme you've proposed having enough of an 
advantage of the CL approach (which also allows prefix aliases) to 
compensate for being novel.

Re: Proper namespaces in Elisp

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1488 bytes --]

On Thu, May 7, 2020 at 10:56 PM Daniel Colascione <dancol@dancol.org> wrote:

>
> It's not that your solution is bad per se --- it's that it's a bespoke
> solution that solves fewer problems than the general CL mechanism. Sure,
> the implementation is simple, but the CL approach isn't really that hard
> to implement either if we decide to do it.
>

Believe me, Daniel, when I tell you it's not me you need to convince ;-)

We should prefer familiar solutions to unfamiliar ones unless the new
> solution comes with some compelling advantage that compensates for its
> novelty, and I don't see the scheme you've proposed having enough of an
> advantage of the CL approach (which also allows prefix aliases) to
> compensate for being novel.
>

I couldn't agree more.  Really, I agree 100%.

Anyway, if you want to play around with my shorthand.el thing, here's
a dirty hack that fixes eldoc.

(defun shorthand-probe (w)
  (cl-loop
   for (short-pat . long-pat) in shorthand-shorthands
   thereis (intern-soft
            (replace-regexp-in-string short-pat long-pat w))))

(defun elisp--current-symbol ()
  (let ((c (char-after (point))))
    (and c
         (memq (char-syntax c) '(?w ?_))
         (let ((w (current-word)))
           (or (intern-soft w)
               (shorthand-probe w))))))

Stefan, can we make elisp--current-symbol use thingatpt.el so that
I only have to do this trick once (since I'll have to do it for M-. , too)

João

[-- Attachment #2: Type: text/html, Size: 2243 bytes --]

Re: Proper namespaces in Elisp

[Vladimir Sedach]

João Távora <joaotavora@gmail.com> writes:
> All namespacing in all languages systems "break" grep if
> you give grep a volatile namespace qualifier.  This just comes
> with the namespacing.

That is not true. A reference like NAMESPACE:SYMBOL makes it obvious
that you should search for SYMBOL. This is why GNU etags works for
Common Lisp, without understanding Common Lisp packages. Because an
explicit namespace separator like : is in most cases unambiguous and
easy to parse, it is also possible to write tools that understand it.
The proposed system has arbitrary prefixes. How do you know what to
search for?

> And in your solution the problem then becomes teaches isearch,
> query-replace and its various variants, a (probably very large)
> number of tools in the wild about this these overlays. You see
> "yalp-" then C-s yalp- and find nothing!.  Another medium-size
> annoyance would be alignment and keeping to column limits,
> like the ones Emacs has.

That is true. I am not advocating for overlays/abbrev. I am pointing
out that it gets you a lot of the same benefits as the proposed
system, while being far less intrusive. As I pointed out in the other
sub-thread, a namespace system that does the wrong thing is a lot
worse than the current status quo of not having a namespace system at
all. Why not come up with something that works well, instead of
relying on magic comment reader hacks? You could get a lot of
auxiliary benefits out of a good solution as well (for example, real
namespace declarations such as CL defpackage/R6RS library will let
you generate autoloads automatically for exported symbols).

A question related to isearch: how would the proposed system work
with apropos?

> But most crucially: All this started because there are a zillion
> packages in the wild using s.el.  They all invoke its functions
> with "s-foo". You would  ALL need to be massively and
> intrusively changed and the authors convinced to use your system
> instead. With my system we can mostly (if not entirely) keep their
> code untouched, so these files that (require 's) will not know they
> are actually working with modern-string.el instead of s.el.

I am sorry, I don't follow. The proposal is for a file-local setting
that requires magic comments. Why wouldn't package authors be
required to change their code to use this? Or are you saying that
this:

;;; Shorthand: (("vl-" . "very-long-prefix-")
;;;             ("yalp-" . "yet-another-long-prefix-"))

Goes in the file defining the namespace, and everything read after
that turns yalp-foo into yet-another-long-prefix-foo? Wouldn't that
affect completely unrelated code that happens to mention yalp-bar
somewhere (like throw tags)?

--
Vladimir Sedach
Software engineering services in Los Angeles https://oneofus.la

Re: Proper namespaces in Elisp

[João Távora]

On Fri, May 8, 2020 at 4:10 AM Vladimir Sedach <vas@oneofus.la> wrote:
> João Távora <joaotavora@gmail.com> writes:
> > All namespacing in all languages systems "break" grep if
> > you give grep a volatile namespace qualifier.  This just comes
> > with the namespacing.
> That is not true. A reference like NAMESPACE:SYMBOL makes it obvious
> that you should search for SYMBOL. This is why GNU etags works

I was talking about grep. And if you teach etags about ":" you can
teach it about shorthands, too.   Again this comes with the problem,
I don't think it's even worth discussing.

It's like hunger. The daily problem "hunger" is not solved by "eating".
It's solved by dying, I presume.  That sucks, unless you're a zombie
in which case you're still hungry.

> That is true. I am not advocating for overlays/abbrev. I am pointing
> out that it gets you a lot of the same benefits as the proposed
> system, while being far less intrusive.

Heh, sounds a lot like advocacy to me. Anyway, it's hideous, IMHO.
Even if you think that's just taste (it might be), the main reason
I'm not interested in discussing it is that it doesn't solve the s.el, f.el,
dash.el situation at all, because it's not a namespacing system.

> A question related to isearch: how would the proposed system work
> with apropos?

The actual symbol is always the longhand version.

> I am sorry, I don't follow. The proposal is for a file-local setting
> that requires magic comments. Why wouldn't package authors be
> required to change their code

We can try to guess notable cases like s.el if they use the cookie
;; Package-Requires or even by watching the "require" calls.
Anyway, even if we don't do that, adding a couple of lines
in a backward-compatible way is infinitely better than renaming
a zillion calls.

Bye,
João

Re: Proper namespaces in Elisp

[Vladimir Sedach]

João Távora <joaotavora@gmail.com> writes:
> I was talking about grep.

With grep you still know to search for SYMBOL because you know where
NAMESPACE ends and SYMBOL begins when you delimited namespaces
explicitly, like NAMESPACE:SYMBOL

> Again this comes with the problem, I don't think it's even worth
> discussing.

It affects debugging Elisp code, so it is very much worth discussing.

>> A question related to isearch: how would the proposed system work
>> with apropos?
>
> The actual symbol is always the longhand version.

So apropos would do the prefix abbreviation to symbol name
translation?

>> I am sorry, I don't follow. The proposal is for a file-local setting
>> that requires magic comments. Why wouldn't package authors be
>> required to change their code
>
> We can try to guess notable cases like s.el if they use the cookie
> ;; Package-Requires or even by watching the "require" calls.

Let me ask this a different way:

Who determines what prefix abbreviation to use for a given package?
Does the library author declare that "jennys-awesome-string-library"
has a prefix abbreviation of "jasl-" that everyone is obligated to
use, or does each user of the library declare the prefix abbreviation
when they require the library?

What delimits the scope of when the prefix abbreviation is active in
the reader?

Can you be more specific about the algorithm for "try to guess
notable cases"?

Why is the prefix abbreviation declared with magic comments (bad),
but the use of the prefix abbreviation with a require s-expression
(good)?

There are other ways to load code other than require. How does this
proposal work load/eval/etc.?

--
Vladimir Sedach
Software engineering services in Los Angeles https://oneofus.la

Re: Proper namespaces in Elisp

[João Távora]

On Fri, May 8, 2020 at 7:12 PM Vladimir Sedach <vas@oneofus.la> wrote:

> > Again this comes with the problem, I don't think it's even worth
> > discussing.
> It affects debugging Elisp code, so it is very much worth discussing.

OK, I'll rephrase.  It is my opinion that it's not worthwhile discussing it
with you, given you and I are not talking of the same thing.  I hold
this opinion. But you can discuss it with whomever you see fit.

> use, or does each user of the library declare the prefix abbreviation
> when they require the library?

The latter, including the library itself.

I've shown a proof of concept here and improved it since.
You can look it up in my Github account, get answers to your
questions from  the code there, open issues, or help me solve
problems. There are many other namespacing systems out
there, expressed in actual code, I encourage you to contribute
to one, or create one from scratch.

João

Re: Proper namespaces in Elisp

[Vladimir Sedach]

Daniel Colascione <dancol@dancol.org> writes:
> Common Lisp does deal with this problem in some cases, e.g., in LOOP.
> We should be moving forwards keywords anyway.

LOOP would not even have to deal with that if namespaces were over
bindings, like Andrea suggested, and what Scheme module systems do.

> FWIW, I think elisp should just literally copy Common Lisp's
> approach.

Please see this other sub-thread where I explain why this is a really
bad idea:

https://lists.gnu.org/archive/html/emacs-devel/2020-05/msg00623.html

> It's tried and tested, and the pitfalls are well-understood.

Well-understood enough where people want to throw away most of how it
does namespace imports, and have a backward-compatible replacement
ready this year:

https://gist.github.com/phoe/2b63f33a2a4727a437403eceb7a6b4a3

There is no need to repeat the same mistakes. Andrea's proposal for
namespacing over bindings/Scheme's approach avoids mangling the
reader, does not cause problems for the printer, and does not make
code resort to symbol-name hacks like LOOP does. Scheme's approach to
namespacing has a lot of advantages over Common Lisp's.

We can also avoid the "namespace multiple inheritance" problems of
unintended re-definition and import conflicts if we combine João's
approach of declaring prefixes with the ideas of package-local
nicknames and namespaces over bindings.

Here is how that might look:

--8<---------------cut here---------------start------------->8---
(declare-namespace jennys-awesome-string-library
  (use elisp29)
  (export foo))

(declare-namespace jonnys-amazing-syntax-library
  (use elisp29)
  (export foo))

(declare-namespace some-new-library
  (use elisp30)
  (import jennys-awesome-string-library st-)
  (import jonnys-amazing-syntax-library sy-))
--8<---------------cut here---------------end--------------->8---

In some-new-library, st-foo refers unambiguously to the binding of
foo in jennys-awesome-string-library, and sy-foo to the binding of
foo in jonnys-amazing-syntax-library. If a package wants a literal
quoted symbol like 'string as input somewhere, or as a literal in a
macro (LOOP), that is no problem (it would be in Common Lisp!).

No modifications to the reader or printer needed.

You can even specify the prefix to end in a semicolon, and it looks
like Common Lisp. I think a visually distinct separator like a
semicolon has a lot of legibility advantages, and this scheme lets
you choose whatever you prefer.

--
Vladimir Sedach
Software engineering services in Los Angeles https://oneofus.la

Re: Proper namespaces in Elisp

[Daniel Colascione]

On 5/8/20 11:59 AM, Vladimir Sedach wrote:
> 
> Daniel Colascione <dancol@dancol.org> writes:
>> Common Lisp does deal with this problem in some cases, e.g., in LOOP.
>> We should be moving forwards keywords anyway.
> 
> LOOP would not even have to deal with that if namespaces were over
> bindings, like Andrea suggested, and what Scheme module systems do.
> 
>> FWIW, I think elisp should just literally copy Common Lisp's
>> approach.
> 
> Please see this other sub-thread where I explain why this is a really
> bad idea:
> 
> https://lists.gnu.org/archive/html/emacs-devel/2020-05/msg00623.html
> 
>> It's tried and tested, and the pitfalls are well-understood.
> 
> Well-understood enough where people want to throw away most of how it
> does namespace imports, and have a backward-compatible replacement
> ready this year:
> 
> https://gist.github.com/phoe/2b63f33a2a4727a437403eceb7a6b4a3

I wouldn't call package-local nicknames "throw[ing] away" the namespace 
system. The post describes a logical extension. Is your argument that 
the CL namespace system (suitably extended) *allows* people to do silly 
things (like :use) even though it doesn't require that they do?

> There is no need to repeat the same mistakes. Andrea's proposal for
> namespacing over bindings/Scheme's approach avoids mangling the
> reader, does not cause problems for the printer, and does not make
> code resort to symbol-name hacks like LOOP does. Scheme's approach to
> namespacing has a lot of advantages over Common Lisp's.
> 
> We can also avoid the "namespace multiple inheritance" problems of
> unintended re-definition and import conflicts if we combine João's
> approach of declaring prefixes with the ideas of package-local
> nicknames and namespaces over bindings.
> 
> Here is how that might look:
> 
> --8<---------------cut here---------------start------------->8---
> (declare-namespace jennys-awesome-string-library
>    (use elisp29)
>    (export foo))
> 
> (declare-namespace jonnys-amazing-syntax-library
>    (use elisp29)
>    (export foo))
> 
> (declare-namespace some-new-library
>    (use elisp30)
>    (import jennys-awesome-string-library st-)
>    (import jonnys-amazing-syntax-library sy-))

Fair enough. What about requiring that a colon separate the module 
prefix from the remainder of the symbol? Using a dash seems ripe for 
misinterpretation, but someone reading a symbol containing colon (in a 
way that looks vaguely reminiscent of CL) would know to look for a 
namespace alias instead of searching in vain for some global definition 
of st-foo or sy-foo.

Re: Proper namespaces in Elisp

[Andrea Corallo]

Vladimir Sedach <vas@oneofus.la> writes:

> There is no need to repeat the same mistakes. Andrea's proposal for
> namespacing over bindings/Scheme's approach avoids mangling the
> reader, does not cause problems for the printer, and does not make
> code resort to symbol-name hacks like LOOP does. Scheme's approach to
> namespacing has a lot of advantages over Common Lisp's.
>
> We can also avoid the "namespace multiple inheritance" problems of
> unintended re-definition and import conflicts if we combine João's
> approach of declaring prefixes with the ideas of package-local
> nicknames and namespaces over bindings.
>
> Here is how that might look:
>
> (declare-namespace jennys-awesome-string-library
>   (use elisp29)
>   (export foo))
>
> (declare-namespace jonnys-amazing-syntax-library
>   (use elisp29)
>   (export foo))
>
> (declare-namespace some-new-library
>   (use elisp30)
>   (import jennys-awesome-string-library st-)
>   (import jonnys-amazing-syntax-library sy-))

Hi Vladimir,

This looks pretty similar as a concept to the raw prototype I've put
together and posted this evening:

https://lists.gnu.org/archive/html/emacs-devel/2020-05/msg01034.html

I do not have the prefix for importing but is something I wanted to add.

I'm learning also here that Scheme works on bindings too, good to be
ignorant!! :)  Now I've very curios to see how different is from what I
came up with.

  Andrea

-- 
akrl@sdf.org

Re: Proper namespaces in Elisp

[Stefan Monnier]

> I'm learning also here that Scheme works on bindings too, good to be
> ignorant!! :)

Note that Scheme doesn't have `symbol-value`, `set`, nor does it have
symbol properties.  That makes things quite a bit easier (basically,
symbols and variables are completely decoupled in Scheme, whereas in
Elisp they are siamese brethren).


        Stefan

Re: Proper namespaces in Elisp

[Vladimir Sedach]

Daniel Colascione <dancol@dancol.org> writes:
> I wouldn't call package-local nicknames "throw[ing] away" the
> namespace system. The post describes a logical extension. Is your
> argument that the CL namespace system (suitably extended) *allows*
> people to do silly things (like :use) even though it doesn't require
> that they do?

Yes, it is silly, and people do it. They will do it in Elisp if you
let them. I think unintended re-definition of short names due to
"namespace multiple inheritance" is going to be a problem for Elisp,
because ordinary Emacs users load code from MELPA. It is a lot
different than encountering the problem in C++ or something when you
are a distribution maintainer compiling binaries. Neither the R6RS
approach of throwing an error, nor clobbering the definition and
showing a warning (what some Common Lisp implementations do) that
will not be paid attention to is appropriate for what should be a
backwards-compatible package update.

> Fair enough. What about requiring that a colon separate the module
> prefix from the remainder of the symbol?

Good question. I think if you leave it open-ended, that can solve the
"s.el prefix is short and bad, but we don't want people to have to
change too much code" problem. Ideally all they would need to do is:

--8<---------------cut here---------------start------------->8---
(declare-namespace legacy-library
  (use elisp29)
  (import modern-string s-))
--8<---------------cut here---------------end--------------->8---

> Using a dash seems ripe for misinterpretation, but someone reading
> a symbol containing colon (in a way that looks vaguely reminiscent
> of CL) would know to look for a namespace alias instead of
> searching in vain for some global definition of st-foo or sy-foo.

That is true. If you use a dash for legacy code, and suggest the use
of semicolon for new code as a convention, that will help minimize
the confusion. I cannot think of a better way.

--
Vladimir Sedach
Software engineering services in Los Angeles https://oneofus.la

Re: Proper namespaces in Elisp

[Daniel Colascione]

On May 8, 2020 5:13:00 PM Vladimir Sedach <vas@oneofus.la> wrote:

> Daniel Colascione <dancol@dancol.org> writes:
>> I wouldn't call package-local nicknames "throw[ing] away" the
>> namespace system. The post describes a logical extension. Is your
>> argument that the CL namespace system (suitably extended) *allows*
>> people to do silly things (like :use) even though it doesn't require
>> that they do?
>
> Yes, it is silly, and people do it. They will do it in Elisp if you
> let them. I think unintended re-definition of short names due to
> "namespace multiple inheritance" is going to be a problem for Elisp,
> because ordinary Emacs users load code from MELPA. It is a lot
> different than encountering the problem in C++ or something when you
> are a distribution maintainer compiling binaries. Neither the R6RS
> approach of throwing an error, nor clobbering the definition and
> showing a warning (what some Common Lisp implementations do) that
> will not be paid attention to is appropriate for what should be a
> backwards-compatible package update.


So there are three basic operations we can support: in Python syntax, 1) 
from PKG import * (exposing PKG.foo as foo), 2) import PKG as p (exposing 
PKG.foo as p.foo) and 3), from PKG import foo (exposing PKG.foo as foo).  
CL supports all three. I'm most interested in supporting #2, since that's 
closest to existing use. The lexspace prototype posted earlier today 
supports #3 and #1 (the latter via lexspace inheritance) only, but I think 
we should do #2 instead, and I think we can do it without runtime overhead.

Re: Proper namespaces in Elisp

[Andrea Corallo]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I'm learning also here that Scheme works on bindings too, good to be
>> ignorant!! :)
>
> Note that Scheme doesn't have `symbol-value`, `set`, nor does it have
> symbol properties.  That makes things quite a bit easier (basically,
> symbols and variables are completely decoupled in Scheme, whereas in
> Elisp they are siamese brethren).

I see.  I guess properties should have a dedicated binding in my proto
(as for function and value has).

-- 
akrl@sdf.org

Re: Proper namespaces in Elisp

[Andrea Corallo]

Daniel Colascione <dancol@dancol.org> writes:

> So there are three basic operations we can support: in Python syntax,
> 1) from PKG import * (exposing PKG.foo as foo), 2) import PKG as p
> (exposing PKG.foo as p.foo) and 3), from PKG import foo (exposing
> PKG.foo as foo).  CL supports all three. I'm most interested in
> supporting #2, since that's closest to existing use. The lexspace
> prototype posted earlier today supports #3 and #1 (the latter via
> lexspace inheritance) only, but I think we should do #2 instead

I suspect we'll need all threes anyway.

> and I
> think we can do it without runtime overhead.

I agree with you.  I think only one additional pointer chase is
sufficent for that but I'd do this only for (byte-)compiled code (see my
replay on the prototype thread).

Andrea

-- 
akrl@sdf.org

Re: Proper namespaces in Elisp

[Tuấn-Anh Nguyễn]

On Fri, May 8, 2020 at 3:29 AM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> > I don't like reader magic. It complicates code logic. I know we talked about
> > symbol-macrolet and macrolet as possible options and know why neither is
> > quite the right thing, but I still think that a macro of the same general
> > kind would work fine. Macros don't need to recognize defining forms if they
> > macroexpand their bodies before walking them.
>
> You need not only discover the defining forms but also the "using"
> forms.  So you need to know that the arguments to `make-local-variable`,
> `add-hook`, `symbol-value`, `facep`, ... (and all function that pass
> their arg to one of those) are not just random symbols (that you should
> leave untouched) but symbols that obey the namespace rules.
>

Alternatively, the design could be:

1. Each symbol’s associated namespace is a first-class concept. So is
   the fact whether a symbol is fully qualified.
2. Each namespace is itself a first-class construct, which holds
   information about symbols imported directly from other namespaces
   (used), and desired aliases of other namespaces.
3. Resolving an unqualified symbol into a qualified one is done by
   consulting a “current namespace” context, and is likewise a first-class
   operation available not just to the reader.
4. The reader returns qualified symbols only when explicitly asked
   (through a "reserved namespace separator"), or configured (i.e.
   imported symbols). It normalizes locally-defined namespace aliases
   while doing so.
5. Primitives that deal with symbols directly become namespace-aware.
   They resolve unqualified symbols when encountering them, using the
   mechanism in 3. These include `set`, `defalias`,
   `make-local-variable`, `funcall`, `symbol-value`, ...

--
Tuấn-Anh Nguyễn
Software Engineer

Re: Proper namespaces in Elisp

[Daniel Colascione]

On May 9, 2020 1:37:16 AM Andrea Corallo <akrl@sdf.org> wrote:

> Daniel Colascione <dancol@dancol.org> writes:
>
>> So there are three basic operations we can support: in Python syntax,
>> 1) from PKG import * (exposing PKG.foo as foo), 2) import PKG as p
>> (exposing PKG.foo as p.foo) and 3), from PKG import foo (exposing
>> PKG.foo as foo).  CL supports all three. I'm most interested in
>> supporting #2, since that's closest to existing use. The lexspace
>> prototype posted earlier today supports #3 and #1 (the latter via
>> lexspace inheritance) only, but I think we should do #2 instead
>
> I suspect we'll need all threes anyway.

Why? What problem is actually being solved by #1 and #3? If the problem is 
that package names long enough to avoid collisions are too annoying to 
type, we can address this problem with #2.

Re: Proper namespaces in Elisp

[Andrea Corallo]

Daniel Colascione <dancol@dancol.org> writes:

> On May 9, 2020 1:37:16 AM Andrea Corallo <akrl@sdf.org> wrote:
>
>> Daniel Colascione <dancol@dancol.org> writes:
>>
>>> So there are three basic operations we can support: in Python syntax,
>>> 1) from PKG import * (exposing PKG.foo as foo), 2) import PKG as p
>>> (exposing PKG.foo as p.foo) and 3), from PKG import foo (exposing
>>> PKG.foo as foo).  CL supports all three. I'm most interested in
>>> supporting #2, since that's closest to existing use. The lexspace
>>> prototype posted earlier today supports #3 and #1 (the latter via
>>> lexspace inheritance) only, but I think we should do #2 instead
>>
>> I suspect we'll need all threes anyway.
>
> Why? What problem is actually being solved by #1 and #3? If the
> problem is that package names long enough to avoid collisions are too
> annoying to type, we can address this problem with #2.

I think #1 would be the way to derive an entire package and add some
functionality to it without changing the names that you'll finally want
to re-expose.  Is this correct?

-- 
akrl@sdf.org

Re: Proper namespaces in Elisp

[Daniel Colascione]

On May 9, 2020 10:25:13 AM Andrea Corallo <akrl@sdf.org> wrote:

> Daniel Colascione <dancol@dancol.org> writes:
>
>> On May 9, 2020 1:37:16 AM Andrea Corallo <akrl@sdf.org> wrote:
>>
>>> Daniel Colascione <dancol@dancol.org> writes:
>>>
>>>> So there are three basic operations we can support: in Python syntax,
>>>> 1) from PKG import * (exposing PKG.foo as foo), 2) import PKG as p
>>>> (exposing PKG.foo as p.foo) and 3), from PKG import foo (exposing
>>>> PKG.foo as foo).  CL supports all three. I'm most interested in
>>>> supporting #2, since that's closest to existing use. The lexspace
>>>> prototype posted earlier today supports #3 and #1 (the latter via
>>>> lexspace inheritance) only, but I think we should do #2 instead
>>>
>>> I suspect we'll need all threes anyway.
>>
>> Why? What problem is actually being solved by #1 and #3? If the
>> problem is that package names long enough to avoid collisions are too
>> annoying to type, we can address this problem with #2.
>
> I think #1 would be the way to derive an entire package and add some
> functionality to it without changing the names that you'll finally want
> to re-expose.  Is this correct?

Can you give a concrete example of a situation in which it'd be useful to 
do that? I think other people on the thread are right about discouraging 
bulk naked symbol imports.


>
> --
> akrl@sdf.org

Re: Proper namespaces in Elisp

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 554 bytes --]

On Sat, May 9, 2020 at 6:45 PM Daniel Colascione <dancol@dancol.org> wrote:

>
> Can you give a concrete example of a situation in which it'd be useful to
> do that? I think other people on the thread are right about discouraging
> bulk naked symbol imports.
>

For a submodule that you do control, it's not such a bad idea.

And you should think about what to do with Elisp's existing  "bare"
namespace, which isn't specced or frozen (and likely never will be).
I think Vlamidir suggested a decent idea for that some messages ago.

João

[-- Attachment #2: Type: text/html, Size: 949 bytes --]

Re: Proper namespaces in Elisp

[Andrea Corallo]

Daniel Colascione <dancol@dancol.org> writes:

> On May 9, 2020 10:25:13 AM Andrea Corallo <akrl@sdf.org> wrote:
>
>> Daniel Colascione <dancol@dancol.org> writes:
>>
>>> On May 9, 2020 1:37:16 AM Andrea Corallo <akrl@sdf.org> wrote:
>>>
>>>> Daniel Colascione <dancol@dancol.org> writes:
>>>>
>>>>> So there are three basic operations we can support: in Python syntax,
>>>>> 1) from PKG import * (exposing PKG.foo as foo), 2) import PKG as p
>>>>> (exposing PKG.foo as p.foo) and 3), from PKG import foo (exposing
>>>>> PKG.foo as foo).  CL supports all three. I'm most interested in
>>>>> supporting #2, since that's closest to existing use. The lexspace
>>>>> prototype posted earlier today supports #3 and #1 (the latter via
>>>>> lexspace inheritance) only, but I think we should do #2 instead
>>>>
>>>> I suspect we'll need all threes anyway.
>>>
>>> Why? What problem is actually being solved by #1 and #3? If the
>>> problem is that package names long enough to avoid collisions are too
>>> annoying to type, we can address this problem with #2.
>>
>> I think #1 would be the way to derive an entire package and add some
>> functionality to it without changing the names that you'll finally want
>> to re-expose.  Is this correct?
>
> Can you give a concrete example of a situation in which it'd be useful
> to do that? I think other people on the thread are right about
> discouraging bulk naked symbol imports.

Say we have a library exposing functions A B C D ... Z and you want to
make a package that is exaclty the same except for functions F and G.
You'd import everything (or what you need with #3) and (re)define your F
and G versions of it.

-- 
akrl@sdf.org

Re: Proper namespaces in Elisp

[Daniel Colascione]

On 5/9/20 11:23 AM, João Távora wrote:
> On Sat, May 9, 2020 at 6:45 PM Daniel Colascione <dancol@dancol.org 
> <mailto:dancol@dancol.org>> wrote:
> 
> 
>     Can you give a concrete example of a situation in which it'd be
>     useful to
>     do that? I think other people on the thread are right about
>     discouraging
>     bulk naked symbol imports.
> 
> 
> For a submodule that you do control, it's not such a bad idea.

Concrete example? If you want inheritance, you can get it with EIEIO.

Re: Proper namespaces in Elisp

[Daniel Colascione]

On 5/9/20 11:30 AM, Andrea Corallo wrote:
> Daniel Colascione <dancol@dancol.org> writes:
> 
>> On May 9, 2020 10:25:13 AM Andrea Corallo <akrl@sdf.org> wrote:
>>
>>> Daniel Colascione <dancol@dancol.org> writes:
>>>
>>>> On May 9, 2020 1:37:16 AM Andrea Corallo <akrl@sdf.org> wrote:
>>>>
>>>>> Daniel Colascione <dancol@dancol.org> writes:
>>>>>
>>>>>> So there are three basic operations we can support: in Python syntax,
>>>>>> 1) from PKG import * (exposing PKG.foo as foo), 2) import PKG as p
>>>>>> (exposing PKG.foo as p.foo) and 3), from PKG import foo (exposing
>>>>>> PKG.foo as foo).  CL supports all three. I'm most interested in
>>>>>> supporting #2, since that's closest to existing use. The lexspace
>>>>>> prototype posted earlier today supports #3 and #1 (the latter via
>>>>>> lexspace inheritance) only, but I think we should do #2 instead
>>>>>
>>>>> I suspect we'll need all threes anyway.
>>>>
>>>> Why? What problem is actually being solved by #1 and #3? If the
>>>> problem is that package names long enough to avoid collisions are too
>>>> annoying to type, we can address this problem with #2.
>>>
>>> I think #1 would be the way to derive an entire package and add some
>>> functionality to it without changing the names that you'll finally want
>>> to re-expose.  Is this correct?
>>
>> Can you give a concrete example of a situation in which it'd be useful
>> to do that? I think other people on the thread are right about
>> discouraging bulk naked symbol imports.
> 
> Say we have a library exposing functions A B C D ... Z and you want to
> make a package that is exaclty the same except for functions F and G.
> You'd import everything (or what you need with #3) and (re)define your F
> and G versions of it.

When does this situation ever actually come up? And why should we pick 
an approach that incurs runtime overhead everywhere just for this?

Re: Proper namespaces in Elisp

[João Távora]

On Sat, May 9, 2020 at 7:32 PM Daniel Colascione <dancol@dancol.org> wrote:
>
>
>
> On 5/9/20 11:23 AM, João Távora wrote:
> > On Sat, May 9, 2020 at 6:45 PM Daniel Colascione <dancol@dancol.org
> > <mailto:dancol@dancol.org>> wrote:
> >
> >
> >     Can you give a concrete example of a situation in which it'd be
> >     useful to
> >     do that? I think other people on the thread are right about
> >     discouraging
> >     bulk naked symbol imports.
> >
> >
> > For a submodule that you do control, it's not such a bad idea.
>
> Concrete example? If you want inheritance, you can get it with EIEIO.

When writing a namespaced foo-tests.el library for your namespaced
foo.el library, for example.  Or for a big enough foo program, a foo-utils.el
that exports utils that only foo.el needs and noone else.

EIEIO is Elisp's CLOS, AFAIK.  I use it, but don't know what it has to do with
namespacing.

João

Re: Proper namespaces in Elisp

[Daniel Colascione]

On 5/9/20 11:35 AM, João Távora wrote:
> On Sat, May 9, 2020 at 7:32 PM Daniel Colascione <dancol@dancol.org> wrote:
>>
>>
>>
>> On 5/9/20 11:23 AM, João Távora wrote:
>>> On Sat, May 9, 2020 at 6:45 PM Daniel Colascione <dancol@dancol.org
>>> <mailto:dancol@dancol.org>> wrote:
>>>
>>>
>>>      Can you give a concrete example of a situation in which it'd be
>>>      useful to
>>>      do that? I think other people on the thread are right about
>>>      discouraging
>>>      bulk naked symbol imports.
>>>
>>>
>>> For a submodule that you do control, it's not such a bad idea.
>>
>> Concrete example? If you want inheritance, you can get it with EIEIO.
> 
> When writing a namespaced foo-tests.el library for your namespaced
> foo.el library, for example.  Or for a big enough foo program, a foo-utils.el
> that exports utils that only foo.el needs and noone else.

Why not just re-enter the same namespace in your test file then? What 
problem are you actually trying to solve here? If the problem is just 
package name prefix length, you can fix it with symbol rewriting without 
a ton of other complexity. I don't see a *need* for module inheritance 
or bulk imports or anything like that just to address the problem of 
symbol names being inconveniently long in source code.

> EIEIO is Elisp's CLOS, AFAIK.  I use it, but don't know what it has to do with
> namespacing.

It's a way of getting implementation inheritance. So is the module 
repackaging you're describing.

Re: Proper namespaces in Elisp

[Andrea Corallo]

Daniel Colascione <dancol@dancol.org> writes:


>>
>> Say we have a library exposing functions A B C D ... Z and you want to
>> make a package that is exaclty the same except for functions F and G.
>> You'd import everything (or what you need with #3) and (re)define your F
>> and G versions of it.
>
> When does this situation ever actually come up?

Never in Elisp because we have no namespace facility, but for instance
it could be a cleaner way to change the behavior of a certain package
respect to just advising its functions.  I think this is a pretty common
case for us.

> And why should we pick
> an approach that incurs runtime overhead everywhere just for this?

I'm *not* sure we should and I'm not stating that now.  I suspect it
would be worth, but we are really in early discussion and investigation.

-- 
akrl@sdf.org

Re: Proper namespaces in Elisp

[João Távora]

On Sat, May 9, 2020 at 7:39 PM Daniel Colascione <dancol@dancol.org> wrote:
>
> On 5/9/20 11:35 AM, João Távora wrote:
> > On Sat, May 9, 2020 at 7:32 PM Daniel Colascione <dancol@dancol.org> wrote:
> >>
> >>
> >>
> >> On 5/9/20 11:23 AM, João Távora wrote:
> >>> On Sat, May 9, 2020 at 6:45 PM Daniel Colascione <dancol@dancol.org
> >>> <mailto:dancol@dancol.org>> wrote:
> >>>
> >>>
> >>>      Can you give a concrete example of a situation in which it'd be
> >>>      useful to
> >>>      do that? I think other people on the thread are right about
> >>>      discouraging
> >>>      bulk naked symbol imports.
> >>>
> >>>
> >>> For a submodule that you do control, it's not such a bad idea.
> >>
> >> Concrete example? If you want inheritance, you can get it with EIEIO.
> >
> > When writing a namespaced foo-tests.el library for your namespaced
> > foo.el library, for example.  Or for a big enough foo program, a foo-utils.el
> > that exports utils that only foo.el needs and noone else.
>
> Why not just re-enter the same namespace in your test file then?
> What problem are you actually trying to solve here?

In my foo-test.el file I have definitions, too. I don't want those definitions
to enter my foo namespace.  It's very simple to follow.

I don't know what you call module inheritance.   Look, you know the CL package
system right?  I'll just speak CL then, if you don't  mind:

;; foo-utils .lisp
(defpackage :foo-utils (:use :cl) (:export #:add-two))
(in-package :foo-utils)

(defun contorted-two-adder (x) (+ 1 1 x))
(defun add-two (x) (contorted-two-adder x))

;; foo.lisp
(defpackage :foo (:use :cl :foo-utils)  (:export #:bar))
(in-package :foo)

(defun bar () (add-two 40))

;; foo-tests.lisp
(defpackage :foo-tests (:use :cl :foo))
(in-package :foo-tests)

(defun check-forty-two-ness ()  (unless (eq (bar) 42)) (error "oh noes")))

Hope I didn't make many mistakes typing this in Gmail...

> package name prefix length, you can fix it with symbol rewriting without
> a ton of other complexity. I don't see a *need* for module inheritance

Again I don't know what you call "module inheritance". Maybe we're
just mis-communicating.  I'm just describing bog-standard "hide the
implementation" stuff.  I was suggesting reasonable uses of :USE, or
what I think you called "bulk symbol imports", i.e. merging two
namespaces.

João

Why :USE sucks in the Common Lisp package system

[Michał "phoe" Herda]

On 09.05.2020 20:48, Andrea Corallo wrote:
> I'm *not* sure we should and I'm not stating that now.  I suspect it
> would be worth, but we are really in early discussion and investigation.

Hey,

just noticed that one of my posts was mentioned here earlier; I thought
that I could provide a bit more context from the perspective of a Common
Lisp programmer, especially in the context of the namespacing discussion
for elisp.

In CL, we have packages that can :USE each other, and we have had them
ever since Common Lisp had become standardized. Recently (and I mean,
like, very recently - two, three years?), the support for a language
extension called package-local nicknames has emerged, mostly to resolve
the growing issue of global nicknames polluting global namespaces: BT
stood for Binary Types, but BT also stood for Bordeaux Threads, but BT
also stood for... And so on, and so on. This, however, has given the CL
community to work with one more issue in the CL ecosystem, which is :USE
abuse.

The major issue with :USE is its ubiquity - packages :USE literally
everything, a lot of other packages, even (and especially) code that
they have no control over. This means that they now depend on these
packages' exports not interfering with their code. Who can guarantee
that? No one, really, unless some manual or automated inspection of
packages is done.

This has led to de-facto ossification of some libraries upon which lots
of code depends, the most famous example being Alexandria, the most
popular CL library. If everyone uses Alexandria along with 20 or so
other packages, then if Alexandria introduces a new exported symbol,
then it may either generate package conflicts with other used packages
(this visibly breaks code) or its value/function/class/whatever cells
may be silently overwritten by a package that previously used to have
its own symbol with that name but now inherits it from Alexandria (which
*invisibly* breaks code).

I bet that a massive amount of Common Lisp code is written that way,
simply because PLNs did not even exist back when that code was written.
Previously, it was cumbersome to manually write (alexandria:assoc-value
...), so people resorted to :USE; nowadays, with PLNs, it is much easier
to write (a:assoc-value ...), and :USE is not required to write readable
code.

My point of view is that the package system with PLNs is good enough for
my use cases: I can :USE the COMMON-LISP package or perhaps one of its
replacements, such as COMMON-LISP+QT, and locally nickname all libraries
I use.

I hope that, at one point (meaning: when the CLISP implementation gets
package-local nicknames implemented), we will be able to go through all
of the code in Quicklisp and adjust it to reduce the :USE abuse, and
therefore free existing libraries from this unfortunate ossification.
It's a non-trivial amount of work to do, but I hope it'll be done.

While I'm no emacs programmer (this is my first post here!), I think
that a package system with PLNs could work for elisp as a tool for
symbol namespacing, as long as it is clear that elisp programmers know
not to repeat the mistake from the CL ecosystem and only use :USE for
packages that either do not change (in our case, it's the COMMON-LISP
package), or a package that changes but is the only used package (and
therefore has nothing to conflict with), or a combination of packages
that do not change and we have full control over so we can manually see
and resolve the symbol conflicts as they arise. And then, we locally
nickname everything, e.g. importing ALEXANDRIA as A, FLEXI-STREAMS as F,
BORDEAUX-THREADS as BT or just T, and write our code that way.

BR
Michał "phoe" Herda

Re: Why :USE sucks in the Common Lisp package system

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 662 bytes --]

On Sat, May 9, 2020 at 9:34 PM Michał "phoe" Herda <phoe@disroot.org> wrote:
Hi  Michał, welcome to the list, your post came up in a thread I was
participating in.

> The major issue with :USE is its ubiquity - packages :USE literally
> everything, a lot of other packages, even (and especially) code that
> they have no control over.

But don't _you_ ever use :USE it for packages that you _do_ have
control over?  Like an internal utils package?  Or a test package
that uses the package under testing?  Otherwise I agree with you
about the ":USE abuse".

See also:
https://lists.gnu.org/archive/html/emacs-devel/2020-05/msg01264.html

João

[-- Attachment #2: Type: text/html, Size: 1093 bytes --]

Re: Why :USE sucks in the Common Lisp package system

[Michał "phoe" Herda]

[-- Attachment #1: Type: text/plain, Size: 936 bytes --]

Thanks for the welcome.

On 09.05.2020 23:47, João Távora wrote:
> But don't _you_ ever use :USE it for packages that you _do_ have
> control over?  Like an internal utils package?  Or a test package
> that uses the package under testing?  Otherwise I agree with you
> about the ":USE abuse". 

Of course I do! I've mentioned in my earlier mail,

> or a combination of packages that do not change and we have full
control over so we can manually see and resolve the symbol conflicts as
they arise

Perhaps that wasn't clear enough from me: I meant that it is fine to
:USE packages that are known to not change and to :USE packages that we
fully control, such as internal util packages that you mention; and it
is fine to combine using the two within a single package.

I actually disagree when it comes to :USE-ing packages under test; I
think that they should be tested via their fully qualified name, or
local nickname.

~phoe


[-- Attachment #2: Type: text/html, Size: 1699 bytes --]

Re: Why :USE sucks in the Common Lisp package system

[Daniel Colascione]

On 5/9/20 2:55 PM, Michał "phoe" Herda wrote:
> Thanks for the welcome.
> 
> On 09.05.2020 23:47, João Távora wrote:
>> But don't _you_ ever use :USE it for packages that you _do_ have
>> control over?  Like an internal utils package?  Or a test package
>> that uses the package under testing?  Otherwise I agree with you
>> about the ":USE abuse".
> 
> Of course I do! I've mentioned in my earlier mail,
> 
>  > or a combination of packages that do not change and we have full 
> control over so we can manually see and resolve the symbol conflicts as 
> they arise
> 
> Perhaps that wasn't clear enough from me: I meant that it is fine to 
> :USE packages that are known to not change and to :USE packages that we 
> fully control, such as internal util packages that you mention; and it 
> is fine to combine using the two within a single package.

It's just as easy to use local nicknames here though. I still haven't 
seen a compelling case for :use that outweighs the risk of imprudent 
use. Why is prefix-less import of some internal utils package so 
important?  If the utils package and the user of the utils package have 
the same author, they can just share a namespace instead of one using 
the other.

Re: Why :USE sucks in the Common Lisp package system

[Michał "phoe" Herda]

On 10.05.2020 00:01, Daniel Colascione wrote:
> It's just as easy to use local nicknames here though. I still haven't
> seen a compelling case for :use that outweighs the risk of imprudent
> use. Why is prefix-less import of some internal utils package so
> important?  If the utils package and the user of the utils package
> have the same author, they can just share a namespace instead of one
> using the other.

Sure, that is a convincing argument. The single compelling use case for
:USE is not needing to type CL:IF everywhere you want to type IF,
CL:DEFUN everywhere you want DEFUN, et cetera; literally everything else
is achievable via package-local-nicknames instead of that.

The big issue is that decades- or even years-old code authors did not
have PLNs at their disposal, and even though they were available in SBCL
for a long time, they were not widespread in the CL ecosystem. Now the
situation has changed and they are close enough to be supported by all
contemporary alive implementations - the people who wrote Lisp code back
in the day were unable to use such luxuries while maintaining readable
code of their own libraries.

Re: Why :USE sucks in the Common Lisp package system

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 641 bytes --]

On Sat, May 9, 2020 at 10:55 PM Michał "phoe" Herda <phoe@disroot.org>
wrote:

> Thanks for the welcome.
>
> On 09.05.2020 23:47, João Távora wrote:
>
> But don't _you_ ever use :USE it for packages that you _do_ have
> control over?  Like an internal utils package?  Or a test package
> that uses the package under testing?  Otherwise I agree with you
> about the ":USE abuse".
>
> Of course I do! I've mentioned in my earlier mail,
>
Doh, sorry.  I read so many mails todays that I read yours
in diagonal.   You're right. The subject doesn't help, tho.
It sounds  a bit like you're making a blanket statement.

João

[-- Attachment #2: Type: text/html, Size: 1363 bytes --]

Re: Why :USE sucks in the Common Lisp package system

[Andrea Corallo]

"Michał \"phoe\" Herda" <phoe@disroot.org> writes:

> While I'm no emacs programmer (this is my first post here!), I think
> that a package system with PLNs could work for elisp as a tool for
> symbol namespacing, as long as it is clear that elisp programmers know
> not to repeat the mistake from the CL ecosystem and only use :USE for
> packages that either do not change (in our case, it's the COMMON-LISP
> package), or a package that changes but is the only used package (and
> therefore has nothing to conflict with), or a combination of packages
> that do not change and we have full control over so we can manually see
> and resolve the symbol conflicts as they arise. And then, we locally
> nickname everything, e.g. importing ALEXANDRIA as A, FLEXI-STREAMS as F,
> BORDEAUX-THREADS as BT or just T, and write our code that way.
>
> BR
> Michał "phoe" Herda

Hi Michał,

Nice to read you here!

From your write to me sounds that the real trouble with :USE is that
namespace versioning is missing.  Very interesting.

-- 
akrl@sdf.org

Re: Proper namespaces in Elisp

[Vladimir Sedach]

Daniel Colascione <dancol@dancol.org> writes:
> So there are three basic operations we can support: in Python syntax,
> 1) from PKG import * (exposing PKG.foo as foo), 2) import PKG as p
> (exposing PKG.foo as p.foo) and 3), from PKG import foo (exposing
> PKG.foo as foo).  CL supports all three. I'm most interested in
> supporting #2, since that's closest to existing use. The lexspace
> prototype posted earlier today supports #3 and #1 (the latter via
> lexspace inheritance) only, but I think we should do #2 instead, and
> I think we can do it without runtime overhead.

Before I write anything else, I have to call attention to a fact you
might already be aware of: Python cannot do code re-loading reliably,
because depending on which of these 3 ways a function is imported,
the new definition might or might not be seen.

This is a really good example of the way that namespace systems can
have unintended consequences, and why we have to be careful designing
one for Elisp.

#3 is ok, because that way you know exactly what you are importing,
and will not get any surprises if PKG exports another definition
later.

Most use cases want a single #1 to pull in the language that you are
working in (in Common Lisp, this would be the "COMMON-LISP" package).

As I pointed out in another sub-thread, this can be used to advantage
if you version Elisp namespaces to major Emacs versions (elisp29,
elisp30, etc.). It is a lot easier to support backwards compatibility
and make major changes to Emacs that way.

The only problem is retrofitting this on top of the global namespace.
The global namespace is going to have to be a fallback for all symbol
lookups. Existing libraries also need to have their declared exports
put into the global namespace with their chosen package prefix and
appropriate autoloads, to be compatible with packages that do not use
namespaces (advice/redefinition is going to have to do the
appropriate thing in both places, and dynamic bindings have to be
shared!). Modifying internal functions is already a gamble on
updates, so I think it is ok to tell people "you need to access this
internal function with a namespace qualifier because the package now
uses namespaces."

The reason #1 becomes a problem is if you do it two or more times.
This is akin to "namespace multiple inheritance." This is different
from multiple inheritance in OO because it is not transitive (you do
not want to import the dependencies of the package you are
importing!).

--
Vladimir Sedach
Software engineering services in Los Angeles https://oneofus.la

Re: Why :USE sucks in the Common Lisp package system

[Andreas Schwab]

On Mai 09 2020, Andrea Corallo wrote:

> "Michał \"phoe\" Herda" <phoe@disroot.org> writes:
>
>> While I'm no emacs programmer (this is my first post here!), I think
>> that a package system with PLNs could work for elisp as a tool for
>> symbol namespacing, as long as it is clear that elisp programmers know
>> not to repeat the mistake from the CL ecosystem and only use :USE for
>> packages that either do not change (in our case, it's the COMMON-LISP
>> package), or a package that changes but is the only used package (and
>> therefore has nothing to conflict with), or a combination of packages
>> that do not change and we have full control over so we can manually see
>> and resolve the symbol conflicts as they arise. And then, we locally
>> nickname everything, e.g. importing ALEXANDRIA as A, FLEXI-STREAMS as F,
>> BORDEAUX-THREADS as BT or just T, and write our code that way.
>>
>> BR
>> Michał "phoe" Herda
>
> Hi Michał,
>
> Nice to read you here!
>
> From your write to me sounds that the real trouble with :USE is that
> namespace versioning is missing.  Very interesting.

In python or perl, you can control which identifiers to import from a
module into the current namespace.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Why :USE sucks in the Common Lisp package system

[Helmut Eller]

On Sun, May 10 2020, Andreas Schwab wrote:

> On Mai 09 2020, Andrea Corallo wrote:
>
>> From your write to me sounds that the real trouble with :USE is that
>> namespace versioning is missing.  Very interesting.
>
> In python or perl, you can control which identifiers to import from a
> module into the current namespace.

Common Lisp also has something similar with, ahem, :IMPORT-FROM or
manually with IMPORT.  But as usual, it's more fun to rant about how
unusable Common Lisp is.

Helmut

Re: Why :USE sucks in the Common Lisp package system

[Michał "phoe" Herda]

On 10.05.2020 10:53, Helmut Eller wrote:
> Common Lisp also has something similar with, ahem, :IMPORT-FROM or
> manually with IMPORT.  But as usual, it's more fun to rant about how
> unusable Common Lisp is.

I'm not ranting how unusable it is here; I'm saying what I consider to
be a mistake in retrospect that other languages IMO should learn from as
they build their namespacing.

Thirty years ago, people did not really care about namespace pollution,
namespace versioning, or even about library versioning much; nowadays,
it turns out to be a big problem, as someone mentioned earlier in the
thread.

Common Lisp, including ASDF and Quicklisp, has no good solution for
library versioning, let alone package export or namespace versioning.
All standard Quicklisp dists are just bundled together after verifying
that they load on a single image, and version pinning of individual
dists is left as an exercise for the programmer. (Even Alexandria stayed
on version 0.0.0[0] for *years* before I've actually asked it to be
fixed[1].)

It is not *too* much of an issue since the Common Lisp world is not a
very big one compared to other programming languages, but as the CL
ecosystem grows, I expect that it'll start to become a somewhat pressing
matter eventually.

~phoe

[0] https://gitlab.common-lisp.net/alexandria/alexandria/-/commit/6bb56b6c
[1] https://gitlab.common-lisp.net/alexandria/alexandria/-/issues/7

Re: Why :USE sucks in the Common Lisp package system

[Michał "phoe" Herda]

[-- Attachment #1: Type: text/plain, Size: 279 bytes --]

On 10.05.2020 00:12, João Távora wrote:
> Doh, sorry.  I read so many mails todays that I read yours
> in diagonal.   You're right. The subject doesn't help, tho.
> It sounds  a bit like you're making a blanket statement.

Yes. I could likely have worded it better.

~phoe


[-- Attachment #2: Type: text/html, Size: 849 bytes --]