Help sought understanding shorthands wrt modules/packages

Help sought understanding shorthands wrt modules/packages

[Gerd Möllmann]

Since Richard is swamped with mails, I'd like to try and ask some
questions here.

To start with---I have no problem understanding the shorthands
mechanism, that's easy enough.  (To be honest, I have to admit I don't
like the idea, for reasons I won't go into here, but that's another
story.  A mechanism will find its uses, I guess :-).

My problem is understanding why anyone would think shorthands are an
alternative to or substitute for packages.  I mean in principle, and
regardless of packages being Common Lisp packages or any other form of
module system.  I simply can't get it.

So, I guess my first question is: does anyone here think shorthands are
a substitute got packages, and if so could you please share your thought
why, and how one would use them for that?  If not, it might also be
helpful to know why not.

And my second question os: Because I don't get it, I'm considering the
possibility that shorthands are still incomplete, and some features are
missing that would make it a substitute for a module system.  Something
that's not documented anywhere, apparently.  If so, what is it?  Or
where is it described?  Or does someone have an idea?  You know what I
mean.

Finally, here's why I don't get the idea:

According to me, shorthands don't address the problem
packages/modules/namespaces are trying to do deal with _at all_, which
is programming in the large, which means that large programs with many
and diverse contributors suffer from name conflicts.

The reason why I think shorthands don't address that problem is pretty
simple, according to me again: The underlying one obarray can only
contain one name per symbol at a time.  So, whatever name lands in the
obarray must be unique, as it always was.  And this of course leads to
conflicts.  Whether or not the unique name can otherwise be used in
abbreviated form or not doesn't change anything in this regard.

Or am I going nuts and overlooking the obvious?

Re: Help sought understanding shorthands wrt modules/packages

[tomas]

[-- Attachment #1: Type: text/plain, Size: 349 bytes --]

On Sun, Oct 30, 2022 at 07:45:39AM +0100, Gerd Möllmann wrote:
> Since Richard is swamped with mails, I'd like to try and ask some
> questions here.

I had similar feelings about shorthands vs. (module) name spaces.

I just couldn't figure out whether this was due to habituation
or whether I was feeling a "real" problem

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Help sought understanding shorthands wrt modules/packages

[Eli Zaretskii]

> Date: Sun, 30 Oct 2022 09:38:41 +0100
> From: <tomas@tuxteam.de>
> 
> On Sun, Oct 30, 2022 at 07:45:39AM +0100, Gerd Möllmann wrote:
> > Since Richard is swamped with mails, I'd like to try and ask some
> > questions here.
> 
> I had similar feelings about shorthands vs. (module) name spaces.
> 
> I just couldn't figure out whether this was due to habituation
> or whether I was feeling a "real" problem

Before everyone starts presenting here that we made some nonsensical
decision when we added shorthands, let me remind us that it was added
to allow us to use packages like "s" (https://github.com/magnars/s.el),
"f" (https://github.com/rejeep/f.el) and other similar ones, which use
"problematic" function names.  We wanted to be able to use packages
which have dependencies of those, while at the same time providing
reasonably-named aliases for them.  (Gerd wasn't around back then, so
he is excused, but the rest of us were here, so please let's not
pretend those discussions never happened.)

See the following long discussions:

  https://lists.gnu.org/archive/html/emacs-devel/2020-04/msg02055.html
  https://lists.gnu.org/archive/html/emacs-devel/2020-05/msg00016.html

Re: Help sought understanding shorthands wrt modules/packages

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1254 bytes --]

On Sun, Oct 30, 2022 at 10:54:54AM +0200, Eli Zaretskii wrote:
> > Date: Sun, 30 Oct 2022 09:38:41 +0100
> > From: <tomas@tuxteam.de>
> > 
> > On Sun, Oct 30, 2022 at 07:45:39AM +0100, Gerd Möllmann wrote:
> > > Since Richard is swamped with mails, I'd like to try and ask some
> > > questions here.
> > 
> > I had similar feelings about shorthands vs. (module) name spaces.
> > 
> > I just couldn't figure out whether this was due to habituation
> > or whether I was feeling a "real" problem
> 
> Before everyone starts presenting here that we made some nonsensical
> decision when we added shorthands, let me remind us that it was added
> to allow us to use packages like "s" (https://github.com/magnars/s.el),
> "f" (https://github.com/rejeep/f.el) and other similar ones, which use
> "problematic" function names.  We wanted to be able to use packages
> which have dependencies of those, while at the same time providing
> reasonably-named aliases for them.  (Gerd wasn't around back then, so
> he is excused, but the rest of us were here, so please let's not
> pretend those discussions never happened.)

I don't know who you mean by "us". I'm not pretending anything. I do
know that discussion happened.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Help sought understanding shorthands wrt modules/packages

[Gerd Möllmann]

I was asking specifically about shorthands wrt to their use as a module 
replacement, and I think tomas replied in that context.

If anyone wants to know why I think shorthands should be used only in 
exceptional circumstances, and not broadly, I could explain that, but 
that's not what I'm currently after.

And I guess, after reading the threads, at least the parts that didn't 
veer off completely,  I'm no longer excused :-)

Re: Help sought understanding shorthands wrt modules/packages

[Eli Zaretskii]

> Date: Sun, 30 Oct 2022 10:14:10 +0100
> From: tomas@tuxteam.de
> Cc: emacs-devel@gnu.org
> 
> > Before everyone starts presenting here that we made some nonsensical
> > decision when we added shorthands, let me remind us that it was added
> > to allow us to use packages like "s" (https://github.com/magnars/s.el),
> > "f" (https://github.com/rejeep/f.el) and other similar ones, which use
> > "problematic" function names.  We wanted to be able to use packages
> > which have dependencies of those, while at the same time providing
> > reasonably-named aliases for them.  (Gerd wasn't around back then, so
> > he is excused, but the rest of us were here, so please let's not
> > pretend those discussions never happened.)
> 
> I don't know who you mean by "us". I'm not pretending anything. I do
> know that discussion happened.

Well, may I suggest that you re-read your message, then?  "I had
similar feelings" can be easily interpreted as saying that no one took
such opinions seriously, or maybe that they were not discussed at all.
Which is not what happened.

People have short memories.  A discussion that happened 2.5 years ago
tends to be restarted for no good reason.  You dwell here long enough
to know that.  So when you respond to a request from someone who
wasn't here when those discussions happened, my recommendation is to
mention those discussions, so that this one at least had a good chance
of being a continuation of those ones, not a re-iterations of them.

I'm sorry to be a bit blunt, but the level of noise on this list is
unbearable.  I urge everyone to please keep that in mind, and make an
effort to lower the noise.

Re: Help sought understanding shorthands wrt modules/packages

[Eli Zaretskii]

> Date: Sun, 30 Oct 2022 11:26:45 +0100
> Cc: eliz@gnu.org, emacs-devel@gnu.org
> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> 
> I was asking specifically about shorthands wrt to their use as a module 
> replacement, and I think tomas replied in that context.

The shorthands were added for a specific purpose that should be clear
from those discussions.  It can also be used as a "poor-man's
packaging system", and some contend that it is more than enough for
that purpose.  But that's not why we added it, and I don't necessarily
agree with the opinions which consider shorthands to be a basis for
Emacs package system, let alone a good one.

Moreover, from my POV, the jury is still out on the question of
whether we at all need packages in Emacs.  "Programming in the large"
doesn't sound very relevant to how Emacs Lisp is used.  It is not a GP
language for writing programs, it is a language for writing Emacs
applications.  The largest Emacs Lisp applications -- Gnus and Org --
are nowhere near the bar which I'd consider to be "programming in the
large".

I may change my mind, of course, but I didn't yet see any cogent
explanation of why we would need packages in Emacs.  People just tend
to consider it self-evident, like some axiom.  (And if you or someone
want to start a discussion of that, please change the Subject.)

Thanks.

Re: Help sought understanding shorthands wrt modules/packages

[tomas]

[-- Attachment #1: Type: text/plain, Size: 2465 bytes --]

On Sun, Oct 30, 2022 at 12:40:45PM +0200, Eli Zaretskii wrote:
> > Date: Sun, 30 Oct 2022 10:14:10 +0100
> > From: tomas@tuxteam.de
> > Cc: emacs-devel@gnu.org
> > 
> > > Before everyone starts presenting here that we made some nonsensical
> > > decision when we added shorthands, let me remind us that it was added
> > > to allow us to use packages like "s" (https://github.com/magnars/s.el),
> > > "f" (https://github.com/rejeep/f.el) and other similar ones, which use
> > > "problematic" function names.  We wanted to be able to use packages
> > > which have dependencies of those, while at the same time providing
> > > reasonably-named aliases for them.  (Gerd wasn't around back then, so
> > > he is excused, but the rest of us were here, so please let's not
> > > pretend those discussions never happened.)
> > 
> > I don't know who you mean by "us". I'm not pretending anything. I do
> > know that discussion happened.
> 
> Well, may I suggest that you re-read your message, then?  "I had
> similar feelings" can be easily interpreted as saying that no one took
> such opinions seriously,

Perhaps it can be interpreted this way. That interpretation would be
wrong. I think it's clear that I didn't articulate those feelings,
so that's on me. I hadn't enough "hard reasoning" to be worth presenting.

I'm not putting that blame on anyone but me. Please, don't over-interpret.

>                     or maybe that they were not discussed at all.
> Which is not what happened.

Again, I'm at a loss on how you can read that into my message.

> People have short memories.  A discussion that happened 2.5 years ago
> tends to be restarted for no good reason.  You dwell here long enough
> to know that.  So when you respond to a request from someone who
> wasn't here when those discussions happened, my recommendation is to
> mention those discussions, so that this one at least had a good chance
> of being a continuation of those ones, not a re-iterations of them.

Please, Eli. I think Gerd is sufficiently grown up to differentiate
between one "I had similar feelings" and what went on in the list,
which is in the archives and he has surely gone over.

> I'm sorry to be a bit blunt, but the level of noise on this list is
> unbearable.  I urge everyone to please keep that in mind, and make an
> effort to lower the noise.

Gerd's thoughts are everything but noise. You know that best.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Help sought understanding shorthands wrt modules/packages

[Eli Zaretskii]

> Date: Sun, 30 Oct 2022 12:06:36 +0100
> From: tomas@tuxteam.de
> Cc: emacs-devel@gnu.org
> 
> > I'm sorry to be a bit blunt, but the level of noise on this list is
> > unbearable.  I urge everyone to please keep that in mind, and make an
> > effort to lower the noise.
> 
> Gerd's thoughts are everything but noise. You know that best.

I was trying to prevent future noise from others.

Is there a need for a module system

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sun, 30 Oct 2022 11:26:45 +0100
>> Cc: eliz@gnu.org, emacs-devel@gnu.org
>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> 
>> I was asking specifically about shorthands wrt to their use as a module 
>> replacement, and I think tomas replied in that context.
>
> The shorthands were added for a specific purpose that should be clear
> from those discussions.  It can also be used as a "poor-man's
> packaging system", and some contend that it is more than enough for
> that purpose.

That's what I wanted to know more about.  Specifically, the whys and
hows.

> But that's not why we added it, and I don't necessarily
> agree with the opinions which consider shorthands to be a basis for
> Emacs package system, let alone a good one.

Ok.

> Moreover, from my POV, the jury is still out on the question of
> whether we at all need packages in Emacs.  "Programming in the large"
> doesn't sound very relevant to how Emacs Lisp is used.  It is not a GP
> language for writing programs, it is a language for writing Emacs
> applications.  The largest Emacs Lisp applications -- Gnus and Org --
> are nowhere near the bar which I'd consider to be "programming in the
> large".

Ok, here's a new thread, because that's interesting, and something I
don't agree with.

The reason being that I think one cannot look at specific Lisp programs
like gnus or org in isolation. If I look at Melpa, it says it has 5299
packages.  Add GNU Elpa, for which I couldn't readily find the number,
and Emacs' Lisp, and subtract what's in common.  That's pretty large
from my POV.

> I may change my mind, of course, but I didn't yet see any cogent
> explanation of why we would need packages in Emacs.  People just tend
> to consider it self-evident, like some axiom.

Need is a strong requirement.  Do we need it now?  Obviously no, it's
working using naming schemes, more or less as dash.el and s.el show.

I'd rather ask is that a good enough solution?  Can we do better?  Is it
worth the effort?  Can it be done in the first place?  Does it perhaps
enable additional things?

Opinions welcome.

Re: Is there a need for a module system

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: tomas@tuxteam.de,  emacs-devel@gnu.org
> Date: Sun, 30 Oct 2022 12:24:15 +0100
> 
> The reason being that I think one cannot look at specific Lisp programs
> like gnus or org in isolation. If I look at Melpa, it says it has 5299
> packages.  Add GNU Elpa, for which I couldn't readily find the number,
> and Emacs' Lisp, and subtract what's in common.  That's pretty large
> from my POV.

Does someone really use 5299 packages (or any number close to that) in
the same Lisp program?

IOW, why should I care how many unused packages lie around on my disk?
As long as I don't have them in my program, they will never cause any
trouble.

> I'd rather ask is that a good enough solution?  Can we do better?  Is it
> worth the effort?  Can it be done in the first place?  Does it perhaps
> enable additional things?

All good questions, but they must be considered in some relevant
context, IMO.  "Good enough solution" for what problems and what
purposes?

Emacs is enormously large.  Already there's not a single person who
can be familiar with everything we have, even in the core packages,
like subr.el, simple.el, etc.  Or even come close to that.  Myself and
Lars are learning something new almost every day.  We definitely miss
some problems that get introduced and we are definitely doing an
imperfect job of keeping Emacs clean, due to this enormity alone.
Making Emacs even larger and more complex in these conditions needs
very good reasons in my book.  So the question of need are very
pertinent, IMO.

Re: Help sought understanding shorthands wrt modules/packages

[Stefan Monnier]

> The reason why I think shorthands don't address that problem is pretty
> simple, according to me again: The underlying one obarray can only
> contain one name per symbol at a time.  So, whatever name lands in the

I thought CL packages had the same problem: you can't (globally) have
two packages with the same name.


        Stefan "Trying hard not to get pulled into this discussion"

Re: Help sought understanding shorthands wrt modules/packages

[Gerd Möllmann]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The reason why I think shorthands don't address that problem is pretty
>> simple, according to me again: The underlying one obarray can only
>> contain one name per symbol at a time.  So, whatever name lands in the
>
> I thought CL packages had the same problem: you can't (globally) have
> two packages with the same name.

That's true, in this case that's a problem with package names.  I guess
everything having a name can conflict with the same name being used by
someone else.  But,

- You can handle it with rename-package.  I've used that in CMUCL, while
  working its CLOS implementation), to load two versions of PCL in the
  same image at the same time.

- There is an extension "hierarchical packages" which gives packages a
  tree structure.  It's more or less like with Java packages.  An
  implementation on top of standard packages isn't very hard.
  https://cmucl.org/docs/cmu-user/html/Introduction-To-Hierarchical-Packages.html

>         Stefan "Trying hard not to get pulled into this discussion"

C'mon :-)

Re: Is there a need for a module system

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

> Does someone really use 5299 packages (or any number close to that) in
> the same Lisp program?
>
> IOW, why should I care how many unused packages lie around on my disk?
> As long as I don't have them in my program, they will never cause any
> trouble.

As a user you don't care.

A program/programming environment allowing the use of 5000 packages
should care, IMO.  As good as it goes.

>> I'd rather ask is that a good enough solution?  Can we do better?  Is it
>> worth the effort?  Can it be done in the first place?  Does it perhaps
>> enable additional things?
>
> All good questions, but they must be considered in some relevant
> context, IMO.  "Good enough solution" for what problems and what
> purposes?
>
> Emacs is enormously large.  Already there's not a single person who
> can be familiar with everything we have, even in the core packages,
> like subr.el, simple.el, etc.  Or even come close to that.  Myself and
> Lars are learning something new almost every day.  We definitely miss
> some problems that get introduced and we are definitely doing an
> imperfect job of keeping Emacs clean, due to this enormity alone.

You're doing a good job, no complaints from me.  We're all mere humans.

> Making Emacs even larger and more complex in these conditions needs
> very good reasons in my book.  So the question of need are very
> pertinent, IMO.

A modules solution doesn't have to be gigantic.  If you take
features/pkg as an example, you'll see that it's small compared to other
stuff.  Ok, it's not complete and whatnot, and maybe never will because
it's an experiment after all, but there is nothing missing that will
require another umpteen lines of code.  And I find it not that
complicated either, at least I tried to make it so.

Re: Help sought understanding shorthands wrt modules/packages

[Stefan Monnier]

Gerd Möllmann [2022-10-30 14:48:20] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>> The reason why I think shorthands don't address that problem is pretty
>>> simple, according to me again: The underlying one obarray can only
>>> contain one name per symbol at a time.  So, whatever name lands in the
>>
>> I thought CL packages had the same problem: you can't (globally) have
>> two packages with the same name.
>
> That's true, in this case that's a problem with package names.  I guess
> everything having a name can conflict with the same name being used by
> someone else.

Indeed, the standard ELisp coding style is to always type the full
`<PKGNAME>-<LOCAL>` name.  With CL packages, the full names look like
`<PKGNAME>:<LOCAL>` instead but are otherwise similarly forced to
be unique.

The main benefit of CL packages is that you get to skip the `<PKG>:`
prefix, and/or that you can locally use a different (e.g. shorter) prefix.

The shorthands thingy doesn't let you skip the prefix altogether but
does let you locally use shorter prefixes.

> But,
>
> - You can handle it with rename-package.  I've used that in CMUCL, while
>   working its CLOS implementation), to load two versions of PCL in the
>   same image at the same time.

Indeed you get to manipulate your namespace more directly (you can also
add global package aliases).

Another benefit is that there's a clear "inside" and "outside"
of a package, whereas with ELisp's convention there's no immediate
downside for authors to define a few vars&functions outside of their
namespace-prefix.

> - There is an extension "hierarchical packages" which gives packages a
>   tree structure.  It's more or less like with Java packages.  An
>   implementation on top of standard packages isn't very hard.
>   https://cmucl.org/docs/cmu-user/html/Introduction-To-Hierarchical-Packages.html

Note that those still need to be globally unique, so it just pushes the
problem a bit further.  To solve it "for real", you can use something
similar but where all your names a "relative".
IIRC https://people.cs.uchicago.edu/~blume/papers/cm-TOPLAS.pdf
is such an example.


        Stefan

Re: Help sought understanding shorthands wrt modules/packages

[Helmut Eller]

On Sun, Oct 30 2022, Gerd Möllmann wrote:

> So, I guess my first question is: does anyone here think shorthands are
> a substitute got packages, and if so could you please share your thought
> why, and how one would use them for that?  If not, it might also be
> helpful to know why not.

I think that shorthands do reduce the visual clutter in the source text
a bit.  And that's about all they do.

CL-like packages also provide runtime data structures to
export/import/inherit symbols between packages.  Obviously shorthands
can't do that.

In CL, it's also possible to create "empty" packages, i.e., packages
that don't import symbols from the CL package.  With this, one can
create a private function +.  That's also not possible with shorthands.

> And my second question os: Because I don't get it, I'm considering the
> possibility that shorthands are still incomplete, and some features are
> missing that would make it a substitute for a module system.  Something
> that's not documented anywhere, apparently.  If so, what is it?  Or
> where is it described?  Or does someone have an idea?  You know what I
> mean.

One thing that was talked about in the past is something like "local
shorthands".  I think the idea is that a certain prefix is only active
in a certain region of the text.  Preferably delimited by an sexp (and
not by special tokens as IN-PACKAGE in CL).  Or maybe it was about doing
something like namespace defaulting in XML.

I don't know if this is documented anywhere.  My impression was that
shorthands appeared out of the blue.  I also had the impression that
nobody liked the name "shorthands" much; or maybe that was just me :-).

[...]
> Or am I going nuts and overlooking the obvious?

Well, its pretty obvious that RMS doesn't like CL packages.

The other obvious problem are Lisp macros: macros make it darn hard to
introduce a "real" module system as in Python or Haskell.  I think the
options are:

 1. continue to use prefixes
 2. adopt half measures like shortands
 3. add namespaces to symbols, like CL or XML; keeps macros simple, but
    doesn't support renaming or aliases
 4. introduce hygienic macros as in Scheme

I think 4. is technically too difficult.

There may also be technical problems with 3., but the CL community
probably knows them and the workarounds.  But I think 3. will not happen
mainly for political/ideological reasons.

Helmut

Re: Help sought understanding shorthands wrt modules/packages

[Gerd Möllmann]

Thank you very much for your point of view, Helmut!

> There may also be technical problems with 3., but the CL community
> probably knows them and the workarounds.  

Yes to both.

But let me add, as notorious trouble-maker, that shorthands consist of 
problems only.  The fact that no-one seems to want to ask why I think 
that, I interpret as meaning that this is a well-known fact :-).

> But I think 3. will not happen
> mainly for political/ideological reasons.

Let's see.  I hope that ideology doesn't play a role in Emacs, and 
Richard simply didn't have the necessary information to change his mind.

Re: Help sought understanding shorthands wrt modules/packages

[Gerd Möllmann]

> IIRC https://people.cs.uchicago.edu/~blume/papers/cm-TOPLAS.pdf
> is such an example.

Thanks for the link.  That looks like an interesting read.

Re: Help sought understanding shorthands wrt modules/packages

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 559 bytes --]

On Mon, Oct 31, 2022 at 7:28 AM Gerd Möllmann <gerd.moellmann@gmail.com>
wrote:

>
>
But let me add, as notorious trouble-maker, that shorthands consist of
> problems only.  The fact that no-one seems to want to ask why I think
> that, I interpret as meaning that this is a well-known fact :-).
>

Please elaborate.


> Let's see.  I hope that ideology doesn't play a role in Emacs, and
> Richard simply didn't have the necessary information to change his mind.
>

Ideology was one motivating factor to choose Bazar and not git, back then.

[-- Attachment #2: Type: text/html, Size: 1198 bytes --]

Re: Help sought understanding shorthands wrt modules/packages

[Gerd Möllmann]

Juanma Barranquero <lekktu@gmail.com> writes:

> On Mon, Oct 31, 2022 at 7:28 AM Gerd Möllmann <gerd.moellmann@gmail.com> wrote:
>
>   
>  But let me add, as notorious trouble-maker, that shorthands consist of 
>  problems only.  The fact that no-one seems to want to ask why I think 
>  that, I interpret as meaning that this is a well-known fact :-).
>
> Please elaborate.

Hi Juanma, glad you asked :-).

The basic problem is that the shorthands are 100% ephemeral, and are
not managed in any way.

Example from the Info manual

     ;; read-symbol-shorthands: (("snu-" . "some-nice-string-utils-"))

The "snu-" definition is not recorded anywhere, nothing is preventing
someone else from also using "snu-", or warning about it.  You can't
find out that "snu-" was used to read it from looking at a symbol, or if
an abbreviation was used at all, which is in my eyes unlispy, because it
allows no introspection.

Add to that that read-file-names can be bound locally, by developers and
users, as they see fit (file-local, buffer-local, dynamically).  Now we
have a whole new type of possible conflicts, in addition to what we
already have.

And good look with xref, grep and alike.

Does that make sense?

>  Let's see.  I hope that ideology doesn't play a role in Emacs, and 
>  Richard simply didn't have the necessary information to change his mind.
>
> Ideology was one motivating factor to choose Bazar and not git, back then. 

Ok, that's true.

Although, I don't really see how GNU has any stakes in Lisp standards.
Maybe with the Guile thing that cropped up 20 years ago, but I'd hope
that that's no longer an issue.

But who knows?  Hm, thinking of this--wouldn't it be a good idea to
integrate ECL with Emacs?  Hm :-).

Re: Help sought understanding shorthands wrt modules/packages

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 326 bytes --]

On Mon, Oct 31, 2022 at 1:57 PM Gerd Möllmann <gerd.moellmann@gmail.com>
wrote:


> Does that make sense?
>

Yep.

BTW, I really hope your CL packages' implementation is eventually accepted.


> Maybe with the Guile thing that cropped up 20 years ago, but I'd hope
> that that's no longer an issue.
>

Yeah.

[-- Attachment #2: Type: text/html, Size: 851 bytes --]

Re: Help sought understanding shorthands wrt modules/packages

[Stefan Monnier]

> And good look with xref, grep and alike.

FWIW, I think these are about equally problematic for *all*
namespace/module systems.
In my book, they are the strongest argument for the status quo.


        Stefan

Re: Help sought understanding shorthands wrt modules/packages

[Ag Ibragimov]

I have a question slightly outside of the conversation.

I tried recently using shorthands, and I liked it. But then I realized,
if I'm to use them in a package, it won't be compatible with older
versions of Emacs, right?

Can I somehow use the shorthands and still make the package work in
let's say Emacs 27?

Re: Help sought understanding shorthands wrt modules/packages

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 2623 bytes --]

Hi Gerd, I'm there one who implemented shorthands in Emacs, and i sure
don't think they are a substitute for CL packages.

They can solve some namespacing problems and they can also offer some
convenience for library writing (see github.com/joaotavora/beardbolt for an
example). But they're infinitely inferior to proper CL packages, especially
with the package-local nickname extension.

It's "what could be arranged" to use a silly portuguesism.

João

On Sun, Oct 30, 2022, 06:46 Gerd Möllmann <gerd.moellmann@gmail.com> wrote:

> Since Richard is swamped with mails, I'd like to try and ask some
> questions here.
>
> To start with---I have no problem understanding the shorthands
> mechanism, that's easy enough.  (To be honest, I have to admit I don't
> like the idea, for reasons I won't go into here, but that's another
> story.  A mechanism will find its uses, I guess :-).
>
> My problem is understanding why anyone would think shorthands are an
> alternative to or substitute for packages.  I mean in principle, and
> regardless of packages being Common Lisp packages or any other form of
> module system.  I simply can't get it.
>
> So, I guess my first question is: does anyone here think shorthands are
> a substitute got packages, and if so could you please share your thought
> why, and how one would use them for that?  If not, it might also be
> helpful to know why not.
>
> And my second question os: Because I don't get it, I'm considering the
> possibility that shorthands are still incomplete, and some features are
> missing that would make it a substitute for a module system.  Something
> that's not documented anywhere, apparently.  If so, what is it?  Or
> where is it described?  Or does someone have an idea?  You know what I
> mean.
>
> Finally, here's why I don't get the idea:
>
> According to me, shorthands don't address the problem
> packages/modules/namespaces are trying to do deal with _at all_, which
> is programming in the large, which means that large programs with many
> and diverse contributors suffer from name conflicts.
>
> The reason why I think shorthands don't address that problem is pretty
> simple, according to me again: The underlying one obarray can only
> contain one name per symbol at a time.  So, whatever name lands in the
> obarray must be unique, as it always was.  And this of course leads to
> conflicts.  Whether or not the unique name can otherwise be used in
> abbreviated form or not doesn't change anything in this regard.
>
> Or am I going nuts and overlooking the obvious?
>
>
>
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 3267 bytes --]

Re: Help sought understanding shorthands wrt modules/packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > But let me add, as notorious trouble-maker, that shorthands consist of 
  > problems only.

If you say that, I guess you don't see how they are useful.

I would like to write to explain it, but I can't do that just now.  I
am far more overloaded than usual.  I am backlogged by about 1500
emails MORE than I was backlogged 10 days ago, and struggling to deal
with that.

In addition, I have to write some important things to post about
bigger issues, such as the risky and harmful idea replacing the
standard GPL license notice with an SPDX tag.

There is no hurry about this issue, so please stop pressuring people
to rush it.  Let's wait a couple of weeks, so that I have time to read
the article about CL packages that you refered to, and then write
something about it.

  > Let's see.  I hope that ideology doesn't play a role in Emacs,

"Ideology" is a derogatory term for "general political stance which is
used as the basis for prsctical decisions."  The GNU Project is part
of the free software movement, which holds a stance of that kind which
can properly play a role in any of our decisions.  When you talk about that,
please don't use derogatory words.

I don't see that it directly enters into this issue, though.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Help sought understanding shorthands wrt modules/packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

This code comes from Fload.

  /* If file name is magic, call the handler.  */
  handler = Ffind_file_name_handler (file, Qload);
  if (!NILP (handler))
    return call5 (handler, Qload, file, noerror, nomessage, nosuffix);

Is it a bug that it does not pass along the argument must_suffix?
It looks that way.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Help sought understanding shorthands wrt modules/packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I just wrote changes which ought to provide the missing
functionality in the shorthands feature.  It adds an argument to Fload
which specifies additional shorthands to apply to that file.
See below.

I have made it compile but I have not had a chance yet to test running
it.  I hope to find a chance to do that soon.

This is intended to make it possible to have a file strings.el which
loads the file s.el (unmodified) and renames all the functions in it
so that they start with `string-' (or whatever prefix we like).
We could then include `strings.el' in the standard Emacs Lisp namespace
and document its facilities in the Emacs Lisp Reference Manual.

Because this avoids modifying s.el, we would not have to maintain
our own version of that file.  That would be good for cooperating with
its developer.


diff --git a/src/lread.c b/src/lread.c
index dfa4d9afb51..abd818cfa5a 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -151,7 +151,7 @@ static int read_emacs_mule_char (int, int (*) (int, Lisp_Object),
 
 static void readevalloop (Lisp_Object, struct infile *, Lisp_Object, bool,
                           Lisp_Object, Lisp_Object,
-                          Lisp_Object, Lisp_Object);
+                          Lisp_Object, Lisp_Object, Lisp_Object);
 
 static void build_load_history (Lisp_Object, bool);
 
@@ -1170,7 +1170,7 @@ loadhist_initialize (Lisp_Object filename)
   specbind (Qcurrent_load_list, Fcons (filename, Qnil));
 }
 
-DEFUN ("load", Fload, Sload, 1, 5, 0,
+DEFUN ("load", Fload, Sload, 1, 6, 0,
        doc: /* Execute a file of Lisp code named FILE.
 First try FILE with `.elc' appended, then try with `.el', then try
 with a system-dependent suffix of dynamic modules (see `load-suffixes'),
@@ -1206,6 +1206,13 @@ When searching suffixes, this function normally stops at the first
 one that exists.  If the option `load-prefer-newer' is non-nil,
 however, it tries all suffixes, and uses whichever file is the newest.
 
+ADD-SHORTHANDS specifies a list of additional shorthands (symbol
+renamings) to perform when loading this file,
+in addition to those specified in the file itself.
+Normally you should call `load-with-shorthands' to specify this,
+rather than calling `load' with nil for all args
+except for FILE and ADD-SHORTHANDS.
+
 Loading a file records its definitions, and its `provide' and
 `require' calls, in an element of `load-history' whose
 car is the file name loaded.  See `load-history'.
@@ -1216,7 +1223,7 @@ is bound to the file's name.
 
 Return t if the file exists and loads successfully.  */)
   (Lisp_Object file, Lisp_Object noerror, Lisp_Object nomessage,
-   Lisp_Object nosuffix, Lisp_Object must_suffix)
+   Lisp_Object nosuffix, Lisp_Object must_suffix, Lisp_Object add_shorthands)
 {
   FILE *stream UNINIT;
   int fd;
@@ -1236,7 +1243,8 @@ Return t if the file exists and loads successfully.  */)
   /* If file name is magic, call the handler.  */
   handler = Ffind_file_name_handler (file, Qload);
   if (!NILP (handler))
-    return call5 (handler, Qload, file, noerror, nomessage, nosuffix);
+    return call7 (handler, Qload, file, noerror, nomessage,
+		  nosuffix, must_suffix, add_shorthands);
 
   /* The presence of this call is the result of a historical accident:
      it used to be in every file-operation and when it got removed
@@ -1325,7 +1333,8 @@ Return t if the file exists and loads successfully.  */)
       else
 	handler = Ffind_file_name_handler (found, Qload);
       if (! NILP (handler))
-	return call5 (handler, Qload, found, noerror, nomessage, Qt);
+	return call6 (handler, Qload, found, noerror, nomessage, Qt,
+		      add_shorthands);
 #ifdef DOS_NT
       /* Tramp has to deal with semi-broken packages that prepend
 	 drive letters to remote files.  For that reason, Tramp
@@ -1480,9 +1489,11 @@ Return t if the file exists and loads successfully.  */)
 	      emacs_close (fd);
 	      clear_unwind_protect (fd_index);
 	    }
-	  val = call4 (Vload_source_file_function, found, hist_file_name,
+	  val = call5 (Vload_source_file_function, found, hist_file_name,
 		       NILP (noerror) ? Qnil : Qt,
-		       (NILP (nomessage) || force_load_messages) ? Qnil : Qt);
+		       (NILP (nomessage) || force_load_messages) ? Qnil : Qt,
+		       add_shorthands);
+	  /* ???  Fix  ‘load-with-code-conversion’.  */
 	  return unbind_to (count, val);
 	}
     }
@@ -1557,6 +1568,7 @@ Return t if the file exists and loads successfully.  */)
 
   if (is_module)
     {
+      /* ??? Does this require fixing for shorthands?  */
 #ifdef HAVE_MODULES
       loadhist_initialize (found);
       Fmodule_load (found);
@@ -1570,6 +1582,7 @@ Return t if the file exists and loads successfully.  */)
     {
 #ifdef HAVE_NATIVE_COMP
       loadhist_initialize (hist_file_name);
+      /* ??? Does this require fixing for shorthands?  */
       Fnative_elisp_load (found, Qnil);
       build_load_history (hist_file_name, true);
 #else
@@ -1585,14 +1598,14 @@ Return t if the file exists and loads successfully.  */)
 
       if (! version || version >= 22)
         readevalloop (Qget_file_char, &input, hist_file_name,
-                      0, Qnil, Qnil, Qnil, Qnil);
+                      0, Qnil, Qnil, Qnil, Qnil, add_shorthands);
       else
         {
           /* We can't handle a file which was compiled with
              byte-compile-dynamic by older version of Emacs.  */
           specbind (Qload_force_doc_strings, Qt);
           readevalloop (Qget_emacs_mule_file_char, &input, hist_file_name,
-                        0, Qnil, Qnil, Qnil, Qnil);
+                        0, Qnil, Qnil, Qnil, Qnil, add_shorthands);
         }
     }
   unbind_to (count, Qnil);
@@ -1633,7 +1646,8 @@ save_match_data_load (Lisp_Object file, Lisp_Object noerror,
 {
   specpdl_ref count = SPECPDL_INDEX ();
   record_unwind_save_match_data ();
-  Lisp_Object result = Fload (file, noerror, nomessage, nosuffix, must_suffix);
+  Lisp_Object result
+    = Fload (file, noerror, nomessage, nosuffix, must_suffix, Qnil);
   return unbind_to (count, result);
 }
 \f
@@ -2166,7 +2180,9 @@ readevalloop_eager_expand_eval (Lisp_Object val, Lisp_Object macroexpand)
    READFUN, if non-nil, is used instead of `read'.
 
    START, END specify region to read in current buffer (from eval-region).
-   If the input is not from a buffer, they must be nil.  */
+   If the input is not from a buffer, they must be nil.
+
+   ADD_SHORTHANDS is copied from the argument to `load' of the same name.  */
 
 static void
 readevalloop (Lisp_Object readcharfun,
@@ -2174,7 +2190,7 @@ readevalloop (Lisp_Object readcharfun,
 	      Lisp_Object sourcename,
 	      bool printflag,
 	      Lisp_Object unibyte, Lisp_Object readfun,
-	      Lisp_Object start, Lisp_Object end)
+	      Lisp_Object start, Lisp_Object end, Lisp_Object add_shorthands)
 {
   int c;
   Lisp_Object val;
@@ -2225,6 +2241,7 @@ readevalloop (Lisp_Object readcharfun,
 	    (NILP (lex_bound) || BASE_EQ (lex_bound, Qunbound)
 	     ? Qnil : list1 (Qt)));
   specbind (Qmacroexp__dynvars, Vmacroexp__dynvars);
+  specbind (Qread_symbol_added_shorthands, add_shorthands);
 
   /* Ensure sourcename is absolute, except whilst preloading.  */
   if (!will_dump_p ()
@@ -2414,7 +2431,7 @@ This function preserves the position of point.  */)
   specbind (Qlexical_binding, lisp_file_lexically_bound_p (buf) ? Qt : Qnil);
   BUF_TEMP_SET_PT (XBUFFER (buf), BUF_BEGV (XBUFFER (buf)));
   readevalloop (buf, 0, filename,
-		!NILP (printflag), unibyte, Qnil, Qnil, Qnil);
+		!NILP (printflag), unibyte, Qnil, Qnil, Qnil, Qnil);
   return unbind_to (count, Qnil);
 }
 
@@ -2449,7 +2466,7 @@ This function does not move point.  */)
   /* `readevalloop' calls functions which check the type of start and end.  */
   readevalloop (cbuf, 0, BVAR (XBUFFER (cbuf), filename),
 		!NILP (printflag), Qnil, read_function,
-		start, end);
+		start, end, Qnil);
 
   return unbind_to (count, Qnil);
 }
@@ -4933,43 +4950,65 @@ oblookup_considering_shorthand (Lisp_Object obarray, const char *in,
 				ptrdiff_t size, ptrdiff_t size_byte, char **out,
 				ptrdiff_t *size_out, ptrdiff_t *size_byte_out)
 {
-  Lisp_Object tail = Vread_symbol_shorthands;
+  Lisp_Object tail = Qnil;
+  int stage = 0;
 
   /* First, assume no transformation will take place.  */
   *out = NULL;
-  /* Then, iterate each pair in Vread_symbol_shorthands.  */
-  FOR_EACH_TAIL_SAFE (tail)
+
+  /* Check each of the shorthands now in effect.  */
+  while (1)
     {
-      Lisp_Object pair = XCAR (tail);
-      /* Be lenient to 'read-symbol-shorthands': if some element isn't a
-	 cons, or some member of that cons isn't a string, just skip
-	 to the next element.  */
-      if (!CONSP (pair))
-	continue;
-      Lisp_Object sh_prefix = XCAR (pair);
-      Lisp_Object lh_prefix = XCDR (pair);
-      if (!STRINGP (sh_prefix) || !STRINGP (lh_prefix))
-	continue;
-      ptrdiff_t sh_prefix_size = SBYTES (sh_prefix);
-
-      /* Compare the prefix of the transformation pair to the symbol
-	 name.  If a match occurs, do the renaming and exit the loop.
-	 In other words, only one such transformation may take place.
-	 Calculate the amount of memory to allocate for the longhand
-	 version of the symbol name with xrealloc.  This isn't
-	 strictly needed, but it could later be used as a way for
-	 multiple transformations on a single symbol name.  */
-      if (sh_prefix_size <= size_byte
-	  && memcmp (SSDATA (sh_prefix), in, sh_prefix_size) == 0)
+      /* Do stage 1: iterate each pair in Vread_symbol_added_shorthands.  */
+      if (stage == 0)
 	{
-	  ptrdiff_t lh_prefix_size = SBYTES (lh_prefix);
-	  ptrdiff_t suffix_size = size_byte - sh_prefix_size;
-	  *out = xrealloc (*out, lh_prefix_size + suffix_size);
-	  memcpy (*out, SSDATA(lh_prefix), lh_prefix_size);
-	  memcpy (*out + lh_prefix_size, in + sh_prefix_size, suffix_size);
-	  *size_out = SCHARS (lh_prefix) - SCHARS (sh_prefix) + size;
-	  *size_byte_out = lh_prefix_size + suffix_size;
-	  break;
+	  tail = Vread_symbol_added_shorthands;
+	  stage = 1;
+	}
+      /* Then stage 2: iterate each pair in Vread_symbol_shorthands.  */
+      else if (stage == 1)
+	{
+	  tail = Vread_symbol_shorthands;
+	  stage = 2;
+	}
+      /* There are no more stages after stage 2.  */
+      else
+	break;
+
+      /* Check each of the shorthands of this stage.  */
+      FOR_EACH_TAIL_SAFE (tail)
+	{
+	  Lisp_Object pair = XCAR (tail);
+	  /* Be lenient to 'read-symbol-shorthands': if some element isn't a
+	     cons, or some member of that cons isn't a string, just skip
+	     to the next element.  */
+	  if (!CONSP (pair))
+	    continue;
+	  Lisp_Object sh_prefix = XCAR (pair);
+	  Lisp_Object lh_prefix = XCDR (pair);
+	  if (!STRINGP (sh_prefix) || !STRINGP (lh_prefix))
+	    continue;
+	  ptrdiff_t sh_prefix_size = SBYTES (sh_prefix);
+
+	  /* Compare the prefix of the transformation pair to the symbol
+	     name.  If a match occurs, do the renaming and exit the loop.
+	     In other words, only one such transformation may take place.
+	     Calculate the amount of memory to allocate for the longhand
+	     version of the symbol name with xrealloc.  This isn't
+	     strictly needed, but it could later be used as a way for
+	     multiple transformations on a single symbol name.  */
+	  if (sh_prefix_size <= size_byte
+	      && memcmp (SSDATA (sh_prefix), in, sh_prefix_size) == 0)
+	    {
+	      ptrdiff_t lh_prefix_size = SBYTES (lh_prefix);
+	      ptrdiff_t suffix_size = size_byte - sh_prefix_size;
+	      *out = xrealloc (*out, lh_prefix_size + suffix_size);
+	      memcpy (*out, SSDATA(lh_prefix), lh_prefix_size);
+	      memcpy (*out + lh_prefix_size, in + sh_prefix_size, suffix_size);
+	      *size_out = SCHARS (lh_prefix) - SCHARS (sh_prefix) + size;
+	      *size_byte_out = lh_prefix_size + suffix_size;
+	      break;
+	    }
 	}
     }
   /* Now, as promised, call oblookup with the "final" symbol name to
@@ -5715,6 +5754,13 @@ that are loaded before your customizations are read!  */);
 This variable's value can only be set via file-local variables.
 See Info node `(elisp)Shorthands' for more details.  */);
   Vread_symbol_shorthands = Qnil;
+  DEFVAR_LISP ("read-symbol-added-shorthands", Vread_symbol_added_shorthands,
+          doc: /* Alist of symbol-name shorthands added by `load'.
+This variable's value comes from the ADDED-SHORTHANDS argument to `load'.
+See Info node `(elisp)Shorthands' for more details.  */);
+  Vread_symbol_added_shorthands = Qnil;
+
+  DEFSYM (Qread_symbol_added_shorthands, "read-symbol-added-shorthands");
   DEFSYM (Qobarray_cache, "obarray-cache");
   DEFSYM (Qobarrayp, "obarrayp");
 

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Help sought understanding shorthands wrt modules/packages

[Gerd Möllmann]

João Távora <joaotavora@gmail.com> writes:

> Hi Gerd, I'm there one who implemented shorthands in Emacs, and i sure
> don't think they are a substitute for CL packages.

Thanks for letting me know, Joao. 

Re: Help sought understanding shorthands wrt modules/packages

[Gerd Möllmann]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > But let me add, as notorious trouble-maker, that shorthands consist of 
>   > problems only.
>
> If you say that, I guess you don't see how they are useful.

I think I understand pretty well when they are useful, in the same way
as defvar-alias and such are useful.  Anything going beyond that I
consider potentially harmful.

> I would like to write to explain it, but I can't do that just now.  I
> am far more overloaded than usual.  I am backlogged by about 1500
> emails MORE than I was backlogged 10 days ago, and struggling to deal
> with that.

No problem, take your time.

> There is no hurry about this issue, so please stop pressuring people
> to rush it.  Let's wait a couple of weeks, so that I have time to read
> the article about CL packages that you refered to, and then write
> something about it.

I'm not pressuring anyone intentially.  I don't even know how that could
be.  By the existence of the branch with CL packages?  Or by asking
questions and giving, hopefully rational, arguments?

And even if someone feels pressured, honestly, so what?

And how is that related to your personal time situation, in the first
place?  I at least am not expecting or waiting for some kind of verdict.
I do what I feel like doing to the degree possible.  Take it or leave
it, so to say.

>   > Let's see.  I hope that ideology doesn't play a role in Emacs,
>
> I don't see that it directly enters into this issue, though.

That's good.

Re: Help sought understanding shorthands wrt modules/packages

[Michael Albinus]

Richard Stallman <rms@gnu.org> writes:

Hi Richard,

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> This code comes from Fload.
>
>   /* If file name is magic, call the handler.  */
>   handler = Ffind_file_name_handler (file, Qload);
>   if (!NILP (handler))
>     return call5 (handler, Qload, file, noerror, nomessage, nosuffix);
>
> Is it a bug that it does not pass along the argument must_suffix?
> It looks that way.

Tramp is able to handle the must_suffix argument.

However, ange-ftp-load doesn't know this argument. No problem, we can
fix this by adding the argument to ange-ftp-load, and ignore it
then. Will do.

I suppose this was due to backward compatibility, which isn't needed
anymore for ange-ftp.

Best regards, Michael.

Re: Help sought understanding shorthands wrt modules/packages

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Wed, 02 Nov 2022 23:17:19 -0400
> 
> I just wrote changes which ought to provide the missing
> functionality in the shorthands feature.  It adds an argument to Fload
> which specifies additional shorthands to apply to that file.
> See below.
> 
> I have made it compile but I have not had a chance yet to test running
> it.  I hope to find a chance to do that soon.

Please do test it, but in any case please don't install before the
emacs-29 release branch is cut.  This change is significant enough to
not risk it for the upcoming release of Emacs 29.

> -DEFUN ("load", Fload, Sload, 1, 5, 0,
> +DEFUN ("load", Fload, Sload, 1, 6, 0,
>         doc: /* Execute a file of Lisp code named FILE.
>  First try FILE with `.elc' appended, then try with `.el', then try
>  with a system-dependent suffix of dynamic modules (see `load-suffixes'),
> @@ -1206,6 +1206,13 @@ When searching suffixes, this function normally stops at the first
>  one that exists.  If the option `load-prefer-newer' is non-nil,
>  however, it tries all suffixes, and uses whichever file is the newest.
>  
> +ADD-SHORTHANDS specifies a list of additional shorthands (symbol
> +renamings) to perform when loading this file,
> +in addition to those specified in the file itself.
> +Normally you should call `load-with-shorthands' to specify this,
> +rather than calling `load' with nil for all args
> +except for FILE and ADD-SHORTHANDS.
> +
>  Loading a file records its definitions, and its `provide' and
>  `require' calls, in an element of `load-history' whose
>  car is the file name loaded.  See `load-history'.

This needs a NEWS entry and a suitable change in the ELisp manual.

>  #ifdef HAVE_NATIVE_COMP
>        loadhist_initialize (hist_file_name);
> +      /* ??? Does this require fixing for shorthands?  */
>        Fnative_elisp_load (found, Qnil);
>        build_load_history (hist_file_name, true);

Andrea, can you please comment on the above question?

> +  DEFVAR_LISP ("read-symbol-added-shorthands", Vread_symbol_added_shorthands,
> +          doc: /* Alist of symbol-name shorthands added by `load'.
> +This variable's value comes from the ADDED-SHORTHANDS argument to `load'.
> +See Info node `(elisp)Shorthands' for more details.  */);
> +  Vread_symbol_added_shorthands = Qnil;

This variable should be documented in the ELisp manual and mentioned
in the NEWS entry.

Thanks.

P.S. It would be good to add tests for this new feature; see the test
suite we have in tests/.

A short defense of shorthands.el (but CL packages are still better) (Was: Help sought understanding shorthands wrt modules/packages)

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 4443 bytes --]

On Thu, Nov 3, 2022 at 5:12 AM Gerd Möllmann <gerd.moellmann@gmail.com>
wrote:

    João Távora <joaotavora@gmail.com> writes:

    > Hi Gerd, I'm there one who implemented shorthands in Emacs, and i sure
    > don't think they are a substitute for CL packages.

    Thanks for letting me know, Joao.

No problem.  Allow me to mount a short defense of shorthands.  In doing
so, I don't mean to imply -- by any measure -- that CL packages are a
much welcomed addition to Emacs. They are, absolutely.

* shorthands are designed to perform namespacing operations with minimal
  or even no changes to the Lisp forms of an Elisp file.  The
  hypothetical file x.el:

  (defvar x--bar 42)
  (defun x-foo () x--bar)
  (provide 'x)
  ;; x.el ends here
  ;; Local Variables:
  ;; read-symbol-shorthands: (("x-" . "xenomorph-"))
  ;; End:

  which will pollute the global namespace when loaded into Emacs 27,
  will, when loaded into Emacs >28, intern xenomorph--bar and
  xenomorph-foo instead of x--bar and x-foo.

  Its user file yummy.el

  (require 'x)
  (defun yummy () (x-foo))
  ;; yummy.el ends here
  ;; Local Variables:
  ;; read-symbol-shorthands: (("x-" . "xenomorph-"))
  ;; End:

  can also be loaded into Emacs 27 and Emacs 28.  The interaction
  between the two packages works in both cases, but in Emacs 28 the
  global namespace won't be polluted.

* The above use case was motivated by the s.el, dash.el and f.el
  libraries which incur in this namespace pollution.  To be clear, all
  packages pollute the namespace but these short prefixed ones were
  especially heavy polluters, since short names naturally appear more
  frequently in completion lists.

* If CL packages had been used instead, this "double duty" wouldn't have
  been possible, because x.el and y.el would have to be changed
  considerably (admittedly in a rather straightforward fashion)

* When shorthands were presented, much criticism was leveled at it, some
  stemming from a misunderstanding of the specific problem this attempts
  to solve.

* But one of the criticisms is pretty reasonable: this "breaks grep"
  because the same symbol can now be referenced by two different
  character strings from two different contexts.  Also two different
  symbols are designated by the same character string, again in two
  different contexts.

  However, the same is true for every namespacing facility by
  definition.  This is what namespacing systems do.

 * Anyway, the problem is that

    grep xenomorph-foo x.el yummy.el

   doesn't return anything, even though xenomorph-foo is really
   the name of the symbol in obarray.  Obviously, grep doesn't
   understand ELisp.

* In my opinion, the part that is missing from shorthands is a tool that
  replaces grep (for Lisp symbolic uses, of course) and understands and
  can be used as a backend for xref-find-references.  One can think
  of different approaches for realizing this tool

* The most promising approach, IMO, to fix this is to create a new
  binary program, call it 'sexgrep' (for "Symbolic Expression Grep")
  which can be run separately from Emacs but which uses Emacs's reader
  syntax.  It could reuse lread.c maybe, or reimplement relevant parts
  of it.  The program's input is a full symbol name and a number of
  Elisp files.  By using a source-tracking reader and understanding the
  relatively simple syntax of shorthand definitions, it'll make

    sexpgrep xenomorph-foo yummy.el x.el

  find matches for the search pattern in those two files (on line 2)
  respectively.

* Taking the approach with CL packages would be more difficult, because
  the program would need to have a Lisp evaluator that understands
  CL:DEFPACKAGE and CL:IN-PACKAGE, not just a reader.  This is why, in
  the Common Lisp implementations that I've used, a global in-memory
  database of symbols is used instead.  The database is kept up-to-date
  whenever the code is read and loaded (which may occur in two different
  moments in time, if compiled files are used).

  This also works nicely (SLIME and SLY use it to great effect).  But it
  fails to search code that isn't loaded and is subject to some annoying
  but resolvable problems (like when loading CL fasls that were compiled
  on a different machine, for example.  Maybe Helmut Eller, SLIME
  author, has some good input in this regard)

[-- Attachment #2: Type: text/html, Size: 5007 bytes --]

Re: A short defense of shorthands.el (but CL packages are still better) (Was: Help sought understanding shorthands wrt modules/packages)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Does shorthands break grep?  Let's look at a few cases.

* The case you brought up.

    > (defvar x--bar 42)
    > (defun x-foo () x--bar)
    > (provide 'x)
    > ;; x.el ends here
    > ;; Local Variables:
    > ;; read-symbol-shorthands: (("x-" . "xenomorph-"))
    > ;; End:

    > Its user file yummy.el

    > (require 'x)
    > (defun yummy () (x-foo))
    > ;; yummy.el ends here
    > ;; Local Variables:
    > ;; read-symbol-shorthands: (("x-" . "xenomorph-"))
    > ;; End:

If that's what files x.el and yummy.el contain, 
I don't think anyone will grep for `xenomorph-foo'.
The code in yummy.el actually says `x-foo', so that
is what someone will grep for, and grep will find that in x.el.
So I don't think this case causes any problem with grep.

* A user program foo.el requires strings.el.

Consider strings.el and s.el, strings.el defines strings- aliases for
the functions in s.el by loading s.el with a renaming.

foo.el should call functions named `strings-foo', etc.,
But strings.el does not explicitly contain those names.
So grep will work properly on foo.el
but fail on strings.el but not on foo.el.

We can fix that by adding to strings.el
text such as

    (ignore ;;This is for grep to find.  The definitions are in s.el.
    (defun strings-foo ())
    )

so that grep will find those function names there, and the user
will see to look in s.el for the definitions.

* Something like what CL packages do.

Suppose you want a file foo.el to refer to functions in the frob "package"
while eliding its name prefix, `frob:'.  For instance, `activate' in foo.el
would stand for `frob:activate'.

foo.el can have a shorthand to rename `activate' to `frob:activate'.
If you grep for `activate', grep will find uses of it in foo.el,
but also that renaming in foo.el.

This case does not pose a problem for grep.  When you see the
renaming of `activate' to `frob:activate' in foo.el, you will grep
for `frob:activate' and find that in the frob "package".

This does the same job as CL packages except that it avoids
heuristics and search paths.  Renamings are specified explicitly
rather than found due to the existence or nonexistence of
certain symbols in various packages.



-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: A short defense of shorthands.el (but CL packages are still better)

[João Távora]

Richard Stallman <rms@gnu.org> writes:

> If that's what files x.el and yummy.el contain, 
> I don't think anyone will grep for `xenomorph-foo'.
> The code in yummy.el actually says `x-foo', so that
> is what someone will grep for, and grep will find that in x.el.
> So I don't think this case causes any problem with grep.

As you know, he underlying interned symbol is xenomorph-foo, and that's
the symbol that the at-point documentation system ElDoc shows you in the
echo area, and that's what shows up C-h f.

But I see what you mean: in this particular example, we're lucky enough
that both yummy.el and x.el define the same shorthand.  So it wasn't a
good example.  But we could have a perfectly reasonable z.el that
defines:

   ;; read-symbol-shorthands: (("xeno-" . "xenomorph-"))

And here, xeno-foo now means the very same symbol xenomorph-foo.  And
this "breaks grep".  Not literally, of course just in that it undermines
the user's ability to use grep to know where in source code the Lisp
symbol xenomorph-foo is mentioned.

My point is that this is common to all namespacing systems, across every
language.  Although in different languages and systems we may have
different ways to work around the problem.

Re: Help sought understanding shorthands wrt modules/packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Hi Gerd, I'm there one who implemented shorthands in Emacs, and i sure
  > > don't think they are a substitute for CL packages.

To ask whether something is a "substitute" for XYZ
presumes that XYZ is a good thing and does something useful.

If CL packages still have the misfeature of searching a list of
packages for one that has a symbol 'foobar' in it, and deciding
what `foobar' in your code means based on that,
then they are inexcusable bad design and we must not implement them.

If they no longer have that misfeature, maybe they are ok.

If you want `foo', when it occurs in certain files,
to refer to the symbol `hack:foo', the best way to do this
is to use a shorthand.  If CL doesn't have that search-list flaw,
maybe it should be implemented using shorthands as the underlying
mechanism.



-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Help sought understanding shorthands wrt modules/packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Please do test it, but in any case please don't install before the
  > emacs-29 release branch is cut.  This change is significant enough to
  > not risk it for the upcoming release of Emacs 29.

That is unreasonable.  People have been installing very large changes
just lately.  This is an added small feature that won't affect any code
unless it explicitly uses this, which currently nothing does.

Since it finishes an existing feature and makes it usable for the most
important case, we should definitely install it before Emacs 29.


I now have tested it -- a simple case now works.  If someone tells me
how to install the s package from NonGNU ELPA, I could test that too.
(I have never installed anything from ELPA.)

I wrote text for etc/NEWS:

    ** Shorthands for loading a file can be specified from "outside".

    The new function `load-with-shorthands' loads a file
    and specifies additional shorthands for reading it.

I wrote this simple interface function in shorthands.el.

    (defun load-with-shorthands (file add-shorthands)
      "Load FILE adding the additional shorthands in ADD-SHORTHANDS.
    ADD-SHORTHANDS should be an alist of symbol renamings to apply
    when loading FILE; see Info node `(elisp)Shorthands' for more details.
    These shorthands act before the shorthands specified in the file itself."
      (load file nil nil nil nil add-shorthands))

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Help sought understanding shorthands wrt modules/packages

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: akrl@sdf.org, emacs-devel@gnu.org
> Date: Sat, 05 Nov 2022 12:49:27 -0400
> 
>   > Please do test it, but in any case please don't install before the
>   > emacs-29 release branch is cut.  This change is significant enough to
>   > not risk it for the upcoming release of Emacs 29.
> 
> That is unreasonable.  People have been installing very large changes
> just lately.

No, they haven't, not lately anyway.

> This is an added small feature that won't affect any code
> unless it explicitly uses this, which currently nothing does.

It is a minor feature, indeed, but its effect on low-level code is
quite significant and could potentially destabilize Emacs.  I would
like to have a pretest period for Emacs 29 that is as short as
possible, and that means last-minute non-trivial changes in low-level
code are out.

> Since it finishes an existing feature and makes it usable for the most
> important case, we should definitely install it before Emacs 29.

I disagree.  Please wait with installing until the release branch is
cut, it will happen in a couple of weeks.

Re: Help sought understanding shorthands wrt modules/packages

[Eduardo Ochs]

On Sat, 5 Nov 2022 at 13:50, Richard Stallman <rms@gnu.org> wrote:
>
> I now have tested it -- a simple case now works.  If someone tells me
> how to install the s package from NonGNU ELPA, I could test that too.
> (I have never installed anything from ELPA.)

Hi Richard,

I sometimes run emacs -Q in empty/fake home directories, with:

  rm -Rv /tmp/fake-home-dir/
  mkdir  /tmp/fake-home-dir/
  cd     /tmp/fake-home-dir/
  HOME=$PWD emacs -Q

and then I install packages by executing some of the sexps below with
my favorite variant of `C-e C-x C-e':

  (require 'package)
  (package-initialize)
  (add-to-list 'package-archives '("nongnu" .
"https://elpa.nongnu.org/nongnu/"))
  (add-to-list 'package-archives '("melpa"  . "https://melpa.org/packages/"))
  (package-refresh-contents)
  (package-install 's)
  (describe-package 's)

Notes:

  1) If you don't want NonGNU ELPA ou MELPA just skip the
     corresponding `add-to-list's,

  2) `(package-refresh-contents)' usually takes several seconds to run,

  3) The usual instructions for installing packages are like this:

       (info "(emacs)Package Menu")

Buenas Salenas,
  Eduardo Ochs
  http://angg.twu.net/eepitch.html

Re: Help sought understanding shorthands wrt modules/packages

[Michael Albinus]

Richard Stallman <rms@gnu.org> writes:

Hi Richard,

>   > Please do test it, but in any case please don't install before the
>   > emacs-29 release branch is cut.  This change is significant enough to
>   > not risk it for the upcoming release of Emacs 29.
>
> That is unreasonable.  People have been installing very large changes
> just lately.  This is an added small feature that won't affect any code
> unless it explicitly uses this, which currently nothing does.
>
> Since it finishes an existing feature and makes it usable for the most
> important case, we should definitely install it before Emacs 29.

Handling of shorthands is implemented in lread.c. File name handlers for
`load' don't know (yet) what to do.

Should the file name handlers ignore shorthands? Or could you provide
the shorthands implementation in lread.c as a Lisp function, which could
be called from the handlers? Or do you expect the file name handlers to
implement shorthands handling on their own?

> I wrote text for etc/NEWS:
>
>     ** Shorthands for loading a file can be specified from "outside".
>
>     The new function `load-with-shorthands' loads a file
>     and specifies additional shorthands for reading it.

Please use APOSTROPHE only in etc/NEWS, and not a GRAVE ACCENT.

Best regards, Michael.

Re: Help sought understanding shorthands wrt modules/packages

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1963 bytes --]

On Sun, Nov 6, 2022 at 9:06 AM Michael Albinus <michael.albinus@gmx.de>
wrote:

> > I wrote text for etc/NEWS:
> >
> >     ** Shorthands for loading a file can be specified from "outside".
> >
> >     The new function `load-with-shorthands' loads a file
> >     and specifies additional shorthands for reading it.

I think it's fair to say, as the person who originally implemented
shorthands,
that I don't agree with this change.

Providing two different ways to load an .el file so that each interns
different
symbols into obarray would lead to unfortunate consequences where it would
be hard or impossible for humans or programs to understand the provenance of
a given symbol.

I cannot understand Richard's assertion that "shorthands are incomplete".
Shorthands were primarily designed to solve the "the s.el problem", and the
current
implementation available since Emacs 28 solves that problem already, IMO.

I've described how to do so multiple times in the past, but here goes again:

Anyway, here are the instructions again summarized:

1. Grab s.el from its upstream repository
2. Don't touch any of its code.  Just add this local variable
   definition to the end of the file.

   ;; Local Variables:
   ;; read-symbol-shorthands: (("s-" . "magnars-string-"))
   ;; End:

3. Rename the file from s.el to magnars-string.el.  Change its line

     (provide 's)

   to

     (provide 'magnars-string)

4. Include magnars-string.el in Emacs core or GNU/NonGNU Elpa
5. Tell the current users of s.el, say im-a-s-user.el, in the world to type

     (require 'magnars-string)

   instead of

     (require 's)

6. In the local variables footer of im-a-s-user.el, also add

   ;; Local Variables:
   ;; read-symbol-shorthands: (("s-" . "magnars-string-"))
   ;; End:

That's it.  Minimal code was changed, and im-a-s-user.el continues to work
as
previously, except that the "namespace pollution" of many symbols with a
very
short prefix is not present anymore.

[-- Attachment #2: Type: text/html, Size: 2762 bytes --]

Re: Help sought understanding shorthands wrt modules/packages

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 693 bytes --]

On Sat, Nov 5, 2022 at 3:13 AM Richard Stallman <rms@gnu.org> wrote:

> If CL packages still have the misfeature of searching a list of
> packages for one that has a symbol 'foobar' in it, and deciding
> what `foobar' in your code means based on that,
> then they are inexcusable bad design and we must not implement them.
>
> If they no longer have that misfeature, maybe they are ok.

If you're talking about the :USE directive, you don't have to employ it:
it's not
mandatory for CL packages to be immensely better. But it's very useful  and
convenient in specific, well-understood situations.  If you're talking
about
something else, I don't know what it might me.

João

[-- Attachment #2: Type: text/html, Size: 925 bytes --]

Re: A short defense of shorthands.el (but CL packages are still better)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > As you know, he underlying interned symbol is xenomorph-foo, and that's
  > the symbol that the at-point documentation system ElDoc shows you in the
  > echo area, and that's what shows up C-h f.

I've never used ElDoc and I don't know what it does.  Maybe you're
right about C-h f, but you have not specified the scenario fully.
What does the user actually type, to invoke it?

Perhaps the user type C-h f x-foo RET, after seeing `x-foo' in the
code and gets the answer that it doesn't exist.  That would be an
unhelpful result.

So maybe each file's shorthands, as actually used when it was loaded,
should be remembered in an alist, so that C-h f and friends can look
them up and DTRT.

As for the grep shortcoming, it can't be addressed by changing the code
of grep.  But I have an idea for how coding conventions can do it.

We can identify the usual scenarios in which shorthands affect grepping,
and write dummy definitions to deal with them.

For instance, in the file string.el, we can have dummy definitions for all
the renamings that it does.

;;; (defalias 'string-foo  's-foo)  ;; in s.el
;;; (defalias 'string-bar  's-bar)  ;; in s.el

Grepping for string-foo will find that line, which will tell you to
look at s.el and find s-foo.

We could conceivably make the commands for finding that grep hit
understand that and go straight to the right line in s.el.


In the case of doing something like CL packages, where the shorthands
look like ("frobnicate" . "frob:frobnicate") and ("scramble"
. "frob:scramble"), we may not need to do anything.  If you see a call
to `frobnicate' and grep for `frobnicate', you will find the
definition of `frob:frobnicate', as well as perhaps a few other
functions named frobnicate in various other packages.

You will need to study the code to see which of those functions is
actually used in the program you are studying.

I think the same grep issue will occur with actual CL packages, or
anything that mimics them.  Shorthands are neither better nor worse in
this case.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Help sought understanding shorthands wrt modules/packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > That is unreasonable.  People have been installing very large changes
  > > just lately.

  > No, they haven't, not lately anyway.

I think package-vc was just installed in master this weekend, and
noverlays a few days ago.  I think there was another in the past week.

  > It is a minor feature, indeed, but its effect on low-level code is
  > quite significant and could potentially destabilize Emacs.

It adds a feature that nothing in Emacs actually uses.
It is potentially significant, but it won't affect anything
if you don't use it.  I think you have exaggerated a .01 into .99.

Basically, you haven't convinced me that there is a basis for this
conclusion.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: A short defense of shorthands.el (but CL packages are still better)

[João Távora]

Richard Stallman <rms@gnu.org> writes:

> I've never used ElDoc and I don't know what it does.

ElDoc is Emacs's built-in at-point documentation system.  It displays
the docstring of the symbol under point in the echo are.  It is active
by default in Elisp files, and nothing has to be done by the user to
benefit from it, so you've probably used it unknowingly.

If x.el's contents were:

   (defun x-foo (point)
      "Conjure xenomorph foos."
      (interactive "d")
      (message "42 %s" point))
   (provide 'x)

   ;; x.el ends here
   ;; Local Variables:
   ;; read-symbol-shorthands: (("x-" . "xenomorph-"))
   ;; End:


You would see, in the Emacs echo area line, the indication

  xenomorph-foo: (POINT)

every time point is over a reference to the function

> Maybe you're right about C-h f, but you have not specified the
> scenario fully.  What does the user actually type, to invoke it?

Type this:

   C-h f x e n o TAB RET

This will most likely lead to a *Help* buffer displaying xenomorph-foo's
docstring.

But here's a much better example.  Even without any clients of x.el the
way you invoke the interactive function defined above is

   M-x xenomorph-foo RET

_not_

   M-x x-foo

And if the function ever signals an error, and you have the debugger
activated, then this is what shows up in the Backtrace:

   Debugger entered--Lisp error: (error "ohno")
     signal(error ("ohno"))
     error("ohno")
     xenomorph-foo(104)
     funcall-interactively(xenomorph-foo 104)

This is all consistent with the fact that the symbol actually interned
is always xenomorph-foo, not x-foo or any other context-dependent
shorthand to xenomorph-foo.

Understanding that a shorthand is just a convened-upon shortcut to a
name is key to a full proper name, with given spatial and temporal
validity, is key to operating within a namespacing system.

Curiously, most people have no problem grasping this concept when
operating namespacing systems of non-programming realities: it's well
understood that in this conversation the string "Richard" refers to you.
"RMS", a shorter string, also refers to you.  But if Richard Wagner were
to join the conversation, the first string would become ambiguous and we
could decide to replace it with something else.  We could use the latter
"RMS" shorthand for you and "WRW" for the famous composer.  But if this
were a discussion on electrical engineering, then also the shorthand
"RMS" would become ambiguous as it usually means "Root-Mean-Square" in
that context.

> I think the same grep issue will occur with actual CL packages, or
> anything that mimics them.  Shorthands are neither better nor worse in
> this case.

We are absolutely in agreement here.  This "breaks grep" characteristic
is an innate characteristc of any namespacing system.

But it is not impossible to design cross-referencing systems that
understand namespaces.  There is a good amount of prior art for the CL
package system, for example.

João

Re: Help sought understanding shorthands wrt modules/packages

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: akrl@sdf.org, emacs-devel@gnu.org
> Date: Mon, 07 Nov 2022 02:47:34 -0500
> 
>   > > That is unreasonable.  People have been installing very large changes
>   > > just lately.
> 
>   > No, they haven't, not lately anyway.
> 
> I think package-vc was just installed in master this weekend

package-vc is a new Lisp package with a small number of new commands
which provide new functionality.  The package doesn't touch any
internals, and thus cannot have any effect on any commands other than
those it adds.  And the discussion of merging this package's branch
started more than a month ago.  (The branch itself is much older.)

> and noverlays a few days ago.

The decision to install noverlays was made in September, and since
then Stefan, Gerd, Matt, and others worked on putting the final polish
on it.  The branch itself was reviewed and discussed long ago.

>   > It is a minor feature, indeed, but its effect on low-level code is
>   > quite significant and could potentially destabilize Emacs.
> 
> It adds a feature that nothing in Emacs actually uses.
> It is potentially significant, but it won't affect anything
> if you don't use it.  I think you have exaggerated a .01 into .99.

I don't think I exaggerated.  Here's the list of the aspects of the
patch which I think make it incomplete and potentially risky:

  . it changes oblookup_considering_shorthand in fundamental ways,
    which affects all of its callers, notably intern and unintern;
  . it includes unresolved issues: how to handle this with
    natively-compiled code and with modules (I personally don't yet
    have a clear idea whether and how to do any of these two);
  . it affects file handlers: they will need to support the new
    feature (and such a support is not yet in the patch, so it will
    have to be written and tested);
  . there's no lispref documentation for this

Finally, there's nothing important enough in this feature that we
should hurry with installing it.

It will take time to figure out the aspects we didn't and complete the
implementation and the docs.  Given the short time till branching, I
don't see how we could make it, and I definitely don't see why we
should hurry.  Why is it a problem to have it on master starting one
month from now?

> Basically, you haven't convinced me that there is a basis for this
> conclusion.

Given the above, I think it's the other way around: you should
convince me.

Re: Help sought understanding shorthands wrt modules/packages

[Matt Armstrong]

João Távora <joaotavora@gmail.com> writes:

> On Sat, Nov 5, 2022 at 3:13 AM Richard Stallman <rms@gnu.org> wrote:
>
>> If CL packages still have the misfeature of searching a list of
>> packages for one that has a symbol 'foobar' in it, and deciding what
>> `foobar' in your code means based on that, then they are inexcusable
>> bad design and we must not implement them.
>>
>> If they no longer have that misfeature, maybe they are ok.
>
> If you're talking about the :USE directive, you don't have to employ
> it: it's not mandatory for CL packages to be immensely better. But
> it's very useful and convenient in specific, well-understood
> situations.  If you're talking about something else, I don't know what
> it might me.

My understanding is that Richard is concerned about ambiguities, perhaps
not even flagged as errors at load time, that occurred in a version of
CL packages he implemented or otherwise worked with in the past, but
that may no longer occur in Common Lisp implementations conforming to
the newest standard.  I believe he described the "misfeature" he is
concerned about more clearly in
https://lists.gnu.org/archive/html/emacs-devel/2022-10/msg02165.html.
There, I think Gerd made the convincing argument that the situation is
acceptable in current CL standards.

If so, then Richard may be closer to a "maybe they are ok" judgment
about CL packages than he has been in decades.  ;-)

Re: Help sought understanding shorthands wrt modules/packages

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1591 bytes --]

On Tue, Nov 8, 2022 at 12:27 AM Matt Armstrong <matt@rfc20.org> wrote:

> João Távora <joaotavora@gmail.com> writes:
>
> > If you're talking about the :USE directive, you don't have to employ
> > it: it's not mandatory for CL packages to be immensely better. But
> > it's very useful and convenient in specific, well-understood
> > situations.  If you're talking about something else, I don't know what
> > it might me.
>
> My understanding is that Richard is concerned about ambiguities, perhaps
> not even flagged as errors at load time, that occurred in a version of
> CL packages he implemented or otherwise worked with in the past, but
> that may no longer occur in Common Lisp implementations conforming to
> the newest standard.  I believe he described the "misfeature" he is
> concerned about more clearly in
> https://lists.gnu.org/archive/html/emacs-devel/2022-10/msg02165.html.
> There, I think Gerd made the convincing argument that the situation is
> acceptable in current CL standards.
>

Thanks, I read that description, and as far as I can gather, only
USE-PACKAGE vaguely fits that description,

http://www.lispworks.com/documentation/HyperSpec/Body/f_use_pk.htm#use-package

It's an optional feature that shouldn't be used -- much like -rf to rm or
really any flag to any program -- if you one doesn't understand what it
does.
It will error and offer restarts if two used packages export symbols with
the
same name.

Which reminds me.... I wonder if Gerd's branch also comes with a
programmable
interactive restart system :-)

João

[-- Attachment #2: Type: text/html, Size: 2509 bytes --]

Re: A short defense of shorthands.el (but CL packages are still better)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > You would see, in the Emacs echo area line, the indication

  >   xenomorph-foo: (POINT)

  > every time point is over a reference to the function x-foo.

Is that behavior bar?  What behavior would we prefer ElDoc to
implememt for such cases?  Would we want it to show `x-foo' instead of
`xenomorph-foo' in the echo area?,

How does ElDoc know to apply the shorthands for x.el when looking at
the text of x.el?  Does ElDoc have code to do this explicitly?

  > > Maybe you're right about C-h f, but you have not specified the
  > > scenario fully.  What does the user actually type, to invoke it?

  > Type this:

  >    C-h f x e n o TAB RET

  > This will most likely lead to a *Help* buffer displaying xenomorph-foo's
  > docstring.

That seems correct to me.  `xenomorph-foo' is a defined function; why
shouldn't C-h f know about it?

  > way you invoke the interactive function defined above is

  >    M-x xenomorph-foo RET

  > _not_

  >    M-x x-foo

That too seems correct to me.

When I consider the case of `string.el', which would rename s-* to
string-*, these all seem like correct behavior (though M-x won't allow
these functions under any name, since they are not interactive).

  > And if the function ever signals an error, and you have the debugger
  > activated, then this is what shows up in the Backtrace:

  >    Debugger entered--Lisp error: (error "ohno")
  >      signal(error ("ohno"))
  >      error("ohno")
  >      xenomorph-foo(104)
  >      funcall-interactively(xenomorph-foo 104)

I think that is correct, too.  It might be confusing
if you are looking at the source of x.el and you don't
notice it uses a shorthand.  Nonetheless, it is correct.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: A short defense of shorthands.el (but CL packages are still better)

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1791 bytes --]

On Tue, Nov 8, 2022 at 5:02 AM Richard Stallman <rms@gnu.org> wrote:

> Is that behavior bar?  What behavior would we prefer ElDoc to
> implememt for such cases?  Would we want it to show `x-foo' instead of
> `xenomorph-foo' in the echo area?,

I wouldn't.  It would be misleading.  The shorthand x-foo is only available
locally.

> That seems correct to me.  `xenomorph-foo' is a defined function; why
> shouldn't C-h f know about it?

Yes it is the correct behaviour,  and that's why I implemented it like so.

>   > way you invoke the interactive function defined above is
>   >    M-x xenomorph-foo RET
>   > _not_
>   >    M-x x-foo
> That too seems correct to me.

Same here.

>   >      xenomorph-foo(104)
>   >      funcall-interactively(xenomorph-foo 104)
> I think that is correct, too.  It might be confusing
> if you are looking at the source of x.el and you don't
> notice it uses a shorthand.  Nonetheless, it is correct.

Right again.

My point with these examples was simply to illustrate that a tool
like grep looking for text strings in files "x-foo" isn't guaranteed
to serve us in the presence of a namespacing system.  Here, any
grep for one of "xenomorph-foo", "x-foo" or "xeno-foo" will come up
short in some way, even though these are all ways to reference the
same symbol in some specific context.

My point was also that we shouldn't try to force or limit our uses of
said namespacing system to try and accommodate grep. So while
the "breaks grep" objection does hold some water, it's because grep
is "broken" extremely easily.

But if a tool other than grep were to understand the locally defined
shorthands for every file it visits, then it could very well be useful.
I described such a tool some emails ago.

João

[-- Attachment #2: Type: text/html, Size: 2450 bytes --]

Re: Help sought understanding shorthands wrt modules/packages

[Gerd Möllmann]

On 08.11.22 05:52, João Távora wrote:
> Which reminds me.... I wonder if Gerd's branch also comes with a 
> programmable
> interactive restart system :-)

Pssst, not so loud... :-)

Re: Help sought understanding shorthands wrt modules/packages

[Gerd Möllmann]

> It adds a feature that nothing in Emacs actually uses.
> It is potentially significant, but it won't affect anything
> if you don't use it.  I think you have exaggerated a .01 into .99.

I don't agree.  Symbols are at the heart of Lisp.  Changing their
semantics is a big deal, and as soon as that features is in, it will be
used.

BTW, in light of what I read now on emacs-devel in various threads,
I take my questions about the design of shorthands and there use instead
of modules/packages back, because there is no design.

This belongs into a feature branch, IMO.

Re: Help sought understanding shorthands wrt modules/packages

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1210 bytes --]

On Tue, Nov 8, 2022, 06:20 Gerd Möllmann <gerd.moellmann@gmail.com> wrote:

> .
>
> BTW, in light of what I read now on emacs-devel in various threads,
> I take my questions about the design of shorthands and there use instead
> of modules/packages back, because there is no design.
>

I find this characterization a bit unfair. There was and is a design, a
fairly small one and one aimed primarily at solving a particular problem,
which wouldn't be possible with CL packages. You can't introduce CL
packages into an existing library file without significantly changing it
and without breaking compatibility to earlier versions of Emacs.

Nevertheless, shorthands are not a replacement for CL packages, as I've
explained before.  They were meant to plug a specific hole regarding
namespace pollution derived from packages with very short (and commonly
desired) prefixes.  That was the main goal. In the process, shorthands also
offer a fraction of the programming convenience of CL packages. It's a
very, very small fraction, but given that after many years of discussion
there was still nothing in Emacs and now there is something, i count that
as an improvement of infinite%.

>
>

[-- Attachment #2: Type: text/html, Size: 1918 bytes --]

Re: Help sought understanding shorthands wrt modules/packages

[Gerd Möllmann]

On 08.11.22 10:54, João Távora wrote:
> On Tue, Nov 8, 2022, 06:20 Gerd Möllmann <gerd.moellmann@gmail.com 
> <mailto:gerd.moellmann@gmail.com>> wrote:
> 
>     .
> 
>     BTW, in light of what I read now on emacs-devel in various threads,
>     I take my questions about the design of shorthands and there use instead
>     of modules/packages back, because there is no design.
> 
> 
> I find this characterization a bit unfair. There was and is a design, a 
> fairly small one and one aimed primarily at solving a particular 
> problem, which wouldn't be possible with CL packages. 

Sorry for having been a battleaxe :-).

I know that part, and I think it's likely a good tool when meeting
realities like dash and s and so on.

What I meant was using shorthands as the basis for a general mechanism
that somehow solves a "problem" that other languages use modules for.  I
mean, a design, or at least what I understand under a design, should
have at least briefly considered things like Eldoc, xref, completions,
and what else there is.  I didn't see/hear this, and it only slowly
comes up and generates "we could do this or that".  TBH, that reminds me
a bit of Greenspun's tenth.

But anyways, the ship has sailed.

Re: Help sought understanding shorthands wrt modules/packages

[João Távora]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> What I meant was using shorthands as the basis for a general mechanism
> that somehow solves a "problem" that other languages use modules for.

We're in violent agreement here.  Shorthands are not a general mechanism
for doing that, nor was it ever their purpose when I proposed them.

> I mean, a design, or at least what I understand under a design, should
> have at least briefly considered things like Eldoc, xref, completions,
> and what else there is.

But shorthands _do_ work with ElDoc, Xref, and completions, and likely
anything that uses standard mechanisms to get the Elisp symbol at point.
xref-find-definition (the M-. shortcut) works.  The only main thing that
I know of that doesn't work is xref-find-references, the C-? shortcut.
And that's because the underlying backend for Elisp is based on... you
guessed it, grep!  See

   (cl-defmethod semantic-symref-perform-search ((tool semantic-symref-tool-grep))

Also note that shorthands are not the only thing that "break grep",
undermining such naive xref-find-references backends.  Symbol
indirection techniques such as (intern (format ...)) as found in
e.g. xref-create-fetcher of xref.el are just as problematic.

> I didn't see/hear this, and it only slowly comes up and generates "we
> could do this or that".

The only thing that remains to be done in shorthands is the
xref-find-references support.  Do you have that bit covered in your CL
packages branch?  I didn't check, but last I investigated, and saw
real-world implementations of cross-referencing systems (in SBCL and
Allegro CL), they were workable but not perfect either.

> TBH, that reminds me a bit of Greenspun's tenth. But anyways, the ship
> has sailed.

Maybe. Ultimately, Greenspun and CL are misunderstood sailors at heart,
but it shouldn't stop us from reaching for that boat.  So I commend you
for this work on CL packages, and hope you keep pushing it.  If you want
to discuss integration with other Emacs's facilities am I familiar with
(ElDoc, completions, xref backend ideas), feel free to reach out.

João

Re: Help sought understanding shorthands wrt modules/packages

[Stefan Kangas]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> What I meant was using shorthands as the basis for a general mechanism
> that somehow solves a "problem" that other languages use modules for.  I
> mean, a design, or at least what I understand under a design, should
> have at least briefly considered things like Eldoc, xref, completions,
> and what else there is.  I didn't see/hear this, and it only slowly
> comes up and generates "we could do this or that".  TBH, that reminds me
> a bit of Greenspun's tenth.
>
> But anyways, the ship has sailed.

Has it?  Richard's most recent patch has not been installed, AFAICT.

As João explained, shorthands was added mainly for the benefit of s.el
and dash.el (none of which uses it yet, I think?).  But it was not clear
from the discussion at the time that it was to be extended into a
general module system.  That's a more recent development.  Until and
unless changes in that direction have been installed, it seems premature
to talk about sailing ships.

Re: Help sought understanding shorthands wrt modules/packages

[João Távora]

Stefan Kangas <stefankangas@gmail.com> writes:

> As João explained, shorthands was added mainly for the benefit of s.el
> and dash.el (none of which uses it yet, I think?).  But it was not clear
> from the discussion at the time that it was to be extended into a
> general module system.

I wasn't and it cannot be extended into a general module system.  Emacs
already has one always has, albeit one with severe limitations and based
on cumbersome and easy-to-break convention.  Shorthands are just a way
to make live slightly easier within it.

> That's a more recent development.  Until and
> unless changes in that direction have been installed, it seems premature
> to talk about sailing ships.

FWIW I don't agree with those changes, it seems like they would lead to
truly odd situations like having an el file load different sets of
symbols depending on how it is loaded (what happens if one M-. to a
definition, fixes a bug, and types C-M-x?)

But even if they are installed, I don't think they negate the need for
CL packages or make them any harder to implement.

João

Re: Help sought understanding shorthands wrt modules/packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >   I believe he described the "misfeature" he is
  > concerned about more clearly in
  > https://lists.gnu.org/archive/html/emacs-devel/2022-10/msg02165.html.
  > There, I think Gerd made the convincing argument that the situation is
  > acceptable in current CL standards.

I have all the mail on emacs-devel in my saved inbox
but that reference to a web archives does not enable me to find this one.

If you would show the From and Date fields of the message,
with those I can actually find it.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Help sought understanding shorthands wrt modules/packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > If you're talking about the :USE directive,

I implemented CL packages 40 years ago -- I don't remember the details.

There was a feature that let you specify, in one package FOO, "Let me
omit package prefixes when I refer to symbols from package BAR."
Then if BAR defines a symbol BAR:HACK, and you write HACK in package FOO
with no prefix, the package system would search for a symbol HACK, find
it in package BAR, and treat HACK as an alias for BAR:HACK.

Is that what :USE does?  If so, it would seem that :USE is the bad design
I am talking about.

The general idea is a nice one, but it doesn't work in Lisp.

A Lisp symbol has a value, a function definition, a property list, and
an identity.  Since CL packages handle this by interning one symbol in
multiple alists, it gangs together the symbol's slots with its
identity.  If the function FOO:HACK means BAR:HACK, then the variable
FOO:HACK means BAR:HACK, and the propery list FOO:HACK means that of
BAR:HACK, and FOO:HACK as a property name means BAR:HACK.

Even examining the value of HACK with FOO selected is enough to do
intern that particular symbol HACK in FOO.

Making it even worse, once BAR:HACK is recorded as the meaning of HACK
in FOO's symbol table (obarray), there was no way to make it mean some
other symbol HACK.

Suppose that package FOO says to allow reference without explicit
prefix to symbols from packages ADDED and BAR.  Suppose BAR has a
symbol HACK and ADDED does not have one.  In FOO you the HACK, and it
becomes aliased to BAR:HACK.

Suppose you then define another symbol called ADDED:HACK.  Now the
meaning of HACK in FOO has been changed.  But it can't really be changed,
because the obarray for FOO already contains the HACK that is BAR:HACK.

This is a gross bug.  We should not implement such a bug.  No one ever
should.

We can make package systems that behave naturally and consistently.
But they can't be compatible with CL packages.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Help sought understanding shorthands wrt modules/packages

[Matt Armstrong]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   >   I believe he described the "misfeature" he is
>   > concerned about more clearly in
>   > https://lists.gnu.org/archive/html/emacs-devel/2022-10/msg02165.html.
>   > There, I think Gerd made the convincing argument that the situation is
>   > acceptable in current CL standards.
>
> I have all the mail on emacs-devel in my saved inbox
> but that reference to a web archives does not enable me to find this one.
>
> If you would show the From and Date fields of the message,
> with those I can actually find it.

From: 	 Richard Stallman
Subject: Re: No to CL packages
Date: 	 Fri, 28 Oct 2022 17:59:32 -0400

Re: Help sought understanding shorthands wrt modules/packages

[Yuri Khan]

On Wed, 9 Nov 2022 at 11:04, Richard Stallman <rms@gnu.org> wrote:

> There was a feature that let you specify, in one package FOO, "Let me
> omit package prefixes when I refer to symbols from package BAR."

> The general idea is a nice one, but it doesn't work in Lisp.

This feature exists in C++, Python, Scala, and many other languages,
and most code style guides frown upon using it, for the very reasons
you cite. It is tolerated sometimes for tightly coupled modules, like
a library and its automated tests, or a public interface module of a
library importing its internal implementation modules wholesale.

Point is: wholesale import is a sharp tool best reserved for very
special circumstances.

Re: Help sought understanding shorthands wrt modules/packages

[tomas]

[-- Attachment #1: Type: text/plain, Size: 874 bytes --]

On Wed, Nov 09, 2022 at 12:42:24PM +0700, Yuri Khan wrote:
> On Wed, 9 Nov 2022 at 11:04, Richard Stallman <rms@gnu.org> wrote:
> 
> > There was a feature that let you specify, in one package FOO, "Let me
> > omit package prefixes when I refer to symbols from package BAR."
> 
> > The general idea is a nice one, but it doesn't work in Lisp.
> 
> This feature exists in C++, Python, Scala, and many other languages,

...but it doesn't lump together a variable, a function (and a couple
of other things). Be it because, at heart, they're a Lisp-1 (Python,
probably Scala), because they disambiguate types with a sigil (Perl,
PHP) -- thus making them a Lisp-1.

Perhaps (I've to defer to Richard's experience here; I'm one of those
wary of shorthands and wishing for a "real" namespace/module system)
Elisp Lisp-2-ness makes things harder.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Help sought understanding shorthands wrt modules/packages

[Matt Armstrong]

Richard Stallman <rms@gnu.org> writes:

> I implemented CL packages 40 years ago -- I don't remember the details.
>
> There was a feature that let you specify, in one package FOO, "Let me
> omit package prefixes when I refer to symbols from package BAR."
> Then if BAR defines a symbol BAR:HACK, and you write HACK in package FOO
> with no prefix, the package system would search for a symbol HACK, find
> it in package BAR, and treat HACK as an alias for BAR:HACK.
>
> Is that what :USE does?  If so, it would seem that :USE is the bad design
> I am talking about.

[...]

(I have clipped the, I believe, well understood downsides of :use)

> This is a gross bug.  We should not implement such a bug.  No one ever
> should.
>
> We can make package systems that behave naturally and consistently.
> But they can't be compatible with CL packages.

How about this goal: Any package system adopted by Emacs should allow
new symbols to be exported from a package without risk of breaking its
users.

I wonder if a consensus could be reached here.  Namely, just omit :use
from any Emacs implementation of lisp packages.

I would like to hear more from people with more Common Lisp experience
than I do, but importing all symbols from one package into another does
not strike me as an essential feature, or a good idea in general.  It is
popular in some languages, even idiomatic.  Others, such as Go, make it
impossible to achieve for reasons similar to the ones Richard outlined.
Namely, eventually, heavily used packages cannot add anything new for
fear of breaking something.

Re: Help sought understanding shorthands wrt modules/packages

[Gerd Möllmann]

João Távora <joaotavora@gmail.com> writes:

> But even if they are installed, I don't think they negate the need for
> CL packages or make them any harder to implement.

I suspect it could make it harder to achieve backwards-compatibility.  I
can't prove it, though, because I set shorthands out of scope for my
experiement.

Let me try to explain:

The main problem with staying backwards-compatible was the triple of
functions (intern, intern-soft, symbol-name), alone and in combination,
what semantics they currently expect and how that fits with packages.
That's what I found during my experiment.

The simplest example is keywords: (intern ":a") currently makes a
keyword, with the semantics that any symbol whose name starts with a
colon is a keyword, and symbol-name will return a name that includes the
colon.  That doesn't fit well with CL packages, where the colon is
interpreted as part of a qualified symbol name, the symbol-name is "a"
and the empty string in front of the ":" is the (empty) nickname of the
keyword package.

Like I said, that's the super-simple form of the problem.  I guess one
can find every possible variation of that problem somewhere in the wild.

Without taking shorthands into account, one can make an educated guess
what semantics the code expects, and use that to become
backwards-compatible.  I suspected that was the case, which was the
initial reason I started the experimeent.  Yeah, I know, what a nerd
:-).  And it proved true, without shorthands at least.  Experiment done.

With taking shorthands into account, these "guesses" don't really hold
much water.  Or so is my gut feeling ATM.

P.S.

I have an alternative idea how to "guess", but I don't know if it's
practically feasible, works well enough, and yadda yadda.  See
admin/cl-packages.org in the branch.

Just a side note :-).

Re: Help sought understanding shorthands wrt modules/packages

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 247 bytes --]

You're into something. Adopting Gerd's implementation of CL packages as the
basis of
Elisp packages does not mean we need to replicate the things we think are
pitfalls.
It's still a better foundation than inventing a new package system
wholesale.

[-- Attachment #2: Type: text/html, Size: 469 bytes --]

Re: Help sought understanding shorthands wrt modules/packages

[Gerd Möllmann]

On 08.11.22 16:47, Stefan Kangas wrote:
> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>> But anyways, the ship has sailed.
> 
> Has it? 

I meant the pedal boat I'm in :-).

Re: Help sought understanding shorthands wrt modules/packages

[Gerd Möllmann]

> I wonder if a consensus could be reached here.  Namely, just omit :use
> from any Emacs implementation of lisp packages.
> 
> I would like to hear more from people with more Common Lisp experience
> than I do, but importing all symbols from one package into another does
> not strike me as an essential feature, or a good idea in general.  

Use-package (which is what :use is using in defpackage :-) is kind of a 
natural thing, for me.  For example, this use is ubiquitous and might 
even be in the standard, I don't remember:

(in-package "CL-USER")
(use-package "COMMON-LISP")

which basically says "let me use all of CL without qualification" in my 
interactive session.  So, you don't have to write cl:car, you write car.

And that's the spirit in which I've seen use-package is used in general:
You are writing a package doing X11 stuff - you might (or might not) use
package xlib.  You don't like cl:loop, you might want to use the iterate
package.  You're in a compiler package, you might want to use the API of
the assembler package.  And so on.

I think one important part to also keep in mind is that packages allow
internal and external symbols.  The external symbols are the API.  The
internals one can access (with X::Y), if you need to, but you should
feel guilty :-).  The evolution of APIs is of course another can of worms.

Re: Help sought understanding shorthands wrt modules/packages

[Helmut Eller]

On Tue, Nov 08 2022, Richard Stallman wrote:

> Making it even worse, once BAR:HACK is recorded as the meaning of HACK
> in FOO's symbol table (obarray), there was no way to make it mean some
> other symbol HACK.

That's technically wrong: it is possibly to "shadow" inherited symbols.
E.g. so:

(defvar *foo* (make-package "FOO" :use '()))
(defvar *bar* (make-package "BAR" :use (list *foo*)))

(export (intern "HACK" *foo*) *foo*)
(assert (eq (intern "HACK" *bar*)
	    (intern "HACK" *foo*)))
(shadowing-import (make-symbol "HACK") *bar*)
(assert (not (eq (intern "HACK" *bar*)
		 (intern "HACK" *foo*))))

This shows that FOO::HACK and BAR::HACK can be different symbols even
though FOO exports HACK and BAR uses FOO.

Helmut

Re: Help sought understanding shorthands wrt modules/packages

[Matt Armstrong]

Helmut Eller <eller.helmut@gmail.com> writes:

> On Tue, Nov 08 2022, Richard Stallman wrote:
>
>> Making it even worse, once BAR:HACK is recorded as the meaning of HACK
>> in FOO's symbol table (obarray), there was no way to make it mean some
>> other symbol HACK.
>
> That's technically wrong: it is possibly to "shadow" inherited symbols.

Yes, Common Lisp apparently grew mitigations for the pitfalls of
:use/use-package after Richard had experience with them.  I do not think
the pitfalls went away, though.

Re: Help sought understanding shorthands wrt modules/packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > What I meant was using shorthands as the basis for a general mechanism
  > that somehow solves a "problem" that other languages use modules for.

Modules in other languages work correctly because they operate on
variable _meanings_.  For instance, on function definitions, on
variables, on classes, on class methods.  If one name is the name of a
function and the name of a variable, the module system handles each of
those meanings separately.  

It's not possible to do that job correctly in Lisp at read time,
because that precludss handling each meaning of the symbol separately.
Inevitably, CL packages are flawed compared with other languages'
module systems.  The basic design decision to make them work at the
level of reading made that inevitable.  We can call this the
"polyvalency problem."

Module systems in other languages fit together with lexical scoping.
Lexically scoped local variables, functions, or classes don't interact
with the module system.  There is no way for the Lisp reader to tell
which uses of a symbol are local.  Inevitably, 

  (defun my-member (elt list) ...)

is going to inherit `elt' and `list' as symbols from the package
of standard Lisp functions, rather than the current package.

The grave problem of :USE in CL packages could be fixed by replacing
it with a new construct that specifies a list of symbols to be
inherited from each other package, perhaos with renaming.  Instead of
(:USE "bar") meaning to inherit all the external symbols of bar, you
would write (:INHERIT "bar" "entrypoint" ("barhack" . "hack")).
Instead of the kludgy "get an error if you :USE two packages that
clash", with :INHERIT you could avoid or fix conflicts one by one.

We could call this system "corrected CL packages."
It would not be compatible but it would be better.

This sort of inheritance could be implemented by means of shorthands,
but it could also be implemented directly in lread.c the same way
shorthands are implemented.

The idea of CL packages is to use multiple obarrays, but that
implementation adds unnecessary complexity and unnecessary change.  We
can treat names with package prefixes, such as `bar:entrypoint', as
ordinary names and put them all in one obarray.  We could implement
all of the corrected CL packages system with just one obarray.

With this approach, reading a file in package bar would transform
every name by concatenating `bar:' at the front.

Normally name concatenation is messy and desirable to avoid, but
perhaos if done in a controlled way through a package system it would
not be messy.  At least, not any more messy than is inevitable in a
package system that operates at read time.

We could hide this concatenation most of the time by making print
cut off the package prefixes that are redundant in their context.

I wish I could have written all this in a coherent way at the start of
this discussion.  But this is an issue I dealt with 40 years ago,
before GNU, and it had become buried in my memory.  It has come back
to me only gradually.

Eli wrote:

  > Moreover, from my POV, the jury is still out on the question of
  > whether we at all need packages in Emacs.  "Programming in the large"
  > doesn't sound very relevant to how Emacs Lisp is used.

I share this feeling, except that I take it a bit further.  In the
1980s I thought about whether we should have packages in Emacs, and
concluded that using a naming conventions is simpler, easier and more
flexible.  So I chose that.

If CL packages were unproblematical, I'd say, "Sure, why not?  We
aren't desperate to keep Emacs small nowadays."  But they are
problematical, and that is a reason to choose to do without them.

The only drawback of our current use of naming conventions is the need
to write long names in your code when referring to functions and
variables defined in other parts of Emacs.  If your code refers to
many such entry points of a certain part of Emacs, maybe you want
to abbreviate those.

If that is what you want perhaps shorthands are enough.  You can
define abbreviations for the entry points that your package uses
often.

That gives you total control over what abbreviations to use in your
package, does not impinge on any other code including the code that
defines the entry points you want to abbreviate, and it causes no
confusions once you understand what shorthands are.

The only problem we can't avoid is the inconvenience in grep, but that
will happen with any sort of package system that permits abbreviation
of names.  It is not a reason to prefer CL packages.

You can even avoid the polyvalency problem this way.  For instance,
you can define `foo-fun' as a shorthand for `find-outer-otter' the
function, and `foo-var' as a shorthand for `find-outer-otter' the
variable.

In fact, either abbreviation _could_ be used for either the function
or the variable; there is nothing to enforce the distinction between
those two abbreviations.  But if you wish to maintain that distinction
in your code, you can.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Help sought understanding shorthands wrt modules/packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > One thing that was talked about in the past is something like "local
  > shorthands".  I think the idea is that a certain prefix is only active
  > in a certain region of the text.

I think we could do this easily enough using the existing shorthands
feature.  Would you like to write it?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Help sought understanding shorthands wrt modules/packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Example from the Info manual

  >      ;; read-symbol-shorthands: (("snu-" . "some-nice-string-utils-"))

  > The "snu-" definition is not recorded anywhere, nothing is preventing
  > someone else from also using "snu-", or warning about it.

There is no need to prevent someone else from using it.  Each file's
shorthands are local: they do not conflict with other files.  If you
want to define `snu-' as a shorthand in your file, go ahead.

                                                               You can't
  > find out that "snu-" was used to read it from looking at a symbol,

Why does that matter?
If what appears in the code is `some-nice-string-utils-concat', when is
it crucial to find out whether a shorthand was used to enter it?

Is this about the grep shortcoming?  I don't think CL packages avoid that.

                                                                       or if
  > an abbreviation was used at all, which is in my eyes unlispy, because it
  > allows no introspection.

I don't think we need to complicate Emacs Lisp to make all kinds of
introspection possible at the s-expression level.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Help sought understanding shorthands wrt modules/packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Since it finishes an existing feature and makes it usable for the most
  > > important case, we should definitely install it before Emacs 29.

  > Handling of shorthands is implemented in lread.c. File name handlers for
  > `load' don't know (yet) what to do.

Is this in regard to the already-installed shorthands feature, or to
my new feature which adds shorthands to a file when loading it?

If the latter, I think all that file name handlers need to do for `load'
is to pass along the new 6th argument in the recursive call to `load'.

If that solution doesn't do the job, what is the problem?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Help sought understanding shorthands wrt modules/packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I don't agree.  Symbols are at the heart of Lisp.  Changing their
  > semantics is a big deal, and as soon as that features is in, it will be
  > used.

The shorthands feature is already installed, but it doesn't change the
sematics of symbols.  It is just an aliasing feature.

The new feature of loading a file with added shorthands
is just a variation on this aliasing feature.  It will not
change the sematics of symbols.

  > This belongs into a feature branch, IMO.

That's a good idea.  Can someone tell me how to do this?  My knowledge
of git is just barely enough to install changes in master.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Help sought understanding shorthands wrt modules/packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > >     ** Shorthands for loading a file can be specified from "outside".
  > > >
  > > >     The new function `load-with-shorthands' loads a file
  > > >     and specifies additional shorthands for reading it.

  > I think it's fair to say, as the person who originally implemented
  > shorthands,
  > that I don't agree with this change.

Now I see why you think this is not necessary.  Your idea for how to
handle s.el is to use an edited version, edited to specify shorthands.

Thus, we would have a file magnars-string.el in GNU ELPA, or perhaps
in the core, and a nearly identical file s.el in NonGNU ELPA for
compatibility's sake.

(Thanks for reminding me that the developer's name is Magnar.
I was trying to remember it, but lacking a copy of s.el I knew
where to find, I couldn't check.)

We would need to update the two files in sync every time.

I did not think of doing it that way, because I think it is asking for
trouble.  We should have only one copy of s.el, and it should be the
usual released version.

Thus, instead of making a modified version magnars-string.el,
we would have string.el load s.el with added shorthands.
We would never have two versions of that file, only one.

Another point of disagreement is that you envision advising users
to load magnars-string.el rather than string.el, and make it
something that only some users would do.

People have said that s.el is very useful.  So I envision making it
standard and documenting those functions in the Emacs Lisp Reference
Manual.

Maybe we should even preload them.

  > Providing two different ways to load an .el file so that each interns
  > different
  > symbols into obarray would lead to unfortunate consequences where it would
  > be hard or impossible for humans or programs to understand the provenance of
  > a given symbol.

I don't follow you here.  What problems do you see?


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Help sought understanding shorthands wrt modules/packages

[Helmut Eller]

On Thu, Nov 10 2022, Richard Stallman wrote:

>   > One thing that was talked about in the past is something like "local
>   > shorthands".  I think the idea is that a certain prefix is only active
>   > in a certain region of the text.
>
> I think we could do this easily enough using the existing shorthands
> feature.  Would you like to write it?

No, I'm not interested.  I general, I think it was unnecessary to
implement shorthands in the reader.  It could have been done with a
macro: just substitute those symbols that start with a shorthand prefix
with the corresponding fully prefixed symbol.

Helmut

Re: Help sought understanding shorthands wrt modules/packages

[Michael Albinus]

Richard Stallman <rms@gnu.org> writes:

Hi Richard,

>   > Handling of shorthands is implemented in lread.c. File name handlers for
>   > `load' don't know (yet) what to do.
>
> Is this in regard to the already-installed shorthands feature, or to
> my new feature which adds shorthands to a file when loading it?
>
> If the latter, I think all that file name handlers need to do for `load'
> is to pass along the new 6th argument in the recursive call to `load'.
>
> If that solution doesn't do the job, what is the problem?

That would work, yes. Thanks.

Best regards, Michael.

Re: Help sought understanding shorthands wrt modules/packages

[Gerd Möllmann]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > What I meant was using shorthands as the basis for a general mechanism
>   > that somehow solves a "problem" that other languages use modules for.
>
> Modules in other languages work correctly because they operate on
> variable _meanings_.  For instance, on function definitions, on
> variables, on classes, on class methods.  If one name is the name of a
> function and the name of a variable, the module system handles each of
> those meanings separately.

I must admit that I don't understand what you are referring to.  Could
you please give an example of a module system doing that?  At first
glance It sounds like visibility rules, or something, but I don't
remember a language with different visibility rules for functions and
variables, ATM.

> It's not possible to do that job correctly in Lisp at read time,
> because that precludss handling each meaning of the symbol separately.
> Inevitably, CL packages are flawed compared with other languages'
> module systems.  The basic design decision to make them work at the
> level of reading made that inevitable.  We can call this the
> "polyvalency problem."

Well.  Let me start by saying that I don't think of CL packages as
modules, but, if anything but packages, as namespaces.  The term module
is somewhat open to definition, different languages use the term
differently.  Some even have both with completely different menaings,
like C++.

>
> Module systems in other languages fit together with lexical scoping.
> Lexically scoped local variables, functions, or classes don't interact
> with the module system.  There is no way for the Lisp reader to tell
> which uses of a symbol are local.  Inevitably, 
>
>   (defun my-member (elt list) ...)
>
> is going to inherit `elt' and `list' as symbols from the package
> of standard Lisp functions, rather than the current package.

I think Michael Albinus already showed that this is not inevitable in CL
(shadow, shadowing-import).  If one wants, you can have your own elt and
car.

>
> The grave problem of :USE in CL packages could be fixed by replacing
> it with a new construct that specifies a list of symbols to be
> inherited from each other package, perhaos with renaming.

I fail to understand the "grave" problem, sorry.  Could you please give
a more concrete example?

BTW, in CL, if you don't want to use-package you can also import just
the list of symbols you want.  Without renaming, but I guess that could
be added, if somone wants that.  (I mention the idea of renaming in
cl-packages.org, BTW, on the branch, but I'm still not sure it's worth
it.)

> Instead of
> (:USE "bar") meaning to inherit all the external symbols of bar, you
> would write (:INHERIT "bar" "entrypoint" ("barhack" . "hack")).
> Instead of the kludgy "get an error if you :USE two packages that
> clash", with :INHERIT you could avoid or fix conflicts one by one.

I think what you mean can be done with import, but I'm not sure.  Like I
said, import doesn't have the renaming part.

>
> We could call this system "corrected CL packages."
> It would not be compatible but it would be better.

It could be compatible, at least I don't see a reason right now why it
couldn't.

>
> This sort of inheritance could be implemented by means of shorthands,
> but it could also be implemented directly in lread.c the same way
> shorthands are implemented.
>
> The idea of CL packages is to use multiple obarrays, but that
> implementation adds unnecessary complexity and unnecessary change.  We
> can treat names with package prefixes, such as `bar:entrypoint', as
> ordinary names and put them all in one obarray.  We could implement
> all of the corrected CL packages system with just one obarray.
>
> With this approach, reading a file in package bar would transform
> every name by concatenating `bar:' at the front.

Let me just remark that this is a read-time solution like CL packages,
so all you said above applies to shorthands as well as packages.

So, that's 1:1.

>
> Normally name concatenation is messy and desirable to avoid, but
> perhaos if done in a controlled way through a package system it would
> not be messy.  At least, not any more messy than is inevitable in a
> package system that operates at read time.

Perhaps, but I find CL packages not messy at all, TBH.  Instead, I quite
strongly feel that shorthand-packages are messy.  And I still fail to
see how shorthands address the problem a name conflicts in the first
place.  One obarray = name conflicts.  But we've been there, I think.

>
> We could hide this concatenation most of the time by making print
> cut off the package prefixes that are redundant in their context.

See, that's what I meant when I wrote "there is no design".  We could,
perhaps, maybe, maybe not, hm.  Why not use what's there and works?

And it's not even difficult to implement or understand, at least to the
degree I did, before I lost impetus.  It's not even 0.01% of redisplay,
for example.

> I wish I could have written all this in a coherent way at the start of
> this discussion.  But this is an issue I dealt with 40 years ago,
> before GNU, and it had become buried in my memory.  It has come back
> to me only gradually.

No problem.

>
> Eli wrote:
>
>   > Moreover, from my POV, the jury is still out on the question of
>   > whether we at all need packages in Emacs.  "Programming in the large"
>   > doesn't sound very relevant to how Emacs Lisp is used.
>
> I share this feeling, except that I take it a bit further.  In the
> 1980s I thought about whether we should have packages in Emacs, and
> concluded that using a naming conventions is simpler, easier and more
> flexible.  So I chose that.
>
> If CL packages were unproblematical, I'd say, "Sure, why not?  We
> aren't desperate to keep Emacs small nowadays."  But they are
> problematical, and that is a reason to choose to do without them.

Please spend some effort to explain the problematic cases. 

> The only drawback of our current use of naming conventions is the need
> to write long names in your code when referring to functions and
> variables defined in other parts of Emacs.  If your code refers to
> many such entry points of a certain part of Emacs, maybe you want
> to abbreviate those.
>
> If that is what you want perhaps shorthands are enough.  You can
> define abbreviations for the entry points that your package uses
> often.
>
> That gives you total control over what abbreviations to use in your
> package, does not impinge on any other code including the code that
> defines the entry points you want to abbreviate, and it causes no
> confusions once you understand what shorthands are.
>
> The only problem we can't avoid is the inconvenience in grep, but that
> will happen with any sort of package system that permits abbreviation
> of names.  It is not a reason to prefer CL packages.
>
> You can even avoid the polyvalency problem this way.  For instance,
> you can define `foo-fun' as a shorthand for `find-outer-otter' the
> function, and `foo-var' as a shorthand for `find-outer-otter' the
> variable.
>
> In fact, either abbreviation _could_ be used for either the function
> or the variable; there is nothing to enforce the distinction between
> those two abbreviations.  But if you wish to maintain that distinction
> in your code, you can.

I think to all of this I've already said something, so I won't bore
anyone with more :-).

Re: Help sought understanding shorthands wrt modules/packages

[Gerd Möllmann]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Example from the Info manual
>
>   >      ;; read-symbol-shorthands: (("snu-" . "some-nice-string-utils-"))
>
>   > The "snu-" definition is not recorded anywhere, nothing is preventing
>   > someone else from also using "snu-", or warning about it.
>
> There is no need to prevent someone else from using it.  Each file's
> shorthands are local: they do not conflict with other files.  If you
> want to define `snu-' as a shorthand in your file, go ahead.

There is no control over this, at least in the present implementation.
Nothing makes sure or warns if you bind read-symbol-shorthands in
whichever way you want.

>
>                                                                You can't
>   > find out that "snu-" was used to read it from looking at a symbol,
>
> Why does that matter?
> If what appears in the code is `some-nice-string-utils-concat', when is
> it crucial to find out whether a shorthand was used to enter it?

It's the other way round: there is snu-concat in some buffer, and you
want to get to some-nice-string-utils-concat.  How do you get there?  Or
is the plan that one can find that out in source file, and not in
*scratch*?  And so on.

> Is this about the grep shortcoming?  I don't think CL packages avoid
> that.

No, that was a bad example of me.

Re: Help sought understanding shorthands wrt modules/packages

[Gerd Möllmann]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > I don't agree.  Symbols are at the heart of Lisp.  Changing their
>   > semantics is a big deal, and as soon as that features is in, it will be
>   > used.
>
> The shorthands feature is already installed, but it doesn't change the
> sematics of symbols.

I don't agree.  Before shorthands, a symbol had one name, after, it can
have many.

Re: Help sought understanding shorthands wrt modules/packages

[João Távora]

Richard Stallman <rms@gnu.org> writes:

> Now I see why you think this is not necessary.  Your idea for how to
> handle s.el is to use an edited version, edited to specify shorthands.

The magnars-string.el file is meant to supersede s.el and be a
well-behaved version of it.  The s.el file, as it exists today, is a
namespace-polluting liability, we shouldn't maintain it.  The good thing
about shorthands here is that minimal changes need to be done to both
the library and its users, so making packages in (Non)GNU ELPAs and core
start using magnars-string.el is very little effort which is well worth
spending.

>   > Providing two different ways to load an .el file so that each interns
>   > different
>   > symbols into obarray would lead to unfortunate consequences where it would
>   > be hard or impossible for humans or programs to understand the provenance of
>   > a given symbol.
>
> I don't follow you here.  What problems do you see?

If I find a bug or want to add some function to that presumed s.el that
your string.el is loading specially, I have to somehow remember to load
it with shorthands.  The usual commands C-M-x, M-x load-file, M-x
emacs-lisp-byte-compile-and-load will all produce wrong results that I
won't know how to fix.  This is uncharted and dangerous territory: after
your change, information about where to intern symbols of an Emacs Lisp
file would no longer be completely contained in the file itself.

For non-human users, violating this invariant would possibly also break
tools that map files to symbols, such as C-h f (describe-function) and
M-. (xref-find-definition).

João

Re: Help sought understanding shorthands wrt modules/packages

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1564 bytes --]

On Fri, Nov 11, 2022, 09:35 Gerd Möllmann <gerd.moellmann@gmail.com> wrote:

> Richard Stallman <rms@gnu.org> writes:
>
> > [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> > [[[ whether defending the US Constitution against all enemies,     ]]]
> > [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> >
> >   > I don't agree.  Symbols are at the heart of Lisp.  Changing their
> >   > semantics is a big deal, and as soon as that features is in, it will
> be
> >   > used.
> >
> > The shorthands feature is already installed, but it doesn't change the
> > sematics of symbols.
>
> I don't agree.  Before shorthands, a symbol had one name, after, it can
> have many.
>

This is incorrect. You're confusing the text manifestation of a symbol in a
Lisp form before it is read (as in CL:READ) with the symbol itself, which
has only one name. This didn't and couldn't change with shorthands.

Deciding to use it not use a shorthand is no different from deciding to use
or not use package qualification for a symbol in CL packages.

Neither changes the name of a symbol, just the manifestation is different.
If you force them to be the same thing, then no namespacing symbol is
possible at all

If you conflate symbol name and symbol designation/manifestation in source
files , you'll have problems implementing any package system (CL,
shorthands, whatever) and you confuse people trying to understand any Lisp
package system, i.e. you confuse this discussion. Let's try to avoid that :)

[-- Attachment #2: Type: text/html, Size: 2332 bytes --]

Re: Help sought understanding shorthands wrt modules/packages

[Gerd Möllmann]

João Távora <joaotavora@gmail.com> writes:

> On Fri, Nov 11, 2022, 09:35 Gerd Möllmann <gerd.moellmann@gmail.com> wrote:
>  I don't agree.  Before shorthands, a symbol had one name, after, it can
>  have many.
>
> This is incorrect. You're confusing the text manifestation of a symbol
> in a Lisp form before it is read (as in CL:READ) with the symbol
> itself, which has only one name. This didn't and couldn't change with
> shorthands.

Then let me try to express myself clearer.  (I hoped to get away with
something more informal.)

Before shorthands there was a 1:1 correspondence between the printed
representation of a symbol and the symbol you get when reading the
printed representation.  Symbol-name returned a string that's the
printed representation.  (And let's please not also consider escaping in
general, print-escape and print1 vs princ and such.)

After shorthands, there is a printed representation in the code, which
when read gives you a symbol with a name that can be different from the
printed representation.  In fact, many printed represenations exist,
theoretically mapping to the same symbol.

If that's not changing semantics, I don't know.  Changing the semantics
of symbols, because of the different meaning of what symbol-name
returns.

> Deciding to use it not use a shorthand is no different from deciding
> to use or not use package qualification for a symbol in CL packages.

I wasn't talking about CL packages at all, just before/after shorthands.

>
> Neither changes the name of a symbol, just the manifestation is
> different. If you force them to be the same thing, then no namespacing
> symbol is possible at all
>
> If you conflate symbol name and symbol designation/manifestation in
> source files , you'll have problems implementing any package system
> (CL, shorthands, whatever) and you confuse people trying to understand
> any Lisp package system, i.e. you confuse this discussion. Let's try
> to avoid that :)

Who's confused?  Anyone?  :-)

Re: Help sought understanding shorthands wrt modules/packages

[João Távora]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> João Távora <joaotavora@gmail.com> writes:
>
>> On Fri, Nov 11, 2022, 09:35 Gerd Möllmann <gerd.moellmann@gmail.com> wrote:
>>  I don't agree.  Before shorthands, a symbol had one name, after, it can
>>  have many.
>>
>> This is incorrect. You're confusing the text manifestation of a symbol
>> in a Lisp form before it is read (as in CL:READ) with the symbol
>> itself, which has only one name. This didn't and couldn't change with
>> shorthands.
>
> Then let me try to express myself clearer.  (I hoped to get away with
> something more informal.)
>
> Before shorthands there was a 1:1 correspondence between the printed
> representation of a symbol and the symbol you get when reading the
> printed representation.  Symbol-name returned a string that's the
> printed representation.  (And let's please not also consider escaping in
> general, print-escape and print1 vs princ and such.)
>
> After shorthands, there is a printed representation in the code, which
> when read gives you a symbol with a name that can be different from the
> printed representation.  In fact, many printed represenations exist,
> theoretically mapping to the same symbol.

But that's _exactly_ what a namespacing system does: it creates this
indirection.  Which by the way, also enables converse: the _same_
printed representation in two different contexts maping to different
symbols.

So you seem to be somehow lamenting that shorthands is a namespacing
system! :-) 

Anyway, I can't understand how the presence of shorthands can negatively
impact CL packages.  If co-existence is complicated (but is it?) it
doesn't seem hard to make either shorthands or CL-packages a noop in in
files that prefer one of the systems.  For example, the reader can just
throw away any shorthand info altogether as soon as it detects that
CL:*CURRENT-PACKAGE* (or whatever equivalent you're envisioning) is not
the default one.  Or CL:IN-PACKAGE can just error out if it detects that
read-symbol-shorthands is non-nil.

But even if the read logic didn't do that, and considered the two
systems at once, I'm still not sure there would be any ambiguity.

Regardless, if/when CL packages ever make it to core (I hope they do, of
course), I can't see why someone would want to continue to combine their
use with shorthands.  The convenience aspect of shorthands would be
completely dwarfed by CL packages.

João

Re: Help sought understanding shorthands wrt modules/packages

[Gerd Möllmann]

João Távora <joaotavora@gmail.com> writes:

> So you seem to be somehow lamenting that shorthands is a namespacing
> system! :-)

Hehe :-).  No, I wanted to disagree with Richard saying that there is no
change of semantics at all.  You know--somone being wrong on the
Internet and so :-).

>
> Anyway, I can't understand how the presence of shorthands can negatively
> impact CL packages.  If co-existence is complicated (but is it?) it
> doesn't seem hard to make either shorthands or CL-packages a noop in in
> files that prefer one of the systems.  For example, the reader can just
> throw away any shorthand info altogether as soon as it detects that
> CL:*CURRENT-PACKAGE* (or whatever equivalent you're envisioning) is not
> the default one.  Or CL:IN-PACKAGE can just error out if it detects that
> read-symbol-shorthands is non-nil.

If I implied that then I have to apologize.  I meant to convey that

(a) I haven't done any experimeent whatsoever with shorthands in
feature/pkg because I set that delibaretly out-of-scope.  One has to
draw a line somewhere.

(b) I can't exclude the possibility of non-backwards-compatibility due
to the use of shorthands.  At least they way I've been doing things in
the branch.  It's also not proven to be incompatible.  It's unknown.

(c) I might have an idea how do things differently to achieve
backwards-compatibility.  To what extent that works, or if it works at
all, I also don't know, and likely won't find out any time soon.

> But even if the read logic didn't do that, and considered the two
> systems at once, I'm still not sure there would be any ambiguity.

From my point of view, the gist of the matter is
backwards-compatibility.  That's the pain part.

In the case of packages, how to load/execute code from Emacs <= 29 in an
Emacs having packages.  In Emacs <= 29 we find one set of rules, like
allowed symbol name constituents, the behaviour of
intern/intern-soft/symbol-name, when used with symbol-concing, what is a
keyword and so on.  In Emacs with CL packages, some of these rules are
changed, apparently in a way that mostly works, i.e. I can run it with
unchanged packages compiled for 29.

The situation when using shorthands is only insofar different, that I
didn't try to cope with it.  I only have kind of an educated gut
feeling, if something like that exists, especially with
dynamically-bound read-symbol-shorthands that this is, let's say,
difficult.

Or maybe it isn't.  I don't know.  Someone has to invest the time to
find out.

>
> Regardless, if/when CL packages ever make it to core (I hope they do, of
> course), I can't see why someone would want to continue to combine their
> use with shorthands.  The convenience aspect of shorthands would be
> completely dwarfed by CL packages.

Some will, some won't, and some maybe want to use both.  Once it's there
it's there, which means we have to cope with both at the same time.

Ce la vie :-).

Re: Help sought understanding shorthands wrt modules/packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The magnars-string.el file is meant to supersede s.el and be a
  > well-behaved version of it.  The s.el file, as it exists today, is a
  > namespace-polluting liability, we shouldn't maintain it.

No one would be happer than I to eliminate te usual version of s.el,
but my understanding is that that would cause a big practical problem.

Lots of packages do (require 's), and they call those functions by the
names s-... defined in s.el.  If we don't have s.el, they will break.
They can't use your proposed magnars-string.el (which I would call
stris.el), since the names it would define are wrong _for them_.

To modify all those files would satisfy all our goals, but it's
not feasible.  We don't maintain them.  I suppose we don't distribute
most of them.

We could distribute both s.el and magnars-string.el (strings.el), two
files which are almost the same and should be changed in sync.  That
would do the job, but it is undesirable in terms of maintenance.

`load-with-shorthands' is a way of making strings.el use a modified
virtual copy of s.el, in a way that avoids actually duplicating the
contents of s.el.

  > If I find a bug or want to add some function to that presumed s.el that
  > your string.el is loading specially, I have to somehow remember to load
  > it with shorthands.

What is the precise scenario here?  I don't see it.  If we found a
need to modify s.el, we would modify our version of s.el.  With my
implementation, that would DTRT for programs that use s.el and for
programs that use strings.el.

If we wanted to add something to strings.el, we would add it directly
to strings.el.  The shorthands would not apply to the contents of
strings.el itself.  They would only apply to the contents of s.el
when it is loaded by strings.el.

  >   The usual commands C-M-x, M-x load-file, M-x
  > emacs-lisp-byte-compile-and-load will all produce wrong results that I
  > won't know how to fix.

I don't follow you.  I don't see why anything would produce wrong
results.  If you show me a specific scenario, I might understand.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Help sought understanding shorthands wrt modules/packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I don't agree.  Before shorthands, a symbol had one name, after, it can
  > have many.

That is a conceptual misunderstanding.  Making a local alias for a symbol
does not give it "many names".

In this regard, there is no deep difference between shorthands and
:USE.  They both create alternate read syntax for a specific symbol
but it still has only one name.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Help sought understanding shorthands wrt modules/packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I must admit that I don't understand what you are referring to.  Could
  > you please give an example of a module system doing that?

I can't.  I have never written code in a language which has such a
module system.  I read about these module systems decades ago.
For more details, or examples, you will need to ask people who have
actually used them.

                                                               At first
  > glance It sounds like visibility rules, or something, but I don't
  > remember a language with different visibility rules for functions and
  > variables, ATM.

What I recall is NOT that there are "different visibility rules",
but rather that the visibility rules would apply to specific definitions.

Perhaps in those languages a given symbol can have only one active
definition in any scope.  So a symbol could be have a function definition
or a variable definition in any given module, but never both.

I think that is the situation in Scheme.  But I have never programmed
in Scheme.

  > > The grave problem of :USE in CL packages could be fixed by replacing
  > > it with a new construct that specifies a list of symbols to be
  > > inherited from each other package, perhaos with renaming.

  > I fail to understand the "grave" problem, sorry.  Could you please give
  > a more concrete example?

Perhaps "shadow" or "shadowing-import" amount to the construct I was
envisioning.  I saw those terms only today, and I don't know what they
mean.

  > > We could call this system "corrected CL packages."
  > > It would not be compatible but it would be better.

  > It could be compatible, at least I don't see a reason right now why it
  > couldn't.

It could be made upward-compatible, but not fully compatible: programs
using :INHERIT would not load in standard Common Lisp.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Help sought understanding shorthands wrt modules/packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > We could hide this concatenation most of the time by making print
  > > cut off the package prefixes that are redundant in their context.

  > See, that's what I meant when I wrote "there is no design".

I am brainstorming, and you rebuke me because it isn't a finished
design.

                                                                 We could,
  > perhaps, maybe, maybe not, hm.

That is what brainstorming looks like.  You describe the process of
brainstorming with words designed to ridicule.  That is unreasonable
and it is unkind.

Please don't do that.

   Why not use what's there and works?

Because it has a bad design flaw in :USE, and it is a lot of added
complexity that we are doing fine without.  It exists "there", for
some value of "there", but I am glad it is not _here_ in Emacs.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Help sought understanding shorthands wrt modules/packages

[João Távora]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

>> Anyway, I can't understand how the presence of shorthands can negatively
>> impact CL packages.  If co-existence is complicated (but is it?) it
>> doesn't seem hard to make either shorthands or CL-packages a noop in in
>> files that prefer one of the systems.  For example, the reader can just
>> throw away any shorthand info altogether as soon as it detects that
>> CL:*CURRENT-PACKAGE* (or whatever equivalent you're envisioning) is not
>> the default one.  Or CL:IN-PACKAGE can just error out if it detects that
>> read-symbol-shorthands is non-nil.
>
> If I implied that then I have to apologize.  I meant to convey that

No need to apologize at all :-) 

> (a) I haven't done any experimeent whatsoever with shorthands in
> feature/pkg because I set that delibaretly out-of-scope.  One has to
> draw a line somewhere.

>
> (b) I can't exclude the possibility of non-backwards-compatibility due
> to the use of shorthands.  At least they way I've been doing things in
> the branch.  It's also not proven to be incompatible.  It's unknown.

You can make some tests.  If one goes boom, then we'll know for sure.
If none go boom, well, it's a good indication, but it's very hard to
prove a negative.  I mean, can you really prove Emacs didn't cause the
Hindeburg??

> (c) I might have an idea how do things differently to achieve
> backwards-compatibility.  To what extent that works, or if it works at
> all, I also don't know, and likely won't find out any time soon.
>
>> But even if the read logic didn't do that, and considered the two
>> systems at once, I'm still not sure there would be any ambiguity.
>
> From my point of view, the gist of the matter is
> backwards-compatibility.  That's the pain part.

To be honest, I don't think you should focus effort on this.  Just
declare shorthands out of bounds when CL packages are used, and then go
move on with your project.

IOW proclaim that shorthands only ever worked in the default package and
will continue working only in the default package (the main obarray).
This is absolutely true and saying so is not breaking backward
compatibility.  What's more, no-one will ever request that shorthands
work outside the main package, where they are useless anyway.  It would
solve any compatibility packages.  In practice, this can be as simple as
putting some check for a future CL:*CURRENT-PACKAGE* somewhere in the
current shorthand-consulting reader.

João

Re: Help sought understanding shorthands wrt modules/packages

[João Távora]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > The magnars-string.el file is meant to supersede s.el and be a
>   > well-behaved version of it.  The s.el file, as it exists today, is a
>   > namespace-polluting liability, we shouldn't maintain it.
>
> No one would be happer than I to eliminate te usual version of s.el,
> but my understanding is that that would cause a big practical problem.
>
> Lots of packages do (require 's), and they call those functions by the
> names s-... defined in s.el.  If we don't have s.el, they will break.
> They can't use your proposed magnars-string.el (which I would call
> stris.el), since the names it would define are wrong _for them_.

I don't think you're following my plan, which was the plan all along.

Here's the story from the beginning:

* Once upon a time, s.el was a namespace polluting liability, and this
  was part of the reason why it wasn't in GNU Emacs or GNU ELPA.

* s.el was a very popular kid.  Many other packages foo.el and bar.el
  depended on it.  They were also popular.  But the namespacing
  pollution of s.el prevented us from including those in GNU ELPA, too.

* Right, so we wanted to integrate s.el into Emacs core or GNU Elpa
  "somehow", but how could we do that?

  We developed shorthands for Emacs 28.

* OK, but what did we do with them?

  We made a new file magnars-string.el which was absolutely identical to
  s.el but for the local read-symbol-shorthands cookie (and the provide
  form.)

  Then we submitted mangnars-string.el to Emacs core or GNU ELPA.

* When foo.el and bar.el wanted to enter GNU ELPA, they just upgraded
  their string library from s.el to magnars-string.el, of course.

* Oh no, but wasn't upgrading them so very complicated?

  No at all, it amounted to changing a single line, in foo.el:
    (require 's)
  to
    (require 'magnars-string)

  And adding the SAME read-symbol-shorthands cookie in the trailer of
  the file.

  ;; foo.el ends here
  ;; Local Variables:
  ;; read-symbol-shorthands: (("s-" . "magnars-string-"))
  ;; End:

* But but, what about all the the references to s-trim and s-append in
  lines 1234 and 876 of foo.el?  

  They became, magically, references to the symbols magnars-string-trim
  and magnars-string-append.

* And that's all?  They lived happily ever after?

  Yes, that's about it.  The usual copyright stuff was changed in foo.el
  and bar.el, the name of the package was endelessly bikeshed, Stefan
  Monnier submitted many patches (mostly for pcase usage), and yes, they
  lived happily.

Epilogue:

* One night, Saint Ignucius came down to tell the world not to use s.el
  anymore, because a better version had appeared in GNU land.  This
  being a world of sinners, most of it didn't listen or care.  Splitters!

  Saint Ignucius then also asked the maintainers to stop maintaining
  their s.el.  Let's presume they did listen.

  One day, some package fossil.el, which didn't listen, found a bug in
  its s.el dependency.  Should that bug have been fixed in s.el?  No, it
  was fixed in magnars-string.el.  fossil.el was told to upgrade to
  magnars-string.el, because it is very easy.  Now fossil.el was
  enlightened, too.

  Not so fast! fossil.el was meant to work on Emacs 27 and Emacs 6 and
  Emacs 500BC!  No problem, that's didn't change the plan.  fossil.el
  was upgraded to use magnars-string.el anyway.

  And that new fossil.el was loaded into Emacs 27 along with
  magnars-string.el.  They loaded fine, and the presence of the
  read-symbol-shorthands file-local variables there didn't affect it.

  And fossil.el worked just fine there.  The namespace was polluted with
  s- symbols, that's true, but the situation was no worse than before.

  One day, the user simply upgraded to Emacs 28, kept all his packages
  in the same place, and the namespace pollution simply went away.

>   >   The usual commands C-M-x, M-x load-file, M-x
>   > emacs-lisp-byte-compile-and-load will all produce wrong results that I
>   > won't know how to fix.
>
> I don't follow you.  I don't see why anything would produce wrong
> results.  If you show me a specific scenario, I might understand.

If you do these actions in the s.el file, the one where the actual code
is and the one you want to keep unchanged, it will just intern new
s-prefixed symbols in the obarray!  I.e. you won't change the renamed
symbol's definition -- the one begotten by that load-with-shorthands
call -- at all.

Re: Help sought understanding shorthands wrt modules/packages

[Gerd Möllmann]

João Távora <joaotavora@gmail.com> writes:

> You can make some tests.  If one goes boom, then we'll know for sure.
> If none go boom, well, it's a good indication, but it's very hard to
> prove a negative.  I mean, can you really prove Emacs didn't cause the
> Hindeburg??

Now that you say it, I think it blew up Northstream :-).

Nah, I'd rather procrastinate.

>> (c) I might have an idea how do things differently to achieve
>> backwards-compatibility.  To what extent that works, or if it works at
>> all, I also don't know, and likely won't find out any time soon.
>>
>>> But even if the read logic didn't do that, and considered the two
>>> systems at once, I'm still not sure there would be any ambiguity.
>>
>> From my point of view, the gist of the matter is
>> backwards-compatibility.  That's the pain part.
>
> To be honest, I don't think you should focus effort on this.  Just
> declare shorthands out of bounds when CL packages are used, and then go
> move on with your project.

I could and maybe I would, but I've lost interest.

Maybe you've encountered that yourself, as a programmer?  You suddenly
realize at some point that the interesting part of a project is over,
and all what follows will be boring?  That's kind of where I am.

Re: Help sought understanding shorthands wrt modules/packages

[Dmitry Gutov]

On 12.11.2022 12:11, João Távora wrote:
>    Saint Ignucius then also asked the maintainers to stop maintaining
>    their s.el.  Let's presume they did listen.

Did they?

Re: Help sought understanding shorthands wrt modules/packages

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 421 bytes --]

On Sat, Nov 12, 2022, 14:36 Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 12.11.2022 12:11, João Távora wrote:
> >    Saint Ignucius then also asked the maintainers to stop maintaining
> >    their s.el.  Let's presume they did listen.
>
> Did they?
>

Did the whole fairy-tale schtick fly over your head? None of this has
happened yet: I'm just presenting a plan. If people play along is not up to
me.

>

[-- Attachment #2: Type: text/html, Size: 918 bytes --]

Re: Help sought understanding shorthands wrt modules/packages

[Dmitry Gutov]

On 12.11.2022 17:20, João Távora wrote:
> On Sat, Nov 12, 2022, 14:36 Dmitry Gutov <dgutov@yandex.ru 
> <mailto:dgutov@yandex.ru>> wrote:
> 
>     On 12.11.2022 12:11, João Távora wrote:
>      >    Saint Ignucius then also asked the maintainers to stop maintaining
>      >    their s.el.  Let's presume they did listen.
> 
>     Did they?
> 
> 
> Did the whole fairy-tale schtick fly over your head? None of this has 
> happened yet: I'm just presenting a plan. If people play along is not up 
> to me.

It would seem useful to do some dev-rel work in advance. Or maybe now?

Otherwise, we've just acquired some extra complexity in the reader, xref 
and elisp completion code, while the anticipated benefits are very slow 
to materialize.

Half a year has passed now since Emacs 28.1's release, and even longer 
than that since the "shorthands" have been installed on master, and I 
don't see a single mention of them on s.el's issue tracker.

Ditto for dash.el and f.el.

Re: Help sought understanding shorthands wrt modules/packages

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 12.11.2022 17:20, João Távora wrote:
>> On Sat, Nov 12, 2022, 14:36 Dmitry Gutov <dgutov@yandex.ru
>> <mailto:dgutov@yandex.ru>> wrote:
>>     On 12.11.2022 12:11, João Távora wrote:
>>      >    Saint Ignucius then also asked the maintainers to stop maintaining
>>      >    their s.el.  Let's presume they did listen.
>>     Did they?
>> Did the whole fairy-tale schtick fly over your head? None of this
>> has happened yet: I'm just presenting a plan. If people play along
>> is not up to me.
>
> It would seem useful to do some dev-rel work in advance. Or maybe now?

Yes, maybe.  But maybe we should first agree on the plan.  We're not
there yet: Richard has a completely different idea, that I don't think
is technically sound.

I've tried to explain my original plan again.  It would seem you have
read it: do you understand it?  Do you see anything technically wrong
with it?  If you don't then maybe you could help convince others that
it's a viable plan

> Otherwise, we've just acquired some extra complexity in the reader,
> xref and elisp completion code, while the anticipated benefits are
> very slow to materialize.

I'm not sure the complexity is that much (it was a long time ago, but I
believe it was reasonably low).  But you're right, the benefits of
shorthands so far are just programmer convenience for libraries outside
of Emacs.  I've enjoyed using them, and I would guess a small number of
programmers are also taking advantage of them them in some side
projects, judging from a quick GitHub code search.

> Half a year has passed now since Emacs 28.1's release, and even longer
> than that since the "shorthands" have been installed on master, and I
> don't see a single mention of them on s.el's issue tracker.
>
> Ditto for dash.el and f.el.

For the record, I think you're completely right.  But hey, this is
Emacs, it moves slowly.  Again, if you understand the plan I put forth
and you think it's viable, then speak up and let's get things moving.

João

Re: Help sought understanding shorthands wrt modules/packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > There is no control over this, at least in the present implementation.
  > Nothing makes sure or warns if you bind read-symbol-shorthands in
  > whichever way you want.

That's the Lisp tradition -- you get a lot of control, and we don't
try to stop you from messing up various Lisp-level data structures
used in the system.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Help sought understanding shorthands wrt modules/packages

[Gerd Möllmann]

On 13.11.22 05:17, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>    > There is no control over this, at least in the present implementation.
>    > Nothing makes sure or warns if you bind read-symbol-shorthands in
>    > whichever way you want.

Out of context.

Re: Help sought understanding shorthands wrt modules/packages

[Dmitry Gutov]

On 12.11.2022 20:45, João Távora wrote:

> I've tried to explain my original plan again.  It would seem you have
> read it: do you understand it?  Do you see anything technically wrong
> with it?  If you don't then maybe you could help convince others that
> it's a viable plan

It sounds workable, but I'm not convinced myself that this approach is 
going to be attractive enough for third-party package authors. As well 
as for the author of s.el. And if I don't see it as attractive, what 
hope would I have of promoting it?

First: would the author want to rename it to magnars-string or strings? 
I don't know, both seem more generic than 's.el' which is pretty 
well-known by now. And if you look at some discussions, he doesn't want 
to break compatibility with older Emacs without very good reason.

Second: why is the existence of s.el a problem? Does somebody else want 
to use this name? I could understand it if we wanted to introduce such 
namespace in the core, but as some related discussion has shown, the 
maintainers don't value this kind of naming scheme (consistent prefixed 
naming and very short names), so in the end what? It will sit unused?

The main value of s.el IIUC is in its superficial qualities: the 
easy-to-grasp names, the searchable README.md which lists all the 
methods, and the naming of functions which is friendlier to Clojure 
users. If we take away the naming, what's left is perhaps a dozen of 
added convenience functions.

>> Otherwise, we've just acquired some extra complexity in the reader,
>> xref and elisp completion code, while the anticipated benefits are
>> very slow to materialize.
> 
> I'm not sure the complexity is that much (it was a long time ago, but I
> believe it was reasonably low).  But you're right, the benefits of
> shorthands so far are just programmer convenience for libraries outside
> of Emacs.  I've enjoyed using them, and I would guess a small number of
> programmers are also taking advantage of them them in some side
> projects, judging from a quick GitHub code search.

My own search on GitHub has yielded nothing so far.

>> Half a year has passed now since Emacs 28.1's release, and even longer
>> than that since the "shorthands" have been installed on master, and I
>> don't see a single mention of them on s.el's issue tracker.
>>
>> Ditto for dash.el and f.el.
> 
> For the record, I think you're completely right.  But hey, this is
> Emacs, it moves slowly.  Again, if you understand the plan I put forth
> and you think it's viable, then speak up and let's get things moving.

Third-party packages also usually want to keep compatibility with 
several older Emacs versions. So perhaps we can see some movement when 
the year 2028 rolls over, and I will be proven wrong.

Re: Help sought understanding shorthands wrt modules/packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Half a year has passed now since Emacs 28.1's release, and even longer 
  > than that since the "shorthands" have been installed on master, and I 
  > don't see a single mention of them on s.el's issue tracker.

Why would they be mentioned there, of all places?
We are not proposing that Magnars make changes in his release.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Help sought understanding shorthands wrt modules/packages

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 12.11.2022 20:45, João Távora wrote:
>> I've tried to explain my original plan again.  It would seem you have
>> read it: do you understand it?  Do you see anything technically wrong
>> with it?  If you don't then maybe you could help convince others that
>> it's a viable plan
> It sounds workable, but I'm not convinced myself that this approach is
> going to be attractive enough for third-party package authors. As well
> as for the author of s.el. And if I don't see it as attractive, what
> hope would I have of promoting it?

It's very simple: either you think there's a problem with s.el and this
mini-refix gunk, or you don't.  If you don't, you should probably be
writing to someone else: this discussion is about a plan to solve it,
but it presume it exists.

If, OTOH, you think it is a problem, and you've a plan for solving it
that somehow doesn't involve changing things, I'd like to hear it.  My
plan changes a name, adds two lines of boilerplate, keeps the rest
intact -- and solves the problem. It's not only "workable": it's the
only one.

>> believe it was reasonably low).  But you're right, the benefits of
>> shorthands so far are just programmer convenience for libraries outside
>> of Emacs.  I've enjoyed using them, and I would guess a small number of
>> programmers are also taking advantage of them them in some side
>> projects, judging from a quick GitHub code search.
>
> My own search on GitHub has yielded nothing so far.

I searched for 'read-symbol-shorthands' and turned up hits like this

https://github.com/include-yy/yyorg-bookmark/blob/47602869ad19853957cddfe01a81a04a9e3bda1c/yyorg-bookmark.el
https://github.com/mm--/dot-emacs/blob/2387d914a93fbd72b54e389a030b5dd0459a1c3b/elisp/jmm-notmuch-stuff.el
https://github.com/abgruszecki/dotfiles/blob/ba6763d0eade0f62ecc543e9a46501241aa88138/spacemacs/.spacemacs.d/bespoke.el
https://github.com/mm--/dot-emacs/blob/3ae1274e9de1641c6338a98b9659ce69477fbd0b/elisp/jmm-inkscape.el
https://github.com/ilmotta/dotemacs/blob/a6ecf9b415468b023ac070f958a084021c4db398/lisp/lib-media.el

along with my beardbolt.el library.  I don't know what you were
searching for.

Re: Help sought understanding shorthands wrt modules/packages

[Dmitry Gutov]

On 14.11.2022 08:33, João Távora wrote:
>>> believe it was reasonably low).  But you're right, the benefits of
>>> shorthands so far are just programmer convenience for libraries outside
>>> of Emacs.  I've enjoyed using them, and I would guess a small number of
>>> programmers are also taking advantage of them them in some side
>>> projects, judging from a quick GitHub code search.
>> My own search on GitHub has yielded nothing so far.
> I searched for 'read-symbol-shorthands' and turned up hits like this
> 
> https://github.com/include-yy/yyorg-bookmark/blob/47602869ad19853957cddfe01a81a04a9e3bda1c/yyorg-bookmark.el
> https://github.com/mm--/dot-emacs/blob/2387d914a93fbd72b54e389a030b5dd0459a1c3b/elisp/jmm-notmuch-stuff.el
> https://github.com/abgruszecki/dotfiles/blob/ba6763d0eade0f62ecc543e9a46501241aa88138/spacemacs/.spacemacs.d/bespoke.el
> https://github.com/mm--/dot-emacs/blob/3ae1274e9de1641c6338a98b9659ce69477fbd0b/elisp/jmm-inkscape.el
> https://github.com/ilmotta/dotemacs/blob/a6ecf9b415468b023ac070f958a084021c4db398/lisp/lib-media.el
> 
> along with my beardbolt.el library.  I don't know what you were
> searching for.

Searched for exact same thing. Got pages and pages of hits in different 
forks of the Emacs repo.

Cool, so some people do enjoy it.

Re: Help sought understanding shorthands wrt modules/packages

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1502 bytes --]

On Mon, Nov 14, 2022 at 11:41 AM Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 14.11.2022 08:33, João Távora wrote:
> >>> believe it was reasonably low).  But you're right, the benefits of
> >>> shorthands so far are just programmer convenience for libraries outside
> >>> of Emacs.  I've enjoyed using them, and I would guess a small number of
> >>> programmers are also taking advantage of them them in some side
> >>> projects, judging from a quick GitHub code search.
> >> My own search on GitHub has yielded nothing so far.
> > I searched for 'read-symbol-shorthands' and turned up hits like this
> >
> >
> https://github.com/include-yy/yyorg-bookmark/blob/47602869ad19853957cddfe01a81a04a9e3bda1c/yyorg-bookmark.el
> >
> https://github.com/mm--/dot-emacs/blob/2387d914a93fbd72b54e389a030b5dd0459a1c3b/elisp/jmm-notmuch-stuff.el
> >
> https://github.com/abgruszecki/dotfiles/blob/ba6763d0eade0f62ecc543e9a46501241aa88138/spacemacs/.spacemacs.d/bespoke.el
> >
> https://github.com/mm--/dot-emacs/blob/3ae1274e9de1641c6338a98b9659ce69477fbd0b/elisp/jmm-inkscape.el
> >
> https://github.com/ilmotta/dotemacs/blob/a6ecf9b415468b023ac070f958a084021c4db398/lisp/lib-media.el
> >
> > along with my beardbolt.el library.  I don't know what you were
> > searching for.
>
> Searched for exact same thing. Got pages and pages of hits in different
> forks of the Emacs repo.


Yes, Github code search is very poor, but I think it's the best we have
to judge what is out there.

[-- Attachment #2: Type: text/html, Size: 2698 bytes --]

RE: [External] : Re: Help sought understanding shorthands wrt modules/packages

[Drew Adams]

> Perhaps "shadow" or "shadowing-import" amount
> to the construct I was envisioning.  I saw
> those terms only today, and I don't know what
> they mean.

FWIW, I'd (passively) like to see this topic progress.

I was thinking that the back & forth in this thread
was  starting to be helpful.  I'm sorry to see it
dropped (apparently).  Maybe it's a hard slog, but
maybe it's a slog worth working through.

If Richard, Gerd, and whomever else would lock
themselves together in a room (virtual or real) for
a weekend and come to senses & consensus, that
might result in a Hallelujah, maybe even a Eureka!,
moment.  Might or might not.  In lieu of that room,
there's this thread...

How about it?  How about hassling a bit more over
coming up with something useful for Emacs Lisp?  It
doesn't need to be CL-compatible.  Maybe it doesn't
need to be Elisp backward-compatible.  What it
should be, I think, is what's most useful for Elisp
(its future) - and Emacs users.
____

Some CL homework maybe wouldn't hurt.  If you (RMS)
saw the term "shadow" only today (a week ago) then
maybe it's time to catch up on CL packages as
they've been for a few decades now.  Just sayin' - 
wouldn't hurt.  And yes, your arguments are good
ones - thanks for helping advance the schmilblick.


CL's consistency rules:

https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node112.html

Shadowing and name conflicts:

https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node115.html

https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node116.html

CL package functions, altogether:

https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node118.html

____

Now go DTRT! ;-)

____

And then, for better or worse, there are modules...
Those are more like what Emacs calls "packages", no?

https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node119.html

Re: Help sought understanding shorthands wrt modules/packages

[Eli Zaretskii]

> From: Drew Adams <drew.adams@oracle.com>
> CC: "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> Date: Sat, 19 Nov 2022 22:51:15 +0000
> 
> FWIW, I'd (passively) like to see this topic progress.

What kind of progress?  Is deciding that we don't (yet) need packages in
Emacs Lisp going to be considered "progress" in your opinion?

> If Richard, Gerd, and whomever else would lock
> themselves together in a room (virtual or real) for
> a weekend and come to senses & consensus, that
> might result in a Hallelujah, maybe even a Eureka!,
> moment.  Might or might not.  In lieu of that room,
> there's this thread...

Sorry, but it isn't just up to Richard and Gerd.  As I wrote at the
beginning of this thread, I'm not convinced we need this kind of
functionality in Emacs Lisp (and I explained why I thought so).  Until
someone comes up with convincing arguments, I still submit that adding this
kind of complexity into Emacs internals for the sake of functionality that
is not really needed is not a good idea.  (And no, "CL has it" is not a
convincing argument in my book.)

Re: [External] : Re: Help sought understanding shorthands wrt modules/packages

[Gerd Möllmann]

Drew Adams <drew.adams@oracle.com> writes:

>> Perhaps "shadow" or "shadowing-import" amount
>> to the construct I was envisioning.  I saw
>> those terms only today, and I don't know what
>> they mean.
>
> FWIW, I'd (passively) like to see this topic progress.
>
> I was thinking that the back & forth in this thread
> was  starting to be helpful.  I'm sorry to see it
> dropped (apparently).  Maybe it's a hard slog, but
> maybe it's a slog worth working through.

Sorry, but no thanks from me.

Why?  I don't think there is anything to talk about.  Or did you find
out what Richard's plan is?  Last I read was that it's not what Joao had
in mind, at least, and that Richard is brainstorming.

And, to be honest, the papal bulls concerning CL packages, in ignorance
of one's own ignorance, the non-answers, etc.---I don't want to have to
deal with that.

> Now go DTRT! ;-)

I do :-).

RE: [External] : Re: Help sought understanding shorthands wrt modules/packages

[Drew Adams]

> > FWIW, I'd (passively) like to see this topic progress.
> 
> What kind of progress?  Is deciding that we don't (yet) need packages in
> Emacs Lisp going to be considered "progress" in your opinion?

To me, progress on the question would include more
investigation and discussion.  It's not just about
deciding.

But if such a decision has already been made, so
be it.  I didn't realize that the question was
closed.

> > If Richard, Gerd, and whomever else would lock
> > themselves together in a room (virtual or real) for
> > a weekend and come to senses & consensus, that
> > might result in a Hallelujah, maybe even a Eureka!,
> > moment.  Might or might not.  In lieu of that room,
> > there's this thread...
> 
> Sorry, but it isn't just up to Richard and Gerd.  

Sorry, but I didn't say or suggest that _anything_
was up to Richard and Gerd, let alone just up to
them.

> As I wrote at the
> beginning of this thread, I'm not convinced we need this kind of
> functionality in Emacs Lisp (and I explained why I thought so).

What kind of functionality is "this kind"?  Would
you mind repeating that, if you explained it, and
summarizing your "why" - or else point to where
you've already done that?

If you decided (e.g., at the outset) that this
kind of functionality isn't needed, I guess I
missed that announcement.  But what kind?  Just
CL packages, or anything at all that resembles
them?  Or any attempt to improve handling of
namespaces?

Certainly, if there's no problem then there's no
need to look for a solution.  I didn't realize
that the premise had been decided.

> Until someone comes up with convincing arguments, I still submit that adding
> this kind of complexity into Emacs internals for the sake of functionality
> that is not really needed is not a good idea.  (And no, "CL has it" is not a
> convincing argument in my book.)

I see.  So be it.

Did someone say that "CL has it" is a convincing
argument for why Elisp should have it - whatever
you might think "it" is?  I don't think so.

In any case, given that you're not yet convinced
of a need/use for better namespace control, and
given that Gerd's given up, I guess that if there
is such a need/benefit then the question will be
off the table for a while.  Thanks for making
this clear - I missed it.

Re: Help sought understanding shorthands wrt modules/packages

[Eli Zaretskii]

> From: Drew Adams <drew.adams@oracle.com>
> CC: "rms@gnu.org" <rms@gnu.org>,
>         "gerd.moellmann@gmail.com"
> 	<gerd.moellmann@gmail.com>,
>         "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> Date: Sun, 20 Nov 2022 18:55:54 +0000
> 
> > As I wrote at the
> > beginning of this thread, I'm not convinced we need this kind of
> > functionality in Emacs Lisp (and I explained why I thought so).
> 
> What kind of functionality is "this kind"?  Would
> you mind repeating that, if you explained it, and
> summarizing your "why" - or else point to where
> you've already done that?

Sorry, I cannot afford repeating that.  Please read the beginning of the
thread.

> If you decided (e.g., at the outset) that this
> kind of functionality isn't needed, I guess I
> missed that announcement.  But what kind?

Packages.

> Did someone say that "CL has it" is a convincing
> argument for why Elisp should have it - whatever
> you might think "it" is?

That's the only argument I've heard.

Re: [External] : Re: Help sought understanding shorthands wrt modules/packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I am sorry I have not been able to say anything about shorthands for a
week or more.  I have been getting 1000 messages a day lately and I've
been unable to catch up.

Same thing with the Eglot manual, which I want to take a look at.  The
question is, should Eglot be documented as a feature, in one place?
Or is it an aspect that exists in many different features, and that
aspect of features F and G should be documented in the section on
feature F and the section on feature G.



-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [External] : Re: Help sought understanding shorthands wrt modules/packages

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: eliz@gnu.org, gerd.moellmann@gmail.com, emacs-devel@gnu.org
> Date: Tue, 22 Nov 2022 07:14:43 -0500
> 
> Same thing with the Eglot manual, which I want to take a look at.  The
> question is, should Eglot be documented as a feature, in one place?
> Or is it an aspect that exists in many different features, and that
> aspect of features F and G should be documented in the section on
> feature F and the section on feature G.

There are aspects of Eglot that belong to both classes.  So we have an Eglot
manual, and we also document its support for relevant Emacs features where
those features are documented.

Re: Help sought understanding shorthands wrt modules/packages

[Matt Armstrong]

Richard Stallman <rms@gnu.org> writes:

> The only drawback of our current use of naming conventions is the need
> to write long names in your code when referring to functions and
> variables defined in other parts of Emacs.  If your code refers to
> many such entry points of a certain part of Emacs, maybe you want to
> abbreviate those.

I think there is another drawback of Emacs' prefix based namespace
convention: that of longer local (private, non-public, unexported, etc.)
names.  In C, this is achieved with static functions and variables.  In
Common Lisp this is achieved by exporting only the public symbols.

Re: Help sought understanding shorthands wrt modules/packages

[Eli Zaretskii]

> From: Matt Armstrong <matt@rfc20.org>
> Cc: emacs-devel@gnu.org
> Date: Tue, 22 Nov 2022 10:01:33 -0800
> 
> I think there is another drawback of Emacs' prefix based namespace
> convention: that of longer local (private, non-public, unexported, etc.)
> names.

Why is that a drawback?  (It's a serious question.)

Re: Help sought understanding shorthands wrt modules/packages

[Matt Armstrong]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

>> The grave problem of :USE in CL packages could be fixed by replacing
>> it with a new construct that specifies a list of symbols to be
>> inherited from each other package, perhaos with renaming.
>
> I fail to understand the "grave" problem, sorry.  Could you please give
> a more concrete example?

A few weeks ago, in this thread or another, I attempted to explain my
understanding of the point Richard was raising.  As far as I can tell I
still share the same concern Richard raises.

The issue is that `(use-package "A")' brings all of A's exported symbols
into the current namespace, and so allows them to be used unqualified.
This implies that any addition to a package A's exported symbol list has
a chance of breaking code in any package that "use-package"'s it.  The
breakage takes the form of an introduced ambiguity.

This is a problem for "programming in the large".  If package's can't
add new things without risking breaking their users then the approach is
arguably worse than Emacs' current approach, at least in this respect.
Adding a new name for a new piece of functionality is one of the most
common ways packages evolve over time.


> BTW, in CL, if you don't want to use-package you can also import just
> the list of symbols you want.  Without renaming, but I guess that
> could be added, if somone wants that.  (I mention the idea of renaming
> in cl-packages.org, BTW, on the branch, but I'm still not sure it's
> worth it.)

Yes, Common Lisp provides ways to fix things *after* the breakage
occurs, but the harm comes with the breakage, and is an inevitable
consequence of the design.

Yes, the danger can be avoided by refraining from using `use-package' at
all.  This is the root my suggestion to consider CL packages for Emacs
but subtract `use-package'.

Yes, this kind of problem exists not only in Common Lisp but many other
languages, to the extent that some might come to believe this sort of
problem feels "natural" or is perhaps worth it for the convenience of
using unqualified names from other packages.  Python's "from foo
import*" is like CL's use-package and is now widely considered poor
style in the Python community (for the reasons I've stated).  Guile has
something like Common Lisp, with the same problem and the same kinds of
mitigating workarounds.  Ada has "use" with semantics similar to CL's.
Perl has "use" but, presumably through experience, recognized a problem
and has a "use VERSION" mechanism to limit potential harm from API
changes.  And on and on.

The argument is that all of these languages are all flawed in the same
way, and that a package/module system designed today should aim for
better.

There are languages that avoid this problem.  Java avoids it, as does
Go, and as far as I can tell Rust does too.  These languages simply do
not providing a way to slurp entire namespaces into other namespaces.
In these languages referring to names in other packages requires either
a package name qualifier, which can be renamed locally to something
short if desired, or by explicitly pulling individual names, one by one,
into the current scope.  With this there is no way for an addition of a
name in one package to cause a naming conflict or ambiguity in any
other.


>> Eli wrote:
>>
>>   > Moreover, from my POV, the jury is still out on the question of
>>   > whether we at all need packages in Emacs.  "Programming in the large"
>>   > doesn't sound very relevant to how Emacs Lisp is used.
>>
>> I share this feeling, except that I take it a bit further.  In the
>> 1980s I thought about whether we should have packages in Emacs, and
>> concluded that using a naming conventions is simpler, easier and more
>> flexible.  So I chose that.
>>
>> If CL packages were unproblematical, I'd say, "Sure, why not?  We
>> aren't desperate to keep Emacs small nowadays."  But they are
>> problematical, and that is a reason to choose to do without them.
>
> Please spend some effort to explain the problematic cases.

Done, I think.  I suspect you already understood the point, but merely
disagree about its importance?

To Eli's "jury is still out" point:

So far I have not seen a clear description of the problems with Emacs'
current approach, so there is not even a common understanding of what
the problems are, much less any consensus over how serious they are.
Without that, I think there is little hope of weighing different
alternatives productively.

As far as Common Lisp style packages, well, Common Lisp is a stable set
of compromises designed to unify disparate lisp implementations in the
80's.  It is stable and well understood, which has some benefits.  But
it is not clear to me that, when considering designs for Emacs today,
that there is any sort of "implied fitness for purpose" that comes from
Common Lisp designs.  The question and problem of adding a packaging
system to Emacs today strikes me as a very different thing from the
kinds of issues and problems originally considered in the 80's for
Common Lisp.  Specifically, back then "use-package" may have been a
compromise seen as a necessary, pragmatic, but suboptimal solution to
some problem of the day (e.g. perhaps making old code work without much
change while still retrofitting a package system).  It isn't obvious to
me that Emacs has the same problem(s) or that it should, necessarily,
adopt the same solutions.

Re: Help sought understanding shorthands wrt modules/packages

[Matt Armstrong]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Matt Armstrong <matt@rfc20.org>
>> Cc: emacs-devel@gnu.org
>> Date: Tue, 22 Nov 2022 10:01:33 -0800
>>
>> I think there is another drawback of Emacs' prefix based namespace
>> convention: that of longer local (private, non-public, unexported, etc.)
>> names.
>
> Why is that a drawback?  (It's a serious question.)

Thanks for asking.

I have to admit that my answer boils down to familiarity.  I'd like
programming in Emacs Lisp to feel more like the other programming
languages I'm more familiar with.

When I have written Emacs Lisp packages the package name prefixes have
been a bit of a chore to deal with.  I'm happier programming in
languages where the namespace context is a property of a scope, usually
a file level context.  This way, the "current" namespace need not be
re-stated throughout a package's code every time a name is mentioned.

This is a different concern than how code references names names from
other packages.  If I'm writing code that uses something from some other
package, I'm fine stating that other package's name, especially if the
language lets me abbreviate longer package names (usually for the local
file/scope).

I believe the general dislike for having to re-state the associated
package within the package code itself is one reason why people write
packages like https://elpa.gnu.org/packages/nameless.html.

My guess is that avoiding this kind of repetitive verbosity, at least
for the common cases, is what many people want to get out of "Emacs
Packages/Modules".  That is just a guess, though.  I can't really tell
what benefits people actually want when they "it'd be great if Emacs had
CL style packages."

I have to say that, personally, CL style packages didn't "cut it" for me
either.  They did make it easier to have package-private names that were
boilerplate free, but they didn't make it obvious which of my symbols
were exported and which weren't.  That was a property of not of the
symbol name, not of its `defun/defvar', but of whether it appeared in
some export list somewhere else, often in a different file.  I didn't
think that was particularly great, either.

Re: Help sought understanding shorthands wrt modules/packages

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 932 bytes --]

On Tue, Nov 22, 2022 at 8:38 PM Matt Armstrong <matt@rfc20.org> wrote:

> The argument is that all of these languages are all flawed in the same
> way, and that a package/module system designed today should aim for
> better.

That's an overgeneralization. Ada, for example, is free from Beaujolais
effects, and it supports overloading of names, so use'ing a package will,
at most, cause collisions detected at compile time and easily fixed. (And
even these collisions will only happen if the overloaded functions have a
similar parameter profile and you happen to use one of them in your code).

Of course in Ada it is, perhaps, more common today to rename imported
packages as a short prefix, instead of "using" them, but that's for
readability, which is a big emphasis in Ada culture.

I won't claim Ada packages are a perfect solution, but certainly they don't
seem to me "flawed in the same way" that some others you mentioned.

[-- Attachment #2: Type: text/html, Size: 1173 bytes --]

Re: Help sought understanding shorthands wrt modules/packages

[Gerd Möllmann]

On 22.11.22 20:37, Matt Armstrong wrote:
> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
> 
>>> The grave problem of :USE in CL packages could be fixed by replacing
>>> it with a new construct that specifies a list of symbols to be
>>> inherited from each other package, perhaos with renaming.
>>
>> I fail to understand the "grave" problem, sorry.  Could you please give
>> a more concrete example?
> 
> A few weeks ago, in this thread or another, I attempted to explain my
> understanding of the point Richard was raising.  As far as I can tell I
> still share the same concern Richard raises.
> 
> The issue is that `(use-package "A")' brings all of A's exported symbols
> into the current namespace, and so allows them to be used unqualified.
> This implies that any addition to a package A's exported symbol list has
> a chance of breaking code in any package that "use-package"'s it.  The
> breakage takes the form of an introduced ambiguity.

Hm, I must admit that I don't see an ambiguity here.  Maybe you are 
thinking of something like the following?

Assume packages A and B, A using B and containing a defun for f.  Next, 
B is modified to export B:f, which A inherits.  A is recompiled, and the 
compiler sees "(defun f () 42)".  In this case, the 'f' refers to B:f, 
but unambiguously.

If that's what you are thinking about, yes that's a known problem, which 
is, in most, if not all, CL implementations, solved with "package 
locks".  In CLisp:

https://clisp.sourceforge.io/impnotes/pack-lock.html

> 
> This is a problem for "programming in the large".  If package's can't
> add new things without risking breaking their users then the approach is
> arguably worse than Emacs' current approach, at least in this respect.
> Adding a new name for a new piece of functionality is one of the most
> common ways packages evolve over time.

But that's not unique to adding symbols.  The same is potentially true 
for all sorts of API changes.

>> BTW, in CL, if you don't want to use-package you can also import just
>> the list of symbols you want.  Without renaming, but I guess that
>> could be added, if somone wants that.  (I mention the idea of renaming
>> in cl-packages.org, BTW, on the branch, but I'm still not sure it's
>> worth it.)
> 
> Yes, Common Lisp provides ways to fix things *after* the breakage
> occurs, but the harm comes with the breakage, and is an inevitable
> consequence of the design.
> 
> Yes, the danger can be avoided by refraining from using `use-package' at
> all.  This is the root my suggestion to consider CL packages for Emacs
> but subtract `use-package'.

Here we have different opinions.  I see no need to forbid use-package. 
Imagine a CL without use-package, being in some package.  I don't want 
to write cl:car everywhere.  What for?

> 
> Yes, this kind of problem exists not only in Common Lisp but many other
> languages, to the extent that some might come to believe this sort of
> problem feels "natural" or is perhaps worth it for the convenience of
> using unqualified names from other packages.  Python's "from foo
> import*" is like CL's use-package and is now widely considered poor
> style in the Python community (for the reasons I've stated). 

Does Python have internal symbols?

> Guile has
> something like Common Lisp, with the same problem and the same kinds of
> mitigating workarounds.  Ada has "use" with semantics similar to CL's.
> Perl has "use" but, presumably through experience, recognized a problem
> and has a "use VERSION" mechanism to limit potential harm from API
> changes.  And on and on.
> 
> The argument is that all of these languages are all flawed in the same
> way, and that a package/module system designed today should aim for
> better.

More power to you :-).

Re: Help sought understanding shorthands wrt modules/packages

[Gerd Möllmann]

On 23.11.22 01:55, Matt Armstrong wrote:
> 
> My guess is that avoiding this kind of repetitive verbosity, at least
> for the common cases, is what many people want to get out of "Emacs
> Packages/Modules".  That is just a guess, though.  I can't really tell
> what benefits people actually want when they "it'd be great if Emacs had
> CL style packages."

Speaking for me, I want more modularity, more structure.  That's also 
why I prefer C++.

Re: Help sought understanding shorthands wrt modules/packages

[Eli Zaretskii]

> From: Matt Armstrong <matt@rfc20.org>
> Cc: rms@gnu.org, gerd.moellmann@gmail.com, emacs-devel@gnu.org
> Date: Tue, 22 Nov 2022 16:55:18 -0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Matt Armstrong <matt@rfc20.org>
> >> Cc: emacs-devel@gnu.org
> >> Date: Tue, 22 Nov 2022 10:01:33 -0800
> >>
> >> I think there is another drawback of Emacs' prefix based namespace
> >> convention: that of longer local (private, non-public, unexported, etc.)
> >> names.
> >
> > Why is that a drawback?  (It's a serious question.)
> 
> Thanks for asking.
> 
> I have to admit that my answer boils down to familiarity.  I'd like
> programming in Emacs Lisp to feel more like the other programming
> languages I'm more familiar with.
> [...]

Thanks.  However, my question was specifically about the "longer local
names" aspect, and it looks like you've answered a slightly different
question...

Re: Help sought understanding shorthands wrt modules/packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Of course in Ada it is, perhaps, more common today to rename imported
  > packages as a short prefix, instead of "using" them, but that's for
  > readability, which is a big emphasis in Ada culture.

Our package system could adopt that practice instead of USE.
That would avoid this problem entirely.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Help sought understanding shorthands wrt modules/packages

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 456 bytes --]

On Sun, Nov 27, 2022 at 12:32 AM Richard Stallman <rms@gnu.org> wrote:

> Our package system could adopt that practice instead of USE.
> That would avoid this problem entirely.

That already exists in Common Lisp packages, with nicknames.

AFAIU, the main difference between what you want and the Common Lisp
package implementation Gerd was proposing is that you want to ban "use" and
he wants to keep it. That doesn't seem like a technical problem to me.

[-- Attachment #2: Type: text/html, Size: 946 bytes --]

Re: Help sought understanding shorthands wrt modules/packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Our package system could adopt that practice instead of USE.
  > > That would avoid this problem entirely.

  > That already exists in Common Lisp packages, with nicknames.

That may be a good thing.  Can you show me a description of the
nicknames feature?

  > AFAIU, the main difference between what you want and the Common Lisp
  > package implementation Gerd was proposing is that you want to ban "use" and
  > he wants to keep it. That doesn't seem like a technical problem to me.

The problem caused by using USE is a problem of human behavior, shaped
by the technology.  Sometimes these problems, about what happens when people
use the technology they are given, are very important.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Help sought understanding shorthands wrt modules/packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

The message mentions :USE, and also use-package.
I thought these were totally unrelated, and that
usepackage realated to the sort of packages in ELPA.

Could someone plesae clear this up for me?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Help sought understanding shorthands wrt modules/packages

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: matt@rfc20.org, emacs-devel@gnu.org
> Date: Wed, 14 Dec 2022 17:21:54 -0500
> 
> The message mentions :USE, and also use-package.
> I thought these were totally unrelated, and that
> usepackage realated to the sort of packages in ELPA.
> 
> Could someone plesae clear this up for me?

They are completely unrelated.  use-package is basically a fancy
'require' facility, which (a) arranges for the library to be loaded
only when it is actually used, and (b) allows the users to specify in
their init file some code that needs to be run immediately before and
after the library is loaded, such as setting up some variables from
the library or which affect the library, turning on and off some
optional features of the library, loading dependency libraries, etc.

Re: Help sought understanding shorthands wrt modules/packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Could someone plesae clear this up for me?

  > They are completely unrelated.

Thanks.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)