Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Phil Sainty]

This has probably been covered in earlier discussions (apologies for
not being across those), but...

Won't this break a ton of basic tooling for locating things, if the
symbol in the file is not the actual symbol?

Simplicity can be a *very* good thing, and knowing that what you see
is in fact what you get is a benefit which shouldn't be undervalued,
IMO.

Name-spaces in other languages I've seen seem to come at a very
significant cost, complicating the tooling and forcing a lot of
things which could previously be take for granted to need changing
(if even possible) so that they are reliant on new features which
"understand" what's going on behind the scenes.

Is whatever we're gaining actually worth the resulting obfuscation?

Long names being "tedious" (quoting the new info manual) to read
and write seems like an insufficient reason, IMHO.


-Phil (who is no doubt WAY too late to the party)

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Lars Ingebrigtsen]

Phil Sainty <psainty@orcon.net.nz> writes:

> Won't this break a ton of basic tooling for locating things, if the
> symbol in the file is not the actual symbol?

Yes, it'll break a lot of tooling.  For the record, I was against adding
this functionality, but I was outvoted.  (Which is fine.)

My argument was: If we want to add something along these lines, it
should be more like Common Lisp's package system.  (Not that that's a
perfect system -- I wouldn't want to add it wholesale, but it's pretty
good, and we could take the good parts.)  And, yes, doing that would
also require a lot of tooling changes, which I'd want to see happen
before doing it.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Phil Sainty <psainty@orcon.net.nz> writes:
>
>> Won't this break a ton of basic tooling for locating things, if the
>> symbol in the file is not the actual symbol?
>
> Yes, it'll break a lot of tooling. 

What "lot of tooling"?  I don't remmeber seeing a list.  If you have
one, I'd like to see it, so that I can work on it, eventually.

> My argument was: If we want to add something along these lines, it
> should be more like Common Lisp's package system.  (Not that that's a
> perfect system -- I wouldn't want to add it wholesale, but it's pretty
> good, and we could take the good parts.)

Yes, I concur.  I would have liked CL packages too.  I've been waiting
for those to "magically" appear in Elisp ever since I started using
Emacs.  What I don't understand is how Shorthands is making that any
more difficult.  If anything, I would cautiously suspect that it might
make it _easier_.

> And, yes, doing that would also require a lot of tooling changes,
> which I'd want to see happen before doing it.

Then get to work! :-)

João

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Yes, it'll break a lot of tooling.

Can you report any specific problems?

  > My argument was: If we want to add something along these lines, it
  > should be more like Common Lisp's package system.

That is far more complex and has deep flaws.  The symbol prefix renaming
feature is much cleaner.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Gregory Heytings]

>> Yes, it'll break a lot of tooling.
>
> Can you report any specific problems?
>

A simple example: suppose you want to check which ELPA package activates 
tab-bar-mode.  That's easy to do with "grep -R tab-bar-mode" in a clone of 
the ELPA repository.  With symbol prefix renaming, a package author might 
decide to add ("tb-" . "tab-bar-") in the shorthands of the package, and 
"grep -R tab-bar-mode" will not show anything.  Likewise for tag systems, 
the symbols that are recorded will possibly be different in each package, 
and a search for tab-bar-mode will not return occurrences of tb-mode.

>> My argument was: If we want to add something along these lines, it 
>> should be more like Common Lisp's package system.
>
> That is far more complex and has deep flaws.  The symbol prefix renaming 
> feature is much cleaner.
>

Indeed, but IMO only if it comes with an unambiguous textual indication 
that the symbol will be transformed by the Lisp reader into another 
symbol, for example "::" between the shorthand and the rest of the symbol. 
Not only will those who read the code immediately see that tb::mode is a 
shorthand and not a regular Elisp symbol, but it becomes much easier to 
instruct tools such as grep or tag systems to preprocess el files by first 
expanding tb::mode into tab-bar-mode.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[André A. Gomes]

Gregory Heytings <gregory@heytings.org> writes:

> A simple example: suppose you want to check which ELPA package
> activates tab-bar-mode.  That's easy to do with "grep -R tab-bar-mode"
> in a clone of the ELPA repository.  With symbol prefix renaming, a
> package author might decide to add ("tb-" . "tab-bar-") in the
> shorthands of the package, and "grep -R tab-bar-mode" will not show
> anything.  Likewise for tag systems, the symbols that are recorded
> will possibly be different in each package, and a search for
> tab-bar-mode will not return occurrences of tb-mode.

I don't think this is a problem.  Grep comes the world of Unix and its
mantras.  But Lisp REPLs come from another world.

Using grep and tag systems to reason about a Lisp program is like eating
soup with a fork.  You can do it, but it's the wrong tool.


--
André A. Gomes
"Free Thought, Free World"

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Alan Mackenzie]

Hello, André.

On Thu, Sep 30, 2021 at 13:31:34 +0300, André A. Gomes wrote:
> Gregory Heytings <gregory@heytings.org> writes:

> > A simple example: suppose you want to check which ELPA package
> > activates tab-bar-mode.  That's easy to do with "grep -R tab-bar-mode"
> > in a clone of the ELPA repository.  With symbol prefix renaming, a
> > package author might decide to add ("tb-" . "tab-bar-") in the
> > shorthands of the package, and "grep -R tab-bar-mode" will not show
> > anything.  Likewise for tag systems, the symbols that are recorded
> > will possibly be different in each package, and a search for
> > tab-bar-mode will not return occurrences of tb-mode.

> I don't think this is a problem.  Grep comes the world of Unix and its
> mantras.  But Lisp REPLs come from another world.

> Using grep and tag systems to reason about a Lisp program is like eating
> soup with a fork.  You can do it, but it's the wrong tool.

That is a very negative and unhelpful thing to say.  Do you have the
requisite background to say it?  What precisely would you use in place
of grep, which is a powerful, easily learnt, fast, universal tool?

My experience is that grep is an essential tool for Emacs maintenance.

> --
> André A. Gomes
> "Free Thought, Free World"

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

On Thu, Sep 30, 2021 at 11:54 AM Alan Mackenzie <acm@muc.de> wrote:

> > Using grep and tag systems to reason about a Lisp program is like eating
> > soup with a fork.  You can do it, but it's the wrong tool.
>
> That is a very negative and unhelpful thing to say.  Do you have the
> requisite background to say it?

Is there a "requisite background" to express opinions on this mailing list?
Must have missed that memo.

João

PS: Agree with André, btw. Don't know if I have the "requisite
background", either.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[André A. Gomes]

João Távora <joaotavora@gmail.com> writes:

> On Thu, Sep 30, 2021 at 11:54 AM Alan Mackenzie <acm@muc.de> wrote:
>
>> > Using grep and tag systems to reason about a Lisp program is like eating
>> > soup with a fork.  You can do it, but it's the wrong tool.
>>
>> That is a very negative and unhelpful thing to say.  Do you have the
>> requisite background to say it?
>
> Is there a "requisite background" to express opinions on this mailing list?
> Must have missed that memo.

I believe Alan sensed some gratuitous arrogance in my words, and that's
certainly unplesant.  The soup joke went perhaps too far.  

I am genuinely interested in discussing the technical side of the issue
at hand, not in "shaming" anyone.  I'm certainly not an "authority", so
expect a humble attitude from my side.


-- 
André A. Gomes
"Free Thought, Free World"

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Alan Mackenzie]

Hello, João.

On Thu, Sep 30, 2021 at 12:18:41 +0100, João Távora wrote:
> On Thu, Sep 30, 2021 at 11:54 AM Alan Mackenzie <acm@muc.de> wrote:

> > > Using grep and tag systems to reason about a Lisp program is like eating
> > > soup with a fork.  You can do it, but it's the wrong tool.

> > That is a very negative and unhelpful thing to say.  Do you have the
> > requisite background to say it?

> Is there a "requisite background" to express opinions on this mailing list?

Of course.  That is a genuine interest in the development of Emacs.  It
would appear that André has such a background.  However, his earlier
post, the one to which I replied, looked like it could have come from a
troll.  We have had at least one troll on this list in the last few
years, and he caused a great deal of wasted time for a lot of people
before I told him to go away.  I was prepared to do the same again, but
this time sooner.  Happily it turned out this was not needed.

> Must have missed that memo.

Amongst quite a few other things.

> João

> PS: Agree with André, btw. Don't know if I have the "requisite
> background", either.

You do, as you know very well.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

On Thu, Sep 30, 2021 at 5:58 PM Alan Mackenzie <acm@muc.de> wrote:

> before I told him to go away.  I was prepared to do the same again, but
> this time sooner.  Happily it turned out this was not needed.
> > Must have missed that memo.
> Amongst quite a few other things.

Indeed, like your promotion to neighbourhood vigilante.

João

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Stefan Monnier]

I think it's time you guys take a break from this thread.


        Stefan


João Távora [2021-09-30 21:25:19] wrote:
> On Thu, Sep 30, 2021 at 5:58 PM Alan Mackenzie <acm@muc.de> wrote:
>> before I told him to go away.  I was prepared to do the same again, but
>> this time sooner.  Happily it turned out this was not needed.
>> > Must have missed that memo.
>> Amongst quite a few other things.
> Indeed, like your promotion to neighbourhood vigilante.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[André A. Gomes]

Alan Mackenzie <acm@muc.de> writes:

> Hello, André.
>
> On Thu, Sep 30, 2021 at 13:31:34 +0300, André A. Gomes wrote:
>> Gregory Heytings <gregory@heytings.org> writes:
>
>> > A simple example: suppose you want to check which ELPA package
>> > activates tab-bar-mode.  That's easy to do with "grep -R tab-bar-mode"
>> > in a clone of the ELPA repository.  With symbol prefix renaming, a
>> > package author might decide to add ("tb-" . "tab-bar-") in the
>> > shorthands of the package, and "grep -R tab-bar-mode" will not show
>> > anything.  Likewise for tag systems, the symbols that are recorded
>> > will possibly be different in each package, and a search for
>> > tab-bar-mode will not return occurrences of tb-mode.
>
>> I don't think this is a problem.  Grep comes the world of Unix and its
>> mantras.  But Lisp REPLs come from another world.
>
>> Using grep and tag systems to reason about a Lisp program is like eating
>> soup with a fork.  You can do it, but it's the wrong tool.
>
> That is a very negative and unhelpful thing to say.  Do you have the
> requisite background to say it?  What precisely would you use in place
> of grep, which is a powerful, easily learnt, fast, universal tool?
>
> My experience is that grep is an essential tool for Emacs maintenance.

I'm sorry that my words had a negative impact towards you or others.  I
acknowledge and value the efforts of any member of this community.

My words were misinterpreted perhaps.  Firstly, I also eat soup with a
fork, i.e. I also grep Lisp source code.  But I must be honest with
myself, and admit it's not the right tool.  Indeed, grep is a universal
tool that can be used to inspect any textual data.  But powerful Lisp
environments (think Slime) are "aware of themselves" and do better than
grep does.

My point is thus simple and far from being original.  A culture shock
between "Unix" and Lisp exists, and it has been discussed to death (for
instance in the "UNIX Haters Handbook").

I think shorthands is a good idea.  While the grep claim is true, I find
it orthogonal in the sense that we're judging the adoption of a new idea
by the wrong standards.

Again, I apologize.


-- 
André A. Gomes
"Free Thought, Free World"

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Alan Mackenzie]

Hello, André.

On Thu, Sep 30, 2021 at 14:30:11 +0300, André A. Gomes wrote:
> Alan Mackenzie <acm@muc.de> writes:


> > On Thu, Sep 30, 2021 at 13:31:34 +0300, André A. Gomes wrote:
> >> Gregory Heytings <gregory@heytings.org> writes:

> >> > A simple example: suppose you want to check which ELPA package
> >> > activates tab-bar-mode.  That's easy to do with "grep -R tab-bar-mode"
> >> > in a clone of the ELPA repository.  With symbol prefix renaming, a
> >> > package author might decide to add ("tb-" . "tab-bar-") in the
> >> > shorthands of the package, and "grep -R tab-bar-mode" will not show
> >> > anything.  Likewise for tag systems, the symbols that are recorded
> >> > will possibly be different in each package, and a search for
> >> > tab-bar-mode will not return occurrences of tb-mode.

> >> I don't think this is a problem.  Grep comes the world of Unix and its
> >> mantras.  But Lisp REPLs come from another world.

> >> Using grep and tag systems to reason about a Lisp program is like eating
> >> soup with a fork.  You can do it, but it's the wrong tool.

> > That is a very negative and unhelpful thing to say.  Do you have the
> > requisite background to say it?  What precisely would you use in place
> > of grep, which is a powerful, easily learnt, fast, universal tool?

> > My experience is that grep is an essential tool for Emacs maintenance.

> I'm sorry that my words had a negative impact towards you or others.  I
> acknowledge and value the efforts of any member of this community.

Thanks for saying that!

> My words were misinterpreted perhaps.  Firstly, I also eat soup with a
> fork, i.e. I also grep Lisp source code.  But I must be honest with
> myself, and admit it's not the right tool.  Indeed, grep is a universal
> tool that can be used to inspect any textual data.  But powerful Lisp
> environments (think Slime) are "aware of themselves" and do better than
> grep does.

For some jobs, I'm sure they will.  Possibly even for most jobs.  But
they have their disadvantages.  They're difficult to combine with other
tools.  They don't run on the command line.  They tend to need your code
to be instrumented in some fashion first, which is slow.  They're
difficult and time-consuming to learn, particularly for a
special-purpose tool.

The environment I have to use in my day job has such tools built in to
it, and they're well implemented.  But I dislike their constraints.  I
dislike the precise format (unchangeable) chosen for the output.  Now
and then, I'd kill to be able to use 'find' and 'grep' on the source
files.  I don't like integrated development environments.

Also, grep is fast.  On my 4 year old machine today it took 0.27 seconds
to grep through our entire set of Lisp source files, from a cold start.
With those files in cache, it took a mere 0.05 seconds.  This is
effectively instantaneous.

> My point is thus simple and far from being original.  A culture shock
> between "Unix" and Lisp exists, and it has been discussed to death (for
> instance in the "UNIX Haters Handbook").

I know that book.  :-)  Most of the people on this list do both "Unix"
and Lisp, and some are experts at both.  There's no need to have to
chose one or the other, we can use both.

> I think shorthands is a good idea.  While the grep claim is true, I find
> it orthogonal in the sense that we're judging the adoption of a new idea
> by the wrong standards.

If it is going to cause problems in Emacs development, we should surely
discuss those problems in advance and evaluate them, rather than just
saying "should be OK, why worry?".  We're likely talking about an
irreversible change taking place here.  It'd better be right.

> Again, I apologize.

No problems!

> -- 
> André A. Gomes
> "Free Thought, Free World"

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Gregory Heytings]

>> A simple example: suppose you want to check which ELPA package 
>> activates tab-bar-mode.  That's easy to do with "grep -R tab-bar-mode" 
>> in a clone of the ELPA repository.  With symbol prefix renaming, a 
>> package author might decide to add ("tb-" . "tab-bar-") in the 
>> shorthands of the package, and "grep -R tab-bar-mode" will not show 
>> anything.  Likewise for tag systems, the symbols that are recorded will 
>> possibly be different in each package, and a search for tab-bar-mode 
>> will not return occurrences of tb-mode.
>
> I don't think this is a problem.  Grep comes the world of Unix and its 
> mantras.  But Lisp REPLs come from another world.
>
> Using grep and tag systems to reason about a Lisp program is like eating 
> soup with a fork.  You can do it
>

You could do it for Emacs 1 up to and including Emacs 28.

>
> but it's the wrong tool.
>

And what is/what are the "right" tool(s) for the above use case?

I agree with you on one point: in some cases grep/tags are not the best 
tools, because they will not display all actual matches.  This is not only 
true for Lisp, it's also true for plain C, in which tokens can be the 
result of preprocessing.  But they are (much used) tools on programmer's 
workbenches, and if something can be done to avoid breaking them, or at 
least to make it easier to adapt them to the change, that would be better.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1250 bytes --]

On Thu, Sep 30, 2021, 12:46 Gregory Heytings <gregory@heytings.org> wrote:

> > but it's the wrong tool.
>
> And what is/what are the "right" tool(s) for the above use case?
>

Tools that understand the symbolic nature of the Lisp family of languages.
For the examplw you have since, Tools that rely on way or the other really
on the 'read' Lisp primitive.

xref-find-definition does. xref-find-references doesn't, yet, as far as i
know. C-h f is fine, and so is completion-at-point.

Grep, as you very well note, is already flawed, not only for Lisp, but for
many languages. By "flawed" I mean: it is not suitable for categorically
answering questions e.g. about how functions relate to each other (callers
and callees). It fails even on C, for example by the mere existence of
comment blocks. Should comment blocks be outlawed in C?

In contrast, in some common lisp IDEs you have such tools and expose this
database. Xref in Emacs was originally derived from work of a Common Lisp
programmer, which created the amazing SLIME, which you may have heard of.
SLIME (and my fork of it Sly) are indeed able to use these databases.

André's comment is very accurate. In SLIME, one eats Lisp with a spoon, not
a fork.

João

[-- Attachment #2: Type: text/html, Size: 2070 bytes --]

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Tomas Hlavaty]

On Thu 30 Sep 2021 at 13:41, João Távora <joaotavora@gmail.com> wrote:
> On Thu, Sep 30, 2021, 12:46 Gregory Heytings <gregory@heytings.org> wrote:
>> > but it's the wrong tool.
>> And what is/what are the "right" tool(s) for the above use case?
> In contrast, in some common lisp IDEs you have such tools and expose this
> database. Xref in Emacs was originally derived from work of a Common Lisp
> programmer, which created the amazing SLIME, which you may have heard of.
> SLIME (and my fork of it Sly) are indeed able to use these databases.
>
> André's comment is very accurate. In SLIME, one eats Lisp with a spoon, not
> a fork.
  
Except it does not remove the need for grep.
It misses a lot of things, you cannot rely on it completely.
It needs the lisp code loaded and compiled sucessfully.
Slime + Common Lisp are great but not perfect.
unique and non-stupid names like "s" => perfect grep and web search

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

On Thu, Sep 30, 2021 at 2:00 PM Tomas Hlavaty <tom@logand.com> wrote:
>
> On Thu 30 Sep 2021 at 13:41, João Távora <joaotavora@gmail.com> wrote:
> > On Thu, Sep 30, 2021, 12:46 Gregory Heytings <gregory@heytings.org> wrote:
> >> > but it's the wrong tool.
> >> And what is/what are the "right" tool(s) for the above use case?
> > In contrast, in some common lisp IDEs you have such tools and expose this
> > database. Xref in Emacs was originally derived from work of a Common Lisp
> > programmer, which created the amazing SLIME, which you may have heard of.
> > SLIME (and my fork of it Sly) are indeed able to use these databases.
> >
> > André's comment is very accurate. In SLIME, one eats Lisp with a spoon, not
> > a fork.
>
> Except it does not remove the need for grep.
> It misses a lot of things, you cannot rely on it completely.
> It needs the lisp code loaded and compiled sucessfully.

No, you're able to build such a tool that merely needs to Lisp code to be
CL:READ.  Grep also "reads" the code, but it reads it line by line. OK for
for C (but not always), not so much for Lisp (In Elisp, CL:READ is 'read', btw).

Also, Lisp is developed iteratively.  You're supposed to have it loaded in your
image while you  develop it.  It's not like C where you do something and have to
recompile the whole thing.

Elisp development is no different here.

CL:READ/'read' areboth  aware of packages/shorthands. It can tell you where
the symbol reference lives. That doesn't tell you if its a function, a variable,
a gremlin, etc... of course [*]

But neither does grep.  Grep is light-years away from that.

And grep fails very horribly in Common Lisp as you well know, if
you include the package qualifier.  Since a package qualifier has
loads of nicknames (much like shorthands to be honest. For those
not in the loop, the package qualifier is the equivalent of the prefix
in Emacs lisp.  So, if in Emacs Lisp, you make it easy to grep for
the previ and non-prefix parts of a given name, then bam!, you have
a achieved Common Lisp grep parity! (which isn't much of course,
as André's point still stands intact: it is a fork for soup).

> Slime + Common Lisp are great but not perfect.

Sure!  But they sure beat anything else I've worked with, including Emacs,
which comes second.

I use grep too, by the way.  But I prefer apropos. Again, it knows a
lot more and
has a French name, so it's frankly impossible not to.

I see some people arguing like shorthands have "broken" grep which was some k
ind of omniscient oracle about code analysis. That's so far away from
the truth.  What
would be the point of all these LSP enhancements we're seeing for all
languages if
it were? Of all Static Code Analysis tools being developed?  In Elisp
many constructs
already escape it or break it.  defmethod or defstruct just to name two.

My point is that Lisp has many much better tools -- available today or
waiting to be
created -- because of its List-based structure.  Grep really is a
middle-ages tool
for Lisp.  Lisp looks like it's from the past, but it's actually from
the future ;-)

João

[*] This, by the way, is _another_ reason why reader-based approaches to
namespacing systems are, IMO, better than macro-based ones.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Tomas Hlavaty]

On Thu 30 Sep 2021 at 14:26, João Távora <joaotavora@gmail.com> wrote:
> On Thu, Sep 30, 2021 at 2:00 PM Tomas Hlavaty <tom@logand.com> wrote:
>> On Thu 30 Sep 2021 at 13:41, João Távora <joaotavora@gmail.com> wrote:
>> > On Thu, Sep 30, 2021, 12:46 Gregory Heytings <gregory@heytings.org> wrote:
>> >> > but it's the wrong tool.
>> >> And what is/what are the "right" tool(s) for the above use case?
>> > In contrast, in some common lisp IDEs you have such tools and expose this
>> > database. Xref in Emacs was originally derived from work of a Common Lisp
>> > programmer, which created the amazing SLIME, which you may have heard of.
>> > SLIME (and my fork of it Sly) are indeed able to use these databases.
>> >
>> > André's comment is very accurate. In SLIME, one eats Lisp with a spoon, not
>> > a fork.
>>
>> Except it does not remove the need for grep.
>> It misses a lot of things, you cannot rely on it completely.
>> It needs the lisp code loaded and compiled sucessfully.
>
> No, you're able to build such a tool that merely needs to Lisp code to be
> CL:READ.

That does not work.  Common Lisp reader is programable.  If you do not
compile and load everything needed sucessfully, the reader will fail for
anything non-trivial.

> You're supposed to have it loaded in your image while you develop it.

You can restrict yourself like that but why do you think that it is a
reasonable restriction for everybody?

I likely search for things not loaded in my image more often than for
things already loaded in my image.

Which image btw?  The one with or without threads, the one with or
without unicode, the normal one or the one with more debuging support
turned on, the one with sbcl or ccl or ecl or clisp...?

> It's not like C where you do something and have to recompile the whole
> thing.

It really depends on what you change.  Sometimes it is necessary to
recompile the whole thing even in Common Lisp.  For example: If you
change a macro, you need to recompile all places where it is used.  If
you change an inlined function, you need to recompile all places where
it is used.

> And grep fails very horribly in Common Lisp as you well know, if
> you include the package qualifier.

That's why good programmers make effort to choose good names independent
of language.

Lots of Common Lisp code is awful because people play clever tricks with
names and packages, symbol import, export, shadowing.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

On Thu, Sep 30, 2021 at 3:27 PM Tomas Hlavaty <tom@logand.com> wrote:

> That does not work.  Common Lisp reader is programable.  If you do not
> compile and load everything needed sucessfully, the reader will fail for
> anything non-trivial.

Those non-trivial things are quite rare, and good reader etiquette makes the
code that is CL:READ with a non-full reader at least make a good deal of
sense.  Unless you're programming basic stuff like `(` and stuff, in which
case you've already lost.

But you're right, almost. The standard reader needs to know packages just to
be able to intern the symbols it reads.  UNLESS you use a non-interning reader
;-) Which is precisely what you should use for a lisp-specific grep-like.

And anyhoo, the point is moot for Elisp.  The reader is not programmable (yet!)
and shorthand definitions are file-local.  So building that grep-like is not
impeded by that question at all.

> > You're supposed to have it loaded in your image while you develop it.
>
> You can restrict yourself like that but why do you think that it is a
> reasonable restriction for everybody?

I don't know if it's a restriction.  It's what I and many consider the most
useful way to work with Lisp.  To have symbolic information at our
fingertips.  You like to have line-based literal-text information, sure go
ahead!  But you'll be missing a lot of free structure.
It's free symbolic real-estate!

> > It's not like C where you do something and have to recompile the whole
> > thing.
>
> It really depends on what you change.  Sometimes it is necessary to
> recompile the whole thing even in Common Lisp.

We're talkinga bout iterative development, which is where development
tools of the sort of grep come about.  I'm not stating that in CL I don't
recompile whole stuff from scratch Oh boy, I _wish_ I could state that.
Last CL job I worked it took at least 20 minutes.  So I and everyone else
developed a bit incrementally, then compiled the whole thing to test.

> > And grep fails very horribly in Common Lisp as you well know, if
> > you include the package qualifier.
> That's why good programmers make effort to choose good names independent
> of language.

So when you program CL you repeat the package qualifier in the symbol name?
I've seen that, yuck. And you never use `:USE` (not even` :USE :CL`) and you
always refer to the same symbol by the very same name every time? And
all your colleagues and the code you use in Common Lisp does that?

> Lots of Common Lisp code is awful because people play clever tricks with
> names and packages, symbol import, export, shadowing.

"With great power comes great responsibility", some beardy sysadmin
probably wrote.

João

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Tomas Hlavaty]

On Thu 30 Sep 2021 at 15:57, João Távora <joaotavora@gmail.com> wrote:
> On Thu, Sep 30, 2021 at 3:27 PM Tomas Hlavaty <tom@logand.com> wrote:
>
>> That does not work.  Common Lisp reader is programable.  If you do not
>> compile and load everything needed sucessfully, the reader will fail for
>> anything non-trivial.
>
> Those non-trivial things are quite rare,

Not really.

> and good reader etiquette makes the code that is CL:READ with a
> non-full reader at least make a good deal of sense.

What is "good reader etiquette"?

What if some library does not have "good reader etiquette"?
Do you give up search because of that?

>> > You're supposed to have it loaded in your image while you develop it.
>>
>> You can restrict yourself like that but why do you think that it is a
>> reasonable restriction for everybody?
>
> I don't know if it's a restriction.  It's what I and many consider the most
> useful way to work with Lisp.  To have symbolic information at our
> fingertips.  You like to have line-based literal-text information, sure go
> ahead!  But you'll be missing a lot of free structure.
> It's free symbolic real-estate!

It is not a choice between one or the other.
I need both.
Please do not break grep and web search.

>> > And grep fails very horribly in Common Lisp as you well know, if
>> > you include the package qualifier.
>> That's why good programmers make effort to choose good names independent
>> of language.
>
> So when you program CL you repeat the package qualifier in the symbol name?
> I've seen that, yuck. And you never use `:USE` (not even` :USE :CL`) and you
> always refer to the same symbol by the very same name every time? And
> all your colleagues and the code you use in Common Lisp does that?

Use common sense and do not work against the available tools.
There are some heuristics for choosing good names.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

Tomas Hlavaty <tom@logand.com> writes:

> On Thu 30 Sep 2021 at 15:57, João Távora <joaotavora@gmail.com> wrote:
>> On Thu, Sep 30, 2021 at 3:27 PM Tomas Hlavaty <tom@logand.com> wrote:
>>
>>> That does not work.  Common Lisp reader is programable.  If you do not
>>> compile and load everything needed sucessfully, the reader will fail for
>>> anything non-trivial.
>>
>> Those non-trivial things are quite rare,
>
> Not really.
>
>> and good reader etiquette makes the code that is CL:READ with a
>> non-full reader at least make a good deal of sense.
>
> What is "good reader etiquette"?

See this section of CLTL2

  https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node191.html

See how some combinations are explicitly reserved for the user.  Stick
to those.

> What if some library does not have "good reader etiquette"?
> Do you give up search because of that?

No, but it's like libraries using

   (intern (format nil "~a~a" "foo" "bar"))

They're not making life easier for their users in that respect.

> It is not a choice between one or the other.  I need both.  Please do
> not break grep and web search.

I explained how they are already "broken".  In CL (the topic of this
particular subthread), by packages and in every language that has any
type of indirection.  I'm saying they were never really "good" to begin
with, not if you want to use the full available power in those
languages.

> There are some heuristics for choosing good names.

Yes there are, and you shouldn't give them up, by any means.

João

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Tomas Hlavaty]

On Thu 30 Sep 2021 at 17:02, João Távora <joaotavora@gmail.com> wrote:
> Tomas Hlavaty <tom@logand.com> writes:
>
>> On Thu 30 Sep 2021 at 15:57, João Távora <joaotavora@gmail.com> wrote:
>>> On Thu, Sep 30, 2021 at 3:27 PM Tomas Hlavaty <tom@logand.com> wrote:
>>>
>>>> That does not work.  Common Lisp reader is programable.  If you do not
>>>> compile and load everything needed sucessfully, the reader will fail for
>>>> anything non-trivial.
>>>
>>> Those non-trivial things are quite rare,
>>
>> Not really.
>>
>>> and good reader etiquette makes the code that is CL:READ with a
>>> non-full reader at least make a good deal of sense.
>>
>> What is "good reader etiquette"?
>
> See this section of CLTL2
>
>   https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node191.html
>
> See how some combinations are explicitly reserved for the user.  Stick
> to those.
>
>> What if some library does not have "good reader etiquette"?
>> Do you give up search because of that?
>
> No, but it's like libraries using
>
>    (intern (format nil "~a~a" "foo" "bar"))
>
> They're not making life easier for their users in that respect.

It is different.

The intern case simply does not appear in the search results.

But if your CL:READ based search breaks, you can no longer search the
codebase at all until you fix it.

>> It is not a choice between one or the other.  I need both.  Please do
>> not break grep and web search.
>
> I explained how they are already "broken".  In CL (the topic of this
> particular subthread), by packages and in every language that has any
> type of indirection.  I'm saying they were never really "good" to begin
> with, not if you want to use the full available power in those
> languages.

Neither of the two options are perfect.
As I said, I need both to work as well as possible.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

Tomas Hlavaty <tom@logand.com> writes:

>>> It is not a choice between one or the other.  I need both.  Please do
>>> not break grep and web search.
>>
>> I explained how they are already "broken".  In CL (the topic of this
>> particular subthread), by packages and in every language that has any
>> type of indirection.  I'm saying they were never really "good" to begin
>> with, not if you want to use the full available power in those
>> languages.
>
> Neither of the two options are perfect.
> As I said, I need both to work as well as possible.

Right.  But it's a trade-off between language expressive power (the
power to express more in less characters) and repetition and verbosity.
That does give the power of locality and enables things as grep (though
they're always flawed as we've seen), but it comes at the expense of
hard and brittle couplings.

From your arguments, it seems you don't like namespace features in
languages.  That strikes me, personally, as odd, from a Common Lisp
programmer as you present yourself, but I respect it.

"Proper" namespaces have been proposed numerous times for Emacs, and
people can and do get around the practical problems with numerous
strategies.  The most prominent of which happened some years ago with
the creation of very popular libraries with very short suffixes: s.el,
dash.el and f.el all above the 99.9 downloads percentile in MELPA.  But
that is by no means the only one: people will use macros and "break
grep" happily with something simple and innocent as flet or labels or
putting anonymous functions in local variables, because in Emacs,
interning a symbol is a big responsibility.  All these things I consider
worse, even just from a grep-lover perspective, than criterious use of
namespacing.  As with any language feature, use it wisely.

João

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[André A. Gomes]

Tomas Hlavaty <tom@logand.com> writes:

> That's why good programmers make effort to choose good names independent
> of language.
>
> Lots of Common Lisp code is awful because people play clever tricks with
> names and packages, symbol import, export, shadowing.

Such claims aren't at all rare, but I wouldn't expect to read them on
the Emacs mailing list.

Unix, why did you spread like a virus?  Why "worse is better"?  This is
backwards...


-- 
André A. Gomes
"Free Thought, Free World"

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Gregory Heytings]

>>> but it's the wrong tool.
>>
>> And what is/what are the "right" tool(s) for the above use case?
> 
> Tools that understand the symbolic nature of the Lisp family of 
> languages. For the examplw you have since, Tools that rely on way or the 
> other really on the 'read' Lisp primitive.
>
> xref-find-definition does. xref-find-references doesn't, yet, as far as 
> i know. C-h f is fine, and so is completion-at-point.
>

Which means that you basically have to load all ELPA packages into your 
Emacs session to answer a question that was so far trivial to answer 
(which packages use such-and-such function or refer to this-or-that 
variable).

>
> Grep, as you very well note, is already flawed, not only for Lisp, but 
> for many languages. By "flawed" I mean: it is not suitable for 
> categorically answering questions e.g. about how functions relate to 
> each other (callers and callees).
>

It's not flawed, it's not perfect, like pretty much any other tool (and it 
has been around long enough that its users are aware of its limits). 
Even a tool that would be aware of the Lisp read primitive cannot 
categorically answer that question, for example when function names are 
created dynamically, or with (apply variable args).

>
> It fails even on C, for example by the mere existence of comment blocks. 
> Should comment blocks be outlawed in C?
>

Of course not.  And comment blocks are not a problem AFAICS, visiting the 
file will clearly show that this is not an actual call.  In C it's 
preprocessing that is a problem.  And function pointers.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

> Of course not.  And comment blocks are not a problem AFAICS, visiting the
> file will clearly show that this is not an actual call.  In C it's
> preprocessing that is a problem.  And function pointers.

Uhhh, indirection, tremble!!!  Nice one, function pointers! Fortunately
no-one ever, ever, stores functions in variables in Lisp.  Phew!

João

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Gregory Heytings]

>> Of course not.  And comment blocks are not a problem AFAICS, visiting 
>> the file will clearly show that this is not an actual call.  In C it's 
>> preprocessing that is a problem.  And function pointers.
>
> Uhhh, indirection, tremble!!!  Nice one, function pointers! Fortunately 
> no-one ever, ever, stores functions in variables in Lisp.  Phew!
>

Inopportune irony.  I mentioned calling function in variables two lines 
above: "Even a tool that would be aware of the Lisp read primitive cannot 
categorically answer that question, for example when function names are 
created dynamically, or with (apply variable args)."

RE: [External] : Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Drew Adams]

> Tools that understand the symbolic nature of the Lisp
> family of languages. For the example you have since,
> Tools that rely on way or the other really on the
> 'read' Lisp primitive.

https://github.com/emacsmirror/el-search

(Not sure that's the latest or best or only location.
Ccing Michael H.)
___

As for grep etc. _versus_ Lisp-aware searching:

Both Lisp-aware and ad-hoc searches (searching for
_anything_ in any kind of code or text, including
Lisp code) are useful.  It's not either/or.

As for the proposal: I do hope Phil S's points get
addressed thoughtfully.

> Grep, as you very well note, is already flawed,
> not only for Lisp, but for many languages.
> By "flawed" I mean: it is not suitable for
> categorically answering questions e.g. about how
> functions relate to each other (callers and
> callees).

Of course.  Regexps are limited.  `grep' is
limited.  It's only "flawed" if you expect it to
be something it's not.  Such tools, and regexp
search in particular, are handy and useful, even
though they have limitations.  Even if you use
regexp search to do nothing more than substring
searching, it's useful.

Some use regexps for things they're awful for
(useless or worse than useless).

Some other people go beyond usefully pointing
out the limitations of regexps to crusade
against using regexps.  Both are misguided, IMO.

A hammer is a useful tool; it's just not the
only tool, and for many jobs it's not the best
tool.  Howling against hammers is, well, as
dumb as thinking they're useful for everything.

/s/hammer/duct tape, if you like. ;-)

Re: [External] : Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

On Thu, Sep 30, 2021 at 5:23 PM Drew Adams <drew.adams@oracle.com> wrote:
>
> > Tools that understand the symbolic nature of the Lisp
> > family of languages. For the example you have since,
> > Tools that rely on way or the other really on the
> > 'read' Lisp primitive.
>
> https://github.com/emacsmirror/el-search

Nice prior art. thanks.

> A hammer is a useful tool; it's just not the
> only tool, and for many jobs it's not the best
> tool.  Howling against hammers is, well, as
> dumb as thinking they're useful for everything.

Sure.  I use grep.  I use the hammer.  When I
see something that looks like a nail mostly.

João

Re: [External] : Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Michael Heerdegen]

João Távora <joaotavora@gmail.com> writes:

> Sure.  I use grep.  I use the hammer.  When I see something that looks
> like a nail mostly.

I'm the opposite.  I use Emacs - however it looks like.

Michael.

Re: [External] : Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Michael Heerdegen]

Drew Adams <drew.adams@oracle.com> writes:

> > Tools that understand the symbolic nature of the Lisp
> > family of languages. For the example you have since,
> > Tools that rely on way or the other really on the
> > 'read' Lisp primitive.
>
> https://github.com/emacsmirror/el-search

Yes, thanks Drew.  Indeed I was wondering whether shorthands would break
el-search, but it was trivial to add support for them because el-search
is (solely) based on `read'ing.  Therefore it's slower than grepping of
course, but it's still acceptable (~ 10 seconds for the entire Emacs
Lisp code base)

> (Not sure that's the latest or best or only location.
> Ccing Michael H.)

Latest and best location is on my HDD, sadly, I did not update for quite
a while (actually since the change of the infrastructure of the Gnu Elpa
repo).  If anyone is interested I can post a newer version.


Michael.

A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[João Távora]

On Thu, Sep 30, 2021 at 11:11 PM Michael Heerdegen
<michael_heerdegen@web.de> wrote:
>
> Drew Adams <drew.adams@oracle.com> writes:
>
> > > Tools that understand the symbolic nature of the Lisp
> > > family of languages. For the example you have since,
> > > Tools that rely on way or the other really on the
> > > 'read' Lisp primitive.
> >
> > https://github.com/emacsmirror/el-search
>
> Yes, thanks Drew.  Indeed I was wondering whether shorthands would break
> el-search, but it was trivial to add support for them because el-search
> is (solely) based on `read'ing.  Therefore it's slower than grepping of
> course, but it's still acceptable (~ 10 seconds for the entire Emacs
> Lisp code base)

The "searching for symbols" problem is one of those embarrassingly parallel
ones, right?

Also, I can of course imaging that the `read' ing sexps is lower than reading
lines, but once you fill up an obarray with all the symbols you read, you don't
pay for the string comparison, since matching symbols to the search pattern
(presumably also a symbol) should be much faster since you can just use
the hashing of the obarray.

I haven't read or tried your code yet, I'm just speculating on possible
approaches.

João

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Michael Heerdegen]

João Távora <joaotavora@gmail.com> writes:

> The "searching for symbols" problem is one of those embarrassingly parallel
> ones, right?

> Also, I can of course imaging that the `read' ing sexps is lower than
> reading
> lines, but once you fill up an obarray with all the symbols you read,
> you don't
> pay for the string comparison, since matching symbols to the search pattern
> (presumably also a symbol) should be much faster since you can just use
> the hashing of the obarray.

Actually, el-search searches expressions.  The search "pattern" is a
description (predicate) for expressions, and it searches for matching
expressions.  I made it so that search patterns are pcase patterns -
while this fits perfectly, it's also the most deterring aspect for most
people :-P

Symbols are compared with `eq', so I think the answer to the question of
fast symbol comparison is "yes".  One surprisingly time consuming part
in simple cases is setting up buffers (to support read and sexp-based
scanning).

IIRC, "elisp-refs" (Melpa) copied the basic approach and created
something more lightweight but easier to use (a bit like xref I think, I
don't use xref - not sure).  I think adding shorthand support to that
package would also be trivial.

Michael.

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[João Távora]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> João Távora <joaotavora@gmail.com> writes:

> Actually, el-search searches expressions.  The search "pattern" is a
> description (predicate) for expressions, and it searches for matching
> expressions.  I made it so that search patterns are pcase patterns -
> while this fits perfectly, it's also the most deterring aspect for most
> people :-P

But of course, but it makes total sense that a Lisp-aware searcher would
be much more powerful like you have designed it.

Much as I applaud that, I think the grep crowd needs to be appeased with
something with a similar interface, though.  The passion I'm seeing in
some messages seems to be about knowing where a particular symbol is
referenced (understanding shorthands, of course).

> Symbols are compared with `eq', so I think the answer to the question of
> fast symbol comparison is "yes".  One surprisingly time consuming part
> in simple cases is setting up buffers (to support read and sexp-based
> scanning).

I'm going to vapourwarely say that can be optimized if you pass a
function as the STREAM argument to `read`.  Maybe?  You'll always need a
character buffer somehow, but perhaps you don't need to setup a big one
expensively.

Also what do you mean scanning, you mean `parse-partial-sexp` aka
syntax-ppss scanning?  That's much more expensive than `read` and used
for font-lock, C-M- navgation and such.  A more basic, faster searcher
wouldn't need that.

> IIRC, "elisp-refs" (Melpa) copied the basic approach and created
> something more lightweight but easier to use (a bit like xref I think, I
> don't use xref - not sure).  I think adding shorthand support to that
> package would also be trivial.

Interesting, and closeness to xref.el is indeed very useful.

João

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Michael Heerdegen]

João Távora <joaotavora@gmail.com> writes:

> The passion I'm seeing in some messages seems to be about knowing
> where a particular symbol is referenced (understanding shorthands, of
> course).

Absolutely understandable.

> I'm going to vapourwarely say that can be optimized if you pass a
> function as the STREAM argument to `read`.  Maybe?  You'll always need a
> character buffer somehow, but perhaps you don't need to setup a big one
> expensively.

`read'ing per se is not difficult.  But (from) where do I read, and
what?  Where does the code start?  What are only comments - what's not
there as code but part of a string (e.g. docstring)?  And there may be
other stuff the file may set up (like, now, shorthands).

> Also what do you mean scanning, you mean `parse-partial-sexp` aka
> syntax-ppss scanning?

Yes.  And sure, that's necessarily a part of such an algorithm.

> That's much more expensive than `read` and used for font-lock, C-M-
> navgation and such.  A more basic, faster searcher wouldn't need that.

That's why I made it so:

There are two steps: the first step is very fast and doesn't use that.
It kicks out all candidates where it's sure that there are no matches,
just by looking at the symbols, without sexp-level parsing.

The second step is slower but only has to look at the rest (a tiny part,
most of the time).


> Interesting, and closeness to xref.el is indeed very useful.

elisp-refs is not unpopular - someone might want to have a look at it.

Michael.

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1400 bytes --]

On Fri, Oct 01, 2021 at 12:38:21AM +0100, João Távora wrote:

[...]

> But of course, but it makes total sense that a Lisp-aware searcher would
> be much more powerful like you have designed it.
> 
> Much as I applaud that, I think the grep crowd needs to be appeased [...]

I must admit I haven't looked into the Lisp-aware searcher, and
I think the idea is intriguing (makes one think of Coccinelle).

So an answer like "RTFM" would be fine with me.

Now the question: does it also find things hidden away in comments?

This is one feature I wouldn't like to miss, as part of the "grep
crowd".

More generally: regexp based search is invaluable whenever I'm
diving in something I don't know well, which is some mix of different
languages (comments are a different language, that's why I brought
that example up), or where "I don't know yet what I'm doing".

All those "magic action at a distance" things (and name prefix
goes in that direction, but many things you see in contemporary
Web frameworks do this too) make life harder for the "grep
crowd".

Now I don't think this is in itself a reason to not do it. I've
missed some kind of name space mechanism in Emacs for ages.

But taking the grep crowd into consideration is a nice touch;
you might find yourself in that place some day, while wading
through the swamps of a foreign country :-)

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[João Távora]

On Fri, Oct 1, 2021 at 8:03 AM <tomas@tuxteam.de> wrote:

> All those "magic action at a distance" things (and name prefix
> goes in that direction, but many things you see in contemporary
> Web frameworks do this too) make life harder for the "grep
> crowd".

Yes, of course.  So-called higher-level languages are all about
indirection. Pointers, function pointers, late binding, garbage
collectors, duck typing, namespaces,,, they all give us something and they
take something away.  Every time.

Grep is a tool, a tool to search text.  It is only able to respond to
questions about a program's source when that source is written in text
files (which it normally is) and when the programming language is
relatively poor on indirection.  Or to put it another way: the
usefulness of grep as a program-related question-answering machine is
inversely proportional to the indirection mechanisms in the language in
question and its expressive power (aka the power to say more with less).

That doesn't mean that one can't devise tools to answer questions about
programs in a way that makes _less_ assumptions about the multiple ways
in which these program's sources are encoded.

And the 'read' based tools that we're discussing in this thread are
efforts in that direction.  They will still be fallible, no doubt, but
not in the ways grep is.  And grep will still be useful.  I'll only be
slightly _less_ useful because a new language feature has increased the
number of questions that can be asked about a source.  But that is what
indirections ALWAYS do.  "What is this pointer pointing to?", "What
version of this C lib am I linking against?", "What the heck is the GC
doing", "Is this object that responds to .quack() really a duck?", etc
etc.

(Curiously, this is _exactly also_ the reason why these features are
also culpable for performance degradations, something that we're much
more trained into accepting, because we experience them less directly)

Language design never has been held back by the particular assumptions
of a search tool, popular and ubiquitous as it may be.

And it's frankly a bit bonkers to even conceive that. When I can, I _do_ put
the C brace on the function definition line so that grep tells me if
it's the definition or the declaration I'm looking at, but how crazy
would it have been to force C not to have forward declarations BECAUSE
it could hurt grep?? (as it quite obviously does!).

That's why these "modern web frameworks" are causing you "grep pain".
But programs today (and for a good number of years) and even more, the
ever complex programs in the higher-level languages of the future will
even cause you more pain!

It's the job of development tools -- with the Editor at the center -- to
help the programmer navigate these things.  Hence M$ IntelliSense, hence
SLIME, hence LSP, hence xref.el, etc, etc, etc.  Emacs is in a very good
place in this regard, to be honest.  We have incredible talent and Elisp
is a very good language, way waaayy better than JS, for example.

> Now the question: does it also find things hidden away in comments?
> This is one feature I wouldn't like to miss, as part of the "grep
> crowd".

The first iteration of the tool I'm thinking of answers questions like
what is referenced by source code, today.  It doesn't answer "what could
someday be referenced when one uncomments lines".  A symbol name stuck
in a comment probably shouldn't be there, at least not in Shorthand
form.  And anyway Shorthands don't operate on comments, so the point
may be moot.  So the answer is that it can be done, but not in this first
iteration.  I will think about it, though.

> Now I don't think this is in itself a reason to not do it. I've
> missed some kind of name space mechanism in Emacs for ages.

Of course!

> But taking the grep crowd into consideration is a nice touch; you
> might find yourself in that place some day, while wading through the
> swamps of a foreign country :-)

Yes.  The things that said here are taken into consideration ;-) And yes I
use grep.  To "wade" into your delightful image, it's like using made-up
sign language in a foreign country: it'll get you places, but you won't
be reading the classics or chatting up the bear that wants to eat you.

João

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1785 bytes --]

On Fri, Oct 01, 2021 at 02:15:35PM +0100, João Távora wrote:
> On Fri, Oct 1, 2021 at 8:03 AM <tomas@tuxteam.de> wrote:
> 
> > All those "magic action at a distance" things (and name prefix
> > goes in that direction, but many things you see in contemporary
> > Web frameworks do this too) make life harder for the "grep
> > crowd".
> 
> Yes, of course.  So-called higher-level languages are all about
> indirection. Pointers, function pointers, late binding, garbage
> collectors, duck typing, namespaces,,, they all give us something and they
> take something away.  Every time.
> 
> Grep is a tool, a tool to search text [...]

I think we agree, basically. Note that I mentioned Coccinelle, which
goes deep into that direction.

> The first iteration of the tool I'm thinking of answers questions like
> what is referenced by source code, today.  It doesn't answer "what could
> someday be referenced when one uncomments lines".  A symbol name stuck
> in a comment probably shouldn't be there, at least not in Shorthand
> form.  And anyway Shorthands don't operate on comments, so the point
> may be moot.  So the answer is that it can be done, but not in this first
> iteration.  I will think about it, though.

Actually I wasn't specifically thinking about a source code snippet
hidden in a comment, but more of a text explaining things about a
variable or function. I like grep to find those, too :-)

[...]

> Yes.  The things that said here are taken into consideration ;-) And yes I
> use grep.  To "wade" into your delightful image, it's like using made-up
> sign language in a foreign country: it'll get you places, but you won't
> be reading the classics or chatting up the bear that wants to eat you.

nice image :)

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Dmitry Gutov]

On 01.10.2021 16:15, João Távora wrote:
> Language design never has been held back by the particular assumptions
> of a search tool, popular and ubiquitous as it may be.

Certain language designers intentionally limit the language's power due 
to usability considerations, keeping in mind their audience.

Speaking of shorthands, if only the "local" part of every symbol's name 
was something reliable (as is often the case in module/package systems 
out there), we could still implement the search for references using 
Grep fairly efficiently: you Grep across the files for the local name, 
and then post-filter the references by looking at the end of the file.

If that approach is not feasible, we're limited to searching for the 
instances of 'require' forms (when the symbol/function can be mapped to 
a package name) and then searching inside every such file on the second 
pass.

read-ing the contents of every Lisp file is pretty expensive, in comparison.

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[João Távora]

On Fri, Oct 1, 2021 at 3:30 PM Dmitry Gutov <dgutov@yandex.ru> wrote:
>
> On 01.10.2021 16:15, João Távora wrote:
> > Language design never has been held back by the particular assumptions
> > of a search tool, popular and ubiquitous as it may be.
>
> Certain language designers intentionally limit the language's power due
> to usability considerations, keeping in mind their audience.

What languages, what evidence for this?  Anyway, many more limit the power
due to performance considerations.  Counts as "usability"? I guess. IME
language  audiences that are interested in performance usually don't
care so much
about ergonomics and vice versa.

> Speaking of shorthands, if only the "local" part of every symbol's name
> was something reliable (as is often the case in module/package systems
> out there), we could still implement the search for references using
> Grep fairly efficiently: you Grep across the files for the local name,
> and then post-filter the references by looking at the end of the file.

Yes, yes, that's basically what happens with Common Lisp, because things
are separated by `:`.  And in many other languages as you say.  Unfortunately,
it's not 100% clean in Elisp because it relies on convention, not syntax. But
possible, yes.  Would you like to work on that `thing-at-pt.el` front?

> If that approach is not feasible, we're limited to searching for the
> instances of 'require' forms (when the symbol/function can be mapped to
> a package name) and then searching inside every such file on the second
> pass.

That's not the approach I was thinking of, but I hope to present a working
prototype soon, which is a better way to present ideas.

> read-ing the contents of every Lisp file is pretty expensive, in comparison.

Have you benchmarked?  What exactly have you benchmarked? Just `read` or
`read` + `parse-partial-sexp` (i.e. building the syntax-ppss cache)?
Versus what?

João Távora

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Dmitry Gutov]

On 01.10.2021 17:40, João Távora wrote:

>> Certain language designers intentionally limit the language's power due
>> to usability considerations, keeping in mind their audience.
> 
> What languages, what evidence for this?  Anyway, many more limit the power
> due to performance considerations.  Counts as "usability"? I guess. IME
> language  audiences that are interested in performance usually don't
> care so much
> about ergonomics and vice versa.

Go would be one example. The reasoning lies largely in the field of 
usability. Their understanding of it, at least.

But being able to use certain common tools belongs to the same area.

>> Speaking of shorthands, if only the "local" part of every symbol's name
>> was something reliable (as is often the case in module/package systems
>> out there), we could still implement the search for references using
>> Grep fairly efficiently: you Grep across the files for the local name,
>> and then post-filter the references by looking at the end of the file.
> 
> Yes, yes, that's basically what happens with Common Lisp, because things
> are separated by `:`.  And in many other languages as you say.  Unfortunately,
> it's not 100% clean in Elisp because it relies on convention, not syntax.

If we moved the declaration of the package prefix from the referring 
file (at the bottom) to the defining file, or just always said that the 
prefix must match the package name, this would make this possible. At 
the cost of some flexibility, of course, but I'm not sure who really 
needs that.

Taking the example from the manual, the clients would be able to write

;; elisp-shorthands: (("snu" . "some-nice-string-utils"))

but not

;; elisp-shorthands: (("sn" . "some-nice"))


and that doesn't sound like a terrible limitation.

> But
> possible, yes.  Would you like to work on that `thing-at-pt.el` front?

thing-at-pt? I'm not sure which particular task you are referring to.

>> If that approach is not feasible, we're limited to searching for the
>> instances of 'require' forms (when the symbol/function can be mapped to
>> a package name) and then searching inside every such file on the second
>> pass.
> 
> That's not the approach I was thinking of, but I hope to present a working
> prototype soon, which is a better way to present ideas.

Looking forward to it.

>> read-ing the contents of every Lisp file is pretty expensive, in comparison.
> 
> Have you benchmarked?  What exactly have you benchmarked? Just `read` or
> `read` + `parse-partial-sexp` (i.e. building the syntax-ppss cache)?
> Versus what?

I'm currently having a discussion where taking 2 seconds to search for a 
(very common) Lisp reference across Emacs code base is considered too 
slow, with all the Lisp overhead of parsing the results and constructing 
the list and rendering.

Simply doing

(benchmark 1 '(dolist (dir load-path)
                 (when (file-exists-p dir)
                   (let ((files (directory-files dir t "\\.el\\'")))
                     (dolist (file files)
                       (unless (file-directory-p file)
                         (with-temp-buffer
                           (insert-file file)
                           (read-from-string (buffer-string)))))))))

Reports 2.5 seconds here.

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[João Távora]

On Fri, Oct 1, 2021 at 4:48 PM Dmitry Gutov <dgutov@yandex.ru> wrote:
>
> On 01.10.2021 17:40, João Távora wrote:
>
> >> Certain language designers intentionally limit the language's power due
> >> to usability considerations, keeping in mind their audience.
> >
> > What languages, what evidence for this?  Anyway, many more limit the power
> > due to performance considerations.  Counts as "usability"? I guess. IME
> > language  audiences that are interested in performance usually don't
> > care so much
> > about ergonomics and vice versa.
>
> Go would be one example. The reasoning lies largely in the field of
> usability. Their understanding of it, at least.

Yes Go, I see what you mean.  But it's been growing with new features,
like generic functions.  And has namespaces.  They didn't design it
around grep, that's for sure.  That's what I meant.

> Taking the example from the manual, the clients would be able to write
> ;; elisp-shorthands: (("snu" . "some-nice-string-utils"))
> but not
> ;; elisp-shorthands: (("sn" . "some-nice"))
> and that doesn't sound like a terrible limitation.

I agree.  We could make it a recommendation, i.e. issue a (stern)
warning when we
detect this.  Or not allow shorthands of other forms in Emacs code, ELPA, etc.

> > But
> > possible, yes.  Would you like to work on that `thing-at-pt.el` front?
>
> thing-at-pt? I'm not sure which particular task you are referring to.

thingatpt.el, sorry.  The library used by other Elisp programs when they
want to pick some text from the buffer, at point, that represents a symbol,
a string, a list.  We could have some kind of "symbol-prefix" "symbol-suffix"
or "symbol-part" for eventually telling grep to go search only for that part.

> > That's not the approach I was thinking of, but I hope to present a working
> > prototype soon, which is a better way to present ideas.
> Looking forward to it.

Just realized the default xref-backend-references uses semantic and ede...

Also realized that you do some kind of (intern (format ""...)) there.
Grep heresy!

> Simply doing
>
> (benchmark 1 '(dolist (dir load-path)
>                  (when (file-exists-p dir)
>                    (let ((files (directory-files dir t "\\.el\\'")))
>                      (dolist (file files)
>                        (unless (file-directory-p file)
>                          (with-temp-buffer
>                            (insert-file file)
>                            (read-from-string (buffer-string)))))))))
>
> Reports 2.5 seconds here.

OK, thanks, gonna start with that snippet. Love a good snippet.
Are these Emacs's files? You can read from the buffer directly.  And
you could reuse buffers.
Dunno if it makes a difference.

But even before I do, a good way to solve this is the good old speed-for-space.
aka cache.  No point in reading a file that we've read and hasn't
changed since last
time right?  mtime caches look plausible, and we could even make it so
that visited
files' buffers are also read.

My first challenge will be to check if the reader does source
tracking, or is somehow
pluggable to call a function when it reads a symbol.

João

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[João Távora]

João Távora <joaotavora@gmail.com> writes:

>> (benchmark 1 '(dolist (dir load-path)
>>                  (when (file-exists-p dir)
>>                    (let ((files (directory-files dir t "\\.el\\'")))
>>                      (dolist (file files)
>>                        (unless (file-directory-p file)
>>                          (with-temp-buffer
>>                            (insert-file file)
>>                            (read-from-string (buffer-string)))))))))
>>
>> Reports 2.5 seconds here.

> Are these Emacs's files?

Nevermind, just saw it's load-path.  Which makes sense.

So I have to ask: is this an 'Emacs -Q' load-path?

João

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[João Távora]

João Távora <joaotavora@gmail.com> writes:

> João Távora <joaotavora@gmail.com> writes:
>
>>> (benchmark 1 '(dolist (dir load-path)
>>>                  (when (file-exists-p dir)
>>>                    (let ((files (directory-files dir t "\\.el\\'")))
>>>                      (dolist (file files)
>>>                        (unless (file-directory-p file)
>>>                          (with-temp-buffer
>>>                            (insert-file file)
>>>                            (read-from-string (buffer-string)))))))))
>>>
>>> Reports 2.5 seconds here.
>
>> Are these Emacs's files?
>
> Nevermind, just saw it's load-path.  Which makes sense.
>
> So I have to ask: is this an 'Emacs -Q' load-path?

Anyway, the good news is that:

 * avoiding read-from-string, and using read isntead
 * using a single buffer
 * using insert-file-contents (insert-file is for interactive use)
 * using the Emacs -Q load-path

speeds your snippet down from 2 to 0.2 seconds.

The bad news is that your snippet was only reading the first form in a
file, so when you read the whole file it takes 2.8 seconds on my corei7
machine to read the whole Emacs sources.

So performance wise, it's not bad but can probably be improved.  I don't think
it's gonna be stupidly hard because:

* this is one of those embarassingly parallel problems.  We could just spawn
  spawn `invocation-name' to use SMP

* designing a caching strategy seems fairly easy here.

João

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Michael Heerdegen]

João Távora <joaotavora@gmail.com> writes:

> * designing a caching strategy seems fairly easy here.

el-search uses caching of the data used by the first algorithm I
described - the one that kicks out most non-matches.  Thus, the second
run of finding references of a symbol that doesn't appear too often
takes around 3 seconds for the Emacs elisp code base, which is around
four times faster than the first run.

Michael.

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Dmitry Gutov]

On 01.10.2021 19:41, João Távora wrote:
> João Távora <joaotavora@gmail.com> writes:
> 
>> João Távora <joaotavora@gmail.com> writes:
>>
>>>> (benchmark 1 '(dolist (dir load-path)
>>>>                   (when (file-exists-p dir)
>>>>                     (let ((files (directory-files dir t "\\.el\\'")))
>>>>                       (dolist (file files)
>>>>                         (unless (file-directory-p file)
>>>>                           (with-temp-buffer
>>>>                             (insert-file file)
>>>>                             (read-from-string (buffer-string)))))))))
>>>>
>>>> Reports 2.5 seconds here.
>>
>>> Are these Emacs's files?
>>
>> Nevermind, just saw it's load-path.  Which makes sense.
>>
>> So I have to ask: is this an 'Emacs -Q' load-path?
> 
> Anyway, the good news is that:
> 
>   * avoiding read-from-string, and using read isntead
>   * using a single buffer
>   * using insert-file-contents (insert-file is for interactive use)
>   * using the Emacs -Q load-path
> 
> speeds your snippet down from 2 to 0.2 seconds.
> 
> The bad news is that your snippet was only reading the first form in a
> file, so when you read the whole file it takes 2.8 seconds on my corei7
> machine to read the whole Emacs sources.

Right.

Note that the above code uses very little Lisp and mostly calls into C. 
Any kind of deeper analysis based on reading the forms would need to 
have more processing implemented in Lisp around it. Which is often the 
slow part.

> So performance wise, it's not bad but can probably be improved.  I don't think
> it's gonna be stupidly hard because:
> 
> * this is one of those embarassingly parallel problems.  We could just spawn
>    spawn `invocation-name' to use SMP

If analysis is implemented in Lisp, the concurrency will be limited by 
that part. Though I guess you could launch multiple Emacs instances and 
then combine their results somehow.

> * designing a caching strategy seems fairly easy here.

Probably doable, yes, though both harder than the approach I described, 
and almost certainly slower even with cache (which needs regular 
invalidation). And the first search in the session (without cache to 
fall back on) will likely hurt the most.

Further, the code above only looks inside load-path one level deep. But 
we'd usually need to scan each directory recursively (there are packages 
like cedet with nested file structure).

Anyway, good luck. I just hope you don't end up with etags-like approach 
where the user will need to built and rebuild the index manually every 
time they want it to be fresh.

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:
> On 01.10.2021 19:41, João Távora wrote:

> If analysis is implemented in Lisp, the concurrency will be limited by
> that part. Though I guess you could launch multiple Emacs instances
> and then combine their results somehow.

Yes, that's exactly what I suggested. The combination is a sort
operation, which isn't very heavy for a 1000  matches.

>> * designing a caching strategy seems fairly easy here.
> Probably doable, yes, though both harder than the approach I
> described, and almost certainly slower even with cache (which needs
> regular invalidation). And the first search in the session (without
> cache to fall back on) will likely hurt the most.

You can very well make big parts of the database when building, for
example when byte-compiling.  And you only invalidate when you change a
file.

> Anyway, good luck. I just hope you don't end up with etags-like
> approach where the user will need to built and rebuild the index
> manually every time they want it to be fresh.

No that's not the goal.  The goal is to make a proper symbolic
xref-find-references for elisp which isn't regexp based (and which takes
around 4-5 seconds here everytime, btw...  and sometimes asks me if I
want to apply local variables for a file it's visiting... bug?)

Anyway, the reason I'm reasonably confident this _can_ be done is also
because this _has_ been done, in Common Lisp implementations.  That's
were xref.el comes from, ultimately.  They use even more advanced stuff,
like macroexpansion, so they catch the make-foo of defstruct.  Some
implementations have even more advanced stuff like proper who-calls,
who-sets, who-macroexpands.  They've had it for decades!  Have a look at
SLIME or Sly and plug them to SBCL or Allegro Common Lisp, for example.

João

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Dmitry Gutov]

On 02.10.2021 04:46, João Távora wrote:

>>> * designing a caching strategy seems fairly easy here.
>> Probably doable, yes, though both harder than the approach I
>> described, and almost certainly slower even with cache (which needs
>> regular invalidation). And the first search in the session (without
>> cache to fall back on) will likely hurt the most.
> 
> You can very well make big parts of the database when building, for
> example when byte-compiling.  And you only invalidate when you change a
> file.

And then someone will change a file using Vim, or an older version of 
Emacs. Or a misconfigured one.

>> Anyway, good luck. I just hope you don't end up with etags-like
>> approach where the user will need to built and rebuild the index
>> manually every time they want it to be fresh.
> 
> No that's not the goal.  The goal is to make a proper symbolic
> xref-find-references for elisp which isn't regexp based (and which takes
> around 4-5 seconds here everytime, btw...

(benchmark 1 '(xref-find-references "wrong-type-argument"))

=> 0.28s

Maybe try the version in emacs-28 branch.

Or if you are on macOS, see bug#50733 (its Grep is very slow).

> and sometimes asks me if I
> want to apply local variables for a file it's visiting... bug?)

Maybe during set-auto-mode in xref--collect-matches? I don't recall 
seeing anything like this recently, though.

> Anyway, the reason I'm reasonably confident this _can_ be done is also
> because this _has_ been done, in Common Lisp implementations.  That's
> were xref.el comes from, ultimately.  They use even more advanced stuff,
> like macroexpansion, so they catch the make-foo of defstruct.  Some
> implementations have even more advanced stuff like proper who-calls,
> who-sets, who-macroexpands.  They've had it for decades!  Have a look at
> SLIME or Sly and plug them to SBCL or Allegro Common Lisp, for example.

I'd be happy to see such features in Elisp, sure.

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[André A. Gomes]

João Távora <joaotavora@gmail.com> writes:

> Anyway, the reason I'm reasonably confident this _can_ be done is also
> because this _has_ been done, in Common Lisp implementations.  That's
> were xref.el comes from, ultimately.  They use even more advanced stuff,
> like macroexpansion, so they catch the make-foo of defstruct.  Some
> implementations have even more advanced stuff like proper who-calls,
> who-sets, who-macroexpands.  They've had it for decades!  Have a look at
> SLIME or Sly and plug them to SBCL or Allegro Common Lisp, for example.

Sadly, things don't change much over decades João!  Take a look at what
John Rose says regarding sophisticated programming environments (like
SLIME and Sly) and "good old" grep.  This an excerpt of a message from
1987.  Here's a fragment.

#begin_quote
When a LispM[achine] loads code into its memory, it loads a lot of
debugging information too.  For example, each function records the names
of its arguments and local variables, the names of all macros expanded
to produce its code, documentation strings, and sometimes an interpreted
definition, just for good measure.

Oh, each function also remembers which file it was defined in.  You have
no idea how useful this is: there’s an editor command called
“meta-point” that immediately transfers you to the source of any
function, without breaking your stride.  ANY function, not just one of a
special predetermined set.  Likewise, there’s a key that causes the
calling sequence of a function to be displayed instantly.

Logged into a Sun for the last few days, my Meta-Point reflex has
continued unabated, but it is completely frustrated.  The program that I
am working on has about 80 files.  If I want to edit the code of a
function Foo, I have to switch to a shell window and grep for named Foo
in various files.  Then I have to type in the name of the appropriate
file.  Then I have to correct my spelling error.  Finally I have to
search inside the file.  What used to take five seconds now takes a
minute or two.  (But what’s an order of magnitude between friends?)  By
this time, I really want to see the Sun at its best, so I’m tempted to
boot it a couple of times.
#end_quote

Take a look at http://www.art.net/~hopkins/Don/unix-haters/preface.html
for more.

I may eat soup with a fork, but at least I admit it's worse.


--
André A. Gomes
"Free Thought, Free World"

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Dmitry Gutov]

On 01.10.2021 19:05, João Távora wrote:
> On Fri, Oct 1, 2021 at 4:48 PM Dmitry Gutov <dgutov@yandex.ru> wrote:
>>
>> On 01.10.2021 17:40, João Távora wrote:
>>
>>>> Certain language designers intentionally limit the language's power due
>>>> to usability considerations, keeping in mind their audience.
>>>
>>> What languages, what evidence for this?  Anyway, many more limit the power
>>> due to performance considerations.  Counts as "usability"? I guess. IME
>>> language  audiences that are interested in performance usually don't
>>> care so much
>>> about ergonomics and vice versa.
>>
>> Go would be one example. The reasoning lies largely in the field of
>> usability. Their understanding of it, at least.
> 
> Yes Go, I see what you mean.  But it's been growing with new features,
> like generic functions.  And has namespaces.  They didn't design it
> around grep, that's for sure.  That's what I meant.

Growing very slowly.

Go wasn't designed around Grep, perhaps. But you made a high-level 
statement, and I make a high-level rebuke.

>> Taking the example from the manual, the clients would be able to write
>> ;; elisp-shorthands: (("snu" . "some-nice-string-utils"))
>> but not
>> ;; elisp-shorthands: (("sn" . "some-nice"))
>> and that doesn't sound like a terrible limitation.
> 
> I agree.  We could make it a recommendation, i.e. issue a (stern)
> warning when we
> detect this.  Or not allow shorthands of other forms in Emacs code, ELPA, etc.

Or make it mandatory, thus making it possible to use the approach to 
searching I've described, and more or less guarantee it working on 
third-party code as well.

>>> But
>>> possible, yes.  Would you like to work on that `thing-at-pt.el` front?
>>
>> thing-at-pt? I'm not sure which particular task you are referring to.
> 
> thingatpt.el, sorry.  The library used by other Elisp programs when they
> want to pick some text from the buffer, at point, that represents a symbol,
> a string, a list.  We could have some kind of "symbol-prefix" "symbol-suffix"
> or "symbol-part" for eventually telling grep to go search only for that part.

I think you're rather looking at altering what

   (get 'emacs-lisp-mode 'find-tag-default-function)

returns.

>>> That's not the approach I was thinking of, but I hope to present a working
>>> prototype soon, which is a better way to present ideas.
>> Looking forward to it.
> 
> Just realized the default xref-backend-references uses semantic and ede...

Uses a certain minor part of Semantic and doesn't use EDE.

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 01.10.2021 19:05, João Távora wrote:

> Go wasn't designed around Grep, perhaps. But you made a high-level
> statement, and I make a high-level rebuke.

Actually no.  Since you bring it up, I wrote:

>..> Language design never has been held back by the particular
>..> assumptions of a search tool, popular and ubiquitous as it may be.
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I really did mean Grep, which is even more obvious from the
context.. _You_ made up another, different, statement about "usability"
and rebuked that.  So you self-rebuked, or something of the sort.  But I
agree with your point, in part, nonetheless.

> Or make it mandatory, thus making it possible to use the approach to
> searching I've described, and more or less guarantee it working on
> third-party code as well.

We could, but it's too soon for that.  Bear in mind, one thing is the
Emacs repo, another thing is what you wish to do with the language.  And
a warning would more than suitably protect the former (once we decide
what needs to be protected and from what).  For the latter, removing a
feature that works fine is almost immoral IMO.  You may as well forbid
small prefixes alltogether, or heretic symbol names, or anaphoric
macros.

Rather, I think we must understand what the grep-inclined want to do.
For example it might make sense to enforce some rules for external
symbols (the ones that one commonly searches for across files) and
others for internal symbols.

>>>> But
>>>> possible, yes.  Would you like to work on that `thing-at-pt.el` front?
>>>
>>> thing-at-pt? I'm not sure which particular task you are referring to.
>> thingatpt.el, sorry.  The library used by other Elisp programs when
>> they
>> want to pick some text from the buffer, at point, that represents a symbol,
>> a string, a list.  We could have some kind of "symbol-prefix" "symbol-suffix"
>> or "symbol-part" for eventually telling grep to go search only for that part.
>
> I think you're rather looking at altering what
>
>   (get 'emacs-lisp-mode 'find-tag-default-function)
>
> returns.

What does that do?  Returns nil here.  I'd like the approach to work
with Leo Liu's ack.el, for example, which is the particular 'grep'
interface I use.  It uses 'thingatpt'.

>>>> That's not the approach I was thinking of, but I hope to present a working
>>>> prototype soon, which is a better way to present ideas.
>>> Looking forward to it.
>> Just realized the default xref-backend-references uses semantic and
>> ede...
> Uses a certain minor part of Semantic and doesn't use EDE.

I wasn't criticizing btw.  Wouldn't be a problem if it used even a big
part.

João

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Dmitry Gutov]

On 02.10.2021 04:30, João Távora wrote:

>> Or make it mandatory, thus making it possible to use the approach to
>> searching I've described, and more or less guarantee it working on
>> third-party code as well.
> 
> We could, but it's too soon for that.  Bear in mind, one thing is the
> Emacs repo, another thing is what you wish to do with the language.  And
> a warning would more than suitably protect the former (once we decide
> what needs to be protected and from what).  For the latter, removing a
> feature that works fine is almost immoral IMO.  You may as well forbid
> small prefixes alltogether, or heretic symbol names, or anaphoric
> macros.

Whatever search feature we end up implementing, should work on Elisp 
code anywhere, inside or outside of Emacs core.

> Rather, I think we must understand what the grep-inclined want to do.

I wasn't really describing what the users of Grep would do. If that was 
the goal, prohibiting shorthands would be the answer.

> For example it might make sense to enforce some rules for external
> symbols (the ones that one commonly searches for across files) and
> others for internal symbols.

I think the "private" symbols are largely irrelevant to this discussion. 
Unless people really are (?) going to use shorthands for them.

>>>>> But
>>>>> possible, yes.  Would you like to work on that `thing-at-pt.el` front?
>>>>
>>>> thing-at-pt? I'm not sure which particular task you are referring to.
>>> thingatpt.el, sorry.  The library used by other Elisp programs when
>>> they
>>> want to pick some text from the buffer, at point, that represents a symbol,
>>> a string, a list.  We could have some kind of "symbol-prefix" "symbol-suffix"
>>> or "symbol-part" for eventually telling grep to go search only for that part.
>>
>> I think you're rather looking at altering what
>>
>>    (get 'emacs-lisp-mode 'find-tag-default-function)
>>
>> returns.
> 
> What does that do?  Returns nil here.  I'd like the approach to work
> with Leo Liu's ack.el, for example, which is the particular 'grep'
> interface I use.  It uses 'thingatpt'.

Yes, sorry. That one was for 'find-tag' and 'xref-find-definitions' with 
etags.

Tools like ack-el indeed use thingatpt, though I wonder whether we 
should add a new "thing" rather than change how 'symbol' works. That is, 
consider whether any of the existing users might not like this change.

>>>>> That's not the approach I was thinking of, but I hope to present a working
>>>>> prototype soon, which is a better way to present ideas.
>>>> Looking forward to it.
>>> Just realized the default xref-backend-references uses semantic and
>>> ede...
>> Uses a certain minor part of Semantic and doesn't use EDE.
> 
> I wasn't criticizing btw.  Wouldn't be a problem if it used even a big
> part.

It's really just find-grep under the hood. Or Global, or id-utils.

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Whatever search feature we end up implementing, should work on Elisp
> code anywhere, inside or outside of Emacs core.

Yes.  xref-find-references is that feature for me.

But my understanding is that you were somehow suggesting another,
simpler method of having grep search by only the suffix.  Which is a
good idea, but -- as always -- needs some assumptions in the code.

I'm just saying we shouldn't _force_ any code to be constrained to such
assumptions.  But if _some code_ happens to want to be constrained by
those assumption, then your idea is valuable and should exist alongside
xref-find-references.

>> Rather, I think we must understand what the grep-inclined want to do.

> I wasn't really describing what the users of Grep would do. If that
> was the goal, prohibiting shorthands would be the answer.

I'm not talking about what they _would_ do.  Obviously they would Grep.
But they don't grep gratutiously, they are trying to answer questions.
If you give them tools that give those answers, maybe they will start
using those tools.

>> For example it might make sense to enforce some rules for external
>> symbols (the ones that one commonly searches for across files) and
>> others for internal symbols.
>
> I think the "private" symbols are largely irrelevant to this
> discussion. Unless people really are (?) going to use shorthands for
> them.

I would.  In fact, I would use them _prominently_ for private symbols,
which are the vast majority of symbols I write and precisely where I
feel most pain.  eglot--this, eglot--that, eglot-test--foo.  For
definitions of external symbols, like '(defun eglot-super-important-bit
() )' I would _not_ use them, precisely to safeguard answering some
basic questions with "grep".

That would be "my rule", at least a draft version of it.  But also, like
probably most people passionately discussing this feature, I too haven't
really had time to even play around with it (I spend my Emacs hack
budget writing e-mails ;-) ).

>> What does that do?  Returns nil here.  I'd like the approach to work
>> with Leo Liu's ack.el, for example, which is the particular 'grep'
>> interface I use.  It uses 'thingatpt'.
>
> Yes, sorry. That one was for 'find-tag' and 'xref-find-definitions'
> with etags.
>
> Tools like ack-el indeed use thingatpt, though I wonder whether we
> should add a new "thing" rather than change how 'symbol' works. That
> is, consider whether any of the existing users might not like this
> change.

Yes, I also think we should add a new "thing".  For your "grep-by-parts"
idea, I mean, I hope that's understood.  For xref-find-references, the
'symbol' is the thing.

João

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Dmitry Gutov]

On 02.10.2021 05:05, João Távora wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>> Whatever search feature we end up implementing, should work on Elisp
>> code anywhere, inside or outside of Emacs core.
> 
> Yes.  xref-find-references is that feature for me.
> 
> But my understanding is that you were somehow suggesting another,
> simpler method of having grep search by only the suffix.  Which is a
> good idea, but -- as always -- needs some assumptions in the code.

Assumptions that most languages with module/namespace systems satisfy.

> I'm just saying we shouldn't _force_ any code to be constrained to such
> assumptions.  But if _some code_ happens to want to be constrained by
> those assumption, then your idea is valuable and should exist alongside
> xref-find-references.

Yes, I was saying the opposite.

>> I think the "private" symbols are largely irrelevant to this
>> discussion. Unless people really are (?) going to use shorthands for
>> them.
> 
> I would.  In fact, I would use them _prominently_ for private symbols,
> which are the vast majority of symbols I write and precisely where I
> feel most pain.  eglot--this, eglot--that, eglot-test--foo.  For
> definitions of external symbols, like '(defun eglot-super-important-bit
> () )' I would _not_ use them, precisely to safeguard answering some
> basic questions with "grep".

That makes sense. But if the goal of shorthands was to shorten internal 
symbols only, it would probably looked a little different.

And anyway, if multiple packages have internal symbols with the same 
local name, the search for is likely to require the same approach and 
hence the same limitations. Unless we're really going to prohibit 
packages from using "private" functions from other packages (thus being 
able to limit the search to the current file). Which is a frowned-upon 
but common practice.

> That would be "my rule", at least a draft version of it.  But also, like
> probably most people passionately discussing this feature, I too haven't
> really had time to even play around with it (I spend my Emacs hack
> budget writing e-mails ;-) ).

Hah. ;-(

>> Tools like ack-el indeed use thingatpt, though I wonder whether we
>> should add a new "thing" rather than change how 'symbol' works. That
>> is, consider whether any of the existing users might not like this
>> change.
> 
> Yes, I also think we should add a new "thing".  For your "grep-by-parts"
> idea, I mean, I hope that's understood.  For xref-find-references, the
> 'symbol' is the thing.

And then we encounter the question, what is a 'symbol', really.

Either way, yes, these are two distinct usage examples with different 
answers to that question.

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Adam Porter]

João Távora <joaotavora@gmail.com> writes:

>> I think the "private" symbols are largely irrelevant to this
>> discussion. Unless people really are (?) going to use shorthands for
>> them.
>
> I would.  In fact, I would use them _prominently_ for private symbols,
> which are the vast majority of symbols I write and precisely where I
> feel most pain.  eglot--this, eglot--that, eglot-test--foo.  For
> definitions of external symbols, like '(defun eglot-super-important-bit
> () )' I would _not_ use them, precisely to safeguard answering some
> basic questions with "grep".

Yes, I'd also like to use them within a package.  For example, in
org-super-agenda.el, rather than "org-super-agenda-this" and
"org-super-agenda--that", it would be nice to be able to write something
like "osa-this" and "osa--that" (or, if I wanted to be cute, something
like "~/this" and "~/-that", since some kind of prefix is still
required).

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Adam Porter]

Dmitry Gutov <dgutov@yandex.ru> writes:

> If we moved the declaration of the package prefix from the referring
> file (at the bottom) to the defining file, or just always said that
> the prefix must match the package name, this would make this
> possible. At the cost of some flexibility, of course, but I'm not sure
> who really needs that.
>
> Taking the example from the manual, the clients would be able to write
>
> ;; elisp-shorthands: (("snu" . "some-nice-string-utils"))
>
> but not
>
> ;; elisp-shorthands: (("sn" . "some-nice"))
>
>
> and that doesn't sound like a terrible limitation.

Maybe I misunderstood you, but having packages declare their own symbol
shorthands doesn't seem practical to me.  For example, if
"some-nice-utils.el" declared its shorthand to be "snu-", that would
conflict with a package "snu.el".  It would seem to make the existing
single-namespace problem worse by allowing packages to claim more of the
namespace, similar to "domain squatting."

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Dmitry Gutov]

On 02.10.2021 11:36, Adam Porter wrote:
> Maybe I misunderstood you, but having packages declare their own symbol
> shorthands doesn't seem practical to me.

No, that's not what I meant.

> For example, if
> "some-nice-utils.el" declared its shorthand to be "snu-", that would
> conflict with a package "snu.el".  It would seem to make the existing
> single-namespace problem worse by allowing packages to claim more of the
> namespace, similar to "domain squatting."

The shorthands would still be decided in the calling file. Just the part 
that is "shortened" will be predetermined. The LONGHAND-PREFIX part.

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

We have two kinds of uses for shorthands.

* Creating longer real names to avoid collisions.
In this kind of case the package that you load
creates shorthands for itself, which rename its symbols to longer
names that won't conflict.  This is what we will do with s.el.

* Creating shorter names for convenience.
In this kind of case, one Lisp file would create shorthand prefixes
for code in other files.  These prefixes might really be shorter
than the symbols' documented name.

People who had the second case in mind suggested name conventions
for the shorthand prefixes.  Suppose we use ":" as the end of a prefix.

Suppose you want to search for calls to find-outer-otter-create-otter.
If there is a call to foo:create-otter in a file wehack.el,
it would be possible to look at wehack.el and see if it defines
`foo:' as a shorthand for `find-outer-otter-'.  If so,
that is a call to find-outer-otter-create-otter.

The hard part is to determine which strings to actually grep for.
There are various ways to do it.

Here's one:

Assume the convention that the shorthand substitute always ends in
`-'.  So grep for `:otter', `:create-otter', `:outer-create-otter',
and so on.  When you find any of them, you can check whether that file
defines a shortcut that would supply what's missing to generate
`find-outer-otter-create-otter'.  For instance, if it defines
`walrus:' as a shorthand for `find-outer', then removing `find-outer'
from the start of `find-outer-otter-create-otter', you see that a
reference to that symbol using that shorthand would be
`walrus:otter-create-otter'.  If the file contains that, it is a hit.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Gregory Heytings]

>
> We have two kinds of uses for shorthands.
>
> * Creating longer real names to avoid collisions. In this kind of case 
> the package that you load creates shorthands for itself, which rename 
> its symbols to longer names that won't conflict.  This is what we will 
> do with s.el.
>
> * Creating shorter names for convenience. In this kind of case, one Lisp 
> file would create shorthand prefixes for code in other files.  These 
> prefixes might really be shorter than the symbols' documented name.
>

From a programming language design viewpoint, a single solution to cope 
with these two use cases is IMO clearly not TRT.  Because:

(1) the need for a solution to the first use case is essentially a 
consequence of the absence of a solution to the second use case (IOW, a 
few library authors have chosen to use short prefixes in their libraries 
to make them more appealing to use),

(2) the first use case is limited to very few libraries,

(3) the only way to cope with these two use cases (which are the opposite 
of each other) with a single solution is to add a preprocessor-like 
feature to the Lisp read primitive, with which any token can be changed 
into any other token, which is unnecessary for the use case that should 
become the only one in the long term (namely the second one),

(4) a solution to the first use case cannot be a long term solution, it 
can only be a temporary workaround, for example because docstrings are not 
modified with such a preprocessor-like feature.

It would IMO be better to add a proper solution for the second use case, 
together with a temporary workaround for the first use case which would 
work during one major Emacs release, to give some time to the authors of 
these few libraries to adapt their code to the new feature (IOW, to 
abandon the short prefix they use and to use the new feature instead).

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

For reference, these are the two use-cases:

  > > * Creating longer real names to avoid collisions. In this kind of case 
  > > the package that you load creates shorthands for itself, which rename 
  > > its symbols to longer names that won't conflict.  This is what we will 
  > > do with s.el.
  > >
  > > * Creating shorter names for convenience. In this kind of case, one Lisp 
  > > file would create shorthand prefixes for code in other files.  These 
  > > prefixes might really be shorter than the symbols' documented name.

You wrote:

  > (1) the need for a solution to the first use case is essentially a 
  > consequence of the absence of a solution to the second use case (IOW, a 
  > few library authors have chosen to use short prefixes in their libraries 
  > to make them more appealing to use),

It might be true that the existence _in the past_ of a system for
shortening the names of symbols in another package would have avoided
making libraries whose own name prefixes are "s-" and "-".  We can't be
sure.

It does not follow that adding such a system now would eliminate the
problem that those short prefixes cause.  I don't think it would.

People have proposed conventions to use particular punctuation characters
to indicate a shortened prefix -- such as `s:', perhaps.  And maybe
that's what we should do.  But programs that do (require 's) don't
start those symbols with `s:'.  They start them with `s-'.

  > (4) a solution to the first use case cannot be a long term solution,

We need a long term solution, and that's what this is intended to be.
It is a good solution because it avoids the need to convince hundreds,
perhaps thousands, of library authors that we are not in communication
with to change the way they write calls to Magnars's libraries.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Gregory Heytings]

Hi Richard,

Thank you for your kind and courteous answer.  Sorry for my belated reply, 
I had to think about this a bit.

> For reference, these are the two use-cases:
>
>>> * Creating longer real names to avoid collisions. In this kind of case 
>>> the package that you load creates shorthands for itself, which rename 
>>> its symbols to longer names that won't conflict.  This is what we will 
>>> do with s.el.
>>>
>>> * Creating shorter names for convenience. In this kind of case, one 
>>> Lisp file would create shorthand prefixes for code in other files. 
>>> These prefixes might really be shorter than the symbols' documented 
>>> name.
>>
>> (4) a solution to the first use case cannot be a long term solution,
>
> We need a long term solution, and that's what this is intended to be. 
> It is a good solution because it avoids the need to convince hundreds, 
> perhaps thousands, of library authors that we are not in communication 
> with to change the way they write calls to Magnars's libraries.
>

I understand that it is intended to be a long-term solution, but it is 
not.  We cannot convince all these library authors to change the way they 
write calls to Magnars' libraries, yet with the chosen solution, it will 
nonetheless be necessary to adapt each Elisp file they write, by changing 
the (require 's) into a (require 'magnars-string) and by adding a 
read-symbol-shorthands local variable at the end of these files.

The proper solution to that problem, which does not require to change a 
single byte in any of these third-party Elisp files, would have been to 
create a hook that is executed on the file contents before it is processed 
by





, it can only be a temporary workaround, for example because docstrings 
are not modified with such a preprocessor-like feature.

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 2107 bytes --]


[Sorry for my previous unfinished message.]

Hi Richard,

Thank you for your kind and courteous answer.  Sorry for my belated reply, 
I had to think about this a bit.

> For reference, these are the two use-cases:
>
>>> * Creating longer real names to avoid collisions. In this kind of case 
>>> the package that you load creates shorthands for itself, which rename 
>>> its symbols to longer names that won't conflict.  This is what we will 
>>> do with s.el.
>>>
>>> * Creating shorter names for convenience. In this kind of case, one 
>>> Lisp file would create shorthand prefixes for code in other files. 
>>> These prefixes might really be shorter than the symbols' documented 
>>> name.
>>
>> (4) a solution to the first use case cannot be a long term solution,
>
> We need a long term solution, and that's what this is intended to be. It 
> is a good solution because it avoids the need to convince hundreds, 
> perhaps thousands, of library authors that we are not in communication 
> with to change the way they write calls to Magnars's libraries.
>

I understand that it is intended to be a long-term solution, but it is 
not.  We cannot convince all these library authors to change the way they 
write calls to Magnars' libraries, yet with the chosen solution, it will 
nonetheless be necessary to adapt each Elisp file they write, by changing 
the (require 's) into a (require 'magnars-string) and by adding a 
read-symbol-shorthands local variable at the end of these files.

The proper solution to that problem, which does not require to change a 
single byte in any of these third-party Elisp files, would have been to 
create a hook that is executed on the file contents before it is processed 
by readevalloop, or IOW, a hook with which it would be possible to 
preprocess Elisp files.

I attach an example function which would be put in that hook: it checks 
whether the file contains a (require 's) and at least one call to one of 
the functions defined in that library, and if so, replaces all symbols 
"exported" by the s.el library (and only those symbols) in the file with 
new ones.

[-- Attachment #2: Type: text/plain, Size: 2033 bytes --]

(defvar magnars-string-symbols '(s-trim s-trim-left s-trim-right
s-collapse-whitespace s-split s-split-up-to s-lines s-join
s-concat s-prepend s-append s-repeat s-chop-suffix
s-chop-suffixes s-chop-prefix s-chop-prefixes s-shared-start
s-shared-end s-chomp s-truncate s-word-wrap s-center s-pad-left
s-pad-right s-left s-right s-ends-with? s-starts-with?
s-contains? s-equals? s-less? s-matches? s-blank? s-blank-str?
s-present? s-presence s-lowercase? s-uppercase? s-mixedcase?
s-capitalized? s-numeric? s-replace s-replace-regexp
s-replace-all s-downcase s-upcase s-capitalize s-titleize s-with
s-index-of s-reverse s-match-strings-all s-matched-positions-all
s-match s-slice-at s-split-words s-lower-camel-case
s-upper-camel-case s-snake-case s-dashed-words
s-capitalized-words s-titleized-words s-word-initials s-format
s-lex-value-as-lisp s-lex-fmt|expand s-lex-format s-count-matches
s-count-matches-all s-wrap s-blank-p s-blank-str-p
s-capitalized-p s-contains-p s-ends-with-p s-equals-p s-less-p
s-lowercase-p s-matches-p s-mixedcase-p s-numeric-p s-prefix-p
s-prefix? s-present-p s-starts-with-p s-suffix-p s-suffix?
s-uppercase-p s--truthy? s--aget s--mapcar-head))

(defun convert-require-magnars-string ()
  (goto-char (point-min))
  (when (search-forward "(require 's)" nil t))
  (setq found nil)
  (dolist (symbol magnars-string-symbols)
    (when (not found)
      (goto-char (point-min))
      (setq sym (substring (format "%s" symbol) 2))
      (when (re-search-forward
             (concat "\\_<s-" (regexp-quote sym) "\\_>") nil t)
        (setq found t))))
  (when found
    (goto-char (point-min))
    (search-forward "(require 's)" nil)
    (replace-match "(require 'magnars-string)")
    (dolist (symbol magnars-string-symbols)
      (goto-char (point-min))
      (setq sym (substring (format "%s" symbol) 2))
      (while (re-search-forward
              (concat "\\_<s-" (regexp-quote sym) "\\_>") nil t)
        (replace-match (format "magnars-string-%s" sym))))))

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I understand that it is intended to be a long-term solution, but it is 
  > not.  We cannot convince all these library authors to change the way they 
  > write calls to Magnars' libraries, yet with the chosen solution, it will 
  > nonetheless be necessary to adapt each Elisp file they write, by changing 
  > the (require 's) into a (require 'magnars-string) and by adding a 
  > read-symbol-shorthands local variable at the end of these files.

I think there is a misunderstanding about how we plan to handle this.
Unless I have misremembered, we plan to replace the file s.el with a
simple file that sets up a shorthand for `s-' and loads magnar-string.el.

The programs that currently use s.el will require no change at all.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Gregory Heytings]

>> I understand that it is intended to be a long-term solution, but it is 
>> not.  We cannot convince all these library authors to change the way 
>> they write calls to Magnars' libraries, yet with the chosen solution, 
>> it will nonetheless be necessary to adapt each Elisp file they write, 
>> by changing the (require 's) into a (require 'magnars-string) and by 
>> adding a read-symbol-shorthands local variable at the end of these 
>> files.
>
> I think there is a misunderstanding about how we plan to handle this. 
> Unless I have misremembered, we plan to replace the file s.el with a 
> simple file that sets up a shorthand for `s-' and loads 
> magnar-string.el.
>
> The programs that currently use s.el will require no change at all.
>

I fear you misremember something, indeed.  The programs that use s.el will 
of course require a similar change.  Otherwise how could Emacs know that 
the calls to s-trim in library frobnicate.el are to be understood as calls 
to magnars-string-trim?

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I fear you misremember something, indeed.  The programs that use s.el will 
  > of course require a similar change.

In the plan, they won't need a change.  

  >   Otherwise how could Emacs know that 
  > the calls to s-trim in library frobnicate.el are to be understood as calls 
  > to magnars-string-trim?

The plan was that requiring s.el would set up shortcuts for the
containing file.

Joao, is that included in what we have installed?


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Gregory Heytings]

>> Speaking of shorthands, if only the "local" part of every symbol's name 
>> was something reliable (as is often the case in module/package systems 
>> out there), we could still implement the search for references using 
>> Grep fairly efficiently: you Grep across the files for the local name, 
>> and then post-filter the references by looking at the end of the file.
>
> Yes, yes, that's basically what happens with Common Lisp, because things 
> are separated by `:`.  And in many other languages as you say. 
> Unfortunately, it's not 100% clean in Elisp because it relies on 
> convention, not syntax.
>

It seems very easy to enforce a syntax for this:

diff --git a/src/lread.c b/src/lread.c
index af0a799459..b66585df7d 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -4657,6 +4657,7 @@ oblookup_considering_shorthand (Lisp_Object obarray, 
const char *in,
        Lisp_Object lh_prefix = XCDR (pair);
        if (!STRINGP (sh_prefix) || !STRINGP (lh_prefix))
         continue;
+      sh_prefix = concat2 (sh_prefix, build_string ("::"));
        ptrdiff_t sh_prefix_size = SBYTES (sh_prefix);

        /* Compare the prefix of the transformation pair to the symbol

The "::" separator has a similar purpose in other languages (C++, Perl), 
it is not used anywhere in Emacs core, and it is used in a single place in 
ELPA (var::append-list in hyperbole).  The ":" separator is already used 
in quite a few places, in Emacs core and in ELPA.

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[João Távora]

On Fri, Oct 1, 2021 at 11:58 PM Gregory Heytings <gregory@heytings.org> wrote:

> > Yes, yes, that's basically what happens with Common Lisp, because things
> > are separated by `:`.  And in many other languages as you say.
> > Unfortunately, it's not 100% clean in Elisp because it relies on
> > convention, not syntax.

> It seems very easy to enforce a syntax for this:

It also very easily breaks one of the main use cases for this
feature: to import s.el, dash.el, f.el and its user libraries with
minimal changes to them.

I've said this before, I prefer Common Lisp packages as a namespacing
system,too  but not only are they unsuitable for the above task, but
they are very negatively regarded within groups of vocal Emacs users
(misunderstood, in my personal opinion).  They have been proposed many
times here, without any success.  There are even Elisp implementations for
them, I think, search in your favourite search engine.

João

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Gregory Heytings]

>>> Yes, yes, that's basically what happens with Common Lisp, because 
>>> things are separated by `:`.  And in many other languages as you say. 
>>> Unfortunately, it's not 100% clean in Elisp because it relies on 
>>> convention, not syntax.
>
>> It seems very easy to enforce a syntax for this:
>
> It also very easily breaks one of the main use cases for this feature: 
> to import s.el, dash.el, f.el and its user libraries with minimal 
> changes to them.
>

I probably misunderstand something, but why would it break that use case? 
Could you indicate what would be put in the elisp-shorthands for (a) the 
s.el library, and (b) a user library that uses s.el?

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 422 bytes --]

On Sat, Oct 2, 2021, 09:51 Gregory Heytings <gregory@heytings.org> wrote:

>
> I probably misunderstand something,


Yep.

but why would it break that use case?
> Could you indicate what would be put in the elisp-shorthands for (a) the
> s.el library, and (b) a user library that uses s.el?
>

Someone else can explain, but it's the same thing for both. Or you can go
read the fine manual.

João

>
>

[-- Attachment #2: Type: text/html, Size: 1189 bytes --]

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Gregory Heytings]

>> but why would it break that use case? Could you indicate what would be 
>> put in the elisp-shorthands for (a) the s.el library, and (b) a user 
>> library that uses s.el?
> 
> Someone else can explain, but it's the same thing for both. Or you can 
> go read the fine manual.
>

RTFM isn't exactly a useful answer.  And no, the manual doesn't explain 
what you want to do.  Apparently you have a specific idea in mind, which 
is, in your own words, "to import s.el, dash.el, f.el and its user 
libraries with minimal changes to them."  Import where?  And why would you 
want to make their prefix names shorter than a single character?

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 392 bytes --]

On Sat, Oct 2, 2021, 12:57 Gregory Heytings <gregory@heytings.org> wrote:

>
> >>
>
> RTFM isn't exactly a useful answer.
>

Then WTFG, watch the fine gif? Can you find out in the top message? Hope
you can. The answer is there in front of your eyes. Watch it well. This
constant email flood a true DOS so I'm sorry I can't be your personal tutor
on the matter here.

João

>

[-- Attachment #2: Type: text/html, Size: 1167 bytes --]

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[João Távora]

Here's a slightly better answer.  Have a look at commit

39a63cda6dcbc4fb79dfd34bec1144057c1c3a10

in the repo.  It adds magnars-string.el.  I didn't include it in main
yet because
of possible copyright issues (should be fine though).  Anyway, it clears up
how it works.

Thanks,
João

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Gregory Heytings]

> 
> Then WTFG, watch the fine gif? Can you find out in the top message? Hope 
> you can. The answer is there in front of your eyes. Watch it well. This 
> constant email flood a true DOS so I'm sorry I can't be your personal 
> tutor on the matter here.
>

I'm certainly not asking for a tutor, no, and no, the answer is not "in 
front of my eyes".  Because your "fine gif" doesn't show what 
"magnars-string" is.  I'm going to assume that it's the "s.el" library, at 
the end of which you added the following local variable:

;; elisp-shorthands: (("s-" . "magnars-string-"))

and in which you replaced (provide 's) by (provide 'magnars-string).

Now, I agree with you what you said in an earlier post: "Language design 
never has been held back by the particular assumptions of a search tool, 
popular and ubiquitous as it may be."  To which I would add that language 
design should not be influenced by particular choices of libraries, 
popular and ubiquitous as they may be.

A handful Elisp libraries have decided (for good reasons, but against the 
common practice so far) to use a short prefix for their functions.  That 
doesn't seem a sufficient reason to design namespaces in Elisp in such a 
way that these particular libraries need to be changed as little as 
possible.  (And they have to be changed more than you apparently think: if 
I do the above and do C-h f magnars-string-presence RET, the docstring 
says "Return S if it's `s-present?', otherwise return nil.".  Of course 
`s-present?' is not considered as a symbol in the *Help* buffer, and C-h o 
s-present? RET does not work.)

If supporting these few libraries is a requirement, a better approach 
would be to introduce a particular and temporary exception for them, 
something like:

diff --git a/src/lread.c b/src/lread.c
index af0a799459..44e94b9604 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -4657,6 +4657,10 @@ oblookup_considering_shorthand (Lisp_Object 
obarray, const char *in,
        Lisp_Object lh_prefix = XCDR (pair);
        if (!STRINGP (sh_prefix) || !STRINGP (lh_prefix))
  	continue;
+      if (strlen (SSDATA (sh_prefix)) < 3)
+	sh_prefix = concat2 (sh_prefix, build_string ("-"));
+      else
+	sh_prefix = concat2 (sh_prefix, build_string ("::"));
        ptrdiff_t sh_prefix_size = SBYTES (sh_prefix);

        /* Compare the prefix of the transformation pair to the symbol

Such short prefixes would raise a warning in Emacs 30, and an error in 
Emacs 32.

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Stefan Kangas]

Gregory Heytings <gregory@heytings.org> writes:

> A handful Elisp libraries have decided (for good reasons, but against the
> common practice so far) to use a short prefix for their functions.  That
> doesn't seem a sufficient reason to design namespaces in Elisp in such a
> way that these particular libraries need to be changed as little as
> possible.

This might be a tangent, but can we really say that shorthands
implements namespaces?  AFAIU, namespaces means that you restrict the
scope of identifiers (or symbols) so that identically named
identifiers in different scopes don't conflict.  But shorthands does
not do that, and would therefore be better described precisely as a
work-around for the lack of namespaces.  Or am I missing something?

But then what are namespaces ? (was: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)))

[João Távora]

On Sat, Oct 2, 2021 at 4:22 PM Stefan Kangas <stefan@marxist.se> wrote:

> This might be a tangent,

Yup, but a good question. Changing the subject again.

> but can we really say that shorthands
> implements namespaces?  AFAIU, namespaces means that you restrict the
> scope of identifiers (or symbols) so that identically named
> identifiers in different scopes don't conflict.  But shorthands does
> not do that, and would therefore be better described precisely as a
> work-around for the lack of namespaces.  Or am I missing something?

There is no canonical definition, as far as I know.

For me "namespaces" are about allowing the same thing to be invoked
by different names, depending on context.  Take names, names of humans.
There is your family namespace, the official state namespace, your
gamer/hacker/hobbyist namespace, your high-school nickname namespace.
Depending on context, you are known by different names there but you are
always you.  Your name is a good example. "Stefan" in the context of
this thread,
means you. In emacs-devel it's not enough, we need your surname.

This is the invariant I find in programming language namespaces, of my
brief investigation.  The Emacs wiki page on namespaces has good humour
about this at the top.

João

Re: But then what are namespaces ? (was: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)))

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 1327 bytes --]


Stefan K:

>
> can we really say that shorthands implements namespaces?  AFAIU, 
> namespaces means that you restrict the scope of identifiers (or symbols) 
> so that identically named identifiers in different scopes don't 
> conflict.  But shorthands does not do that, and would therefore be 
> better described precisely as a work-around for the lack of namespaces.
>

Indeed.  Namespaces partition the set of identifiers in such a way that 
the same identifier can be bound in different places to different 
meanings, and that you can refer to each of the different meanings of an 
identifier either by using their namespace prefix(es), or by importing one 
of the meanings of an identifier ("using" in C++, "import" in Java).

Elisp has a single namespace, and this does not change with shorthands. 
So indeed, as you write, shorthands are better characterized as a 
workaround for the lack of namespaces.  Nonetheless, they provide one of 
the features of namespaces, namely that you can import subsets of the 
possible identifiers of the single namespace and access them through a 
shorter name.

João:

>
> For me "namespaces" are about allowing the same thing to be invoked by 
> different names, depending on context.
>

This is not at all what namespaces are about.  This is aliasing.

Re: But then what are namespaces ?

[Stefan Monnier]

> Elisp has a single namespace, and this does not change with shorthands.

Many/most languages with namespaces map "local names" to a unique global
namespace (e.g using hierarchies like `org.gnu.foo.bar` to avoid conflicts).
So this new mechanism is not that different.


        Stefan

Re: But then what are namespaces ?

[Gregory Heytings]

>> Elisp has a single namespace, and this does not change with shorthands.
>
> Many/most languages with namespaces map "local names" to a unique global 
> namespace (e.g using hierarchies like `org.gnu.foo.bar` to avoid 
> conflicts).
>

Yes, perhaps what I said wasn't clear enough: all languages have a single 
global namespace, and languages with namespaces have a single global 
namespace that is partitioned.  (I'm not sure what you mean by 
"many/most", I at least don't know languages with namespaces that use a 
different strategy.)

>
> So this new mechanism is not that different.
>

It depends how you define "different" ;-)  It is not entirely different 
indeed, from a user viewpoint it provides a similar feature, but it is 
different: in languages with namespaces, identifiers are bound in a 
certain subset of the global namespace, whereas this mechanism only allows 
you to locally expand an identifier prefix to another identifier prefix. 
It's more a preprocessor-like workaround, a poor man's implementation of 
namespaces if you want.

Re: But then what are namespaces ?

[Stefan Monnier]

> (I'm not sure what you mean by "many/most", I at least don't know
> languages with namespaces that use a different strategy.)

Some languages take extra steps to try and avoid relying on a unique
global namespace, and rely instead exclusively on relative names.

See https://dl.acm.org/doi/abs/10.1145/325478.325518 for instance.
Whether that can be considered as having a unique global namespace or
not is somewhat philosophical, admittedly (and the abstract itself
suggests the authors aren't sure either ;-)


        Stefan

Re: But then what are namespaces ?

[João Távora]

On Sat, Oct 2, 2021 at 10:05 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> > (I'm not sure what you mean by "many/most", I at least don't know
> > languages with namespaces that use a different strategy.)
>
> Some languages take extra steps to try and avoid relying on a unique
> global namespace, and rely instead exclusively on relative names.
>
> See https://dl.acm.org/doi/abs/10.1145/325478.325518 for instance.
> Whether that can be considered as having a unique global namespace or
> not is somewhat philosophical, admittedly (and the abstract itself
> suggests the authors aren't sure either ;-)

Didn't you, Stefan Monnier of the University of Montreal (hehe), author
a paper for the ELS where you discuss Elisp and namespaces?  I think
I saw it once, but can't find the link, am I dreaming?

João

Re: But then what are namespaces ?

[Stefan Monnier]

> Didn't you, Stefan Monnier of the University of Montreal (hehe), author
> a paper for the ELS where you discuss Elisp and namespaces?  I think
> I saw it once, but can't find the link, am I dreaming?

It must have been in a dream, I'm afraid.  I only once gave a talk at
ELS, and that was an invited talk about ELisp in general.
I can't remember the details, but I likely mentioned something about
namespace during the talk at some point, but probably more along the
lines of "things that still didn't happen".

The only time I submitted a paper to ELS (so far) it was not about Emacs
(and it was rejected ;-).


        Stefan

Re: But then what are namespaces ?

[João Távora]

On Sat, Oct 2, 2021 at 10:14 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> > Didn't you, Stefan Monnier of the University of Montreal (hehe), author
> > a paper for the ELS where you discuss Elisp and namespaces?  I think
> > I saw it once, but can't find the link, am I dreaming?
>
> It must have been in a dream, I'm afraid.  I only once gave a talk at
> ELS, and that was an invited talk about ELisp in general.
> I can't remember the details, but I likely mentioned something about
> namespace during the talk at some point, but probably more along the
> lines of "things that still didn't happen".

Ah! It wasn't a dream, but it wasn't for ELS either. It's in your
personal webpage:

https://www.iro.umontreal.ca/~monnier/hopl-4-emacs-lisp.pdf

"While Emacs Lisp was designed from the beginning as a real
programming language rather than a
tiny ad-hoc extension language, it was not designed for łprogramming
in the largež as witnessed
by the lack of module or namespace system.
Yet, the Emacs Lisp side of Emacs is now a rather large system, so in
order to avoid name conflicts,
Emacs Lisp relies on a poor man’s namespace system,"

Section 8.11

> The only time I submitted a paper to ELS (so far) it was not about Emacs
> (and it was rejected ;-).

The world isn't ready for your genius,
Stefan.

Re: But then what are namespaces ?

[João Távora]

On Sat, Oct 2, 2021 at 11:41 PM João Távora <joaotavora@gmail.com> wrote:

> > namespace during the talk at some point, but probably more along the
> > lines of "things that still didn't happen".
>
> Ah! It wasn't a dream, but it wasn't for ELS either. It's in your
> personal webpage:
>
> https://www.iro.umontreal.ca/~monnier/hopl-4-emacs-lisp.pdf

> Section 8.11

I'm glad I found it,  the section goes to succinctly list many of the arguments
here and lists links to many macro-driven approaches, that surely
"break grep  and -- contrary to shorthands -- also break M-., ElDoc, completion,
font-lock, etc and many other things. So maybe we should outlaw macros
as they are clearly making all this evil possible.

João

Re: But then what are namespaces ?

[Stefan Monnier]

> Ah! It wasn't a dream, but it wasn't for ELS either. It's in your
> personal webpage:
>
> https://www.iro.umontreal.ca/~monnier/hopl-4-emacs-lisp.pdf

[ That was a temporary URL for the draft.
  The better URL is: https://dl.acm.org/doi/abs/10.1145/3386324  ]

Ah, right.  Well, we only mention existing attempts, rather than present
our own design, but yes, we do discuss namespaces, indeed (a bit like
lexical scoping, it's been a recurring topic for many years now, so it
had to appear in that HOPL paper).


        Stefan

Re: But then what are namespaces ?

[Gregory Heytings]

>> (I'm not sure what you mean by "many/most", I at least don't know 
>> languages with namespaces that use a different strategy.)
>
> Some languages take extra steps to try and avoid relying on a unique 
> global namespace, and rely instead exclusively on relative names.
>
> See https://dl.acm.org/doi/abs/10.1145/325478.325518 for instance. 
> Whether that can be considered as having a unique global namespace or 
> not is somewhat philosophical, admittedly (and the abstract itself 
> suggests the authors aren't sure either ;-)
>

Thanks for the pointer!  It'll take me some time to digest that rather 
long paper.  From what I've understood after glancing through it, what 
they discuss is more an make-like utility that automatically renames the 
identifiers exported by individual modules to avoid name clashes than a 
new namespace strategy at the programming language level.

Re: But then what are namespaces ? (was: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)))

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 660 bytes --]

On Sat, Oct 2, 2021, 20:42 Gregory Heytings <gregory@heytings.org> wrote:

> > For me "namespaces" are about allowing the same thing to be invoked by
> > different names, depending on context.
>
> This is not at all what namespaces are about.  This is aliasing.


Says you, right? Or is the definition of namespace rigidly set down
somewhere?  The Wikipedia definition, FWIW is unavoidably vague and
generic, includes file systems, etc. It does rhyme with my ideas of
"context" and even uses the example of human names.

But for me, you can call this "aliasing", "alberto" or whatever you want.
In the Gregory namespace, of course. ;-)

João

[-- Attachment #2: Type: text/html, Size: 1129 bytes --]

Re: But then what are namespaces ? (was: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)))

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 559 bytes --]


>>> For me "namespaces" are about allowing the same thing to be invoked by 
>>> different names, depending on context.
>> 
>> This is not at all what namespaces are about.  This is aliasing.
> 
> Says you, right? Or is the definition of namespace rigidly set down 
> somewhere?
>

You are free to use whatever definition you want, but aliasing is the 
exact opposite of namespaces: namespaces allow you to use the same (short) 
identifier to denote different objects, aliasing allows you to use 
different identifiers to denote the same object.

Re: But then what are namespaces ? (was: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)))

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 764 bytes --]

As you wish. Beware that "aliasing" means something completely different in
Emacs and symbols.

João

On Sat, Oct 2, 2021, 21:42 Gregory Heytings <gregory@heytings.org> wrote:

>
> >>> For me "namespaces" are about allowing the same thing to be invoked by
> >>> different names, depending on context.
> >>
> >> This is not at all what namespaces are about.  This is aliasing.
> >
> > Says you, right? Or is the definition of namespace rigidly set down
> > somewhere?
> >
>
> You are free to use whatever definition you want, but aliasing is the
> exact opposite of namespaces: namespaces allow you to use the same (short)
> identifier to denote different objects, aliasing allows you to use
> different identifiers to denote the same object.

[-- Attachment #2: Type: text/html, Size: 1159 bytes --]

RE: [External] : Re: But then what are namespaces ? (was: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)))

[Drew Adams]

FWIW (OT) -

There are statements in this thread such as these:

> Elisp has a single namespace

> all languages have a single global namespace,
  and languages with namespaces have a single
  global namespace that is partitioned

> a unique global namespace

This is off-topic, but I just wanted to remind
us that Elisp does have multiple "namespaces",
in a particular sense.

I won't argue that obarrays are namespaces in
the usual senses we think of - import/export
combination etc.  But in one sense they really
do provide for multiple sets/spaces of names.

Those sets are completely separate.  They have
nothing to do with short/relative/unqualified
names vs long/absolute/qualified names.  But
they can be quite handy for some purposes.

Emacs abbrevs use obarrays.  My little library
`synonyms.el' uses an obarray for the words in
the thesaurus.  And `completing-read' can use
obarrays.

https://www.emacswiki.org/emacs/Synonyms

(Now back to your regularly scheduled broadcast...)

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[João Távora]

On Sat, Oct 2, 2021 at 4:03 PM Gregory Heytings <gregory@heytings.org> wrote:

> > Then WTFG, watch the fine gif? Can you find out in the top message? Hope
> > you can. The answer is there in front of your eyes. Watch it well. This
> > constant email flood a true DOS so I'm sorry I can't be your personal
> > tutor on the matter here.

> I'm certainly not asking for a tutor, no, and no, the answer is not "in
> front of my eyes".  Because your "fine gif" doesn't show what
> "magnars-string" is.

It does.  Watch it closely as I said.  At a certain point, I M-. into it and
back with M-,

And I've pointed you to the commit that has that file.

João

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Gregory Heytings]

>> I'm certainly not asking for a tutor, no, and no, the answer is not "in 
>> front of my eyes".  Because your "fine gif" doesn't show what 
>> "magnars-string" is.
>
> It does.  Watch it closely as I said.  At a certain point, I M-. into it 
> and back with M-, And I've pointed you to the commit that has that file.
>

Our mails crossed each other.  And the commit to which you refer contains 
s.el file, modified exactly as I described in the next sentence.  So it 
would have been more helpful to reply to the actual points of my post.

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Gregory Heytings]

>
> Grep is a tool, a tool to search text.  It is only able to respond to 
> questions about a program's source when that source is written in text 
> files (which it normally is) and when the programming language is 
> relatively poor on indirection.
>

The objections raised in this discussion are not only about grep.  They 
are also about tag systems (that can do more complex operations than grep 
on source files), and about the problem of code readability for Elisp 
programmers.  All of a sudden symbols can silently be transformed by 
'read' without any clear indication that such a transformation takes 
place.

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[João Távora]

On Fri, Oct 1, 2021 at 11:58 PM Gregory Heytings <gregory@heytings.org> wrote:

> The objections raised in this discussion are not only about grep.

But in this thread, we're discussing a grep-like, so please don't derail it
with the same points rebutted over and over.

> They
> are also about tag systems (that can do more complex operations than grep
> on source files), and about the problem of code readability for Elisp
> programmers.  All of a sudden symbols can silently be transformed by
> 'read' without any clear indication that such a transformation takes
> place.

I'm getting a bit tired of stressing that things with no "clear indication" are
what higher level-languages are about: anonymous functions, macros,
etc abstraction in general. Namespaces are also that: you may not like them
but they exist everywhere else for a reason.

But for shorthands, it's probably very easy to highlight them with font-lock.
Does this count as "clear indication"? What about ElDoc showing you the
real name of the symbol in the echo area? Anyway, if you want to propose
a patch for font-lock in Elisp, I think it will be accepted.

João

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Gregory Heytings]

>> The objections raised in this discussion are not only about grep.
>
> But in this thread, we're discussing a grep-like, so please don't derail 
> it with the same points rebutted over and over.
>

You started a subthread to discuss possible solutions from the point of 
view of grep, which is fine, but it is IMO important to not forget that 
the fact that shorthands break grep is not the only problem, and perhaps 
even not the most important one.

>> They are also about tag systems (that can do more complex operations 
>> than grep on source files), and about the problem of code readability 
>> for Elisp programmers.  All of a sudden symbols can silently be 
>> transformed by 'read' without any clear indication that such a 
>> transformation takes place.
>
> I'm getting a bit tired of stressing that things with no "clear 
> indication" are what higher level-languages are about: anonymous 
> functions, macros, etc abstraction in general. Namespaces are also that: 
> you may not like them but they exist everywhere else for a reason.
>

AFAIK, no other language use a shorthand-like approach to implement 
namespaces.  That's perhaps not without a reason.

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 472 bytes --]

On Sat, Oct 2, 2021, 10:04 Gregory Heytings <gregory@heytings.org> wrote:

>
> AFAIK, no other language use a shorthand-like approach to implement
> namespaces.  That's perhaps not without a reason.
>

All of the catastrophical grep problems you and others insist on so much
apply equally to other languages' systems. That's just a fact, so this
particular argument of singling out shorthands comparatively is probably
the most absurd one so far.

João

>

[-- Attachment #2: Type: text/html, Size: 1007 bytes --]

Re: A read-based grep-like for symbols (el-search?) (was Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master))

[Michael Heerdegen]

<tomas@tuxteam.de> writes:

> Now the question: does it also find things hidden away in comments?

[Did you ask me?]

For expression based search this is actually not trivial: commented code
might be messed with human language, or discontinued (code in between)
or otherwise broken or incomplete.

For el-search, there is some support for processing docstrings, and I
plan to do something useful for comments too, but ATM comments are
whitespace for el-search.  OTOH, el-search supports expression-based
query-replace, so it can automate some otherwise very time-consuming
tasks... so, while it can be really useful, it is not /the/ one-and-only
tool.

Michael.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Tomas Hlavaty]

On Thu 30 Sep 2021 at 13:31, André A. Gomes <andremegafone@gmail.com> wrote:
> Gregory Heytings <gregory@heytings.org> writes:
>> A simple example: suppose you want to check which ELPA package
>> activates tab-bar-mode.  That's easy to do with "grep -R tab-bar-mode"
>> in a clone of the ELPA repository.  With symbol prefix renaming, a
>> package author might decide to add ("tb-" . "tab-bar-") in the
>> shorthands of the package, and "grep -R tab-bar-mode" will not show
>> anything.  Likewise for tag systems, the symbols that are recorded
>> will possibly be different in each package, and a search for
>> tab-bar-mode will not return occurrences of tb-mode.
>
> I don't think this is a problem.

It is a problem.  It is a shame that it will break grep and search in
general (e.g. web), due to names not being unique anymore.

> Grep comes the world of Unix and its mantras.  But Lisp REPLs come
> from another world.

That is wrong.  I use Common Lisp and still need grep.

> Using grep and tag systems to reason about a Lisp program is like eating
> soup with a fork.  You can do it, but it's the wrong tool.

Some eat soup with sticks.  Sure it can be the wrong tool for those
refusing to learn to work with that.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Eli Zaretskii]

> Date: Tue, 28 Sep 2021 13:38:09 +1300
> From: Phil Sainty <psainty@orcon.net.nz>
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
> This has probably been covered in earlier discussions (apologies for
> not being across those), but...
> 
> Won't this break a ton of basic tooling for locating things, if the
> symbol in the file is not the actual symbol?

Those tools will need to be enhanced to support shorthands, yes.

And this problem is not new, we had it for a while with other Lisp
constructs where generated symbols aren't literally present in the
sources.  For example, try M-. on a call to a constructor defined by
cl-defstruct.  Here's one such use case:

  emacs -Q
  M-x load-file RET lisp/profiler.el RET
  C-x C-f lisp/profiler.el RET
  C-u 10006 M-g c  ;; this puts point on a call to profiler-make-calltree
  M-.

The result is an error message.  This happens to me quite a lot
recently, as more and more code is converted to using cl-defstruct.

> Simplicity can be a *very* good thing, and knowing that what you see
> is in fact what you get is a benefit which shouldn't be undervalued,
> IMO.

Then I guess you dislike cl-defstruct, add-function, pcase, and other
macros and features that change how the source code looks and produce
symbols under the hood?

> Is whatever we're gaining actually worth the resulting obfuscation?

Time will tell.  It currently sounds like its worth it, but as with
any such feature, we could be wrong.

> Long names being "tedious" (quoting the new info manual) to read
> and write seems like an insufficient reason, IMHO.

They are not the real reason, they are just the way to explain the
feature in simple terms.  The real reason is to make namespace
management easier.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

> Those tools will need to be enhanced to support shorthands, yes.

One way grep could start working more to our liking is if one is given
the ability to grep by either the prefix or the name.  Maybe
(thing-at-point 'symbol-component) could be added.  Unfortunately, it is
not always so, even before Shorthands.

CL packages are more structured in this regard: there's the package and
the symbol name.  The family and the identity are not "weakly" mangled
in the symbol name.

> Then I guess you dislike cl-defstruct, add-function, pcase, and other
> macros and features that change how the source code looks and produce
> symbols under the hood?

Those are so-called anaphoric macros.  For the record, the Common Lisp
implementations that I used deal fine with them, because they index the
macroexpanded code in a database after reading the original code with a
"source tracking Lisp reader".  I.e. they are not 'coupled' to the
non-macroexpanded text.  Then, in tools like SLIME or SLY there are
tricks to know where that text has drifted in the buffer.

I don't know what kind of databse is used by Xref, which is relatively
recent, but I suspect it's a more naive database.

Nevertheless, I'd like to note that Shorthands work well with Xref's
main workhorse (M-.) probably because it uses the proper API for
symbols, obarray.  They also work well with Eldoc, C-h f and in-buffer
Completions, to the best of my knowledge.

>> Long names being "tedious" (quoting the new info manual) to read
>> and write seems like an insufficient reason, IMHO.
> They are not the real reason, they are just the way to explain the
> feature in simple terms.

For me, it's a very real reason!  I added that there :-) It's tedious to
type _and_ read.

> The real reason is to make namespace management easier.

That is also another reason I concur with (and this is why it is also
written in the manual).

João

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: Phil Sainty <psainty@orcon.net.nz>,  emacs-devel@gnu.org
> Date: Tue, 28 Sep 2021 08:41:29 +0100
> 
> I don't know what kind of databse is used by Xref, which is relatively
> recent, but I suspect it's a more naive database.

AFAIK, load-history.  See elisp-mode.el for the details.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Helmut Eller]

On Tue, Sep 28 2021, João Távora wrote:

> Nevertheless, I'd like to note that Shorthands work well with Xref's
> main workhorse (M-.) probably because it uses the proper API for
> symbols, obarray.  They also work well with Eldoc, C-h f and in-buffer
> Completions, to the best of my knowledge.

Do Shorthands also work with font-lock?  E.g. are renamed macro names
highlighed?

Helmut

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

On Tue, Sep 28, 2021 at 9:10 AM Helmut Eller <eller.helmut@gmail.com> wrote:
> On Tue, Sep 28 2021, João Távora wrote:
> > Nevertheless, I'd like to note that Shorthands work well with Xref's
> > main workhorse (M-.) probably because it uses the proper API for
> > symbols, obarray.  They also work well with Eldoc, C-h f and in-buffer
> > Completions, to the best of my knowledge.
>
> Do Shorthands also work with font-lock?  E.g. are renamed macro names
> highlighed?

I've now confirmed that this works as expected. Didn't have to touch
any font-lock code.  Both the shorthand form and the longhand form
are highlighted.  If you meant some other situation, please clarify.

João

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Stefan Monnier]

>   emacs -Q
>   M-x load-file RET lisp/profiler.el RET
>   C-x C-f lisp/profiler.el RET
>   C-u 10006 M-g c  ;; this puts point on a call to profiler-make-calltree
>   M-.
>
> The result is an error message.

This is a "loading order" bug: if you somehow load `cl-extra` (which
tends to happen more often than I expected, so the bug "disappeared"
when I tried to dig into it) you won't get the error any more.

IOW the fix is this chunk of code in `cl-extra.el`:

    (with-eval-after-load 'find-func
      (defvar find-function-regexp-alist)
      (add-to-list 'find-function-regexp-alist
                   '(define-type . cl--typedef-regexp)))

we should probably move it to something like `cl-preloaded.el`.

> This happens to me quite a lot recently,

The above is a plain bug.

> as more and more code is converted to using cl-defstruct.

In the above example, the problem is fairly minor (I mean, after fixing
the bug), but it gets worse in two cases:
- when the `cl-defstruct` has several longish constructors:
  `M-.` jumps to the `cl-defstruct` but not to the actual place where
  `profiler-make-calltree` is defined within the `cl-defstruct`.
- when jumping to an accessor (rather than a constructor) because the
  accessor's name is not present literally in the `cl-defstruct`
  so you jumped from `foo-bar` and you end up on a `cl-defstruct` for
  `foo` and need to know that the `bar` symbol you might find a few
  lines below is the one that caused the definition of `foo-bar`.

It shouldn't be very hard [IOW, I encourage some of the readers here to
go out and do it] to refine/extend the `find-function-regexp-alist`
mechanism so that `M-.` jumps directly to the actual `:constructor`
thingy or to the actual field corresponding to an accessor.

> Then I guess you dislike cl-defstruct, add-function, pcase, and other
> macros and features that change how the source code looks and produce
> symbols under the hood?

Neither `pcase` nor `add-function` generate symbols under the hood
(other than gensym'd ones, admittedly, but these aren't relevant in
this discussion).  `cl-defstruct` does, tho only for field accessors and
not for the constructor you showed in your example.

>> Long names being "tedious" (quoting the new info manual) to read
>> and write seems like an insufficient reason, IMHO.

As a researcher in programming languages, I tend to take it for granted
that "syntax doesn't matter", but in reality details of syntax have
enormous impacts on how languages are used and perceived, so I really
wouldn't dismiss "tedious" as an insufficient reason.


        Stefan

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> It shouldn't be very hard [IOW, I encourage some of the readers here to
> go out and do it] to refine/extend the `find-function-regexp-alist`
> mechanism so that `M-.` jumps directly to the actual `:constructor`
> thingy or to the actual field corresponding to an accessor.

Regexps.... Ugh.  Shouldn't we invest in a proper macro-expanding and
source-tracking reader?  I've recently some work for a Common Lisp
annotation-based stepper that correlates each evaluated form with a
source position.  I built a working system and published a paper.  Is
there interest in such a deeper Lisp introspection features for Emacs?

>>> Long names being "tedious" (quoting the new info manual) to read
>>> and write seems like an insufficient reason, IMHO.
> As a researcher in programming languages, I tend to take it for granted
> that "syntax doesn't matter", but in reality details of syntax have
> enormous impacts on how languages are used and perceived,

Yes, when you're transferring programs between two well-behaved robotic
entities, syntax doesn't matter.  But when human's squishy matter is
interacting with them (and we do that a lot, unfortunately), it matters.


João         "You think you have a problem, you use a regexp...."  :-)

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Stefan Monnier]

João Távora [2021-09-28 16:28:13] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> It shouldn't be very hard [IOW, I encourage some of the readers here to
>> go out and do it] to refine/extend the `find-function-regexp-alist`
>> mechanism so that `M-.` jumps directly to the actual `:constructor`
>> thingy or to the actual field corresponding to an accessor.
> Regexps.... Ugh.

Yup.  FWIW, my suggestion above was meant to suggest to extend it so we
can also have functions rather than regexps and let the function search
for the field/constructor (but only after finding the `cl-defstruct`
via regexp search ;-).

> Shouldn't we invest in a proper macro-expanding and
> source-tracking reader?

It's been proposed but the only attempt at such a thing has been Alan's
which had some serious drawbacks (making symbol equality more complex
and costly).

> I've recently some work for a Common Lisp annotation-based stepper
> that correlates each evaluated form with a source position.  I built
> a working system and published a paper.  Is there interest in such
> a deeper Lisp introspection features for Emacs?

Interest, yes.  Do you have a URL for your paper?

>>>> Long names being "tedious" (quoting the new info manual) to read
>>>> and write seems like an insufficient reason, IMHO.
>> As a researcher in programming languages, I tend to take it for granted
>> that "syntax doesn't matter", but in reality details of syntax have
>> enormous impacts on how languages are used and perceived,
> Yes, when you're transferring programs between two well-behaved robotic
> entities, syntax doesn't matter.

For us PL people the reason why it doesn't matter is because we only
look at the typing rules and dynamic semantics.
We care more about how the grammar is written than about how programs
are written.


        Stefan

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Alan Mackenzie]

Hello, Eli.

On Tue, Sep 28, 2021 at 09:25:48 +0300, Eli Zaretskii wrote:
> > Date: Tue, 28 Sep 2021 13:38:09 +1300
> > From: Phil Sainty <psainty@orcon.net.nz>
> > Cc: emacs-devel <emacs-devel@gnu.org>

> > This has probably been covered in earlier discussions (apologies for
> > not being across those), but...

> > Won't this break a ton of basic tooling for locating things, if the
> > symbol in the file is not the actual symbol?

> Those tools will need to be enhanced to support shorthands, yes.

I thought that large features weren't being accepted for Emacs 28 any
more?  These "shorthands" are a gigantic feature which disrupt our way
of developing Emacs.

Can we please delay the release of Emacs 28.1 until we have these tool
enhancements in place?  And until that point, have a moratorium on using
shorthands?

I only found out about this feature today, since there has been nothing
in the Subject: lines of the discussions indicating the breakage that is
occurring.

> And this problem is not new, we had it for a while with other Lisp
> constructs where generated symbols aren't literally present in the
> sources.  For example, try M-. on a call to a constructor defined by
> cl-defstruct.

Surely that is no reason to make the situation worse?

> Here's one such use case:

>   emacs -Q
>   M-x load-file RET lisp/profiler.el RET
>   C-x C-f lisp/profiler.el RET
>   C-u 10006 M-g c  ;; this puts point on a call to profiler-make-calltree
>   M-.

> The result is an error message.  This happens to me quite a lot
> recently, as more and more code is converted to using cl-defstruct.

Recently, I made changes in the jit lock area.  An essential part of
that was to find all uses of the symbol jit-lock-functions.  This was
straightforward:

    $ find . -name '*.el' | xargs grep -n jit-lock-functions

This returned for me _ALL_ uses of jit-lock-functions in Emacs (plus
quite a few comments, too).  I have done this time and time again with
various symbols over the years, and surely you have, too.

As shorthands spread, this won't work any more.  If we must proceed with
this facility, can we please delay it until there's some suitable
replacement for such a use of find and grep and other similar things?

> > Simplicity can be a *very* good thing, and knowing that what you see
> > is in fact what you get is a benefit which shouldn't be undervalued,
> > IMO.

> Then I guess you dislike cl-defstruct, add-function, pcase, and other
> macros and features that change how the source code looks and produce
> symbols under the hood?

I strongly dislike cl-defstruct and add-function (as, I think, you do),
and also, for different reasons, pcase.  Macros which produce symbols
are OK, as long as these symbols, with the same meaning, don't appear in
source code.  Or sometimes even if they do.  I think we would largely
agree on when a macro is objectionable, even if that is hard to pin down
exactly.

> > Is whatever we're gaining actually worth the resulting obfuscation?

> Time will tell.  It currently sounds like its worth it, but as with
> any such feature, we could be wrong.

And if we are wrong, what then?  This feels like an irreversible change.
Why are we not trying it out in a development branch rather than (what
is shortly to become) the release branch?

> > Long names being "tedious" (quoting the new info manual) to read
> > and write seems like an insufficient reason, IMHO.

> They are not the real reason, they are just the way to explain the
> feature in simple terms.  The real reason is to make namespace
> management easier.

I don't think, on balance, it will do this.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Eli Zaretskii]

> Date: Tue, 28 Sep 2021 17:25:56 +0000
> Cc: Phil Sainty <psainty@orcon.net.nz>, joaotavora@gmail.com,
>   emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> I thought that large features weren't being accepted for Emacs 28 any
> more?  These "shorthands" are a gigantic feature which disrupt our way
> of developing Emacs.

The shorthands don't disrupt anything unless they are used.  And using
them is completely opt-in, and intended for specific situations where
it is justified.  Of course, any feature can be abused, but the blame
is on those who abuse it.

> Can we please delay the release of Emacs 28.1 until we have these tool
> enhancements in place?

I see no reason for such a delay, given that our tools are already
imperfect.  We should improve our tools, of course, but there's
nothing in shorthands that justifies delaying Emacs 28.1.

> And until that point, have a moratorium on using shorthands?

I'm not aware of any plans to use shorthands in Emacs itself.  People
talk and discuss these possibilities, and that's okay.  But that's
just talk at this point, certainly for Emacs 28.

> > > Is whatever we're gaining actually worth the resulting obfuscation?
> 
> > Time will tell.  It currently sounds like its worth it, but as with
> > any such feature, we could be wrong.
> 
> And if we are wrong, what then?

Then we will avoid using it, or maybe even recommend that no one does.
And perhaps replace shorthands with something better.  But we aren't
there anymore, and I think your sense of a catastrophe is unjustified,
if not exaggerated.

> > They are not the real reason, they are just the way to explain the
> > feature in simple terms.  The real reason is to make namespace
> > management easier.
> 
> I don't think, on balance, it will do this.

Time will tell.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Alan Mackenzie]

Hello, Eli.

On Tue, Sep 28, 2021 at 21:25:41 +0300, Eli Zaretskii wrote:
> > Date: Tue, 28 Sep 2021 17:25:56 +0000
> > Cc: Phil Sainty <psainty@orcon.net.nz>, joaotavora@gmail.com,
> >   emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > I thought that large features weren't being accepted for Emacs 28 any
> > more?  These "shorthands" are a gigantic feature which disrupt our way
> > of developing Emacs.

> The shorthands don't disrupt anything unless they are used.  And using
> them is completely opt-in, and intended for specific situations where
> it is justified.

I haven't opted in.  How do I opt out of somebody else's use of these?
I.e. so that grep still works for me?

> Of course, any feature can be abused, but the blame is on those who
> abuse it.

And it is you and I who will suffer, through not being able to use grep
reliably on the Emacs sources.

> > Can we please delay the release of Emacs 28.1 until we have these tool
> > enhancements in place?

> I see no reason for such a delay, given that our tools are already
> imperfect.  We should improve our tools, of course, but there's
> nothing in shorthands that justifies delaying Emacs 28.1.

> > And until that point, have a moratorium on using shorthands?

> I'm not aware of any plans to use shorthands in Emacs itself.  People
> talk and discuss these possibilities, and that's okay.  But that's
> just talk at this point, certainly for Emacs 28.

Even if it's just talk, how will we know that it's just talk?  And how
long will it stay just talk?  Clearly there's intent to use this,
otherwise nobody would have bothered implementing it.

> > > > Is whatever we're gaining actually worth the resulting obfuscation?

> > > Time will tell.  It currently sounds like its worth it, but as with
> > > any such feature, we could be wrong.

> > And if we are wrong, what then?

> Then we will avoid using it, or maybe even recommend that no one does.
> And perhaps replace shorthands with something better.  But we aren't
> there anymore, and I think your sense of a catastrophe is unjustified,
> if not exaggerated.

OK, how do you suggest I find all occurrences of jit-lock-functions in
the Emacs Lisp sources after shorthands start being used?  How do I find
occurrences of a symbol in Emacs Lisp sources on the web, which
currently a web search will find?

> > > They are not the real reason, they are just the way to explain the
> > > feature in simple terms.  The real reason is to make namespace
> > > management easier.

> > I don't think, on balance, it will do this.

> Time will tell.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Eli Zaretskii]

> Date: Tue, 28 Sep 2021 19:05:50 +0000
> Cc: psainty@orcon.net.nz, joaotavora@gmail.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > The shorthands don't disrupt anything unless they are used.  And using
> > them is completely opt-in, and intended for specific situations where
> > it is justified.
> 
> I haven't opted in.  How do I opt out of somebody else's use of these?

What use?  No one used it yet in the Emacs sources.

> > I'm not aware of any plans to use shorthands in Emacs itself.  People
> > talk and discuss these possibilities, and that's okay.  But that's
> > just talk at this point, certainly for Emacs 28.
> 
> Even if it's just talk, how will we know that it's just talk?  And how
> long will it stay just talk?  Clearly there's intent to use this,
> otherwise nobody would have bothered implementing it.

No, the intent is different.  And we will know because enough eyes
watch the commits that go in.

> OK, how do you suggest I find all occurrences of jit-lock-functions in
> the Emacs Lisp sources after shorthands start being used?

The same as today.  With the same probability of success.  Grep cannot
guarantee 100% success, because it cannot catch symbols that are
generated at run time, and we already have such features in place.
E.g., search the Emacs tree for "(intern (format ", and you will see
how many of those are already here.

So please calm down, your emotions are misplaced.  There's no
catastrophe, certainly not as long as shorthands aren't used inside
Emacs.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Phil Sainty]

On 2021-09-29 08:29, Eli Zaretskii wrote:
> I'm not aware of any plans to use shorthands in Emacs itself.

No doubt; but that doesn't prevent people using them in
libraries which are subsequently proposed for contribution to
Emacs or ELPA.

Personally I think this feature is detrimental to the ecosystem
at large, but even for Emacs itself I think it represents a
problem waiting to happen in one form or another.


Having caught up with the "s.el" issue, I've come to realise
that there are two *radically* different ideas being discussed
here, and I'd love to see the two issues separated.


The first use-case is to do with the "s" library, and finding
a way to rename all of that code with a longer prefix without
requiring other libraries currently requiring "s" to change
much.  I'm still dubious that this is worth doing; but I must
acknowledge that if it *is* to be supported, then it does indeed
mean providing some way to translate "s-" symbols to their
longer names.

This use-case is an extremely limited one, targeted (AFAIU) at
an *extremely* small number of libraries; so the impact is quite
limited.

However I think some people may be expecting that this is also
the *primary* use-case and that the feature probably won't be
used much otherwise, and I'm predicting that this will not turn
out to be the case.


The second use-case, and the one I think will prove to be FAR
more common if this goes ahead, is this: Some people simply want
to read and write shorter symbol names in their code.

João has certainly confirmed that this is an intended personal
use, and I'm confident that many, many other authors will find
the notion of short symbols to be appealing, because long
symbols can be a surface-level irritant.

I suspect that people who are already used to programming in
object-oriented (especially) languages with all manner of
obfuscated and duplicate naming going on will commonly consider
that short names in elisp would be nicer, and will see this as
the officially-blessed way to write code with short names, and
very soon there will be a lot of elisp in the wild which is
(ab)using shorthands just so that the authors of that code can
read and write shorter names.

Alan's examples will probably be real -- people will put all
kinds of shorthand declarations into their third-party code,
including shorthands for core features, just because they can,
and because it's an officially-supported feature.


The first ("s.el") use-case would seem to require something like
the current shorthands feature.

The second absolutely does not, and I wish that shorthands would
not be used for that.

The goal of the second use-case is purely about the authoring
experience -- reading and writing short symbols -- and Emacs
could facilitate that *without* creating discrepancies between
the source code and the actual symbol names.

Instead, we would need two things:

1. A way of displaying long symbols in the desired short form,
    such that the buffer contains the actual symbol, but the
    user sees the short symbol (i.e. some kind of replacing
    display).

2. Something analogous to abbrev which recognises when someone
    starts typing a symbol with one of the configured short
    prefixes, and expands it to be the full name (but per (1)
    visually displayed as the short form that they typed).

This gives the author the things they wanted, without the need
for any read-time renaming at all.  The user gets what they
want, and no one else suffers any problems at all.

This feature would be opt-in for anyone who wished to use it,
and could then be controlled in much the same way as the
shorthand feature, with a variable mapping short prefixes to
the associated real prefixes.

The benefits of this approach over shorthand symbols are immense:

1. First and very much foremost it *doesn't break anything else*
    because the actual code contains the real symbol names.

2. Users who enable the feature can retrospectively apply it to
    their code without editing the existing code -- by adding the
    abbreviation spec, any existing code can be redisplayed using
    the new abbreviation without anything being edited.

3. Users could even change which abbreviation they personally
    use/prefer and revert the buffer, and the display would
    change to match.  People could use their own configurations
    which were different to the author's and no one is stepping
    on anyone else's toes.

4. Users could even configure these things globally if they
    wished to -- it's fine, because it doesn't affect the actual
    code, or what other people see.

5. If the abbreviated text is a display-only effect, copying and
    pasting for other people will produce the real working code
    with the actual symbols.

I see one negative: If lines lengths were based on the author's
short symbols, then the actual lines may be longer than 80 cols
even if the author sees them as <= 80.  This would only be an
issue for third-party code, though (I'm sure it's already the
case that someone contributing patches with excessively-long
lines would be asked to reformat that code).  In any case, by
comparison with breaking standard tools, I feel that a few
formatting issues should be fairly inconsequential.


I'm sure there would be fiddly elements in implementing what
I've described, but it's a problem with a confined scope;
whereas making every other tool and feature suddenly know how to
seamlessly cope with shorthand symbols (notwithstanding that I
don't think it's possible to even do so in many cases) is a
problem with a very wide scope and overall likely to be much
more difficult.


-Phil

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Gregory Heytings]

>
> Instead, we would need two things:
>
> 1. A way of displaying long symbols in the desired short form, such that 
> the buffer contains the actual symbol, but the user sees the short 
> symbol (i.e. some kind of replacing display).
>
> 2. Something analogous to abbrev which recognises when someone starts 
> typing a symbol with one of the configured short prefixes, and expands 
> it to be the full name (but per (1) visually displayed as the short form 
> that they typed).
>

Not that this already exists (and has existed for years), it's the 
"nameless.el" library.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Gregory Heytings]

>> Instead, we would need two things:
>> 
>> 1. A way of displaying long symbols in the desired short form, such 
>> that the buffer contains the actual symbol, but the user sees the short 
>> symbol (i.e. some kind of replacing display).
>> 
>> 2. Something analogous to abbrev which recognises when someone starts 
>> typing a symbol with one of the configured short prefixes, and expands 
>> it to be the full name (but per (1) visually displayed as the short 
>> form that they typed).
>
> Not that this already exists (and has existed for years), it's the 
> "nameless.el" library.
>

s/Not/Note/

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Joost Kremers]

On Fri, Oct 01 2021, Phil Sainty wrote:
> 1. A way of displaying long symbols in the desired short form,
>    such that the buffer contains the actual symbol, but the
>    user sees the short symbol (i.e. some kind of replacing
>    display).
[...]
> I see one negative: If lines lengths were based on the author's
> short symbols, then the actual lines may be longer than 80 cols
> even if the author sees them as <= 80. 

As Gregory points out, there is a package that does this already. It actually
has another issue, which is that indentation is sometimes wrong. If the second
argument of a function is on a separate line, it's indented to align with the
first argument. This position depends on the length of the prefix of the
function name:

```
(somelongprefix-do-something arg1
                             arg2)
```

The nameless package can align `arg2` so that it matches up with `arg1`, but
unlike the shortening of the prefix, that indentation is saved to the file.
Not dramatic, but it can occasionally be annoying.


-- 
Joost Kremers
Life has its moments

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Adam Porter]

Joost Kremers <joostkremers@fastmail.fm> writes:

> As Gregory points out, there is a package that does this already. It actually
> has another issue, which is that indentation is sometimes wrong. If the second
> argument of a function is on a separate line, it's indented to align with the
> first argument. This position depends on the length of the prefix of the
> function name:
>
> ```
> (somelongprefix-do-something arg1
>                              arg2)
> ```
>
> The nameless package can align `arg2` so that it matches up with `arg1`, but
> unlike the shortening of the prefix, that indentation is saved to the file.
> Not dramatic, but it can occasionally be annoying.

FWIW, this is a dramatically annoying problem that has prevented me from
using Nameless.  :)  Either the code is indented incorrectly while I'm
editing it with Nameless activated, or it's indented incorrectly in the
saved file.  And having incorrect indentation is more annoying than
longer symbols, for me.

Since I usually use aggressive-indent-mode for my Elisp projects, I once
tried to use hooks to automatically reindent a file with full symbol
names before saving, and restore the abbreviated view and indentation
afterward, but I couldn't make it work smoothly (IIRC, buffers always
indicated that they were modified after having saved them and
reactivated the mode, which was an even bigger problem for usability).

If this issue could be solved, it could be as Phil said: user-local
symbol abbreviations that are transparent to everything else.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Stefan Kangas]

Adam Porter <adam@alphapapa.net> writes:

> FWIW, this is a dramatically annoying problem that has prevented me from
> using Nameless.  :)  Either the code is indented incorrectly while I'm
> editing it with Nameless activated, or it's indented incorrectly in the
> saved file.  And having incorrect indentation is more annoying than
> longer symbols, for me.

This is an occasional annoyance with nameless, indeed.

But it doesn't strike me as one that fundamentally can't be solved.
nameless should "just" change the indentation as needed with some
clever use of overlays, or something.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Tomas Hlavaty]

For me it would be enough to have tab autocompletion when writing elisp
(similar to what slime does for common lisp).  Is there something like
that?

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

On Thu, Sep 30, 2021 at 1:52 PM Tomas Hlavaty <tom@logand.com> wrote:
>
> For me it would be enough to have tab autocompletion when writing elisp
> (similar to what slime does for common lisp).  Is there something like
> that?

If you're talking about "Shorthand-aware" tab autocompletion, yes.
It's C-M-i (also known as "completion-at-point").  If you'd like nicer
looking alternative that uses that same information, try Company
mode (search for "emacs company" in your favourite search engine).

João

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Tomas Hlavaty]

On Thu 30 Sep 2021 at 13:55, João Távora <joaotavora@gmail.com> wrote:
> It's C-M-i (also known as "completion-at-point").

Great, thank you!

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Lars Ingebrigtsen]

Phil Sainty <psainty@orcon.net.nz> writes:

> The first use-case is to do with the "s" library, and finding
> a way to rename all of that code with a longer prefix without
> requiring other libraries currently requiring "s" to change
> much.  I'm still dubious that this is worth doing; but I must
> acknowledge that if it *is* to be supported, then it does indeed
> mean providing some way to translate "s-" symbols to their
> longer names.

Yes, that's the implicit intended use case, but it's not spelled out
anywhere.

> The second use-case, and the one I think will prove to be FAR
> more common if this goes ahead, is this: Some people simply want
> to read and write shorter symbol names in their code.

Yes, if we had a more ergonomic syntax for this, then a large portion of
people would be writing

(require 'gnus-summary :as x)
(require 'xref :as gs)

etc etc, since this is what they do in many modern languages (and I'm
not much fan of it there, either).  But since the syntax is the really
awkward

;; Local Variables:
;; elisp-shorthands: (("t-" . "my-tricks-")
;;                    ("sns-" . "some-nice-string-utils-"))
;; End:

it'll hopefully be less appealing to people.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1019 bytes --]

On Thu, Sep 30, 2021 at 2:17 PM Lars Ingebrigtsen <larsi@gnus.org> wrote:

>
> Yes, if we had a more ergonomic syntax for this, then a large portion of
> people would be writing
>
> (require 'gnus-summary :as x)
> (require 'xref :as gs)
>
> etc etc, since this is what they do in many modern languages (and I'm
> not much fan of it there, either).  But since the syntax is the really
> awkward


I get your point ;-), but the syntax is not "awkward" (which it is, I admit)
for no reason.

Until you or someone can specify what happens above and below those
forms (in terms of buffer positions), or what happens if the require
statement
is in conditional, we can't "make" better syntax.  Thus we need file local
values
(which could be hoisted up to the starting comments in a Lisp file, I
think).

But better syntax is possible.  You're looking at what SLIME/SLY does with
Common Lisp's CL:IN-PACKAGE (which I believe you're familiar with, since
you've said you like CL's packages).

João

[-- Attachment #2: Type: text/html, Size: 1545 bytes --]

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Eli Zaretskii]

> Date: Fri, 01 Oct 2021 01:23:48 +1300
> From: Phil Sainty <psainty@orcon.net.nz>
> Cc: Alan Mackenzie <acm@muc.de>, joaotavora@gmail.com, emacs-devel@gnu.org
> 
> On 2021-09-29 08:29, Eli Zaretskii wrote:
> > I'm not aware of any plans to use shorthands in Emacs itself.
> 
> No doubt; but that doesn't prevent people using them in
> libraries which are subsequently proposed for contribution to
> Emacs or ELPA.

If we decide we don't want that except in the few cases which it's
supposed to solve, then we will ask the authors to kindly not use that
in the packages submitted to ELPA, certainly to the core.

> This use-case is an extremely limited one, targeted (AFAIU) at
> an *extremely* small number of libraries; so the impact is quite
> limited.

The problem is not with the number of libraries, the problem is with
the libraries that _use_ those few.  They are a lot.

> Alan's examples will probably be real -- people will put all
> kinds of shorthand declarations into their third-party code,
> including shorthands for core features, just because they can,
> and because it's an officially-supported feature.

If we decide we don't want that, we won't allow that.

> 1. A way of displaying long symbols in the desired short form,
>     such that the buffer contains the actual symbol, but the
>     user sees the short symbol (i.e. some kind of replacing
>     display).

I very much doubt that display-time feature can solve this, because it
will not be supported by Lisp itself.  And then you have worse
problems.

> 2. Something analogous to abbrev which recognises when someone
>     starts typing a symbol with one of the configured short
>     prefixes, and expands it to be the full name (but per (1)
>     visually displayed as the short form that they typed).

Are you thinking about M-/ ?  That already exists, and I'm using it
all the time, not only in Lisp.  Long identifiers are ubiquitous these
days, and long words in human-readable text are as well.

I really don't understand why you and Alan (and others) are so
worried.  Use of this feature is entirely optional, and if we are
afraid this will somehow slip into the parts of Emacs we don't want,
let's augment CONTRIBUTE to prevent that.  Me, I think it's no more
dangerous than someone trying to sneak files without lexical-binding
into Emacs: it will be immediately caught and flagged, because several
people watch the commits closely enough.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

On Thu, Sep 30, 2021 at 3:12 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > 2. Something analogous to abbrev which recognises when someone
> >     starts typing a symbol with one of the configured short
> >     prefixes, and expands it to be the full name (but per (1)
> >     visually displayed as the short form that they typed).
>
> Are you thinking about M-/ ?  That already exists, and I'm using it
> all the time, not only in Lisp.

Not only that, but I'd like to underline that C-M-i (completion-at-point)
will do a context-aware search for Elisp which considers _both_ shorthand-
and longhand- versions of possible symbols.  Nothing wrong with M-/ tho,
also use it lots.

João

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Phil Sainty]

On 2021-10-01 03:12, Eli Zaretskii wrote:
>> This use-case is an extremely limited one, targeted (AFAIU)
>> at an *extremely* small number of libraries; so the impact is
>> quite limited.
> 
> The problem is not with the number of libraries, the problem
> is with the libraries that _use_ those few.  They are a lot.

When I say "the impact is quite limited" I mean that the total
number of symbol names which are being obfuscated is not large
if we're only using this to support "s.el" (and maybe a tiny
number of similar cases), because the obfuscated symbols are
always the same limited set.

The *negative* impact of shorthands is limited iff the use of
shorthands is limited to targeting a small number of problem
libraries.


>> 1. A way of displaying long symbols in the desired short
>>     form, such that the buffer contains the actual symbol,
>>     but the user sees the short symbol (i.e. some kind of
>>     replacing display).
> 
> I very much doubt that display-time feature can solve this,
> because it will not be supported by Lisp itself.  And then you
> have worse problems.

We are now talking about the second use-case, which has nothing
to do with "s.el" (and which I'm proposing should have nothing
to do with 'shorthands').

In this scenario the files that lisp reads will always contain
the real symbols, so no extra lisp support is required.  The
lisp reader never sees the short names (and nor does any person
who has not opted in), because they don't actually exist in the
file.

This use case is purely about enabling the people who opt in to
type and see their short names when editing the source code, but
with the real/longer names being the actual text.


>> 2. Something analogous to abbrev which recognises when
>>     someone starts typing a symbol with one of the configured
>>     short prefixes, and expands it to be the full name (but
>>     per (1) visually displayed as the short form that they
>>     typed).
> 
> Are you thinking about M-/ ?  That already exists, and I'm
> using it all the time, not only in Lisp.  Long identifiers are
> ubiquitous these days, and long words in human-readable text
> are as well.

No, I'm talking about a feature (possibly already implemented by
the "nameless" library; I still need to experiment with this)
whereby people would be able to type their short symbols when
editing their source code and Emacs would recognise the
abbreviation and *automatically* (hence more like abbrev than
dabbrev) change it to the real name.


> I really don't understand why you and Alan (and others) are so
> worried.

I'd be less worried if the release which includes shorthands
(if it's retained) *also* includes "nameless" or something
similar in order to provide for the inevitable second use-case
in a way which doesn't break anything.

We might not allow any shorthand code in Emacs itself, but if
shorthands are understood to be the in-built solution for
reading and writing short names then all kinds of third-party
code is going to start doing exactly that.  A very large
proportion (I would assume a majority) of Emacs users will be
using third-party code in their configs, and hence many of
them/us will inevitably acquire shorthand code in our configs
even if it is not core Emacs code or "s.el"; and the more such
code that emerges, the more problems people will have.

This will affect people debugging their own configs; people
sharing code with other people; people submitting bug reports
or help requests here on the mailing lists and elsewhere.
Whether it's something they can't figure out, or a frustrating
stumbling block that they do figure out, I think it's going
to create a lot of unnecessary problems.

I'm saying that code which does not contain shorthand symbols
is *easier* (on average, for the Emacs user base as a whole)
to deal with than code which does contain shorthand symbols,
and so let's not bless shorthands (even tacitly) as the
standard solution for anything which can be done in other ways.


-Phil

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Stefan Kangas]

Phil Sainty <psainty@orcon.net.nz> writes:

> We are now talking about the second use-case, which has nothing
> to do with "s.el" (and which I'm proposing should have nothing
> to do with 'shorthands').
>
> In this scenario the files that lisp reads will always contain
> the real symbols, so no extra lisp support is required.  The
> lisp reader never sees the short names (and nor does any person
> who has not opted in), because they don't actually exist in the
> file.
>
> This use case is purely about enabling the people who opt in to
> type and see their short names when editing the source code, but
> with the real/longer names being the actual text.
[...]
> No, I'm talking about a feature (possibly already implemented by
> the "nameless" library; I still need to experiment with this)
> whereby people would be able to type their short symbols when
> editing their source code and Emacs would recognise the
> abbreviation and *automatically* (hence more like abbrev than
> dabbrev) change it to the real name.

This idea strikes me as better than what shorthand provides.  On
paper, at least.

The benefit of nameless is that I don't need to read the package name.
With the above I a) would not need to type the full package name, b)
could do this for any external package I want, c) could configure
which abbreviations are used.  Basically this takes the best of
shorthands and nameless, and combines it in a way that will give us
all benefits of shorthands without affecting 'read' or Emacs Lisp on a
fundamental level.  It removes the main reason to use dash.el or s.el,
which is their short prefixes.

The more we discuss this, the more misgivings I have about shorthands.
Something like what is described above could be a serious contender to
shorthands, but we unfortunately do not have time to explore that or
any other option given that shorthands is already going in so close to
the release of Emacs 28.  For now, it is just an idea of course, but
shorthands was also mostly an idea until this week (at least
publicly).

On a separate note, but related to my misgivings:  If this really is
mostly about s.el, dash.el and f.el, I think it is problem that this
is featured so prominently in the ELisp manual.  To my eyes, what we
have now is basically an implicit recommendation and a statement that
it is unproblematic for general use.  I believe there is a clear risk
that users will use this feature in ways that many of us AFAIU (from
reading this thread) would expressly rather avoid.

Finally, I think the existence of shorthands, and the work it will
cause in all of our tools, workflows, etc. means that people will be
more reluctant to accept proper namespacing later.  "Why do you need
that, just use shorthands", or "shorthands caused X, Y and Z to break
and now you want to break everything all over again".  In other words,
as a person who strongly disagrees with the namespace critics, I worry
that shorthands will stand in the way of something more proper.

Just my two cents.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Lars Ingebrigtsen]

Stefan Kangas <stefan@marxist.se> writes:

> On a separate note, but related to my misgivings:  If this really is
> mostly about s.el, dash.el and f.el, I think it is problem that this
> is featured so prominently in the ELisp manual.  To my eyes, what we
> have now is basically an implicit recommendation and a statement that
> it is unproblematic for general use.  I believe there is a clear risk
> that users will use this feature in ways that many of us AFAIU (from
> reading this thread) would expressly rather avoid.

Yes, making the manual less enthusiastic about this sounds like a good
idea to me.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Stefan Kangas <stefan@marxist.se> writes:
>
>> On a separate note, but related to my misgivings:  If this really is
>> mostly about s.el, dash.el and f.el, I think it is problem that this
>> is featured so prominently in the ELisp manual.  To my eyes, what we
>> have now is basically an implicit recommendation and a statement that
>> it is unproblematic for general use.  I believe there is a clear risk
>> that users will use this feature in ways that many of us AFAIU (from
>> reading this thread) would expressly rather avoid.
>
> Yes, making the manual less enthusiastic about this sounds like a good
> idea to me.

I don't care, go ahead.  Make it less enthusiastic about advice,
function pointers and macros while you're at it.

But really, if you want to add some sentences about caveats and grep
woes, go ahead, that's reasonable.  ...or delete the whole section, as I
said I don't care.

João

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: Stefan Kangas <stefan@marxist.se>,  Phil Sainty <psainty@orcon.net.nz>,
>   Alan Mackenzie <acm@muc.de>,  Eli Zaretskii <eliz@gnu.org>,  Emacs
>  developers <emacs-devel@gnu.org>
> Date: Fri, 01 Oct 2021 08:24:27 +0100
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > Stefan Kangas <stefan@marxist.se> writes:
> >
> >> On a separate note, but related to my misgivings:  If this really is
> >> mostly about s.el, dash.el and f.el, I think it is problem that this
> >> is featured so prominently in the ELisp manual.  To my eyes, what we
> >> have now is basically an implicit recommendation and a statement that
> >> it is unproblematic for general use.  I believe there is a clear risk
> >> that users will use this feature in ways that many of us AFAIU (from
> >> reading this thread) would expressly rather avoid.
> >
> > Yes, making the manual less enthusiastic about this sounds like a good
> > idea to me.
> 
> I don't care, go ahead.  Make it less enthusiastic about advice,
> function pointers and macros while you're at it.
> 
> But really, if you want to add some sentences about caveats and grep
> woes, go ahead, that's reasonable.  ...or delete the whole section, as I
> said I don't care.

Adding caveats and disadvantages to that section, as well as
recommendations to use it sparingly etc., is fine by me, but deleting
the section is not.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Eli Zaretskii]

> Date: Fri, 01 Oct 2021 11:50:14 +1300
> From: Phil Sainty <psainty@orcon.net.nz>
> Cc: acm@muc.de, joaotavora@gmail.com, emacs-devel@gnu.org
> 
> The *negative* impact of shorthands is limited iff the use of
> shorthands is limited to targeting a small number of problem
> libraries.

The intent is to use this only when necessary.  So yes, the use is
supposed to be limited to those few cases, at least in Emacs.

> This use case is purely about enabling the people who opt in to
> type and see their short names when editing the source code, but
> with the real/longer names being the actual text.

And I'm saying that this will be a bad solution, since many features
that identify symbols by name will not work with those "display-only"
names that the user sees.  So using something like that is a net loss
for users: it "fixes" a minor readability issue, but introduces major
usability issues.

> We might not allow any shorthand code in Emacs itself, but if
> shorthands are understood to be the in-built solution for
> reading and writing short names then all kinds of third-party
> code is going to start doing exactly that.  A very large
> proportion (I would assume a majority) of Emacs users will be
> using third-party code in their configs, and hence many of
> them/us will inevitably acquire shorthand code in our configs
> even if it is not core Emacs code or "s.el"; and the more such
> code that emerges, the more problems people will have.

We cannot control what third-party code is doing.  If you and others
don't want that to happen in third-party packages you use, the onus is
on you and others to complain to the respective developers, just like
you now (prematurely) complain here, and ask them not to abuse the
feature.

In any case, I don't understand what this has to do with Emacs:
nothing can prevent third-party packages from inventing something like
shorthands and starting to use that in their own packages.  The
existence of shorthands in Emacs core is not a sign to anyone that the
feature can be abused.  We in the Emacs project development cannot be
held responsible for irresponsible acts of others, it is highly unfair
for you and others to hold us to such a standard!

> This will affect people debugging their own configs; people
> sharing code with other people; people submitting bug reports
> or help requests here on the mailing lists and elsewhere.
> Whether it's something they can't figure out, or a frustrating
> stumbling block that they do figure out, I think it's going
> to create a lot of unnecessary problems.

Sorry, not our problem.  If and when this happens, please take it up
with people responsible for making it happen.  That's not us.

> I'm saying that code which does not contain shorthand symbols
> is *easier* (on average, for the Emacs user base as a whole)
> to deal with than code which does contain shorthand symbols,
> and so let's not bless shorthands (even tacitly) as the
> standard solution for anything which can be done in other ways.

Nothing in Emacs is "blessed" unconditionally.  Each feature has its
intended use, and those who attempt to use it outside of the intended
scope are responsible for the consequences.  I can assure you that
nothing like that will ever happen in Emacs, not on our watch.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Phil Sainty]

On 2021-10-01 19:09, Eli Zaretskii wrote:
> And I'm saying that this will be a bad solution, since many features
> that identify symbols by name will not work with those "display-only"
> names that the user sees.  So using something like that is a net loss
> for users: it "fixes" a minor readability issue, but introduces major
> usability issues.

Major usability issues caused by things not being what they appear
to be?  Not to put too finer point on it, but that is precisely what
this thread has been about from the beginning.

You can't do any of these things without causing problems.  The notion
of not using real symbol names is a problem by definition.  I'm merely
suggesting that the problems created by the 'nameless' style of "the
code is not what it appears to be" are not nearly as bad as the
problems caused by the 'shorthands' style of "the code is not what it
appears to be".

The particular issue of not being able to identify things using the
names that you see is (amongst the other issues which have been
discussed) precisely what people will face when looking at any code
where shorthands are in effect (at minimum code using s.el).  At least
with the 'nameless' approach every user who sees the short names has
opted in and is aware in advance that things are not how they appear.


> The existence of shorthands in Emacs core is not a sign to anyone
> that the feature can be abused.  We in the Emacs project development
> cannot be held responsible for irresponsible acts of others, it is
> highly unfair for you and others to hold us to such a standard!

I absolutely agree that you are not responsible for irresponsible acts
of others, but I don't think it was at all unreasonable to raise these
concerns.

I do believe that I've failed to convince you on the subject of
shorthands, however, so I shall hope that documentation dissuading
people from misusing the feature will prove sufficient for avoiding
very many such cases.


-Phil

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Eli Zaretskii]

> Date: Sat, 02 Oct 2021 01:24:22 +1300
> From: Phil Sainty <psainty@orcon.net.nz>
> Cc: acm@muc.de, joaotavora@gmail.com, emacs-devel@gnu.org
> 
> On 2021-10-01 19:09, Eli Zaretskii wrote:
> > And I'm saying that this will be a bad solution, since many features
> > that identify symbols by name will not work with those "display-only"
> > names that the user sees.  So using something like that is a net loss
> > for users: it "fixes" a minor readability issue, but introduces major
> > usability issues.
> 
> Major usability issues caused by things not being what they appear
> to be?  Not to put too finer point on it, but that is precisely what
> this thread has been about from the beginning.

No, because shorthands are known to Lisp.  They aren't a display-time
feature.

> > The existence of shorthands in Emacs core is not a sign to anyone
> > that the feature can be abused.  We in the Emacs project development
> > cannot be held responsible for irresponsible acts of others, it is
> > highly unfair for you and others to hold us to such a standard!
> 
> I absolutely agree that you are not responsible for irresponsible acts
> of others, but I don't think it was at all unreasonable to raise these
> concerns.

Sure, just don't raise them with us.

> I do believe that I've failed to convince you on the subject of
> shorthands, however, so I shall hope that documentation dissuading
> people from misusing the feature will prove sufficient for avoiding
> very many such cases.

We should mention the possible caveats of this, yes.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Phil Sainty]

On 2021-10-02 02:00, Eli Zaretskii wrote:
>> From: Phil Sainty <psainty@orcon.net.nz>
>> Major usability issues caused by things not being what they appear
>> to be?  Not to put too finer point on it, but that is precisely what
>> this thread has been about from the beginning.
> 
> No, because shorthands are known to Lisp.  They aren't a display-time
> feature.

Several usability issues arising from shorthands have been clearly
described by multiple people during this thread.  Breaking basic tooling
*is* a usability problem.  Symbols not existing by the names used in the
code *is* a usability problem.


>> I absolutely agree that you are not responsible for irresponsible acts
>> of others, but I don't think it was at all unreasonable to raise these
>> concerns.
> 
> Sure, just don't raise them with us.

I was advocating for the removal of a new feature from Emacs Lisp -- 
this
is the only place I could do that.


-Phil

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Eli Zaretskii]

> Date: Sat, 02 Oct 2021 13:28:11 +1300
> From: Phil Sainty <psainty@orcon.net.nz>
> Cc: acm@muc.de, joaotavora@gmail.com, emacs-devel@gnu.org
> 
> On 2021-10-02 02:00, Eli Zaretskii wrote:
> >> From: Phil Sainty <psainty@orcon.net.nz>
> >> Major usability issues caused by things not being what they appear
> >> to be?  Not to put too finer point on it, but that is precisely what
> >> this thread has been about from the beginning.
> > 
> > No, because shorthands are known to Lisp.  They aren't a display-time
> > feature.
> 
> Several usability issues arising from shorthands have been clearly
> described by multiple people during this thread.  Breaking basic tooling
> *is* a usability problem.  Symbols not existing by the names used in the
> code *is* a usability problem.

There's a huge difference between breaking literal searches for
symbols by text-searching tools, and breaking basic Emacs commands
because the name the user sees and types is not known to Emacs.

> >> I absolutely agree that you are not responsible for irresponsible acts
> >> of others, but I don't think it was at all unreasonable to raise these
> >> concerns.
> > 
> > Sure, just don't raise them with us.
> 
> I was advocating for the removal of a new feature from Emacs Lisp -- 
> this
> is the only place I could do that.

I see no reason to remove this feature.  The issues raised here are
all manageable as long as we use this only where and as intended.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Phil Sainty]

On 2021-10-02 19:45, Eli Zaretskii wrote:
> There's a huge difference between breaking literal searches for
> symbols by text-searching tools, and breaking basic Emacs commands
> because the name the user sees and types is not known to Emacs.

But shorthands does *both* of those things.

The name the user sees is "s-foo".

The name known to Emacs is "string-library-foo" (or whatever).

The user types "C-h o s-foo RET" and Emacs says "no match".


The huge difference is that with shorthands the above problem happens
to every user, whether or not they're aware that symbols can have
shorthands; whereas with the "nameless" approach it happens only to
the users who have knowingly chosen to have it happen, as a willing
trade-off to achieve something else.

I'm genuinely confused that you're disapproving of a feature that
people must opt into, on the basis of a problem which already happens
with the approved feature that people can't opt out of.

(Bearing in mind that the raison d'être for my idea was specifically
to provide an *alternative to shorthands* for people who wished to opt
into reading and writing short names, regardless of the 'problems',
and to allow them to do so without affecting anyone other than
themselves.)


-Phil

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Eli Zaretskii]

> Date: Sat, 02 Oct 2021 20:44:55 +1300
> From: Phil Sainty <psainty@orcon.net.nz>
> Cc: acm@muc.de, joaotavora@gmail.com, emacs-devel@gnu.org
> 
> On 2021-10-02 19:45, Eli Zaretskii wrote:
> > There's a huge difference between breaking literal searches for
> > symbols by text-searching tools, and breaking basic Emacs commands
> > because the name the user sees and types is not known to Emacs.
> 
> But shorthands does *both* of those things.
> 
> The name the user sees is "s-foo".
> 
> The name known to Emacs is "string-library-foo" (or whatever).
> 
> The user types "C-h o s-foo RET" and Emacs says "no match".

If this is correct (I didn't try), please report it as a bug.  The
difference is that this bug can be solved, because Lisp knows about
the shorthand, whereas display-time features cannot be fixed as
easily.

> I'm genuinely confused that you're disapproving of a feature that
> people must opt into, on the basis of a problem which already happens
> with the approved feature that people can't opt out of.

Existing problems in approved features are bugs that need to be
solved.  They are not reasons for removing the feature.

I explained the reasons for disapproving renaming that is limited to
display; if you still disagree, let's agree to disagree.

bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[Phil Sainty]

Following up on 
https://lists.gnu.org/archive/html/emacs-devel/2021-10/msg00109.html

On 2021-10-02 21:53, Eli Zaretskii wrote:
>> From: Phil Sainty <psainty@orcon.net.nz>
>> On 2021-10-02 19:45, Eli Zaretskii wrote:
>> > There's a huge difference between breaking literal searches for
>> > symbols by text-searching tools, and breaking basic Emacs commands
>> > because the name the user sees and types is not known to Emacs.
>> 
>> But shorthands does *both* of those things.
>> 
>> The name the user sees is "s-foo".
>> The name known to Emacs is "string-library-foo" (or whatever).
>> The user types "C-h o s-foo RET" and Emacs says "no match".
> 
> If this is correct (I didn't try), please report it as a bug.


This was indeed the case in the build I'd compiled for testing:

GNU Emacs 28.0.50 of 2021-09-29
Repository revision: b02a7ad2631b6ac3a95e53cb26a0aa1b1ab7e98a
Repository branch: master

I tested with a copy of so-long.el in which I renamed all of the
so-long-* symbols to sl-* and then configured the local variable
;; elisp-shorthands: (("sl-" . "so-long-"))

Loading the new sl.el confirmed that Emacs didn't recognise the
shorthand symbols generally.

(Until now I was under the impression that this was all part and
parcel of the shorthands feature, but it turns out that it's a bug.)


-Phil

bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[Eli Zaretskii]

> Date: Sat, 02 Oct 2021 22:20:09 +1300
> From: Phil Sainty <psainty@orcon.net.nz>
> 
> Following up on 
> https://lists.gnu.org/archive/html/emacs-devel/2021-10/msg00109.html
> 
> On 2021-10-02 21:53, Eli Zaretskii wrote:
> >> From: Phil Sainty <psainty@orcon.net.nz>
> >> On 2021-10-02 19:45, Eli Zaretskii wrote:
> >> > There's a huge difference between breaking literal searches for
> >> > symbols by text-searching tools, and breaking basic Emacs commands
> >> > because the name the user sees and types is not known to Emacs.
> >> 
> >> But shorthands does *both* of those things.
> >> 
> >> The name the user sees is "s-foo".
> >> The name known to Emacs is "string-library-foo" (or whatever).
> >> The user types "C-h o s-foo RET" and Emacs says "no match".
> > 
> > If this is correct (I didn't try), please report it as a bug.
> 
> 
> This was indeed the case in the build I'd compiled for testing:
> 
> GNU Emacs 28.0.50 of 2021-09-29
> Repository revision: b02a7ad2631b6ac3a95e53cb26a0aa1b1ab7e98a
> Repository branch: master
> 
> I tested with a copy of so-long.el in which I renamed all of the
> so-long-* symbols to sl-* and then configured the local variable
> ;; elisp-shorthands: (("sl-" . "so-long-"))
> 
> Loading the new sl.el confirmed that Emacs didn't recognise the
> shorthand symbols generally.

João, if this problem still exists after your changes yesterday, could
you please look into it?  TIA.

bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

>> Loading the new sl.el confirmed that Emacs didn't recognise the
>> shorthand symbols generally.
>
> João, if this problem still exists after your changes yesterday, could
> you please look into it?  TIA.

It's not after my changes from yesterday, no.  This is by design.  Emacs
doesn't recognize the shorthands symbols generally (if "generally" is to
mean "globally") because shorthands _don't_ exist in the global obarray.
They exist in the buffer.  I'm going to type in the relevant part of my
reply to Phil in emacs-devel, when complaining about C-h o not finding
his shorthand.

    That's only when the user types that in the minibuffer and doesn't
    associate [it] in any way to the buffer where [the user] sets up
    that particular shorthand (remember, shorthands aren't global:
    that's the point).  Much like if I type 'import foo as bar' in my
    Python of JavaScript program and then go search [globally, on the
    internet] for 'bar' I don't get the results for 'foo'.
     
    But (have you seen the animated gif?)  if you type 'C-h o' with point ON
    TOP OF 's-foo', then M-x describe-symbol will be prepolulated with
    string-library-foo, and need only type RET.

In other words, Phil, if you wish me to do anything about this bug, you
must explain exactly what you are doing and what you expect.  What does
it mean precisely for "Emacs to recognise the shorthand symbols
generally".  Currently it "recognizes" them when the buffer where they
are setup is current.

João

bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: Phil Sainty <psainty@orcon.net.nz>,  50959@debbugs.gnu.org
> Date: Sat, 02 Oct 2021 12:02:30 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Loading the new sl.el confirmed that Emacs didn't recognise the
> >> shorthand symbols generally.
> >
> > João, if this problem still exists after your changes yesterday, could
> > you please look into it?  TIA.
> 
> It's not after my changes from yesterday, no.  This is by design.  Emacs
> doesn't recognize the shorthands symbols generally (if "generally" is to
> mean "globally") because shorthands _don't_ exist in the global obarray.
> They exist in the buffer.

Would it be possible to see if the current buffer (from which the
minibuffer was entered) has shorthands, and if so, apply them to
minibuffer input?

bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: João Távora <joaotavora@gmail.com>
>> Cc: Phil Sainty <psainty@orcon.net.nz>,  50959@debbugs.gnu.org
>> Date: Sat, 02 Oct 2021 12:02:30 +0100
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> Loading the new sl.el confirmed that Emacs didn't recognise the
>> >> shorthand symbols generally.
>> >
>> > João, if this problem still exists after your changes yesterday, could
>> > you please look into it?  TIA.
>> 
>> It's not after my changes from yesterday, no.  This is by design.  Emacs
>> doesn't recognize the shorthands symbols generally (if "generally" is to
>> mean "globally") because shorthands _don't_ exist in the global obarray.
>> They exist in the buffer.
>
> Would it be possible to see if the current buffer (from which the
> minibuffer was entered) has shorthands, and if so, apply them to
> minibuffer input?

Yes, much like it is done with C-M-i, basically.

João

bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[João Távora]

João Távora <joaotavora@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:

>> Would it be possible to see if the current buffer (from which the
>> minibuffer was entered) has shorthands, and if so, apply them to
>> minibuffer input?
>
> Yes, much like it is done with C-M-i, basically.

Let me improve on that.  That it _can_ be done doesn't mean that it
_should_ be done, but we can decide on that.  There are various levels
to this integration:

0. no integration

1. This is the current integration.  I.e. when C-h o is pressed on the
   symbol the global name is discovered and used as the default.  This
   is how SLIME work with CL's namespacing system.  SLIME is a very well
   tested and widely appreciated Common Lisp IDE for Emcas.

2. The shorthands from the buffer where the minibuffer was entered are
   _not_ in the completions list, but typing one of them interns the
   symbol with those shorthands present, so you get the desired result.
   This would fix Phil's visually-copy-and-type scenario.

3. (Eli's suggestion): the completion list would be augmented with the
   shorthands from the buffer where the minibuffer was entered from.

In levels 2 and 3 the user might be surprised that what once worked for
one 'C-h o' session no longer works for another 'C-h o' session.  The
only way I can see this being acceptable is if the user is somehow made
aware -- visually or otherwise -- that the list she is seeing is somehow
connected to the origin buffer.

In contrast, C-M-i (where level 3 happens) never really leaves the
buffer: its results are connected to it because they will be inserted
there.

João

bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: psainty@orcon.net.nz,  50959@debbugs.gnu.org
> Date: Sat, 02 Oct 2021 12:54:13 +0100
> 
> 0. no integration
> 
> 1. This is the current integration.  I.e. when C-h o is pressed on the
>    symbol the global name is discovered and used as the default.  This
>    is how SLIME work with CL's namespacing system.  SLIME is a very well
>    tested and widely appreciated Common Lisp IDE for Emcas.
> 
> 2. The shorthands from the buffer where the minibuffer was entered are
>    _not_ in the completions list, but typing one of them interns the
>    symbol with those shorthands present, so you get the desired result.
>    This would fix Phil's visually-copy-and-type scenario.
> 
> 3. (Eli's suggestion): the completion list would be augmented with the
>    shorthands from the buffer where the minibuffer was entered from.

Are 2 and 3 significantly different (from the implementation POV)?

bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[João Távora]

On Sat, Oct 2, 2021 at 1:30 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Cc: psainty@orcon.net.nz,  50959@debbugs.gnu.org
> > Date: Sat, 02 Oct 2021 12:54:13 +0100
> >
> > 0. no integration
> >
> > 1. This is the current integration.  I.e. when C-h o is pressed on the
> >    symbol the global name is discovered and used as the default.  This
> >    is how SLIME work with CL's namespacing system.  SLIME is a very well
> >    tested and widely appreciated Common Lisp IDE for Emcas.
> >
> > 2. The shorthands from the buffer where the minibuffer was entered are
> >    _not_ in the completions list, but typing one of them interns the
> >    symbol with those shorthands present, so you get the desired result.
> >    This would fix Phil's visually-copy-and-type scenario.
> >
> > 3. (Eli's suggestion): the completion list would be augmented with the
> >    shorthands from the buffer where the minibuffer was entered from.
>
> Are 2 and 3 significantly different (from the implementation POV)?

I think so.

I think 2 can be achieved by setting elisp-shorthands buffer-locally
in the minibuffer and removing the "require-match" flag requirement to
whatever completing-read call happens there.

3 is achieved by calculating the list of completions using
'elisp--completion-local-symbols` and then filtering it down as usual.
"require-match" is kept untouched.

João

bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sat, 2 Oct 2021 15:02:41 +0100
> Cc: Phil Sainty <psainty@orcon.net.nz>, 50959@debbugs.gnu.org
> 
> > > 0. no integration
> > >
> > > 1. This is the current integration.  I.e. when C-h o is pressed on the
> > >    symbol the global name is discovered and used as the default.  This
> > >    is how SLIME work with CL's namespacing system.  SLIME is a very well
> > >    tested and widely appreciated Common Lisp IDE for Emcas.
> > >
> > > 2. The shorthands from the buffer where the minibuffer was entered are
> > >    _not_ in the completions list, but typing one of them interns the
> > >    symbol with those shorthands present, so you get the desired result.
> > >    This would fix Phil's visually-copy-and-type scenario.
> > >
> > > 3. (Eli's suggestion): the completion list would be augmented with the
> > >    shorthands from the buffer where the minibuffer was entered from.
> >
> > Are 2 and 3 significantly different (from the implementation POV)?
> 
> I think so.
> 
> I think 2 can be achieved by setting elisp-shorthands buffer-locally
> in the minibuffer and removing the "require-match" flag requirement to
> whatever completing-read call happens there.
> 
> 3 is achieved by calculating the list of completions using
> 'elisp--completion-local-symbols` and then filtering it down as usual.
> "require-match" is kept untouched.

You are saying that 3 is easier than 2?  Then I think we should do 3,
as it's better from the user's POV, right?

bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[João Távora]

On Sat, Oct 2, 2021 at 3:20 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Sat, 2 Oct 2021 15:02:41 +0100
> > Cc: Phil Sainty <psainty@orcon.net.nz>, 50959@debbugs.gnu.org
> >
> > > > 0. no integration
> > > >
> > > > 1. This is the current integration.  I.e. when C-h o is pressed on the
> > > >    symbol the global name is discovered and used as the default.  This
> > > >    is how SLIME work with CL's namespacing system.  SLIME is a very well
> > > >    tested and widely appreciated Common Lisp IDE for Emcas.
> > > >
> > > > 2. The shorthands from the buffer where the minibuffer was entered are
> > > >    _not_ in the completions list, but typing one of them interns the
> > > >    symbol with those shorthands present, so you get the desired result.
> > > >    This would fix Phil's visually-copy-and-type scenario.
> > > >
> > > > 3. (Eli's suggestion): the completion list would be augmented with the
> > > >    shorthands from the buffer where the minibuffer was entered from.
> > >
> > > Are 2 and 3 significantly different (from the implementation POV)?
> >
> > I think so.
> >
> > I think 2 can be achieved by setting elisp-shorthands buffer-locally
> > in the minibuffer and removing the "require-match" flag requirement to
> > whatever completing-read call happens there.
> >
> > 3 is achieved by calculating the list of completions using
> > 'elisp--completion-local-symbols` and then filtering it down as usual.
> > "require-match" is kept untouched.
>
> You are saying that 3 is easier than 2?  Then I think we should do 3,
> as it's better from the user's POV, right?

No, I don't know that for sure.  And I don't think it's better from
the user's POV.
See my reply to Phil.  It would mistakenly provide the idea that Shorthands
are some alias to the symbol (in the sense of defvaralias).  They are not.

The user would then be quite surprised to find the list of completions change
behind his back as he changes the place of origin of those C-h o calls.

It could only make sense if these interactive prompts were clearly tied to
the buffers they originated from.  The current "Describe function" prompt
should become "Describe function inside foobarbaz.el" .  Even then, I think
it is insufficient, and uncharted territory, whereas the current approach isn't
(it's how SLIME/SLY work in a Lisp-based IDE with local namespacing)

That _can_ be done, and I think all of Phil's list will eventually
funnel down to
a few existing function calls.  But it nevertheless needs more profound work
and a careful understanding of the consequences.

In summary, I think that with the exception of a shorthand-aware
'xref-backend-references',
something that I am working on (between the drops of the torrential emails,
some of them bordering on sheer harassment), this feature is currently
consistent
from a tools point of view.

Again, Shorthands are buffer-local textual indirections to symbols.  They
are not the symbol.  This will never change (not with Shorthands): so including
shorthands in a list of symbols is misguided.  Displaying them in
lists of fragments of
text to be completed in the buffer is not.

That is not to say I don't understand Phil's concerns: I do.  I just
don't understand
the feeling of imminent catastrophe.

As I wrote to Phil, I believe much of this anguish is improved if we font-lock
shorthands specially.  That doesn't seem hard at all.  If I understand Phil's
objections from day 1, he is talking about looking at something in an Elisp
buffer and being uncertain about the nature of the thing that his eyes are
focusing.  But if he is _looking_ at it, then we can quench much of that
doubt by changing the way it looks.

If, on the other hand, he is operating on the thing with some other tool [*],
not just his eyes, then I think the current tools are already doing
the right thing.

João

[*]: Yes, except when that tool is Grep or similar, but that's for all namespace
systems ever invented, so if the concern is still Grep, please open another
bug or (yet) another thread.

bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sat, 2 Oct 2021 15:43:03 +0100
> Cc: Phil Sainty <psainty@orcon.net.nz>, 50959@debbugs.gnu.org
> 
> > You are saying that 3 is easier than 2?  Then I think we should do 3,
> > as it's better from the user's POV, right?
> 
> No, I don't know that for sure.  And I don't think it's better from
> the user's POV.
> See my reply to Phil.  It would mistakenly provide the idea that Shorthands
> are some alias to the symbol (in the sense of defvaralias).  They are not.
> 
> The user would then be quite surprised to find the list of completions change
> behind his back as he changes the place of origin of those C-h o calls.

I'm not sure the user will be so much surprised.  We could document
that.  And shorthands aren't supposed to be used massively or
willy-nilly, so these surprises are not necessarily as acute as you
think.  they are certainly not worse than not showing these shorthands
at all.

> It could only make sense if these interactive prompts were clearly tied to
> the buffers they originated from.

But they are: we always know which buffer was the current one when the
minibuffer is entered.

> In summary, I think that with the exception of a shorthand-aware
> 'xref-backend-references',
> something that I am working on (between the drops of the torrential emails,
> some of them bordering on sheer harassment), this feature is currently
> consistent
> from a tools point of view.

You are saying that Help commands should allow asking about
shorthands, except if point is on the shorthand?  That'd be a grave
restriction, I think, worse than "depending on the buffer" which you
don't like: here it depends not only on the buffer, but also on
position of point in that buffer.

> Again, Shorthands are buffer-local textual indirections to symbols.  They
> are not the symbol.  This will never change (not with Shorthands): so including
> shorthands in a list of symbols is misguided.  Displaying them in
> lists of fragments of
> text to be completed in the buffer is not.

I think this is unnecessarily radical POV, and one that will cause
complaints.

bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[João Távora]

On Sat, Oct 2, 2021 at 3:56 PM Eli Zaretskii <eliz@gnu.org> wrote:

> But they are: we always know which buffer was the current one when the
> minibuffer is entered.

We "the Emacs code" do. But it's not obvious to a user.  The commands
that invoke these things are global, remember.  I very very often don't know
where I pulled C-h o from.  And I don't want to know. Sometimes I do,
sometimes I don't.

In summary, I wish we can keep the what has been the invariant of Elisp
ever since it was created, and which is underlined many times in the manual:
there is only one obarray of symbols.  I believe my stance preserves this
accurate mental model more than yours.

> You are saying that Help commands should allow asking about
> shorthands, except if point is on the shorthand?

"should NOT allow".  That's what I am saying, yes.  Any command that operates
on _symbols_ should not offer shorthands as candidates unless the goal of that
command is to directly modify the buffer where those shorthands have meaning.

So C-h o is in the former group.  C-M-i s in the latter.

> That'd be a grave
> restriction, I think, worse than "depending on the buffer" which you
> don't like: here it depends not only on the buffer, but also on
> position of point in that buffer.

I don't agree, but ultimately it's your call.  Notice (maybe watch the
.gif again),
that what happens when you type C-h o on 's-concat' is that the prompt becomes:

   "Describe symbol (default magnar-string-concat): ... "

It does _not_ become:

   "Describe symbol (default s-concat): ... "

Because 's-concat' is _not_ a symbol.

> > Again, Shorthands are buffer-local textual indirections to symbols.  They
> > are not the symbol.  This will never change (not with Shorthands): so including
> > shorthands in a list of symbols is misguided.  Displaying them in
> > lists of fragments of
> > text to be completed in the buffer is not.
> I think this is unnecessarily radical POV, and one that will cause
> complaints.

It hasn't in SLIME/SLY and package-local-nicknames have existed for
quite some time there.

What is your opinion on the visually annotating font-lock idea?  I think
it's useful even if we decide to go with levels 2 or 3 of the above
integration (which, as I said, I think we shouldn't, not for now)

João

bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sat, 2 Oct 2021 16:22:51 +0100
> Cc: Phil Sainty <psainty@orcon.net.nz>, 50959@debbugs.gnu.org
> 
> > That'd be a grave
> > restriction, I think, worse than "depending on the buffer" which you
> > don't like: here it depends not only on the buffer, but also on
> > position of point in that buffer.
> 
> I don't agree, but ultimately it's your call.  Notice (maybe watch the
> .gif again),
> that what happens when you type C-h o on 's-concat' is that the prompt becomes:
> 
>    "Describe symbol (default magnar-string-concat): ... "
> 
> It does _not_ become:
> 
>    "Describe symbol (default s-concat): ... "
> 
> Because 's-concat' is _not_ a symbol.

I don't see the significance of the difference, from the usability
POV.  I'd still like to see Help commands support shorthands even if
point is not on a shorthand.

> What is your opinion on the visually annotating font-lock idea?  I think
> it's useful even if we decide to go with levels 2 or 3 of the above
> integration (which, as I said, I think we shouldn't, not for now)

I don't object, and think it could be useful.  But I don't think it
could supplant the recognition of shorthands in Help commands.

bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[João Távora]

On Sat, Oct 2, 2021 at 4:53 PM Eli Zaretskii <eliz@gnu.org> wrote:
> > Because 's-concat' is _not_ a symbol.
>
> I don't see the significance of the difference, from the usability
> POV.

You don't?  Maybe you are still somewhat thinking of shorthands
as similar to the products of defvaralias or defalias.  s-concat
may mean magnar-string-concat in a buffer, and stream-concat
in another.  In that some other buffer magnar-string-concat may
be accessible by the shorthand ms-concat.

This is how namespaces work also in other languages.  In Elisp
in proposals like Andreas Corallo's
(https://akrl.sdf.org/lexspaces/lexspaces.html)
or if Lars' proposed evolution of shorthands to be somehow local
to a specific region to a buffer (like Common Lisp's IN-PACKAGE).

> I'd still like to see Help commands support shorthands even if
> point is not on a shorthand.

Alright, it does need an implementation.  Of unforeseen complexity.
Certainly the most tricky of all the feature so far, I predict,  as it would
affect many more commands. I'm not confident I can muster the effort
to tackle it for Emacs 28, though I can of course experiment (and maybe
it's easier, especially if you help me restrict the feature in scope).

If it's ever done, I think at the very least it should be made
optional, and that the default should be off.

The least of its problems is that, when set to on, it would bring back
some form of namespace pollution: finding too "inexplicably" short
symbol names when perusing global lists of symbols.  What's more,
finding them there only sometimes.

The more serious of its problems is that it would make the evolution
of Shorthands to be local to a region (for example, as I hinted
above) much harder.

Then, it would raise the bar to adopting _other_ namespacing systems
even higher than it already is.

> I don't object, and think it could be useful.  But I don't think it
> could supplant the recognition of shorthands in Help commands.

I think it could, by helping the user keep the correct mental model
in mind.  It could even be made more sophisticated, like temporarily
displaying the full names of the symbols at the point of the symbols when
C-h f is entered. That would be level of integration 1.5 so to speak (in
the scale I gave a while ago).

I think we must focus on the problem being presented here, not on
the particular solution.  _Why_, fundamentally do you want to see
shorthands listed in an Help command? What, in layman's terms,
are the pieces of information that are missing for you when you _don't_
see them?  Is it more like:

   a. "I wonder what that that 's-thingy' over there is/means or what
    documentation there is for it.?"

Or is it:

  b. "I wonder what help on s-thingies I have at my disposal here
   (and yes I know perfectly well what 'here' is)?  I'm going to type a
   prefix, the TAB to find out."

Or is it both? Or is it a third thing?

The question goes for Phil too, though I think I have a better intuition of
what his problem is: I think it's 'a': he fears he won't be able to immediately
tell visually what he's looking at, say in one of his windows that has
an excerpt of a .el file open.  Disregarding the fact that that already
happens with things like  cl-flet or letf, for example (lexical function
definitions), I think it's a legitimate concern (especially coming from
someone with likely no experience of namespace systems at all).

João Távora

bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sat, 2 Oct 2021 18:46:18 +0100
> Cc: Phil Sainty <psainty@orcon.net.nz>, 50959@debbugs.gnu.org
> 
> On Sat, Oct 2, 2021 at 4:53 PM Eli Zaretskii <eliz@gnu.org> wrote:
> > > Because 's-concat' is _not_ a symbol.
> >
> > I don't see the significance of the difference, from the usability
> > POV.
> 
> You don't?  Maybe you are still somewhat thinking of shorthands
> as similar to the products of defvaralias or defalias.  s-concat
> may mean magnar-string-concat in a buffer, and stream-concat
> in another.  In that some other buffer magnar-string-concat may
> be accessible by the shorthand ms-concat.

We are talking at cross-purposes.  You are talking about the
underlying mechanism; I'm talking about the user's experience.  From
the UX POV, it is unexpected to be unable to get help about a
symbol-like thing the user sees in the buffer.

bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[João Távora]

On Sat, Oct 2, 2021 at 6:58 PM Eli Zaretskii <eliz@gnu.org> wrote:

> We are talking at cross-purposes.  You are talking about the
> underlying mechanism; I'm talking about the user's experience.  From
> the UX POV, it is unexpected to be unable to get help about a
> symbol-like thing the user sees in the buffer.

I'm taking both things into consideration. Does that mean that
your answer to my questionnaire is 'a.' or "mostly" 'a.'?

 a. "I wonder what that that 's-thingy' over there is/means or what
    documentation there is for it.?"

I'm assuming that since you wrote "sees in the buffer".

João

PS: Much better terminology "symbol-like thing" :-)

bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sat, 2 Oct 2021 19:03:43 +0100
> Cc: Phil Sainty <psainty@orcon.net.nz>, 50959@debbugs.gnu.org
> 
> On Sat, Oct 2, 2021 at 6:58 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > We are talking at cross-purposes.  You are talking about the
> > underlying mechanism; I'm talking about the user's experience.  From
> > the UX POV, it is unexpected to be unable to get help about a
> > symbol-like thing the user sees in the buffer.
> 
> I'm taking both things into consideration. Does that mean that
> your answer to my questionnaire is 'a.' or "mostly" 'a.'?
> 
>  a. "I wonder what that that 's-thingy' over there is/means or what
>     documentation there is for it.?"
> 
> I'm assuming that since you wrote "sees in the buffer".

Yes.

bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > We are talking at cross-purposes.  You are talking about the
  > underlying mechanism; I'm talking about the user's experience.  From
  > the UX POV, it is unexpected to be unable to get help about a
  > symbol-like thing the user sees in the buffer.

I agree.  We should implement doing the right thing with these cases,
or at least whatever thing is the most right that we can come up with.
-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I don't object, and think it could be useful.  But I don't think it
  > could supplant the recognition of shorthands in Help commands.

Perhaps it would be useful to make a list of the help commands
(I suggest command names rather than their key bindings) in which
we are considering doing this.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Again, Shorthands are buffer-local textual indirections to symbols.  They
  > > are not the symbol.  This will never change (not with Shorthands): so including
  > > shorthands in a list of symbols is misguided.  Displaying them in
  > > lists of fragments of
  > > text to be completed in the buffer is not.

  > I think this is unnecessarily radical POV, and one that will cause
  > complaints.

It seems valid to me _in principle_, but can you think of specific
cases that might cause complaints?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: joaotavora@gmail.com, psainty@orcon.net.nz, 50959@debbugs.gnu.org
> Date: Sun, 03 Oct 2021 20:14:56 -0400
> 
>   > > Again, Shorthands are buffer-local textual indirections to symbols.  They
>   > > are not the symbol.  This will never change (not with Shorthands): so including
>   > > shorthands in a list of symbols is misguided.  Displaying them in
>   > > lists of fragments of
>   > > text to be completed in the buffer is not.
> 
>   > I think this is unnecessarily radical POV, and one that will cause
>   > complaints.
> 
> It seems valid to me _in principle_, but can you think of specific
> cases that might cause complaints?

I described them: when the user sees an identifier in the buffer, but
cannot call up Help for it unless point is on that identifier.

bug#50959: [PATCH] Re: bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[João Távora]

João Távora <joaotavora@gmail.com> writes:

> On Sat, Oct 2, 2021 at 1:30 PM Eli Zaretskii <eliz@gnu.org> wrote:

>> > 0. no integration
>> >
>> > 1. This is the current integration.  I.e. when C-h o is pressed on the
>> >    symbol the global name is discovered and used as the default.  This
>> >    is how SLIME work with CL's namespacing system.  SLIME is a very well
>> >    tested and widely appreciated Common Lisp IDE for Emcas.
>> >
>> > 2. The shorthands from the buffer where the minibuffer was entered are
>> >    _not_ in the completions list, but typing one of them interns the
>> >    symbol with those shorthands present, so you get the desired result.
>> >    This would fix Phil's visually-copy-and-type scenario.
>> >
>> > 3. The completion list would be augmented with the shorthands from
>> >    the buffer where the minibuffer was entered from.

Hello Eli,

I've implemented a variation on 2 based on the later suggestion you
gave in emacs-devel:

> That is, the user types "C-h o s-foo <SOME KEY SEQUENCE>" and that
> replaces s-foo with the expansion, the "real" symbol.  Is that
> feasible?

Yes, it is.  <SOME KEY SEQUENCE> is, of course, TAB.  Here is a patch
for people to try out which I will push in a few days time if there are
no objections.  Cc-ing completion-style specialist Stefan.

Patch just below,
João

commit f9f64c4b3287d7276c8edeacdecfa9c78194447b
Author: João Távora <joaotavora@gmail.com>
Date:   Wed Oct 6 11:30:29 2021 +0100

    Complete shorthands to longhands for symbol-completing tables
    
    Shorthands aren't symbols, they're text forms that 'read' into
    symbols.  As such, shorthands aren't candidates in these tables of
    symbols.  But in some situations, if no other candidates match the
    pattern, we can e.g. complete "x-foo" to "xavier-foo" if the shorthand
    
      (("x-" . "xavier-"))
    
    is set up in the buffer of origin.
    
    bug#50959
    
    * lisp/help-fns.el (help--symbol-completion-table): Report
    `symbol-help' category.
    
    * lisp/minibuffer.el (completion-styles-alist): New 'shorthand'
    style.
    (completion-category-defaults): Link 'symbol-help' category with
    'shorthand' style.
    (minibuffer--original-buffer): New variable.
    (completing-read-default): Setup minibuffer--original-buffer.
    (completion-shorthand-try-completion)
    (completion-shorthand-all-completions): New helpers.

diff --git a/lisp/help-fns.el b/lisp/help-fns.el
index 6be5cd4a50..03bbc979a9 100644
--- a/lisp/help-fns.el
+++ b/lisp/help-fns.el
@@ -176,8 +176,11 @@ help--symbol-completion-table-affixation
           completions))
 
 (defun help--symbol-completion-table (string pred action)
-  (if (and completions-detailed (eq action 'metadata))
-      '(metadata (affixation-function . help--symbol-completion-table-affixation))
+  (if (eq action 'metadata)
+      `(metadata
+        ,@(when completions-detailed
+            '((affixation-function . help--symbol-completion-table-affixation)))
+        (category . symbol-help))
     (when help-enable-completion-autoload
       (let ((prefixes (radix-tree-prefixes (help-definition-prefixes) string)))
         (help--load-prefixes prefixes)))
diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 1e1a6f852e..48859585bc 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -943,7 +943,12 @@ completion-styles-alist
      completion-initials-try-completion completion-initials-all-completions
      "Completion of acronyms and initialisms.
 E.g. can complete M-x lch to list-command-history
-and C-x C-f ~/sew to ~/src/emacs/work."))
+and C-x C-f ~/sew to ~/src/emacs/work.")
+    (shorthand
+     completion-shorthand-try-completion completion-shorthand-all-completions
+     "Completion of symbol shorthands setup in `read-symbol-shorthands'.
+E.g. can complete \"x-foo\" to \"xavier-foo\" if the shorthand
+((\"x-\" . \"xavier-\")) is set up in the buffer of origin."))
   "List of available completion styles.
 Each element has the form (NAME TRY-COMPLETION ALL-COMPLETIONS DOC):
 where NAME is the name that should be used in `completion-styles',
@@ -990,7 +995,8 @@ completion-category-defaults
     ;; e.g. one that does not anchor to bos.
     (project-file (styles . (substring)))
     (xref-location (styles . (substring)))
-    (info-menu (styles . (basic substring))))
+    (info-menu (styles . (basic substring)))
+    (symbol-help (styles . (basic shorthand substring))))
   "Default settings for specific completion categories.
 Each entry has the shape (CATEGORY . ALIST) where ALIST is
 an association list that can specify properties such as:
@@ -1618,6 +1624,9 @@ minibuffer-confirm-exit-commands
 (defvar minibuffer--require-match nil
   "Value of REQUIRE-MATCH passed to `completing-read'.")
 
+(defvar minibuffer--original-buffer nil
+  "Buffer that was current when `completing-read' was called.")
+
 (defun minibuffer-complete-and-exit ()
   "Exit if the minibuffer contains a valid completion.
 Otherwise, try to complete the minibuffer contents.  If
@@ -4080,6 +4089,40 @@ completion-initials-try-completion
   (let ((newstr (completion-initials-expand string table pred)))
     (when newstr
       (completion-pcm-try-completion newstr table pred (length newstr)))))
+
+;; Shorthand completion
+;;
+;; Iff there is a (("x-" . "string-library-")) shorthand setup and
+;; string-library-foo is in candidates, complete x-foo to it.
+
+(defun completion-shorthand-try-completion (string table pred point)
+  "Try completion with `read-symbol-shorthands' of original buffer."
+  (cl-loop with expanded
+           for (short . long) in
+           (with-current-buffer minibuffer--original-buffer
+             read-symbol-shorthands)
+           for probe =
+           (and (> point (length short))
+                (string-prefix-p short string)
+                (try-completion (setq expanded
+                                      (concat long
+                                              (substring
+                                               string
+                                               (length short))))
+                                table pred))
+           when probe
+           do (message "Shorthand expansion")
+           and return (cons expanded (max (length long)
+                                          (+ (- point (length short))
+                                             (length long))))))
+
+(defun completion-shorthand-all-completions (string table pred _point)
+  ;; no-op: For now, we don't want shorthands to list all the possible
+  ;; locally active longhands.  For the completion categories where
+  ;; this style is active, it could hide other more interesting
+  ;; matches from subsequent styles.
+  nil)
+
 \f
 (defvar completing-read-function #'completing-read-default
   "The function called by `completing-read' to do its work.
@@ -4111,6 +4154,7 @@ completing-read-default
                     ;; in minibuffer-local-filename-completion-map can
                     ;; override bindings in base-keymap.
                     base-keymap)))
+         (buffer (current-buffer))
          (result
           (minibuffer-with-setup-hook
               (lambda ()
@@ -4119,7 +4163,8 @@ completing-read-default
                 ;; FIXME: Remove/rename this var, see the next one.
                 (setq-local minibuffer-completion-confirm
                             (unless (eq require-match t) require-match))
-                (setq-local minibuffer--require-match require-match))
+                (setq-local minibuffer--require-match require-match)
+                (setq-local minibuffer--original-buffer buffer))
             (read-from-minibuffer prompt initial-input keymap
                                   nil hist def inherit-input-method))))
     (when (and (equal result "") def)

bug#50959: [PATCH] Re: bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Yes, it is.  <SOME KEY SEQUENCE> is, of course, TAB.  Here is a patch
> for people to try out which I will push in a few days time if there are
> no objections.  Cc-ing completion-style specialist Stefan.

Implementing it as a completion style seems wrong.
Why not just tweak the completion table?


        Stefan

bug#50959: [PATCH] Re: bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 759 bytes --]

On Wed, Oct 6, 2021 at 1:19 PM Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> > Yes, it is.  <SOME KEY SEQUENCE> is, of course, TAB.  Here is a patch
> > for people to try out which I will push in a few days time if there are
> > no objections.  Cc-ing completion-style specialist Stefan.
>
> Implementing it as a completion style seems wrong.
>

What is "it"? "It" === "Eli's suggestion"?


> Why not just tweak the completion table?


What tweak do you suggest? The explicit point in this patch
was _not_ to list shorthands as completions, for the reasons
that I gave many times already and that led to Eli's suggestion.
If you can implement that without a completion style that's fine
(but it seems it's the nicest way).

João

[-- Attachment #2: Type: text/html, Size: 1369 bytes --]

bug#50959: [PATCH] Re: bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> > Yes, it is.  <SOME KEY SEQUENCE> is, of course, TAB.  Here is a patch
>> > for people to try out which I will push in a few days time if there are
>> > no objections.  Cc-ing completion-style specialist Stefan.
>> Implementing it as a completion style seems wrong.
> What is "it"? "It" === "Eli's suggestion"?

"it" == "making C-x o's completion pay attention to shorthands".


        Stefan

bug#50959: [PATCH] Re: bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 769 bytes --]

On Wed, Oct 6, 2021 at 1:50 PM Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> >> > Yes, it is.  <SOME KEY SEQUENCE> is, of course, TAB.  Here is a patch
> >> > for people to try out which I will push in a few days time if there
> are
> >> > no objections.  Cc-ing completion-style specialist Stefan.
> >> Implementing it as a completion style seems wrong.
> > What is "it"? "It" === "Eli's suggestion"?
>
> "it" == "making C-x o's completion pay attention to shorthands".


In this thread, we've described many ways how that can be done.
C-x o already "pays attention" to them, from day one, but that attention
is somehow deemed insufficient.  So you'll have to be clearer about
what you're suggesting exactly, i.e., "what happens when".

João

[-- Attachment #2: Type: text/html, Size: 1267 bytes --]

bug#50959: [PATCH] Re: bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> In this thread, we've described many ways how that can be done.
> C-x o already "pays attention" to them, from day one, but that attention
> is somehow deemed insufficient.  So you'll have to be clearer about
> what you're suggesting exactly, i.e., "what happens when".

As an ELisp programmer, I'd want (and expect) that if I can successfully
use `st-bar` and `st-foo` in the current buffer, I will see them in the
possible completions at least when I complete `st-` and ideally in other
cases (e.g. it would be good for completion of `sba` to take `st-bar`
into account if `flex` completion style).


        Stefan

bug#50959: [PATCH] Re: bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 2027 bytes --]

On Wed, Oct 6, 2021 at 8:59 PM Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> > In this thread, we've described many ways how that can be done.
> > C-x o already "pays attention" to them, from day one, but that attention
> > is somehow deemed insufficient.  So you'll have to be clearer about
> > what you're suggesting exactly, i.e., "what happens when".
>
> As an ELisp programmer, I'd want (and expect) that if I can successfully
> use `st-bar` and `st-foo` in the current buffer, I will see them in the
> possible completions at least when I complete `st-` and ideally in other
> cases (e.g. it would be good for completion of `sba` to take `st-bar`
> into account if `flex` completion style).
>

1) If by "use in the current buffer" you mean "insert in the current buffer"
then your wish makes full sense to me and is already granted, fully, via
C-M-i.

2) If you're talking about querying the single global database of symbols
using M-x describe-symbols, then you must address symbols by their
full name, as you have always done. It's easy to do so:

2a) by placing point under the form of interest.  If it happens to be
shorthand
it will automatically be picked up.  Shorthands are now visually highlighted
as such.

2b) by typing the shorthands as such and typing TAB to complete them to
the full symbol name.  This is only if you enter the query interface from
the buffer where the shorthands exist.

I think this is more than sufficient. I've worked for years and years with
a CL system that only does 1) and 2a). I've worked side-by-side with
people using it. I even forked and maintained such a system.  I've never
seen requests or complaints for anything more in this regard.  Sure, we
are all different and some of us are more demanding, and that's a good
thing, but it's a fact that the vast majority of Emacs users haven't used
a namespacing system in Elisp for a long time, possibly ever.  Shouldn't
we give this first  conservative approach a shot first?

João

[-- Attachment #2: Type: text/html, Size: 2755 bytes --]

bug#50959: [PATCH] Re: bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[João Távora]

João Távora <joaotavora@gmail.com> writes:

> João Távora <joaotavora@gmail.com> writes:
>
>> On Sat, Oct 2, 2021 at 1:30 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
>>> > 0. no integration
>>> >
>>> > 1. This is the current integration.  I.e. when C-h o is pressed on the
>>> >    symbol the global name is discovered and used as the default.  This
>>> >    is how SLIME work with CL's namespacing system.  SLIME is a very well
>>> >    tested and widely appreciated Common Lisp IDE for Emcas.
>>> >
>>> > 2. The shorthands from the buffer where the minibuffer was entered are
>>> >    _not_ in the completions list, but typing one of them interns the
>>> >    symbol with those shorthands present, so you get the desired result.
>>> >    This would fix Phil's visually-copy-and-type scenario.
>>> >
>>> > 3. The completion list would be augmented with the shorthands from
>>> >    the buffer where the minibuffer was entered from.
>
> Hello Eli,
>
> I've implemented a variation on 2 based on the later suggestion you
> gave in emacs-devel:
>
>> That is, the user types "C-h o s-foo <SOME KEY SEQUENCE>" and that
>> replaces s-foo with the expansion, the "real" symbol.  Is that
>> feasible?
>
> Yes, it is.  <SOME KEY SEQUENCE> is, of course, TAB.  Here is a patch
> for people to try out which I will push in a few days time if there are
> no objections.  Cc-ing completion-style specialist Stefan.
>
> Patch just below,

I've stashed this patch in branch scratch/bug-50959-fix.

The reason is Stefan's strong objections to it.

IIUC, Stefan, who originally designed the completion-style mechanism
would seem to favour a scheme where a new completion table category
'symbol' is linked to a new 'abbrev' style (instead of a 'shorthand'
style).  The 'abbrev' style would then query the table for an
appropriate source of abbreviations, which, in the case of table of the
'symbol' category, would be extracted from 'read-symbol-shorthands'.
Stefan do you confirm this?

If so, the end result would be the same and a functional level, and
Phil's original request would be satisfied.  I don't strongly object to
the above more complex approach, but I don't have much time right now to
implement those extra indirections either (not that they're rocket
science, but they need at least good formats, good names, good
docstrings, etc.), so I'll leave it up to you guys to figure out what to
do here.

João

bug#50959: [PATCH] Re: bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: Phil Sainty <psainty@orcon.net.nz>,  50959@debbugs.gnu.org,  Stefan
>  Monnier <monnier@iro.umontreal.ca>,  rms@gnu.org
> Date: Sun, 10 Oct 2021 11:46:11 +0100
> 
> I'll leave it up to you guys to figure out what to do here.

I already did, and that's why I asked you to install that change.
With it on a scratch branch, the problem is without any solution
whatsoever.  This is bad for users, IMNSHO.

And that is all I have to say on this matter.

bug#50959: [PATCH] Re: bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: João Távora <joaotavora@gmail.com>
>> Cc: Phil Sainty <psainty@orcon.net.nz>,  50959@debbugs.gnu.org,  Stefan
>>  Monnier <monnier@iro.umontreal.ca>,  rms@gnu.org
>> Date: Sun, 10 Oct 2021 11:46:11 +0100
>> 
>> I'll leave it up to you guys to figure out what to do here.
>
> I already did, and that's why I asked you to install that change.
> And that is all I have to say on this matter.

Done.  When someone strongly objects, I always seek confirmation from
the head maintainer before overriding them.

Also marking bug done.  If someone disagrees or finds bugs, let me know.

João

bug#50959: [PATCH] Re: bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: psainty@orcon.net.nz,  50959-done@debbugs.gnu.org,
>   monnier@iro.umontreal.ca,  rms@gnu.org
> Date: Sun, 10 Oct 2021 14:39:03 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: João Távora <joaotavora@gmail.com>
> >> Cc: Phil Sainty <psainty@orcon.net.nz>,  50959@debbugs.gnu.org,  Stefan
> >>  Monnier <monnier@iro.umontreal.ca>,  rms@gnu.org
> >> Date: Sun, 10 Oct 2021 11:46:11 +0100
> >> 
> >> I'll leave it up to you guys to figure out what to do here.
> >
> > I already did, and that's why I asked you to install that change.
> > And that is all I have to say on this matter.
> 
> Done.  When someone strongly objects, I always seek confirmation from
> the head maintainer before overriding them.

Thank you!

bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]


  > 2. The shorthands from the buffer where the minibuffer was entered are
  >    _not_ in the completions list, but typing one of them interns the
  >    symbol with those shorthands present, so you get the desired result.
  >    This would fix Phil's visually-copy-and-type scenario.

Would someone please give an explained example to make this concretely clear?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[Phil Sainty]

Hi João,

On 2021-10-03 00:02, João Távora wrote:
> In other words, Phil, if you wish me to do anything about this bug, you
> must explain exactly what you are doing and what you expect.  What does
> it mean precisely for "Emacs to recognise the shorthand symbols
> generally".  Currently it "recognizes" them when the buffer where they
> are setup is current.

I'm just the messenger on this occasion.  I was describing to Eli how
shorthands exhibit the exact same problem that was objectionable in the
"nameless" approach, and he asked me to raise it as a bug.

I think that the shorthand symbols will need to be interned to avoid
this (or at least I can't think of any other solution which avoids the
problem).  Perhaps define them as aliases -- or even obsolete aliases,
if the long-term preference is for other libraries to migrate from s-*
and the likes to using the proper names?


-Phil

bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[Eli Zaretskii]

> Date: Sun, 03 Oct 2021 00:53:55 +1300
> From: Phil Sainty <psainty@orcon.net.nz>
> Cc: Eli Zaretskii <eliz@gnu.org>, 50959@debbugs.gnu.org
> 
> Hi João,
> 
> On 2021-10-03 00:02, João Távora wrote:
> > In other words, Phil, if you wish me to do anything about this bug, you
> > must explain exactly what you are doing and what you expect.  What does
> > it mean precisely for "Emacs to recognise the shorthand symbols
> > generally".  Currently it "recognizes" them when the buffer where they
> > are setup is current.
> 
> I'm just the messenger on this occasion.  I was describing to Eli how
> shorthands exhibit the exact same problem that was objectionable in the
> "nameless" approach, and he asked me to raise it as a bug.

Let me translate: is "C-h" command the only use case you have in mind
where you saw a problem with shorthands?  If there are others, and
they involve Emacs's internal functionalities (as opposed to, say,
searching with Grep or some other text-oriented tool), please describe
them.

> I think that the shorthand symbols will need to be interned to avoid
> this (or at least I can't think of any other solution which avoids the
> problem).

They are already, or something very similar.

bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[Phil Sainty]

On 2021-10-03 01:10, Eli Zaretskii wrote:
> Let me translate: is "C-h" command the only use case you have in mind
> where you saw a problem with shorthands?  If there are others, and
> they involve Emacs's internal functionalities (as opposed to, say,
> searching with Grep or some other text-oriented tool), please describe
> them.

I think it's literally anything that hasn't has custom support added
as part of the shorthands feature?

E.g.:

* describe-function|-variable|-face|-symbol
* apropos|-command|-variable|-user-option|-symbol
* customize-apropos|-face|-variable|-option
* info-lookup-symbol
* execute-extended-command
* anything using read-command|-variable|-face-name
* anything using these interactive codes:

a -- Function name: symbol with a function definition.
C -- Command name: symbol with interactive function definition.
      This skips events that are integers or symbols.
S -- Any symbol.
v -- Variable name: symbol that is ‘custom-variable-p’.

etc, etc...

Basically everything?


-Phil

bug#50959: 28.0.50; Shorthand symbols are unknown to Emacs

[João Távora]

On Sat, Oct 2, 2021 at 1:37 PM Phil Sainty <psainty@orcon.net.nz> wrote:
>
> On 2021-10-03 01:10, Eli Zaretskii wrote:
> > Let me translate: is "C-h" command the only use case you have in mind
> > where you saw a problem with shorthands?  If there are others, and
> > they involve Emacs's internal functionalities (as opposed to, say,
> > searching with Grep or some other text-oriented tool), please describe
> > them.
>
> I think it's literally anything that hasn't has custom support added
> as part of the shorthands feature?
>
> E.g.:
>
> * describe-function|-variable|-face|-symbol
> * apropos|-command|-variable|-user-option|-symbol
> * customize-apropos|-face|-variable|-option
> * info-lookup-symbol
> * execute-extended-command
> * anything using read-command|-variable|-face-name
> * anything using these interactive codes:
>
> a -- Function name: symbol with a function definition.
> C -- Command name: symbol with interactive function definition.
>       This skips events that are integers or symbols.
> S -- Any symbol.
> v -- Variable name: symbol that is ‘custom-variable-p’.
>
> etc, etc...
>
> Basically everything?

As it is clear, that's not the case.  These things have and will always
deal with the global table of symbols obarray.  These commands operate on
symbols.  Shorthands are _not_ symbols! They are not! They are a textual
indirection to symbols, which -- as the manual explained long before I touched
it -- are objects composed of four things, blablabla.

Maybe this helps you think: it's just like (intern (concat "foo"
"bar")) is another
type of indirection to a symbol. A run-time indirection.  Shorthands
are read-time
indirections.

So: when the commands you reference are invoked in the buffer where
shorthands exist with point on the shorthand you which to describe, lookup, etc
all those functions and the interactive codes do the right thing.  What is that?
They follow the indirection to the symbol and operate on the symbol,
as they always
have.

If you find some case where they DON'T  follow this indirection, where
they clearly
COULD reasonably follow it then that is a bug and or feature request
for  shorthands.
But you should explain also why think it is reasonable.  Consider how SLIME (and
also proprietary Common Lisp IDE's, I think) have dealt with this issue (search
"package local nicknames" in your favourite search engine).

Soon I will propose that we font-lock shorthands specially (or at least make it
optional).  That should be easy to do, and should, in my opinion, extinguish at
least some of your anguish.

João

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

Phil Sainty <psainty@orcon.net.nz> writes:

> On 2021-10-02 19:45, Eli Zaretskii wrote:
>> There's a huge difference between breaking literal searches for
>> symbols by text-searching tools, and breaking basic Emacs commands
>> because the name the user sees and types is not known to Emacs.
>
> But shorthands does *both* of those things.
>
> The name the user sees is "s-foo".
>
> The name known to Emacs is "string-library-foo" (or whatever).
>
> The user types "C-h o s-foo RET" and Emacs says "no match".

That's only when the user types that in the minibuffer and doesn't
associate in any way to the buffer where you set up that particular
shorthand (remember, shorthands aren't global: that's the point).  Much
like if I type 'import foo as bar' in my Python of JavaScript program
and then go search for 'bar' I don't get the results for 'foo'.

But (have you seen the animated gif?)  if you type 'C-h o' with point ON
TOP OF 's-foo', then M-x describe-symbol will be prepolulated with
string-library-foo, and need only type RET.

The same thing with ElDoc, or M-., or C-M-i when used *in the buffer*
that you are looking at.

So in with my suggestion you type half the characters and get the
correct choice.  If however you like to to copy what you see visually
from the buffer to the minibuffer, say, to warm your fingers, then a
visual strategy similar to the "nameless.el" package you have promoted
here can be devised.  You'd never never SEE "s-foo" (though it exists in
the buffer).  I, like others have said, think that is bit chaotic, but
not absurd.  Then you would be able to type C-h s t r i n g - l i b r a
r y - f o o RET

João

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > The name the user sees is "s-foo".
  > >
  > > The name known to Emacs is "string-library-foo" (or whatever).
  > >
  > > The user types "C-h o s-foo RET" and Emacs says "no match".

  > That's only when the user types that in the minibuffer and doesn't
  > associate in any way to the buffer where you set up that particular
  > shorthand (remember, shorthands aren't global: that's the point).  Much
  > like if I type 'import foo as bar' in my Python of JavaScript program
  > and then go search for 'bar' I don't get the results for 'foo'.

This is the intentional behavior: to avoid putting `s-foo' in the
global Emacs Lisp namespace.  It will be available in those programs
which do (require 's).

Other programs will be able to use `s-foo' for some other meaning
and there will be no conflict.
-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > The name known to Emacs is "string-library-foo" (or whatever).
  > >
  > > The user types "C-h o s-foo RET" and Emacs says "no match".

If the user switches to a buffer which contains a Lisp program that
requires s, then types C-h o s-foo RET, or C-h f s-foo RET, I think it
would make sense for it to find s-foo.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Gregory Heytings]

>>> The name known to Emacs is "string-library-foo" (or whatever).
>>>
>>> The user types "C-h o s-foo RET" and Emacs says "no match".
>
> If the user switches to a buffer which contains a Lisp program that 
> requires s, then types C-h o s-foo RET, or C-h f s-foo RET, I think it 
> would make sense for it to find s-foo.
>

That's yet another problem introduced by that system.  Suppose a user has 
an Emacs session opened, with two windows: the top one contains a buffer 
with a (require 'string-library) and a elisp-shorthands (("s-" . 
"string-library-")), and the bottom one contains a text buffer.  Now:

(1) If the currently selected window is the top one:

(1.1) If point is on s-foo, C-h o will suggest string-library-foo.  So far 
so good.

(1.2) If the user is not on s-foo, C-h o s - f o o RET will currently 
return "No match".  Unless of course an s-foo function or variable defined 
is already defined elsewhere, in which case it's that function or variable 
that will be found, and the user will get a possibly wrong information. 
That could perhaps be fixed by copying the value of elisp-shorthands in 
the minibuffer buffer-local variables, in which case s-foo will 
automatically be expanded into string-library-foo.  The same would be 
necessary for M-:, C-h f, C-h v, and perhaps M-x.

(1.3) The fix in (1.2) would however break another use case: the buffer in 
the top window was entered with M-., the function that is visible does not 
use any of the string-library functions, which means that the user does 
not know that s-foo expands into string-library-foo.  What they actually 
want is perhaps to see the documentation of the s-foo function or variable 
that they know and that is defined elsewhere, and C-h o s - f o o RET will 
display the documentation of string-library-foo instead.

(1.4) The fix in (1.2) would also not be enough when recursive minibuffers 
enter the scene, see (2).

(1.5) If the fix in (1.2) is not used for M-x, M-x s-frobnicate will not 
expand into string-library-frobnicate.  If the fix in (1.2) is used for 
M-x, M-x s-mode will become M-x string-library-mode.  Both are obviously 
problematic.

(2) If the currently selected window is not the top one (it is either in 
the minibuffer or in the bottom window), C-h o s - f o o will again return 
"No match", unless of course an s-foo function or variable is already 
defined elsewhere, in which case the user will get a wrong information.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > The name known to Emacs is "string-library-foo" (or whatever).
>   > >
>   > > The user types "C-h o s-foo RET" and Emacs says "no match".
>
> If the user switches to a buffer which contains a Lisp program that
> requires s, then types C-h o s-foo RET, or C-h f s-foo RET, I think it
> would make sense for it to find s-foo.

The system is behaving as designed.

In the sequence C-h o s-foo, the last 5 characters typed are not the
name of a symbol, they are the name of a shorthand (a shorthand is _not_
a symbol) that you are seeing (in the sense of "with your eyes"") in
some buffer.  Note this:

* The display of that shorthand is visually different from the display
  of any other symbol-referring form.

  In particular the prefix and the suffix are font-locked with different
  colors.  Thus, there is little reason to be mistaken that 's-foo' is
  the full name of a symbol.

  There is even less reason to be mistaken than when using using
  'cl-letf' and 'cl-flet', or level 'let' which create local
  definitions.

* If, as you write, you have switched to the buffer where the shorthand
  's-foo' occurs and you type the beginning of the C-h o sequence when
  point is ON that shorthand, the Lisp reader will follow the shorthand
  to the `string-library-foo' symbol and offer to display help on that.

* The C-h o and similar commands operate on symbols and are global.

  They display the same list of symbols irrespective of the locus of
  there invocation.  This makes sense, because Emacs operates with a
  single global collection of symbols.  Shorthands haven't changed that,
  and this was never the intention.

* C-h o and similar commands _could_ theoretically be changed to
  complement that list of symbols with the list of shorthands to symbols
  of the buffer.

  That would be a non-trivial effort and would bring grave confusion.
  That's because, as you have well written elsewhere, the Lisp reader
  follows 's-foo' to 'string-library-foo' on this buffer but may follow
  it to 'system-library-foo' in another buffer.

  That's what makes Shorthands a namespacing system (contrary to what
  some mistaken minds are keen on insiting): there is more than one way
  to refer to the same thing _depending_ on context and the _same name_
  may refer to different tihngs, depending on context.

  The buffer is the context here.

* So, if we follow our first instincts (they were my first instincts,
  too!), it means that the exact same Help input in two different
  situations could bring about different results.

Again, it is theoretically possible to do as you suggest, but the effort
required isn't trivial and, more importantly, the confusion generated is
much, much greater.

João

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: psainty@orcon.net.nz,  acm@muc.de,  eliz@gnu.org,  emacs-devel@gnu.org
> Date: Mon, 04 Oct 2021 16:44:26 +0100
>
> The system is behaving as designed.

Famous last words ;-)

Seriously, though: IME, this is not a popular response to user
requests.

> In the sequence C-h o s-foo, the last 5 characters typed are not the
> name of a symbol, they are the name of a shorthand (a shorthand is _not_
> a symbol) that you are seeing (in the sense of "with your eyes"") in
> some buffer.

If it is impossible, or hard, or inappropriate (in your opinion) to
support s-foo in this use case, would it be possible to have a special
command that would expand the shorthand in the minibuffer?  That is,
the user types "C-h o s-foo <SOME KEY SEQUENCE>" and that replaces
s-foo with the expansion, the "real" symbol.  Is that feasible?

> * So, if we follow our first instincts (they were my first instincts,
>   too!), it means that the exact same Help input in two different
>   situations could bring about different results.

That already happens in any number of situations, and shouldn't
prevent us from adding one more.  The most trivial example is
buffer-local variables; another example is mode-specific commands (a
recent addition in Emacs 28).  And there are many more: a user who
expects identical Emacs behaviors in different buffers will be
extremely disappointed.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

>> The system is behaving as designed.
> Famous last words ;-)

:-)

> Seriously, though: IME, this is not a popular response to user
> requests.

I understand, I really do.  But it's true: the system is behaving as
designed.

That surely doesn't mean one can't design it differently, or augment it
in the direction of those user requests.

It _is_ possible (I hope I left that clear).  I just don't think it is a
good idea, for the reasons I tried to explain.

My intuition is that people that are making this request aren't
imagining how it would behave if you had two files in your Emacs session
(or even across different sessions).

   read-symbol-shorthands: (("s-" . "magnar-string-"))

And another had:

   read-symbol-shorthands: (("s-" . "joes-system-"))

It is my belief that this would bring some level of confusion.  The user
would have to consider very well the locus of her 'C-h o' request.  I
don't usually do that, not for functions at least (for variables, yes,
to a certain extent).

>> In the sequence C-h o s-foo, the last 5 characters typed are not the
>> name of a symbol, they are the name of a shorthand (a shorthand is _not_
>> a symbol) that you are seeing (in the sense of "with your eyes"") in
>> some buffer.
>
> If it is impossible, or hard, or inappropriate (in your opinion) to
> support s-foo in this use case, would it be possible to have a special
> command that would expand the shorthand in the minibuffer?  That is,
> the user types "C-h o s-foo <SOME KEY SEQUENCE>" and that replaces
> s-foo with the expansion, the "real" symbol.  Is that feasible?

When using some "completion styles" (which see), the completion system
already does that, to a large extent.  The 'flex' completion style, for
example, would do that: it is quite likely that 's-foo' would expand to
'magnar-string-foo'.

Note, however, that that expansion is based on the simple string
relation between 's-foo' and 'magnar-string-foo', where the former is a
subsequence of the latter.  If the shorthand were:

  read-symbol-shorthands: (("x-" . "magnar-string-"))

then, 'x-foo' would _not_ expand to 'magnars-string-foo' using the
'flex' completion style.

If you nevertheless wish that 'x-foo' expand to 'magnars-string-foo'
BECAUSE you invoked 'C-h o' from a buffer where those shorthands are
setup, then you are again creating some locality in the 'C-h o'
interface.  But I presume you are prepared to accept that.

So, it _is_ possible to code that.  Perhaps using a buffer-local
completion-style.  At first sight, this new idea of yours is more
elegant than the first request, because it expresses the true vehicular
nature of shorthands much more accurately.  It also seems simpler to
implement, at first sight.  With heavy emphasis on the two "at first
sight"s ;-).

>> * So, if we follow our first instincts (they were my first instincts,
>>   too!), it means that the exact same Help input in two different
>>   situations could bring about different results.
>
> That already happens in any number of situations, and shouldn't
> prevent us from adding one more.  The most trivial example is
> buffer-local variables; another example is mode-specific commands (a
> recent addition in Emacs 28).  

Interesting, I wasn't aware of mode-specific commands.  Can you give one
example of one or two commands specific to a mode?  Does it mean that
the listing itself, i.e. the collection of completions of given by 'C-h
x' is different according to the buffer?

> And there are many more: a user who expects identical Emacs behaviors
> in different buffers will be extremely disappointed.

In what regards _listing_ of symbols brought about by C-h o, C-h f and
C-h x, etc, I will be one of these "disappointed" users.  I'm OK with
seeing the final product of my C-h v request say that 'foo' 's value is
42 in the buffer 'bar.el', because it's so very clearly spelled out
there.  But I'd find it odd not to see a variable, function or anything
listed there at all.

Anyway "disappointed", doesn't mean "extremely disappointed" because
being a catastrophist about such things isn't really my thing.  If this
is the direction where you and others envision Emacs going, I'm on
board!

João

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: rms@gnu.org,  psainty@orcon.net.nz,  acm@muc.de,  emacs-devel@gnu.org
> Date: Mon, 04 Oct 2021 18:43:08 +0100
> 
> >> * So, if we follow our first instincts (they were my first instincts,
> >>   too!), it means that the exact same Help input in two different
> >>   situations could bring about different results.
> >
> > That already happens in any number of situations, and shouldn't
> > prevent us from adding one more.  The most trivial example is
> > buffer-local variables; another example is mode-specific commands (a
> > recent addition in Emacs 28).  
> 
> Interesting, I wasn't aware of mode-specific commands.  Can you give one
> example of one or two commands specific to a mode?

Look for commands that have 'interactive' with an additional argument
that is a mode symbol.

> Does it mean that the listing itself, i.e. the collection of
> completions of given by 'C-h x' is different according to the
> buffer?

According to the buffer's mode, yes.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Gregory Heytings]

>
> The system is behaving as designed.
>

And the question is: was it properly designed?

>
> That's what makes Shorthands a namespacing system (contrary to what some 
> mistaken minds are keen on insiting): [A] there is more than one way to 
> refer to the same thing _depending_ on context and [B] the _same name_ 
> may refer to different tihngs, depending on context.
>

Interesting.  A few days ago you still believed that name spaces are [A], 
and one of the "mistaken minds" told you that they are not [A], but [B]. 
Now you believe that they are both [A] and [B].  Perhaps in a week you'll 
realize that they are [B], not [A].

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 537 bytes --]

On Mon, Oct 4, 2021, 19:38 Gregory Heytings <gregory@heytings.org> wrote:

>
Interesting.  A few days ago you still believed that name spaces are [A],
> and one of the "mistaken minds" told you that they are not [A], but [B].
> Now you believe that they are both [A] and [B].  Perhaps in a week you'll
> realize that they are [B], not [A].
>

Gregory, the statements you make are false. You are only trolling yourself
at this point, and I have no time to indulge your fantasies and
persecutions.

Have a good night,
João

[-- Attachment #2: Type: text/html, Size: 1177 bytes --]

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 834 bytes --]


>> Interesting.  A few days ago you still believed that name spaces are 
>> [A], and one of the "mistaken minds" told you that they are not [A], 
>> but [B]. Now you believe that they are both [A] and [B].  Perhaps in a 
>> week you'll realize that they are [B], not [A].
> 
> the statements you make are false.
>

If you say so.  Anyone can look at the archives.

>
> I have no time to indulge your fantasies and persecutions.
>

I fear you rather don't want to discuss with those who point you to clear 
weaknesses in the approach in which you believe and which you've 
implemented.  Which is understandable, but perhaps not the best way to 
convince those who expressed their doubts.  And telling them that they are 
"trolls" and "mistaken minds" also does not help, of course.

Have a good day||night, too.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I implore each of you to be more kind in your postings.
If someone else does not see the strength of your logic,
it is quite likely a misunderstanding or miscommunication.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

Gregory Heytings <gregory@heytings.org> writes:

>>> Interesting.  A few days ago you still believed that name spaces
>>> are [A], and one of the "mistaken minds" told you that they are not
>>> [A], but [B]. Now you believe that they are both [A] and [B]. 
>>> Perhaps in a week you'll realize that they are [B], not [A].
>> the statements you make are false.
> If you say so.  Anyone can look at the archives.

OK, so let's:

In https://lists.gnu.org/archive/html/emacs-devel/2021-10/msg00125.html,
I wrote:

>> For me "namespaces" are about allowing the same thing to be invoked
>> by different names, depending on context.

It's clear to that if things can have two names _depending on context_
then, _in two different contexts_, the same name might be used for
different things.  Is that corollary not clear from my sentence?  I'm
sorry I wasn't trying to give a scientific definition.

This isn't "the complete opposite of namespacing", or "aliasing" as you
defined it, because aliasing (at least as it exists in Emacs) is global,
i.e. not depending on context.  This is where your mind is mistaken.

Not to mention that bikeshedding what namespaces are is completely
useless to the discussion at hand.  Shorthands are a reality, they have
a real implementation and they do what they do to solve specific Emacs
problems and is what I explained many times and is clear in the manual.

_You_ not I, go out of your way to write an email that twists my
statement so that it seems that I contradicted myself and the sheer
factual reality of the feature I implemented.  That is false.  And then
you say that "in a week" I'll say something else.  That's intentional
and gratutious harrassment, aka "trolling".  I have no time to indulge
that fantasy.

João

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > That's what makes Shorthands a namespacing system (contrary to what
    > some mistaken minds are keen on insiting):

The first part is a statement about Emacs, and it's a fine thing to say.

But the parenthetical statement is a personal criticism of some
person or persons.  Do you see what I mean?

I suggest it would have been better to refrain from saying that.  It's
better to refrain from criticizing any person, and talk only about the
topic.  Also, you have a better chance of convincing people that way.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > * C-h o and similar commands _could_ theoretically be changed to
  >   complement that list of symbols with the list of shorthands to symbols
  >   of the buffer.

  >   That would be a non-trivial effort and would bring grave confusion.

I don't see that it would cause grave confusion.

  >   That's because, as you have well written elsewhere, the Lisp reader
  >   follows 's-foo' to 'string-library-foo' on this buffer but may follow
  >   it to 'system-library-foo' in another buffer.

That is true.

If the extension I proposed for C-h o is made, then C-h o s-foo RET
would present info on 'string-library-foo' in one buffer, and about
'system-library-foo' in another buffer.  But if the help buffer
explains this clearly, it will make sense to the user, and it will be
useful.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1408 bytes --]

On Tue, Oct 5, 2021, 22:20 Richard Stallman <rms@gnu.org> wrote:

  >   That would be a non-trivial effort and would bring grave confusion.
>
> I don't see that it would cause grave confusion.
>

It would for me. I'd have to keep in mind exactly where I'm invoking help
commands from. I've never done that.

>
>   >   That's because, as you have well written elsewhere, the Lisp reader
>   >   follows 's-foo' to 'string-library-foo' on this buffer but may follow
>   >   it to 'system-library-foo' in another buffer.
>
> That is true.
>
> If the extension I proposed for C-h o is made, then C-h o s-foo RET
> would present info on 'string-library-foo' in one buffer, and about
> 'system-library-foo' in another buffer.  But if the help buffer
> explains this clearly, it will make sense to the user, and it will be
> useful.
>

Yes, you're right. If we're clear about what we're sitting the user, it's
not so bad. That option is on the table.

Another simpler, less "revolutionary" option (can't find the right word) is
to do what Eli suggested, having 'C-h f s-foo TAB' expand to the actual
symbol name, according to the buffer, using the normal completion
mechanisms of Emacs. Then the usual invariant of 'C-h f' showing global
symbols would be kept.

We can upgrade from Eli's suggestion (which would seem simpler to
implement), to your suggestion later.

João

>

[-- Attachment #2: Type: text/html, Size: 2417 bytes --]

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Stefan Monnier]

> It would for me.  I'd have to keep in mind exactly where I'm invoking help
> commands from.  I've never done that.

Yet, I think it's very natural.  When you do `C-x o` the current symbol
under point is used as a default choice, so it makes sense that it would
obey the rules that apply to the buffer from which it was invoked.

That's already the case for variable names when it comes to printing
their (buffer-local) value, of course.

More importantly, I think conflicts will be quite rare, so you could
decide to not use this functionality and you'd likely not be
significantly affected.


        Stefan

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

On Tue, Oct 5, 2021 at 11:16 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> > It would for me.  I'd have to keep in mind exactly where I'm invoking help
> > commands from.  I've never done that.
>
> Yet, I think it's very natural.  When you do `C-x o` the current symbol
> under point is used as a default choice, so it makes sense that it would
> obey the rules that apply to the buffer from which it was invoked.

I don't think it's natural at all.  A most common case of me is doing a
C-h f from within the *Help* buffer itself, to find another related
function.  Should I keep track of where all the C-h f chains started to
use those shorthands in those C-h f searches?  That's madness, at least
for me.

To your example, you'll notice that `C-x o` on a shorthand under point
works.  The shorthand is immediately read into the symbol and the
symbol's full name shows in the default.  As I think you understand, a
shorthand is _not_ a symbol, but it's read into a symbol.  So the
listing of symbols is unaffected: it remains a listing of symbols,
coming from the only source of symbols, obarray.  (As Drew noted
elsewhere, you can have obarray store symbols for other purposes, and
that's fine, but the symbols you use in Elisp programs all live in the
global obarray).

> That's already the case for variable names when it comes to printing
> their (buffer-local) value, of course.

Yes, but the "variableness" is a property of a symbol. I can find the
symbol 'foo' in any variable listing, from any buffer.  Eventually the
description eventually shows me that it doesn't have the value I
expected it to in that buffer, but it doesn't hide the symbol from me
depending on where I invoked it from.

> More importantly, I think conflicts will be quite rare, so you could
> decide to not use this functionality and you'd likely not be
> significantly affected.

I've pushed a patch for bug#50959 that implements Eli's suggestion using
completion styles.  The specific original request of that bug is fixed.

I've also experimented with actually listing shorthands in the symbols
listings and it's also possible (if a bit more tricky). If you select
one of them, the shorthand is interned to the symbol and things just
work. I wouldn't _like_ to see them there, it would confuse me a lot to
see them sometimes and not others.  So at least a knob to turn them off
would be needed.

João Távora

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Stefan Monnier]

> I don't think it's natural at all.  A most common case of me is doing a
> C-h f from within the *Help* buffer itself,

Then there won't be any read-symbol-shorthands installed and you won't
be affected either way.

> to find another related function.  Should I keep track of where all
> the C-h f chains started to use those shorthands in those C-h
> f searches?  That's madness, at least for me.

I don't think anyone suggested tracking such chains.


        Stefan

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 885 bytes --]

On Wed, Oct 6, 2021 at 1:21 PM Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> > I don't think it's natural at all.  A most common case of me is doing a
> > C-h f from within the *Help* buffer itself,
>
> Then there won't be any read-symbol-shorthands installed and you won't
> be affected either way.
>

Yes, but if I reach that buffer from a query for s-foo, which was
in the completions list, and then decide, from there, that I want
s-bar instead, I'll be confused if it isn't in the completion list.


> > to find another related function.  Should I keep track of where all
> > the C-h f chains started to use those shorthands in those C-h
> > f searches?  That's madness, at least for me.
>
> I don't think anyone suggested tracking such chains.
>

Right. But they could, legitimately, ask for this to supplant
the confusion explained above.

João

[-- Attachment #2: Type: text/html, Size: 1475 bytes --]

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Stefan Monnier]

> Yes, but if I reach that buffer from a query for s-foo, which was
> in the completions list, and then decide, from there, that I want
> s-bar instead, I'll be confused if it isn't in the completion list.

Yes, shorthands make life a bit more complex.
I don't think there's much we can do about that.

> Right. But they could, legitimately, ask for this to supplant
> the confusion explained above.

No, I think that would make life yet a bit more complex.  Better tell
them to go back to the original buffer before hitting `C-h o` if/when
they want that behavior.


        Stefan

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 580 bytes --]

On Wed, Oct 6, 2021 at 1:46 PM Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> > Yes, but if I reach that buffer from a query for s-foo, which was
> > in the completions list, and then decide, from there, that I want
> > s-bar instead, I'll be confused if it isn't in the completion list.
>
> Yes, shorthands make life a bit more complex.
> I don't think there's much we can do about that.
>

We can _not_ list shorthands in lists of symbols ;-)

Taking SLIME/SLY's cue at that, where people actually use
these things and they aren't all that complex.

João

[-- Attachment #2: Type: text/html, Size: 1009 bytes --]

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 781 bytes --]

On Wed, Oct 6, 2021, 14:16 João Távora <joaotavora@gmail.com> wrote:

>
> On Wed, Oct 6, 2021 at 1:46 PM Stefan Monnier <monnier@iro.umontreal.ca>
> wrote:
>
>> > Yes, but if I reach that buffer from a query for s-foo, which was
>> > in the completions list, and then decide, from there, that I want
>> > s-bar instead, I'll be confused if it isn't in the completion list.
>>
>> Yes, shorthands make life a bit more complex.
>> I don't think there's much we can do about that.
>>
>
> We can _not_ list shorthands in lists of symbols ;-)
>

To be clear, I mean _global_ lists. It's ok to list them in local ones,
such as C-M-i.

In fact, someday more local things such as lexically available variables
and functions can be listed there, maybe.

João

>

[-- Attachment #2: Type: text/html, Size: 1697 bytes --]

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Stefan Monnier]

> In fact, someday more local things such as lexically available variables
> and functions can be listed there, maybe.

M-TAB already lists lexically-available variables, yes.


        Stefan

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 449 bytes --]

On Wed, Oct 6, 2021 at 9:01 PM Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> > In fact, someday more local things such as lexically available variables
> > and functions can be listed there, maybe.
>
> M-TAB already lists lexically-available variables, yes.


... with a jit macroexpander and code walker, nice. I always
thought it was just heuristics. Got to remember never to
launch rockets in macro expansions I guess.

João

[-- Attachment #2: Type: text/html, Size: 856 bytes --]

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Stefan Monnier]

> ... with a jit macroexpander and code walker, nice. I always
> thought it was just heuristics. Got to remember never to
> launch rockets in macro expansions I guess.

Depends if you like fireworks,


        Stefan

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Dmitry Gutov]

On 06.10.2021 01:15, Stefan Monnier wrote:
>> It would for me.  I'd have to keep in mind exactly where I'm invoking help
>> commands from.  I've never done that.
> Yet, I think it's very natural.  When you do `C-x o` the current symbol
> under point is used as a default choice, so it makes sense that it would
> obey the rules that apply to the buffer from which it was invoked.
> 
> That's already the case for variable names when it comes to printing
> their (buffer-local) value, of course.
> 
> More importantly, I think conflicts will be quite rare, so you could
> decide to not use this functionality and you'd likely not be
> significantly affected.

+1

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Another simpler, less "revolutionary" option (can't find the right word) is
  > to do what Eli suggested, having 'C-h f s-foo TAB' expand to the actual
  > symbol name, according to the buffer, using the normal completion
  > mechanisms of Emacs.

That does sound good.

For any system that established file-specific aliases for symbols,
even if it were like Common Lisp packages, it would happen that the
aliases that are meaningful in one buffer may not be meaningful in
another.  With Common Lisp packages, a certain package could be
imported into file A and not into file B, so some or all of its
symbols would get shorter names in A and not B.

So the help commands would face the challenge of how to deal with
that.

It seems to me that we can separate, at least in principle, the
question of how help commands to handling that situation from the way
that the aliasing is implemented.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 677 bytes --]

On Thu, Oct 7, 2021 at 11:21 PM Richard Stallman <rms@gnu.org> wrote:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Another simpler, less "revolutionary" option (can't find the right
> word) is
>   > to do what Eli suggested, having 'C-h f s-foo TAB' expand to the actual
>   > symbol name, according to the buffer, using the normal completion
>   > mechanisms of Emacs.
>
> That does sound good.


An implementation of this idea is awaiting comments over at bug#50959.

João

[-- Attachment #2: Type: text/html, Size: 1031 bytes --]

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Thu, 7 Oct 2021 23:24:15 +0100
> Cc: Phil Sainty <psainty@orcon.net.nz>, Alan Mackenzie <acm@muc.de>, Eli Zaretskii <eliz@gnu.org>, 
> 	emacs-devel <emacs-devel@gnu.org>
> 
>    > Another simpler, less "revolutionary" option (can't find the right word) is
>    > to do what Eli suggested, having 'C-h f s-foo TAB' expand to the actual
>    > symbol name, according to the buffer, using the normal completion
>    > mechanisms of Emacs.
> 
>  That does sound good.
> 
> An implementation of this idea is awaiting comments over at bug#50959.

I think you should install that.  But if you have specific aspects of
that you think need further scrutiny and discussion before installing,
please point out those specific aspects.

Thanks.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Stefan Monnier]

>> An implementation of this idea is awaiting comments over at bug#50959.
> I think you should install that.

I'm not sure if he may be referring to another patch (I haven't checked
the whole thread), but the patch I saw in there implements it with an
ad-hoc `completion-style`.

I consider this as an abuse of the notion of completion style (which
are supposed to be orthogonal to the actual completion data), so I'd
strongly recommend not to use this patch.

I could agree to the use of a new completion-style for it, but then the
code of the completion style should not be specific to
`read-symbol-shorthands`.  Instead it should offer a generic feature
usable by other completion tables (and the part specific to
`read-symbol-shorthands` would be in the completion table of `C-x o`
rather than in the completion style code).


        Stefan

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: João Távora <joaotavora@gmail.com>,
>  psainty@orcon.net.nz,  acm@muc.de, rms@gnu.org,  emacs-devel@gnu.org
> Date: Fri, 08 Oct 2021 08:58:55 -0400
> 
> >> An implementation of this idea is awaiting comments over at bug#50959.
> > I think you should install that.
> 
> I'm not sure if he may be referring to another patch (I haven't checked
> the whole thread), but the patch I saw in there implements it with an
> ad-hoc `completion-style`.
> 
> I consider this as an abuse of the notion of completion style (which
> are supposed to be orthogonal to the actual completion data), so I'd
> strongly recommend not to use this patch.

You'd have to convince me why it's abuse, because I'm not convinced
yet.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Dmitry Gutov]

On 08.10.2021 16:22, Eli Zaretskii wrote:
> You'd have to convince me why it's abuse, because I'm not convinced
> yet.

Completion styles are supposed to be reusable between tables.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 428 bytes --]

On Sun, Oct 10, 2021 at 10:06 PM Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 08.10.2021 16:22, Eli Zaretskii wrote:
> > You'd have to convince me why it's abuse, because I'm not convinced
> > yet.
>
> Completion styles are supposed to be reusable between tables.
>

And FTR this one is.  It's just particularly adapted for tables providing
symbol names, which are tables that declare the 'symbol' category.

João

[-- Attachment #2: Type: text/html, Size: 798 bytes --]

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> An implementation of this idea is awaiting comments over at bug#50959.
>> I think you should install that.
> I'm not sure if he may be referring to another patch (I haven't checked
> the whole thread), but the patch I saw in there implements it with an
> ad-hoc `completion-style`.

Yup, that is the one (it's the only patch there so far).

> I consider this as an abuse of the notion of completion style (which
> are supposed to be orthogonal to the actual completion data), so I'd
> strongly recommend not to use this patch.

What do you mean a style being orthogonal to the completion data? A
substring style will complete "oo" to "foobar" because it knows that
"oo" belongs in "foobar" according to the notion of substring, and
likewise flex will complete "fb" to "foobar" because that belonging is
also there, according to a certain "flex" notion.

So the "shorthand" style makes "x-bar" complete to "string-library-foo"
according to a certain notion of abbreviation based on "shorthands".

How is this not "orthogonal to the data".

> I could agree to the use of a new completion-style for it, but then the
> code of the completion style should not be specific to
> `read-symbol-shorthands`.  Instead it should offer a generic feature
> usable by other completion tables (and the part specific to
> `read-symbol-shorthands` would be in the completion table of `C-x o`
> rather than in the completion style code).

In practice, does this change anything in terms of behavior?

If not, then I suggest we push that patch and then augment it with this
generalization, in case we do really come to the conclusion that this
more generic scheme really is useful.

Currently, the `C-x o` table (and other using
help--symbol-completion-table) declares that it is of the 'symbol'
category.  Up to here I'd say you agree.

Then, a shorthand-considering completion style is associated with the
'symbol' category.  I think this is pretty clean.

Obviously you disagree.  So if your suggestion is to make a an "abbrev"
completion style which consults the table for the specific source of
abbreviations (which happens to be read-symbol-shorthands in the case of
help--symbol-completion-table), than I have no strong points against, in
principle.  It's a question of adding something else to the metadata I
think.

But isn't this a bit of overengineering?  And can't it be left for
'master', as it's kind of a new feature?

João

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Stefan Monnier]

>> I consider this as an abuse of the notion of completion style (which
>> are supposed to be orthogonal to the actual completion data), so I'd
>> strongly recommend not to use this patch.
>
> What do you mean a style being orthogonal to the completion data?

Completion styles are supposed to work meaningfully with "any"
completion table.  Some completion styles make more sense with some
tables than others, admittely (and some like the `backend` completion
style only do something useful with those completion tables setup to
take advantage of it), but the completion styles should be agnostic to
where the completion candidates come from (that's the job of completion
tables), and so far that's been true of all completion styles.

> A substring style will complete "oo" to "foobar" because it knows that
> "oo" belongs in "foobar" according to the notion of substring, and
> likewise flex will complete "fb" to "foobar" because that belonging is
> also there, according to a certain "flex" notion.

Right.  And your completion style uses `read-symbol-shorthands` as
a "style", but it makes no sense to apply `read-symbol-shorthands` to
something else than symbol names, so it can only ever be used for the
completion table which returns symbol names.

> So the "shorthand" style makes "x-bar" complete to "string-library-foo"
> according to a certain notion of abbreviation based on "shorthands".

Yes, and I can accept this principle, but it should not be tied to
a specific source of candidates (symbol names, in this case).

> How is this not "orthogonal to the data".

Does the above answer the question?

>> I could agree to the use of a new completion-style for it, but then the
>> code of the completion style should not be specific to
>> `read-symbol-shorthands`.  Instead it should offer a generic feature
>> usable by other completion tables (and the part specific to
>> `read-symbol-shorthands` would be in the completion table of `C-x o`
>> rather than in the completion style code).
> In practice, does this change anything in terms of behavior?

I don't think so.

> If not, then I suggest we push that patch and then augment it with this
> generalization, in case we do really come to the conclusion that this
> more generic scheme really is useful.

I find the current patch to be a hideous kludge completely subverting
the design, so I'd rather you fix the code first.  It doesn't have to be
difficult.  A bit like with the `backend` completion, you can just make
the `shorthand` style call the completion-table to do the
actual expansion, and thus move the part of the code specific to
`read-symbol-shorthands` to where it belongs (the completion table).

> Obviously you disagree.  So if your suggestion is to make a an "abbrev"
> completion style which consults the table for the specific source of
> abbreviations (which happens to be read-symbol-shorthands in the case of
> help--symbol-completion-table),

That's right.

> than I have no strong points against, in principle.  It's a question
> of adding something else to the metadata I think.

Indeed you can also put the info in the metadata for the `shorthand`
style to use it (instead of calling the completion table with another
argument).

> But isn't this a bit of overengineering?

No, it's just trying to keep the abstraction-breakage to a tolerable level.

> And can't it be left for 'master', as it's kind of a new feature?

I see no need to push any of this to `emacs-28`, so all my
recommendations are for `master` here.

On my end, I intend to implement what I consider is the "proper"
behavior, presumably making it conditional on some user config since you
seem to consider it strongly undesirable whereas I tend to consider it
strongly "The Right Thing".


        Stefan

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  psainty@orcon.net.nz,  acm@muc.de,
>   rms@gnu.org,  emacs-devel@gnu.org
> Date: Fri, 08 Oct 2021 11:55:47 -0400
> 
> Completion styles are supposed to work meaningfully with "any"
> completion table.  Some completion styles make more sense with some
> tables than others, admittely (and some like the `backend` completion
> style only do something useful with those completion tables setup to
> take advantage of it), but the completion styles should be agnostic to
> where the completion candidates come from (that's the job of completion
> tables), and so far that's been true of all completion styles.

That sounds like a principle (a.k.a. "dogma") that doesn't have to be
universally accepted.  E.g., what terrible things will happen if we
allow a style that is NOT agnostic to the source of the completion
candidates?

> > And can't it be left for 'master', as it's kind of a new feature?
> 
> I see no need to push any of this to `emacs-28`, so all my
> recommendations are for `master` here.

Then what do you suggest we do on the release branch?

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Stefan Monnier]

>> I see no need to push any of this to `emacs-28`, so all my
>> recommendations are for `master` here.
> Then what do you suggest we do on the release branch?

Absolutely nothing.  This is a very minor convenience issue and it has
no impact at all for now given the absence of code making use of that
new feature.  So I see absolutely no hurry to add such a feature to
Emacs-28 (as opposed to the hurry there was to add
`read-symbol-shorthands` itself).


        Stefan

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: joaotavora@gmail.com,  psainty@orcon.net.nz,  acm@muc.de,  rms@gnu.org,
>   emacs-devel@gnu.org
> Date: Fri, 08 Oct 2021 14:16:21 -0400
> 
> >> I see no need to push any of this to `emacs-28`, so all my
> >> recommendations are for `master` here.
> > Then what do you suggest we do on the release branch?
> 
> Absolutely nothing.  This is a very minor convenience issue

I disagree that it is minor.

> and it has no impact at all for now given the absence of code making
> use of that new feature.

Then the patch you don't like will not have any impact, either.

> So I see absolutely no hurry to add such a feature to
> Emacs-28 (as opposed to the hurry there was to add
> `read-symbol-shorthands` itself).

I do see a reason to hurry, because we want to release Emacs 28.1
soon.  So if there are no better proposals, we will stay with the
patch you don't like on the release branch.  It is fine with me not to
merge it to master, or do something else on master, though.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> So the "shorthand" style makes "x-bar" complete to "string-library-foo"
>> according to a certain notion of abbreviation based on "shorthands".
> Yes, and I can accept this principle, but it should not be tied to
> a specific source of candidates (symbol names, in this case).

But why?  What bad thing can happen if a specific style is tied to a
specific type of data?

Note _type_, not _source_ of data, which is indeed the table.  The style
can be used on any list of Lisp symbols, regardless of whether they
represent function names, variables names, non-Elisp entities, etc.
That's why this style, as many others, is tied to a category, not a
table.

If there's a mechanism for tables prefer styles via categories, and it's
useful, I don't see why it shouldn't be used.  We need a "backend"
completion style for LSP and SLY, for example, and that doesn't fit
neatly in your delicate frame at all.

> I find the current patch to be a hideous kludge completely subverting
> the design, so I'd rather you fix the code first.

Since we're going for imagery, i think it's this artificial purity which
is a bit malodorous.  

>> But isn't this a bit of overengineering?
> No, it's just trying to keep the abstraction-breakage to a tolerable
> level.

Abstractions should only be kept as long as they're useful to model
behaviour efficiently.  That's not the case with this one.  As we know,
good design is hard to use incorrectly.  So the mere fact that it was
easy to do my patch in an inteligible way and the fact that the
"cleaner" alternative is to do much of the same but with a contrived
mound of indirections is evidence that that particular abstraction that
you're idealizing is relatively weak.

But I agree with Eli, if you want to make the "abbrev" style to replace
"shorthand" (and hopefully with more real use cases than just the
shorthands) on master, I don't object.

João

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The first use-case is to do with the "s" library, and finding
  > a way to rename all of that code with a longer prefix without
  > requiring other libraries currently requiring "s" to change...

Not solely with the `s' library; there are a few others.
But basically I think you are right about this.
And this is indeed the main intended use in Emacs itself.

  > The second use-case, and the one I think will prove to be FAR
  > more common if this goes ahead, is this: Some people simply want
  > to read and write shorter symbol names in their code.

I would not object to making rules about naming or usage conventions
for the shorter names, in the second use-case.

We can't follow those conventions for the first use-case.  They would
not work.  But it is ok to treat the two use-cases differently.

  > Instead, we would need two things:

  > 1. A way of displaying long symbols in the desired short form,
  >     such that the buffer contains the actual symbol, but the
  >     user sees the short symbol (i.e. some kind of replacing
  >     display).

  > 2. Something analogous to abbrev which recognises when someone
  >     starts typing a symbol with one of the configured short
  >     prefixes, and expands it to be the full name (but per (1)
  >     visually displayed as the short form that they typed).

I think #2 might be a good idea, but #1 would lead to horrible
confusion.  If the screen does not match the buffer, that is chaos.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

On Fri, Oct 1, 2021 at 11:38 PM Richard Stallman <rms@gnu.org> wrote:

>   > 2. Something analogous to abbrev which recognises when someone
>   >     starts typing a symbol with one of the configured short
>   >     prefixes, and expands it to be the full name (but per (1)
>   >     visually displayed as the short form that they typed).
>
> I think #2 might be a good idea, but #1 would lead to horrible
> confusion.  If the screen does not match the buffer, that is chaos.

#2 already exists.  It is C-M-i (aka M-x completion-at-point).

João

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

>   > The second use-case, and the one I think will prove to be FAR
>   > more common if this goes ahead, is this: Some people simply want
>   > to read and write shorter symbol names in their code.
>
> I would not object to making rules about naming or usage conventions
> for the shorter names, in the second use-case.
>
> We can't follow those conventions for the first use-case.  They would
> not work.  But it is ok to treat the two use-cases differently.

Note that the two cases are related. The very reason why libraries like
s.el, dash.el and f.el appeared and blossomed in popularity is, in
no small part, that "people simply want to read and write shorter names
in their code".

It's perfectly legitimate not to desire this individually, but it is misguided
to try to forbid others from achieving this desire.

Rather, sensible rules should be drafted and tooling should be developed
that can protect both viewpoints.

João Távora

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Phil Sainty]

On 2021-10-02 11:38, Richard Stallman wrote:
> I think #2 might be a good idea, but #1 would lead to horrible
> confusion.  If the screen does not match the buffer, that is chaos.

No more chaotic than the lisp reader interning symbols using names
other than those that are used in the code.  All such "things are not
as they appear" features are varying degrees of chaos.

The feature for #1/#2 would be opt-in, so any user who enabled it
would be aware in advance of the nature of the chaos, and therefore
not confused by it (it would be the trade-off they were knowingly
making in order to read and write short names); and users who did
not enable the feature would not encounter any chaos at all, as
they would only ever see the real names.


-Phil

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 2228 bytes --]

Richard Stallman <rms@gnu.org> writes:

I'd like to highlight this line from RMS:
<cite>

If the screen does not match the buffer, that is chaos.
</cite>
and enshrine it as a key tenet in everything we design going forward.

Note that there are corner cases where the above can already be violated
via display text properties and overlays.

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > The first use-case is to do with the "s" library, and finding
>   > a way to rename all of that code with a longer prefix without
>   > requiring other libraries currently requiring "s" to change...
>
> Not solely with the `s' library; there are a few others.
> But basically I think you are right about this.
> And this is indeed the main intended use in Emacs itself.
>
>   > The second use-case, and the one I think will prove to be FAR
>   > more common if this goes ahead, is this: Some people simply want
>   > to read and write shorter symbol names in their code.
>
> I would not object to making rules about naming or usage conventions
> for the shorter names, in the second use-case.
>
> We can't follow those conventions for the first use-case.  They would
> not work.  But it is ok to treat the two use-cases differently.
>
>   > Instead, we would need two things:
>
>   > 1. A way of displaying long symbols in the desired short form,
>   >     such that the buffer contains the actual symbol, but the
>   >     user sees the short symbol (i.e. some kind of replacing
>   >     display).
>
>   > 2. Something analogous to abbrev which recognises when someone
>   >     starts typing a symbol with one of the configured short
>   >     prefixes, and expands it to be the full name (but per (1)
>   >     visually displayed as the short form that they typed).
>
> I think #2 might be a good idea, but #1 would lead to horrible
> confusion.  If the screen does not match the buffer, that is chaos.

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Dmitry Gutov]

On 28.09.2021 22:05, Alan Mackenzie wrote:
>> Then we will avoid using it, or maybe even recommend that no one does.
>> And perhaps replace shorthands with something better.  But we aren't
>> there anymore, and I think your sense of a catastrophe is unjustified,
>> if not exaggerated.
> OK, how do you suggest I find all occurrences of jit-lock-functions in
> the Emacs Lisp sources after shorthands start being used?  How do I find
> occurrences of a symbol in Emacs Lisp sources on the web, which
> currently a web search will find?
> 

I imagine the answer will be 'xref-find-references', as soon as it grows 
support for the shorthands *shrug*

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Stefan Monnier]

> Can we please delay the release of Emacs 28.1 until we have these tool
> enhancements in place?

These tool enhancements won't appear magically until people start using
shorthands and bumping into issues that need fixing.
shorthands won't be used widely until it's supported by a large fraction
of installed Emacsen, so we want to add the essential support for it as
soon as possible, so that people can start using it for real in Emacs-29
(knowing then that it also works for people still on Emacs-28).


        Stefan

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Alan Mackenzie]

Hello, Stefan.

On Tue, Sep 28, 2021 at 14:38:02 -0400, Stefan Monnier wrote:
> > Can we please delay the release of Emacs 28.1 until we have these tool
> > enhancements in place?

> These tool enhancements won't appear magically until people start using
> shorthands and bumping into issues that need fixing.

The bumping into has already happened.

How do you propose I find all occurrences of jit-lock-functions in the
Emacs Lisp sources, which up to now I've been able to do with find and
grep?

> shorthands won't be used widely until it's supported by a large fraction
> of installed Emacsen, ....

So we experience the problems for real in 2023 rather than 2021?

> .... so we want to add the essential support for it as soon as
> possible, so that people can start using it for real in Emacs-29
> (knowing then that it also works for people still on Emacs-28).

How am I now supposed to find every occurrence of jit-lock-functions?

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Eli Zaretskii]

> Date: Tue, 28 Sep 2021 19:14:22 +0000
> Cc: Eli Zaretskii <eliz@gnu.org>, Phil Sainty <psainty@orcon.net.nz>,
>   joaotavora@gmail.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> How do you propose I find all occurrences of jit-lock-functions in the
> Emacs Lisp sources, which up to now I've been able to do with find and
> grep?

The same as you did until now.  Nothing's changed.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Alan Mackenzie]

Hello, Eli.

On Tue, Sep 28, 2021 at 22:22:19 +0300, Eli Zaretskii wrote:
> > Date: Tue, 28 Sep 2021 19:14:22 +0000
> > Cc: Eli Zaretskii <eliz@gnu.org>, Phil Sainty <psainty@orcon.net.nz>,
> >   joaotavora@gmail.com, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > How do you propose I find all occurrences of jit-lock-functions in the
> > Emacs Lisp sources, which up to now I've been able to do with find and
> > grep?

> The same as you did until now.  Nothing's changed.

.... unless somebody has decided to shorthand jit-lock-functions to, say,
jl-functions in his file.  How would I arrange for grep to find that?  I
need to find _all_ occurrences of this symbol.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Eli Zaretskii]

> Date: Tue, 28 Sep 2021 19:31:23 +0000
> Cc: monnier@iro.umontreal.ca, psainty@orcon.net.nz, joaotavora@gmail.com,
>   emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> Hello, Eli.
> 
> On Tue, Sep 28, 2021 at 22:22:19 +0300, Eli Zaretskii wrote:
> > > Date: Tue, 28 Sep 2021 19:14:22 +0000
> > > Cc: Eli Zaretskii <eliz@gnu.org>, Phil Sainty <psainty@orcon.net.nz>,
> > >   joaotavora@gmail.com, emacs-devel@gnu.org
> > > From: Alan Mackenzie <acm@muc.de>
> 
> > > How do you propose I find all occurrences of jit-lock-functions in the
> > > Emacs Lisp sources, which up to now I've been able to do with find and
> > > grep?
> 
> > The same as you did until now.  Nothing's changed.
> 
> .... unless somebody has decided to shorthand jit-lock-functions to, say,
> jl-functions in his file.

But no one did.  And I see no reason that we will allow that.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > How do you propose I find all occurrences of jit-lock-functions in the
  > Emacs Lisp sources, which up to now I've been able to do with find and
  > grep?

We can have a rule about when it is ok to define symbol-renaming in
the Emacs sources, just as we have a rule about defining advice in the
Emacs sources.

This way, you could know about what renamings might exist,
and would know which symbols might have multiple names
and what argument to give to grep.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Tomas Hlavaty]

On Thu 30 Sep 2021 at 18:10, Richard Stallman <rms@gnu.org> wrote:
>   > How do you propose I find all occurrences of jit-lock-functions in the
>   > Emacs Lisp sources, which up to now I've been able to do with find and
>   > grep?
>
> We can have a rule about when it is ok to define symbol-renaming in
> the Emacs sources, just as we have a rule about defining advice in the
> Emacs sources.
>
> This way, you could know about what renamings might exist,
> and would know which symbols might have multiple names
> and what argument to give to grep.

What about cases where grep (or alternatives like rg or ag) are invoked
manually?  Or using a tool like eev and not via some clever Emacs
function which computes the argument automatically?

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

Phil Sainty <psainty@orcon.net.nz> writes:

> This has probably been covered in earlier discussions (apologies for
> not being across those), but...
>
> Won't this break a ton of basic tooling for locating things, if the
> symbol in the file is not the actual symbol?

Yes, it will "break" tags, grep, etc and these "dumb" tools.  Depends
what you mean by "break", however.  As a means to find symbols, grep is
_already_ broken by the ability to be utilize both "foo" and "foobar" in
the source.  You simply accept that breakage almost blindly.  Even
smarter tools are "broken" by (intern (concat "foo" "bar")) in that
sense.

I wouldn't give up the power of these indirections just to keep grep.
But indeed I have to be criterious when I use them.

As always, adding expressive power and indirection to a language
"breaks" anything that is too tightly coupled to the implementation.
For symbols, some tools make assumptions that they are always
represented textually by a certain contiguous string in a file.
Fortunately, some other tools don't, and IMO those are the most
important and valuable ones for dealing with symbols.

In these email discussions we use a namespace system, too.  We treat
each other by first names, like people normally do.  I don't type the
surnames of people unless they are explicitly participating in the
discussion.  With that I "break" the ability to grep for the exact
Stefan or Phil that I mean.  I can still grep for 'Stefan', or 'Sainty'
if I want.  It'll bring in some false positives.  But I wouldn't want to
give up the comfort of a first name basis.

So it sounds like you're arguing against namespaces in general, and
you're right, they have disadvantages, in any language.  But the
advantages are immense.  To organize complex problems utilizing many
libraries, I cannot imagine a decent language that does not have them.

João

PS: Do you know Xref and M-. ?  Added around Emacs 25, it's a tool that
uses a cross-referencing database that at least in principle shouldn't
make those assumptions.  It's actually based on a similar tool that I've
used in Common Lisp for many years (very complete namepsacing system),
and it's been fine.  I still use grep for the qualifying part of symbols
(the package, the Surname) and for the identifying part (the "first
name")

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Phil Sainty]

On 2021-09-28 20:21, João Távora wrote:
> So it sounds like you're arguing against namespaces in general,

Yes, I would vote against adding this to elisp.


> and you're right, they have disadvantages, in any language.
> But the advantages are immense.

Are they?  I don't see how they add anything at all of genuine worth.

(FWIW I'm sorry that this sounds harsh, as I'm certain that a lot of
effort has gone into this.  I just have really strong misgivings about
the feature.)

I agree that they make code a little nicer to write for the author
who knows what everything is.  I think this is an *extremely* minor
benefit which everyone can easily manage without (as they have done
for decades); and AFAICS it is the only benefit, and so I don't see
how it is worth all of the trouble that comes with it.

I firmly disagree with the notion that it makes code easier to read
for other people.  Other people actually now have to mentally account
for the fact that every symbol they see might not actually exist by
that name.  The manual even talked about the ability for different
people to use different shorthands for the same library, so a symbol
might actually have *many* names in the codebase.  This makes things
harder.

Allowing things to not be what they seem adds an additional cognitive
load to *everything* you look at, because everything has the potential
to not be what it seems, and so I think this makes codebases harder to
read and understand, generally.

Libraries that you might actually be familiar with can now appear in
an entirely unfamiliar guise, meaning that extra mental effort is
needed to figure out what is going on in situations which you would
otherwise have immediately understood.

It makes it harder for users to share code because, even if you know
which libraries are used, copied and pasted code can now be completely
broken without knowledge of the magic file-local variable.

 From what I've seen it doesn't seem that the shorthand symbols even
exist (they are translated at read time), so if someone reads some
shorthand code and tries to look up the symbols they see, will they
find them?  If they C-h v and use TAB completion for the prefix they
are seeing in the code, will they see all the completions?  If so,
will that work in all buffers, in all locations?  If they get to the
help buffer and they try another look-up, will it then fail?  Or if
they simply have split windows and are reading code from the other
window?  What if different libraries specify the same shorthand for
different dependencies?  Can people only look up certain symbols
from certain places?

(I've just compiled from master to test some of this, and the answer
seems to be that it's impossible to look up the documentation for the
code you can actually see; unless you're in the buffer and asking
about the symbol at point, in which case it figures it out.  Again,
this is making things harder for the reader, not easier.  We have gone
from "if you can see it you can look it up" to "you can only look it
up if you are pointing directly at it, because it isn't what it says
it is".)


> To organize complex problems utilizing many libraries, I cannot
> imagine a decent language that does not have them.

How do namespaces help with that?  Without namespaces you still have
access to all of the same functionality from all of those many
libraries, to organise your complex problem.

Furthermore Emacs Lisp is not other languages, and Emacs is unusual.
It combines a very large amount of global state (which most programs
do not do) with dynamic binding (which most languages do not have).
Other languages are more likely to restrict the global state, and
share things in a far more limited manner.  In Emacs, global symbols
are used pervasively between libraries in a fashion that doesn't
happen in most software; and if the same symbol does not always have
the same name, I think that's a bad thing.

Surely "other languages have this" isn't a reason to add something to
elisp unless there is a clear net win to doing so, and all I'm seeing
are net losses.  What are the advantages that I'm not seeing which are
so immense that they make all the problems worthwhile?


-Phil

p.s. I know you wrote more, but this was already a long reply.
However, I don't think the informal "namespacing" of people's names
in a discussion is an analogue of the formal expressions needed by a
programming language; and I couldn't figure out what the "expressive
power" of shorthands might be.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Dmitry Gutov]

On 28.09.2021 15:49, Phil Sainty wrote:
> Allowing things to not be what they seem adds an additional cognitive
> load to *everything* you look at, because everything has the potential
> to not be what it seems, and so I think this makes codebases harder to
> read and understand, generally.

Perhaps we could alleviate this by requiring that shorthands end with a 
particular character (like '/' or ':'), so that if you see it in a name, 
it's probably a shorthand.

There is code out there which uses those characters as well, but it's 
not that frequent and could be phased out.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Gregory Heytings]

>> Allowing things to not be what they seem adds an additional cognitive 
>> load to *everything* you look at, because everything has the potential 
>> to not be what it seems, and so I think this makes codebases harder to 
>> read and understand, generally.

+1

>
> Perhaps we could alleviate this by requiring that shorthands end with a 
> particular character (like '/' or ':'), so that if you see it in a name, 
> it's probably a shorthand.
>

I'd vote for '@'.  It's not used in core nor in ELPA, it's only used in 
(very few) MELPA packages.

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Adam Porter]

Gregory Heytings <gregory@heytings.org> writes:

>> Perhaps we could alleviate this by requiring that shorthands end
>> with a particular character (like '/' or ':'), so that if you see it
>> in a name, it's probably a shorthand.
>
> I'd vote for '@'.  It's not used in core nor in ELPA, it's only used
> in (very few) MELPA packages.

I wouldn't like seeing "@" in symbol names, IMHO.  And we may be getting
ahead of ourselves.  But let the bikeshedding commence.  :)

    Adam (who may someday use "~/" as a local shorthand in packages)

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 28.09.2021 15:49, Phil Sainty wrote:
>> Allowing things to not be what they seem adds an additional cognitive
>> load to *everything* you look at, because everything has the potential
>> to not be what it seems, and so I think this makes codebases harder to
>> read and understand, generally.
>
> Perhaps we could alleviate this by requiring that shorthands end with
> a particular character (like '/' or ':'), so that if you see it in a
> name, it's probably a shorthand.

I think that's a dreadful idea.

The point of shorthands is aiding typing and manage namespacing
etiquette succesfully.  Your proposal would single-handedly destroy the
ability to import s.el and s.el -using libraries with minimal changes,
for example, which was one of the main motivations for writing it.

If however, you substritute "requiring shorthands to end" with "visually
annotating" shorthands, such as with font-lock, for example, then I
think that's a pretty good idea that should solve the "cognitive load"
bit.  We already do that with macros, functions, variables, uninterned
symbols, why not shorthands, indeed.  It's super consistent.

João

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Gregory Heytings]

>> Perhaps we could alleviate this by requiring that shorthands end with a 
>> particular character (like '/' or ':'), so that if you see it in a 
>> name, it's probably a shorthand.
>
> I think that's a dreadful idea.
>

How so?  The purpose is only to provide an unambiguous textual clue that 
the symbol will be transformed by the Lisp reader into another symbol. 
With your examples (in the manual), the author or user of 
some-nice-string-utils.el would use

;; elisp-shorthands: (("snu" . "some-nice-string-utils-"))

(without the final dash) in its local variables, and the chosen symbol 
(after thinking a bit more about this, I think "::" which was suggested 
earlier is better than "@") between snu and the function/variable names 
throughout the file.

That would also make the task of standard tools like tags or grep easier.

>
> If however, you substritute "requiring shorthands to end" with "visually 
> annotating" shorthands, such as with font-lock, for example
>

So in the end we would have something that very much resembles nameless.el 
(a prefix defined in local variables, and using font lock to "visually 
annotate" that prefix) without its benefits (not breaking any of the 
standard tools and allowing anyone to choose whether and how they want to 
use that feature)?

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[João Távora]

Phil Sainty <psainty@orcon.net.nz> writes:

> On 2021-09-28 20:21, João Távora wrote:
>> So it sounds like you're arguing against namespaces in general,
>
> Yes, I would vote against adding this to elisp.
>
>
>> and you're right, they have disadvantages, in any language.
>> But the advantages are immense.
>
> Are they?  I don't see how they add anything at all of genuine worth.
>
> (FWIW I'm sorry that this sounds harsh, as I'm certain that a lot of
> effort has gone into this.  I just have really strong misgivings about
> the feature.)

No problem.  We can be harsh and polite to each other.  And,
surprisingly, "not much effort" went into this :-) if you see the source
code, you'll see that is fairly terse.  Certainly much less than other
features I've added to Emacs.  A good thing.

> I agree that they make code a little nicer to write for the author
> who knows what everything is.  I think this is an *extremely* minor
> benefit which everyone can easily manage without (as they have done
> for decades); and AFAICS it is the only benefit, and so I don't see
> how it is worth all of the trouble that comes with it.
>
> I firmly disagree with the notion that it makes code easier to read
> for other people.  Other people actually now have to mentally account
> for the fact that every symbol they see might not actually exist by
> that name.

I know you don't like these analogies, but when you are with your
family, do you qualify everybody by their full names?  You don't right.
It's tedious to you, as a speaker, and I suppoe it's tedious to your
counterparts as a listener.  Humans use shorthands for lots and lots of
things.  Hence the common phrase "context is everything".

My position is that human languages are no different from artificial
ones in their interaction with humans (reading and writing).  For
interactions between programs (compilers, etc) I fully agree with Stefan
Monnier that they don't care, and neither should we.

(or should we?  Have you ever needed to debug a C++ program without a
name unmangler?)

> The manual even talked about the ability for different people to use
> different shorthands for the same library, so a symbol might actually
> have *many* names in the codebase.  This makes things harder.

Namespace systems in all languagues I know are about the ability to
refer to the same thing by different names in different contexts.  I
guess I have to refer you to some recent decade's worth of language
design.

> Allowing things to not be what they seem adds an additional cognitive
> load to *everything* you look at,

I don't deny that.  There is an additional cognitive load.  My point is
that that load is comparatively smaller, MUCH smaller, than the load
incurred by reading repetitive code.  Don't forget that typing
this-long-symbol-prefix is a way of code duplication, however primitive.
And this duplication is bad, not for storage reasons, but because it
slows down the human parser tremedously.

> It makes it harder for users to share code because, even if you know
> which libraries are used, copied and pasted code can now be completely
> broken without knowledge of the magic file-local variable.

That's true, yes.  You can't share code without context.  But listen,
that already happen in a myriad of situations.  I can't paste a snippet
that refers to a global variable or funtion and expect it to work UNLESS
I give you that context.  No really, the problem you bring about is not
new.

> From what I've seen it doesn't seem that the shorthand symbols even
> exist (they are translated at read time), so if someone reads some
> shorthand code and tries to look up the symbols they see, will they
> find them?

No.  In Lisp there is the form and there the symbol.  These are not
interchangeable.  A symbol is an object, which is much more than just
its print name.  A form is a written down version of the symbol.

Shorthands are just the forms. 

> If they C-h v and use TAB completion for the prefix they are seeing in
> the code, will they see all the completions?

No. The shorthands are local context.  So if they type C-M-i they will
see it, but with C-h v you see the full name of the symbol.  Which is
always valid, of course.

> If so, will that work in all buffers, in all locations?

No.  You may be known under different nicknames in your family, your
Yatch clubetc.  Mixing up your nicknames in those different contexts
would be disastrous.

> If they get to the help buffer and they try another look-up, will it
> then fail?

You lost m at this point.

> Or if they simply have split windows and are reading code from the
> other window?  What if different libraries specify the same shorthand
> for different dependencies?  Can people only look up certain symbols
> from certain places?

Again, it seems you are against the idea of namespaces in general in any
language.  But indirections is what hight-level programming is about.
You don't have to use all of them, and neither should you, but I firmly
believe you should have access to them.  Namespaces, generic functions,
advice, hooks, macros, functions.  All these things to "hidden" things
that you have to chase down.  If you don't want any of it, you have to
be ultra-literal.  You have to use machine code, not even assembly.

>> To organize complex problems utilizing many libraries, I cannot
>> imagine a decent language that does not have them.
>
> How do namespaces help with that?

By allowing you to compose programs from many different libraries in a
way that you don't write a mammoth prefix all the time.  You can choose
what prefix you assign to each library's symbols.  "fn" for the bunch of
"file-name" stuff you're going to do.  "s" for the bunch of
"magnars-string" stuff you're going to do. "phil" for the bunch of
"phils-incredible-library-of-utils.el" stuff you're going to do.  That
is how.

> Furthermore Emacs Lisp is not other languages, and Emacs is unusual.

Yes, Emacs is unsual.  Especially in the developers ahahah (me included,
of course).

But Emacs Lisp is not much different than other languages.  It's a
Lisp-2 with many features of Common Lisp.  Probably has more much devs
than Common Lisp, and a younger generation at that.  Probably has more
future, I'd say.  Needs a better GC.

> It combines a very large amount of global state (which most programs
> do not do)

Obviously that depends on the programs you write and how you write them.

> with dynamic binding (which most languages do not have).

And which shouldn't be abused, BTW (and here you go dynamic binding!
Another super-useful feature with lots of tricky hidden behaviour.  Oh
the cognitive load!)

> Surely "other languages have this" isn't a reason to add something to
> elisp unless there is a clear net win to doing so, and all I'm seeing
> are net losses.  What are the advantages that I'm not seeing which are
> so immense that they make all the problems worthwhile?

I've described them.  Again I refer you to all major languages that have
namespacing systems.  Or to the fair number of people that have
expressed want of a namespacing system in Emacs.

It's like you're asking me to describe to your why global variables are
useful.  Some people avoid them like the plague, they hide behaviour.
You need context.  There's cognitive load to understand a problem with
global variables.  But just like shorthands, they're opt-in.  You don't
_have_ to use them in your programs.  You can object to them in other
peoples programs, same thing.

> programming language; and I couldn't figure out what the "expressive
> power" of shorthands might be.

That is quite clear to see :-)

João

Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >   Other people actually now have to mentally account
  > for the fact that every symbol they see might not actually exist by
  > that name.

I think that very few Emacs users, and few Emacs Lisp programmers,
will need to account for this.  We will fix various tools to handle
them seamlessly.

Are you assuming that a symbol will have a usual long name,
and someone will define a shorthand so as to refer to it by
a shorter name?

That usage will be possible, but the way I want to use it is
different.

Namely, a package's entry points will have only one name, the same
name people already know.  But -- and this is the change -- that name
will be visible only in programs that require the package.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Elisp LSP Server

[Ag Ibragimov]

Sorry, that thread "Re: Do shorthands break basic tooling (tags, grep, etc)? (was Re: Shorthands have landed on master)" has grown too big.

But scrolling through it, I realized I wanted to ask for a long time:

Has anyone thought about writing a LSP Server for Emacs Lisp?

You know, so Emacs Lisp can be comfortably edited outside of Emacs. "Why would anyone want that?", you ask?

Well, for example, when something breaks in your config, and you can't even start Emacs, and you have to use something else. It's rare, but it happens.

Another reason: VSCode. The damn thing is spreading like a disease. And now it's even possible to browse code on GitHub, in a browser, by simply pressing <.> (the dot).
I would love to be able to comfortably browse through elisp code without having to clone it locally, but none of the existing VSCode Lisp plugins are any good.
For example, there's no equivalent of imenu for Lisp files. You can't jump to a given function.

I don't care much for VSCode, but LSP provides good consistency. I would love to be able to use elisp packages like lsp-ui while writing elisp code [in Emacs].

-- 
Thanks, Ag

Re: Elisp LSP Server

[Po Lu]

Ag Ibragimov <agzam.ibragimov@gmail.com> writes:

> Another reason: VSCode. The damn thing is spreading like a
> disease. And now it's even possible to browse code on GitHub, in a
> browser, by simply pressing <.> (the dot).  I would love to be able to
> comfortably browse through elisp code without having to clone it
> locally, but none of the existing VSCode Lisp plugins are any good.
> For example, there's no equivalent of imenu for Lisp files. You can't
> jump to a given function.

Isn't the code behind that particular feature proprietary?

Re: Elisp LSP Server

[Philip Kaludercic]

Po Lu <luangruo@yahoo.com> writes:

> Ag Ibragimov <agzam.ibragimov@gmail.com> writes:
>
>> Another reason: VSCode. The damn thing is spreading like a
>> disease. And now it's even possible to browse code on GitHub, in a
>> browser, by simply pressing <.> (the dot).  I would love to be able to
>> comfortably browse through elisp code without having to clone it
>> locally, but none of the existing VSCode Lisp plugins are any good.
>> For example, there's no equivalent of imenu for Lisp files. You can't
>> jump to a given function.
>
> Isn't the code behind that particular feature proprietary?

In what sense? GitHub is inherently proprietary, so yes, but what
about this specific feature is different.

-- 
	Philip Kaludercic

Re: Elisp LSP Server

[Po Lu]

Philip Kaludercic <philipk@posteo.net> writes:

> Po Lu <luangruo@yahoo.com> writes:
>
>> Ag Ibragimov <agzam.ibragimov@gmail.com> writes:
>>
>>> Another reason: VSCode. The damn thing is spreading like a
>>> disease. And now it's even possible to browse code on GitHub, in a
>>> browser, by simply pressing <.> (the dot).  I would love to be able to
>>> comfortably browse through elisp code without having to clone it
>>> locally, but none of the existing VSCode Lisp plugins are any good.
>>> For example, there's no equivalent of imenu for Lisp files. You can't
>>> jump to a given function.
>>
>> Isn't the code behind that particular feature proprietary?
>
> In what sense? GitHub is inherently proprietary, so yes, but what
> about this specific feature is different.

I think it is dangerous for Emacs to "integrate" with such proprietary
software.  It would make people gravitate towards that software.

Re: Elisp LSP Server

[Philip Kaludercic]

Po Lu <luangruo@yahoo.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> Po Lu <luangruo@yahoo.com> writes:
>>
>>> Ag Ibragimov <agzam.ibragimov@gmail.com> writes:
>>>
>>>> Another reason: VSCode. The damn thing is spreading like a
>>>> disease. And now it's even possible to browse code on GitHub, in a
>>>> browser, by simply pressing <.> (the dot).  I would love to be able to
>>>> comfortably browse through elisp code without having to clone it
>>>> locally, but none of the existing VSCode Lisp plugins are any good.
>>>> For example, there's no equivalent of imenu for Lisp files. You can't
>>>> jump to a given function.
>>>
>>> Isn't the code behind that particular feature proprietary?
>>
>> In what sense? GitHub is inherently proprietary, so yes, but what
>> about this specific feature is different.
>
> I think it is dangerous for Emacs to "integrate" with such proprietary
> software.  It would make people gravitate towards that software.

That yes, but that is an inherent problem of LSP (e.g. see how VSCode
can automatically download and run a propitiatory LSP server like
Pylance), and not the "." command itself.

I think a better solution to Ag's problem is to make editing online
repositories easier, without to manually clone a directory every time.

-- 
	Philip Kaludercic

Re: Elisp LSP Server

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I think it is dangerous for Emacs to "integrate" with such proprietary
  > > software.  It would make people gravitate towards that software.

Po Lu is right.

  > That yes, but that is an inherent problem of LSP (e.g. see how VSCode
  > can automatically download and run a propitiatory LSP server like
  > Pylance), and not the "." command itself.

Can you please explain the problem that you're talking about?  If it
is true that LSP inherently tends to use nonfree LSP servers, then
using LSP is inherently a problem, and maybe we should not include
that feature in Emacs at all.

Or can we fix that problem?  Maybe loading nonfree servers is not
totally inherent -- maybe we can arrange to avoid it.

  > I think a better solution to Ag's problem is to make editing online
  > repositories easier, without to manually clone a directory every time.

Would you spell out in more detail what you're thinking of?  Perhaps this
would be a solution, but what is the solution, and what is the problem?


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Elisp LSP Server

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > I think it is dangerous for Emacs to "integrate" with such proprietary
>   > > software.  It would make people gravitate towards that software.
>
> Po Lu is right.
>
>   > That yes, but that is an inherent problem of LSP (e.g. see how VSCode
>   > can automatically download and run a propitiatory LSP server like
>   > Pylance), and not the "." command itself.
>
> Can you please explain the problem that you're talking about?  If it
> is true that LSP inherently tends to use nonfree LSP servers, then
> using LSP is inherently a problem, and maybe we should not include
> that feature in Emacs at all.

I don't think that this is currently the case, most LSP servers I know
of are free software.

The difference between Emacs (at least Eglot) and other LSP
implementations, is that the latter can download servers automatically.
This is done without regard to the license, as projects like VSCode are
not focused on software freedom. The intent is to take care of setting
up a server without the user having to care, but the effect is that
actual programs, possibly propitiatory, are being downloaded and run on
the host system.

> Or can we fix that problem?  Maybe loading nonfree servers is not
> totally inherent -- maybe we can arrange to avoid it.

Of course. Currently the problem doesn't exist to begin with, since LSP
servers have to be installed manually for Emacs.  If LSP clients for
Emacs were to automatically configure a server, then I do think that the
licensing should be seriously taken into consideration. 

>   > I think a better solution to Ag's problem is to make editing online
>   > repositories easier, without to manually clone a directory every time.
>
> Would you spell out in more detail what you're thinking of?  Perhaps this
> would be a solution, but what is the solution, and what is the problem?

The problem is when you are browsing the online interface of a source
code repository, you might decide to want to make a change. This
introduces friction in the user's workflow, because you have to find a
directory, the repository has to be cloned into said directory, opened
in Emacs and then you can start working.

GitHub has introduced a feature where a in-browser Editor (derived from
VSCode as far as I see) can be invoked by literally at the press of a
button. Everything has been initialized, and you can make your changes
in a matter of seconds.

Replicating this for Emacs seems hard to do, but it is probably the
wrong approach. My guess would be that most people wouldn't want Emacs
in a browser. A solution could be to provide a script or command that
automatically clones a repository to a temporary directory and opens
Emacs (possibly by making use of emacsclient). The tricky part would be
to integrate this into a web browser, but my guess would be that this
could be done by use of a browser extension. This would recognize the
website, extract the git repository and start the script.

Haven't tried any of this, it is just brainstorming.

-- 
	Philip Kaludercic

Re: Elisp LSP Server

[Stefan Monnier]

> Replicating this for Emacs seems hard to do, but it is probably the
> wrong approach. My guess would be that most people wouldn't want Emacs
> in a browser. A solution could be to provide a script or command that
> automatically clones a repository to a temporary directory and opens
> Emacs (possibly by making use of emacsclient).

Indeed, I'd appreciate that.
Better if the location of the local clone is stable so visiting it
N times only clones it once (and maybe updates it N times).


        Stefan

Re: Elisp LSP Server

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The difference between Emacs (at least Eglot) and other LSP
  > implementations, is that the latter can download servers automatically.
  > This is done without regard to the license, as projects like VSCode are
  > not focused on software freedom.

Yes, that's what we must expect from them.

But what they do does not matter to us
as long as we can avoid doing likewise.

  > Replicating this for Emacs seems hard to do, but it is probably the
  > wrong approach. My guess would be that most people wouldn't want Emacs
  > in a browser. A solution could be to provide a script or command that
  > automatically clones a repository to a temporary directory and opens
  > Emacs (possibly by making use of emacsclient). The tricky part would be
  > to integrate this into a web browser, but my guess would be that this
  > could be done by use of a browser extension. This would recognize the
  > website, extract the git repository and start the script.

It sounds like a reasonable thing to do
by writing a free extension.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Elisp LSP Server

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

> Can you please explain the problem that you're talking about?  If it
> is true that LSP inherently tends to use nonfree LSP servers, then
> using LSP is inherently a problem, and maybe we should not include
> that feature in Emacs at all.

The OP was proposing that Emacs integrate (by providing imenu
functionality to Visual Studio Code through LSP) with a particular
feature of GitHub, where, through proprietary JavaScript and a
proprietary plugin, anyone can press "." to immediately view code from
GitHub in VS Code.

Re: Elisp LSP Server

[Ag Ibragimov]

Po Lu <luangruo@yahoo.com> writes:
> The OP was proposing that Emacs integrate (by providing imenu
> functionality to Visual Studio Code through LSP) with a particular
> feature of GitHub, where, through proprietary JavaScript and a
> proprietary plugin, anyone can press "." to immediately view code from
> GitHub in VS Code.

Not quite. I'm not suggesting at all integrating with proprietary software. I'm merely curious if anyone has ever thought about implementing an LSP server for Emacs Lisp.
LSP is a standard and an open specification. There already exist several dozen implementations of Language Protocol servers for many different languages.
And many of them are perfectly usable in Emacs, thanks to eglot and lsp-mode. And that's great because adhering to the standard makes it a smooth experience for the user.

One of the main reasons people leave Emacs, sometimes even after many years of using it, is the substandard language support.
They often have to start programming in another language, and it takes too much effort to make Emacs work for a specific language.
Simple things like goto definition and autocomplete often won't work as expected. People often have to learn a bunch of independent Elisp packages and figure out ways to make them work together nicely.
So many times I've heard: "I love Emacs, used it for almost [so many years] but I had to work with a language [X] and moved to [another] IDE that has first-class support for it."

LSP, for many Emacsen has become a true game-changer. You don't have to re-invent the wheel for every single different language, no need to map and re-map the keys (separately for every language) or struggle when a language mode package authors fail to include some features.
Wouldn't it be ironic if soon, Emacs gets truly fantastic support for dozens of languages (btw, that list is still growing), yet Elisp would have to ride like a second-class passenger?

I think creating Elisp LSP server would provide consistency with all other languages already supported.
And as a side effect, that would also make it possible for better navigation and code introspection of Elisp outside of Emacs.
Personally, I don't care if a proprietary software vendor like GitHub introduces better features for a language that is the ultimate epitome of free software.
Their non-free software would still be written in Emacs (at least some parts of it) - the ultimate tool with unmatched support for programming languages.

--
Thanks, Ag

Re: Elisp LSP Server

[Daniel Martín]

Ag Ibragimov <agzam.ibragimov@gmail.com> writes:

>
> I think creating Elisp LSP server would provide consistency with all
> other languages already supported.

I agree that having a consistent interface for providing language
"intelligence" might be good (but note that the design of the LSP
interface has its own problems), but I think that the price we'd pay by
using an ELisp LSP server would be too much.  Why pay the price of
serializing from/to JSON, RPC, and having an inferior process running
all the time when the introspection capabilities of Emacs can provide
the same capabilities (with similar, if not the same, quality) but all
happening more efficiently in process?

> And as a side effect, that would also make it possible for better
> navigation and code introspection of Elisp outside of Emacs.

I think this is a good point, but I don't expect there is a lot of
people that work on ELisp without Emacs (even if we consider people that
use a web IDE that supports LSP).  Wouldn't we spend a lot of time
building something that almost no one would use?

Re: Elisp LSP Server

[Philip Kaludercic]

Daniel Martín <mardani29@yahoo.es> writes:

> Ag Ibragimov <agzam.ibragimov@gmail.com> writes:
>
>> And as a side effect, that would also make it possible for better
>> navigation and code introspection of Elisp outside of Emacs.
>
> I think this is a good point, but I don't expect there is a lot of
> people that work on ELisp without Emacs (even if we consider people that
> use a web IDE that supports LSP).  Wouldn't we spend a lot of time
> building something that almost no one would use?

I agree, it seems like too much effort, if all you need is navigation
and introspection. E.g. htmlfontify uses etags (but I guess that could
be updated to use xref) to generate links for definitions.

-- 
	Philip Kaludercic

Re: Elisp LSP Server

[Daniel Martín]

Philip Kaludercic <philipk@posteo.net> writes:

> Daniel Martín <mardani29@yahoo.es> writes:
>
>> Ag Ibragimov <agzam.ibragimov@gmail.com> writes:
>>
>>> And as a side effect, that would also make it possible for better
>>> navigation and code introspection of Elisp outside of Emacs.
>>
>> I think this is a good point, but I don't expect there is a lot of
>> people that work on ELisp without Emacs (even if we consider people that
>> use a web IDE that supports LSP).  Wouldn't we spend a lot of time
>> building something that almost no one would use?
>
> I agree, it seems like too much effort, if all you need is navigation
> and introspection. E.g. htmlfontify uses etags (but I guess that could
> be updated to use xref) to generate links for definitions.

I can also recommend a GNU project here: GNU Global
(https://www.gnu.org/software/global/).  GNU Global is free software
that can generate a code browsing website for a software project where
you can search for symbols, navigate to function definitions, etc.
Here's an example of how a website generated by GNU Global looks like:
https://www.tamacom.com/tour/kernel/linux/

GNU Global can integrate with multiple tagging backends.  One of them is
ctags/etags, which provides support for Emacs Lisp.

Re: Elisp LSP Server

[Po Lu]

Daniel Martín <mardani29@yahoo.es> writes:

> I think this is a good point, but I don't expect there is a lot of
> people that work on ELisp without Emacs (even if we consider people that
> use a web IDE that supports LSP).  Wouldn't we spend a lot of time
> building something that almost no one would use?

Yes, but the use-case the OP put forward seems very plausible: sadly,
many people utilize GitHub to develop their Emacs customizations, and
making it easier to work on Emacs customizations using GitHub and its
bundled proprietary software (VS Code) will only encourage people to
install VS Code.  Even if people will only use it for "Go To Definition"
or "Completion at point", I think it will still be a great setback.

Especially when both features are already present in Emacs, but without
integration with GitHub, as AFAIU Microsoft doesn't allow the protocol
used by VS Code to be used by other programs such as Emacs.

Instead of making it easier for people to put their customizations on
GitHub, how about making it harder?  It would encourage people to move
to systems that do not require running proprietary software.

Re: Elisp LSP Server

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Yes, but the use-case the OP put forward seems very plausible: sadly,
  > many people utilize GitHub to develop their Emacs customizations, and
  > making it easier to work on Emacs customizations using GitHub and its
  > bundled proprietary software (VS Code) will only encourage people to
  > install VS Code.  Even if people will only use it for "Go To Definition"
  > or "Completion at point", I think it will still be a great setback.

Thank you for pointing this out and arguing the point.

On the technical level, we have been talking about two different issues:

* providing to Emacs the features of the language server for
editing Emacs Lisp.

* making then available through the Language Server so that
people can take advantage of them in VS Code.

The first is clearly a step forward; the second one be an own goal.
Actually, two own goals at once.

In addition to the own-goal of encouraging use of VS Code,
we would score the own goal of encouraging use of GitHub.
GitHub has done tremendous harm to the free software movement,
by spreading confusion and misinformation about licenses.

We need to do more to point out the wrongs that GitHub has done
(and continues to do) -- not try to "cooperate" with it.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Elisp LSP Server

[Ag Ibragimov]

> Instead of making it easier for people to put their customizations on
> GitHub, how about making it harder?  It would encourage people to move
> to systems that do not require running proprietary software.

Do you seriously believe that? We should deliberately hinder our own
progress, hoping it somehow would hurt GitHub?

I don't know if you know this, but Emacs Lisp is probably one of the
most popular Lisps out there hosted on GitHub and Gitlab. After over six
decades of Lisp history, today, there's more Elisp code out there than
of any other Lisps, perhaps with the exclusion of maybe Clojure (?).

I can assure you Emacs ecosystem (as we have it today) wouldn't exist
without fantastic authors and maintainers of many great Emacs packages
hosted there.

We can debate all day long about the morality of their choices, but
making decisions that would create some obstacles for them? That would
only annoy people and force them to get creative.

I understand that many people view GitHub as a big evil thing and many
others think it's the second greatest website after Wikipedia. But it
isn't as simple as that - it's not all black and white.

People are the greatest and most precious resource in any programming
language community. Making things nicer for them is the best and the
only way to guarantee the prosperity and growth of that community.

The mission we all are fighting for to promote computer user freedom is
about people. If, for whatever reason, some people like a proprietary
service or software, we need to understand those reasons and try to give
them better alternatives instead of saying: "screw you, we won't let you
integrate with that thing. It's evil."

Re: Elisp LSP Server

[Po Lu]

Ag Ibragimov <agzam.ibragimov@gmail.com> writes:

>> Instead of making it easier for people to put their customizations on
>> GitHub, how about making it harder?  It would encourage people to move
>> to systems that do not require running proprietary software.

> Do you seriously believe that? We should deliberately hinder our own
> progress, hoping it somehow would hurt GitHub?

But making it easy to host Emacs customizations on GitHub isn't progress
for Emacs.  In fact, it would be very damaging to free software in
general, by encouraging more users to migrate to GitHub.

> The mission we all are fighting for to promote computer user freedom is
> about people. If, for whatever reason, some people like a proprietary
> service or software, we need to understand those reasons and try to give
> them better alternatives instead of saying: "screw you, we won't let you
> integrate with that thing. It's evil."

But if nobody has made such an alternative, it would be bad to simply
encourage people to use the proprietary software.

Re: Elisp LSP Server

[Ag Ibragimov]

Po Lu <luangruo@yahoo.com> writes:

> But if nobody has made such an alternative, it would be bad to simply
> encourage people to use the proprietary software.

You keep saying that we must avoid integrating Emacs with proprietary software. In a general sense, I agree. But I'm not sure I can completely agree with the reasons you're stating: "Doing otherwise would encourage people to use non-free software". There's also the other side of that coin: If Emacs doesn't work well in some environments - users simply may stop using Emacs.

But I think that's just two opposite opinions, nothing more. I don't think there's any data to support either of these claims.

I, for one, would love to see Emacs working nicely on everything - in every operating system, any browser, on a toaster, on embedded systems, on every earth-orbiting satellite, on my TV, on my watch, on my phone, in my car, everything. Excuse me, but I can't see how this would be "isn't progress for Emacs and very damaging to free software in general". But again, it's just an opinion. I don't have any proof to support the claim opposite to yours that enhancing the compatibility of Emacs (regardless of platform's nature) is better for the future of Emacs; I'm just speculating.

But, yes, I would love to see Emacs spreading far and wide, and sadly, this is not happening. And maybe it has something to do with this (irrational in my opinion) fear that we should hold it back and prevent any attempts to integrate it or make it work nicely with non-free software. I firmly support your beliefs, and I always try to advocate for free and open-source software myself, but I don't think anything in this world is black and white (otherwise, it's just a borderline religious fanatism), and sometimes making compromises do wonderful things.

Look around, and maybe I hope you'd see the fantastic success of Vim, shortly after its maintainers decided to host the source code on a popular, albeit proprietary platform. Incredible Emacs-powered projects launched: Spacemacs, Doom, Org-roam, et al.; See the abundance of truly amazing Emacs packages: completion frameworks, editing tools, spellcheckers, themes, version control integrations, chatting apps, etc. Most of them are hosted on GitHub, not because it has first-class support for Elisp, but despite its nearly complete absence. Frankly, I don't like the fact that they are on GitHub, but I think if someone advocates for moving their projects anywhere else (and the choices aren't unequivocally better alternatives), the authors would be like: "Here's the blowtorch. Blow me. Or torch m
 e. I don't care. I'm not moving my stuff over. That thing over there, it sucks."

Check out the phenomenal rise of VSCode. In only five years, they've succeeded in achieving more than twenty years of Emacs evolution. And honestly, VSCode's enormous pace of spreading is very alarming. It feels like in a few years we'd be living in an xkcd comic where one NASA console operator talks to another: "Damn it, Steve. You forgot to send the licensing fee acknowledgment signal yesterday, and I think now we've lost control of one of the rovers on Mars. VSCode remote daemon has died."

I don't want to live in a world dominated by proprietary software because it integrates better with FOSS, and Emacs gets abandoned because it just can't be integrated with non-free soft. Thanks to the ingenious and tireless Emacs fans who are willing to make compromises and keep finding creative ways to overcome Emacs limitations, we're not there yet. And I hope we never will be.

Re: Elisp LSP Server

[Stefan Kangas]

Ag Ibragimov <agzam.ibragimov@gmail.com> writes:

> I, for one, would love to see Emacs working nicely on everything - in
> every operating system, any browser, on a toaster, on embedded
> systems, on every earth-orbiting satellite, on my TV, on my watch, on
> my phone, in my car, everything. Excuse me, but I can't see how this
> would be "isn't progress for Emacs and very damaging to free software
> in general".

Emacs in general does accept patches porting it to new operating
systems, AFAICT even esoteric ones.  For example, we even have a port
for MS-DOS, if that happens to be your thing.

> Check out the phenomenal rise of VSCode. In only five years, they've
> succeeded in achieving more than twenty years of Emacs evolution. And
> honestly, VSCode's enormous pace of spreading is very alarming.

VSCode is less ambitious than Emacs, and is built on worse foundations.
That means that their development will inevitably hit a ceiling, whereas
ours won't.  We might develop slower than them for various reasons, but
check back in 40 more years, and then we'll see who's still around.

That said, I agree that Emacs has a lot of work to do if it wants to
stay relevant.  On this list, we are busy working on those things.
If you want to help, head over to the bug tracker and get cracking.
Or, you know, write an Emacs Lisp LSP server.  ;-)

Re: Elisp LSP Server

[dick]

[I have been individually instructed
[here](https://lists.gnu.org/archive/html/emacs-tangents/2021-10/msg00001.html)
to express my differences with GNU on a mailing list no one
reads gnu-misc-discuss@gnu.org, but being as no one else appears bound to that
directive, and being as I consider myself having more to contribute both
technically and philosophically than they, I too willfully flout that
directive]

SK> VSCode is less ambitious than Emacs, and is built on worse foundations.
SK> That means that their development will inevitably hit a ceiling, whereas
SK> ours won't.

Everything about the above is false.  Emacs development, for better or worse,
is hemmed by philosophical considerations that VSCode is not.

Silliness including EMBA and the handwringing over LSP and Github remind me,
ironically given GNU's ballyhooed respect for freedoms, of China's
miniaturization of the internet behind their so-called Great Firewall.  Like
GNU, Chinese officials maintain that the wider internet at large is plagued by
vice, and thus every conceivable service must be replicated *ab initio* under
their enlightened moral code.  The difference is China, given that it's, well,
a huge country, actually has a chance of succeeding.

Re: Elisp LSP Server

[Eli Zaretskii]

> From: dick <dick.r.chiang@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Tue, 12 Oct 2021 16:53:29 -0400
> 
> [I have been individually instructed
> [here](https://lists.gnu.org/archive/html/emacs-tangents/2021-10/msg00001.html)
> to express my differences with GNU on a mailing list no one
> reads gnu-misc-discuss@gnu.org, but being as no one else appears bound to that
> directive, and being as I consider myself having more to contribute both
> technically and philosophically than they, I too willfully flout that
> directive]

Flout it too much, and I will be forced to do something I didn't yet
do to anyone: ban you from the Emacs lists.

Re: Elisp LSP Server

[dick]

Strange, I seem to be banned everywhere I go, and yet I feel like I have the
most to contribute.

Re: Elisp LSP Server

[Eli Zaretskii]

> From: dick <dick.r.chiang@gmail.com>
> Date: Wed, 13 Oct 2021 07:56:03 -0400
> Cc: emacs-devel@gnu.org
> 
> Strange, I seem to be banned everywhere I go, and yet I feel like I have the
> most to contribute.

You are not yet banned, that was just a warning.  And your
contributions are generally accepted, thanks.

Re: Elisp LSP Server

[Ag Ibragimov]

Re: Elisp LSP Server

[Po Lu]

Ag Ibragimov <agzam.ibragimov@gmail.com> writes:

> You keep saying that we must avoid integrating Emacs with proprietary
> software. In a general sense, I agree. But I'm not sure I can
> completely agree with the reasons you're stating: "Doing otherwise
> would encourage people to use non-free software". There's also the
> other side of that coin: If Emacs doesn't work well in some
> environments - users simply may stop using Emacs.

If that really is a problem, I suggest to work on a replacement for
GitHub that does not involve proprietary software.

Re: Elisp LSP Server

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > You keep saying that we must avoid integrating Emacs with
  > proprietary software. In a general sense, I agree. But I'm not
  > sure I can completely agree with the reasons you're stating:
  > "Doing otherwise would encourage people to use non-free
  > software". There's also the other side of that coin: If Emacs
  > doesn't work well in some environments - users simply may stop
  > using Emacs.

We are failing to communicate because you are making the _popularity_
of Emacs the highest priority.  But that's not our goal.  I did not
write Emacs just to have a success.  There are more important things
at stake here.

The GNU Project has a goal that is more important than its own
success.  Its goal is to help eliminate injustice in computing.  Every
nonfree program does injustice to its users.  See fsf.org/tedx (14
minutes) for explanastion.

The reason we must not encourage people to use nonfree programs
is because we would be leading them into being vicims of injustice.
That would be culpable on our part.

We say, "Foo is a nonfree program so it does injustice to you if you use it."
If we also said, "If you're using Emacs, to solve this problem, use Foo,"
that would be a moral contradiction.  We must not present a nonfree program
as a solution, because it's the problem.

We can make Emacs function for people who use GitHub, but we must not
give GitHub privileges over the other forges that treat their users
ethically.



-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Elisp LSP Server

[dick]

RS> We must not encourage people to use nonfree programs.

"We begrudgingly make emacs work under closed-source platforms like Windows,
OSX, but they are unjust."

I suppose this is no more or less defensible than all of U.S. foreign policy,
of which I've been a longtime beneficiary merely by being a citizen.  The
proceeds from nonfree software have also sheltered and clothed me, and
provided the leisure time required to write free software.  So I guess I'm
rotten to the core.

There's a cult of personality at play here that the ten or so core developers
feel obliged to respect.  In that sense, your continued exhortations,
impassioned but illogical, are working spectacularly well.

Re: Elisp LSP Server

[tomas]

[-- Attachment #1: Type: text/plain, Size: 172 bytes --]

On Wed, Oct 27, 2021 at 02:04:05PM -0400, dick wrote:

[rant elided]

Do you have anything, like, useful or constructive or funny to contribute
here?

C'm on.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Elisp LSP Server

[Eli Zaretskii]

> From: dick <dick.r.chiang@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Wed, 27 Oct 2021 14:04:05 -0400
> 
> RS> We must not encourage people to use nonfree programs.
> 
> "We begrudgingly make emacs work under closed-source platforms like Windows,
> OSX, but they are unjust."
> 
> I suppose this is no more or less defensible than all of U.S. foreign policy,
> of which I've been a longtime beneficiary merely by being a citizen.  The
> proceeds from nonfree software have also sheltered and clothed me, and
> provided the leisure time required to write free software.  So I guess I'm
> rotten to the core.
> 
> There's a cult of personality at play here that the ten or so core developers
> feel obliged to respect.  In that sense, your continued exhortations,
> impassioned but illogical, are working spectacularly well.

Once again, please stop using the GNU mailing list for anti-GNU
propaganda.  We have a special list for this, which I mentioned
before, please take this there.

This is your final warning.  There will be no other.

Re: Elisp LSP Server

[Karl Fogel]

On 27 Oct 2021, dick wrote:
>RS> We must not encourage people to use nonfree programs.
>
>"We begrudgingly make emacs work under closed-source platforms 
>like Windows,
>OSX, but they are unjust."
>
>I suppose this is no more or less defensible than all of 
>U.S. foreign policy,
>of which I've been a longtime beneficiary merely by being a 
>citizen.  The
>proceeds from nonfree software have also sheltered and clothed 
>me, and
>provided the leisure time required to write free software.  So I 
>guess I'm
>rotten to the core.
>
>There's a cult of personality at play here that the ten or so 
>core developers
>feel obliged to respect.  In that sense, your continued 
>exhortations,
>impassioned but illogical, are working spectacularly well.

It's not a cult of personality, it's a well-articulated and 
consistent philosophical position that many people here, not just 
the "ten or so core developers", respect and (to varying extents) 
agree with.

It's also a position that has been pretty openly part of the 
charter of this mailing list and this software project for a long 
time.  You don't have to agree with it -- no one can force you to 
-- but there's no point coming to tango school and complaining 
that they're not teaching salsa.  You're free to start your own 
school (i.e., fork, which includes making your own forums) if you 
want salsa.

Best regards,
-Karl

Re: Elisp LSP Server

[dick]

> it's a well-articulated and consistent philosophical position that many
> people here, not just the "ten or so core developers", respect and (to
> varying extents) agree with.

Well, it doesn't matter as much to me, as someone who cares about emacs and
yes, its popularity, what religion the non-core people subscribe to.

I do enjoy reading the fsf website sometimes to marvel at the doublespeak.

Re: Elisp LSP Server

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1316 bytes --]

On Tue, Oct 12, 2021 at 12:29:31AM -0500, Ag Ibragimov wrote:

[...]

> I don't know if you know this, but Emacs Lisp is probably one of the
> most popular Lisps out there hosted on GitHub and Gitlab [...]

You could start your little contribution by reversing the order above
(e.g. "Gitlab and Github". Or even better: Codeberg, Sourcehut, Gitlab
and so on).

See, Microsoft has at last recognised it can't beat free software in
a direct confrontation ("cancer" remember?). So it is now engaged in
a war of perception. Git is not Git anymore (a decentralised VCS),
but Github [2]. And so on. Accompanied by that "we really, honestly
love Open Source [1], promised".

Now I don't think I'd want to deliberately make Microsoft's user's
(or anyone's) lives deliberately harder. That would not only be highly
unfair, it would also be engaging in the very same misbehaviour we
rightfully denounce the non-free side for.

But I don't think I want to do Microsoft's advertising department
job either. Not without being paid handsomely by them.

Cheers

[0] In the $COMPANY I used to work for, none of the Web developers
   (save one!) knew the difference between Github and Git. And that
   was befote Microsoft's Acquisition.
[1] Yeah, with that spelling: freedom seems to be kind of a F* word
   for Corporatopia.

 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Elisp LSP Server

[Ag Ibragimov]

<tomas@tuxteam.de> writes:

> On Tue, Oct 12, 2021 at 12:29:31AM -0500, Ag Ibragimov wrote:
>
> [...]
>
>> I don't know if you know this, but Emacs Lisp is probably one of the
>> most popular Lisps out there hosted on GitHub and Gitlab [...]
>
> You could start your little contribution by reversing the order above
> (e.g. "Gitlab and Github". Or even better: Codeberg, Sourcehut, Gitlab
> and so on).

I was stating the fact I know. I don't know for sure if other Git Forges
do contain more Elisp than, for example, Common Lisp.

>
> See, Microsoft has at last recognised it can't beat free software in
> a direct confrontation ("cancer" remember?). So it is now engaged in
> a war of perception. Git is not Git anymore (a decentralised VCS),
> but Github [2]. And so on. Accompanied by that "we really, honestly
> love Open Source [1], promised".

I am not in disagreement with everyone saying that GitHub is
bad. Microsoft's acquisition of it has made things even worse. It was as
if the Vatican would've announced that they'd be now in charge of
running CERN, but firmly assured everyone that all existing and future
projects would continue, and the LCH remains an independent research
organization.

> Now I don't think I'd want to deliberately make Microsoft's user's
> (or anyone's) lives deliberately harder. That would not only be highly
> unfair, it would also be engaging in the very same misbehaviour we
> rightfully denounce the non-free side for.

I do agree with that 100%. 

Re: Elisp LSP Server

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Instead of making it easier for people to put their customizations on
  > > GitHub, how about making it harder?  It would encourage people to move
  > > to systems that do not require running proprietary software.

  > Do you seriously believe that? We should deliberately hinder our own
  > progress, hoping it somehow would hurt GitHub?

The issue here is what constitutes "progress".  What does progress
mean, in any given situation?  Would a language server for Emacs Lisp
be progress?  It would be a mistake to assume so.

Our work proceeds by replacing nonfree programs.  Why do that?
Because each one is an injustice -- denies freedom to its users by
giving its developer power over them.  A nonfree program is not an
inferior solution; it is a problem.  We can fix that problem by
replacing it with free software, so that it can't do wrong to people
any more.

When we can't supersede a particular nonfree program in the near
future, we should be careful not to be led into enhancing its use.
Thus, when someone says, "Let's implement XYZ; it will be convenient
for users that want to do ABC," we must ask: What is ABC, and is
making ABC more convenient a good thing?

Popular nonfree programs have many users, and they will innocently
suggest we direct our work to make their use of nonfree software more
convenient.  I don't blame them for suggesting this, but it is not
what we should do.

We have to do careful thinking about what constitutes progress in this
particular situation.  We have to find the various options --
including mixtures of the obvious options -- and think about what good
and bad effects they would have.  Finding the right choice may be
subtle and complex.

But we can't find the right choice if we don't evaluate it based on
the right values and goals.  Facilitating the use of VS Code or GitHub
is not a goal; encouraging people to stop using them is a goal.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Elisp LSP Server

[Ag Ibragimov]

Richard Stallman <rms@gnu.org> writes:

> Our work proceeds by replacing nonfree programs.  Why do that?
> Because each one is an injustice -- denies freedom to its users by
> giving its developer power over them.  A nonfree program is not an
> inferior solution; it is a problem.  We can fix that problem by
> replacing it with free software, so that it can't do wrong to people
> any more.
>
> When we can't supersede a particular nonfree program in the near
> future, we should be careful not to be led into enhancing its use.
> Thus, when someone says, "Let's implement XYZ; it will be convenient
> for users that want to do ABC," we must ask: What is ABC, and is
> making ABC more convenient a good thing?

> Popular nonfree programs have many users, and they will innocently
> suggest we direct our work to make their use of nonfree software more
> convenient.  I don't blame them for suggesting this, but it is not
> what we should do.
>
> We have to do careful thinking about what constitutes progress in this
> particular situation.  We have to find the various options --
> including mixtures of the obvious options -- and think about what good
> and bad effects they would have.  Finding the right choice may be
> subtle and complex.
>
> But we can't find the right choice if we don't evaluate it based on
> the right values and goals.  Facilitating the use of VS Code or GitHub
> is not a goal; encouraging people to stop using them is a goal.
>

I profoundly appreciate you for graciously explaining this all. I
understand the points, and I don't think I'm in complete disagreement
with them. My mistake was to speculate about an idea. Which, upon the
materialization (if ever happens), may or may not be efficacious and
valuable for Emacs users. But my greatest mistake was to suggest that it
could potentially bring convenience to users of nonfree
services and proprietary software.

To be clear, I wasn't suggesting that someone should work on it. I was
merely curious if anyone had attempted to build it. And the potential
technical challenges to watch for.

I still think that there's a significant utilitarian value of simply
consolidating and unifying numerous different dissimilarities existing
today for different programming languages in Emacs. Polyglot programming
has become a norm in the industry, and sadly, Emacs has earned quite an
infamous reputation for being difficult to learn. And programming
language books and tutorials, sometimes even after acknowledging the
incredible power of Emacs and recommending to learn it, often warn the
learners to avoid making the mistake of trying to learn both the
language and Emacs simultaneously.

And as I mentioned before, even experienced users of Emacs often have to
abandon it after fruitless attempts to make it work nicely with a
different language. Something simple as: "it keeps screwing up the
formatting" can become the source of enormous frustration.

But finally, eglot and lsp-mode started challenging this status-quo. And
I honestly think we should embrace these efforts and pave the way for
Emacs to become the ultimate, egalitarian programming environment for
everyone. I hate to bring this up again, but VSCode (if you haven't yet
noticed) is already ahead of us in this competition. No, I don't think
I'm exaggerating. I think it's an accurate assessment. Yes, Emacs still
has a few cards up the sleeve, but for how long?

You have had witnessed the rise and fall of legendary Lisp machines with
your own eyes. And I think Emacs, even though maybe not ideal, but
indeed the best "Lisp machine" that we have today. And I don't think
Emacs is guaranteed not to face a similar fate. 

I'm sure people would use classic counter-argumentation like: "it's not
a sprint, it's a marathon. Just wait another twenty years, and you'd
see...", and I have used the same argument many times myself. But the
pace of growth and expansion of VSCode, various plugins, and services
that support it come across as almost unprecedented.

Unfortunately, I cannot offer you sound, reasonable suggestions that are
guaranteed not to challenge to the slightest your firm beliefs and
uncompromising philosophy, and I wouldn't even try. But I plead you to
take this adversary seriously, and I am pretty sure I am not the first
one to do that. We probably need a better strategy. Better than just:
"encouraging people to stop using them is a goal".

Thank you.

Re: Elisp LSP Server

[Po Lu]

Ag Ibragimov <agzam.ibragimov@gmail.com> writes:

> But finally, eglot and lsp-mode started challenging this status-quo. And
> I honestly think we should embrace these efforts and pave the way for
> Emacs to become the ultimate, egalitarian programming environment for
> everyone. I hate to bring this up again, but VSCode (if you haven't yet
> noticed) is already ahead of us in this competition. No, I don't think
> I'm exaggerating. I think it's an accurate assessment. Yes, Emacs still
> has a few cards up the sleeve, but for how long?

But again, making a language server for Emacs Lisp will not improve
Emacs' support for other programing languages.

And it will encourage Emacs users to use non-free software.

So, while supporting the language server protocol for _other_ servers is
something worthwhile, exposing Emacs functionality to other programs
through the LSP is not.

Re: Elisp LSP Server

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: rms@gnu.org,  philipk@posteo.net,  psainty@orcon.net.nz,
>  emacs-devel@gnu.org,  joaotavora@gmail.com,  mardani29@yahoo.es
> Date: Wed, 13 Oct 2021 13:20:43 +0800
> 
> But again, making a language server for Emacs Lisp will not improve
> Emacs' support for other programing languages.

Yes, and developing feature XYZ for Emacs will not improve Emacs
support for some other feature ABC.  How is that relevant?

> And it will encourage Emacs users to use non-free software.
> 
> So, while supporting the language server protocol for _other_ servers is
> something worthwhile, exposing Emacs functionality to other programs
> through the LSP is not.

This makes very little sense to me: we are supposed to refrain from
developing an Emacs-specific feature whose primary audience is Emacs
itself, because someone might find a way of using it with proprietary
software?  By the same logic, the GNU Project should discontinue GCC
and Binutils, because they can be used to develop and produce
proprietary programs.

Please tell me that I misunderstood something very important in what
you had in mind.

Re: Elisp LSP Server

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> Please tell me that I misunderstood something very important in what
> you had in mind.

Yes, I was stating that this feature won't find an audience in Emacs
users, but will instead find one in people who want to use GitHub and VS
Code to edit Emacs Lisp.

Re: Elisp LSP Server

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: agzam.ibragimov@gmail.com,  rms@gnu.org,  philipk@posteo.net,
>   psainty@orcon.net.nz,  emacs-devel@gnu.org,  joaotavora@gmail.com,
>   mardani29@yahoo.es
> Date: Wed, 13 Oct 2021 20:49:11 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Please tell me that I misunderstood something very important in what
> > you had in mind.
> 
> Yes, I was stating that this feature won't find an audience in Emacs
> users, but will instead find one in people who want to use GitHub and VS
> Code to edit Emacs Lisp.

How can it NOT find any users in Emacs?  We edit Lisp all the time,
don't we?  Shouldn't advanced features for indentation, syntax
highlight, refactoring, etc. of Emacs Lisp programs be very welcome in
Emacs?

Re: Elisp LSP Server

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> How can it NOT find any users in Emacs?  We edit Lisp all the time,
> don't we?  Shouldn't advanced features for indentation, syntax
> highlight, refactoring, etc. of Emacs Lisp programs be very welcome in
> Emacs?

Yes, but what the OP proposed was not to implement these new features in
Emacs, but instead to expose existing features such as imenu to other
programs by making Emacs act as a language server implementing the
language server protocol.

It would be nice if Emacs does gain the advanced features you mentioned
above, but they shouldn't be implemented as a language server, and
either way that's not what the OP was proposing.

Thanks.

Re: Elisp LSP Server

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: philipk@posteo.net,  rms@gnu.org,  psainty@orcon.net.nz,
>   emacs-devel@gnu.org,  joaotavora@gmail.com,  mardani29@yahoo.es,
>   agzam.ibragimov@gmail.com
> Date: Wed, 13 Oct 2021 21:37:09 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > How can it NOT find any users in Emacs?  We edit Lisp all the time,
> > don't we?  Shouldn't advanced features for indentation, syntax
> > highlight, refactoring, etc. of Emacs Lisp programs be very welcome in
> > Emacs?
> 
> Yes, but what the OP proposed was not to implement these new features in
> Emacs, but instead to expose existing features such as imenu to other
> programs by making Emacs act as a language server implementing the
> language server protocol.

And Emacs itself couldn't be that server's client?

> It would be nice if Emacs does gain the advanced features you mentioned
> above, but they shouldn't be implemented as a language server, and
> either way that's not what the OP was proposing.

They should be implemented in Emacs, but the data necessary for the
implementation is supposed to come from the language server.  Right?

Re: Elisp LSP Server

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> They should be implemented in Emacs, but the data necessary for the
> implementation is supposed to come from the language server.  Right?

No, it works the other way around; once Emacs implements the feature, it
can be exposed as a language server, but if Emacs doesn't, there is
nothing in the language server protocol that will magically give Emacs
that functionality.

Thanks.

Re: Elisp LSP Server

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Po Lu <luangruo@yahoo.com>
>> Cc: agzam.ibragimov@gmail.com,  rms@gnu.org,  philipk@posteo.net,
>>   psainty@orcon.net.nz,  emacs-devel@gnu.org,  joaotavora@gmail.com,
>>   mardani29@yahoo.es
>> Date: Wed, 13 Oct 2021 20:49:11 +0800
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Please tell me that I misunderstood something very important in what
>> > you had in mind.
>> 
>> Yes, I was stating that this feature won't find an audience in Emacs
>> users, but will instead find one in people who want to use GitHub and VS
>> Code to edit Emacs Lisp.
>
> How can it NOT find any users in Emacs?  We edit Lisp all the time,
> don't we?  Shouldn't advanced features for indentation, syntax
> highlight, refactoring, etc. of Emacs Lisp programs be very welcome in
> Emacs?

I think the question here is would an Elisp LSP server be able to offer
anything over a direct implementation within Emacs. That this seems
improbable was mentioned previously, so in effect this would only
improve the ELisp support for non-Emacs editors (why there would be
interest for that I find hard to understand, but that is not the point).

-- 
	Philip Kaludercic

Re: Elisp LSP Server

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > How can it NOT find any users in Emacs?  We edit Lisp all the time,
  > don't we?  Shouldn't advanced features for indentation, syntax
  > highlight, refactoring, etc. of Emacs Lisp programs be very welcome in
  > Emacs?

We want those features (and, indeed, we have a lot of them already).
I doubt anyone disagrees about that.

The question is whether to encapsulate them as a "language server"
so that ther editors can use them as a sort of "back end".

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Elisp LSP Server

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > This makes very little sense to me: we are supposed to refrain from
  > developing an Emacs-specific feature whose primary audience is Emacs
  > itself,

I think that is a misunderstanding of what the issue is.

I think we all agree that we want to develop code within Emacs to
analyze Emacs Lisp code in whatever ways are useful.  Those results
will be available in Emacs for whatever use we want to make of them.

The question at issue -- unless I have misunderstood -- is NOT about
doing that.

            Because someone might find a way of using it with proprietary
  > software?

The proposal at issue -- unless I have misunderstood -- is to develop
a server interface for Emacs specifically to make those analysis
results available to other editors (which may mean, principally
VScode).

Please correct me if I have misunderstood the facts of the proposal
and its presuppositions.

If I understood right, this can't be of any benefit to editing with
Emacs.  Because all the analysis results we can produce will be
available anyway for editing in Emacs.  Editing Emacs Lisp code with
Emacs will not be improved by this.  The only benefit of this
additional work will be to help people edit it with VScode.

That's why I think this would be an own goal.

Would it be possible to make Emacs launch a child Emacs and use it as
a language server to get the results of these analyses?  I suppose so.
Just as it is possible to launch Emacs inside M-x term and edit in
that child Emacs.

Is there is any practical advantage to starting a child Emacs to do
this analysis in?  I don't see it.  Why not do it in the initial Emacs
process?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Elisp LSP Server

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: luangruo@yahoo.com, agzam.ibragimov@gmail.com, philipk@posteo.net,
> 	psainty@orcon.net.nz, emacs-devel@gnu.org,
> 	joaotavora@gmail.com, mardani29@yahoo.es
> Date: Thu, 14 Oct 2021 18:26:01 -0400
> 
> The proposal at issue -- unless I have misunderstood -- is to develop
> a server interface for Emacs specifically to make those analysis
> results available to other editors (which may mean, principally
> VScode).

That interface will be first and foremost useful for Emacs, because
that's the main/only editor people use to edit Emacs Lisp programs.
It's true that other editors will be able to use that, but the chances
of someone using VSCode to edit Emacs Lisp are slim at best.

> If I understood right, this can't be of any benefit to editing with
> Emacs.  Because all the analysis results we can produce will be
> available anyway for editing in Emacs.

This assumption/conclusion misses an important point, see below.

> Would it be possible to make Emacs launch a child Emacs and use it as
> a language server to get the results of these analyses?  I suppose so.
> Just as it is possible to launch Emacs inside M-x term and edit in
> that child Emacs.
> 
> Is there is any practical advantage to starting a child Emacs to do
> this analysis in?  I don't see it.  Why not do it in the initial Emacs
> process?

Doing that in the same process will make Emacs slower, because Emacs
is fundamentally single-threaded.  On modern systems that routinely
have 8 or 16 execution units, off-loading some of the processing to
asynchronous code that exploits additional execution units is a
significant advantage, and allows us to be much more scalable.  If
Emacs had true concurrency, we could do that in the same process by
arranging for additional threads.  But since that is not available,
running a language processor in a separate process is the next best.
And since a protocol for that already exists, reusing the protocol in
this case sounds like an idea worth exploring.  Of course, if it turns
out the LSP protocol doesn't help much, we could always implement our
own protocol (which would then solve the fear of VSCode using that,
although I think that's us being haunted by the shadow of a dwarf).

Re: Elisp LSP Server

[Tim Cross]

Eli Zaretskii <eliz@gnu.org> writes:

> Doing that in the same process will make Emacs slower, because Emacs
> is fundamentally single-threaded.  On modern systems that routinely
> have 8 or 16 execution units, off-loading some of the processing to
> asynchronous code that exploits additional execution units is a
> significant advantage, and allows us to be much more scalable.  If
> Emacs had true concurrency, we could do that in the same process by
> arranging for additional threads.  But since that is not available,
> running a language processor in a separate process is the next best.
> And since a protocol for that already exists, reusing the protocol in
> this case sounds like an idea worth exploring.

That is an aspect I hadn't considered. Given the number of desktop
systems sitting there with most cores doing nothing, this could be a
useful addition indeed. 

> Of course, if it turns
> out the LSP protocol doesn't help much, we could always implement our
> own protocol (which would then solve the fear of VSCode using that,
> although I think that's us being haunted by the shadow of a dwarf).

I love that expression - it succinctly sums up my feelings concerning
the fears raised in this thread.  

Re: Elisp LSP Server

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

In the GNU Project we recommend avoiding the term "ecosystem" to
describe software development, because it frames the situation in
amoral terms.

See https://gnu.org/philosophy/words-to-avoid.html#Ecosystem
for more explanation.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Elisp LSP Server

[Ag Ibragimov]

Richard Stallman <rms@gnu.org> writes:

> In the GNU Project we recommend avoiding the term "ecosystem" to
> describe software development, because it frames the situation in
> amoral terms.

An honest question, sir. May I? First of all, thank you for pointing
that out. I was unaware. You, see (as you have probably noticed), I am
not a native English speaker. I don't have an extended
vocabulary. That's just so you know - I'm not trolling you. Even though
that term is widely used in our industry, I would try to avoid it in
written communication from now on. But I'm afraid I may involuntarily
use it while talking. I tried searching for some better alternatives,
but I couldn't find anything that sounds like a good replacement for
it. Even dictionaries use examples such as: "Silicon Valley's
entrepreneurial ecosystem". And I have to say, after reading that
paragraph from words-to-avoid.html, that example made my eyes
twitch. What would you suggest to use instead?

Re: Elisp LSP Server

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > An honest question, sir. May I? First of all, thank you for pointing
  > that out. I was unaware. You, see (as you have probably noticed), I am
  > not a native English speaker. I don't have an extended
  > vocabulary.

That's nothing to apologize for.  Most of us grow up speaking one
language, or maybe two; after that, to learn other languages is hard
work.

  >  I tried searching for some better alternatives,
  > but I couldn't find anything that sounds like a good replacement for
  > it. Even dictionaries use examples such as: "Silicon Valley's
  > entrepreneurial ecosystem".

The term "ecosystem" -- and the amoral approach associated with it --
are widespread in the computing field.  Using that term encourages
that amoral approach.

In the free software movement that we have a different approach: free
software is just and nonfree software is unjust.  So we don't want to
find another word to use instead for the same meaning.  The meaning
is what we want to shun.

I suggest you watch fsf.org/tedx as an introduction to the
free software movement.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Elisp LSP Server

[Dmitry Gutov]

On 10.10.2021 15:48, Ag Ibragimov wrote:
> LSP, for many Emacsen has become a true game-changer. You don't have to re-invent the wheel for every single different language, no need to map and re-map the keys (separately for every language) or struggle when a language mode package authors fail to include some features.
> Wouldn't it be ironic if soon, Emacs gets truly fantastic support for dozens of languages (btw, that list is still growing), yet Elisp would have to ride like a second-class passenger?

I see where you going with this: if we just consolidated our languages 
support code as an LSP client and moved Elisp implementation to an LSP 
server, we might be better able to focus on a language support UI which 
is the same across languages.

Unfortunately, it's not so simple: even aside from the JSON transport 
(which we could get around by offering different transport options), the 
LSP protocol has limitations which would limit us on certain features.

For example, you can 'C-u M-.' in an Emacs Lisp buffer, then press TAB 
and see all the available symbols you can navigate to; I'm told this is 
either impossible to support with the current LSP protocol, or pretty 
hard to do.

Other examples are in how completions information is presented; it's 
basically tailored to VS Code's completion popup. LSP clients are 
saddled with certain drawbacks when you use company-mode.

Maybe someday.

Instead, we're currently focusing on reusable UI workflows.

Re: Elisp LSP Server

[Po Lu]

Ag Ibragimov <agzam.ibragimov@gmail.com> writes:

> Not quite. I'm not suggesting at all integrating with proprietary
> software. I'm merely curious if anyone has ever thought about
> implementing an LSP server for Emacs Lisp.  LSP is a standard and an
> open specification. There already exist several dozen implementations
> of Language Protocol servers for many different languages.

But an LSP server for Emacs Lisp would, for the reasons you have
mentioned, be used for proprietary software.  The ability of people to
utilize imenu in Emacs Lisp code on GitHub would encourage people to use
the proprietary software that GitHub requires.

> And many of them are perfectly usable in Emacs, thanks to eglot and
> lsp-mode. And that's great because adhering to the standard makes it a
> smooth experience for the user.

> One of the main reasons people leave Emacs, sometimes even after many
> years of using it, is the substandard language support.  They often
> have to start programming in another language, and it takes too much
> effort to make Emacs work for a specific language.  Simple things like
> goto definition and autocomplete often won't work as expected. People
> often have to learn a bunch of independent Elisp packages and figure
> out ways to make them work together nicely.  So many times I've heard:
> "I love Emacs, used it for almost [so many years] but I had to work
> with a language [X] and moved to [another] IDE that has first-class
> support for it."

> LSP, for many Emacsen has become a true game-changer. You don't have
> to re-invent the wheel for every single different language, no need to
> map and re-map the keys (separately for every language) or struggle
> when a language mode package authors fail to include some features.

But that's an example of other software being useful in Emacs, which
encourages people to switch to free software such as Emacs.

> Wouldn't it be ironic if soon, Emacs gets truly fantastic support for
> dozens of languages (btw, that list is still growing), yet Elisp would
> have to ride like a second-class passenger?
> I think creating Elisp LSP server would provide consistency with all
> other languages already supported.

How is Emacs Lisp a second class passenger in Emacs?  And how will
making a language server out of the existing Lisp support make that
better?

> And as a side effect, that would also make it possible for better
> navigation and code introspection of Elisp outside of Emacs.

But is there any real use-case for this, outside of proprietary
software?

> Personally, I don't care if a proprietary software vendor like GitHub
> introduces better features for a language that is the ultimate epitome
> of free software.  Their non-free software would still be written in
> Emacs (at least some parts of it) - the ultimate tool with unmatched
> support for programming languages.

Proprietary software written in Emacs is still proprietary software.
And besides, there is no reason to make the job of proprietary software
dealers easier.

Re: Elisp LSP Server

[Alexandre Garreau]

Le lundi 11 octobre 2021, 09:34:56 CEST Po Lu a écrit :
> Ag Ibragimov <agzam.ibragimov@gmail.com> writes:
> > Not quite. I'm not suggesting at all integrating with proprietary
> > software. I'm merely curious if anyone has ever thought about
> > implementing an LSP server for Emacs Lisp.  LSP is a standard and an
> > open specification. There already exist several dozen implementations
> > of Language Protocol servers for many different languages.
> 
> But an LSP server for Emacs Lisp would, for the reasons you have
> mentioned, be used for proprietary software.  The ability of people to
> utilize imenu in Emacs Lisp code on GitHub would encourage people to use
> the proprietary software that GitHub requires.

Seeing those regular discussions about anti-freedom interoperability 
(while usually interoperability is a user freedom), I still hope that 
there is a solution, such as copyleft was decades ago… and can’t help but 
think that if SSPL is juridically sound, it could help so (like, extending 
copyleft beyond dynamic linking, to general inter-process communication 
and services).  Has FSF advanced on that subject?

Re: Elisp LSP Server

[Po Lu]

Alexandre Garreau <galex-713@galex-713.eu> writes:

> Seeing those regular discussions about anti-freedom interoperability 
> (while usually interoperability is a user freedom), I still hope that 
> there is a solution, such as copyleft was decades ago… and can’t help but 
> think that if SSPL is juridically sound, it could help so (like, extending 
> copyleft beyond dynamic linking, to general inter-process communication 
> and services).  Has FSF advanced on that subject?

AFAIU, the FSF considers the SSPL a proprietary software license.

I don't understand the specifics though, and it would be nice if someone
explained.  Thanks in advance :)

Re: Elisp LSP Server

[Alexandre Garreau]

Le lundi 11 octobre 2021, 12:46:47 CEST Po Lu a écrit :
> Alexandre Garreau <galex-713@galex-713.eu> writes:
> > Seeing those regular discussions about anti-freedom interoperability
> > (while usually interoperability is a user freedom), I still hope that
> > there is a solution, such as copyleft was decades ago… and can’t help
> > but think that if SSPL is juridically sound, it could help so (like,
> > extending copyleft beyond dynamic linking, to general inter-process
> > communication and services).  Has FSF advanced on that subject?
> 
> AFAIU, the FSF considers the SSPL a proprietary software license.
> 
> I don't understand the specifics though, and it would be nice if someone
> explained.  Thanks in advance :)

Where did you read that? I still can’t see it mentioned in licenses/
licenses-list, and last time I asked they said me they were still 
reviewing it

Re: Elisp LSP Server

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

We have not yet studied the SSPL.  Many other things have seemed
to be higher priority.  But since it has been rejected as "open source",
I suppose it won't qualify as free either.

I tend to think that trying to impose copyright-based rules
that stretch beyond the bounds of one single program
is abuse of copyright, based on what I learned in the past.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Elisp LSP Server

[tomas]

[-- Attachment #1: Type: text/plain, Size: 688 bytes --]

On Mon, Oct 11, 2021 at 05:16:31PM -0400, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
> We have not yet studied the SSPL.  Many other things have seemed
> to be higher priority.  But since it has been rejected as "open source",
> I suppose it won't qualify as free either.

I think the OP was confusing things. The Open Source Initiative
(OSI) has, as you hint, categorised SSPL as "not an open source
license" [1].

Cheers

[1] https://opensource.org/node/1099

 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Elisp LSP Server

[Alexandre Garreau]

Le lundi 11 octobre 2021 23:16:31 CEST, vous avez écrit :
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
> We have not yet studied the SSPL.  Many other things have seemed
> to be higher priority.  But since it has been rejected as "open source",
> I suppose it won't qualify as free either.

I wasn’t sure, and was hoping to see yet another occasion to show a 
distinction between free and open-source philosophy.  The only case where 
SSPL obligates the (non-end) user to do something, is actually in the 
interest of the end-user… “if it’s the same service, publish everything”…

> I tend to think that trying to impose copyright-based rules
> that stretch beyond the bounds of one single program
> is abuse of copyright, based on what I learned in the past.

That may be very wise.  But I’m not sure…

What if such a thing was used for a GCC LSP-server? THAT would be 
extreeemely useful, and was never done because proprietary software is so 
ubiquitous that it would benefit proprietary software… because currently 
dynamic linkage between gcc and emacs would be fine because it would force 
both to be free, while the *same usage* using a binary-agnostic protocol 
would not spread the GPL…  to me it seems non-technical, and it’s 
preventing technical integration between GNU software so…

Re: Elisp LSP Server

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

We have moral criteria for judging whether a license qualifies as
free.  See https://gnu.org/philosophy/free-sw.html.  When we evaluate
the SSPL, must judge whether it complies with those rules.
If not, we can't accept it.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Elisp LSP Server

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > The OP was proposing that Emacs integrate (by providing imenu
  > > functionality to Visual Studio Code through LSP) with a particular
  > > feature of GitHub, where, through proprietary JavaScript and a
  > > proprietary plugin, anyone can press "." to immediately view code from
  > > GitHub in VS Code.

  > Not quite. I'm not suggesting at all integrating with proprietary
  > software. I'm merely curious if anyone has ever thought about
  > implementing an LSP server for Emacs Lisp.

To implement an LSP server for Emacs Lisp, in the abstract, is a good
thing to do.  However, doing it in special relation with GitHub is the
wrong way.  Would someone please explain concretely how this proposal
relates to GitHub?

  > Personally, I don't care if a proprietary software vendor like
  > GitHub introduces better features for a language that is the
  > ultimate epitome of free software.

The GNU Project is very concerned to help replace GitHub, _not_ to
enhance it.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Elisp LSP Server

[Mathias Dahl]

[-- Attachment #1: Type: text/plain, Size: 805 bytes --]

> > ... with a particular feature of GitHub, where, through proprietary
> > JavaScript and a proprietary plugin, anyone can press "." to
> > immediately view code from GitHub in VS Code.

About this convenience feature of being able to very easily open VS
Code when browsing GitHub, we could develop our own
Chrome/Edge/Firefox extension to allow the same workflow if we
wanted. I haven't written web browser extensions for Firefox but I
have for Chrome and I have some idea on how that could
happen. Conceptually it's not very hard, but there are some work
involved of course, especially if we wanted it to be working across
multiple platforms.

Do we think having such a feature would greatly increase the chance
that someone would be contributing to Emacs development or development
of Emacs packages?

[-- Attachment #2: Type: text/html, Size: 927 bytes --]

Re: Elisp LSP Server

[Dmitry Gutov]

On 22.10.2021 19:23, Mathias Dahl wrote:
> About this convenience feature of being able to very easily open VS
> Code when browsing GitHub, we could develop our own
> Chrome/Edge/Firefox extension to allow the same workflow if we
> wanted.

The OP was referring to being able to open "VS Code" (a version of it) 
inside the browser. It's a "cloud IDE" sort of thing.

Re: Elisp LSP Server

[Alexandre Garreau]

Le vendredi 22 octobre 2021, 18:40:06 CEST Dmitry Gutov a écrit :
> On 22.10.2021 19:23, Mathias Dahl wrote:
> > About this convenience feature of being able to very easily open VS
> > Code when browsing GitHub, we could develop our own
> > Chrome/Edge/Firefox extension to allow the same workflow if we
> > wanted.
> 
> The OP was referring to being able to open "VS Code" (a version of it)
> inside the browser. It's a "cloud IDE" sort of thing.

Is there an implementation/backend of VS Code implemented in javascript, 
or transpiled to it?

It would anyway, without having to transpile C to js (which already 
exists, but not including X ofc (hopefully?)), or elisp to js (which may 
already exist or be easily doable from what exists), be possible to 
develop not really an extension but a plugin that would allow open emacs 
inside frames such as it was possible to watch videos before <video> and 
firefox to include its own player

Re: Elisp LSP Server

[Tim Cross]

Alexandre Garreau <galex-713@galex-713.eu> writes:

> Le vendredi 22 octobre 2021, 18:40:06 CEST Dmitry Gutov a écrit :
>> On 22.10.2021 19:23, Mathias Dahl wrote:
>> > About this convenience feature of being able to very easily open VS
>> > Code when browsing GitHub, we could develop our own
>> > Chrome/Edge/Firefox extension to allow the same workflow if we
>> > wanted.
>> 
>> The OP was referring to being able to open "VS Code" (a version of it)
>> inside the browser. It's a "cloud IDE" sort of thing.
>

> Is there an implementation/backend of VS Code implemented in javascript, 
> or transpiled to it?
>
I think vs code is an electron app, which means it is largely
implemented in javascript. It uses javascript for it's extension system.

> It would anyway, without having to transpile C to js (which already 
> exists, but not including X ofc (hopefully?)), or elisp to js (which may 
> already exist or be easily doable from what exists), be possible to 
> develop not really an extension but a plugin that would allow open emacs 
> inside frames such as it was possible to watch videos before <video> and 
> firefox to include its own player

I think the best phrase for here is "Nothing to see here, please move
on". There are already solutions to pass off content from the browser to
Emacs (e.g. org protocol and the edit with emacs extension).

I also think what is really needed in the free software community is not
an extension which allows emacs to run inside a browser or an LSP mode,
but rather a well designed free javascript library which would
facilitate the development of web UIs that would allow sites like
savannah.gnu.org to have an interface as functional and modern looking
as github using free software. I'm sure more people would be just as
happy to use savannah for their project hosting if it didn't look and
feel like being beamed back to 1999 every time you used the site. Such a
library could even be designed and developed with a focus which makes
integration with other free software projects even easier, further
increasing the likelihood of free software being used/preferred in
modern web environments.  

Re: Elisp LSP Server

[Stefan Kangas]

(Moved to emacs-tangents.)

Tim Cross <theophilusx@gmail.com> writes:

> I also think what is really needed in the free software community is
> [...] a well designed free javascript library which would facilitate
> the development of web UIs that would allow sites like
> savannah.gnu.org to have an interface as functional and modern looking
> as github using free software.

What's wrong with e.g. React (MIT License) or Angular (MIT License)?

Re: Elisp LSP Server

[Tim Cross]

Stefan Kangas <stefankangas@gmail.com> writes:

> (Moved to emacs-tangents.)
>
> Tim Cross <theophilusx@gmail.com> writes:
>
>> I also think what is really needed in the free software community is
>> [...] a well designed free javascript library which would facilitate
>> the development of web UIs that would allow sites like
>> savannah.gnu.org to have an interface as functional and modern looking
>> as github using free software.
>
> What's wrong with e.g. React (MIT License) or Angular (MIT License)?

There is quite a bit considered "wrong" about the MIT License (see the
GNU Project page on free licenses). However, I probably should have said
"framework" rather than library. The problem with both react and angular
is that much of the time, sites are created based on those technologies,
but using higher level abstractions for components or infrastructure
that sit on top of a react or angular base which do not have free
licenses.

What I was meaning to say is that what is needed is a free technology
stack where all the components - e.g. authentication layer, middleware,
model/database, ui/view components and controllers are all free
software and which have appropriate free software licenses. 

Re: Elisp LSP Server

[Dmitry Gutov]

On 22.10.2021 22:59, Tim Cross wrote:
> but rather a well designed free javascript library which would
> facilitate the development of web UIs that would allow sites like
> savannah.gnu.org to have an interface as functional and modern looking
> as github using free software. I'm sure more people would be just as
> happy to use savannah for their project hosting if it didn't look and
> feel like being beamed back to 1999 every time you used the site. Such a
> library could even be designed and developed with a focus which makes
> integration with other free software projects even easier, further
> increasing the likelihood of free software being used/preferred in
> modern web environments.

JS frameworks, CSS frameworks and libraries, collections of components 
and icons, modern-looking and Free, all are available.

What is missing are people who are familiar with such tools that are, 
somehow, interested in improving Savannah.

Re: Elisp LSP Server

[Tim Cross]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 22.10.2021 22:59, Tim Cross wrote:
>> but rather a well designed free javascript library which would
>> facilitate the development of web UIs that would allow sites like
>> savannah.gnu.org to have an interface as functional and modern looking
>> as github using free software. I'm sure more people would be just as
>> happy to use savannah for their project hosting if it didn't look and
>> feel like being beamed back to 1999 every time you used the site. Such a
>> library could even be designed and developed with a focus which makes
>> integration with other free software projects even easier, further
>> increasing the likelihood of free software being used/preferred in
>> modern web environments.
>
> JS frameworks, CSS frameworks and libraries, collections of components and
> icons, modern-looking and Free, all are available.
>
> What is missing are people who are familiar with such tools that are, somehow,
> interested in improving Savannah.


To a point, that is very true. However, assembling a full stack
framework where all the components are free and work well together is
not as straight-forward as it may seem. Considerable knowledge and
experience is required. If a defined free framework existed, then the
search for someone to update savannah would be more likely to succeed
because all you would then need is someone with the necessary drive and
knowledge of JS. More generally, someone wanting to create a new site
could just use the framework and have confidence that all the components
are free - so much broader benefit than just savannah. 

Re: Elisp LSP Server

[Dmitry Gutov]

On 23.10.2021 01:48, Tim Cross wrote:
> To a point, that is very true. However, assembling a full stack
> framework where all the components are free and work well together is
> not as straight-forward as it may seem. Considerable knowledge and
> experience is required. If a defined free framework existed, then the
> search for someone to update savannah would be more likely to succeed
> because all you would then need is someone with the necessary drive and
> knowledge of JS.

A number of frameworks (popular and well-known) exist as well. But a 
framework is not a "blank site to be filled with contents". That would 
be more like a CMS (which exist as well, but serve a different purpose).

> More generally, someone wanting to create a new site
> could just use the framework and have confidence that all the components
> are free - so much broader benefit than just savannah.

There are existing projects, to be forked/adapted/customized. Even 
Gitlab, for example. Take the community edition (it's Free) and modify 
to your heart's content. But that, again, needs people.

Re: Elisp LSP Server

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

It would be fine to release a free Javascript front end for Savannah,
but it should be released as a program users can install, not as
something that comes at them automatically from a server.

Making the program separately installable enables users to get the
full benefits of free software as described in
https://gnu.org/philosophy/free-software-even-more-important.html.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Elisp LSP Server

[Alexandre Garreau]

Le vendredi 22 octobre 2021, 21:59:26 CEST Tim Cross a écrit :
> Alexandre Garreau <galex-713@galex-713.eu> writes:
> > Le vendredi 22 octobre 2021, 18:40:06 CEST Dmitry Gutov a écrit :
> >> On 22.10.2021 19:23, Mathias Dahl wrote:
> >> > About this convenience feature of being able to very easily open VS
> >> > Code when browsing GitHub, we could develop our own
> >> > Chrome/Edge/Firefox extension to allow the same workflow if we
> >> > wanted.
> >> 
> >> The OP was referring to being able to open "VS Code" (a version of
> >> it)
> >> inside the browser. It's a "cloud IDE" sort of thing.
> > 
> > Is there an implementation/backend of VS Code implemented in
> > javascript, or transpiled to it?
> 
> I think vs code is an electron app, which means it is largely
> implemented in javascript. It uses javascript for it's extension system.
> > It would anyway, without having to transpile C to js (which already
> > exists, but not including X ofc (hopefully?)), or elisp to js (which
> > may already exist or be easily doable from what exists), be possible
> > to develop not really an extension but a plugin that would allow open
> > emacs inside frames such as it was possible to watch videos before
> > <video> and firefox to include its own player
> 
> I think the best phrase for here is "Nothing to see here, please move
> on". There are already solutions to pass off content from the browser to
> Emacs (e.g. org protocol and the edit with emacs extension).
> 
> I also think what is really needed in the free software community is not
> an extension which allows emacs to run inside a browser or an LSP mode,
> but rather a well designed free javascript library which would
> facilitate the development of web UIs that would allow sites like
> savannah.gnu.org to have an interface as functional and modern looking
> as github using free software. I'm sure more people would be just as
> happy to use savannah for their project hosting if it didn't look and
> feel like being beamed back to 1999 every time you used the site. Such
> a library could even be designed and developed with a focus which makes
> integration with other free software projects even easier, further
> increasing the likelihood of free software being used/preferred in
> modern web environments.

Of course not.  The big monopolies, that is AngularJS, React, CodeIgniter, 
etc. are all already free-software.  Most of the software running server-
side also is free-software.

What is very anti-freedom is the very paradigm they enforce, that is “the 
cloud”, or more specifically: to make everything depend on servers, that 
send on-the-go obfuscated (“minified”, which actually often includes some 
steps of compilation such as open-coding, constant-propagation, code 
suppression, etc.) javascript, that could change at each page load, 
resulting on something that, even if it had a free-software license, would 
be very non-free in term of collective users’ rights (each user cannot 
inspect each script individually, that they will be the only one to see, 
so even less modify them, see the WWWorst AppStore), all that while 
favorizing a server-side workflow that’s here to simplify “scaling”, that 
is evolution toward an end such as one computer is not enough to run the 
service, and the software is tailormade to be easily movable (“scalable”) 
to many interconnected servers that will do load-balancing, hence the need 
“not to care where runs what”, because computers start to be seen as a 
fungible, anonymous, unidentifiable, cessible, unimportant ressource among 
others…

…while, especially *politically*, this is very untrue! that’s what makes 
most of modern developers&sysadmins (“DevOps”) run toward AWS and, most 
generally, cloud (any VPS in general, just forgetting forever about owning 
your own computers), and any sovereignity illusory.

So no: running *out* of the current javascript web is the best idea, and 
improving or advertising something such as this org-protocol I never heard 
about looks like good.

PS: this still makes me dubious as I always used org-mode without any link 
with any browser, without ever becoming aware of it (which would be an 
important step in advertising it: making some part of the normal workflow 
(such as exporting) present it as an option), plus org-mode has 
unfortunately infamously been notorious to use/promote proprietary 
software or interaction with it.

Re: Elisp LSP Server

[Po Lu]

Alexandre Garreau <galex-713@galex-713.eu> writes:

> Is there an implementation/backend of VS Code implemented in javascript, 
> or transpiled to it?
>
> It would anyway, without having to transpile C to js (which already 
> exists, but not including X ofc (hopefully?)), or elisp to js (which may 
> already exist or be easily doable from what exists), be possible to 
> develop not really an extension but a plugin that would allow open emacs 
> inside frames such as it was possible to watch videos before <video> and 
> firefox to include its own player

BTW, there seems to already be an NaCL port of Emacs.  Could that help?
Does NaCL work in free browsers, such as IceCat, Epiphany or Chromium?

Re: Elisp LSP Server

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It would anyway, without having to transpile C to js (which already 
  > exists, but not including X ofc (hopefully?)), or elisp to js (which may 
  > already exist or be easily doable from what exists), be possible to 
  > develop not really an extension but a plugin that would allow open emacs 
  > inside frames such as it was possible to watch videos before <video> and 
  > firefox to include its own player

I suppose it would work... but what is the benefit?
-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Elisp LSP Server

[Alexandre Garreau]

Le lundi 25 octobre 2021, 04:17:18 CEST Richard Stallman a écrit :
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>   > It would anyway, without having to transpile C to js (which already
>   > exists, but not including X ofc (hopefully?)), or elisp to js (which
>   > may already exist or be easily doable from what exists), be
>   > possible to develop not really an extension but a plugin that would
>   > allow open emacs inside frames such as it was possible to watch
>   > videos before <video> and firefox to include its own player
> 
> I suppose it would work... but what is the benefit?

as you said: the same workflow as what’s currently proposed with VS Code as 
a webapp.  Having the program’s UI *inside* the page instead of another 
window is more ergonomical and less confusing.

Actually I find the very concept of (floating) windowing very anti-
ergonomic, cumbersome and confusing, and I seem not to be the only one: 
not only because of the popularity of tiling WM (among whose who 
understand the thing), but because I saw many eldery people in course of 
learning windowing systems be very stressed and frightened by stuff that 
pops out at random places on the screen and that you have to consciously 
think about where to place and how to dimension on the screen…

More of it: it supports the concept of running software natively and 
locally, instead of on top of javascript/DOM as it is becoming very 
popular (maybe mainly for that reason, as well as easiness to launch from 
a webpage with js), which would go in the interest of Free Software, I 
believe (as supported in the WWWorst AppStore).

Re: Elisp LSP Server

[Mathias Dahl]

[-- Attachment #1: Type: text/plain, Size: 448 bytes --]

>
>
> On 22.10.2021 19:23, Mathias Dahl wrote:
> > About this convenience feature of being able to very easily open VS
> > Code when browsing GitHub, we could develop our own
> > Chrome/Edge/Firefox extension to allow the same workflow if we
> > wanted.
>
> The OP was referring to being able to open "VS Code" (a version of it)
> inside the browser. It's a "cloud IDE" sort of thing.
>

Ah. We could open Emacs locally though, using an extension.

[-- Attachment #2: Type: text/html, Size: 725 bytes --]

Re: Elisp LSP Server

[Dmitry Gutov]

On 22.10.2021 19:55, Mathias Dahl wrote:
> 
>     On 22.10.2021 19:23, Mathias Dahl wrote:
>      > About this convenience feature of being able to very easily open VS
>      > Code when browsing GitHub, we could develop our own
>      > Chrome/Edge/Firefox extension to allow the same workflow if we
>      > wanted.
> 
>     The OP was referring to being able to open "VS Code" (a version of it)
>     inside the browser. It's a "cloud IDE" sort of thing.
> 
> 
> Ah. We could open Emacs locally though, using an extension.

I think the idea is more along the lines of having a read-to-use VM with 
an appropriate version of Emacs installed, with all declared package 
dependencies and probably additional tools as well (e.g. third-party 
community sometimes uses stuff like eldev, or buttercup for testing).

Then you only need to hack up a change, you're able to test it without 
having to install anything, then you do a commit/push/PR.

All this is perhaps less critical for Emacs Lisp than for other 
languages and environments, but even so, it could lower the barrier for 
contribution further.

Re: Elisp LSP Server

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > The OP was referring to being able to open "VS Code" (a version of it)
  > > inside the browser. It's a "cloud IDE" sort of thing.
  > >

  > Ah. We could open Emacs locally though, using an extension.

You can already start Emacs locally.  What's the actual point or goal
here?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Elisp LSP Server

[Alexandre Garreau]

Le lundi 25 octobre 2021, 04:17:51 CEST Richard Stallman a écrit :
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>   > > The OP was referring to being able to open "VS Code" (a version of
>   > > it)
>   > > inside the browser. It's a "cloud IDE" sort of thing.
>   > 
>   > Ah. We could open Emacs locally though, using an extension.
> 
> You can already start Emacs locally.  What's the actual point or goal
> here?

you cannot from webpages, yet these represent most of the usage of their 
computer from modern users.  This is because of their hypertext nature: by 
being heavily interconnected, they lead to some kind of addiction/
accumulation (that’s easily seen when you “get lost on wikipedia” by 
compulsively reading stuff that have something little but interesting in 
common).  The issue is you can follow an hyperlink from emacs (or any 
software, for the matter) to the webbrowser, but more hardly from the 
webbrowser to some specific function of emacs (but a new file that would be 
stored in /tmp, hence limiting the usefulness of emacs in the time, since 
the file won’t be there in the future to edit again), and, worse, 
impossible from any software that’s not a browser nor emacs, to emacs (at 
least through a hyperlink, which in most software always triggers a 
browser).

Re: Elisp LSP Server

[Theodor Thornhill]

On October 25, 2021 9:22:36 AM GMT+02:00, Alexandre Garreau <galex-713@galex-713.eu> wrote:
>Le lundi 25 octobre 2021, 04:17:51 CEST Richard Stallman a écrit :
>> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>> [[[ whether defending the US Constitution against all enemies,     ]]]
>> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>> 
>>   > > The OP was referring to being able to open "VS Code" (a version of
>>   > > it)
>>   > > inside the browser. It's a "cloud IDE" sort of thing.
>>   > 
>>   > Ah. We could open Emacs locally though, using an extension.
>> 
>> You can already start Emacs locally.  What's the actual point or goal
>> here?
>
>you cannot from webpages, yet these represent most of the usage of their 
>computer from modern users.  This is because of their hypertext nature: by 
>being heavily interconnected, they lead to some kind of addiction/
>accumulation (that’s easily seen when you “get lost on wikipedia” by 
>compulsively reading stuff that have something little but interesting in 
>common).  The issue is you can follow an hyperlink from emacs (or any 
>software, for the matter) to the webbrowser, but more hardly from the 
>webbrowser to some specific function of emacs (but a new file that would be 
>stored in /tmp, hence limiting the usefulness of emacs in the time, since 
>the file won’t be there in the future to edit again), and, worse, 
>impossible from any software that’s not a browser nor emacs, to emacs (at 
>least through a hyperlink, which in most software always triggers a 
>browser).
>


https://github.com/emacs-lsp/lsp-gitpod

This looks like what you are thinking of, doesn't it? 

Theodor Thornhill 

Re: Elisp LSP Server

[Alexandre Garreau]

Le lundi 25 octobre 2021, 09:45:21 CEST Theodor Thornhill a écrit :
> On October 25, 2021 9:22:36 AM GMT+02:00, Alexandre Garreau 
<galex-713@galex-713.eu> wrote:
> >Le lundi 25 octobre 2021, 04:17:51 CEST Richard Stallman a écrit :
> >> [[[ To any NSA and FBI agents reading my email: please consider   
> >> ]]]
> >> [[[ whether defending the US Constitution against all enemies,    
> >> ]]]
> >> [[[ foreign or domestic, requires you to follow Snowden's example.
> >> ]]]
> >> 
> >>   > > The OP was referring to being able to open "VS Code" (a version
> >>   > > of
> >>   > > it)
> >>   > > inside the browser. It's a "cloud IDE" sort of thing.
> >>   > 
> >>   > Ah. We could open Emacs locally though, using an extension.
> >> 
> >> You can already start Emacs locally.  What's the actual point or goal
> >> here?
> >
> >you cannot from webpages, yet these represent most of the usage of
> >their computer from modern users.  This is because of their hypertext
> >nature: by being heavily interconnected, they lead to some kind of
> >addiction/ accumulation (that’s easily seen when you “get lost on
> >wikipedia” by compulsively reading stuff that have something little
> >but interesting in common).  The issue is you can follow an hyperlink
> >from emacs (or any software, for the matter) to the webbrowser, but
> >more hardly from the webbrowser to some specific function of emacs
> >(but a new file that would be stored in /tmp, hence limiting the
> >usefulness of emacs in the time, since the file won’t be there in the
> >future to edit again), and, worse, impossible from any software that’s
> >not a browser nor emacs, to emacs (at least through a hyperlink, which
> >in most software always triggers a browser).
> 
> https://github.com/emacs-lsp/lsp-gitpod
> 
> This looks like what you are thinking of, doesn't it?

Absolutely not, because it takes the whole page (an entire tab), while I 
mean a single block *inside* an already webpage.  Furthermore: this seems 
to require javascript (my idea would only be to require a simple bar link, 
or <embed> element, to any emacs-editable file, to trigger the opening of 
an emacs X window *inside* a webpage), and worse: the opening, hence 
execution (though not usage) of VS Code.

Re: Elisp LSP Server

[Theodor Thornhill]

>> https://github.com/emacs-lsp/lsp-gitpod
>> 
>> This looks like what you are thinking of, doesn't it?
>
>Absolutely not, because it takes the whole page (an entire tab), while I 
>mean a single block *inside* an already webpage.  Furthermore: this seems 
>to require javascript (my idea would only be to require a simple bar link, 
>or <embed> element, to any emacs-editable file, to trigger the opening of 
>an emacs X window *inside* a webpage), and worse: the opening, hence 
>execution (though not usage) of VS Code.
>
>

Yeah. I wasn't thinking of the implementation per se, more in the likes of "press a button on $GITFORGE to start hacking in Emacs" with niceties already enabled. I guess that in the name of gathering inspiration this is useful. Politics aside, I can see people wanting to use this. 

Theo

Re: Elisp LSP Server

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > You can already start Emacs locally.  What's the actual point or goal
  > > here?

  > you cannot from webpages,

I can't imagine what it would mean to "start Emacs from a web page".
Can you please explain concretely?  Please keep in mind that some of
us have never seen it.

With a concrete picture of the practice in question, we could start to
think about the practical and _moral_ implications of supporting that
in Emacs.  How would it affect the development and use of the GNU system?
How would it affect our fight against unjust computing?

We could also think about how we would want to implement it, if we
decide to try.

The more closely and seamlessly Emacs becomes integrated with some
other program's user interface, the more it undermines our goal of
making users aware of GNU Emacs, and of the ethical goals that we
develop GNU Emacs (and GNU as a whole) to promote.

That suggests that perhaps the best way to do this job is via emacsclient.

  >   The issue is you can follow an hyperlink from emacs (or any 
  > software, for the matter) to the webbrowser, but more hardly from the 
  > webbrowser to some specific function of emacs

I'm not sure how to understand the idea of invoking a "specific
function of Emacs" from a browser.  Would you please give a couple of
concrete examples?  Which functions of Emacs would you suggest we
support imvoking in this way, and how would they be used in the
scenario?

  >  yet these represent most of the usage of their 
  > computer from modern users.

That is not necessarily a recommendation of it.  Quite the contrary:
there are a lot of bad developments in modern computing.
Most of the computing people do nowadays is for suckers.
Surveillance companies have a lot of influence over what people
do on the internet, and how they do it.

The question of when to go along, and how far, is sometimes
obvious, and sometimes subtle.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Elisp LSP Server

[Alexandre Garreau]

Le samedi 30 octobre 2021, 08:51:37 CET Richard Stallman a écrit :
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>   > > You can already start Emacs locally.  What's the actual point or
>   > > goal
>   > > here?
>   > 
>   > you cannot from webpages,
> 
> I can't imagine what it would mean to "start Emacs from a web page".
> Can you please explain concretely?  Please keep in mind that some of
> us have never seen it.

To me it would mean to have something written in the page’s source that 
would trigger emacs to be launched, and possibly its window to be 
displayed as part of the page (that is: without decoration or ability to 
be moved, and it would scroll with the page’s content and wouldn’t be 
displayed if the browser’s window’s not).

What I would imagine would be for instance <embed src="file.c" /> or 
possibly with attributes specifying that we want to open it with emacs or 
at line n (I’m sure standards exist for those, there is certainly some 
anchor syntax for within github to point at a line, something like 
file.c#l123).

> With a concrete picture of the practice in question, we could start to
> think about the practical and _moral_ implications of supporting that
> in Emacs.  How would it affect the development and use of the GNU
> system? How would it affect our fight against unjust computing?
> 
> We could also think about how we would want to implement it, if we
> decide to try.
> 
> The more closely and seamlessly Emacs becomes integrated with some
> other program's user interface, the more it undermines our goal of
> making users aware of GNU Emacs, and of the ethical goals that we
> develop GNU Emacs (and GNU as a whole) to promote.

Oh that’s a valid concern I didn’t thought about…

> That suggests that perhaps the best way to do this job is via
> emacsclient.

Indeed.

>   >   The issue is you can follow an hyperlink from emacs (or any
>   > 
>   > software, for the matter) to the webbrowser, but more hardly from
>   > the
>   > webbrowser to some specific function of emacs
> 
> I'm not sure how to understand the idea of invoking a "specific
> function of Emacs" from a browser.  Would you please give a couple of
> concrete examples?  Which functions of Emacs would you suggest we
> support imvoking in this way, and how would they be used in the
> scenario?

open at line, open with a certain mode enabled, open several files at once, 
open an svg file either as an image or as source, etc.

the main one being “open at line”

>   >  yet these represent most of the usage of their
>   > 
>   > computer from modern users.
> 
> That is not necessarily a recommendation of it.  Quite the contrary:
> there are a lot of bad developments in modern computing.
> Most of the computing people do nowadays is for suckers.
> Surveillance companies have a lot of influence over what people
> do on the internet, and how they do it.

Yes of course.

> The question of when to go along, and how far, is sometimes
> obvious, and sometimes subtle.

As I said: I think the way I propose would be strictly in benefit of emacs 
(it would drag people from the browser to emacs, which is going to be more 
full-featured and faster anyway). It would also for users because it would 
remove the need for javascript in certain cases that are going to 
proliferate (people programming inside their browser), so they could avoid 
it.

Re: Elisp LSP Server

[Tim Cross]

Alexandre Garreau <galex-713@galex-713.eu> writes:

> Le samedi 30 octobre 2021, 08:51:37 CET Richard Stallman a écrit :
>> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>> [[[ whether defending the US Constitution against all enemies,     ]]]
>> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>> 
>>   > > You can already start Emacs locally.  What's the actual point or
>>   > > goal
>>   > > here?
>>   > 
>>   > you cannot from webpages,
>> 
>> I can't imagine what it would mean to "start Emacs from a web page".
>> Can you please explain concretely?  Please keep in mind that some of
>> us have never seen it.
>
> To me it would mean to have something written in the page’s source that 
> would trigger emacs to be launched, and possibly its window to be 
> displayed as part of the page (that is: without decoration or ability to 
> be moved, and it would scroll with the page’s content and wouldn’t be 
> displayed if the browser’s window’s not).
>
> What I would imagine would be for instance <embed src="file.c" /> or 
> possibly with attributes specifying that we want to open it with emacs or 
> at line n (I’m sure standards exist for those, there is certainly some 
> anchor syntax for within github to point at a line, something like 
> file.c#l123).
>
>> With a concrete picture of the practice in question, we could start to
>> think about the practical and _moral_ implications of supporting that
>> in Emacs.  How would it affect the development and use of the GNU
>> system? How would it affect our fight against unjust computing?
>> 
>> We could also think about how we would want to implement it, if we
>> decide to try.
>> 
>> The more closely and seamlessly Emacs becomes integrated with some
>> other program's user interface, the more it undermines our goal of
>> making users aware of GNU Emacs, and of the ethical goals that we
>> develop GNU Emacs (and GNU as a whole) to promote.
>
> Oh that’s a valid concern I didn’t thought about…
>
>> That suggests that perhaps the best way to do this job is via
>> emacsclient.
>
> Indeed.
>
>>   >   The issue is you can follow an hyperlink from emacs (or any
>>   > 
>>   > software, for the matter) to the webbrowser, but more hardly from
>>   > the
>>   > webbrowser to some specific function of emacs
>> 
>> I'm not sure how to understand the idea of invoking a "specific
>> function of Emacs" from a browser.  Would you please give a couple of
>> concrete examples?  Which functions of Emacs would you suggest we
>> support imvoking in this way, and how would they be used in the
>> scenario?
>
> open at line, open with a certain mode enabled, open several files at once, 
> open an svg file either as an image or as source, etc.
>
> the main one being “open at line”
>
>>   >  yet these represent most of the usage of their
>>   > 
>>   > computer from modern users.
>> 
>> That is not necessarily a recommendation of it.  Quite the contrary:
>> there are a lot of bad developments in modern computing.
>> Most of the computing people do nowadays is for suckers.
>> Surveillance companies have a lot of influence over what people
>> do on the internet, and how they do it.
>
> Yes of course.
>
>> The question of when to go along, and how far, is sometimes
>> obvious, and sometimes subtle.
>
> As I said: I think the way I propose would be strictly in benefit of emacs 
> (it would drag people from the browser to emacs, which is going to be more 
> full-featured and faster anyway). It would also for users because it would 
> remove the need for javascript in certain cases that are going to 
> proliferate (people programming inside their browser), so they could avoid 
> it.

I'm sorry, but this argument makes no sense to me.

This whole argument assumes the user has emacs installed. If they don't
use emacs, it is unlikely they have it installed and if they do use it,
then it isn't going to attract any new users.

I also don't understand the argument of avoiding javascript - the only
way to interact with an API or a LSP server or emacsclient from within a
web page is via some form of script execution and the only scripting
language universally supported inside a browser is javascript (possible
exception is with a browser extensions, but that would need a browser
specific extensions for each browser type).

I also don't see how this relates at all to an elisp language LSP
server. So far in this thread, the only person who seems to have come up
with a valid justification for an LSP server is Eli, with his suggestion
that an elisp LSP server running in a separate Emacs instance could
provide an efficient mechanism for offloading processing to a separate
process, thereby improving emacs performance (a poor mans
multi-tasking if you like).

If you had access to an elisp LSP server from within a web browser, that
would still require lots of javascript to handle the embedded 'window' -
having an elisp LSP server is not going to suddenly enable Emacs to
embed an Emacs frame/window inside a web page. The best you could do is
use the elisp LSP server for parsing, adding face properties, completion
etc. You would still need javascript to manage the display and
navigation within the embedded 'window'. .

Note also that there already exists a browser extension (chrome and
firefox, though I don't know about the firefox extension post the
firefox API changes a few years ago), which allows you to use Emacs to
edit any editable field in a web page. The 'with-editor' extension adds
an 'emacs' button to editable fields in a web page - clicking on that
pulls up a new emacs frame (via emacsclient) which allows you to use
Emacs to edit the data. Once you close the frame, the data is updated in
the browser page field.

I disagree with RMS and others fears regarding the implications of an
elisp LSP server. I feel this is 'tilting at windmills'. Emacs and elisp
are great technologies, but they are of little to no interest to
non-emacs users. Just because it would be technically possible to call
an elisp LSP server from a non-free program does not mean that provision
of such an LSP server would result in either Emacs/elisp being used by
non-free software or in some way encouraging free software users to use
non-free software. The real benefit of elisp is in the whole environment
that Emacs provides. The limited functionality provided via the LSP
protocol cannot reproduce that environment and at best, would only
provide access to functionality which is already available via other LSP
servers which are likely to be more familiar to non-Emacs users than
elisp.

... and of course, all of this is primarily vapourware based on a
fleeting thought with no real substance. I would encourage anyone who
feels it is worth fleshing out further to do so and the rest of us wait
until something of substance exists before debating pros and cons. For
all we know, implementation of an elisp LSP server may result in other
benefits not yet thought of which could be extremely valuable to the
Emacs community. We should encourage such developments rather than
discourage them based on vague and unsubstantiated fears. 

Re: Elisp LSP Server

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > To me it would mean to have something written in the page’s source that 
  > would trigger emacs to be launched, and possibly its window to be 
  > displayed as part of the page (that is: without decoration or ability to 
  > be moved, and it would scroll with the page’s content and wouldn’t be 
  > displayed if the browser’s window’s not).

It sounds like this would turn Emacs into a sort of widget for use by
websites.  That's not what we want Emacs to be.

  > What I would imagine would be for instance <embed src="file.c" /> or 
  > possibly with attributes specifying that we want to open it with emacs or 
  > at line n (I’m sure standards exist for those, there is certainly some 
  > anchor syntax for within github to point at a line, something like 
  > file.c#l123).

How does a server know the names of files on your computer?
Why does it want you to edit some specific file?

  > open at line, open with a certain mode enabled, open several files at once, 
  > open an svg file either as an image or as source, etc.

  > the main one being “open at line”

I can understand what it means to specify to go to a certain position
in a file while visiting it in Emacs, but why would a web site
do such a thing?

The scenarios that I can envision are unethical ones, where your computing
is done by a web site run by someone else, and thus not under your control.
I can't think of an ethical scenario that would use this.

Can you describe one?


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Elisp LSP Server

[Jostein Kjønigsen]

[-- Attachment #1: Type: text/plain, Size: 2637 bytes --]

> How does a server know the names of files on your computer?

Hey Richard. 

An LSP-server is actually software running on your own machine, but in a different process. And it’s very often free software too!

Thus a client-server architecture with corresponding terms. 

For someone not intimately into LSP as a protocol, I can see how that can be confusing. 

--
Vennlig hilsen
Jostein Kjønigsen

jostein@kjonigsen.net 🍵 jostein@gmail.com
https://jostein.kjønigsen.no <https://jostein.xn--kjnigsen-64a.no/>


On Wed, Nov 3, 2021, at 04:18, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>   > To me it would mean to have something written in the page’s source that 
>   > would trigger emacs to be launched, and possibly its window to be 
>   > displayed as part of the page (that is: without decoration or ability to 
>   > be moved, and it would scroll with the page’s content and wouldn’t be 
>   > displayed if the browser’s window’s not).
> 
> It sounds like this would turn Emacs into a sort of widget for use by
> websites.  That's not what we want Emacs to be.
> 
>   > What I would imagine would be for instance <embed src="file.c" /> or 
>   > possibly with attributes specifying that we want to open it with emacs or 
>   > at line n (I’m sure standards exist for those, there is certainly some 
>   > anchor syntax for within github to point at a line, something like 
>   > file.c#l123).
> 
> How does a server know the names of files on your computer?
> Why does it want you to edit some specific file?
> 
>   > open at line, open with a certain mode enabled, open several files at once, 
>   > open an svg file either as an image or as source, etc.
> 
>   > the main one being “open at line”
> 
> I can understand what it means to specify to go to a certain position
> in a file while visiting it in Emacs, but why would a web site
> do such a thing?
> 
> The scenarios that I can envision are unethical ones, where your computing
> is done by a web site run by someone else, and thus not under your control.
> I can't think of an ethical scenario that would use this.
> 
> Can you describe one?
> 
> 
> -- 
> Dr Richard Stallman (https://stallman.org)
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)
> 
> 
> 
> 

[-- Attachment #2: Type: text/html, Size: 4234 bytes --]

Re: Elisp LSP Server

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > An LSP-server is actually software running on your own machine, but in a different process. And it’s very often free software too!

I understand about having Emacs invoke a language server.
(Is an "LSP server" the same thing as a language server?
It isn't obvious.)

But we were talking about having a web browser invoke Emacs.
How does the language server relate to that?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Elisp LSP Server

[Daniel Brooks]

Richard Stallman <rms@gnu.org> writes:

>   > An LSP-server is actually software running on your own machine,
>   > but in a different process. And it’s very often free software too!
>
> I understand about having Emacs invoke a language server.
> (Is an "LSP server" the same thing as a language server?
> It isn't obvious.)

It is. LSP stands for “language–server protocol”. It’s a protocol for an
editor and a language compiler or runtime to exchange information. The
editor sends queries that generally correspond to the user’s actions,
and the server sends back information the editor can use to complete
that action.

For example, if I am editing a C program in Emacs, I can call
`xref-find-definitions' and it will find the identifier at point,
consult my TAGS file, and jump to the definition of that identifier.

With an LSP server, Emacs instead sends the filename and the location of
point to the server, and the server sends back a list of filenames and
locations where the definition(s) are. This frees Emacs from having to
know very much about the specific language, and frees language designers
from having to teach every text editor in the world about their new
language.

> But we were talking about having a web browser invoke Emacs.
> How does the language server relate to that?

I admit I wasn’t following the discussion, but I think that it is
becoming more common for people to run large applications inside the
browser. Specifically, GitHub is doing something with VSCode. By running
VSCode in the browser, GitHub can provide an IDE for any project hosted
there, which is hoped to be a powerful feature for attracting new
developers to submit code to those projects that use it. I haven’t tried
it myself, but I see no technical obstacles to this. I myself helped in
some small ways to get MAME emulated arcade machines and consoles
running in the browser for https://archive.org/.

Anyway, VSCode tries to make it as easy as possible for their users to
get started programming in their favorite languages. The VSCode user can
generally go to a list of plugins inside of VSCode, find one for their
language, check the checkbox next to it, and more or less immediately
start programming in that language, complete with LSP integration. (This
is not much different from running `package-install', for example, except
that it can and does install binaries on your system).

Within Emacs, `lsp-mode' offers integration with LSP servers, and it has
a function `lsp-install-server' which can automatically install some of
them.

If Emacs were to act as an LSP server, then in principle VSCode could
offer an Elisp extension which would download Emacs for the user. It
wouldn’t be for use as an editor, but just for use as the LSP server for
the elisp code that the user is editing in VSCode.

Incidentally, Microsoft owns GitHub and VSCode, and they started the LSP
protocol specification specifically for use with VSCode back in 2016
(although the direct inspiration, I suspect, comes from a post by Steve
Yegge at Google a few years prior).

From the point of view of most developers of text editors, and most
language designers, the LSP protocol makes sense. It reduces the amount
of work that they all have to do to support each other. On the other
hand, the specification is quite closely tied to the needs of VSCode. It
is called an open standard, but having taken a look at it I can see that
it is just whatever VSCode happened to support at the time it was
published, plus some extensions since then. Perhaps it can grow from
there.

I don’t see any particular need for Emacs to be a language server for
Elisp, but there is nevertheless an argument for it. For example, it is
not much different than allowing users to export word–processor
documents in formats that can be read by your competitors. As far as I
know, Microsoft Office does not give the user the option of exporting
their document as an OpenOffice file, but OpenOffice does allow
exporting documents as Microsoft Office files. Microsoft Office tries to
keep people from straying from Microsoft products, while OpenOffice does
not.

Even so, I think it would be significant work for not much benefit.

I hope something in that ramble was helpful!

db48x

Re: Elisp LSP Server

[Jostein Kjønigsen]

[-- Attachment #1: Type: text/plain, Size: 2830 bytes --]


On 05.11.2021 06:55, Daniel Brooks wrote:
>
> I admit I wasn’t following the discussion, but I think that it is
> becoming more common for people to run large applications inside the
> browser.

I admit not following 100% either, but I think the original argument was 
that users sometimes likes or have the need to edit Emacs LISP-files  
outside Emacs, like in online code-forges like GitHub, GitLab and other 
places.

For the time being these forges, or no editor outside Emacs really, 
offers adequate support for Emacs LISP.

I guess the argument was that if there was an Emasc LISP LSP-server 
available for them to use, that might change, and that it could in 
theory advance the case of Emacs through making Emacs LISP more 
accessible. If that's how it will turn out in the real world is anyone's 
guess :)

>   
> Anyway, VSCode tries to make it as easy as possible for their users to
> get started programming in their favorite languages. The VSCode user can
> generally go to a list of plugins inside of VSCode, find one for their
> language, check the checkbox next to it, and more or less immediately
> start programming in that language, complete with LSP integration.

VSCode has seen huge adoptation, also among free software proponents, 
now being the #1 most popular code-editor in the world (from not even 
existing 10 years ago).

I believe this aspect here is a very important reason for VSCode having 
reached the number of users it have.

If we were to apply similar attention to how new Emacs users get started 
with Emacs, it would most likely affect the amount of new Emacs-users 
staying around as long-term Emacs users.

A somewhat usable out-of-the-box experience is now considered a pretty 
fundamental requirement for most software, and not taking that into 
consideration is by most people considered not viable in the long term.

> As far as I
> know, Microsoft Office does not give the user the option of exporting
> their document as an OpenOffice file, but OpenOffice does allow
> exporting documents as Microsoft Office files. Microsoft Office tries to
> keep people from straying from Microsoft products, while OpenOffice does
> not.

Clearly somewhat off-topic, but Microsoft Office actually offers support 
for the OpenOffice/LibreOffice OpenDocument file-formats. Both for 
opening and saving.

When you start Microsoft Office for the first time you are also asked 
which one of these file-format families you would want to be your default.

> I hope something in that ramble was helpful!
Oh certainly. You expressed yourself much more in depth and much clearer 
than I would, had I answered the same email.

-- 
Vennlig hilsen
*Jostein Kjønigsen*

jostein@kjonigsen.net 🍵 jostein@gmail.com 🍵 jostein@hufleslufs.no
https://jostein.kjønigsen.no <https://jostein.kjønigsen.no>

[-- Attachment #2: Type: text/html, Size: 4173 bytes --]

Re: Elisp LSP Server

[Alexandre Garreau]

Le mercredi 3 novembre 2021, 04:18:07 CET Richard Stallman a écrit :
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>   > To me it would mean to have something written in the page’s source
>   > that
>   > would trigger emacs to be launched, and possibly its window to be
>   > displayed as part of the page (that is: without decoration or
>   > ability to be moved, and it would scroll with the page’s content
>   > and wouldn’t be displayed if the browser’s window’s not).
> 
> It sounds like this would turn Emacs into a sort of widget for use by
> websites.  That's not what we want Emacs to be.

Why so? is necessarily a widget a bad or diminutive thing? emacs is so 
important, that if emacs was a widget, then it’s not emacs that would be 
diminuished, but the notion of widget that would be enormously widened and 
improved.

All I say is I think emacs is often the best piece of software to look at 
source, and to edit anything textual in general.  Having emacs into a 
fully-functional graphical browser view is to me the easiest and most 
useful step before having a fully-functional graphical browser inside 
emacs.

Ofc emacs is deemed to dominate all userland anyway, once it’s mastered 
and sufficiently well integrated into workflow.  It’s the modern unix and 
userland implementation of what a LISP machine seemed to be, partially.

>   > What I would imagine would be for instance <embed src="file.c" /> or
>   > possibly with attributes specifying that we want to open it with
>   > emacs or at line n (I’m sure standards exist for those, there is
>   > certainly some anchor syntax for within github to point at a line,
>   > something like file.c#l123).
> 
> How does a server know the names of files on your computer?

Here I was not talking about that: a such «src="file.c"» would mean the 
“file.c” file *relative to the url of the page* containing that.

But an url could as well point to a local file, I’m pretty sure that 1) 
it’s accessible with javascript via the html form element to select files, 
2) you can guess/reconstruct it according some previous value given by 
user, so you could have «src="file:///home/rms/Downloads/foo-repo/file.c”».

> Why does it want you to edit some specific file?

Possibly because you asked for it, because you were viewing source through 
a web browser, for instance because you didn’t cared to download the whole 
source tree locally.

>   > open at line, open with a certain mode enabled, open several files
>   > at once, open an svg file either as an image or as source, etc.
>   > 
>   > the main one being “open at line”
> 
> I can understand what it means to specify to go to a certain position
> in a file while visiting it in Emacs, but why would a web site
> do such a thing?

because currently github does it both in html pre (read-only), in html 
textarea (strictly less practical than emacs (except if you’re using w3m 
or some other browser that’s already launching emacs (actually $EDITOR) 
for editing any textarea)), or in VS Code (which is proprietary).

Actually, ideally I think viewing and editing of certain files in a web 
page should be agnostic of the application, so you could decide that html 
<video> should be viewed using vlc or mpv (preferably in-page, because 
it’s more ergonomical, it classify better your activities, and doesn’t 
break the flow with disruptives floating windows), or that file.c, or in 
general all textareas, should be viewed/edited using emacs.  It would be 
nice if, until emacs is a browser on par with firefox, firefox could trigger 
emacs (at least emacsclient) to edit stuff and view other such as source.

> The scenarios that I can envision are unethical ones, where your
> computing is done by a web site run by someone else, and thus not under
> your control. I can't think of an ethical scenario that would use this.
> 
> Can you describe one?

Source fragment samples on a personal blog, and the feature of allowing 
the user to benefit from personalized syntactic coloration, indentation, 
editing, and running what’s in the source snippet.

Many blogs do that nowadays, but all of them do that using proprietary 
javascript applications I think.  Or maybe some of them are free.  Given 
javascript “ecosystem” (I know you dislike this word and me too and I’m 
using it ironically) and culture, it would even be hard to know.  And even 
if it had a free-software license, I personally consider that javascript 
sent once-per-page-load is unarchived, unauditable, hence have many common 
issues with proprietary software.  I’d prefer if that was done with emacs 
and slime instead (and that’s already totally possible, what is lacking is 
an url convention to specify lines, <embed>-ing of arbitrary content 
(given the user has any software that can display/edit it), and 
mostimportantly and lacking and difficult: to display emacs inside a 
webppage).

Re: Elisp LSP Server

[Jostein Kjønigsen]

[-- Attachment #1: Type: text/plain, Size: 1661 bytes --]


On 05.10.2021 13:27, Po Lu wrote:
> Philip Kaludercic<philipk@posteo.net>  writes:
>
>> Po Lu<luangruo@yahoo.com>  writes:
>>
>>> Ag Ibragimov<agzam.ibragimov@gmail.com>  writes:
>>>
>>>> Another reason: VSCode. The damn thing is spreading like a
>>>> disease. And now it's even possible to browse code on GitHub, in a
>>>> browser, by simply pressing <.> (the dot).  I would love to be able to
>>>> comfortably browse through elisp code without having to clone it
>>>> locally, but none of the existing VSCode Lisp plugins are any good.
>>>> For example, there's no equivalent of imenu for Lisp files. You can't
>>>> jump to a given function.
>>> Isn't the code behind that particular feature proprietary?
>> In what sense? GitHub is inherently proprietary, so yes, but what
>> about this specific feature is different.
> I think it is dangerous for Emacs to "integrate" with such proprietary
> software.  It would make people gravitate towards that software.
>
Just to be clear, and so that we're all on the same page:

* Whatever software Github runs is proprietary.
* The LSP protocol-specification it is based on is fully open, and 
implemented by lots of (FLOSS) programming-languages.

What Ag Ibragimov is asking for here is not integrating with a 
proprietary product, but implementing the open protocol, to provider 
better support for Emacs Lisp as a language.

If that is perceived as a danger, then I honestly think one might 
consider re-adjusting once own perception :)

-- 
Vennlig hilsen
*Jostein Kjønigsen*

jostein@kjonigsen.net 🍵 jostein@gmail.com 🍵 jostein@hufleslufs.no
https://jostein.kjønigsen.no <https://jostein.kjønigsen.no>

[-- Attachment #2: Type: text/html, Size: 3406 bytes --]

Re: Elisp LSP Server

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Isn't the code behind that particular feature proprietary?

  > In what sense?

The definition of "free software" and its antonym, "proprietary software",
are in https://gnu.org/philosophy/free-sw.html.  When we speak of
whether certain code is proprietary, that's the sense we mean.

What is not necessarily so clear is this other question: which code
are we asking about?  Which code should we ask about?

We should ask about all the code that you need to install onto a
free GNU/Linux system installation, so as to make the feature work.
Everything needed to make the feature work is crucial, because all of
that needs to be free in order to have this in Emacs.

                   GitHub is inherently proprietary, so yes, but what
  > about this specific feature is different.

GitHub has grave moral faults and has done great harm to the free
software community, but saying it is "proprietary" is not well
defined.  That is because GitHub is not a program -- it is a service.
The distinction of free vs proprietary is defined for programs, but it
is not defined for services.

For more explanation, see
https://www.gnu.org/philosophy/network-services-arent-free-or-nonfree.html.

Because of GitHub's moral faults, GNU software should never presume
use of GitHub.  It is ok to support use of GitHub _or_ some
freedom-respecting alternatives.  However, a feature that works _only_
with GitHib (or other bad alternatives) must not be supported by
Emacs.

However, I think we're talking about a different kind of feature here.

  >   I would love to be able to
  > >> comfortably browse through elisp code without having to clone it
  > >> locally, but none of the existing VSCode Lisp plugins are any good.

You perceive a similarity between this and the GitHub example, and
maybe you're right, but I don't follow.  Would you please describe the
similarity you perceive?  Does this propose integrating Emacs somehow
with some nonfree software?  If so, please state the details: what
nonfree software is it, what does it do, and how would Emacs interact
with it?

We need to analyze this clearly to determine what our moral goal
requires.



-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Elisp LSP Server

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

>                    GitHub is inherently proprietary, so yes, but what
>   > about this specific feature is different.
>
> GitHub has grave moral faults and has done great harm to the free
> software community, but saying it is "proprietary" is not well
> defined.  That is because GitHub is not a program -- it is a service.
> The distinction of free vs proprietary is defined for programs, but it
> is not defined for services.
>
> [...]
> 
> However, I think we're talking about a different kind of feature here.

Yes, that was phrased inadequately, but I do think that the term
"proprietary" is right here, because this is not about
GitHub-the-service but the editor that is actually running in the
browser. Apparently, you can even download extensions, which can also be
non-free.

The documentation[0] is supposed to explain the licensing situation, but
I cannot find a clear statement.  And since VSCode is a propitiatory
fork of "Code" (the editor maintained by Microsoft, published on GitHub
under the MIT license), I assume that the in-browser-VSCode
("Codespaces") is propitiatory too.

[0] https://docs.github.com/en/github/site-policy/github-terms-for-additional-products-and-features#codespaces

-- 
	Philip Kaludercic

Re: Elisp LSP Server

[Joost Kremers]

On Tue, Oct 05 2021, Ag Ibragimov wrote:
> You know, so Emacs Lisp can be comfortably edited outside of Emacs. "Why would
> anyone want that?", you ask?
>
> Well, for example, when something breaks in your config, and you can't even
> start Emacs, and you have to use something else. It's rare, but it happens.

That's the sort of situation I solve with `emacs -Q`. Sure, it's annoying that
my custom key bindings aren't available, but it's rare enough not to be a real
issue. (I actually once made a minimal init.el file with my most used custom key
bindings.)

> I don't care much for VSCode, but LSP provides good consistency. I would love to
> be able to use elisp packages like lsp-ui while writing elisp code [in Emacs].

So, if you have an itch, scratch it! ;-)

Personally, I'm happy with the facilities Emacs already provides (with el-spice,
eldoc and eldoc-box) that I don't really feel the need to use LSP. I do use
lsp-(ui)-mode for Python, and yes, I could see myself using it for Elisp as well
if there were an LSP server for it, but I can't say I really miss it.

-- 
Joost Kremers
Life has its moments

Re: Elisp LSP Server

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I don't care much for VSCode, but LSP provides good consistency. I would love to
  > > be able to use elisp packages like lsp-ui while writing elisp code [in Emacs].

Can this function effectively in a system that contains no nonfree software?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Elisp LSP Server

[Ag Ibragimov]

Richard Stallman <rms@gnu.org> writes:

>   > > I don't care much for VSCode, but LSP provides good consistency. I would love to
>   > > be able to use elisp packages like lsp-ui while writing elisp code [in Emacs].
>
> Can this function effectively in a system that contains no nonfree software?

lsp-ui is an Emacs package. It already works with many (maybe most?) languages that work with LSP.

Here's a list of languages that lsp-mode (another Emacs package) currently supports:

ActionScript, Ada, Angular, Bash, Beancount, C++ (ccls), C++ (clangd),
C# (omnisharp-roslyn), C# (csharp-ls), Clojure, CMake, Crystal,
CSS/LessCSS/SASS/SCSS, D, Dart, Dhall, Dockerfile, Elixir, Elm, Erlang,
Eslint, F#, Fortran, GDScript, Go (gopls), Grammarly, GraphQL, Groovy,
Hack, HTML, Haskell, Java, Javascript/Typescript (deno),
JavaScript/TypeScript (sourcegraph), JavaScript/TypeScript (theia-ide),
JavaScript Flow, Json, Julia, Kotlin, LanguageTool (LTEX), Lua
(EmmyLua), Lua (Lua Language Server), Lua (Lua-Lsp), Markdown, MSSQL,
Nim, Nix, OCaml (ocaml-lsp), Pascal/Object Pascal, Perl, PHP
(intelephense), PHP (Serenata), PHP (felixbecker), PHP (phpactor),
Powershell, Prolog, PureScript, Python (Jedi Language Server), Python
(Palantir), Python (Pyright), Python (Microsoft), R, Racket
(jeapostrophe), Racket (Theia), Ruby, Rust (rust-analyzer), Rust (rls),
Scala, SQL (sqls), Svelte, Swift, Terraform, TeX, LaTeX, etc (digestif),
TeX, LaTeX, etc (texlab), TeX, LaTeX, etc (texlab, external), V, Vala,
Verilog/SystemVerilog (hdl-checker), Verilog/SystemVerilog
(svlangserver), VHDL, Vimscript, Vue, XML, YAML, Zig

And as someone who loves Emacs Lisp, I would like to have it added to
this list. Seriously, Vimscript and Grammarly (which is not even a
programming language) have their own LSP servers. Why can't Elisp have
one?
 

Re: Elisp LSP Server

[dick]

AI> ActionScript, Ada, Angular, Bash, Beancount, C++ (ccls), C++ (clangd), C#...

How spamming this laundry list furthers your cause is anyone's guess.

AI> And as someone who loves Emacs Lisp

Dp you?  It's clear from your correspondence thus far that you don't write
much of it.  If you did, you would 1. be able to do your own dirty work, and
2. understand that writing elisp outside emacs is a non-starter.

Re: Elisp LSP Server

[Stefan Kangas]

Ag Ibragimov <agzam.ibragimov@gmail.com> writes:

> Richard Stallman <rms@gnu.org> writes:
>
>>   > > I don't care much for VSCode, but LSP provides good consistency. I would love to
>>   > > be able to use elisp packages like lsp-ui while writing elisp code [in Emacs].
>>
>> Can this function effectively in a system that contains no nonfree software?
>
> lsp-ui is an Emacs package. It already works with many (maybe most?) languages that work with LSP.
>
> Here's a list of languages that lsp-mode (another Emacs package) currently supports:

The by far most interesting Emacs package is eglot, as that is the only
contender that bothered with the copyright assignment.  That means that
this is the only package that we can reasonably imagine will be shipped
with Emacs itself at some point.[1]  I think this would be an important
advance.

Incidentally, discussing a) how we could make that happen, and b) how to
improve eglot, seems much more important to me than an LSP server for
Emacs Lisp.

(I hope that lsp-mode will decide to sort out the copyright assignment
issue, preferably sooner rather than later, so that we could consider
including them in Emacs on technical grounds.  For now, I think that any
efforts to improve them are better spent on eglot.)

Footnotes:
[1]  This is true as long as the Emacs maintainers insist on keeping the
     copyright assignment requirement.  I unfortunately don't see that
     changing.

Re: Elisp LSP Server

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Footnotes:
  > [1]  This is true as long as the Emacs maintainers insist on keeping the
  >      copyright assignment requirement.

This is a GNU Project legal policy.
See gnu.org/gnu/gnu-structure.html.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Elisp LSP Server

[Stefan Monnier]

> And as someone who loves Emacs Lisp, I would like to have it added to
> this list. Seriously, Vimscript and Grammarly (which is not even a
> programming language) have their own LSP servers. Why can't Elisp have
> one?

There is no reason not to have one, *except* that nobody has written it.
So if you want to see it happen, I suggest you start hacking.

Personally, I think it would be fine to have such a thing, but it's not
something I'm going to put on my TODO list.


        Stefan

Re: Elisp LSP Server

[Po Lu]

Ag Ibragimov <agzam.ibragimov@gmail.com> writes:

> lsp-ui is an Emacs package. It already works with many (maybe most?)
> languages that work with LSP.

Isn't lsp-ui part of lsp-mode?

> Here's a list of languages that lsp-mode (another Emacs package) currently supports:

Aside from what Stefan mentioned, lsp-mode has another problem: it
encourages users to, and even automatically downloads proprietary
software from the internet.

Some time ago, I was told to install it to work on programs in Dart (a
programming language).  It tried to download a compressed file
containing ambiguously licensed JavaScript code.

Re: Elisp LSP Server

[Ag Ibragimov]

Po Lu <luangruo@yahoo.com> writes:

> Aside from what Stefan mentioned, lsp-mode has another problem: it
> encourages users to, and even automatically downloads proprietary
> software from the internet.

Can I beg you to disregard for a minute the aspects of non-free software
issues that may or may not arise due to someone trying to figure this
out, and focus on the technical side of this?

It's becoming a Russian idiom about "skinning a bear that never been
hunted". I think it was a simple question of: "Has anyone attempted
doing this?" and I expected some conversation about technical
challenges, not a lecture about the dangers of non-free software. It was
my mistake to even mention godforsaken VSCode - I don't even use it, I
don't know why I brought it up. I apologize; Can we please start it over
and forget about GitHub, VSCode, etc.?

From what I gather, the answer to my original question is: "no, there
wasn't (at least known) attempt to create an LSP server for Elisp"

So let me then ask follow-up questions:

- Has anyone ever tried figuring out something similar? Surely, in 45
  years of Emacs, someone must have done something like that?

- If someone attempts to create a cli tool that potentially could
introspect, lint, format, and maybe even refactor elisp outside of
Emacs, where would they need to be looking?

- Is it possible to build something like this by examining relevant
pieces of C in Emacs codebase?

- Maybe there are parts in REmacs project that would be easier to go
  through?

Thank you!

Re: Elisp LSP Server

[Philip Kaludercic]

Ag Ibragimov <agzam.ibragimov@gmail.com> writes:

> It was my mistake to even mention godforsaken VSCode - I don't even
> use it, I don't know why I brought it up. I apologize; Can we please
> start it over and forget about GitHub, VSCode, etc.?

I do think there is some association, because it seems there is little
use for a Elisp language server outside of these use-cases. When someone
wants to debug a broken configuration, they'd usually emacs -Q, not open
VSCode.

> - Has anyone ever tried figuring out something similar? Surely, in 45
>   years of Emacs, someone must have done something like that?

LSP has not existed for that long, and I am not sure if there were any
comparable projects of the same ambition before. At most "language
servers" have existed for specific languages, such as Common Lisp.

> - If someone attempts to create a cli tool that potentially could
> introspect, lint, format, and maybe even refactor elisp outside of
> Emacs, where would they need to be looking?

If I wanted to do this, I'd just write Elisp scripts that would be
invoked via shell scripts or something of that sort using Emacs' batch
mode.

An alternative would be to implement everything yourself from group up,
and follow up on the development whenever something changes.

Another one would be to extend something like tree sitter by Elisp
support.

> - Is it possible to build something like this by examining relevant
> pieces of C in Emacs codebase?

Not everything relating to Elisp development in implemented in C. You'd
also have to read the Elisp parts.

> - Maybe there are parts in REmacs project that would be easier to go
>   through?

I don't know about that, but AFAIK the project has stalled.

-- 
	Philip Kaludercic

Re: Elisp LSP Server

[tomas]

[-- Attachment #1: Type: text/plain, Size: 672 bytes --]

On Tue, Oct 12, 2021 at 10:21:57AM +0000, Philip Kaludercic wrote:
> Ag Ibragimov <agzam.ibragimov@gmail.com> writes:
> 
> > It was my mistake to even mention godforsaken VSCode - I don't even
> > use it, I don't know why I brought it up. I apologize; Can we please
> > start it over and forget about GitHub, VSCode, etc.?
> 
> I do think there is some association, because it seems there is little
> use for a Elisp language server outside of these use-cases. When someone
> wants to debug a broken configuration, they'd usually emacs -Q, not open
> VSCode.

FWIW there are LSP bindings for vim. Exotic, I know, but hey, it's free
software.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Elisp LSP Server

[Philip Kaludercic]

<tomas@tuxteam.de> writes:

> On Tue, Oct 12, 2021 at 10:21:57AM +0000, Philip Kaludercic wrote:
>> Ag Ibragimov <agzam.ibragimov@gmail.com> writes:
>> 
>> > It was my mistake to even mention godforsaken VSCode - I don't even
>> > use it, I don't know why I brought it up. I apologize; Can we please
>> > start it over and forget about GitHub, VSCode, etc.?
>> 
>> I do think there is some association, because it seems there is little
>> use for a Elisp language server outside of these use-cases. When someone
>> wants to debug a broken configuration, they'd usually emacs -Q, not open
>> VSCode.
>
> FWIW there are LSP bindings for vim. Exotic, I know, but hey, it's free
> software.

From my understanding of vim, this seems more practical, since Vimscript
isn't evaluated at runtime (like in a *scratch* buffer or M-:), or am I
mistaken?

> Cheers
>  - t
>

P.S. could you Cc all the participants in the thread. I wouldn't have
noticed your repose if I hadn't checked the gmane.

-- 
	Philip Kaludercic

Re: Elisp LSP Server

[tomas]

[-- Attachment #1: Type: text/plain, Size: 722 bytes --]

On Tue, Oct 12, 2021 at 12:51:33PM +0000, Philip Kaludercic wrote:
> <tomas@tuxteam.de> writes:

[...]

> > FWIW there are LSP bindings for vim [...]

> >From my understanding of vim, this seems more practical, since Vimscript
> isn't evaluated at runtime (like in a *scratch* buffer or M-:), or am I
> mistaken?

I'm not that much into vim. Happy user, yes, but when it comes to
hacking at it (even config), my brain is already full with Emacs :)

> P.S. could you Cc all the participants in the thread. I wouldn't have
> noticed your repose if I hadn't checked the gmane.

Hm. I thought I do "global reply" on this list already, since it seems
to be local custom. Probably some "list reply" slipped, sorry.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Elisp LSP Server

[Stefan Monnier]

> - Has anyone ever tried figuring out something similar?  Surely, in 45
>   years of Emacs, someone must have done something like that?

The idea of an ELisp LSP server has been mentioned back when ELisp LSP
clients (lsp-mode and eglot) got off the ground.
I don't know of anyone having attempted to implement it, tho.

> - If someone attempts to create a cli tool that potentially could
> introspect, lint, format, and maybe even refactor elisp outside of
> Emacs, where would they need to be looking?

I don't understand what kind of answer you're looking for.
AFAIK most of the pieces are readily available in Emacs itself.
So I'd expect it's largely an effort of integrating those pieces (but
there's probably a fair bit of code missing to implement the server
part of the LSP protocol).

> - Is it possible to build something like this by examining relevant
> pieces of C in Emacs codebase?

I'd expect an ELisp LSP server would be written 100% in ELisp.


        Stefan

Re: Elisp LSP Server

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Aside from what Stefan mentioned, lsp-mode has another problem: it
  > encourages users to, and even automatically downloads proprietary
  > software from the internet.

This suggests that lsp-mode is poison.  Recommending it in Emacs might
put us in a morally contradictory position.

Would you please explain more about this?  Perhaps give some names to
the programs in the scenario, and say how they relate to each other?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Elisp LSP Server

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

>   > Aside from what Stefan mentioned, lsp-mode has another problem: it
>   > encourages users to, and even automatically downloads proprietary
>   > software from the internet.
>
> This suggests that lsp-mode is poison.  Recommending it in Emacs might
> put us in a morally contradictory position.
>
> Would you please explain more about this?  Perhaps give some names to
> the programs in the scenario, and say how they relate to each other?

lsp-dart-dap-setup tries to download a VS Code extension from the
following URL:

"https://github.com/Dart-Code/Dart-Code/releases/download/v3.25.1/dart-code-3.25.1.vsix"

But the situation seems to have improved since I last tried it, as the
zip archive seems to contain a license and non-minified code now.

Re: Elisp LSP Server

[Dmitry Gutov]

On 13.10.2021 03:00, Po Lu wrote:
> lsp-dart-dap-setup tries to download a VS Code extension from the
> following URL:
> 
> "https://github.com/Dart-Code/Dart-Code/releases/download/v3.25.1/dart-code-3.25.1.vsix"
> 
> But the situation seems to have improved since I last tried it, as the
> zip archive seems to contain a license and non-minified code now.

Did you intentionally present the situation as worse than it was?

Whether the license file is inside the archive or not, the program 
inside can be (and was) free. Same goes for minified/non-minified (one 
*could* hide proprietary code inside such bundle, but checking whether 
it corresponds to the officially published code is not too difficult).

This file is 5 years old: 
https://github.com/Dart-Code/Dart-Code/blob/master/LICENSE

And it's been there since the project's start.

Re: Elisp LSP Server

[Tim Cross]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Aside from what Stefan mentioned, lsp-mode has another problem: it
>   > encourages users to, and even automatically downloads proprietary
>   > software from the internet.
>
> This suggests that lsp-mode is poison.  Recommending it in Emacs might
> put us in a morally contradictory position.
>
> Would you please explain more about this?  Perhaps give some names to
> the programs in the scenario, and say how they relate to each other?

I think the LSP situation has been both misrepresented and
misunderstood.

There is nothing inherently evil about LSP. It is just a protocol. There
is nothing in LSP which requires proprietary software or use of non-free
services. There can be instances of LSP implementations which might do
this, but that is about the implementation and not about support of the
protocol. Ironically, the first place I ever heard about/discussed a LSP
like solution was on this list many years ago during discussions about
how to provide better and more consistent support for things like
completion, syntax highlighting, linting etc. At that time, it was
acknowledged that one reason many editors built for a specific language
were able to provide more superior support in these areas was because they
had a built-in parser for the language being used and that for Emacs to
provide something similar would require it had the ability to parse each
language being supported. This triggered efforts like the one to allow
Emacs to leverage of Eclipse to provide enhanced language support for
Java and to some extents, efforts like CEDET and Semantic.

The basic idea behind LSP is an extension of this idea. In an ideal
world, every programming language would also include an LSP language
server. It would be maintained in parallel with the development of the
language. Any editor which wanted to support that language would
implement the client side of the LSP protocol and install the language
provided LSP language server, avoiding the need for each editor to
implement it's own version of a language parser in order to provide more
'intelligent' support for things like completion, API documentation,
linting, etc.  

The fact MS has used LSP to provide enhanced features on github has
nothing to do with LSP directly - Github integration is not a required
part of the protocol and is not mentioned in the LSP sepcs. You are not
required to use github because you use LSP. Likewise, your not required
to use VSCode extensions, although some languages might distribute their
LSP language server components via that service. Likewise, language
servers for LSP may or may not be free and may or may not encourage use
of non-free software - it needs to be evaluated based on each LSP
language server.

Even if a language server is distributed via github and to use it you
have to download the server from github, this is not sufficient justification
not to support LSP. We already have a number of packages in ELPA and
nongnu ELPA where the main repository is on github. 

I am ambivalent about an LSP language server for ELISP. This is mainly
because all the bits an LSP language server provides are already part of
Emacs and partly because I don't see any real benefit or use case for
ELISP outside of Emacs (I know there are some out there who would like
to use Elisp for everything, but I feel that is an example of a 'hammer
looking for a nail' type syndrome - just because Elisp can be used as a
general purpose programming language doesn't mean it should be).

I suspect that if you configured Emacs to use the same key bindings for
Elisp support features, so that when developing Elisp code it 'felt' the
same as when developing code for an LSP supported language, you would
satisfy much of the requirements raised by the OP.

Having said that, if someone wants to have a go at developing an LSP
language server for Elisp, I would encourage them to do so. While I
suspect they will soon find the amount of work required and the benefits
it provides difficult to justify/maintain, it does no harm and it may
result in other benefits and is a project which would likely provide a
good platform for expanding/enhancing an individual's elisp knowledge at
a deeper level.

In summary, for the OP,
- No, I don't think anyone has tried developing an LSP server for Elisp
- Sure, if you want to do it, go for it
- No idea what language to use or how to implement the server. It has
been suggested elisp would be the obvious choice, but I don't see how
you can implement elisp outside Emacs without pretty much implementing
most of emacs. Other language servers have used Rust or Go to implement
the language parser. I guess it will be whatever language you feel is
best to implement an elisp parser outside of elisp. 

Re: Elisp LSP Server

[Joost Kremers]

On Wed, Oct 13 2021, Tim Cross wrote:
> just because Elisp can be used as a
> general purpose programming language doesn't mean it should be).

Why not? In fact, it arguably already is. You can't really say that something
like Gnus is just an Emacs extension. I think it's more usefully thought of as a
program that just happens to run on the Elisp VM that is Emacs. :-)

> - No idea what language to use or how to implement the server. It has
> been suggested elisp would be the obvious choice, but I don't see how
> you can implement elisp outside Emacs without pretty much implementing
> most of emacs.

Why not just use Emacs? Run an Emacs daemon process (properly isolated, so it
doesn't interfere with the user's Emacs process), feed it the Elisp code and
have it return the analytics that LSP requires. That doesn't sound too difficult.


-- 
Joost Kremers
Life has its moments

Re: Elisp LSP Server

[Dmitry Gutov]

On 13.10.2021 20:15, Joost Kremers wrote:
>> - No idea what language to use or how to implement the server. It has
>> been suggested elisp would be the obvious choice, but I don't see how
>> you can implement elisp outside Emacs without pretty much implementing
>> most of emacs.
> Why not just use Emacs? Run an Emacs daemon process (properly isolated, so it
> doesn't interfere with the user's Emacs process), feed it the Elisp code and
> have it return the analytics that LSP requires. That doesn't sound too difficult.

That seems like the only practical way to implement such a server, indeed.

Re: Elisp LSP Server

[Tim Cross]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 13.10.2021 20:15, Joost Kremers wrote:
>>> - No idea what language to use or how to implement the server. It has
>>> been suggested elisp would be the obvious choice, but I don't see how
>>> you can implement elisp outside Emacs without pretty much implementing
>>> most of emacs.
>> Why not just use Emacs? Run an Emacs daemon process (properly isolated, so it
>> doesn't interfere with the user's Emacs process), feed it the Elisp code and
>> have it return the analytics that LSP requires. That doesn't sound too difficult.
>
> That seems like the only practical way to implement such a server, indeed.

The main objective put forward by the OP was to make it so that working
with Elisp was the same as working with other languages supported by LSP
- at least that was the core message I read from the OP. The ability of
non-Emacs editors to also have this access was secondary.

IFF this is the case, having to start a whole second Emacs instance as a
daemon, just to edit elisp with emacs seems like a significant resource
sink with no real benefit other than adding additional failure points.

One of the points the OP mentioned was inconsistency in programmer
experience - having one interface when editing other LSP supported
languages and another when editing elisp. My point is that you can
eliminate most of that difference with a little work on key bindings and
possibly some interface changes. It doesn't have to be an LSP language
server provided it feels like one to the user.

The use for an elisp language server for non-Emacs editors seems like a
very very small use case to me. Even the proposed use case of being able
to edit a broken init.el file with a different editor seems unlikely. In
fact, I doubt very much elisp is of any interest to anyone who isn't an
Emacs user and emacs -Q is probably still more convenient than using a
completely different editor to edit your init.el file. 

I doubt very mush that anyone will ever bother to implement an LSP
language server for elisp. There simply isn't sufficient benefit given
the work required. Even less likely for non-Emacs users if to have an
elisp LSP language server you need to install Emacs to provide the
daemon, which will be slow to start and resource hungry compared to
other LSP language servers. I think it also largely misses the point -
the idea of LSP is to provide a small, fast and easy to install basic
service which can assist an editor to provide intelligent completion,
linting, syntax highlighting etc for a language. All of the other LSP
language servers I've used are small, fast to start and easy to install.
Emacs as a daemon is none of these.

At any rate, I only entered this thread to address the misinformation
about LSP being evil and encouraging the use of non-free software. LSP
is simply a protocol - how it is implemented and how it is used is down
to developers and users - arguing that providing an LSP server would
encourage the use of non-free software is extremely unlikely. Elisp is
of no interest to non-Emacs users and while it might seem that elisp is
the greatest language ever to those who have drunk the Emacs kool-aid,
to those outside Emacs, it is just some esoteric editor configuration
language of little interest. 

Re: Elisp LSP Server

[Stefan Monnier]

> IFF this is the case, having to start a whole second Emacs instance as a
> daemon, just to edit elisp with emacs seems like a significant resource
> sink with no real benefit other than adding additional failure points.

FWIW, starting a second Emacs process is exactly what Flymake does in
`emacs-lisp-mode` ;-)


        Stefan

Re: Elisp LSP Server

[Dmitry Gutov]

On 13.10.2021 23:00, Tim Cross wrote:
> IFF this is the case, having to start a whole second Emacs instance as a
> daemon, just to edit elisp with emacs seems like a significant resource
> sink with no real benefit other than adding additional failure points.

FWIW, 'emacs -Q' has a much faster start than most of language servers 
out there. So it shouldn't be a problem. What *would* -- is that most of 
the language smarts we have implemented for Elisp relies on edited code 
being loaded in the current session. Which seems hard to synchronize to 
the external Emacs instance.

But nothing should stop us from adding an additional "transport" for the 
LSP protocol -- e.g. an in-process one.

It's still probably not a good idea (basing all our IDE features solely 
on the LSP protocol). See my first message in this thread.

Re: Elisp LSP Server

[Eli Zaretskii]

> From: Tim Cross <theophilusx@gmail.com>
> Date: Thu, 14 Oct 2021 07:00:25 +1100
> Cc: Joost Kremers <joostkremers@fastmail.fm>, emacs-devel@gnu.org
> 
> I doubt very mush that anyone will ever bother to implement an LSP
> language server for elisp. There simply isn't sufficient benefit given
> the work required.

The main benefit, AFAIU, is to use the system's execution units more
efficiently by off-loading some of the CPU-intensive work to another
Emacs process.

> Even less likely for non-Emacs users if to have an
> elisp LSP language server you need to install Emacs to provide the
> daemon, which will be slow to start and resource hungry compared to
> other LSP language servers.

I'm surprised.  How large are memory footprints of "other LSP language
servers", and how large do you envision the footprint of Emacs-based
server to be?  Emacs has ceased to be THE memory hog long ago, it's
actually quite modest in its memory requirements nowadays.  So I don't
see why this particular job would require Emacs to become such a
memory-hungry application.

> I think it also largely misses the point -
> the idea of LSP is to provide a small, fast and easy to install basic
> service which can assist an editor to provide intelligent completion,
> linting, syntax highlighting etc for a language. All of the other LSP
> language servers I've used are small, fast to start and easy to install.
> Emacs as a daemon is none of these.

I think the time to start is not very important, since it happens
once.

Re: Elisp LSP Server

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> The main benefit, AFAIU, is to use the system's execution units more
> efficiently by off-loading some of the CPU-intensive work to another
> Emacs process.

On the other hand, this doesn't require making the Emacs instance act as
a language server, which will also come with the benefit of not having
to use a mechanism as bloated as JSON RPC for communication.

Implementing an entire Language Server when the goal is to offload work
onto another execution unit seems overkill to me.

> I'm surprised.  How large are memory footprints of "other LSP language
> servers", and how large do you envision the footprint of Emacs-based
> server to be?  Emacs has ceased to be THE memory hog long ago, it's
> actually quite modest in its memory requirements nowadays.  So I don't
> see why this particular job would require Emacs to become such a
> memory-hungry application.

FWIW, CCLS, a popular language server for C/C++ code can easily take
many gigabytes of RAM when working on relatively large C++ codebases.

I don't remember exactly how much CCLS took on the Emacs codebase
(everything in lib, lib-src, and src), but it was somewhere around 600
megabytes.

Re: Elisp LSP Server

[Po Lu]

Tim Cross <theophilusx@gmail.com> writes:

> The use for an elisp language server for non-Emacs editors seems like a
> very very small use case to me. Even the proposed use case of being able
> to edit a broken init.el file with a different editor seems unlikely. In
> fact, I doubt very much elisp is of any interest to anyone who isn't an
> Emacs user and emacs -Q is probably still more convenient than using a
> completely different editor to edit your init.el file. 

Very very small, apart from the use-case the OP mentioned: to benefit
users of GitHub, which will drive users to GitHub and make them use VS
Code.

> At any rate, I only entered this thread to address the misinformation
> about LSP being evil and encouraging the use of non-free software. LSP
> is simply a protocol - how it is implemented and how it is used is down
> to developers and users - arguing that providing an LSP server would
> encourage the use of non-free software is extremely unlikely. Elisp is
> of no interest to non-Emacs users and while it might seem that elisp is
> the greatest language ever to those who have drunk the Emacs kool-aid,
> to those outside Emacs, it is just some esoteric editor configuration
> language of little interest. 

I didn't suggest that LSP is evil, but providing an LSP server for Emacs
Lisp is; it is likely to make life lucrative for the significant portion
of Emacs users who develop their configurations with GitHub, which will
in turn lead to more and more Emacs users using GitHub.

Which is precisely because there is no use for such a language server
anywhere else.

Re: Elisp LSP Server

[dick]

PL> ... is likely to make life lucrative ...

I don't think that word means what you think it means (Princess Bride, 1987).

The current and foreseeable quality of lsp-mode is reason enough not to bother
writing an elisp service.  Until that changes, pontificating about its moral
suitability is, on the face of it, quite silly, but also a bad case of
premature optimization.

Re: Elisp LSP Server

[Augusto Stoffel]

On Thu, 14 Oct 2021 at 07:00, Tim Cross <theophilusx@gmail.com> wrote:

> At any rate, I only entered this thread to address the misinformation
> about LSP being evil and encouraging the use of non-free software. LSP
> is simply a protocol - how it is implemented and how it is used is down
> to developers and users - arguing that providing an LSP server would
> encourage the use of non-free software is extremely unlikely. Elisp is
> of no interest to non-Emacs users and while it might seem that elisp is
> the greatest language ever to those who have drunk the Emacs kool-aid,
> to those outside Emacs, it is just some esoteric editor configuration
> language of little interest. 

I wouldn't paint LSP as being that neutral.  It is defined by Microsoft
and primarily with VSCode in mind.  Yes, they do take input from the
community and encourage other editors to support the protocol.  But
looking into the details, you notice that this aspect tends to be an
afterthought.

As a silly example, note that LSP defines methods such as "hover" and
"code lenses".  Whatever those VSCode features are, they could have used
a common name in the protocol.

As a more technical example, one can mention that the LSP completion
method doesn't return the bounds of the thing being completed (the
equivalent of START and END in `completion-at-point-functions').  There
have been several discussions on LSP's issue tracker about this problem,
cf. [1] and linked threads.  The conclusion, AFAICT, is that VSCode can
live with this limitation so the LSP protocol won't be fixed.

I think it's great to have an LSP client for Emacs, but it's important
to keep it as a simple translation layer between Emacs's editing
features and the current popular API for code analyzers (especially when
it's not a particularly well designed one).

[1]: https://github.com/microsoft/language-server-protocol/issues/648

Re: Elisp LSP Server

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1232 bytes --]

On Wed, Oct 13, 2021, 18:34 Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 13.10.2021 20:15, Joost Kremers wrote:
> >> - No idea what language to use or how to implement the server. It has
> >> been suggested elisp would be the obvious choice, but I don't see how
> >> you can implement elisp outside Emacs without pretty much implementing
> >> most of emacs.
> > Why not just use Emacs? Run an Emacs daemon process (properly isolated,
> so it
> > doesn't interfere with the user's Emacs process), feed it the Elisp code
> and
> > have it return the analytics that LSP requires. That doesn't sound too
> difficult.
>
> That seems like the only practical way to implement such a server, indeed.
>

Not only practical, but also the right one, IMHO

The use of jsonrpc.el, an existing Emacs library for JSONRPC communication,
as well as parts of the Eglot LSP client (also slated to be in Emacs core)
can more than likely help when doing this.

Even though Eglot is a LSP client, a very significant part of its code
could be used in the server.

If there is serious interest in making a eglot-servel.el, I can arrange the
split of eglot.el into eglot-common.el and eglot-client.el libraries.

João

>
>

[-- Attachment #2: Type: text/html, Size: 1943 bytes --]

Re: Elisp LSP Server

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I ask those who support the free software movement to decline to
develop code to help VScode profit from using GNU Emacs as a back end.

I don't think that misfeature would put an end to Emacs as a real
editor.  But supporting it would suggest that we are naifs who can
be convinced to help our adversaries because "it's the nice thing to do."

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Elisp LSP Server

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 497 bytes --]

On Fri, Oct 15, 2021, 23:51 Richard Stallman <rms@gnu.org> wrote:

I ask those who support the free software movement to decline to
> develop code to help VScode profit from using GNU Emacs as a back end.
>

Ok. I'm sorry I'm not following every message in this thread, but does this
means that the Elisp LSP server is not wanted after all?  Just
double-checking.

I don't have many strong feelings towards that, but I had the impression
that you and others once wanted this.

João

[-- Attachment #2: Type: text/html, Size: 998 bytes --]

Re: Elisp LSP Server

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Ok. I'm sorry I'm not following every message in this thread, but does this
  > means that the Elisp LSP server is not wanted after all?

Indeed, it does mean that.  An Emacs Lisp LSP server won't help us
give Emacs whatever editing features for Emacs Lisp we want.  All it
would help is adding those features to VScode.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Elisp LSP Server

[Dmitry Gutov]

On 18.10.2021 02:49, Richard Stallman wrote:
> All it
> would help is adding those features to VScode.

And Vim. And Atom, Kate, GNOME Builder, Acme, Kakoune, Sublime Text, 
Brackets, Theira... and other editors which either support LSP ootb, or 
have plugins for that.

Re: Elisp LSP Server

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > And Vim. And Atom, ... and other editors which either support LSP
  > ootb, or have plugins for that.

That side effect is not weighty enough to affect the decision.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Elisp LSP Server

[Tim Cross]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Ok. I'm sorry I'm not following every message in this thread, but does this
>   > means that the Elisp LSP server is not wanted after all?
>
> Indeed, it does mean that.  An Emacs Lisp LSP server won't help us
> give Emacs whatever editing features for Emacs Lisp we want.  All it
> would help is adding those features to VScode.


this does of course totally ignore the very valid point made by Eli re:
off loading some processing to a separate process which could make Emacs
more responsive and to some extent, overcome the inherent limitations
imposed by a single threaded editor.

It also seems like premature fear optimisation - the use case for an
elisp LSP server for non-Emacs users is very small and the use case for
those who want that who also use VSCode is even smaller. I'm not even
convinced there is really a use case for editing elisp outside of Emacs
(even with VS Code). You would still need to run emacs in order to
execute/use the code. It could even be that having such a facility might
result in more people actually using Emacs once they realise the
potential of elisp and the power that gives an editor. 

It makes no sense to discourage such a development over fears which are
based on such weak grounds.

Re: Elisp LSP Server

[Stefan Monnier]

> ... sense ... fears ...

Hmm.. those two don't mix well, IIRC?


        Stefan

Re: Elisp LSP Server

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Sun, 17 Oct 2021 19:49:56 -0400
> Cc: joostkremers@fastmail.fm, theophilusx@gmail.com, dgutov@yandex.ru,
>  emacs-devel@gnu.org
> 
>   > Ok. I'm sorry I'm not following every message in this thread, but does this
>   > means that the Elisp LSP server is not wanted after all?
> 
> Indeed, it does mean that.  An Emacs Lisp LSP server won't help us
> give Emacs whatever editing features for Emacs Lisp we want.  All it
> would help is adding those features to VScode.

I don't share that conclusion.

Re: Elisp LSP Server

[Stefan Monnier]

>> Indeed, it does mean that.  An Emacs Lisp LSP server won't help us
>> give Emacs whatever editing features for Emacs Lisp we want.  All it
>> would help is adding those features to VScode.
>
> I don't share that conclusion.

+1

In any case, there is no such LSP server and I don't know of anyone who
has even real plans to start working on it any time soon, so this is all
very hypothetical.

But I really see no reason to try and discourage anyone to start working
on it.


        Stefan

Re: Elisp LSP Server

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: rms@gnu.org,  joostkremers@fastmail.fm,  theophilusx@gmail.com,
>   emacs-devel@gnu.org,  joaotavora@gmail.com,  dgutov@yandex.ru
> Date: Mon, 18 Oct 2021 08:43:30 -0400
> 
> >> Indeed, it does mean that.  An Emacs Lisp LSP server won't help us
> >> give Emacs whatever editing features for Emacs Lisp we want.  All it
> >> would help is adding those features to VScode.
> >
> > I don't share that conclusion.
> 
> +1
> 
> In any case, there is no such LSP server and I don't know of anyone who
> has even real plans to start working on it any time soon, so this is all
> very hypothetical.

Yes, as long as this is a theoretical issue, I see no reason to
discuss is seriously, let alone have a dispute about it.  We can cross
that bridge if and when we get to it.

Re: Elisp LSP Server

[Po Lu]

Joost Kremers <joostkremers@fastmail.fm> writes:

> Why not? In fact, it arguably already is.

It isn't IMO.  For instance, I haven't seen a general purpose
programming language that doesn't have file streams yet.

> You can't really say that something like Gnus is just an Emacs
> extension.

Why is that?  Gnus is a newsreader, and net news is usually just text,
so it fits into a text editor very well.

[OT] Elisp as a general purpose programming language (was: Elisp LSP Server)

[Joost Kremers]

On Thu, Oct 14 2021, Po Lu wrote:
> It isn't IMO.  For instance, I haven't seen a general purpose
> programming language that doesn't have file streams yet.

Yes, but where is the definition of "general-purpose programming language" that
explicitly includes file streams as a requirement? ;-)

OK, that sounded more pedantic that I wish... The point is that I think this
would be quibbling over semantics, so probably not a very useful discussion.

>> You can't really say that something like Gnus is just an Emacs
>> extension.
>
> Why is that?  Gnus is a newsreader, and net news is usually just text,
> so it fits into a text editor very well.

But news (and mail) readers exist as standalone programs. Such applications
involve editing text, which happens to be something Emacs is good at. So
building them on top of Emacs is an added benefit for those applications,
because they get all the power of Emacs for an important part of their
functionality.

In my understanding of an Emacs extension (or perhaps "plug-in" would be a
better term), however, the benefit flows in the other direction: from the
extension to Emacs. An extension makes text editing (or more broadly, using
Emacs) more convenient. (An extension would not make sense as a standalone
program, for example.)

I'm sure there's gonna be a large grey area, though, (Org, Magit?) so again I'm
afraid this is gonna end in quibbling over semantics...

Anyway, long story short, I do see Elisp as a general-purpose programming
language, but I did say "arguably". :-)

-- 
Joost Kremers
Life has its moments

Re: [OT] Elisp as a general purpose programming language (was: Elisp LSP Server)

[Jean-Christophe Helary]

> On Oct 14, 2021, at 17:43, Joost Kremers <joostkremers@fastmail.fm> wrote:
> 
> I'm sure there's gonna be a large grey area, though, (Org, Magit?) so again I'm
> afraid this is gonna end in quibbling over semantics...

Magit is *clearly* not about text editing.



-- 
Jean-Christophe Helary @brandelune
https://mac4translators.blogspot.com
https://sr.ht/~brandelune/omegat-as-a-book/

Re: Elisp LSP Server

[Po Lu]

Tim Cross <theophilusx@gmail.com> writes:

> I think the LSP situation has been both misrepresented and
> misunderstood.
>
> There is nothing inherently evil about LSP. It is just a protocol.

There isn't, but in the message Richard replied to I wasn't talking
about LSP, just lsp-mode.

There is another LSP client for Emacs, eglot, that has none of these
problems.