CL packages landed

CL packages landed

[Gerd Möllmann]

I've pushed the branch "pkg" which implements CL packages for Emacs.

Before Someone (tm) gets the idea of blowing up something like Internet
cables instead of pipelines, or we have blackouts here, or whatever, I
thought I'd better make this accessible.

What to expect:

- It runs on macOS with unchanged init file.  I'm not using much, but I
  can say that it works with Helm and Lsp,

- I know of a bug when connecting to Gmail imap using Gnus that's not
  there in master.

- YMMV, it's all only a few weeks old, if at all.


What it does:

- Lisp_Package is a new Lisp type.

- There are 2 predefined packages "emacs" and "keyword", and a
  current package *package* (buffer-local IIRC).

- The whole package stuff is available very early in C. It's initialized
  right after init_alloc.

- Obarrays are gone.  Good riddance. I removed them in a
  backward-compatible way.

- Symbols have packages (symbol-package).  Uninterned symbols have
  a nil package, others have "emacs" or "keyword".

- Lisp_Symbol::next is gone.  So, the size of Lisp_Symbol is unchanged.

- The symbol name of keywords does NOT include the colon.  That turned
  out the only sane way to do it, after some experimentation.

- A number of places in C that assumed that symbol names of keywords
  start with ':' are changed.  On the Lisp side, symbol-name for
  keywords returns a string starting with ':'.  I tried without, but
  that wasn't compatible enough.  There is a cl-symbol-name not
  prepending the ':'.

- Some functions like list-all-packages, make-package are implemented in
  Lisp.  Many others are still missing, defpackage most notably.  I plan
  to add them based on public-domain code from CMUCL.

- Reading symbols and printing symbols is changed.

- A new variable package-prefixes controls if ':' in symbol names is
  interpreted as a separating package names from symbol names.  Default
  is not to.  Astonishingly many code uses such symbol names, the best
  one I've seen is a complete URL 'http://...' :-).


What it is not/does not have:

- Bug-free, for sure, but it is pretty usable.

- Documentation.

- A complete Lisp-side implementation of packages (defpackage,
  use-package, ...

- Tests of the new stuff.  There is a stub under test/, but it doesn't
  do much.

- Fixes for tests of old stuff.  In fact, I haven't run make check yet.
  Weeding out all bugs is currently not my focus.

- shorthand.el is currently not supported.

- Support for pure space is missing.  I have cherry-picked a commit by
  Stefan that removes uses of pure_alloc.  I didn't want to deal with
  pure space, and I can't test it anyway here.

- Other stuff that I currently forget.

And the code is not entirely terrible, at least IMO :-).


Plans:

I think I'll continue using this branch.  No plans, no commitments
though.


Have fun :-)

Re: CL packages landed

[Po Lu]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> - Support for pure space is missing.  I have cherry-picked a commit by
>   Stefan that removes uses of pure_alloc.  I didn't want to deal with
>   pure space, and I can't test it anyway here.

This is definitely a Good Thing.

As long as the unexec build remains working, that is.

Re: CL packages landed

[Stefan Kangas]

[-- Attachment #1: Type: text/plain, Size: 801 bytes --]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> I've pushed the branch "pkg" which implements CL packages for Emacs.

I saw these GCC warnings while compiling:

pkg.c:43:3: warning: function ‘pkg_error’ might be a candidate for
‘gnu_printf’ format attribute [-Wsuggest-attribute=format]
   43 |   verror (fmt, ap);
      |   ^~~~~~
pkg.c:39:1: warning: function might be candidate for attribute
‘noreturn’ [-Wsuggest-attribute=noreturn]
   39 | pkg_error (const char *fmt, ...)
      | ^~~~~~~~~

See the attached attempt to fix them.

While playing around with this, I also came up with the attached small
additions to the test suite (loosely based on the HyperSpec).

BTW, I think according to our branch naming conventions this should be
named "feature/pkg"?

[-- Attachment #2: 0001-src-lisp.h-pkg_error-Fix-GCC-warning-in-declaration.patch --]
[-- Type: text/x-diff, Size: 963 bytes --]

From 6987c5bbb794a001f6e5654bd9ea9c0e097bac09 Mon Sep 17 00:00:00 2001
From: Stefan Kangas <stefankangas@gmail.com>
Date: Fri, 21 Oct 2022 08:12:59 +0200
Subject: [PATCH 1/2] * src/lisp.h (pkg_error): Fix GCC warning in declaration.

---
 src/lisp.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/lisp.h b/src/lisp.h
index 2042dbbd2f..f60583a150 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -2272,7 +2272,7 @@ PACKAGE_NAMEX (Lisp_Object package)
 extern void init_pkg (void);
 extern void syms_of_pkg (void);
 extern Lisp_Object pkg_qualified_symbol (Lisp_Object name, Lisp_Object package, bool external);
-extern void pkg_error (const char *fmt, ...);
+extern _Noreturn void pkg_error (const char *fmt, ...) ATTRIBUTE_FORMAT_PRINTF (1, 0);
 extern Lisp_Object pkg_unqualified_symbol (Lisp_Object name);
 extern bool pkg_keywordp (Lisp_Object obj);
 extern Lisp_Object pkg_define_symbol (Lisp_Object sym, Lisp_Object package);
-- 
2.35.1


[-- Attachment #3: 0002-Fix-cl-intern-cl-unintern-tests.patch --]
[-- Type: text/x-diff, Size: 1144 bytes --]

From dd02db5702e46af6543be14cb05d981fa261ad1a Mon Sep 17 00:00:00 2001
From: Stefan Kangas <stefankangas@gmail.com>
Date: Fri, 21 Oct 2022 08:13:25 +0200
Subject: [PATCH 2/2] Fix cl-intern/cl-unintern tests

* test/src/pkg-tests.el (pkg-tests-cl-intern)
(pkg-tests-cl-unintern): Fix tests.
---
 test/src/pkg-tests.el | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/test/src/pkg-tests.el b/test/src/pkg-tests.el
index 1cfea6a632..aff6ff0eb5 100644
--- a/test/src/pkg-tests.el
+++ b/test/src/pkg-tests.el
@@ -126,10 +126,21 @@ pkg-tests-find-symbol
   (should nil))
 
 (ert-deftest pkg-tests-cl-intern ()
-  (should nil))
+  (cl-assert (not (find-symbol "foo")))
+  (unwind-protect
+      (progn
+        (cl-intern "foo")
+        (should (find-symbol "foo")))
+    (cl-unintern 'foo)))
 
 (ert-deftest pkg-tests-cl-unintern ()
-  (should nil))
+  (cl-assert (not (find-symbol "foo")))
+  (unwind-protect
+      (progn
+        (cl-intern "foo")
+        (cl-unintern 'foo)
+        (should-not (find-symbol "foo")))
+    (cl-unintern 'foo)))
 
 (ert-deftest pkg-tests-export ()
   (should nil))
-- 
2.35.1

Re: CL packages landed

[Gerd Möllmann]

Stefan Kangas <stefankangas@gmail.com> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> I've pushed the branch "pkg" which implements CL packages for Emacs.
>
> I saw these GCC warnings while compiling:
>
> pkg.c:43:3: warning: function ‘pkg_error’ might be a candidate for
> ‘gnu_printf’ format attribute [-Wsuggest-attribute=format]
>    43 |   verror (fmt, ap);
>       |   ^~~~~~
> pkg.c:39:1: warning: function might be candidate for attribute
> ‘noreturn’ [-Wsuggest-attribute=noreturn]
>    39 | pkg_error (const char *fmt, ...)
>       | ^~~~~~~~~
>
> See the attached attempt to fix them.
>
> While playing around with this, I also came up with the attached small
> additions to the test suite (loosely based on the HyperSpec).
>

Thanks!  Please feel free to push on that branch.

> BTW, I think according to our branch naming conventions this should be
> named "feature/pkg"?

No idea, is that so?  Should I rename it? 

Re: CL packages landed

[Gerd Möllmann]

Stefan Kangas <stefankangas@gmail.com> writes:

> pkg.c:39:1: warning: function might be candidate for attribute
> ‘noreturn’ [-Wsuggest-attribute=noreturn]
>    39 | pkg_error (const char *fmt, ...)
>       | ^~~~~~~~~

Err, and BTW, signaling the right type of error (package-error, for
instance) is also not yet done :-).

Re: CL packages landed

[Po Lu]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> No idea, is that so?  Should I rename it? 

While we're on the subject of naming, wouldn't "package.c" be a better
name for the file?

Re: CL packages landed

[Gerd Möllmann]

Po Lu <luangruo@yahoo.com> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> No idea, is that so?  Should I rename it? 
>
> While we're on the subject of naming, wouldn't "package.c" be a better
> name for the file?

I tried that in the beginning, but it resulted in a conflict in the tests.

Re: CL packages landed

[Andrea Corallo]

Hi Gerd,

I've no time to comment on the patch, but this looks like great
improvement!

Thanks

  Andrea

Re: CL packages landed

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Fri, 21 Oct 2022 08:48:13 +0200
> 
> > BTW, I think according to our branch naming conventions this should be
> > named "feature/pkg"?
> 
> No idea, is that so?  Should I rename it? 

Yes, please.

Re: CL packages landed

[Gerd Möllmann]

Andrea Corallo <akrl@sdf.org> writes:

> I've no time to comment on the patch, but this looks like great
> improvement!

Thanks!

Re: CL packages landed

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> No idea, is that so?  Should I rename it? 
>
> Yes, please.

Done.  I also deleted origin/pkg.

BTW, can I read somewhere what the meaning of scratch/ and feature/ is?

Re: CL packages landed

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: stefankangas@gmail.com,  emacs-devel@gnu.org
> Date: Fri, 21 Oct 2022 10:01:24 +0200
> 
> BTW, can I read somewhere what the meaning of scratch/ and feature/ is?

The obvious meanings: feature/ is for feature branches that are
intended to live for some time, scratch/ is for one-off
throw-away-after-use branches.

Re: CL packages landed

[Gerd Möllmann]

On 22-10-21 12:48 , Eli Zaretskii wrote:
>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: stefankangas@gmail.com,  emacs-devel@gnu.org
>> Date: Fri, 21 Oct 2022 10:01:24 +0200
>>
>> BTW, can I read somewhere what the meaning of scratch/ and feature/ is?
> 
> The obvious meanings: feature/ is for feature branches that are
> intended to live for some time, scratch/ is for one-off
> throw-away-after-use branches.

Thanks.

Re: CL packages landed

[Stefan Monnier]

>>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>>> Cc: stefankangas@gmail.com,  emacs-devel@gnu.org
>>> Date: Fri, 21 Oct 2022 10:01:24 +0200
>>>
>>> BTW, can I read somewhere what the meaning of scratch/ and feature/ is?
>> The obvious meanings: feature/ is for feature branches that are
>> intended to live for some time, scratch/ is for one-off
>> throw-away-after-use branches.

Among other things: force-pushes are tolerated on `scratch/*` as well as
commits with poor style, incomplete commit messages etc... because we do
not intend to `git merge` from scratch branches, whereas we do expect to
`git merge` from `feature/*` so: no force push, and no commits that
don't have a proper commit message.


        Stefan

Re: CL packages landed

[Gerd Möllmann]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>>>> Cc: stefankangas@gmail.com,  emacs-devel@gnu.org
>>>> Date: Fri, 21 Oct 2022 10:01:24 +0200
>>>>
>>>> BTW, can I read somewhere what the meaning of scratch/ and feature/ is?
>>> The obvious meanings: feature/ is for feature branches that are
>>> intended to live for some time, scratch/ is for one-off
>>> throw-away-after-use branches.
>
> Among other things: force-pushes are tolerated on `scratch/*` as well as
> commits with poor style, incomplete commit messages etc... because we do
> not intend to `git merge` from scratch branches, whereas we do expect to
> `git merge` from `feature/*` so: no force push, and no commits that
> don't have a proper commit message.

That makes sense, thanks. 

I don't think I've force-pushed in my life, so that's not a problem.
But the existing commit messages might be.  It was originally mainly an
experiment, after all.

Please, could someone with enough time at their hands take a look at
the log if that's good enough?

  git log --author=gerd master...

Without the --author, stuff from a merge will be shown.

RE: [External] : CL packages landed

[Drew Adams]

> I've pushed the branch "pkg" which implements CL packages for Emacs.

Bonne initiative !  Really appreciated, Gerd.
___

Now we should figure out a good way to disambiguate
the use of the word "package", for our docs etc.,
to help users understand that this and the Emacs
"package" system are completely different species.

Re: [External] : CL packages landed

[Gerd Möllmann]

I call them symbol-packages if in doubt. 

> Am 21.10.2022 um 17:06 schrieb Drew Adams <drew.adams@oracle.com>:
> 
> 
>> 
>> I've pushed the branch "pkg" which implements CL packages for Emacs.
> 
> Bonne initiative !  Really appreciated, Gerd.
> ___
> 
> Now we should figure out a good way to disambiguate
> the use of the word "package", for our docs etc.,
> to help users understand that this and the Emacs
> "package" system are completely different species.

RE: [External] : CL packages landed

[Drew Adams]

> > Now we should figure out a good way to disambiguate
> > the use of the word "package", for our docs etc.,
> > to help users understand that this and the Emacs
> > "package" system are completely different species.
>
> I call them symbol-packages if in doubt.

Sure.  But a simpler name might be useful.  And folks
might anyway end up calling them "packages" - e.g.,
when they _think_ what's meant isn't in doubt.

"Obarray" was an obscure word, but it had the upside
of being one of a kind - unambiguous.
___

(I hear a whispered "snobarray", for symbol-name
obarray.  No, I'm not serious.)

Re: CL packages landed

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> BTW, can I read somewhere what the meaning of scratch/ and feature/ is?
>>> The obvious meanings: feature/ is for feature branches that are
>>> intended to live for some time, scratch/ is for one-off
>>> throw-away-after-use branches.
>
> Among other things: force-pushes are tolerated on `scratch/*` as well as
> commits with poor style, incomplete commit messages etc... because we do
> not intend to `git merge` from scratch branches, whereas we do expect to
> `git merge` from `feature/*` so: no force push, and no commits that
> don't have a proper commit message.

Among other things: automatic tests are run for feature/* branches on
emba. See https://emba.gnu.org/emacs/emacs/-/pipelines

>         Stefan

Best regards, Michael.

Re: CL packages landed

[Gerd Möllmann]

On 22-10-22 11:40 , Michael Albinus wrote:
> 
> Among other things: automatic tests are run for feature/* branches on
> emba. See https://emba.gnu.org/emacs/emacs/-/pipelines

That's good to know, thanks.

BTW, I don't know what's confusing emba (probably something I did :-)), 
but it seems not to be on HEAD of the branch, but on some commit I can't 
even find here.

Re: CL packages landed

[Stefan Kangas]

Michael Albinus <michael.albinus@gmx.de> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>>>> BTW, can I read somewhere what the meaning of scratch/ and feature/ is?
>>>> The obvious meanings: feature/ is for feature branches that are
>>>> intended to live for some time, scratch/ is for one-off
>>>> throw-away-after-use branches.
>>
>> Among other things: force-pushes are tolerated on `scratch/*` as well as
>> commits with poor style, incomplete commit messages etc... because we do
>> not intend to `git merge` from scratch branches, whereas we do expect to
>> `git merge` from `feature/*` so: no force push, and no commits that
>> don't have a proper commit message.
>
> Among other things: automatic tests are run for feature/* branches on
> emba. See https://emba.gnu.org/emacs/emacs/-/pipelines

I added the above to admin/notes/repo.  Maybe we should move it to a
separate file to make it easier to find though.

In any case, feel free to fix, extend, revert, etc.

Re: CL packages landed

[Philip Kaludercic]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> I've pushed the branch "pkg" which implements CL packages for Emacs.
>
> Before Someone (tm) gets the idea of blowing up something like Internet
> cables instead of pipelines, or we have blackouts here, or whatever, I
> thought I'd better make this accessible.

I am very glad to see this!  Could you comment on what larger influence
with might have on Emacs Lisp as a language.  E.g. in bug#57907 there
was a discussion how adding keyword support for cl-loop was not
necessary for Emacs Lisp, but now that symbol packages might become a
thing, would that have to change?  How would the support look like from
a backwards compatibility perspective?  I am assuming all old code would
be loaded into the default packages, right?

Re: CL packages landed

[Gerd Möllmann]

Philip Kaludercic <philipk@posteo.net> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> I've pushed the branch "pkg" which implements CL packages for Emacs.
>>
>> Before Someone (tm) gets the idea of blowing up something like Internet
>> cables instead of pipelines, or we have blackouts here, or whatever, I
>> thought I'd better make this accessible.
>
> I am very glad to see this!  Could you comment on what larger influence
> with might have on Emacs Lisp as a language.

I can say something about what my goal is. If this ever lands in master
is a different story, I haven't heard something like that being said.
But packages are now there, and won't disappear :-).

So, my goal:

- Be backwards-compatible for users and developers.

- Provide the benefits of packages to developers, mainly.  Where users
  writing their own stuff would be developers as well, who use currently
  "my-" prefixes for own functions etc.
  
  Some of the benefits: The ability to export only a subset of
  functions/variables, while being able to use unobstrusive names.  It
  would no longer be neccessary to use prefixes like or "xyz-", or
  "xyz--".  You could define a package "xyz", and reference definitionsi
  with something like "xyz:..." or "xyz::..." (use only if you must
  access internals).  Or use no prefix at all by importing symbols or
  packages.

  Emacs itself could hide tons of internal definitions like that, and
  export only what is in the public interface.

  And so on, you get the idea.

And one could think of extensions, which are currently not in scope for
me, like hierarchical packages (think Java, x.y.z.collections) that
could be added on top, to structure the package namespace.  CMUCL and
Allegro CL have something like that.

> E.g. in bug#57907 there was a discussion how adding keyword support
> for cl-loop was not necessary for Emacs Lisp, but now that symbol
> packages might become a thing, would that have to change?

I remember that.  The implementation of cl-loop would have to be changed
to work like it does in Common Lisp, that is, it would accept something
like 'for' regardless of for's package.  That's not hard.

> How would the support look like from a backwards compatibility
> perspective?  I am assuming all old code would be loaded into the
> default packages, right?

Right, and that's already the case in feature/pkg, and it seems to work
well, except that Gnus doesn't start for me.  I couldn't figure that one
out yet.  Gnus is large, and tedious to debug if one doesn't know its
internals.

Other than that, what I use (Magit, Helm, Lsp, ...) seems to work just
fine, and meanwhile the existing tests also succeed.  That looks like
good news in the compatibility department so far.

Performance seems to be more or less the same, unsurprisingly, a full
build is a tad faster in pkg.  No difference noticeable interactively.

Re: CL packages landed

[Michael Albinus]

Stefan Kangas <stefankangas@gmail.com> writes:

> Michael Albinus <michael.albinus@gmx.de> writes:
>
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>
>>>>>> BTW, can I read somewhere what the meaning of scratch/ and feature/ is?
>>>>> The obvious meanings: feature/ is for feature branches that are
>>>>> intended to live for some time, scratch/ is for one-off
>>>>> throw-away-after-use branches.
>>>
>>> Among other things: force-pushes are tolerated on `scratch/*` as well as
>>> commits with poor style, incomplete commit messages etc... because we do
>>> not intend to `git merge` from scratch branches, whereas we do expect to
>>> `git merge` from `feature/*` so: no force push, and no commits that
>>> don't have a proper commit message.
>>
>> Among other things: automatic tests are run for feature/* branches on
>> emba. See https://emba.gnu.org/emacs/emacs/-/pipelines
>
> I added the above to admin/notes/repo.  Maybe we should move it to a
> separate file to make it easier to find though.
>
> In any case, feel free to fix, extend, revert, etc.

Thanks. In admin/notes/emba, the branches to be considered are described
more precisely:

--8<---------------cut here---------------start------------->8---
Only branches whose name starts with 'master', 'emacs', 'feature', or
'fix' are considered.  This is declared in the workflow rules of file
'test/infra/gitlab-ci.yml'.
--8<---------------cut here---------------end--------------->8---

Best regards, Michael.

Re: CL packages landed

[Michael Albinus]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

Hi Gerd,

> BTW, I don't know what's confusing emba (probably something I did
> :-)), but it seems not to be on HEAD of the branch, but on some commit
> I can't even find here.

I've checked for the last reports on emba for your branch, and the
commit ids are also mentioned in the commit messages seen on the
emacs-diffs@gnu.org ML.

Best regards, Michael.

Re: CL packages landed

[Gerd Möllmann]

Yes, it works now, thanks, I think since Stefan K pushed :-). 
Build and tests successful. Nice!

> Am 22.10.2022 um 17:09 schrieb Michael Albinus <michael.albinus@gmx.de>:
> 
> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
> 
> Hi Gerd,
> 
>> BTW, I don't know what's confusing emba (probably something I did
>> :-)), but it seems not to be on HEAD of the branch, but on some commit
>> I can't even find here.
> 
> I've checked for the last reports on emba for your branch, and the
> commit ids are also mentioned in the commit messages seen on the
> emacs-diffs@gnu.org ML.
> 
> Best regards, Michael.

Re: CL packages landed

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

CL packages are the wrong way to implement packages in Lisp.
As I explained in a discussion two years ago, packages implemented
using obarrays (or equivalent) don't work reliably.

We have a much better basis for Lisp packages
in the shorthands mechanism.  It only needs to be completed.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: CL packages landed

[Gerd Möllmann]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> CL packages are the wrong way to implement packages in Lisp.
> As I explained in a discussion two years ago, packages implemented
> using obarrays (or equivalent) don't work reliably.

I can't find your explanation.  Do you have a pointer?

> We have a much better basis for Lisp packages
> in the shorthands mechanism.  It only needs to be completed.

Can you point me to a full design?  I've read the Elisp info page and
the sources.