Re: we need *modularity* [last problem] (was: Re: as for Calc and the math library)

Re: we need *modularity* [last problem] (was: Re: as for Calc and the math library)

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 8662 bytes --]

>> I have seen your previous answers to others, and I do
>> understand the point you make, but I think the argument
>> fails because a module can be just a proxy to
>> a non-free library.
>
>Emacs is notoriously not modular. Everything is everywhere,
>you seem to have everything but if fact you have nothing.
>There is so much stuff you can't find the basics stuff that
>isn't there.
>
>Here take a look: what does this name say?
>
>  cl--defalias
>
>Ah, that's a local function in the Common Lisp package, right?
>
>Well, YES, but ...
>
>It is actually an Elisp defun. And that prefix isn't really
>a package qualifier, it is just a convention, part of
>the name. And the double dash, likewise just a convention, we
>don't have package modularity or any of that kind of data
>hiding, abstraction, encapsulation ... what is it more called?
>Well, modularity!
>
>In Emacs everything is intermixed, Elisp-Emacs,
>non-interactive/interactive, data/data processing/the
>interface (buffer, buffer, and buffer), what is a file, what
>is a package, what is a library, and what is a program - no
>one can tell.
>
>Emacs Lisp isn't is fun, but absolutely not any kind
>of powerhouse. It is inconsistent in itself and on top of that
>all those zillion interfaces, cl-, seq-, slime- (yes, had to
>use that today, for `slime-transpose-lists').
>
>It is pretty fast to get something going but complexity
>skyrockets.
>
>What we should do, in general: modularity! clear divisions, in
>particular, libraries for everything, including those that provide basic
>stuff in a complete and consistent way.
>
>We should get real package modularity, transform the prefixes
>into real package qualifiers, and throw away the ugly
>local ones.
>
>With Elisp, optimize everything for development speed and
>convenience, have consistent interfaces as much as possible,
>acknowledge that Elisp was underpowered, which is why all the
>interfaces came, now, core Elisp and the interfaces should
>make peace, one should identify what it is exactly, what are
>the to 10 things that people use with cl-, pcase-, seq- etc,
>and bring them into a new core Elisp, and the few CL experts
>can still use all the rest.
>
>Now, with native compilation we have speed - we have power
>with the interfaces if one could harvest it, which one
>currently - well, it is very slow and difficult, anyway - the
>very, very last spet into maturity and getting out of the
>Elisp ghetto (and getting visitors!) are modularity, modern
>software principles - by modularity I mean real modularity,
>based on technology - not silly conventions - I forgot where
>I was - the last step is modularity in general, and libraries
>in particular.
>
>Clear cut interfaces. Get away with the ugly prefixes and
>error--prone conventions that don't even do anything.
>
>Elisp 3.0! \o/
>
>PS. This is the problem. This series is completed, I said
>    I would write 10, I wrote 4. Here are the other parts.
>    It is all based on the same theme. Modularity,
>    consistency, libraries, modern software practises.
>
>    https://lists.gnu.org/archive/html/emacs-devel/2024-08/msg00154.html
>    https://lists.gnu.org/archive/html/emacs-devel/2024-08/msg00380.html
>    https://lists.gnu.org/archive/html/emacs-devel/2024-08/msg00388.html
>
>PPS. So who is working on real modularity based on the package
>     with real public and local functions as we speak? I know
>     there are always some guy - at least - working on such
>     a basic idea. I know some way. And I know some day.

Emacs Lisp like all other programming languages has its problems. I am not sure
though all the things you have taken up in those mails are problems. I haven't
answered on previous ones, more than a cursory suggestion about adding some
explanation to the manual in one mail about FFI which seem to be just ingored.
IMO, we have to understand Emacs as application and what it has to do to
achieve its goal of an interactive text editor, in addition to be also a Lisp
implementation.

At low level, there is the central idiom, buffer, implemented as a gap buffer,
which has to be manipulated in some way. IMO, something like "goto-char", sounds
much better than "move-gap" in the context of text editing. They need to have
some low-level primitives on which to implement the higher level ones. Compare
to some other editors, like VSC or whichever. They are all in pretty similar
situation, since very few programming languages do have a "text" data structure
built into the language.

They need some sort of "assembly" to build on. As discussed in the mentioned
mail from yesterday or the day before, I think it could be added to the manual
about the programming model used in low-level text and buffer management: make
an object current, and than act upon it. This is not so uncommon, stack-based
procedural approach. PostScript and OpenGL are two notable languages/API that
use similar approach. There are probably others I am not familiar with. I don't
know if that was popular programming model during 80's when Emacs was designed
or not, but Emacs Lisp is not alone in its design.

I don't know what is a better alternative? Look at Xlib and numerous arguments
their functions take. Win32 uses structures to pass arguments, which leads to
less function arguments passed around, but one has to setup structures before
they are passed to a function. It is also ads verbosity. Yuri mentioned
iterators, but I think iterators would have similar problem as markers: one has
to update all iterators into a buffer if one of them writes to the buffer. I am
not sure, but iterator could be seen as "functional marker", i.e. a functional
interface to markers? Since Emacs Lisp does not have doted notation for
functions that work on objects, like some other languages that use iterators
extensively (C++, Java), I think iterators and markers are basically the same
thing in EmacsLisp? I haven't think of it too much, so I might be wrong.

So, I don't know, while yes, you are correct that the programmatic interface is
a low-level, I don't think it is a bad one, or that they had so much more choice
there. There is though nothing that forbids anyone to make a higher-level
interface on top of the low-level one. I think, as someone mentioned in some
earlier mail, org-mode, or perhaps font-lock, regular expressions, or various
programming modes could be seen as a higher level interfaces. At least
conceptually? They could be implemented on top of goto-char, char-before/after
and point, but for the performance reasons are not (just my guess).

Also Emacs has to implement its own command loop and renderer, and since most of
us would like some control over those from Lisp, they need to expose those parts
to Lisp as well. I don't think that that Emacs rendering and text processing is
all very entangled in some bad way. Emacs does implement a console-like
(character) renderer. The design is (somehwat) similar to other similar
projects. For example MS console API implements a similar design, a 2d matrix
of characters, with text properties describing colors and other rendering
attributes for characters. There is a long comment by Gerd in xdisp.c explaining
the design in Emacs.

I will definitely agree with you that Emacs needs modular development, but now
you are opening another can of worms into an already unwelcome discussion :).

I also agree with you that EmacsLisp and C core could get a big cleanup, the API
surface is big, especially now when Lisp can be compiled to machine code via
GCC. There also seem to be some dissonance with the manual, as seen from my
other mail about subrs and some other functions (which I hope will get an
answer), so I definitely also agree that Elisp manual could get a facelift too.

I actually wrote this explanation, including a small CL program ~100 sloc, that
implements toy version of Emacs text processing API for the illustrative purpose
days before, on your first mail about goto-char, but never sent because of
explicitly not spending time of maintainers on the discussion that they probably
don't care much of, so not to waste their time and understanding that the topic,
while interesting is probably unwelcome, and yet, Eli accused me of delibarately
wasting maintainers time and energy. Personally, I think it is interesting to
talk about low-level implementation details, at least to check and further my
own udnerstanding, but it is probably best to continue the discussion via PM if
you are interested.

best regards
/a

[-- Attachment #2: Type: text/html, Size: 27449 bytes --]

Re: we need *modularity* [last problem] (was: Re: as for Calc and the math library)

[Emanuel Berg]

arthur miller wrote:

> [...]

Thanks for your post.

Yes, I should stop rant but this realization that Emacs is
like a huge legacy application - no modularity, no clear
divisions, no sound policy how it happened - just go by
instinct and when problem comes - put a convention in a book
and add more stuff.

At the same time people say Lisp is the most powerful language
by definition and that cl-lib and pcase has ruined Emacs (both
of these are very clean cut in general; in the Emacs jungle,
they are a miracle of order).

With ELPA we have some modularity, we should move stuff from
core Emacs to ELPA and, as said, get real modularity based on
the package with real prefixes and - well, everything real!
And, based on technology and not some wishful thinking
a--human-convention can even reduce such huge disadvantages to
other technology.

So it is _really bad_, no excuses, had I known it was like
this - I don't know what would have happened - anyway ... it
is what it is.

But if we get real modularity we get real libraries and if we
do, there is hope.

-- 
underground experts united
https://dataswamp.org/~incal

a eieio-like interface we could use for functions (was: Re: we need *modularity* [last problem] (was: Re: as for Calc and the math library))

[Emanuel Berg]

Skip 4 paragraphs of meta-discussion:

  (re-search-forward "Anyway,")

arthur miller wrote:

> while interesting is probably unwelcome, and yet, Eli
> accused me of delibarately wasting maintainers time and
> energy.

Yeah, too much bad blood this time.

Maintainers should be busy with Emacs - and they are, I'm
sure. But some spend too much time visiting and commenting
almost _every one_ of my threads, crazy. There is no need,
I know what you are saying by now.

Some maintainers are also web cops on several of our lists.
If I were them, I would focus (even) more on Emacs and less on
policing these lists, and saying negative things about people
who have been regulars for decades.

Anyway, back to Emacs then, you remembered I said the
interface programming, that that was a crazy time consumer for
no reason. I found an interface in eieio - this is actually
what I suggested exactly (they did it long before) - maybe
some additional things are needed, but in principle - only it
is much more Lispy.

Fast and uniform, includes type and default and, if you know
eieio better than I, I'm sure you know a lot more of
those fields.

Again this is better since an un-lispy syntax would scare
people away and create controversy. Maybe I can get back to
that issue/idea as well, then!

(defclass box ()
  ((buf :initarg :buf
        :type string
        :initform "*bad*")
   (xc :initarg :xc
       :type number
       :initform 12)
   (yc :initarg :yc
       :type number
       :initform 8)))

https://dataswamp.org/~incal/emacs-init/bad.el

-- 
underground experts united
https://dataswamp.org/~incal

Re: we need *modularity* [last problem] (was: Re: as for Calc and the math library)

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 3634 bytes --]

>arthur miller wrote:
>
>> [...]
>
>Thanks for your post.

Thanks for the answer.

>Yes, I should stop rant but this realization that Emacs is
>like a huge legacy application - no modularity, no clear
>divisions, no sound policy how it happened - just go by
>instinct and when problem comes - put a convention in a book
>and add more stuff.

Well, yes, 40 years of development does leave some traces I guess.
Different people, professions, epochs, styles, etc.

The big API surface also might contribute to this. Some duplication is
innevitable, but some problems are solved independently, multiple times,
either because of people not being aware of something already existing (myself
been guilty) or simply NIH-syndrome.

I don't think EmacsLisp nor Lisps in general are alone in this. However, Lisp(s)
are very easy to hack, so it is perhaps seen more often in Lisp world: it is
easier to hack your own than to learn and use someone elses code. Examples in
Emacs world are numerous, but I don't want to offend anyone, and I dont' think
it would change anything, so I won't enumerate neither built-in stuff nor
external and well-known packages.

>At the same time people say Lisp is the most powerful language

I don't know. Lisp is a family of languages. As I have learned from a post by
K. Pitman to comp.lang.lisp, McCarthy asked explicitly for none of Lisp dialects
(or languages) to be named "Lisp".

I don't think, any of Lisp languages in popular use is *the* "most powerful
language" if such a qualification among programming languages would even be
possible, certainly not EmacsLisp. But I do think that Lisp languages can be
very elegant. Or at least they have a potential to be elegant, EmacsLisp not
being an exception.

For the reflection on Python, I have stated my opinion here a couple of years or
more ago, and I still stand with the same. I prefer Lisp to Python, if I may
choose, and since I don't consult any more, I can :-). IMO opinion, there is
nothing really interesting about Python. It is a C-like language with some
extensions found in other C-like languages (OO, etc), with somewhat cleaner
syntax and executed in an interpreter. Its real value is interpreter and dynamic
nature. It is indeed an easy and useful language to use, but as interesting as
VisualBasic to me. It is simple, easy, and boring! These are necessary not bad
characteristic, it is a useful language since it is used by many people, and
there is lots of software written for it.

>by definition and that cl-lib and pcase has ruined Emacs (both
>of these are very clean cut in general; in the Emacs jungle,
>they are a miracle of order).

Well, everyone is entitled to their opinion, aren't they? I don't think you can
do much about it. Not everyone has to agree about everything either. Some people
don't like pcase or cl-lib, so be it.

>With ELPA we have some modularity, we should move stuff from
>core Emacs to ELPA and, as said, get real modularity based on

As Drew use to write: +1 (for moving stuff out to Elpa).

>the package with real prefixes and - well, everything real!
>And, based on technology and not some wishful thinking
>a--human-convention can even reduce such huge disadvantages to
>other technology.
>
>So it is _really bad_, no excuses, had I known it was like
>this - I don't know what would have happened - anyway ... it
>is what it is.
>
>But if we get real modularity we get real libraries and if we
>do, there is hope.

As said in the previous mail, modularity would help, but let us not open another
big, off-topic discussion.

[-- Attachment #2: Type: text/html, Size: 12315 bytes --]

Re: as for Calc and the math library

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 1127 bytes --]

> > If Emacs is to have a math library, the library must use high-quality
> > mathematical and numerical algorithms that are well-known and
> > described in many textbooks on this subject matter...
>
> I agree with this point, but I suspect that writing such a library from
> scratch might be a bit difficult.
>
> May I suggest an alternative: abstract out GNU Calc math routines to be
> available from Elisp without having to read Calc sources.
> Such a task would require:
> 1. Documenting some Calc data structures
> 2. Possibly refactoring some Calc functions to use simpler API.
>
> It would be a much easier task and can improve the existing
> functionality.

Even better, given people proper FFI so they can import and use any of the tens
or hundreds of quality math libraries available.

That would also help to keep C core smaller, and relieve the dependency on core
devs to add bindings to future libraries when users want or need them.

A third of Emacs C core could have easily been implemented by users if they had
proper FFI exposed to Lisp instead of C modules which very few people use.

[-- Attachment #2: Type: text/html, Size: 3943 bytes --]

Re: as for Calc and the math library

[Eli Zaretskii]

> From: arthur miller <arthur.miller@live.com>
> Date: Mon, 12 Aug 2024 05:30:08 +0000
> 
> Even better, given people proper FFI so they can import and use any of the tens
> or hundreds of quality math libraries available.

Not going to happen, and you know it.

Re: as for Calc and the math library

[Nicolas Martyanoff]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: arthur miller <arthur.miller@live.com>
>> Date: Mon, 12 Aug 2024 05:30:08 +0000
>> 
>> Even better, given people proper FFI so they can import and use any of the tens
>> or hundreds of quality math libraries available.
>
> Not going to happen, and you know it.

Naïve question, why? I had this problem not so long ago because I wanted
to bind libpq, and I had to abandon the whole idea because dealing with
dynamic modules was way too inconvenient.

Having something similar to Common Lisp FFIs would make it trivial.

Best,

-- 
Nicolas Martyanoff
https://n16f.net
nicolas@n16f.net

Re: as for Calc and the math library

[Eli Zaretskii]

> From: Nicolas Martyanoff <nicolas@n16f.net>
> Cc: arthur miller <arthur.miller@live.com>,  emacs-devel@gnu.org
> Date: Mon, 12 Aug 2024 13:23:27 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: arthur miller <arthur.miller@live.com>
> >> Date: Mon, 12 Aug 2024 05:30:08 +0000
> >> 
> >> Even better, given people proper FFI so they can import and use any of the tens
> >> or hundreds of quality math libraries available.
> >
> > Not going to happen, and you know it.
> 
> Naïve question, why?

Because providing FFI would allow using non-free libraries in Emacs.

We made the effort to allow loading dynamic modules precisely for this
reason: to allow free libraries to be used, but not non-free ones.

Re: as for Calc and the math library

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Nicolas Martyanoff <nicolas@n16f.net>
>> Cc: arthur miller <arthur.miller@live.com>,  emacs-devel@gnu.org
>> Date: Mon, 12 Aug 2024 13:23:27 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: arthur miller <arthur.miller@live.com>
>> >> Date: Mon, 12 Aug 2024 05:30:08 +0000
>> >> 
>> >> Even better, given people proper FFI so they can import and use any of the tens
>> >> or hundreds of quality math libraries available.
>> >
>> > Not going to happen, and you know it.
>> 
>> Naïve question, why?
>
> Because providing FFI would allow using non-free libraries in Emacs.
>
> We made the effort to allow loading dynamic modules precisely for this
> reason: to allow free libraries to be used, but not non-free ones.

There is an item in etc/TODO about adding FFI, as an important feature:

  ** FFI (foreign function interface)
  See e.g. https://lists.gnu.org/r/emacs-devel/2013-10/msg00246.html

  One way of doing this is to start with fx's dynamic loading, and use it
  to implement things like auto-loaded buffer parsers and database
  access in cases which need more than Lisp.

Now I'm confused.

Re: as for Calc and the math library

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Nicolas Martyanoff <nicolas@n16f.net>
>>> Cc: arthur miller <arthur.miller@live.com>,  emacs-devel@gnu.org
>>> Date: Mon, 12 Aug 2024 13:23:27 +0200
>>> 
>>> Eli Zaretskii <eliz@gnu.org> writes:
>>> 
>>> >> From: arthur miller <arthur.miller@live.com>
>>> >> Date: Mon, 12 Aug 2024 05:30:08 +0000
>>> >> 
>>> >> Even better, given people proper FFI so they can import and use any of the tens
>>> >> or hundreds of quality math libraries available.
>>> >
>>> > Not going to happen, and you know it.
>>> 
>>> Naïve question, why?
>>
>> Because providing FFI would allow using non-free libraries in Emacs.
>>
>> We made the effort to allow loading dynamic modules precisely for this
>> reason: to allow free libraries to be used, but not non-free ones.
>
> There is an item in etc/TODO about adding FFI, as an important feature:
>
>   ** FFI (foreign function interface)
>   See e.g. https://lists.gnu.org/r/emacs-devel/2013-10/msg00246.html
>
>   One way of doing this is to start with fx's dynamic loading, and use it
>   to implement things like auto-loaded buffer parsers and database
>   access in cases which need more than Lisp.
>
> Now I'm confused.

I guess an FFI is already there? Or was there because it looks kind of
dead:

  https://github.com/tromey/emacs-ffi

Tom Tromey says in one of the issues

  https://github.com/tromey/emacs-ffi/issues/20

that John Wigley asked him on IRC to submit it for inclusion. Can't find
anything on emacs-devel though.

Re: as for Calc and the math library

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: Nicolas Martyanoff <nicolas@n16f.net>,  arthur.miller@live.com,
>   emacs-devel@gnu.org
> Date: Wed, 14 Aug 2024 06:11:00 +0200
> 
> I guess an FFI is already there? Or was there because it looks kind of
> dead:
> 
>   https://github.com/tromey/emacs-ffi
> 
> Tom Tromey says in one of the issues
> 
>   https://github.com/tromey/emacs-ffi/issues/20
> 
> that John Wigley asked him on IRC to submit it for inclusion. Can't find
> anything on emacs-devel though.

It doesn't ensure GPL-compliance, AFAICT, so in its current form it
cannot be accepted, sorry.  And I'm not sure I understand how can one
technically enforce GPL compliance in FFI-style loading of arbitrary
shared libraries.  The only idea that comes to mind is allow-list of
known libraries, the way we do in sqlite.c, but I'm not sure that
method is scalable to the basically infinite world of arbitrary
libraries.

Re: as for Calc and the math library

[Suhail Singh]

Eli Zaretskii <eliz@gnu.org> writes:

>> I guess an FFI is already there? Or was there because it looks kind of
>> dead:
>> 
>>   https://github.com/tromey/emacs-ffi
>> 
>> Tom Tromey says in one of the issues
>> 
>>   https://github.com/tromey/emacs-ffi/issues/20
>> 
>> that John Wigley asked him on IRC to submit it for inclusion. Can't find
>> anything on emacs-devel though.
>
> It doesn't ensure GPL-compliance, AFAICT, so in its current form it
> cannot be accepted, sorry. And I'm not sure I understand how can one
> technically enforce GPL compliance in FFI-style loading of arbitrary
> shared libraries.  The only idea that comes to mind is allow-list of
> known libraries, the way we do in sqlite.c, but I'm not sure that
> method is scalable to the basically infinite world of arbitrary
> libraries.

On the topic of what would be acceptable for an FFI, wouldn't something
akin to what's done for modules be sufficient ?  I.e., have the users of
the interface explicitly state that they are compliant.

It would scale better than an allow-list.  IIUC, Arthur mentioned this
in another thread.  If this wouldn't be sufficient for an FFI, could you
please elaborate on why that's the case ?

The question here isn't about what does or does not constitute GPL
compliance, but what Emacs, as a project, is trying to ensure when it
comes to such compliance.  In the case of modules, at least, a claim of
compliance is sufficient (without any additional verification).

-- 
Suhail

Re: as for Calc and the math library

[Eli Zaretskii]

> From: Suhail Singh <suhailsingh247@gmail.com>
> Cc: Gerd Möllmann <gerd.moellmann@gmail.com>,
>   nicolas@n16f.net,
>   arthur.miller@live.com,  emacs-devel@gnu.org
> Date: Wed, 14 Aug 2024 10:00:18 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > It doesn't ensure GPL-compliance, AFAICT, so in its current form it
> > cannot be accepted, sorry. And I'm not sure I understand how can one
> > technically enforce GPL compliance in FFI-style loading of arbitrary
> > shared libraries.  The only idea that comes to mind is allow-list of
> > known libraries, the way we do in sqlite.c, but I'm not sure that
> > method is scalable to the basically infinite world of arbitrary
> > libraries.
> 
> On the topic of what would be acceptable for an FFI, wouldn't something
> akin to what's done for modules be sufficient ?  I.e., have the users of
> the interface explicitly state that they are compliant.
> 
> It would scale better than an allow-list.  IIUC, Arthur mentioned this
> in another thread.  If this wouldn't be sufficient for an FFI, could you
> please elaborate on why that's the case ?

What exactly are you suggesting?  IOW, please describe what you have
in mind in more detail, because I don't think I understand it.

> The question here isn't about what does or does not constitute GPL
> compliance, but what Emacs, as a project, is trying to ensure when it
> comes to such compliance.

It tries to ensure that Emacs is not used as front-end for non-free
software.

Re: as for Calc and the math library

[Suhail Singh]

Eli Zaretskii <eliz@gnu.org> writes:

>> On the topic of what would be acceptable for an FFI, wouldn't something
>> akin to what's done for modules be sufficient ?  I.e., have the users of
>> the interface explicitly state that they are compliant.
>> 
>> It would scale better than an allow-list.  IIUC, Arthur mentioned this
>> in another thread.  If this wouldn't be sufficient for an FFI, could you
>> please elaborate on why that's the case ?
>
> What exactly are you suggesting?  IOW, please describe what you have
> in mind in more detail, because I don't think I understand it.

Specifically, modify the `define-ffi-library' macro that emacs-ffi
provides.

Presently, it takes two arguments: a SYMBOL and a NAME.  I am proposing
that it be updated to take three arguments: a SYMBOL, a NAME and a
GPL-COMPATIBLE-P token.  A value of `t' would be necessary for creating
a reference to the library.

If the value of `t' would be insufficient for Emacs' standards for
encouraging GPL compliance (in case it's considered "not explicit
enough"), the third argument could instead be something like
GPL-COMPATIBILITY where the value of 'library-is-GPL-compatible would be
necessary to be able to create a reference to the library.

A further variant/extension of the above proposal is also possible.
This extension modifies both `define-ffi-library' as well as
`define-ffi-function'.  When `define-ffi-library' gets an acceptable
claim of GPL compatibility, it attaches a property (which acts as a
witness for observing the claim of GPL compatibility) to the library
symbol.  The presence of said property is then checked by
`define-ffi-function'.  If the property is missing,
`define-ffi-function' complains.

-- 
Suhail

Re: as for Calc and the math library

[Eli Zaretskii]

> From: Suhail Singh <suhailsingh247@gmail.com>
> Cc: Suhail Singh <suhailsingh247@gmail.com>,  gerd.moellmann@gmail.com,
>  nicolas@n16f.net,  arthur.miller@live.com,  emacs-devel@gnu.org
> Date: Wed, 14 Aug 2024 11:08:02 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> On the topic of what would be acceptable for an FFI, wouldn't something
> >> akin to what's done for modules be sufficient ?  I.e., have the users of
> >> the interface explicitly state that they are compliant.
> >> 
> >> It would scale better than an allow-list.  IIUC, Arthur mentioned this
> >> in another thread.  If this wouldn't be sufficient for an FFI, could you
> >> please elaborate on why that's the case ?
> >
> > What exactly are you suggesting?  IOW, please describe what you have
> > in mind in more detail, because I don't think I understand it.
> 
> Specifically, modify the `define-ffi-library' macro that emacs-ffi
> provides.
> 
> Presently, it takes two arguments: a SYMBOL and a NAME.  I am proposing
> that it be updated to take three arguments: a SYMBOL, a NAME and a
> GPL-COMPATIBLE-P token.  A value of `t' would be necessary for creating
> a reference to the library.

And if the value is not t, then the load will fail?  If so, then this
additional argument makes very little sense: you could instead say
that just by loading the library, the Lisp program which uses
emacs-ffi "declares" the loaded library to be GPL-compatible.  And we
are back where we began.

The way we do it when loading modules requires the _loaded_ library to
declare itself compatible, by exporting a symbol of a certain name.
That is an action by the library we load, not by the Lisp program
which loads it.

Sv: as for Calc and the math library

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 4461 bytes --]

> > Eli Zaretskii <eliz@gnu.org> writes:
> >
> > >> On the topic of what would be acceptable for an FFI, wouldn't something
> > >> akin to what's done for modules be sufficient ?  I.e., have the users of
> > >> the interface explicitly state that they are compliant.
> > >>
> > >> It would scale better than an allow-list.  IIUC, Arthur mentioned this
> > >> in another thread.  If this wouldn't be sufficient for an FFI, could you
> > >> please elaborate on why that's the case ?
> > >
> > > What exactly are you suggesting?  IOW, please describe what you have
> > > in mind in more detail, because I don't think I understand it.
> >
> > Specifically, modify the `define-ffi-library' macro that emacs-ffi
> > provides.
> >
> > Presently, it takes two arguments: a SYMBOL and a NAME.  I am proposing
> > that it be updated to take three arguments: a SYMBOL, a NAME and a
> > GPL-COMPATIBLE-P token.  A value of `t' would be necessary for creating
> > a reference to the library.
>
> And if the value is not t, then the load will fail?  If so, then this
> additional argument makes very little sense: you could instead say

It makes as much sense as it makes in C library. The token is basically an
agreement between Emacs developers and the user not to load (link) closed
source libraries into Emacs.

> that just by loading the library, the Lisp program which uses
> emacs-ffi "declares" the loaded library to be GPL-compatible.  And we
> are back where we began.

Yes, you could. It would just completely remove the barrier. However,
the token is an explicit acknowledgment of Emacs policy and license terms, by
the person who loads the library into Emacs.

> The way we do it when loading modules requires the _loaded_ library to
> declare itself compatible, by exporting a symbol of a certain name.
> That is an action by the library we load, not by the Lisp program
> which loads it.

True. But as you said yourself, a malicious user can easily cicrumvent it, even
in C and there is nothing we can do to prevent them other than possibly taking
legal actions against them.

If some company or a state would use Emacs or any other GNU program, as a
front-end to closed-source software, there is very little one can do
technically. It is only the license and the agreement that actually protects it.
________________________________
Från: Eli Zaretskii <eliz@gnu.org>
Skickat: den 14 augusti 2024 17:31
Till: Suhail Singh <suhailsingh247@gmail.com>
Kopia: gerd.moellmann@gmail.com <gerd.moellmann@gmail.com>; nicolas@n16f.net <nicolas@n16f.net>; arthur.miller@live.com <arthur.miller@live.com>; emacs-devel@gnu.org <emacs-devel@gnu.org>
Ämne: Re: as for Calc and the math library

> From: Suhail Singh <suhailsingh247@gmail.com>
> Cc: Suhail Singh <suhailsingh247@gmail.com>,  gerd.moellmann@gmail.com,
>  nicolas@n16f.net,  arthur.miller@live.com,  emacs-devel@gnu.org
> Date: Wed, 14 Aug 2024 11:08:02 -0400
>
> Eli Zaretskii <eliz@gnu.org> writes:
>
> >> On the topic of what would be acceptable for an FFI, wouldn't something
> >> akin to what's done for modules be sufficient ?  I.e., have the users of
> >> the interface explicitly state that they are compliant.
> >>
> >> It would scale better than an allow-list.  IIUC, Arthur mentioned this
> >> in another thread.  If this wouldn't be sufficient for an FFI, could you
> >> please elaborate on why that's the case ?
> >
> > What exactly are you suggesting?  IOW, please describe what you have
> > in mind in more detail, because I don't think I understand it.
>
> Specifically, modify the `define-ffi-library' macro that emacs-ffi
> provides.
>
> Presently, it takes two arguments: a SYMBOL and a NAME.  I am proposing
> that it be updated to take three arguments: a SYMBOL, a NAME and a
> GPL-COMPATIBLE-P token.  A value of `t' would be necessary for creating
> a reference to the library.

And if the value is not t, then the load will fail?  If so, then this
additional argument makes very little sense: you could instead say
that just by loading the library, the Lisp program which uses
emacs-ffi "declares" the loaded library to be GPL-compatible.  And we
are back where we began.

The way we do it when loading modules requires the _loaded_ library to
declare itself compatible, by exporting a symbol of a certain name.
That is an action by the library we load, not by the Lisp program
which loads it.

[-- Attachment #2: Type: text/html, Size: 10787 bytes --]

Re: as for Calc and the math library

[Eli Zaretskii]

> From: arthur miller <arthur.miller@live.com>
> CC: "gerd.moellmann@gmail.com" <gerd.moellmann@gmail.com>, "nicolas@n16f.net"
> 	<nicolas@n16f.net>, "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> Date: Thu, 15 Aug 2024 05:00:06 +0000
> 
> > > Specifically, modify the `define-ffi-library' macro that emacs-ffi
> > > provides.
> > >
> > > Presently, it takes two arguments: a SYMBOL and a NAME.  I am proposing
> > > that it be updated to take three arguments: a SYMBOL, a NAME and a
> > > GPL-COMPATIBLE-P token.  A value of `t' would be necessary for creating
> > > a reference to the library.
> >
> > And if the value is not t, then the load will fail?  If so, then this
> > additional argument makes very little sense: you could instead say
> 
> It makes as much sense as it makes in C library. The token is basically an
> agreement between Emacs developers and the user not to load (link) closed
> source libraries into Emacs.

No.  What we need is the declaration _by_the_library_ being loaded
that it complies.  It is not an agreement between Emacs and the
library, it's a _requirement_ on our part that only the library itself
can fulfill.

> > that just by loading the library, the Lisp program which uses
> > emacs-ffi "declares" the loaded library to be GPL-compatible.  And we
> > are back where we began.
> 
> Yes, you could. It would just completely remove the barrier. However,
> the token is an explicit acknowledgment of Emacs policy and license terms, by
> the person who loads the library into Emacs.

No, see above.

> > The way we do it when loading modules requires the _loaded_ library to
> > declare itself compatible, by exporting a symbol of a certain name.
> > That is an action by the library we load, not by the Lisp program
> > which loads it.
> 
> True. But as you said yourself, a malicious user can easily cicrumvent it, even
> in C and there is nothing we can do to prevent them other than possibly taking
> legal actions against them.

Malicious agents can do all kinds of bad and even unlawful things, but
we are not under any obligations to cater to them.  They can always
change the Emacs source code to do whatever they like, but why do you
expect _us_ to do it for them, or make it easier for them to do it?

> If some company or a state would use Emacs or any other GNU program, as a
> front-end to closed-source software, there is very little one can do
> technically. It is only the license and the agreement that actually protects it.

That's a separate issue, although it could be related.  What is at
stake here is the fact that we don't want to encourage use of Emacs as
such a front-end.  That policy of the project is not new, it has many
different expressions, as already mentioned in this thread.  We don't
mention non-free software or fonts in our documentation, we don't
install changes that specifically favor or are designed to support
non-free programs, we don't link against non-free libraries except if
they are system libraries, we don't install functionality enhancements
on non-free systems if there's no equivalent on free systems, etc.
etc.  So I wonder what is this particular fuss all about.  Its
only effect is to waste maintainers' time and energy on replying to
these posts, and I very much doubt this was the intent.  So could we
please stop this Nth instance of the same discussions?  I cannot
change the project policies even if I wanted, because I promised to
uphold them when Richard nominated me.

Sv: as for Calc and the math library

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 9547 bytes --]

>> > > Specifically, modify the `define-ffi-library' macro that emacs-ffi
>> > > provides.
>> > >
>> > > Presently, it takes two arguments: a SYMBOL and a NAME.  I am proposing
>> > > that it be updated to take three arguments: a SYMBOL, a NAME and a
>> > > GPL-COMPATIBLE-P token.  A value of `t' would be necessary for creating
>> > > a reference to the library.
>> >
>> > And if the value is not t, then the load will fail?  If so, then this
>> > additional argument makes very little sense: you could instead say
>>
>> It makes as much sense as it makes in C library. The token is basically an
>> agreement between Emacs developers and the user not to load (link) closed
>> source libraries into Emacs.
>
>No.  What we need is the declaration _by_the_library_ being loaded
>that it complies.  It is not an agreement between Emacs and the
>library, it's a _requirement_ on our part that only the library itself
>can fulfill.

I have seen your previous answers to others, and I do understand the point you
make, but I think the argument fails because a module can be just a proxy to a
non-free library. If I can put it in other words, I wonder if it is a "false
sense of security", because it is a guarantee by the person who writes the C
module, not the 3rd party library they link to. Of course they are violating the
license if they link to a non-free library, but that we are agreeing about we
can't prevent.

>> > that just by loading the library, the Lisp program which uses
>> > emacs-ffi "declares" the loaded library to be GPL-compatible.  And we
>> > are back where we began.
>>
>> Yes, you could. It would just completely remove the barrier. However,
>> the token is an explicit acknowledgment of Emacs policy and license terms, by
>> the person who loads the library into Emacs.
>
>No, see above.
>
>> > The way we do it when loading modules requires the _loaded_ library to
>> > declare itself compatible, by exporting a symbol of a certain name.
>> > That is an action by the library we load, not by the Lisp program
>> > which loads it.
>>
>> True. But as you said yourself, a malicious user can easily cicrumvent it,
>> even
>> in C and there is nothing we can do to prevent them other than possibly taking
>> legal actions against them.
>
>Malicious agents can do all kinds of bad and even unlawful things, but
>we are not under any obligations to cater to them.  They can always
>change the Emacs source code to do whatever they like, but why do you

Exactly.

>expect _us_ to do it for them, or make it easier for them to do it?

No, of course I don't expect that from you. My point was (is?) that it is a
trade-off between convinience for Emacs users who would use it as intendended,
only with free software, vs the inconvience it causes.

If you look at some other projects which are more interesting than Emacs to use
as front-ends for non-free software, do we see influx of companies misusing
them? I am thinking of Gtk, KDE/Gnome libs and Libre/OpenOffice for example. All
of those projects could be used to plug-in proprietary software, yet, there are
not so many cases, if any? Even Qt which comes as a dual licensed does not seem
to be misused that way. Perhaps I am just uninformed?

You have mentioned llvm, but I am not familiar enough with them. As I understand
they are financied by Apple, Nvidia and few other big tech companies with
explicit goal to have an alternative to GCC just for the more permissive
licensing.

>> If some company or a state would use Emacs or any other GNU program, as a
>> front-end to closed-source software, there is very little one can do
>> technically. It is only the license and the agreement that actually protects
>> it.
>
>That's a separate issue, although it could be related.  What is at
>stake here is the fact that we don't want to encourage use of Emacs as
>such a front-end.  That policy of the project is not new, it has many
>different expressions, as already mentioned in this thread.  We don't
>mention non-free software or fonts in our documentation, we don't
>install changes that specifically favor or are designed to support
>non-free programs, we don't link against non-free libraries except if
>they are system libraries, we don't install functionality enhancements
>on non-free systems if there's no equivalent on free systems, etc.
>etc.  So I wonder what is this particular fuss all about.  Its

I said it in the beginning: sometimes it is useful to revisit policies
when circumstancies have changed.

>only effect is to waste maintainers' time and energy on replying to
>these posts, and I very much doubt this was the intent.  So could we

Have I ever trolled you before? Why would I want to waste your time and energy?
I am certainly the last person in the world to do that, and you should know that.
I understand it takes time and energy, and I appologize for that, but it is
useful to learn why you consider FFI unnaceptable, and to hear the arguments
from the first hand. For the record: I have never brought up this issue before.

>please stop this Nth instance of the same discussions?  I cannot
>change the project policies even if I wanted, because I promised to
>uphold them when Richard nominated me.

I wouldn't expect you to change policy on your own neither. I did hope for RMS,
you and other experienced developers to perhaps take a look at the history,
perhaps at other software and how Emacs development could be, and reflect over
the policy if it is still worth keeping or perhaps it can be changed.
________________________________
Från: Eli Zaretskii <eliz@gnu.org>
Skickat: den 15 augusti 2024 09:02
Till: arthur miller <arthur.miller@live.com>
Kopia: suhailsingh247@gmail.com <suhailsingh247@gmail.com>; gerd.moellmann@gmail.com <gerd.moellmann@gmail.com>; nicolas@n16f.net <nicolas@n16f.net>; emacs-devel@gnu.org <emacs-devel@gnu.org>
Ämne: Re: as for Calc and the math library

> From: arthur miller <arthur.miller@live.com>
> CC: "gerd.moellmann@gmail.com" <gerd.moellmann@gmail.com>, "nicolas@n16f.net"
>        <nicolas@n16f.net>, "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> Date: Thu, 15 Aug 2024 05:00:06 +0000
>
> > > Specifically, modify the `define-ffi-library' macro that emacs-ffi
> > > provides.
> > >
> > > Presently, it takes two arguments: a SYMBOL and a NAME.  I am proposing
> > > that it be updated to take three arguments: a SYMBOL, a NAME and a
> > > GPL-COMPATIBLE-P token.  A value of `t' would be necessary for creating
> > > a reference to the library.
> >
> > And if the value is not t, then the load will fail?  If so, then this
> > additional argument makes very little sense: you could instead say
>
> It makes as much sense as it makes in C library. The token is basically an
> agreement between Emacs developers and the user not to load (link) closed
> source libraries into Emacs.

No.  What we need is the declaration _by_the_library_ being loaded
that it complies.  It is not an agreement between Emacs and the
library, it's a _requirement_ on our part that only the library itself
can fulfill.

> > that just by loading the library, the Lisp program which uses
> > emacs-ffi "declares" the loaded library to be GPL-compatible.  And we
> > are back where we began.
>
> Yes, you could. It would just completely remove the barrier. However,
> the token is an explicit acknowledgment of Emacs policy and license terms, by
> the person who loads the library into Emacs.

No, see above.

> > The way we do it when loading modules requires the _loaded_ library to
> > declare itself compatible, by exporting a symbol of a certain name.
> > That is an action by the library we load, not by the Lisp program
> > which loads it.
>
> True. But as you said yourself, a malicious user can easily cicrumvent it, even
> in C and there is nothing we can do to prevent them other than possibly taking
> legal actions against them.

Malicious agents can do all kinds of bad and even unlawful things, but
we are not under any obligations to cater to them.  They can always
change the Emacs source code to do whatever they like, but why do you
expect _us_ to do it for them, or make it easier for them to do it?

> If some company or a state would use Emacs or any other GNU program, as a
> front-end to closed-source software, there is very little one can do
> technically. It is only the license and the agreement that actually protects it.

That's a separate issue, although it could be related.  What is at
stake here is the fact that we don't want to encourage use of Emacs as
such a front-end.  That policy of the project is not new, it has many
different expressions, as already mentioned in this thread.  We don't
mention non-free software or fonts in our documentation, we don't
install changes that specifically favor or are designed to support
non-free programs, we don't link against non-free libraries except if
they are system libraries, we don't install functionality enhancements
on non-free systems if there's no equivalent on free systems, etc.
etc.  So I wonder what is this particular fuss all about.  Its
only effect is to waste maintainers' time and energy on replying to
these posts, and I very much doubt this was the intent.  So could we
please stop this Nth instance of the same discussions?  I cannot
change the project policies even if I wanted, because I promised to
uphold them when Richard nominated me.

[-- Attachment #2: Type: text/html, Size: 22892 bytes --]

we need *modularity* [last problem] (was: Re: as for Calc and the math library)

[Emanuel Berg]

arthur miller wrote:

> I have seen your previous answers to others, and I do
> understand the point you make, but I think the argument
> fails because a module can be just a proxy to
> a non-free library.

Emacs is notoriously not modular. Everything is everywhere,
you seem to have everything but if fact you have nothing.
There is so much stuff you can't find the basics stuff that
isn't there.

Here take a look: what does this name say?

  cl--defalias

Ah, that's a local function in the Common Lisp package, right?

Well, YES, but ...

It is actually an Elisp defun. And that prefix isn't really
a package qualifier, it is just a convention, part of
the name. And the double dash, likewise just a convention, we
don't have package modularity or any of that kind of data
hiding, abstraction, encapsulation ... what is it more called?
Well, modularity!

In Emacs everything is intermixed, Elisp-Emacs,
non-interactive/interactive, data/data processing/the
interface (buffer, buffer, and buffer), what is a file, what
is a package, what is a library, and what is a program - no
one can tell.

Emacs Lisp isn't is fun, but absolutely not any kind
of powerhouse. It is inconsistent in itself and on top of that
all those zillion interfaces, cl-, seq-, slime- (yes, had to
use that today, for `slime-transpose-lists').

It is pretty fast to get something going but complexity
skyrockets.

What we should do, in general: modularity! clear divisions, in
particular, libraries for everything, including those that provide basic
stuff in a complete and consistent way.

We should get real package modularity, transform the prefixes
into real package qualifiers, and throw away the ugly
local ones.

With Elisp, optimize everything for development speed and
convenience, have consistent interfaces as much as possible,
acknowledge that Elisp was underpowered, which is why all the
interfaces came, now, core Elisp and the interfaces should
make peace, one should identify what it is exactly, what are
the to 10 things that people use with cl-, pcase-, seq- etc,
and bring them into a new core Elisp, and the few CL experts
can still use all the rest.

Now, with native compilation we have speed - we have power
with the interfaces if one could harvest it, which one
currently - well, it is very slow and difficult, anyway - the
very, very last spet into maturity and getting out of the
Elisp ghetto (and getting visitors!) are modularity, modern
software principles - by modularity I mean real modularity,
based on technology - not silly conventions - I forgot where
I was - the last step is modularity in general, and libraries
in particular.

Clear cut interfaces. Get away with the ugly prefixes and
error--prone conventions that don't even do anything.

Elisp 3.0! \o/

PS. This is the problem. This series is completed, I said
    I would write 10, I wrote 4. Here are the other parts.
    It is all based on the same theme. Modularity,
    consistency, libraries, modern software practises.
    
    https://lists.gnu.org/archive/html/emacs-devel/2024-08/msg00154.html
    https://lists.gnu.org/archive/html/emacs-devel/2024-08/msg00380.html
    https://lists.gnu.org/archive/html/emacs-devel/2024-08/msg00388.html

PPS. So who is working on real modularity based on the package
     with real public and local functions as we speak? I know
     there are always some guy - at least - working on such
     a basic idea. I know some way. And I know some day.

-- 
underground experts united
https://dataswamp.org/~incal