Re: as for Calc and the math library

Re: as for Calc and the math library

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 1127 bytes --]

> > If Emacs is to have a math library, the library must use high-quality
> > mathematical and numerical algorithms that are well-known and
> > described in many textbooks on this subject matter...
>
> I agree with this point, but I suspect that writing such a library from
> scratch might be a bit difficult.
>
> May I suggest an alternative: abstract out GNU Calc math routines to be
> available from Elisp without having to read Calc sources.
> Such a task would require:
> 1. Documenting some Calc data structures
> 2. Possibly refactoring some Calc functions to use simpler API.
>
> It would be a much easier task and can improve the existing
> functionality.

Even better, given people proper FFI so they can import and use any of the tens
or hundreds of quality math libraries available.

That would also help to keep C core smaller, and relieve the dependency on core
devs to add bindings to future libraries when users want or need them.

A third of Emacs C core could have easily been implemented by users if they had
proper FFI exposed to Lisp instead of C modules which very few people use.

[-- Attachment #2: Type: text/html, Size: 3943 bytes --]

Re: as for Calc and the math library

[Eli Zaretskii]

> From: arthur miller <arthur.miller@live.com>
> Date: Mon, 12 Aug 2024 05:30:08 +0000
> 
> Even better, given people proper FFI so they can import and use any of the tens
> or hundreds of quality math libraries available.

Not going to happen, and you know it.

Re: as for Calc and the math library

[Nicolas Martyanoff]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: arthur miller <arthur.miller@live.com>
>> Date: Mon, 12 Aug 2024 05:30:08 +0000
>> 
>> Even better, given people proper FFI so they can import and use any of the tens
>> or hundreds of quality math libraries available.
>
> Not going to happen, and you know it.

Naïve question, why? I had this problem not so long ago because I wanted
to bind libpq, and I had to abandon the whole idea because dealing with
dynamic modules was way too inconvenient.

Having something similar to Common Lisp FFIs would make it trivial.

Best,

-- 
Nicolas Martyanoff
https://n16f.net
nicolas@n16f.net

Re: as for Calc and the math library

[Eli Zaretskii]

> From: Nicolas Martyanoff <nicolas@n16f.net>
> Cc: arthur miller <arthur.miller@live.com>,  emacs-devel@gnu.org
> Date: Mon, 12 Aug 2024 13:23:27 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: arthur miller <arthur.miller@live.com>
> >> Date: Mon, 12 Aug 2024 05:30:08 +0000
> >> 
> >> Even better, given people proper FFI so they can import and use any of the tens
> >> or hundreds of quality math libraries available.
> >
> > Not going to happen, and you know it.
> 
> Naïve question, why?

Because providing FFI would allow using non-free libraries in Emacs.

We made the effort to allow loading dynamic modules precisely for this
reason: to allow free libraries to be used, but not non-free ones.

Re: as for Calc and the math library

[Nicolas Martyanoff]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Nicolas Martyanoff <nicolas@n16f.net>
>> Cc: arthur miller <arthur.miller@live.com>,  emacs-devel@gnu.org
>> Date: Mon, 12 Aug 2024 13:23:27 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: arthur miller <arthur.miller@live.com>
>> >> Date: Mon, 12 Aug 2024 05:30:08 +0000
>> >> 
>> >> Even better, given people proper FFI so they can import and use any of the tens
>> >> or hundreds of quality math libraries available.
>> >
>> > Not going to happen, and you know it.
>> 
>> Naïve question, why?
>
> Because providing FFI would allow using non-free libraries in Emacs.
>
> We made the effort to allow loading dynamic modules precisely for this
> reason: to allow free libraries to be used, but not non-free ones.

You absolutely can use non-free dynamic modules: enforcing the existence
of a symbol saying "I promise I'm free" does not change anything. One
could also easily add a free dynamic module that calls non-free
libraries.

It is sad to see Emacs being hamstrung and users being limited because
someone could do something that would have no impact whatsoever with the
project or any other user.

But your software your rules.

-- 
Nicolas Martyanoff
https://n16f.net
nicolas@n16f.net

Re: as for Calc and the math library

[Eli Zaretskii]

> From: Nicolas Martyanoff <nicolas@n16f.net>
> Cc: Nicolas Martyanoff <nicolas@n16f.net>,  arthur.miller@live.com,
>   emacs-devel@gnu.org
> Date: Mon, 12 Aug 2024 14:11:07 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > We made the effort to allow loading dynamic modules precisely for this
> > reason: to allow free libraries to be used, but not non-free ones.
> 
> You absolutely can use non-free dynamic modules: enforcing the existence
> of a symbol saying "I promise I'm free" does not change anything. One
> could also easily add a free dynamic module that calls non-free
> libraries.

The need to declare that a library is free and have its sources freely
available does serve as an obstacle for non-free software.  And using
non-free library with a free module is against the GPL, so it is
illegal.  We cannot prevent people from lying and doing illegal
things, we can only make it harder.

> It is sad to see Emacs being hamstrung and users being limited because
> someone could do something that would have no impact whatsoever with the
> project or any other user.

This is a sad world for more reasons than one, but discussing that is
off-topic here.

as for Calc and the math library

[Christopher Dimech]

----- Kristof Đymek
Administrator General - Chilkat Design Build - Naiad Informatics - Gnu Project

Society has become too quick to pass judgement and declare someone
Persona Non-Grata, the most extreme form of censure a country can
bestow.

In a new era of destructive authoritarianism, I support Richard
Stallman.  Times of great crisis are also times of great
opportunity.  I call upon you to make this struggle yours as well !

https://www.gnu.org     https://www.fsf.org/


> Sent: Tuesday, August 13, 2024 at 1:22 AM
> From: "Eli Zaretskii" <eliz@gnu.org>
> To: "Nicolas Martyanoff" <nicolas@n16f.net>
> Cc: arthur.miller@live.com, emacs-devel@gnu.org
> Subject: Re: as for Calc and the math library
>
> > From: Nicolas Martyanoff <nicolas@n16f.net>
> > Cc: Nicolas Martyanoff <nicolas@n16f.net>,  arthur.miller@live.com,
> >   emacs-devel@gnu.org
> > Date: Mon, 12 Aug 2024 14:11:07 +0200
> > 
> > Eli Zaretskii <eliz@gnu.org> writes:
> > 
> > > We made the effort to allow loading dynamic modules precisely for this
> > > reason: to allow free libraries to be used, but not non-free ones.
> > 
> > You absolutely can use non-free dynamic modules: enforcing the existence
> > of a symbol saying "I promise I'm free" does not change anything. One
> > could also easily add a free dynamic module that calls non-free
> > libraries.
> 
> The need to declare that a library is free and have its sources freely
> available does serve as an obstacle for non-free software.  And using
> non-free library with a free module is against the GPL, so it is
> illegal.  We cannot prevent people from lying and doing illegal
> things, we can only make it harder.

License terms do not prohibit the software's use, such a restriction would 
not be enforceable. 
 
> > It is sad to see Emacs being hamstrung and users being limited because
> > someone could do something that would have no impact whatsoever with the
> > project or any other user.
> 
> This is a sad world for more reasons than one, but discussing that is
> off-topic here.
 
What you can do is remove non-free software from free-software tools.
You are wasting time for no real returns.

Re: as for Calc and the math library

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

GPL3 explicitly does not place rules on what changes you can make
in a covered progra _for your private use_.

A user can lawfully write a nonfree program for private use that
dynamically links in a GPL3-covered module, just as a user can
lawfully copy GPL3-covered code plus nonfree code into a program for
private use.

However, _distribution_ of such a nonfree combination would violate
GPLv3.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

as for Calc and the math library

[Christopher Dimech]

> Sent: Thursday, August 15, 2024 at 1:59 PM
> From: "Richard Stallman" <rms@gnu.org>
> To: "Christopher Dimech" <dimech@gmx.com>
> Cc: emacs-devel@gnu.org
> Subject: Re: as for Calc and the math library
>
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
> GPL3 explicitly does not place rules on what changes you can make
> in a covered progra _for your private use_.
> 
> A user can lawfully write a nonfree program for private use that
> dynamically links in a GPL3-covered module, just as a user can
> lawfully copy GPL3-covered code plus nonfree code into a program for
> private use.
> 
> However, _distribution_ of such a nonfree combination would violate
> GPLv3.
> 
> -- 
> Dr Richard Stallman (https://stallman.org)
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)

Despite this clarity, there's a recurring problem with the community:
some maintainers go beyond the requirements of the GPL3 by creating
additional barriers within the source code to prevent what they perceive
as undesirable use cases.

This could involve adding checks to verify that only certain libraries
are used or implementing "allowed-lists" to restrict what modifications
can be made or with what the software can interact.

It's quite right to argue that these additional obstacles are
unnecessary and, in some cases, counterproductive. The GPL3 already sets
clear boundaries — if a nonfree module or library is being used in a way
that is kept private and not distributed, it is perfectly legal under
the license. The problem arises when maintainers attempt to enforce
additional restrictions that go beyond the license’s requirements,
sometimes under the misconception that doing so aligns with the GPL's
spirit.

For instance, claims that using a nonfree library with a GPL3-covered
module is inherently illegal reflect a misunderstanding of the
license.  As clarified, such usage is only problematic if the
software is distributed. Unfortunately, these misunderstandings can lead
to a unpleasant environment where maintainers unnecessarily police the
actions of users or other developers, potentially stifling innovation
and cooperation.

Moreover, some maintainers might believe they are in a better position
to judge what is or isn't permissible under the GPL3. While it’s true
that interpreting the nuances of free-licenses requires a good
understanding, dismissing the judgments of others, such as those with
significant experience or legal knowledge, is unproductive. The GPL3 was
crafted to empower users, not to restrict them unnecessarily, and the
community should focus on upholding that ethos rather than imposing
additional, unwarranted barriers.

Re: as for Calc and the math library

[Eli Zaretskii]

> From: Christopher Dimech <dimech@gmx.com>
> Cc: emacs-devel@gnu.org
> Date: Thu, 15 Aug 2024 05:06:47 +0200
> 
> > However, _distribution_ of such a nonfree combination would violate
> > GPLv3.
> 
> Despite this clarity, there's a recurring problem with the community:
> some maintainers go beyond the requirements of the GPL3 by creating
> additional barriers within the source code to prevent what they perceive
> as undesirable use cases.
> 
> This could involve adding checks to verify that only certain libraries
> are used or implementing "allowed-lists" to restrict what modifications
> can be made or with what the software can interact.
> 
> It's quite right to argue that these additional obstacles are
> unnecessary and, in some cases, counterproductive. The GPL3 already sets
> clear boundaries — if a nonfree module or library is being used in a way
> that is kept private and not distributed, it is perfectly legal under
> the license. The problem arises when maintainers attempt to enforce
> additional restrictions that go beyond the license’s requirements,
> sometimes under the misconception that doing so aligns with the GPL's
> spirit.

The maintainers only add these obstacles when there's a request for
Emacs as a project to distribute code which would allow making Emacs a
front-end for non-free software.  As long as such code is used
privately by someone, or even left on some repository outside of
Emacs, it is not our business (although distributing non-compliant
software which claims to be compliant is against the law).  But please
understand that you cannot request _us_ to include such code in Emacs,
because then _we_ will be either violating the GPL or encouraging use
of non-free software.

And please don't forget or ignore that in addition to GPL violations,
there's one more aspect involved here, and that is not to encourage
use of non-free software.  For the same reason we don't mention
non-free programs or libraries or fonts in our sources and
documentation, we do not intend to make it easy for people to use
non-free shared libraries by providing _our_ code that caters to such
use cases.  Emacs is Free Software, so anyone can take its sources and
modify them to do whatever they want, but don't expect _us_ to do that
as part of the official Emacs sources.  This is not new, although some
people tend to raise the same issues here time and again for some
reason.

> For instance, claims that using a nonfree library with a GPL3-covered
> module is inherently illegal reflect a misunderstanding of the
> license.

It would be illegal for a library to claim GPL compliance when in fact
there's no compliance, yes.  Other than that, no one said anything
about the legal aspects; the issues discussed in this thread are our
usual ethics that precludes us from encouraging use of non-free
software in conjunction with Emacs.

> As clarified, such usage is only problematic if the
> software is distributed. Unfortunately, these misunderstandings can lead
> to a unpleasant environment where maintainers unnecessarily police the
> actions of users or other developers, potentially stifling innovation
> and cooperation.

It could, but it doesn't, not in this case.

> Moreover, some maintainers might believe they are in a better position
> to judge what is or isn't permissible under the GPL3.

This is a strawman: no one said anything about GPL; the fact that the
symbol required by loading dynamic modules has "GPL" in its name does
not contradict this, because the requirement is to be GPL-compatible,
not GPLv3.

as for Calc and the math library

[Christopher Dimech]

> Sent: Thursday, August 15, 2024 at 6:43 PM
> From: "Eli Zaretskii" <eliz@gnu.org>
> To: "Christopher Dimech" <dimech@gmx.com>
> Cc: rms@gnu.org, emacs-devel@gnu.org
> Subject: Re: as for Calc and the math library
>
> > From: Christopher Dimech <dimech@gmx.com>
> > Cc: emacs-devel@gnu.org
> > Date: Thu, 15 Aug 2024 05:06:47 +0200
> > 
> > > However, _distribution_ of such a nonfree combination would violate
> > > GPLv3.
> > 
> > Despite this clarity, there's a recurring problem with the community:
> > some maintainers go beyond the requirements of the GPL3 by creating
> > additional barriers within the source code to prevent what they perceive
> > as undesirable use cases.
> > 
> > This could involve adding checks to verify that only certain libraries
> > are used or implementing "allowed-lists" to restrict what modifications
> > can be made or with what the software can interact.
> > 
> > It's quite right to argue that these additional obstacles are
> > unnecessary and, in some cases, counterproductive. The GPL3 already sets
> > clear boundaries — if a nonfree module or library is being used in a way
> > that is kept private and not distributed, it is perfectly legal under
> > the license. The problem arises when maintainers attempt to enforce
> > additional restrictions that go beyond the license’s requirements,
> > sometimes under the misconception that doing so aligns with the GPL's
> > spirit.
> 
> The maintainers only add these obstacles when there's a request for
> Emacs as a project to distribute code which would allow making Emacs a
> front-end for non-free software.  As long as such code is used
> privately by someone, or even left on some repository outside of
> Emacs, it is not our business (although distributing non-compliant
> software which claims to be compliant is against the law).  But please
> understand that you cannot request _us_ to include such code in Emacs,
> because then _we_ will be either violating the GPL or encouraging use
> of non-free software.

> And please don't forget or ignore that in addition to GPL violations,
> there's one more aspect involved here, and that is not to encourage
> use of non-free software.  For the same reason we don't mention
> non-free programs or libraries or fonts in our sources and
> documentation, we do not intend to make it easy for people to use
> non-free shared libraries by providing _our_ code that caters to such
> use cases.  

The focus in free software is on creating clear, understandable, and
accessible code. The core principle of free software is that users have
the freedom to study, modify, and distribute the software as they see
fit. This freedom is best supported by writing clear and well-documented
code that users can easily understand and adapt to their needs.

Encouraging free software adoption through software design or structural
means — such as intentionally complicating the code to guide or limit how
users interact with it — is problematic. This approach runs counter to
the spirit of free software because it resembles the kind of code
obfuscation often seen in proprietary software, where the intent is to
prevent users from understanding or modifying the code.

In free software, the goal should be to empower users, not restrict
them. Any attempt to influence how users engage with the software
through code structure, rather than through education or advocacy,
undermines the principles of transparency and user freedom that are
central to the free software movement. The power of free software lies
in its openness, and that openness should be reflected in the clarity
and accessibility of the code itself.


> Emacs is Free Software, so anyone can take its sources and
> modify them to do whatever they want, but don't expect _us_ to do that
> as part of the official Emacs sources.  This is not new, although some
> people tend to raise the same issues here time and again for some
> reason.
> 
> > For instance, claims that using a nonfree library with a GPL3-covered
> > module is inherently illegal reflect a misunderstanding of the
> > license.
> 
> It would be illegal for a library to claim GPL compliance when in fact
> there's no compliance, yes.  Other than that, no one said anything
> about the legal aspects; the issues discussed in this thread are our
> usual ethics that precludes us from encouraging use of non-free
> software in conjunction with Emacs.

Emacs, as a project, must always ensure that its releases are fully 
compliant with the GPL, which governs its distribution and use.  

There is ambiguity in determining software freedom.  One cannot always
determine a priori whether external software is free or non-free without
examining its license. Imposing restrictions or enforcing certain ways
of doing things to ensure compliance with the free software ethos is an
overreaching approach. Forcing a particular method of interaction and
then claiming that this makes the software free misses the point that
freedom is inherent in the license itself, not in the prescribed usage.
 
> > As clarified, such usage is only problematic if the
> > software is distributed. Unfortunately, these misunderstandings can lead
> > to a unpleasant environment where maintainers unnecessarily police the
> > actions of users or other developers, potentially stifling innovation
> > and cooperation.
> 
> It could, but it doesn't, not in this case.
> 
> > Moreover, some maintainers might believe they are in a better position
> > to judge what is or isn't permissible under the GPL3.
> 
> This is a strawman: no one said anything about GPL; the fact that the
> symbol required by loading dynamic modules has "GPL" in its name does
> not contradict this, because the requirement is to be GPL-compatible,
> not GPLv3.

The statement you’re referring to touches on an important aspect of the
free software ecosystem - namely, that the GPL (GNU General Public
License) is just one of many licenses under which free software can be
distributed. The idea that software needs to be GPL-compliant is indeed
a common misconception, but it's more accurate to say that it needs to
be compatible with the GPL, especially when interacting with GPL-licensed 
code.  

But only when distributed.  For software that is not distributed (e.g.,
used privately or within an organization), these requirements do not
apply.  Suggesting or enforcing limitations on what users can code, even
in private use is where the controversy lies.  The GPL, as stated, does
not place restrictions on private modifications or use. Users are free
to experiment, modify, and even create non-free software for their
private use, without any obligation to comply with the GPL’s
distribution requirements.  But you decided to hinder that possibility.
An additional item to the list of bad ideas.

Re: as for Calc and the math library

[Eli Zaretskii]

> From: Christopher Dimech <dimech@gmx.com>
> Cc: rms@gnu.org, emacs-devel@gnu.org
> Date: Thu, 15 Aug 2024 15:28:24 +0200
> 
> The focus in free software is on creating clear, understandable, and
> accessible code.

That's _part_ of our focus here.  Another part, and definitely no less
important, is to make sure the software is free.

> The core principle of free software

Just one of the core principles, not the only one.

> is that users have the freedom to study, modify, and distribute the
> software as they see fit.

Not "as they see fit": there are some restrictions on what they can
do.  For example, they cannot distribute the modified software without
its complete source code, including the modifications and additions.

> This freedom is best supported by writing clear and well-documented
> code that users can easily understand and adapt to their needs.

And by other measures, some of which are no less important.

> Encouraging free software adoption through software design or structural
> means — such as intentionally complicating the code to guide or limit how
> users interact with it — is problematic.

It's a judgment call, yes.  There's a trade-off to be considered.

> This approach runs counter to
> the spirit of free software because it resembles the kind of code
> obfuscation often seen in proprietary software, where the intent is to
> prevent users from understanding or modifying the code.

No, nothing in the specific issue at hand causes any kind of
obfuscation.

> In free software, the goal should be to empower users, not restrict
> them.

To empower users to be able to use, modify, and redistribute the
software they receive.  Not to empower them to do everything
imaginable.

> Any attempt to influence how users engage with the software
> through code structure, rather than through education or advocacy,
> undermines the principles of transparency and user freedom that are
> central to the free software movement.

No, nothing in our practice undermines any transparency nor any user
freedom to use, study, and modify the software.

> The power of free software lies in its openness, and that openness
> should be reflected in the clarity and accessibility of the code
> itself.

Nothing in our practice makes the Emacs code less open and clear.

> There is ambiguity in determining software freedom.  One cannot always
> determine a priori whether external software is free or non-free without
> examining its license. Imposing restrictions or enforcing certain ways
> of doing things to ensure compliance with the free software ethos is an
> overreaching approach.

Once again, this is the approach taken by the GNU Project, and if you
want to argue against it, you are in the wrong place.  Kindly take
this dispute elsewhere.

> The statement you’re referring to touches on an important aspect of the
> free software ecosystem - namely, that the GPL (GNU General Public
> License) is just one of many licenses under which free software can be
> distributed. The idea that software needs to be GPL-compliant is indeed
> a common misconception, but it's more accurate to say that it needs to
> be compatible with the GPL, especially when interacting with GPL-licensed 
> code.  
> 
> But only when distributed.  For software that is not distributed (e.g.,
> used privately or within an organization), these requirements do not
> apply.

Nothing in what we do prevents users from doing whatever they like
with Emacs as long as they do it for their private use.

> Suggesting or enforcing limitations on what users can code, even
> in private use is where the controversy lies.

No one enforces users to do anything nor prevents them from doing
anything they like.  The request was to force the Emacs project to
actively help people do something that we do not want to endorse. Such
kind of request is ridiculous: a software project is not obliged to do
everything its users request, only those parts with which it (the
project) agrees.

> The GPL, as stated, does not place restrictions on private
> modifications or use.

Yes.

> Users are free to experiment, modify, and even create non-free
> software for their private use, without any obligation to comply
> with the GPL’s distribution requirements.

Yes.

> But you decided to hinder that possibility.

No.

Please stop accusing the project with these false accusations.

Sv: as for Calc and the math library

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 4754 bytes --]

> > Eli Zaretskii <eliz@gnu.org> writes:
> >
> > > We made the effort to allow loading dynamic modules precisely for this
> > > reason: to allow free libraries to be used, but not non-free ones.

Emacs users are "linking" Emacs to all kind of proprietary, non-free services
these days. You are reading /r/Emacs yourself and seen packages popping up
targeting this or that proprietary service almost every day. Shared libraries
are no longer the staple of distributing closed source. The computing landscape
has change a lot since 1980s/1990s.

> > You absolutely can use non-free dynamic modules: enforcing the existence
> > of a symbol saying "I promise I'm free" does not change anything. One
> > could also easily add a free dynamic module that calls non-free
> > libraries.
>
> The need to declare that a library is free and have its sources freely
> available does serve as an obstacle for non-free software.  And using
> non-free library with a free module is against the GPL, so it is
> illegal.  We cannot prevent people from lying and doing illegal
> things, we can only make it harder.

IMO, licenses are to restrict the usage, not arbitrary technical
limitations. We could similary have a token declaration in FFI interface, when
loading a library, no?

You are preventing  people not familiar with programming who can't write a
simple C wrapper to load a proprietary library, but it ain't stop any malicious
company anyway. At the same time, the strategy is slowing down
Emacs development and make it harder for talented people to actually write
useful code for Emacs. It is also growing the C core unnecessary.

There is a plethora of MIT licensed math libraries, with big API surfaces,
well optimized for many architectures which users could bring into Emacs
 themselves.

Anyway, Guile is the "gnu extension language", and there are no
problems to expose FFI:

https://www.gnu.org/software/guile/manual/html_node/Foreign-Function-Interface.html

How come there is no decision against loading shared objects in Guile? There was
even a discussion, on this very mailing list, of Emacs core re-written in Guile
(which seems to not happen). That would auto-expose Guile FFI, but that seemed
to be OK?

Similar for CLisp and GCL, both are GNU projects and expose FFI.

With all respect to Emacs developers, especially RMS and you, I am very grateful
for all your work and the software you gave us, but I don't understand how is
this strategy helping Emacs and GNU development?

> > It is sad to see Emacs being hamstrung and users being limited because
> > someone could do something that would have no impact whatsoever with the
> > project or any other user.
>
> This is a sad world for more reasons than one, but discussing that is
> off-topic here.

It shouldn't be Eli, because it is about Emacs development, and you know as well
that it is rather shooting us in the foot, than helping.

I am honesly not trying to be PITA. I understand the subject is sensitive. I have
seen the history, and (I think) I understand why the decision was made. I
won't press the subject more, but I think it may be worth to revisit the
 decision since times have changed.

With all respect
/arthur
________________________________
Från: Eli Zaretskii <eliz@gnu.org>
Skickat: den 12 augusti 2024 15:22
Till: Nicolas Martyanoff <nicolas@n16f.net>
Kopia: arthur.miller@live.com <arthur.miller@live.com>; emacs-devel@gnu.org <emacs-devel@gnu.org>
Ämne: Re: as for Calc and the math library

> From: Nicolas Martyanoff <nicolas@n16f.net>
> Cc: Nicolas Martyanoff <nicolas@n16f.net>,  arthur.miller@live.com,
>   emacs-devel@gnu.org
> Date: Mon, 12 Aug 2024 14:11:07 +0200
>
> Eli Zaretskii <eliz@gnu.org> writes:
>
> > We made the effort to allow loading dynamic modules precisely for this
> > reason: to allow free libraries to be used, but not non-free ones.
>
> You absolutely can use non-free dynamic modules: enforcing the existence
> of a symbol saying "I promise I'm free" does not change anything. One
> could also easily add a free dynamic module that calls non-free
> libraries.

The need to declare that a library is free and have its sources freely
available does serve as an obstacle for non-free software.  And using
non-free library with a free module is against the GPL, so it is
illegal.  We cannot prevent people from lying and doing illegal
things, we can only make it harder.

> It is sad to see Emacs being hamstrung and users being limited because
> someone could do something that would have no impact whatsoever with the
> project or any other user.

This is a sad world for more reasons than one, but discussing that is
off-topic here.

[-- Attachment #2: Type: text/html, Size: 14404 bytes --]

Re: as for Calc and the math library

[Eli Zaretskii]

> From: arthur miller <arthur.miller@live.com>
> CC: "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> Date: Tue, 13 Aug 2024 07:16:35 +0000
> 
> > > Eli Zaretskii <eliz@gnu.org> writes:
> > >
> > > > We made the effort to allow loading dynamic modules precisely for this
> > > > reason: to allow free libraries to be used, but not non-free ones.
> 
> Emacs users are "linking" Emacs to all kind of proprietary, non-free services
> these days. You are reading /r/Emacs yourself and seen packages popping up
> targeting this or that proprietary service almost every day. Shared libraries
> are no longer the staple of distributing closed source. The computing landscape
> has change a lot since 1980s/1990s.
> 
> > > You absolutely can use non-free dynamic modules: enforcing the existence
> > > of a symbol saying "I promise I'm free" does not change anything. One
> > > could also easily add a free dynamic module that calls non-free
> > > libraries.
> >
> > The need to declare that a library is free and have its sources freely
> > available does serve as an obstacle for non-free software.  And using
> > non-free library with a free module is against the GPL, so it is
> > illegal.  We cannot prevent people from lying and doing illegal
> > things, we can only make it harder.
> 
> IMO, licenses are to restrict the usage, not arbitrary technical
> limitations. We could similary have a token declaration in FFI interface, when
> loading a library, no? 

You are basically reiterating what Nicolas already said, and I
answered that.  I see no reason to repeat my answers to these
arguments, they are still the same.

> You are preventing  people not familiar with programming who can't write a
> simple C wrapper to load a proprietary library, but it ain't stop any malicious
> company anyway.

People not familiar with programming will be unable to use FFI for
anything serious anyway.

> At the same time, the strategy is slowing down
> Emacs development and make it harder for talented people to actually write
> useful code for Emacs. It is also growing the C core unnecessary. 

I don't share this view of the Emacs development, of course.  And with
all due respect, I don't think you are in a good position to judge
that: you are not involved in this deep enough and long enough to have
the perspective and experience to make such judgments.

> There is a plethora of MIT licensed math libraries, with big API surfaces, 
> well optimized for many architectures which users could bring into Emacs
>  themselves.

AFAIU, there should be no reasons not to be able to load MIT licensed
libraries via the emacs-module machinery.

> Anyway, Guile is the "gnu extension language", and there are no
> problems to expose FFI:
> 
> https://www.gnu.org/software/guile/manual/html_node/Foreign-Function-Interface.html
> 
> How come there is no decision against loading shared objects in Guile? There was
> even a discussion, on this very mailing list, of Emacs core re-written in Guile
> (which seems to not happen). That would auto-expose Guile FFI, but that seemed
> to be OK?
> 
> Similar for CLisp and GCL, both are GNU projects and expose FFI.

These questions are not for me to answer.  I'm not responsible for
these other projects.  I think they are mistaken, but then the Guile
folks never listened to what I had to say on quite a few subjects, so
I'm not surprised.  (I know nothing about how Common Lisp is developed
and what are its goals.)

For the record: there are other GNU projects that use the same
"restrictions" on plugins: Gawk, GNU Make, and GCC, to mention those I
know about.  So it isn't like Emacs is alone in this.

Re: as for Calc and the math library

[Nicolas Martyanoff]

Eli Zaretskii <eliz@gnu.org> writes:

> AFAIU, there should be no reasons not to be able to load MIT licensed
> libraries via the emacs-module machinery.

IIRC the reason I abandoned was that you can load a dynamic module but
there is no mechanism to reload it once modified, meaning one cannot
develop packages with dynamic modules iteratively.

There is also nothing in place to facilitate building these dynamic
modules as part of an Emacs package, and it is not that simple: locate C
files distributed with the package, build the library with the right
flags depending on the current platform, cache it in a place that makes
sense…

-- 
Nicolas Martyanoff
https://n16f.net
nicolas@n16f.net

Re: as for Calc and the math library

[Eli Zaretskii]

> From: Nicolas Martyanoff <nicolas@n16f.net>
> Cc: arthur miller <arthur.miller@live.com>,  nicolas@n16f.net,
>   emacs-devel@gnu.org
> Date: Tue, 13 Aug 2024 15:10:28 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > AFAIU, there should be no reasons not to be able to load MIT licensed
> > libraries via the emacs-module machinery.
> 
> IIRC the reason I abandoned was that you can load a dynamic module but
> there is no mechanism to reload it once modified, meaning one cannot
> develop packages with dynamic modules iteratively.

This is a technical problem with loading shared libraries, so it will
also happen with FFI, AFAIU.

In general, unloading and reloading doesn't work in Emacs well even in
Lisp, less so with native-compiled Lisp.  Not sure this can be
improved and how, but it would be a welcome addition and enhancement.

> There is also nothing in place to facilitate building these dynamic
> modules as part of an Emacs package, and it is not that simple: locate C
> files distributed with the package, build the library with the right
> flags depending on the current platform, cache it in a place that makes
> sense…

That again is mostly common with any solution based on loading shared
libraries.  We have elaborate documentation of then interfaces for
writing and using dynamic modules, but of course if someone will
submit some boilerplate code to ease this, or come up with automation
of some of this, I'm sure that would be welcome.

Re: as for Calc and the math library

[Nicolas Martyanoff]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Nicolas Martyanoff <nicolas@n16f.net>
>> Cc: arthur miller <arthur.miller@live.com>,  nicolas@n16f.net,
>>   emacs-devel@gnu.org
>> Date: Tue, 13 Aug 2024 15:10:28 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > AFAIU, there should be no reasons not to be able to load MIT licensed
>> > libraries via the emacs-module machinery.
>> 
>> IIRC the reason I abandoned was that you can load a dynamic module but
>> there is no mechanism to reload it once modified, meaning one cannot
>> develop packages with dynamic modules iteratively.
>
> This is a technical problem with loading shared libraries, so it will
> also happen with FFI, AFAIU.
>
> In general, unloading and reloading doesn't work in Emacs well even in
> Lisp, less so with native-compiled Lisp.  Not sure this can be
> improved and how, but it would be a welcome addition and enhancement.

The main difference is that with a dynamic module, if you want to add a
new C function available in elisp or modify an existing one, you have to
update the C glue code defining the function and restart Emacs to build
and reload it.

With the kind of FFI API we are talking about, once the
foreign library has been loaded (e.g. libpq), one can write Elisp
functions calling foreign functions incrementally.

This is not the end of the world if you are just binding a couple
functions and you know everything you need. When you are binding a
large and complex library, being able to go at it incrementally and
testing everything in Emacs at each step makes it much comfortable.

Ultimately all of this is moot since the political aspect takes
precedence. But thinking about it, it should be possible to build a
dynamic module exposing a couple functions whose job would be to open
arbitrary shared libraries and call functions with libffi (which is MIT
licensed so no license violation here). Some Emacs code could be used to
extract constant values, structure layouts and other information
necessary at runtime, but it is not that hard (see cffi-grovel in Common
Lisp for example).

-- 
Nicolas Martyanoff
https://n16f.net
nicolas@n16f.net

Sv: as for Calc and the math library

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 10345 bytes --]

> > > > Eli Zaretskii <eliz@gnu.org> writes:
> > > >
> > > > > We made the effort to allow loading dynamic modules precisely for this
> > > > > reason: to allow free libraries to be used, but not non-free ones.
> >
> > Emacs users are "linking" Emacs to all kind of proprietary, non-free services
> > these days. You are reading /r/Emacs yourself and seen packages popping up
> > targeting this or that proprietary service almost every day. Shared libraries
> > are no longer the staple of distributing closed source. The computing
> > landscape
> > has change a lot since 1980s/1990s.
> >
> > > > You absolutely can use non-free dynamic modules: enforcing the existence
> > > > of a symbol saying "I promise I'm free" does not change anything. One
> > > > could also easily add a free dynamic module that calls non-free
> > > > libraries.
> > >
> > > The need to declare that a library is free and have its sources freely
> > > available does serve as an obstacle for non-free software.  And using
> > > non-free library with a free module is against the GPL, so it is
> > > illegal.  We cannot prevent people from lying and doing illegal
> > > things, we can only make it harder.
> >
> > IMO, licenses are to restrict the usage, not arbitrary technical
> > limitations. We could similary have a token declaration in FFI interface, when
> > loading a library, no?
>
> You are basically reiterating what Nicolas already said, and I
> answered that.  I see no reason to repeat my answers to these
> arguments, they are still the same.

Just justifiying the statement that Emacs could have something similar as
GPL_COMPATIBLE token in Lisp API, just as there is in module API. I don't
 think I have seen anyone suggesting that before, at least not what I know of.

> > You are preventing  people not familiar with programming who can't write a
> > simple C wrapper to load a proprietary library, but it ain't stop any
> > malicious
> > company anyway.
>
> People not familiar with programming will be unable to use FFI for
> anything serious anyway.

I think you have misunderstand what I am saying: my point was that this
situation is making it more tedious for experienced people to work with Emacs
while possibly being a stopper *only* for unexperienced programmers.

> > At the same time, the strategy is slowing down
> > Emacs development and make it harder for talented people to actually write
> > useful code for Emacs. It is also growing the C core unnecessary.
>
> I don't share this view of the Emacs development, of course.  And with
> all due respect, I don't think you are in a good position to judge
> that: you are not involved in this deep enough and long enough to have
> the perspective and experience to make such judgments.

Sure, I certainly am not experienced nor involved in Emacs as long and
as much as you or many others here.

However, I think everyone understands possible fears of a company, or even
just individuals loading proprietary libraries, which free world (GNU) does not
want to endorse and promote such usage. I don't even disagree that it is unjustified.

However, pragmatically, I don't think putting an obstacle is the best way. On the
 contrary, I think, making it as easy as possible to use and extend Emacs for
 whichever purpose, gives a positive incentive for people to see qualities in Emacs
 in particular, and implicitly in GNU project.

As with every important decision, there is a balance between the gain and the
damage. In this case, considering how easy is to work with C libraries in some
other Lisp implementations, and how faster it would be to write extensions and
experiment with Emacs, the gain is perhaps bigger than the possible loss.

> > There is a plethora of MIT licensed math libraries, with big API surfaces,
> > well optimized for many architectures which users could bring into Emacs
> >  themselves.
>
> AFAIU, there should be no reasons not to be able to load MIT licensed
> libraries via the emacs-module machinery.

Nobody is stopping anyone to implement anything as modules, outside of Emacs
core, and I didn't say anything against modules either. I just say FFI would
make it easier and faster.

> > Anyway, Guile is the "gnu extension language", and there are no
> > problems to expose FFI:
> >
> > https://www.gnu.org/software/guile/manual/html_node/Foreign-Function-Interface.html
> >
> > How come there is no decision against loading shared objects in Guile? There
> > was
> > even a discussion, on this very mailing list, of Emacs core re-written in
> > Guile
> > (which seems to not happen). That would auto-expose Guile FFI, but that seemed
> > to be OK?
> >
> > Similar for CLisp and GCL, both are GNU projects and expose FFI.
>
> These questions are not for me to answer.  I'm not responsible for
> these other projects.  I think they are mistaken, but then the Guile
> folks never listened to what I had to say on quite a few subjects, so
> I'm not surprised.  (I know nothing about how Common Lisp is developed
> and what are its goals.)

Of course you are not responsible for other projects, nobody thinks you are. My
point is that it is not unthinkable for a GNU project to implement FFI. Also, we
are not directly seeing commercial companies running toward Guile and linking
bunch of proprietary binaries into Guile runtime, and Guile is de facto, a
popular Scheme implementation.

> For the record: there are other GNU projects that use the same
> "restrictions" on plugins: Gawk, GNU Make, and GCC, to mention those I
> know about.  So it isn't like Emacs is alone in this.

I am not sure if it is relevant with non-lisp projects. There are lots of
projects with different needs, some are less, others are more self-contained. It
all depends how the software is used. Lisp(s), Emacs Lisp included, are usually
in somewhat special situation of being relatively isolated islands when it comes
to available software. Common Lisp perhaps being a biggest exception there.
An easy to use FFI makes it easier to take advantage of already existing software,
such as mathematical libraries for example.

Anyway, thanks for the arguments; I think it is useful to re-think the arguments,
regardless on which side of the arguments I am.

best regards
________________________________
Från: Eli Zaretskii <eliz@gnu.org>
Skickat: den 13 augusti 2024 14:12
Till: arthur miller <arthur.miller@live.com>
Kopia: nicolas@n16f.net <nicolas@n16f.net>; emacs-devel@gnu.org <emacs-devel@gnu.org>
Ämne: Re: as for Calc and the math library

> From: arthur miller <arthur.miller@live.com>
> CC: "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> Date: Tue, 13 Aug 2024 07:16:35 +0000
>
> > > Eli Zaretskii <eliz@gnu.org> writes:
> > >
> > > > We made the effort to allow loading dynamic modules precisely for this
> > > > reason: to allow free libraries to be used, but not non-free ones.
>
> Emacs users are "linking" Emacs to all kind of proprietary, non-free services
> these days. You are reading /r/Emacs yourself and seen packages popping up
> targeting this or that proprietary service almost every day. Shared libraries
> are no longer the staple of distributing closed source. The computing landscape
> has change a lot since 1980s/1990s.
>
> > > You absolutely can use non-free dynamic modules: enforcing the existence
> > > of a symbol saying "I promise I'm free" does not change anything. One
> > > could also easily add a free dynamic module that calls non-free
> > > libraries.
> >
> > The need to declare that a library is free and have its sources freely
> > available does serve as an obstacle for non-free software.  And using
> > non-free library with a free module is against the GPL, so it is
> > illegal.  We cannot prevent people from lying and doing illegal
> > things, we can only make it harder.
>
> IMO, licenses are to restrict the usage, not arbitrary technical
> limitations. We could similary have a token declaration in FFI interface, when
> loading a library, no?

You are basically reiterating what Nicolas already said, and I
answered that.  I see no reason to repeat my answers to these
arguments, they are still the same.

> You are preventing  people not familiar with programming who can't write a
> simple C wrapper to load a proprietary library, but it ain't stop any malicious
> company anyway.

People not familiar with programming will be unable to use FFI for
anything serious anyway.

> At the same time, the strategy is slowing down
> Emacs development and make it harder for talented people to actually write
> useful code for Emacs. It is also growing the C core unnecessary.

I don't share this view of the Emacs development, of course.  And with
all due respect, I don't think you are in a good position to judge
that: you are not involved in this deep enough and long enough to have
the perspective and experience to make such judgments.

> There is a plethora of MIT licensed math libraries, with big API surfaces,
> well optimized for many architectures which users could bring into Emacs
>  themselves.

AFAIU, there should be no reasons not to be able to load MIT licensed
libraries via the emacs-module machinery.

> Anyway, Guile is the "gnu extension language", and there are no
> problems to expose FFI:
>
> https://www.gnu.org/software/guile/manual/html_node/Foreign-Function-Interface.html
>
> How come there is no decision against loading shared objects in Guile? There was
> even a discussion, on this very mailing list, of Emacs core re-written in Guile
> (which seems to not happen). That would auto-expose Guile FFI, but that seemed
> to be OK?
>
> Similar for CLisp and GCL, both are GNU projects and expose FFI.

These questions are not for me to answer.  I'm not responsible for
these other projects.  I think they are mistaken, but then the Guile
folks never listened to what I had to say on quite a few subjects, so
I'm not surprised.  (I know nothing about how Common Lisp is developed
and what are its goals.)

For the record: there are other GNU projects that use the same
"restrictions" on plugins: Gawk, GNU Make, and GCC, to mention those I
know about.  So it isn't like Emacs is alone in this.

[-- Attachment #2: Type: text/html, Size: 27709 bytes --]

Re: as for Calc and the math library

[Eli Zaretskii]

> From: arthur miller <arthur.miller@live.com>
> CC: "nicolas@n16f.net" <nicolas@n16f.net>, "emacs-devel@gnu.org"
> 	<emacs-devel@gnu.org>
> Date: Tue, 13 Aug 2024 21:43:00 +0000
> 
> > > IMO, licenses are to restrict the usage, not arbitrary technical
> > > limitations. We could similary have a token declaration in FFI interface, when
> > > loading a library, no?
> >
> > You are basically reiterating what Nicolas already said, and I
> > answered that.  I see no reason to repeat my answers to these
> > arguments, they are still the same.
> 
> Just justifiying the statement that Emacs could have something similar as
> GPL_COMPATIBLE token in Lisp API, just as there is in module API. I don't
>  think I have seen anyone suggesting that before, at least not what I know of.

Then I don't understand what you are suggesting here.  What do you
mean by "token in Lisp API"?  And if you mean something like what we
have in the emacs-module machinery, then how is that different from
what we already have?

> > > You are preventing  people not familiar with programming who can't write a
> > > simple C wrapper to load a proprietary library, but it ain't stop any
> > > malicious
> > > company anyway.
> >
> > People not familiar with programming will be unable to use FFI for
> > anything serious anyway.
> 
> I think you have misunderstand what I am saying: my point was that this
> situation is making it more tedious for experienced people to work with Emacs
> while possibly being a stopper *only* for unexperienced programmers.

I understood what you were saying, and I disagree with your assertion
that using emacs-module API makes the job harder.  If anything, it
makes it somewhat easier: it gives the module a way to call Emacs
primitives and provides various auxiliary facilities, without which
adjusting a library to Emacs would be a very hard job, because Emacs
has its own object system that is quite different from that of any
programming language that can be used to code a module.  I suggest to
re-read the "Writing Dynamically-Loaded Modules" appendix to the ELisp
manual and imagine how hard it would be to use FFI without many of
those facilities provided specifically for dynamic modules.

> However, pragmatically, I don't think putting an obstacle is the best way. On the
>  contrary, I think, making it as easy as possible to use and extend Emacs for
>  whichever purpose, gives a positive incentive for people to see qualities in Emacs
>  in particular, and implicitly in GNU project.
> 
> As with every important decision, there is a balance between the gain and the
> damage. In this case, considering how easy is to work with C libraries in some
> other Lisp implementations, and how faster it would be to write extensions and
> experiment with Emacs, the gain is perhaps bigger than the possible loss.

There are examples out there what happens when projects take the route
you suggest.  I cannot say I'm encouraged with LLVM being used for
proprietary closed-code extensions and targets, nor am I overwhelmed
with the popularity of Guile, which does have unrestricted FFI.  So I
see no reason to believe what you suggest will bring such significant
advantages to Emacs that we should seriously consider lifting the
GPL-compatibility restriction.

> > > There is a plethora of MIT licensed math libraries, with big API surfaces,
> > > well optimized for many architectures which users could bring into Emacs
> > >  themselves.
> >
> > AFAIU, there should be no reasons not to be able to load MIT licensed
> > libraries via the emacs-module machinery.
> 
> Nobody is stopping anyone to implement anything as modules, outside of Emacs
> core, and I didn't say anything against modules either. I just say FFI would
> make it easier and faster.

See above: I think you are mistaken in your assessment.  I don't see
any reasons why using FFI would be so much easier than using the
existing modules API.  Given some simple boilerplate (like the DEFUN
macro you can see in test/src/emacs-module-resources/mod-test.c, and
other utility functions there), the job becomes quite simple.  By
contrast, using FFI the programmer would need to figure all this out
on their own, and that requires very good familiarity with Emacs
internals.

> > For the record: there are other GNU projects that use the same
> > "restrictions" on plugins: Gawk, GNU Make, and GCC, to mention those I
> > know about.  So it isn't like Emacs is alone in this.
> 
> I am not sure if it is relevant with non-lisp projects.

How so?  GNU Make has its own scripting language, and Gawk implements
the Awk programming language, which is a full-fledged specialized
language for data processing.  How is this "not relevant" to our case?
Gawk even comes with a small set of extensions written using the
plug-in protocol, and I invite you to review them to see how useful
they are, even though they are relatively small and do simple tasks.

> There are lots of
> projects with different needs, some are less, others are more self-contained. It
> all depends how the software is used. Lisp(s), Emacs Lisp included, are usually
> in somewhat special situation of being relatively isolated islands when it comes
> to available software. Common Lisp perhaps being a biggest exception there.
> An easy to use FFI makes it easier to take advantage of already existing software, 
> such as mathematical libraries for example. 

I think you greatly underestimate the complexity of using FFI-style
API to allow Emacs to use a mathematical library.  For starters,
floating-point values have a very specific representation in Emacs, so
you cannot just take a value from the library and use it.  Next, most
such libraries allocate memory for its data structures (matrices
etc.), so using them will need to provide GC glue for freeing the
resources when an instance of a data structure is no longer needed.
And that is just the tip of a very large iceberg.

> Anyway, thanks for the arguments; I think it is useful to re-think the arguments,
> regardless on which side of the arguments I am.

As a practical example, I suggest to take a library and integrate it
with Emacs using the modules API, if you haven't done that already.
It is sometimes amazing how many little details need to be figured out
and coded for that, when the target is Emacs Lisp.

Sv: as for Calc and the math library

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 18449 bytes --]

> >
> > > > IMO, licenses are to restrict the usage, not arbitrary technical
> > > > limitations. We could similary have a token declaration in FFI interface,
> > > > when
> > > > loading a library, no?
> > >
> > > You are basically reiterating what Nicolas already said, and I
> > > answered that.  I see no reason to repeat my answers to these
> > > arguments, they are still the same.
> >
> > Just justifiying the statement that Emacs could have something similar as
> > GPL_COMPATIBLE token in Lisp API, just as there is in module API. I don't
> >  think I have seen anyone suggesting that before, at least not what I know of.
>
> Then I don't understand what you are suggesting here.  What do you
> mean by "token in Lisp API"?  And if you mean something like what we

I suggest exactly that, as it sounds: a compliance token, as you have for modules.

Something like:

(defun load-library (library-name, gpl-compatible)
"Load a shared object with LIBRARY-NAME. The module has to be compatible with
GPL licence, for the deatils see https://www.gnu.org/licenses/gpl-3.0.txt, which
is asserted by GPL-COMPATIBLE been non-nil."
 ( ... ))

> have in the emacs-module machinery, then how is that different from
> what we already have?

It is not different at all; it is the main point and *the* reason why I suggest
it. Being in Lisp is obviously a lesser obstacle than being in C, but the same
argument(s) you presented for module code, and possible violations of the
licence, are valid for the Lisp code as well.

By the way I was about to snitch FFI from SXEmacs, but I see now Tromeys code is
in Emacs, thanks to Madhus mail. I know Tom made one, but I didn't know it was
still included in git sources.

> > > > You are preventing  people not familiar with programming who can't write a
> > > > simple C wrapper to load a proprietary library, but it ain't stop any
> > > > malicious
> > > > company anyway.
> > >
> > > People not familiar with programming will be unable to use FFI for
> > > anything serious anyway.
> >
> > I think you have misunderstand what I am saying: my point was that this
> > situation is making it more tedious for experienced people to work with Emacs
> > while possibly being a stopper *only* for unexperienced programmers.
>
> I understood what you were saying, and I disagree with your assertion
> that using emacs-module API makes the job harder.  If anything, it
> makes it somewhat easier: it gives the module a way to call Emacs
> primitives and provides various auxiliary facilities, without which
> adjusting a library to Emacs would be a very hard job, because Emacs
> has its own object system that is quite different from that of any
> programming language that can be used to code a module.  I suggest to
> re-read the "Writing Dynamically-Loaded Modules" appendix to the ELisp
> manual and imagine how hard it would be to use FFI without many of
> those facilities provided specifically for dynamic modules.

I understand what you say as well. From my relatively short experience of
writing some FFI code for SBCL, I find it easier to work iteratively from a Lisp
repl, than writing a C code, recompile, reload and test. I can just C-x C-e in
Slime and have my glue code re-evaled and re-installed, instead of recompiling
and reloading everything.

> > However, pragmatically, I don't think putting an obstacle is the best way. On
> > the
> >  contrary, I think, making it as easy as possible to use and extend Emacs for
> >  whichever purpose, gives a positive incentive for people to see qualities in
> > Emacs
> >  in particular, and implicitly in GNU project.
> >
> > As with every important decision, there is a balance between the gain and the
> > damage. In this case, considering how easy is to work with C libraries in some
> > other Lisp implementations, and how faster it would be to write extensions and
> > experiment with Emacs, the gain is perhaps bigger than the possible loss.
>
> There are examples out there what happens when projects take the route
> you suggest.  I cannot say I'm encouraged with LLVM being used for
> proprietary closed-code extensions and targets, nor am I overwhelmed
> with the popularity of Guile, which does have unrestricted FFI.  So I
> see no reason to believe what you suggest will bring such significant
> advantages to Emacs that we should seriously consider lifting the
> GPL-compatibility restriction.

I don't say you should allow unrestricted access. Just to furntiture around a
bit, and allow access to a library from Lisp, not just from C.

> > > > There is a plethora of MIT licensed math libraries, with big API surfaces,
> > > > well optimized for many architectures which users could bring into Emacs
> > > >  themselves.
> > >
> > > AFAIU, there should be no reasons not to be able to load MIT licensed
> > > libraries via the emacs-module machinery.
> >
> > Nobody is stopping anyone to implement anything as modules, outside of Emacs
> > core, and I didn't say anything against modules either. I just say FFI would
> > make it easier and faster.
>
> See above: I think you are mistaken in your assessment.  I don't see
> any reasons why using FFI would be so much easier than using the
> existing modules API.  Given some simple boilerplate (like the DEFUN
> macro you can see in test/src/emacs-module-resources/mod-test.c, and
> other utility functions there), the job becomes quite simple.  By
> contrast, using FFI the programmer would need to figure all this out
> on their own, and that requires very good familiarity with Emacs
> internals.
>
> > > For the record: there are other GNU projects that use the same
> > > "restrictions" on plugins: Gawk, GNU Make, and GCC, to mention those I
> > > know about.  So it isn't like Emacs is alone in this.
> >
> > I am not sure if it is relevant with non-lisp projects.
>
> How so?  GNU Make has its own scripting language, and Gawk implements

Actually two scripting lanugages. More recent one is Guile. I have never felt
need to extend make, so no idea if Guile FFI is usable from make.

> the Awk programming language, which is a full-fledged specialized
> language for data processing.  How is this "not relevant" to our case?

I didn't thought it is relevant since neither make, nor awk, nor gcc are used
interactively to write Emacs applications, to solve computational problems and
similar in the way Emacs is used.

> Gawk even comes with a small set of extensions written using the
> plug-in protocol, and I invite you to review them to see how useful
> they are, even though they are relatively small and do simple tasks.

I have basically stopped using Bash & Co (tr, awk, sed etc) for any scripting. I
don't think those are bad, but as you have seen from my numerous posts on Reddit
promoting Elisp and debugging in Elisp, I do honestly find scripting in Lisp and
Emacs much more enjoyable than debugging shell scripts. I don't think it is even
comparable.

Perhaps Emanuel could write an awk implementation in Elisp, that works on Emacs
buffers, as a (one of) higher level interface. It would be probably more useful
and enjoybale for text processing than extending awk with C++ modules or
implementing some sort of DSL based on a bunch of cryptic acronyms. At least awk
is well documented. Just my opinion, I hope Emanuel is not angry on me for
reflecting and suggesting it.

By the way, a regression (sorry), perhaps someone more literally gifted than me
and better used to English language, could add a few words to Emacs manual about
how Emacs API works, regarding text processing and buffer management at
least. Something like this:
________________________________________________________________________________
Emacs Lisp uses a simple procedural model to manipulate windows, files, buffers,
and other objects.

In Emacs API, we typically manipulate a current object, whether it is a buffer,
point, window, frame, etc. It means we act on a selected buffer, selected
window, frame which we then manipulate by calling an appropriate function. Which
one we need is best looked up in the manual. The details on the calling
arguments and returned results are easiset looked up via C-h f function-name
RET.

Some of functions can take an optional object to act on, instead of acting on
the current-one, but the basic model of working with Emacs API buffer, window
and frame management, as well as text processing, is to select something (make
it current), and than acting on it.
________________________________________________________________________________

Something in that style, I don't see anything similar in introduction to the
manual, and I think it might be useful to clarify it. At least I have understand
Emacs API so.

One can also speculate how API could look like, and for illustrative purposes,
we can compare for example to similar API model like PostScript or OpenGL vs
Xlib or win32, or to iterator based design as suggested by Yuri. But I don't
think it belongs to the manual or emacs-devel list.

> > There are lots of
> > projects with different needs, some are less, others are more self-contained.
> > It
> > all depends how the software is used. Lisp(s), Emacs Lisp included, are
> > usually
> > in somewhat special situation of being relatively isolated islands when it
> > comes
> > to available software. Common Lisp perhaps being a biggest exception there.
> > An easy to use FFI makes it easier to take advantage of already existing
> > software,
> > such as mathematical libraries for example.
>
> I think you greatly underestimate the complexity of using FFI-style
> API to allow Emacs to use a mathematical library.  For starters,
> floating-point values have a very specific representation in Emacs, so
> you cannot just take a value from the library and use it.  Next, most
> such libraries allocate memory for its data structures (matrices
> etc.), so using them will need to provide GC glue for freeing the
> resources when an instance of a data structure is no longer needed.
> And that is just the tip of a very large iceberg.

Agree. I haven't seen Tromeys code yet, but I have seen and tried FFI in
SXEmacs, and as said in SBCL.

SBCL does not use libffi under the hood (unlike more portable CFFI), and they do
give one access to raw pointers and raw memory allocated by C runtime, as well
as the responsibility to free that allocated memory. For the good and for the
bad.

Now, Emacs also has an interface to GCC, I wonder how hard/easy is to expose C
runtime via native compiler to Lisp? I am not familiar with libgccjit, but as I
udnerstand eln files are shared objects under the hood? Does it not make the ffi
implementation easier? Sorry if it is an uninformed question, I haven't looked
that up myself.

> > Anyway, thanks for the arguments; I think it is useful to re-think the
> > arguments,
> > regardless on which side of the arguments I am.
>
> As a practical example, I suggest to take a library and integrate it
> with Emacs using the modules API, if you haven't done that already.
> It is sometimes amazing how many little details need to be figured out
> and coded for that, when the target is Emacs Lisp.
>

I have done some FFI with SBCL, and yes, I agree there are lots of details to
take care of. Error checking via Lisp is not trivial, compared to simple
if-statement in C. But still, if I f-up in SBCL with my glue, I can just rewrite
and C-x C-e in Slime and switch to repl window to test. As said above, I don't
think it is comparable to writing C code.

________________________________
Från: Eli Zaretskii <eliz@gnu.org>
Skickat: den 14 augusti 2024 07:10
Till: arthur miller <arthur.miller@live.com>
Kopia: nicolas@n16f.net <nicolas@n16f.net>; emacs-devel@gnu.org <emacs-devel@gnu.org>
Ämne: Re: as for Calc and the math library

> From: arthur miller <arthur.miller@live.com>
> CC: "nicolas@n16f.net" <nicolas@n16f.net>, "emacs-devel@gnu.org"
>        <emacs-devel@gnu.org>
> Date: Tue, 13 Aug 2024 21:43:00 +0000
>
> > > IMO, licenses are to restrict the usage, not arbitrary technical
> > > limitations. We could similary have a token declaration in FFI interface, when
> > > loading a library, no?
> >
> > You are basically reiterating what Nicolas already said, and I
> > answered that.  I see no reason to repeat my answers to these
> > arguments, they are still the same.
>
> Just justifiying the statement that Emacs could have something similar as
> GPL_COMPATIBLE token in Lisp API, just as there is in module API. I don't
>  think I have seen anyone suggesting that before, at least not what I know of.

Then I don't understand what you are suggesting here.  What do you
mean by "token in Lisp API"?  And if you mean something like what we
have in the emacs-module machinery, then how is that different from
what we already have?

> > > You are preventing  people not familiar with programming who can't write a
> > > simple C wrapper to load a proprietary library, but it ain't stop any
> > > malicious
> > > company anyway.
> >
> > People not familiar with programming will be unable to use FFI for
> > anything serious anyway.
>
> I think you have misunderstand what I am saying: my point was that this
> situation is making it more tedious for experienced people to work with Emacs
> while possibly being a stopper *only* for unexperienced programmers.

I understood what you were saying, and I disagree with your assertion
that using emacs-module API makes the job harder.  If anything, it
makes it somewhat easier: it gives the module a way to call Emacs
primitives and provides various auxiliary facilities, without which
adjusting a library to Emacs would be a very hard job, because Emacs
has its own object system that is quite different from that of any
programming language that can be used to code a module.  I suggest to
re-read the "Writing Dynamically-Loaded Modules" appendix to the ELisp
manual and imagine how hard it would be to use FFI without many of
those facilities provided specifically for dynamic modules.

> However, pragmatically, I don't think putting an obstacle is the best way. On the
>  contrary, I think, making it as easy as possible to use and extend Emacs for
>  whichever purpose, gives a positive incentive for people to see qualities in Emacs
>  in particular, and implicitly in GNU project.
>
> As with every important decision, there is a balance between the gain and the
> damage. In this case, considering how easy is to work with C libraries in some
> other Lisp implementations, and how faster it would be to write extensions and
> experiment with Emacs, the gain is perhaps bigger than the possible loss.

There are examples out there what happens when projects take the route
you suggest.  I cannot say I'm encouraged with LLVM being used for
proprietary closed-code extensions and targets, nor am I overwhelmed
with the popularity of Guile, which does have unrestricted FFI.  So I
see no reason to believe what you suggest will bring such significant
advantages to Emacs that we should seriously consider lifting the
GPL-compatibility restriction.

> > > There is a plethora of MIT licensed math libraries, with big API surfaces,
> > > well optimized for many architectures which users could bring into Emacs
> > >  themselves.
> >
> > AFAIU, there should be no reasons not to be able to load MIT licensed
> > libraries via the emacs-module machinery.
>
> Nobody is stopping anyone to implement anything as modules, outside of Emacs
> core, and I didn't say anything against modules either. I just say FFI would
> make it easier and faster.

See above: I think you are mistaken in your assessment.  I don't see
any reasons why using FFI would be so much easier than using the
existing modules API.  Given some simple boilerplate (like the DEFUN
macro you can see in test/src/emacs-module-resources/mod-test.c, and
other utility functions there), the job becomes quite simple.  By
contrast, using FFI the programmer would need to figure all this out
on their own, and that requires very good familiarity with Emacs
internals.

> > For the record: there are other GNU projects that use the same
> > "restrictions" on plugins: Gawk, GNU Make, and GCC, to mention those I
> > know about.  So it isn't like Emacs is alone in this.
>
> I am not sure if it is relevant with non-lisp projects.

How so?  GNU Make has its own scripting language, and Gawk implements
the Awk programming language, which is a full-fledged specialized
language for data processing.  How is this "not relevant" to our case?
Gawk even comes with a small set of extensions written using the
plug-in protocol, and I invite you to review them to see how useful
they are, even though they are relatively small and do simple tasks.

> There are lots of
> projects with different needs, some are less, others are more self-contained. It
> all depends how the software is used. Lisp(s), Emacs Lisp included, are usually
> in somewhat special situation of being relatively isolated islands when it comes
> to available software. Common Lisp perhaps being a biggest exception there.
> An easy to use FFI makes it easier to take advantage of already existing software,
> such as mathematical libraries for example.

I think you greatly underestimate the complexity of using FFI-style
API to allow Emacs to use a mathematical library.  For starters,
floating-point values have a very specific representation in Emacs, so
you cannot just take a value from the library and use it.  Next, most
such libraries allocate memory for its data structures (matrices
etc.), so using them will need to provide GC glue for freeing the
resources when an instance of a data structure is no longer needed.
And that is just the tip of a very large iceberg.

> Anyway, thanks for the arguments; I think it is useful to re-think the arguments,
> regardless on which side of the arguments I am.

As a practical example, I suggest to take a library and integrate it
with Emacs using the modules API, if you haven't done that already.
It is sometimes amazing how many little details need to be figured out
and coded for that, when the target is Emacs Lisp.

[-- Attachment #2: Type: text/html, Size: 44796 bytes --]

Re: Sv: as for Calc and the math library

[Nicolas Martyanoff]

arthur miller <arthur.miller@live.com> writes:

> I suggest exactly that, as it sounds: a compliance token, as you have for modules.
>
> Something like:
>
> (defun load-library (library-name, gpl-compatible)
> "Load a shared object with LIBRARY-NAME. The module has to be compatible with
> GPL licence, for the deatils see https://www.gnu.org/licenses/gpl-3.0.txt, which
> is asserted by GPL-COMPATIBLE been non-nil."
>  ( ... ))
>
>> have in the emacs-module machinery, then how is that different from
>> what we already have?

I would be fine with that, but I still do not see the point. IANAL but
as far as I know, using dlopen/dlsym to call a function in another
library does not create a new program, there is no notion of derived
work of any kind, this has nothing to do with either the GPL or the
license of the library.

If this is a purely ideological issue, and it feels to me like it is, then
why is it fine for Emacs to have functions to call external programs and
read their output? What if these programs are proprietary?

-- 
Nicolas Martyanoff
https://n16f.net
nicolas@n16f.net

Re: as for Calc and the math library

[Eli Zaretskii]

> From: Nicolas Martyanoff <nicolas@n16f.net>
> Cc: Eli Zaretskii <eliz@gnu.org>,  "nicolas@n16f.net" <nicolas@n16f.net>,
>   "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> Date: Wed, 14 Aug 2024 11:56:01 +0200
> 
> as far as I know, using dlopen/dlsym to call a function in another
> library does not create a new program, there is no notion of derived
> work of any kind, this has nothing to do with either the GPL or the
> license of the library.

AFAIU, according to GPL, using a shared library cannot even be subject
to LGPL, only to GPL.  The libgcc shared library is one frequent case
in point.  So I think you are wrong.

> If this is a purely ideological issue, and it feels to me like it is, then
> why is it fine for Emacs to have functions to call external programs and
> read their output? What if these programs are proprietary?

Exactly because they are separate programs, and because we don't
distribute them (and don't encourage anyone to use them).

In any case, if you, or someone else wants to discuss GPL-related
issues and their utility, please take the discussion elsewhere, for
example gnu-misc-discuss.  This list is not for such discussions.

Re: as for Calc and the math library

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Nicolas Martyanoff <nicolas@n16f.net>
>> Cc: arthur miller <arthur.miller@live.com>,  emacs-devel@gnu.org
>> Date: Mon, 12 Aug 2024 13:23:27 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: arthur miller <arthur.miller@live.com>
>> >> Date: Mon, 12 Aug 2024 05:30:08 +0000
>> >> 
>> >> Even better, given people proper FFI so they can import and use any of the tens
>> >> or hundreds of quality math libraries available.
>> >
>> > Not going to happen, and you know it.
>> 
>> Naïve question, why?
>
> Because providing FFI would allow using non-free libraries in Emacs.
>
> We made the effort to allow loading dynamic modules precisely for this
> reason: to allow free libraries to be used, but not non-free ones.

There is an item in etc/TODO about adding FFI, as an important feature:

  ** FFI (foreign function interface)
  See e.g. https://lists.gnu.org/r/emacs-devel/2013-10/msg00246.html

  One way of doing this is to start with fx's dynamic loading, and use it
  to implement things like auto-loaded buffer parsers and database
  access in cases which need more than Lisp.

Now I'm confused.

Re: as for Calc and the math library

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Nicolas Martyanoff <nicolas@n16f.net>
>>> Cc: arthur miller <arthur.miller@live.com>,  emacs-devel@gnu.org
>>> Date: Mon, 12 Aug 2024 13:23:27 +0200
>>> 
>>> Eli Zaretskii <eliz@gnu.org> writes:
>>> 
>>> >> From: arthur miller <arthur.miller@live.com>
>>> >> Date: Mon, 12 Aug 2024 05:30:08 +0000
>>> >> 
>>> >> Even better, given people proper FFI so they can import and use any of the tens
>>> >> or hundreds of quality math libraries available.
>>> >
>>> > Not going to happen, and you know it.
>>> 
>>> Naïve question, why?
>>
>> Because providing FFI would allow using non-free libraries in Emacs.
>>
>> We made the effort to allow loading dynamic modules precisely for this
>> reason: to allow free libraries to be used, but not non-free ones.
>
> There is an item in etc/TODO about adding FFI, as an important feature:
>
>   ** FFI (foreign function interface)
>   See e.g. https://lists.gnu.org/r/emacs-devel/2013-10/msg00246.html
>
>   One way of doing this is to start with fx's dynamic loading, and use it
>   to implement things like auto-loaded buffer parsers and database
>   access in cases which need more than Lisp.
>
> Now I'm confused.

I guess an FFI is already there? Or was there because it looks kind of
dead:

  https://github.com/tromey/emacs-ffi

Tom Tromey says in one of the issues

  https://github.com/tromey/emacs-ffi/issues/20

that John Wigley asked him on IRC to submit it for inclusion. Can't find
anything on emacs-devel though.

Re: as for Calc and the math library

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: Nicolas Martyanoff <nicolas@n16f.net>,  arthur.miller@live.com,
>   emacs-devel@gnu.org
> Date: Wed, 14 Aug 2024 06:11:00 +0200
> 
> I guess an FFI is already there? Or was there because it looks kind of
> dead:
> 
>   https://github.com/tromey/emacs-ffi
> 
> Tom Tromey says in one of the issues
> 
>   https://github.com/tromey/emacs-ffi/issues/20
> 
> that John Wigley asked him on IRC to submit it for inclusion. Can't find
> anything on emacs-devel though.

It doesn't ensure GPL-compliance, AFAICT, so in its current form it
cannot be accepted, sorry.  And I'm not sure I understand how can one
technically enforce GPL compliance in FFI-style loading of arbitrary
shared libraries.  The only idea that comes to mind is allow-list of
known libraries, the way we do in sqlite.c, but I'm not sure that
method is scalable to the basically infinite world of arbitrary
libraries.

Re: as for Calc and the math library

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: Nicolas Martyanoff <nicolas@n16f.net>,  arthur.miller@live.com,
>>   emacs-devel@gnu.org
>> Date: Wed, 14 Aug 2024 06:11:00 +0200
>> 
>> I guess an FFI is already there? Or was there because it looks kind of
>> dead:
>> 
>>   https://github.com/tromey/emacs-ffi
>> 
>> Tom Tromey says in one of the issues
>> 
>>   https://github.com/tromey/emacs-ffi/issues/20
>> 
>> that John Wigley asked him on IRC to submit it for inclusion. Can't find
>> anything on emacs-devel though.
>
> It doesn't ensure GPL-compliance, AFAICT, so in its current form it
> cannot be accepted, sorry.  And I'm not sure I understand how can one
> technically enforce GPL compliance in FFI-style loading of arbitrary
> shared libraries.  The only idea that comes to mind is allow-list of
> known libraries, the way we do in sqlite.c, but I'm not sure that
> method is scalable to the basically infinite world of arbitrary
> libraries.

Maybe one should remove the entry from etc/TODO then?

Re: as for Calc and the math library

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: nicolas@n16f.net,  arthur.miller@live.com,  emacs-devel@gnu.org
> Date: Wed, 14 Aug 2024 08:28:38 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> >> Cc: Nicolas Martyanoff <nicolas@n16f.net>,  arthur.miller@live.com,
> >>   emacs-devel@gnu.org
> >> Date: Wed, 14 Aug 2024 06:11:00 +0200
> >> 
> >> I guess an FFI is already there? Or was there because it looks kind of
> >> dead:
> >> 
> >>   https://github.com/tromey/emacs-ffi
> >> 
> >> Tom Tromey says in one of the issues
> >> 
> >>   https://github.com/tromey/emacs-ffi/issues/20
> >> 
> >> that John Wigley asked him on IRC to submit it for inclusion. Can't find
> >> anything on emacs-devel though.
> >
> > It doesn't ensure GPL-compliance, AFAICT, so in its current form it
> > cannot be accepted, sorry.  And I'm not sure I understand how can one
> > technically enforce GPL compliance in FFI-style loading of arbitrary
> > shared libraries.  The only idea that comes to mind is allow-list of
> > known libraries, the way we do in sqlite.c, but I'm not sure that
> > method is scalable to the basically infinite world of arbitrary
> > libraries.
> 
> Maybe one should remove the entry from etc/TODO then?

Maybe.

Re: as for Calc and the math library

[Suhail Singh]

Eli Zaretskii <eliz@gnu.org> writes:

>> I guess an FFI is already there? Or was there because it looks kind of
>> dead:
>> 
>>   https://github.com/tromey/emacs-ffi
>> 
>> Tom Tromey says in one of the issues
>> 
>>   https://github.com/tromey/emacs-ffi/issues/20
>> 
>> that John Wigley asked him on IRC to submit it for inclusion. Can't find
>> anything on emacs-devel though.
>
> It doesn't ensure GPL-compliance, AFAICT, so in its current form it
> cannot be accepted, sorry. And I'm not sure I understand how can one
> technically enforce GPL compliance in FFI-style loading of arbitrary
> shared libraries.  The only idea that comes to mind is allow-list of
> known libraries, the way we do in sqlite.c, but I'm not sure that
> method is scalable to the basically infinite world of arbitrary
> libraries.

On the topic of what would be acceptable for an FFI, wouldn't something
akin to what's done for modules be sufficient ?  I.e., have the users of
the interface explicitly state that they are compliant.

It would scale better than an allow-list.  IIUC, Arthur mentioned this
in another thread.  If this wouldn't be sufficient for an FFI, could you
please elaborate on why that's the case ?

The question here isn't about what does or does not constitute GPL
compliance, but what Emacs, as a project, is trying to ensure when it
comes to such compliance.  In the case of modules, at least, a claim of
compliance is sufficient (without any additional verification).

-- 
Suhail

Re: as for Calc and the math library

[Eli Zaretskii]

> From: Suhail Singh <suhailsingh247@gmail.com>
> Cc: Gerd Möllmann <gerd.moellmann@gmail.com>,
>   nicolas@n16f.net,
>   arthur.miller@live.com,  emacs-devel@gnu.org
> Date: Wed, 14 Aug 2024 10:00:18 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > It doesn't ensure GPL-compliance, AFAICT, so in its current form it
> > cannot be accepted, sorry. And I'm not sure I understand how can one
> > technically enforce GPL compliance in FFI-style loading of arbitrary
> > shared libraries.  The only idea that comes to mind is allow-list of
> > known libraries, the way we do in sqlite.c, but I'm not sure that
> > method is scalable to the basically infinite world of arbitrary
> > libraries.
> 
> On the topic of what would be acceptable for an FFI, wouldn't something
> akin to what's done for modules be sufficient ?  I.e., have the users of
> the interface explicitly state that they are compliant.
> 
> It would scale better than an allow-list.  IIUC, Arthur mentioned this
> in another thread.  If this wouldn't be sufficient for an FFI, could you
> please elaborate on why that's the case ?

What exactly are you suggesting?  IOW, please describe what you have
in mind in more detail, because I don't think I understand it.

> The question here isn't about what does or does not constitute GPL
> compliance, but what Emacs, as a project, is trying to ensure when it
> comes to such compliance.

It tries to ensure that Emacs is not used as front-end for non-free
software.

Re: as for Calc and the math library

[Suhail Singh]

Eli Zaretskii <eliz@gnu.org> writes:

>> On the topic of what would be acceptable for an FFI, wouldn't something
>> akin to what's done for modules be sufficient ?  I.e., have the users of
>> the interface explicitly state that they are compliant.
>> 
>> It would scale better than an allow-list.  IIUC, Arthur mentioned this
>> in another thread.  If this wouldn't be sufficient for an FFI, could you
>> please elaborate on why that's the case ?
>
> What exactly are you suggesting?  IOW, please describe what you have
> in mind in more detail, because I don't think I understand it.

Specifically, modify the `define-ffi-library' macro that emacs-ffi
provides.

Presently, it takes two arguments: a SYMBOL and a NAME.  I am proposing
that it be updated to take three arguments: a SYMBOL, a NAME and a
GPL-COMPATIBLE-P token.  A value of `t' would be necessary for creating
a reference to the library.

If the value of `t' would be insufficient for Emacs' standards for
encouraging GPL compliance (in case it's considered "not explicit
enough"), the third argument could instead be something like
GPL-COMPATIBILITY where the value of 'library-is-GPL-compatible would be
necessary to be able to create a reference to the library.

A further variant/extension of the above proposal is also possible.
This extension modifies both `define-ffi-library' as well as
`define-ffi-function'.  When `define-ffi-library' gets an acceptable
claim of GPL compatibility, it attaches a property (which acts as a
witness for observing the claim of GPL compatibility) to the library
symbol.  The presence of said property is then checked by
`define-ffi-function'.  If the property is missing,
`define-ffi-function' complains.

-- 
Suhail

Re: as for Calc and the math library

[Eli Zaretskii]

> From: Suhail Singh <suhailsingh247@gmail.com>
> Cc: Suhail Singh <suhailsingh247@gmail.com>,  gerd.moellmann@gmail.com,
>  nicolas@n16f.net,  arthur.miller@live.com,  emacs-devel@gnu.org
> Date: Wed, 14 Aug 2024 11:08:02 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> On the topic of what would be acceptable for an FFI, wouldn't something
> >> akin to what's done for modules be sufficient ?  I.e., have the users of
> >> the interface explicitly state that they are compliant.
> >> 
> >> It would scale better than an allow-list.  IIUC, Arthur mentioned this
> >> in another thread.  If this wouldn't be sufficient for an FFI, could you
> >> please elaborate on why that's the case ?
> >
> > What exactly are you suggesting?  IOW, please describe what you have
> > in mind in more detail, because I don't think I understand it.
> 
> Specifically, modify the `define-ffi-library' macro that emacs-ffi
> provides.
> 
> Presently, it takes two arguments: a SYMBOL and a NAME.  I am proposing
> that it be updated to take three arguments: a SYMBOL, a NAME and a
> GPL-COMPATIBLE-P token.  A value of `t' would be necessary for creating
> a reference to the library.

And if the value is not t, then the load will fail?  If so, then this
additional argument makes very little sense: you could instead say
that just by loading the library, the Lisp program which uses
emacs-ffi "declares" the loaded library to be GPL-compatible.  And we
are back where we began.

The way we do it when loading modules requires the _loaded_ library to
declare itself compatible, by exporting a symbol of a certain name.
That is an action by the library we load, not by the Lisp program
which loads it.

Re: as for Calc and the math library

[Suhail Singh]

Eli Zaretskii <eliz@gnu.org> writes:

> The way we do it when loading modules requires the _loaded_ library to
> declare itself compatible, by exporting a symbol of a certain name.
> That is an action by the library we load, not by the Lisp program
> which loads it.

Thank you for elaborating. In the context of Emacs, having a claim of
GPL compatibility is not sufficient; it also matters who is making such
a claim and asserting compliance with the license terms.

IIUC, the claim has to be made either by the library itself (the case of
modules), or by the Emacs maintainers (the case of an allow-list).
However, a claim made by the user (or a repackager) of a library is
insufficient.  Is my understanding correct?

To test my understanding of the above, as a counterfactual, if there was
a standardized way by which libraries identified their license (perhaps
by exporting a pre-determined symbol containing something like an SPDX
license identifier), that too would have been acceptable as long as
emacs-ffi checked against an allow-list of compatible-with-GPL-licenses.
Is that correct?

-- 
Suhail

Re: as for Calc and the math library

[Eli Zaretskii]

> From: Suhail Singh <suhailsingh247@gmail.com>
> Cc: Suhail Singh <suhailsingh247@gmail.com>,  gerd.moellmann@gmail.com,
>   nicolas@n16f.net,  arthur.miller@live.com,  emacs-devel@gnu.org
> Date: Wed, 14 Aug 2024 12:00:35 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > The way we do it when loading modules requires the _loaded_ library to
> > declare itself compatible, by exporting a symbol of a certain name.
> > That is an action by the library we load, not by the Lisp program
> > which loads it.
> 
> Thank you for elaborating. In the context of Emacs, having a claim of
> GPL compatibility is not sufficient; it also matters who is making such
> a claim and asserting compliance with the license terms.
> 
> IIUC, the claim has to be made either by the library itself (the case of
> modules), or by the Emacs maintainers (the case of an allow-list).
> However, a claim made by the user (or a repackager) of a library is
> insufficient.  Is my understanding correct?

Yes.

> To test my understanding of the above, as a counterfactual, if there was
> a standardized way by which libraries identified their license (perhaps
> by exporting a pre-determined symbol containing something like an SPDX
> license identifier), that too would have been acceptable as long as
> emacs-ffi checked against an allow-list of compatible-with-GPL-licenses.
> Is that correct?

Yes, I think so.

Re: as for Calc and the math library

[Emanuel Berg]

> IIUC

We cannot have ... because.

Simple functions: those are trivial, and amount to just
a bunch of 3-4 lines of Elisp each anyway; they
add complexity.

Advanced functions: they belong to proprietary and otherwise
incompatible libraries; they are too complex to write.

-- 
underground experts united
https://dataswamp.org/~incal

Sv: as for Calc and the math library

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 4461 bytes --]

> > Eli Zaretskii <eliz@gnu.org> writes:
> >
> > >> On the topic of what would be acceptable for an FFI, wouldn't something
> > >> akin to what's done for modules be sufficient ?  I.e., have the users of
> > >> the interface explicitly state that they are compliant.
> > >>
> > >> It would scale better than an allow-list.  IIUC, Arthur mentioned this
> > >> in another thread.  If this wouldn't be sufficient for an FFI, could you
> > >> please elaborate on why that's the case ?
> > >
> > > What exactly are you suggesting?  IOW, please describe what you have
> > > in mind in more detail, because I don't think I understand it.
> >
> > Specifically, modify the `define-ffi-library' macro that emacs-ffi
> > provides.
> >
> > Presently, it takes two arguments: a SYMBOL and a NAME.  I am proposing
> > that it be updated to take three arguments: a SYMBOL, a NAME and a
> > GPL-COMPATIBLE-P token.  A value of `t' would be necessary for creating
> > a reference to the library.
>
> And if the value is not t, then the load will fail?  If so, then this
> additional argument makes very little sense: you could instead say

It makes as much sense as it makes in C library. The token is basically an
agreement between Emacs developers and the user not to load (link) closed
source libraries into Emacs.

> that just by loading the library, the Lisp program which uses
> emacs-ffi "declares" the loaded library to be GPL-compatible.  And we
> are back where we began.

Yes, you could. It would just completely remove the barrier. However,
the token is an explicit acknowledgment of Emacs policy and license terms, by
the person who loads the library into Emacs.

> The way we do it when loading modules requires the _loaded_ library to
> declare itself compatible, by exporting a symbol of a certain name.
> That is an action by the library we load, not by the Lisp program
> which loads it.

True. But as you said yourself, a malicious user can easily cicrumvent it, even
in C and there is nothing we can do to prevent them other than possibly taking
legal actions against them.

If some company or a state would use Emacs or any other GNU program, as a
front-end to closed-source software, there is very little one can do
technically. It is only the license and the agreement that actually protects it.
________________________________
Från: Eli Zaretskii <eliz@gnu.org>
Skickat: den 14 augusti 2024 17:31
Till: Suhail Singh <suhailsingh247@gmail.com>
Kopia: gerd.moellmann@gmail.com <gerd.moellmann@gmail.com>; nicolas@n16f.net <nicolas@n16f.net>; arthur.miller@live.com <arthur.miller@live.com>; emacs-devel@gnu.org <emacs-devel@gnu.org>
Ämne: Re: as for Calc and the math library

> From: Suhail Singh <suhailsingh247@gmail.com>
> Cc: Suhail Singh <suhailsingh247@gmail.com>,  gerd.moellmann@gmail.com,
>  nicolas@n16f.net,  arthur.miller@live.com,  emacs-devel@gnu.org
> Date: Wed, 14 Aug 2024 11:08:02 -0400
>
> Eli Zaretskii <eliz@gnu.org> writes:
>
> >> On the topic of what would be acceptable for an FFI, wouldn't something
> >> akin to what's done for modules be sufficient ?  I.e., have the users of
> >> the interface explicitly state that they are compliant.
> >>
> >> It would scale better than an allow-list.  IIUC, Arthur mentioned this
> >> in another thread.  If this wouldn't be sufficient for an FFI, could you
> >> please elaborate on why that's the case ?
> >
> > What exactly are you suggesting?  IOW, please describe what you have
> > in mind in more detail, because I don't think I understand it.
>
> Specifically, modify the `define-ffi-library' macro that emacs-ffi
> provides.
>
> Presently, it takes two arguments: a SYMBOL and a NAME.  I am proposing
> that it be updated to take three arguments: a SYMBOL, a NAME and a
> GPL-COMPATIBLE-P token.  A value of `t' would be necessary for creating
> a reference to the library.

And if the value is not t, then the load will fail?  If so, then this
additional argument makes very little sense: you could instead say
that just by loading the library, the Lisp program which uses
emacs-ffi "declares" the loaded library to be GPL-compatible.  And we
are back where we began.

The way we do it when loading modules requires the _loaded_ library to
declare itself compatible, by exporting a symbol of a certain name.
That is an action by the library we load, not by the Lisp program
which loads it.

[-- Attachment #2: Type: text/html, Size: 10787 bytes --]

Re: as for Calc and the math library

[Eli Zaretskii]

> From: arthur miller <arthur.miller@live.com>
> CC: "gerd.moellmann@gmail.com" <gerd.moellmann@gmail.com>, "nicolas@n16f.net"
> 	<nicolas@n16f.net>, "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> Date: Thu, 15 Aug 2024 05:00:06 +0000
> 
> > > Specifically, modify the `define-ffi-library' macro that emacs-ffi
> > > provides.
> > >
> > > Presently, it takes two arguments: a SYMBOL and a NAME.  I am proposing
> > > that it be updated to take three arguments: a SYMBOL, a NAME and a
> > > GPL-COMPATIBLE-P token.  A value of `t' would be necessary for creating
> > > a reference to the library.
> >
> > And if the value is not t, then the load will fail?  If so, then this
> > additional argument makes very little sense: you could instead say
> 
> It makes as much sense as it makes in C library. The token is basically an
> agreement between Emacs developers and the user not to load (link) closed
> source libraries into Emacs.

No.  What we need is the declaration _by_the_library_ being loaded
that it complies.  It is not an agreement between Emacs and the
library, it's a _requirement_ on our part that only the library itself
can fulfill.

> > that just by loading the library, the Lisp program which uses
> > emacs-ffi "declares" the loaded library to be GPL-compatible.  And we
> > are back where we began.
> 
> Yes, you could. It would just completely remove the barrier. However,
> the token is an explicit acknowledgment of Emacs policy and license terms, by
> the person who loads the library into Emacs.

No, see above.

> > The way we do it when loading modules requires the _loaded_ library to
> > declare itself compatible, by exporting a symbol of a certain name.
> > That is an action by the library we load, not by the Lisp program
> > which loads it.
> 
> True. But as you said yourself, a malicious user can easily cicrumvent it, even
> in C and there is nothing we can do to prevent them other than possibly taking
> legal actions against them.

Malicious agents can do all kinds of bad and even unlawful things, but
we are not under any obligations to cater to them.  They can always
change the Emacs source code to do whatever they like, but why do you
expect _us_ to do it for them, or make it easier for them to do it?

> If some company or a state would use Emacs or any other GNU program, as a
> front-end to closed-source software, there is very little one can do
> technically. It is only the license and the agreement that actually protects it.

That's a separate issue, although it could be related.  What is at
stake here is the fact that we don't want to encourage use of Emacs as
such a front-end.  That policy of the project is not new, it has many
different expressions, as already mentioned in this thread.  We don't
mention non-free software or fonts in our documentation, we don't
install changes that specifically favor or are designed to support
non-free programs, we don't link against non-free libraries except if
they are system libraries, we don't install functionality enhancements
on non-free systems if there's no equivalent on free systems, etc.
etc.  So I wonder what is this particular fuss all about.  Its
only effect is to waste maintainers' time and energy on replying to
these posts, and I very much doubt this was the intent.  So could we
please stop this Nth instance of the same discussions?  I cannot
change the project policies even if I wanted, because I promised to
uphold them when Richard nominated me.

Sv: as for Calc and the math library

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 9547 bytes --]

>> > > Specifically, modify the `define-ffi-library' macro that emacs-ffi
>> > > provides.
>> > >
>> > > Presently, it takes two arguments: a SYMBOL and a NAME.  I am proposing
>> > > that it be updated to take three arguments: a SYMBOL, a NAME and a
>> > > GPL-COMPATIBLE-P token.  A value of `t' would be necessary for creating
>> > > a reference to the library.
>> >
>> > And if the value is not t, then the load will fail?  If so, then this
>> > additional argument makes very little sense: you could instead say
>>
>> It makes as much sense as it makes in C library. The token is basically an
>> agreement between Emacs developers and the user not to load (link) closed
>> source libraries into Emacs.
>
>No.  What we need is the declaration _by_the_library_ being loaded
>that it complies.  It is not an agreement between Emacs and the
>library, it's a _requirement_ on our part that only the library itself
>can fulfill.

I have seen your previous answers to others, and I do understand the point you
make, but I think the argument fails because a module can be just a proxy to a
non-free library. If I can put it in other words, I wonder if it is a "false
sense of security", because it is a guarantee by the person who writes the C
module, not the 3rd party library they link to. Of course they are violating the
license if they link to a non-free library, but that we are agreeing about we
can't prevent.

>> > that just by loading the library, the Lisp program which uses
>> > emacs-ffi "declares" the loaded library to be GPL-compatible.  And we
>> > are back where we began.
>>
>> Yes, you could. It would just completely remove the barrier. However,
>> the token is an explicit acknowledgment of Emacs policy and license terms, by
>> the person who loads the library into Emacs.
>
>No, see above.
>
>> > The way we do it when loading modules requires the _loaded_ library to
>> > declare itself compatible, by exporting a symbol of a certain name.
>> > That is an action by the library we load, not by the Lisp program
>> > which loads it.
>>
>> True. But as you said yourself, a malicious user can easily cicrumvent it,
>> even
>> in C and there is nothing we can do to prevent them other than possibly taking
>> legal actions against them.
>
>Malicious agents can do all kinds of bad and even unlawful things, but
>we are not under any obligations to cater to them.  They can always
>change the Emacs source code to do whatever they like, but why do you

Exactly.

>expect _us_ to do it for them, or make it easier for them to do it?

No, of course I don't expect that from you. My point was (is?) that it is a
trade-off between convinience for Emacs users who would use it as intendended,
only with free software, vs the inconvience it causes.

If you look at some other projects which are more interesting than Emacs to use
as front-ends for non-free software, do we see influx of companies misusing
them? I am thinking of Gtk, KDE/Gnome libs and Libre/OpenOffice for example. All
of those projects could be used to plug-in proprietary software, yet, there are
not so many cases, if any? Even Qt which comes as a dual licensed does not seem
to be misused that way. Perhaps I am just uninformed?

You have mentioned llvm, but I am not familiar enough with them. As I understand
they are financied by Apple, Nvidia and few other big tech companies with
explicit goal to have an alternative to GCC just for the more permissive
licensing.

>> If some company or a state would use Emacs or any other GNU program, as a
>> front-end to closed-source software, there is very little one can do
>> technically. It is only the license and the agreement that actually protects
>> it.
>
>That's a separate issue, although it could be related.  What is at
>stake here is the fact that we don't want to encourage use of Emacs as
>such a front-end.  That policy of the project is not new, it has many
>different expressions, as already mentioned in this thread.  We don't
>mention non-free software or fonts in our documentation, we don't
>install changes that specifically favor or are designed to support
>non-free programs, we don't link against non-free libraries except if
>they are system libraries, we don't install functionality enhancements
>on non-free systems if there's no equivalent on free systems, etc.
>etc.  So I wonder what is this particular fuss all about.  Its

I said it in the beginning: sometimes it is useful to revisit policies
when circumstancies have changed.

>only effect is to waste maintainers' time and energy on replying to
>these posts, and I very much doubt this was the intent.  So could we

Have I ever trolled you before? Why would I want to waste your time and energy?
I am certainly the last person in the world to do that, and you should know that.
I understand it takes time and energy, and I appologize for that, but it is
useful to learn why you consider FFI unnaceptable, and to hear the arguments
from the first hand. For the record: I have never brought up this issue before.

>please stop this Nth instance of the same discussions?  I cannot
>change the project policies even if I wanted, because I promised to
>uphold them when Richard nominated me.

I wouldn't expect you to change policy on your own neither. I did hope for RMS,
you and other experienced developers to perhaps take a look at the history,
perhaps at other software and how Emacs development could be, and reflect over
the policy if it is still worth keeping or perhaps it can be changed.
________________________________
Från: Eli Zaretskii <eliz@gnu.org>
Skickat: den 15 augusti 2024 09:02
Till: arthur miller <arthur.miller@live.com>
Kopia: suhailsingh247@gmail.com <suhailsingh247@gmail.com>; gerd.moellmann@gmail.com <gerd.moellmann@gmail.com>; nicolas@n16f.net <nicolas@n16f.net>; emacs-devel@gnu.org <emacs-devel@gnu.org>
Ämne: Re: as for Calc and the math library

> From: arthur miller <arthur.miller@live.com>
> CC: "gerd.moellmann@gmail.com" <gerd.moellmann@gmail.com>, "nicolas@n16f.net"
>        <nicolas@n16f.net>, "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> Date: Thu, 15 Aug 2024 05:00:06 +0000
>
> > > Specifically, modify the `define-ffi-library' macro that emacs-ffi
> > > provides.
> > >
> > > Presently, it takes two arguments: a SYMBOL and a NAME.  I am proposing
> > > that it be updated to take three arguments: a SYMBOL, a NAME and a
> > > GPL-COMPATIBLE-P token.  A value of `t' would be necessary for creating
> > > a reference to the library.
> >
> > And if the value is not t, then the load will fail?  If so, then this
> > additional argument makes very little sense: you could instead say
>
> It makes as much sense as it makes in C library. The token is basically an
> agreement between Emacs developers and the user not to load (link) closed
> source libraries into Emacs.

No.  What we need is the declaration _by_the_library_ being loaded
that it complies.  It is not an agreement between Emacs and the
library, it's a _requirement_ on our part that only the library itself
can fulfill.

> > that just by loading the library, the Lisp program which uses
> > emacs-ffi "declares" the loaded library to be GPL-compatible.  And we
> > are back where we began.
>
> Yes, you could. It would just completely remove the barrier. However,
> the token is an explicit acknowledgment of Emacs policy and license terms, by
> the person who loads the library into Emacs.

No, see above.

> > The way we do it when loading modules requires the _loaded_ library to
> > declare itself compatible, by exporting a symbol of a certain name.
> > That is an action by the library we load, not by the Lisp program
> > which loads it.
>
> True. But as you said yourself, a malicious user can easily cicrumvent it, even
> in C and there is nothing we can do to prevent them other than possibly taking
> legal actions against them.

Malicious agents can do all kinds of bad and even unlawful things, but
we are not under any obligations to cater to them.  They can always
change the Emacs source code to do whatever they like, but why do you
expect _us_ to do it for them, or make it easier for them to do it?

> If some company or a state would use Emacs or any other GNU program, as a
> front-end to closed-source software, there is very little one can do
> technically. It is only the license and the agreement that actually protects it.

That's a separate issue, although it could be related.  What is at
stake here is the fact that we don't want to encourage use of Emacs as
such a front-end.  That policy of the project is not new, it has many
different expressions, as already mentioned in this thread.  We don't
mention non-free software or fonts in our documentation, we don't
install changes that specifically favor or are designed to support
non-free programs, we don't link against non-free libraries except if
they are system libraries, we don't install functionality enhancements
on non-free systems if there's no equivalent on free systems, etc.
etc.  So I wonder what is this particular fuss all about.  Its
only effect is to waste maintainers' time and energy on replying to
these posts, and I very much doubt this was the intent.  So could we
please stop this Nth instance of the same discussions?  I cannot
change the project policies even if I wanted, because I promised to
uphold them when Richard nominated me.

[-- Attachment #2: Type: text/html, Size: 22892 bytes --]

Re: as for Calc and the math library

[Eli Zaretskii]

> From: arthur miller <arthur.miller@live.com>
> CC: "suhailsingh247@gmail.com" <suhailsingh247@gmail.com>,
> 	"gerd.moellmann@gmail.com" <gerd.moellmann@gmail.com>, "nicolas@n16f.net"
> 	<nicolas@n16f.net>, "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> Date: Thu, 15 Aug 2024 20:09:56 +0000
> 
> >No.  What we need is the declaration _by_the_library_ being loaded
> >that it complies.  It is not an agreement between Emacs and the
> >library, it's a _requirement_ on our part that only the library itself
> >can fulfill.
> 
> I have seen your previous answers to others, and I do understand the point you
> make, but I think the argument fails because a module can be just a proxy to a
> non-free library.

A module that declares GPL compatibility and loads non-free library is
either lying or (if the library is distributed with the module, as I'd
expect) in clear violation of the GPL.

> >expect _us_ to do it for them, or make it easier for them to do it?
> 
> No, of course I don't expect that from you. My point was (is?) that it is a
> trade-off between convinience for Emacs users who would use it as intendended,
> only with free software, vs the inconvience it causes.

Yes, it's a trade-off.  The GNU Project decided where to take this
trade-off, and I promised to uphold that policy.  So can we please
stop this futile discussion, which just went one more circle?

> If you look at some other projects which are more interesting than Emacs to use
> as front-ends for non-free software, do we see influx of companies misusing
> them? I am thinking of Gtk, KDE/Gnome libs and Libre/OpenOffice for example. All
> of those projects could be used to plug-in proprietary software, yet, there are
> not so many cases, if any? Even Qt which comes as a dual licensed does not seem
> to be misused that way. Perhaps I am just uninformed?
> 
> You have mentioned llvm, but I am not familiar enough with them. As I understand
> they are financied by Apple, Nvidia and few other big tech companies with
> explicit goal to have an alternative to GCC just for the more permissive
> licensing.

Once again, please take these off-topic discussions elsewhere.

> I said it in the beginning: sometimes it is useful to revisit policies
> when circumstancies have changed.

You are talking to the wrong guy about revisiting policies.  As
already explained more than once.

we need *modularity* [last problem] (was: Re: as for Calc and the math library)

[Emanuel Berg]

arthur miller wrote:

> I have seen your previous answers to others, and I do
> understand the point you make, but I think the argument
> fails because a module can be just a proxy to
> a non-free library.

Emacs is notoriously not modular. Everything is everywhere,
you seem to have everything but if fact you have nothing.
There is so much stuff you can't find the basics stuff that
isn't there.

Here take a look: what does this name say?

  cl--defalias

Ah, that's a local function in the Common Lisp package, right?

Well, YES, but ...

It is actually an Elisp defun. And that prefix isn't really
a package qualifier, it is just a convention, part of
the name. And the double dash, likewise just a convention, we
don't have package modularity or any of that kind of data
hiding, abstraction, encapsulation ... what is it more called?
Well, modularity!

In Emacs everything is intermixed, Elisp-Emacs,
non-interactive/interactive, data/data processing/the
interface (buffer, buffer, and buffer), what is a file, what
is a package, what is a library, and what is a program - no
one can tell.

Emacs Lisp isn't is fun, but absolutely not any kind
of powerhouse. It is inconsistent in itself and on top of that
all those zillion interfaces, cl-, seq-, slime- (yes, had to
use that today, for `slime-transpose-lists').

It is pretty fast to get something going but complexity
skyrockets.

What we should do, in general: modularity! clear divisions, in
particular, libraries for everything, including those that provide basic
stuff in a complete and consistent way.

We should get real package modularity, transform the prefixes
into real package qualifiers, and throw away the ugly
local ones.

With Elisp, optimize everything for development speed and
convenience, have consistent interfaces as much as possible,
acknowledge that Elisp was underpowered, which is why all the
interfaces came, now, core Elisp and the interfaces should
make peace, one should identify what it is exactly, what are
the to 10 things that people use with cl-, pcase-, seq- etc,
and bring them into a new core Elisp, and the few CL experts
can still use all the rest.

Now, with native compilation we have speed - we have power
with the interfaces if one could harvest it, which one
currently - well, it is very slow and difficult, anyway - the
very, very last spet into maturity and getting out of the
Elisp ghetto (and getting visitors!) are modularity, modern
software principles - by modularity I mean real modularity,
based on technology - not silly conventions - I forgot where
I was - the last step is modularity in general, and libraries
in particular.

Clear cut interfaces. Get away with the ugly prefixes and
error--prone conventions that don't even do anything.

Elisp 3.0! \o/

PS. This is the problem. This series is completed, I said
    I would write 10, I wrote 4. Here are the other parts.
    It is all based on the same theme. Modularity,
    consistency, libraries, modern software practises.
    
    https://lists.gnu.org/archive/html/emacs-devel/2024-08/msg00154.html
    https://lists.gnu.org/archive/html/emacs-devel/2024-08/msg00380.html
    https://lists.gnu.org/archive/html/emacs-devel/2024-08/msg00388.html

PPS. So who is working on real modularity based on the package
     with real public and local functions as we speak? I know
     there are always some guy - at least - working on such
     a basic idea. I know some way. And I know some day.

-- 
underground experts united
https://dataswamp.org/~incal

first-is (3 versions, Elisp hangup) (was: Re: we need *modularity* [last problem])

[Emanuel Berg]

Here is one example. How to check if a word starts with
something? E.g., a "-" (not a dash anymore but
a "hyphen-minus", haha) then that should denote something
else, a line in this case, such situations.

How do you check that? And it cannot break for data that is
(not stringp), should just be nil then

I was doing something else, now I did this instead. I'm sure
you have a much better method that will make me look like
a fool, but ... it is like this all the time with Elisp.

Why isn't there just a (first-is THIS ALL) and it would not
crash, just eval to nil on type incompatibilities, other than
that it would work on arbitrary data? THIS could even be
a predicate that would take FRST as the argument.

Hey ... not bad! I should write that!

See the problem? :P

(defun elpat--first-is-3 (s str)
  (and (stringp s)
       (stringp str)
       (string= s (char-to-string (seq-first str)))))

(defun elpat--first-is-2 (s str)
  (and (stringp s)
       (stringp str)
       (string= s (substring str 0 1))))

(defun elpat--first-is (s str)
  (and (cl-every #'stringp (list s str))
       #1=(string-match s str)
       (zerop #1#)))

;; (elpat--first-is   "s" "same")
;; (elpat--first-is-2 "s" "same")
;; (elpat--first-is-3 "s" "same")

;; (elpat--first-is   "s" "lame")
;; (elpat--first-is-2 "s" "lame")
;; (elpat--first-is-3 "s" "lame")

-- 
underground experts united
https://dataswamp.org/~incal

Re: first-is (3 versions, Elisp hangup) (was: Re: we need *modularity* [last problem])

[Emanuel Berg]

Oh, no!

;; (elpat--first-is   "" "same") ; t
;; (elpat--first-is-2 "" "same") ; nil
;; (elpat--first-is-3 "" "same") ; nil

;; (elpat--first-is   "s" "") ; nil
;; (elpat--first-is-2 "s" "") ; crash
;; (elpat--first-is-3 "s" "") ; crash

;; (elpat--first-is   "" "") ; t
;; (elpat--first-is-2 "" "") ; crash
;; (elpat--first-is-3 "" "") ; crash

But hang on, I have some thing for that as well. Here:

(defun string-data-p (str)
  (and (stringp str) (not (string= str "")) str))

Heh, see where this is going?

It is clear to me that Elisp isn't a language designed or
optimized for devel speed and productivity.

Lisp could be (I _think_, much more so anyway) - but I guess
it won't happen.

> (defun elpat--first-is-3 (s str)
>   (and (stringp s)
>        (stringp str)
>        (string= s (char-to-string (seq-first str)))))
>
> (defun elpat--first-is-2 (s str)
>   (and (stringp s)
>        (stringp str)
>        (string= s (substring str 0 1))))
>
> (defun elpat--first-is (s str)
>   (and (cl-every #'stringp (list s str))
>        #1=(string-match s str)
>        (zerop #1#)))
>
> ;; (elpat--first-is   "s" "same")
> ;; (elpat--first-is-2 "s" "same")
> ;; (elpat--first-is-3 "s" "same")
>
> ;; (elpat--first-is   "s" "lame")
> ;; (elpat--first-is-2 "s" "lame")
> ;; (elpat--first-is-3 "s" "lame")

-- 
underground experts united
https://dataswamp.org/~incal

Re: first-is (3 versions, Elisp hangup) (was: Re: we need *modularity* [last problem])

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Fri, 16 Aug 2024 11:35:23 +0200
> 
> Here is one example. How to check if a word starts with
> something?

By using string-prefix-p.  Which you could have found by using
"M-x shortdoc RET string RET"

Re: as for Calc and the math library

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >No.  What we need is the declaration _by_the_library_ being loaded
  > >that it complies.  It is not an agreement between Emacs and the
  > >library, it's a _requirement_ on our part that only the library itself
  > >can fulfill.

  > I have seen your previous answers to others, and I do understand
  > the point you make, but I think the argument fails because a
  > module can be just a proxy to a non-free library. If I can put it
  > in other words, I wonder if it is a "false sense of security",
  > because it is a guarantee by the person who writes the C module,
  > not the 3rd party library they link to. Of course they are
  > violating the license if they link to a non-free library, but that
  > we are agreeing about we can't prevent.

You're right, and this is a possible problem, but it raises a real
copyleft compliance issue only in a specific kind of occasion: when
someone writes such an intermediary module that loads a nonfree
library, _and distributes that intermediary module_ for such use.

Just writing such an intermediary module and using it oneself does not
raise a copyleft compliance issue, because GPLv3 says one can make any
sort of modificatoin and use it privately -- even actually copying in
nonfree code.  Where the copyleft requirement comes into pkay is for
making a modified version and distributing it to others.

So it could be good to keep an eye on the modules that people are
releasing, to make sure they do not load in other nonfree code.
If they do, the FSF should tell their distributors to stop.

Given this situation, a spcial label as a flag for "this library is
free" might be useful, We just have to remember taht such a flag label
is not proof of anything.  Its presence is only a statement (which can
be false) of intent that the module be entirely GPL3-compatible free
code and that it link in only GPL3-compatible other modules.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

as for Calc and the math library

[Christopher Dimech]

> Sent: Monday, August 19, 2024 at 4:38 AM
> From: "Richard Stallman" <rms@gnu.org>
> To: "arthur miller" <arthur.miller@live.com>
> Cc: emacs-devel@gnu.org
> Subject: Re: as for Calc and the math library
>
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > >No.  What we need is the declaration _by_the_library_ being loaded
>   > >that it complies.  It is not an agreement between Emacs and the
>   > >library, it's a _requirement_ on our part that only the library itself
>   > >can fulfill.
>
>   > I have seen your previous answers to others, and I do understand
>   > the point you make, but I think the argument fails because a
>   > module can be just a proxy to a non-free library. If I can put it
>   > in other words, I wonder if it is a "false sense of security",
>   > because it is a guarantee by the person who writes the C module,
>   > not the 3rd party library they link to. Of course they are
>   > violating the license if they link to a non-free library, but that
>   > we are agreeing about we can't prevent.
>
> You're right, and this is a possible problem, but it raises a real
> copyleft compliance issue only in a specific kind of occasion: when
> someone writes such an intermediary module that loads a nonfree
> library, _and distributes that intermediary module_ for such use.
>
> Just writing such an intermediary module and using it oneself does not
> raise a copyleft compliance issue, because GPLv3 says one can make any
> sort of modificatoin and use it privately -- even actually copying in
> nonfree code.  Where the copyleft requirement comes into pkay is for
> making a modified version and distributing it to others.
>
> So it could be good to keep an eye on the modules that people are
> releasing, to make sure they do not load in other nonfree code.
> If they do, the FSF should tell their distributors to stop.
>
> Given this situation, a spcial label as a flag for "this library is
> free" might be useful, We just have to remember taht such a flag label
> is not proof of anything.  Its presence is only a statement (which can
> be false) of intent that the module be entirely GPL3-compatible free
> code and that it link in only GPL3-compatible other modules.

Emacs could recommend and endorse specific libraries without judging or
comparing them based on any criterion other than freedom.  As is done
with the Free GNU/Linux Distributions.  Emacs itself could report a
flag when someone uses such libraries, rather than having the libraries
themselves report the information to the editor.  However GPL Compliance
would be too restrictive.  Knowing a library is free should be enough.
This way users would know what they are doing without harm.

Sv: as for Calc and the math library

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 9075 bytes --]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > >No.  What we need is the declaration _by_the_library_ being loaded
>   > >that it complies.  It is not an agreement between Emacs and the
>   > >library, it's a _requirement_ on our part that only the library itself
>   > >can fulfill.
>
>   > I have seen your previous answers to others, and I do understand
>   > the point you make, but I think the argument fails because a
>   > module can be just a proxy to a non-free library. If I can put it
>   > in other words, I wonder if it is a "false sense of security",
>   > because it is a guarantee by the person who writes the C module,
>   > not the 3rd party library they link to. Of course they are
>   > violating the license if they link to a non-free library, but that
>   > we are agreeing about we can't prevent.
>
> You're right, and this is a possible problem, but it raises a real
> copyleft compliance issue only in a specific kind of occasion: when
> someone writes such an intermediary module that loads a nonfree
> library, _and distributes that intermediary module_ for such use.
>
> Just writing such an intermediary module and using it oneself does not
> raise a copyleft compliance issue, because GPLv3 says one can make any
> sort of modificatoin and use it privately -- even actually copying in
> nonfree code.  Where the copyleft requirement comes into pkay is for
> making a modified version and distributing it to others.

Hi, and thank you very much for the clarification RMS. I saw you previous post
saying this also. I think it is useful to remind people that the GPL is about
the distribution and the rights to distribute the changed code not so much about
using the code (I think, I hope I am not sailing out of the clear water here).

> So it could be good to keep an eye on the modules that people are
> releasing, to make sure they do not load in other nonfree code.
> If they do, the FSF should tell their distributors to stop.

Yes, and this perhaps the only thing we can do, since technically it is very
easy to circumwent the license.

> Given this situation, a spcial label as a flag for "this library is
> free" might be useful, We just have to remember taht such a flag label
> is not proof of anything.  Its presence is only a statement (which can
> be false) of intent that the module be entirely GPL3-compatible free
> code and that it link in only GPL3-compatible other modules.

Indeed. I don't know how realisict, technically and resource-wise it would be to
use Elpa as a place to distribute modules and to have each library in Elpa
"signed" or "approved" by someone FSF-trusted who could ensure the GPL license
is not violated. The problem I see is that after the initial review, libraries
are pulled and build automaticly from Git sources, so that would be a constant
process, but perhaps could be automated by a script which checks for presence of
certain system/library calls (dlopen & co) and warns an approved "reviewer" to
take a look at that library? I perhaps also misunderstand how Elpa works since I
am not so familiar with Elpa implementation. Problem with automation is that a
malicious author could, at least in theory, initially commit a perfect library,
and after some time introduce malicious or GPL-incompatible code. I am thinking
of the recent XZ-incident.

Since you have chimmed in, and in the light of your clarifications, may I ask
what do you think of the difference between doing the same thing from the Lisp?
Is there any conceptual difference between loading an external library from a
Lisp library as it is from a C library?

Since you have worked a lot with Lisp(s) in your past, and as I have seen from
the transcripts of the work on the CommonLisp standard, work in which you also
took part, you seem to always been concerned with how easy/difficult constructs
are to use and understand for users. In several contributions you tried to
simplify some constructs, at least as I understand (setf comes to mind for
example, but I don't remember the details at the moment, would have to look up
for the exact reference).

IMO it would be much easier and more convenient for EmacsLisp users and
programmers to use a Lisp interface to work with modules than C. Admittedly it
lowers the technical barrier for violating the GPL license, but that technical
barrier is a hindrance only for very innexperience programmers, who probably
wouldn't be able to use FFI from Lisp neither. As a layman when it comes to
licensing, I don't see the conceptual diffrence between a C library or a Lisp
library when it comes to licensing, and the very same "agreement" that should
uphold between a Lisp library writer, as there is between C library writer and
the GPL code they use.

If you can reflect on the conceptual difference I would be very thankful. I
promised to Eli I won't push for FFI, and I am not doing it either, but I am
interested to learn why it would be different to load a library via Lisp from
loading it from C, more than technical difference of doing it via Lisp is much
easier and convenient. If anyone, you have done lots of work in both Lisp and
C. I think it is very interesting to hear your opinion on the matter.

I hope Eli don't get too angry on me for asking you this, since I understand he
considers this off-topic. I understand it takes time and energy from you, when
we talk about this matters, and I appologize for that. This is also an
opportunity to hear what RMS opinion is and the reason why FFI is bad in
EmacsLisp, if he would be kind to explaing and clarify that one. I am sure I am
not the only one who will find this useful to hear. Perhaps I am just an idiot,
but I am not an enemy, and I don't mean harm as you seem to believe (since you
epressed the concern that I am just trolling to take your time). If you want to
send me the warning, or ban me from this list, so be it.

PS: I hope I didn't sent this twice. In a recent couple of weeks I have a problem with GNUs.
It sometimes works and sometimes not. Several of these mails I had to copy over to a
different client, happend this time, while some went out without problems from Emacs.
________________________________
Från: Richard Stallman <rms@gnu.org>
Skickat: den 18 augusti 2024 18:38
Till: arthur miller <arthur.miller@live.com>
Kopia: emacs-devel@gnu.org <emacs-devel@gnu.org>
Ämne: Re: as for Calc and the math library

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >No.  What we need is the declaration _by_the_library_ being loaded
  > >that it complies.  It is not an agreement between Emacs and the
  > >library, it's a _requirement_ on our part that only the library itself
  > >can fulfill.

  > I have seen your previous answers to others, and I do understand
  > the point you make, but I think the argument fails because a
  > module can be just a proxy to a non-free library. If I can put it
  > in other words, I wonder if it is a "false sense of security",
  > because it is a guarantee by the person who writes the C module,
  > not the 3rd party library they link to. Of course they are
  > violating the license if they link to a non-free library, but that
  > we are agreeing about we can't prevent.

You're right, and this is a possible problem, but it raises a real
copyleft compliance issue only in a specific kind of occasion: when
someone writes such an intermediary module that loads a nonfree
library, _and distributes that intermediary module_ for such use.

Just writing such an intermediary module and using it oneself does not
raise a copyleft compliance issue, because GPLv3 says one can make any
sort of modificatoin and use it privately -- even actually copying in
nonfree code.  Where the copyleft requirement comes into pkay is for
making a modified version and distributing it to others.

So it could be good to keep an eye on the modules that people are
releasing, to make sure they do not load in other nonfree code.
If they do, the FSF should tell their distributors to stop.

Given this situation, a spcial label as a flag for "this library is
free" might be useful, We just have to remember taht such a flag label
is not proof of anything.  Its presence is only a statement (which can
be false) of intent that the module be entirely GPL3-compatible free
code and that it link in only GPL3-compatible other modules.

--
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)



[-- Attachment #2: Type: text/html, Size: 21676 bytes --]

Re: as for Calc and the math library

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > So it could be good to keep an eye on the modules that people are
  > > releasing, to make sure they do not load in other nonfree code.
  > > If they do, the FSF should tell their distributors to stop.

  > Yes, and this perhaps the only thing we can do, since technically it is very
  > easy to circumwent the license.

The crucial preparation for this is to make sure we have effective,
practical control over what modules are being visibly distribued by
others.  that means teachng a substantial part of the community to
understand that this is a kind of deception, and to tell us if they
spot one.

We could also think of some other measures, such as including in Emacs
a list of hash codes of such deceptive modules, and code to refuse
to load them and explain why.

Of course, this would not actually _prevent_ users from loading those
modules.  A user could delete the pertinent item from the list, then
go ahead.  But I expect this would be fairly effective at leading the
community away from doing that.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: as for Calc and the math library

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Since you have chimmed in, and in the light of your
  > clarifications, may I ask what do you think of the difference
  > between doing the same thing from the Lisp?  Is there any
  > conceptual difference between loading an external library from a
  > Lisp library as it is from a C library?

In principle, they are the same issue; however, specific details of
the situation might affect the conclusion, and the choice of language
might make a difference to what those details are.  So I can't say
anything more concrete without details (and perhaps consulting a
lawyer).



-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Emacs ffi (was: Re: as for Calc and the math library)

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Suhail Singh <suhailsingh247@gmail.com>
>> Cc: Suhail Singh <suhailsingh247@gmail.com>,  gerd.moellmann@gmail.com,
>>  nicolas@n16f.net,  arthur.miller@live.com,  emacs-devel@gnu.org
>> Date: Wed, 14 Aug 2024 11:08:02 -0400
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> On the topic of what would be acceptable for an FFI, wouldn't something
>> >> akin to what's done for modules be sufficient ?  I.e., have the users of
>> >> the interface explicitly state that they are compliant.
>> >> 
>> >> It would scale better than an allow-list.  IIUC, Arthur mentioned this
>> >> in another thread.  If this wouldn't be sufficient for an FFI, could you
>> >> please elaborate on why that's the case ?
>> >
>> > What exactly are you suggesting?  IOW, please describe what you have
>> > in mind in more detail, because I don't think I understand it.
>> 
>> Specifically, modify the `define-ffi-library' macro that emacs-ffi
>> provides.
>> 
>> Presently, it takes two arguments: a SYMBOL and a NAME.  I am proposing
>> that it be updated to take three arguments: a SYMBOL, a NAME and a
>> GPL-COMPATIBLE-P token.  A value of `t' would be necessary for creating
>> a reference to the library.
>
> And if the value is not t, then the load will fail?  If so, then this
> additional argument makes very little sense: you could instead say
> that just by loading the library, the Lisp program which uses
> emacs-ffi "declares" the loaded library to be GPL-compatible.  And we
> are back where we began.

Agree

> The way we do it when loading modules requires the _loaded_ library to
> declare itself compatible, by exporting a symbol of a certain name.
> That is an action by the library we load, not by the Lisp program
> which loads it.

But we could do the same for our hypothetical ffi machinery, that is:
`define-ffi-library' could fail if the shared library is not exporting
the predetermined symbol we expect no?

  Andrea

Re: Emacs ffi (was: Re: as for Calc and the math library)

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: Suhail Singh <suhailsingh247@gmail.com>,  gerd.moellmann@gmail.com,
>   nicolas@n16f.net,  arthur.miller@live.com,  emacs-devel@gnu.org
> Date: Thu, 15 Aug 2024 05:31:49 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > The way we do it when loading modules requires the _loaded_ library to
> > declare itself compatible, by exporting a symbol of a certain name.
> > That is an action by the library we load, not by the Lisp program
> > which loads it.
> 
> But we could do the same for our hypothetical ffi machinery, that is:
> `define-ffi-library' could fail if the shared library is not exporting
> the predetermined symbol we expect no?

Of course.  But how many such libraries do that?  You are in effect
talking about libraries specifically written or adapted for this
hypothetical FFI machinery that is Emacs- or at least GNU-specific.
Whereas the main motivation behind the request is to allow loading
arbitrary libraries out there that already exist and definitely don't
export any such symbols.  At the very least, the interested individual
will have to fork the library, verify it has a GPL-compatible license,
modify its code to export such a symbol, and then rebuild it.

So, while this arrangement is definitely possible and okay for Emacs,
I very much doubt that it will be practical enough to be of use.
Nevertheless, if someone submits such an implementation of such FFI
machinery, I see no obstacles to accepting it (except the usual: clean
code, good documentation, etc.).

Re: Emacs ffi

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <acorallo@gnu.org>
>> Cc: Suhail Singh <suhailsingh247@gmail.com>,  gerd.moellmann@gmail.com,
>>   nicolas@n16f.net,  arthur.miller@live.com,  emacs-devel@gnu.org
>> Date: Thu, 15 Aug 2024 05:31:49 -0400
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > The way we do it when loading modules requires the _loaded_ library to
>> > declare itself compatible, by exporting a symbol of a certain name.
>> > That is an action by the library we load, not by the Lisp program
>> > which loads it.
>> 
>> But we could do the same for our hypothetical ffi machinery, that is:
>> `define-ffi-library' could fail if the shared library is not exporting
>> the predetermined symbol we expect no?
>
> Of course.  But how many such libraries do that?

Dunno, ATM very few if not zero I guess.

> You are in effect
> talking about libraries specifically written or adapted for this
> hypothetical FFI machinery that is Emacs- or at least GNU-specific.
> Whereas the main motivation behind the request is to allow loading
> arbitrary libraries out there that already exist and definitely don't
> export any such symbols.  At the very least, the interested individual
> will have to fork the library, verify it has a GPL-compatible license,
> modify its code to export such a symbol, and then rebuild it.

Maybe over time libraries interested to e used by Emacs could accept the
simple patch necessary to define end export the symbol?

> So, while this arrangement is definitely possible and okay for Emacs,
> I very much doubt that it will be practical enough to be of use.
> Nevertheless, if someone submits such an implementation of such FFI
> machinery, I see no obstacles to accepting it (except the usual: clean
> code, good documentation, etc.).


Okay thanks, we are on the same page on this.


  Andrea

Re: Emacs ffi

[Andrea Corallo]

Christopher Dimech <dimech@gmx.com> writes:

>> Sent: Friday, August 16, 2024 at 8:32 AM
>> From: "Andrea Corallo" <acorallo@gnu.org>
>> To: "Eli Zaretskii" <eliz@gnu.org>
>> Cc: suhailsingh247@gmail.com, gerd.moellmann@gmail.com, nicolas@n16f.net, arthur.miller@live.com, emacs-devel@gnu.org
>> Subject: Re: Emacs ffi
>>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>> >> From: Andrea Corallo <acorallo@gnu.org>
>> >> Cc: Suhail Singh <suhailsingh247@gmail.com>,  gerd.moellmann@gmail.com,
>> >>   nicolas@n16f.net,  arthur.miller@live.com,  emacs-devel@gnu.org
>> >> Date: Thu, 15 Aug 2024 05:31:49 -0400
>> >>
>> >> Eli Zaretskii <eliz@gnu.org> writes:
>> >>
>> >> > The way we do it when loading modules requires the _loaded_ library to
>> >> > declare itself compatible, by exporting a symbol of a certain name.
>> >> > That is an action by the library we load, not by the Lisp program
>> >> > which loads it.
>> >>
>> >> But we could do the same for our hypothetical ffi machinery, that is:
>> >> `define-ffi-library' could fail if the shared library is not exporting
>> >> the predetermined symbol we expect no?
>> >
>> > Of course.  But how many such libraries do that?
>>
>> Dunno, ATM very few if not zero I guess.
>
> If there are no existing libraries that export the required symbol, it
> raises legitimate concerns about the feasibility and utility of the
> approach. Essentially, if the mechanism put in place is not supported by
> any available libraries, then the mechanism is ineffective and overly
> idealistic.
>
> There exists a large disconnect between theoretical compliance and
> practical usability. If the requirements for compliance are so stringent
> or specific that no existing libraries can meet them, then it could be
> argued that the approach is not grounded in the reality of software
> development.

I disagree, this is normal evolution for software ex: every time a new
programming language is created (or a new feature is added to it)
compilers implement the related support, even if no programs are using
that language or that new extension at present.  Is this a large
disconnect between theoretical compliance and practical usability?  No,
is just that users will come later.

If Emacs requires the symbol maybe compatible libraries will just export
it afterwards (given the cost is close to zero).

  Andrea

Re: Emacs ffi

[Christopher Dimech]

> Sent: Saturday, August 17, 2024 at 8:07 AM
> From: "Andrea Corallo" <acorallo@gnu.org>
> To: "Christopher Dimech" <dimech@gmx.com>
> Cc: suhailsingh247@gmail.com, gerd.moellmann@gmail.com, nicolas@n16f.net, arthur.miller@live.com, emacs-devel@gnu.org, "Eli Zaretskii" <eliz@gnu.org>
> Subject: Re: Emacs ffi
>
> Christopher Dimech <dimech@gmx.com> writes:
> 
> >> Sent: Friday, August 16, 2024 at 8:32 AM
> >> From: "Andrea Corallo" <acorallo@gnu.org>
> >> To: "Eli Zaretskii" <eliz@gnu.org>
> >> Cc: suhailsingh247@gmail.com, gerd.moellmann@gmail.com, nicolas@n16f.net, arthur.miller@live.com, emacs-devel@gnu.org
> >> Subject: Re: Emacs ffi
> >>
> >> Eli Zaretskii <eliz@gnu.org> writes:
> >>
> >> >> From: Andrea Corallo <acorallo@gnu.org>
> >> >> Cc: Suhail Singh <suhailsingh247@gmail.com>,  gerd.moellmann@gmail.com,
> >> >>   nicolas@n16f.net,  arthur.miller@live.com,  emacs-devel@gnu.org
> >> >> Date: Thu, 15 Aug 2024 05:31:49 -0400
> >> >>
> >> >> Eli Zaretskii <eliz@gnu.org> writes:
> >> >>
> >> >> > The way we do it when loading modules requires the _loaded_ library to
> >> >> > declare itself compatible, by exporting a symbol of a certain name.
> >> >> > That is an action by the library we load, not by the Lisp program
> >> >> > which loads it.
> >> >>
> >> >> But we could do the same for our hypothetical ffi machinery, that is:
> >> >> `define-ffi-library' could fail if the shared library is not exporting
> >> >> the predetermined symbol we expect no?
> >> >
> >> > Of course.  But how many such libraries do that?
> >>
> >> Dunno, ATM very few if not zero I guess.
> >
> > If there are no existing libraries that export the required symbol, it
> > raises legitimate concerns about the feasibility and utility of the
> > approach. Essentially, if the mechanism put in place is not supported by
> > any available libraries, then the mechanism is ineffective and overly
> > idealistic.
> >
> > There exists a large disconnect between theoretical compliance and
> > practical usability. If the requirements for compliance are so stringent
> > or specific that no existing libraries can meet them, then it could be
> > argued that the approach is not grounded in the reality of software
> > development.
> 
> I disagree, this is normal evolution for software ex: every time a new
> programming language is created (or a new feature is added to it)
> compilers implement the related support, even if no programs are using
> that language or that new extension at present.  Is this a large
> disconnect between theoretical compliance and practical usability?  No,
> is just that users will come later.
> 
> If Emacs requires the symbol maybe compatible libraries will just export
> it afterwards (given the cost is close to zero). - Andrea

It is imposing an additional requirement that isn't typically necessary 
in most other environments.

Typically, mathematical libraries are designed to be widely applicable
and used across various platforms and applications. Requiring these
libraries to export a specific symbol or be designed with editor
compatibility in mind is an unnecessary hurdle. Developers of libraries
do not prioritize compatibility with editors like Emacs.

To overcome the hurdle that Emacs is bringing upon itself, of requiring
GPL compliance for mathematical libraries, it should provide its own
built-in library as suggested by E. Berg. It is the pragmatic way to do
it.  I disagreed with Berg initially, but I can now understand his
frustrations with the whole emacs design.

Some free software simply cannot be used with GPL-licensed software like 
Emacs without violating the terms of one or both licenses.  Requiring GPL 
compliance for integration is banal.  Particularly when Emacs is not the 
one providing the library.  It will be the user who will be using.  One 
simply cannot accuse a user to breaking any license if the library is not
GPL compliant.  The landscape of free software licenses is diverse, with 
many different licenses with many not compatible with each other. You’d 
have to put every teenager in the world in jail, and you can’t do that ! 

But this message does not resonate within the minds of the core emacs maintainers, 
with the trend continuing for many years to come.  Some believe that everybody 
in the world doesn’t get it about free software, and even that everybody in the 
world is a crook and that everybody in the world is trying to steal free software 
and make bad use of it.  I do not approve of such ideas.  Many think they know
everything about copyright infringement, but never been thrown out of court on 
a motion to dismiss.  

It's always the same, you'll get prima donna maintainers who are at the same 
level of priests, who preach a lot about licensing scenarios.  But have never 
been there, never done it, but preaching a lot about what others can do and what 
they cannot do.  Too much coercion was surely not what we wanted to apply.

As customary I get "Your message was deemed inappropriate by the moderator."
from emacs-devel-owner@gnu.org.  I wonder who it is.  Actually I wonder why
I wonder.

We need to understand how that working together purposively brings us to the 
point where everyone is not afraid of free software anymore and we are not 
worried about their complying anymore.  We are just all engaging and leading 
the task of making free software.

Re: Emacs ffi

[Eli Zaretskii]

> From: Christopher Dimech <dimech@gmx.com>
> Cc: suhailsingh247@gmail.com, gerd.moellmann@gmail.com, nicolas@n16f.net,
>  arthur.miller@live.com, emacs-devel@gnu.org, Eli Zaretskii <eliz@gnu.org>
> Date: Fri, 16 Aug 2024 23:21:48 +0200
> 
> 
> But this message does not resonate within the minds of the core emacs maintainers, 
> with the trend continuing for many years to come.  Some believe that everybody 
> in the world doesn’t get it about free software, and even that everybody in the 
> world is a crook and that everybody in the world is trying to steal free software 
> and make bad use of it.  I do not approve of such ideas.  Many think they know
> everything about copyright infringement, but never been thrown out of court on 
> a motion to dismiss.  
> 
> It's always the same, you'll get prima donna maintainers who are at the same 
> level of priests, who preach a lot about licensing scenarios.  But have never 
> been there, never done it, but preaching a lot about what others can do and what 
> they cannot do.  Too much coercion was surely not what we wanted to apply.

For the umpteenth time: the maintainers are not at liberty to change
the policies they promised to uphold when they were appointed.  If you
want to discuss these policies, please take this to the proper place.

> As customary I get "Your message was deemed inappropriate by the moderator."
> from emacs-devel-owner@gnu.org.  I wonder who it is.  Actually I wonder why
> I wonder.

You were asked, several times, to take this subject off this list.
What did you expect to happen when you keep ignoring those requests,
time and again?

Re: Emacs ffi

[Christopher Dimech]

> Sent: Saturday, August 17, 2024 at 6:06 PM
> From: "Eli Zaretskii" <eliz@gnu.org>
> To: "Christopher Dimech" <dimech@gmx.com>
> Cc: acorallo@gnu.org, suhailsingh247@gmail.com, gerd.moellmann@gmail.com, nicolas@n16f.net, arthur.miller@live.com, emacs-devel@gnu.org
> Subject: Re: Emacs ffi
>
> > From: Christopher Dimech <dimech@gmx.com>
> > Cc: suhailsingh247@gmail.com, gerd.moellmann@gmail.com, nicolas@n16f.net,
> >  arthur.miller@live.com, emacs-devel@gnu.org, Eli Zaretskii <eliz@gnu.org>
> > Date: Fri, 16 Aug 2024 23:21:48 +0200
> > 
> > 
> > But this message does not resonate within the minds of the core emacs maintainers, 
> > with the trend continuing for many years to come.  Some believe that everybody 
> > in the world doesn’t get it about free software, and even that everybody in the 
> > world is a crook and that everybody in the world is trying to steal free software 
> > and make bad use of it.  I do not approve of such ideas.  Many think they know
> > everything about copyright infringement, but never been thrown out of court on 
> > a motion to dismiss.  
> > 
> > It's always the same, you'll get prima donna maintainers who are at the same 
> > level of priests, who preach a lot about licensing scenarios.  But have never 
> > been there, never done it, but preaching a lot about what others can do and what 
> > they cannot do.  Too much coercion was surely not what we wanted to apply.
> 
> For the umpteenth time: the maintainers are not at liberty to change
> the policies they promised to uphold when they were appointed.  If you
> want to discuss these policies, please take this to the proper place.
> 
> > As customary I get "Your message was deemed inappropriate by the moderator."
> > from emacs-devel-owner@gnu.org.  I wonder who it is.  Actually I wonder why
> > I wonder.
> 
> You were asked, several times, to take this subject off this list.
> What did you expect to happen when you keep ignoring those requests,
> time and again?
 
I have no illusions about the outcome. Decisions will be made regardless, 
but it's important to present the contrarian perspective. While I typically 
refrain from commenting, I do speak on the rare occasions when licensing 
is at issue.  It's about enhancing functionality, not complicating it with 
licensing hurdles that don't really apply.  

Incorporating a mathematical library into Emacs, based on Calc, is not
just feasible but highly beneficial.  Without all the perceived worries
you described.  You do a great job on the technical aspects, but when
you step outside your field and level of specialty, sometimes it is like
stepping off a cliff.

Emacs is a great idea.  In order to make it appeal to a younger generation 
of people who write programs for sharing, we need to make it simpler to use, 
quicker to understand, and better at doing all the jobs it’s supposed to do. 
And we need to refrain from going unnecessarily to war.

Re: Emacs ffi

[Eli Zaretskii]

> From: Christopher Dimech <dimech@gmx.com>
> Cc: acorallo@gnu.org, suhailsingh247@gmail.com, gerd.moellmann@gmail.com,
>  nicolas@n16f.net, arthur.miller@live.com, emacs-devel@gnu.org
> Date: Sat, 17 Aug 2024 11:05:43 +0200
> 
> > > As customary I get "Your message was deemed inappropriate by the moderator."
> > > from emacs-devel-owner@gnu.org.  I wonder who it is.  Actually I wonder why
> > > I wonder.
> > 
> > You were asked, several times, to take this subject off this list.
> > What did you expect to happen when you keep ignoring those requests,
> > time and again?
>  
> I have no illusions about the outcome. Decisions will be made regardless, 
> but it's important to present the contrarian perspective. While I typically 
> refrain from commenting, I do speak on the rare occasions when licensing 
> is at issue.  It's about enhancing functionality, not complicating it with 
> licensing hurdles that don't really apply.  

You are welcome to comment on this, just not here.  Discussing the
policies of the GNU Project is off-topic on this list.  So please
stop.

Re: Emacs ffi

[Stefan Kangas]

Christopher Dimech <dimech@gmx.com> writes:

> Incorporating a mathematical library into Emacs, based on Calc, is not
> just feasible but highly beneficial.

I think we all agree that a general purpose mathematical library would
be interesting.  The point is that the specifics of how it is
implemented matters.  If you think basing it on Calc is the best way
forward, then I invite you to get started.  I mean this sincerely.

Unfortunately, this discussion currently seems to be going in circles.
Since nothing helps ground a discussion more than working code, I
propose that we revisit this topic when we have something more concrete
to consider.

Re: Emacs ffi

[Joel Reicher]

Stefan Kangas <stefankangas@gmail.com> writes:

> Christopher Dimech <dimech@gmx.com> writes:
>
>> Incorporating a mathematical library into Emacs, based on Calc, 
>> is not just feasible but highly beneficial.
>
> I think we all agree that a general purpose mathematical library 
> would be interesting.  The point is that the specifics of how it 
> is implemented matters.

"Interesting" is different to "useful", and I think this is why 
the implementation is unclear, because the requirements of a 
library can only be made clear by a multiplicity of consumers.

What packages, other than Calc, would make use of such a library? 
(That's not rhetorical; I don't pretend to have an overview of all 
such packages.)

> If you think basing it on Calc is the best way forward, then I 
> invite you to get started.

It needs to be based on potential consumers. If Calc is the only 
consumer, then a library does not make sense yet. 

But that's not to say an FFI wouldn't improve Calc's 
implementation. If that's the issue, then it's a discussion that 
differs from one about library.

Regards,

        - Joel

Re: Emacs ffi

[Christopher Dimech]

> Sent: Sunday, August 18, 2024 at 2:30 AM
> From: "Joel Reicher" <joel.reicher@gmail.com>
> To: "Stefan Kangas" <stefankangas@gmail.com>
> Cc: "Christopher Dimech" <dimech@gmx.com>, "Eli Zaretskii" <eliz@gnu.org>, acorallo@gnu.org, suhailsingh247@gmail.com, gerd.moellmann@gmail.com, nicolas@n16f.net, arthur.miller@live.com, emacs-devel@gnu.org
> Subject: Re: Emacs ffi
>
> Stefan Kangas <stefankangas@gmail.com> writes:
>
> > Christopher Dimech <dimech@gmx.com> writes:
> >
> >> Incorporating a mathematical library into Emacs, based on Calc,
> >> is not just feasible but highly beneficial.
> >
> > I think we all agree that a general purpose mathematical library
> > would be interesting.  The point is that the specifics of how it
> > is implemented matters.
>
> "Interesting" is different to "useful", and I think this is why
> the implementation is unclear, because the requirements of a
> library can only be made clear by a multiplicity of consumers.
>
> What packages, other than Calc, would make use of such a library?
> (That's not rhetorical; I don't pretend to have an overview of all
> such packages.)
>
> > If you think basing it on Calc is the best way forward, then I
> > invite you to get started.
>
> It needs to be based on potential consumers. If Calc is the only
> consumer, then a library does not make sense yet.

Agreed.  I cannot see an Emacs package that would make extensive use
of the library.

> But that's not to say an FFI wouldn't improve Calc's
> implementation. If that's the issue, then it's a discussion that
> differs from one about library. - Joel

Calc is not bad, and outside libraries would not improve it much.
The mathematical tools of calc could be redesigned as an emacs
built-in library that Calc would then use.  Calc would then become
just a normal package.  Direct work on Calc has stopped.

My comments were directed at not enforcing licensing checks on external
libraries, but rather to maintain only our own internal licensing compliance
and provide guidance to users.  Users should make their own decisions about
integrating libraries.  Individuals who primarily write code have no serious
experience on licensing matters and legal implications.

Licensing issues require specialized knowledge beyond coding skills, including
an understanding of legal frameworks, compliance requirements, and the nuances
of different software licenses. Therefore, it's crucial to involve those with
a solid background in licensing to ensure proper handling and adherence to legal
standards.

Re: Emacs ffi

[Emanuel Berg]

Joel Reicher wrote:

> What packages, other than Calc, would make use of such
> a library? (That's not rhetorical; I don't pretend to have
> an overview of all such packages.)

All packages that do math, science, stats, scientific
computing, and so on; and all users who do any of that with
and from Emacs.

-- 
underground experts united
https://dataswamp.org/~incal

Sv: Emacs ffi

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 3686 bytes --]

Joel Reicher <joel.reicher@gmail.com> writes:

> Stefan Kangas <stefankangas@gmail.com> writes:
>
>> Christopher Dimech <dimech@gmx.com> writes:
>>
>>> Incorporating a mathematical library into Emacs, based on Calc, is not just
>>> feasible but highly beneficial.
>>
>> I think we all agree that a general purpose mathematical library would be
>> interesting.  The point is that the specifics of how it is implemented
>> matters.
>
> "Interesting" is different to "useful", and I think this is why the
> implementation is unclear, because the requirements of a library can only be
> made clear by a multiplicity of consumers.

Exactly the reason I suggested FFI in the very first place. "Mathematics" are a
broad term. Someone will want a linear algebra library, someone will want
symbolic manipulations, statistics, analysis, theorem provers and so on. There
are lots of mathematical libraries that could be useful to Emacs users in
different areas, since Emacs is used in some many different places by different
people.

> What packages, other than Calc, would make use of such a library? (That's not
> rhetorical; I don't pretend to have an overview of all such packages.)
>
>> If you think basing it on Calc is the best way forward, then I invite you to
>> get started.
>
> It needs to be based on potential consumers. If Calc is the only consumer, then
> a library does not make sense yet. But that's not to say an FFI wouldn't improve
> Calc's implementation. If that's the issue, then it's a discussion that differs
> from one about library.

As an alternative, it would be interesing to have everything in Lisp, no doubt
about that one. Imagine if Macsyma was available for Elisp. Maxima is the GPL
alternative, but it is all very uncommon CommonLisp, unfortunately.

Perhaps Calc could be brought to those levels, but I believe the amount of the
work and doubling on the implementation of something that already exists in form
of loadable C libraries, would not be trivial.
________________________________
Från: Joel Reicher <joel.reicher@gmail.com>
Skickat: den 17 augusti 2024 16:30
Till: Stefan Kangas <stefankangas@gmail.com>
Kopia: Christopher Dimech <dimech@gmx.com>; Eli Zaretskii <eliz@gnu.org>; acorallo@gnu.org <acorallo@gnu.org>; suhailsingh247@gmail.com <suhailsingh247@gmail.com>; gerd.moellmann@gmail.com <gerd.moellmann@gmail.com>; nicolas@n16f.net <nicolas@n16f.net>; arthur.miller@live.com <arthur.miller@live.com>; emacs-devel@gnu.org <emacs-devel@gnu.org>
Ämne: Re: Emacs ffi

Stefan Kangas <stefankangas@gmail.com> writes:

> Christopher Dimech <dimech@gmx.com> writes:
>
>> Incorporating a mathematical library into Emacs, based on Calc,
>> is not just feasible but highly beneficial.
>
> I think we all agree that a general purpose mathematical library
> would be interesting.  The point is that the specifics of how it
> is implemented matters.

"Interesting" is different to "useful", and I think this is why
the implementation is unclear, because the requirements of a
library can only be made clear by a multiplicity of consumers.

What packages, other than Calc, would make use of such a library?
(That's not rhetorical; I don't pretend to have an overview of all
such packages.)

> If you think basing it on Calc is the best way forward, then I
> invite you to get started.

It needs to be based on potential consumers. If Calc is the only
consumer, then a library does not make sense yet.

But that's not to say an FFI wouldn't improve Calc's
implementation. If that's the issue, then it's a discussion that
differs from one about library.

Regards,

        - Joel

[-- Attachment #2: Type: text/html, Size: 9197 bytes --]

Re: Emacs ffi

[Christopher Dimech]

> Sent: Sunday, August 18, 2024 at 1:21 AM
> From: "Stefan Kangas" <stefankangas@gmail.com>
> To: "Christopher Dimech" <dimech@gmx.com>, "Eli Zaretskii" <eliz@gnu.org>
> Cc: acorallo@gnu.org, suhailsingh247@gmail.com, gerd.moellmann@gmail.com, nicolas@n16f.net, arthur.miller@live.com, emacs-devel@gnu.org
> Subject: Re: Emacs ffi
>
> Christopher Dimech <dimech@gmx.com> writes:
>
> > Incorporating a mathematical library into Emacs, based on Calc, is not
> > just feasible but highly beneficial.
>
> I think we all agree that a general purpose mathematical library would
> be interesting.  The point is that the specifics of how it is
> implemented matters.  If you think basing it on Calc is the best way
> forward, then I invite you to get started.  I mean this sincerely.

Others thought that basing it on Calc is a good way forward.  It does
make for a good library having looked into its mathematical capabilities.

E. Berg's initial inclination to create a mathematical library for Emacs
was a commendable idea to expand the capabilities within the editor.
However, he did not want to delve into Calc, but on his own
implementations.

The disappointment to some was grounded in a strong argument from the
core Emacs developers, who emphasized the importance of having a robust,
well-integrated library that could match or exceed the functionality of
the existing Calc package.  The high bar set for quality and integration
within the Emacs ecosystem means that such a project would be a major
undertaking, likely contributing to the reluctance to pursue it.

> Unfortunately, this discussion currently seems to be going in circles.
> Since nothing helps ground a discussion more than working code, I
> propose that we revisit this topic when we have something more concrete
> to consider.

It is for the core emacs groups leading the development to establish the
principal design that is user-friendly and flexible to allow for easy
extension, without imposing overly stringent requirements beyond what is
typically expected for standard Emacs packages. My role is not to
overstep but to provide occasional input.

At this moment, it's unclear whether the decision will lean toward
implementing an FFI (Foreign Function Interface) or pursuing a different
approach. However, having Emacs provide a built-in solution seems more
conducive to seamless integration, allowing for a more cohesive
experience.

I could contribute concrete mathematical operations and assist in
refining the library as it evolves. While it doesn't need to be a
comprehensive solution, it should be robust enough to meet current needs
and provide a foundation for further development as required in a
collaborative approach.  Ultimately, I adhere to established design,
working within the framework set by the core Emacs team to create a
valuable tool.

Re: Emacs ffi

[Emanuel Berg]

Stefan Kangas wrote:

> Unfortunately, this discussion currently seems to be going
> in circles. Since nothing helps ground a discussion more
> than working code, I propose that we revisit this topic when
> we have something more concrete to consider.

Well, what is it that the code should do?

Provide a simpler interface to Calc?

What kind of interface?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Emacs ffi

[Andrea Corallo]

Christopher Dimech <dimech@gmx.com> writes:

>> Sent: Saturday, August 17, 2024 at 8:07 AM
>> From: "Andrea Corallo" <acorallo@gnu.org>
>> To: "Christopher Dimech" <dimech@gmx.com>
>> Cc: suhailsingh247@gmail.com, gerd.moellmann@gmail.com, nicolas@n16f.net, arthur.miller@live.com, emacs-devel@gnu.org, "Eli Zaretskii" <eliz@gnu.org>
>> Subject: Re: Emacs ffi
>>
>> Christopher Dimech <dimech@gmx.com> writes:
>> 
>> >> Sent: Friday, August 16, 2024 at 8:32 AM
>> >> From: "Andrea Corallo" <acorallo@gnu.org>
>> >> To: "Eli Zaretskii" <eliz@gnu.org>
>> >> Cc: suhailsingh247@gmail.com, gerd.moellmann@gmail.com, nicolas@n16f.net, arthur.miller@live.com, emacs-devel@gnu.org
>> >> Subject: Re: Emacs ffi
>> >>
>> >> Eli Zaretskii <eliz@gnu.org> writes:
>> >>
>> >> >> From: Andrea Corallo <acorallo@gnu.org>
>> >> >> Cc: Suhail Singh <suhailsingh247@gmail.com>,  gerd.moellmann@gmail.com,
>> >> >>   nicolas@n16f.net,  arthur.miller@live.com,  emacs-devel@gnu.org
>> >> >> Date: Thu, 15 Aug 2024 05:31:49 -0400
>> >> >>
>> >> >> Eli Zaretskii <eliz@gnu.org> writes:
>> >> >>
>> >> >> > The way we do it when loading modules requires the _loaded_ library to
>> >> >> > declare itself compatible, by exporting a symbol of a certain name.
>> >> >> > That is an action by the library we load, not by the Lisp program
>> >> >> > which loads it.
>> >> >>
>> >> >> But we could do the same for our hypothetical ffi machinery, that is:
>> >> >> `define-ffi-library' could fail if the shared library is not exporting
>> >> >> the predetermined symbol we expect no?
>> >> >
>> >> > Of course.  But how many such libraries do that?
>> >>
>> >> Dunno, ATM very few if not zero I guess.
>> >
>> > If there are no existing libraries that export the required symbol, it
>> > raises legitimate concerns about the feasibility and utility of the
>> > approach. Essentially, if the mechanism put in place is not supported by
>> > any available libraries, then the mechanism is ineffective and overly
>> > idealistic.
>> >
>> > There exists a large disconnect between theoretical compliance and
>> > practical usability. If the requirements for compliance are so stringent
>> > or specific that no existing libraries can meet them, then it could be
>> > argued that the approach is not grounded in the reality of software
>> > development.
>> 
>> I disagree, this is normal evolution for software ex: every time a new
>> programming language is created (or a new feature is added to it)
>> compilers implement the related support, even if no programs are using
>> that language or that new extension at present.  Is this a large
>> disconnect between theoretical compliance and practical usability?  No,
>> is just that users will come later.
>> 
>> If Emacs requires the symbol maybe compatible libraries will just export
>> it afterwards (given the cost is close to zero). - Andrea
>
> It is imposing an additional requirement that isn't typically necessary 
> in most other environments.
>
> Typically, mathematical libraries are designed to be widely applicable
> and used across various platforms and applications. Requiring these
> libraries to export a specific symbol or be designed with editor
> compatibility in mind is an unnecessary hurdle. Developers of libraries
> do not prioritize compatibility with editors like Emacs.
>
> To overcome the hurdle that Emacs is bringing upon itself, of requiring
> GPL compliance for mathematical libraries, it should provide its own
> built-in library as suggested by E. Berg. It is the pragmatic way to do
> it.  I disagreed with Berg initially, but I can now understand his
> frustrations with the whole emacs design.
>
> Some free software simply cannot be used with GPL-licensed software like 
> Emacs without violating the terms of one or both licenses.  Requiring GPL 
> compliance for integration is banal.  Particularly when Emacs is not the 
> one providing the library.  It will be the user who will be using.  One 
> simply cannot accuse a user to breaking any license if the library is not
> GPL compliant.  The landscape of free software licenses is diverse, with 
> many different licenses with many not compatible with each other. You’d 
> have to put every teenager in the world in jail, and you can’t do that ! 
>
> But this message does not resonate within the minds of the core emacs maintainers, 
> with the trend continuing for many years to come.  Some believe that everybody 
> in the world doesn’t get it about free software, and even that everybody in the 
> world is a crook and that everybody in the world is trying to steal free software 
> and make bad use of it.  I do not approve of such ideas.  Many think they know
> everything about copyright infringement, but never been thrown out of court on 
> a motion to dismiss.  
>
> It's always the same, you'll get prima donna maintainers who are at the same 
> level of priests, who preach a lot about licensing scenarios.  But have never 
> been there, never done it, but preaching a lot about what others can do and what 
> they cannot do.  Too much coercion was surely not what we wanted to apply.

Looking at your Emacs contribution track I guess we have "been there"
and "did it" more than you, so I'm sorry but I don't understand where
*your* preaching is coming from.

This is my last (of very few) replies to you on this subject 👋

Re: Emacs ffi

[Björn Bidar]

Christopher Dimech <dimech@gmx.com> writes:

>> Sent: Saturday, August 17, 2024 at 8:07 AM
>> From: "Andrea Corallo" <acorallo@gnu.org>
>> To: "Christopher Dimech" <dimech@gmx.com>
>> Cc: suhailsingh247@gmail.com, gerd.moellmann@gmail.com, nicolas@n16f.net, arthur.miller@live.com, emacs-devel@gnu.org, "Eli Zaretskii" <eliz@gnu.org>
>> Subject: Re: Emacs ffi
>>
>> Christopher Dimech <dimech@gmx.com> writes:
>> 
>> >> Sent: Friday, August 16, 2024 at 8:32 AM
>> >> From: "Andrea Corallo" <acorallo@gnu.org>
>> >> To: "Eli Zaretskii" <eliz@gnu.org>
>> >> Cc: suhailsingh247@gmail.com, gerd.moellmann@gmail.com, nicolas@n16f.net, arthur.miller@live.com, emacs-devel@gnu.org
>> >> Subject: Re: Emacs ffi
>> >>
>> >> Eli Zaretskii <eliz@gnu.org> writes:
>> >>
>> >> >> From: Andrea Corallo <acorallo@gnu.org>
>> >> >> Cc: Suhail Singh <suhailsingh247@gmail.com>,  gerd.moellmann@gmail.com,
>> >> >>   nicolas@n16f.net,  arthur.miller@live.com,  emacs-devel@gnu.org
>> >> >> Date: Thu, 15 Aug 2024 05:31:49 -0400
>> >> >>
>> >> >> Eli Zaretskii <eliz@gnu.org> writes:
>> >> >>
>> >> >> > The way we do it when loading modules requires the _loaded_ library to
>> >> >> > declare itself compatible, by exporting a symbol of a certain name.
>> >> >> > That is an action by the library we load, not by the Lisp program
>> >> >> > which loads it.
>> >> >>
>> >> >> But we could do the same for our hypothetical ffi machinery, that is:
>> >> >> `define-ffi-library' could fail if the shared library is not exporting
>> >> >> the predetermined symbol we expect no?
>> >> >
>> >> > Of course.  But how many such libraries do that?
>> >>
>> >> Dunno, ATM very few if not zero I guess.
>> >
>> > If there are no existing libraries that export the required symbol, it
>> > raises legitimate concerns about the feasibility and utility of the
>> > approach. Essentially, if the mechanism put in place is not supported by
>> > any available libraries, then the mechanism is ineffective and overly
>> > idealistic.
>> >
>> > There exists a large disconnect between theoretical compliance and
>> > practical usability. If the requirements for compliance are so stringent
>> > or specific that no existing libraries can meet them, then it could be
>> > argued that the approach is not grounded in the reality of software
>> > development.
>> 
>> I disagree, this is normal evolution for software ex: every time a new
>> programming language is created (or a new feature is added to it)
>> compilers implement the related support, even if no programs are using
>> that language or that new extension at present.  Is this a large
>> disconnect between theoretical compliance and practical usability?  No,
>> is just that users will come later.
>> 
>> If Emacs requires the symbol maybe compatible libraries will just export
>> it afterwards (given the cost is close to zero). - Andrea
>
> It is imposing an additional requirement that isn't typically necessary 
> in most other environments.
>
> Typically, mathematical libraries are designed to be widely applicable
> and used across various platforms and applications. Requiring these
> libraries to export a specific symbol or be designed with editor
> compatibility in mind is an unnecessary hurdle. Developers of libraries
> do not prioritize compatibility with editors like Emacs.

> To overcome the hurdle that Emacs is bringing upon itself, of requiring
> GPL compliance for mathematical libraries, it should provide its own
> built-in library as suggested by E. Berg. It is the pragmatic way to do
> it.  I disagreed with Berg initially, but I can now understand his
> frustrations with the whole emacs design.

> Some free software simply cannot be used with GPL-licensed software like 
> Emacs without violating the terms of one or both licenses.  Requiring GPL 
> compliance for integration is banal.  Particularly when Emacs is not the 
> one providing the library.  It will be the user who will be using.  One 
> simply cannot accuse a user to breaking any license if the library is not
> GPL compliant.  The landscape of free software licenses is diverse, with 
> many different licenses with many not compatible with each other. You’d 
> have to put every teenager in the world in jail, and you can’t do that ! 


I think GPL compliance could be checked but I agree we can't require
external libraries where Emacs is the minority to impose such
requirements. Well technically we can but who will honor them? Does the
FFI interface have a system for such things?
I think a warning or a "do you really want this" would be the least
intrusive way.
Similarly to spoiler warnings in media when there's graphic depections
of violence: Do you really want to enable this, the libraries could
restrict your freedom or similar. 

I think the XEmacs implementation of FFI would be a good start to look
how such an interface could be added.


> But this message does not resonate within the minds of the core emacs maintainers, 
> with the trend continuing for many years to come.  Some believe that everybody 
> in the world doesn’t get it about free software, and even that everybody in the 
> world is a crook and that everybody in the world is trying to steal free software 
> and make bad use of it.  I do not approve of such ideas.  Many think they know
> everything about copyright infringement, but never been thrown out of court on 
> a motion to dismiss.  

I don't really want to go into the politics besides that I have seen
worse on this list from user or maintainer such as to tell user to not
use gender neutral pronouns if they so chose to do so.

> It's always the same, you'll get prima donna maintainers who are at the same 
> level of priests, who preach a lot about licensing scenarios.  But have never 
> been there, never done it, but preaching a lot about what others can do and what 
> they cannot do.  Too much coercion was surely not what we wanted to apply.
>
> As customary I get "Your message was deemed inappropriate by the moderator."
> from emacs-devel-owner@gnu.org.  I wonder who it is.  Actually I wonder why
> I wonder.
>
> We need to understand how that working together purposively brings us to the 
> point where everyone is not afraid of free software anymore and we are not 
> worried about their complying anymore.  We are just all engaging and leading 
> the task of making free software.

I had this experience in the past too where some judge first and then
read. I.e. the recent discussion about OAuth2 where there was a
compliant that we shouldn't implement support for non-free platforms
even thou that wasn't really the point.

Sometimes it would be better if the more experienced contributors would
try to be more welcoming or helping when it comes to new contributors.

Re: Emacs ffi

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I think GPL compliance could be checked but I agree we can't require
  > external libraries where Emacs is the minority to impose such
  > requirements. Well technically we can but who will honor them? Does the
  > FFI interface have a system for such things?

Instead of thinking of this question as an abstract one, it will be
more productive to look at exanples.  I think we will find that most
things people might want in the area of mathematical libraries are
available as GPL-compatible free software, and that most of the rest
come with an operating system's major components and would qualify for
the "system library" exception.

So maybe in practice the problematical case only rarely occurs.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Emacs ffi

[Christopher Dimech]

> Sent: Tuesday, August 20, 2024 at 4:57 AM
> From: "Richard Stallman" <rms@gnu.org>
> To: "Björn Bidar" <bjorn.bidar@thaodan.de>
> Cc: emacs-devel@gnu.org, eliz@gnu.org
> Subject: Re: Emacs ffi
>
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>   > I think GPL compliance could be checked but I agree we can't require
>   > external libraries where Emacs is the minority to impose such
>   > requirements. Well technically we can but who will honor them? Does the
>   > FFI interface have a system for such things?
> 
> Instead of thinking of this question as an abstract one, it will be
> more productive to look at exanples.  I think we will find that most
> things people might want in the area of mathematical libraries are
> available as GPL-compatible free software, and that most of the rest
> come with an operating system's major components and would qualify for
> the "system library" exception.
> 
> So maybe in practice the problematical case only rarely occurs.

Why are we moving away from the built-in mathematical implementations in 
Emacs' Calc package ?  Calc is a powerful tool and deeply integrated within 
Emacs, offering a wide range of mathematical functionalities that cater to 
many common and advanced needs.  It supports arithmetic, algebra, calculus, 
and even symbolic mathematics, making it very versatile.  

Is its interface daunting for users ?  Would we need dedicated people to focus
on it again ? 

Still, the discussion has been shifting towards a General FFI Interface, beyond
mathematical libraries.

Re: Emacs ffi

[Emanuel Berg]

Emacs is not the only free software out there, how does
everyone else do this?

In Debian we opt-in with non-free in /etc/apt/sources.list
e.g. with a line like this:

  deb http://deb.debian.org/debian bookworm main contrib non-free non-free-firmware

Using this command, you can see that they give the user access
to software repositories of a size that is mindboggling.

Do they have another policy than Emacs?

Also, are there no free or FOSS libraries we can start with,
and possibly upgrade later if the others really are superior?

IME when people say the proprietary version is much better,
like for CAD and such specific applications, where the
free/FOSS world is supposedly behind - I dont know who can
tell, but what we have is good enough.

Is the situation not the same for libraries?

Here is the shell command. Yes

$ apt-cache pkgnames | wc -l
64 986

- that is amazing digit to behold.

-- 
underground experts united
https://dataswamp.org/~incal

Re: as for Calc and the math library

[Gerd Möllmann]

Suhail Singh <suhailsingh247@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> I guess an FFI is already there? Or was there because it looks kind of
>>> dead:
>>> 
>>>   https://github.com/tromey/emacs-ffi
>>> 
>>> Tom Tromey says in one of the issues
>>> 
>>>   https://github.com/tromey/emacs-ffi/issues/20
>>> 
>>> that John Wigley asked him on IRC to submit it for inclusion. Can't find
>>> anything on emacs-devel though.
>>
>> It doesn't ensure GPL-compliance, AFAICT, so in its current form it
>> cannot be accepted, sorry. And I'm not sure I understand how can one
>> technically enforce GPL compliance in FFI-style loading of arbitrary
>> shared libraries.  The only idea that comes to mind is allow-list of
>> known libraries, the way we do in sqlite.c, but I'm not sure that
>> method is scalable to the basically infinite world of arbitrary
>> libraries.
>
> On the topic of what would be acceptable for an FFI, wouldn't something
> akin to what's done for modules be sufficient ?  I.e., have the users of
> the interface explicitly state that they are compliant.
>
> It would scale better than an allow-list.  IIUC, Arthur mentioned this
> in another thread.  If this wouldn't be sufficient for an FFI, could you
> please elaborate on why that's the case ?

Another question is of course why something like Tom's FFI must be made
part of Emacs when, AFIU, it could be a package like vterm which also
contains an Emacs module.

Re: as for Calc and the math library

[Nicolas Martyanoff]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Another question is of course why something like Tom's FFI must be made
> part of Emacs when, AFIU, it could be a package like vterm which also
> contains an Emacs module.

This is what I had in mind when I suggested a dynamic module using
libffi (apparently exactly what this emacs-ffi does) . I do not see the
point in trying to get this kind of module included in Emacs since it is
obviously not wanted here.

Just distribute it as a package and be done with it. People wanting to
use it will just add it as a dependency.

-- 
Nicolas Martyanoff
https://n16f.net
nicolas@n16f.net

Re: as for Calc and the math library

[Gerd Möllmann]

Nicolas Martyanoff <nicolas@n16f.net> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> Another question is of course why something like Tom's FFI must be made
>> part of Emacs when, AFIU, it could be a package like vterm which also
>> contains an Emacs module.
>
> This is what I had in mind when I suggested a dynamic module using
> libffi (apparently exactly what this emacs-ffi does) . I do not see the
> point in trying to get this kind of module included in Emacs since it is
> obviously not wanted here.
>
> Just distribute it as a package and be done with it. People wanting to
> use it will just add it as a dependency.

+1

Re: as for Calc and the math library

[Eli Zaretskii]

> From: Nicolas Martyanoff <nicolas@n16f.net>
> Cc: Suhail Singh <suhailsingh247@gmail.com>,  Eli Zaretskii <eliz@gnu.org>,
>  nicolas@n16f.net,  arthur.miller@live.com,  emacs-devel@gnu.org
> Date: Wed, 14 Aug 2024 16:40:41 +0200
> 
> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
> 
> > Another question is of course why something like Tom's FFI must be made
> > part of Emacs when, AFIU, it could be a package like vterm which also
> > contains an Emacs module.
> 
> This is what I had in mind when I suggested a dynamic module using
> libffi (apparently exactly what this emacs-ffi does) . I do not see the
> point in trying to get this kind of module included in Emacs since it is
> obviously not wanted here.
> 
> Just distribute it as a package and be done with it. People wanting to
> use it will just add it as a dependency.

Perfectly fine.  Then people who do that will be responsible for
making sure the libraries they load are compatible, license-wise, with
the emacs-ffi module (whose license is not explicitly stated, btw,
AFAICT).

Re: as for Calc and the math library

[Madhu]

* Nicolas Martyanoff <87a5hi0yts.fsf@valhala.localdomain> :
Wrote on Mon, 12 Aug 2024 13:23:27 +0200:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: arthur miller <arthur.miller@live.com>
>>> Date: Mon, 12 Aug 2024 05:30:08 +0000
>>> Even better, given people proper FFI so they can import and use any
>>> of the tens
>>> or hundreds of quality math libraries available.
>>
>> Not going to happen, and you know it.
>
> Naïve question, why? I had this problem not so long ago because I wanted
> to bind libpq, and I had to abandon the whole idea because dealing with
> dynamic modules was way too inconvenient.
>
> Having something similar to Common Lisp FFIs would make it trivial.

There is a workable promising start, with a single lisp module from
Tromey:

https://github.com/tromey/emacs-ffi

I have a clone on gh (user enometh), but I haven't found the time to
polish it up enough to post

[ffi] Re: as for Calc and the math library

[Madhu]

* Madhu <m3bk1vn03w.fsf@leonis4.robolove.meer.net> :
Wrote on Wed, 14 Aug 2024 10:59:23 +0530:

> * Nicolas Martyanoff <87a5hi0yts.fsf@valhala.localdomain> :
> Wrote on Mon, 12 Aug 2024 13:23:27 +0200:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> From: arthur miller <arthur.miller@live.com> Date: Mon, 12 Aug 2024
>>>> 05:30:08 +0000 Even better, given people proper FFI so they can
>>>> import and use any of the tens or hundreds of quality math
>>>> libraries available.

I forgot to mention, this was also already possbible and being done
For example (also has examples using emacs-ffi)

https://github.com/jkitchin/emacs-modules


>>> Not going to happen, and you know it.
>>a Naïve question, why? I had this problem not so long ago because I
>>wanted to bind libpq, and I had to abandon the whole idea because
>>dealing with dynamic modules was way too inconvenient.
>> Having something similar to Common Lisp FFIs would make it trivial.

> There is a workable promising start, with a single lisp module from
> Tromey:
> https://github.com/tromey/emacs-ffi
> I have a clone on gh (user enometh), but I haven't found the time to
> polish it up enough to post

as for Calc and the math library

[Emanuel Berg]

;; __________________________________________________________
;; ``````````````````````````````````````````````````````````
;; |                         
;; 3                        CCUG-ML                        60
;;                                                          |
;; __________________________________________________________
;; `-`-`-`-`-`-`-`-`-`-`-`-`-`-`-`-`-`-`-`-`-`-`-`-`-`-`-`-`-

I have now been using the Calc library a bit!

Not a lot. Very little.

But I can already say about Calc that it is very good.
The code is good as well and as for math it is at a very
high level.

But it it is too complicated and the entry level is too high.
It is advanced math but also an advanced program.

We still need a math library that covers like up to the
introductory courses at the university. This should be
accessible with the familiar Elisp syntax.

Every function in math from school and again, maybe including
the introductory courses at the university, should be
available with Emacs notation.

So now, we don't have

(** 4 1)
( ^ 4 1)

(We do have `expt' but notation is important and should be,
for a math library, using "their" symbols and conventions,
not ours, as much as possible.)

We don't have

(defun distance-point (min max)
  (+ min (/ (- max min) 2.0)) )

Or from Logic, we don't have `nand' i.e. (not (and a b)). [last]

All that we don't have, or we do, as Calc can do it - easily.

Okay, so we don't need a new generic library. We need
a generic library which is a complement to all the basic
stuff, we don't have readily available as for now.

Out of respect for Calc there is no need to move it, now need
for that and no practical reason to.

However the new stuff must be totally transparent. When some
dude comes home from school/university and wants to try some
half-basic stuff, this should work instantly using the Elisp
he is used to. It should be "Oh, cool, Emacs has it!" And for
a long time, it should be like that for him.

If at some much later point, he (or she) moves on to "real
Calc", then he is so good we don't have to worry about
him anymore.

So we should have a CCUG-ML project: the Generic Complementary
Math Library Using Calc.

Yes, it would be to some small extent *using* Calc but to
a huge extent it would just be an interface. But 100%
transparent, so one can use Elisp and the common math
notation people know from school.

No long prefixes, no special notation, nothing, as I now know
there is _a lot_ of in Calc. People should be allowed to use
the basics of Calc without knowing it!

It should just be just like when I looked for `**' and `nand',
only this time, they should exist, thru the interface,
using Calc.

IMO!

Here is nand BTW:

(defun nand (&rest conditions)
  (when (member nil conditions)
    t) )

;; (nand)           ; nil
;; (nand 1 2 3)     ; nil
;; (nand nil)       ; t
;; (nand 1 2 nil 3) ; t

;; (not (and))           ; nil
;; (not (and 1 2 3))     ; nil
;; (not (and nil))       ; t
;; (not (and 1 2 nil 3)) ; t

And here is some Calc:

(require 'calc-ext)
(require 'cl-lib)

;; (infx "0 < 10")

(defun infx (str)
  (let* ((exp (math-read-expr str))
         (fun (car exp))
         (new (cons (intern (caar (cl-member fun math-expr-opers :key #'cadr :test #'equal)))
                    (cdr exp))))
    (apply (car new) (cdr new))))

-- 
underground experts united
https://dataswamp.org/~incal

Re: as for Calc and the math library

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sun, 11 Aug 2024 00:48:13 +0200
> 
> We don't have
> 
> (defun distance-point (min max)
>   (+ min (/ (- max min) 2.0)) )

Once again, if we want this and other functions for a math library in
Emacs, we should make sure the algorithms used are solid and stable,
and in particular don't cause errors unless the result really cannot
be expressed as a valid number.  For example, in the above, the
expression (- min max) could overflow when min/2 - max/2 that it
calculates is a valid number.

If Emacs is to have a math library, the library must use high-quality
mathematical and numerical algorithms that are well-known and
described in many textbooks on this subject matter.  This high quality
should start from the simplest calculations, and go all the way to the
most complex ones.  For example, even summing many float values
naïvely will eventually lose accuracy.  E.g., try adding 10 values of
1e-16 followed by the value of 1, that is

  (+ 1.0 1.e-16 1.e-16 1.e-16 1.e-16 1.e-16 1.e-16 1.e-16 1.e-16 1.e-16 1.e-16)

Doing this naïvely will lose all the small values, as if they were not
there at all.  So even such seemingly-simple operations might need
non-trivial algorithm to avoid losing accuracy when it could be kept.

Re: as for Calc and the math library

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> If Emacs is to have a math library, the library must use high-quality
> mathematical and numerical algorithms that are well-known and
> described in many textbooks on this subject matter...

I agree with this point, but I suspect that writing such a library from
scratch might be a bit difficult.

May I suggest an alternative: abstract out GNU Calc math routines to be
available from Elisp without having to read Calc sources.
Such a task would require:
1. Documenting some Calc data structures
2. Possibly refactoring some Calc functions to use simpler API.

It would be a much easier task and can improve the existing
functionality.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

as for Calc and the math library

[Christopher Dimech]

> Sent: Monday, August 12, 2024 at 4:45 AM
> From: "Ihor Radchenko" <yantar92@posteo.net>
> To: "Eli Zaretskii" <eliz@gnu.org>
> Cc: "Emanuel Berg" <incal@dataswamp.org>, emacs-devel@gnu.org
> Subject: Re: as for Calc and the math library
>
> Eli Zaretskii <eliz@gnu.org> writes:
>
> > If Emacs is to have a math library, the library must use high-quality
> > mathematical and numerical algorithms that are well-known and
> > described in many textbooks on this subject matter...
>
> I agree with this point, but I suspect that writing such a library from
> scratch might be a bit difficult.
>
> May I suggest an alternative: abstract out GNU Calc math routines to be
> available from Elisp without having to read Calc sources.
> Such a task would require:
> 1. Documenting some Calc data structures
> 2. Possibly refactoring some Calc functions to use simpler API.
>
> It would be a much easier task and can improve the existing
> functionality.  - Ihor Radchenko

That would be my recommendation.  Calc already offer robust mathematical
routines.  Doing a mathematical library is not difficult but required focus
and the ability to deal with complicated implementation, not just applying
direct solutions.

Re: as for Calc and the math library

[Emanuel Berg]

Ihor Radchenko wrote:

> I suspect that writing such a library from scratch might be
> a bit difficult.
>
> May I suggest an alternative: abstract out GNU Calc math
> routines to be available from Elisp without having to read
> Calc sources. Such a task would require:
;
> 1. Documenting some Calc data structures
> 2. Possibly refactoring some Calc functions to use simpler API.
>
> It would be a much easier task and can improve the
> existing functionality.

Very smart suggestion.

-- 
underground experts united
https://dataswamp.org/~incal

Re: as for Calc and the math library

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Emanuel Berg <incal@dataswamp.org>, emacs-devel@gnu.org
> Date: Sun, 11 Aug 2024 16:45:54 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > If Emacs is to have a math library, the library must use high-quality
> > mathematical and numerical algorithms that are well-known and
> > described in many textbooks on this subject matter...
> 
> I agree with this point, but I suspect that writing such a library from
> scratch might be a bit difficult.

It depends on the scope and the person who'd like to work on that.

> May I suggest an alternative: abstract out GNU Calc math routines to be
> available from Elisp without having to read Calc sources.
> Such a task would require:
> 1. Documenting some Calc data structures
> 2. Possibly refactoring some Calc functions to use simpler API.
> 
> It would be a much easier task and can improve the existing
> functionality.

I don't think I understand well enough what will this entail in
practice.  Can you show an example of doing this for a couple of
functions?

In any case, this job, as any other job in Emacs, needs its
volunteer(s).  Until we have such volunteers, it's all academic.

as for Calc and the math library

[Christopher Dimech]

> Sent: Monday, August 12, 2024 at 5:59 AM
> From: "Eli Zaretskii" <eliz@gnu.org>
> To: "Ihor Radchenko" <yantar92@posteo.net>
> Cc: incal@dataswamp.org, emacs-devel@gnu.org
> Subject: Re: as for Calc and the math library
>
> > From: Ihor Radchenko <yantar92@posteo.net>
> > Cc: Emanuel Berg <incal@dataswamp.org>, emacs-devel@gnu.org
> > Date: Sun, 11 Aug 2024 16:45:54 +0000
> >
> > Eli Zaretskii <eliz@gnu.org> writes:
> >
> > > If Emacs is to have a math library, the library must use high-quality
> > > mathematical and numerical algorithms that are well-known and
> > > described in many textbooks on this subject matter...
> >
> > I agree with this point, but I suspect that writing such a library from
> > scratch might be a bit difficult.
>
> It depends on the scope and the person who'd like to work on that.
>
> > May I suggest an alternative: abstract out GNU Calc math routines to be
> > available from Elisp without having to read Calc sources.
> > Such a task would require:
> > 1. Documenting some Calc data structures
> > 2. Possibly refactoring some Calc functions to use simpler API.
> >
> > It would be a much easier task and can improve the existing
> > functionality.
>
> I don't think I understand well enough what will this entail in
> practice.  Can you show an example of doing this for a couple of
> functions?
>
> In any case, this job, as any other job in Emacs, needs its
> volunteer(s).  Until we have such volunteers, it's all academic.

For good mathematical libraries, you require a practicing applied
mathematician.  A programmer is not enough.

Re: as for Calc and the math library

[Eli Zaretskii]

> From: Christopher Dimech <dimech@gmx.com>
> Cc: Ihor Radchenko <yantar92@posteo.net>, incal@dataswamp.org,
>  emacs-devel@gnu.org
> Date: Sun, 11 Aug 2024 20:17:48 +0200
> 
> For good mathematical libraries, you require a practicing applied
> mathematician.  A programmer is not enough.

That's not true.  A programmer experienced in numerical computations
is more than enough, because the relevant algorithms are known and
described in the literature, so there's no need to invent them.

as for Calc and the math library

[Christopher Dimech]

> Sent: Monday, August 12, 2024 at 6:32 AM
> From: "Eli Zaretskii" <eliz@gnu.org>
> To: "Christopher Dimech" <dimech@gmx.com>
> Cc: yantar92@posteo.net, incal@dataswamp.org, emacs-devel@gnu.org
> Subject: Re: as for Calc and the math library
>
> > From: Christopher Dimech <dimech@gmx.com>
> > Cc: Ihor Radchenko <yantar92@posteo.net>, incal@dataswamp.org,
> >  emacs-devel@gnu.org
> > Date: Sun, 11 Aug 2024 20:17:48 +0200
> >
> > For good mathematical libraries, you require a practicing applied
> > mathematician.  A programmer is not enough.
>
> That's not true.  A programmer experienced in numerical computations
> is more than enough, because the relevant algorithms are known and
> described in the literature, so there's no need to invent them.

A mathematician can usually do better than the standard techniques.
Some descriptions in the literature can be quite old and not so up
to date.  Most of the original fortran routines was not done by programmers.

Re: as for Calc and the math library

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> > If Emacs is to have a math library, the library must use high-quality
>> > mathematical and numerical algorithms that are well-known and
>> > described in many textbooks on this subject matter...
>> 
>> I agree with this point, but I suspect that writing such a library from
>> scratch might be a bit difficult.
>
> It depends on the scope and the person who'd like to work on that.

Yes, of course. If somebody is up to writing a serious math lib, it
would be better. But having a bit simpler task idea can increase the pool
of potential volunteers.

>> May I suggest an alternative: abstract out GNU Calc math routines to be
>> available from Elisp without having to read Calc sources.
>> Such a task would require:
>> 1. Documenting some Calc data structures
>> 2. Possibly refactoring some Calc functions to use simpler API.
>> 
>> It would be a much easier task and can improve the existing
>> functionality.
>
> I don't think I understand well enough what will this entail in
> practice.  Can you show an example of doing this for a couple of
> functions?

For (1), I was referring to ";;;; Arithmetic routines." in
calc.el. Now, people must simply know the internal format of special
number representations to work with Calc from Elisp.
It might be worth (a) putting them into a proper manual (some kind of
Calc Lisp API description); (b) maybe creating constructor functions to
abstract the types away.

For (2), for starters, Calc functions often do not even have
docstrings. One simply has to guess(?) or read the code to understand
what kinds of values can be passed. For example, take `math-add' - no
docstring, and go try adding invalid arguments:

M-: (math-add "foo" '(frac 1 2))

Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p "foo")
  (math-mul "foo" 2)
  (calc-add-fractions "foo" (frac 1 2))
  (math-add "foo" (frac 1 2))

The error checking could be improved, especially since it is a
polymorphic function that is capable of doing things like

(math-add '(vec (frac 1 2) 3 1) '(frac 1 2)) ; => (vec 1 (frac 7 2) (frac 3 2))

but somehow not

M-: (math-add 0.1 '(frac 1 10))

Debugger entered--Lisp error: (wrong-type-argument integer-or-marker-p 2.0)
  (math-gcd 2.0 10)
  (math-make-frac 2.0 10)
  (calc-add-fractions 0.1 (frac 1 10))
  (math-add 0.1 (frac 1 10))

> In any case, this job, as any other job in Emacs, needs its
> volunteer(s).  Until we have such volunteers, it's all academic.

Yes, of course. I was hoping that Emanual (who is apparently interested
in this topic, or at least in improving Elisp) can be the one.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: as for Calc and the math library

[Emanuel Berg]

Ihor Radchenko wrote:

> Yes, of course. I was hoping that Emanual (who is apparently
> interested in this topic, or at least in improving Elisp)
> can be the one.

Thanks for that but after the controversy I lost the
enthusiasm just a little bit since that was very unpleasant,
but it is all good because now my own programming ideas keep
me busy instead!

You need to reach out to the young guys studying math at
universities today, one thinks all they do all day is math but
almost _all_ of them are into computers as well, how many will
use Emacs I don't know but this is the person you want for
this job :)

-- 
underground experts united
https://dataswamp.org/~incal

Re: as for Calc and the math library

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: incal@dataswamp.org, emacs-devel@gnu.org
> Date: Sun, 11 Aug 2024 18:27:16 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> 1. Documenting some Calc data structures
> >> 2. Possibly refactoring some Calc functions to use simpler API.
> >> 
> >> It would be a much easier task and can improve the existing
> >> functionality.
> >
> > I don't think I understand well enough what will this entail in
> > practice.  Can you show an example of doing this for a couple of
> > functions?
> 
> For (1), I was referring to ";;;; Arithmetic routines." in
> calc.el.

(1) is documentation of Calc data structures.  If you mean the large
commentary that describes the various form of objects that Calc can
handle, then what is left to document?

> Now, people must simply know the internal format of special
> number representations to work with Calc from Elisp.
> It might be worth (a) putting them into a proper manual (some kind of
> Calc Lisp API description); (b) maybe creating constructor functions to
> abstract the types away.

Doesn't (a) already exist in the Calc manual chapter "Data Types"?
And if not, then you are talking about enhancing the documentation of
Calc, which is always welcome, but I fail to see how it is a step
towards a separate library.

> For (2), for starters, Calc functions often do not even have
> docstrings. One simply has to guess(?) or read the code to understand
> what kinds of values can be passed.

And how is this relevant to (2), which is about refactoring some of
Calc code into hopefully easier-to-use functions?

> For example, take `math-add' - no
> docstring, and go try adding invalid arguments:
> 
> M-: (math-add "foo" '(frac 1 2))

It's a small wonder that you cannot math-add a string, especially
since none of the objects described in the commentary are any kind of
string.  More generally, AFAIU these don't have doc strings because
they are internal functions.

But still, I don't understand the relevance.

> but somehow not
> 
> M-: (math-add 0.1 '(frac 1 10))
> 
> Debugger entered--Lisp error: (wrong-type-argument integer-or-marker-p 2.0)
>   (math-gcd 2.0 10)
>   (math-make-frac 2.0 10)
>   (calc-add-fractions 0.1 (frac 1 10))
>   (math-add 0.1 (frac 1 10))

You again play against the rules: 0.1 is not a valid object for these
functions.  Try

  M-: (math-add '(float 1 -1) '(frac 1 10))

Re: as for Calc and the math library

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> For (1), I was referring to ";;;; Arithmetic routines." in
>> calc.el.
>
> (1) is documentation of Calc data structures.  If you mean the large
> commentary that describes the various form of objects that Calc can
> handle, then what is left to document?

Put this commentary, and other relevant APIs into .texi, explained and
organized aiming for Elisp API users.

>> Now, people must simply know the internal format of special
>> number representations to work with Calc from Elisp.
>> It might be worth (a) putting them into a proper manual (some kind of
>> Calc Lisp API description); (b) maybe creating constructor functions to
>> abstract the types away.
>
> Doesn't (a) already exist in the Calc manual chapter "Data Types"?

Hmm. Yes. Last time, I failed to see them.

> And if not, then you are talking about enhancing the documentation of
> Calc, which is always welcome, but I fail to see how it is a step
> towards a separate library.

The idea is to have a description and overview of what the library
does. It will make the design easier later.

>> For (2), for starters, Calc functions often do not even have
>> docstrings. One simply has to guess(?) or read the code to understand
>> what kinds of values can be passed.
>
> And how is this relevant to (2), which is about refactoring some of
> Calc code into hopefully easier-to-use functions?

If these functions are going to be used from Elisp, they must have
docstrings.

>> For example, take `math-add' - no
>> docstring, and go try adding invalid arguments:
>> 
>> M-: (math-add "foo" '(frac 1 2))
>
> It's a small wonder that you cannot math-add a string, especially
> since none of the objects described in the commentary are any kind of
> string.  More generally, AFAIU these don't have doc strings because
> they are internal functions.
>
> But still, I don't understand the relevance.

I meant this function as a candidate for math library we are
discussing.

>> but somehow not
>> 
>> M-: (math-add 0.1 '(frac 1 10))
>> 
>> Debugger entered--Lisp error: (wrong-type-argument integer-or-marker-p 2.0)
>>   (math-gcd 2.0 10)
>>   (math-make-frac 2.0 10)
>>   (calc-add-fractions 0.1 (frac 1 10))
>>   (math-add 0.1 (frac 1 10))
>
> You again play against the rules: 0.1 is not a valid object for these
> functions.  Try
>
>   M-: (math-add '(float 1 -1) '(frac 1 10))

My point is that the error message could be nicer.
Something like (wrong-type-argument calc-value-p 2.0)
"integer-or-marker-p" creates an impression that you can only pass
integers there.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: as for Calc and the math library

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: incal@dataswamp.org, emacs-devel@gnu.org
> Date: Sun, 11 Aug 2024 19:13:16 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > And if not, then you are talking about enhancing the documentation of
> > Calc, which is always welcome, but I fail to see how it is a step
> > towards a separate library.
> 
> The idea is to have a description and overview of what the library
> does. It will make the design easier later.

I think if someone will step forward to do this kind of job, it will
be up to them what steps should lead to the goal.

> >> M-: (math-add 0.1 '(frac 1 10))
> >> 
> >> Debugger entered--Lisp error: (wrong-type-argument integer-or-marker-p 2.0)
> >>   (math-gcd 2.0 10)
> >>   (math-make-frac 2.0 10)
> >>   (calc-add-fractions 0.1 (frac 1 10))
> >>   (math-add 0.1 (frac 1 10))
> >
> > You again play against the rules: 0.1 is not a valid object for these
> > functions.  Try
> >
> >   M-: (math-add '(float 1 -1) '(frac 1 10))
> 
> My point is that the error message could be nicer.

For internal functions that don't expect to see objects other than
what they were designed to handle?  Nice error messages are for the
application level, not for this level, IMO.

as for Calc and the math library

[Christopher Dimech]

> Sent: Monday, August 12, 2024 at 6:52 AM
> From: "Eli Zaretskii" <eliz@gnu.org>
> To: "Ihor Radchenko" <yantar92@posteo.net>
> Cc: incal@dataswamp.org, emacs-devel@gnu.org
> Subject: Re: as for Calc and the math library
>
> > From: Ihor Radchenko <yantar92@posteo.net>
> > Cc: incal@dataswamp.org, emacs-devel@gnu.org
> > Date: Sun, 11 Aug 2024 18:27:16 +0000
> >
> > Eli Zaretskii <eliz@gnu.org> writes:
> >
> > >> 1. Documenting some Calc data structures
> > >> 2. Possibly refactoring some Calc functions to use simpler API.
> > >>
> > >> It would be a much easier task and can improve the existing
> > >> functionality.
> > >
> > > I don't think I understand well enough what will this entail in
> > > practice.  Can you show an example of doing this for a couple of
> > > functions?
> >
> > For (1), I was referring to ";;;; Arithmetic routines." in
> > calc.el.
>
> (1) is documentation of Calc data structures.  If you mean the large
> commentary that describes the various form of objects that Calc can
> handle, then what is left to document?
>
> > Now, people must simply know the internal format of special
> > number representations to work with Calc from Elisp.
> > It might be worth (a) putting them into a proper manual (some kind of
> > Calc Lisp API description); (b) maybe creating constructor functions to
> > abstract the types away.
>
> Doesn't (a) already exist in the Calc manual chapter "Data Types"?
> And if not, then you are talking about enhancing the documentation of
> Calc, which is always welcome, but I fail to see how it is a step
> towards a separate library.
>
> > For (2), for starters, Calc functions often do not even have
> > docstrings. One simply has to guess(?) or read the code to understand
> > what kinds of values can be passed.
>
> And how is this relevant to (2), which is about refactoring some of
> Calc code into hopefully easier-to-use functions?

Well, if the task is focused on refactoring, then we should not change
any implementations if we are to test the results before and after, just
to be sure things are as we think should be.

> > For example, take `math-add' - no
> > docstring, and go try adding invalid arguments:
> >
> > M-: (math-add "foo" '(frac 1 2))
>
> It's a small wonder that you cannot math-add a string, especially
> since none of the objects described in the commentary are any kind of
> string.  More generally, AFAIU these don't have doc strings because
> they are internal functions.
>
> But still, I don't understand the relevance.
>
> > but somehow not
> >
> > M-: (math-add 0.1 '(frac 1 10))
> >
> > Debugger entered--Lisp error: (wrong-type-argument integer-or-marker-p 2.0)
> >   (math-gcd 2.0 10)
> >   (math-make-frac 2.0 10)
> >   (calc-add-fractions 0.1 (frac 1 10))
> >   (math-add 0.1 (frac 1 10))
>
> You again play against the rules: 0.1 is not a valid object for these
> functions.  Try
>
>   M-: (math-add '(float 1 -1) '(frac 1 10))
>
>