we need *modularity* [last problem] (was: Re: as for Calc and the math library)

we need *modularity* [last problem] (was: Re: as for Calc and the math library)

[Emanuel Berg]

arthur miller wrote:

> I have seen your previous answers to others, and I do
> understand the point you make, but I think the argument
> fails because a module can be just a proxy to
> a non-free library.

Emacs is notoriously not modular. Everything is everywhere,
you seem to have everything but if fact you have nothing.
There is so much stuff you can't find the basics stuff that
isn't there.

Here take a look: what does this name say?

  cl--defalias

Ah, that's a local function in the Common Lisp package, right?

Well, YES, but ...

It is actually an Elisp defun. And that prefix isn't really
a package qualifier, it is just a convention, part of
the name. And the double dash, likewise just a convention, we
don't have package modularity or any of that kind of data
hiding, abstraction, encapsulation ... what is it more called?
Well, modularity!

In Emacs everything is intermixed, Elisp-Emacs,
non-interactive/interactive, data/data processing/the
interface (buffer, buffer, and buffer), what is a file, what
is a package, what is a library, and what is a program - no
one can tell.

Emacs Lisp isn't is fun, but absolutely not any kind
of powerhouse. It is inconsistent in itself and on top of that
all those zillion interfaces, cl-, seq-, slime- (yes, had to
use that today, for `slime-transpose-lists').

It is pretty fast to get something going but complexity
skyrockets.

What we should do, in general: modularity! clear divisions, in
particular, libraries for everything, including those that provide basic
stuff in a complete and consistent way.

We should get real package modularity, transform the prefixes
into real package qualifiers, and throw away the ugly
local ones.

With Elisp, optimize everything for development speed and
convenience, have consistent interfaces as much as possible,
acknowledge that Elisp was underpowered, which is why all the
interfaces came, now, core Elisp and the interfaces should
make peace, one should identify what it is exactly, what are
the to 10 things that people use with cl-, pcase-, seq- etc,
and bring them into a new core Elisp, and the few CL experts
can still use all the rest.

Now, with native compilation we have speed - we have power
with the interfaces if one could harvest it, which one
currently - well, it is very slow and difficult, anyway - the
very, very last spet into maturity and getting out of the
Elisp ghetto (and getting visitors!) are modularity, modern
software principles - by modularity I mean real modularity,
based on technology - not silly conventions - I forgot where
I was - the last step is modularity in general, and libraries
in particular.

Clear cut interfaces. Get away with the ugly prefixes and
error--prone conventions that don't even do anything.

Elisp 3.0! \o/

PS. This is the problem. This series is completed, I said
    I would write 10, I wrote 4. Here are the other parts.
    It is all based on the same theme. Modularity,
    consistency, libraries, modern software practises.
    
    https://lists.gnu.org/archive/html/emacs-devel/2024-08/msg00154.html
    https://lists.gnu.org/archive/html/emacs-devel/2024-08/msg00380.html
    https://lists.gnu.org/archive/html/emacs-devel/2024-08/msg00388.html

PPS. So who is working on real modularity based on the package
     with real public and local functions as we speak? I know
     there are always some guy - at least - working on such
     a basic idea. I know some way. And I know some day.

-- 
underground experts united
https://dataswamp.org/~incal

Re: we need *modularity* [last problem] (was: Re: as for Calc and the math library)

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 8662 bytes --]

>> I have seen your previous answers to others, and I do
>> understand the point you make, but I think the argument
>> fails because a module can be just a proxy to
>> a non-free library.
>
>Emacs is notoriously not modular. Everything is everywhere,
>you seem to have everything but if fact you have nothing.
>There is so much stuff you can't find the basics stuff that
>isn't there.
>
>Here take a look: what does this name say?
>
>  cl--defalias
>
>Ah, that's a local function in the Common Lisp package, right?
>
>Well, YES, but ...
>
>It is actually an Elisp defun. And that prefix isn't really
>a package qualifier, it is just a convention, part of
>the name. And the double dash, likewise just a convention, we
>don't have package modularity or any of that kind of data
>hiding, abstraction, encapsulation ... what is it more called?
>Well, modularity!
>
>In Emacs everything is intermixed, Elisp-Emacs,
>non-interactive/interactive, data/data processing/the
>interface (buffer, buffer, and buffer), what is a file, what
>is a package, what is a library, and what is a program - no
>one can tell.
>
>Emacs Lisp isn't is fun, but absolutely not any kind
>of powerhouse. It is inconsistent in itself and on top of that
>all those zillion interfaces, cl-, seq-, slime- (yes, had to
>use that today, for `slime-transpose-lists').
>
>It is pretty fast to get something going but complexity
>skyrockets.
>
>What we should do, in general: modularity! clear divisions, in
>particular, libraries for everything, including those that provide basic
>stuff in a complete and consistent way.
>
>We should get real package modularity, transform the prefixes
>into real package qualifiers, and throw away the ugly
>local ones.
>
>With Elisp, optimize everything for development speed and
>convenience, have consistent interfaces as much as possible,
>acknowledge that Elisp was underpowered, which is why all the
>interfaces came, now, core Elisp and the interfaces should
>make peace, one should identify what it is exactly, what are
>the to 10 things that people use with cl-, pcase-, seq- etc,
>and bring them into a new core Elisp, and the few CL experts
>can still use all the rest.
>
>Now, with native compilation we have speed - we have power
>with the interfaces if one could harvest it, which one
>currently - well, it is very slow and difficult, anyway - the
>very, very last spet into maturity and getting out of the
>Elisp ghetto (and getting visitors!) are modularity, modern
>software principles - by modularity I mean real modularity,
>based on technology - not silly conventions - I forgot where
>I was - the last step is modularity in general, and libraries
>in particular.
>
>Clear cut interfaces. Get away with the ugly prefixes and
>error--prone conventions that don't even do anything.
>
>Elisp 3.0! \o/
>
>PS. This is the problem. This series is completed, I said
>    I would write 10, I wrote 4. Here are the other parts.
>    It is all based on the same theme. Modularity,
>    consistency, libraries, modern software practises.
>
>    https://lists.gnu.org/archive/html/emacs-devel/2024-08/msg00154.html
>    https://lists.gnu.org/archive/html/emacs-devel/2024-08/msg00380.html
>    https://lists.gnu.org/archive/html/emacs-devel/2024-08/msg00388.html
>
>PPS. So who is working on real modularity based on the package
>     with real public and local functions as we speak? I know
>     there are always some guy - at least - working on such
>     a basic idea. I know some way. And I know some day.

Emacs Lisp like all other programming languages has its problems. I am not sure
though all the things you have taken up in those mails are problems. I haven't
answered on previous ones, more than a cursory suggestion about adding some
explanation to the manual in one mail about FFI which seem to be just ingored.
IMO, we have to understand Emacs as application and what it has to do to
achieve its goal of an interactive text editor, in addition to be also a Lisp
implementation.

At low level, there is the central idiom, buffer, implemented as a gap buffer,
which has to be manipulated in some way. IMO, something like "goto-char", sounds
much better than "move-gap" in the context of text editing. They need to have
some low-level primitives on which to implement the higher level ones. Compare
to some other editors, like VSC or whichever. They are all in pretty similar
situation, since very few programming languages do have a "text" data structure
built into the language.

They need some sort of "assembly" to build on. As discussed in the mentioned
mail from yesterday or the day before, I think it could be added to the manual
about the programming model used in low-level text and buffer management: make
an object current, and than act upon it. This is not so uncommon, stack-based
procedural approach. PostScript and OpenGL are two notable languages/API that
use similar approach. There are probably others I am not familiar with. I don't
know if that was popular programming model during 80's when Emacs was designed
or not, but Emacs Lisp is not alone in its design.

I don't know what is a better alternative? Look at Xlib and numerous arguments
their functions take. Win32 uses structures to pass arguments, which leads to
less function arguments passed around, but one has to setup structures before
they are passed to a function. It is also ads verbosity. Yuri mentioned
iterators, but I think iterators would have similar problem as markers: one has
to update all iterators into a buffer if one of them writes to the buffer. I am
not sure, but iterator could be seen as "functional marker", i.e. a functional
interface to markers? Since Emacs Lisp does not have doted notation for
functions that work on objects, like some other languages that use iterators
extensively (C++, Java), I think iterators and markers are basically the same
thing in EmacsLisp? I haven't think of it too much, so I might be wrong.

So, I don't know, while yes, you are correct that the programmatic interface is
a low-level, I don't think it is a bad one, or that they had so much more choice
there. There is though nothing that forbids anyone to make a higher-level
interface on top of the low-level one. I think, as someone mentioned in some
earlier mail, org-mode, or perhaps font-lock, regular expressions, or various
programming modes could be seen as a higher level interfaces. At least
conceptually? They could be implemented on top of goto-char, char-before/after
and point, but for the performance reasons are not (just my guess).

Also Emacs has to implement its own command loop and renderer, and since most of
us would like some control over those from Lisp, they need to expose those parts
to Lisp as well. I don't think that that Emacs rendering and text processing is
all very entangled in some bad way. Emacs does implement a console-like
(character) renderer. The design is (somehwat) similar to other similar
projects. For example MS console API implements a similar design, a 2d matrix
of characters, with text properties describing colors and other rendering
attributes for characters. There is a long comment by Gerd in xdisp.c explaining
the design in Emacs.

I will definitely agree with you that Emacs needs modular development, but now
you are opening another can of worms into an already unwelcome discussion :).

I also agree with you that EmacsLisp and C core could get a big cleanup, the API
surface is big, especially now when Lisp can be compiled to machine code via
GCC. There also seem to be some dissonance with the manual, as seen from my
other mail about subrs and some other functions (which I hope will get an
answer), so I definitely also agree that Elisp manual could get a facelift too.

I actually wrote this explanation, including a small CL program ~100 sloc, that
implements toy version of Emacs text processing API for the illustrative purpose
days before, on your first mail about goto-char, but never sent because of
explicitly not spending time of maintainers on the discussion that they probably
don't care much of, so not to waste their time and understanding that the topic,
while interesting is probably unwelcome, and yet, Eli accused me of delibarately
wasting maintainers time and energy. Personally, I think it is interesting to
talk about low-level implementation details, at least to check and further my
own udnerstanding, but it is probably best to continue the discussion via PM if
you are interested.

best regards
/a

[-- Attachment #2: Type: text/html, Size: 27449 bytes --]

Re: we need *modularity* [last problem] (was: Re: as for Calc and the math library)

[Emanuel Berg]

arthur miller wrote:

> [...]

Thanks for your post.

Yes, I should stop rant but this realization that Emacs is
like a huge legacy application - no modularity, no clear
divisions, no sound policy how it happened - just go by
instinct and when problem comes - put a convention in a book
and add more stuff.

At the same time people say Lisp is the most powerful language
by definition and that cl-lib and pcase has ruined Emacs (both
of these are very clean cut in general; in the Emacs jungle,
they are a miracle of order).

With ELPA we have some modularity, we should move stuff from
core Emacs to ELPA and, as said, get real modularity based on
the package with real prefixes and - well, everything real!
And, based on technology and not some wishful thinking
a--human-convention can even reduce such huge disadvantages to
other technology.

So it is _really bad_, no excuses, had I known it was like
this - I don't know what would have happened - anyway ... it
is what it is.

But if we get real modularity we get real libraries and if we
do, there is hope.

-- 
underground experts united
https://dataswamp.org/~incal

Re: we need *modularity* [last problem] (was: Re: as for Calc and the math library)

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 3634 bytes --]

>arthur miller wrote:
>
>> [...]
>
>Thanks for your post.

Thanks for the answer.

>Yes, I should stop rant but this realization that Emacs is
>like a huge legacy application - no modularity, no clear
>divisions, no sound policy how it happened - just go by
>instinct and when problem comes - put a convention in a book
>and add more stuff.

Well, yes, 40 years of development does leave some traces I guess.
Different people, professions, epochs, styles, etc.

The big API surface also might contribute to this. Some duplication is
innevitable, but some problems are solved independently, multiple times,
either because of people not being aware of something already existing (myself
been guilty) or simply NIH-syndrome.

I don't think EmacsLisp nor Lisps in general are alone in this. However, Lisp(s)
are very easy to hack, so it is perhaps seen more often in Lisp world: it is
easier to hack your own than to learn and use someone elses code. Examples in
Emacs world are numerous, but I don't want to offend anyone, and I dont' think
it would change anything, so I won't enumerate neither built-in stuff nor
external and well-known packages.

>At the same time people say Lisp is the most powerful language

I don't know. Lisp is a family of languages. As I have learned from a post by
K. Pitman to comp.lang.lisp, McCarthy asked explicitly for none of Lisp dialects
(or languages) to be named "Lisp".

I don't think, any of Lisp languages in popular use is *the* "most powerful
language" if such a qualification among programming languages would even be
possible, certainly not EmacsLisp. But I do think that Lisp languages can be
very elegant. Or at least they have a potential to be elegant, EmacsLisp not
being an exception.

For the reflection on Python, I have stated my opinion here a couple of years or
more ago, and I still stand with the same. I prefer Lisp to Python, if I may
choose, and since I don't consult any more, I can :-). IMO opinion, there is
nothing really interesting about Python. It is a C-like language with some
extensions found in other C-like languages (OO, etc), with somewhat cleaner
syntax and executed in an interpreter. Its real value is interpreter and dynamic
nature. It is indeed an easy and useful language to use, but as interesting as
VisualBasic to me. It is simple, easy, and boring! These are necessary not bad
characteristic, it is a useful language since it is used by many people, and
there is lots of software written for it.

>by definition and that cl-lib and pcase has ruined Emacs (both
>of these are very clean cut in general; in the Emacs jungle,
>they are a miracle of order).

Well, everyone is entitled to their opinion, aren't they? I don't think you can
do much about it. Not everyone has to agree about everything either. Some people
don't like pcase or cl-lib, so be it.

>With ELPA we have some modularity, we should move stuff from
>core Emacs to ELPA and, as said, get real modularity based on

As Drew use to write: +1 (for moving stuff out to Elpa).

>the package with real prefixes and - well, everything real!
>And, based on technology and not some wishful thinking
>a--human-convention can even reduce such huge disadvantages to
>other technology.
>
>So it is _really bad_, no excuses, had I known it was like
>this - I don't know what would have happened - anyway ... it
>is what it is.
>
>But if we get real modularity we get real libraries and if we
>do, there is hope.

As said in the previous mail, modularity would help, but let us not open another
big, off-topic discussion.

[-- Attachment #2: Type: text/html, Size: 12315 bytes --]