Development Speed

Development Speed

[xenodasein--- via Emacs development discussions.]

  Over some time now I have been trying to familiarize myself with Emacs
code-base and to follow emacs-devel.  I have an observation I would like to
ask about, because of course I alone can not assess the level of it's validity
as a mere mortal who doesn't have the man-hours spent on Emacs internals.

  Is Emacs developing too fast?  Contributions seem to focus on increasing
code size rather than reducing it, on adding features and not on
"paying technical debt"?  Maybe these tendencies oscillate healthily over
years and I only observed a period of the former.

  Especially the graphical parts seem to suffer from high-coupling and
low-cohesion?  Where for example the Neovim project seems to have the ability
to arbitrarily change what GUI implementation it uses. This mail seems to have
the same observation about GUI situation:
https://lists.gnu.org/archive/html/emacs-devel/2021-12/msg01866.html

  I couldn't find a "milestones" type of guide, and do not know of future
plans except the ones in TODO file, so maybe there is something regarding to
this topic?

  Large-scale refactoring would make it possible to pursue TODO list items
like "Emacs as word processor" or "Concurrency", or allow us to use C17?

Thoughts?

Re: Development Speed

[Po Lu]

xenodasein--- via "Emacs development discussions." <emacs-devel@gnu.org>
writes:

>   Especially the graphical parts seem to suffer from high-coupling and
> low-cohesion?  Where for example the Neovim project seems to have the ability
> to arbitrarily change what GUI implementation it uses.

We certainly have the same ability in Emacs.  I was able to develop a
fully functional graphical port to Haiku in less than 2 months.

>   Large-scale refactoring would make it possible to pursue TODO list items
> like "Emacs as word processor" or "Concurrency", or allow us to use C17?

Refactoring for refactoring's sake never accomplishes much, and why
should we use C17?  What advantages does it hold?

Thanks.

Re: Development Speed

[xenodasein--- via Emacs development discussions.]

Quoting: https://lists.gnu.org/archive/html/emacs-devel/2021-12/msg01900.html
From: Po Lu
Date: Mon, 20 Dec 2021 08:31:18 +0800

> We certainly have the same ability in Emacs.  I was able to develop a
> fully functional graphical port to Haiku in less than 2 months.

I fail to see how this is accurate or relevant.

> Refactoring for refactoring's sake never accomplishes much

I don't know what refactoring for refactoring's sake is but;
What I mention is primarily done to reduce manhours needed to achieve
some things and and make new contributions easier/possible.
If making changes take too long, it can also indicate that the project is dead.
Or again from Neovim's example, could cause social issues, forks.
Software engineering textbooks have more to say on the topic.

> and why should we use C17?  What advantages does it hold?

This is a legitimate question we can assess.
Features like restricted pointers could improve performance?
But I think the better question is what is the excuse for striving not to
keep up with the latest language standard? (Assuming someone wants to work on
it.)

Re: Development Speed

[Po Lu]

xenodasein--- via "Emacs development discussions." <emacs-devel@gnu.org>
writes:

> I fail to see how this is accurate or relevant.

It shows that you don't need major refactoring to be able to add a new
GUI port of Emacs.

> I don't know what refactoring for refactoring's sake is but;
> What I mention is primarily done to reduce manhours needed to achieve
> some things and and make new contributions easier/possible.

I find any kind of rewrite to increase the number of manhours needed to
achieve things.  Especially when Emacs is involved.

> This is a legitimate question we can assess.  Features like restricted
> pointers could improve performance?  But I think the better question
> is what is the excuse for striving not to keep up with the latest
> language standard? (Assuming someone wants to work on it.)

I think we already use restricted pointers (including the GCC extension
for it) when they are available, as we do with some other features of
newer C standards, such as `_Noreturn'.

There is no "excuse" needed to not drop older language standards, when
the new one is not even universally available.  For example, it wasn't
really available on Haiku until this month, and even now the standard
library (libroot) seems to lack support for some of its features.

We dropped K&R because nobody was using a K&R compiler to build Emacs,
and because it didn't build with such a compiler either.  I regularly
see people building Emacs with a C99 compiler, and even more people
building with a C11 compiler.

For example, I don't think there's a version of GCC that will compile
the DJGPP port and supports C17 either.

Re: Development Speed

[Philip Kaludercic]

xenodasein--- via "Emacs development discussions." <emacs-devel@gnu.org>
writes:

> Quoting: https://lists.gnu.org/archive/html/emacs-devel/2021-12/msg01900.html
> From: Po Lu
> Date: Mon, 20 Dec 2021 08:31:18 +0800
>
>> and why should we use C17?  What advantages does it hold?
>
> This is a legitimate question we can assess.
> Features like restricted pointers could improve performance?
> But I think the better question is what is the excuse for striving not to
> keep up with the latest language standard? (Assuming someone wants to work on
> it.)

I think the main issue is that by raising the language standard, you
break compatibility with systems that do not provide the compiler yet.

-- 
	Philip Kaludercic

Re: Development Speed

[Lars Ingebrigtsen]

xenodasein--- via "Emacs development discussions." <emacs-devel@gnu.org>
writes:

>   Is Emacs developing too fast?  Contributions seem to focus on increasing
> code size rather than reducing it, on adding features and not on
> "paying technical debt"?  Maybe these tendencies oscillate healthily over
> years and I only observed a period of the former.

Yes, I don't think that's accurate.  (But it's certainly the impression
you get when reading emacs-devel, so it's understandable why you'd think
so.)

There's an enormous amount of maintenance work going on in Emacs
development.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Development Speed

[Stefan Monnier]

Regarding paying technical debt, we could do a lot more of it, indeed,
but it takes time and effort and is hard to do without introducing
backward incompatible changes.

More importantly, we can't force people to do it.

This said, I think a lot of the work Emacs maintainers do (as opposed
to developers of 3rd party packages) is indeed just that.  It's just
that some of our debt are "too hard to tackle", so we focus on the
smaller things instead :-(

>> and why should we use C17?  What advantages does it hold?
> This is a legitimate question we can assess.

We have slowly moved from K&R C to ANSI C to newer versions of the
standard over the years, tho by and large we don't really pay attention
to particular versions of the standard but rather to specific features,
since the main issue is whether it's supported without too many bugs in
the various C compilers that we can expect people to use to build Emacs.

> Features like restricted pointers could improve performance?

My guess is that you're going to have a hell of a time coming up with
a contorted enough benchmark for such an improvement to be reliably
measurable in Emacs's code.

If it can lead to cleaner code and/or better errors&warnings, I'm all
for it, but I don't expect performance improvements to come from
the compiler.

> But I think the better question is what is the excuse for striving not
> to keep up with the latest language standard? (Assuming someone wants
> to work on it.)

I suggest you choose your words more carefully: "excuse" above implies
you think we lie, which doesn't seem like a good starting point for an
honest and constructive conversation.


        Stefan

Re: Development Speed

[xenodasein--- via Emacs development discussions.]

Quoting: list-was-down?
From: Po Lu
Date: -

> I looked at neovim, and in fact, I tried to use it for around a year and
> half, and my typical response to these questions is "let's wait for
> neovim to grow as old as Emacs, and see what happens to it".

> Neovim is IMO still very immature: for example, it still doesn't have
> the ability to open multiple frames or display images and multiple fonts
> in single window, which is a limitation shared with Vim, a relatively
> unsophisticated editor.

I completely agree, what I refer to is modularity of it's GUI, on that front
it is better, on anything else it is as you said.

> Most of what you read in a modern software engineering book doesn't
> apply to Emacs. Not to mention that those books tend to be discredited
> as often as psychology books, which is a rather ominous sign.

I mean, there has to be something more to all that research other than just
"Yo, it's like, psychology, man."

> Nobody running a modern GNU/Linux system will realistically suffer the
> deficiencies of such a platform. Care to list some?

Anything that will have been caused by not using latest the C?  Could mean
more bugs, less features...  M-x butterfly

Re: Development Speed

[xenodasein--- via Emacs development discussions.]

Quoting: list-was-down?
From: Lars Ingebrigtsen
Date: -

> Yes, I don't think that's accurate. (But it's certainly the impression
> you get when reading emacs-devel, so it's understandable why you'd think
> so.)

> There's an enormous amount of maintenance work going on in Emacs
> development.

I have indeed read the 10% series, and the monumental effort speaks for
itself. I also see commits from lots of people fixing and tidying things up
everyday, it's great. My sincere thanks as an Emacs user.

I only know of the communication through lists.gnu.org and the TODO file
and I missed any large scale, coordinated plans to structural changes.
I do not mean to say those must exist; but to merely inquire whether there
should be any such, or if they exist or are they documented and can be
accessed without directly requesting comments.

Quoting: list-was-down?
From: Stefan Monnier
Date: -

> Regarding paying technical debt, we could do a lot more of it, indeed,
> but it takes time and effort and is hard to do without introducing
> backward incompatible changes.

> More importantly, we can't force people to do it.

> This said, I think a lot of the work Emacs maintainers do (as opposed
> to developers of 3rd party packages) is indeed just that. It's just
> that some of our debt are "too hard to tackle", so we focus on the
> smaller things instead :-(

This is exactly what I thought I understood about the situation, feels good
to have been able to send at least some thoughts through, from all
my gibberish :)

> ...
> If it can lead to cleaner code and/or better errors&warnings, I'm all
> for it, but I don't expect performance improvements to come from
> the compiler.

I see, that makes sense. Thanks.

> I suggest you choose your words more carefully: "excuse" above implies
> you think we lie, which doesn't seem like a good starting point for an
> honest and constructive conversation.

That wasn't my intention at all, apologies for any offense by mistake.
I am aware of my need of tone tuning; second language problems. I welcome
tips like this, please don't refrain.

Re: Development Speed

[Po Lu]

xenodasein--- via "Emacs development discussions." <emacs-devel@gnu.org>
writes:

> I completely agree, what I refer to is modularity of it's GUI, on that front
> it is better, on anything else it is as you said.

It's not much more modular than what we have in Emacs.  The only real
difference is that it runs in a separate process, which I think also
poses freedom issues.  AFAIK there is at least one proprietary frontend
for neovim.

>> Nobody running a modern GNU/Linux system will realistically suffer the
>> deficiencies of such a platform. Care to list some?

> Anything that will have been caused by not using latest the C?  Could
> mean more bugs, less features...

Details, please.  We want to fix any bugs that crop up anywhere, and I
don't think we're missing out on any features.

Re: Development Speed

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >   Is Emacs developing too fast?  Contributions seem to focus on increasing
  > code size rather than reducing it, on adding features and not on
  > "paying technical debt"?  Maybe these tendencies oscillate healthily over
  > years and I only observed a period of the former.

I think we sometimes go too fast.  However, I don't think we should
be putting a priority on making the code smaller.  Most of the time,
smaller won't do users much good.

There may be specific areas where some cleaning up would be desirable
-- what you think of as "paying technical debt".  But we should limit
that to specific aspects where it could be a big improvement.  Every
change risks causing new bugs.  I think most cleanups would be
counterproductive even if they do make cleaner code.  

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Development Speed

[Eli Zaretskii]

On December 21, 2021 6:15:29 AM GMT+02:00, Richard Stallman <rms@gnu.org> wrote:
> 
> I think we sometimes go too fast.  However, I don't think we should
> be putting a priority on making the code smaller.  Most of the time,
> smaller won't do users much good.
> 
> There may be specific areas where some cleaning up would be desirable
> -- what you think of as "paying technical debt".  But we should limit
> that to specific aspects where it could be a big improvement.  Every
> change risks causing new bugs.  I think most cleanups would be
> counterproductive even if they do make cleaner code.  
> 

IMNSHO, there's no such thing as "too fast development".  There's development that makes too many mistakes at a too high rate, but that's not what we have in Emacs, far from that.

As for the risks of excess cleanups: I share your concern, but if some contributor likes cleaning up code, and he/she does a clean job (pun intended), it is better to sustain that risk than risking to lose that contributor, who might one day proceed to developing new features.

Re: Development Speed

[xenodasein--- via Emacs development discussions.]

(This is a resend, it got lost when list was down.)

Quoting: https://lists.gnu.org/archive/html/emacs-devel/2021-12/msg01930.html
From: Po Lu
Date: Mon, 20 Dec 2021 13:12:40 +0800

First, thanks for your input on this, I really appreciate it.

> It shows that you don't need major refactoring to be able to add a new
> GUI port of Emacs.

Please take a look at Neovim, there is no comparison.
Also the entanglement of TUI and GUI; there is no clear separation without
breaking a lot of lisp code. For example trying to make a minibuffer frame on TUI
just returns nil, as they use the same functions.

> I find any kind of rewrite to increase the number of manhours needed to
> achieve things. Especially when Emacs is involved.

Maybe Emacs lives in a dimension where SE principles and research don't apply,
and if you see no room for improvement whatsoever so be it.

> I think we already use restricted pointers...
> ...
> I regularly see people building Emacs with a C99 compiler,
> and even more people building with a C11 compiler.
> For example, I don't think there's a version of GCC that will compile
> the DJGPP port and supports C17 either.

AFAIK there is almost no visible difference between C11 and C17; a
bug-fix release. And if features like you described are common then Emacs is
already mostly there. Nice.

> There is no "excuse" needed to not drop older language standards, when
> the new one is not even universally available.

Of course the targeting a platform without support is a valid reason but
making GNU/Linux or Windows users suffer the deficiencies of a platform
that cannot compile C11 in this day and age is somewhat cruel IMO.

We needn't use the whole C runtime but the core language should be fair game.

Re: Development Speed

[Po Lu]

xenodasein--- via "Emacs development discussions." <emacs-devel@gnu.org>
writes:

> (This is a resend, it got lost when list was down.)

FWIW, I think I already replied.

Re: Development Speed

[xenodasein--- via Emacs development discussions.]

Quoting: https://lists.gnu.org/archive/html/emacs-devel/2021-12/msg01977.html
From: Richard Stallman
Date: Mon, 20 Dec 2021 23:15:29 -0500

> I think we sometimes go too fast. However, I don't think we should
> be putting a priority on making the code smaller. Most of the time,
> smaller won't do users much good.

I agree if the amount of regular contributions and contributors scale
up with the code size.  Otherwise, wouldn't compromises be necessary?

> There may be specific areas where some cleaning up would be desirable
> -- what you think of as "paying technical debt". But we should limit
> that to specific aspects where it could be a big improvement. Every
> change risks causing new bugs. I think most cleanups would be
> counterproductive even if they do make cleaner code.

That is true, they should be done when a direction necessitates them.
Ones that came to my mind were the TODO items I mentioned, but they
are enormous.

Re: Development Speed

[Eli Zaretskii]

> Date: Tue, 21 Dec 2021 11:25:06 +0100 (CET)
> From: xenodasein--- via "Emacs development discussions." <emacs-devel@gnu.org>
> Please take a look at Neovim, there is no comparison.

Neovim is very young.  Emacs is being developed 4 times as long as
Neovim.  Let's talk when Neovim is comparably old (if it's still
around at that time).

> Also the entanglement of TUI and GUI; there is no clear separation without
> breaking a lot of lisp code. For example trying to make a minibuffer frame on TUI
> just returns nil, as they use the same functions.

This is a feature: Lisp programs almost never need to know whether
they run on text-mode or GUI displays.  If a Lisp program had to ask
on every step whether the display is TTY or GUI, the application could
would be much more complex and hard to read and maintain.  It used to
be that way at some distant past, but we deliberately implemented most
of the GUI features on TTYs: multiple frames, colors (with transparent
translation of X colors), mouse support, menus and dialogs -- in order
to eliminate most of the differences on the Lisp level.  I'm amazed
that you think this is a disadvantage in Emacs.

> > I regularly see people building Emacs with a C99 compiler,
> > and even more people building with a C11 compiler.

We didn't yet move to C11, we require C99.

> > There is no "excuse" needed to not drop older language standards, when
> > the new one is not even universally available.
> 
> Of course the targeting a platform without support is a valid reason but
> making GNU/Linux or Windows users suffer the deficiencies of a platform
> that cannot compile C11 in this day and age is somewhat cruel IMO.

We are not talking about platforms, we are talking about GNU/Linux
systems that still use old GCC versions.  There are quite a lot of
those.  Not everyone runs the latest alpha of the Linux kernel with
the latest snapshot of GCC.  One of the worst "surprises" when
building a new version of a package is to find out it no longer
supports your compiler/linker/libraries, because that's a beginning of
a long journey down the rabbit hole of upgrading your entire system,
and finally probably the hardware as well.  There's nothing nastier
software developers can do to their users than being the tail that
wags the dog.

I'm very happy to say that Emacs was never like that, and I hope it
will never change in this regard.

Re: Development Speed

[Eli Zaretskii]

> Date: Tue, 21 Dec 2021 12:09:36 +0100 (CET)
> Cc: Emacs Devel <emacs-devel@gnu.org>
> From: xenodasein--- via "Emacs development discussions." <emacs-devel@gnu.org>
> > I think we sometimes go too fast. However, I don't think we should
> > be putting a priority on making the code smaller. Most of the time,
> > smaller won't do users much good.
> 
> I agree if the amount of regular contributions and contributors scale
> up with the code size.  Otherwise, wouldn't compromises be necessary?

Our Git repository is public, so everyone can study its statistics.
Do you have any evidence that the rate of changes and contributions in
Emacs deteriorates with the years?

> That is true, they should be done when a direction necessitates them.
> Ones that came to my mind were the TODO items I mentioned, but they
> are enormous.

That's a tribute to the many features that we already have, and their
development over the years.  It isn't like we have a lot of holes in
the feature set filling which is a low-hanging fruit.

RE: [External] : Re: Development Speed

[Drew Adams]

> > Is Emacs developing too fast?  Contributions seem to focus on
> increasing
> > code size rather than reducing it, on adding features and not on
> > "paying technical debt"?  Maybe these tendencies oscillate healthily
> > over years and I only observed a period of the former.
> 
> I think we sometimes go too fast.  However, I don't think we should
> be putting a priority on making the code smaller.  Most of the time,
> smaller won't do users much good.
> 
> There may be specific areas where some cleaning up would be desirable
> -- what you think of as "paying technical debt".  But we should limit
> that to specific aspects where it could be a big improvement.  Every
> change risks causing new bugs.  I think most cleanups would be
> counterproductive even if they do make cleaner code.

+1.

Re: Development Speed

[xenodasein--- via Emacs development discussions.]

Quoting: https://lists.gnu.org/archive/html/emacs-devel/2021-12/msg02015.html
From: Eli Zaretskii
Subject: Re: Development Speed
Date: Tue, 21 Dec 2021 16:52:13 +0200

> Our Git repository is public, so everyone can study its statistics.
> Do you have any evidence that the rate of changes and contributions in
> Emacs deteriorates with the years?

I did not necessarily want to imply that it was an existing problem,
just a what-if.  I do not have it but I suppose some numbers could be
produced using amounts of active contributors, contributions, total LOC,
etc. and ratios as evidence?  There are also software that analyze code
complexity using metrics such as cyclomatic complexity I believe.

My reason own to bring it up was that I absolutely hated what I saw
here: GOOGLE ALERT https://trends.google.com/trends/explore?date=all&q=%2Fm%2F01yp0m,%2Fm%2F07zh7,%2Fm%2F0134xwrk

Re: Development Speed

[Eli Zaretskii]

> Date: Wed, 22 Dec 2021 01:55:56 +0100 (CET)
> From: xenodasein@tutanota.de
> Cc: emacs-devel@gnu.org, Richard Stallman <rms@gnu.org>
> 
> Quoting: https://lists.gnu.org/archive/html/emacs-devel/2021-12/msg02015.html
> From: Eli Zaretskii
> Subject: Re: Development Speed
> Date: Tue, 21 Dec 2021 16:52:13 +0200
> 
> > Our Git repository is public, so everyone can study its statistics.
> > Do you have any evidence that the rate of changes and contributions in
> > Emacs deteriorates with the years?
> 
> I did not necessarily want to imply that it was an existing problem,
> just a what-if.  I do not have it but I suppose some numbers could be
> produced using amounts of active contributors, contributions, total LOC,
> etc. and ratios as evidence?  There are also software that analyze code
> complexity using metrics such as cyclomatic complexity I believe.

We have enough real issues to take up all my free time, I don't need
hypothetical what-if's added to them.  You are welcome to produce such
numbers, and if you think they are of interest, post them.

> My reason own to bring it up was that I absolutely hated what I saw
> here: GOOGLE ALERT https://trends.google.com/trends/explore?date=all&q=%2Fm%2F01yp0m,%2Fm%2F07zh7,%2Fm%2F0134xwrk

The challenge is to identify the _real_ causes of that, and then see
what we can do about them.