C and Emacs Lisp code parts

C and Emacs Lisp code parts

[Andreas Röhler]

Hi emacs-devel,

last years parts of C code have been switched into the Lisp area. There 
are pro and cons, the cons seems to be an easier maintenance, to protect 
against the lack of skilled C-developers.

The backside is a general slowness, not felt in details of such a 
change, but cumulated.Would liketo see this strategy changed. Rather 
focus at a fast and small core. Reduce the rate of changes maybe. My 
preferred Emacs must not provide everything, but be quick and reliable 
and easy to extend. Emacs Lisp seen as designed for the user-space.

Cheers,

Andreas

Re: C and Emacs Lisp code parts

[Eli Zaretskii]

> From: Andreas Röhler <andreas.roehler@online.de>
> Date: Fri, 1 Jul 2016 10:03:07 +0200
> 
> last years parts of C code have been switched into the Lisp area. There 
> are pro and cons, the cons seems to be an easier maintenance, to protect 
> against the lack of skilled C-developers.
> 
> The backside is a general slowness, not felt in details of such a 
> change, but cumulated.Would liketo see this strategy changed. Rather 
> focus at a fast and small core. Reduce the rate of changes maybe. My 
> preferred Emacs must not provide everything, but be quick and reliable 
> and easy to extend. Emacs Lisp seen as designed for the user-space.

To compare performance, we need a performance test suite.  Without
measuring the impact of these changes, we cannot rationally discuss
the alleged slowdown.  This has been discussed before, but AFAIK no
one is working on that.

Volunteers are welcome to work on such a performance test suite.  Once
available, we can make it mandatory for the results to be presented
when someone comes with a suggestion to move code from C to Lisp, or
vice versa.

Re: C and Emacs Lisp code parts

[Andreas Röhler]

On 01.07.2016 10:13, Eli Zaretskii wrote:
>> From: Andreas Röhler <andreas.roehler@online.de>
>> Date: Fri, 1 Jul 2016 10:03:07 +0200
>>
>> last years parts of C code have been switched into the Lisp area. There
>> are pro and cons, the cons seems to be an easier maintenance, to protect
>> against the lack of skilled C-developers.
>>
>> The backside is a general slowness, not felt in details of such a
>> change, but cumulated.Would liketo see this strategy changed. Rather
>> focus at a fast and small core. Reduce the rate of changes maybe. My
>> preferred Emacs must not provide everything, but be quick and reliable
>> and easy to extend. Emacs Lisp seen as designed for the user-space.
> To compare performance, we need a performance test suite.  Without
> measuring the impact of these changes, we cannot rationally discuss
> the alleged slowdown.

You need a number to believe function running from Emacs Lisp is slower 
than an implementation in C?

Re: C and Emacs Lisp code parts

[Alan Mackenzie]

Hello, Andreas.

On Fri, Jul 01, 2016 at 10:39:26AM +0200, Andreas Röhler wrote:


> On 01.07.2016 10:13, Eli Zaretskii wrote:
> >> From: Andreas Röhler <andreas.roehler@online.de>
> >> Date: Fri, 1 Jul 2016 10:03:07 +0200

> >> last years parts of C code have been switched into the Lisp area. There
> >> are pro and cons, the cons .....

I think you meant "pros" here.

> >> ..... seems to be an easier maintenance, to protect against the
> >> lack of skilled C-developers.

Of course, that fails to protect against any lack of skilled Emacs Lisp
developers.  Which is more likely?

> >> The backside is a general slowness, not felt in details of such a
> >> change, but cumulated.Would liketo see this strategy changed.r

Can you quantify this alleged general slowness?  It could well be (and I
suspect it is) that the rewriting of certain parts of Emacs in Lisp have
had a negligible, unmeasureable impact on Emacs's speed.

Can you be more specific, and identify some of these C -> Lisp changes
which you suspect have slowed Emacs?  To be honest, I'm not aware of any
of these changes at all, with just a vague background awareness they may
have taken place.

> >> Rather focus at a fast and small core. Reduce the rate of changes
> >> maybe. My preferred Emacs must not provide everything, but be quick
> >> and reliable and easy to extend.

I think it is all these things, though there is room for improvement
(which is steadily happening).

> >> Emacs Lisp seen as designed for the user-space.

> > To compare performance, we need a performance test suite.  Without
> > measuring the impact of these changes, we cannot rationally discuss
> > the alleged slowdown.

> You need a number to believe function running from Emacs Lisp is slower 
> than an implementation in C?

You need numbers to show that any such difference is important enough to
suffer the disadvantages of C over Lisp for (slower, more difficult,
maintenance, mainly).

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: C and Emacs Lisp code parts

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 630 bytes --]

>>>>> Andreas Röhler <andreas.roehler@online.de> writes:

> You need a number to believe function running from Emacs Lisp is slower than
> an implementation in C?

Performance and efficiency have many dimensions. Switching out a bus for a
race car might could mean very little if your map is bad.

We've put out the call for a performance czar before, but thus far no takers.
It's an area where much help is needed, and could be an interesting puzzle to
take on.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: C and Emacs Lisp code parts

[Eli Zaretskii]

> From: Andreas Röhler <andreas.roehler@online.de>
> Cc: Eli Zaretskii <eliz@gnu.org>
> Date: Fri, 1 Jul 2016 10:39:26 +0200
> 
> You need a number to believe function running from Emacs Lisp is slower 
> than an implementation in C?

Any Emacs function implemented in C will be run by the Emacs Lisp
interpreter, so the interpreter is always involved, whether you want
it or not, and values need to be converted from their Lisp
representation to the corresponding C representation.  Against this
background of Lisp interpreter calling functions implemented in C, a
reimplementation in Lisp might not be significantly slower in
practical use cases.  Moreover, if the Lisp implementation uses a
different algorithm, it could be even faster.

And then there are functions whose speed doesn't matter at all, like
functions which wait, or interface with slow external devices.

So yes, we need numbers to make rational decisions about this.

Re: C and Emacs Lisp code parts

[Andreas Röhler]

On 01.07.2016 11:16, Alan Mackenzie wrote:
> Hello, Andreas.
>
> Can you quantify this alleged general slowness?

No, just experienced. With several hundred abbrevs in a mode, feels like 
taking minutes when M-x edit-abbrevs is called: 
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=14518


>   It could well be (and I
> suspect it is) that the rewriting of certain parts of Emacs in Lisp have
> had a negligible, unmeasureable impact on Emacs's speed.
>
> Can you be more specific, and identify some of these C -> Lisp changes
> which you suspect have slowed Emacs?

NEWS.23: "* Changes in Specialized Modes and Packages in Emacs 23.1

** Abbrev has been rewritten in Elisp and extended with more flexibility."

Re: C and Emacs Lisp code parts

[Andreas Röhler]

On 01.07.2016 11:25, Eli Zaretskii wrote:
>
> So yes, we need numbers to make rational decisions about this.

Sorry, but that's irrational. Compare doesn't need numbers. To compare 
implementations in C an exec is needed - which doesn't exist yet. If 
written, the reason basically is gone, because C will be faster - unless 
something went wrong, which is a case for debugging rather than benchmarks.

Re: C and Emacs Lisp code parts

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Andreas Röhler <andreas.roehler@online.de>
> Date: Fri, 1 Jul 2016 14:25:22 +0200
> 
> On 01.07.2016 11:25, Eli Zaretskii wrote:
> >
> > So yes, we need numbers to make rational decisions about this.
> 
> Sorry, but that's irrational.

??? Comparing without numbers, based on "feelings", is irrational!

It is a well-known fact that reasoning about code optimizations
without measurements is a fallacy that causes optimizations in wrong
places.

> Compare doesn't need numbers. To compare implementations in C an
> exec is needed - which doesn't exist yet. If written, the reason
> basically is gone, because C will be faster - unless something went
> wrong, which is a case for debugging rather than benchmarks.

I'm not against writing it, I'm just saying that before accepting a C
implementation instead of a Lisp one, we should consider the speedup
factor, because rewriting in C has its downsides (which you mentioned)
that have to be weighed against the advantages.

Re: C and Emacs Lisp code parts

[Eli Zaretskii]

> From: Andreas Röhler <andreas.roehler@online.de>
> Date: Fri, 1 Jul 2016 14:07:37 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, John Wiegley <johnw@gnu.org>
> 
> > Can you quantify this alleged general slowness?
> 
> No, just experienced. With several hundred abbrevs in a mode, feels like 
> taking minutes when M-x edit-abbrevs is called: 
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=14518

In which you were asked to provide some numbers, but gave none.
Please do provide them, and perhaps that bug could be taken care of.

Re: C and Emacs Lisp code parts

[Andy Moreton]

On Fri 01 Jul 2016, Andreas Röhler wrote:

> On 01.07.2016 11:25, Eli Zaretskii wrote:
>>
>> So yes, we need numbers to make rational decisions about this.
>
> Sorry, but that's irrational. Compare doesn't need numbers. To compare
> implementations in C an exec is needed - which doesn't exist yet. If written,
> the reason basically is gone, because C will be faster - unless something went
> wrong, which is a case for debugging rather than benchmarks.

Eli is right. You believe that C is always faster, but most programmers'
intuitions about performance are wrong.  Benchmarks and numbers give a
more objective basis for making rational design decisions.

    AndyM

Re: C and Emacs Lisp code parts

[Andreas Röhler]

On 01.07.2016 11:17, John Wiegley wrote:
>>>>>> Andreas Röhler <andreas.roehler@online.de> writes:
>> You need a number to believe function running from Emacs Lisp is slower than
>> an implementation in C?
> Performance and efficiency have many dimensions. Switching out a bus for a
> race car might could mean very little if your map is bad.
>
> We've put out the call for a performance czar before, but thus far no takers.
> It's an area where much help is needed, and could be an interesting puzzle to
> take on.
>

Indeed - do you have any special task in mind? Would my disclaimer be 
sufficient for the test cases provided maybe?
Or --maybe better-- keep that stuff out of distributed Emacs?

Re: C and Emacs Lisp code parts

[Karl Fogel]

Andy Moreton <andrewjmoreton@gmail.com> writes:
>> On 01.07.2016 11:25, Eli Zaretskii wrote:
>>>
>>> So yes, we need numbers to make rational decisions about this.
>>
>> Sorry, but that's irrational. Compare doesn't need numbers. To compare
>> implementations in C an exec is needed - which doesn't exist yet. If written,
>> the reason basically is gone, because C will be faster - unless something went
>> wrong, which is a case for debugging rather than benchmarks.
>
>Eli is right. You believe that C is always faster, but most programmers'
>intuitions about performance are wrong.  Benchmarks and numbers give a
>more objective basis for making rational design decisions.

Yes, Eli is exactly right, and is being completely rational.

Andreas, for example:

Suppose 100 functions have been converted from C to Lisp.  Of those 100, say that 3 are responsible for 95% of the slowdown that resulted (resulted from the conversion of the group of 100, that is).  Meanwhile, the other 97 are only rarely called, or are never called in loops, whatever -- in other words, they are not responsible for much slowdown.

If you convert all 100 functions back to C, then you lose the maintainability advantages of having the innocent 97 in Lisp, for very little real gain; you could have gotten almost all the gain from just identifying the 3 important functions and converting only them.

That's why Eli's point about profiling first, and basing any decisions on the results, is so important.

The adage "C is faster" is not always correct in any case, as others here have pointed out.  But what I'm saying above is that, even if it *were* always true, it would still not be an argument for converting lots of functions back to C, or for avoiding converting more functions from C to Lisp.  You have to know *which* ones need to be in C.  Manual testing can maybe identify a few of those, and if you can perform such tests and identify some such functions, that would be great.  An even more general way to do it (with correspondingly greater up-front investment, of course) is a performance test suite.

But to just recommend that Emacs stop moving things from C to Lisp, or move things back to C, without knowing *which* things and why, is to make a recommendation too vague to guide anyone's actions.

Best regards,
-Karl

Re: C and Emacs Lisp code parts

[Andreas Röhler]

On 01.07.2016 17:14, Karl Fogel wrote:
> [ ... ]
> But to just recommend that Emacs stop moving things from C to Lisp, or move things back to C, without knowing *which* things and why, is to make a recommendation too vague to guide anyone's actions.
>
> Best regards,
> -Karl
>

Hi Karl and all,

The relative slowness of high-level languages against C is a known thing 
beyond Emacs.

Sure, any new C-replacement needs to be checked and also benchmarked. 
But thats not an action to start with.
BTW does anyone still need benchmarks to learn about slowness of 
propertize and fontification stuff?
The whole display-engine IMO is a hot candidate for a thoroughly 
re-consideration.

Cheers,

Andreas

Re: C and Emacs Lisp code parts

[Eli Zaretskii]

> From: Andreas Röhler <andreas.roehler@online.de>
> Date: Fri, 1 Jul 2016 18:52:42 +0200
> 
> The whole display-engine IMO is a hot candidate for a thoroughly 
> re-consideration.

The display engine is entirely in C already, so I'm not sure what are
you talking about.

Re: C and Emacs Lisp code parts

[Andreas Röhler]

On 01.07.2016 19:13, Eli Zaretskii wrote:
>> From: Andreas Röhler <andreas.roehler@online.de>
>> Date: Fri, 1 Jul 2016 18:52:42 +0200
>>
>> The whole display-engine IMO is a hot candidate for a thoroughly
>> re-consideration.
> The display engine is entirely in C already, so I'm not sure what are
> you talking about.

So stuff in font-lock.el isn't display-related?

Re: C and Emacs Lisp code parts

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Andreas Röhler <andreas.roehler@online.de>
> Date: Fri, 1 Jul 2016 19:36:23 +0200
> 
> > The display engine is entirely in C already, so I'm not sure what are
> > you talking about.
> 
> So stuff in font-lock.el isn't display-related?

font-lock.el puts 'face' text properties on portions of the buffer.
The display engine processes the properties when it renders text for
display.  These are two separate parts of Emacs.

IOW, "display-related" isn't the same as the display engine.  Almost
everything in Emacs is display-related, since every change in buffer
text causes the display to be updated.  But that doesn't mean every
Emacs feature is display-related.

As for font-lock, its features are based on regular expression and
syntax tables, both of which are implemented in C as well.

Re: C and Emacs Lisp code parts

[Alan Mackenzie]

Hello, Andreas.

On Fri, Jul 01, 2016 at 07:36:23PM +0200, Andreas Röhler wrote:


> On 01.07.2016 19:13, Eli Zaretskii wrote:
> >> From: Andreas Röhler <andreas.roehler@online.de>
> >> Date: Fri, 1 Jul 2016 18:52:42 +0200

> >> The whole display-engine IMO is a hot candidate for a thoroughly
> >> re-consideration.
> > The display engine is entirely in C already, so I'm not sure what are
> > you talking about.

> So stuff in font-lock.el isn't display-related?

It is display "related", but then pretty much everything else in Emacs
is display "related" too.

font-lock.el is not part of the display engine, in the sense that term
is used in Emacs.

Do you think that rewriting font-lock.el in C would help speed up Emacs
much?  If you do, why not back up your opinion with some numbers.  You
can get the numbers with M-x profiler-start (followed at a later stage
by M-x profiler-report) or M-x elp-instrument-package (followed at a
later stage by M-x elp-results).

My own feeling is that font-lock.el works fast enough (i.e. even
eliminating the time it takes wouldn't give a noticeable speed up), but
I'm willing to be persuaded otherwise by some good numbers.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: C and Emacs Lisp code parts

[Andreas Röhler]

On 01.07.2016 19:55, Alan Mackenzie wrote:
> Hello, Andreas.
>
> On Fri, Jul 01, 2016 at 07:36:23PM +0200, Andreas Röhler wrote:
>
>
>
> My own feeling is that font-lock.el works fast enough

That was only an example for displayed-related stuff in Emacs Lisp.

If there is no problem with font-lock and updating buffers after change 
etc., the better.
So my impression, a remarkable amount of reports deal with slowness of 
display/re-display were wrong.

Re: C and Emacs Lisp code parts

[Alan Mackenzie]

Hello, Andreas.

On Fri, Jul 01, 2016 at 08:31:11PM +0200, Andreas Röhler wrote:


> On 01.07.2016 19:55, Alan Mackenzie wrote:
> > On Fri, Jul 01, 2016 at 07:36:23PM +0200, Andreas Röhler wrote:

> > My own feeling is that font-lock.el works fast enough

> That was only an example for displayed-related stuff in Emacs Lisp.

> If there is no problem with font-lock and updating buffers after change 
> etc., the better.
> So my impression, a remarkable amount of reports deal with slowness of 
> display/re-display were wrong.

There have been quite a lot of (justified) complaints about the slowness
of C Mode's redisplay.  These wouldn't be helped by rewriting
font-lock.el as font-lock.c.  They might be helped by rewriting parts of
CC Mode in C.  This has already been suggested (by Eli), but hasn't been
explored in any great detail, as yet.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: C and Emacs Lisp code parts

[Andreas Röhler]

On 01.07.2016 20:57, Alan Mackenzie wrote:
> Hello, Andreas.
>
> On Fri, Jul 01, 2016 at 08:31:11PM +0200, Andreas Röhler wrote:
>
>
>> On 01.07.2016 19:55, Alan Mackenzie wrote:
>>> On Fri, Jul 01, 2016 at 07:36:23PM +0200, Andreas Röhler wrote:
>>> My own feeling is that font-lock.el works fast enough
>> That was only an example for displayed-related stuff in Emacs Lisp.
>> If there is no problem with font-lock and updating buffers after change
>> etc., the better.
>> So my impression, a remarkable amount of reports deal with slowness of
>> display/re-display were wrong.
> There have been quite a lot of (justified) complaints about the slowness
> of C Mode's redisplay.  These wouldn't be helped by rewriting
> font-lock.el as font-lock.c.  They might be helped by rewriting parts of
> CC Mode in C.  This has already been suggested (by Eli), but hasn't been
> explored in any great detail, as yet.
>

Seems an interesting approach.

BTW think at the fate of bazaar. Here Python was at stake and for a time 
it growed quickly. Assume the easiness of Python as a factor of success. 
But finally a C-based tool won the race, while bazaar went into bigger 
and bigger issues. Why? :-)

Re: C and Emacs Lisp code parts

[Davis Herring]

> BTW think at the fate of bazaar. Here Python was at stake and for a time
> it growed quickly. Assume the easiness of Python as a factor of success.
> But finally a C-based tool won the race, while bazaar went into bigger
> and bigger issues. Why? :-)

This has officially gotten ridiculous beyond the reach of emoticons.

$ cd /usr/libexec/git-core
$ file * | grep text
git-add--interactive:    a /usr/bin/perl -w script text executable
git-am:                  POSIX shell script text executable
git-bisect:              POSIX shell script text executable
git-difftool:            a /usr/bin/perl script text executable
git-difftool--helper:    POSIX shell script text executable
git-filter-branch:       POSIX shell script text executable
git-instaweb:            POSIX shell script text executable
git-lost-found:          POSIX shell script text executable
git-merge-octopus:       POSIX shell script text executable
git-merge-one-file:      POSIX shell script text executable
git-merge-resolve:       POSIX shell script text executable
git-mergetool:           POSIX shell script text executable
git-mergetool--lib:      POSIX shell script text executable
git-parse-remote:        POSIX shell script text executable
git-pull:                POSIX shell script text executable
git-quiltimport:         POSIX shell script text executable
git-rebase:              POSIX shell script text executable
git-rebase--interactive: POSIX shell script text executable
git-relink:              a /usr/bin/perl script text executable
git-repack:              POSIX shell script text executable
git-request-pull:        POSIX shell script text executable
git-sh-setup:            POSIX shell script text executable
git-stash:               POSIX shell script text executable
git-submodule:           POSIX shell script text executable
git-web--browse:         POSIX shell script text executable

Git, like init (and /etc/init.d), Emacs, NumPy, and basically every 
other non-trivial software system in existence, consists of a mix of 
compiled and interpreted code according to the strengths and weaknesses 
of each.  Everyone knows this.

Everyone also knows that no practical benefit is to be had from the 
(substantial!) effort of rewriting all the high-level code in C (or why 
not hand-tuned assembler?) because of 
<https://en.wikipedia.org/wiki/Amdahl's_law>.  The numbers that govern 
your life in that law are the ones that, erm, everyone has told you need 
to be measured in order to make any kind of useful decisions.

Git is widely celebrated for its efficiency.  Much of that comes from 
the design of its (on-disk) data structures; minimizing the cost of disk 
access is rather more important than the language that is used to search 
them.

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or 
too sparse, it is because mass-energy conversion has occurred during 
shipping.

Re: C and Emacs Lisp code parts

[Tom Tromey]

>>>>> "Andreas" == Andreas Röhler <andreas.roehler@online.de> writes:

Andreas> last years parts of C code have been switched into the Lisp
Andreas> area. There are pro and cons, the cons seems to be an easier
Andreas> maintenance, to protect against the lack of skilled C-developers.

Andreas> The backside is a general slowness, not felt in details of such a
Andreas> change, but cumulated.Would liketo see this strategy changed. Rather
Andreas> focus at a fast and small core. Reduce the rate of changes maybe. My
Andreas> preferred Emacs must not provide everything, but be quick and reliable
Andreas> and easy to extend. Emacs Lisp seen as designed for the user-space.

Another approach is to just write Emacs Lisp, and use a compiler to
translate important parts to C.  I've done a lot of this work, but it
needs some love to get to the finish line.

    https://github.com/tromey/el-compilador

I wrote up the result of one very simple benchmark there -- I hacked the
compiler enough to generate compile-able C.

Tom

Re: C and Emacs Lisp code parts

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 1468 bytes --]

On 2016-07-01 23:32, Tom Tromey wrote:
>>>>>> "Andreas" == Andreas Röhler <andreas.roehler@online.de> writes:
> 
> Andreas> last years parts of C code have been switched into the Lisp
> Andreas> area. There are pro and cons, the cons seems to be an easier
> Andreas> maintenance, to protect against the lack of skilled C-developers.
> 
> Andreas> The backside is a general slowness, not felt in details of such a
> Andreas> change, but cumulated.Would liketo see this strategy changed. Rather
> Andreas> focus at a fast and small core. Reduce the rate of changes maybe. My
> Andreas> preferred Emacs must not provide everything, but be quick and reliable
> Andreas> and easy to extend. Emacs Lisp seen as designed for the user-space.
> 
> Another approach is to just write Emacs Lisp, and use a compiler to
> translate important parts to C.  I've done a lot of this work, but it
> needs some love to get to the finish line.

This. Quoting from your page (which I +1'd a while ago):

    I think Emacs should move more strongly toward self-hosting. Too much of
    Emacs is written in C, and in the long term this should be migrated to
    lisp.

Couldn't agree more.  There are so many benefits to code written in Elisp: readability, simplicity of debugging, simplicity of editing and replacing code, safety, etc.  Moving code to C by hand may be a way to make some functions faster, but there are many other less drastic ways.

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: C and Emacs Lisp code parts

[Eli Zaretskii]

> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> From: Andreas Röhler <andreas.roehler@online.de>
> Date: Fri, 1 Jul 2016 22:04:08 +0200
> 
> BTW think at the fate of bazaar. Here Python was at stake and for a time 
> it growed quickly. Assume the easiness of Python as a factor of success. 
> But finally a C-based tool won the race, while bazaar went into bigger 
> and bigger issues. Why? :-)

Bazaar is fast enough.  Its dropping out of favor was due to the fact
that that it stopped being actively maintained.

Re: C and Emacs Lisp code parts

[Andreas Röhler]

On 01.07.2016 22:31, Davis Herring wrote:
>> BTW think at the fate of bazaar. Here Python was at stake and for a time
>> it growed quickly. Assume the easiness of Python as a factor of success.
>> But finally a C-based tool won the race, while bazaar went into bigger
>> and bigger issues. Why? :-)
>
> This has officially gotten ridiculous beyond the reach of emoticons.
>
> $ cd /usr/libexec/git-core
> $ file * | grep text
> git-add--interactive:    a /usr/bin/perl -w script text executable
> git-am:                  POSIX shell script text executable
> git-bisect:              POSIX shell script text executable
> git-difftool:            a /usr/bin/perl script text executable
> git-difftool--helper:    POSIX shell script text executable
> git-filter-branch:       POSIX shell script text executable
> git-instaweb:            POSIX shell script text executable
> git-lost-found:          POSIX shell script text executable
> git-merge-octopus:       POSIX shell script text executable
> git-merge-one-file:      POSIX shell script text executable
> git-merge-resolve:       POSIX shell script text executable
> git-mergetool:           POSIX shell script text executable
> git-mergetool--lib:      POSIX shell script text executable
> git-parse-remote:        POSIX shell script text executable
> git-pull:                POSIX shell script text executable
> git-quiltimport:         POSIX shell script text executable
> git-rebase:              POSIX shell script text executable
> git-rebase--interactive: POSIX shell script text executable
> git-relink:              a /usr/bin/perl script text executable
> git-repack:              POSIX shell script text executable
> git-request-pull:        POSIX shell script text executable
> git-sh-setup:            POSIX shell script text executable
> git-stash:               POSIX shell script text executable
> git-submodule:           POSIX shell script text executable
> git-web--browse:         POSIX shell script text executable
>
> Git, like init (and /etc/init.d), Emacs, NumPy, and basically every 
> other non-trivial software system in existence, consists of a mix of 
> compiled and interpreted code according to the strengths and 
> weaknesses of each.  Everyone knows this.

So why you tell that?

My suggestion is about re-considerating, reversing a kind of dogmatic 
pro-elisp policy - as I felt it.
While completely agree with Clément's posts here --stressing the 
benefits of Lisp debugging etc.--
these strength is less important WRT lifetime of core functions.

The idea is about reflecting the reasons which part to keep in C, which 
in Lisp. Don't have a quick answer for that. Roughly would favor to look 
what belongs to user-space, what not. Also stuff which is used seldom 
probably doesn't deserve C, etc.


>
> Everyone also knows that no practical benefit is to be had from the 
> (substantial!) effort of rewriting all the high-level code in C (or 
> why not hand-tuned assembler?) because of 
> <https://en.wikipedia.org/wiki/Amdahl's_law>.  The numbers that govern 
> your life in that law are the ones that, erm, everyone has told you 
> need to be measured in order to make any kind of useful decisions.

Amdahl is about processing power raised by number of kernels, no idea 
what the example should contribute here.


>
> Git is widely celebrated for its efficiency.  Much of that comes from 
> the design of its (on-disk) data structures; minimizing the cost of 
> disk access is rather more important than the language that is used to 
> search them.
>
> Davis
>

Still don't understand why qualifying git "C-based" should be wrong.

$ cd git
$ ls

abspath.c                   mailinfo.h
aclocal.m4                  mailmap.c
advice.c                    mailmap.h
advice.h                    Makefile
alias.c                     match-trees.c
alloc.c                     merge-blobs.c
archive.c                   merge-blobs.h
archive.h                   merge.c
archive-tar.c               merge-recursive.c
archive-zip.c               merge-recursive.h
argv-array.c                mergesort.c
argv-array.h                mergesort.h
attr.c                      mergetools
attr.h                      name-hash.c
base85.c                    notes.c
bisect.c                    notes-cache.c
bisect.h                    notes-cache.h
blob.c                      notes.h
blob.h                      notes-merge.c
block-sha1                  notes-merge.h
branch.c                    notes-utils.c
branch.h                    notes-utils.h
builtin                     object.c
builtin.h                   object.h
bulk-checkin.c              pack-bitmap.c
bulk-checkin.h              pack-bitmap.h
bundle.c                    pack-bitmap-write.c
bundle.h                    pack-check.c
cache.h                     pack.h
cache-tree.c                pack-objects.c
cache-tree.h                pack-objects.h
check_bindir                pack-revindex.c
check-builtins.sh           pack-revindex.h
check-racy.c                pack-write.c
ci                          pager.c
color.c                     parse-options.c
color.h                     parse-options-cb.c
column.c                    parse-options.h
column.h                    patch-delta.c
combine-diff.c              patch-ids.c
command-list.txt            patch-ids.h
commit.c                    path.c
commit.h                    pathspec.c
commit-slab.h               pathspec.h
compat                      perl
config.c                    pkt-line.c
config.mak.in               pkt-line.h
config.mak.uname            po
configure.ac                ppc
connect.c                   preload-index.c
connected.c                 pretty.c
connected.h                 prio-queue.c
connect.h                   prio-queue.h
contrib                     progress.c
convert.c                   progress.h
convert.h                   prompt.c
copy.c                      prompt.h
COPYING                     quote.c
credential.c                quote.h
credential-cache.c          reachable.c
credential-cache--daemon.c  reachable.h
credential.h                read-cache.c
credential-store.c          README.md
csum-file.c                 ref-filter.c
csum-file.h                 ref-filter.h
ctype.c                     reflog-walk.c
daemon.c                    reflog-walk.h
date.c                      refs
decorate.c                  refs.c
decorate.h                  refs.h
delta.h                     RelNotes
diff.c                      remote.c
diffcore-break.c            remote-curl.c
diffcore-delta.c            remote.h
diffcore.h                  remote-testsvn.c
diffcore-order.c            replace_object.c
diffcore-pickaxe.c          rerere.c
diffcore-rename.c           rerere.h
diff-delta.c                resolve-undo.c
diff.h                      resolve-undo.h
diff-lib.c                  revision.c
diff-no-index.c             revision.h
dir.c                       run-command.c
dir.h                       run-command.h
Documentation               send-pack.c
editor.c                    send-pack.h
entry.c                     sequencer.c
environment.c               sequencer.h
ewah                        server-info.c
exec_cmd.c                  setup.c
exec_cmd.h                  sha1-array.c
fast-import.c               sha1-array.h
fetch-pack.c                sha1_file.c
fetch-pack.h                sha1-lookup.c
fmt-merge-msg.h             sha1-lookup.h
fsck.c                      sha1_name.c
fsck.h                      shallow.c
generate-cmdlist.sh         shell.c
gettext.c                   sh-i18n--envsubst.c
gettext.h                   shortlog.h
git-add--interactive.perl   show-index.c
git-archimport.perl         sideband.c
git-bisect.sh               sideband.h
git.c                       sigchain.c
git-compat-util.h           sigchain.h
git-cvsexportcommit.perl    split-index.c
git-cvsimport.perl          split-index.h
git-cvsserver.perl          strbuf.c
git-difftool--helper.sh     strbuf.h
git-difftool.perl           streaming.c
git-filter-branch.sh        streaming.h
git-gui                     string-list.c
git-instaweb.sh             string-list.h
gitk-git                    submodule.c
git-merge-octopus.sh        submodule-config.c
git-merge-one-file.sh       submodule-config.h
git-merge-resolve.sh        submodule.h
git-mergetool--lib.sh       symlinks.c
git-mergetool.sh            t
git-p4.py                   tag.c
git-parse-remote.sh         tag.h
git-quiltimport.sh          tar.h
git.rc                      tempfile.c
git-rebase--am.sh           tempfile.h
git-rebase--interactive.sh  templates
git-rebase--merge.sh        thread-utils.c
git-rebase.sh               thread-utils.h
git-relink.perl             trace.c
git-remote-testgit.sh       trace.h
git-request-pull.sh         trailer.c
git-send-email.perl         trailer.h
git-sh-i18n.sh              transport.c
git-sh-setup.sh             transport.h
git-stash.sh                transport-helper.c
git-submodule.sh            tree.c
git-svn.perl                tree-diff.c
GIT-VERSION-GEN             tree.h
gitweb                      tree-walk.c
git-web--browse.sh          tree-walk.h
gpg-interface.c             unicode_width.h
gpg-interface.h             unimplemented.sh
graph.c                     unix-socket.c
graph.h                     unix-socket.h
grep.c                      unpack-trees.c
grep.h                      unpack-trees.h
hashmap.c                   update_unicode.sh
hashmap.h                   upload-pack.c
help.c                      url.c
help.h                      url.h
hex.c                       urlmatch.c
http-backend.c              urlmatch.h
http.c                      usage.c
http-fetch.c                userdiff.c
http.h                      userdiff.h
http-push.c                 utf8.c
http-walker.c               utf8.h
ident.c                     varint.c
imap-send.c                 varint.h
INSTALL                     vcs-svn
khash.h                     version.c
kwset.c                     versioncmp.c
kwset.h                     version.h
levenshtein.c               walker.c
levenshtein.h               walker.h
LGPL-2.1                    wildmatch.c
line-log.c                  wildmatch.h
line-log.h                  worktree.c
line-range.c                worktree.h
line-range.h                wrap-for-bin.sh
list-objects.c              wrapper.c
list-objects.h              write_or_die.c
ll-merge.c                  ws.c
ll-merge.h                  wt-status.c
lockfile.c                  wt-status.h
lockfile.h                  xdiff
log-tree.c                  xdiff-interface.c
log-tree.h                  xdiff-interface.h
mailinfo.c                  zlib.c
$

Re: C and Emacs Lisp code parts

[Andreas Röhler]

On 02.07.2016 08:32, Eli Zaretskii wrote:
>> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
>> From: Andreas Röhler <andreas.roehler@online.de>
>> Date: Fri, 1 Jul 2016 22:04:08 +0200
>>
>> BTW think at the fate of bazaar. Here Python was at stake and for a time
>> it growed quickly. Assume the easiness of Python as a factor of success.
>> But finally a C-based tool won the race, while bazaar went into bigger
>> and bigger issues. Why? :-)
> Bazaar is fast enough.  Its dropping out of favor was due to the fact
> that that it stopped being actively maintained.

AFAIK before Emacs switched to it and after considerable efforts been 
put into it. Nonetheless the complains being slow multiplied. You must 
have seen the echo here on this list.

Assume from a certain point it grow that complex, it ended up 
unmaintainable.
Too many commits and extensions, not enough tests, not enough discuss 
and reflection.

Re: C and Emacs Lisp code parts

[Eli Zaretskii]

> Cc: acm@muc.de, emacs-devel@gnu.org
> From: Andreas Röhler <andreas.roehler@online.de>
> Date: Sat, 2 Jul 2016 09:01:55 +0200
> 
> > Bazaar is fast enough.  Its dropping out of favor was due to the fact
> > that that it stopped being actively maintained.
> 
> AFAIK before Emacs switched to it and after considerable efforts been 
> put into it. Nonetheless the complains being slow multiplied. You must 
> have seen the echo here on this list.

That was years before we switched from it to Git, and the reason for
slowness had nothing to do with it being written in Python.  The
reason was the inefficient protocol, one of many offered by Bazaar,
set up on Savannah.  Once the efficient protocol was installed and
configured, the slowness problems disappeared.

> Assume from a certain point it grow that complex, it ended up 
> unmaintainable.

FUD.

> Too many commits and extensions, not enough tests, not enough discuss 
> and reflection.

FUD.

Look, if you want your ideas and arguments be taken seriously, you may
wish to study better the use cases you bring up, and back up your
arguments with facts and measurements, instead of hand-waving and
FUD.  Do that, and you will see a very different response from this
community: a real inefficiency, evident to everyone, is something
quite a few people here will be fast to dig into, analyze, and propose
ways for improvements.

Re: C and Emacs Lisp code parts

[Paul Eggert]

On 07/02/2016 08:34 AM, Andreas Röhler wrote:
> My suggestion is about re-considerating, reversing a kind of dogmatic 
> pro-elisp policy - as I felt it.
Perhaps you felt more dogmatism than actually exists. As I understand 
it, the attitude is more to prefer Elisp, but to use C when Elisp is not 
technically feasible or when performance would be too slow in Elisp.

Some inertia is involved; when something is first written in one 
language and thoroughly debugged, there is a natural and understandable 
reluctance to change it to the other. So the issue typically comes up 
more often when considering a new feature or a major change, and the 
merits of Elisp vs C can be discussed on a change-by-change basis. If 
you want this process to move forward I suggest proposing a specific 
change along these lines; I think you'll find less dogmatism than the 
above comment would suggest.

Re: C and Emacs Lisp code parts

[Fabrice Popineau]

Eli Zaretskii <eliz <at> gnu.org> writes:

> As for font-lock, its features are based on regular expression and
> syntax tables, both of which are implemented in C as well.

The fact that regexp are implemented in C is not a guarantee of 
the best speed.
Some years ago, when the Common Lisp CL-PPCRE library appeared, 
it was benchmarked 5-20 times faster than C regexp libraries, because
the automaton was compiled directly to machine code, whereas the C
counterpart had to do lookups into the table that describes the
automaton.

I am no specialist of the the current variants of regexp libraries
but some of them seem to make use of jit compiler.
(See http://sljit.sourceforge.net/regex_perf.html for example)
So isnt't there some response time to gain around regexps?
Because I don't see that src/regex.c is using a jit.

Fabrice

Re: C and Emacs Lisp code parts

[Eli Zaretskii]

> From: Fabrice Popineau <fabrice.popineau@gmail.com>
> Date: Sat, 2 Jul 2016 16:41:05 +0000 (UTC)
> 
> Eli Zaretskii <eliz <at> gnu.org> writes:
> 
> > As for font-lock, its features are based on regular expression and
> > syntax tables, both of which are implemented in C as well.
> 
> The fact that regexp are implemented in C is not a guarantee of 
> the best speed.
> Some years ago, when the Common Lisp CL-PPCRE library appeared, 
> it was benchmarked 5-20 times faster than C regexp libraries, because
> the automaton was compiled directly to machine code, whereas the C
> counterpart had to do lookups into the table that describes the
> automaton.

Andreas was proposing to push parts of Emacs to C, so the fact that
the C implementation can be improved is not related to the argument.

> I am no specialist of the the current variants of regexp libraries
> but some of them seem to make use of jit compiler.
> (See http://sljit.sourceforge.net/regex_perf.html for example)
> So isnt't there some response time to gain around regexps?

Maybe there is, but that's a separate issue.  Of course, volunteers
are welcome to come up with more efficient implementations.

Re: C and Emacs Lisp code parts

[Davis Herring]

>> Git, like init (and /etc/init.d), Emacs, NumPy, and basically every
>> other non-trivial software system in existence, consists of a mix of
>> compiled and interpreted code according to the strengths and
>> weaknesses of each.  Everyone knows this.
>
> So why you tell that?
> [...]
> Still don't understand why qualifying git "C-based" should be wrong.

Because you drew attention to Git's C basis as (what appeared to be) an 
argument for moving Emacs in that direction, despite the fact that Emacs 
is already C-based in much the same fashion as Git (other than that 
there are many C functions which take Lisp_Object).

> My suggestion is about re-considerating, reversing a kind of dogmatic
> pro-elisp policy - as I felt it.
> While completely agree with Clément's posts here --stressing the
> benefits of Lisp debugging etc.--
> these strength is less important WRT lifetime of core functions.

The extent of that policy is "put in in Elisp unless there's a good 
reason not to", which is merely a question of burden of proof.  And that 
direction of burden makes sense: it is widely accepted that C 
programming takes longer, so why pay for it in the case where no one can 
point out a reason one way or the other?

> The idea is about reflecting the reasons which part to keep in C, which
> in Lisp. Don't have a quick answer for that. Roughly would favor to look
> what belongs to user-space, what not. Also stuff which is used seldom
> probably doesn't deserve C, etc.

No disagreement here -- but the utility of this conversation is quite 
limited until you have such an answer (quick or otherwise).  Merely 
suggesting that people (who, say, are implementing a new feature and 
need to decide what language to use) consider carefully which language 
to use is not usefully distinct from the current "policy".

> Amdahl is about processing power raised by number of kernels, no idea
> what the example should contribute here.

Amdahl's law is not so limited: see more precisely 
<https://en.wikipedia.org/wiki/Amdahl%27s_law#Speedup_in_a_serial_program>.

Spelled out explicitly: if, in some context, we spend only (say) 20% of 
our time running Lisp code (vs., say, redisplay), then we can improve 
the speed of that context by no more than 25% by rewriting that Lisp in 
any fashion whatsoever (including C).

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or 
too sparse, it is because mass-energy conversion has occurred during 
shipping.

Re: C and Emacs Lisp code parts

[Andreas Röhler]

On 05.07.2016 19:02, Davis Herring wrote:
>
> Amdahl's law is not so limited: see more precisely 
> <https://en.wikipedia.org/wiki/Amdahl%27s_law#Speedup_in_a_serial_program>.
>
> Spelled out explicitly: if, in some context, we spend only (say) 20% 
> of our time running Lisp code (vs., say, redisplay), then we can 
> improve the speed of that context by no more than 25% by rewriting 
> that Lisp in any fashion whatsoever (including C).
>
> Davis
>

Ah, see, thanks! So it should be not even be specific to computers, but 
also apply when cleaning a car, care for a garden etc.

Re: C and Emacs Lisp code parts

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 742 bytes --]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> In which you were asked to provide some numbers, but gave none. Please do
> provide them, and perhaps that bug could be taken care of.

Until we have numbers, no action will be taken here. I am still of the
position that *more* code should be moved from C to Emacs Lisp, and not the
other way around.

If there are cases where this might be a bad idea, measurement is needed to
convince me. As others have said, performance is a complex beast, and human
guesses about how a modern machine operates are more often wrong than not.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: C and Emacs Lisp code parts

[Andreas Röhler]

On 06.07.2016 00:48, John Wiegley wrote:
>>>>>> Eli Zaretskii <eliz@gnu.org> writes:
>> In which you were asked to provide some numbers, but gave none. Please do
>> provide them, and perhaps that bug could be taken care of.
> Until we have numbers, no action will be taken here. I am still of the
> position that *more* code should be moved from C to Emacs Lisp, and not the
> other way around.
>
> If there are cases where this might be a bad idea, measurement is needed to
> convince me. As others have said, performance is a complex beast, and human
> guesses about how a modern machine operates are more often wrong than not.
>

M-x elp-instrument-function RET define-abbrevs RET
do some edits
M-x elp-results RET


abbrev--describe               36050       15.224345396 0.0004223119
abbrev-edit-save-buffer        1           8.034011581   8.034011581
abbrev-edit-save-to-file       1           8.033998314   8.033998314
abbrev--write                  18025       5.8414600190 0.0003240754
define-abbrevs                 1           1.334075568   1.334075568
abbrev-get                     107835      0.5151135200 4.776...e-06
abbrev-table-get               36964       0.1173239610 3.174...e-06
abbrev-table-put               18320       0.0690998279 3.771...e-06
abbrev-table-empty-p           584         0.010703864 1.832...e-05
abbrev-table-p                 584         0.0050513859 8.649...e-06
abbreviate-file-name           91          0.002899625 3.186...e-05
abbrev-table-name              2           0.000254252   0.000127126
abbrev-mode                    1           1.4463e-05    1.4463e-05

Re: C and Emacs Lisp code parts

[Andreas Röhler]

On 06.07.2016 09:25, Andreas Röhler wrote:
>
> M-x elp-instrument-function RET define-abbrevs RET
> do some edits
> M-x elp-results RET
>


"do some edits" is meaningless here, sorry.
define-abbrevs is called when abbrevs are written down into abbrev-file.

Re: C and Emacs Lisp code parts

[Phillip Lord]

Andreas Röhler <andreas.roehler@online.de> writes:

> On 06.07.2016 00:48, John Wiegley wrote:
>>>>>>> Eli Zaretskii <eliz@gnu.org> writes:
>>> In which you were asked to provide some numbers, but gave none. Please do
>>> provide them, and perhaps that bug could be taken care of.
>> Until we have numbers, no action will be taken here. I am still of the
>> position that *more* code should be moved from C to Emacs Lisp, and not the
>> other way around.
>>
>> If there are cases where this might be a bad idea, measurement is needed to
>> convince me. As others have said, performance is a complex beast, and human
>> guesses about how a modern machine operates are more often wrong than not.
>>
>
> M-x elp-instrument-function RET define-abbrevs RET
> do some edits
> M-x elp-results RET
>
>
> abbrev--describe               36050       15.224345396 0.0004223119
> abbrev-edit-save-buffer        1           8.034011581   8.034011581
> abbrev-edit-save-to-file       1           8.033998314   8.033998314
> abbrev--write                  18025       5.8414600190 0.0003240754
> define-abbrevs                 1           1.334075568   1.334075568
> abbrev-get                     107835      0.5151135200 4.776...e-06
> abbrev-table-get               36964       0.1173239610 3.174...e-06
> abbrev-table-put               18320       0.0690998279 3.771...e-06
> abbrev-table-empty-p           584         0.010703864 1.832...e-05
> abbrev-table-p                 584         0.0050513859 8.649...e-06
> abbreviate-file-name           91          0.002899625 3.186...e-05
> abbrev-table-name              2           0.000254252   0.000127126


But what do these numbers mean? Is this better or worse than previously,
and what are you saving?

Phil