Optimising Elisp code

Optimising Elisp code

[Davin Pearson]

Suppose that you have a function:

(defun foo ()
   (bar))

And bar is a function that has no side effects and returns 123 then calling
the function code-optimize (the name of my optimisation routine)

M-x code-optimize on the function foo will result in the following defun 

(defun foo ()
    123)

Re: Optimising Elisp code

[Davin Pearson]

On Friday, October 5, 2018 at 3:15:32 PM UTC+13, Davin Pearson wrote:
> Suppose that you have a function:
> 
> (defun foo ()
>    (bar))
> 
> And bar is a function that has no side effects and returns 123 then calling
> the function code-optimize (the name of my optimisation routine)
> 
> M-x code-optimize on the function foo will result in the following defun 
> 
> (defun foo ()
>     123)

Please could you admise me about whether or not such as function is part of Gnu Emacs or otherwise available as a third party tool.

If not then my code will be reinventing the wheel :-(

Re: Optimising Elisp code

[Emanuel Berg]

Davin Pearson wrote:

> Please could you admise me about whether or
> not such as function is part of Gnu Emacs or
> otherwise available as a third party tool.

Hm, perhaps it will be more educational as for
the purpose and advantages of your tool if you
posted a more live or in-world example of how
you have made use of it?

> If not then my code will be reinventing the
> wheel :-(

Reinventing the wheel is not easy. If you have
done that, cred to you, and it will be
interesting to have a look at it!

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Optimising Elisp code

[tomas]

[-- Attachment #1: Type: text/plain, Size: 676 bytes --]

On Thu, Oct 04, 2018 at 07:15:30PM -0700, Davin Pearson wrote:
> Suppose that you have a function:
> 
> (defun foo ()
>    (bar))
> 
> And bar is a function that has no side effects and returns 123 then calling
> the function code-optimize (the name of my optimisation routine)
> 
> M-x code-optimize on the function foo will result in the following defun 
> 
> (defun foo ()
>     123)

Have you already looked at the code produced by the elisp compiler? As
in doing "compile-function" and "disassemble-function"?

Have you had a look at eval-and-compile and friends?

Perhaps elisp is already doing (parts of) what you have in mind?

Cheers
-- tomás

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Optimising Elisp code

[Noam Postavsky]

On Thu, 4 Oct 2018 at 22:20, Davin Pearson <davin.pearson@gmail.com> wrote:
>
> On Friday, October 5, 2018 at 3:15:32 PM UTC+13, Davin Pearson wrote:
> > Suppose that you have a function:
> >
> > (defun foo ()
> >    (bar))
> >
> > And bar is a function that has no side effects and returns 123 then calling
> > the function code-optimize (the name of my optimisation routine)
> >
> > M-x code-optimize on the function foo will result in the following defun
> >
> > (defun foo ()
> >     123)
>
> Please could you admise me about whether or not such as function is part of Gnu Emacs or otherwise available as a third party tool.

Not exactly, but if you use defsubst to define bar (defsubst bar ()
123), then the result of byte-compiling foo will be equivalent to your
optimized version.

Re: Optimising Elisp code

[Barry Margolin]

In article <638fb7dc-6fc5-4645-8793-97a00038a3a8@googlegroups.com>,
 Davin Pearson <davin.pearson@gmail.com> wrote:

> Suppose that you have a function:
> 
> (defun foo ()
>    (bar))
> 
> And bar is a function that has no side effects and returns 123 then calling
> the function code-optimize (the name of my optimisation routine)
> 
> M-x code-optimize on the function foo will result in the following defun 
> 
> (defun foo ()
>     123)

What you're describing is called inline expansion. AFAIK, the Elisp 
compiler doesn't do this automatically.

You can define bar as a macro -- those HAVE to be expanded at compile 
time, since that's how they work.

You can also define an inline function using defsubst. From the Elisp 
manual:

An inline function works just like an ordinary function except for one 
thing: when you compile a call to the function, the function's 
definition is open-coded into the caller.

Making a function inline makes explicit calls run faster. But it also 
has disadvantages. For one thing, it reduces flexibility; if you change 
the definition of the function, calls already inlined still use the old 
definition until you recompile them.

There are more caveats given in the documentation.

-- 
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

Re: Optimising Elisp code

[Emanuel Berg]

Barry Margolin wrote:

> You can also define an inline function using
> defsubst. From the Elisp manual:
>
> An inline function works just like an
> ordinary function except for one thing: when
> you compile a call to the function, the
> function's definition is open-coded into
> the caller.

This rings a bell from the C++ days when/where
you could define a member function as "inline".
That's all I remember, but maybe it was
something similar to this? I think it amounted
to, instead of making a call, that function was
coded into the object itself, and for this
reason it was suited for really short,
simple functions.

Yes, what does it mean that "the function's
definition is open-coded into the caller"?

If the caller is a function, does this mean
instead of calling the other function, the
other function's code is put into the first
function so there isn't a second call but
everything is executed in the first function?

> Making a function inline makes explicit calls
> run faster.

Well, yeah, obviously since there is no second
invocation with just the original function put
onto the stack!

What are "explicit calls"? Regular calls,
right? But then what are implicit calls?
Anonymous functions? Or is an inline function
being executed an implicit call to
that function?

> But it also has disadvantages. For one thing,
> it reduces flexibility; if you change the
> definition of the function, calls already
> inlined still use the old definition until
> you recompile them.

Another good point, but that's obvious as well.

(Hey, for a person who doesn't understand this,
I sure sound confident enough :))

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Optimising Elisp code

[Emanuel Berg]

> This rings a bell from the C++ days
> when/where you could define a member function
> as "inline". That's all I remember, but maybe
> it was something similar to this? I think it
> amounted to, instead of making a call, that
> function was coded into the object itself,
> and for this reason it was suited for really
> short, simple functions. [...]
>
> If the caller is a function, does this mean
> instead of calling the other function, the
> other function's code is put into the first
> function so there isn't a second call but
> everything is executed in the first function?

OK, in C++ it works like this:

If a member function is defined within the
class definition (not recommended), or if it is
defined outside of it (recommended) with the
word "inline" preceding the return type, then
the function becomes an inline function.

What this means is that when the code is
compiled into machine code, instead of having
one place for the function, and jumping back
and forth every time that function is called,
the machine code for the inline function is
placed, duplicated wherever it is used.

So you get longer machine code, but faster,
since there is just constant execution for the
inline functions, without jumping back
and forth.

When should it be used? If the function is very
short, say 1-3 lines, the jumping back and
forth requires more machine instructions than
the function itself. Then the organizational
gain of having it neatly at one place is
diminished by the speed advantage of not having
to jump back and forth to that place all
the time.

"Make the common case fast, and the complicated
case work" -- M. Yoda

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Optimising Elisp code

[Óscar Fuentes]

Emanuel Berg <moasen@zoho.com> writes:

> OK, in C++ it works like this:
>
> If a member function is defined within the
> class definition (not recommended), or if it is
> defined outside of it (recommended) with the
> word "inline" preceding the return type, then
> the function becomes an inline function.
>
> What this means is that when the code is
> compiled into machine code, instead of having
> one place for the function, and jumping back
> and forth every time that function is called,
> the machine code for the inline function is
> placed, duplicated wherever it is used.
>
> So you get longer machine code, but faster,
> since there is just constant execution for the
> inline functions, without jumping back
> and forth.
>
> When should it be used? If the function is very
> short, say 1-3 lines, the jumping back and
> forth requires more machine instructions than
> the function itself. Then the organizational
> gain of having it neatly at one place is
> diminished by the speed advantage of not having
> to jump back and forth to that place all
> the time.

Just in case any bystander finds this post: don't rely on anything that is
mentioned above, nor for C++ `inline' keyword nor for code inlining in
general.

Re: Optimising Elisp code

[Emanuel Berg]

Óscar Fuentes wrote:

> Emanuel Berg <moasen@zoho.com> writes:
>
>> OK, in C++ it works like this: If a member
>> function is defined within the class
>> definition (not recommended), or if it is
>> defined outside of it (recommended) with the
>> word "inline" preceding the return type,
>> then the function becomes an inline
>> function. What this means is that when the
>> code is compiled into machine code, instead
>> of having one place for the function, and
>> jumping back and forth every time that
>> function is called, the machine code for the
>> inline function is placed, duplicated
>> wherever it is used. So you get longer
>> machine code, but faster, since there is
>> just constant execution for the inline
>> functions, without jumping back and forth.
>> When should it be used? If the function is
>> very short, say 1-3 lines, the jumping back
>> and forth requires more machine instructions
>> than the function itself. Then the
>> organizational gain of having it neatly at
>> one place is diminished by the speed
>> advantage of not having to jump back and
>> forth to that place all the time.
>
> Just in case any bystander finds this post:
> don't rely on anything that is mentioned
> above, nor for C++ `inline' keyword nor for
> code inlining in general.

I read it in this book:

    @book{cpp-direkt,
      author     = {Jan Skansholm},
      ISBN       = {91-44-47931-X},
      publisher  = {Studentlitteratur},
      title      = {C++ direkt},
      year       = {1996}
    }

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Optimising Elisp code

[Óscar Fuentes]

Emanuel Berg <moasen@zoho.com> writes:

>> Just in case any bystander finds this post:
>> don't rely on anything that is mentioned
>> above, nor for C++ `inline' keyword nor for
>> code inlining in general.
>
> I read it in this book:
>
>     @book{cpp-direkt,
>       author     = {Jan Skansholm},
>       ISBN       = {91-44-47931-X},
>       publisher  = {Studentlitteratur},
>       title      = {C++ direkt},
>       year       = {1996}
>     }

The key data point there is "1996". But even then things were not as
simple as you described.

Since long time ago C++ "inline" is really about the ODR ("One
Definition Rule"), not about how the compiler generates code.

Re: Optimising Elisp code

[James K. Lowden]

On Fri, 05 Oct 2018 17:04:17 +0200
Emanuel Berg <moasen@zoho.com> wrote:

> "Make the common case fast, and the complicated
> case work" -- M. Yoda

	Complex solutions are slow for small N, and 
	N is almost always small.  
	-- Rob Pike

--jkl

Re: Optimising Elisp code

[Emanuel Berg]

Óscar Fuentes wrote:

>>> Just in case any bystander finds this post:
>>> don't rely on anything that is mentioned
>>> above, nor for C++ `inline' keyword nor for
>>> code inlining in general.
>>
>> I read it in this book:
>>
>>     @book{cpp-direkt,
>>       author     = {Jan Skansholm},
>>       ISBN       = {91-44-47931-X},
>>       publisher  = {Studentlitteratur},
>>       title      = {C++ direkt},
>>       year       = {1996}
>>     }
>
> The key data point there is "1996". But even
> then things were not as simple as
> you described.

It was basically a translation of what it says.

> Since long time ago C++ "inline" is really
> about the ODR ("One Definition Rule"), not
> about how the compiler generates code.

I have now look it up in another book, namely

    @book{c-programming-language,
      author     = {Bjarne Stroustrup},
      ISBN       = {0-201-53992-6},
      publisher  = {Addison Wesley},
      title      = {The C++ Programming Language},
      year       = 1992
    }

here it says much less, with no attempt to
explain what actually "inline" means, still, it
seems to contradict what you say, because it
says the keyword inline is a "hint to the
compiler" to generate the code inline rather
than have the function called the usual way
(page 124).

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Optimising Elisp code

[Emanuel Berg]

James K. Lowden wrote:

> 	Complex solutions are slow for small N, and 
> 	N is almost always small.  
> 	-- Rob Pike

    Advance too fast, you catch up with death.
    But advance too slow, death catches up with YOU!
    -- The Metabaron

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Optimising Elisp code

[Emanuel Berg]

> I have now look it up in another book, namely
>
>     @book{c-programming-language,
>       author     = {Bjarne Stroustrup},
>       ISBN       = {0-201-53992-6},
>       publisher  = {Addison Wesley},
>       title      = {The C++ Programming Language},
>       year       = 1992
>     }
>
> here it says much less, with no attempt to
> explain what actually "inline" means, still, it
> seems to contradict what you say, because it
> says the keyword inline is a "hint to the
> compiler" to generate the code inline rather
> than have the function called the usual way
> (page 124).

I was about to say "I can do this all night",
but then I'd be a lier because I'm running out
of C++ books. This is the last one

    @book{small-c-how-to-program,
      author     = {H M Deitel and P J Deitel},
      ISBN       = {0-13-185758-4},
      publisher  = {Pearson},
      title      = {Small C++ How to Program},
      year       = 2005
    }

and it says virtually the same thing: "inline"
advices the compiler to "generate a copy of the
function's code in place" rather than the usual
call. (It also says the compiler can choose not
to do this; page 246.)

But I'm not saying you are wrong. I was once
~fluent in *writing* C++, but I didn't claim
then, and certainly do not claim now, to
understand what goes on under the hood.

To young, promising programmers, two pieces of
advice:

1) don't be a professional programmer; and

2) stay away from these books:

%%%% C++

@book{small-c-how-to-program,
  author     = {H M Deitel and P J Deitel},
  ISBN       = {0-13-185758-4},
  publisher  = {Pearson},
  title      = {Small C++ How to Program},
  year       = 2005
}

@book{c-programming-language,
  author     = {Bjarne Stroustrup},
  ISBN       = {0-201-53992-6},
  publisher  = {Addison Wesley},
  title      = {The C++ Programming Language},
  year       = 1992
}

@book{cpp-direkt,
    author     = {Jan Skansholm},
    ISBN       = {91-44-47931-X},
    publisher  = {Studentlitteratur},
    title      = {C++ direkt},
    year       = 1996
}

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Optimising Elisp code

[Emanuel Berg]

> C++ blah blah blah

For anyone reading this, don't be confused by
all the C++ talk, please carry on the Elisp
discussion if that is your desire!

As well as the C++ discussion for that matter.
Programming is NEVER off-topic on an
Emacs newsgroup!

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Optimising Elisp code

[Emanuel Berg]

> I was about to say "I can do this all night",
> but then I'd be a lier because I'm running
> out of C++ books. This is the last one

I was wrong, I have yet another one, but not on
C++, but on C, namely

    @book{vägen-till-c,
      author     = {Ulf Bilting and Jan Skansholm},
      ISBN       = {978-91-44-01468-5},
      publisher  = {Studentlitteratur},
      title      = {Vägen till C},
      year       = 2000
    }

it says, page 279, that into the "new" C99
standard, from C++, they have brought the
"inline" keyword. Then they just say the same
thing, instead of a call, the insertion of
code.

They also say the previous method of getting
somewhat the same stuff has been the use of
macros. By macros, they mean the #preprocessor
replacement of text! So they think the new
"inline" is really good! And one can
appreciate that :)

Now I really need to drink a couple of beers
and ride my bike (bicycle). I haven't written
this much on gnu.emacs.help in ages. Maybe I'm
not getting older after all! That, I'd
appreciate as well...

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Optimising Elisp code

[Stefan Monnier]

>>> function. What this means is that when the
>>> code is compiled into machine code, instead
[...]
>>> function is called, the machine code for the
>>> inline function is placed, duplicated
                    ^^
Maybe, maybe not: it's up to the compiler, because you won't be able to
tell the difference without looking under the hood.

And for the same reason the compiler can also decide to inline when you
*don't* say `inline`, as long as it doesn't affect the behavior of
the program.

Similarly, the compiler may decide to "outline" a chunk of code (take
a part of your function, move it to a new separate function, and
replace it with a call to that new function), or do all kinds of other
fun stuff, such as compile your C++ to Elisp code and combine it with an
Elisp interpreter.


        Stefan

Re: Optimising Elisp code

[Emanuel Berg]

Stefan Monnier wrote:

> Similarly, the compiler may decide to
> "outline" a chunk of code (take a part of
> your function, move it to a new separate
> function, and replace it with a call to that
> new function), or do all kinds of other fun
> stuff, such as compile your C++ to Elisp code
> and combine it with an Elisp interpreter.

Can you explain what inline means in terms
of Elisp?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Knowing where a function has been used (e.g. for optimizing) [Was: Re: Optimising Elisp code]

[Garreau, Alexandre]

On 2018-10-05 at 10:28, Barry Margolin wrote:
> You can define bar as a macro -- those HAVE to be expanded at compile 
> time, since that's how they work.

But what’s the relationship since here, it’s not?

> You can also define an inline function using defsubst. From the Elisp 
> manual:

> Making a function inline makes explicit calls run faster. But it also 
> has disadvantages. For one thing, it reduces flexibility; if you change 
> the definition of the function, calls already inlined still use the old 
> definition until you recompile them.

So aren’t there any dynamical system that would keep a track of
dependencies so that to allow inlining while recompiling the appropriate
functions when changing definitions of inlined stuff?

> An inline function works just like an ordinary function except for one 
> thing: when you compile a call to the function, the function's 
> definition is open-coded into the caller.

Doesn’t lexical scoping also lead to that behavior?

> What you're describing is called inline expansion. AFAIK, the Elisp 
> compiler doesn't do this automatically.

Does it automatically for lexically scoped function definitions?

Is it only because then the behavior would differ and because there’s
currently no way to, say, tell to defun, each time it redefines
something that has been inlined, to also recompile every functions which
included it?

There’s already, for each currently defined function, a handy way to get
back to where it was defined.  But I always noticed this relation is
one-way: there’s no way, for each currently defined function, to get
back to where it was called.  I always missed it because it would gives
a quick way to look at exemples of how a function is used (examples are
the other great way of learning: inherenthly more imperfect than a
definition but quicker and easier), but I feel like, there, if it
existed, it would also allow to generalize some optimizations that
already are handy for compiled, non-dynamic code (because afaik
recursively do this and other expansions, reducing, simplification,
could lead to making abstraction free of performance overhead, right?).

Re: Optimising Elisp code

[Stefan Monnier]

> What you're describing is called inline expansion.
> AFAIK, the Elisp compiler doesn't do this automatically.

Indeed, it doesn't do it automatically because it doesn't know how to
(automatically) undo it, and it has visible effects w.r.t advice,
debug-on-entry, etc...

> You can define bar as a macro -- those HAVE to be expanded at compile 
> time, since that's how they work.
>
> You can also define an inline function using defsubst. From the Elisp 
> manual:

And also with cl-defsubst, with (declare (compiler-macro ...))
and more recently with define-inline.


        Stefan

Re: Optimising Elisp code

[Garreau, Alexandre]

On 2018-10-06 at 10:40, Stefan Monnier wrote:
>> What you're describing is called inline expansion.
>> AFAIK, the Elisp compiler doesn't do this automatically.
>
> Indeed, it doesn't do it automatically because it doesn't know how to
> (automatically) undo it, and it has visible effects w.r.t advice,
> debug-on-entry, etc...

Why “undo”? do you mean take the byte-compiled form and programmatically
do on it something like find and extract the part that comes from
byte-compiling the inlined function?

Or would “reevaluate every function that where defined with the
(now-redefined) inlined function inside” work too?

Re: Optimising Elisp code

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1234 bytes --]

On Sat, Oct 06, 2018 at 06:55:54PM +0200, Garreau, Alexandre wrote:
> On 2018-10-06 at 10:40, Stefan Monnier wrote:
> >> What you're describing is called inline expansion.
> >> AFAIK, the Elisp compiler doesn't do this automatically.
> >
> > Indeed, it doesn't do it automatically because it doesn't know how to
> > (automatically) undo it, and it has visible effects w.r.t advice,
> > debug-on-entry, etc...
> 
> Why “undo”? [...]

because...

 - when debugging, you want to know that function "foo" is being
   called
 - you want, perhaps, to advice/instrument the function after its
   entrails have been spread across some other byte code
 - because you expect it to be possible to redefine the function
   at run time (the run time system would have to know how to
   excise the inlined and perhaps optimised bits and pieces and
   replace them by something else)

> Or would “reevaluate every function that where defined with the
> (now-redefined) inlined function inside” work too?

Hmm. Not reevaluate, but recompile, and not "inside", but rather
"outside" -- like finding all places where this function was
inlined and do some magic there. Perhaps doable, but not trivial.

Cheers
-- t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Garreau, Alexandre]

Le 06/10/2018 à 21h24, tomas@tuxteam.de a écrit :
> On Sat, Oct 06, 2018 at 06:55:54PM +0200, Garreau, Alexandre wrote:
>> Or would “reevaluate every function that where defined with the
>> (now-redefined) inlined function inside” work too?
>
> Hmm. Not reevaluate, but recompile, and not "inside", but rather
> "outside" -- like finding all places where this function was
> inlined 

I don’t get how “outside” is the correct form here, as this is precisely
what I was meaning: I mentioned it in my other message titled “Knowing
where a function has been used (e.g. for optimizing)” (message-id: [0],
url: [1]), as I found a such functionality, symetrical to
find-definition, would be quite generally useful for lot of stuff.

> and do some magic there.

What other kind of magic than (maybe recursively)
reevaluating/recompiling could be needed there?

> Perhaps doable, but not trivial.

I have really no precise idea of where and how is stored the place where
each function was defined wrt the related function’s symbol (I tried to
search somewhere ago and found it must be linked with load-history, but,
because of the absence of a such feature ^^, I don’t know where and how
it is updated, without further searching (by grepping?)), but would it
be so hard storing a symbol at the right place (near the place where it
is said where that symbol is defined (and/or other informations about
this symbol)) for each function (why not also variable or even macro?
symbol generically) refered in the function body?

>> Why “undo”? [...]
>
> because...
>
>  - when debugging, you want to know that function "foo" is being
>    called
>  - you want, perhaps, to advice/instrument the function after its
>    entrails have been spread across some other byte code
>  - because you expect it to be possible to redefine the function
>    at run time (the run time system would have to know how to
>    excise the inlined and perhaps optimised bits and pieces and
>    replace them by something else)

So, except maybe wrt debugging (which there should imply reevaluation,
like with edebug-defun), reevaluating (or rather recompiling indeed,
since afaiu then, simple evaluation will never optimize anything, unlike
byte-compiling (right?)).

[0] <9oqtevzzzzzz.yga.xxuns.g6.gal_-_@portable.galex-713.eu>
[1] https://lists.gnu.org/archive/html/help-gnu-emacs/2018-10/msg00068.html

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[tomas]

[-- Attachment #1: Type: text/plain, Size: 2695 bytes --]

On Sat, Oct 06, 2018 at 09:55:09PM +0200, Garreau, Alexandre wrote:
> Le 06/10/2018 à 21h24, tomas@tuxteam.de a écrit :
> > On Sat, Oct 06, 2018 at 06:55:54PM +0200, Garreau, Alexandre wrote:
> >> Or would “reevaluate every function that where defined with the
> >> (now-redefined) inlined function inside” work too?
> >
> > Hmm. Not reevaluate, but recompile, and not "inside", but rather
> > "outside" -- like finding all places where this function was
> > inlined 
> 
> I don’t get how “outside” is the correct form here, as this is precisely
> what I was meaning: I mentioned it in my other message titled “Knowing
> where a function has been used (e.g. for optimizing)” (message-id: [0],
> url: [1]), as I found a such functionality, symetrical to
> find-definition, would be quite generally useful for lot of stuff.

Then I didn't get you right: I gather we both mean the "context" where
the function was "inlined into". If you're lucky, no optimization has
happened *after" you've inlined. Otherwise, you'll have to recompile
the whole context (i.e. parts of the caller).

> > and do some magic there.
> 
> What other kind of magic than (maybe recursively)
> reevaluating/recompiling could be needed there?

In the case of a redefinition of the inlined function you'll have to
find all call sites and do something.
> 
> > Perhaps doable, but not trivial.
> 
> I have really no precise idea of where and how is stored the place where
> each function was defined

Which functions you mean by "each function"? You mean the sites at which
the inlined function is "called" (i.e. inlined), I guess.

Well, I haven't a precise idea either, but for a pretty lucid description
(in the context of Guile, but with pointers to other implementations),
cf

  https://wingolog.org/archives/2018/02/07/design-notes-on-inline-caches-in-guile

> >> Why “undo”? [...]

[...]

> So, except maybe wrt debugging (which there should imply reevaluation,

you mean "recompile"

> like with edebug-defun), reevaluating (or rather recompiling indeed,
> since afaiu then, simple evaluation will never optimize anything, unlike
> byte-compiling (right?)).

Not only debugging, but any kind of redefinition. Emacs lisp is a
dynamic language, and the users expect that when you redefine a
function, the new definition is active; unless the magic is under
user control (you've said "defmacro" or "def-inline", then you know
you've to recompile the callers), the run system has to manage the
mess, i.e. it has to remember all the sites where things have to
be inlined, like in the "inline cache" from the reference above.

Cheers
-- t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Garreau, Alexandre]

Le 06/10/2018 à 22h27, tomas@tuxteam.de a écrit :
> On Sat, Oct 06, 2018 at 09:55:09PM +0200, Garreau, Alexandre wrote:
>> Le 06/10/2018 à 21h24, tomas@tuxteam.de a écrit :
>> > On Sat, Oct 06, 2018 at 06:55:54PM +0200, Garreau, Alexandre wrote:
>> >> Or would “reevaluate every function that where defined with the
>> >> (now-redefined) inlined function inside” work too?
>> >
>> > Hmm. Not reevaluate, but recompile, and not "inside", but rather
>> > "outside" -- like finding all places where this function was
>> > inlined 
>> 
>> I don’t get how “outside” is the correct form here, as this is precisely
>> what I was meaning: I mentioned it in my other message titled “Knowing
>> where a function has been used (e.g. for optimizing)” (message-id: [0],
>> url: [1]), as I found a such functionality, symetrical to
>> find-definition, would be quite generally useful for lot of stuff.
>
> Then I didn't get you right: I gather we both mean the "context" where
> the function was "inlined into".

Yes.

> If you're lucky, no optimization has happened *after" you've inlined.

Lucky? Why? if it didn’t happen, will you really be able to extract the
inlined part to patch it? won’t recompiling it be simplier?

I like the idea of inline optimization precisely because if done
recursively even on a reasonable level it could sometimes lead to great
expansion/reduction and simplification, not only of one function call.

> Otherwise, you'll have to recompile the whole context (i.e. parts of
> the caller).

How do you recompile only a part? you can do this (I’m pretty ignorant
of elisp’s byte-compiler capabilities)?

>> > and do some magic there.
>> 
>> What other kind of magic than (maybe recursively)
>> reevaluating/recompiling could be needed there?
>
> In the case of a redefinition of the inlined function you'll have to
> find all call sites and do something.

that is, recursively recompiling, as I said, right? “do something” is no
more precise than “do some magic”, and “there” (in both messages)
already meant “at the call sites”.

>> > Perhaps doable, but not trivial.
>> 
>> I have really no precise idea of where and how is stored the place where
>> each function was defined
>
> Which functions you mean by "each function"? You mean the sites at which
> the inlined function is "called" (i.e. inlined), I guess.

Not really, including this, but I was speaking about the find-definition
feature of emacs specifically, since afterwards I talk about a symmetric
feature, that is, store the list of symbols refered by each function, even (or
rather, especially) if it is compiled, in a place linked to each
function definition: so if the functions `f', `g' and `h' refer to `a',
having in something linked to the `a' symbol (an alist, a plist,
anything) a list of these.  So the same way you can find-definition of
`a' and see how and where it was defined (like with C-h f, C-h v… and
xref generally), you could see where it has been used: that’s at the
same time a good way of see the use/meaning of `a' by examples rather
than by definition (if the definition didn’t suffice), and that would
allow to recursively draw some dependency graph showing what needs to be
recompiled if `a' is changed.

Maybe even someone could try some experimental optimizations features
such as, let this `a' be a variable, not a thunk, recompile everything
referring to `a' when you change it (thus leading to the same behavior
of a thunk, without side-effects, and with statical optimization)

> Well, I haven't a precise idea either, but for a pretty lucid description
> (in the context of Guile, but with pointers to other implementations),
> cf
>
>   https://wingolog.org/archives/2018/02/07/design-notes-on-inline-caches-in-guile

I was speaking of something more static, not at eval-time, but at
compile-time.  The idea was to keep a big but exhaustive list of usages
(that would be filled at each definition, I guess the same way as
load-history already is), growing with time (possibly directly linked to
the symbol).

>> >> Why “undo”? [...]
>
> [...]
>
>> So, except maybe wrt debugging (which there should imply reevaluation,
>
> you mean "recompile"

Afaik edebug-defun works by reevaluation everywhere, even for
not-compiled functions.  So I meant reevaluation.  Except, you, indeed,
were speaking about the case I was saying I wasn’t speaking about, that
is debugging an already existing (possibly compiled) function, without
reusing its definition (that is reevaluating, or recompiling if
necessary (because it was already compiled)).

>> like with edebug-defun), reevaluating (or rather recompiling indeed,
>> since afaiu then, simple evaluation will never optimize anything, unlike
>> byte-compiling (right?)).
>
> Not only debugging, but any kind of redefinition. Emacs lisp is a
> dynamic language, and the users expect that when you redefine a
> function, the new definition is active; unless the magic is under
> user control (you've said "defmacro" or "def-inline", then you know
> you've to recompile the callers),

That’s why I’m talking about recompiling.

> the run system has to manage the mess, i.e. it has to remember all the
> sites where things have to be inlined, like in the "inline cache" from
> the reference above.

In wingo’s article he speaks about a cache put *inside* a function,
that’s updated *while* evaluation, depending on it.  I’m speaking of the
statical, pre-compilation case.  With this same information, not
guessed, but exhaustive, and not stored *inside* the function, but in
some place related, be it a global dictionary (hashtable, alist or
plist), or in something related to the symbol (its plist? some modified
symbol?): so you can programatically use this list of locations
typically in xref/help buffers, in some experimental paradigm of
programming, etc.  IC is only an optimization, is VM/evaluation-side, is
non-exhaustive/guessed by nature, and is, afaiu, hidden from the user
point-of-view.

Re: Optimising Elisp code

[Barry Margolin]

In article <86r2h44fqg.fsf@zoho.com>, Emanuel Berg <moasen@zoho.com> 
wrote:

> Barry Margolin wrote:
> 
> > You can also define an inline function using
> > defsubst. From the Elisp manual:
> >
> > An inline function works just like an
> > ordinary function except for one thing: when
> > you compile a call to the function, the
> > function's definition is open-coded into
> > the caller.
> 
> This rings a bell from the C++ days when/where
> you could define a member function as "inline".
> That's all I remember, but maybe it was
> something similar to this? I think it amounted
> to, instead of making a call, that function was
> coded into the object itself, and for this
> reason it was suited for really short,
> simple functions.
> 
> Yes, what does it mean that "the function's
> definition is open-coded into the caller"?
> 
> If the caller is a function, does this mean
> instead of calling the other function, the
> other function's code is put into the first
> function so there isn't a second call but
> everything is executed in the first function?

Yes, that's what "open-coded" means. 

> 
> > Making a function inline makes explicit calls
> > run faster.
> 
> Well, yeah, obviously since there is no second
> invocation with just the original function put
> onto the stack!
> 
> What are "explicit calls"? Regular calls,
> right? But then what are implicit calls?
> Anonymous functions? Or is an inline function
> being executed an implicit call to
> that function?

Calling the function with funcall or apply, e.g.

(funcall some-fun x y z)

The variable some-fun might contain an ordinary function or an inline 
function, but the compiler doesn't know what it will be so it can't 
expand it.

> 
> > But it also has disadvantages. For one thing,
> > it reduces flexibility; if you change the
> > definition of the function, calls already
> > inlined still use the old definition until
> > you recompile them.
> 
> Another good point, but that's obvious as well.
> 
> (Hey, for a person who doesn't understand this,
> I sure sound confident enough :))

Yep, you're picking it up OK.

-- 
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1183 bytes --]

On Sat, Oct 06, 2018 at 11:42:12PM +0200, Garreau, Alexandre wrote:

[...]

> Not really, including this, but I was speaking about the find-definition
> feature of emacs specifically, since afterwards I talk about a symmetric
> feature, that is, store the list of symbols refered by each function, even (or
> rather, especially) if it is compiled, in a place linked to each
> function definition: so if the functions `f', `g' and `h' refer to `a',
> having in something linked to the `a' symbol [...]

OK, I roughly got your idea. I'm a bit out of time to discuss those
things at depth (I'd have to do much more reading than I could ATM
to be able to do more than just handwaving), but one last observation
from me: I think what is going to kill you is the inherently dynamic
nature of LISP (think eval: once you reach such a point, potentially
*every* function, even those not yet known at compile time, can be
referenced). This is a trait LISP shares with other languages like
Javascript. Your strategy might make more sense in a context like
Haskell, where the compiler "knows everything" (yeah, sloppy, but
I think you get what I mean).

Cheers
-- tomás

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Knowing where a function has been used (e.g. for optimizing) [Was: Re: Optimising Elisp code]

[Emanuel Berg]

Garreau, Alexandre wrote:

> (examples are the other
> great way of learning: inherenthly more
> imperfect than a definition but quicker and
> easier)

There should always first be the definition,
then one simple example, then a more
complicated example, then the example that
confirms the rule.

In programming, because computers are
deterministic, example number three shouldn't
exist - but sometimes, it does.

For example, when... err

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Emanuel Berg]

Garreau, Alexandre wrote:

> I have really no precise idea of where and
> how is stored the place where each function
> was defined wrt the related function’s symbol
> (I tried to search somewhere ago and found it
> must be linked with load-history, but,
> because of the absence of a such feature ^^,
> I don’t know where and how it is updated,
> without further searching (by grepping?)),
> but would it be so hard storing a symbol at
> the right place (near the place where it is
> said where that symbol is defined (and/or
> other informations about this symbol)) for
> each function (why not also variable or even
> macro? symbol generically) refered in the
> function body?

Style hint: Don't try to say so many things in
such a hurry! Say one thing. OK, now
that's done.

New paragraph about something else.
Say another thing.

Simple.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Emanuel Berg]

tomas wrote:

> Not only debugging, but any kind of
> redefinition. Emacs lisp is a dynamic
> language, and the users expect that when you
> redefine a function, the new definition is
> active; unless the magic is under user
> control (you've said "defmacro" or
> "def-inline", then you know you've to
> recompile the callers), the run system has to
> manage the mess, i.e. it has to remember all
> the sites where things have to be inlined,
> like in the "inline cache" from the
> reference above.

No one in this thread, unless I missed
something, has made an attempt at defining,
explaining, and showing with examples how
inlining is done and what it means in Elisp:

(defun add-two-digits (a b)
  (+ a b) )
  
(defsubst add-two-digits-inline (i j)
  (+ i j) )

(defun add-it ()
  (add-two-digits 5 10)
  (add-two-digits-inline 15 5) )

This byte-compiles with no warnings/errors.

So what happens then, and what happens when
"add-it" is called?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Emanuel Berg]

tomas wrote:

> OK, I roughly got your idea. I'm a bit out of
> time to discuss those things at depth

We should start by discussing *the basics*.
And I'm not just saying that because I don't
understand the finer points. I'm mean, I...
whatever

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Stefan Monnier]

> (defun add-two-digits (a b)
>   (+ a b) )
>   
> (defsubst add-two-digits-inline (i j)
>   (+ i j) )
>
> (defun add-it ()
>   (add-two-digits 5 10)
>   (add-two-digits-inline 15 5) )
>
> This byte-compiles with no warnings/errors.
>
> So what happens then, and what happens when
> "add-it" is called?

Try M-x disassemble RET add-it RET
to see how the two calls where compiled.


        Stefan

Getting functions usage examples [Was: Re: Knowing where a function has been used (e.g. for optimizing) [Was: Re: Optimising Elisp code]]

[Garreau, Alexandre]

On 2018-10-07 at 14:41, Emanuel Berg wrote:
> Garreau, Alexandre wrote:
>
>> (examples are the other
>> great way of learning: inherenthly more
>> imperfect than a definition but quicker and
>> easier)
>
> There should always first be the definition,

Agree.  Yet sometimes people are bad at abstraction, so when the
abstraction grow complex, in order to *more easily* understand the
example, after one trye, one can begin by getting a guess of what it
does, through an example (with a context, of course, because
side-effects (and imperative programming generally) and complex data
stractures), and then come back at the definition.

> then one simple example, then a more
> complicated example, then the example that
> confirms the rule.

as not everybody needs examples, and ideally we should just keep using
only definitions, nobody (we hope) will waste a lot of precious time
putting 2-3 call examples in the docstrings (especially if it’s
something complex so for each you need to describe a valid context).

Since the function is only dynamically typed, unless you begin
inference-typing or basing on symbols names, you will have a hard time
at automatically generating the simplest meaningful examples.

So having a grasp of “how is this function used in reality” may be quite
useful… and *then*, afterwards, someone could try something more
complex, intuitive and experimental, such as of these (potentially
thousands, or more) examples, automatically sorting out the most simple,
the most idiomatics, the better ones.

Then of course I get this may go from quite straightforward for one
special library function used one or three times around, to grow
especially complex as for “list of functions where `+' / `point' /
`progn' is used”.

And, regard to optimization, again, for these examples (the most used
functions), if one got automatically inlined, then it would require too
much recompiling, then the compiler might figure out it shouldn’t inline
it (unless the programmer accept to do so (maybe it could suggests her
about it (wrt its magnitude of usage through the rest of emacs)?)).

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Garreau, Alexandre]

Le 07/10/2018 à 10h10, tomas@tuxteam.de a écrit :
> On Sat, Oct 06, 2018 at 11:42:12PM +0200, Garreau, Alexandre wrote:
>
> [...]
>
>> Not really, including this, but I was speaking about the find-definition
>> feature of emacs specifically, since afterwards I talk about a symmetric
>> feature, that is, store the list of symbols refered by each function, even (or
>> rather, especially) if it is compiled, in a place linked to each
>> function definition: so if the functions `f', `g' and `h' refer to `a',
>> having in something linked to the `a' symbol [...]
>
> OK, I roughly got your idea. I'm a bit out of time to discuss those
> things at depth (I'd have to do much more reading than I could ATM
> to be able to do more than just handwaving), but one last observation
> from me: I think what is going to kill you is the inherently dynamic
> nature of LISP

some calls will be more interesting to inline than others, or more
expensive, especially as their re-inlining (and recompilation of callee)
might be expensive, so the compiler could use that to choose whether
it’s interesting or not, I guess.

In the case in which it’s interesting, I think that *dynamically*
recompiling is enough.

> (think eval: once you reach such a point, potentially *every*
> function, even those not yet known at compile time, can be
> referenced).

eval will references what it gets as argument, when speaking of
inlining, we’re only speaking about body, not arguments: *here* maybe
(and I’m unsure) dynamic IC could do something, indeed.  But it’s not
what we’re talking about: at the most, if they were in lisp (in emacs
they’re not afaik), we could inline something such as apply and get eval
as a simply recursive function (maybe even tail-recursive in some cases?
or that would imply too much of optimized rearranging of calls
probably…) or something alike that would be useful.  But eval is in C
and if we talk about elisp optimization we only talk about stuff in
lisp:  afaiu there’s no way anything can be inlined inside eval, or eval
be inlined inside anything.

Re: Getting functions usage examples [Was: Re: Knowing where a function has been used (e.g. for optimizing) [Was: Re: Optimising Elisp code]]

[Emanuel Berg]

Garreau, Alexandre wrote:

> as not everybody needs examples, and ideally
> we should just keep using only definitions,
> nobody (we hope) will waste a lot of precious
> time putting 2-3 call examples in the
> docstrings

I was speaking broadly, not about Elisp
docstrings. Like computer books. Learn to do it
all in 21 days. A friend once asked if you
really could learn it all in 21 days. I said,
"yes, if you already know it."

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Emanuel Berg]

Stefan Monnier wrote:

> Try M-x disassemble RET add-it RET
> to see how the two calls where compiled.

OK, here goes:

(defsubst add-it-inline (i j)
  (+ i j) )

(defun add-it (a b)
  (+ a b) )

(defun test-normal ()
  (add-it 1 2) )
;; (test-normal)

;; byte code for test-normal:
;;   args: nil
;; 0       constant  add-it
;; 1       constant  1
;; 2       constant  2
;; 3       call      2
;; 4       return

(defun test-inline ()
  (add-it-inline 10 20) )
;; (test-inline)

;; byte code for test-inline:
;;   args: nil
;; 0       constant  10
;; 1       constant  20
;; 2       varbind   j
;; 3       dup
;; 4       varbind   i
;; 5       varref    j
;; 6       plus
;; 7       unbind    2
;; 8       return

OK, so there is no call, instead the code
is inserted. I suppose this is a glitch in the
Matrix, becuase what I can reCALL this is what
I've been saying this whole time...

But anyway, instead of dwelling on the past,
meditate on this:

;; byte code for add-it:
;;  args: (a b)
;; 0       varref    a
;; 1       varref    b
;; 2       plus      
;; 3       return

So as for machine instructions, the inline
version, "test-inline", has 9.
The "test-normal" has 5. The non-inlined
"add-it" has 4. 5 + 4 = 9. And I've saved the
best forst last: 9 = 9!

So there is no gain as to the number of machine
instructions. So where is the gain? The `call'
instruction specifically, or the implied
funcall overhead not visible in
the disassembly?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Emanuel Berg]

Stefan Monnier wrote:

> Try M-x disassemble RET add-it RET
> to see how the two calls where compiled.

Next question (first was the number of machine
instructions vs. call overhead), next question:
I have 479 `defun's in my Elisp, any general
pointers as how to determine which of those
I should consider inlining?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Optimising Elisp code [again]

[Garreau, Alexandre]

On 2018-10-07 at 17:59, Emanuel Berg wrote:
> Stefan Monnier wrote:
> So there is no gain as to the number of machine instructions. So where
> is the gain? The `call' instruction specifically, or the implied
> funcall overhead not visible in the disassembly?

A smarter compiler could have optimized the discarded value (since we
don’t use its return value, and the inlined function don’t have any
side-effects (which otherwise would have been preserved)), or could have
simplified the result (I checked and with + 2 then - 2 it seems elisp
compiler don’t do this, while I’m sure gcc can) if something was going
to be done with it.

However, inlining is not an optimization made to gain space or reduce
the number of instruction (even worse: normally it could *increase* the
total amount of instruction and therefore use more memory), it’s here to
gain *speed*.  Not by executing less instructions, but by removing a
jump (the funcall), as in today’s computers, jumps (like function calls,
or conditionals) can increase execution time.  So at the end,
duplicating instructions can lead to faster execution, albeit increased
memory usage.

I kind of forgot why jumps are slow, but afair, either because the cache
being little it’s faster to execute nearby code, either because as
modern cpus try to do as much as possible in the same time (pipeline
instructions optimization, found in today’s superscalar cpu, iirc), they
need to know what will be executed next, and they stop searching what’s
next after a jump…  but that (+ loop unrolling) might be mitigated by
the fact that if you use too much memory at same time, it might not fit
in the smaller and faster cache, and then get slower…  so I guess here
it gets complex and architecture dependant, and becomes the job of
complex and native compilers to guess what is best to do.

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Garreau, Alexandre]

On 2018-10-07 at 18:10, Emanuel Berg wrote:
> Stefan Monnier wrote:
>
>> Try M-x disassemble RET add-it RET
>> to see how the two calls where compiled.
>
> Next question (first was the number of machine
> instructions vs. call overhead), next question:
> I have 479 `defun's in my Elisp, any general
> pointers as how to determine which of those
> I should consider inlining?

I’d say the most called ones, and the ones you don’t mind not being able
to modify once you began to use them: like if you inline `f', then you
use it in hundreds of functions, you won’t be able to just redefun `f'
and get all these functions benefit from the change.  So it’s a matter
of tradeof between speed (if that’s that much important) and dynamism.

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Barry Margolin]

In article <86d0slrb4h.fsf@zoho.com>, Emanuel Berg <moasen@zoho.com> 
wrote:

> Stefan Monnier wrote:
> 
> > Try M-x disassemble RET add-it RET
> > to see how the two calls where compiled.
> 
> Next question (first was the number of machine
> instructions vs. call overhead), next question:
> I have 479 `defun's in my Elisp, any general
> pointers as how to determine which of those
> I should consider inlining?

Only worry about the really small ones, less than 5 lines in the body. 
The more time spent in the function itself, the less significant the 
calling overhead is.

And as another answer said, you should only bother with functions that 
are called very frequently. You're only saving a few milliseconds per 
call, so if you only call the function once an hour the savings is 
insignificant.

-- 
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

Re: Getting functions usage examples [Was: Re: Knowing where a function has been used (e.g. for optimizing) [Was: Re: Optimising Elisp code]]

[Van L]

>> then one simple example, then a more
>> complicated example, then the example that
>> confirms the rule.
> 
> as not everybody needs examples, and ideally we should just keep using
> only definitions

Examples are likely cache hits for 80% of people for bridging their gap.

Re: Optimising Elisp code [again]

[Emanuel Berg]

Garreau, Alexandre wrote:

> I kind of forgot why jumps are slow

Isn't it as simple as all the overhead of
placing the new function on the stack with
return pointers and stuff?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Emanuel Berg]

If you have studied the Gnus source code, you
find that the defuns are insanely long. They go
on all but forever. This is because Gnus is
already slow, and perhaps Elisp is as well, so
they don't want to brake it up into modules
(smaller defuns) because then it would require
the funcall overhead. Perhaps Gnus would
benefit from inlining stuff?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Emanuel Berg]

Barry Margolin wrote:

> And as another answer said, you should only
> bother with functions that are called very
> frequently. You're only saving a few
> milliseconds per call, so if you only call
> the function once an hour the savings
> is insignificant.

Is there a tool to find out which functions are
called the most often or do you do that by
using your mind and instinct alone?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Optimising Elisp code [again]

[tomas]

[-- Attachment #1: Type: text/plain, Size: 511 bytes --]

On Mon, Oct 08, 2018 at 04:11:14PM +0200, Emanuel Berg wrote:
> Garreau, Alexandre wrote:
> 
> > I kind of forgot why jumps are slow
> 
> Isn't it as simple as all the overhead of
> placing the new function on the stack with
> return pointers and stuff?

And cold caches, and pipeline stalls and...

Modern processors are quite complex beasts, and much of it has to
do with the sad fact that RAM is much, much, MUCH slower than the
CPU, and they try to lie to you about it...

Cheers
-- tomás

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Garreau, Alexandre]

On 2018-10-08 at 16:18, Emanuel Berg wrote:
> Barry Margolin wrote:
>
>> And as another answer said, you should only
>> bother with functions that are called very
>> frequently. You're only saving a few
>> milliseconds per call, so if you only call
>> the function once an hour the savings
>> is insignificant.
>
> Is there a tool to find out which functions are
> called the most often or do you do that by
> using your mind and instinct alone?

I guess both: knowing what you did you normally should know what
functions are the most “low-level” and “basics” (like “used everywhere”
in loops, or in loops loops, etc.).

But afair, there are also integrated profiling tools for emacs lisp :
(info "(elisp) Profiling")

Naming, and Gnus functions length [Was: Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]]

[Garreau, Alexandre]

On 2018-10-08 at 16:14, Emanuel Berg wrote:
> If you have studied the Gnus source code, you find that the defuns are
> insanely long. They go on all but forever. This is because Gnus is
> already slow, and perhaps Elisp is as well, so they don't want to
> brake it up into modules (smaller defuns) because then it would
> require the funcall overhead. Perhaps Gnus would benefit from inlining
> stuff?

Maybe Gnus functions length are just a question of style.  I guess their
developers are experienced enough to know how to properly use manual
inlining.  And anyway, as, as you pointed it, Gnus can already be
sometimes quite slow, it’s probably more because of I/O than function
call overhead, so this later must be neglictible in comparison.  So Gnus
functions length is probably unrelated.

I put Gnus dev in copy (hoping it’s relevant to the conversation and not
inapropriated to the situation) in case that can help getting more
relevant answers for this.

I personally myself tend to write really big functions, because I like
to have well-named things, and dislike to repeat myself and/or have a
lot of intermediary values and functions with bad names and poor
semantics, so until I come with a proper and nice semantic abstraction I
keep stuff in the same function.

This is unlike some languages like scheme, ocaml, etc. which potentially
requires recursion for loops in some cases, therefore obligating you to
give a name to each loop of your programs (thus potentially resulting in
needing a lot of names, and even potentially bad or complicated ones).

Also unlike pre-algebraic languages which may force you in using lot and
lot of temporary variables, which will inevitably result in either using
very, I find, poor names (such as “double_x = x * 2”), either totally
non-sensical and confusing names such as “A”, “B”, “C”, etc. (TeX is a
notable example where a lot of the loops use recursions, and stuff like
that, you end up needing many names and ending with names with strange
semantics such as “@”, double-“@”, etc. everywhere, resulting in cryptic
code for the unexperienced).

Re: Getting functions usage examples [

[Garreau, Alexandre]

On 2018-10-08 at 11:07, Van L wrote:
>>> then one simple example, then a more
>>> complicated example, then the example that
>>> confirms the rule.
>> 
>> as not everybody needs examples, and ideally we should just keep using
>> only definitions
>
> Examples are likely cache hits for 80% of people for bridging their gap.

Can you be more precise about what do you exactely mean by “bridging
their gap” (to what?) in this context?

Re: Naming, and Gnus functions length [Was: Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]]

[Emanuel Berg]

Garreau, Alexandre wrote:

>> If you have studied the Gnus source code,
>> you find that the defuns are insanely long.
>> They go on all but forever. This is because
>> Gnus is already slow, and perhaps Elisp is
>> as well, so they don't want to brake it up
>> into modules (smaller defuns) because then
>> it would require the funcall overhead.
>> Perhaps Gnus would benefit from
>> inlining stuff?
>
> Maybe Gnus functions length are just
> a question of style. I guess their developers
> are experienced enough to know how to
> properly use manual inlining. And anyway, as,
> as you pointed it, Gnus can already be
> sometimes quite slow, it’s probably more
> because of I/O than function call overhead,
> so this later must be neglictible in
> comparison. So Gnus functions length is
> probably unrelated.

I once said to the Gnus developers, why don't
you break down those age-long functions, to
make it much easier to read and maintain the
code?

They said Gnus is slow, Elisp is slow, and
breaking the defuns up into neat modules would
make it even slower.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Naming, and Gnus functions length [

[Garreau, Alexandre]

On 2018-10-08 at 17:43, Emanuel Berg wrote:
> Garreau, Alexandre wrote:
>>> If you have studied the Gnus source code, you find that the defuns
>>> are insanely long.  They go on all but forever. This is because Gnus
>>> is already slow, and perhaps Elisp is as well, so they don't want to
>>> brake it up into modules (smaller defuns) because then it would
>>> require the funcall overhead.  Perhaps Gnus would benefit from
>>> inlining stuff?
>>
>> Maybe Gnus functions length are just a question of style. I guess
>> their developers are experienced enough to know how to properly use
>> manual inlining. And anyway, as, as you pointed it, Gnus can already
>> be sometimes quite slow, it’s probably more because of I/O than
>> function call overhead, so this later must be neglictible in
>> comparison. So Gnus functions length is probably unrelated.
>
> I once said to the Gnus developers, why don't you break down those
> age-long functions, to make it much easier to read and maintain the
> code?
>
> They said Gnus is slow, Elisp is slow, and breaking the defuns up into
> neat modules would make it even slower.

Seriously? is Gnus so complex (algorithmically at run-time I mean)?
damn, I wouldn’t have figured out…  Then indeed maybe it could… as these
“neat modules” would be only used by Gnus and not all the way around…
but yet: keeping the dynamic behavior (which must be wanted since no
defsubst/define-inline is already used to fix this), while having the
neat modules, if they’re ubiquitous enough, might mean recompiling a big
part of Gnus at each changed function.  So in the thesis of an
automatically-optimizing compiler, maybe the compiler wouldn’t inline
that much…

Re: Naming, and Gnus functions length [

[Emanuel Berg]

Garreau, Alexandre wrote:

> Seriously? is Gnus so complex

Surely you jest! Gnus is probably the most
complex communication facility ever...

Anyway, in the Gnus source, there are
4112 "defun"s, and 122 "defsubst"s.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Naming, and Gnus functions length [Was: Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]]

[Eli Zaretskii]

> From: Emanuel Berg <moasen@zoho.com>
> Date: Mon, 08 Oct 2018 17:43:11 +0200
> 
> I once said to the Gnus developers, why don't
> you break down those age-long functions, to
> make it much easier to read and maintain the
> code?
> 
> They said Gnus is slow, Elisp is slow, and
> breaking the defuns up into neat modules would
> make it even slower.

They are right.

Why is Elisp slow? (was: Re: Naming, and Gnus functions length [Was: Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]])

[Emanuel Berg]

Eli Zaretskii wrote:

>> I once said to the Gnus developers, why
>> don't you break down those age-long
>> functions, to make it much easier to read
>> and maintain the code?
>> 
>> They said Gnus is slow, Elisp is slow, and
>> breaking the defuns up into neat modules
>> would make it even slower.
>
> They are right.

Why is Elisp slow?

Is it funcall overhead that is slow in
particular or is Elisp slow in other aspects
as well?

Is Lisp slow in general? For example, compared
to C? If so, what is the reason for this?

How much faster does it (Elisp) get with the
byte compiler?

How much faster can one get it with inlining,
assuming it is done by a skilled hand?

Are there other speed-up methods other than
inlining. (I mean specifically, not "write
better code".)

Is there like a list of things, functions
perhaps, or data structures, that shouldn't be
used unless necessary, because they are slow?

Is there a tool that I can enable, then use
Emacs a whole day (or night rather), and then
the tool tells me "everything is OK, except for
when you do this, that sticks out as something
that drags down the tempo, so maybe you should
have a look at it"?

Facts for fans: This post contains 12 "?"s!

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Emanuel Berg]

Barry Margolin wrote:

> Only worry about the really small ones, less
> than 5 lines in the body. The more time spent
> in the function itself, the less significant
> the calling overhead is.

OK!

> And as another answer said, you should only
> bother with functions that are called
> very frequently.

How do I know that? Is there a tool to make
a ranking which functions were used the most
during a session?

Excluding the `self-insert-command' and such
functions, of course.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Why is Elisp slow?

[Garreau, Alexandre]

elisp is mostly interpreted (contrarily to C which is always compiled to
native code, so directly executed by the CPU) and very very dynamic
afaik, not that much optimized at byte-compilation afaiu from what I
observed, and I guess not that much optimized from the VM point of view…

Also aren’t you talking about the tool you’re asking about the, I think,
previously mentioned profiling features of elisp? Like mentioned in
(info "(elisp) Profiling") (you do `M-x profiler-start' to start and
then after a while `M-x profiler-report' to get the relevant
informations).

Re: Optimising Elisp code [again]

[Barry Margolin]

In article <mailman.1852.1539009893.1284.help-gnu-emacs@gnu.org>,
 <tomas@tuxteam.de> wrote:

> On Mon, Oct 08, 2018 at 04:11:14PM +0200, Emanuel Berg wrote:
> > Garreau, Alexandre wrote:
> > 
> > > I kind of forgot why jumps are slow
> > 
> > Isn't it as simple as all the overhead of
> > placing the new function on the stack with
> > return pointers and stuff?
> 
> And cold caches, and pipeline stalls and...
> 
> Modern processors are quite complex beasts, and much of it has to
> do with the sad fact that RAM is much, much, MUCH slower than the
> CPU, and they try to lie to you about it...

Since we're talking about a byte-code interpreter, not machine code, the 
pipeline is probably not relevant.

-- 
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Barry Margolin]

In article <86d0sknoud.fsf@zoho.com>, Emanuel Berg <moasen@zoho.com> 
wrote:

> > And as another answer said, you should only
> > bother with functions that are called
> > very frequently.
> 
> How do I know that? Is there a tool to make
> a ranking which functions were used the most
> during a session?

Emacs Lisp has a profiler. Read the section on it in the Elisp manual.

In general you shouldn't even be worrying about this unless your code is 
running too slow to begin with. Then you use the profiler to find the 
bottlenecks and try to improve them. Inline coding is likely to be the 
LEAST effective way to improve performance, except for the simplest 
functions.

-- 
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

Re: Why is Elisp slow?

[Emanuel Berg]

Garreau, Alexandre wrote:

> elisp is mostly interpreted (contrarily to
> C which is always compiled to native code, so
> directly executed by the CPU) and very very
> dynamic afaik

Why can't Elisp be compiled as well?

With "dynamic", do you mean for example one can
change a defun, re-eval it, and the next call,
still from the same process, will execute the
new, modified version?

If so, that's a feature we all love and that's
why Elisp is so addicting because it gives
instant feedback. Still, we do it just to
a tiny tiny fraction of mostly our own Elisp.
The rest stays the same.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Optimising Elisp code [again]

[Emanuel Berg]

Barry Margolin wrote:

> Since we're talking about a byte-code
> interpreter, not machine code, the pipeline
> is probably not relevant.

Cannot one have a three-layer architecture, C,
compiled Lisp, interpretated Lisp?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Emanuel Berg]

Barry Margolin wrote:

> Inline coding is likely to be the LEAST
> effective way to improve performance, except
> for the simplest functions.

If we are talking my personal Elisp - of
course! It will rather be like having an impact
driver in my tool box - it is a good feeling to
have and to understand, but will it improve my
life quality in terms of its functionality,
which obviously is the reason for its
existence? Probably not or at least
very rarely!

If we are talking the insanely long functions
of Gnus, if the funcall overhead is the reason
for that, I don't see why modularization plus
inlining couldn't be one puzzle piece of
the remedy.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Why is Elisp slow?

[Garreau, Alexandre]

On 2018-10-09 at 08:23, Emanuel Berg wrote:
> Garreau, Alexandre wrote:
>
>> elisp is mostly interpreted (contrarily to
>> C which is always compiled to native code, so
>> directly executed by the CPU) and very very
>> dynamic afaik
>
> Why can't Elisp be compiled as well?

Afaik it can, as any turing-complete language, that just requires hard
work: either writing a full native compiler, or interfacing with one
(like gcc).

> With "dynamic", do you mean for example one can change a defun,
> re-eval it, and the next call, still from the same process, will
> execute the new, modified version?
>
> If so, that's a feature we all love and that's why Elisp is so
> addicting because it gives instant feedback. Still, we do it just to a
> tiny tiny fraction of mostly our own Elisp.  The rest stays the same.

For instance.  There are other details such as garbage collection /
dynamic allocation of everything, having everything on heap, dynamic
typing, etc.  But it’s not going to be the most important I guess.

However being able to change anything (or most things) in emacs is a
stated key feature of it, so it makes an full interpreter and the
presence of almost all (elisp) sources mandatory, as well as a fast
(byte-)compiler.

Even more: ideally, to be able to tweak absolutely everything,
everything should be in lisp and nothing in C (or the strict minimal to
make it portable without having to reinvent the wheel of porting to
different architectures in emacs), but afair currently there is still
much in C because indeed elisp is too slow, and the most important
features are needed to be in C so emacs stays usable.

So more work would be needed to give elisp a native compiler… the most
similar approach I recall are gcl (GNU Common Lisp, common lisp is quite
near to elisp) wanting to become a gcc frontend (it’s already compiling
to C), and Guile proposing to become the elisp engine… as guile is
normally faster because more optimizations and stuff, it should have
made elisp faster, but Guile Emacs is actually slow because, afair,
buffer-local variables, dynamic scoping, and low-level stuff like that
are bad supported, and in the end there is really few work on it (and
Guile is far from being as stable as emacs anyway), so as they want to
implement native compilation (but more-or-less (forgot) independently of
gcc afair) too, that’d make a native compiler for elisp… but as the
dynamic stuff is slow on their VM anyway…

Re: Optimising Elisp code [again]

[Garreau, Alexandre]

On 2018-10-09 at 08:26, Emanuel Berg wrote:
> Barry Margolin wrote:
>
>> Since we're talking about a byte-code interpreter, not machine code,
>> the pipeline is probably not relevant.
>
> Cannot one have a three-layer architecture, C, compiled Lisp,
> interpretated Lisp?

It’s already like this… or what are you talking about otherwise?  We got
C, which interprets directly both byte-compiled and interpreted lisp,
which is all the high-level stuff.

Re: Getting functions usage examples [

[Van L]

>>>> then one simple example, then a more
>>>> complicated example, then the example that
>>>> confirms the rule.
>>> 
>>> as not everybody needs examples, and ideally we should just keep using
>>> only definitions
>> 
>> Examples are likely cache hits for 80% of people for bridging their gap.
> 
> Can you be more precise about what do you exactely mean by “bridging
> their gap” (to what?) in this context?

Most people have a goal wanting to be achieved.

They consult the documentation examples to get their 
fill-in the gap in understanding to get there.

Picture Ikea furniture assembly as a way of getting
the thing done. Some non-technology experts or new comers
learn technology in comic or graphic novel reading form.

Re: Why is Elisp slow?

[Stefan Monnier]

> elisp is mostly interpreted (contrarily to C which is always compiled to
                                                      ^^^
                                                    almost

> native code, so directly executed by the CPU) and very very dynamic
> afaik, not that much optimized at byte-compilation afaiu from what I
> observed, and I guess not that much optimized from the VM point of view…

Emacs's implementation of Elisp is fairly naive and not carefully
optimized.  As a result it's not blazing fast.  This said, on the rare
occasions Elisp was compared to other systems it didn't seem nearly as
slow as I thought.  E.g. I wouldn't be surprised to hear that it's about
as fast as Python (not talking about things like PyPy, tho).


        Stefan

Re: Optimising Elisp code [again]

[Emanuel Berg]

Garreau, Alexandre wrote:

>> Cannot one have a three-layer architecture,
>> C, compiled Lisp, interpretated Lisp?
>
> It’s already like this… or what are you
> talking about otherwise? We got C, which
> interprets directly both byte-compiled and
> interpreted lisp, which is all the
> high-level stuff.

Isn't there an FOSS CL compiler around one can
duplicate/modify?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Optimising Elisp code [again]

[Garreau, Alexandre]

On 2018-10-09 at 15:28, Emanuel Berg wrote:
> Garreau, Alexandre wrote:
>>> Cannot one have a three-layer architecture,
>>> C, compiled Lisp, interpretated Lisp?
>>
>> It’s already like this… or what are you
>> talking about otherwise? We got C, which
>> interprets directly both byte-compiled and
>> interpreted lisp, which is all the
>> high-level stuff.
>
> Isn't there an FOSS CL compiler around one can
> duplicate/modify?

gcl. It’s GNU, it uses gcc (for now it generates C and compile that with
gcc, but it was planned to make it a gcc frontend a long time ago, but
iso support comes first afair).  The only other compiler I know is the
most used/popular (afaik), sbcl, but it’s not GNU (so it would be less
meaningful since there are a bunch of GNU lisp implementations)… and
clisp has a *VM* too (and its byte-compiler), but I guess it’s better
than emacs’ one.

However if you’re talking about using a cl compiler to host emacs,
*iirc*, rms said cl was too big for emacs which he wanted not too big
(and indeed, unless some modifications are maid, afaiu, it might imply
to make gcc a mandatory dependency of emacs, which is pretty big)…

Interestingly, on my (debian stable) system, gcl [0] appears to be 7,8M
(but there is a 15M gcl static library [3]), emacs [1] 11M, and libguile
[2] 1,6M… so is gcl smaller than emacs (also maybe it is the static that
contain all gcl, and maybe gcl should be bigger when everything is
complete and standard?)? I can’t measure clisp as I didn’t attempted
(yet) to recompile here, and it’s not packaged for the current debian
stable.

Btw, what would it change to make emacs hosted by gcl, compared to by
guile (modulo the obvious facilities of having elisp and cl more near
than elisp and scheme)? would the gcc runtime dep hard to remove? would
that be so big? would that be easier to get buffer-local variables,
dynamic scoping, et al?

[0] resolves to /usr/lib/gcl-2.6.12/unixport/saved_gcl 
[1] /usr/bin/emacs25-x
[2] /usr/lib/i386-linux-gnu/libguile-2.0.so.22.8.1
[3] /usr/lib/gcl-2.6.12/unixport/libgcl.a

Re: Why is Elisp slow?

[Eli Zaretskii]

> From: "Garreau\, Alexandre" <galex-713@galex-713.eu>
> Date: Tue, 09 Oct 2018 10:05:43 +0200
> 
> > Why can't Elisp be compiled as well?
> 
> Afaik it can, as any turing-complete language, that just requires hard
> work: either writing a full native compiler, or interfacing with one
> (like gcc).

Please take a look at the libjit branch in the Emacs repository, it
implements one idea of doing that.

Re: Optimising Elisp code [again]

[Emanuel Berg]

Garreau, Alexandre wrote:

> Btw, what would it change to make emacs
> hosted by gcl

I mean, compile the Lisp that isn't user
defined and unlikely to change - compile for
real, not byte-compile for
sped-up interpretation.

I.e.: fun-and-play Elisp on the top
      compiled Elisp in the middle
      C downmost

?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Optimising Elisp code [again]

[Garreau, Alexandre]

On 2018-10-09 at 17:26, Emanuel Berg wrote:
> Garreau, Alexandre wrote:
>
>> Btw, what would it change to make emacs
>> hosted by gcl
>
> I mean, compile the Lisp that isn't user
> defined and unlikely to change - compile for
> real, not byte-compile for
> sped-up interpretation.
>
> I.e.: fun-and-play Elisp on the top
>       compiled Elisp in the middle
>       C downmost
>
> ?

We’d get the same problems as mentioned before about loss of dynamism
(you’ll need to recompile everything if changing a core thing, which you
currently don’t need).  Yet the opposite would also be there: some core
functions might be reimplemented in lisp, and then these could be easier
to hack for beginners, or any people uncomfortable with C.

But beside that you would get awesome performances I guess.  Not
*always* like C, because, I guess there are still mallocs and garbage
collection all the way around, types may not be determined as you’d
like, etc.  but especially as elisp is quite imperative (but potentially
less well used, sometimes), you may get quite near performances (I did
hear about 1,2/1,5 more slow than C for compiled elisp).

I’d like to see how the performances of libjit (native jit compiler I
guess) emacs, as previously mentioned, I’ll try it I think.

Re: Optimising Elisp code [again]

[Barry Margolin]

In article <86ftxfm98q.fsf@zoho.com>, Emanuel Berg <moasen@zoho.com> 
wrote:

> Garreau, Alexandre wrote:
> 
> > Btw, what would it change to make emacs
> > hosted by gcl
> 
> I mean, compile the Lisp that isn't user
> defined and unlikely to change - compile for
> real, not byte-compile for
> sped-up interpretation.
> 
> I.e.: fun-and-play Elisp on the top
>       compiled Elisp in the middle
>       C downmost
> 
> ?

The Elisp interpreter is probably tightly intertwined with the Emacs 
editor code. Disentangling them so you can compile Elisp to machine code 
would be a big project, and probably not worth it.

Byte-code interpretation is common in many modern languages, it makes it 
easier for the languages to be portable. Python and PHP both compile to 
byte code rather than machine code. It's generally fast enough for 
interactive applications. For computationally-intensive applications, 
there are often libraries that can be linked in, like Numpy for Python.

-- 
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Barry Margolin]

In article <86tvlvmxtz.fsf@zoho.com>, Emanuel Berg <moasen@zoho.com> 
wrote:

> If we are talking the insanely long functions
> of Gnus, if the funcall overhead is the reason
> for that, I don't see why modularization plus
> inlining couldn't be one puzzle piece of
> the remedy.

If a function is "insanely long" then inlining will have negligible 
effect. Funcall overhead is only relevant if the function is really 
short, so it spends nearly as much time calling the function as doing 
the actual work of the function, AND you call the function frequently 
enough that this overhead adds up to something significant.

-- 
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Emanuel Berg]

Barry Margolin wrote:

>> If we are talking the insanely long
>> functions of Gnus, if the funcall overhead
>> is the reason for that, I don't see why
>> modularization plus inlining couldn't be one
>> puzzle piece of the remedy.
>
> If a function is "insanely long" then
> inlining will have negligible effect.
> Funcall overhead is only relevant if the
> function is really short, so it spends nearly
> as much time calling the function as doing
> the actual work of the function, AND you call
> the function frequently enough that this
> overhead adds up to something significant.

The insanely long functions would be shorter if
they, instead of doing one million things,
called other functions to help them with that.
This isn't done because it isn't fast enough.
But if inline works as you describe it, some of
that code could in fact be moved out of the
insanely long function, while still not slowing
it down with funcall overhead.


-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Óscar Fuentes]

Barry Margolin <barmar@alum.mit.edu> writes:

> If a function is "insanely long" then inlining will have negligible 
> effect. Funcall overhead is only relevant if the function is really 
> short, so it spends nearly as much time calling the function as doing 
> the actual work of the function, AND you call the function frequently 
> enough that this overhead adds up to something significant.

Inlining is not only about funcall overhead, although most compilers use
code length as the unique heuristic for deciding when to inline a
function.

Re: Optimising Elisp code [again]

[Óscar Fuentes]

Barry Margolin <barmar@alum.mit.edu> writes:

> The Elisp interpreter is probably tightly intertwined with the Emacs 
> editor code. Disentangling them so you can compile Elisp to machine code 
> would be a big project, and probably not worth it.

http://git.savannah.gnu.org/cgit/emacs.git/log/?h=feature/libjit

https://github.com/tromey/el-compilador

Re: Optimising Elisp code [again]

[Garreau, Alexandre]

On 2018-10-10 at 12:18, Barry Margolin wrote:
> In article <86ftxfm98q.fsf@zoho.com>, Emanuel Berg <moasen@zoho.com> 
> wrote:
>> I mean, compile the Lisp that isn't user
>> defined and unlikely to change - compile for
>> real, not byte-compile for
>> sped-up interpretation.
>> 
>> I.e.: fun-and-play Elisp on the top
>>       compiled Elisp in the middle
>>       C downmost
>> 
>> ?
>
> The Elisp interpreter is probably tightly intertwined with the Emacs 
> editor code. Disentangling them so you can compile Elisp to machine code 
> would be a big project, and probably not worth it.

Why would that need distangling?  Couldn’t editor code be compiled as
well?  I understand things need to stay dynamic and redefinable, but a
large portion would remain unchanged anyway for most of time and the
things that would most likely be changed would be less stable and
less low-level things that wouldn’t break other stuff wouldn’t it?

Re: Optimising Elisp code [again]

[Emanuel Berg]

Óscar Fuentes wrote:

>> The Elisp interpreter is probably tightly
>> intertwined with the Emacs editor code.
>> Disentangling them so you can compile Elisp
>> to machine code would be a big project, and
>> probably not worth it.
>
> http://git.savannah.gnu.org/cgit/emacs.git/log/?h=feature/libjit
>
> https://github.com/tromey/el-compilador

Wonderful! But how does one integrate the
compiled code with the rest of Emacs?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Barry Margolin]

In article <86woqprcdb.fsf@zoho.com>, Emanuel Berg <moasen@zoho.com> 
wrote:

> Barry Margolin wrote:
> 
> >> If we are talking the insanely long
> >> functions of Gnus, if the funcall overhead
> >> is the reason for that, I don't see why
> >> modularization plus inlining couldn't be one
> >> puzzle piece of the remedy.
> >
> > If a function is "insanely long" then
> > inlining will have negligible effect.
> > Funcall overhead is only relevant if the
> > function is really short, so it spends nearly
> > as much time calling the function as doing
> > the actual work of the function, AND you call
> > the function frequently enough that this
> > overhead adds up to something significant.
> 
> The insanely long functions would be shorter if
> they, instead of doing one million things,
> called other functions to help them with that.
> This isn't done because it isn't fast enough.
> But if inline works as you describe it, some of
> that code could in fact be moved out of the
> insanely long function, while still not slowing
> it down with funcall overhead.

Long functions are often long because of complex logic, which can be 
hard to extract into separate functions.

Also, we tend to extract code into separate functions only when there's 
some logic to it -- it has to have a meaningful identity outside the 
function where it exists. If a function can only be used in one place, 
and refers to lots of the variables in that caller, then it's often hard 
to justify splitting it out -- it actually makes things more confusing 
because the modularity doesn't make sense.

I doubt that anyone makes decision about whether to split large 
functions into separate functions based on the overhead of function 
calling.

-- 
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Emanuel Berg]

Barry Margolin wrote:

> Long functions are often long because of
> complex logic, which can be hard to extract
> into separate functions.
>
> Also, we tend to extract code into separate
> functions only when there's some logic to it
> -- it has to have a meaningful identity
> outside the function where it exists.
> If a function can only be used in one place,
> and refers to lots of the variables in that
> caller, then it's often hard to justify
> splitting it out -- it actually makes things
> more confusing because the modularity doesn't
> make sense.

Complexity isn't reduced by having tons of
stuff in a single function that goes on
forever- on the contrary, this makes it more
difficult to read, understand, and maintain.

But of course the modularity should make sense!

> I doubt that anyone makes decision about
> whether to split large functions into
> separate functions based on the overhead of
> function calling.

This is what the Gnus people told me when
I suggested not writing functions that are
longer and deeper than a novel by Dostoevsky -
they said, ~"Gnus is slow, Elisp is slow,
making neat modules of those defuns will make
it even slower."

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Optimising Elisp code [again]

[Robert Thorpe]

Emanuel Berg <moasen@zoho.com> writes:

> Óscar Fuentes wrote:
>
>>> The Elisp interpreter is probably tightly
>>> intertwined with the Emacs editor code.
>>> Disentangling them so you can compile Elisp
>>> to machine code would be a big project, and
>>> probably not worth it.
>>
>> http://git.savannah.gnu.org/cgit/emacs.git/log/?h=feature/libjit
>>
>> https://github.com/tromey/el-compilador
>
> Wonderful! But how does one integrate the
> compiled code with the rest of Emacs?

Firstly, I think it's worth clarifying how things work now.  Emacs has
two interpreters.  There's the interactive interpreter, that's what's run
when you type elisp straight into Emacs.  It is used to interpret files
ending in ".el".  Then there's the bytecode interpreter.  To use that
you have to compile a program to bytecode first.  It's the bytecode
interpreter that deals with ".elc" files.  The bytecode interpreter is
much faster than the interactive interpreter.  (I expect you know all
this Emanuel, I'm just writing it here for others who are reading who
might not).

The libjit branch works by compiling bytecode into native code.  As I
understand it, this is done at runtime, so the user doesn't have to do
anything.  I think there was another project that did something similar
at compile time.  Compiling bytecode sounds a bit strange, but it makes
sense.  In many cases what the JIT does is emit the same code that
is inside the bytecode interpreter.  In a very simplified sense, it
works like this....  So, the bytecode interpreter is a big switch/case
statement.  Somewhere there is instruction FOO.  When the JIT compiler
sees FOO it inserts into it's output buffer the code that was in the
bytecoding interpreter to deal with that opcode.  There are some
optimizations beyond that.

I think it's a good idea and I hope it will be merged into mainline
Emacs one day.

BR,
Robert Thorpe

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Barry Margolin]

In article <8636tcl05a.fsf@zoho.com>, Emanuel Berg <moasen@zoho.com> 
wrote:

> Barry Margolin wrote:
> 
> > I doubt that anyone makes decision about
> > whether to split large functions into
> > separate functions based on the overhead of
> > function calling.
> 
> This is what the Gnus people told me when
> I suggested not writing functions that are
> longer and deeper than a novel by Dostoevsky -
> they said, ~"Gnus is slow, Elisp is slow,
> making neat modules of those defuns will make
> it even slower."

Sounds like a bullshit rationalization. The real reason is probably that 
refactoring is a PITA.

-- 
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Eric Abrahamsen]

Barry Margolin <barmar@alum.mit.edu> writes:

> In article <8636tcl05a.fsf@zoho.com>, Emanuel Berg <moasen@zoho.com> 
> wrote:
>
>> Barry Margolin wrote:
>> 
>> > I doubt that anyone makes decision about
>> > whether to split large functions into
>> > separate functions based on the overhead of
>> > function calling.
>> 
>> This is what the Gnus people told me when
>> I suggested not writing functions that are
>> longer and deeper than a novel by Dostoevsky -
>> they said, ~"Gnus is slow, Elisp is slow,
>> making neat modules of those defuns will make
>> it even slower."
>
> Sounds like a bullshit rationalization. The real reason is probably that 
> refactoring is a PITA.

As someone who's currently trying to refactor Gnus (replacing obarray
hashtables with actual hashtables), I can confirm that it is indeed a
PITA.

Re: Optimising Elisp code [again]

[Emanuel Berg]

Robert Thorpe wrote:

> The libjit branch works by compiling bytecode
> into native code. As I understand it, this is
> done at runtime, so the user doesn't have to
> do anything. I think there was another
> project that did something similar at compile
> time. Compiling bytecode sounds a bit
> strange, but it makes sense. In many cases
> what the JIT does is emit the same code that
> is inside the bytecode interpreter. In a very
> simplified sense, it works like this.... So,
> the bytecode interpreter is a big switch/case
> statement. Somewhere there is instruction
> FOO. When the JIT compiler sees FOO it
> inserts into it's output buffer the code that
> was in the bytecoding interpreter to deal
> with that opcode. There are some
> optimizations beyond that.

So it is a JIT native compiler of bytecode.

> I think it's a good idea and I hope it will
> be merged into mainline Emacs one day.

Thanks for a great post and also cred to the
people who have been, are, and/or in the future
will work on the libjit project!

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Emanuel Berg]

Barry Margolin wrote:

> Sounds like a bullshit rationalization.
> The real reason is probably that refactoring
> is a PITA.

If so, why did they write the code that way to
begin with? Was everything even slower then or
did they simply not give it much thought?
Refactoring may be difficult but writing
modular code to begin with is obviously much
easier than the opposite as small modules can
be developed and tested individually so
debugging and everything else is much easier.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1260 bytes --]

On Sat, Oct 13, 2018 at 10:51:56PM +0200, Emanuel Berg wrote:
> Barry Margolin wrote:
> 
> > Sounds like a bullshit rationalization.
> > The real reason is probably that refactoring
> > is a PITA.
> 
> If so, why did they write the code that way to
> begin with? [...]

Code "grows", one single (minimum-effort) step by step, following
a path of minimum resistance over a time-changing fractal landscape.

As one colleague of me used to put it (and he's one of the most
gifted programmers I know personally), code grows all by itself,
and the programmer's job is to prune it and to keep the pruning
rate high enough so to not be drowned by the jungle.

Put in more sober terms, usually you start with one idea and a
rough sketch. Most of the time, this sketch is wrong (because
you aren't God), but when it's right enough you start getting
users, so you keep going, and fixing up things as they come.

From time to time the tension between your original vision and
the one you have now (because you're older, or because the folks
pushing the project have changed, or because reality, or your
users, or the weather evolved) is so big that you take the hurdle
of a Big Refactor. Then your users will hate you :-)

Cheers
-- tomás

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Emanuel Berg]

tomas wrote:

> Code "grows", one single (minimum-effort)
> step by step, following a path of minimum
> resistance over a time-changing
> fractal landscape.
>
> As one colleague of me used to put it (and
> he's one of the most gifted programmers
> I know personally), code grows all by itself,
> and the programmer's job is to prune it and
> to keep the pruning rate high enough so to
> not be drowned by the jungle.

Code grows, but not necessarily by enlonging
the defuns that are already there. Instead,
when some new situation arrives, write yet
another defun to handle it. Then call it from
the original function. That way the original
function grows, but only by a single line.

To do this is much, much simpler than to write
an insanely long defun, and when insanity
becomes a problem, try to "refactor" it!

[This is a general remark; I never did any work
on Gnus so I don't know what explanation holds
the most truth.]

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1470 bytes --]

On Sun, Oct 14, 2018 at 09:10:56PM +0200, Emanuel Berg wrote:
> tomas wrote:
> 
> > Code "grows", one single (minimum-effort)
> > step by step [...]

> Code grows, but not necessarily by enlonging
> the defuns that are already there [...]

Most of the time it does, though: *this* is the
path of least resistance, for small steps.

> when some new situation arrives, write yet
> another defun to handle it. Then call it from
> the original function. That way the original
> function grows, but only by a single line.

On the long term this tends lead to horrible code,
where the defuns reflect all the history of the
situations, as they arrived, not the problem's
structure as it is *now*. But yes, you see something
of that in actual code -- real life is always a
compromise.

> To do this is much, much simpler than to write
> an insanely long defun, and when insanity
> becomes a problem, try to "refactor" it!

I've seen lots of code evolve. I'm an old guy. My
experience seems to contradict your assessment above.

Keeping code clean, always finding the right abstractions
and enforcing them throughout takes a non-trivial amount
of effort. I tend to think that "it's worth it", at least
in terms of self-esteem and developer satisfaction, and
I like to think that, in the long term, it amortises itself
in terms of effort, but for the last one I can't say whether
(or when) it is wishful thinking or not.

Cheers
-- tomás

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Emanuel Berg]

tomas wrote:

> On the long term this tends lead to horrible
> code, where the defuns reflect all the
> history of the situations, as they arrived,
> not the problem's structure as it is *now*.

Why is this horrible code? Isn't it just
natural that the code reflects how problems
have been solved over time, with changes in
technology, etc.? Isn't it much better to have
this "reflection" in neat modules than in
single functions spanning 3 A4 each?

Besides, if some of these defuns are really
part of history with no relevance for the
present situation, it is much easier to delete
them [1] than to mess with the
original huge functions!

[1] Better yet, delete *the call* to them and
    to move the modules to an out-of-action
    source file, as history might
    repeat itself.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Van L]

> [1] Better yet, delete *the call* to them and
>    to move the modules to an out-of-action
>    source file, as history might
>    repeat itself.

Surely, this is an easy problem for Deep Learning on call-traces to prune.

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Garreau, Alexandre]

On 2018-10-15 at 21:27, Emanuel Berg wrote:
> tomas wrote:
>
>> On the long term this tends lead to horrible
>> code, where the defuns reflect all the
>> history of the situations, as they arrived,
>> not the problem's structure as it is *now*.
>
> Why is this horrible code? Isn't it just
> natural that the code reflects how problems
> have been solved over time, with changes in
> technology, etc.?

No, VCSes a here for that.

> Isn't it much better to have this "reflection" in neat modules than in
> single functions spanning 3 A4 each?

If these modules have no meaning, no, because it will harden the work of
understanding, thus, for instance, of debugging, the function.  While
the same functions may have piece that will cancel or make useless
pieces of other functions, if you just “add up historically”, you may,
if you want to solve the problem as it is *now*, simplify the whole
thing.  And it won’t simply be refactoring by adding functions or
macros, it will be removing stuff.

> Besides, if some of these defuns are really
> part of history with no relevance for the
> present situation, it is much easier to delete
> them [1] than to mess with the
> original huge functions!

> [1] Better yet, delete *the call* to them and
>     to move the modules to an out-of-action
>     source file, as history might
>     repeat itself.

VCS, search tools, or otherwise devs memories, are here for that.

And even if it stayed in the source code anyway people are interested in
functions that are documented and that they use, directly or indirectly.
They might as well not see it if it’s not used.

Re: Knowing where a function has been used (bis) [Was: Re: Optimising Elisp code]

[Emanuel Berg]

Garreau, Alexandre wrote:

>>> On the long term this tends lead to
>>> horrible code, where the defuns reflect all
>>> the history of the situations, as they
>>> arrived, not the problem's structure as it
>>> is *now*.
>>
>> Why is this horrible code? Isn't it just
>> natural that the code reflects how problems
>> have been solved over time, with changes in
>> technology, etc.?
>
> No, VCSes a here for that.

It is a fact of life that code develops and
grows in different directions. Different parts
of the code reflect more so the original idea,
other parts of the code refect more so
situations that arrived without anyone thinking
of them day one. This is completely natural and
not something so you can use the code as
a history book, if anyone now got that idea.

>> Isn't it much better to have this
>> "reflection" in neat modules than in single
>> functions spanning 3 A4 each?
>
> If these modules have no meaning, no, because
> it will harden the work of understanding,
> thus, for instance, of debugging, the
> function. While the same functions may have
> piece that will cancel or make useless pieces
> of other functions, if you just “add up
> historically”, you may, if you want to solve
> the problem as it is *now*, simplify the
> whole thing. And it won’t simply be
> refactoring by adding functions or macros, it
> will be removing stuff.

The modules should have have meaning and
reflect how the problem is solved now. If they,
at some point, become totally obsolete, they
are much easier to drop as modules than to try
to extract that exact portion of the original,
3 A4 long defun, without creating bugs while
doing that.

>> [1] Better yet, delete *the call* to them
>> and to move the modules to an out-of-action
>> source file, as history might repeat itself.
>
> VCS, search tools, or otherwise devs
> memories, are here for that. And even if it
> stayed in the source code anyway people are
> interested in functions that are documented
> and that they use, directly or indirectly.
> They might as well not see it if it’s
> not used.

OK, if you have found a spot where they are the
most visible, go ahead and use it. To me, an
"obsolete-source" directory is fine.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Why is Elisp slow?

[Van L]

Emanuel Berg <moasen@zoho.com> writes:

> Why can't Elisp be compiled as well?

I came across this tip of an iceberg in the commentary.

#+name: xxx-byte-opt-el
#+begin_src emacs-lisp
  ;; "No matter how hard you try, you can't make a racehorse out of a pig.
  ;; You can, however, make a faster pig."
  ;;
  ;; Or, to put it another way, the Emacs byte compiler is a VW Bug.  This code
  ;; makes it be a VW Bug with fuel injection and a turbocharger...  You're
  ;; still not going to make it go faster than 70 mph, but it might be easier
  ;; to get it there.
#+end_src

If a big advance from an Industry player, such as FB, were to release a
JIT, that the GNU Emacs legal dept. does not object to, does that help?

-- 
© 2019 Van L
gpg using EEF2 37E9 3840 0D5D 9183  251E 9830 384E 9683 B835
                             "How did we get to this point?" - William P. Barr

Re: Why is Elisp slow?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1778 bytes --]

On Thu, May 02, 2019 at 02:49:22PM +1000, Van L wrote:

>  ;; "No matter how hard you try, you can't make a racehorse out of a pig.
>  ;; You can, however, make a faster pig."

[...]

> If a big advance from an Industry player, such as FB, were to release a
> JIT, that the GNU Emacs legal dept. does not object to, does that help?

I, for one, appreciate Emacs much more for other characteristics than
its speed.

Its extremely judicious developer community, which manages advances
at a steady pace (in a 30 year old project) while generally avoiding
disrupting surprises for its regular users is... amazing.

I've yet to run into a serious limitation posed by its "slow" elisp
implementation.

It's the same outside Emacs. For most of my day-to-day problems, I
use known-slow language implementations. For those rare exceptions,
there are other tools.

Of course Common Lispers (and to some extent Schemers, and other
functional programmers [1]) are spoilt by extremely clever implementations,
and this is what the comment's author was hinting at -- but it takes
a wizard to master those tools. It takes a grand wizard to hack them.

The real feat here is that Emacs offers an environment which is still
amenable to non-wizards. In practice, I think, the biggest optimization
potential still exists in making contributors better than in making
the core implementation faster. In that metric, I'd say, Emacs succeeds
very well [2].

Cheers

[1] This is an example of what I'm talking about:
    https://dl.acm.org/citation.cfm?id=3276489

[2] Here I can just give that reference back to you
    "Evolution of Emacs Lisp" by Stefan Monnier & Michael Sperber
    https://www.iro.umontreal.ca/~monnier/hopl-4-emacs-lisp.pdf

-- tomás

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Why is Elisp slow?

[Marcin Borkowski]

On 2019-05-02, at 09:56, tomas@tuxteam.de wrote:

>> If a big advance from an Industry player, such as FB, were to release a
>> JIT, that the GNU Emacs legal dept. does not object to, does that help?
>
> I, for one, appreciate Emacs much more for other characteristics than
> its speed.

Agreed, but...

> Its extremely judicious developer community, which manages advances
> at a steady pace (in a 30 year old project) while generally avoiding
> disrupting surprises for its regular users is... amazing.
>
> I've yet to run into a serious limitation posed by its "slow" elisp
> implementation.

...well, Org-mode agenda takes almost 30 seconds to generate for me...

Best,

-- 
Marcin Borkowski
http://mbork.pl

Re: Why is Elisp slow?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 534 bytes --]

On Thu, May 02, 2019 at 01:06:03PM +0200, Marcin Borkowski wrote:
> 
> On 2019-05-02, at 09:56, tomas@tuxteam.de wrote:

[on elisp speed]

> > I've yet to run into a serious limitation posed by its "slow" elisp
> > implementation.
> 
> ...well, Org-mode agenda takes almost 30 seconds to generate for me...

Perhaps this is more a case for fine tuning Org mode's algorithms &
data structures? (I don't use agenda, so I don't really know what
I'm talking about. Perhaps all of this is already done?)

Cheers
-- tomás

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Why is Elisp slow?

[Eli Zaretskii]

> Date: Thu, 2 May 2019 15:18:27 +0200
> From: tomas@tuxteam.de
> Cc: help-gnu-emacs@gnu.org
> 
> > ...well, Org-mode agenda takes almost 30 seconds to generate for me...
> 
> Perhaps this is more a case for fine tuning Org mode's algorithms &
> data structures? (I don't use agenda, so I don't really know what
> I'm talking about. Perhaps all of this is already done?)

It's certainly a reason to report this to the Org developers, and ask
them to speed this up.

(My personal rant to package authors is that they are way too eager to
implement stuff in Lisp which cannot possibly be fast enough or
scalable enough, instead of urging Emacs to provide new C primitives
and core features, or, better, submitting patches to implement such
features in C.  Two cases in point are linum-mode and fci-mode.  End
of rant.)

Re: Why is Elisp slow?

[Marcin Borkowski]

On 2019-05-02, at 15:18, tomas@tuxteam.de wrote:

> On Thu, May 02, 2019 at 01:06:03PM +0200, Marcin Borkowski wrote:
>> 
>> On 2019-05-02, at 09:56, tomas@tuxteam.de wrote:
>
> [on elisp speed]
>
>> > I've yet to run into a serious limitation posed by its "slow" elisp
>> > implementation.
>> 
>> ...well, Org-mode agenda takes almost 30 seconds to generate for me...
>
> Perhaps this is more a case for fine tuning Org mode's algorithms &
> data structures? (I don't use agenda, so I don't really know what
> I'm talking about. Perhaps all of this is already done?)

I don't think so.  I just happen to have several thousand headlines in
my agenda files.  And AFAIK, agenda generation is already pretty
optimized.  So, this might be one (rare, admittedly) example of such
a limitation.

Best,

-- 
Marcin Borkowski
http://mbork.pl

Re: Why is Elisp slow?

[Marcin Borkowski]

On 2019-05-02, at 17:34, Eli Zaretskii <eliz@gnu.org> wrote:

>> Date: Thu, 2 May 2019 15:18:27 +0200
>> From: tomas@tuxteam.de
>> Cc: help-gnu-emacs@gnu.org
>> 
>> > ...well, Org-mode agenda takes almost 30 seconds to generate for me...
>> 
>> Perhaps this is more a case for fine tuning Org mode's algorithms &
>> data structures? (I don't use agenda, so I don't really know what
>> I'm talking about. Perhaps all of this is already done?)
>
> It's certainly a reason to report this to the Org developers, and ask
> them to speed this up.

Well, I don't consider it a bug in Orgmode - rather a bug in my workflow
(and I more or less know how to fix that, only it will take me some
time).

> (My personal rant to package authors is that they are way too eager to
> implement stuff in Lisp which cannot possibly be fast enough or
> scalable enough, instead of urging Emacs to provide new C primitives
> and core features, or, better, submitting patches to implement such
> features in C.  Two cases in point are linum-mode and fci-mode.  End
> of rant.)

That is an interesting POV, although I'm not sure if I agree.  IOW,
I fully understand why people might do so.  I, for one, consider myself
pretty fluent in Elisp and hardly literate in C.

Best,

-- 
Marcin Borkowski
http://mbork.pl

Re: Why is Elisp slow?

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

> (My personal rant to package authors is that they are way too eager to
> implement stuff in Lisp which cannot possibly be fast enough or
> scalable enough, instead of urging Emacs to provide new C primitives
> and core features, or, better, submitting patches to implement such
> features in C.  Two cases in point are linum-mode and fci-mode.  End
> of rant.)

On addition to what Marcin Borkowski says, it is a great convenience to
implement something yourself that you can deliver on your own schedule
instead of depending on something that can take years to be available to
your user base. Apart from the extra work of explaining and justifying
what you need and convincing someone to implement it.

(And then, what is "fast enough for me" can be unbearably slow for
others, etc)

Re: Why is Elisp slow?

[Eli Zaretskii]

> From: Marcin Borkowski <mbork@mbork.pl>
> Cc: help-gnu-emacs@gnu.org
> Date: Thu, 02 May 2019 21:13:28 +0200
> 
> > It's certainly a reason to report this to the Org developers, and ask
> > them to speed this up.
> 
> Well, I don't consider it a bug in Orgmode - rather a bug in my workflow

Maybe improvements in Org could make your bug less prominent...

Then again, it could be that there's some trick up Org developers'
sleeve that they could teach you.

In general, anything surprising/inconvenient/buggy should be reported,
you can never know what will come out of that.

> > (My personal rant to package authors is that they are way too eager to
> > implement stuff in Lisp which cannot possibly be fast enough or
> > scalable enough, instead of urging Emacs to provide new C primitives
> > and core features, or, better, submitting patches to implement such
> > features in C.  Two cases in point are linum-mode and fci-mode.  End
> > of rant.)
> 
> That is an interesting POV, although I'm not sure if I agree.  IOW,
> I fully understand why people might do so.  I, for one, consider myself
> pretty fluent in Elisp and hardly literate in C.

Submitting changes in C is a bonus, it's good enough to describe the
use case and ask for primitives to support it.  Someone else might get
motivated.  It could even be very easy, like with fci-mode.

I'm quite sure that some of the bad reputation of Emacs is due to
incredible things people do in Lisp, which then make Emacs look
sluggish and bloated.

Re: Why is Elisp slow?

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Thu, 02 May 2019 21:33:55 +0200
> 
> On addition to what Marcin Borkowski says, it is a great convenience to
> implement something yourself that you can deliver on your own schedule
> instead of depending on something that can take years to be available to
> your user base. Apart from the extra work of explaining and justifying
> what you need and convincing someone to implement it.

I have no problem with people who code something for their personal
use.  That's none of my concern.

As for convincing: that is not required.  Just put the request in the
open, and be done.

Re: Why is Elisp slow?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1163 bytes --]

On Thu, May 02, 2019 at 09:13:28PM +0200, Marcin Borkowski wrote:

[...]

> > (My personal rant to package authors is that they are way too eager to
> > implement stuff in Lisp which cannot possibly be fast enough or
> > scalable enough, instead of urging Emacs to provide new C primitives
> > and core features, or, better, submitting patches to implement such
> > features in C.  Two cases in point are linum-mode and fci-mode.  End
> > of rant.)
> 
> That is an interesting POV, although I'm not sure if I agree.  IOW,
> I fully understand why people might do so.  I, for one, consider myself
> pretty fluent in Elisp and hardly literate in C.

This approach is known as "Ousterhout's dichotomy" [1], after John Ousterhout,
author of Tcl (which was conceived as a very simple scripting language
meant to extend existing applications). Tcl grew later a bytecode
interpreter (as did elisp) -- making the C interface significantly
more complex (but the interpreter significantly faster).

Walking the trade-off between simple and efficient is interesting.

Cheers

[1] https://en.wikipedia.org/wiki/Ousterhout%27s_dichotomy

-- tomás

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Why is Elisp slow?

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Óscar Fuentes <ofv@wanadoo.es>
>> Date: Thu, 02 May 2019 21:33:55 +0200
>> 
>> On addition to what Marcin Borkowski says, it is a great convenience to
>> implement something yourself that you can deliver on your own schedule
>> instead of depending on something that can take years to be available to
>> your user base. Apart from the extra work of explaining and justifying
>> what you need and convincing someone to implement it.
>
> I have no problem with people who code something for their personal
> use.  That's none of my concern.

The sceneario I described was about a package maintainer, such as Org.

> As for convincing: that is not required.  Just put the request in the
> open, and be done.

That's not a solution if you are package maintainer trying to solve a
performance problem within a given time frame.

Re: Why is Elisp slow?

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Thu, 02 May 2019 22:12:50 +0200
> 
> > I have no problem with people who code something for their personal
> > use.  That's none of my concern.
> 
> The sceneario I described was about a package maintainer, such as Org.

That was exactly my gripe.  I'm saying that implementing something for
others should be considered more seriously than just "let's code it
because we can".

> > As for convincing: that is not required.  Just put the request in the
> > open, and be done.
> 
> That's not a solution if you are package maintainer trying to solve a
> performance problem within a given time frame.

Of course it is: with the Lisp implementation already in place, you
don't need to wait for anyone or anything.  But requesting a feature
in addition to that does two things: (a) it alerts others, including
Emacs developers, to the need; and (b) it announces load and clear
that the package maintainer is not really happy with the current
solution.  Without such a request, no one will even know that there's
a problem here waiting for a volunteer.

Re: Why is Elisp slow?

[Ergus]

One reason why people implemented everything in elisp instead of
contributing in C (or sending patches) is:

1) in many documents around it says that it is the recommended way to
extend emacs so new users (like me) think that everything is possible
that way.

2) Because emacs didn't have a C api for modules until very recently and
even with that all the infrastructure is Lisp based. (and it still needs
a LOT of documentation) 

3) It was the only available API with documentation and that didn't
require recompile all emacs and they could redistribute, share and
maintain independently without doing all the complex process to
contribute.

Probably (I have a dream) if someone decides to develop a new emacs in
2019 most of the functions and API will be made in pure and clear C (or
any other compiled language), with a full C api that gives the same
access level than what elisp gives in Emacs today (with C lists, arrays
and structs), so it will be not only faster but also simpler to extend
it with other languages like Scheme, Python, Lua, C++, rust and so
on. (There are modules projects going in that direction) And the editor
don't even need to provide a compiler or interpreter for them. (there
will be Guile/gcc/python and so on for that)

But the emacs modules need a lot of time before they can provide the
same power and access than Elisp gives these days, but most of the
infrastructure actually calls indirectly Elisp functions and create Lisp
objects, so there will be not any performance advantage in that part of
the code until it goes back to C or the Lisp->C or the JIT compilers
will be finished.

Vim, for example, added a C api for that reason, but we are talking
about a very different beast here. They don't have their own interpreter
to maintain, no duality with GUI and TUI to maintain, no Lisp objects,
bytecode, debugger... so a very different beast.  



On Thu, May 02, 2019 at 11:20:54PM +0300, Eli Zaretskii wrote:
>> From: ?scar Fuentes <ofv@wanadoo.es>
>> Date: Thu, 02 May 2019 22:12:50 +0200
>>
>> > I have no problem with people who code something for their personal
>> > use.  That's none of my concern.
>>
>> The sceneario I described was about a package maintainer, such as Org.
>
>That was exactly my gripe.  I'm saying that implementing something for
>others should be considered more seriously than just "let's code it
>because we can".
>
>> > As for convincing: that is not required.  Just put the request in the
>> > open, and be done.
>>
>> That's not a solution if you are package maintainer trying to solve a
>> performance problem within a given time frame.
>
>Of course it is: with the Lisp implementation already in place, you
>don't need to wait for anyone or anything.  But requesting a feature
>in addition to that does two things: (a) it alerts others, including
>Emacs developers, to the need; and (b) it announces load and clear
>that the package maintainer is not really happy with the current
>solution.  Without such a request, no one will even know that there's
>a problem here waiting for a volunteer.
>

Re: Why is Elisp slow?

[조성빈]

> Probably (I have a dream) if someone decides to develop a new emacs in
> 2019 most of the functions and API will be made in pure and clear C (or
> any other compiled language), with a full C api that gives the same
> access level than what elisp gives in Emacs today (with C lists, arrays
> and structs), so it will be not only faster but also simpler to extend
> it with other languages like Scheme, Python, Lua, C++, rust and so
> on. (There are modules projects going in that direction) And the editor
> don't even need to provide a compiler or interpreter for them. (there
> will be Guile/gcc/python and so on for that)

This, very much sounds like Guile Emacs. Anyone knows how Guile Emacs is doing?̊̈ It’s very much looks like vaporware these days :-(
Is upstream considering Guile Emacs as a valid solution? 
Is there any development ongoing? (Official or not?)

Re: Why is Elisp slow?

[Ergus]

WARNING: This email has plenty of personal opinions.

On Fri, May 03, 2019 at 08:39:43AM +0900, ????????? wrote:
>
>> Probably (I have a dream) if someone decides to develop a new emacs in
>> 2019 most of the functions and API will be made in pure and clear C (or
>> any other compiled language), with a full C api that gives the same
>> access level than what elisp gives in Emacs today (with C lists, arrays
>> and structs), so it will be not only faster but also simpler to extend
>> it with other languages like Scheme, Python, Lua, C++, rust and so
>> on. (There are modules projects going in that direction) And the editor
>> don't even need to provide a compiler or interpreter for them. (there
>> will be Guile/gcc/python and so on for that)
>
>This, very much sounds like Guile Emacs. Anyone knows how Guile Emacs
>is doing????? It???s very much looks like vaporware these days :-(

Yes, that is the dream (at least the closest we have ever been)

>Is upstream considering Guile Emacs as a valid solution?

I made a similar question some time ago and the answer is that there is
not action. :( In fact there is not too much action in
Guile's development .

Not many projects feel interested in Guile because the weak support it
has so there are few users and few developers (But also because
Lisp-like family languages (common Lisp, Scheme and the others) are a
museum piece for young generations that are more used to Python, Ruby,
javascript, and most of OOP (also because there are more jobs with
that))

Actually I think that these days will be easier to find new C/C++
developers for emacs than Lisp developers. Lisp and Scheme are
beautiful, but they require a different way of thinking and a lot of
time (own experience, I am just starting with it.) One reason why I
can't convince my friends to use Emacs is actually how Lisp scares them
((())()'()).

>Is there any development ongoing? (Official or not?)
>
>
I don't think so :( :( :(. It will require that some core emacs
developers feel more interested in implementing this, because it is a
lot of work... (specially to do it right and keeping the so sacred
backward compatibility) These days only few people know the core parts
of emacs for such a task.

Having Guile as a dependency will grow emacs size significantly
and using it as an external dependency (not usually the emacs way...)
will require to port Guile to more systems (AFAIK). BUt such desition
could benefit very much both projects.

There are even some emacs ports to rust (remacs) but they still have the
elisp as the core languaje. Because most of the code and functionalities
are already in elisp so changing that will be like reimplementing all
emacs from scratch.

(Some time ago there was a lot of effort and time invested in porting C
functions to Elisp in emacs)

So with our actual emacs maybe the JIT or the Lisp->C source to source
is actually the more reliable option in a possible (realistic) future.

Personally I feel the Lisp->C compiler feels like the faster and more
robust solution for me, but it will create a dependency with
binutils... which is for multiple reasons undesired.

The alternative JIT may be based in libJIT which is not very active
either... and has serious issues/limitations that has not been solved in
years.

[1] https://tromey.com/blog/?p=982
[2] https://lists.gnu.org/archive/html/emacs-devel/2018-08/msg00393.html

Re: Why is Elisp slow?

[조성빈]

I became curious after reading your email :-)

Since implementing a new language/byte interpreter (Guile, for example) is hard, and making it fast is harder, why doesn't Emacs leverage the Common Lisp environment?̊̈

For example, (since CL and elisp’s syntax is mostly similar) can’t we implement Emacs based on CL (looks like there were ongoing efforts before like Climacs, Hemlock, etc…) and expose elisp functions in a special package named ‘elisp’?̊̈ It would allow elisp compatibility (by running all elisp code in the special elisp package) while giving package writers access to CL libraries. Since legacy cruft can go inside the package ‘elisp’, CL emacs APIs can remove/redesign some old APIs and remove technical debt. This would also greatly accerlate Emacs speed as CL is super fast especially when using with SBCL.

Are there any reasons (that I do not know) why CL was not considered or was considered but rejected as an elisp alternative? 
Are there license problems to build on previous attempts?̊̈ 

IMHO building emacs on CL would be a wonderful idea... and would fix many problems that current Emacs have.

> 2019. 5. 3. 오전 9:44, Ergus <spacibba@aol.com> 작성:
> 
> WARNING: This email has plenty of personal opinions.
> 
>> On Fri, May 03, 2019 at 08:39:43AM +0900, ????????? wrote:
>> 
>>> Probably (I have a dream) if someone decides to develop a new emacs in
>>> 2019 most of the functions and API will be made in pure and clear C (or
>>> any other compiled language), with a full C api that gives the same
>>> access level than what elisp gives in Emacs today (with C lists, arrays
>>> and structs), so it will be not only faster but also simpler to extend
>>> it with other languages like Scheme, Python, Lua, C++, rust and so
>>> on. (There are modules projects going in that direction) And the editor
>>> don't even need to provide a compiler or interpreter for them. (there
>>> will be Guile/gcc/python and so on for that)
>> 
>> This, very much sounds like Guile Emacs. Anyone knows how Guile Emacs
>> is doing????? It???s very much looks like vaporware these days :-(
> 
> Yes, that is the dream (at least the closest we have ever been)
> 
>> Is upstream considering Guile Emacs as a valid solution?
> 
> I made a similar question some time ago and the answer is that there is
> not action. :( In fact there is not too much action in
> Guile's development .
> 
> Not many projects feel interested in Guile because the weak support it
> has so there are few users and few developers (But also because
> Lisp-like family languages (common Lisp, Scheme and the others) are a
> museum piece for young generations that are more used to Python, Ruby,
> javascript, and most of OOP (also because there are more jobs with
> that))
> 
> Actually I think that these days will be easier to find new C/C++
> developers for emacs than Lisp developers. Lisp and Scheme are
> beautiful, but they require a different way of thinking and a lot of
> time (own experience, I am just starting with it.) One reason why I
> can't convince my friends to use Emacs is actually how Lisp scares them
> ((())()'()).
> 
>> Is there any development ongoing? (Official or not?)
> I don't think so :( :( :(. It will require that some core emacs
> developers feel more interested in implementing this, because it is a
> lot of work... (specially to do it right and keeping the so sacred
> backward compatibility) These days only few people know the core parts
> of emacs for such a task.
> 
> Having Guile as a dependency will grow emacs size significantly
> and using it as an external dependency (not usually the emacs way...)
> will require to port Guile to more systems (AFAIK). BUt such desition
> could benefit very much both projects.
> 
> There are even some emacs ports to rust (remacs) but they still have the
> elisp as the core languaje. Because most of the code and functionalities
> are already in elisp so changing that will be like reimplementing all
> emacs from scratch.
> 
> (Some time ago there was a lot of effort and time invested in porting C
> functions to Elisp in emacs)
> 
> So with our actual emacs maybe the JIT or the Lisp->C source to source
> is actually the more reliable option in a possible (realistic) future.
> 
> Personally I feel the Lisp->C compiler feels like the faster and more
> robust solution for me, but it will create a dependency with
> binutils... which is for multiple reasons undesired.
> 
> The alternative JIT may be based in libJIT which is not very active
> either... and has serious issues/limitations that has not been solved in
> years.
> 
> [1] https://tromey.com/blog/?p=982
> [2] https://lists.gnu.org/archive/html/emacs-devel/2018-08/msg00393.html

Re: Why is Elisp slow?

[Paul W. Rankin]

On Fri, May 03 2019, Ergus wrote:
> Actually I think that these days will be easier to find new C/C++
> developers for emacs than Lisp developers. Lisp and Scheme are
> beautiful, but they require a different way of thinking and a lot of
> time (own experience, I am just starting with it.) One reason why I
> can't convince my friends to use Emacs is actually how Lisp scares 
> them
> ((())()'()).

Emacs Lisp was the fist programming language I learnt (really as a means 
to an end to make a major mode I wanted). I found Elisp elegant and easy 
to understand; the parens make it clear where each expression begins and 
ends, functions are always the first element, and the documentation is 
incredibly accessible for a noob/hobbyist programmer. I find other 
languages difficult/inaccessible by comparison.

-- 
https://www.paulwrankin.com

Re: Why is Elisp slow?

[Stefan Monnier]

> One reason why people implemented everything in elisp instead of
> contributing in C (or sending patches) is:

Eli's point is quite different (and actually goes well beyond adding
new C primitives): there is a world of Elisp hackers developing great
packages but never engaging with Emacs development at all, in the sense
that they take Emacs as an immutable base whose limitations and bugs are
just obstacles around which to work.

I regularly bump into code "out there" with random hacks to work around
Emacs bugs I've never heard of.

All Eli is saying is: when the Emacs infrastructure isn't as good as
you'd like for your package, please report it as a bug (no need to do
anything more than that).  That doesn't mean we'll necessarily fix those
bugs (sometimes they're hard to fix, or simply nobody is interested in
fixing them), but it helps to know about them, and can guide
future redesigns.


        Stefan

Re: Why is Elisp slow?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Thu, 02 May 2019 22:39:46 -0400
> 
> > One reason why people implemented everything in elisp instead of
> > contributing in C (or sending patches) is:
> 
> Eli's point is quite different (and actually goes well beyond adding
> new C primitives): there is a world of Elisp hackers developing great
> packages but never engaging with Emacs development at all, in the sense
> that they take Emacs as an immutable base whose limitations and bugs are
> just obstacles around which to work.

Well said, thanks.

> I regularly bump into code "out there" with random hacks to work around
> Emacs bugs I've never heard of.

Not just bugs: also limitations and missing features.

> All Eli is saying is: when the Emacs infrastructure isn't as good as
> you'd like for your package, please report it as a bug (no need to do
> anything more than that).  That doesn't mean we'll necessarily fix those
> bugs (sometimes they're hard to fix, or simply nobody is interested in
> fixing them), but it helps to know about them, and can guide
> future redesigns.

Right.

Re: Why is Elisp slow?

[Eli Zaretskii]

> From: 조성빈 <pcr910303@icloud.com>
> Date: Fri, 3 May 2019 08:39:43 +0900
> Cc: Eli Zaretskii <eliz@gnu.org>, help-gnu-emacs@gnu.org
> 
> Is upstream considering Guile Emacs as a valid solution? 

IMO, it is not in a state that it needs ti be considered seriously,
not yet.

Re: Why is Elisp slow?

[Eli Zaretskii]

> Date: Fri, 3 May 2019 02:44:16 +0200
> From: Ergus <spacibba@aol.com>
> Cc: help-gnu-emacs@gnu.org
> 
> Having Guile as a dependency will grow emacs size significantly
> and using it as an external dependency (not usually the emacs way...)
> will require to port Guile to more systems (AFAIK).

Which systems we care about are not already supported by Guile?

> The alternative JIT may be based in libJIT which is not very active
> either... and has serious issues/limitations that has not been solved in
> years.

There's a branch in the Emacs repository that uses libJIT.  It has
problems with some Emacs configurations, which need to be solved in
libJIT.  Alas, those problems, reported months ago to the libJIT
developers, are still not fixed.

More importantly, the libJIT build failed to show any significant
speed-up wrt byte code, so it sounds like maybe the whole idea was
either wrong or its design couldn't possibly provide any gains.  Or
maybe we just measured the speed-up in wrong scenarios.

Re: Why is Elisp slow?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1717 bytes --]

On Fri, May 03, 2019 at 02:44:16AM +0200, Ergus wrote:
> WARNING: This email has plenty of personal opinions.
> 
> On Fri, May 03, 2019 at 08:39:43AM +0900, ????????? wrote:
> >
> >>Probably (I have a dream [...]

> >This, very much sounds like Guile Emacs. Anyone knows how Guile Emacs
> >is doing????? It???s very much looks like vaporware these days :-(
> 
> Yes, that is the dream (at least the closest we have ever been)
> 
> >Is upstream considering Guile Emacs as a valid solution?
> 
> I made a similar question some time ago and the answer is that there is
> not action. :( In fact there is not too much action in
> Guile's development .
> 
> Not many projects feel interested in Guile [...]

C'm on, folks. Dreaming is OK, but once you wake up, you can
try to exercise your search engine skills. Guile is well and
alive (the Guilers meet every year at FOSDEM, talk videos seem
to be available), Guile's maintainer (Andy Wingo) is doing an
excellent job, and one of the more active Guile projects (Guix)
has just celebrated its 1.0 release.

The official web site is a good starting point:

  https://www.gnu.org/software/guile/

Ah, and Andy Wingo's blog is also recommended reading:

  https://wingolog.org

As to guile-emacs, there are two things here: first, the project
needs hands, it seems. I see four hands out there ;-) Second,
not everyone here likes Scheme (and they might be right), so it
would need some convincing (be sure to read the Emacs history by
Stefan Monnier and Michael Sperber, posted elsewhere in this list).

A good starting point on guile-emacs would be

  https://www.emacswiki.org/emacs/GuileEmacs

So -- lots to do!

Cheers
-- t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Why is Elisp slow?

[Ergus]

On Fri, May 03, 2019 at 10:00:45AM +0300, Eli Zaretskii wrote:
>> Date: Fri, 3 May 2019 02:44:16 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: help-gnu-emacs@gnu.org
>>
>> Having Guile as a dependency will grow emacs size significantly
>> and using it as an external dependency (not usually the emacs way...)
>> will require to port Guile to more systems (AFAIK).
>
>Which systems we care about are not already supported by Guile?
>

Actually in windows I have a friend who tried it and the experience was
not convincing for him. And the native code compiler was in fact
tremendously slow in a power9 machine. It felt like a hack in both
cases more than a real native application.

>> The alternative JIT may be based in libJIT which is not very active
>> either... and has serious issues/limitations that has not been solved in
>> years.
>
>There's a branch in the Emacs repository that uses libJIT.  It has
>problems with some Emacs configurations, which need to be solved in
>libJIT.  Alas, those problems, reported months ago to the libJIT
>developers, are still not fixed.
>
>More importantly, the libJIT build failed to show any significant
>speed-up wrt byte code, so it sounds like maybe the whole idea was
>either wrong or its design couldn't possibly provide any gains.  Or
>maybe we just measured the speed-up in wrong scenarios.
>
This is not surprising. My work is 80% performance measurement and
benchmarking and the real improvements with JIT compilation (in my
experience) and specially with libJIT is not as good as many people
expect in most of the common scenarios. That's because the generated
code is usually very generic (so it does not take advantage of
architecture specific features), and strategies like vectorization and
branch prediction are very difficult to hint (most of the times
impossible). So the only real difference with a bytecode interpreter is
the bytecode parsing part, but not too much more.

There is also the typical array of struct vs struct of arrays problem, type
aligns and the optimal memory management what are also very dark to
advice for a JIT compiler. The use of dark types and pointers to
functions produces huge overheads and they are extensively used in JIT
compilers internally generated code, the impossibility to do proper
inlinings, and of course the fact that coming from a higher level
language like Lisp we need more dynamic/runtime functionalities like
garbage collector and generic function types. Plus the hashtable search
costs and the extensively used list searches in Lisp instead of vectors
or similar cheaper containers and callbacks in the user code.

That's why I support the idea of the Lisp->C compiler more than the JIT
at least to use it for the emacs core Lisp functions because gcc is very
optimized and migrate the bottleneck back to C code as much as
possible. (many of the very used in simple.el for example that are
called very frequently)

That's the strategy that made python become so successfully (plus the
language simplicity). The existence of Cython, Numba + Pure C/Fortran
core libraries like numpy and a simple C API to extend it. So when a
user implements anything in 100% python; if it has performance issues
she only needs to change the python lists with numpy arrays and numpy
functions and that's it, the difference will be observed
immediately. But if a specific function still represents a problem
she can implement it easily in C or use a C library directly with
CTypes, Pybind11 or boost, that uses dlopen instead of create processes
and parse outputs and writes in numpy arrays/types directly.

So in spite of that python's virtual machine is very slow to load, the
rest of the execution is not necessarily slow and can be improved as
much as the developer desires. Also as there are more users the bytecode
part (interpreter and generator) is improved and optimized constantly.

Re: Why is Elisp slow?

[Ergus]

On Fri, May 03, 2019 at 10:06:56AM +0900, ????????? wrote:
>I became curious after reading your email :-)
>
>Since implementing a new language/byte interpreter (Guile, for example) is hard, and making it fast is harder, why doesn't Emacs leverage the Common Lisp environment?????
>
It is not a language issue I think, but a compiler/interpreter
optimization level.

>For example, (since CL and elisp???s syntax is mostly similar) can???t we implement Emacs based on CL (looks like there were ongoing efforts before like Climacs, Hemlock, etc???) and expose elisp functions in a special package named ???elisp???????? It would allow elisp compatibility (by running all elisp code in the special elisp package) while giving package writers access to CL libraries. Since legacy cruft can go inside the package ???elisp???, CL emacs APIs can remove/redesign some old APIs and remove technical debt. This would also greatly accerlate Emacs speed as CL is super fast especially when using with SBCL.
>
That was the emacs-guile approach. The problem was that nobody finished
it. And if the whole project (means the core and more important
developers) do not decide to go in that direction for a good reason
(like they want to evict the bytecode interpreter from within emacs or
they consider important to use a specific Guile functionality) then it
won't happen. Also, doing big changes like that in emacs core code
becomes a big discussion that takes loooooong time in the mailing list. 

>Are there any reasons (that I do not know) why CL was not considered or was considered but rejected as an elisp alternative?
>Are there license problems to build on previous attempts?????
>
maybe it has to do with this:

https://www.gnu.org/gnu/rms-lisp.en.html

>IMHO building emacs on CL would be a wonderful idea... and would fix many problems that current Emacs have.
>
And create others, it is always a trade of.

Any way

I have to say that comparing to other interpreters around the Elisp is
not the slowest one I have tried by far. Of course it could be way
faster, but some optimization like user code and provide new containers
and datatypes in the library level based in C (and recomend them) code
could potentially make more difference in real user extensions.

>> 2019. 5. 3. ?????? 9:44, Ergus <spacibba@aol.com> ??????:
>>
>> WARNING: This email has plenty of personal opinions.
>>
>>> On Fri, May 03, 2019 at 08:39:43AM +0900, ????????? wrote:
>>>
>>>> Probably (I have a dream) if someone decides to develop a new emacs in
>>>> 2019 most of the functions and API will be made in pure and clear C (or
>>>> any other compiled language), with a full C api that gives the same
>>>> access level than what elisp gives in Emacs today (with C lists, arrays
>>>> and structs), so it will be not only faster but also simpler to extend
>>>> it with other languages like Scheme, Python, Lua, C++, rust and so
>>>> on. (There are modules projects going in that direction) And the editor
>>>> don't even need to provide a compiler or interpreter for them. (there
>>>> will be Guile/gcc/python and so on for that)
>>>
>>> This, very much sounds like Guile Emacs. Anyone knows how Guile Emacs
>>> is doing????? It???s very much looks like vaporware these days :-(
>>
>> Yes, that is the dream (at least the closest we have ever been)
>>
>>> Is upstream considering Guile Emacs as a valid solution?
>>
>> I made a similar question some time ago and the answer is that there is
>> not action. :( In fact there is not too much action in
>> Guile's development .
>>
>> Not many projects feel interested in Guile because the weak support it
>> has so there are few users and few developers (But also because
>> Lisp-like family languages (common Lisp, Scheme and the others) are a
>> museum piece for young generations that are more used to Python, Ruby,
>> javascript, and most of OOP (also because there are more jobs with
>> that))
>>
>> Actually I think that these days will be easier to find new C/C++
>> developers for emacs than Lisp developers. Lisp and Scheme are
>> beautiful, but they require a different way of thinking and a lot of
>> time (own experience, I am just starting with it.) One reason why I
>> can't convince my friends to use Emacs is actually how Lisp scares them
>> ((())()'()).
>>
>>> Is there any development ongoing? (Official or not?)
>> I don't think so :( :( :(. It will require that some core emacs
>> developers feel more interested in implementing this, because it is a
>> lot of work... (specially to do it right and keeping the so sacred
>> backward compatibility) These days only few people know the core parts
>> of emacs for such a task.
>>
>> Having Guile as a dependency will grow emacs size significantly
>> and using it as an external dependency (not usually the emacs way...)
>> will require to port Guile to more systems (AFAIK). BUt such desition
>> could benefit very much both projects.
>>
>> There are even some emacs ports to rust (remacs) but they still have the
>> elisp as the core languaje. Because most of the code and functionalities
>> are already in elisp so changing that will be like reimplementing all
>> emacs from scratch.
>>
>> (Some time ago there was a lot of effort and time invested in porting C
>> functions to Elisp in emacs)
>>
>> So with our actual emacs maybe the JIT or the Lisp->C source to source
>> is actually the more reliable option in a possible (realistic) future.
>>
>> Personally I feel the Lisp->C compiler feels like the faster and more
>> robust solution for me, but it will create a dependency with
>> binutils... which is for multiple reasons undesired.
>>
>> The alternative JIT may be based in libJIT which is not very active
>> either... and has serious issues/limitations that has not been solved in
>> years.
>>
>> [1] https://tromey.com/blog/?p=982
>> [2] https://lists.gnu.org/archive/html/emacs-devel/2018-08/msg00393.html
>

Re: Why is Elisp slow?

[조성빈]

>> For example, (since CL and elisp???s syntax is mostly similar) can???t we implement Emacs based on CL (looks like there were ongoing efforts before like Climacs, Hemlock, etc???) and expose elisp functions in a special package named ???elisp???????? It would allow elisp compatibility (by running all elisp code in the special elisp package) while giving package writers access to CL libraries. Since legacy cruft can go inside the package ???elisp???, CL emacs APIs can remove/redesign some old APIs and remove technical debt. This would also greatly accerlate Emacs speed as CL is super fast especially when using with SBCL.
>> 
> That was the emacs-guile approach. The problem was that nobody finished
> it. And if the whole project (means the core and more important
> developers) do not decide to go in that direction for a good reason
> (like they want to evict the bytecode interpreter from within emacs or
> they consider important to use a specific Guile functionality) then it
> won't happen. Also, doing big changes like that in emacs core code
> becomes a big discussion that takes loooooong time in the mailing list. 

The difference between guile and CL is that
* Guile is arguably not popular and not used by many programs.I’ve never heard a program that is popular and uses guile as an extension language. CL is, well, much popular than Guile, and is used by industry-strength programs.
* Guile (which is a scheme) has a radically different syntax with elisp (which means that to implement elisp in guile, one has to hack the byte code interpreter AFAIK). Elisp can be implemented inside CL with full compatibility, which mitigates lots of problems between interfacing between CL code and elisp code.
* I’m pretty sure CL will be much, much faster than guile. CL was optimized for about 30 years or so, and SBCL compiles CL all the way to native code, which is comparable to C. I’m not sure if guile can do that.

>> Are there any reasons (that I do not know) why CL was not considered or was considered but rejected as an elisp alternative?
>> Are there license problems to build on previous attempts?????
>> 
> maybe it has to do with this:
> 
> https://www.gnu.org/gnu/rms-lisp.en.html <https://www.gnu.org/gnu/rms-lisp.en.html>

I read the link; it’s a pity to not consider CL because it isn’t ‘lispy’ enough. 

> 
>> IMHO building emacs on CL would be a wonderful idea... and would fix many problems that current Emacs have.
>> 
> And create others, it is always a trade off.

What problems would CL based Emacs can make?̊̈ Can you show some examples?̊̈ I am genuinely interested.

> 
> Any way
> 
> I have to say that comparing to other interpreters around the Elisp is
> not the slowest one I have tried by far. Of course it could be way
> faster, but some optimization like user code and provide new containers
> and datatypes in the library level based in C (and recomend them) code
> could potentially make more difference in real user extensions.

Well, CL is as fast as C…. 

> 
>>> 2019. 5. 3. ?????? 9:44, Ergus <spacibba@aol.com> ??????:
>>> 
>>> WARNING: This email has plenty of personal opinions.
>>> 
>>>> On Fri, May 03, 2019 at 08:39:43AM +0900, ????????? wrote:
>>>> 
>>>>> Probably (I have a dream) if someone decides to develop a new emacs in
>>>>> 2019 most of the functions and API will be made in pure and clear C (or
>>>>> any other compiled language), with a full C api that gives the same
>>>>> access level than what elisp gives in Emacs today (with C lists, arrays
>>>>> and structs), so it will be not only faster but also simpler to extend
>>>>> it with other languages like Scheme, Python, Lua, C++, rust and so
>>>>> on. (There are modules projects going in that direction) And the editor
>>>>> don't even need to provide a compiler or interpreter for them. (there
>>>>> will be Guile/gcc/python and so on for that)
>>>> 
>>>> This, very much sounds like Guile Emacs. Anyone knows how Guile Emacs
>>>> is doing????? It???s very much looks like vaporware these days :-(
>>> 
>>> Yes, that is the dream (at least the closest we have ever been)
>>> 
>>>> Is upstream considering Guile Emacs as a valid solution?
>>> 
>>> I made a similar question some time ago and the answer is that there is
>>> not action. :( In fact there is not too much action in
>>> Guile's development .
>>> 
>>> Not many projects feel interested in Guile because the weak support it
>>> has so there are few users and few developers (But also because
>>> Lisp-like family languages (common Lisp, Scheme and the others) are a
>>> museum piece for young generations that are more used to Python, Ruby,
>>> javascript, and most of OOP (also because there are more jobs with
>>> that))
>>> 
>>> Actually I think that these days will be easier to find new C/C++
>>> developers for emacs than Lisp developers. Lisp and Scheme are
>>> beautiful, but they require a different way of thinking and a lot of
>>> time (own experience, I am just starting with it.) One reason why I
>>> can't convince my friends to use Emacs is actually how Lisp scares them
>>> ((())()'()).
>>> 
>>>> Is there any development ongoing? (Official or not?)
>>> I don't think so :( :( :(. It will require that some core emacs
>>> developers feel more interested in implementing this, because it is a
>>> lot of work... (specially to do it right and keeping the so sacred
>>> backward compatibility) These days only few people know the core parts
>>> of emacs for such a task.
>>> 
>>> Having Guile as a dependency will grow emacs size significantly
>>> and using it as an external dependency (not usually the emacs way...)
>>> will require to port Guile to more systems (AFAIK). BUt such desition
>>> could benefit very much both projects.
>>> 
>>> There are even some emacs ports to rust (remacs) but they still have the
>>> elisp as the core languaje. Because most of the code and functionalities
>>> are already in elisp so changing that will be like reimplementing all
>>> emacs from scratch.
>>> 
>>> (Some time ago there was a lot of effort and time invested in porting C
>>> functions to Elisp in emacs)
>>> 
>>> So with our actual emacs maybe the JIT or the Lisp->C source to source
>>> is actually the more reliable option in a possible (realistic) future.
>>> 
>>> Personally I feel the Lisp->C compiler feels like the faster and more
>>> robust solution for me, but it will create a dependency with
>>> binutils... which is for multiple reasons undesired.
>>> 
>>> The alternative JIT may be based in libJIT which is not very active
>>> either... and has serious issues/limitations that has not been solved in
>>> years.
>>> 
>>> [1] https://tromey.com/blog/?p=982
>>> [2] https://lists.gnu.org/archive/html/emacs-devel/2018-08/msg00393.html
>> 
> 

Re: Why is Elisp slow?

[Eli Zaretskii]

> Date: Fri, 3 May 2019 11:58:49 +0200
> From: Ergus <spacibba@aol.com>
> Cc: help-gnu-emacs@gnu.org
> 
> >Which systems we care about are not already supported by Guile?
> >
> 
> Actually in windows I have a friend who tried it and the experience was
> not convincing for him.

I don't know what that means.  Guile 2.0.x compiles on Windows, and
(32-bit) binaries are available from the ezwinports site.  Those
binaries are good enough to build GDB with Guile support on Windows.

It's possible that later versions of Guile "need work" to be able to
compile and work well on Windows, but the amount of problems cannot be
too large, given all the fixes that went into 2.0.x.

Re: Why is Elisp slow?

[Eli Zaretskii]

> From: 조성빈 <pcr910303@icloud.com>
> Date: Fri, 3 May 2019 20:52:06 +0900
> Cc: help-gnu-emacs@gnu.org
> 
> * Guile (which is a scheme) has a radically different syntax with elisp (which means that to implement elisp in guile, one has to hack the byte code interpreter AFAIK). Elisp can be implemented inside CL with full compatibility, which mitigates lots of problems between interfacing between CL code and elisp code.

This is not a problem with Guile, because Guile includes a compiler
and interpreter for Emacs Lisp.

Re: Why is Elisp slow?

[Ergus]

On Fri, May 03, 2019 at 08:52:06PM +0900, ????????? wrote:
>
>
>The difference between guile and CL is that * Guile is arguably not
>popular and not used by many programs.I???ve never heard a program that
>is popular and uses guile as an extension language. CL is, well, much
>popular than Guile, and is used by industry-strength programs.

> * Guile (which is a scheme) has a radically different syntax with
>elisp (which means that to implement elisp in guile, one has to hack
>the byte code interpreter AFAIK). Elisp can be implemented inside CL
>with full compatibility, which mitigates lots of problems between
>interfacing between CL code and elisp code.

But guile is in fact developed in the gnu project, so there are the
guile developers (very few but they are) that could potentially
provide and fix any issue. But also the strong part of guile is not the
language itself but the potential of the infrastructure.

>* I???m pretty sure CL will be much, much faster than guile.

Probably, that depends of the compiler not only the language.

>CL was optimized for about 30 years or so, and SBCL compiles CL all the way to native code, which is comparable to C. I???m not sure if guile can do that.
>
Guile has been also optimized (in spite of there is a lot of work to do,
that's true) and yes, it can generate native code, but also
bytecode. But it also offers a lot of functionalities to extend and work
as a glue language which is actually most important in our days.

At the end (in my opinion) if emacs wants to survive other 40 years it
will need to start looking for integration with python, lua, C++, rust,
Ruby and similar languages with a more "promising" future than common
lisp. Because we really need to attract new developers with new ideas,
visions and experiences... and 99% of programmers don't use any lisp
like language.

To make big changes we need more developers, that will not come if we
don't have more users first. And those new users/developers can enforce
the new changes looking at the future and not to backward compatibility.


>> https://www.gnu.org/gnu/rms-lisp.en.html <https://www.gnu.org/gnu/rms-lisp.en.html>
>
>I read the link; it???s a pity to not consider CL because it isn???t ???lispy??? enough.
>
>>
>>> IMHO building emacs on CL would be a wonderful idea... and would fix many problems that current Emacs have.
>>>
>> And create others, it is always a trade off.
>
>What problems would CL based Emacs can make????? Can you show some examples????? I am genuinely interested.
>

Dynamic scoping rules vs exical scoping rules in common lisp is just the
first one I can remember, but maybe someone else will reply you with more details.

>>
>> Any way
>>
>> I have to say that comparing to other interpreters around the Elisp is
>> not the slowest one I have tried by far. Of course it could be way
>> faster, but some optimization like user code and provide new containers
>> and datatypes in the library level based in C (and recomend them) code
>> could potentially make more difference in real user extensions.
>
>Well, CL is as fast as C???.
>

But again, common lisp is just a language. Performance is more associated
with the compiler you use. Cython makes python very fast too for
example AND IT IS PYTHON!!!.  The latest version of the GNU common lisp
compiler (GCL) was in 2014... so its developement goes slower than
Guile.

If we had a native compiler for Elisp it could be as fast as common lisp
(ideally). But in any case it needs to be something that goes embedded
within emacs (otherwise emacs won't be emacs anymore).

Think also in the licenses issues and legal consequences of such strong
dependency for a key part of the editor.

On the other hand it is true that in that case that will be a part of
the code that we won't need to maintain if we rely on other packages.

Let me say something. I am not opposed at all to switch to common lisp,
(or scheme or python, actually I think that at some point it will be
needed to change the core language for something more flexible)
what I mean is that this specific problem is not a language issue, but the
compiler. And I don't think that we can just copy the SBCL code into
emacs for many reasons.

Re: Why is Elisp slow?

[Ergus]

On Fri, May 03, 2019 at 03:44:17PM +0300, Eli Zaretskii wrote:
>> From: ????????? <pcr910303@icloud.com>
>> Date: Fri, 3 May 2019 20:52:06 +0900
>> Cc: help-gnu-emacs@gnu.org
>>
>> * Guile (which is a scheme) has a radically different syntax with elisp (which means that to implement elisp in guile, one has to hack the byte code interpreter AFAIK). Elisp can be implemented inside CL with full compatibility, which mitigates lots of problems between interfacing between CL code and elisp code.
>
>This is not a problem with Guile, because Guile includes a compiler
>and interpreter for Emacs Lisp.
>
Hi Eli, now I am the curious.

If that's already done (which sems to be the harder part), where is the
real problem to migrate to guile? I know there should be many, but what
are the known ones??

Re: Why is Elisp slow?

[조성빈]

Looks like the point of getting Guile Emacs recurs to:

* How much Guile Emacs is polished(seamless integration with Elisp, full compatibility, etc..)
* How much Guile supports other languages such as JS or Lua
* How much Guile manages to be fast

Guile Emacs was developed for at least 15 years or more, right?

How much can Guile Emacs do these in current situation?̊̈
Is Guile Emacs usable day-to-day?


> 2019. 5. 3. 오후 9:51, Ergus <spacibba@aol.com> 작성:
> 
> On Fri, May 03, 2019 at 08:52:06PM +0900, 조성빈 wrote:
> 
>> The difference between Guile and CL is that
>> * Guile is arguably not popular and not used by many programs. I've never heard a program that is popular and uses Guile as an extension language. CL is, well, much more popular than Guile, and is used by industry-strength programs.
> 
>> * Guile (which is a scheme) has a radically different syntax with
>> elisp (which means that to implement elisp in Guile, one has to hack
>> the byte code interpreter AFAIK). Elisp can be implemented inside CL
>> with full compatibility, which mitigates lots of problems between
>> interfacing between CL code and elisp code.
> 
> But Guile is in fact developed in the gnu project, so there are the
> Guile developers (very few but they are) that could potentially
> provide and fix any issue. But also the strong part of Guile is not the
> language itself but the potential of the infrastructure.
> 
>> * I'm pretty sure CL will be much, much faster than Guile.
> 
> Probably, that depends of the compiler not only the language.
> 
>> CL was optimized for about 30 years or so, and SBCL compiles CL all the way to native code, which is comparable to C. I'm not sure if Guile can do that.
> 
> Guile has been also optimized (in spite of there is a lot of work to do,
> that's true) and yes, it can generate native code, but also
> bytecode. But it also offers a lot of functionalities to extend and work
> as a glue language which is actually most important in our days.
> 
> At the end (in my opinion) if emacs wants to survive other 40 years it
> will need to start looking for integration with python, lua, C++, rust,
> Ruby and similar languages with a more "promising" future than common
> lisp. Because we really need to attract new developers with new ideas,
> visions and experiences... and 99% of programmers don't use any lisp
> like language.
> 
> To make big changes we need more developers, that will not come if we
> don't have more users first. And those new users/developers can enforce
> the new changes looking at the future and not to backward compatibility.
> 
> 
>>> https://www.gnu.org/gnu/rms-lisp.en.html <https://www.gnu.org/gnu/rms-lisp.en.html>
>> 
>> I read the link; it's a pity to not consider CL because it isn't "lispy" enough.
>> 
>>> 
>>>> IMHO building emacs on CL would be a wonderful idea... and would fix many problems that current Emacs have.
>>>> 
>>> And create others, it is always a trade off.
>> 
>> What problems would CL based Emacs can make? Can you show some examples? I am genuinely interested.
>> 
> 
> Dynamic scoping rules vs exical scoping rules in common lisp is just the
> first one I can remember, but maybe someone else will reply you with more details.
> 
>>> 
>>> Any way
>>> 
>>> I have to say that comparing to other interpreters around the Elisp is
>>> not the slowest one I have tried by far. Of course it could be way
>>> faster, but some optimization like user code and provide new containers
>>> and datatypes in the library level based in C (and recomend them) code
>>> could potentially make more difference in real user extensions.
>> 
>> Well, CL is as fast as C.
> 
> But again, common lisp is just a language. Performance is more associated
> with the compiler you use. Cython makes python very fast too for
> example AND IT IS PYTHON!!!.  The latest version of the GNU common lisp
> compiler (GCL) was in 2014... so its developement goes slower than
> Guile.
> 
> If we had a native compiler for Elisp it could be as fast as common lisp
> (ideally). But in any case it needs to be something that goes embedded
> within emacs (otherwise emacs won't be emacs anymore).
> 
> Think also in the licenses issues and legal consequences of such strong
> dependency for a key part of the editor.
> 
> On the other hand it is true that in that case that will be a part of
> the code that we won't need to maintain if we rely on other packages.
> 
> Let me say something. I am not opposed at all to switch to common lisp,
> (or scheme or python, actually I think that at some point it will be
> needed to change the core language for something more flexible)
> what I mean is that this specific problem is not a language issue, but the
> compiler. And I don't think that we can just copy the SBCL code into
> emacs for many reasons.

Re: Why is Elisp slow?

[Ergus]

On Fri, May 03, 2019 at 10:16:32PM +0900, ????????? wrote:
>Looks like the point of getting Guile Emacs recurs to:
>
>* How much Guile Emacs is polished(seamless integration with Elisp, full compatibility, etc..)
Maybe here is the actual problem.. I actually asked Eli.

>* How much Guile supports other languages such as JS or Lua
The C api is pretty nice and easy. I have some code done on it. JS is
a different beast. But having C it is straightforward to interface
python and Lua. Actually the Lua C API is much more complicated that
than Guile's.

>* How much Guile manages to be fast

It is decently fine and slowly is getting better.
>
>Guile Emacs was developed for at least 15 years or more, right?
>
>How much can Guile Emacs do these in current situation?????
>Is Guile Emacs usable day-to-day?
>
The problem is maintenance and manpower. The emacs developers are very
busy with bugfixes and there is A LOT of code to maintain/update/fix.

It needs just a voluntary to do the work. I am fine with the C
integration but all Lisp is still a bit esoteric for me. 
>
>> 2019. 5. 3. ?????? 9:51, Ergus <spacibba@aol.com> ??????:
>>
>> On Fri, May 03, 2019 at 08:52:06PM +0900, ????????? wrote:
>>
>>> The difference between Guile and CL is that
>>> * Guile is arguably not popular and not used by many programs. I've never heard a program that is popular and uses Guile as an extension language. CL is, well, much more popular than Guile, and is used by industry-strength programs.
>>
>>> * Guile (which is a scheme) has a radically different syntax with
>>> elisp (which means that to implement elisp in Guile, one has to hack
>>> the byte code interpreter AFAIK). Elisp can be implemented inside CL
>>> with full compatibility, which mitigates lots of problems between
>>> interfacing between CL code and elisp code.
>>
>> But Guile is in fact developed in the gnu project, so there are the
>> Guile developers (very few but they are) that could potentially
>> provide and fix any issue. But also the strong part of Guile is not the
>> language itself but the potential of the infrastructure.
>>
>>> * I'm pretty sure CL will be much, much faster than Guile.
>>
>> Probably, that depends of the compiler not only the language.
>>
>>> CL was optimized for about 30 years or so, and SBCL compiles CL all the way to native code, which is comparable to C. I'm not sure if Guile can do that.
>>
>> Guile has been also optimized (in spite of there is a lot of work to do,
>> that's true) and yes, it can generate native code, but also
>> bytecode. But it also offers a lot of functionalities to extend and work
>> as a glue language which is actually most important in our days.
>>
>> At the end (in my opinion) if emacs wants to survive other 40 years it
>> will need to start looking for integration with python, lua, C++, rust,
>> Ruby and similar languages with a more "promising" future than common
>> lisp. Because we really need to attract new developers with new ideas,
>> visions and experiences... and 99% of programmers don't use any lisp
>> like language.
>>
>> To make big changes we need more developers, that will not come if we
>> don't have more users first. And those new users/developers can enforce
>> the new changes looking at the future and not to backward compatibility.
>>
>>
>>>> https://www.gnu.org/gnu/rms-lisp.en.html <https://www.gnu.org/gnu/rms-lisp.en.html>
>>>
>>> I read the link; it's a pity to not consider CL because it isn't "lispy" enough.
>>>
>>>>
>>>>> IMHO building emacs on CL would be a wonderful idea... and would fix many problems that current Emacs have.
>>>>>
>>>> And create others, it is always a trade off.
>>>
>>> What problems would CL based Emacs can make? Can you show some examples? I am genuinely interested.
>>>
>>
>> Dynamic scoping rules vs exical scoping rules in common lisp is just the
>> first one I can remember, but maybe someone else will reply you with more details.
>>
>>>>
>>>> Any way
>>>>
>>>> I have to say that comparing to other interpreters around the Elisp is
>>>> not the slowest one I have tried by far. Of course it could be way
>>>> faster, but some optimization like user code and provide new containers
>>>> and datatypes in the library level based in C (and recomend them) code
>>>> could potentially make more difference in real user extensions.
>>>
>>> Well, CL is as fast as C.
>>
>> But again, common lisp is just a language. Performance is more associated
>> with the compiler you use. Cython makes python very fast too for
>> example AND IT IS PYTHON!!!.  The latest version of the GNU common lisp
>> compiler (GCL) was in 2014... so its developement goes slower than
>> Guile.
>>
>> If we had a native compiler for Elisp it could be as fast as common lisp
>> (ideally). But in any case it needs to be something that goes embedded
>> within emacs (otherwise emacs won't be emacs anymore).
>>
>> Think also in the licenses issues and legal consequences of such strong
>> dependency for a key part of the editor.
>>
>> On the other hand it is true that in that case that will be a part of
>> the code that we won't need to maintain if we rely on other packages.
>>
>> Let me say something. I am not opposed at all to switch to common lisp,
>> (or scheme or python, actually I think that at some point it will be
>> needed to change the core language for something more flexible)
>> what I mean is that this specific problem is not a language issue, but the
>> compiler. And I don't think that we can just copy the SBCL code into
>> emacs for many reasons.
>
>

Re: Why is Elisp slow?

[Eli Zaretskii]

> Date: Fri, 3 May 2019 14:58:32 +0200
> From: Ergus <spacibba@aol.com>
> Cc: help-gnu-emacs@gnu.org
> 
> >This is not a problem with Guile, because Guile includes a compiler
> >and interpreter for Emacs Lisp.
> >
> Hi Eli, now I am the curious.
> 
> If that's already done (which sems to be the harder part), where is the
> real problem to migrate to guile? I know there should be many, but what
> are the known ones??

There were past discussions about that, I suggest to look them up.

One of the problems I remember is the basic difference in philosophy
regarding raw bytes in text strings: Guile disallows that, and in many
cases signals errors, while ELisp allows that, and Emacs as a project
is unlikely to change that attitude (while Guile developers seemed to
be convinced theirs is the right philosophy).  Since this directly
affects the internal representation of strings, we have a problem
here.  Maybe this is not the most important problem, but until it's
solved, we cannot even think about moving on.

In general, the integration of Guile into Emacs is simply not finished
yet, not even close.  We cannot land such unstable code on master.
And of course, the longer the guile-emacs project is left as an
unmaintained fork, the more bitrot is accrues, for example it doesn't
support latest Emacs features like bignums in ELisp (although Guile
itself does support that).  In short, without an active team working
on it, this will never happen.  (As everything else in Emacs, except
that this feature is very complex and needs a significant effort.)

Re: Why is Elisp slow?

[Eli Zaretskii]

> From: 조성빈 <pcr910303@icloud.com>
> Date: Fri, 3 May 2019 22:16:32 +0900
> Cc: help-gnu-emacs@gnu.org
> 
> Is Guile Emacs usable day-to-day?

I don't think so, but you can try it and find out yourself.

I think in the past the person who worked on guile-emacs posted a
summary of the status and the problems still to be solved, so maybe
look that up.

Re: Why is Elisp slow?

[Óscar Fuentes]

Ergus <spacibba@aol.com> writes:

>>More importantly, the libJIT build failed to show any significant
>>speed-up wrt byte code, so it sounds like maybe the whole idea was
>>either wrong or its design couldn't possibly provide any gains.  Or
>>maybe we just measured the speed-up in wrong scenarios.
>>
> This is not surprising. My work is 80% performance measurement and
> benchmarking and the real improvements with JIT compilation (in my
> experience) and specially with libJIT is not as good as many people
> expect in most of the common scenarios. That's because the generated
> code is usually very generic (so it does not take advantage of
> architecture specific features), and strategies like vectorization and
> branch prediction are very difficult to hint (most of the times
> impossible). So the only real difference with a bytecode interpreter is
> the bytecode parsing part, but not too much more.

On Emacs case, I'm pretty sure whatever advantages comes from good
architecture-specific code, accurate branch prediction, etc are below
the noise level. As you pointed out below, Elisp is a dynamic language
and for turning this

(let ((acc 0))
  (dotimes (i 10)
    (setq acc (+ acc (foo i)))))
    
into this

int acc = 0;
for(int i = 0; i < 10; ++i) {
  acc += foo(i);
}

you need either sophisticated analysis (that, in practice, only works
for the "easy" cases) or annotate the code with type declarations (and
enforce then).

Because otherwise handling variables as containers for generic values is
incompatible with "C-like" performance.

And an Elisp -> C translator does not magically solve this.

Re: Why is Elisp slow?

[Stefan Monnier]

> If that's already done (which sems to be the harder part), where is the
> real problem to migrate to guile? I know there should be many, but what
> are the known ones??

There's also the question of what would be the advantages.


        Stefan

Re: Why is Elisp slow?

[Samuel Wales]

On 5/2/19, Marcin Borkowski <mbork@mbork.pl> wrote:
> ...well, Org-mode agenda takes almost 30 seconds to generate for me...

[replying in general]

org agenda is the slowest part of emacs for me.

it would be *great* if devs could speed it up with c.  i am curious
what would they do?  take a look at the profiles.  maybe speed up
regexps?

a complete rewrite in lisp might or might not speed it up.  it might
or might not allow devs to speed it up with c.  i think a complete
rewrite has been proposed, but dunno if it's going to be done or will
keep compatibility or be known to be likely faster.

i /think/ a rewrite /has already/ been done, with a nice lisp syntax
to replace the existing non-lisp tags search syntax.  but last i heard
it leaves out a small number of features and it isn't faster yet.
iirc part of the reason given for that was that the existing one was
optimized to death.  however, idk if it is a complete rewrite or if
speed was a desideratum.  perhaps it or parts of it could be rewritten
or sped up in c.  dunno.

as for cl, it's so similar to el that it is a natural comparison.
which makes me wonder why a participant in this thread said it was for
wizards?  elisp has buffers, syntax tables, and font locking.

-- 
The Kafka Pandemic

What is misopathy?
https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html

The disease DOES progress. MANY people have died from it. And ANYBODY
can get it at any time.

Re: Why is Elisp slow?

[Emanuel Berg]

Ergus wrote:

> At the end (in my opinion) if emacs wants to
> survive other 40 years it will need to start
> looking for integration with python, lua,
> C++, rust, Ruby and similar languages with
> a more "promising" future than common lisp.

Maybe there are some modules or interfaces
around that to some degree makes it possible to
use other languages than Lisp the Emacs
extension and configuration language.

However even so I think it'll stay at that.
People won't accept an Emacs w/o Lisp. Its half
the fun of it.

> Ruby and similar languages with a more
> "promising" future than common lisp.
> Because we really need to attract new
> developers with new ideas, visions and
> experiences... and 99% of programmers don't
> use any lisp like language.

IMO there is no need to do anything in
particular to attract anyone. If they are
attracted by Emacs enough, but for some reason
don't want Lisp with it, they will come anyway,
fork it and try (perhaps even succeed) in
having another language replace Lisp.

But even if this happens this breakaway Emacs
will be a very little ship in the Emacs world.

People like Lisp, some even love it! Just take
a look at the [M]ELPA archives and people's
Elisp dotfiles all around the world. (Now it
sounded like I do that, only I don't. But 1)
I have my own dotfiles and 2) I don't have to
see them other guys' to know they are
there. :))

How about Emacs w/ Haskell for example?
Side-effect free Emacs - no matter what button
you push, nothing happens! :)

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Why is Elisp slow?

[Emanuel Berg]

Ergus wrote:

> That's the strategy that made python become
> so successfully (plus the language
> simplicity). The existence of Cython, Numba +
> Pure C/Fortran core libraries like numpy and
> a simple C API to extend it. So when a user
> implements anything in 100% python; if it has
> performance issues she only needs to change
> the python lists with numpy arrays and numpy
> functions and that's it, the difference will
> be observed immediately. But if a specific
> function still represents a problem she can
> implement it easily in C or use a C library
> directly with CTypes, Pybind11 or boost, that
> uses dlopen instead of create processes and
> parse outputs and writes in numpy
> arrays/types directly.

Jesus these days you need a PhD to write
a program.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Why is Elisp slow?

[Eli Zaretskii]

> From: Samuel Wales <samologist@gmail.com>
> Date: Fri, 3 May 2019 16:34:21 -0700
> Cc: help-gnu-emacs@gnu.org
> 
> org agenda is the slowest part of emacs for me.
> 
> it would be *great* if devs could speed it up with c.  i am curious
> what would they do?  take a look at the profiles.  maybe speed up
> regexps?

A profile of a long run with all its tree branches expanded would be
nice, preferably with the relevant Org Lisp files loaded as *.el, to
increase the accuracy of the profile.  Bonus points for providing Org
file(s) needed to reproduce such a long run, which could serve as
basis for trying various speed-up approaches.

If someone wants to do that, please use the bug-reporting features in
Emacs (so that the relevant system configuration aspects are captured,
and the data is recorded by the issue tracker), and please CC the
relevant Org addresses (mailing list, issue tracker, etc.).

Re: Why is Elisp slow?

[Marcin Borkowski]

On 2019-05-04, at 02:32, Emanuel Berg <moasenwood@zoho.eu> wrote:

> Side-effect free Emacs - no matter what button
> you push, nothing happens! :)

Well, Vim has already taken that niche (though it's not exactly
"nothing", but "nothing with a beep").

;-)

-- 
Marcin Borkowski
http://mbork.pl

Re: Why is Elisp slow?

[Marcin Borkowski]

On 2019-05-02, at 22:00, tomas@tuxteam.de wrote:

> On Thu, May 02, 2019 at 09:13:28PM +0200, Marcin Borkowski wrote:
>
> [...]
>
>> > (My personal rant to package authors is that they are way too eager to
>> > implement stuff in Lisp which cannot possibly be fast enough or
>> > scalable enough, instead of urging Emacs to provide new C primitives
>> > and core features, or, better, submitting patches to implement such
>> > features in C.  Two cases in point are linum-mode and fci-mode.  End
>> > of rant.)
>> 
>> That is an interesting POV, although I'm not sure if I agree.  IOW,
>> I fully understand why people might do so.  I, for one, consider myself
>> pretty fluent in Elisp and hardly literate in C.
>
> This approach is known as "Ousterhout's dichotomy" [1], after John Ousterhout,
> author of Tcl (which was conceived as a very simple scripting language
> meant to extend existing applications). Tcl grew later a bytecode
> interpreter (as did elisp) -- making the C interface significantly
> more complex (but the interpreter significantly faster).
>
> Walking the trade-off between simple and efficient is interesting.

Thanks.  As usual, I learned something new from you...

Best,

-- 
Marcin Borkowski
http://mbork.pl

Re: Why is Elisp slow?

[Marcin Borkowski]

On 2019-05-02, at 21:45, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Marcin Borkowski <mbork@mbork.pl>
>> Cc: help-gnu-emacs@gnu.org
>> Date: Thu, 02 May 2019 21:13:28 +0200
>> 
>> > It's certainly a reason to report this to the Org developers, and ask
>> > them to speed this up.
>> 
>> Well, I don't consider it a bug in Orgmode - rather a bug in my workflow
>
> Maybe improvements in Org could make your bug less prominent...
>
> Then again, it could be that there's some trick up Org developers'
> sleeve that they could teach you.

As it was already said, there's probably not much room for optimization
here - it was raised many times on the Org-mode mailing list.

> In general, anything surprising/inconvenient/buggy should be reported,
> you can never know what will come out of that.
>
>> > (My personal rant to package authors is that they are way too eager to
>> > implement stuff in Lisp which cannot possibly be fast enough or
>> > scalable enough, instead of urging Emacs to provide new C primitives
>> > and core features, or, better, submitting patches to implement such
>> > features in C.  Two cases in point are linum-mode and fci-mode.  End
>> > of rant.)
>> 
>> That is an interesting POV, although I'm not sure if I agree.  IOW,
>> I fully understand why people might do so.  I, for one, consider myself
>> pretty fluent in Elisp and hardly literate in C.
>
> Submitting changes in C is a bonus, it's good enough to describe the
> use case and ask for primitives to support it.  Someone else might get
> motivated.  It could even be very easy, like with fci-mode.
>
> I'm quite sure that some of the bad reputation of Emacs is due to
> incredible things people do in Lisp, which then make Emacs look
> sluggish and bloated.

Possibly.  Still, I guess I never coded something inefficient/big enough
to need the speedup from a potential new C-level feature.  Though I'm
going to remember what you said here, just in case.

Best,

-- 
Marcin Borkowski
http://mbork.pl

Re: Why is Elisp slow?

[Ergus]

On Sat, May 04, 2019 at 12:18:29AM +0200, ?scar Fuentes wrote:
>Ergus <spacibba@aol.com> writes:
>
>>>More importantly, the libJIT build failed to show any significant
>>>speed-up wrt byte code, so it sounds like maybe the whole idea was
>>>either wrong or its design couldn't possibly provide any gains.  Or
>>>maybe we just measured the speed-up in wrong scenarios.
>>>
>> This is not surprising. My work is 80% performance measurement and
>> benchmarking and the real improvements with JIT compilation (in my
>> experience) and specially with libJIT is not as good as many people
>> expect in most of the common scenarios. That's because the generated
>> code is usually very generic (so it does not take advantage of
>> architecture specific features), and strategies like vectorization and
>> branch prediction are very difficult to hint (most of the times
>> impossible). So the only real difference with a bytecode interpreter is
>> the bytecode parsing part, but not too much more.
>
>On Emacs case, I'm pretty sure whatever advantages comes from good
>architecture-specific code, accurate branch prediction, etc are below
>the noise level. As you pointed out below, Elisp is a dynamic language
>and for turning this
>
>(let ((acc 0))
>  (dotimes (i 10)
>    (setq acc (+ acc (foo i)))))
>
>into this
>
>int acc = 0;
>for(int i = 0; i < 10; ++i) {
>  acc += foo(i);
>}
>
>you need either sophisticated analysis (that, in practice, only works
>for the "easy" cases) or annotate the code with type declarations (and
>enforce then).
>
>Because otherwise handling variables as containers for generic values is
>incompatible with "C-like" performance.
>
>And an Elisp -> C translator does not magically solve this.
>
That's true. The code quality will be not very good (as what happen in
Android Java native compilers and Cython) but some of the optimizations
in the low level can be applied. That depends of the optimizations
implemented and the information provided to the compilers (both of
them).

A Lisp-C compiler for example can reduce significantly the function call
overheads and callback overheads cause thanks to the Lisp syntax it is
very easy to apply inline optimizations which in C represent a VERY
important improvement. In your example code foo, setq and operator + are
functions called in runtime, which interpreted means go to the symbol
hashtable, find the pointer to the function, interpret the inputs and
execute... compiling that... just think how it can change.

With a simple optimization like having the hardcoded 10 the compiler
will know the number of iterations to execute and the C compiler applies
many good optimizations in those cases.

SO it won't be the same than your second code, but performance could be
in the same order in many cases.

Re: Why is Elisp slow?

[Ergus]

On Fri, May 03, 2019 at 06:57:57PM -0400, Stefan Monnier wrote:
>> If that's already done (which sems to be the harder part), where is the
>> real problem to migrate to guile? I know there should be many, but what
>> are the known ones??
>
>There's also the question of what would be the advantages.
>
>
>        Stefan
Inter-language interaction (which is one of the strong features in
Guile out of the box) and native compilation. Plus some people taking
care of the compiler/interpreter/virtual machine parts and specialized
in that specific section, which is good because we need manpower.

Re: Why is Elisp slow?

[조성빈]

2019. 5. 4. 오후 10:27, Ergus <spacibba@aol.com> 작성:

>> On Sat, May 04, 2019 at 12:18:29AM +0200, ?scar Fuentes wrote:
>> Ergus <spacibba@aol.com> writes:
>> 
>>>> More importantly, the libJIT build failed to show any significant
>>>> speed-up wrt byte code, so it sounds like maybe the whole idea was
>>>> either wrong or its design couldn't possibly provide any gains.  Or
>>>> maybe we just measured the speed-up in wrong scenarios.
>>>> 
>>> This is not surprising. My work is 80% performance measurement and
>>> benchmarking and the real improvements with JIT compilation (in my
>>> experience) and specially with libJIT is not as good as many people
>>> expect in most of the common scenarios. That's because the generated
>>> code is usually very generic (so it does not take advantage of
>>> architecture specific features), and strategies like vectorization and
>>> branch prediction are very difficult to hint (most of the times
>>> impossible). So the only real difference with a bytecode interpreter is
>>> the bytecode parsing part, but not too much more.
>> 
>> On Emacs case, I'm pretty sure whatever advantages comes from good
>> architecture-specific code, accurate branch prediction, etc are below
>> the noise level. As you pointed out below, Elisp is a dynamic language
>> and for turning this
>> 
>> (let ((acc 0))
>> (dotimes (i 10)
>>   (setq acc (+ acc (foo i)))))
>> 
>> into this
>> 
>> int acc = 0;
>> for(int i = 0; i < 10; ++i) {
>> acc += foo(i);
>> }
>> 
>> you need either sophisticated analysis (that, in practice, only works
>> for the "easy" cases) or annotate the code with type declarations (and
>> enforce then).
>> 
>> Because otherwise handling variables as containers for generic values is
>> incompatible with "C-like" performance.
>> 
>> And an Elisp -> C translator does not magically solve this.
>> 
> That's true. The code quality will be not very good (as what happen in
> Android Java native compilers and Cython) but some of the optimizations
> in the low level can be applied. That depends of the optimizations
> implemented and the information provided to the compilers (both of
> them).
> 
> A Lisp-C compiler for example can reduce significantly the function call
> overheads and callback overheads cause thanks to the Lisp syntax it is
> very easy to apply inline optimizations which in C represent a VERY
> important improvement.

I’m pretty sure that it is hard to do that, because lisp is a dynamic language, and it is hard to be sure if that function I(the compiler) wants to inline is really the function that the code writer wants to call.
The transpiled code essentially will be C code that contains a small embedded elisp interpreter, which imposes the same overhead as running elisp code through the elisp engine from Emacs.

> In your example code foo, setq and operator + are
> functions called in runtime, which interpreted means go to the symbol
> hashtable, find the pointer to the function, interpret the inputs and
> execute... compiling that... just think how it can change.

Transpiled C code will have the same overhead.

> 
> With a simple optimization like having the hardcoded 10 the compiler
> will know the number of iterations to execute and the C compiler applies
> many good optimizations in those cases.
> 
> SO it won't be the same than your second code, but performance could be
> in the same order in many cases.

Re: Why is Elisp slow?

[Stefan Monnier]

> Inter-language interaction (which is one of the strong features in
> Guile out of the box)

That's fairly abstract (details matter: e.g. how are you going to use
macros like `define-minor-mode` from non-Elisp languages?). 

Also while I much prefer Scheme to Elisp (or Common-Lisp for that
matter), I'm not looking forward to maintaining Emacs packages in two
(and more) different languages, and other maintainers are probably
imagining such maintenance nightmare with a lot more dread than I.

So again: what are the advantages?

From that point of view, moving Emacs to a Common-Lisp implementation
would make more sense, since there is a chance we can *replace* Elisp
with Common-Lisp in the long-run, rather than adding another language on
the side.

> and native compilation.

That's irrelevant.  I think you mean "efficiency" or something like that.
But that presumes that Elisp-on-Guile is indeed more efficient than
Elisp-on-Emacs.  Is there actual evidence that this is the case?

Also "efficiency" is broad: there's the execution time of some
benchmarks, of course, but there's also the startup time, the VM size,
the heap size, the GC interruptions, the time to load a big package
(e.g. org-mode), ...

Guile would hopefully win on some of those, but by how much?
And how much worse does it make the others?

> Plus some people taking care of the compiler/interpreter/virtual
> machine parts and specialized in that specific section, which is good
> because we need manpower.

AFAIC that's the main goal, indeed.

But last I checked Guile didn't have much manpower either.  And there's
a good chance that tracking Guile development (after switching to Guile)
will also require manpower on Emacs's side.  So it's not clear that it
would turn out to be an advantage.


        Stefan

Re: Why is Elisp slow?

[Eli Zaretskii]

> Date: Sat, 4 May 2019 15:32:18 +0200
> From: Ergus <spacibba@aol.com>
> Cc: help-gnu-emacs@gnu.org
> 
> because we need manpower.

For some reason, I very much doubt that integration with Guile will
add a significant amount of manpower to our team.

Re: Why is Elisp slow?

[Emanuel Berg]

Stefan Monnier wrote:

> Also while I much prefer Scheme to Elisp (or
> Common-Lisp for that matter), I'm not looking
> forward to maintaining Emacs packages in two
> (and more) different languages, and other
> maintainers are probably imagining such
> maintenance nightmare with a lot more dread
> than I.

Except for some other advantages with CL
compared to Elisp, which I take it are there
for natural reasons, is the reason CL is much
faster than Elisp, and even as fast as C,
reportedly, using the Steel Bank compiler - is
the reason CL is much faster than Elisp that CL
can be compiled for real, and there is even
a "a high performance" [1] compiler, SBCL,
readily available to do it, *meanwhile* with
Elisp you're stuck with the mere byte-compiler?

So then the first question, literally [2], as
formulated by a wise but in this case ignorant
man, is:

    Why can't Elisp be compiled as well?

> From that point of view, moving Emacs to
> a Common-Lisp implementation would make more
> sense, since there is a chance we can
> *replace* Elisp with Common-Lisp in the
> long-run, rather than adding another language
> on the side.

I think that'd be great, but instead of having
CL on the side (or "Elisp on the side", if you
will) is it unthinkable to add a module _to CL_
so that _CL can deal with Elisp_ with its
usual toolchain?


[1] http://www.sbcl.org/
[2] http://lists.gnu.org/archive/html/help-gnu-emacs/2019-05/msg00008.html

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Why is Elisp slow?

[Stefan Monnier]

>     Why can't Elisp be compiled as well?

The question is meaningless because it's premise is wrong: Elisp can
be compiled as well.


        Stefan

Re: Why is Elisp slow?

[Emanuel Berg]

Stefan Monnier wrote:

>> Why can't Elisp be compiled as well?
>
> The question is meaningless because it's
> premise is wrong: Elisp can be compiled
> as well.

Well, someone is edgy all of a sudden?

But very well then, why can't Elisp be compiled
as fast as CL (or C), and then be used on top
of (C) Emacs?

Or if it can, why isn't it?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Why is Elisp slow?

[조성빈]

2019. 5. 5. 오전 7:56, Stefan Monnier <monnier@iro.umontreal.ca> 작성:

>>    Why can't Elisp be compiled as well?
> 
> The question is meaningless because it's premise is wrong: Elisp can
> be compiled as well.
> 
I’m pretty sure Elisp can’t compile down to raw machine code (like JIT) in current infrastructure. If you’re saying about compiling to bytecode, you’re losing his/her points.
Elisp bytecode is terribly slow :-(
> 
>        Stefan
> 

Re: Why is Elisp slow?

[Ergus]

Hi Stefan:

I mostly agree with you and I get your points but I have some comments about.

On May 4, 2019 4:03:28 PM GMT+02:00, Stefan Monnier <monnier@IRO.UMontreal.CA> wrote:
>> Inter-language interaction (which is one of the strong features in
>> Guile out of the box)
>
>That's fairly abstract (details matter: e.g. how are you going to use
>macros like `define-minor-mode` from non-Elisp languages?). 
>
>Also while I much prefer Scheme to Elisp (or Common-Lisp for that
>matter), I'm not looking forward to maintaining Emacs packages in two
>(and more) different languages, and other maintainers are probably
>imagining such maintenance nightmare with a lot more dread than I.
>
>So again: what are the advantages?
>
I totally agree with the complexity of mixing languages; but if we look around it is very difficult to attract new users and developers if they need to learn a new (old fashion) language only for Emacs (as I have been doing the last months). New programmers generations only know python and some C-like languages but specially OOP and maybe is the moment to think a bit in the future of the project more than in the past. Even if that implies taking some risks and make hard desitions. Just give a look how popular are sublime text or athom, that compared to Emacs those are kid toys.

So at some point it will be needed to make this desitions because the old developers will not live forever. I see the beauty of Lisp, but I am talking more about a human issue here, that is more critical for Emacs right now and for the future.

>From that point of view, moving Emacs to a Common-Lisp implementation
>would make more sense, since there is a chance we can *replace* Elisp
>with Common-Lisp in the long-run, rather than adding another language
>on
>the side.
>
IF we could do so (embed the CL compiler as a submodule within emacs) and this can improve performance (while reduce the amount of code we need to maintain, and the number of field experts we need to reach and attract ourself) I'll totally support such desition. It is not THE solution we need but it will reduce some of the important issues. And will make the project development more dynamic and sutainable. But if we can't take such desition right now is because we have few people to do the work and the new potential users/developers live in a different world with python java rust lua C++ Ruby but also saturated of cheaper (easier to use and that do work without learning curve) alternatives like notepad++, visual studio code, geany, eclipse, kdevelop... and so on. And some of them have put a lot of effort in improving the editing capabilities, programming functionalities and ergonomy more than implementing a pdf reader or games withing the tool. So they only do one thing but they do it right (or good enough for the user general needs) without initial configurations.

For the developers it is also easier to join to those projects because they are hosted on Github/gitlab with a more familiar workflow and interface, no copyright procedure, no mailing list.... and everything in the same please and integrate with a fork based workflow. You can see where I'm going right?

I don't say that all this is better or worst than what we have, it is just what the new generations understand, learn in the universities. Otherwise we are developing Emacs for us and the already existing users only. Ignoring where is the programming world now and where it is going.


>> and native compilation.
>
>That's irrelevant.  I think you mean "efficiency" or something like
>that.
>But that presumes that Elisp-on-Guile is indeed more efficient than
>Elisp-on-Emacs.  Is there actual evidence that this is the case?
>
>Also "efficiency" is broad: there's the execution time of some
>benchmarks, of course, but there's also the startup time, the VM size,
>the heap size, the GC interruptions, the time to load a big package
>(e.g. org-mode), ...
>
>Guile would hopefully win on some of those, but by how much?
>And how much worse does it make the others?
>
>> Plus some people taking care of the compiler/interpreter/virtual
>> machine parts and specialized in that specific section, which is good
>> because we need manpower.
>
>AFAIC that's the main goal, indeed.
>
>But last I checked Guile didn't have much manpower either.  And there's
>a good chance that tracking Guile development (after switching to
>Guile)

Vicious circle no-projects-interested means no users with issues which means no potential developers. Any way I was talking about guile because it was the official gnu language for extensions similar somehow to elisp and with a full platform more or less ready to use without license issues.

>will also require manpower on Emacs's side.  So it's not clear that it
>would turn out to be an advantage.
>
>
>        Stefan

-- 
Enviado desde mi dispositivo Android con K-9 Mail. Por favor, disculpa mi brevedad.

Re: Why is Elisp slow?

[조성빈]

The more I think about Guile Emacs and the Guile project itself, I start to think that it’s not a problem of manpower, but something more fundamental. Is it feasible (or even possible) to make two different high level languages to interoperate in the engine level?̊̈
For example, will it even make sense to have some Elisp macros in scheme land, or even lua land, python land, JS land?̊̈

Even if Emacs has limited success in these problem by e.g. transpiling JS to elisp or doing macro expansion at runtime by the Elisp engine, I would like a JSish API for emacs when writing packages, not a low level wrapper around Elisp APIs.
Same thing for Python, Lua, Scheme, etc...
Different languages introduce different concepts that just don’t match.
What would I do to use a Python module in JS?̊̈ Unless the language itself is designed to be easily interoperable with elisp by superseding concepts (such as Clojure superseding Java), I’m pretty sure many people won’t be satisfied in any kind of implementation. 

CL is one (and presumably the only) case that (at least looks like) supersedes Elisp in concepts. (Actually it would be something more like Elisp designed like that.) It’s a Lisp-2, it has macros, it uses S-exps, etc, etc. It also has the advantage that we can leverage the state-of-art CL compilers that produce insane speeds from such a dynamic language.

2019. 5. 5. 오전 9:43, Ergus <spacibba@aol.com> 작성:

> Hi Stefan:
> 
> I mostly agree with you and I get your points but I have some comments about.
> 
> On May 4, 2019 4:03:28 PM GMT+02:00, Stefan Monnier <monnier@IRO.UMontreal.CA> wrote:
>>> Inter-language interaction (which is one of the strong features in
>>> Guile out of the box)
>> 
>> That's fairly abstract (details matter: e.g. how are you going to use
>> macros like `define-minor-mode` from non-Elisp languages?). 
>> 
>> Also while I much prefer Scheme to Elisp (or Common-Lisp for that
>> matter), I'm not looking forward to maintaining Emacs packages in two
>> (and more) different languages, and other maintainers are probably
>> imagining such maintenance nightmare with a lot more dread than I.
>> 
>> So again: what are the advantages?
> 
> I totally agree with the complexity of mixing languages; but if we look around it is very difficult to attract new users and developers if they need to learn a new (old fashion) language only for Emacs (as I have been doing the last months). New programmers generations only know python and some C-like languages but specially OOP and maybe is the moment to think a bit in the future of the project more than in the past. Even if that implies taking some risks and make hard desitions. Just give a look how popular are sublime text or athom, that compared to Emacs those are kid toys.
> 
> So at some point it will be needed to make this desitions because the old developers will not live forever. I see the beauty of Lisp, but I am talking more about a human issue here, that is more critical for Emacs right now and for the future.
> 
>> From that point of view, moving Emacs to a Common-Lisp implementation
>> would make more sense, since there is a chance we can *replace* Elisp
>> with Common-Lisp in the long-run, rather than adding another language
>> on
>> the side.
>> 
> IF we could do so (embed the CL compiler as a submodule within emacs) and this can improve performance (while reduce the amount of code we need to maintain, and the number of field experts we need to reach and attract ourself) I'll totally support such desition. It is not THE solution we need but it will reduce some of the important issues. And will make the project development more dynamic and sutainable. But if we can't take such desition right now is because we have few people to do the work and the new potential users/developers live in a different world with python java rust lua C++ Ruby but also saturated of cheaper (easier to use and that do work without learning curve) alternatives like notepad++, visual studio code, geany, eclipse, kdevelop... and so on. And some of them have put a lot of effort in improving the editing capabilities, programming functionalities and ergonomy more than implementing a pdf reader or games withing the tool. So they only do one thing but they do it right (or good enough for the user general needs) without initial configurations.
> 
> For the developers it is also easier to join to those projects because they are hosted on Github/gitlab with a more familiar workflow and interface, no copyright procedure, no mailing list.... and everything in the same please and integrate with a fork based workflow. You can see where I'm going right?
> 
> I don't say that all this is better or worst than what we have, it is just what the new generations understand, learn in the universities. Otherwise we are developing Emacs for us and the already existing users only. Ignoring where is the programming world now and where it is going.
> 
> 
>>> and native compilation.
>> 
>> That's irrelevant.  I think you mean "efficiency" or something like
>> that.
>> But that presumes that Elisp-on-Guile is indeed more efficient than
>> Elisp-on-Emacs.  Is there actual evidence that this is the case?
>> 
>> Also "efficiency" is broad: there's the execution time of some
>> benchmarks, of course, but there's also the startup time, the VM size,
>> the heap size, the GC interruptions, the time to load a big package
>> (e.g. org-mode), ...
>> 
>> Guile would hopefully win on some of those, but by how much?
>> And how much worse does it make the others?
>> 
>>> Plus some people taking care of the compiler/interpreter/virtual
>>> machine parts and specialized in that specific section, which is good
>>> because we need manpower.
>> 
>> AFAIC that's the main goal, indeed.
>> 
>> But last I checked Guile didn't have much manpower either.  And there's
>> a good chance that tracking Guile development (after switching to
>> Guile)
> 
> Vicious circle no-projects-interested means no users with issues which means no potential developers. Any way I was talking about guile because it was the official gnu language for extensions similar somehow to elisp and with a full platform more or less ready to use without license issues.
> 
>> will also require manpower on Emacs's side.  So it's not clear that it
>> would turn out to be an advantage.
>> 
>> 
>>       Stefan
> 
> -- 
> Enviado desde mi dispositivo Android con K-9 Mail. Por favor, disculpa mi brevedad.
> 

Re: Why is Elisp slow?

[Stefan Monnier]

>>>    Why can't Elisp be compiled as well?
>> The question is meaningless because it's premise is wrong: Elisp can
>> be compiled as well.
> I’m pretty sure Elisp can’t compile down to raw machine code (like JIT) in
> current infrastructure.

Maybe there is no compiler that does that currently, but that's not
relevant to the question of whether there can be such a compiler.
[ And FWIW, you can probably compile Elisp to "raw" machine code with
  https://sourceforge.net/p/clocc/hg/ci/default/tree/src/cllib/elisp.lisp  ]

> Elisp bytecode is terribly slow :-(

There's been very little effort put into making it fast, indeed.
This said, when we compared it to Guile and some byte-code-based
Common-Lisp (I think it was ECL, but I can't remember) some years ago it
wasn't nearly as bad as I thought.

If someone's interested in speeding it up.  I'd recommend two
approaches:
- Either a JIT based on something like basic-block-versioning (BBV)
- Or a fast byte-code such as those generated by vmgen


        Stefan

Re: Why is Elisp slow?

[Paul W. Rankin]

On Sun, May 05 2019, Ergus wrote:
> I totally agree with the complexity of mixing languages; but if we 
> look around it is very difficult to attract new users and developers 
> if they need to learn a new (old fashion) language only for Emacs (as 
> I have been doing the last months). New programmers generations only 
> know python and some C-like languages but specially OOP and maybe is 
> the moment to think a bit in the future of the project more than in 
> the past. Even if that implies taking some risks and make hard 
> desitions. Just give a look how popular are sublime text or athom, 
> that compared to Emacs those are kid toys.
>
> So at some point it will be needed to make this desitions because the 
> old developers will not live forever. I see the beauty of Lisp, but I 
> am talking more about a human issue here, that is more critical for 
> Emacs right now and for the future.

I don't understand this sentiment. Is there some emergency in Emacs 
development that I'm unaware of? Development seems faster and stronger 
than it ever has in the decade or so I've been using Emacs -- there's a 
new major release every year or so, and with the addition of package.el 
there's the introduction of third-party package archives and the 
explosion that is MELPA.

The possibility of the current Emacs developers all dying out is not 
something anyone needs to worry about for at least another 30-40 years, 
and I'm pretty sure we (the human race) will have vastly different 
concerns by then. But even if we reach a point where there is no one 
left maintaining/developing/using Emacs, it's open source code... 
someone will discover it and if they find it useful or interesting, they 
will continue developing it. Open source code can never really die.

Where does this fear come from that an open source project will die if 
it doesn't keep changing?

> For the developers it is also easier to join to those projects because 
> they are hosted on Github/gitlab with a more familiar workflow and 
> interface, no copyright procedure, no mailing list.... and everything 
> in the same please and integrate with a fork based workflow. You can 
> see where I'm going right?

If a possible contributor has cloned a project repository to their own 
machine and has made some changes, the fork-based workflow requires that 
they: create an account at the origin project's GitHub or GitLab (or the 
project's GitLab instance), create a forked repository there, add the 
fork as a remote on their machine, push the changes, then open a pull 
request.

Once you use a git send-email workflow, this fork-based workflow will 
seem convoluted and unnecessarily centralised. All a contributor need do 
is clone the project repository, commit some changes then run:

    git send-email HEAD^

And send the email to the project's owner/mailing list. No account 
creation necessary. Check out:

    https://git-send-email.io/

Yes the copyright assignment procedure is a deterrent to contributing to 
Emacs/ELPA; this discussion is probably for another thread.

--
https://www.paulwrankin.com

Re: Why is Elisp slow?

[Stefan Monnier]

> the last months). New programmers generations only know python and some
> C-like languages but specially OOP and maybe is the moment to think a bit in
> the future of the project more than in the past. Even if that implies taking

This describes the situation from today just as well as that from 10
years ago, 20 years, and even 30 years ago.

Rewriting all Emacs packages in the language-du-jour every 10 years
doesn't sound very realistic.


        Stefan

Re: Why is Elisp slow?

[Óscar Fuentes]

"Paul W. Rankin" <hello@paulwrankin.com> writes:

>> For the developers it is also easier to join to those projects
>> because they are hosted on Github/gitlab with a more familiar
>> workflow and interface, no copyright procedure, no mailing list....
>> and everything in the same please and integrate with a fork based
>> workflow. You can see where I'm going right?
>
> If a possible contributor has cloned a project repository to their own
> machine and has made some changes, the fork-based workflow requires
> that they: create an account at the origin project's GitHub or GitLab
> (or the project's GitLab instance),

This is an one-off task.

> create a forked repository there,

Same.

> add the fork as a remote on their machine,

Same.

> push the changes, then open a pull request.

Which, in my experience, takes half a minute with the Github UI and
possibly less with Magit's Github plugin.

> Once you use a git send-email workflow, this fork-based workflow will
> seem convoluted and unnecessarily centralised. All a contributor need
> do is clone the project repository, commit some changes then run:
>
>    git send-email HEAD^

You skipped over quite a few issues here. Plus, a branch is easy to
access, in case someone wishes to grab the changes, while fishing mails
on a mailing list requires more setup and constant attention.

> And send the email to the project's owner/mailing list. No account
> creation necessary. Check out:
>
>    https://git-send-email.io/

Yes, that explains the job on a detailed way, and you can see that it is
not as simple as you pretend. Plus, that's the setup for sending
changes, no mention about how to grab them from the ml.

Re: Why is Elisp slow?

[Ergus]

Hi Paul:

On Sun, May 05, 2019 at 03:25:59PM +1000, Paul W. Rankin wrote:
>
>
>I don't understand this sentiment. Is there some emergency in Emacs 
>development that I'm unaware of? Development seems faster and stronger 
>than it ever has in the decade or so I've been using Emacs -- there's 
>a new major release every year or so, and with the addition of 
>package.el there's the introduction of third-party package archives 
>and the explosion that is MELPA.
>
I agree with you in that development is still active. But the grown in
number of users and developers is pretty constant, while the number of
developers and computer users in the world has grow exponentially. But
also there are sections and full parts in the emacs code without expert
to maintain it right now.

Mane old developers has left and deep/infrastructure/architectural big
changes are every time more difficult to do and the tendency goes worst
with time (more and more code).

Just give a look to the rating of the other editors I
mentioned, and their git repository. In fact in the last year, the
articles recommending text editors for programmers rarely include Emacs
while vim is in all of them. Also in my work there are more
than 400 programmers and I am the only Emacs user. Same when I go to
conferences around. So this is a bad statistical signal. Without users
there are not potential contributor.

We don't have a technical problem because emacs is more powerful than
most of the editors in the lists around, but we have a social problem.

Package.el same as modules are the best and more important steps emacs
has made in the last years. It has motivated and organized many things
that were very problematic before, which is very important, but it is
something the GNU/Linux distros figured out like 25 years ago
(aptitude, yast, pacman, synaptic), so we were (and are) late (not
wrong, just late, delayed in time for the decisions).

Just looking at the number of code reviewers we have, it is a reason to
worry. We have much less developers/contributors in emacs than what vim
has these days, and at the same time we have more than 2 times more code
to maintain with more languages and functionalities. 

>
>The possibility of the current Emacs developers all dying out is not 
>something anyone needs to worry about for at least another 30-40 
>years, and I'm pretty sure we (the human race) will have vastly 
>different concerns by then.

Not only dying, but also leaving, getting busy with the years, or
keeping old standards and paradigms, workflows, tools, concerns... and
this is reflected automatically in the project. Emacs needs new blood,
new people with new needs, ideas and concerns.

>But even if we reach a point where there 
>is no one left maintaining/developing/using Emacs, it's open source 
>code... someone will discover it and if they find it useful or 
>interesting, they will continue developing it. Open source code can 
>never really die.
>

The idea (and our reponsability) is not to reach there if we can avoid
that. Otherwise we failed as maintainers/developers. 

>
>Where does this fear come from that an open source project will die if 
>it doesn't keep changing?
>

Is not keep changing, is keep usefulness for its main purpose. What
emacs does as main purpose "edit text" will always gonna be useful. But,
are we being "competitive" in this field? The emacs infrastructure and
philosophy has all the needed for that (and much much muuuuuch more),
but are we using it properly for that purpose?

I think it is the best infrastructure to edit text, but right now we
are not providing the best editor (which should be the main goal). And
we add more and more and more functionalities and code. It is common
to heard that "emacs is a good OS, it only lacks a good editor."
which makes me very angry, but from the current editors standards
point of view it is right.

We don't need to keed changing, we need to do the right changes for
the project long path. FOr example: provide interaction with other
systems, languajes, optimize the compiler or rely in some other one
that already do it well and we trust in, cleanup code and portions of
code not fundamental for the emacs functionality or move them to
elpa/melpa. Standarize a bit more the bindings and function names.

For example the performance issues many people complain about
frequently is just a consequence of the extreme generalization in
functionalities and overheads in the infrastructure added for doing
"everything", but many of those things we barely do well in 2019
standards (some of then were added, and abandoned), so most users use
something different for that (like web browsing, reading PDF, play
tetris, play music, create buffers with buttons, have duality for TUI
and GUI in the same program, support very old architectures that are
inexistent these days) And all this is more and more and more code to
maintain, and more fields of expertise needed in our domain.

Same happens with development, we have files with more than 33000 lines
that we need to maintain, and any change in it is veeeeery complicated
and complex, but most of the code is to give support to some
functionalities very rarely used in practice.

And the infrastructure itself is poorly documented and complex
(specially the C interaction with Lisp from within emacs files.)
Thanks Eli, Stefan and the others who takes care of new users like me,
but that method to introduce developers is not scalable and makes them
to spend a lot of time.

>
>
>If a possible contributor has cloned a project repository to their own 
>machine and has made some changes, the fork-based workflow requires 
>that they: create an account at the origin project's GitHub or GitLab 
>(or the project's GitLab instance), create a forked repository there, 
>add the fork as a remote on their machine, push the changes, then open 
>a pull request.
>
>Once you use a git send-email workflow, this fork-based workflow will 
>seem convoluted and unnecessarily centralised. All a contributor need 
>do is clone the project repository, commit some changes then run:
>
>   git send-email HEAD^
>
>And send the email to the project's owner/mailing list. No account 
>creation necessary. Check out:
>
>   https://git-send-email.io/
>
>Yes the copyright assignment procedure is a deterrent to contributing 
>to Emacs/ELPA; this discussion is probably for another thread.
>
>--
>https://www.paulwrankin.com
>

This is not better than having a fork-merge request workflow. But also
forces to handmake a lot of the steps the new interfaces do for you like
reading the patch and make comments, follow those comments and reply to
specific parts of them, make then visible in the web, automatic
test/benchmark the functionalities, changes, close the requests or deny
them, give access for all that to the users in the future. Group people
in different forks working in the same functionality, bug, issue,
extension; test the code in the fork head automatically before merging
it in master. Is not impossible to do all this, it is just more manual,
time consumming and unfamiliar for 90% of the developers I know.

But also it is somehow dangerous, because actually everybody writes in
the master branch, while in my opinion only the managers (Eli, Stefan,
Alan...) should write there and the rest (me) should write only in the
forks and make pull requests (similar to what we do with branches, but
safer).

Now for the project it looks like:

bug: bugtracker
developement: savannah
developer interaction: mailing list
user interaction: reddit + help mailing list
documentation: within emacs
project main page: gnu.org
contribution: copyright process+mailing list+savannah
questions: another mailing list
packages: elpa
more packages: melpa
user hacks and advices: emacswiki

Re: Why is Elisp slow?

[Stefan Monnier]

> But very well then, why can't Elisp be compiled
> as fast as CL (or C), and then be used on top
> of (C) Emacs?

Same meaningless question.

> Or if it can, why isn't it?

That's the valid question.
AFAIK, the answer is that nobody was motivated enough to do it.


        Stefan

Re: Why is Elisp slow?

[Stefan Monnier]

> The more I think about Guile Emacs and the Guile project itself, I start to
> think that it’s not a problem of manpower, but something more
> fundamental.

Quite right.

> Is it feasible (or even possible) to make two different high
> level languages to interoperate in the engine level?̊̈

Language interoperation is very difficult, and even more so if you want
both languages to be usable "equally" (as opposed to a high-level
language with FFI bindings, say).  The .NET platform aims to do just
that, and note that it's fairly complex and the most successful
languages there were specifically designed/tweaked for that platform.

> For example, will it even make sense to have some Elisp macros in scheme

Interoperation between "similar" languages like Elisp and Scheme can
probably be made to be tolerable (e.g. I guess most Emacs-specific
special forms of Elisp, such as `save-excursion`, could be provided on
the Scheme side without too much trouble (famous last words)), but
macros seem pretty hard to handle in general (i.e. will require manual
work on a macro-by-macro case).

> land, or even lua land, python land, JS land?̊̈

For these, macros aren't the only problem, since there's also the
problem of the different kinds of datatypes used.  What would JS objects
look like in Elisp, and what would Elisp objects look like in JS?

> I would like a JSish API for emacs when writing packages, not a low
> level wrapper around Elisp APIs.

Indeed.

> What would I do to use a Python module in JS?̊̈ Unless the language itself is
> designed to be easily interoperable with elisp by superseding concepts (such
> as Clojure superseding Java), I’m pretty sure many people won’t be satisfied
> in any kind of implementation. 

Exactly.


        Stefan

Re: Why is Elisp slow?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 477 bytes --]

On Sun, May 05, 2019 at 01:19:08AM +0200, Emanuel Berg wrote:
> Stefan Monnier wrote:
> 
> >> Why can't Elisp be compiled as well?
> >
> > The question is meaningless because it's
> > premise is wrong: Elisp can be compiled
> > as well.
> 
> Well, someone is edgy all of a sudden?

Sorry to put it somewhat bluntly, Emanuel, but my impression
is that you tend to talk much and listen little.

This makes communication... somewhat difficult.

Cheers
-- tomás

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Why is Elisp slow?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 553 bytes --]

On Sun, May 05, 2019 at 03:46:01PM +0200, Ergus wrote:
> Hi Paul:
> 
> On Sun, May 05, 2019 at 03:25:59PM +1000, Paul W. Rankin wrote:

[...]

> I agree with you in that development is still active. But the grown in
> number of users and developers is pretty constant, while the number of
> developers and computer users in the world has grow exponentially [...]

Exponential growth is a sign of immaturity. Only the beginning of the
logistic curve [1] looks exponential.

Cheers

[1] https://en.wikipedia.org/wiki/Logistic_curve
-- t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Why is Elisp slow?

[Tadeus Prastowo]

On Sun, May 5, 2019 at 5:51 PM Stefan Monnier
<monnier@iro.umontreal.ca> wrote:> Language interoperation is very
difficult, and even more so if you want
> both languages to be usable "equally" (as opposed to a high-level
> language with FFI bindings, say).  The .NET platform aims to do just
> that, and note that it's fairly complex and the most successful
> languages there were specifically designed/tweaked for that platform.

Out of curiosity, do you mind to elaborate on how it is to be usable
"equally"?  (I think the best way to understand it is by studying the
.NET platform, but maybe you could explain it better).

Thank you very much.

>         Stefan

-- 
Best regards,
Tadeus

Re: Why is Elisp slow?

[Emanuel Berg]

tomas wrote:

>>>> Why can't Elisp be compiled as well?
>>>
>>> The question is meaningless because it's
>>> premise is wrong: Elisp can be compiled
>>> as well.
>> 
>> Well, someone is edgy all of a sudden?
>
> Sorry to put it somewhat bluntly, Emanuel,
> but my impression is that you tend to talk
> much and listen little.

Well, maybe people get the idea I listen
proportionally poorly, just because my talk
speaks for itself?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Why is Elisp slow?

[조성빈]

나의 iPhone에서 보냄

2019. 5. 6. 오후 4:33, Tadeus Prastowo <tadeus.prastowo@unitn.it> 작성:

> On Sun, May 5, 2019 at 5:51 PM Stefan Monnier
> <monnier@iro.umontreal.ca> wrote:> Language interoperation is very
> difficult, and even more so if you want
>> both languages to be usable "equally" (as opposed to a high-level
>> language with FFI bindings, say).  The .NET platform aims to do just
>> that, and note that it's fairly complex and the most successful
>> languages there were specifically designed/tweaked for that platform.
> 
> Out of curiosity, do you mind to elaborate on how it is to be usable
> "equally"?  (I think the best way to understand it is by studying the
> .NET platform, but maybe you could explain it better).

I would say as ‘to do something in a way that is considered idiomatic in the language’.
For example, some APIs (in general) makes sense when they use C++ templates or Rust `impl` blocks; it doesn’t make sense to enforce type unsafety.
Some APIs makes sense when they are provided as Lisp macros or babel macros; it doesn’t make sense to enforce boilerplate code.

Have you ever done FFI between two high level languages, e.g. Swift and Rust? It’s a real pain, because the interfacing code can’t use any of the language features as they should be defined by their common dominators: the SysV ABI.
You can’t use generics, tuples, lambdas, etc... even though both languages have them.

It’s well, a somewhat similar experience.

> Thank you very much.
> 
>>        Stefan
> 
> -- 
> Best regards,
> Tadeus
> 

Re: Why is Elisp slow?

[Tadeus Prastowo]

On Mon, May 6, 2019 at 11:03 AM 조성빈 <pcr910303@icloud.com> wrote:
>
> 2019. 5. 6. 오후 4:33, Tadeus Prastowo <tadeus.prastowo@unitn.it> 작성:
>
> > On Sun, May 5, 2019 at 5:51 PM Stefan Monnier
> > <monnier@iro.umontreal.ca> wrote:> Language interoperation is very
> > difficult, and even more so if you want
> >> both languages to be usable "equally" (as opposed to a high-level
> >> language with FFI bindings, say).  The .NET platform aims to do just
> >> that, and note that it's fairly complex and the most successful
> >> languages there were specifically designed/tweaked for that platform.
> >
> > Out of curiosity, do you mind to elaborate on how it is to be usable
> > "equally"?  (I think the best way to understand it is by studying the
> > .NET platform, but maybe you could explain it better).
>
> I would say as ‘to do something in a way that is considered idiomatic in the language’.
> For example, some APIs (in general) makes sense when they use C++ templates or Rust `impl` blocks; it doesn’t make sense to enforce type unsafety.
> Some APIs makes sense when they are provided as Lisp macros or babel macros; it doesn’t make sense to enforce boilerplate code.
>
> Have you ever done FFI between two high level languages, e.g. Swift and Rust? It’s a real pain, because the interfacing code can’t use any of the language features as they should be defined by their common dominators: the SysV ABI.
> You can’t use generics, tuples, lambdas, etc... even though both languages have them.
>
> It’s well, a somewhat similar experience.

Okay, I got it now.  In that case, the difficulty of using both
languages "equally" is a natural problem that is found also in, for
example, the English vs. Chinese vs. other natural language vocabulary
mismatches.

Thank you very much for taking the time to explain the things through.

-- 
Best regards,
Tadeus

Re: Why is Elisp slow?

[Ergus]

On Sun, May 05, 2019 at 11:50:16AM -0400, Stefan Monnier wrote:
>> The more I think about Guile Emacs and the Guile project itself, I start to
>> think that it???s not a problem of manpower, but something more
>> fundamental.
>
>Quite right.
>
>> Is it feasible (or even possible) to make two different high
>> level languages to interoperate in the engine level?????
>
>Language interoperation is very difficult, and even more so if you want
>both languages to be usable "equally" (as opposed to a high-level
>language with FFI bindings, say).  The .NET platform aims to do just
>that, and note that it's fairly complex and the most successful
>languages there were specifically designed/tweaked for that platform.
>
It looks more complex because the starting point is a High level
language as Lisp so any port will need many data types conversion and
copies with the corresponding overheads (and ugly code).

All the infrastructure is based in the Lisp Objects, types and data
structures and functions. But IF the base API were designed with basic C
types (int, doubles, char*) and structs, and language agnostic, the
interface with Lua, Python or Ruby will be pretty simple (even Fortran,
or anything else).

That's what vim did and they now support many languages for extending
and create plugins. (I know it is a different system and I am not saying
we have to move in that direction right now.)

But, instead of create lisp object from C the only we need it to create
an interface for the C functions that need access from Lisp and use them
in the interactive or high level lisp specific functions. But the low
level functions will be all exposed in the C library and we could access
then even with dlopen in some cases.

Of course I am not saying that we should develop code inside emacs core
with all the different languages or change language every 10 years. For
me it is fine Elisp (or common lisp even better if that reduces the
complexity of maintaining our on compiler and potentially improves
performance). But to give the user the possibility to create plugins; a
C API is much more useful and simple. And if we want to port some of
those to elisp will be also easier. I specially don't share the choice
of developing core parts (API) in Lisp directly because having them in C
is faster, simpler, more accessible from outside if needed and usually
gives less overheads to the starting time, and GC.

We already have the API with the modules, but if you see most of the
complexity comes from creating and converting lisp types to something
else and do callbacks to lisp functions, and parsing the arguments.

Defun or any other macro could be emulated if we have to, but again,
that's just a complexity of making everything Lisp centric instead of C
centric. 

Bringing a C api and use Lisp for high level or interactive functions
reduces overheads, simplify the C code we have and enables a more
modular design. But I know that try to do that now is not realistic with
the amount of code and people we have. So that's why I was talking about
Guile, but after all it seems not to be realistic either.

So whats the alternative? Keep our interpreter as is and deal with few
developers (and decreasing) and performance issues and a lot of code to
maintain?




>> For example, will it even make sense to have some Elisp macros in scheme
>
>Interoperation between "similar" languages like Elisp and Scheme can
>probably be made to be tolerable (e.g. I guess most Emacs-specific
>special forms of Elisp, such as `save-excursion`, could be provided on
>the Scheme side without too much trouble (famous last words)), but
>macros seem pretty hard to handle in general (i.e. will require manual
>work on a macro-by-macro case).
>
>> land, or even lua land, python land, JS land?????
>
>For these, macros aren't the only problem, since there's also the
>problem of the different kinds of datatypes used.  What would JS objects
>look like in Elisp, and what would Elisp objects look like in JS?
>
>> I would like a JSish API for emacs when writing packages, not a low
>> level wrapper around Elisp APIs.
>
>Indeed.
>
>> What would I do to use a Python module in JS????? Unless the language itself is
>> designed to be easily interoperable with elisp by superseding concepts (such
>> as Clojure superseding Java), I???m pretty sure many people won???t be satisfied
>> in any kind of implementation.
>
>Exactly.
>
>
>        Stefan
>
>

Re: Why is Elisp slow?

[Stefan Monnier]

> That's what vim did and they now support many languages for extending
> and create plugins. (I know it is a different system and I am not saying
> we have to move in that direction right now.)

Yes, it's a very different design.  Elisp was designed to be the
*implementation* language of Emacs (the use of C for some of the
implementation was a necessary evil given the lack of wide availability
of Lisp), rather than just an *extension* language.


        Stefan

Re: Why is Elisp slow?

[조성빈]

(As I am not currently familiar with how the Emacs codebase is structured) How does Emacs do FFI currently? If I recall correctly aren’t there ways to call C functions in elisp? Would it be hard to expose C functions (by compiling to the SysV ABI) and calling them in elisp (to write plugins in another language)? If I understand the current architecture of Vim correctly, Vim is also doing that; it’s impossible to not use VimScript to make plugins, glue code is essential.

> 2019. 5. 6. 오후 9:58, Ergus <spacibba@aol.com> 작성:
> 
> On Sun, May 05, 2019 at 11:50:16AM -0400, Stefan Monnier wrote:
>>> The more I think about Guile Emacs and the Guile project itself, I start to
>>> think that it???s not a problem of manpower, but something more
>>> fundamental.
>> 
>> Quite right.
>> 
>>> Is it feasible (or even possible) to make two different high
>>> level languages to interoperate in the engine level?????
>> 
>> Language interoperation is very difficult, and even more so if you want
>> both languages to be usable "equally" (as opposed to a high-level
>> language with FFI bindings, say).  The .NET platform aims to do just
>> that, and note that it's fairly complex and the most successful
>> languages there were specifically designed/tweaked for that platform.
>> 
> It looks more complex because the starting point is a High level
> language as Lisp so any port will need many data types conversion and
> copies with the corresponding overheads (and ugly code).
> 
> All the infrastructure is based in the Lisp Objects, types and data
> structures and functions. But IF the base API were designed with basic C
> types (int, doubles, char*) and structs, and language agnostic, the
> interface with Lua, Python or Ruby will be pretty simple (even Fortran,
> or anything else).
> 
> That's what vim did and they now support many languages for extending
> and create plugins. (I know it is a different system and I am not saying
> we have to move in that direction right now.)
> 
> But, instead of create lisp object from C the only we need it to create
> an interface for the C functions that need access from Lisp and use them
> in the interactive or high level lisp specific functions. But the low
> level functions will be all exposed in the C library and we could access
> then even with dlopen in some cases.
> 
> Of course I am not saying that we should develop code inside emacs core
> with all the different languages or change language every 10 years. For
> me it is fine Elisp (or common lisp even better if that reduces the
> complexity of maintaining our on compiler and potentially improves
> performance). But to give the user the possibility to create plugins; a
> C API is much more useful and simple. And if we want to port some of
> those to elisp will be also easier. I specially don't share the choice
> of developing core parts (API) in Lisp directly because having them in C
> is faster, simpler, more accessible from outside if needed and usually
> gives less overheads to the starting time, and GC.
> 
> We already have the API with the modules, but if you see most of the
> complexity comes from creating and converting lisp types to something
> else and do callbacks to lisp functions, and parsing the arguments.
> 
> Defun or any other macro could be emulated if we have to, but again,
> that's just a complexity of making everything Lisp centric instead of C
> centric. 
> Bringing a C api and use Lisp for high level or interactive functions
> reduces overheads, simplify the C code we have and enables a more
> modular design. But I know that try to do that now is not realistic with
> the amount of code and people we have. So that's why I was talking about
> Guile, but after all it seems not to be realistic either.
> 
> So whats the alternative? Keep our interpreter as is and deal with few
> developers (and decreasing) and performance issues and a lot of code to
> maintain?
> 
> 
> 
> 
>>> For example, will it even make sense to have some Elisp macros in scheme
>> 
>> Interoperation between "similar" languages like Elisp and Scheme can
>> probably be made to be tolerable (e.g. I guess most Emacs-specific
>> special forms of Elisp, such as `save-excursion`, could be provided on
>> the Scheme side without too much trouble (famous last words)), but
>> macros seem pretty hard to handle in general (i.e. will require manual
>> work on a macro-by-macro case).
>> 
>>> land, or even lua land, python land, JS land?????
>> 
>> For these, macros aren't the only problem, since there's also the
>> problem of the different kinds of datatypes used.  What would JS objects
>> look like in Elisp, and what would Elisp objects look like in JS?
>> 
>>> I would like a JSish API for emacs when writing packages, not a low
>>> level wrapper around Elisp APIs.
>> 
>> Indeed.
>> 
>>> What would I do to use a Python module in JS????? Unless the language itself is
>>> designed to be easily interoperable with elisp by superseding concepts (such
>>> as Clojure superseding Java), I???m pretty sure many people won???t be satisfied
>>> in any kind of implementation.
>> 
>> Exactly.
>> 
>> 
>>       Stefan
>> 
>> 
> 

Re: Why is Elisp slow?

[Óscar Fuentes]

Ergus <spacibba@aol.com> writes:

> Bringing a C api and use Lisp for high level or interactive functions
> reduces overheads, simplify the C code we have and enables a more
> modular design. But I know that try to do that now is not realistic with
> the amount of code and people we have. So that's why I was talking about
> Guile, but after all it seems not to be realistic either.
>
> So whats the alternative? Keep our interpreter as is and deal with few
> developers (and decreasing) and performance issues and a lot of code to
> maintain?

Emacs doesn't suffer from scarcity of Elisp programmers. Emacs lacks C
programmers.

Maybe it is related to how easy it is to get something working on each
language. Elisp has a huge advantage over C on that aspect, not only
because Elisp is more simple and way more expressive, but also because
of the REPL (no edit-compile-run-crash hamster wheel) and because Emacs
is, in practice, a Lisp machine with the associated advantages.

Elisp is a hacker's dream. Not so C.

Re: Why is Elisp slow?

[조성빈]

> 2019. 5. 6. 오후 10:33, Óscar Fuentes <ofv@wanadoo.es> 작성:
> 
> Ergus <spacibba@aol.com> writes:
> 
>> Bringing a C api and use Lisp for high level or interactive functions
>> reduces overheads, simplify the C code we have and enables a more
>> modular design. But I know that try to do that now is not realistic with
>> the amount of code and people we have. So that's why I was talking about
>> Guile, but after all it seems not to be realistic either.
>> 
>> So whats the alternative? Keep our interpreter as is and deal with few
>> developers (and decreasing) and performance issues and a lot of code to
>> maintain?
> 
> Emacs doesn't suffer from scarcity of Elisp programmers. Emacs lacks C
> programmers.

1. Arguably this is because Elisp *is* the extension language of Emacs.
2. Arguable Emacs suffers from both. :-( I have yet to find a good flow (a static type checking variant of JS from facebook) mode. I have yet to find a good debugging package (dap-mode comes close). I have yet to find a good <insert arbitrary package here>. Most Emacs packages have an equivalent one (if possible0 in Atom, VSCode, etc… but there are *lots* of packages that lack in Melpa. 
3. 

> Maybe it is related to how easy it is to get something working on each
> language. Elisp has a huge advantage over C on that aspect, not only
> because Elisp is more simple and way more expressive, but also because
> of the REPL (no edit-compile-run-crash hamster wheel) and because Emacs
> is, in practice, a Lisp machine with the associated advantages.

Also, um… comparing Elisp with C for ‘easy’?̊̈ Maybe comparing with JS, Python, Scheme, CL would be better, and Elisp isn’t always superior. Elisp is more expressive and allows better abstractions than most ALGOL-derived languages, but esoteric function names and scoping rules that default to dynamic scoping, etc… CL is superior than Elisp in almost all viewpoints. Scheme also has superior points to Elisp.

I wouldn’t say that Elisp is the selling point of Emacs; why did Atom succeed much more than Emacs?̊̈
The reason Emacs succeeded is that Emacs allows tweaking the editor to the extreme; Atom was successful for similar reasons albeit using JS instead of Elisp.

> Elisp is a hacker's dream. Not so C.

Lisp was, and is a hacker’s language.
C++, Java is a corporate language. 
C was, and is a hacker’s language.

Re: Why is Elisp slow?

[Ergus]

On Mon, May 06, 2019 at 09:08:03AM -0400, Stefan Monnier wrote:
>> That's what vim did and they now support many languages for extending
>> and create plugins. (I know it is a different system and I am not saying
>> we have to move in that direction right now.)
>
>Yes, it's a very different design.  Elisp was designed to be the
>*implementation* language of Emacs (the use of C for some of the
>implementation was a necessary evil given the lack of wide availability
>of Lisp), rather than just an *extension* language.
>
>
>        Stefan
>
Yes, but then we need to put much more emphasis (and work) improving and
keep good performance in the compiler (which is a much more complex task
than maintaining the editor, and where very few changes are made these
days). Apply the new tools and techniques in compiler sciences and so on
requires a very high level of expertise. OR consider migrate to
something else that someone else maintain like SBCL or whatever the
managers consider better. (I know that only this proposal will open a
sacred war (as it did before (but the problems are the same)) but being
sincere is more valuable than being politically correct to have
progress)

Because many editors around can already do what emacs do (edit text, be
extensible, free, and portable), but they have faster interpreter and
language, better (more familiar or standardized) keybindings, bigger
communities, simpler (in spite of more limited or not) customization and
simpler code or "more pretty/intuitive" UI.

Lisp was the right option 40 years ago because it was probably the most
dynamic language and easier to understand than C for the computer users
that time. But now there are more dynamic and capable languages. And a
language interpreter is not so easy to maintain for a formal/big project
like emacs.

Summing up. In Spanish we say that whoever takes a lot of space, the
less he tightens up (bad translation: Quien mucho abarca poco aprieta).

IN MY OPINION, we should put more attention in the emacs editing
capabilities/functionalities/interface and rely the compiling
functionalities to a compiler we can (with limitations of course)
trust. Even if that means remove the E letter from our language. But
after that big change any compiler released in future will be easier to
use as a module and we could even give the option to the user to
chose.

(I know this sounds a bit heretic) but being a bit more DOTADIW will be
more scalable and maintainable now and in the future. And we can center
the manpower in update the gui, optimizations, cleanup
old/unused/unmaintained code, solve bugreports, standardize keybindings
and so on.

Re: Why is Elisp slow?

[Stefan Monnier]

I totally agree with the sentiment; we'd benefit from "out-sourcing" the
maintenance of the language implementation, but we can't just replace
Elisp with something else, so we need to find another well-maintained
Elisp implementation that's at least as good as that we have now.

AFAIK the options are:
- Keep what we have
- Move to Guile
- Move to CLISP
- Move to SBCL
CLISP is not actively maintained nowadays.
AFAIK SBCL is alive and kicking, but I'm not sure how well it works on
platforms other than GNU/Linux-on-AMD64.

Of course, Guile has the advantage that someone has already spent a fair
bit of time implementing support for Elisp, whereas for CLISP and SBCL
that would be extra work (Elisp is close to a subset of CL but not
quite).

Another approach would be to implement an Elisp-to-JS compiler and
then use one of the heavily-optimized JIT-compilers for JS.
Compiling Elisp to JS should be much easier than compiling to
native code.


        Stefan

Re: Why is Elisp slow?

[조성빈]

> 2019. 5. 7. 오전 2:25, Stefan Monnier <monnier@iro.umontreal.ca> 작성:
> 
> I totally agree with the sentiment; we'd benefit from "out-sourcing" the
> maintenance of the language implementation, but we can't just replace
> Elisp with something else, so we need to find another well-maintained
> Elisp implementation that's at least as good as that we have now.

I’m curious: How likely is this to happen? This would be truly great if happening :-) 

> AFAIK the options are:
> - Keep what we have
> - Move to Guile
> - Move to CLISP
> - Move to SBCL
> CLISP is not actively maintained nowadays.
> AFAIK SBCL is alive and kicking, but I'm not sure how well it works on
> platforms other than GNU/Linux-on-AMD64.

I can confirm that SBCL works great on at least these platforms:
* Arch Linux(GNU userspace) on AMD64 (my friend’s confirmations)
* macOS(Darwin) on AMD64 (I’m currently using macOS)
* Windows 10 on AMD64 (my another friend’s confirmations)
* Raspbian(GNU userspace) on ARM (at least when using SBCL about a year before…)

CCL is similar (actually provides more support on non-GNU platforms).

> Of course, Guile has the advantage that someone has already spent a fair
> bit of time implementing support for Elisp, whereas for CLISP and SBCL
> that would be extra work (Elisp is close to a subset of CL but not
> quite).

Would that extra work outweigh than implementing Guile’s language integration features?
Perhaps an insightful person may know?
May I ask what part of elisp makes implementing in CL hard?

> Another approach would be to implement an Elisp-to-JS compiler and
> then use one of the heavily-optimized JIT-compilers for JS.
> Compiling Elisp to JS should be much easier than compiling to
> native code.

If possible, this would be more than great as we would be able to use the *big* number of JS packages in npm registry out there.

> 
>        Stefan
> 

Re: Why is Elisp slow?

[Stefan Monnier]

> I’m curious: How likely is this to happen?

As long as noone works on it, I'd say 0% likelihood.

> I can confirm that SBCL works great on at least these platforms:
> * Arch Linux(GNU userspace) on AMD64 (my friend’s confirmations)
> * macOS(Darwin) on AMD64 (I’m currently using macOS)
> * Windows 10 on AMD64 (my another friend’s confirmations)
> * Raspbian(GNU userspace) on ARM (at least when using SBCL about a year before…)

[ IIUC of the 4 cases above, at most 2 run the same version, so we'd
  need to make sure the same Emacs version can be compiled against all
  of those versions.  No idea if it would impose a significant extra
  burden or not, but it's something to be considered.  Also the fact
  that the latest release doesn't work on all those platforms is rather
  worrying.  ]

>> Of course, Guile has the advantage that someone has already spent a fair
>> bit of time implementing support for Elisp, whereas for CLISP and SBCL
>> that would be extra work (Elisp is close to a subset of CL but not
>> quite).
> Would that extra work outweigh than implementing Guile’s language
> integration features?

No idea.

> May I ask what part of elisp makes implementing in CL hard?

Haven't thought too much about it, so I don't even know if it would be
hard (the elisp.lisp implementation I mentioned recently shows that
large parts can be done easily enough).
The obvious issue is buffer-local and terminal-local variables.

>> Another approach would be to implement an Elisp-to-JS compiler and
>> then use one of the heavily-optimized JIT-compilers for JS.
>> Compiling Elisp to JS should be much easier than compiling to
>> native code.
>
> If possible, this would be more than great as we would be able to use the
> *big* number of JS packages in npm registry out there.

Note that compiling to JS doesn't *directly* let you access random JS
data-structures and functions any more than implementing Elisp in C lets
you access random C functions and data-structures.
[ language-interoperation, again.  ]


        Stefan

Re: Why is Elisp slow?

[조성빈]

> 2019. 5. 7. 오전 3:08, Stefan Monnier <monnier@iro.umontreal.ca> 작성:
> 
>> I’m curious: How likely is this to happen?
> 
> As long as noone works on it, I'd say 0% likelihood.
> 
>> I can confirm that SBCL works great on at least these platforms:
>> * Arch Linux(GNU userspace) on AMD64 (my friend’s confirmations)
>> * macOS(Darwin) on AMD64 (I’m currently using macOS)
>> * Windows 10 on AMD64 (my another friend’s confirmations)
>> * Raspbian(GNU userspace) on ARM (at least when using SBCL about a year before…)
> 
> [ IIUC of the 4 cases above, at most 2 run the same version, so we'd
>  need to make sure the same Emacs version can be compiled against all
>  of those versions.  No idea if it would impose a significant extra
>  burden or not, but it's something to be considered.  Also the fact
>  that the latest release doesn't work on all those platforms is rather
>  worrying.  ]

Hmm…? I can’t understand :-(
Why can’t Emacs can include a specific version of SBCL’s source (e.g. as a git module) and compile them all together?̊̈
I’m pretty sure SBCL’s platform-specific code is self-contained.

>>> Of course, Guile has the advantage that someone has already spent a fair
>>> bit of time implementing support for Elisp, whereas for CLISP and SBCL
>>> that would be extra work (Elisp is close to a subset of CL but not
>>> quite).
>> Would that extra work outweigh than implementing Guile’s language
>> integration features?
> 
> No idea.
> 
>> May I ask what part of elisp makes implementing in CL hard?
> 
> Haven't thought too much about it, so I don't even know if it would be
> hard (the elisp.lisp implementation I mentioned recently shows that
> large parts can be done easily enough).
> The obvious issue is buffer-local and terminal-local variables.
> 
>>> Another approach would be to implement an Elisp-to-JS compiler and
>>> then use one of the heavily-optimized JIT-compilers for JS.
>>> Compiling Elisp to JS should be much easier than compiling to
>>> native code.
>> 
>> If possible, this would be more than great as we would be able to use the
>> *big* number of JS packages in npm registry out there.
> 
> Note that compiling to JS doesn't *directly* let you access random JS
> data-structures and functions any more than implementing Elisp in C lets
> you access random C functions and data-structures.
> [ language-interoperation, again.  ]

Ah, I was saying libraries in general; libraries like immutable.js or left-pad :-)

> 
>        Stefan

Re: Why is Elisp slow?

[Stefan Monnier]

>> [ IIUC of the 4 cases above, at most 2 run the same version, so we'd
>>  need to make sure the same Emacs version can be compiled against all
>>  of those versions.  No idea if it would impose a significant extra
>>  burden or not, but it's something to be considered.  Also the fact
>>  that the latest release doesn't work on all those platforms is rather
>>  worrying.  ]
>
> Hmm…? I can’t understand :-(
> Why can’t Emacs can include a specific version of SBCL’s source (e.g. as
> a git module) and compile them all together?̊̈

Exactly because the version that can run on ARM is not the same as the
one that can run under Windows, which is not the same as the one that
runs under AMD64.

Of course, maybe I'm confused by the table of available binaries (at
http://sbcl.org/platform-table.html), and in reality the latest version
works fine on all supported platforms.


        Stefan

Re: Why is Elisp slow?

[Ergus]

On May 6, 2019 8:47:12 PM GMT+02:00, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>>> [ IIUC of the 4 cases above, at most 2 run the same version, so we'd
>>>  need to make sure the same Emacs version can be compiled against
>all
>>>  of those versions.  No idea if it would impose a significant extra
>>>  burden or not, but it's something to be considered.  Also the fact
>>>  that the latest release doesn't work on all those platforms is
>rather
>>>  worrying.  ]
>>
>> Hmm…? I can’t understand :-(
>> Why can’t Emacs can include a specific version of SBCL’s source (e.g.
>as
>> a git module) and compile them all together?̊̈
>
>Exactly because the version that can run on ARM is not the same as the
>one that can run under Windows, which is not the same as the one that
>runs under AMD64.
>
>Of course, maybe I'm confused by the table of available binaries (at
>http://sbcl.org/platform-table.html), and in reality the latest version
>works fine on all supported platforms.
>
>
>        Stefan

What about just ask to sbcl maintainers about portability? On the other hand, if Emacs doesn't use extremely fancy stuff in the language we could just rely on the system's sbcl version (available in the distro), for gnu/linux systems that will be pretty fine and of course that's something that needs specific tests. But nothing compared to maintain a compiler. 

Stefan if you need to test sbcl in some specific (weird) architecture just send/tell me the tests and the architectured you need to test. (Except windows)  and I can do that.

Btw. Should we move this thread to the developers list? (and start the usual war)

Please don't go for JS stuff... Please... performance is terrible even with jit.

Re: Why is Elisp slow?

[조성빈]

> 2019. 5. 7. 오전 5:23, Ergus <spacibba@aol.com> 작성:
> 
> 
> 
> On May 6, 2019 8:47:12 PM GMT+02:00, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>>>> [ IIUC of the 4 cases above, at most 2 run the same version, so we'd
>>>> need to make sure the same Emacs version can be compiled against
>> all
>>>> of those versions.  No idea if it would impose a significant extra
>>>> burden or not, but it's something to be considered.  Also the fact
>>>> that the latest release doesn't work on all those platforms is
>> rather
>>>> worrying.  ]
>>> 
>>> Hmm…? I can’t understand :-(
>>> Why can’t Emacs can include a specific version of SBCL’s source (e.g.
>> as
>>> a git module) and compile them all together?̊̈
>> 
>> Exactly because the version that can run on ARM is not the same as the
>> one that can run under Windows, which is not the same as the one that
>> runs under AMD64.
>> 
>> Of course, maybe I'm confused by the table of available binaries (at
>> http://sbcl.org/platform-table.html), and in reality the latest version
>> works fine on all supported platforms.
>> 
>> 
>>       Stefan
> 
> What about just ask to sbcl maintainers about portability? On the other hand, if Emacs doesn't use extremely fancy stuff in the language we could just rely on the system's sbcl version (available in the distro), for gnu/linux systems that will be pretty fine and of course that's something that needs specific tests. But nothing compared to maintain a compiler. 
> 
> Stefan if you need to test sbcl in some specific (weird) architecture just send/tell me the tests and the architectured you need to test. (Except windows)  and I can do that.
> 
> Btw. Should we move this thread to the developers list? (and start the usual war)
> 
> Please don't go for JS stuff... Please... performance is terrible even with jit.
> 

Um… JS’s performance is pretty good, almost comparable with static languages like Java and C.
I remember a better benchmark, but I couldn’t find that, but look at this: https://attractivechaos.github.io/plb/ <https://attractivechaos.github.io/plb/>
Javascript V8 engine is almost always in about x2 ~ x3 of C.
Compare that to Elisp….uh…which is about comparable with python maybe?

Re: Why is Elisp slow?

[Óscar Fuentes]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I totally agree with the sentiment; we'd benefit from "out-sourcing" the
> maintenance of the language implementation, but we can't just replace
> Elisp with something else, so we need to find another well-maintained
> Elisp implementation that's at least as good as that we have now.
>
> AFAIK the options are:
> - Keep what we have
> - Move to Guile
> - Move to CLISP
> - Move to SBCL

- Pick an Elisp subset that can be compiled to efficient machine code
  and has a runtime model that is amenable to C. You retain much of the
  benefits of Elisp but gain fast execution and the possibility of using
  a miriad of libraries. Of course, do not replace Elisp with this, use
  it just when it matters.
  
There are several C-with-a-Lisp-coat implementations out there. I have
one (not public) and know that this method works.

> Another approach would be to implement an Elisp-to-JS compiler and
> then use one of the heavily-optimized JIT-compilers for JS.
> Compiling Elisp to JS should be much easier than compiling to
> native code.

Javascript's JITs seem fast because the interpreters are so slow. Those
JITs only provide near-C code efficiency only on selected cases.
Translating Elisp to JS does not inspire much confidence about the
efficiency of the resulting machine code, not to mention the required
work to interface with current Emacs C base and external libraries.

Re: Why is Elisp slow?

[Jean-Christophe Helary]

> On May 7, 2019, at 3:47, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
> Of course, maybe I'm confused by the table of available binaries (at
> http://sbcl.org/platform-table.html), and in reality the latest version
> works fine on all supported platforms.

It looks like the table is not up to date. Version 1.5.2 works fine on the latest version of macOS (intel). I installed it with brew.

Jean-Christophe Helary
-----------------------------------------------
http://mac4translators.blogspot.com @brandelune

Re: Why is Elisp slow?

[Ergus]

Hi again:

Sorry for my ignorance. Syntactically speaking which are the key differences between CL and Elisp you think may produce the bigger issues?


On May 6, 2019 8:47:12 PM GMT+02:00, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>>> [ IIUC of the 4 cases above, at most 2 run the same version, so we'd
>>>  need to make sure the same Emacs version can be compiled against
>all
>>>  of those versions.  No idea if it would impose a significant extra
>>>  burden or not, but it's something to be considered.  Also the fact
>>>  that the latest release doesn't work on all those platforms is
>rather
>>>  worrying.  ]
>>
>> Hmm…? I can’t understand :-(
>> Why can’t Emacs can include a specific version of SBCL’s source (e.g.
>as
>> a git module) and compile them all together?̊̈
>
>Exactly because the version that can run on ARM is not the same as the
>one that can run under Windows, which is not the same as the one that
>runs under AMD64.
>
>Of course, maybe I'm confused by the table of available binaries (at
>http://sbcl.org/platform-table.html), and in reality the latest version
>works fine on all supported platforms.
>
>
>        Stefan

-- 
Enviado desde mi dispositivo Android con K-9 Mail. Por favor, disculpa mi brevedad.

Re: Why is Elisp slow?

[Ergus]

Hi again:

Sorry for my ignorance. Syntactically speaking which are the key differences between CL and Elisp you think may produce the bigger issues?


On May 6, 2019 8:47:12 PM GMT+02:00, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>>> [ IIUC of the 4 cases above, at most 2 run the same version, so we'd
>>>  need to make sure the same Emacs version can be compiled against
>all
>>>  of those versions.  No idea if it would impose a significant extra
>>>  burden or not, but it's something to be considered.  Also the fact
>>>  that the latest release doesn't work on all those platforms is
>rather
>>>  worrying.  ]
>>
>> Hmm…? I can’t understand :-(
>> Why can’t Emacs can include a specific version of SBCL’s source (e.g.
>as
>> a git module) and compile them all together?̊̈
>
>Exactly because the version that can run on ARM is not the same as the
>one that can run under Windows, which is not the same as the one that
>runs under AMD64.
>
>Of course, maybe I'm confused by the table of available binaries (at
>http://sbcl.org/platform-table.html), and in reality the latest version
>works fine on all supported platforms.
>
>
>        Stefan

-- 
Enviado desde mi dispositivo Android con K-9 Mail. Por favor, disculpa mi brevedad.
-- 
Enviado desde mi dispositivo Android con K-9 Mail. Por favor, disculpa mi brevedad.

Re: Why is Elisp slow?

[Emanuel Berg]

Óscar Fuentes wrote:

> Emacs doesn't suffer from scarcity of Elisp
> programmers. Emacs lacks C programmers.

What more exactly would those
Emacs C programmers do, if they existed and
were "motivated"?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Why is Elisp slow?

[조성빈]

> 2019. 5. 7. 오전 5:51, Óscar Fuentes <ofv@wanadoo.es> 작성:
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
>> I totally agree with the sentiment; we'd benefit from "out-sourcing" the
>> maintenance of the language implementation, but we can't just replace
>> Elisp with something else, so we need to find another well-maintained
>> Elisp implementation that's at least as good as that we have now.
>> 
>> AFAIK the options are:
>> - Keep what we have
>> - Move to Guile
>> - Move to CLISP
>> - Move to SBCL
> 
> - Pick an Elisp subset that can be compiled to efficient machine code
>  and has a runtime model that is amenable to C. You retain much of the
>  benefits of Elisp but gain fast execution and the possibility of using
>  a miriad of libraries. Of course, do not replace Elisp with this, use
>  it just when it matters.
> 
> There are several C-with-a-Lisp-coat implementations out there. I have
> one (not public) and know that this method works.

Does this have a solution to interfacing the Elisp subset translated to C and usual Elisp?
It feels like there’s almost no gains compared to using Guile, except that it would be marginally faster...

>> Another approach would be to implement an Elisp-to-JS compiler and
>> then use one of the heavily-optimized JIT-compilers for JS.
>> Compiling Elisp to JS should be much easier than compiling to
>> native code.
> 
> Javascript's JITs seem fast because the interpreters are so slow. Those
> JITs only provide near-C code efficiency only on selected cases.
> Translating Elisp to JS does not inspire much confidence about the
> efficiency of the resulting machine code, not to mention the required
> work to interface with current Emacs C base and external libraries.

Yeah, and the slow interpreters are what Emacs has in itself.

Re: Why is Elisp slow?

[Ergus]

Hi all:

After reading the whole SBCL manual and the internals I have some
comments maybe useful if we decide to go in that direction. (If some of
the core developers starts with this I will be very pleased to work on
that, but I am too new in emacs development to start it myself alone.)

1) + Portability does not seems to be a big issue, they already support
 our architectures (and some extra) and they are putting a lot of effort
 on that. There is work to do, but in general, they already have what we
 need from Common Lisp.

2) + They have documented how to port SBCL to new architectures, and some
of the optimizations they implement, so, if we have a compiler
specialist on board and finally our developers choose not to go for
SBCL, then maybe some of those optimization could be considered for our
Elisp compiler.

3) - There is not C binding for SBCL. To extend it, there is not a
public-documented API and their approach is wrapping the C functions
from Lisp (FFI). I wrote in their mailing list and they redirected me to
the CFFI page.

    + The only advantage (I see) in this approach is that as CFFI has
    been ported to almost all common list compilers/interpreters; if we
    go in that direction for our C code we could potentially use any of
    those and jump from one to another at configuration time. So we
    don't have this problem (of being stock with a "bad" compiler)
    anymore.

    - The overhead produced by wrapping in the high level instead of the
      low level in my experience (python, ruby, Julia) is usually much
      higher than having C bindings. 

    - Writing wrappers in lisp for all our C functions exposed to Lisp I
      think is not an option if we don't create an automatic method. So
      if no one creates a plugin for sbcl to support Elisp, I really
      doubt the feasibility of using SBCL except if many of our current
      core developers agree on that and join effort for that. Else it
      will happen as with guile.

4) + With FFI as they implemented, it will be pretty easy to interface
and call C libraries and functions without needing a C wrapper to expose
it to lisp, so maybe an important part of our C code could be
substituted with Lisp (simpler and smaller) glue code. But also we will
be allowed to use C dynamic libraries directly without all the modules
complexity. (Somehow I feel that not everyone will like that if we
comment it in the developers mailing list :p ).



On Mon, May 06, 2019 at 02:47:12PM -0400, Stefan Monnier wrote:
>>> [ IIUC of the 4 cases above, at most 2 run the same version, so we'd
>>>  need to make sure the same Emacs version can be compiled against all
>>>  of those versions.  No idea if it would impose a significant extra
>>>  burden or not, but it's something to be considered.  Also the fact
>>>  that the latest release doesn't work on all those platforms is rather
>>>  worrying.  ]
>>
>> Hmm???? I can???t understand :-(
>> Why can???t Emacs can include a specific version of SBCL???s source (e.g. as
>> a git module) and compile them all together?????
>
>Exactly because the version that can run on ARM is not the same as the
>one that can run under Windows, which is not the same as the one that
>runs under AMD64.
>
>Of course, maybe I'm confused by the table of available binaries (at
>http://sbcl.org/platform-table.html), and in reality the latest version
>works fine on all supported platforms.
>
>
>        Stefan
>
>

Re: Why is Elisp slow?

[조성빈]

> 2019. 5. 7. 오후 7:49, Ergus <spacibba@aol.com> 작성:
> 
> Hi all:
> 
> After reading the whole SBCL manual and the internals I have some
> comments maybe useful if we decide to go in that direction. (If some of
> the core developers starts with this I will be very pleased to work on
> that, but I am too new in emacs development to start it myself alone.)
> 
> 1) + Portability does not seems to be a big issue, they already support
> our architectures (and some extra) and they are putting a lot of effort
> on that. There is work to do, but in general, they already have what we
> need from Common Lisp.
> 
> 2) + They have documented how to port SBCL to new architectures, and some
> of the optimizations they implement, so, if we have a compiler
> specialist on board and finally our developers choose not to go for
> SBCL, then maybe some of those optimization could be considered for our
> Elisp compiler.
> 
> 3) - There is not C binding for SBCL. To extend it, there is not a
> public-documented API and their approach is wrapping the C functions
> from Lisp (FFI). I wrote in their mailing list and they redirected me to
> the CFFI page.
> 
>   + The only advantage (I see) in this approach is that as CFFI has
>   been ported to almost all common list compilers/interpreters; if we
>   go in that direction for our C code we could potentially use any of
>   those and jump from one to another at configuration time. So we
>   don't have this problem (of being stock with a "bad" compiler)
>   anymore.
>   
>   - The overhead produced by wrapping in the high level instead of the
>     low level in my experience (python, ruby, Julia) is usually much
>     higher than having C bindings. 
>   - Writing wrappers in lisp for all our C functions exposed to Lisp I
>     think is not an option if we don't create an automatic method. So
>     if no one creates a plugin for sbcl to support Elisp, I really
>     doubt the feasibility of using SBCL except if many of our current
>     core developers agree on that and join effort for that. Else it
>     will happen as with guile.

There is an automatic method of generating C FFI bindings from header files.
See https://github.com/rpav/cl-autowrap <https://github.com/rpav/cl-autowrap> :-)
What I was thinking about using CL to support Elisp is to define a new
namespace for symbols (which, in CL terms, is a so-called ‘package’) named ‘elisp’.
All Cl-needed functions/macros are defined in the ‘cl’ package,
which means that you can call any cl function/macro by
prefixing ‘cl:’ in the ‘elisp’ package.
As CL and Elisp has many common things (such as the similarity of
many, many macros like ‘defun’ or ‘defvar’, etc… or that it is a Lisp-2),
we can implement elisp entirely inside CL itself.
No need for developing a plugin specially for SBCL; any CL implementation will do.
For functions defined in C, we can make or generate wrappers to get
passed-in-arguments and dynamically wrap values into a Lisp_Obj C struct
(AFAIU all lisp objects are represented as a struct of type info and some pointers,
 am I right?) and call the underlying function generated by cl-autowrap.

> 4) + With FFI as they implemented, it will be pretty easy to interface
> and call C libraries and functions without needing a C wrapper to expose
> it to lisp, so maybe an important part of our C code could be
> substituted with Lisp (simpler and smaller) glue code. But also we will
> be allowed to use C dynamic libraries directly without all the modules
> complexity. (Somehow I feel that not everyone will like that if we
> comment it in the developers mailing list :p ).
> 
> On Mon, May 06, 2019 at 02:47:12PM -0400, Stefan Monnier wrote:
>>>> [ IIUC of the 4 cases above, at most 2 run the same version, so we'd
>>>> need to make sure the same Emacs version can be compiled against all
>>>> of those versions.  No idea if it would impose a significant extra
>>>> burden or not, but it's something to be considered.  Also the fact
>>>> that the latest release doesn't work on all those platforms is rather
>>>> worrying.  ]
>>> 
>>> Hmm? I can't understand :-(
>>> Why can't Emacs can include a specific version of SBCL's source (e.g. as
>>> a git module) and compile them all together?
>> 
>> Exactly because the version that can run on ARM is not the same as the
>> one that can run under Windows, which is not the same as the one that
>> runs under AMD64.
>> 
>> Of course, maybe I'm confused by the table of available binaries (at
>> http://sbcl.org/platform-table.html), and in reality the latest version
>> works fine on all supported platforms.
>> 
>>       Stefan
>> 
> 

Re: Why is Elisp slow?

[Ergus]

On Tue, May 07, 2019 at 08:51:27PM +0900, ????????? wrote:
>
>
>> 2019. 5. 7. ?????? 7:49, Ergus <spacibba@aol.com> ??????:
>>
>> Hi all:
>>
>> After reading the whole SBCL manual and the internals I have some
>> comments maybe useful if we decide to go in that direction. (If some of
>> the core developers starts with this I will be very pleased to work on
>> that, but I am too new in emacs development to start it myself alone.)
>>
>> 1) + Portability does not seems to be a big issue, they already support
>> our architectures (and some extra) and they are putting a lot of effort
>> on that. There is work to do, but in general, they already have what we
>> need from Common Lisp.
>>
>> 2) + They have documented how to port SBCL to new architectures, and some
>> of the optimizations they implement, so, if we have a compiler
>> specialist on board and finally our developers choose not to go for
>> SBCL, then maybe some of those optimization could be considered for our
>> Elisp compiler.
>>
>> 3) - There is not C binding for SBCL. To extend it, there is not a
>> public-documented API and their approach is wrapping the C functions
>> from Lisp (FFI). I wrote in their mailing list and they redirected me to
>> the CFFI page.
>>
>>   + The only advantage (I see) in this approach is that as CFFI has
>>   been ported to almost all common list compilers/interpreters; if we
>>   go in that direction for our C code we could potentially use any of
>>   those and jump from one to another at configuration time. So we
>>   don't have this problem (of being stock with a "bad" compiler)
>>   anymore.
>>
>>   - The overhead produced by wrapping in the high level instead of the
>>     low level in my experience (python, ruby, Julia) is usually much
>>     higher than having C bindings.
>>   - Writing wrappers in lisp for all our C functions exposed to Lisp I
>>     think is not an option if we don't create an automatic method. So
>>     if no one creates a plugin for sbcl to support Elisp, I really
>>     doubt the feasibility of using SBCL except if many of our current
>>     core developers agree on that and join effort for that. Else it
>>     will happen as with guile.
>
>There is an automatic method of generating C FFI bindings from header files.
>See https://github.com/rpav/cl-autowrap <https://github.com/rpav/cl-autowrap> :-)
>What I was thinking about using CL to support Elisp is to define a new
>namespace for symbols (which, in CL terms, is a so-called ???package???) named ???elisp???.
>All Cl-needed functions/macros are defined in the ???cl??? package,
>which means that you can call any cl function/macro by
>prefixing ???cl:??? in the ???elisp??? package.

This is what I understand as a plugin for SBCL. Because we will need
some steroid for that :p 

>As CL and Elisp has many common things (such as the similarity of
>many, many macros like ???defun??? or ???defvar???, etc??? or that it is a Lisp-2),
>we can implement elisp entirely inside CL itself.
>No need for developing a plugin specially for SBCL; any CL implementation will do.
>For functions defined in C, we can make or generate wrappers to get
>passed-in-arguments and dynamically wrap values into a Lisp_Obj C struct
>(AFAIU all lisp objects are represented as a struct of type info and some pointers,
> am I right?) and call the underlying function generated by cl-autowrap.
>
This is so huge that I don't know where to start looking :(. What I was
looking was a representation of the types in SBCL, but I couldn't find
that.

AFAIK cl-autowrap is to wrap simple C functions with C types and
structs, but Elisp actually has it's own representation of lisp objects
and functions. And the exposed functions already have a unified function
format to store the pointer in the function's hash table. If we use a
wrapper on the hight level we will be packing and unpacking structs
(copies and more copies) constantly and wasting the advantage of having
a similar representation. Or we won't be allowed to pass Elisp object to
CL functions.

So the first to do should be to unify the Elisp types with the CL
representation as much as possible and check the differences we can't
solve implicitly to create an explicit conversion when needed. Because
with guile, after doing the hard work, we found that there was a problem
with strings representations (if I understood right what Stefan
explained)

We also need to find a method to create lisp objects from C to implement
DEFVAR_LISP, DEFSYM and similes in C if we don't want to reimplement big
portions of code (like the display engine for example). There is still
the problem with buffer local variable that we don't have a solution for
that yet either.

Then we can start checking the functions we have that are Elisp agnostic
and can be compiled with SBCL. We could also have another set of
functions we have implemented in C we will not need anymore and finally
there will be functions we need to wrap (like everything in editfns.c
for example)... But again... this is huge.
>
>> 4) + With FFI as they implemented, it will be pretty easy to interface
>> and call C libraries and functions without needing a C wrapper to expose
>> it to lisp, so maybe an important part of our C code could be
>> substituted with Lisp (simpler and smaller) glue code. But also we will
>> be allowed to use C dynamic libraries directly without all the modules
>> complexity. (Somehow I feel that not everyone will like that if we
>> comment it in the developers mailing list :p ).
>>
>> On Mon, May 06, 2019 at 02:47:12PM -0400, Stefan Monnier wrote:
>>>>> [ IIUC of the 4 cases above, at most 2 run the same version, so we'd
>>>>> need to make sure the same Emacs version can be compiled against all
>>>>> of those versions.  No idea if it would impose a significant extra
>>>>> burden or not, but it's something to be considered.  Also the fact
>>>>> that the latest release doesn't work on all those platforms is rather
>>>>> worrying.  ]
>>>>
>>>> Hmm? I can't understand :-(
>>>> Why can't Emacs can include a specific version of SBCL's source (e.g. as
>>>> a git module) and compile them all together?
>>>
>>> Exactly because the version that can run on ARM is not the same as the
>>> one that can run under Windows, which is not the same as the one that
>>> runs under AMD64.
>>>
>>> Of course, maybe I'm confused by the table of available binaries (at
>>> http://sbcl.org/platform-table.html), and in reality the latest version
>>> works fine on all supported platforms.
>>>
>>>       Stefan
>>>
>>
>

Re: Why is Elisp slow?

[Stefan Monnier]

>>   - Writing wrappers in lisp for all our C functions exposed to Lisp I

All those are defined with a "DEFUN" macro on the C side.
Whatever change is needed on this side can likely be made largely
mechanically, so I'm not worried.

Similarly, you'll need to rewrite all the functions/macros like CONSP,
SYMBOLP, FIXNUMP, XCAR, XCDR, make_fixnum, ...  Performance of those
is important.

> What I was thinking about using CL to support Elisp is to define a new
> namespace for symbols (which, in CL terms, is a so-called ‘package’)
> named ‘elisp’.

As I already mentioned, this already exists: elisp.lisp.


        Stefan

Re: Why is Elisp slow?

[조성빈]

> 2019. 5. 7. 오후 9:56, Stefan Monnier <monnier@IRO.UMontreal.CA> 작성:
> 
>>>  - Writing wrappers in lisp for all our C functions exposed to Lisp I
> 
> All those are defined with a "DEFUN" macro on the C side.
> Whatever change is needed on this side can likely be made largely
> mechanically, so I'm not worried.
> 
> Similarly, you'll need to rewrite all the functions/macros like CONSP,
> SYMBOLP, FIXNUMP, XCAR, XCDR, make_fixnum, ...  Performance of those
> is important.

Why would you not use the default CL’s defun, car, cdr, symbol-p, cons-p, etc, etc?

>> What I was thinking about using CL to support Elisp is to define a new
>> namespace for symbols (which, in CL terms, is a so-called ‘package’)
>> named ‘elisp’.
> 
> As I already mentioned, this already exists: elisp.lisp.
> 
> 
>        Stefan
> 

Re: Why is Elisp slow?

[Stefan Monnier]

>> Similarly, you'll need to rewrite all the functions/macros like CONSP,
>> SYMBOLP, FIXNUMP, XCAR, XCDR, make_fixnum, ...  Performance of those
>> is important.
>
> Why would you not use the default CL’s defun, car, cdr, symbol-p,
> cons-p, etc, etc?

I'm talking the work needed to adapt Emacs's C code, e.g:

    DEFUN ("get-buffer-window", Fget_buffer_window, Sget_buffer_window, 0, 2, 0,
           doc: /* Return a window currently displaying BUFFER-OR-NAME, or nil if none.
    BUFFER-OR-NAME may be a buffer or a buffer name and defaults to
    the current buffer.
    
    The optional argument ALL-FRAMES specifies the frames to consider:
    
    - t means consider all windows on all existing frames.
    
    - `visible' means consider all windows on all visible frames.
    
    - 0 (the number zero) means consider all windows on all visible
        and iconified frames.
    
    - A frame means consider all windows on that frame only.
    
    Any other value of ALL-FRAMES means consider all windows on the
    selected frame and no others.  */)
         (Lisp_Object buffer_or_name, Lisp_Object all_frames)
    {
      Lisp_Object buffer;
    
      if (NILP (buffer_or_name))
        buffer = Fcurrent_buffer ();
      else
        buffer = Fget_buffer (buffer_or_name);
    
      if (BUFFERP (buffer))
        return window_loop (GET_BUFFER_WINDOW, buffer, true, all_frames);
      else
        return Qnil;
    }


-- Stefan

Re: Why is Elisp slow?

[Ergus]

On Tue, May 07, 2019 at 09:04:31AM -0400, Stefan Monnier wrote:
>>> Similarly, you'll need to rewrite all the functions/macros like CONSP,
>>> SYMBOLP, FIXNUMP, XCAR, XCDR, make_fixnum, ...  Performance of those
>>> is important.
>>
>> Why would you not use the default CL???s defun, car, cdr, symbol-p,
>> cons-p, etc, etc?
>
>I'm talking the work needed to adapt Emacs's C code, e.g:
>
>    DEFUN ("get-buffer-window", Fget_buffer_window, Sget_buffer_window, 0, 2, 0,
>           doc: /* Return a window currently displaying BUFFER-OR-NAME, or nil if none.
>    BUFFER-OR-NAME may be a buffer or a buffer name and defaults to
>    the current buffer.
>
>    The optional argument ALL-FRAMES specifies the frames to consider:
>
>    - t means consider all windows on all existing frames.
>
>    - `visible' means consider all windows on all visible frames.
>
>    - 0 (the number zero) means consider all windows on all visible
>        and iconified frames.
>
>    - A frame means consider all windows on that frame only.
>
>    Any other value of ALL-FRAMES means consider all windows on the
>    selected frame and no others.  */)
>         (Lisp_Object buffer_or_name, Lisp_Object all_frames)
>    {
>      Lisp_Object buffer;
>
>      if (NILP (buffer_or_name))
>        buffer = Fcurrent_buffer ();
>      else
>        buffer = Fget_buffer (buffer_or_name);
>
>      if (BUFFERP (buffer))
>        return window_loop (GET_BUFFER_WINDOW, buffer, true, all_frames);
>      else
>        return Qnil;
>    }
>
>
>-- Stefan
>
That's why I was wondering about the C binds and the C types
representations in SBCL more than the lisp part of the implementation.

Re: Why is Elisp slow?

[조성빈]

Ahh… I now understood what’s the problem. :-(

> 2019. 5. 7. 오후 10:04, Stefan Monnier <monnier@IRO.UMontreal.CA> 작성:
> 
>>> Similarly, you'll need to rewrite all the functions/macros like CONSP,
>>> SYMBOLP, FIXNUMP, XCAR, XCDR, make_fixnum, ...  Performance of those
>>> is important.
>> 
>> Why would you not use the default CL’s defun, car, cdr, symbol-p,
>> cons-p, etc, etc?
> 
> I'm talking the work needed to adapt Emacs's C code, e.g:
> 
>    DEFUN ("get-buffer-window", Fget_buffer_window, Sget_buffer_window, 0, 2, 0,
>           doc: /* Return a window currently displaying BUFFER-OR-NAME, or nil if none.
>    BUFFER-OR-NAME may be a buffer or a buffer name and defaults to
>    the current buffer.
> 
>    The optional argument ALL-FRAMES specifies the frames to consider:
> 
>    - t means consider all windows on all existing frames.
> 
>    - `visible' means consider all windows on all visible frames.
> 
>    - 0 (the number zero) means consider all windows on all visible
>        and iconified frames.
> 
>    - A frame means consider all windows on that frame only.
> 
>    Any other value of ALL-FRAMES means consider all windows on the
>    selected frame and no others.  */)
>         (Lisp_Object buffer_or_name, Lisp_Object all_frames)
>    {
>      Lisp_Object buffer;
> 
>      if (NILP (buffer_or_name))
>        buffer = Fcurrent_buffer ();
>      else
>        buffer = Fget_buffer (buffer_or_name);
> 
>      if (BUFFERP (buffer))
>        return window_loop (GET_BUFFER_WINDOW, buffer, true, all_frames);
>      else
>        return Qnil;
>    }
> 
> 
> -- Stefan
> 

Re: Why is Elisp slow?

[Ergus]

On Tue, May 07, 2019 at 10:16:42PM +0900, ????????? wrote:

Yes, it is not straight forward, but that's what makes it more
interesting (and useful).

I still think that provide a plugin for SBCL is the best to do it native
and efficient, but I have zero knowledge about the SBCL infrastructure
for that. But the idea of adding it some C bindings is not
crazy. Specially because all the functions have the same signature. We
just need some documentation about the internal data structure
representations in SBCL.

>Ahh??? I now understood what???s the problem. :-(
>
>> 2019. 5. 7. ?????? 10:04, Stefan Monnier <monnier@IRO.UMontreal.CA> ??????:
>>
>>>> Similarly, you'll need to rewrite all the functions/macros like CONSP,
>>>> SYMBOLP, FIXNUMP, XCAR, XCDR, make_fixnum, ...  Performance of those
>>>> is important.
>>>
>>> Why would you not use the default CL???s defun, car, cdr, symbol-p,
>>> cons-p, etc, etc?
>>
>> I'm talking the work needed to adapt Emacs's C code, e.g:
>>
>>    DEFUN ("get-buffer-window", Fget_buffer_window, Sget_buffer_window, 0, 2, 0,
>>           doc: /* Return a window currently displaying BUFFER-OR-NAME, or nil if none.
>>    BUFFER-OR-NAME may be a buffer or a buffer name and defaults to
>>    the current buffer.
>>
>>    The optional argument ALL-FRAMES specifies the frames to consider:
>>
>>    - t means consider all windows on all existing frames.
>>
>>    - `visible' means consider all windows on all visible frames.
>>
>>    - 0 (the number zero) means consider all windows on all visible
>>        and iconified frames.
>>
>>    - A frame means consider all windows on that frame only.
>>
>>    Any other value of ALL-FRAMES means consider all windows on the
>>    selected frame and no others.  */)
>>         (Lisp_Object buffer_or_name, Lisp_Object all_frames)
>>    {
>>      Lisp_Object buffer;
>>
>>      if (NILP (buffer_or_name))
>>        buffer = Fcurrent_buffer ();
>>      else
>>        buffer = Fget_buffer (buffer_or_name);
>>
>>      if (BUFFERP (buffer))
>>        return window_loop (GET_BUFFER_WINDOW, buffer, true, all_frames);
>>      else
>>        return Qnil;
>>    }
>>
>>
>> -- Stefan
>>
>
>

Re: Why is Elisp slow?

[조성빈]

Thinking about this, instead of SBCL (which is super-fast, but imposes restrictions on functions being called from the outside world), why not embed ECL (Embeddable Common Lisp), which 
* is significantly slower than SBCL, about 2~3x slower? but is still much faster than Elisp.
* compiles CL code to C
* has great interoperability with C (have never done this before :-( but based on what I hear, can both freely call CL functions from C and the other way around, generate C inside CL as a ECL extension, etc, etc…)

Looks like if Emacs code has Elisp & C interwined, ECL might be a better shot?

> 2019. 5. 7. 오후 10:14, Ergus <spacibba@aol.com> 작성:
> 
> On Tue, May 07, 2019 at 09:04:31AM -0400, Stefan Monnier wrote:
>>>> Similarly, you'll need to rewrite all the functions/macros like CONSP,
>>>> SYMBOLP, FIXNUMP, XCAR, XCDR, make_fixnum, ...  Performance of those
>>>> is important.
>>> 
>>> Why would you not use the default CL???s defun, car, cdr, symbol-p,
>>> cons-p, etc, etc?
>> 
>> I'm talking the work needed to adapt Emacs's C code, e.g:
>> 
>>   DEFUN ("get-buffer-window", Fget_buffer_window, Sget_buffer_window, 0, 2, 0,
>>          doc: /* Return a window currently displaying BUFFER-OR-NAME, or nil if none.
>>   BUFFER-OR-NAME may be a buffer or a buffer name and defaults to
>>   the current buffer.
>> 
>>   The optional argument ALL-FRAMES specifies the frames to consider:
>> 
>>   - t means consider all windows on all existing frames.
>> 
>>   - `visible' means consider all windows on all visible frames.
>> 
>>   - 0 (the number zero) means consider all windows on all visible
>>       and iconified frames.
>> 
>>   - A frame means consider all windows on that frame only.
>> 
>>   Any other value of ALL-FRAMES means consider all windows on the
>>   selected frame and no others.  */)
>>        (Lisp_Object buffer_or_name, Lisp_Object all_frames)
>>   {
>>     Lisp_Object buffer;
>> 
>>     if (NILP (buffer_or_name))
>>       buffer = Fcurrent_buffer ();
>>     else
>>       buffer = Fget_buffer (buffer_or_name);
>> 
>>     if (BUFFERP (buffer))
>>       return window_loop (GET_BUFFER_WINDOW, buffer, true, all_frames);
>>     else
>>       return Qnil;
>>   }
>> 
>> 
>> -- Stefan
>> 
> That's why I was wondering about the C binds and the C types
> representations in SBCL more than the lisp part of the implementation.

Re: Why is Elisp slow?

[Stefan Monnier]

> embed ECL (Embeddable Common Lisp), which
> * is significantly slower than SBCL, about 2~3x slower? but is still
>   much faster than Elisp.

Last time this discussion came up, ECL seemed like the most promising
option indeed, but the performance was not that impressive compared to
Emacs.  Maybe the situation has changed?
Also in terms of maintenance, it's minimal, so it wouldn't help very
much on the side of manpower.


        Stefan

Re: Why is Elisp slow?

[Ergus]

Personally I think is a bad decision go for ECL. It is unmaintained and
performance is not as good that worth the effort. It will be pretty
much the same than going for GCL.

SBCL is very actively maintained and is free and open source why
couldn't we invest some time doing a plugin for our specific use case??
If we ask the maintainers we could even get some help from them?? Is it
more complex than maintain our interpreter and libraries as we are
doing?? Maybe they have some undocumented functionalities we could take
advantage of. If they have FFI they should have some infrastructure
already there, but hidden for normal users (as python does with CTypes
and the python C API).

Actually the bigger problem I see are the buffer local variables and
similar Emacs specific functionalities. (Maybe because I understand
better C than Lisp)

On Tue, May 07, 2019 at 10:22:39AM -0400, Stefan Monnier wrote:
>> embed ECL (Embeddable Common Lisp), which
>> * is significantly slower than SBCL, about 2~3x slower? but is still
>>   much faster than Elisp.
>
>Last time this discussion came up, ECL seemed like the most promising
>option indeed, but the performance was not that impressive compared to
>Emacs.  Maybe the situation has changed?
>Also in terms of maintenance, it's minimal, so it wouldn't help very
>much on the side of manpower.
>
>
>        Stefan
>
>

Re: Why is Elisp slow?

[Ergus]

Hi:

After some mails in the SBCL mailing list and make some
question-reply-question there; I arrived the conclusion that SBCL is not
an alternative :( I was very wrong.

I'll copy-paste the main answers here just in case someone wants to go
this way in the future. Or in case someone more expert see some possible
workaround for the issues exposed here.

----------------------------------------

Your question as stated is too open-ended for me to discern the exact
intent, but I think's it's something like this: there is much common-lisp
code that people run in emacs, and you'd like not to have to maintain a
common-lisp compiler as well as elisp compiler. Instead you'd like to
retarget the output of the SBCL compiler to produce elisp byte code for
execution.
It's unclear whether you'd expect this to be an out-of-process compiler or
an in-process compiler.  If in-process, then you've got larger problems
than figuring out how to call C.  SBCL is not intended to be a "subsystem"
of other lisps, nor even to allow main() to be anything other than its own
main(), though that limitation has been relaxed a bit.

If out-of-process, then I would imagine that SBCL would operate as a file
compiler only, and not a compiler of arbitrary forms. You need to figure
out how to treat emacs as the virtual machine which the SBCL compiler
targets, and a suitable interface for getting the compiled code out of an
SBCL '.fasl' file and into emacs.   The "machine code" in the '.fasl' file
would actually be emacs byte code. (I'm assuming that you plan to keep
more-or-less the same byte code execution engine)

I don't see why C functions are a particular problem.  SBCL uses C
functions all the time.  In fact it is extremely interoperable with C, but
that's really asking the wrong question imho.
Portability is the real question, since you brought it up. Are you implying
that SBCL's compiler should produce machine code that would run within
emacs? This is highly unlikely; I won't say impossible, but darn near so.
 First of all, SBCL can produce machine code only for a tiny fraction of
the CPUs that emacs supports. Secondly, producing machine code assumes that
the entirety of the runtime (comprised of the memory layout, heap objects,
and support routines) is exactly as SBCL expects it to be - symbols look
like SBCL symbols, special variables get bound in exactly the way that SBCL
expects them to get bound.  SBCL's compiled lisp files are not a portable
abstraction like a '.o' file (or '.so' if you like) that could be loaded
into *any* other program (a C program, Java program, etc) and just run.

But as I said initially, I think you're trying to suggest that you would
produce code that is executed by the emacs lisp engine, not the CPU
directly.
So that would be essentially 1 new architecture that SBCL would have to
support, namely the emacs virtual machine.  That sounds more plausible and
there's no reason to care about the set of CPUs that we can directly
compile for, nor even how the existing architectures would call C code.
You'd have to invent that as part of the retargeting to the emacs vm.

Doug

----------------------------------------

I believe the most viable way to use SBCL for Emacs would be to rethink
the approach: instead of a graphics engine written in C - in addition to
many Lisp functions - and Elisp used to "script" that, with SBCL you no
longer need to have much of that C code for performance reasons.

The entry point to this hypothetical new Emacs would be in Common Lisp,
as well as an Elisp environment that would be mostly macrology on top of
CL (to start with). After that, you could consider rewriting much of the
rendering engine into Lisp too, and leave only a tiny amount of C
strictly for interfacing with the operating system.

I know some people have discussed about turning SBCL into a shared
library but the amount of work would be considerable and with little
gain because I doubt that the SBCL maintainers are interested in
committing to a stable C ABI to the compiler internals - otherwise
embedding SBCL into a C program would be of little use except for maybe
running CL away from the main thread.

Stelian Ionescu

----------------------------------------

I expect portability to be a real issue here. If you *do* want to rely on S=
BCL to generate native machine code for you, and abandon supporting platfor=
ms that SBCL doesn't, that would be OK. This would entail significantly mod=
ifying the Emacs runtime though. The SBCL compiler is married pretty heavil=
y to its runtime (mainly through the GC, which is written in C), and SBCL i=
s a small C program which simply jumps to the Lisp entry point in a large L=
isp image and stays there in its own world, with its own calling convention=
 and object layour, occasionally calling back out to C for operating system=
 tools and GC, which understands Lisp objects and calling conventions. So L=
isp/C interoperation works very well, but as others have stated, it works i=
n a very SBCL-specific manner. I reckon the hard part would be getting the =
C engine for Emacs to fit in the picture, hence the suggestion to rewrite m=
uch of it in Lisp.
But the reality is I sort of doubt that eliminating support for a wide vari=
ety of other architectures is acceptable to Emacs. Hence the suggestion to =
create an Emacs bytecode backend. This would solve the portability problem,=
 at the cost of some efficiency. SBCL tries much harder than Emacs to produ=
ce good code (by utilizing high level optimizations such as constraint (e.g=
. type) propagation), however, so even then the bytecode produced by SBCL w=
ould be superior compared to what Emacs can produce now. A problem with thi=
s is that the compiler really does expect to be targeting a CPU ISA rather =
than something higher level, so you'll have to be clever to figure out how =
to set up the translation from the machine independent intermediate represe=
ntation (IR2), and Elisp bytecode.
Finally, another option to reap the benefits of SBCL's fastish code generat=
ion while not sacrificing portability is to figure out how to make Emacs a =
portable Common Lisp program, through use of portable libraries for foreign=
 code interop, graphics, and Elisp (through macrology, most likely). Then i=
t will run on all platforms that have C compilers through other Lisp implem=
entations which do not target machine code (like ECL, CLISP, etc...) and us=
e SBCL for the platforms it supports. This approach is taken by StumpWM, wh=
ich is fairly similar to Emacs.
=20
Charles

----------------------------------------

On Tue, May 07, 2019 at 10:22:39AM -0400, Stefan Monnier wrote:
>> embed ECL (Embeddable Common Lisp), which
>> * is significantly slower than SBCL, about 2~3x slower? but is still
>>   much faster than Elisp.
>
>Last time this discussion came up, ECL seemed like the most promising
>option indeed, but the performance was not that impressive compared to
>Emacs.  Maybe the situation has changed?
>Also in terms of maintenance, it's minimal, so it wouldn't help very
>much on the side of manpower.
>
>
>        Stefan
>
>

Re: Why is Elisp slow?

[Van L]

Stefan Monnier writes:

> Guile would hopefully win on some of those, but by how much?
>
>>8 [snip]
>>8 [snip]
>> because we need manpower.
>
>8 [snip]
>
> But last I checked Guile didn't have much manpower either.  And there's
> a good chance that tracking Guile development (after switching to Guile)
> will also require manpower on Emacs's side.

Can a big advance in funding boost womanpower participation in making
programming and testing great again? recently, big companies that shal
go nameless have been fined amounts like $USD5B, $USD30B plus, and
another one is enqueued for a $USD5B wet lettuce slap across the wrist.

What if the FSF (1) is given a percentage of those fines? say 2%, to
spray on the next evolution of the Internet's GNU Emacs with GC once
every two weeks.

(1) https://nlnet.nl/NGI/reports/NGI-Study-ISBN-9789279864667.pdf

-- 
© 2019 Van L
gpg using EEF2 37E9 3840 0D5D 9183  251E 9830 384E 9683 B835
                               "The pH of lemon juice is 2." - EdX SPU27.1x

Re: Why is Elisp slow?

[조성빈]

I’m just dreaming, but what if Emacs goes the Stumpwm-way and implement everything in CL? There are previous efforts; We might be able to build on them...? 

나의 iPhone에서 보냄

2019. 5. 9. 오후 6:49, Ergus <spacibba@aol.com> 작성:

> Hi:
> 
> After some mails in the SBCL mailing list and make some
> question-reply-question there; I arrived the conclusion that SBCL is not
> an alternative :( I was very wrong.
> 
> I'll copy-paste the main answers here just in case someone wants to go
> this way in the future. Or in case someone more expert see some possible
> workaround for the issues exposed here.
> 
> ----------------------------------------
> 
> Your question as stated is too open-ended for me to discern the exact
> intent, but I think's it's something like this: there is much common-lisp
> code that people run in emacs, and you'd like not to have to maintain a
> common-lisp compiler as well as elisp compiler. Instead you'd like to
> retarget the output of the SBCL compiler to produce elisp byte code for
> execution.
> It's unclear whether you'd expect this to be an out-of-process compiler or
> an in-process compiler.  If in-process, then you've got larger problems
> than figuring out how to call C.  SBCL is not intended to be a "subsystem"
> of other lisps, nor even to allow main() to be anything other than its own
> main(), though that limitation has been relaxed a bit.
> 
> If out-of-process, then I would imagine that SBCL would operate as a file
> compiler only, and not a compiler of arbitrary forms. You need to figure
> out how to treat emacs as the virtual machine which the SBCL compiler
> targets, and a suitable interface for getting the compiled code out of an
> SBCL '.fasl' file and into emacs.   The "machine code" in the '.fasl' file
> would actually be emacs byte code. (I'm assuming that you plan to keep
> more-or-less the same byte code execution engine)
> 
> I don't see why C functions are a particular problem.  SBCL uses C
> functions all the time.  In fact it is extremely interoperable with C, but
> that's really asking the wrong question imho.
> Portability is the real question, since you brought it up. Are you implying
> that SBCL's compiler should produce machine code that would run within
> emacs? This is highly unlikely; I won't say impossible, but darn near so.
> First of all, SBCL can produce machine code only for a tiny fraction of
> the CPUs that emacs supports. Secondly, producing machine code assumes that
> the entirety of the runtime (comprised of the memory layout, heap objects,
> and support routines) is exactly as SBCL expects it to be - symbols look
> like SBCL symbols, special variables get bound in exactly the way that SBCL
> expects them to get bound.  SBCL's compiled lisp files are not a portable
> abstraction like a '.o' file (or '.so' if you like) that could be loaded
> into *any* other program (a C program, Java program, etc) and just run.
> 
> But as I said initially, I think you're trying to suggest that you would
> produce code that is executed by the emacs lisp engine, not the CPU
> directly.
> So that would be essentially 1 new architecture that SBCL would have to
> support, namely the emacs virtual machine.  That sounds more plausible and
> there's no reason to care about the set of CPUs that we can directly
> compile for, nor even how the existing architectures would call C code.
> You'd have to invent that as part of the retargeting to the emacs vm.
> 
> Doug
> 
> ----------------------------------------
> 
> I believe the most viable way to use SBCL for Emacs would be to rethink
> the approach: instead of a graphics engine written in C - in addition to
> many Lisp functions - and Elisp used to "script" that, with SBCL you no
> longer need to have much of that C code for performance reasons.
> 
> The entry point to this hypothetical new Emacs would be in Common Lisp,
> as well as an Elisp environment that would be mostly macrology on top of
> CL (to start with). After that, you could consider rewriting much of the
> rendering engine into Lisp too, and leave only a tiny amount of C
> strictly for interfacing with the operating system.
> 
> I know some people have discussed about turning SBCL into a shared
> library but the amount of work would be considerable and with little
> gain because I doubt that the SBCL maintainers are interested in
> committing to a stable C ABI to the compiler internals - otherwise
> embedding SBCL into a C program would be of little use except for maybe
> running CL away from the main thread.
> 
> Stelian Ionescu
> 
> ----------------------------------------
> 
> I expect portability to be a real issue here. If you *do* want to rely on S=
> BCL to generate native machine code for you, and abandon supporting platfor=
> ms that SBCL doesn't, that would be OK. This would entail significantly mod=
> ifying the Emacs runtime though. The SBCL compiler is married pretty heavil=
> y to its runtime (mainly through the GC, which is written in C), and SBCL i=
> s a small C program which simply jumps to the Lisp entry point in a large L=
> isp image and stays there in its own world, with its own calling convention=
> and object layour, occasionally calling back out to C for operating system=
> tools and GC, which understands Lisp objects and calling conventions. So L=
> isp/C interoperation works very well, but as others have stated, it works i=
> n a very SBCL-specific manner. I reckon the hard part would be getting the =
> C engine for Emacs to fit in the picture, hence the suggestion to rewrite m=
> uch of it in Lisp.
> But the reality is I sort of doubt that eliminating support for a wide vari=
> ety of other architectures is acceptable to Emacs. Hence the suggestion to =
> create an Emacs bytecode backend. This would solve the portability problem,=
> at the cost of some efficiency. SBCL tries much harder than Emacs to produ=
> ce good code (by utilizing high level optimizations such as constraint (e.g=
> . type) propagation), however, so even then the bytecode produced by SBCL w=
> ould be superior compared to what Emacs can produce now. A problem with thi=
> s is that the compiler really does expect to be targeting a CPU ISA rather =
> than something higher level, so you'll have to be clever to figure out how =
> to set up the translation from the machine independent intermediate represe=
> ntation (IR2), and Elisp bytecode.
> Finally, another option to reap the benefits of SBCL's fastish code generat=
> ion while not sacrificing portability is to figure out how to make Emacs a =
> portable Common Lisp program, through use of portable libraries for foreign=
> code interop, graphics, and Elisp (through macrology, most likely). Then i=
> t will run on all platforms that have C compilers through other Lisp implem=
> entations which do not target machine code (like ECL, CLISP, etc...) and us=
> e SBCL for the platforms it supports. This approach is taken by StumpWM, wh=
> ich is fairly similar to Emacs.
> =20
> Charles
> 
> ----------------------------------------
> 
> On Tue, May 07, 2019 at 10:22:39AM -0400, Stefan Monnier wrote:
>>> embed ECL (Embeddable Common Lisp), which
>>> * is significantly slower than SBCL, about 2~3x slower? but is still
>>>  much faster than Elisp.
>> 
>> Last time this discussion came up, ECL seemed like the most promising
>> option indeed, but the performance was not that impressive compared to
>> Emacs.  Maybe the situation has changed?
>> Also in terms of maintenance, it's minimal, so it wouldn't help very
>> much on the side of manpower.
>> 
>> 
>>       Stefan
>> 
>> 
> 

Re: Why is Elisp slow?

[Ergus]

I considered that possibility.

Give a look to the files sizes inside src in Emacs sources. Only xdisp.c, for example, is more than 33000 lines. Projects like remacs are migrating the C code to rust and they have needed years and only moved small portions. Also the lisp compilers aren't as good optimized and mature as GCC for example. And big changes will make the code unfamiliar for our developers.

But we have the scoping differences with common lisp, so that will also force big updates in the lisp code...


On May 10, 2019 5:20:29 AM GMT+02:00, "조성빈" <pcr910303@icloud.com> wrote:
>I’m just dreaming, but what if Emacs goes the Stumpwm-way and implement
>everything in CL? There are previous efforts; We might be able to build
>on them...? 
>
>나의 iPhone에서 보냄
>
>2019. 5. 9. 오후 6:49, Ergus <spacibba@aol.com> 작성:
>
>> Hi:
>> 
>> After some mails in the SBCL mailing list and make some
>> question-reply-question there; I arrived the conclusion that SBCL is
>not
>> an alternative :( I was very wrong.
>> 
>> I'll copy-paste the main answers here just in case someone wants to
>go
>> this way in the future. Or in case someone more expert see some
>possible
>> workaround for the issues exposed here.
>> 
>> ----------------------------------------
>> 
>> Your question as stated is too open-ended for me to discern the exact
>> intent, but I think's it's something like this: there is much
>common-lisp
>> code that people run in emacs, and you'd like not to have to maintain
>a
>> common-lisp compiler as well as elisp compiler. Instead you'd like to
>> retarget the output of the SBCL compiler to produce elisp byte code
>for
>> execution.
>> It's unclear whether you'd expect this to be an out-of-process
>compiler or
>> an in-process compiler.  If in-process, then you've got larger
>problems
>> than figuring out how to call C.  SBCL is not intended to be a
>"subsystem"
>> of other lisps, nor even to allow main() to be anything other than
>its own
>> main(), though that limitation has been relaxed a bit.
>> 
>> If out-of-process, then I would imagine that SBCL would operate as a
>file
>> compiler only, and not a compiler of arbitrary forms. You need to
>figure
>> out how to treat emacs as the virtual machine which the SBCL compiler
>> targets, and a suitable interface for getting the compiled code out
>of an
>> SBCL '.fasl' file and into emacs.   The "machine code" in the '.fasl'
>file
>> would actually be emacs byte code. (I'm assuming that you plan to
>keep
>> more-or-less the same byte code execution engine)
>> 
>> I don't see why C functions are a particular problem.  SBCL uses C
>> functions all the time.  In fact it is extremely interoperable with
>C, but
>> that's really asking the wrong question imho.
>> Portability is the real question, since you brought it up. Are you
>implying
>> that SBCL's compiler should produce machine code that would run
>within
>> emacs? This is highly unlikely; I won't say impossible, but darn near
>so.
>> First of all, SBCL can produce machine code only for a tiny fraction
>of
>> the CPUs that emacs supports. Secondly, producing machine code
>assumes that
>> the entirety of the runtime (comprised of the memory layout, heap
>objects,
>> and support routines) is exactly as SBCL expects it to be - symbols
>look
>> like SBCL symbols, special variables get bound in exactly the way
>that SBCL
>> expects them to get bound.  SBCL's compiled lisp files are not a
>portable
>> abstraction like a '.o' file (or '.so' if you like) that could be
>loaded
>> into *any* other program (a C program, Java program, etc) and just
>run.
>> 
>> But as I said initially, I think you're trying to suggest that you
>would
>> produce code that is executed by the emacs lisp engine, not the CPU
>> directly.
>> So that would be essentially 1 new architecture that SBCL would have
>to
>> support, namely the emacs virtual machine.  That sounds more
>plausible and
>> there's no reason to care about the set of CPUs that we can directly
>> compile for, nor even how the existing architectures would call C
>code.
>> You'd have to invent that as part of the retargeting to the emacs vm.
>> 
>> Doug
>> 
>> ----------------------------------------
>> 
>> I believe the most viable way to use SBCL for Emacs would be to
>rethink
>> the approach: instead of a graphics engine written in C - in addition
>to
>> many Lisp functions - and Elisp used to "script" that, with SBCL you
>no
>> longer need to have much of that C code for performance reasons.
>> 
>> The entry point to this hypothetical new Emacs would be in Common
>Lisp,
>> as well as an Elisp environment that would be mostly macrology on top
>of
>> CL (to start with). After that, you could consider rewriting much of
>the
>> rendering engine into Lisp too, and leave only a tiny amount of C
>> strictly for interfacing with the operating system.
>> 
>> I know some people have discussed about turning SBCL into a shared
>> library but the amount of work would be considerable and with little
>> gain because I doubt that the SBCL maintainers are interested in
>> committing to a stable C ABI to the compiler internals - otherwise
>> embedding SBCL into a C program would be of little use except for
>maybe
>> running CL away from the main thread.
>> 
>> Stelian Ionescu
>> 
>> ----------------------------------------
>> 
>> I expect portability to be a real issue here. If you *do* want to
>rely on S=
>> BCL to generate native machine code for you, and abandon supporting
>platfor=
>> ms that SBCL doesn't, that would be OK. This would entail
>significantly mod=
>> ifying the Emacs runtime though. The SBCL compiler is married pretty
>heavil=
>> y to its runtime (mainly through the GC, which is written in C), and
>SBCL i=
>> s a small C program which simply jumps to the Lisp entry point in a
>large L=
>> isp image and stays there in its own world, with its own calling
>convention=
>> and object layour, occasionally calling back out to C for operating
>system=
>> tools and GC, which understands Lisp objects and calling conventions.
>So L=
>> isp/C interoperation works very well, but as others have stated, it
>works i=
>> n a very SBCL-specific manner. I reckon the hard part would be
>getting the =
>> C engine for Emacs to fit in the picture, hence the suggestion to
>rewrite m=
>> uch of it in Lisp.
>> But the reality is I sort of doubt that eliminating support for a
>wide vari=
>> ety of other architectures is acceptable to Emacs. Hence the
>suggestion to =
>> create an Emacs bytecode backend. This would solve the portability
>problem,=
>> at the cost of some efficiency. SBCL tries much harder than Emacs to
>produ=
>> ce good code (by utilizing high level optimizations such as
>constraint (e.g=
>> . type) propagation), however, so even then the bytecode produced by
>SBCL w=
>> ould be superior compared to what Emacs can produce now. A problem
>with thi=
>> s is that the compiler really does expect to be targeting a CPU ISA
>rather =
>> than something higher level, so you'll have to be clever to figure
>out how =
>> to set up the translation from the machine independent intermediate
>represe=
>> ntation (IR2), and Elisp bytecode.
>> Finally, another option to reap the benefits of SBCL's fastish code
>generat=
>> ion while not sacrificing portability is to figure out how to make
>Emacs a =
>> portable Common Lisp program, through use of portable libraries for
>foreign=
>> code interop, graphics, and Elisp (through macrology, most likely).
>Then i=
>> t will run on all platforms that have C compilers through other Lisp
>implem=
>> entations which do not target machine code (like ECL, CLISP, etc...)
>and us=
>> e SBCL for the platforms it supports. This approach is taken by
>StumpWM, wh=
>> ich is fairly similar to Emacs.
>> =20
>> Charles
>> 
>> ----------------------------------------
>> 
>> On Tue, May 07, 2019 at 10:22:39AM -0400, Stefan Monnier wrote:
>>>> embed ECL (Embeddable Common Lisp), which
>>>> * is significantly slower than SBCL, about 2~3x slower? but is
>still
>>>>  much faster than Elisp.
>>> 
>>> Last time this discussion came up, ECL seemed like the most
>promising
>>> option indeed, but the performance was not that impressive compared
>to
>>> Emacs.  Maybe the situation has changed?
>>> Also in terms of maintenance, it's minimal, so it wouldn't help very
>>> much on the side of manpower.
>>> 
>>> 
>>>       Stefan
>>> 
>>> 
>> 

-- 
Enviado desde mi dispositivo Android con K-9 Mail. Por favor, disculpa mi brevedad.

Re: Why is Elisp slow?

[조성빈]

I understand why this is impossible or near-to-impossible; but...

> 2019. 5. 10. 오후 1:26, Ergus <spacibba@aol.com> 작성:
> 
> I considered that possibility.
> 
> Give a look to the files sizes inside src in Emacs sources. Only xdisp.c, for example, is more than 33000 lines. Projects like remacs are migrating the C code to rust and they have needed years and only moved small portions. Also the lisp compilers aren't as good optimized and mature as GCC for example. And big changes will make the code unfamiliar for our developers.

xdisp.c is a cross-platform (#ifdef-containing) low-level function that handles displaying… C functions like that can just be imported with FFI and be used.
Many CL compilers (such as SBCL, CCL, etc) are actually super-optimized, and emit code as fast as most C compilers as the language was used for a long time;
With some type hints in low-level functions, SBCL emits assembly as fast as C :-)

As for remacs, I’m pretty sure GNU Emacs have much more manpower than Remacs; and Emacs don’t need to migrate all of C code; only the portion that implements Elisp functions.

> But we have the scoping differences with common lisp, so that will also force big updates in the lisp code…

This can be mitigated by implementing Elisp in CL (as we would have to do that) and run the code in the Elisp package.

> On May 10, 2019 5:20:29 AM GMT+02:00, "조성빈" <pcr910303@icloud.com> wrote:
> I’m just dreaming, but what if Emacs goes the Stumpwm-way and implement everything in CL? There are previous efforts; We might be able to build on them...? 
> 
> 나의 iPhone에서 보냄
> 
> 2019. 5. 9. 오후 6:49, Ergus <spacibba@aol.com> 작성:
> 
> Hi:
> 
> After some mails in the SBCL mailing list and make some
> question-reply-question there; I arrived the conclusion that SBCL is not
> an alternative :( I was very wrong.
> 
> I'll copy-paste the main answers here just in case someone wants to go
> this way in the future. Or in case someone more expert see some possible
> workaround for the issues exposed here.
> Your question as stated is too open-ended for me to discern the exact
> intent, but I think's it's something like this: there is much common-lisp
> code that people run in emacs, and you'd like not to have to maintain a
> common-lisp compiler as well as elisp compiler. Instead you'd like to
> retarget the output of the SBCL compiler to produce elisp byte code for
> execution.
> It's unclear whether you'd expect this to be an out-of-process compiler or
> an in-process compiler.  If in-process, then you've got larger problems
> than figuring out how to call C.  SBCL is not intended to be a "subsystem"
> of other lisps, nor even to allow main() to be anything other than its own
> main(), though that limitation has been relaxed a bit.
> 
> If out-of-process, then I would imagine that SBCL would operate as a file
> compiler only, and not a compiler of arbitrary forms. You need to figure
> out how to treat emacs as the virtual machine which the SBCL compiler
> targets, and a suitable interface for getting the compiled code out of an
> SBCL '.fasl' file and into emacs.   The "machine code" in the '.fasl' file
> would actually be emacs byte code. (I'm assuming that you plan to keep
> more-or-less the same byte code execution engine)
> 
> I don't see why C functions are a particular problem.  SBCL uses C
> functions all the time.  In fact it is extremely interoperable with C, but
> that's really asking the wrong question imho.
> Portability is the real question, since you brought it up. Are you implying
> that SBCL's compiler should produce machine code that would run within
> emacs? This is highly unlikely; I won't say impossible, but darn near so.
> First of all, SBCL can produce machine code only for a tiny fraction of
> the CPUs that emacs supports. Secondly, producing machine code assumes that
> the entirety of the runtime (comprised of the memory layout, heap objects,
> and support routines) is exactly as SBCL expects it to be - symbols look
> like SBCL symbols, special variables get bound in exactly the way that SBCL
> expects them to get bound.  SBCL's compiled lisp files are not a portable
> abstraction like a '.o' file (or '.so' if you like) that could be loaded
> into *any* other program (a C program, Java program, etc) and just run.
> 
> But as I said initially, I think you're trying to suggest that you would
> produce code that is executed by the emacs lisp engine, not the CPU
> directly.
> So that would be essentially 1 new architecture that SBCL would have to
> support, namely the emacs virtual machine.  That sounds more plausible and
> there's no reason to care about the set of CPUs that we can directly
> compile for, nor even how the existing architectures would call C code.
> You'd have to invent that as part of the retargeting to the emacs vm.
> 
> Doug
> I believe the most viable way to use SBCL for Emacs would be to rethink
> the approach: instead of a graphics engine written in C - in addition to
> many Lisp functions - and Elisp used to "script" that, with SBCL you no
> longer need to have much of that C code for performance reasons.
> 
> The entry point to this hypothetical new Emacs would be in Common Lisp,
> as well as an Elisp environment that would be mostly macrology on top of
> CL (to start with). After that, you could consider rewriting much of the
> rendering engine into Lisp too, and leave only a tiny amount of C
> strictly for interfacing with the operating system.
> 
> I know some people have discussed about turning SBCL into a shared
> library but the amount of work would be considerable and with little
> gain because I doubt that the SBCL maintainers are interested in
> committing to a stable C ABI to the compiler internals - otherwise
> embedding SBCL into a C program would be of little use except for maybe
> running CL away from the main thread.
> 
> Stelian Ionescu
> I expect portability to be a real issue here. If you *do* want to rely on S=
> BCL to generate native machine code for you, and abandon supporting platfor=
> ms that SBCL doesn't, that would be OK. This would entail significantly mod=
> ifying the Emacs runtime though. The SBCL compiler is married pretty heavil=
> y to its runtime (mainly through the GC, which is written in C), and SBCL i=
> s a small C program which simply jumps to the Lisp entry point in a large L=
> isp image and stays there in its own world, with its own calling convention=
> and object layour, occasionally calling back out to C for operating system=
> tools and GC, which understands Lisp objects and calling conventions. So L=
> isp/C interoperation works very well, but as others have stated, it works i=
> n a very SBCL-specific manner. I reckon the hard part would be getting the =
> C engine for Emacs to fit in the picture, hence the suggestion to rewrite m=
> uch of it in Lisp.
> But the reality is I sort of doubt that eliminating support for a wide vari=
> ety of other architectures is acceptable to Emacs. Hence the suggestion to =
> create an Emacs bytecode backend. This would solve the portability problem,=
> at the cost of some efficiency. SBCL tries much harder than Emacs to produ=
> ce good code (by utilizing high level optimizations such as constraint (e.g=
> . type) propagation), however, so even then the bytecode produced by SBCL w=
> ould be superior compared to what Emacs can produce now. A problem with thi=
> s is that the compiler really does expect to be targeting a CPU ISA rather =
> than something higher level, so you'll have to be clever to figure out how =
> to set up the translation from the machine independent intermediate represe=
> ntation (IR2), and Elisp bytecode.
> Finally, another option to reap the benefits of SBCL's fastish code generat=
> ion while not sacrificing portability is to figure out how to make Emacs a =
> portable Common Lisp program, through use of portable libraries for foreign=
> code interop, graphics, and Elisp (through macrology, most likely). Then i=
> t will run on all platforms that have C compilers through other Lisp implem=
> entations which do not target machine code (like ECL, CLISP, etc...) and us=
> e SBCL for the platforms it supports. This approach is taken by StumpWM, wh=
> ich is fairly similar to Emacs.
> =20
> Charles
> On Tue, May 07, 2019 at 10:22:39AM -0400, Stefan Monnier wrote:
> embed ECL (Embeddable Common Lisp), which
> * is significantly slower than SBCL, about 2~3x slower? but is still
>  much faster than Elisp.
> 
> Last time this discussion came up, ECL seemed like the most promising
> option indeed, but the performance was not that impressive compared to
> Emacs.  Maybe the situation has changed?
> Also in terms of maintenance, it's minimal, so it wouldn't help very
> much on the side of manpower.
> 
> 
>       Stefan
> 
> 
> 
> 
> 
> 
> -- 
> Enviado desde mi dispositivo Android con K-9 Mail. Por favor, disculpa mi brevedad.

Re: Why is Elisp slow?

[Van L]

> Yes, it's a very different design.  Elisp was designed to be the
> *implementation* language of Emacs (the use of C for some of the
> implementation was a necessary evil given the lack of wide availability
> of Lisp), rather than just an *extension* language.

Is it possible to drop the C and re-introduce the missing microcodes
from the Lisp Machine for the full lispy experience on a CuBox Pulse? (1)

(1) https://www.solid-run.com/nxp-family/cubox-pulse/

-- 
© 2019 Van L
gpg using EEF2 37E9 3840 0D5D 9183  251E 9830 384E 9683 B835
                               "The pH of lemon juice is 2." - EdX SPU27.1x

Re: Why is Elisp slow?

[Van L]

> Lisp was, and is a hacker’s language.
> C++, Java is a corporate language. 
> C was, and is a hacker’s language.

Is it possible to compare programming languages to fighter jets at trade
shows demonstrating what can be done w/wo the capability.  For example,
F-22's vectored thrust, F-35's lackof.  They fly their signature flight
paths.  When RMS says CL isn't lispy does that mean no vectored thrust?

-- 
© 2019 Van L
gpg using EEF2 37E9 3840 0D5D 9183  251E 9830 384E 9683 B835
                               "The pH of lemon juice is 2." - EdX SPU27.1x

Re: Why is Elisp slow?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 233 bytes --]

On Fri, May 10, 2019 at 06:26:57AM +0200, Ergus wrote:
> I considered that possibility.
> 
> [...] Also the lisp compilers aren't as good optimized and mature
> as GCC for example [...]

Hmmm. Citation needed.

Cheers
-- t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Why is Elisp slow?

[Van L]

Ergus writes:

> Actually I think that these days will be easier to find new C/C++
> developers for emacs than Lisp developers. Lisp and Scheme are
> beautiful, but they require a different way of thinking and a lot of
> time (own experience, I am just starting with it.) One reason why I
> can't convince my friends to use Emacs is actually how Lisp scares them
> ((())()'()).

Alice wasn't afraid of the cat in Wonderland.

-- 
© 2019 Van L
gpg using EEF2 37E9 3840 0D5D 9183  251E 9830 384E 9683 B835
                  "The iPhone was a really great invention." - Elon Musk

Re: Why is Elisp slow?

[Michael Heerdegen]

Van L <van@scratch.space> writes:

> > ((())()'()).

I wonder how this would look like in C/C++.

Michael.

Lisp, C, and C++ (was: Re: Why is Elisp slow?)

[Emanuel Berg]

Van L wrote:

>> Lisp was, and is a hacker’s language. C++,
>> Java is a corporate language. C was, and is
>> a hacker’s language.
>
> [...]

Who said that? I must'a missed that message and
Van L didn't provide an attribution.

Anyway I agree both Lisp and C are hacker
languages! [figure: 1] They are just different
hacker personalities.

And those personalities can coexist in the same
hacker, for that matter. Just like I enjoyed
"Battle Angel Alita" (2019) just as much as
"Everest" (2015) despite them not being in the
same genre [OT list: 2].

As for Java, I agree it is a corporate language
- or whatever is the appropriate term, but it
isn't a hacker language for sure.

C++ tho I don't know! I think it is a hacker
language as well. A lot of the bad rep it has
taken is because of people's adversity to OO,
which, in turn, is a reaction to the
super OO hype that once was, and also because
of people using it in crazy ways (all but
infinite inheritance trees etc) But, 1) OO is
not necessarily bad, it depends how you use it
as with (almost) everything else; and 2) while
huge in the 90s, most C++ programmers of today
don't put half the emphasis on OO, neither
orally or in practice.


[1] https://dataswamp.org/~incal/pics/lang.png
[2] https://dataswamp.org/~incal/FILM

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Why is Elisp slow?

[Emanuel Berg]

Michael Heerdegen wrote:

>> ((())()'()).
>
> I wonder how this would look like in C/C++.

That question does not compute, as C and C++
are two different languages :P

Actually, it doesn't look much in Lisp either -
it doesn't even evaluate.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Lisp, C, and C++ (was: Re: Why is Elisp slow?)

[Emanuel Berg]

> A lot of the bad rep it has taken is because
> of people's adversity to OO [...]

*aversion

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Why is Elisp slow?

[Emanuel Berg]

If it is possible to have Elisp as fast as CL
or even as fast as C (!), the only problem is
no one has been motivated to do it! ???

Are there not young people and university
students who would love such a challenge?

Or for that matter experienced people who would
love to tinker with it once in a while?

??

I'm sure someone already does it, in some form,
somewhere on the planet! If he reads this,
hello to YOU! Impressive but "as fast as C" was
an exaggeration, right?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Why is Elisp slow?

[Stefan Monnier]

> Are there not young people and university
> students who would love such a challenge?

There are, but they usually consider that applying those ideas to
Javascript/Python/younameit is more worthwhile than applying it
to Elisp.


        Stefan

Re: Why is Elisp slow?

[Emanuel Berg]

Stefan Monnier wrote:

>> Are there not young people and university
>> students who would love such a challenge?
>
> There are, but they usually consider that
> applying those ideas to
> Javascript/Python/younameit is more
> worthwhile than applying it to Elisp.

Is there a tutorial to get them going? I only
read "Compiler Design" at the university, which
means I'm not qualified. But I'm sure you are.
It doesn't have to be a long article. Like
this [1] is enough. If you or anyone else does
it, I can help with the edit, only... if you do
it, I think you'd do all that as well.


[1] https://dataswamp.org/~incal/cols/www/COLORS

-- 
underground experts united
http://user.it.uu.se/~embe8573

Is Elisp really that slow? (was: Why is Elisp slow?)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 2542 bytes --]

On Sat, May 11, 2019 at 02:38:58AM +0200, Emanuel Berg wrote:
> If it is possible to have Elisp as fast as CL
> or even as fast as C (!), the only problem is
> no one has been motivated to do it! ???
> 
> Are there not young people and university
> students who would love such a challenge?

One last attempt: this whole thread is based on the premise
that there is one clear definition of slow. Yet, there isn't.

Back in the 1980s, the (micro-) computers were so slow, that
access to main memory wasn't a problem at all. I know of one
design where two Z80 ran off the same static RAM, taking turns
to access it.

Somewhat later, RAM was so slow that we had to interpose several
layers of cache between it and the CPU (that is still the situation
nowadays, which brought upon us Meltdown and Spectre).

A tad later, CPUs stopped getting faster at sequential execution.
Making pipelines deeper and things stopped being fun, and transistors
are at the current limit of how fast they can do. But, we can make
them cheaper and smaller (and perhaps somewhat less greedy on
electrons), so packing more CPUs on the chip is becoming more
fashionable [1].

If you want to be really fast these days, you have a gang of
fairly dumb processors all doing the same instruction at a
time (SIMD). It's weird to program that: if you take a branch,
you have to decide for one side, and throw away those members
of your gang which went the other way (well, you throw their
results away, actually). Perhaps pick the second branch up
in a second pass.

Of course, the definition of "fast software" changes radically
when your bedrock moves that quickly. The second "generation"
above made cache-efficient algorithms interesting (somehow
a rebirth of the 70ies, where you had very limited RAM and
somewhat bigger external memory). In the third generation,
parallelizable algorithms became more interesting (did you
know that bubble sort, the laughing stock of CS, parallelizes
pretty well? If you have more processors than time, this one
isn't that bad). The fourth generation... well. Interestingly,
functional languages help in making fast programs for those
strange beasts without getting crazy.

What I don't like about this thread is that "fast" is being
treated as the only metrics (it isn't) and as if it were a
metrics at all (it isn't either).

Cheers

[1] Sometimes you see that coming and bet too early on it.
    Alas, Sun is no more:
    https://en.wikipedia.org/wiki/Sun_Niagara 

-- tomás

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Is Elisp really that slow? (was: Why is Elisp slow?)

[조성빈]

I understand your points; but IMHO Elisp is not doing well in all kinds of aspects, Isn’t it?̊̈

나의 iPhone에서 보냄

2019. 5. 11. 오후 4:32, tomas@tuxteam.de 작성:

>> On Sat, May 11, 2019 at 02:38:58AM +0200, Emanuel Berg wrote:
>> If it is possible to have Elisp as fast as CL
>> or even as fast as C (!), the only problem is
>> no one has been motivated to do it! ???
>> 
>> Are there not young people and university
>> students who would love such a challenge?
> 
> One last attempt: this whole thread is based on the premise
> that there is one clear definition of slow. Yet, there isn't.
> 
> Back in the 1980s, the (micro-) computers were so slow, that
> access to main memory wasn't a problem at all. I know of one
> design where two Z80 ran off the same static RAM, taking turns
> to access it.
> 
> Somewhat later, RAM was so slow that we had to interpose several
> layers of cache between it and the CPU (that is still the situation
> nowadays, which brought upon us Meltdown and Spectre).
> 
> A tad later, CPUs stopped getting faster at sequential execution.
> Making pipelines deeper and things stopped being fun, and transistors
> are at the current limit of how fast they can do. But, we can make
> them cheaper and smaller (and perhaps somewhat less greedy on
> electrons), so packing more CPUs on the chip is becoming more
> fashionable [1].
> 
> If you want to be really fast these days, you have a gang of
> fairly dumb processors all doing the same instruction at a
> time (SIMD). It's weird to program that: if you take a branch,
> you have to decide for one side, and throw away those members
> of your gang which went the other way (well, you throw their
> results away, actually). Perhaps pick the second branch up
> in a second pass.
> 
> Of course, the definition of "fast software" changes radically
> when your bedrock moves that quickly. The second "generation"
> above made cache-efficient algorithms interesting (somehow
> a rebirth of the 70ies, where you had very limited RAM and
> somewhat bigger external memory). In the third generation,
> parallelizable algorithms became more interesting (did you
> know that bubble sort, the laughing stock of CS, parallelizes
> pretty well? If you have more processors than time, this one
> isn't that bad). The fourth generation... well. Interestingly,
> functional languages help in making fast programs for those
> strange beasts without getting crazy.
> 
> What I don't like about this thread is that "fast" is being
> treated as the only metrics (it isn't) and as if it were a
> metrics at all (it isn't either).
> 
> Cheers
> 
> [1] Sometimes you see that coming and bet too early on it.
>    Alas, Sun is no more:
>    https://en.wikipedia.org/wiki/Sun_Niagara 
> 
> -- tomás

Re: Is Elisp really that slow? (was: Why is Elisp slow?)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 564 bytes --]

On Sat, May 11, 2019 at 04:42:44PM +0900, 조성빈 wrote:
> I understand your points; but IMHO Elisp is not doing well in all kinds of aspects, Isn’t it?̊̈

(quoting myself):

> > What I don't like about this thread is that "fast" is being
> > treated as the only metrics (it isn't)

Accessibility/hackability: elisp is doing pretty well in this
one (and efficient compilation does put a burden on that).

>                                      and as if it were a
> > metrics at all (it isn't either).

Show us the benchmarks.

Cheers
-- t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Is Elisp really that slow? (was: Why is Elisp slow?)

[조성빈]

2019. 5. 11. 오후 4:57, tomas@tuxteam.de 작성:

>> On Sat, May 11, 2019 at 04:42:44PM +0900, 조성빈 wrote:
>> I understand your points; but IMHO Elisp is not doing well in all kinds of aspects, Isn’t it?̊̈
> 
> (quoting myself):
> 
>>> What I don't like about this thread is that "fast" is being
>>> treated as the only metrics (it isn't)
> 
> Accessibility/hackability: elisp is doing pretty well in this
> one (and efficient compilation does put a burden on that).

Javascript? Common Lisp? PyPy(Implementation of Python)?

>>                                     and as if it were a
>>> metrics at all (it isn't either).
> 
> Show us the benchmarks.
> 
> Cheers
> -- t

Re: Is Elisp really that slow? (was: Why is Elisp slow?)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 805 bytes --]

On Sat, May 11, 2019 at 11:30:55PM +0900, 조성빈 wrote:
> 
> 2019. 5. 11. 오후 4:57, tomas@tuxteam.de 작성:
> 
> >> On Sat, May 11, 2019 at 04:42:44PM +0900, 조성빈 wrote:
> >> I understand your points; but IMHO Elisp is not doing well in all kinds of aspects, Isn’t it?̊̈
> > 
> > (quoting myself):
> > 
> >>> What I don't like about this thread is that "fast" is being
> >>> treated as the only metrics (it isn't)
> > 
> > Accessibility/hackability: elisp is doing pretty well in this
> > one (and efficient compilation does put a burden on that).
> 
> Javascript? Common Lisp? PyPy(Implementation of Python)?

What's the last time you hacked the innards of those?

;-)

IOW: I was talking also about accessibility/hackability "down the
stack".

Cheers
-- tomás

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Is Elisp really that slow? (was: Why is Elisp slow?)

[Emanuel Berg]

tomas wrote:

> Show us the benchmarks.

How do people do benchmarks when it comes to
comparing the speed of different
computer languages?

Do you solve a wide/varied set of problems (the
same set) with different languages and then
compare the wall clock execution time?

Perhaps using an atomic clock in
a sterile environment?

Now, while that would be interesting to do,
I think it's pretty clear Elisp would benefit
from more speed. Maybe it is not slow for its
purposes, but it ain't fast either, sure is
my impression.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Is Elisp really that slow? (was: Why is Elisp slow?)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 2719 bytes --]

On Sun, May 12, 2019 at 01:09:09AM +0200, Emanuel Berg wrote:
> tomas wrote:
> 
> > Show us the benchmarks.
> 
> How do people do benchmarks when it comes to
> comparing the speed of different
> computer languages?

My favourite search engine recommends [1], [2].

> Do you solve a wide/varied set of problems (the
> same set) with different languages and then
> compare the wall clock execution time?

More or less, yes. See references.

> Perhaps using an atomic clock in
> a sterile environment?

Clock accuracy/precision/resolution will be the least of your problems
in a cross-language benchmark. Different languages exist for a reason,
and carry with them a whole set of different cultures, so they'll tend
to express a problem in subtly different way (leading to subtly differing
semantics). How do you account for the fact that some problem expressed
in C won't do array bounds checking, while in Java it will do them
dynamically all the time (costing runtime), whereas in some smart
functional language the compiler will do most of that work most of
the time, due to advanced compiler magic? How do you account for the
fact that in language A the programmer will take 100 times longer to
master the language, but one-tenth of the time to write a program?
The fact that this compiler will produce runtimes twice as fast, but
ten times as fat, and take 50 times the compile time?

Yeah. Numbers for illustration only.

> Now, while that would be interesting to do,
> I think it's pretty clear Elisp would benefit
> from more speed.

My whole point is that this is /not/ so clear. Emphatically so.
Speed comes at a price (did you do your research about Spectre
and Meltdown?). Perhaps Elisp has found an optimum in this tradeoff,
for the job it's supposed to do.

Well, actually this optimum point shifts around with time, since
the environment Emacs lives in changes (people using it, the kind
of tasks it is put to, but also the hardware it lives on).

Actually I'm convinced Emacs is tracking this optimum pretty nicely,
since it is looking back to 34 years of life (and still receives a
steady stream of patches [3]!). Few pieces of software can say that
of them.

>                 Maybe it is not slow for its
> purposes,

...and that is exactly the point!

>           but it ain't fast either, sure is
> my impression.

I haven't the time currently to illustrate my point more, sorry. Do
your experiments, find out for yourself.

Cheers

[1] https://en.wikipedia.org/wiki/The_Computer_Language_Benchmarks_Game
[2] https://benchmarksgame-team.pages.debian.net/benchmarksgame/
[3] http://git.savannah.gnu.org/cgit/emacs.git

-- tomás

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Is Elisp really that slow? (was: Why is Elisp slow?)

[Emanuel Berg]

tomas wrote:

> Different languages exist for a reason, and
> carry with them a whole set of different
> cultures, so they'll tend to express
> a problem in subtly different way (leading to
> subtly differing semantics). How do you
> account for the fact that some problem
> expressed in C won't do array bounds
> checking, while in Java it will do them
> dynamically all the time (costing runtime),
> whereas in some smart functional language the
> compiler will do most of that work most of
> the time, due to advanced compiler magic?
> How do you account for the fact that in
> language A the programmer will take 100 times
> longer to master the language, but one-tenth
> of the time to write a program? The fact that
> this compiler will produce runtimes twice as
> fast, but ten times as fat, and take 50 times
> the compile time?

One program will finish execution with
a correct result at time X and another at
time Y and it will not be X = Y. So one
language, with associated tools, is faster!

If one language has a better compiler, a better
programmer, and so on, that's part of the game.

You can't have a better computer, that's all.

> Clock accuracy/precision/resolution will be
> the least of your problems in
> a cross-language benchmark.

How about a zero-gravity chamber then?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Is Elisp really that slow? (was: Why is Elisp slow?)

[조성빈]

2019. 5. 12. 오후 4:54, tomas@tuxteam.de 작성:

>> On Sun, May 12, 2019 at 01:09:09AM +0200, Emanuel Berg wrote:
>> tomas wrote:
>> 
>>> Show us the benchmarks.
>> 
>> How do people do benchmarks when it comes to
>> comparing the speed of different
>> computer languages?
> 
> My favourite search engine recommends [1], [2].
> 
>> Do you solve a wide/varied set of problems (the
>> same set) with different languages and then
>> compare the wall clock execution time?
> 
> More or less, yes. See references.
> 
>> Perhaps using an atomic clock in
>> a sterile environment?
> 
> Clock accuracy/precision/resolution will be the least of your problems
> in a cross-language benchmark. Different languages exist for a reason,
> and carry with them a whole set of different cultures, so they'll tend
> to express a problem in subtly different way (leading to subtly differing
> semantics). How do you account for the fact that some problem expressed
> in C won't do array bounds checking, while in Java it will do them
> dynamically all the time (costing runtime), whereas in some smart
> functional language the compiler will do most of that work most of
> the time, due to advanced compiler magic? How do you account for the
> fact that in language A the programmer will take 100 times longer to
> master the language, but one-tenth of the time to write a program?
> The fact that this compiler will produce runtimes twice as fast, but
> ten times as fat, and take 50 times the compile time?

So, you would compare languages that are ‘safe and dynamic enough’ as Elisp.
E.g. Python, JS, CL, Ruby, etc.

> Yeah. Numbers for illustration only.
> 
>> Now, while that would be interesting to do,
>> I think it's pretty clear Elisp would benefit
>> from more speed.
> 
> My whole point is that this is /not/ so clear. Emphatically so.
> Speed comes at a price (did you do your research about Spectre
> and Meltdown?). Perhaps Elisp has found an optimum in this tradeoff,
> for the job it's supposed to do.

Speed comes at a price, but as we see Pypy, JS, CL, and other high languages that are fast, we can see that it is /so/ clear that we can pursue more speed without sacrificing the language’s expressiveness, dynamicness, etc.

> Well, actually this optimum point shifts around with time, since
> the environment Emacs lives in changes (people using it, the kind
> of tasks it is put to, but also the hardware it lives on).
> 
> Actually I'm convinced Emacs is tracking this optimum pretty nicely,
> since it is looking back to 34 years of life (and still receives a
> steady stream of patches [3]!). Few pieces of software can say that
> of them.

Well, look at Atom and VS Code which is using the V8 engine (through node) to execute JS to tweak the editor. 
Now compare that to Emacs.
And Atom gets criticized by it’s slow bootup time, etc...
It’s pretty sure Emacs can benefit from some speed...

>>                Maybe it is not slow for its
>> purposes,
> 
> ...and that is exactly the point!
> 
>>          but it ain't fast either, sure is
>> my impression.
> 
> I haven't the time currently to illustrate my point more, sorry. Do
> your experiments, find out for yourself.
> 
> Cheers
> 
> [1] https://en.wikipedia.org/wiki/The_Computer_Language_Benchmarks_Game
> [2] https://benchmarksgame-team.pages.debian.net/benchmarksgame/
> [3] http://git.savannah.gnu.org/cgit/emacs.git
> 
> -- tomás

Re: Is Elisp really that slow? (was: Why is Elisp slow?)

[Eli Zaretskii]

> From: 조성빈 <pcr910303@icloud.com>
> Date: Sun, 12 May 2019 18:46:20 +0900
> Cc: help-gnu-emacs@gnu.org
> 
> Well, look at Atom and VS Code which is using the V8 engine (through node) to execute JS to tweak the editor. 
> Now compare that to Emacs.

This is meaningless, unless you describe the use case for comparison
(and while at that, how about some timings instead of hand-waving?).

Most stuff Emacs routinely does is something the other editors could
only dream about, so by and large you are comparing apples to peanuts.

And while I'm here, allow me a few comments regarding this thread's
topic:

 . Speed of ELisp is a valid topic for discussion, but it has very
   little, if anything to do with whether Emacs as a whole is or isn't
   slow.  Why? because speed-critical stuff is not supposed to be
   implemented in Lisp in Emacs, that is not how Emacs is designed and
   implemented.  If you find some operation that is slow in Emacs, it
   just means it might either use more efficient algorithms, or should
   have some of it recoded in C.

 . The speed of various Emacs operations is always a trade-off between
   how fast we want Emacs to be, on the one hand, and how flexible and
   controllable from Lisp we want it to be, OTOH.  When neither better
   algorithms nor reimplementation in C are likely to speed up some
   operation, it means we decided to give up some speed in order to
   have more control and flexibility.  There's nothing wrong with
   that, IMO.

 . Some Emacs operations are slow because no one felt the itch to sit
   down and speed them up -- here's your chance to do something about
   that.

Re: Is Elisp really that slow?

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

> Most stuff Emacs routinely does is something the other editors could
> only dream about, so by and large you are comparing apples to peanuts.

Examples?

Emacs is amazing, but as far as productivity goes, at least as a
programmer's editor for popular languages, it is far behind current
contenders.

Re: Is Elisp really that slow?

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Sun, 12 May 2019 16:45:47 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Most stuff Emacs routinely does is something the other editors could
> > only dream about, so by and large you are comparing apples to peanuts.
> 
> Examples?

  M-: (add-text-properties 1 2 '(display "Look, ma!")) RET

> Emacs is amazing, but as far as productivity goes, at least as a
> programmer's editor for popular languages, it is far behind current
> contenders.

You've changed the subject, yes?  This was never about productivity.

I agree that our situation with IDE features is quite bleak, but
that's entirely unrelated to the issues discussed in this thread, and
its reasons IMO have nothing to do with either ELisp speed or
flexibility of Emacs in general.

Re: Is Elisp really that slow?

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> Emacs is amazing, but as far as productivity goes, at least as a
>> programmer's editor for popular languages, it is far behind current
>> contenders.
>
> You've changed the subject, yes?  This was never about productivity.
>
> I agree that our situation with IDE features is quite bleak, but
> that's entirely unrelated to the issues discussed in this thread, and
> its reasons IMO have nothing to do with either ELisp speed or
> flexibility of Emacs in general.

It has a lot to do. Analyzing source code is cpu-intensive, because it
is algorithmically complex and lots of data needs to be processed.

The ill-fated Semantic package, despite all the effort invested on
implementing speed-ups, suffered badly from this and AFAIK it was one of
the main causes of its failure.

Even if you defer the actual analysis to an external tool (which is the
sane thing to do, BTW) the amount of data to be processed on the Emacs
end can be quite large, more than enough to introduce perceptible delays
that spoil the experience.

Re: Is Elisp really that slow?

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Sun, 12 May 2019 17:46:54 +0200
> 
> > I agree that our situation with IDE features is quite bleak, but
> > that's entirely unrelated to the issues discussed in this thread, and
> > its reasons IMO have nothing to do with either ELisp speed or
> > flexibility of Emacs in general.
> 
> It has a lot to do. Analyzing source code is cpu-intensive, because it
> is algorithmically complex and lots of data needs to be processed.
> 
> The ill-fated Semantic package, despite all the effort invested on
> implementing speed-ups, suffered badly from this and AFAIK it was one of
> the main causes of its failure.
> 
> Even if you defer the actual analysis to an external tool (which is the
> sane thing to do, BTW) the amount of data to be processed on the Emacs
> end can be quite large, more than enough to introduce perceptible delays
> that spoil the experience.

It makes very little sense to me to build a case about bad performance
on a package that is incomplete, not very actively maintained, and
whose features are nowhere near what is needed for a good modern IDE.
It can maybe serve as an opportunity to let off steam, perhaps that's
what you wanted.

Or are you saying that this situation is due to the fact that ELisp is
a bad implementation language?

Re: Is Elisp really that slow?

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Óscar Fuentes <ofv@wanadoo.es>
>> Date: Sun, 12 May 2019 17:46:54 +0200
>> 
>> > I agree that our situation with IDE features is quite bleak, but
>> > that's entirely unrelated to the issues discussed in this thread, and
>> > its reasons IMO have nothing to do with either ELisp speed or
>> > flexibility of Emacs in general.
>> 
>> It has a lot to do. Analyzing source code is cpu-intensive, because it
>> is algorithmically complex and lots of data needs to be processed.
>> 
>> The ill-fated Semantic package, despite all the effort invested on
>> implementing speed-ups, suffered badly from this and AFAIK it was one of
>> the main causes of its failure.
>> 
>> Even if you defer the actual analysis to an external tool (which is the
>> sane thing to do, BTW) the amount of data to be processed on the Emacs
>> end can be quite large, more than enough to introduce perceptible delays
>> that spoil the experience.
>
> It makes very little sense to me to build a case about bad performance
> on a package that is incomplete, not very actively maintained, and
> whose features are nowhere near what is needed for a good modern IDE.

Maybe this is because they hit some walls, like slowness on Elisp.

> It can maybe serve as an opportunity to let off steam, perhaps that's
> what you wanted.

When I started with Emacs around 20 years ago, whenever something was
incorrectly indented or fontified on C++ code I was writing it indicated
an error on my part. It was way ahead of any other editor. Nowadays CC
Mode can't handle C++ code styles and constructs that became available
shortly afterwards the period I mentioned. For languages that became
popular since the 1990's Emacs is barely adequate on some cases and a
non-starter on others.

I'm certainly frustrated because this is a self-inflicted problem.
Emacs' high-governance (and I mean "really high", you know what I mean)
forced some very bad decisions upon the project (and not only on Emacs)
which put us all, developers and users, on a really difficult position.
It is terrible that now I have to choose between a superb editor that is
dumb about what I'm writing and an average editor that is imbricated
with the language's tooling and thus greatly helps me to be more
productive.

As someone that swears for Emacs, yes, I need to let off some steam
whenever I see how certain topics are discussed.

> Or are you saying that this situation is due to the fact that ELisp is
> a bad implementation language?

Elisp is really good, I don't even dare to propose a migration to
another language.

I have on my to-do list for this year to grab Tromey's JIT branch and
migrate it to LLVM and see what the optimization passes can squeeze,
possibly with some simple static analysis and, if I'm in the mood, type
annotations on Elisp code.

Just for fun, because I know on advance that it is unacceptable on Emacs
repo.

Re: Is Elisp really that slow?

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Sun, 12 May 2019 20:37:08 +0200
> 
> When I started with Emacs around 20 years ago, whenever something was
> incorrectly indented or fontified on C++ code I was writing it indicated
> an error on my part. It was way ahead of any other editor. Nowadays CC
> Mode can't handle C++ code styles and constructs that became available
> shortly afterwards the period I mentioned. For languages that became
> popular since the 1990's Emacs is barely adequate on some cases and a
> non-starter on others.
> 
> I'm certainly frustrated because this is a self-inflicted problem.
> Emacs' high-governance (and I mean "really high", you know what I mean)
> forced some very bad decisions upon the project (and not only on Emacs)
> which put us all, developers and users, on a really difficult position.

If you want to explain the problems of CC Mode by some "governance",
you are dead wrong, AFAIK.  The real reasons, IMO, are entirely
different.  At the very least, it was never proven that this has
anything to do with the problems we have.  Anyone who tracks
emacs-devel could easily formulate an alternative hypothesis, which
IMO is much more probable.

In general, problems in Emacs are rarely if ever due to management
(which doesn't really exist), they are almost always due to lack of
manpower, expertise, talent, free time, motivation, etc.

> I have on my to-do list for this year to grab Tromey's JIT branch and
> migrate it to LLVM and see what the optimization passes can squeeze,
> possibly with some simple static analysis and, if I'm in the mood, type
> annotations on Elisp code.

The JIT branch doesn't really work, and I have yet to see any speedup
from it in the configurations where it does compile.  So from my POV,
that experiment is a failure, and (having spent more than one or two
hours on making it work better) I see no reason why the failure could
be fixed by using LLVM, because libjit doesn't use any compiler in the
first place.

So, here too, I see a very weak connection, to say the least, between
your suspected reasons and the reality as I know it.

Re: Is Elisp really that slow?

[Stefan Monnier]

>> I agree that our situation with IDE features is quite bleak, but
>> that's entirely unrelated to the issues discussed in this thread, and
>> its reasons IMO have nothing to do with either ELisp speed or
>> flexibility of Emacs in general.
>
> It has a lot to do. Analyzing source code is cpu-intensive, because it
> is algorithmically complex and lots of data needs to be processed.

That's part of the problem, but I think it's a minor part of it: most
successful IDEs talk directly to the compiler instead of reimplementing
their own language-implementation-front-end (that requires changes to
the compiler, admittedly, but that's what the `I` is all about).

IOW besides performance, another big challenge is having to
(re)implement a complete language front-end.

I think that's why no effort in this direction took off until LSP:
without LSP, providing IDE-like features in Emacs requires either
hacking your compiler so it gives Emacs the relevant info (this has
been done for a few languages where Emacs users represent a large part
of the population (Lisp and Agda come to mind), but it's rather rare),
or reimplementing a front-end which is hard to justify (and requires
a lot of up-front work before you get any benefit at all).

Most major modes have just enough manpower to get a highlighter and
basic indenter up and running with incremental improvements after that.

Of course, the fact that performance may get in the way makes this even
less attractive.

> Even if you defer the actual analysis to an external tool (which is the
> sane thing to do, BTW)

Indeed.

> the amount of data to be processed on the Emacs end can be quite
> large, more than enough to introduce perceptible delays that spoil
> the experience.

With LSP front-ends using the C-implemented json parser present in recent
Emacsen, this should hopefully be less problematic, but indeed this is
a part where we need to put some of the code in C in order to get good
enough performance.

And the single-threadedness is an additional issue in this respect.


        Stefan

Re: Is Elisp really that slow?

[Stefan Monnier]

> I have on my to-do list for this year to grab Tromey's JIT branch and
> migrate it to LLVM and see what the optimization passes can squeeze,
> possibly with some simple static analysis and, if I'm in the mood, type
> annotations on Elisp code.

Elisp's current implementation is sufficiently naive that it should be
"easy" to make it significantly faster at least on some problems.

This said, it's also easy to overlook the fact that in most
circumstances time is spent running within functions implemented in
C and no amount of clever compilation will help (and clever compilation
can easily end up counterproductive because we spend a lot of resources
(heap space, CPU time, you name it) trying to optimize code that's
largely irrelevant, so the end result is slower startup and no speedup,
or worse).

If you want to do a JIT thingy that speeds up Elisp "in general" then
you need to make sure it does a good job eliminating redundant type
checks and things like that.  You'll probably easily get good speed ups
for the usual tight loops this way.  I encourage people to try something
like basic block versioning for that.

But I'm not sure such a "faster Elisp" will make a big difference to the
kind of code one can write in Elisp.  More specifically, I think it'll
take a lot of effort for a JIT to be good enough that we can implement
the json printer/parser in Elisp instead of C.

So another alternative is to make it possible to write "Elisp-ish" code
that's specifically tailored for compilation to efficient code and
that's specifically marked as such.  This has 2 very significant
advantages:
- the compiler can take its time because it's only applied to those
  specific cases where we (presumably) know it's worthwhile.  We may
  even run the compiler ahead of time rather than in a JIT fashion so it
  doesn't impact startup.
- the compiler doesn't need to accept all Elisp code and reproduce all
  its intricate details faithfully.  It can even signal errors and burp
  on some constructs that it doesn't want to support.
Then again, maybe this "Elisp-ish" could be fairly far removed
from Elisp.  E.g. it could even be plain old C (tho something in
between is likely more interesting).


        Stefan

Re: Is Elisp really that slow?

[Óscar Fuentes]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I have on my to-do list for this year to grab Tromey's JIT branch and
>> migrate it to LLVM and see what the optimization passes can squeeze,
>> possibly with some simple static analysis and, if I'm in the mood, type
>> annotations on Elisp code.
>
> Elisp's current implementation is sufficiently naive that it should be
> "easy" to make it significantly faster at least on some problems.

I'm not interested on making Elisp 30% faster. I want something that can
replace C for most purposes.

> This said, it's also easy to overlook the fact that in most
> circumstances time is spent running within functions implemented in
> C and no amount of clever compilation will help

Exactly. Elisp acts as glue code for the C base. This plus the fact that
many elisp primitives reduce to calls to C functions explains the
observed lack of impact of the JIT.

>(and clever compilation
> can easily end up counterproductive because we spend a lot of resources
> (heap space, CPU time, you name it) trying to optimize code that's
> largely irrelevant, so the end result is slower startup and no speedup,
> or worse).

Agreed.

> If you want to do a JIT thingy that speeds up Elisp "in general" then
> you need to make sure it does a good job eliminating redundant type
> checks and things like that.  You'll probably easily get good speed ups
> for the usual tight loops this way.  I encourage people to try something
> like basic block versioning for that.
>
> But I'm not sure such a "faster Elisp" will make a big difference to the
> kind of code one can write in Elisp.  More specifically, I think it'll
> take a lot of effort for a JIT to be good enough that we can implement
> the json printer/parser in Elisp instead of C.
>
> So another alternative is to make it possible to write "Elisp-ish" code
> that's specifically tailored for compilation to efficient code and
> that's specifically marked as such.  This has 2 very significant
> advantages:
> - the compiler can take its time because it's only applied to those
>   specific cases where we (presumably) know it's worthwhile.  We may
>   even run the compiler ahead of time rather than in a JIT fashion so it
>   doesn't impact startup.
> - the compiler doesn't need to accept all Elisp code and reproduce all
>   its intricate details faithfully.  It can even signal errors and burp
>   on some constructs that it doesn't want to support.
> Then again, maybe this "Elisp-ish" could be fairly far removed
> from Elisp.  E.g. it could even be plain old C (tho something in
> between is likely more interesting).

I've been advocating this Elisp-ish C for a long time. Even on your
stint as maintainer I offered my implementation of my own C-with-sexps
language.

I'm interested on islotating certain parts of Elisp, add some stuff to
help the optimizers and feed it to LLVM. Ideally, an Elisp hacker
working within those limits could implement some cpu-intensive code that
would perform similar to C with minimal retraining. At the same time,
the language must retain enough expressiveness to bring significant
advantages on development over C. defmacro is a must.

Re: Is Elisp really that slow?

[Samuel Wales]

can this be sped up?

               (while (re-search-forward from nil t)
                 (replace-match to t))

Re: Is Elisp really that slow?

[Emanuel Berg]

Eli Zaretskii wrote:

> I agree that our situation with IDE features
> is quite bleak

What IDE features are those? (Not
a rhetorical question.)

Actually in the literal sense of the word
(acronym) I think Emacs is the _ultimate_ IDE
for doing anything and everything with
a computer.

I couldn't stand having one program for HTML,
one for LaTeX, one for Lisp, and so on. And if
they (the dedicated IDEs) have special features
which we don't in our major modes, isn't this
also a question of ~"nobody has been motivated
to do it"?

And if nobody has or is, why care/complain
about it? Let it be, man.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Is Elisp really that slow?

[Emanuel Berg]

Óscar Fuentes wrote:

> When I started with Emacs around 20 years
> ago, whenever something was incorrectly
> indented or fontified on C++ code I was
> writing it indicated an error on my part.
> It was way ahead of any other editor.
> Nowadays CC Mode can't handle C++ code styles
> and constructs that became available shortly
> afterwards the period I mentioned.
> For languages that became popular since the
> 1990's Emacs is barely adequate on some cases
> and a non-starter on others.
>
> [...] It is terrible that now I have to
> choose between a superb editor that is dumb
> about what I'm writing and an average editor
> that is imbricated with the language's
> tooling and thus greatly helps me to be
> more productive.

OK, I see what you mean. Perhaps I was wrong in
my pervious message just now. You want it but
you can't improve it because you are busy
_using_ it. Not everyone are derailed like me
and can thus do whatever they want for no
money. Well, that ain't no good, what
you describe.

> I'm certainly frustrated because this is
> a self-inflicted problem.
> Emacs' high-governance (and I mean "really
> high", you know what I mean) forced some very
> bad decisions upon the project (and not only
> on Emacs) which put us all, developers and
> users, on a really difficult position.

What is this about? Please explain.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Is Elisp really that slow?

[Emanuel Berg]

Eli Zaretskii wrote:

> In general, problems in Emacs are rarely if
> ever due to management (which doesn't really
> exist), they are almost always due to lack of
> manpower, expertise, talent, free time,
> motivation, etc.

Perhaps one could strive to make the general
atmosphere more welcoming and attractive to
newcomers and young programmers... as
a collective effort, I mean.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Is Elisp really that slow?

[Emanuel Berg]

Stefan Monnier wrote:

> So another alternative is to make it possible
> to write "Elisp-ish" code that's specifically
> tailored for compilation to efficient code
> and that's specifically marked as such.
> This has 2 very significant advantages:
>
> - the compiler can take its time because it's
>   only applied to those specific cases where
>   we (presumably) know it's worthwhile.
>   We may even run the compiler ahead of time
>   rather than in a JIT fashion so it doesn't
>   impact startup.
>
> - the compiler doesn't need to accept all
>   Elisp code and reproduce all its intricate
>   details faithfully. It can even signal
>   errors and burp on some constructs that it
>   doesn't want to support.
>
> Then again, maybe this "Elisp-ish" could be
> fairly far removed from Elisp. E.g. it could
> even be plain old C (tho something in between
> is likely more interesting).

Yes, I thought about this as well! That would
be so cool and, as you say, interesting!
Only one would keep the Lisp syntax, right?
Just remove the compilation trouble spots and
see what's left. Probably quite a lot! And then
with the project started, people would get to
them trouble spots/areas and make them smaller
and smaller until some would disappear even!

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Why is Elisp slow?

[Van L]

> Michael Heerdegen writes:
> 
>>> ((())()'()).
> 
> I wonder how this would look like in C/C++.

You may find with luck an image archive (1)
of obfuscatory C/C++ source code ascii art.

(1) https://www.space.com/rosetta-comet-67p-huge-photo-archive.html

Re: Is Elisp really that slow?

[Stefan Monnier]

> can this be sped up?
>
>                (while (re-search-forward from nil t)
>                  (replace-match to t))

That's a perfect example where no matter how many man-years you invest
into a super-duper fancy compiler for Elisp, the above code will not be
affected one bit, because all the time is normally spent within
`re-search-forward` and `replace-match` which are already implemented
in C.

As for whether it can be sped up: yes, no doubt it can.  But how to
speed it up will depend on `from`, on the density of matches in the
text, and on the current major-mode, mostly (the complexity of `from`
impacts the amount of time spent in re-search-forward and which
approaches can reduce it, while the major-mode impacts the overhead of
modifying the buffer (via before/after-change-functions)).

I'd expect that in most cases, there won't be many matches so most of
the time is spent in re-search-forward.  This is already pretty fast in
the case where the pattern is simple, but if the pattern is complex, our
regexp-engine can be somewhere between not very efficient and
pathologically slow.  Fixing this would be great (even better if we can
do it by re-using an existing regexp implementation), especially if we
can do it without slowing down the simple case too much.


        Stefan

Re: Is Elisp really that slow?

[Emanuel Berg]

Stefan Monnier wrote:

>> can this be sped up?
>>
>>  (while (re-search-forward from nil t)
>>    (replace-match to t))
>
> That's a perfect example where no matter how
> many man-years you invest into a super-duper
> fancy compiler for Elisp, the above code will
> not be affected one bit, because all the time
> is normally spent within `re-search-forward`
> and `replace-match` which are already
> implemented in C.

How about this [code last]? I just wrote it.

Comments on code also welcome, as always, but
then perhaps a subject change is called for so
not to enrage anyone for being "OT" (...?).

The goal was

    - case sensitive regexp search

    - interactively, whole buffer if not
      region, otherwise region

    - from Lisp, whole buffer, if no optional
      args delimiting

What in particular will slow it down save for
the same old regexp matcher?

And how do you know what is slow and what
is fast?

And should you really think about that when you
write Elisp?

Anyway:


(defun case-sensitive-regexp-search-and-replace (regexp to-string &optional start stop)
  (interactive
   `(,(read-from-minibuffer "regexp: ")
     ,(read-from-minibuffer "replace: ")
     ,@(if (use-region-p) (list (region-beginning) (region-end))
         (list (point-min) (point-max))) ))
  (save-excursion
    (let ((beg (or start (point-min)))
          (end (or stop  (point-max))) )
      (goto-char beg)
      (let ((case-fold-search nil))
        (while (re-search-forward regexp end t)
          (replace-match to-string) )))))
(defalias 'cs-replace #'case-sensitive-regexp-search-and-replace)


File: https://dataswamp.org/~incal/emacs-init/edit.el

--
underground experts united
http://user.it.uu.se/~embe8573

Re: Is Elisp really that slow?

[Emanuel Berg]

Stefan Monnier wrote:

>> can this be sped up?
>>
>>                (while (re-search-forward from nil t)
>>                  (replace-match to t))
>
> That's a perfect example where no matter how
> many man-years you invest into a super-duper
> fancy compiler for Elisp, the above code will
> not be affected one bit, because all the time
> is normally spent within `re-search-forward`
> and `replace-match` which are already
> implemented in C.

Wait, if so, isn't this the solution to the
whole problem?

1) Identify what is slow in Elisp.

2) Rewrite it in C?

What am I missing?

Is there some things that are slow _and_ cannot
easily be rewritten in C?

Or is there an Elisp-to-C transition penalty?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Is Elisp really that slow?

[Eli Zaretskii]

> From: Emanuel Berg <moasenwood@zoho.eu>
> Date: Mon, 13 May 2019 03:27:54 +0200
> 
> Eli Zaretskii wrote:
> 
> > I agree that our situation with IDE features
> > is quite bleak
> 
> What IDE features are those? (Not
> a rhetorical question.)

Out of the important features listed in

  https://en.wikipedia.org/wiki/Integrated_development_environment

we completely lack support for code completion and refactoring. Our
debugging support is only up to speed with GDB, which means we are
limited in the programming languages we can support reasonably well.
(And even gdb-mi is just good, e.g., it cannot display complex
structures unless you install Python plugins; and its development
stagnated).  Our code search capabilities are rudimentary, and don't
support well modern languages like Java and C++.  So 4 out of 6
important features are either missing or have serious deficiencies.

Sure, add-on packages are available to provide some of the missing
features, but they don't always work well together, and ion any case,
Emacs should have these features built-in, if we want (and we do want)
to claim the title of programmer's editor.

> Actually in the literal sense of the word
> (acronym) I think Emacs is the _ultimate_ IDE
> for doing anything and everything with
> a computer.

In theory.  But not in practice, not at this time.

> And if nobody has or is, why care/complain
> about it? Let it be, man.

Because we want to be "the ultimate IDE".

Re: Is Elisp really that slow?

[Emanuel Berg]

Eli Zaretskii wrote:

> Out of the important features listed in
>
>   https://en.wikipedia.org/wiki/Integrated_development_environment
>
> we completely lack support for code
> completion and refactoring. Our debugging
> support is only up to speed with GDB, which
> means we are limited in the programming
> languages we can support reasonably well.

OK, so if I understand you correctly we are not
happy to be just an editor that writes code and
then relies on other program to the rest, like
compiling and debugging, only we do actually
want that, but we want it to be
more integrated?

And we want more language-specific features
like I guess Java has with Eclipse or C# with
.NET/Mono?

And we also disregard things like Dired, Gnus,
ERC, Emacs-w3m, and what have you, which are
a great help when developing... if you don't
talk to much instead of working, that is :$

We also disregard that when you work on
a project there are often several languages
involved, documentation, a home page, and so on
and we have all that all in one house with the
same finger-habits, help system, and
extension/configuration interface.

I think the whole (point) with Emacs is that it
is ready for everything, with much the same
approach. Want to learn a new language?
You can start immediately, as you don't have to
learn a new editor or "IDE" first!

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Is Elisp really that slow?

[John Yates]

On Mon, May 13, 2019 at 10:38 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> Our> debugging support is only up to speed with GDB ...

Separate from the display limitations Eli mentions a bit later
I would argue that the experience of debugging using gdb-mi with
gdb-many-windows set is quite painful.  At work various of my
team members use gdb via VSCode.  On unit and package tests the
performance is comparable.  Once one progresses to the company's
enormous full application the performance difference is startling.

/john

Re: Is Elisp really that slow?

[Eli Zaretskii]

> From: John Yates <john@yates-sheets.org>
> Date: Mon, 13 May 2019 11:02:41 -0400
> Cc: help-gnu-emacs@gnu.org
> 
> I would argue that the experience of debugging using gdb-mi with
> gdb-many-windows set is quite painful.

So even what I thought was still "good enough" isn't, really...
Hardly surprising, given that it's unmaintained for the last few
years.

> At work various of my
> team members use gdb via VSCode.  On unit and package tests the
> performance is comparable.  Once one progresses to the company's
> enormous full application the performance difference is startling.

Is this perhaps due to parsing the GDB/MI output?  Did you try to
profile that case?  If it's due to parsing, maybe switching it to use
the built-in JSON available with Emacs 27 could help (assuming that
profiling points to that as a hot spot)?

Re: Is Elisp really that slow?

[Eli Zaretskii]

> From: Emanuel Berg <moasenwood@zoho.eu>
> Date: Mon, 13 May 2019 17:00:01 +0200
> 
> Eli Zaretskii wrote:
> 
> > Out of the important features listed in
> >
> >   https://en.wikipedia.org/wiki/Integrated_development_environment
> >
> > we completely lack support for code
> > completion and refactoring. Our debugging
> > support is only up to speed with GDB, which
> > means we are limited in the programming
> > languages we can support reasonably well.
> 
> OK, so if I understand you correctly we are not
> happy to be just an editor that writes code and
> then relies on other program to the rest, like
> compiling and debugging, only we do actually
> want that, but we want it to be
> more integrated?

No, you misunderstood.  We want these features to be as programmers
expect nowadays.  Whether some of the features depend on other
programs or not is immaterial, but if they do, the respective Emacs
front-ends should provide what the users expect.

GDB is not an IDE, it's a debugging engine.  It's okay to make an IDE
whose debugging facilities use GDB as their engine, but the front end
needs to provide the functionalities expected from debugging function
in an IDE.  And if GDB doesn't support some popular language, like
Python, we need an alternative for those who want to develop Python
programs; we don't want to tell them to go to the shell and invoke pdb
as a console program.

> And we want more language-specific features
> like I guess Java has with Eclipse or C# with
> .NET/Mono?

They are not language-specific features, they are in general features
needed in any language.

> And we also disregard things like Dired, Gnus,
> ERC, Emacs-w3m, and what have you, which are
> a great help when developing... if you don't
> talk to much instead of working, that is :$
> 
> We also disregard that when you work on
> a project there are often several languages
> involved, documentation, a home page, and so on
> and we have all that all in one house with the
> same finger-habits, help system, and
> extension/configuration interface.

No, we don't disregard anything.  But having Dired or the other
niceties is little comfort if what you need is to refactor your code
or debug it.

> I think the whole (point) with Emacs is that it
> is ready for everything, with much the same
> approach. Want to learn a new language?
> You can start immediately, as you don't have to
> learn a new editor or "IDE" first!

What if I don't want to learn a language, but to program in it?
I want the tools programmers nowadays expect to have from an
environment that claims to be for programmers.

Re: Is Elisp really that slow?

[Van L]

Emanuel Berg writes:

> Eli Zaretskii writes:
>
> Actually in the literal sense of the word
> (acronym) I think Emacs is the _ultimate_ IDE
> for doing anything and everything with
> a computer.
>
>8 [snip]
>
> And if nobody has or is, why care/complain
> about it? Let it be, man.

There's the `Crit 1' need for a new
interdisciplinarians's field, you name it
`machine behavior,' that affords Emacs the opportunity to be the fittest
naturally in the niche.

-- 
© 2019 VanL
gpg using EEF2 37E9 3840 0D5D 9183  251E 9830 384E 9683 B835
                    "I had to learn through trial and error" - Takao Morita

Re: Is Elisp really that slow?

[Dmitry Gutov]

On 13.05.2019 17:38, Eli Zaretskii wrote:
> Out of the important features listed in
> 
>    https://en.wikipedia.org/wiki/Integrated_development_environment
> 
> we completely lack support for code completion and refactoring.

JFYI: Eglot, published in GNU ELPA, is a an adequate LSP client, and it 
integrates with Company (for completion) and Flymake (to show syntax 
errors, missing imports, etc). Most of the time the user just needs to 
install a corresponding language server program.

It doesn't offer as comprehensive set of refactorings as the 
contemporary Java IDEs, but it does support some.

> Our code search capabilities are rudimentary, and don't
> support well modern languages like Java and C++.

See above. There are language servers for Java and C++.

> Sure, add-on packages are available to provide some of the missing
> features, but they don't always work well together, and ion any case,
> Emacs should have these features built-in, if we want (and we do want)
> to claim the title of programmer's editor.

I think we'll get there after we figure out how to bundle some ELPA 
packages together with the core for releases.

Re: Is Elisp really that slow?

[Paul W. Rankin]

On Tue, May 14 2019, Eli Zaretskii wrote:

> Emacs should have these features built-in, if we want (and we do want) 
> to claim the title of programmer's editor.

<aside>
Just because it sometimes gets overlooked, I think it's worth not losing 
sight of Emacs as a text editor for non-programmers also.
</aside>

--
https://www.paulwrankin.com

Re: Is Elisp really that slow?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 498 bytes --]

On Mon, May 13, 2019 at 04:33:12PM +0200, Emanuel Berg wrote:

[...]

> 1) Identify what is slow in Elisp.
> 
> 2) Rewrite it in C?

3) Write a regexp-to-assembler compiler. Bonus points if that runs
on the GPU, using its SIMD features.

(Cuz the slow parts upthread were the regexp parts anyway, which
are implemented in a sub-language independent of Elisp and embedded
in C).

Caveat: the regexp compiler already does parts of this. Perhaps
not always optimally.

Cheers
-- t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Is Elisp really that slow?

[Emanuel Berg]

Eli Zaretskii wrote:

>> OK, so if I understand you correctly we are
>> not happy to be just an editor that writes
>> code and then relies on other program to the
>> rest, like compiling and debugging, only we
>> do actually want that, but we want it to be
>> more integrated?
>
> No, you misunderstood. We want these features
> to be as programmers expect nowadays.
> Whether some of the features depend on other
> programs or not is immaterial, but if they
> do, the respective Emacs front-ends should
> provide what the users expect.
>
> GDB is not an IDE, it's a debugging engine.
> It's okay to make an IDE whose debugging
> facilities use GDB as their engine, but the
> front end needs to provide the
> functionalities expected from debugging
> function in an IDE.

Yes, that's what I meant. I never did lots'a
debugging myself so perhaps I used the
incorrect/misleading words. But that's what
I meant.

> And if GDB doesn't support some popular
> language, like Python, we need an alternative
> for those who want to develop Python
> programs; we don't want to tell them to go to
> the shell and invoke pdb as
> a console program.

Again, that's what I thought I said, or wanted
to say - you are fine with using external
programs to do things, which I think is 100%
the right thing to do (to be), only you want it
integrated with Emacs.

Exactly what kind of integration, I don't know,
as I lack experience doing that even as a user,
_also_ I certainly don't feel there is
_anything wrong at all_ to "to go to the shell
and invoke pdb as a console program." I do such
things all day every day and feel no need to
change that.

>> And we want more language-specific features
>> like I guess Java has with Eclipse or C#
>> with .NET/Mono?
>
> They are not language-specific features, they
> are in general features needed in any language.

OK, FTR "needed" isn't a word to my liking...
But what I mean is they are language-specific
in terms of how one would go about and
implement them. font-lock and `forward-char'
and much of Emacs isn't.

And actually, _some_ of them features must
really be language-specific also in the sense
you refer to (which, I agree, is the correct
literal interpretation of what I said) - right?
Because I can't imagine you want all the same
little helpers for every single language
there is?

>> And we also disregard things like Dired,
>> Gnus, ERC, Emacs-w3m, and what have you,
>> which are a great help when developing... if
>> you don't talk to much instead of working,
>> that is :$
>> 
>> We also disregard that when you work on
>> a project there are often several languages
>> involved, documentation, a home page, and so
>> on and we have all that all in one house
>> with the same finger-habits, help system,
>> and extension/configuration interface.
>
> No, we don't disregard anything. But having
> Dired or the other niceties is little comfort
> if what you need is to refactor your code or
> debug it.

Right, but when we compare our editor to the
supposedly superior ones [1], I mean. They are
perhaps better IDEs in terms of
_a single language_, but we are the better
(the best?) "IDE" in terms of the
whole computer!

No doubt they will have an advantage for their
particular language as they put all their
effort into that and only that. I think that is
completely natural: a young guy can be in
terrific shape mentally and physically but not
one such guy on the whole geosphere can be the
best in every single Olympic discipline.
Impossible!

>> I think the whole (point) with Emacs is that
>> it is ready for everything, with much the
>> same approach. Want to learn a new language?
>> You can start immediately, as you don't have
>> to learn a new editor or "IDE" first!
>
> What if I don't want to learn a language, but
> to program in it? I want the tools
> programmers nowadays expect to have from an
> environment that claims to be
> for programmers.

I never used those tools so I don't know what
I'm missing. Programming in Emacs I've done in
some 20ish languages with no problems - not
form Emacs, at least :)

Sure, I didn't do it with a deadline, a 40+h
a work week, a deadline, and a I don't know how
many digit salary. And this is probably part of
the difference. I don't really care about being
the best. I want to be something I like and
enjoy. Emacs is certainly good enough for that.
But I respect your efforts and different POV,
for sure, and God willing I can find a spot to
help you some day.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Is Elisp really that slow?

[Emanuel Berg]

Paul W. Rankin wrote:

>> Emacs should have these features built-in,
>> if we want (and we do want) to claim the
>> title of programmer's editor.
>
> Just because it sometimes gets overlooked,
> I think it's worth not losing sight of Emacs
> as a text editor for non-programmers also.

No one is forgetting about that, here the topic
is the speed of Elisp and the lack of some
programmer editor's features tho :)

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Is Elisp really that slow?

[Emanuel Berg]

tomas wrote:

>> 1) Identify what is slow in Elisp.
>> 
>> 2) Rewrite it in C?
>
> 3) Write a regexp-to-assembler compiler.
> Bonus points if that runs on the GPU, using
> its SIMD features.

Please DIY Mr. Zerolo, personally I'd be happy
to try out translating Elisp into C.

I could start with some of my own Elisp,
I mean, I certainly have enough of it...

I know how to write C in general, but how does
one integrate it into Emacs?

Also, I suppose you just don't write the same
Elisp program, only this time in C, but instead
there are certain building blocks that should
correspond to certain things? Not all of it,
but to some extent?

The easiest way to understand this is probably
to see one Elisp defun, and then see the
C version, and then if there is a table
somewhere with the specific building blocks
and conventions all mentioned?

Or if there's a tutorial ~"How to become an
Emacs C programmer" for those who already know,
imperfectly of course but still, Elisp and C?

At this point I'm not talking about changing
actual Emacs, I just want to try it out for my
own purposes.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Is Elisp really that slow?

[Stefan Monnier]

Óscar Fuentes <ofv@wanadoo.es> writes:
> I've been advocating this Elisp-ish C for a long time.  Even on your
> stint as maintainer I offered my implementation of my own C-with-sexps
> language.

Damn!  That doesn't ring a bell :-(

Óscar Fuentes <ofv@wanadoo.es> writes:
> I'm interested on islotating certain parts of Elisp, add some stuff to
> help the optimizers and feed it to LLVM. Ideally, an Elisp hacker
> working within those limits could implement some cpu-intensive code that
> would perform similar to C with minimal retraining. At the same time,
> the language must retain enough expressiveness to bring significant
> advantages on development over C. defmacro is a must.

Emanuel Berg <moasenwood@zoho.eu> writes:
> Yes, I thought about this as well! That would
> be so cool and, as you say, interesting!
> Only one would keep the Lisp syntax, right?

Of course, there's the question of how what this language should look
like, but I think more importantly, there's the question of how its
compiler is used:
- can any random Emacs user download your elisp-ish-c code and run it
  with the same ease as if it were written in Elisp?
- I.e. does it require re-starting Emacs?
  Does it require installing a C compiler or some such external tool?
  Does it work on all the architectures supported by Emacs?
- can incorrect elisp-ish-c code cause your Emacs to seg-fault?


        Stefan

Re: Is Elisp really that slow?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1637 bytes --]

On Tue, May 14, 2019 at 03:21:54PM +0200, Emanuel Berg wrote:
> tomas wrote:
> 
> >> 1) Identify what is slow in Elisp.
> >> 
> >> 2) Rewrite it in C?
> >
> > 3) Write a regexp-to-assembler compiler.
> > Bonus points if that runs on the GPU, using
> > its SIMD features.
> 
> Please DIY Mr. Zerolo, personally I'd be happy
> to try out translating Elisp into C.

But regexp match /was/ the bottleneck in the code sample
you posted, whose core loop is:

  (while (re-search-forward regexp end t)
          (replace-match to-string) )

(go benchmark it, Emacs offers great facilities for that).
So no amount of Elisp -> C translation or other Elisp
speedup will make your code noticeably faster.

[...]

> I know how to write C in general, but how does
> one integrate it into Emacs?

Again, Emacs is happy to tell you. Just do C-f (aka
describe-function), enter re-search-forward <ENTER> and
you get a small blurb:

    re-search-forward is an interactive built-in function in
    ‘C source code’.

    (re-search-forward REGEXP &optional BOUND NOERROR COUNT)

    Probably introduced at or before Emacs version 19.29.

    Search forward from point for regular expression REGEXP.
    [...more interesting text here...]

Notice how this ‘C source code’ above is a link? Follow it,
either by putting point on it and hitting ENTER or by mouse-
clicking on it, and presto, you are in the middle of a C
function written to be called from Elisp (you compile your
Emacs from source code, do you?).

In the manual, there is "E7. Writing Emacs Primitives".

Have fun...

cheers
-- t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Is Elisp really that slow?

[Óscar Fuentes]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Of course, there's the question of how what this language should look
> like, but I think more importantly, there's the question of how its
> compiler is used:
> - can any random Emacs user download your elisp-ish-c code and run it
>   with the same ease as if it were written in Elisp?

See below.

> - I.e. does it require re-starting Emacs?

I hope not.

>   Does it require installing a C compiler or some such external tool?

No. Just LLVM as a library.

>   Does it work on all the architectures supported by Emacs?

It would work on all architectures supported by LLVM.

> - can incorrect elisp-ish-c code cause your Emacs to seg-fault?

For code that the user can download or write himself, this is
undesirable and therefore automatic checks must be added. For code
written by the Emacs developers and meant to compete with C, explicit
checks make sense.

One thing that cannot be left out from this C-ish Elisp is garbage
collection, else the language would have a very different runtime model.
This is probably the most difficult part because GC is something that
you must add on top of LLVM.

Re: Is Elisp really that slow?

[Eli Zaretskii]

> From: Emanuel Berg <moasenwood@zoho.eu>
> Date: Tue, 14 May 2019 13:54:15 +0200
> 
> _also_ I certainly don't feel there is
> _anything wrong at all_ to "to go to the shell
> and invoke pdb as a console program."

Modern debugging front-ends show multiple windows every time the
debuggee stops: a window with the source, another with local
variables, yet another with callstack (a.k.a. "backtrace"), one more
with program input/output, optionally threads and registers, etc.
Displaying that on the console requires the user to type separate
commands for each display, which is time consuming, and also pushes
the other stuff off the visible portion of the console window (so you
need to retype the same commands, or scroll the window with the mouse,
to see that again).  So using a debugger from a console is much less
convenient, when you deal with a complex program, and users nowadays
expect much more from an IDE.

> >> And we want more language-specific features
> >> like I guess Java has with Eclipse or C#
> >> with .NET/Mono?
> >
> > They are not language-specific features, they
> > are in general features needed in any language.
> 
> OK, FTR "needed" isn't a word to my liking...
> But what I mean is they are language-specific
> in terms of how one would go about and
> implement them. font-lock and `forward-char'
> and much of Emacs isn't.

Actually, font-lock needs to be customized for each language, and
there are commands that move by functions, blocks, and other lexical
units, which also need to understand the language to some extent.
IOW, someone must write the code to support each language with the
infrastructure provided by Emacs; if we didn't write that, we cannot
claim we support the language "because the tools are there".

> Because I can't imagine you want all the same
> little helpers for every single language
> there is?

Not sure what you mean by "all the same little helpers", but in
general every language needs its own variety of the same "helpers",
yes.

> > No, we don't disregard anything. But having
> > Dired or the other niceties is little comfort
> > if what you need is to refactor your code or
> > debug it.
> 
> Right, but when we compare our editor to the
> supposedly superior ones [1], I mean. They are
> perhaps better IDEs in terms of
> _a single language_, but we are the better
> (the best?) "IDE" in terms of the
> whole computer!

This argument doesn't fly with users, because newcomers usually need
just one language, but they need a good support for it.  They will go
away if we don't have it, and telling them we support a dozen others
will not make them change their minds.

To keep Emacs alive and kicking for the observable future, we need to
be sure we will have enough developers and contributors.  And since
developers and contributors start as users, we want to attract new
users.  If we fail to attract them, we will quite literally lose our
future.

> I never used those tools so I don't know what
> I'm missing. Programming in Emacs I've done in
> some 20ish languages with no problems - not
> form Emacs, at least :)
> 
> Sure, I didn't do it with a deadline, a 40+h
> a work week, a deadline, and a I don't know how
> many digit salary. And this is probably part of
> the difference.

I think a more important factor is the size of the project you work
on.

Re: Is Elisp really that slow?

[Emanuel Berg]

Eli Zaretskii wrote:

> Modern debugging front-ends show multiple
> windows every time the debuggee stops:
> a window with the source, another with local
> variables, yet another with callstack (a.k.a.
> "backtrace"), one more with program
> input/output, optionally threads and
> registers, etc. Displaying that on the
> console requires the user to type separate
> commands for each display, which is time
> consuming, and also pushes the other stuff
> off the visible portion of the console window
> (so you need to retype the same commands, or
> scroll the window with the mouse, to see that
> again). So using a debugger from a console is
> much less convenient, when you deal with
> a complex program, and users nowadays expect
> much more from an IDE.

OK, point taken. Users, newcomers, aren't
impressed by Emacs. The don't dig the console
and they use the mouse. And they want
everything integrated, all tho, which is
amusing, integration is limited to
a single language.

(Maybe MS IDEs can do all of C#, VB(A), and
MS Access now that I think about it...?)

> Actually, font-lock needs to be customized
> for each language, and there are commands
> that move by functions, blocks, and other
> lexical units, which also need to understand
> the language to some extent. IOW, someone
> must write the code to support each language
> with the infrastructure provided by Emacs; if
> we didn't write that, we cannot claim we
> support the language "because the tools are
> there".

OK, now it gets totally confused %) The
different language major modes are the absolute
_backbone_ of Emacs as a programmer editor,
indispensible! Obviously we want them for every
conceivable language to be as good as possible!
Setting up font-lock and indentation for
a programming language major mode (which even
I have done BTW) isn't what I thought we
were talking about rather much more advanced
stuff, the refactoring stuff (how ever that
works), integrating the debugger and build
process, and so on, and as for
language-specifics, qualitatively different
stuff, not just configuring and tweaking with
the same old interface!
<http://user.it.uu.se/~embe8573/fps/>

> This argument doesn't fly with users, because
> newcomers usually need just one language, but
> they need a good support for it. They will go
> away if we don't have it, and telling them we
> support a dozen others will not make them
> change their minds.
>
> To keep Emacs alive and kicking for the
> observable future, we need to be sure we will
> have enough developers and contributors.
> And since developers and contributors start
> as users, we want to attract new users. If we
> fail to attract them, we will quite literally
> lose our future.

OK, good point.

>> I never used those tools so I don't know
>> what I'm missing. Programming in Emacs I've
>> done in some 20ish languages with no
>> problems - not form Emacs, at least :)
>> 
>> Sure, I didn't do it with a deadline, a 40+h
>> a work week, a deadline, and a I don't know
>> how many digit salary. And this is probably
>> part of the difference.
>
> I think a more important factor is the size
> of the project you work on.

Well, obviously you worked on much bigger
projects than I, but I did a university project
(check out the URL in my signature). The report
- compiled LaTeX and Biblatex, which I wrote in
Emacs - is 153 pages. I used C++, Lisp, zsh,
gnuplot, groff, and pic(1), and wrote all of
that in Emacs. Even the Makefile and several
textfiles :)

So how large should a project be before Emacs
becomes insufficient? A Quake? If so, then no,
regretfully I wasn't part of the team...

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Is Elisp really that slow?

[Eli Zaretskii]

> From: Emanuel Berg <moasenwood@zoho.eu>
> Date: Tue, 14 May 2019 19:05:21 +0200
> 
> So how large should a project be before Emacs
> becomes insufficient?

It depends on the language, and I don't really know where to draw the
line.  On my daytime job I deal with C++ projects that have a few
million lines of code, and Emacs OOB is only borderline useful there.

Re: Is Elisp really that slow?

[Emanuel Berg]

tomas wrote:

>>>> 1) Identify what is slow in Elisp.
>>>> 
>>>> 2) Rewrite it in C?
>>>
>>> 3) Write a regexp-to-assembler compiler.
>>>    Bonus points if that runs on the GPU,
>>>    using its SIMD features.
>> 
>> Please DIY Mr. Zerolo, personally I'd be
>> happy to try out translating Elisp into C.
>
> But regexp match /was/ the bottleneck in the
> code sample you posted, whose core loop is:
>
>   (while (re-search-forward regexp end t)
>           (replace-match to-string) )
>
> (go benchmark it, Emacs offers great
> facilities for that). So no amount of Elisp
> -> C translation or other Elisp speedup will
> make your code noticeably faster.

:) I believe you! It is not that. Personally,
I don't really care that much for Emacs' speed.
I don't really care if other people use it or
not [1]. I mean, I'd _like_ it to be faster and for
more people to use it. But it doesn't bug me on
a daily basis, if you get me.

So from a personal POV, I would be
interesting/fun to write some of my Elisp into
C, and if I get good enough at it, maybe
I could even do it for the project.

You know my Elisp, hardly brilliant but
capable, and I think my C would be good again
if I picked it up after not doing it for
many years.

So the practical issue is how to do it not in
the Elisp sense, not in the C sense, but in the
_Elisp-to-C sense in the context of Emacs_.
And for that purpose, the Elisp I posted and
you referred to (thank you) is too complicated!
Baby steps first.

So, how do I write this in C with correct
conventions?

(defun hello-elisp-world ()
  "This docstring is for display purposes only."
  (interactive)
  (message "hello Elisp world!")
  )
;; (hello-elisp-world) ; eval me, otherwise you don't know what it does

Them, where do I put it (assuming it gets its
own file for test purposes) and do I need to
change the Makefile or build process in general
to make it work?

> Notice how this ‘C source code’ above is
> a link? Follow it, either by putting point on
> it and hitting ENTER or by mouse- clicking on
> it, and presto, you are in the middle of
> a C function written to be called from Elisp

Yes, I know about that, of course :) I think
you suspected that, but thanks anyway as I'm
not the only one reading this...

> (you compile your Emacs from source code, do
> you?).

I don't because I didn't get Emacs-w3m to work
last time I tried, and I always prefered the
repos, anyway. But installation of Emacs itself
was amazingly smooth. I solved the problem with
Emacs-w3m with "apt pinning", which I described
here: [2]

If your comment refers to the Emacs C source,
one can set that up even tho one has Emacs from
the repos. Apparently now it isn't for me, as
it doesn't work, but I have done it before, so
I know it's possible :) If your comment refers
to ~"make a manual install and start mucking
around", I can do that as well :)

> In the manual, there is "E7. Writing Emacs
> Primitives".

You mean this one:

    (info "(elisp) Writing Emacs Primitives")

?


[1] "I don't really care if other people use it
    or not" - this sounds like... something
    that isn't my impression, that other people
    don't use it! Because almost _every_
    programmer I meet on the street and we
    start to talk about our stuff, they use
    some Unix system, most often Linux,
    sometimes BSD. "What editor do you use?"
    "Emacs." ~2/3 times. Only perhaps they can
    read my mind and give me the answer I so
    much desire... Wait! I just said I didn't
    care! What the heck am I talking about?!?

[2] http://lists.gnu.org/archive/html/help-gnu-emacs/2019-04/msg00261.html

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Is Elisp really that slow?

[Emanuel Berg]

Eli Zaretskii wrote:

>> So how large should a project be before
>> Emacs becomes insufficient?
>
> It depends on the language, and I don't
> really know where to draw the line. On my
> daytime job I deal with C++ projects that
> have a few million lines of code, and Emacs
> OOB is only borderline useful there.

And what makes it not useful is the lack of
refactoring, and poor integration with
debugging and the build process?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Is Elisp really that slow?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 3624 bytes --]

On Wed, May 15, 2019 at 01:25:25PM +0200, Emanuel Berg wrote:
> tomas wrote:

[...]

> :) I believe you! It is not that. Personally,
> I don't really care that much for Emacs' speed.

Me neither. But some might...

> I don't really care if other people use it or
> not [1]. I mean, I'd _like_ it to be faster and for
> more people to use it. But it doesn't bug me on
> a daily basis, if you get me.
> 
> So from a personal POV, I would be
> interesting/fun to write some of my Elisp into
> C, and if I get good enough at it, maybe
> I could even do it for the project.
> 
> You know my Elisp, hardly brilliant but
> capable, and I think my C would be good again
> if I picked it up after not doing it for
> many years.
> 
> So the practical issue is how to do it not in
> the Elisp sense, not in the C sense, but in the
> _Elisp-to-C sense in the context of Emacs_.
> And for that purpose, the Elisp I posted and
> you referred to (thank you) is too complicated!
> Baby steps first.
> 
> So, how do I write this in C with correct
> conventions?
> 
> (defun hello-elisp-world ()
>   "This docstring is for display purposes only."
>   (interactive)
>   (message "hello Elisp world!")
>   )
> ;; (hello-elisp-world) ; eval me, otherwise you don't know what it does
> 
> Them, where do I put it (assuming it gets its
> own file for test purposes) and do I need to
> change the Makefile or build process in general
> to make it work?

A bit pressed at the moment to provide you with
a working example, but just following the forward-char
pattern might get you far. But see below...

> > Notice how this ‘C source code’ above is
> > a link? Follow it, either by putting point on
> > it and hitting ENTER or by mouse- clicking on
> > it, and presto, you are in the middle of
> > a C function written to be called from Elisp
> 
> Yes, I know about that, of course :) I think
> you suspected that, but thanks anyway as I'm
> not the only one reading this...

Great :)

The point I wanted to make is that Emacs has been
very good at keeping itself hackable, and this seems
a more important metrics than raw speed, given Emacs's
"mission".

> > (you compile your Emacs from source code, do
> > you?).
> 
> I don't because I didn't get Emacs-w3m to work
> last time I tried, and I always prefered the
> repos, anyway. But installation of Emacs itself
> was amazingly smooth. I solved the problem with
> Emacs-w3m with "apt pinning", which I described
> here: [2]

Hm. I see. Did you file a bug report?

> If your comment refers to the Emacs C source,
> one can set that up even tho one has Emacs from
> the repos. Apparently now it isn't for me, as
> it doesn't work, but I have done it before, so
> I know it's possible :) If your comment refers
> to ~"make a manual install and start mucking
> around", I can do that as well :)

I think I dimly remember something in this list.

> > In the manual, there is "E7. Writing Emacs
> > Primitives".
> 
> You mean this one:
> 
>     (info "(elisp) Writing Emacs Primitives")

Yes. Following this has the downside that you end
up with a "modified Emacs". Thus, perhaps "E8 Writing
Dynamically-Loaded Modules", where your C functions
end up in a separate library, loaded by Emacs at
run time, is more interesting.

That said, it is always a trade-off: stuff written
in Elisp is far more hackable, so writing extensions
in C takes a lot of consideration and a good interface
design (how could people want to use my new function?),
given the "hackability cost".

Cheers
-- tomás

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Is Elisp really that slow?

[Eli Zaretskii]

> From: Emanuel Berg <moasenwood@zoho.eu>
> Date: Wed, 15 May 2019 13:27:31 +0200
> 
> > It depends on the language, and I don't
> > really know where to draw the line. On my
> > daytime job I deal with C++ projects that
> > have a few million lines of code, and Emacs
> > OOB is only borderline useful there.
> 
> And what makes it not useful is the lack of
> refactoring, and poor integration with
> debugging and the build process?

Debugging works well enough, but code search is very simplistic
(brings a lot of false positives, because it's basically dumb text
search, not sensitive enough to the language), and refactoring doesn't
work at all, because we have no facilities for that.  For small
projects, you could use 'A' or 'Q' in Dired, but that quickly becomes
unworkable in large projects.

Re: Is Elisp really that slow?

[Emanuel Berg]

tomas wrote:

> The point I wanted to make is that Emacs has
> been very good at keeping itself hackable,
> and this seems a more important metrics than
> raw speed, given Emacs's "mission".

To make Emacs less hackable isn't allowed :)

And Emacs in general is already fast. I do
things much faster than most people who use
modern GUIs. I type faster, I switch between
programs faster, everything is more integrated,
and so on - it is fast!

That doesn't mean I do more productive stuff
for every given computer hour. Sometimes it
works the opposite way, one discovers some
detail, gets an idea, and spend an hour on some
thing which really doesn't matter. But it is
fun and rewarding, so in a way it does. It is
meaningless, but if it's meaningless, one might
as well do it :)

But if we return to (in your words)
"raw speed", I like that as well, so it ain't
to dismiss the discussion.

(BTW did the subject change or am I tripping?!?)

> Hm. I see. Did you file a bug report?

I don't remember the details but it was solved
with the help from the gmane.emacs.w3m
guys, yes.

>> You mean this one:
>> 
>>     (info "(elisp) Writing Emacs Primitives")
>
> Yes. Following this has the downside that you
> end up with a "modified Emacs".

Right, but as you still need to recompile it,
you have no choice anyway, you must get the
source and build it manually, right?

But, one can use the repo Emacs for everyday
purposes, and then have one other to experiment
on.

Perhaps that gets tangled up in the initial
phase but I'm sure it gets worked
out eventually.

> Thus, perhaps "E8 Writing Dynamically-Loaded
> Modules", where your C functions end up in
> a separate library, loaded by Emacs at run
> time, is more interesting.

Aha, so you can do that! Wonderful!

Only... I don't find it?

;; DNC
(info "(emacs) Writing Dynamically-Loaded Modules")
(info "(elisp) Writing Dynamically-Loaded Modules")

Where is it?

And what is this "E8" notation?

> That said, it is always a trade-off: stuff
> written in Elisp is far more hackable, so
> writing extensions in C takes a lot of
> consideration and a good interface design
> (how could people want to use my new
> function?), given the "hackability cost".

Well, I've written enough Elisp by now anyway.
Time to move on :)

-- 
underground experts united
http://user.it.uu.se/~embe8573
https://dataswamp.org/~incal

Re: Is Elisp really that slow?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 749 bytes --]

On Thu, May 16, 2019 at 01:02:36AM +0200, Emanuel Berg wrote:
> tomas wrote:

[...]

> > Thus, perhaps "E8 Writing Dynamically-Loaded
> > Modules", where your C functions end up in
> > a separate library, loaded by Emacs at run
> > time, is more interesting.
> 
> Aha, so you can do that! Wonderful!
> 
> Only... I don't find it?

Ah, sorry. Dynamically loaded modules were added to Emacs on version
25, I think. What version are you using?

> And what is this "E8" notation?

This is just the section numbering.

> > That said, it is always a trade-off: stuff
> > written in Elisp is far more hackable [...]

> Well, I've written enough Elisp by now anyway.
> Time to move on :)

Enjoy the landscape :-)

Cheers
-- t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Is Elisp really that slow?

[Noam Postavsky]

On Thu, 16 May 2019 at 02:48, <tomas@tuxteam.de> wrote:

> > > Thus, perhaps "E8 Writing Dynamically-Loaded
> > > Modules", where your C functions end up in
> > > a separate library, loaded by Emacs at run
> > > time, is more interesting.
> >
> > Aha, so you can do that! Wonderful!
> >
> > Only... I don't find it?
>
> Ah, sorry. Dynamically loaded modules were added to Emacs on version
> 25, I think.

The docs were added only in Emacs 26, I believe.

Re: Is Elisp really that slow?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 758 bytes --]

On Thu, May 16, 2019 at 05:37:03AM -0400, Noam Postavsky wrote:
> On Thu, 16 May 2019 at 02:48, <tomas@tuxteam.de> wrote:
> 
> > > > Thus, perhaps "E8 Writing Dynamically-Loaded
> > > > Modules", where your C functions end up in
> > > > a separate library, loaded by Emacs at run
> > > > time, is more interesting.
> > >
> > > Aha, so you can do that! Wonderful!
> > >
> > > Only... I don't find it?
> >
> > Ah, sorry. Dynamically loaded modules were added to Emacs on version
> > 25, I think.
> 
> The docs were added only in Emacs 26, I believe.

Oh, I see. Emanuel: in the meantime, here's the online ref:

  https://www.gnu.org/software/emacs/manual/html_node/elisp/Writing-Dynamic-Modules.html#Writing-Dynamic-Modules

Enjoy
-- t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Is Elisp really that slow?

[Eli Zaretskii]

> From: Emanuel Berg <moasenwood@zoho.eu>
> Date: Thu, 16 May 2019 01:02:36 +0200
> 
> > Thus, perhaps "E8 Writing Dynamically-Loaded
> > Modules", where your C functions end up in
> > a separate library, loaded by Emacs at run
> > time, is more interesting.
> 
> Aha, so you can do that! Wonderful!
> 
> Only... I don't find it?
> 
> ;; DNC
> (info "(emacs) Writing Dynamically-Loaded Modules")
> (info "(elisp) Writing Dynamically-Loaded Modules")
> 
> Where is it?

Get a newer Emacs, it's a recent addition to the manual.

> And what is this "E8" notation?

Appendix E, chapter 8.

Re: Is Elisp really that slow?

[Stefan Monnier]

> So from a personal POV, I would be
> interesting/fun to write some of my Elisp into
> C, and if I get good enough at it, maybe
> I could even do it for the project.

FWIW, compiling existing Elisp to C code is not really interesting in
the sense that if it exists as Elisp code it's usually because it's not
very speed-sensitive.

What is more interesting is to take existing C code and see if it can be
rewritten into a kind of Elisp (or other language that has similar
dynamic properties, most importantly the ability to replace code at
run-time) that can be compiled back to efficient code.

After all, the main problem with the part of Emacs that's written in
C is that you can't change it from your ~/.emacs, contrary to all the
Elisp code.


        Stefan

Re: Is Elisp really that slow?

[Emanuel Berg]

Eli Zaretskii wrote:

> Debugging works well enough, but code search
> is very simplistic (brings a lot of false
> positives, because it's basically dumb text
> search, not sensitive enough to the
> language), and refactoring doesn't work at
> all, because we have no facilities for that.
> For small projects, you could use 'A' or 'Q'
> in Dired, but that quickly becomes unworkable
> in large projects.

OK, thanks!

By refactoring, here, I take it you mean
a program that is parsing the code, analyzing
it, and breaking it up, to have neat building
blocks instead of huge functions, but with the
same functionality, and all this without or
with little human intervention?

Well, that sounds... advanced! So this is
something modern IDEs do? :O

To have that for all our programming modes
sounds like a enormous challenge!

The only way I can see it work in practice is
if the refactoring logic could be separated and
be the same for all languages, so one would
just have to have parsing language-specific and
then hook it to the refactoring engine...

-- 
underground experts united
http://user.it.uu.se/~embe8573
https://dataswamp.org/~incal

Re: Is Elisp really that slow?

[Eli Zaretskii]

> From: Emanuel Berg <moasenwood@zoho.eu>
> Date: Fri, 17 May 2019 01:19:42 +0200
> 
> By refactoring, here, I take it you mean
> a program that is parsing the code, analyzing
> it, and breaking it up, to have neat building
> blocks instead of huge functions, but with the
> same functionality, and all this without or
> with little human intervention?

Refactoring is a large topic.  I suggest to do some Internet search
for something like "tools for refactoring programs".  One of the hits
will be this 10-year old paper, which shows some of what I meant in
practice:

  https://pdfs.semanticscholar.org/ce71/00df509f37e4d1c1fc9fa50f2b98ba035ae8.pdf

> Well, that sounds... advanced! So this is
> something modern IDEs do? :O

Some of them, yes.  A good IDE should support at least some of that.

> To have that for all our programming modes
> sounds like a enormous challenge!

The simplest parts can be based on only rudimentary analysis of the
language, something that we already have in every supported language.

> The only way I can see it work in practice is
> if the refactoring logic could be separated and
> be the same for all languages, so one would
> just have to have parsing language-specific and
> then hook it to the refactoring engine...

Yes.

Re: Is Elisp really that slow?

[Ken Goldman]

On 5/12/2019 10:45 AM, Óscar Fuentes wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
> 
>> Most stuff Emacs routinely does is something the other editors could
>> only dream about, so by and large you are comparing apples to peanuts.
> 
> Examples?
> 
> Emacs is amazing, but as far as productivity goes, at least as a
> programmer's editor for popular languages, it is far behind current
> contenders.

I was hooked on emacs the first time I saw keyboard macros.

Is there another editor that can spell check code?

I appreciate that it's multi-OS and multi-language.

I use emacs for bulk coding.  However, I use a language and OS specific 
IDE for debugging.

Re: Is Elisp really that slow?

[Emanuel Berg]

tomas wrote:

> Oh, I see. Emanuel: in the meantime, here's
> the online ref:
>
>   https://www.gnu.org/software/emacs/manual/html_node/elisp/Writing-Dynamic-Modules.html#Writing-Dynamic-Modules

Thanks, I have been reading them for a couple
of days now at a snail's pace. Perhaps it is
time for me to change style a bit and not just
"do" things but try to understand them as
well...

Also, in [1] it says

    Most of the functions described below
    became available in Emacs 25, the first
    Emacs release that supported
    dynamic modules.

And Emacs 25 is what I use! Amazing - this must
be the first time ever I and the time around me
keep an even pace!


[1] https://www.gnu.org/software/emacs/manual/html_node/elisp/Module-Values.html

-- 
underground experts united
http://user.it.uu.se/~embe8573
https://dataswamp.org/~incal

Re: Is Elisp really that slow?

[Emanuel Berg]

Eli Zaretskii wrote:

>> [...] what is this "E8" notation?
>
> Appendix E, chapter 8.

This notation is better:

    (info "(elisp) Integer Basics")

Here is a little helper, I have it bound to
"s", as in source.

(defun info-copy-current-node-name-elisp ()
  "E.g. kill-ring this line: (info \"(gnus) Mail Source Specifiers\")"
  (interactive)
  (Info-copy-current-node-name 0) ) ; [1]


[1] line 18 @ https://dataswamp.org/~incal/emacs-init/my-info.el

-- 
underground experts united
http://user.it.uu.se/~embe8573
https://dataswamp.org/~incal

RE: Is Elisp really that slow?

[Drew Adams]

> Here is a little helper, I have it bound to
> "s", as in source.
> (defun info-copy-current-node-name-elisp ()
>   "..."
>   (interactive)
>   (Info-copy-current-node-name 0))

Good one.

Somewhat related: This command is available in
Info+ (bound in Info to `G').  I use it to
quickly get the URL to paste into web pages etc.

(defun Info-goto-node-web (node &optional flip-new-win)
  "Use `browse-url' to go to Info node NODE using a Web browser.
With a prefix arg, reverse the effect of option
option`browse-url-new-window-flag'.

NODE is the name of a node in the GNU Emacs or Elisp manual.
Alternatively, NODE can have the form (MANUAL)NODE, where MANUAL is
\"emacs\" or \"elisp\" and NODE is the name of the node in that
manual.  Empty NODE in (MANUAL) defaults to the `Top' node."
  (interactive
   (list (Info-read-node-name "Go to node: " Info-current-node)
         current-prefix-arg))
  (require 'browse-url)
  (unless Info-current-file
    (error "This command must be invoked from Info"))
  (browse-url (Info-url-for-node node) 
              (list (if flip-new-win
                        (not browse-url-new-window-flag)
                      browse-url-new-window-flag))))

https://www.emacswiki.org/emacs/InfoPlus

Re: Is Elisp really that slow?

[Eli Zaretskii]

> From: Emanuel Berg <moasenwood@zoho.eu>
> Date: Thu, 23 May 2019 16:28:35 +0200
> 
> Eli Zaretskii wrote:
> 
> >> [...] what is this "E8" notation?
> >
> > Appendix E, chapter 8.
> 
> This notation is better:
> 
>     (info "(elisp) Integer Basics")

It's a question of taste, and other people can legitimately have
different tastes.  For example, if someone prefers to read the manual
in PDF format, your notation is useless.

Re: Is Elisp really that slow?

[Van L]

Emanuel Berg writes:
> Amazing - this must be the
> first time ever I and the
> time around me keep an
> even pace!

Welcome to the simulation.

-- 
© 2019 Van L
gpg using EEF2 37E9 3840 0D5D 9183  251E 9830 384E 9683 B835
          "I don't believe any anything that the media say." - Ali Abdelaziz

Re: Is Elisp really that slow? (was: Why is Elisp slow?)

[Xavier Maillard]

> From: Emanuel Berg <moasenwood@zoho.eu>
> Date: Sun, 12 May 2019 01:09:09 +0200
> 
> Now, while that would be interesting to do,
> I think it's pretty clear Elisp would benefit
> from more speed. Maybe it is not slow for its
> purposes, but it ain't fast either, sure is
> my impression.

I have read pretty much all this thread and as Tomas said it quite
well, there is more than speed. Speaking for myself, speed is not of
any concern here. It is fast enough for everything. Other aspects are
of much priority for me: hackability is one of them, accessibility,
portability, stability, etc.

What would you do with more speed that you do not do today ?

-- 
Xavier Maillard                      
e/j:xavier@maillard.im                w:www.maillard.im
m: 06 49 60 48 56

GPG:                           9983 DCA1 1FAC 8DA7 653A
                               F9AA BA49 09B7 8F04 DE1B

Re: Is Elisp really that slow?

[Emanuel Berg via help-gnu-emacs]

Stefan Monnier wrote:

>> So from a personal POV, I would be
>> interesting/fun to write some of my Elisp
>> into C, and if I get good enough at it,
>> maybe I could even do it for the project.
>
> FWIW, compiling existing Elisp to C code is
> not really interesting in the sense that if
> it exists as Elisp code it's usually because
> it's not very speed-sensitive.

Yeah, I suspected that much. Still, some Elisp
I'm sure I can find somewhere that is
worth translating?

If not, I'll just translate some of my own
stuff. Here, I wonder how to determine what to
optimize, tho? Obviously it should be something
simple that is used often and can benefit from
a speed-up. Is there an automated way to answer
any of those three questions? Or do you
actually have to use your _brain_ and not just
your fingers to do computer work?

Or maybe I'll write something all new in C...

> What is more interesting is to take existing
> C code and see if it can be rewritten into
> a kind of Elisp (or other language that has
> similar dynamic properties, most importantly
> the ability to replace code at run-time) that
> can be compiled back to efficient code.

That's beyond my capabilities at the moment,
I'm afraid...

> After all, the main problem with the part of
> Emacs that's written in C is that you can't
> change it from your ~/.emacs, contrary to all
> the Elisp code.

Is _that_ the problem with Emacs' C? I thought
there wasn't any problem with Emacs' C, we just
needed more to make the program faster?
For simple, fundamental stuff that people don't
configure anyway because of their
low-level-of-complexity nature? You're saying
all of that is already done?

-- 
underground experts united
http://user.it.uu.se/~embe8573
https://dataswamp.org/~incal

Re: Is Elisp really that slow?

[Emanuel Berg via help-gnu-emacs]

Ken Goldman wrote:

> I was hooked on emacs the first time I saw
> keyboard macros.

I consider keyboard macros poor man's
programming as everything you can do with
keyboard macros, you can do with programming,
but not the other way around.

I've seen people use them to align like three
or four liens or something. One should work on
ones typing and cursor moving speed instead of
doing a macro for that, IMO. And if there are
too many lines, Elisp is better than a macro
IMO as the Elisp is a helpful building block
to have. Maybe the same situation arrives
again, only with some detail changed? If you
keep your Elisp, you can solve that as well
with a minimal change to a duplicate function.

> Is there another editor that can spell
> check code?

I certainly think so :)

> I appreciate that it's multi-OS and
> multi-language.

I don't know what editors the other guys are
talking about, the so called competition.
But there are many multi-OS editors.
Multi-language, I don't know. I can only think
of Vim and .NET/Mono which I think are
C#/VB(A)/MS SQL Server and possibly some more.

> I use emacs for bulk coding. [...]

Bulk coding:

    The process of Coding all members of
    a group of Documents (identified, for
    example, by Deduplication,
    Near-Deduplication, Email Threading, or
    Clustering) based on the review of only one
    or a few members of the group.
    Also referred to as Bulk Tagging. [1]

Okaaay...?


[1] https://www.edrm.net/glossary/bulk-coding/
    
-- 
underground experts united
http://user.it.uu.se/~embe8573
https://dataswamp.org/~incal

Re: Is Elisp really that slow?

[Emanuel Berg via help-gnu-emacs]

Eli Zaretskii wrote:

> It's a question of taste, and other people
> can legitimately have different tastes.
> For example, if someone prefers to read the
> manual in PDF format, your notation
> is useless.

Right... *read* the manual. I always forget
about that!

-- 
underground experts united
http://user.it.uu.se/~embe8573
https://dataswamp.org/~incal

Re: Is Elisp really that slow? (was: Why is Elisp slow?)

[Emanuel Berg via help-gnu-emacs]

Xavier Maillard wrote:

> What would you do with more speed that you do
> not do today ?

Because there isn't more speed, your brain
doesn't produce ideas what to do with it.
At least mine doesn't.

But OK, if there was unlimited speed I'd do
a space simulator game where you traveled
across the galaxies all the while controlling,
setting up, and upgrading the space ship with
Emacs Lisp only.

Including the crew:

    (setq number-of-pushups-every-day 45)
    (setq hire-women-only t)

-- 
underground experts united
http://user.it.uu.se/~embe8573
https://dataswamp.org/~incal

Re: Is Elisp really that slow? (was: Why is Elisp slow?)

[Samuel Wales]

On 5/11/19, tomas@tuxteam.de <tomas@tuxteam.de> wrote:
> On Sat, May 11, 2019 at 02:38:58AM +0200, Emanuel Berg wrote:
>> If it is possible to have Elisp as fast as CL
>> or even as fast as C (!), the only problem is
>> no one has been motivated to do it! ???

> One last attempt: this whole thread is based on the premise
> that there is one clear definition of slow. Yet, there isn't.

well, for me as a mere user, i just want elisp [per se] to run faster.
maybe that has 1000 definitions, but the 3 things that i have read in
this thread wfm as PERFECT targets for speeding up and would make
emacs even better for me.

  1] org mode timestamp [aka "daily/weekly"] agenda to complete faster
for a 2 day [that is my normal span] agenda.  if this has more than
one definition, then i impetuously choose all.  :]  n.b. it could in
principle be mostly spread to 1 core per org agenda file as i doubt
it's teh sorting and displaying that takes up the time.  [not
intending to set cat among pigeons on that last one.]

  2] while re-search-forward do replace-match [iirc] [my question
which stefan nicely addressed] to work faster.  this is a tight loop
called frequently.

  3] elisp to be much faster so that new ideas can be done.

by "per se" i mean, perhaps confusingly, mere users program in a lisp,
preferably elisp, as opposed to a strange new syntax or c-like stuff
or elisp-like c.

for my part, i'll gladly program in cl if it is faster [quite similar
to elisp and standard], or even in scheme if needed for tight loops
[different from elisp but lisp].  i will not be thinking about
anything lower level than lisp.  unless somebody says use this
construct.

-- 
The Kafka Pandemic

What is misopathy?
https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html

The disease DOES progress. MANY people have died from it. And ANYBODY
can get it at any time.

Re: Is Elisp really that slow? (was: Why is Elisp slow?)

[Samuel Wales]

On 6/7/19, Samuel Wales <samologist@gmail.com> wrote:
> anything lower level than lisp.  unless somebody says use this
> construct.

for clarity i mean, merely, like, that great blog on efficiency that
said the cl-loop construct is fast.  [i like fp, but i also like
cl-loop.]

Re: Is Elisp really that slow? (was: Why is Elisp slow?)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1550 bytes --]

On Fri, Jun 07, 2019 at 05:26:10PM -0700, Samuel Wales wrote:

[...]

> well, for me as a mere user, i just want elisp [per se] to run faster.
> maybe that has 1000 definitions, but the 3 things that i have read in
> this thread wfm as PERFECT targets for speeding up and would make
> emacs even better for me.

Buy a faster computer, then :-)

>   1] org mode timestamp [aka "daily/weekly"] agenda to complete faster
> for a 2 day [...]

Dunno about that. Perhaps Org has sub-optimal algorithms there?
Why don't you get your hands dirty and profile the whole thing?

>   2] while re-search-forward do replace-match [iirc] [my question
> which stefan nicely addressed] to work faster.  this is a tight loop
> called frequently.

AFAIR the bottleneck here wasn't Elisp at all, but regexp match,
which is written totally... in C (actually there's a mini-language
layer in-between). So the totally wrong target for this thread:
Elisp ain't slow here because it isn't in the equation at all.

Do I remember incorrectly?

>   3] elisp to be much faster so that new ideas can be done.
> 
> by "per se" i mean, perhaps confusingly, mere users program in a lisp,
> preferably elisp, as opposed to a strange new syntax or c-like stuff
> or elisp-like c.

And this is so generic that it's best served by a generic programming
language, which Elisp isn't, and for a good reason.

If you pose a question, you should be ready to expect that the
answer is... "it depends". That's how learning starts :-)

Cheers
-- tomás

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Is Elisp really that slow? (was: Why is Elisp slow?)

[Samuel Wales]

On 6/8/19, tomas@tuxteam.de <tomas@tuxteam.de> wrote:
> On Fri, Jun 07, 2019 at 05:26:10PM -0700, Samuel Wales wrote:
>> well, for me as a mere user, i just want elisp [per se] to run faster.
>> maybe that has 1000 definitions, but the 3 things that i have read in
>> this thread wfm as PERFECT targets for speeding up and would make
>> emacs even better for me.
>
> Buy a faster computer, then :-)

i was going to add somethning like "without buying a supercomputer"
but figured nobody would say to get a new computer.

an osx help channel was once asked by a disabled person what to do to
increase font sizes for the icons.  these were not configurable at
that time.  therefore their response was that there was nothing wrong
with osx, and the obvious solution was always for the user to get new
glasses.  this included all disabled users of all types.

i suspect that if they were configurable, the answer would have been
that obviously they needed to be configurable.

the topic of the thread is elisp speed.  it's true that if a
significantly faster single-core-throughput computer existed and
everybody could have access to it, that would solve the problem.  but
i don't think it is the question that most of us are discussing in
this particular case.  it definitely is not the quesiton i was
addressing.

>>   1] org mode timestamp [aka "daily/weekly"] agenda to complete faster
>> for a 2 day [...]
>
> Dunno about that. Perhaps Org has sub-optimal algorithms there?
> Why don't you get your hands dirty and profile the whole thing?

we discussed this on this thread.  it is optimized to death.  it could
perhaps be rewritten but has not been and that is a large task.  and
no, i am not capable of it.  and no, that is not the question in this
particular case.  the question is, can we speed it up, as the topic of
the thread states, by speeding up elisp.  for example with a faster re
matcher.

also, take a look at the profile.  it's kind of flat.  there isn't
anything that obviously takes up the time.

>
>>   2] while re-search-forward do replace-match [iirc] [my question
>> which stefan nicely addressed] to work faster.  this is a tight loop
>> called frequently.
>
> AFAIR the bottleneck here wasn't Elisp at all, but regexp match,
> which is written totally... in C (actually there's a mini-language
> layer in-between). So the totally wrong target for this thread:
> Elisp ain't slow here because it isn't in the equation at all.
>
> Do I remember incorrectly?

i don't know.  i know your feeling.  the thread talks about
keybindings too and i struggle to figure out that relevance.

to me, you remember incorrectly, but i could be wrong.

i thoght th thread was abuot speeding uyp elisp.  improving the c core
or the bytecode or whatever seems to me a perfect speedup.

but i am a mere user.

>
>>   3] elisp to be much faster so that new ideas can be done.
> And this is so generic that it's best served by a generic programming
> language, which Elisp isn't, and for a good reason.

well, what do i know.  i thought perhaps c could be sped up, the core
could replaced with a really optimized lisp, a fwe new elisp
constructs with opimized c like a new re engine could be introduced,
etc.

>
> If you pose a question, you should be ready to expect that the
> answer is... "it depends". That's how learning starts :-)

i think i am regressing.

Re: Is Elisp really that slow? (was: Why is Elisp slow?)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 833 bytes --]

On Sat, Jun 08, 2019 at 02:00:23PM -0700, Samuel Wales wrote:
> On 6/8/19, tomas@tuxteam.de <tomas@tuxteam.de> wrote:
> > On Fri, Jun 07, 2019 at 05:26:10PM -0700, Samuel Wales wrote:
> >> well, for me as a mere user, i just want elisp [per se] to run faster.
> >> maybe that has 1000 definitions, but the 3 things that i have read in
> >> this thread wfm as PERFECT targets for speeding up and would make
> >> emacs even better for me.
> >
> > Buy a faster computer, then :-)
> 
> i was going to add somethning like "without buying a supercomputer"
> but figured nobody would say to get a new computer.

[...]

Yeah, I know. That's why I put a smiley in there. Single-thread speed
hasn't advanced for a while (on commodity hardware, at least) and it
seems it'll stay so for yet another while.

Cheers
-- tomás

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Is Elisp really that slow?

[Óscar Fuentes]

tomas@tuxteam.de writes:

> On Sat, Jun 08, 2019 at 02:00:23PM -0700, Samuel Wales wrote:
>> On 6/8/19, tomas@tuxteam.de <tomas@tuxteam.de> wrote:
>> > On Fri, Jun 07, 2019 at 05:26:10PM -0700, Samuel Wales wrote:
>> >> well, for me as a mere user, i just want elisp [per se] to run faster.
>> >> maybe that has 1000 definitions, but the 3 things that i have read in
>> >> this thread wfm as PERFECT targets for speeding up and would make
>> >> emacs even better for me.
>> >
>> > Buy a faster computer, then :-)
>> 
>> i was going to add somethning like "without buying a supercomputer"
>> but figured nobody would say to get a new computer.
>
> [...]
>
> Yeah, I know. That's why I put a smiley in there. Single-thread speed
> hasn't advanced for a while (on commodity hardware, at least) and it
> seems it'll stay so for yet another while.

Yep, we can't count on the hardware guys anymore to resolve our
performance problems, we are forced to improve what we have if we wish
to broaden what our tools can do on terms of data volume and/or
complexity.

Re: Is Elisp really that slow? (was: Why is Elisp slow?)

[Emanuel Berg via help-gnu-emacs]

tomas wrote:

> And this is so generic that it's best served
> by a generic programming language, which
> Elisp isn't, and for a good reason.

Well, I guess it depends what you mean. To me,
Elisp is a domain-specific dialect of Lisp.
And Lisp is as generic as it can be. It is the
Pythagorean theorem theorem of computing, as
Pascal Bourguignon would have put it (long time
I heard from him BTW). [1]

So is Elisp Elisp or is it Lisp with an E?
(NB not Ecstacy.)

It is rather the "ungenericness" of the domain
that makes it ungeneric, if so. And Emacs is
pretty generic, I'd say?

Or what do you mean, more specifically?

But in practice if someone I meet in the
city center has a computer problem, I don't say
"solve it with Elisp". So I suppose you are
right, still...


[1] PJB has a homepage here:
    <https://www.informatimago.com>
    But no e-mail what I can see anywhere.
    I always like to CC people when I mention
    them, so I put this here to show I at least
    made a small effort to find his e-mail.

-- 
underground experts united
http://user.it.uu.se/~embe8573
https://dataswamp.org/~incal

Re: Is Elisp really that slow?

[Emanuel Berg via help-gnu-emacs]

Óscar Fuentes wrote:

> Yep, we can't count on the hardware guys
> anymore to resolve our performance problems,
> we are forced to improve what we have if we
> wish to broaden what our tools can do on
> terms of data volume and/or complexity.

And even if we could count on other people to
do everything for us, we wouldn't want
to anyway.

-- 
underground experts united
http://user.it.uu.se/~embe8573
https://dataswamp.org/~incal

Re: Is Elisp really that slow? (was: Why is Elisp slow?)

[Eli Zaretskii]

> From: Samuel Wales <samologist@gmail.com>
> Date: Sat, 8 Jun 2019 14:00:23 -0700
> Cc: help-gnu-emacs@gnu.org
> 
> the topic of the thread is elisp speed.  it's true that if a
> significantly faster single-core-throughput computer existed and
> everybody could have access to it, that would solve the problem.  but
> i don't think it is the question that most of us are discussing in
> this particular case.  it definitely is not the quesiton i was
> addressing.

The literal question of whether Lisp can be sped up was answered
already: what we have is probably as fast as it can be.  Minor
improvements are still possible, of course, but they are unlikely to
change the situation drastically, where "drastically" meand something
like 30% at least.  We've hit the wall.

That is why there are "out of the box" ideas that float around, like
JIT byte-compilation, compilation to native code instead of byte code,
etc.  Personally, I'm not sure they will bring a significant speedup,
but since none of those projects was finished yet, I think we should
wait until some of them do before concluding whether any of them show
a direction we should go.

> >>   1] org mode timestamp [aka "daily/weekly"] agenda to complete faster
> >> for a 2 day [...]
> >
> > Dunno about that. Perhaps Org has sub-optimal algorithms there?
> > Why don't you get your hands dirty and profile the whole thing?
> 
> we discussed this on this thread.  it is optimized to death.  it could
> perhaps be rewritten but has not been and that is a large task.  and
> no, i am not capable of it.  and no, that is not the question in this
> particular case.  the question is, can we speed it up, as the topic of
> the thread states, by speeding up elisp.  for example with a faster re
> matcher.

But the re matcher is written in C, and is also optimized to death.
Replacing it with another regexp engine is a huge task, certainly not
smaller than rewriting the Org agenda code, because Emacs has some
unusual requirements from the regexp code (which is why we use a
forked version of an old glibc regexp code, and didn't yet succeed to
migrate to the current glibc code).

Re: Is Elisp really that slow?

[Xavier Maillard]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Sat, 08 Jun 2019 23:44:15 +0200
> 
> tomas@tuxteam.de writes:
> 
> > On Sat, Jun 08, 2019 at 02:00:23PM -0700, Samuel Wales wrote:
> >> On 6/8/19, tomas@tuxteam.de <tomas@tuxteam.de> wrote:
> >> > On Fri, Jun 07, 2019 at 05:26:10PM -0700, Samuel Wales wrote:
> >> >> well, for me as a mere user, i just want elisp [per se] to run faster.
> >> >> maybe that has 1000 definitions, but the 3 things that i have read in
> >> >> this thread wfm as PERFECT targets for speeding up and would make
> >> >> emacs even better for me.
> >> >
> >> > Buy a faster computer, then :-)
> >> 
> >> i was going to add somethning like "without buying a supercomputer"
> >> but figured nobody would say to get a new computer.
> >
> > [...]
> >
> > Yeah, I know. That's why I put a smiley in there. Single-thread speed
> > hasn't advanced for a while (on commodity hardware, at least) and it
> > seems it'll stay so for yet another while.
> 
> Yep, we can't count on the hardware guys anymore to resolve our
> performance problems, we are forced to improve what we have if we wish
> to broaden what our tools can do on terms of data volume and/or
> complexity.

I would say that it is fortunate. I did not buy the argument to let
the hardware optimize all the sucky software.

At least, this is going to be fun and all we need need is skilled
people (which I am not ;) ).

        - xma

GPG:  BA4909B7 8F04DE1B

Re: Is Elisp really that slow? (was: Why is Elisp slow?)

[Samuel Wales]

On 6/8/19, Eli Zaretskii <eliz@gnu.org> wrote:
> The literal question of whether Lisp can be sped up was answered
> already: what we have is probably as fast as it can be.  Minor

> That is why there are "out of the box" ideas that float around, like
> JIT byte-compilation, compilation to native code instead of byte code,

to me those are valid attempts to answer the literal question.