How should this profiler report interpreted?

How should this profiler report interpreted?

[scame]

I have a function which I use to process some files and it works
slowly with bigger files. I tried to profile it and I got this output.
Only the relevant part is shown:

 - save-excursion                                     9817  99%
  - insert                                               2   0%
   - jit-lock-after-change                               1   0%
    - run-hook-with-args                                 1   0%
       font-lock-extend-jit-lock-region-after-change     1   0%

Does this mean the save-excursion call itself consumes 99% of the CPU?

Re: How should this profiler report interpreted?

[Emanuel Berg via Users list for the GNU Emacs text editor]

scame wrote:

> I have a function which I use to process some files and it
> works slowly with bigger files. I tried to profile it and
> I got this output. Only the relevant part is shown:
>
>  - save-excursion                                     9817  99%
>   - insert                                               2   0%
>    - jit-lock-after-change                               1   0%
>     - run-hook-with-args                                 1   0%
>        font-lock-extend-jit-lock-region-after-change     1   0%
>
> Does this mean the save-excursion call itself consumes 99%
> of the CPU?

To me it looks like it means the tool isn't up for the job :)

Interesting question BTW! How did you do it, exactly?

-- 
underground experts united
https://dataswamp.org/~incal

Re: How should this profiler report interpreted?

[Eli Zaretskii]

> Date: Fri, 26 Mar 2021 13:59:36 +0000
> From: scame <laszlomail@protonmail.com>
> 
> I have a function which I use to process some files and it works
> slowly with bigger files. I tried to profile it and I got this output.
> Only the relevant part is shown:
> 
>  - save-excursion                                     9817  99%
>   - insert                                               2   0%
>    - jit-lock-after-change                               1   0%
>     - run-hook-with-args                                 1   0%
>        font-lock-extend-jit-lock-region-after-change     1   0%
> 
> Does this mean the save-excursion call itself consumes 99% of the CPU?

Yes (assuming that nothing else went on in that Emacs session when you
profiled the function you were interested in).

Re: How should this profiler report interpreted?

[scame]

> To me it looks like it means the tool isn't up for the job :)
> Interesting question BTW! How did you do it, exactly?

I process a generated file where certain lines are connected
by an id. I extract the id from a line, then search backwards
for the matching line using the extracted id and there I insert
the line after the matching line in the file.


  ...
  (while (looking-at "INSERT INTO `test2`.+,\\([0-9]+\\));")
    (let ((item (match-string 1))
          (line (buffer-substring (line-beginning-position) (line-end-position))))
      (message item)
      (save-excursion
        (re-search-backward (concat "INSERT INTO `test1`.+"
                                    "VALUES ([^,]+,'[^']*','[^']*','[^']*','[^']*',[^,]+,[^,]+,"
                                    item
                                    ","))
        (end-of-line)
        (insert "\n" line))
      (delete-region (line-beginning-position) (1+ (line-end-position)))
      )))

This is the save-excursion which takes 99% CPU and very slow.

I probably could change up the code bit, so the problem does not appear,
but I'm curious what the problem is here. Is it a bug?

Re: How should this profiler report interpreted?

[scame via Users list for the GNU Emacs text editor]

> > Does this mean the save-excursion call itself consumes 99% of the CPU?
> Yes (assuming that nothing else went on in that Emacs session when you
> profiled the function you were interested in).

Nothing else which consumes significant CPU, just the tested function.

Re: How should this profiler report interpreted?

[Eli Zaretskii]

> Date: Fri, 26 Mar 2021 15:28:26 +0000
> From:  scame via Users list for the GNU Emacs text editor <help-gnu-emacs@gnu.org>
> 
> > > Does this mean the save-excursion call itself consumes 99% of the CPU?
> > Yes (assuming that nothing else went on in that Emacs session when you
> > profiled the function you were interested in).
> 
> Nothing else which consumes significant CPU, just the tested function.

Then I suggest to run this function many times with benchmark-run,
profile that run, and show the fully expanded profile you get.

Re: How should this profiler report interpreted?

[Emanuel Berg via Users list for the GNU Emacs text editor]

scame wrote:

>> To me it looks like it means the tool isn't up for the job
>> :) Interesting question BTW! How did you do it, exactly?
>
> I process a generated file where certain lines are connected
> by an id. I extract the id from a line, then search
> backwards for the matching line using the extracted id and
> there I insert the line after the matching line in the file.

Sounds like relational algebra, almost. To what extent can you
do that in Elisp? Need SQL? Well, obviously you can do
everything in Elisp...

>   ...
>   (while (looking-at "INSERT INTO `test2`.+,\\([0-9]+\\));")
>     (let ((item (match-string 1))
>           (line (buffer-substring (line-beginning-position) (line-end-position))))
>       (message item)
>       (save-excursion
>         (re-search-backward (concat "INSERT INTO `test1`.+"
>                                     "VALUES ([^,]+,'[^']*','[^']*','[^']*','[^']*',[^,]+,[^,]+,"
>                                     item
>                                     ","))
>         (end-of-line)
>         (insert "\n" line))
>       (delete-region (line-beginning-position) (1+ (line-end-position)))
>       )))

:) Very nice.

> This is the save-excursion which takes 99% CPU and
> very slow.

Well, these regexps in a loop - working on a huge file - which
you are modifying as you go along - is that why
`save-excursion' gets so expensive, some Gordian knot that
needs to be cut up by - Julius Caesar - Che Guevara -
Alexander the Great? - but just keeps tying itself back?

> I probably could change up the code bit, so the problem does
> not appear, but I'm curious what the problem is here. Is it
> a bug?

What happens if you try your code on a very small data file?

-- 
underground experts united
https://dataswamp.org/~incal

Re: How should this profiler report interpreted?

[Emanuel Berg via Users list for the GNU Emacs text editor]

scame via Users list for the GNU Emacs text editor wrote:

>>> Does this mean the save-excursion call itself consumes 99%
>>> of the CPU?
>>
>> Yes (assuming that nothing else went on in that Emacs
>> session when you profiled the function you were interested
>> in).
>
> Nothing else which consumes significant CPU, just the
> tested function.

No, of course not.

-- 
underground experts united
https://dataswamp.org/~incal

Re: How should this profiler report interpreted?

[Stefan Monnier]

>   ...
>   (while (looking-at "INSERT INTO `test2`.+,\\([0-9]+\\));")
>     (let ((item (match-string 1))
>           (line (buffer-substring (line-beginning-position) (line-end-position))))
>       (message item)
>       (save-excursion
>         (re-search-backward (concat "INSERT INTO `test1`.+"
>                                     "VALUES ([^,]+,'[^']*','[^']*','[^']*','[^']*',[^,]+,[^,]+,"
>                                     item
>                                     ","))
>         (end-of-line)
>         (insert "\n" line))
>       (delete-region (line-beginning-position) (1+ (line-end-position)))
>       )))

If that's indeed the code in which all the time is spent, then the
profile is very misleading (IOW bogus) because the time won't be spent
in `save-excursion` but in `re-search-backward`.

If you can reproduce this bogus profile, please `M-x report-emacs-bug`
(ideally with a corresponding sample file on which to run the code).


        Stefan

Re: How should this profiler report interpreted?

[Emanuel Berg via Users list for the GNU Emacs text editor]

Stefan Monnier wrote:

> If that's indeed the code in which all the time is spent,
> then the profile is very misleading (IOW bogus) because the
> time won't be spent in `save-excursion` but in
> `re-search-backward`.

Yes, but the regexps, expensive as they may be, shouldn't make
the whole computer non-responsive, right? 99% CPU?

-- 
underground experts united
https://dataswamp.org/~incal

Re: How should this profiler report interpreted?

[Stefan Monnier]

Emanuel Berg via Users list for the GNU Emacs text editor [2021-03-26 19:48:07] wrote:
> Stefan Monnier wrote:
>> If that's indeed the code in which all the time is spent,
>> then the profile is very misleading (IOW bogus) because the
>> time won't be spent in `save-excursion` but in
>> `re-search-backward`.
> Yes, but the regexps, expensive as they may be, shouldn't make
> the whole computer non-responsive, right?

Nothing in Emacs should ever make the whole computer non-responsive,
indeed, tho this is largely the responsibility of the OS rather than of
Emacs itself.
Did the OP say that his whole computer was unresponsive?

> 99% CPU?

The regexp matcher will happily use 99% of a CPU core given the
chance, yes.


        Stefan

Re: How should this profiler report interpreted?

[scame via Users list for the GNU Emacs text editor]

> If you can reproduce this bogus profile, please `M-x report-emacs-bug`
> (ideally with a corresponding sample file on which to run the code).

I'll do that.

The computer was not unresponsive, only emacs was slow and emacs profiler said 99% CPU.

Re: How should this profiler report interpreted?

[Joost Kremers]

On Fri, Mar 26 2021, scame via Users list for the GNU Emacs text editor wrote:
>> If you can reproduce this bogus profile, please `M-x report-emacs-bug`
>> (ideally with a corresponding sample file on which to run the code).
>
> I'll do that.
>
> The computer was not unresponsive, only emacs was slow and emacs profiler said
> 99% CPU.

I don't think that means it took 99% of the CPU (as you would see with e.g.,
top). I think it means that 99% of the CPU resources used to execute the
relevant Elisp fragment were spent in the `save-excursion`.

-- 
Joost Kremers
Life has its moments

Re: How should this profiler report interpreted?

[Emanuel Berg via Users list for the GNU Emacs text editor]

Stefan Monnier wrote:

> Nothing in Emacs should ever make the whole computer
> non-responsive, indeed, tho this is largely the
> responsibility of the OS rather than of Emacs itself.
> Did the OP say that his whole computer was unresponsive?

No, I said that, so don't blame the OP. I was wrong in this
particular issue and hope - in time - to be forgiven.

:)

>> 99% CPU?
>
> The regexp matcher will happily use 99% of a CPU core given
> the chance, yes.

Right, everything will.

With systemd(1), if you all remember how it was a couple of
years ago - it was like an echo, over and over, "systemd.
Don't know anything about it. Don't like it" - but with
systemd there is an easy way to limit resource use tho, e.g.

  $ systemd-run --scope -p CPUQuota=SLICE% CMD

Remember init? Run levels were hardcoded - in *file names*!
Unbelievable. And when Debian went for systemd, all the
derivates quickly followed suit. So much for "Ubuntu isn't
Debian, Debian is loosing its significance and market share."

And as I remember Debian was forked (for the nth time) again
for this very reason, the great init/systemd polemic into yet
another Debian fork, this one with the claim for fame that it
stuck to init. Where are these guys now? he-ll-o?

But comparisons to the Sino-Soviet split are maybe
exaggerated...

-- 
underground experts united
https://dataswamp.org/~incal

Re: How should this profiler report interpreted?

[scame via Users list for the GNU Emacs text editor]

> >

If you can reproduce this bogus profile, please `M-x report-emacs-bug`

> >

(ideally with a corresponding sample file on which to run the code).

>
> I'll do that.

It's here with a sample file. I'm curious what you make of it:

https://lists.gnu.org/archive/html/bug-gnu-emacs/2021-03/msg02017.html

Re: How should this profiler report interpreted?

[Emanuel Berg via Users list for the GNU Emacs text editor]

scame via Users list for the GNU Emacs text editor wrote:

> It's here with a sample file. I'm curious what you make of it:
>
> https://lists.gnu.org/archive/html/bug-gnu-emacs/2021-03/msg02017.html

I like `format' more than `concat' other than that that code
looks great.

-- 
underground experts united
https://dataswamp.org/~incal

Re: How should this profiler report interpreted?

[Emanuel Berg via Users list for the GNU Emacs text editor]

> Remember init? Run levels were hardcoded - in *file names*!
> Unbelievable. And when Debian went for systemd, all the
> derivates quickly followed suit. So much for "Ubuntu isn't
> Debian, Debian is loosing its significance and market share."
>
> And as I remember Debian was forked (for the nth time) [...]

Speaking of distributions...

Why can't we have one installer

idea ( Unix ) # no other option, hit RET
implementation ( Linux | BSD | ... ) # whatever
set of default software ( GNU | BSD | ...) # so much more versatility (options) with GNU so choose that
style ( TUI | GUI ) # not cut in rock, just an indicator to the installer software what to focus oohn just ahh little bit more
editor ( Emacs | Vim | ... ) # IMPORTANT, here you are supposed to choose Emacs!
desktop # isn't that parallel to the distro, what is it and who needs it? *skipped*
window system protocol ( Wayland | X11 ) # for the experts
window manager ( Openbox | openbsd-cwm ) # :) ha
etc etc
done

why do you need a distribution at all?

What has the distribution ever done for us???

tell me, why doesn't that work?

-- 
underground experts united
https://dataswamp.org/~incal

Re: How should this profiler report interpreted?

[Emanuel Berg via Users list for the GNU Emacs text editor]

> idea ( Unix ) # no other option, hit RET
> implementation ( Linux | BSD | ... ) # whatever
> set of default software ( GNU | BSD | ...) # so much more
> versatility (options) with GNU so choose that
> style ( TUI | GUI ) # not cut in rock, just an indicator to
> the installer software what to focus oohn just ahh little bit
> more
> editor ( Emacs | Vim | ... ) # IMPORTANT, here you are supposed to choose Emacs!
> desktop # isn't that parallel to the distro, what is it and
> who needs it? *skipped*
> window system protocol ( Wayland | X11 ) # for the experts
> window manager ( Openbox | openbsd-cwm ) # :) ha
> etc etc
> done

"True that most forks seem stupid but once in a while you
know, so diversity is our strength in the long run"

Sure, but you can have diversity here as well:

editor ( Emacs | Vim | ... | (waaay back) | yag-emacs )

yag-emacs = yet another... no. you-are-gay-emacs, fork of
Emacs adapted to gay people

... but that doesn't make sense? it has nothing to do with it?

Yes, I told you it is a stupid fork, that's why it is downmost
in a veery long list of editors!

But it IS there!

Diversity: found (mission accomplished)

-- 
underground experts united
https://dataswamp.org/~incal