28.0.50; Proposal: slightly more efficient package-quickstart.el

28.0.50; Proposal: slightly more efficient package-quickstart.el

[Arthur Miller]

[-- Attachment #1: Type: text/plain, Size: 1006 bytes --]


The puropose of "quickstart" file is to perform some calculations
offline to speed up startup time, and what I see in generated code that
every package is added to load-path with `add-to-list' function, which
will ensure that each object is added only once.

Since quickstart file is created by iterating through each and one
installed package whose path is already known, this calculation can be
performed once at generation time, rather than every time when Emacs
starts. It is relatively simple to alter the generator routine for this,
I have included patch. I have been doing something similar in my
personal init file since quite some time now and have not had problems,
but I am not that sofisticated user.

I guess this won't speed thing much, maybe people who use large number
of packages could see some difference. Regardless it is still an
unnecessary computation and thus waste of cpu (and battery) to do this
every startup if it can be done only once when package-quickstart.el is
generated.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: package.patch --]
[-- Type: text/x-patch, Size: 1887 bytes --]

--- ../emacs/lisp/emacs-lisp/package.el	2021-07-11 12:20:59.257657537 +0200
+++ ./package.el	2021-07-20 02:01:38.050169037 +0200
@@ -4137,7 +4137,8 @@
         (package-activated-list ())
         ;; Make sure we can load this file without load-source-file-function.
         (coding-system-for-write 'emacs-internal)
-        (Info-directory-list '("")))
+        (Info-directory-list '(""))
+        paths)
     (dolist (elt package-alist)
       (condition-case err
           (package-activate (car elt))
@@ -4155,6 +4156,7 @@
                 (let ((load-suffixes '(".el" ".elc")))
                   (locate-library (package--autoloads-file-name pkg))))
                (pfile (prin1-to-string file)))
+          (push (package-desc-dir pkg) paths)
           (insert "(let ((load-true-file-name " pfile ")\
 (load-file-name " pfile "))\n")
           (insert-file-contents file)
@@ -4164,6 +4166,10 @@
               (replace-match (if (match-end 1) "" pfile) t t)))
           (unless (bolp) (insert "\n"))
           (insert ")\n")))
+      (goto-char (point-min))
+      (while (re-search-forward "^(add-to-list.*load-path" nil t)
+        (goto-char (line-beginning-position))
+        (kill-sexp))
       (pp `(setq package-activated-list
                  (append ',(mapcar #'package-desc-name package--quickstart-pkgs)
                          package-activated-list))
@@ -4175,6 +4181,10 @@
                       (setq Info-directory-list
                             (append ',info-dirs Info-directory-list)))
               (current-buffer))))
+      (goto-char (point-min))
+      (forward-line 3)
+      (insert (concat "\n(nconc load-path '" (prin1-to-string paths) ")\n"))
+      (goto-char (point-max))
       ;; Use `\s' instead of a space character, so this code chunk is not
       ;; mistaken for an actual file-local section of package.el.
       (insert "\f

Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Stefan Monnier]

Arthur Miller [2021-07-20 02:27:26] wrote:
> Since quickstart file is created by iterating through each and one
> installed package whose path is already known, this calculation can be
> performed once at generation time, rather than every time when Emacs
> starts.

It's indeed an optimization we can perform.  I'm not sure it's worth the
extra code, tho, since there's always a risk it will break something.

Have you made some measurements to see if it the difference is noticeable?

> -        (Info-directory-list '("")))
> +        (Info-directory-list '(""))
> +        paths)

The GNU coding standard would call this `dirs` since a "path" is a list
of directories used for searching.  I guess you could also call it
`path` (singular).

> @@ -4164,6 +4166,10 @@
>                (replace-match (if (match-end 1) "" pfile) t t)))
>            (unless (bolp) (insert "\n"))
>            (insert ")\n")))
> +      (goto-char (point-min))
> +      (while (re-search-forward "^(add-to-list.*load-path" nil t)
> +        (goto-char (line-beginning-position))
> +        (kill-sexp))
>        (pp `(setq package-activated-list
>                   (append ',(mapcar #'package-desc-name package--quickstart-pkgs)
>                           package-activated-list))

Beside the hypothetical risk that the regexp matches an `add-to-list` to
an unrelated variable, I think this risks introducing real bugs for
packages which use (add-to-list 'load-path <subdir>) to add
*sub*directories.

> @@ -4175,6 +4181,10 @@
>                        (setq Info-directory-list
>                              (append ',info-dirs Info-directory-list)))
>                (current-buffer))))
> +      (goto-char (point-min))
> +      (forward-line 3)
> +      (insert (concat "\n(nconc load-path '" (prin1-to-string paths) ")\n"))
> +      (goto-char (point-max))

This inserts the dirs at the end of `load-path` whereas `add-to-list`
usually adds them to the beginning.  I think this will break configs
where a newer version of Org is installed (because the built-in Org
will then take priority).

Of course, we can fix those problems, but unless there's a compelling
performance argument, I think it's a waste of time.


        Stefan

Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Arthur Miller [2021-07-20 02:27:26] wrote:
>> Since quickstart file is created by iterating through each and one
>> installed package whose path is already known, this calculation can be
>> performed once at generation time, rather than every time when Emacs
>> starts.
>
> It's indeed an optimization we can perform.  I'm not sure it's worth the
> extra code, tho, since there's always a risk it will break something.
>
> Have you made some measurements to see if it the difference is
> noticeable?

Not so much, but it is not so much about noticable difference, more
about not performing unnecessary computation. As the loop progresses,
for every dir N, the loop does N-1 path comparisons. These are
unnecessary comparisons since when we know for sure that the string is
not added yet. Trend in Emacs seems to go towards having more and more
packages installed. This removes N(N-1)) comparisons.

>> -        (Info-directory-list '("")))
>> +        (Info-directory-list '(""))
>> +        paths)
>
> The GNU coding standard would call this `dirs` since a "path" is a list
> of directories used for searching.  I guess you could also call it
> `path` (singular).
Ok.
>> @@ -4164,6 +4166,10 @@
>>                (replace-match (if (match-end 1) "" pfile) t t)))
>>            (unless (bolp) (insert "\n"))
>>            (insert ")\n")))
>> +      (goto-char (point-min))
>> +      (while (re-search-forward "^(add-to-list.*load-path" nil t)
>> +        (goto-char (line-beginning-position))
>> +        (kill-sexp))
>>        (pp `(setq package-activated-list
>>                   (append ',(mapcar #'package-desc-name package--quickstart-pkgs)
>>                           package-activated-list))
>
> Beside the hypothetical risk that the regexp matches an `add-to-list` to
> an unrelated variable, I think this risks introducing real bugs for
> packages which use (add-to-list 'load-path <subdir>) to add
> *sub*directories.
Yes, I was aware of it, but I am not sure how to write the regex, to
avoid that risk. Maybe I could check if the path refers to itself.

>> @@ -4175,6 +4181,10 @@
>>                        (setq Info-directory-list
>>                              (append ',info-dirs Info-directory-list)))
>>                (current-buffer))))
>> +      (goto-char (point-min))
>> +      (forward-line 3)
>> +      (insert (concat "\n(nconc load-path '" (prin1-to-string paths) ")\n"))
>> +      (goto-char (point-max))
>
> This inserts the dirs at the end of `load-path` whereas `add-to-list`
> usually adds them to the beginning.  I think this will break configs
> where a newer version of Org is installed (because the built-in Org
> will then take priority).
Ok. I didn't notice anything on my own computer, but it says nothing
about other people setups. Since nconc has to loop throug entire list to
append to it's tail, I thought it is less to loop through existing
load-path, since it probably contains only Emacs lisp directories at
that point. But I can exchange the order.

> Of course, we can fix those problems, but unless there's a compelling
> performance argument, I think it's a waste of time.

It's quite simple to do, it's not much of the code, Performance here is
probably not spending cpu and saving battery life. I don't know, at
least in theory :).

Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Stefan Monnier]

> Not so much, but it is not so much about noticable difference, more
> about not performing unnecessary computation.

But Emacs is more often limited by manpower than CPU, so the impact on
code complexity should not be discounted.

> As the loop progresses, for every dir N, the loop does N-1 path
> comparisons. These are unnecessary comparisons since when we know for
> sure that the string is not added yet. Trend in Emacs seems to go
> towards having more and more packages installed. This removes
> N(N-1)) comparisons.

I know.  But you have to get to a list of at least 2000 elements
before this N² starts making a noticable (0.2s) difference [tested on
an old 1GHz Cortex-A8.  ]

By the time the user has 2000 packages installed, there are several
other performance problems that will be more noticeable, I'm afraid.
E.g. the length of `load-path` itself will be a source of slowdown.

I know 200 packages is fairly common nowadays, so 1000 or more in not an
outlandish idea, and there are indeed performance issues that come up
when we have many packages, but I suspect that we'll need more thorough
changes to tackle this, and in any case it should be driven by concrete
measurements, otherwise we'll waste our time optimizing details that
don't actually matter.

>> Beside the hypothetical risk that the regexp matches an `add-to-list` to
>> an unrelated variable, I think this risks introducing real bugs for
>> packages which use (add-to-list 'load-path <subdir>) to add
>> *sub*directories.
> Yes, I was aware of it, but I am not sure how to write the regex, to
> avoid that risk.

I don't think you can avoid that problem by fixing the regexp, but
rather by `read`ing the sexp and looking at the directory it adds to
make sure it is indeed one of the ones you're adding "by hand".

Or maybe a better approach is to do something like what we do with
`Info-directory-list`, tho it might also be tricky to "get it right".

>> Of course, we can fix those problems, but unless there's a compelling
>> performance argument, I think it's a waste of time.
> It's quite simple to do,

It's not if you want to handle correctly the corner cases, as this email
discussion shows.

> I don't know, at least in theory :).

Maybe there's a theoretical gain.  But there's a very real practical
loss in time spent getting the code to work correctly and maintaining it
in the future.


        Stefan

Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Arthur Miller]

[-- Attachment #1: Type: text/plain, Size: 5753 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Not so much, but it is not so much about noticable difference, more
>> about not performing unnecessary computation.
I agree that simplicity and code clarity is important, on many
levels. But maybe we can have the cake and it it too, as you said 
for wdired?

> But Emacs is more often limited by manpower than CPU, so the impact on
> code complexity should not be discounted.
>
>> As the loop progresses, for every dir N, the loop does N-1 path
>> comparisons. These are unnecessary comparisons since when we know for
>> sure that the string is not added yet. Trend in Emacs seems to go
>> towards having more and more packages installed. This removes
>> N(N-1)) comparisons.
>
> I know.  But you have to get to a list of at least 2000 elements
> before this N² starts making a noticable (0.2s) difference [tested on
> an old 1GHz Cortex-A8.  ]

Of course. I agree that number of packages has to be big to see the
noticable difference. I don't know though how high it shold be. I do a
lot of testing in my own init file, but I can't say anything for sure,
since I do some weird stuff that are not representable of "normal" Emacs
usage.

> By the time the user has 2000 packages installed, there are several
> other performance problems that will be more noticeable, I'm afraid.
> E.g. the length of `load-path` itself will be a source of slowdown.
>
> I know 200 packages is fairly common nowadays, so 1000 or more in not an
> outlandish idea, and there are indeed performance issues that come up
> when we have many packages, but I suspect that we'll need more thorough
> changes to tackle this, and in any case it should be driven by concrete
> measurements, otherwise we'll waste our time optimizing details that
> don't actually matter.
Last weekend I tested actually myself to restructure how my packages are
loaded. I noticed that init time increased after I added ~100 packages,
just for test, and I didn't required anything of that into Emacs. So I
tested the idea to put all .elc file into a single place and skipp in
entirety this monstrosity of load-path that results after 200 packages
are loaded. I got it to work to a degree, it least I got running Emacs,
native compiler not complaining and most packages loaded, but I also got
some cyclic dependency, notably for dired and semantic of all things,
that actually rendered entire session unusable for the most part. I'll
leave that for another day when I have some more time.

>>> Beside the hypothetical risk that the regexp matches an `add-to-list` to
>>> an unrelated variable, I think this risks introducing real bugs for
>>> packages which use (add-to-list 'load-path <subdir>) to add
>>> *sub*directories.
>> Yes, I was aware of it, but I am not sure how to write the regex, to
>> avoid that risk.
>
> I don't think you can avoid that problem by fixing the regexp, but
> rather by `read`ing the sexp and looking at the directory it adds to
> make sure it is indeed one of the ones you're adding "by hand".
The problem is that .* matches everything but new line as I learned form
Emacs Wiki. After fixing regexp to match the newline, it blows the regex
stack as warned on the wiki page.

I have encountered another problem, I hope you can explain it, because I
don't understand what is going on. When I run this piece which actually
tries to match adding to load path: 

(when (re-search-forward rx-path-beg nil t)
            (goto-char (line-beginning-position))
            (setq temp-point (point))
            (forward-sexp)
            (when (search-backward file nil t 1)
              (goto-char temp-point)
              (kill-sexp)))

In search backward, regardless of which bound I give to
re-search-backward or just search-backward, the search failes. My
original thought was to use (line-beginning-position) and I also tested
this temp-point. Interesting is that same code works fine from M-:. If I
setq those variables I need to run that little piece from M-: and position
point in a spot before a statement to remove, it works fine, but in the
defun it always fails so I had to run it with nil for the bound. Is it
something I don't understand there or a bug? 

(when (re-search-forward rx-path-beg nil t)
            (goto-char (line-beginning-position))
            (setq temp-point (point))
            (forward-sexp)
            (when (search-backward file (line-beginning-position)  t 1)
              (goto-char temp-point)
              (kill-sexp)))

> Or maybe a better approach is to do something like what we do with
> `Info-directory-list`, tho it might also be tricky to "get it right".
>
>>> Of course, we can fix those problems, but unless there's a compelling
>>> performance argument, I think it's a waste of time.
>> It's quite simple to do,
>
> It's not if you want to handle correctly the corner cases, as this email
> discussion shows.
>
>> I don't know, at least in theory :).
>
> Maybe there's a theoretical gain.  But there's a very real practical
> loss in time spent getting the code to work correctly and maintaining it
> in the future.

I understand what you mean, it is up to you to decide. The patch does add
few lines, but it is not some tricky, advanced code hard to
understand. I could also refactor some code out of
package-quickstart-refresh into a smaller support defun if that would 
make things easier to maintain.

By the way, I haven't even touched on those empty let closures, that
really just produce work for garbage collector in most cases. I think I
have seen 2 packages that used some of those two vars that get declared
in those let statements. :)


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: package.patch --]
[-- Type: text/x-patch, Size: 2349 bytes --]

--- ../emacs/lisp/emacs-lisp/package.el	2021-07-11 12:20:59.257657537 +0200
+++ ./package.el	2021-07-22 23:37:15.167925681 +0200
@@ -4137,7 +4137,9 @@
         (package-activated-list ())
         ;; Make sure we can load this file without load-source-file-function.
         (coding-system-for-write 'emacs-internal)
-        (Info-directory-list '("")))
+        (Info-directory-list '(""))
+        (rx-path-beg "^(add-to-list 'load-path (directory-file-name")
+        path insertion-point temp-point)
     (dolist (elt package-alist)
       (condition-case err
           (package-activate (car elt))
@@ -4155,9 +4157,24 @@
                 (let ((load-suffixes '(".el" ".elc")))
                   (locate-library (package--autoloads-file-name pkg))))
                (pfile (prin1-to-string file)))
+          ;; prepare list of all dirs to add to load-path
+          (push (package-desc-dir pkg) path)
           (insert "(let ((load-true-file-name " pfile ")\
 (load-file-name " pfile "))\n")
           (insert-file-contents file)
+          (setq insertion-point (point))
+          ;; since we are precomputing and pasting list of package directories
+          ;; to add to load-path, remove statements to add individual package from
+          ;; autoloads file
+          (goto-char insertion-point)
+          (when (re-search-forward rx-path-beg nil t)
+            (goto-char (line-beginning-position))
+            (setq temp-point (point))
+            (forward-sexp)
+            (when (search-backward file nil t 1)
+              (goto-char temp-point)
+              (kill-sexp)))
+          (goto-char insertion-point)
           ;; Fixup the special #$ reader form and throw away comments.
           (while (re-search-forward "#\\$\\|^;\\(.*\n\\)" nil 'move)
             (unless (nth 8 (syntax-ppss))
@@ -4175,6 +4192,10 @@
                       (setq Info-directory-list
                             (append ',info-dirs Info-directory-list)))
               (current-buffer))))
+      (goto-char (point-min))
+      (forward-line 3)
+      (insert (concat "\n(setq load-path (nconc '" (prin1-to-string path) " load-path))\n"))
+      (goto-char (point-max))
       ;; Use `\s' instead of a space character, so this code chunk is not
       ;; mistaken for an actual file-local section of package.el.
       (insert "\f

Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Stefan Monnier]

>>> Not so much, but it is not so much about noticable difference, more
>>> about not performing unnecessary computation.
> I agree that simplicity and code clarity is important, on many
> levels. But maybe we can have the cake and it it too, as you said 
> for wdired?

In the case of wdired there was a concrete gain.
Here it's only hypothetical, so the positive motivation is quite different.
Also `package-quickstart` is fairly tricky to troubleshoot (beyond removing
or refreshing the file).  To the end user it's largely a magical button,
so it's really important to make it work reliably.

IOW the incentives are strongly opposed to your proposition.

> Last weekend I tested actually myself to restructure how my packages are
> loaded. I noticed that init time increased after I added ~100 packages,
> just for test, and I didn't required anything of that into Emacs. So I
> tested the idea to put all .elc file into a single place and skipp in
> entirety this monstrosity of load-path that results after 200 packages
> are loaded. I got it to work to a degree, it least I got running Emacs,
> native compiler not complaining and most packages loaded, but I also got
> some cyclic dependency, notably for dired and semantic of all things,
> that actually rendered entire session unusable for the most part. I'll
> leave that for another day when I have some more time.

Moving the .elc files to a separate (short) list of directories indeed
one way we could address the situation where there are too many entries
on `load-path`.

Another way would be to scan `load-path` "once" and populate
a hash-table from that, after which (load "foo" ...) could be sped up by
looking up "foo" in the hash-table.

Still, that presumes that finding a file is the main issue, but I don't
know if that would indeed be true.

> (when (re-search-forward rx-path-beg nil t)
>             (goto-char (line-beginning-position))
>             (setq temp-point (point))
>             (forward-sexp)
>             (when (search-backward file nil t 1)
>               (goto-char temp-point)
>               (kill-sexp)))

I'd do something like

    (while (re-search-forward "^(add-to-list" nil t)
      (goto-char (match-beginning 0))
        (let ((start (point))
              (x (read (current-buffer))))
          ...)))


-- Stefan

Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 2766 bytes --]


One place where we could gain perceptible speedup is when
package-quickstart is rebuilt after updating packages; could we make
that function asyn?
Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> Not so much, but it is not so much about noticable difference, more
>>>> about not performing unnecessary computation.
>> I agree that simplicity and code clarity is important, on many
>> levels. But maybe we can have the cake and it it too, as you said 
>> for wdired?
>
> In the case of wdired there was a concrete gain.
> Here it's only hypothetical, so the positive motivation is quite different.
> Also `package-quickstart` is fairly tricky to troubleshoot (beyond removing
> or refreshing the file).  To the end user it's largely a magical button,
> so it's really important to make it work reliably.
>
> IOW the incentives are strongly opposed to your proposition.
>
>> Last weekend I tested actually myself to restructure how my packages are
>> loaded. I noticed that init time increased after I added ~100 packages,
>> just for test, and I didn't required anything of that into Emacs. So I
>> tested the idea to put all .elc file into a single place and skipp in
>> entirety this monstrosity of load-path that results after 200 packages
>> are loaded. I got it to work to a degree, it least I got running Emacs,
>> native compiler not complaining and most packages loaded, but I also got
>> some cyclic dependency, notably for dired and semantic of all things,
>> that actually rendered entire session unusable for the most part. I'll
>> leave that for another day when I have some more time.
>
> Moving the .elc files to a separate (short) list of directories indeed
> one way we could address the situation where there are too many entries
> on `load-path`.
>
> Another way would be to scan `load-path` "once" and populate
> a hash-table from that, after which (load "foo" ...) could be sped up by
> looking up "foo" in the hash-table.
>
> Still, that presumes that finding a file is the main issue, but I don't
> know if that would indeed be true.
>
>> (when (re-search-forward rx-path-beg nil t)
>>             (goto-char (line-beginning-position))
>>             (setq temp-point (point))
>>             (forward-sexp)
>>             (when (search-backward file nil t 1)
>>               (goto-char temp-point)
>>               (kill-sexp)))
>
> I'd do something like
>
>     (while (re-search-forward "^(add-to-list" nil t)
>       (goto-char (match-beginning 0))
>         (let ((start (point))
>               (x (read (current-buffer))))
>           ...)))
>
>
> -- Stefan
>
>

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> Not so much, but it is not so much about noticable difference, more
>>>> about not performing unnecessary computation.
>> I agree that simplicity and code clarity is important, on many
>> levels. But maybe we can have the cake and it it too, as you said 
>> for wdired?
>
> In the case of wdired there was a concrete gain.
> Here it's only hypothetical, so the positive motivation is quite different.
> Also `package-quickstart` is fairly tricky to troubleshoot (beyond removing
> or refreshing the file).  To the end user it's largely a magical button,
> so it's really important to make it work reliably.
>
> IOW the incentives are strongly opposed to your proposition.
For me the positive motivation here is to not do unnecessary
calculations, not so much the speed gain. I don't understand why speed
is the only measurement of improvement. Even replacing 'add-to-list' with
a 'push' to load-path would be an improvemnt, but the patch would be not
much different. Anyway, if there is no incentive to change this so it is
:).

>> Last weekend I tested actually myself to restructure how my packages are
>> loaded. I noticed that init time increased after I added ~100 packages,
>> just for test, and I didn't required anything of that into Emacs. So I
>> tested the idea to put all .elc file into a single place and skipp in
>> entirety this monstrosity of load-path that results after 200 packages
>> are loaded. I got it to work to a degree, it least I got running Emacs,
>> native compiler not complaining and most packages loaded, but I also got
>> some cyclic dependency, notably for dired and semantic of all things,
>> that actually rendered entire session unusable for the most part. I'll
>> leave that for another day when I have some more time.
>
> Moving the .elc files to a separate (short) list of directories indeed
> one way we could address the situation where there are too many entries
> on `load-path`.
>
> Another way would be to scan `load-path` "once" and populate
> a hash-table from that, after which (load "foo" ...) could be sped up by
> looking up "foo" in the hash-table.

Sure either could work, hash table would probably be faster since it
would effectively be a database in memory, but it also means it will
cost more in RAM and also be even more complicated since you will have
to modify how require works. Copying over .elc files in some kind of
"package-cache" after the package installation is done is probably
less change.

> Still, that presumes that finding a file is the main issue, but I don't
> know if that would indeed be true.

For me the main issue here was to skip needless work :). I think the
things wouold be speedier, but I don't know how much of that would be
percieved by the end user. Requiring features is not very frequent
issue, and most of it is done anyway at some well defined points like
emacs starting up, or loading a mode or something similar.

>> (when (re-search-forward rx-path-beg nil t)
>>             (goto-char (line-beginning-position))
>>             (setq temp-point (point))
>>             (forward-sexp)
>>             (when (search-backward file nil t 1)
>>               (goto-char temp-point)
>>               (kill-sexp)))
>
> I'd do something like
>
>     (while (re-search-forward "^(add-to-list" nil t)
>       (goto-char (match-beginning 0))
>         (let ((start (point))
>               (x (read (current-buffer))))
>           ...)))
>
Ok, thank you. I haven't used 'read' yet so I wasn't thinking in that
direction.

Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

I just did some tests. I wrote small program and installed all pacakges
from elpa. I got ~700 packages installed alltogether (not all are from
elpa). 

I did following tests: I generated file with original quickstart
generator and patched one. I also removed half of the packages and
generated quickstart file with original generator from package.el

Previous to that of course, I have removed my init file, early init,
byte compiled versions etc, and also found out that I had to build emacs
version without native compiler. Nothing was required, Emacs started
with blank init file, so to only measure difference in generated
quickstart file.

What I found is that to happy you :), it was not much difference between
modified and original generator. I got pretty much same timings for
both, ~0.47 sec. There is an important thing: if I nconed load-path to
generated path, than startup time was significantly higher, it went to
~0.75 sec. When I switched to nconc load-path to generated one (the one
that was unsafe) than timing went back to ~0.47. I did 3 mesurements,
and I am not sure if iterating throuhg ~700 strings should take that
much time, but that happened.

Finally original routine, ~700 packages vs 350 packages. Timings where
~0.47 vs ~0.3 (slightly below each time). So obvoisly it is noticable in
startup time, even when adding ~300 more packages without doing any 
customizations in init file.

Interesting, when I "bake in" quickstart file into my init file, startup
time is ~0.35 sec and that with all my customizations. Maybe disk acces,
but since files are in same dir, I expect them both to be fetched in by
the driver. I have no other explanation more than so.




For the failures, if that is of interest to authors or maintainers:

Failed to verify signature realgud-ipdb-1.0.0.tar.sig:
Bad signature from 066DAFCB81E42C40 GNU ELPA Signing Agent (2019) <elpasign@elpa.gnu.org>
Command output:
gpg: Signatur gjord lör 21 sep 2019 19:54:19 CEST
gpg:              med RSA-nyckeln C433554766D3DDC64221BFAA066DAFCB81E42C40
gpg: FELAKTIG signatur från "GNU ELPA Signing Agent (2019) <elpasign@elpa.gnu.org>" [okänd]

Failed to verify signature scanner-0.2.tar.sig:
Bad signature from 066DAFCB81E42C40 GNU ELPA Signing Agent (2019) <elpasign@elpa.gnu.org>
Command output:
gpg: Signatur gjord tis 15 dec 2020 11:05:02 CET
gpg:              med RSA-nyckeln C433554766D3DDC64221BFAA066DAFCB81E42C40
gpg: FELAKTIG signatur från "GNU ELPA Signing Agent (2019) <elpasign@elpa.gnu.org>" [okänd]

Even some Ada + 1 more package, but it was before I saved the output.

;; refine-autoloads.el ends here

rec-mode - must comment away line below in autoload.el otherwise
generation of package-quickstart.el fails:

(put rec-open-mode 'safe-local-variable (lambda (x) (member x '(edit navigation))

Compiling file /home/arthur/.emacs.d/elpa/dired-du-0.5.2/dired-du-tests.el at Fri Jul 23 21:39:36 2021
dired-du-tests.el:61:27: Error: ‘add-to-list’ can’t use lexical var ‘buffers’;
    use ‘push’ or ‘cl-pushnew’
dired-du-tests.el:65:54: Error: ‘add-to-list’ can’t use lexical var ‘buffers’;
    use ‘push’ or ‘cl-pushnew’
dired-du-tests.el:103:11: Warning: Unused lexical variable `info'
dired-du-tests.el:110:32: Error: ‘add-to-list’ can’t use lexical var
    ‘buffers’; use ‘push’ or ‘cl-pushnew’
dired-du-tests.el:165:27: Warning: Unused lexical variable `name2'
dired-du-tests.el:155:27: Warning: Unused lexical variable `name3'
dired-du-tests.el:157:32: Error: ‘add-to-list’ can’t use lexical var
    ‘buffers’; use ‘push’ or ‘cl-pushnew’
dired-du-tests.el:194:11: Warning: Unused lexical variable `name2'
dired-du-tests.el:178:23: Warning: Unused lexical variable `name3'
dired-du-tests.el:178:23: Warning: Unused lexical variable `marks'
dired-du-tests.el:183:34: Error: ‘add-to-list’ can’t use lexical var
    ‘buffers’; use ‘push’ or ‘cl-pushnew’
dired-du-tests.el:230:11: Warning: Unused lexical variable `name2'
dired-du-tests.el:212:27: Warning: Unused lexical variable `name3'
dired-du-tests.el:213:27: Warning: Unused lexical variable `marks'
dired-du-tests.el:215:34: Error: ‘add-to-list’ can’t use lexical var
    ‘buffers’; use ‘push’ or ‘cl-pushnew’
dired-du-tests.el:275:11: Warning: Unused lexical variable
    `filled-subdir-size'
dired-du-tests.el:251:27: Error: ‘add-to-list’ can’t use lexical var
    ‘buffers’; use ‘push’ or ‘cl-pushnew’

Compiling file /home/arthur/.emacs.d/elpa/notes-mode-1.30/notes-index-mode.el at Fri Jul 23 21:42:23 2021
notes-index-mode.el:26:1: Error: Symbol’s value as variable is void: Use



Entering directory ‘/home/arthur/.emacs.d/elpa/parsec-0.1.3/examples/’
full-csv-parser-tests.el:28:1: Error: Cannot open load file: Filen eller
katalogen finns inte, full-csv-parser


Compiling file /home/arthur/.emacs.d/elpa/phps-mode-0.4.3/admin/phps-mode-automation.el at Fri Jul 23 21:47:38 2021
phps-mode-automation.el:30:1: Error: Cannot open load file: Filen eller
katalogen finns inte, phps-mode-automation-grammar



Compiling file /home/arthur/.emacs.d/elpa/psgml-1.3.4/sgmldecl/fum.el at Fri Jul 23 21:47:46 2021
Entering directory ‘/home/arthur/.emacs.d/elpa/psgml-1.3.4/sgmldecl/’

In replace-grammar:
fum.el:52:6: Warning: ‘insert-buffer’ is for interactive use only; use
    ‘insert-buffer-substring’ instead.

In spt-synexp:
fum.el:74:17: Warning: ‘case’ is an obsolete alias (as of 27.1); use ‘cl-case’
    instead.
fum.el:227:1: Error: Symbol’s function definition is void: spt-nt-name
Leaving directory ‘/home/arthur/.emacs.d/elpa/psgml-1.3.4/sgmldecl/’

Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Arthur Miller]

Arthur Miller <arthur.miller@live.com> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
> I just did some tests. I wrote small program and installed all pacakges
> from elpa. I got ~700 packages installed alltogether (not all are from
> elpa). 
>
> I did following tests: I generated file with original quickstart
> generator and patched one. I also removed half of the packages and
> generated quickstart file with original generator from package.el
>
> Previous to that of course, I have removed my init file, early init,
> byte compiled versions etc, and also found out that I had to build emacs
> version without native compiler. Nothing was required, Emacs started
> with blank init file, so to only measure difference in generated
> quickstart file.
>
> What I found is that to happy you :), it was not much difference between
> modified and original generator. I got pretty much same timings for
> both, ~0.47 sec. There is an important thing: if I nconed load-path to
> generated path, than startup time was significantly higher, it went to
> ~0.75 sec. When I switched to nconc load-path to generated one (the one
> that was unsafe) than timing went back to ~0.47. I did 3 mesurements,
> and I am not sure if iterating throuhg ~700 strings should take that
> much time, but that happened.
>
> Finally original routine, ~700 packages vs 350 packages. Timings where
> ~0.47 vs ~0.3 (slightly below each time). So obvoisly it is noticable in
> startup time, even when adding ~300 more packages without doing any 
> customizations in init file.
>
> Interesting, when I "bake in" quickstart file into my init file, startup
> time is ~0.35 sec and that with all my customizations. Maybe disk acces,
> but since files are in same dir, I expect them both to be fetched in by
> the driver. I have no other explanation more than so.
>
>
>
>
> For the failures, if that is of interest to authors or maintainers:
>
> Failed to verify signature realgud-ipdb-1.0.0.tar.sig:
> Bad signature from 066DAFCB81E42C40 GNU ELPA Signing Agent (2019) <elpasign@elpa.gnu.org>
> Command output:
> gpg: Signatur gjord lör 21 sep 2019 19:54:19 CEST
> gpg:              med RSA-nyckeln C433554766D3DDC64221BFAA066DAFCB81E42C40
> gpg: FELAKTIG signatur från "GNU ELPA Signing Agent (2019) <elpasign@elpa.gnu.org>" [okänd]
>
> Failed to verify signature scanner-0.2.tar.sig:
> Bad signature from 066DAFCB81E42C40 GNU ELPA Signing Agent (2019) <elpasign@elpa.gnu.org>
> Command output:
> gpg: Signatur gjord tis 15 dec 2020 11:05:02 CET
> gpg:              med RSA-nyckeln C433554766D3DDC64221BFAA066DAFCB81E42C40
> gpg: FELAKTIG signatur från "GNU ELPA Signing Agent (2019) <elpasign@elpa.gnu.org>" [okänd]
>
> Even some Ada + 1 more package, but it was before I saved the output.
>
> ;; refine-autoloads.el ends here
>
> rec-mode - must comment away line below in autoload.el otherwise
> generation of package-quickstart.el fails:
>
> (put rec-open-mode 'safe-local-variable (lambda (x) (member x '(edit navigation))
>
> Compiling file /home/arthur/.emacs.d/elpa/dired-du-0.5.2/dired-du-tests.el at Fri Jul 23 21:39:36 2021
> dired-du-tests.el:61:27: Error: ‘add-to-list’ can’t use lexical var ‘buffers’;
>     use ‘push’ or ‘cl-pushnew’
> dired-du-tests.el:65:54: Error: ‘add-to-list’ can’t use lexical var ‘buffers’;
>     use ‘push’ or ‘cl-pushnew’
> dired-du-tests.el:103:11: Warning: Unused lexical variable `info'
> dired-du-tests.el:110:32: Error: ‘add-to-list’ can’t use lexical var
>     ‘buffers’; use ‘push’ or ‘cl-pushnew’
> dired-du-tests.el:165:27: Warning: Unused lexical variable `name2'
> dired-du-tests.el:155:27: Warning: Unused lexical variable `name3'
> dired-du-tests.el:157:32: Error: ‘add-to-list’ can’t use lexical var
>     ‘buffers’; use ‘push’ or ‘cl-pushnew’
> dired-du-tests.el:194:11: Warning: Unused lexical variable `name2'
> dired-du-tests.el:178:23: Warning: Unused lexical variable `name3'
> dired-du-tests.el:178:23: Warning: Unused lexical variable `marks'
> dired-du-tests.el:183:34: Error: ‘add-to-list’ can’t use lexical var
>     ‘buffers’; use ‘push’ or ‘cl-pushnew’
> dired-du-tests.el:230:11: Warning: Unused lexical variable `name2'
> dired-du-tests.el:212:27: Warning: Unused lexical variable `name3'
> dired-du-tests.el:213:27: Warning: Unused lexical variable `marks'
> dired-du-tests.el:215:34: Error: ‘add-to-list’ can’t use lexical var
>     ‘buffers’; use ‘push’ or ‘cl-pushnew’
> dired-du-tests.el:275:11: Warning: Unused lexical variable
>     `filled-subdir-size'
> dired-du-tests.el:251:27: Error: ‘add-to-list’ can’t use lexical var
>     ‘buffers’; use ‘push’ or ‘cl-pushnew’
>
> Compiling file /home/arthur/.emacs.d/elpa/notes-mode-1.30/notes-index-mode.el at Fri Jul 23 21:42:23 2021
> notes-index-mode.el:26:1: Error: Symbol’s value as variable is void: Use
>
>
>
> Entering directory ‘/home/arthur/.emacs.d/elpa/parsec-0.1.3/examples/’
> full-csv-parser-tests.el:28:1: Error: Cannot open load file: Filen eller
> katalogen finns inte, full-csv-parser
>
>
> Compiling file /home/arthur/.emacs.d/elpa/phps-mode-0.4.3/admin/phps-mode-automation.el at Fri Jul 23 21:47:38 2021
> phps-mode-automation.el:30:1: Error: Cannot open load file: Filen eller
> katalogen finns inte, phps-mode-automation-grammar
>
>
>
> Compiling file /home/arthur/.emacs.d/elpa/psgml-1.3.4/sgmldecl/fum.el at Fri Jul 23 21:47:46 2021
> Entering directory ‘/home/arthur/.emacs.d/elpa/psgml-1.3.4/sgmldecl/’
>
> In replace-grammar:
> fum.el:52:6: Warning: ‘insert-buffer’ is for interactive use only; use
>     ‘insert-buffer-substring’ instead.
>
> In spt-synexp:
> fum.el:74:17: Warning: ‘case’ is an obsolete alias (as of 27.1); use ‘cl-case’
>     instead.
> fum.el:227:1: Error: Symbol’s function definition is void: spt-nt-name
> Leaving directory ‘/home/arthur/.emacs.d/elpa/psgml-1.3.4/sgmldecl/’

And I did some incredibly stupid error, I just saw last few lines in
dired showing 704, and thought there were ~700 ṕkgs; turnes out there
are only ~460. Sorting and grouping directories before files is a good
thing :-). 

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Stefan Monnier]

> One place where we could gain perceptible speedup is when
> package-quickstart is rebuilt after updating packages; could we make
> that function asyn?

Yes, that would be a welcome change.


        Stefan


> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>>>> Not so much, but it is not so much about noticable difference, more
>>>>> about not performing unnecessary computation.
>>> I agree that simplicity and code clarity is important, on many
>>> levels. But maybe we can have the cake and it it too, as you said 
>>> for wdired?
>>
>> In the case of wdired there was a concrete gain.
>> Here it's only hypothetical, so the positive motivation is quite different.
>> Also `package-quickstart` is fairly tricky to troubleshoot (beyond removing
>> or refreshing the file).  To the end user it's largely a magical button,
>> so it's really important to make it work reliably.
>>
>> IOW the incentives are strongly opposed to your proposition.
>>
>>> Last weekend I tested actually myself to restructure how my packages are
>>> loaded. I noticed that init time increased after I added ~100 packages,
>>> just for test, and I didn't required anything of that into Emacs. So I
>>> tested the idea to put all .elc file into a single place and skipp in
>>> entirety this monstrosity of load-path that results after 200 packages
>>> are loaded. I got it to work to a degree, it least I got running Emacs,
>>> native compiler not complaining and most packages loaded, but I also got
>>> some cyclic dependency, notably for dired and semantic of all things,
>>> that actually rendered entire session unusable for the most part. I'll
>>> leave that for another day when I have some more time.
>>
>> Moving the .elc files to a separate (short) list of directories indeed
>> one way we could address the situation where there are too many entries
>> on `load-path`.
>>
>> Another way would be to scan `load-path` "once" and populate
>> a hash-table from that, after which (load "foo" ...) could be sped up by
>> looking up "foo" in the hash-table.
>>
>> Still, that presumes that finding a file is the main issue, but I don't
>> know if that would indeed be true.
>>
>>> (when (re-search-forward rx-path-beg nil t)
>>>             (goto-char (line-beginning-position))
>>>             (setq temp-point (point))
>>>             (forward-sexp)
>>>             (when (search-backward file nil t 1)
>>>               (goto-char temp-point)
>>>               (kill-sexp)))
>>
>> I'd do something like
>>
>>     (while (re-search-forward "^(add-to-list" nil t)
>>       (goto-char (match-beginning 0))
>>         (let ((start (point))
>>               (x (read (current-buffer))))
>>           ...)))
>>
>>
>> -- Stefan
>>
>>

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 3363 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

So looking at package.el:

package-quickstart-refresh is called from
package--quickstart-maybe-refresh and looking at the code in
package-quickstart-refresh, it does not block on user input, so it may
just be sufficient to change the call to package-quickstart-refresh
to (make-thread  #'package-quickstart-refresh) --

Thoughts?

>> One place where we could gain perceptible speedup is when
>> package-quickstart is rebuilt after updating packages; could we make
>> that function asyn?
>
> Yes, that would be a welcome change.
>
>
>         Stefan
>
>
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>
>>>>>> Not so much, but it is not so much about noticable difference, more
>>>>>> about not performing unnecessary computation.
>>>> I agree that simplicity and code clarity is important, on many
>>>> levels. But maybe we can have the cake and it it too, as you said 
>>>> for wdired?
>>>
>>> In the case of wdired there was a concrete gain.
>>> Here it's only hypothetical, so the positive motivation is quite different.
>>> Also `package-quickstart` is fairly tricky to troubleshoot (beyond removing
>>> or refreshing the file).  To the end user it's largely a magical button,
>>> so it's really important to make it work reliably.
>>>
>>> IOW the incentives are strongly opposed to your proposition.
>>>
>>>> Last weekend I tested actually myself to restructure how my packages are
>>>> loaded. I noticed that init time increased after I added ~100 packages,
>>>> just for test, and I didn't required anything of that into Emacs. So I
>>>> tested the idea to put all .elc file into a single place and skipp in
>>>> entirety this monstrosity of load-path that results after 200 packages
>>>> are loaded. I got it to work to a degree, it least I got running Emacs,
>>>> native compiler not complaining and most packages loaded, but I also got
>>>> some cyclic dependency, notably for dired and semantic of all things,
>>>> that actually rendered entire session unusable for the most part. I'll
>>>> leave that for another day when I have some more time.
>>>
>>> Moving the .elc files to a separate (short) list of directories indeed
>>> one way we could address the situation where there are too many entries
>>> on `load-path`.
>>>
>>> Another way would be to scan `load-path` "once" and populate
>>> a hash-table from that, after which (load "foo" ...) could be sped up by
>>> looking up "foo" in the hash-table.
>>>
>>> Still, that presumes that finding a file is the main issue, but I don't
>>> know if that would indeed be true.
>>>
>>>> (when (re-search-forward rx-path-beg nil t)
>>>>             (goto-char (line-beginning-position))
>>>>             (setq temp-point (point))
>>>>             (forward-sexp)
>>>>             (when (search-backward file nil t 1)
>>>>               (goto-char temp-point)
>>>>               (kill-sexp)))
>>>
>>> I'd do something like
>>>
>>>     (while (re-search-forward "^(add-to-list" nil t)
>>>       (goto-char (match-beginning 0))
>>>         (let ((start (point))
>>>               (x (read (current-buffer))))
>>>           ...)))
>>>
>>>
>>> -- Stefan
>>>
>>>
>
>

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Stefan Monnier]

T.V Raman [2021-08-03 19:24:59] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> So looking at package.el:
> package-quickstart-refresh is called from
> package--quickstart-maybe-refresh and looking at the code in
> package-quickstart-refresh, it does not block on user input, so it may
> just be sufficient to change the call to package-quickstart-refresh
> to (make-thread  #'package-quickstart-refresh) --
> Thoughts?

The fact that it doesn't block on user input makes `make-thread` fairly
useless, I'm afraid.


        Stefan

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[T.V Raman]

Interesting:-) Other discussons on emacs-devel have declared that
make-thread should never be used if the async call expects user-input
-- so combining both conclusions, we might as well declare make-thread
useless

Stefan Monnier writes:
 > T.V Raman [2021-08-03 19:24:59] wrote:
 > > Stefan Monnier <monnier@iro.umontreal.ca> writes:
 > > So looking at package.el:
 > > package-quickstart-refresh is called from
 > > package--quickstart-maybe-refresh and looking at the code in
 > > package-quickstart-refresh, it does not block on user input, so it may
 > > just be sufficient to change the call to package-quickstart-refresh
 > > to (make-thread  #'package-quickstart-refresh) --
 > > Thoughts?
 > 
 > The fact that it doesn't block on user input makes `make-thread` fairly
 > useless, I'm afraid.
 > 
 > 
 >         Stefan

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
♉ Id: kg:/m/0285kf1  🦮

--

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
♉ Id: kg:/m/0285kf1  🦮

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Stefan Monnier]

T.V Raman [2021-08-04 07:20:49] wrote:
> Interesting:-) Other discussons on emacs-devel have declared that
> make-thread should never be used if the async call expects user-input
> -- so combining both conclusions, we might as well declare make-thread
> useless

Good point.  Indeed I mis(spoke|wrote): the fact that it doesn't need
user input makes it *possible* to use `make-thread`, but the fact that
it doesn't block on subprocess output (and instead spends all its time
interpreting ELisp code) makes `make-thread` useless because it will
basically never yield ;-(

Here, we really want to start a new Emacs process (in this specific case
we even *want* that process to be (mostly) fresh&clean, contrary to many
other situations where we'd want a new process that "configured just
like the current process").


        Stefan


> Stefan Monnier writes:
>  > T.V Raman [2021-08-03 19:24:59] wrote:
>  > > Stefan Monnier <monnier@iro.umontreal.ca> writes:
>  > > So looking at package.el:
>  > > package-quickstart-refresh is called from
>  > > package--quickstart-maybe-refresh and looking at the code in
>  > > package-quickstart-refresh, it does not block on user input, so it may
>  > > just be sufficient to change the call to package-quickstart-refresh
>  > > to (make-thread  #'package-quickstart-refresh) --
>  > > Thoughts?
>  > 
>  > The fact that it doesn't block on user input makes `make-thread` fairly
>  > useless, I'm afraid.
>  > 
>  > 
>  >         Stefan
>
> -- 
>
> Thanks,
>
> --Raman(I Search, I Find, I Misplace, I Research)
> ♉ Id: kg:/m/0285kf1  🦮

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[T.V Raman]

So then perhaps something like
(shell-command "emacs -batch -q -l package -f
package-quickstart-refresh")

 > T.V Raman [2021-08-04 07:20:49] wrote:
 > > Interesting:-) Other discussons on emacs-devel have declared that
 > > make-thread should never be used if the async call expects user-input
 > > -- so combining both conclusions, we might as well declare make-thread
 > > useless
 > 
 > Good point.  Indeed I mis(spoke|wrote): the fact that it doesn't need
 > user input makes it *possible* to use `make-thread`, but the fact that
 > it doesn't block on subprocess output (and instead spends all its time
 > interpreting ELisp code) makes `make-thread` useless because it will
 > basically never yield ;-(
 > 
 > Here, we really want to start a new Emacs process (in this specific case
 > we even *want* that process to be (mostly) fresh&clean, contrary to many
 > other situations where we'd want a new process that "configured just
 > like the current process").
 > 
 > 
 >         Stefan
 > 
 > 
 > > Stefan Monnier writes:
 > >  > T.V Raman [2021-08-03 19:24:59] wrote:
 > >  > > Stefan Monnier <monnier@iro.umontreal.ca> writes:
 > >  > > So looking at package.el:
 > >  > > package-quickstart-refresh is called from
 > >  > > package--quickstart-maybe-refresh and looking at the code in
 > >  > > package-quickstart-refresh, it does not block on user input, so it may
 > >  > > just be sufficient to change the call to package-quickstart-refresh
 > >  > > to (make-thread  #'package-quickstart-refresh) --
 > >  > > Thoughts?
 > >  > 
 > >  > The fact that it doesn't block on user input makes `make-thread` fairly
 > >  > useless, I'm afraid.
 > >  > 
 > >  > 
 > >  >         Stefan
 > >
 > > -- 
 > >
 > > Thanks,
 > >
 > > --Raman(I Search, I Find, I Misplace, I Research)
 > > ♉ Id: kg:/m/0285kf1  🦮

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
♉ Id: kg:/m/0285kf1  🦮

--

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
♉ Id: kg:/m/0285kf1  🦮

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 866 bytes --]

Here is a simple patch that rebuilds package-quickstart-file in a
separate emacs process:

git diff
diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 37dcbe36c8..d525cf4db9 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -4138,7 +4138,11 @@ package--quickstart-maybe-refresh
   (if package-quickstart
       ;; FIXME: Delay refresh in case we're installing/deleting
       ;; several packages!
-      (package-quickstart-refresh)
+      (start-process
+       "quickstart" "*quickstart*" "emacs"
+       "-batch" "-q"
+       "-l" "package"
+       "-f" "package-quickstart-refresh")
     (delete-file (concat package-quickstart-file "c"))
     (delete-file package-quickstart-file)))
 

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Eli Zaretskii]

> From: "T.V Raman" <raman@google.com>
> Cc: arthur.miller@live.com,  emacs-devel@gnu.org
> Date: Wed, 04 Aug 2021 17:22:23 -0700
> 
> Here is a simple patch that rebuilds package-quickstart-file in a
> separate emacs process:

Thanks.  I have one minor comment:

> -      (package-quickstart-refresh)
> +      (start-process
> +       "quickstart" "*quickstart*" "emacs"
> +       "-batch" "-q"
> +       "-l" "package"
> +       "-f" "package-quickstart-refresh")

The 3rd argument of start-process should use

  (expand-file-name invocation-name invocation-directory)

instead of the literal "emacs", so that it invokes the same Emacs
binary as the one in which the command runs.

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[T.V Raman]

Thanks Eli, I didn't know about invocation-name, though I did want to
use the binary that had invoked Emacs, thanks for pointing it
out. I'll send an updated patch, I also need to make the process
buffer "hidden" by adding a space at the front of its name.
Eli Zaretskii writes:
 > > From: "T.V Raman" <raman@google.com>
 > > Cc: arthur.miller@live.com,  emacs-devel@gnu.org
 > > Date: Wed, 04 Aug 2021 17:22:23 -0700
 > > 
 > > Here is a simple patch that rebuilds package-quickstart-file in a
 > > separate emacs process:
 > 
 > Thanks.  I have one minor comment:
 > 
 > > -      (package-quickstart-refresh)
 > > +      (start-process
 > > +       "quickstart" "*quickstart*" "emacs"
 > > +       "-batch" "-q"
 > > +       "-l" "package"
 > > +       "-f" "package-quickstart-refresh")
 > 
 > The 3rd argument of start-process should use
 > 
 >   (expand-file-name invocation-name invocation-directory)
 > 
 > instead of the literal "emacs", so that it invokes the same Emacs
 > binary as the one in which the command runs.

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
♉ Id: kg:/m/0285kf1  🦮

--

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
♉ Id: kg:/m/0285kf1  🦮

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 1642 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:


Here is the updated patch:
git diff master
diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 37dcbe36c8..4e59623cbc 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -4138,7 +4138,12 @@ package--quickstart-maybe-refresh
   (if package-quickstart
       ;; FIXME: Delay refresh in case we're installing/deleting
       ;; several packages!
-      (package-quickstart-refresh)
+      (start-process
+       "quickstart" " *quickstart*"
+       (expand-file-name invocation-name invocation-directory)
+       "-batch" "-q"
+       "-l" "package"
+       "-f" "package-quickstart-refresh")
     (delete-file (concat package-quickstart-file "c"))
     (delete-file package-quickstart-file)))
 
>> From: "T.V Raman" <raman@google.com>
>> Cc: arthur.miller@live.com,  emacs-devel@gnu.org
>> Date: Wed, 04 Aug 2021 17:22:23 -0700
>> 
>> Here is a simple patch that rebuilds package-quickstart-file in a
>> separate emacs process:
>
> Thanks.  I have one minor comment:
>
>> -      (package-quickstart-refresh)
>> +      (start-process
>> +       "quickstart" "*quickstart*" "emacs"
>> +       "-batch" "-q"
>> +       "-l" "package"
>> +       "-f" "package-quickstart-refresh")
>
> The 3rd argument of start-process should use
>
>   (expand-file-name invocation-name invocation-directory)
>
> instead of the literal "emacs", so that it invokes the same Emacs
> binary as the one in which the command runs.
>

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Stefan Monnier]

>    (if package-quickstart
>        ;; FIXME: Delay refresh in case we're installing/deleting
>        ;; several packages!
> -      (package-quickstart-refresh)
> +      (start-process
> +       "quickstart" " *quickstart*"
> +       (expand-file-name invocation-name invocation-directory)
> +       "-batch" "-q"
> +       "-l" "package"
> +       "-f" "package-quickstart-refresh")
>      (delete-file (concat package-quickstart-file "c"))
>      (delete-file package-quickstart-file)))

I think this will build an incorrect file for those users who change
`package`s config in their .emacs (e.g. setting
`package-directory-list`).


        Stefan

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 1014 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:


Interesting.

I went with -batch -q because your earlier message pointed toward doing
something that  was independent of the user's personal config?
>>    (if package-quickstart
>>        ;; FIXME: Delay refresh in case we're installing/deleting
>>        ;; several packages!
>> -      (package-quickstart-refresh)
>> +      (start-process
>> +       "quickstart" " *quickstart*"
>> +       (expand-file-name invocation-name invocation-directory)
>> +       "-batch" "-q"
>> +       "-l" "package"
>> +       "-f" "package-quickstart-refresh")
>>      (delete-file (concat package-quickstart-file "c"))
>>      (delete-file package-quickstart-file)))
>
> I think this will build an incorrect file for those users who change
> `package`s config in their .emacs (e.g. setting
> `package-directory-list`).
>
>
>         Stefan
>
>

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Stefan Monnier]

> I went with -batch -q because your earlier message pointed toward doing
> something that  was independent of the user's personal config?

Yes, we need `--batch`, but we also need to obey the user's settings.


        Stefan

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Eli Zaretskii]

> From: "T.V Raman" <raman@google.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  arthur.miller@live.com,  emacs-devel@gnu.org
> Date: Thu, 05 Aug 2021 08:21:28 -0700
> 
> I went with -batch -q because your earlier message pointed toward doing
> something that  was independent of the user's personal config?

I think we can address Stefan's concerns by using the suitable --eval
command-line arguments to the inferior Emacs subprocess, to pass it
the actual values of package-directory-list and other relevant
variables.

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[T.V Raman]

SG.

Do we know what other "relevant variables" are here? I'm not
intimately familiar with  the package system design, so at this point
I'm happy to let someone else take over this patch.

Eli Zaretskii writes:
 > > From: "T.V Raman" <raman@google.com>
 > > Cc: Eli Zaretskii <eliz@gnu.org>,  arthur.miller@live.com,  emacs-devel@gnu.org
 > > Date: Thu, 05 Aug 2021 08:21:28 -0700
 > > 
 > > I went with -batch -q because your earlier message pointed toward doing
 > > something that  was independent of the user's personal config?
 > 
 > I think we can address Stefan's concerns by using the suitable --eval
 > command-line arguments to the inferior Emacs subprocess, to pass it
 > the actual values of package-directory-list and other relevant
 > variables.

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
♉ Id: kg:/m/0285kf1  🦮

--

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
♉ Id: kg:/m/0285kf1  🦮

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Eli Zaretskii]

> From: "T.V Raman" <raman@google.com>
> Date: Thu, 5 Aug 2021 09:57:01 -0700
> Cc: raman@google.com,
>     monnier@iro.umontreal.ca,
>     arthur.miller@live.com,
>     emacs-devel@gnu.org
> 
> SG.
> 
> Do we know what other "relevant variables" are here? I'm not
> intimately familiar with  the package system design, so at this point
> I'm happy to let someone else take over this patch.

I personally don't know.  Basically, one should look at the defcustoms
in package.el and decide which ones are relevant, I hope very few are.

Perhaps Stefan could provide the list of such variables?

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Stefan Monnier]

> Do we know what other "relevant variables" are here?

If we're lucky the vars defined in package.el are all we need.
But of course, the users may also be using a setup where the installed
packages only work after running some of the `.emacs`.

So one option is to transfer to the subprocess the current value of the
vars defined in `package.el`.  Another is to tell the subprocess to read
the init files.

BTW, all these issues are faced (and partly solved) by the `async`
package we have in GNU ELPA.  We really should try and include it into
Emacs core.


        Stefan

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[T.V Raman]

SG, Perhaps we wait till async is in emacs core and then revisit this.

Stefan Monnier writes:
 > > Do we know what other "relevant variables" are here?
 > 
 > If we're lucky the vars defined in package.el are all we need.
 > But of course, the users may also be using a setup where the installed
 > packages only work after running some of the `.emacs`.
 > 
 > So one option is to transfer to the subprocess the current value of the
 > vars defined in `package.el`.  Another is to tell the subprocess to read
 > the init files.
 > 
 > BTW, all these issues are faced (and partly solved) by the `async`
 > package we have in GNU ELPA.  We really should try and include it into
 > Emacs core.
 > 
 > 
 >         Stefan

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
♉ Id: kg:/m/0285kf1  🦮

--

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
♉ Id: kg:/m/0285kf1  🦮

On The Use(fullness) of make-thread Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 1210 bytes --]


I remember that a lot of work went into integrating thread-support into
Emacs a few years ago and it sounded really promising.

I just did the following somewhat unscientific test to measure how much
usage it gets:

In the Emacs/lisp directory

find . -name '*.el' | xargs grep '(make-thread'
shows 0 hits.

I have about 200 packages installed in ~/.emacs.d/elpa and the same
command there also yields 0 hits.


Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Do we know what other "relevant variables" are here?
>
> If we're lucky the vars defined in package.el are all we need.
> But of course, the users may also be using a setup where the installed
> packages only work after running some of the `.emacs`.
>
> So one option is to transfer to the subprocess the current value of the
> vars defined in `package.el`.  Another is to tell the subprocess to read
> the init files.
>
> BTW, all these issues are faced (and partly solved) by the `async`
> package we have in GNU ELPA.  We really should try and include it into
> Emacs core.
>
>
>         Stefan
>
>

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I went with -batch -q because your earlier message pointed toward doing
>> something that  was independent of the user's personal config?

Why do you even care for -Q option there? You would like to do it async,
so just start the process with user settings and forgett about it and
let it chew until it's done. You wouldn't need that thing until you
restart Emacs next time anyway, so if it takes cuple of seconds extras
do you care?

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Do we know what other "relevant variables" are here?
>
> If we're lucky the vars defined in package.el are all we need.
> But of course, the users may also be using a setup where the installed
> packages only work after running some of the `.emacs`.
>
> So one option is to transfer to the subprocess the current value of the
> vars defined in `package.el`.  Another is to tell the subprocess to read
> the init files.
>
> BTW, all these issues are faced (and partly solved) by the `async`
> package we have in GNU ELPA.  We really should try and include it into
> Emacs core.

Stefan prefer to save manpowers vs complicating things, so I think the
most pragmatic thing would be just to start emacs with user settings and
let create quickstart file. As I wrote in another mail, it is an async
process and result is not needed until next restart. Here I definitely
don't think it is worth complicating things to save a second or few.

If the user is sitting and waiting for the quicstart file to refresh to
restart Emacs, than the user would be using this synchronized anyway,
and miss the point of doing it async, isn't it so?

I guess the problem is that use-package has that :ensure flag, which
will download and install package when Emacs starts. I personally have
gone away from use-package, and do all isntallation "offline", not when
Emacs starts, i.e. I suggest manual installation (package-install) if
user would like to install a package and have it immidiately avialable
in current seesion. I don't know if that is problem that motivates
Raman for this change, just guessing, so forgive me if I am wrong here.

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Cc: "T.V Raman" <raman@google.com>,  Eli Zaretskii <eliz@gnu.org>,
>   emacs-devel@gnu.org
> Date: Fri, 06 Aug 2021 07:18:00 +0200
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> >> I went with -batch -q because your earlier message pointed toward doing
> >> something that  was independent of the user's personal config?
> 
> Why do you even care for -Q option there? You would like to do it async,
> so just start the process with user settings and forgett about it and
> let it chew until it's done.

Starting "emacs -batch" with user settings could easily fail, since
many settings in the init files are for interactive sessions, and will
signal errors when done in batch mode.

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Cc: "T.V Raman" <raman@google.com>,  Eli Zaretskii <eliz@gnu.org>,
>>   emacs-devel@gnu.org
>> Date: Fri, 06 Aug 2021 07:18:00 +0200
>> 
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> 
>> >> I went with -batch -q because your earlier message pointed toward doing
>> >> something that  was independent of the user's personal config?
>> 
>> Why do you even care for -Q option there? You would like to do it async,
>> so just start the process with user settings and forgett about it and
>> let it chew until it's done.
>
> Starting "emacs -batch" with user settings could easily fail, since
> many settings in the init files are for interactive sessions, and will
> signal errors when done in batch mode.

Aha. Ok. Fair enough. But why runing in batch mode at all. 

Why not start normal emacs server process on another socket and
ask it to refresh quickstart file form another client? Two processes,
but no tinkering needed? Would that work?

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Cc: monnier@iro.umontreal.ca,  raman@google.com,  emacs-devel@gnu.org
> Date: Fri, 06 Aug 2021 15:20:19 +0200
> 
> >> Why do you even care for -Q option there? You would like to do it async,
> >> so just start the process with user settings and forgett about it and
> >> let it chew until it's done.
> >
> > Starting "emacs -batch" with user settings could easily fail, since
> > many settings in the init files are for interactive sessions, and will
> > signal errors when done in batch mode.
> 
> Aha. Ok. Fair enough. But why runing in batch mode at all. 
> 
> Why not start normal emacs server process on another socket and
> ask it to refresh quickstart file form another client? Two processes,
> but no tinkering needed? Would that work?

You mean, start a new interactive session, open an Emacs frame, run
all the customizations, including perhaps restoring the last session
via desktop, etc.?  Doesn't sound right to me.

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Cc: monnier@iro.umontreal.ca,  raman@google.com,  emacs-devel@gnu.org
>> Date: Fri, 06 Aug 2021 15:20:19 +0200
>> 
>> >> Why do you even care for -Q option there? You would like to do it async,
>> >> so just start the process with user settings and forgett about it and
>> >> let it chew until it's done.
>> >
>> > Starting "emacs -batch" with user settings could easily fail, since
>> > many settings in the init files are for interactive sessions, and will
>> > signal errors when done in batch mode.
>> 
>> Aha. Ok. Fair enough. But why runing in batch mode at all. 
>> 
>> Why not start normal emacs server process on another socket and
>> ask it to refresh quickstart file form another client? Two processes,
>> but no tinkering needed? Would that work?
>
> You mean, start a new interactive session, open an Emacs frame, run
> all the customizations, including perhaps restoring the last session
> via desktop, etc.?

No, start new emacs server proces on it's own socket. Yes, the point is
to run user init file, since you said batch mode does not do that. So
what if interactive settings are re-read? It is async proces, and that
is what I said a second or two extra, who cares? I don't think you need
new frame. I don't remember if new emacsclient process is needed to just
send it message over socket to eval (package-quickstart-refresh) or not,
but even if it is needed, it does not matter. I don't use to do this so
I don't remember deatisl of the call, but wasn't there some thread
with some guy running multiplse servers for evaling completions like not
so long time ago?

>  Doesn't sound right to me.

Why? At least it is sure user session will be the same, and the extra
time to load emacs does not matter in this case. It is just a pragmatic
approach.

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Stefan Monnier]

> Starting "emacs -batch" with user settings could easily fail, since
> many settings in the init files are for interactive sessions, and will
> signal errors when done in batch mode.

Even without these concerns, there's the fact that `emacs --batch`
implies `-q` and that undoing/overriding this `-q` is not trivial if we
want to do it faithfully.  I think it might require changes to
`startup.el` to better expose the code that selects and loads the init
file(s).

As for the risk of those init files burping, maybe we can "live with it"
and claim it to be a pilot error: either the users fix their init files
or they can't use `package-quickstart`.  Luckily for us, daemon mode has
made it a bit less common for init files to fail in batch mode, because
if they do, they will also fail in daemon mode.


        Stefan

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[T.V Raman]

If we dont want to wait for async, another option is to create a
with-package-env macro in package.el that captures any package
variables that are deemed necessary, then do --eval  (with-pacjage-env
(package-quickstart-refresh))

this way we can progressively add whatever vars we discover are
necessary to snapshot.

Stefan Monnier writes:
 > > Starting "emacs -batch" with user settings could easily fail, since
 > > many settings in the init files are for interactive sessions, and will
 > > signal errors when done in batch mode.
 > 
 > Even without these concerns, there's the fact that `emacs --batch`
 > implies `-q` and that undoing/overriding this `-q` is not trivial if we
 > want to do it faithfully.  I think it might require changes to
 > `startup.el` to better expose the code that selects and loads the init
 > file(s).
 > 
 > As for the risk of those init files burping, maybe we can "live with it"
 > and claim it to be a pilot error: either the users fix their init files
 > or they can't use `package-quickstart`.  Luckily for us, daemon mode has
 > made it a bit less common for init files to fail in batch mode, because
 > if they do, they will also fail in daemon mode.
 > 
 > 
 >         Stefan

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
♉ Id: kg:/m/0285kf1  🦮

--

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
♉ Id: kg:/m/0285kf1  🦮

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Cc: monnier@iro.umontreal.ca,  raman@google.com,  emacs-devel@gnu.org
>> Date: Fri, 06 Aug 2021 15:20:19 +0200
>> 
>> >> Why do you even care for -Q option there? You would like to do it async,
>> >> so just start the process with user settings and forgett about it and
>> >> let it chew until it's done.
>> >
>> > Starting "emacs -batch" with user settings could easily fail, since
>> > many settings in the init files are for interactive sessions, and will
>> > signal errors when done in batch mode.
>> 
>> Aha. Ok. Fair enough. But why runing in batch mode at all. 
>> 
>> Why not start normal emacs server process on another socket and
>> ask it to refresh quickstart file form another client? Two processes,
>> but no tinkering needed? Would that work?
>
> You mean, start a new interactive session, open an Emacs frame, run
> all the customizations, including perhaps restoring the last session
> via desktop, etc.?  Doesn't sound right to me.

I mean something like this:

(start-process
       "quickstartserver" " *quickstartserver*"
       (expand-file-name invocation-name invocation-directory)
       "--daemon=quickstart-refresher")

(start-process
       "quickstartclient" " *quickstartclient*"
       "emacsclient"
       "--socket-name=quickstart-refresher"
       "--eval "
       "'(progn (require 'package) (package-quickstart-refresh) (kill-emacs))'")

I am not sure how do I pass eval argument to cient in
start-process. When I run above start process for server it runs, and I
can run belov progn to generate quickstart file and kill emacs from
terminal. But when I start client with start-process it does not seem to
run. I guess I am not getting arguments correctly, but that is the
idea. Sure less efficient then with --batch, but I don't think it
matters; it is run once in a while in async process.

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[chad]

[-- Attachment #1: Type: text/plain, Size: 3110 bytes --]

As near as I can tell, the moderate-sized potential problem with "just
start another full interactive session, and communicate with it" is the
near-certainty that some people's interactive startup isn't (effectively)
reentrant. There are lots of ways this can happen -- package auto-loading
and refreshing, cache files like desktop/places/recent/history/etc,
configurations that automatically open dedicated frames in various places,
multiple servers confusing emacsclient, and the like. (Back in the 90's and
early 00's, I used to see user configurations like this all the time.)

There is some mitigation for this effect in that it overlaps with
daemon-mode non-rerentrancy, and also some in that only a subset of users
will use package-quickstart. That's probably enough to put together an
experiment, but I would expect some (solvable!) problems to crop up.

On the other hand, if I understand T.V Raman's suggestion to put together a
package that learns which parts of the environment are necessary to share
between a configured emacs and a subsequent emacs batch process, it sounds
like a reasonable approach toward a similar middle ground, but starting
from the cleaner -Q state. If I'm reading async.el correctly, it already
has machinery for this in, for example, async-inject-variable.

(defun async-inject-variables

    (include-regexp &optional predicate exclude-regexp noprops)  "Return a
> `setq' form that replicates part of the calling environment.
> It sets the value for every variable matching INCLUDE-REGEXP and
> also PREDICATE.  It will not perform injection for any variable
> matching EXCLUDE-REGEXP (if present) or representing a `syntax-table'
> i.e. ending by \"-syntax-table\".
> When NOPROPS is non nil it tries to strip out text properties of each
> variable's value with `async-variables-noprops-function'.
> It is intended to be used as follows:
>     (async-start
>        \\=`(lambda ()
>           (require \\='smtpmail)
>           (with-temp-buffer
>             (insert ,(buffer-substring-no-properties (point-min)
> (point-max)))
>             ;; Pass in the variable environment for smtpmail
>             ,(async-inject-variables
> \"\\\\=`\\(smtpmail\\|\\(user-\\)?mail\\)-\")
>             (smtpmail-send-it)))
>        \\='ignore)"


It also has this option, which suggests that this issue comes up at least
some of the time, and is worked around in situ rather than solved at the
package level:

(defvar async-quiet-switch "-Q"
>   "The Emacs parameter to use to call emacs without config.
> Can be one of \"-Q\" or \"-q\".
> Default is \"-Q\" but it is sometimes useful to use \"-q\" to have a
> enhanced config or some more variables loaded.")


I think there's an interesting area to explore here, especially given the
relatively low utility of make-thread, the way cores are multiplying, and
the extra crunching-power desired by potential changes like native-comp,
json, lsp, and tree-sitter (which I assume, perhaps wrongly, will end up
with loadable binary objects for some/most languages), this seems like a
good problem to look at.

Hope that helps,
~Chad

[-- Attachment #2: Type: text/html, Size: 3925 bytes --]

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Clément Pit-Claudel]

On 8/5/21 2:11 PM, Stefan Monnier wrote:
>> Do we know what other "relevant variables" are here?> If we're
>> lucky the vars defined in package.el are all we need. But of
> course, the users may also be using a setup where the installed 
> packages only work after running some of the `.emacs`.

Why do we need to activate packages before regenerating package-quickstart?  Isn't it just a concatenation of all available autoload files?

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Eli Zaretskii]

> From: chad <yandros@gmail.com>
> Date: Fri, 6 Aug 2021 17:46:43 -0700
> Cc: Eli Zaretskii <eliz@gnu.org>, EMACS development team <emacs-devel@gnu.org>, 
> 	Stefan Monnier <monnier@iro.umontreal.ca>, raman <raman@google.com>
> 
> On the other hand, if I understand T.V Raman's suggestion to put together a package that learns which parts
> of the environment are necessary to share between a configured emacs and a subsequent emacs batch
> process, it sounds like a reasonable approach toward a similar middle ground, but starting from the cleaner
> -Q state.

I agree.

> If I'm reading async.el correctly, it already has machinery for this in, for example,
> async-inject-variable.

The machinery exists, but we'd still need to decide which variables
should be "injected".  Once we make that decision, do we really need
all the heavy lifting that Async lets us use?  Because adding --eval
to the command line for each variable is not really rocket science, is
it?

Bottom line: I don't think we need to wait for Async being in core to
solve this issue in a very satisfactory manner.

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Arthur Miller]

Clément Pit-Claudel <cpitclaudel@gmail.com> writes:

> On 8/5/21 2:11 PM, Stefan Monnier wrote:
>>> Do we know what other "relevant variables" are here?> If we're
>>> lucky the vars defined in package.el are all we need. But of
>> course, the users may also be using a setup where the installed 
>> packages only work after running some of the `.emacs`.
>
> Why do we need to activate packages before regenerating package-quickstart?  Isn't it just a concatenation of all available autoload files?

Yes it is, thats what package-quickstart-refresh does, just concatenate
autoloads file for activated packages.

Re: Async rebuild package-quickstart after packages update? was Re: 28.0.50; Proposal: slightly more efficient package-quickstart.el

[Arthur Miller]

chad <yandros@gmail.com> writes:

> As near as I can tell, the moderate-sized potential problem with "just start another full interactive session, and communicate with
> it" is the near-certainty that some people's interactive startup isn't (effectively) reentrant. There are lots of ways this can happen
> -- package auto-loading and refreshing, cache files like desktop/places/recent/history/etc, configurations that automatically open
> dedicated frames in various places, multiple servers confusing emacsclient, and the like. (Back in the 90's and early 00's, I used to
> see user configurations like this all the time.)

I thought that emacs does not open frames in daemon mode (more than one
invisible or how it works).


> There is some mitigation for this effect in that it overlaps with daemon-mode non-rerentrancy, and also some in that only a subset
> of users will use package-quickstart. That's probably enough to put together an experiment, but I would expect some (solvable!)
> problems to crop up.
>
> On the other hand, if I understand T.V Raman's suggestion to put together a package that learns which parts of the environment
> are necessary to share between a configured emacs and a subsequent emacs batch process, it sounds like a reasonable approach
> toward a similar middle ground, but starting from the cleaner -Q state. If I'm reading async.el correctly, it already has machinery
> for this in, for example, async-inject-variable.
>
>  (defun async-inject-variables
>
>      (include-regexp &optional predicate exclude-regexp noprops)  "Return a `setq' form that replicates part of the calling
>  environment.
>  It sets the value for every variable matching INCLUDE-REGEXP and
>  also PREDICATE.  It will not perform injection for any variable
>  matching EXCLUDE-REGEXP (if present) or representing a `syntax-table'
>  i.e. ending by \"-syntax-table\".
>  When NOPROPS is non nil it tries to strip out text properties of each
>  variable's value with `async-variables-noprops-function'.
>  It is intended to be used as follows:
>      (async-start
>         \\=`(lambda ()
>            (require \\='smtpmail)
>            (with-temp-buffer
>              (insert ,(buffer-substring-no-properties (point-min) (point-max)))
>              ;; Pass in the variable environment for smtpmail
>              ,(async-inject-variables \"\\\\=`\\(smtpmail\\|\\(user-\\)?mail\\)-\")
>              (smtpmail-send-it)))
>         \\='ignore)"  
>
> It also has this option, which suggests that this issue comes up at least some of the time, and is worked around in situ rather than
> solved at the package level:
>
>  (defvar async-quiet-switch "-Q"
>    "The Emacs parameter to use to call emacs without config.
>  Can be one of \"-Q\" or \"-q\".
>  Default is \"-Q\" but it is sometimes useful to use \"-q\" to have a
>  enhanced config or some more variables loaded.") 
>
> I think there's an interesting area to explore here, especially given the relatively low utility of make-thread, the way cores are
> multiplying, and the extra crunching-power desired by potential changes like native-comp, json, lsp, and tree-sitter (which I
> assume, perhaps wrongly, will end up with loadable binary objects for some/most languages), this seems like a good problem to
> look at.

I agree that it is an interesting problem, it seems like a lot of work.