bug#35909: 27.0.50; Speed up untarring (and hence package installation)

bug#35909: 27.0.50; Speed up untarring (and hence package installation)

[Noam Postavsky]

Severity: wishlist

package.el uses tar-mode.el to unpack tar files.  This can be very slow
for tar files which have many small files.  As an example see
https://elpa.gnu.org/packages/yasnippet-classic-snippets-1.0.2.tar.

tar-untar-buffer for that file currently takes 11.5 seconds on my
machine.  After (setq write-region-inhibit-fsync t) it takes 1.2
seconds.  When further binding inhibit-message to t around the call, it
takes 0.3 seconds.  Which is the same as running tar -xf.

Patch to follow.

bug#35909: 27.0.50; Speed up untarring (and hence package installation)

[Noam Postavsky]

[-- Attachment #1: Type: text/plain, Size: 612 bytes --]

tags 35909 + patch
quit

> package.el uses tar-mode.el to unpack tar files.  This can be very slow
> for tar files which have many small files.  As an example see
> https://elpa.gnu.org/packages/yasnippet-classic-snippets-1.0.2.tar.
>
> tar-untar-buffer for that file currently takes 11.5 seconds on my
> machine.  After (setq write-region-inhibit-fsync t) it takes 1.2
> seconds.  When further binding inhibit-message to t around the call, it
> takes 0.3 seconds.  Which is the same as running tar -xf.

The patch replaces message with (an enhanced) progress-reporter-update
rather than using inhibit-message.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: patch --]
[-- Type: text/x-diff, Size: 9706 bytes --]

From dba330e96a99e20a57ce4a1efd3d5851dcdfb6ae Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs@gmail.com>
Date: Sat, 25 May 2019 19:44:41 -0400
Subject: [PATCH] Let untarring (and hence package installation) go faster
 (Bug#35909)

* lisp/subr.el (progress-reporter-update)
(progress-reporter-force-update, progress-reporter-do-update): Accept
new optional argument, SUFFIX.
* doc/lispref/display.texi (Progress): Document it.
* etc/NEWS: Announce it.
* lisp/tar-mode.el (tar-untar-buffer): Use a progress reporter instead
of calling message.  Suppress message from write-region.  Let-bind
write-region-inhibit-fsync to t.
---
 doc/lispref/display.texi | 11 ++++++++---
 etc/NEWS                 |  4 ++++
 lisp/subr.el             | 49 ++++++++++++++++++++++++++++++------------------
 lisp/tar-mode.el         | 13 ++++++++-----
 4 files changed, 51 insertions(+), 26 deletions(-)

diff --git a/doc/lispref/display.texi b/doc/lispref/display.texi
index a2ed4b3891..22b9b7ff0b 100644
--- a/doc/lispref/display.texi
+++ b/doc/lispref/display.texi
@@ -426,7 +426,7 @@ Progress
 message is printed immediately.
 @end defun
 
-@defun progress-reporter-update reporter &optional value
+@defun progress-reporter-update reporter &optional value suffix
 This function does the main work of reporting progress of your
 operation.  It displays the message of @var{reporter}, followed by
 progress percentage determined by @var{value}.  If percentage is zero,
@@ -440,6 +440,11 @@ Progress
 @code{make-progress-reporter}.  For instance, if you scan a buffer,
 then @var{value} should be the result of a call to @code{point}.
 
+Optional argument @var{suffix} is a string to be displayed after
+@var{reporter}'s main message and progress text.  If REPORTER is a
+non-numerical reporter, then @var{value} should be @code{nil}, or a
+string to use instead of @var{suffix}.
+
 This function respects @var{min-change} and @var{min-time} as passed
 to @code{make-progress-reporter} and so does not output new messages
 on every invocation.  It is thus very fast and normally you should not
@@ -447,11 +452,11 @@ Progress
 likely negate your effort.
 @end defun
 
-@defun progress-reporter-force-update reporter &optional value new-message
+@defun progress-reporter-force-update reporter &optional value new-message suffix
 This function is similar to @code{progress-reporter-update} except
 that it prints a message in the echo area unconditionally.
 
-The first two arguments have the same meaning as for
+@var{reporter}, @var{value}, and @var{suffix} have the same meaning as for
 @code{progress-reporter-update}.  Optional @var{new-message} allows
 you to change the message of the @var{reporter}.  Since this function
 always updates the echo area, such a change will be immediately
diff --git a/etc/NEWS b/etc/NEWS
index 8e8172a81b..99d8d2ad10 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -370,6 +370,10 @@ in tooltips, as it is not useful there.
 There are 2 new buffer local variables and 1 face to customize this
 mode they are described in the manual "(emacs) Display".
 
++++
+** 'progress-reporter-update' accepts a suffix string to display.
+
+
 \f
 * Editing Changes in Emacs 27.1
 
diff --git a/lisp/subr.el b/lisp/subr.el
index c97d9b96bd..bde4db192e 100644
--- a/lisp/subr.el
+++ b/lisp/subr.el
@@ -5010,7 +5010,8 @@ set-transient-map
 ;;			      MAX-VALUE
 ;;			      MESSAGE
 ;;			      MIN-CHANGE
-;;			      MIN-TIME])
+;;                            MIN-TIME
+;;                            MESSAGE-SUFFIX])
 ;;
 ;; This weirdness is for optimization reasons: we want
 ;; `progress-reporter-update' to be as fast as possible, so
@@ -5020,7 +5021,7 @@ set-transient-map
 ;; digits of precision, it doesn't really matter here.  On the other
 ;; hand, it greatly simplifies the code.
 
-(defsubst progress-reporter-update (reporter &optional value)
+(defsubst progress-reporter-update (reporter &optional value suffix)
   "Report progress of an operation in the echo area.
 REPORTER should be the result of a call to `make-progress-reporter'.
 
@@ -5029,14 +5030,17 @@ progress-reporter-update
  `make-progress-reporter'---then VALUE should be a number between
  MIN-VALUE and MAX-VALUE.
 
-If REPORTER is a non-numerical reporter, VALUE should be nil.
+Optional argument SUFFIX is a string to be displayed after
+REPORTER's main message and progress text.  If REPORTER is a
+non-numerical reporter, then VALUE should be nil, or a string to
+use instead of SUFFIX.
 
 This function is relatively inexpensive.  If the change since
 last update is too small or insufficient time has passed, it does
 nothing."
   (when (or (not (numberp value))      ; For pulsing reporter
 	    (>= value (car reporter))) ; For numerical reporter
-    (progress-reporter-do-update reporter value)))
+    (progress-reporter-do-update reporter value suffix)))
 
 (defun make-progress-reporter (message &optional min-value max-value
 				       current-value min-change min-time)
@@ -5080,26 +5084,28 @@ make-progress-reporter
 		       max-value
 		       message
 		       (if min-change (max (min min-change 50) 1) 1)
-		       min-time))))
+                       min-time
+                       ;; SUFFIX
+                       nil))))
     (progress-reporter-update reporter (or current-value min-value))
     reporter))
 
-(defun progress-reporter-force-update (reporter &optional value new-message)
+(defun progress-reporter-force-update (reporter &optional value new-message suffix)
   "Report progress of an operation in the echo area unconditionally.
 
-The first two arguments are the same as in `progress-reporter-update'.
+REPORTER, VALUE, and SUFFIX are the same as in `progress-reporter-update'.
 NEW-MESSAGE, if non-nil, sets a new message for the reporter."
   (let ((parameters (cdr reporter)))
     (when new-message
       (aset parameters 3 new-message))
     (when (aref parameters 0)
       (aset parameters 0 (float-time)))
-    (progress-reporter-do-update reporter value)))
+    (progress-reporter-do-update reporter value suffix)))
 
 (defvar progress-reporter--pulse-characters ["-" "\\" "|" "/"]
   "Characters to use for pulsing progress reporters.")
 
-(defun progress-reporter-do-update (reporter value)
+(defun progress-reporter-do-update (reporter value &optional suffix)
   (let* ((parameters   (cdr reporter))
 	 (update-time  (aref parameters 0))
 	 (min-value    (aref parameters 1))
@@ -5134,18 +5140,25 @@ progress-reporter-do-update
 	       (setcar reporter (ceiling (car reporter))))
 	     ;; Only print message if enough time has passed
 	     (when enough-time-passed
-	       (if (> percentage 0)
-		   (message "%s%d%%" text percentage)
-		 (message "%s" text)))))
+               (if suffix
+                   (aset parameters 6 suffix)
+                 (setq suffix (or (aref parameters 6) "")))
+               (if (> percentage 0)
+                   (message "%s%d%% %s" text percentage suffix)
+                 (message "%s %s" text suffix)))))
 	  ;; Pulsing indicator
 	  (enough-time-passed
-	   (let ((index (mod (1+ (car reporter)) 4))
-		 (message-log-max nil))
+           (when (and value (not suffix))
+             (setq suffix value))
+           (if suffix
+               (aset parameters 6 suffix)
+             (setq suffix (or (aref parameters 6) "")))
+           (let* ((index (mod (1+ (car reporter)) 4))
+                  (message-log-max nil)
+                  (pulse-char (aref progress-reporter--pulse-characters
+                                    index)))
 	     (setcar reporter index)
-	     (message "%s %s"
-		      text
-		      (aref progress-reporter--pulse-characters
-			    index)))))))
+             (message "%s %s %s" text pulse-char suffix))))))
 
 (defun progress-reporter-done (reporter)
   "Print reporter's message followed by word \"done\" in echo area."
diff --git a/lisp/tar-mode.el b/lisp/tar-mode.el
index c75fe7c373..26fa5e06b0 100644
--- a/lisp/tar-mode.el
+++ b/lisp/tar-mode.el
@@ -522,7 +522,8 @@ tar-untar-buffer
   "Extract all archive members in the tar-file into the current directory."
   (interactive)
   ;; FIXME: make it work even if we're not in tar-mode.
-  (let ((descriptors tar-parse-info))   ;Read the var in its buffer.
+  (let ((descriptors tar-parse-info)    ;Read the var in its buffer.
+        (reporter (make-progress-reporter "Extracting")))
     (with-current-buffer
         (if (tar-data-swapped-p) tar-data-buffer (current-buffer))
       (set-buffer-multibyte nil)          ;Hopefully, a no-op.
@@ -535,17 +536,19 @@ tar-untar-buffer
                (start (tar-header-data-start descriptor))
                (end (+ start (tar-header-size descriptor))))
           (unless (file-directory-p name)
-            (message "Extracting %s" name)
+            (progress-reporter-update reporter name)
             (if (and dir (not (file-exists-p dir)))
                 (make-directory dir t))
             (unless (file-directory-p name)
-	      (let ((coding-system-for-write 'no-conversion))
+              (let ((coding-system-for-write 'no-conversion)
+                    (write-region-inhibit-fsync t))
                 (when link-desc
                   (lwarn '(tar link) :warning
                          "Extracted `%s', %s, as a normal file"
                          name link-desc))
-		(write-region start end name)))
-            (set-file-modes name (tar-header-mode descriptor))))))))
+                (write-region start end name nil :nomessage)))
+            (set-file-modes name (tar-header-mode descriptor)))))
+      (progress-reporter-done reporter))))
 
 (defun tar-summarize-buffer ()
   "Parse the contents of the tar file in the current buffer."
-- 
2.11.0

bug#35909: 27.0.50; Speed up untarring (and hence package installation)

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@gmail.com>
> Date: Sat, 25 May 2019 20:24:46 -0400
> 
> +Optional argument @var{suffix} is a string to be displayed after
> +@var{reporter}'s main message and progress text.  If REPORTER is a
                                                        ^^^^^^^^
@var{reporter}

Thanks.

bug#35909: 27.0.50; Speed up untarring (and hence package installation)

[Noam Postavsky]

tags 35909 fixed
close 35909 27.1
quit

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Noam Postavsky <npostavs@gmail.com>
>> Date: Sat, 25 May 2019 20:24:46 -0400
>> 
>> +Optional argument @var{suffix} is a string to be displayed after
>> +@var{reporter}'s main message and progress text.  If REPORTER is a
>                                                         ^^^^^^^^
> @var{reporter}
>
> Thanks.

Fixed and pushed to master.

f81b812d75 2019-06-03T20:18:19-04:00 "Let untarring (and hence package installation) go faster (Bug#35909)"
https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=f81b812d75d33a7d24c8c0f46455bde9a7ee6840