bug#28756: Substitute download progress bar doesn't reach 100%

bug#28756: Substitute download progress bar doesn't reach 100%

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1679 bytes --]

I just upgraded to the latest Guix and, while downloading substitutes, I
noticed the progress bar seems to never reach 100%, as shown below. I'm
not sure if this started recently or not.

Downloading https://mirror.hydra.gnu.org/guix/nar/gzip/drm4pj1k5mkb5784i0rkqb0bg2z8lmyw-libabw-0.1.1...
 libabw-0.1.1  346KiB                                                                                                                              585KiB/s 00:00 [############        ]  60.0%

Downloading https://mirror.hydra.gnu.org/guix/nar/gzip/djqkmf30zrlwfprdqhmk4nm92iw8ac53-libe-book-0.1.2...
 libe-book-0.1.2  215KiB                                                                                                                           628KiB/s 00:00 [################### ]  96.6%

Downloading https://mirror.hydra.gnu.org/guix/nar/gzip/qjmg7jjcqvclxia9s9b5mw8whm6lpqbz-liblangtag-0.5.8...
 liblangtag-0.5.8  269KiB                                                                                                                          614KiB/s 00:00 [###############     ]  77.2%

Downloading https://mirror.hydra.gnu.org/guix/nar/gzip/7fz9q215cw3nnihr31zmc372zjl00py8-libvisio-0.1.0...
 libvisio-0.1.0  975KiB                                                                                                                            887KiB/s 00:01 [############        ]  60.7%

Downloading https://mirror.hydra.gnu.org/guix/nar/gzip/544jcd4141xgg72dk5xxbs4zjzvxvvxi-nspr-4.17...
 nspr-4.17  404KiB                                                                                                                                 622KiB/s 00:00 [##########          ]  51.5%


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#28756: Substitute download progress bar doesn't reach 100%

[Ludovic Courtès]

Hello,

Leo Famulari <leo@famulari.name> skribis:

> I just upgraded to the latest Guix and, while downloading substitutes, I
> noticed the progress bar seems to never reach 100%, as shown below. I'm
> not sure if this started recently or not.
>
> Downloading https://mirror.hydra.gnu.org/guix/nar/gzip/drm4pj1k5mkb5784i0rkqb0bg2z8lmyw-libabw-0.1.1...
>  libabw-0.1.1  346KiB

Indeed, I’ve noticed too, and I think it relates to the new progress
reporters.  宋文武, do you experience this as well?  Thoughts?

Thanks,
Ludo’.

bug#28756: Substitute download progress bar doesn't reach 100%

[宋文武]

ludo@gnu.org (Ludovic Courtès) writes:

> Hello,
>
> Leo Famulari <leo@famulari.name> skribis:
>
>> I just upgraded to the latest Guix and, while downloading substitutes, I
>> noticed the progress bar seems to never reach 100%, as shown below. I'm
>> not sure if this started recently or not.
>>
>> Downloading https://mirror.hydra.gnu.org/guix/nar/gzip/drm4pj1k5mkb5784i0rkqb0bg2z8lmyw-libabw-0.1.1...
>>  libabw-0.1.1  346KiB
>
> Indeed, I’ve noticed too, and I think it relates to the new progress
> reporters.  宋文武, do you experience this as well?  Thoughts?
>

Yes, thanks for the report, commit abaee53c8 should fix it.

I thought close the 'input' port in the 'progress-substitution' will
close the 'progress' port too, but it's not true.

bug#28756: Substitute download progress bar doesn't reach 100%

[宋文武]

[-- Attachment #1: Type: text/plain, Size: 819 bytes --]

iyzsong@member.fsf.org (宋文武) writes:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Hello,
>>
>> Leo Famulari <leo@famulari.name> skribis:
>>
>>> I just upgraded to the latest Guix and, while downloading substitutes, I
>>> noticed the progress bar seems to never reach 100%, as shown below. I'm
>>> not sure if this started recently or not.
>>>
>>> Downloading https://mirror.hydra.gnu.org/guix/nar/gzip/drm4pj1k5mkb5784i0rkqb0bg2z8lmyw-libabw-0.1.1...
>>>  libabw-0.1.1  346KiB
>>
>> Indeed, I’ve noticed too, and I think it relates to the new progress
>> reporters.  宋文武, do you experience this as well?  Thoughts?
>>
>
> Yes, thanks for the report, commit abaee53c8 should fix it.
>

Well, it didn't...  I have to learn it's a child process will read and
report the process:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-guix-substitute-Report-the-last-process-in-the-child.patch --]
[-- Type: text/x-patch, Size: 2221 bytes --]

From 93b42f62ece1ad5181ed1119fc750bcbb74c5d3c Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=E5=AE=8B=E6=96=87=E6=AD=A6?= <iyzsong@member.fsf.org>
Date: Sat, 14 Oct 2017 22:45:55 +0800
Subject: [PATCH] guix: substitute: Report the last process in the child
 process.

Fixes <https://bugs.gnu.org/28756>.

* guix/utils.scm (filtered-port): Close the 'input' port in the child process.
* guix/scripts/substitute.scm (progress-substitution): Close the 'progress'
port before 'restore-file'.
---
 guix/scripts/substitute.scm | 10 ++++++----
 guix/utils.scm              |  1 +
 2 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index 921a7c679..d175d3e84 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -959,16 +959,18 @@ DESTINATION as a nar file.  Verify the substitute against ACL."
                    (decompressed-port (and=> (narinfo-compression narinfo)
                                              string->symbol)
                                       progress)))
+      ;; A child process of 'decompressed-port' will read from this 'process'
+      ;; port and thus report the actual progress to the console.  As the
+      ;; parent process, we should close it at the start.
+      (close-port progress)
       ;; Unpack the Nar at INPUT into DESTINATION.
       (restore-file input destination)
       (close-port input)
-      (close-port progress)
+      (every (compose zero? cdr waitpid) pids)
 
       ;; Skip a line after what 'progress-reporter/file' printed, and another
       ;; one to visually separate substitutions.
-      (display "\n\n" (current-error-port))
-
-      (every (compose zero? cdr waitpid) pids))))
+      (display "\n\n" (current-error-port)))))
 
 \f
 ;;;
diff --git a/guix/utils.scm b/guix/utils.scm
index de4aa6531..f8cf11ccb 100644
--- a/guix/utils.scm
+++ b/guix/utils.scm
@@ -153,6 +153,7 @@ buffered data is lost."
                   (close-port in)
                   (dump-port input out))
                 (lambda ()
+                  (close input)
                   (false-if-exception (close out))
                   (primitive-_exit 0))))
              (child
-- 
2.13.3


[-- Attachment #3: Type: text/plain, Size: 100 bytes --]


With this patch, update the guix snapshot and do a system reconfigure,
this issue should be fixed!

bug#28756: Substitute download progress bar doesn't reach 100%

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 2315 bytes --]

Hello,

iyzsong@member.fsf.org (宋文武) skribis:

> iyzsong@member.fsf.org (宋文武) writes:
>
>> ludo@gnu.org (Ludovic Courtès) writes:
>>
>>> Hello,
>>>
>>> Leo Famulari <leo@famulari.name> skribis:
>>>
>>>> I just upgraded to the latest Guix and, while downloading substitutes, I
>>>> noticed the progress bar seems to never reach 100%, as shown below. I'm
>>>> not sure if this started recently or not.
>>>>
>>>> Downloading https://mirror.hydra.gnu.org/guix/nar/gzip/drm4pj1k5mkb5784i0rkqb0bg2z8lmyw-libabw-0.1.1...
>>>>  libabw-0.1.1  346KiB
>>>
>>> Indeed, I’ve noticed too, and I think it relates to the new progress
>>> reporters.  宋文武, do you experience this as well?  Thoughts?
>>>
>>
>> Yes, thanks for the report, commit abaee53c8 should fix it.
>>
>
> Well, it didn't...  I have to learn it's a child process will read and
> report the process:
>
> From 93b42f62ece1ad5181ed1119fc750bcbb74c5d3c Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?=E5=AE=8B=E6=96=87=E6=AD=A6?= <iyzsong@member.fsf.org>
> Date: Sat, 14 Oct 2017 22:45:55 +0800
> Subject: [PATCH] guix: substitute: Report the last process in the child
>  process.
>
> Fixes <https://bugs.gnu.org/28756>.
>
> * guix/utils.scm (filtered-port): Close the 'input' port in the child process.
> * guix/scripts/substitute.scm (progress-substitution): Close the 'progress'
> port before 'restore-file'.

[...]

> +      ;; A child process of 'decompressed-port' will read from this 'process'
                                                                  ^^^^^^^^^^^^^
… will read from PROGRESS

> +      ;; port and thus report the actual progress to the console.  As the
> +      ;; parent process, we should close it at the start.
> +      (close-port progress)
>        ;; Unpack the Nar at INPUT into DESTINATION.
>        (restore-file input destination)
>        (close-port input)
> -      (close-port progress)
> +      (every (compose zero? cdr waitpid) pids)

Since INPUT is a wrapper around PROGRESS, it seems weird to close
PROGRESS beforehand.

Shouldn’t ‘filtered-port’ always close INPUT in the parent process (the
second part of the patch you sent closes it in the child process before
quitting, but we should additionally close it in the parent):


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 542 bytes --]

diff --git a/guix/utils.scm b/guix/utils.scm
index de4aa6531..2cf9be36d 100644
--- a/guix/utils.scm
+++ b/guix/utils.scm
@@ -153,9 +153,11 @@ buffered data is lost."
                   (close-port in)
                   (dump-port input out))
                 (lambda ()
+                  (close-port input)
                   (false-if-exception (close out))
                   (primitive-_exit 0))))
              (child
+              (close-port input)
               (close-port out)
               (loop in (cons child pids)))))))))
 

[-- Attachment #3: Type: text/plain, Size: 21 bytes --]


Thanks,
Ludo’.

bug#28756: Substitute download progress bar doesn't reach 100%

[宋文武]

[-- Attachment #1: Type: text/plain, Size: 2457 bytes --]

ludo@gnu.org (Ludovic Courtès) writes:

> Hello,
>
> iyzsong@member.fsf.org (宋文武) skribis:
>
>> iyzsong@member.fsf.org (宋文武) writes:
>>
>>> ludo@gnu.org (Ludovic Courtès) writes:
>>>
>>>> Hello,
>>>>
>>>> Leo Famulari <leo@famulari.name> skribis:
>>>>
>>>>> I just upgraded to the latest Guix and, while downloading substitutes, I
>>>>> noticed the progress bar seems to never reach 100%, as shown below. I'm
>>>>> not sure if this started recently or not.
>>>>>
>>>>> Downloading https://mirror.hydra.gnu.org/guix/nar/gzip/drm4pj1k5mkb5784i0rkqb0bg2z8lmyw-libabw-0.1.1...
>>>>>  libabw-0.1.1  346KiB
>>>>
>>>> Indeed, I’ve noticed too, and I think it relates to the new progress
>>>> reporters.  宋文武, do you experience this as well?  Thoughts?
>>>>
>>>
>>> Yes, thanks for the report, commit abaee53c8 should fix it.
>>>
>>
>> Well, it didn't...  I have to learn it's a child process will read and
>> report the process:
>>
>> From 93b42f62ece1ad5181ed1119fc750bcbb74c5d3c Mon Sep 17 00:00:00 2001
>> From: =?UTF-8?q?=E5=AE=8B=E6=96=87=E6=AD=A6?= <iyzsong@member.fsf.org>
>> Date: Sat, 14 Oct 2017 22:45:55 +0800
>> Subject: [PATCH] guix: substitute: Report the last process in the child
>>  process.
>>
>> Fixes <https://bugs.gnu.org/28756>.
>>
>> * guix/utils.scm (filtered-port): Close the 'input' port in the child process.
>> * guix/scripts/substitute.scm (progress-substitution): Close the 'progress'
>> port before 'restore-file'.
>
> [...]
>
>> +      ;; A child process of 'decompressed-port' will read from this 'process'
>                                                                   ^^^^^^^^^^^^^
> … will read from PROGRESS
>
>> +      ;; port and thus report the actual progress to the console.  As the
>> +      ;; parent process, we should close it at the start.
>> +      (close-port progress)
>>        ;; Unpack the Nar at INPUT into DESTINATION.
>>        (restore-file input destination)
>>        (close-port input)
>> -      (close-port progress)
>> +      (every (compose zero? cdr waitpid) pids)
>
> Since INPUT is a wrapper around PROGRESS, it seems weird to close
> PROGRESS beforehand.
>
> Shouldn’t ‘filtered-port’ always close INPUT in the parent process (the
> second part of the patch you sent closes it in the child process before
> quitting, but we should additionally close it in the parent):
>

Yes, it makes sense:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-guix-substitute-Report-the-last-progress-in-the-chil.patch --]
[-- Type: text/x-patch, Size: 2573 bytes --]

From 6a3fa69f96fc2e3e074a9275066640ddfee57fd2 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=E5=AE=8B=E6=96=87=E6=AD=A6?= <iyzsong@member.fsf.org>
Date: Sat, 14 Oct 2017 22:45:55 +0800
Subject: [PATCH] guix: substitute: Report the last progress in the child
 process.

Fixes <https://bugs.gnu.org/28756>.

* guix/utils.scm (filtered-port): Close the 'input' port for the current
process, and close it upon exit in the child process.
* guix/scripts/substitute.scm (progress-substitution): Display "\n\n" after
the reporter has finished.
---
 guix/scripts/substitute.scm | 11 +++++++----
 guix/utils.scm              |  2 ++
 2 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index 921a7c679..60dbdb176 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -956,19 +956,22 @@ DESTINATION as a nar file.  Verify the substitute against ACL."
                                      #:abbreviation nar-uri-abbreviation)))
                      (progress-report-port reporter raw)))
                   ((input pids)
+                   ;; NOTE: This 'progress' port of current process will be
+                   ;; closed here, while the child process doing the
+                   ;; reporting will close it upon exit.
                    (decompressed-port (and=> (narinfo-compression narinfo)
                                              string->symbol)
                                       progress)))
       ;; Unpack the Nar at INPUT into DESTINATION.
       (restore-file input destination)
       (close-port input)
-      (close-port progress)
+
+      ;; Wait for the reporter to finish.
+      (every (compose zero? cdr waitpid) pids)
 
       ;; Skip a line after what 'progress-reporter/file' printed, and another
       ;; one to visually separate substitutions.
-      (display "\n\n" (current-error-port))
-
-      (every (compose zero? cdr waitpid) pids))))
+      (display "\n\n" (current-error-port)))))
 
 \f
 ;;;
diff --git a/guix/utils.scm b/guix/utils.scm
index de4aa6531..2cf9be36d 100644
--- a/guix/utils.scm
+++ b/guix/utils.scm
@@ -153,9 +153,11 @@ buffered data is lost."
                   (close-port in)
                   (dump-port input out))
                 (lambda ()
+                  (close-port input)
                   (false-if-exception (close out))
                   (primitive-_exit 0))))
              (child
+              (close-port input)
               (close-port out)
               (loop in (cons child pids)))))))))
 
-- 
2.13.3


[-- Attachment #3: Type: text/plain, Size: 34 bytes --]



Thanks for the help and review!

bug#28756: Substitute download progress bar doesn't reach 100%

[Ludovic Courtès]

iyzsong@member.fsf.org (宋文武) skribis:

> From 6a3fa69f96fc2e3e074a9275066640ddfee57fd2 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?=E5=AE=8B=E6=96=87=E6=AD=A6?= <iyzsong@member.fsf.org>
> Date: Sat, 14 Oct 2017 22:45:55 +0800
> Subject: [PATCH] guix: substitute: Report the last progress in the child
>  process.
>
> Fixes <https://bugs.gnu.org/28756>.
>
> * guix/utils.scm (filtered-port): Close the 'input' port for the current
> process, and close it upon exit in the child process.
> * guix/scripts/substitute.scm (progress-substitution): Display "\n\n" after
> the reporter has finished.

LGTM!

You could commit the utils.scm change separately because it’s a bug-fix
in its own right.

Thank you!

Ludo’.

bug#28756: Substitute download progress bar doesn't reach 100%

[宋文武]

ludo@gnu.org (Ludovic Courtès) writes:

> iyzsong@member.fsf.org (宋文武) skribis:
>
>> From 6a3fa69f96fc2e3e074a9275066640ddfee57fd2 Mon Sep 17 00:00:00 2001
>> From: =?UTF-8?q?=E5=AE=8B=E6=96=87=E6=AD=A6?= <iyzsong@member.fsf.org>
>> Date: Sat, 14 Oct 2017 22:45:55 +0800
>> Subject: [PATCH] guix: substitute: Report the last progress in the child
>>  process.
>>
>> Fixes <https://bugs.gnu.org/28756>.
>>
>> * guix/utils.scm (filtered-port): Close the 'input' port for the current
>> process, and close it upon exit in the child process.
>> * guix/scripts/substitute.scm (progress-substitution): Display "\n\n" after
>> the reporter has finished.
>
> LGTM!
>
> You could commit the utils.scm change separately because it’s a bug-fix
> in its own right.
>

Pushed, thanks!  I think the utils.scm change alone (without the remove
of '(close-port progress') and comment) didn't fix this particular bug
clearly so I didn't seperate it..

bug#28756: Substitute download progress bar doesn't reach 100%

[Ludovic Courtès]

iyzsong@member.fsf.org (宋文武) skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> iyzsong@member.fsf.org (宋文武) skribis:
>>
>>> From 6a3fa69f96fc2e3e074a9275066640ddfee57fd2 Mon Sep 17 00:00:00 2001
>>> From: =?UTF-8?q?=E5=AE=8B=E6=96=87=E6=AD=A6?= <iyzsong@member.fsf.org>
>>> Date: Sat, 14 Oct 2017 22:45:55 +0800
>>> Subject: [PATCH] guix: substitute: Report the last progress in the child
>>>  process.
>>>
>>> Fixes <https://bugs.gnu.org/28756>.
>>>
>>> * guix/utils.scm (filtered-port): Close the 'input' port for the current
>>> process, and close it upon exit in the child process.
>>> * guix/scripts/substitute.scm (progress-substitution): Display "\n\n" after
>>> the reporter has finished.
>>
>> LGTM!
>>
>> You could commit the utils.scm change separately because it’s a bug-fix
>> in its own right.
>>
>
> Pushed, thanks!  I think the utils.scm change alone (without the remove
> of '(close-port progress') and comment) didn't fix this particular bug
> clearly so I didn't seperate it..

Sounds reasonable.  Thank you!

Ludo’.

bug#28756: closed (Re: bug#28756: Substitute download progress bar doesn't reach 100%)

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 596 bytes --]

On Tue, Oct 17, 2017 at 11:12:04AM +0000, GNU bug Tracking System wrote:
> Your bug report
> 
> #28756: Substitute download progress bar doesn't reach 100%
> 
> which was filed against the guix package, has been closed.
> 
> The explanation is attached below, along with your original report.
> If you require more details, please reply to 28756@debbugs.gnu.org.

I noticed today that the progress bar does reach 100%, but then resets
to 0% after the download is complete. For example:

 python-2.7.13  15.0MiB                               0B/s 00:09 [                    ]   0.0%

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#28756: closed (Re: bug#28756: Substitute download progress bar doesn't reach 100%)

[Ludovic Courtès]

Leo Famulari <leo@famulari.name> skribis:

> On Tue, Oct 17, 2017 at 11:12:04AM +0000, GNU bug Tracking System wrote:
>> Your bug report
>> 
>> #28756: Substitute download progress bar doesn't reach 100%
>> 
>> which was filed against the guix package, has been closed.
>> 
>> The explanation is attached below, along with your original report.
>> If you require more details, please reply to 28756@debbugs.gnu.org.
>
> I noticed today that the progress bar does reach 100%, but then resets
> to 0% after the download is complete. For example:
>
>  python-2.7.13  15.0MiB                               0B/s 00:09 [                    ]   0.0%

I think that’s because we haven’t updated our ‘guix’ package to include
that commit, though.

Did you try with guix-daemon from master, as in:

  sudo -E ./pre-inst-env guix-daemon …

?

Ludo’.