Borg takes forever to upgrade

Borg takes forever to upgrade

[HiPhish]

Hello everyone,

I was trying to upgrade my packages and Borg (the backup program) has been
taking over an hour already. I have eventually decided to cancel the upgrade
process and tried upgrading all packages except Borg. The upgrade process
finished without problems in a reasonable amount of time, so it is definitely
Borg where things get stuck.

I'm running Guix version 8cc3983a4d02a15ad4a863671c1a5a8b2b542625 after
having run `guix pull` today. Here is the output of top:

    $ top -u guixbuilder01

    top - 21:11:24 up  8:53,  4 users,  load average: 1.24, 1.54, 1.36
    Tasks: 252 total,   2 running, 189 sleeping,   0 stopped,   0 zombie
    %Cpu(s): 30.3 us,  4.4 sy,  0.0 ni, 63.0 id,  2.1 wa,  0.0 hi,  0.2 si,  0.0 st
    KiB Mem :  6094692 total,   324704 free,  3076416 used,  2693572 buff/cache
    KiB Swap:  2097148 total,  2088176 free,     8972 used.  2595796 avail Mem 

      PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                      
    28132 guixbui+  20   0  272972 264756  12804 R 100.0  4.3 118:56.74 .py.test-real                                                
     4312 guixbui+  20   0   49068  29268  11508 S   0.0  0.5   0:00.61 python                                                       
    27994 guixbui+  20   0  132792  18296   9452 S   0.0  0.3   0:00.18 guile

Does anyone have an idea of what is going on here?

Re: Borg takes forever to upgrade

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 863 bytes --]

On Thu, Mar 14, 2019 at 09:12:58PM +0100, HiPhish wrote:
> I was trying to upgrade my packages and Borg (the backup program) has been
> taking over an hour already. I have eventually decided to cancel the upgrade
> process and tried upgrading all packages except Borg. The upgrade process
> finished without problems in a reasonable amount of time, so it is definitely
> Borg where things get stuck.

It's probably stuck in the tests:

https://github.com/borgbackup/borg/issues/4350

Whether or not it gets stuck seems host-specific. It works on some
machines and not others. I recommend just waiting a while for a
substitute to become available.

I don't want to disable the tests here until we understand what's going
on — otherwise we might miss some real issue in how the software is
being built on the machines where the tests get stuck.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Borg takes forever to upgrade

[HiPhish]

On Thursday, 14 March 2019 21:31:15 CET you wrote:
> It's probably stuck in the tests:
> 
> https://github.com/borgbackup/borg/issues/4350
> 
> Whether or not it gets stuck seems host-specific. It works on some
> machines and not others. I recommend just waiting a while for a
> substitute to become available.
Sounds plausible. I'll just wait then, it's not like I need the update 
urgently.

> I don't want to disable the tests here until we understand what's going
> on — otherwise we might miss some real issue in how the software is
> being built on the machines where the tests get stuck.
Yes, sounds reasonable.

Re: Borg takes forever to upgrade

[Christopher Lemmer Webber]

HiPhish writes:

> On Thursday, 14 March 2019 21:31:15 CET you wrote:
>> It's probably stuck in the tests:
>> 
>> https://github.com/borgbackup/borg/issues/4350
>> 
>> Whether or not it gets stuck seems host-specific. It works on some
>> machines and not others. I recommend just waiting a while for a
>> substitute to become available.
> Sounds plausible. I'll just wait then, it's not like I need the update 
> urgently.
>
>> I don't want to disable the tests here until we understand what's going
>> on — otherwise we might miss some real issue in how the software is
>> being built on the machines where the tests get stuck.
> Yes, sounds reasonable.

I'll note that I hit this problem today too.  I did a guix package
upgrade and came back about 6 hours later to see what packages had been
upgraded.  Sadly it had spent all day spinning its wheels on Borg. :(

Re: Borg takes forever to upgrade

[Tobias Geerinckx-Rice]

HiPhish,

HiPhish wrote:
> Hello everyone,
>
> I was trying to upgrade my packages and Borg (the backup 
> program) has been
> taking over an hour already.

[…]

> I have eventually decided to cancel the upgrade
> process and tried upgrading all packages except Borg. The 
> upgrade process
> finished without problems in a reasonable amount of time, so it 
> is definitely
> Borg where things get stuck.

Interrupting after an hour is a bit too soon to say it's stuck. 
Just from experience, many backup programmes tend to have long 
test suites (a trait which they share with many databases and 
compressors, probably because they're a bit of both).  The build 
was chugging along.  What did the log say was happening?

What is strange here is that you didn't get a substitute for Borg. 
I did[0].  Are you deliberately building Borg locally?

Kind regards,

T G-R

[0]: For

  /gnu/store/qplgyx9h65k509l02bfc4cz9dlid00ck-borg-1.1.9.drv
→ /gnu/store/5fazf5kwalvv3r4bin74fzgxp8hmfdha-borg-1.1.9

which is a graft of

  /gnu/store/pgib6ji5792y739jwv90zk679jqxji1q-borg-1.1.9

Re: Borg takes forever to upgrade

[HiPhish]

No, I just let it upgrade like any other of my packages, and the other 
packages upgraded within a reasonable timeframe (I haven't timed it, but let's 
just say ten minutes or so). My first attempt to upgrade ran for I think three 
hours (after typing `guix package -u` to upgrade all packages), but that's 
just what it felt like, I was not timing it. Borg was the first package to get 
upgraded, so I wasn't sure if it was a problem with Borg or a general 
upgrading problem. Then I upgraded all packages except Borg and finally tried 
upgrading only Borg alone. After that second attempt I sent the email.

On Thursday, 14 March 2019 21:39:06 CET you wrote:
> HiPhish,
> [...]
> 
> Interrupting after an hour is a bit too soon to say it's stuck.
> Just from experience, many backup programmes tend to have long
> test suites (a trait which they share with many databases and
> compressors, probably because they're a bit of both).  The build
> was chugging along.  What did the log say was happening?
> 
> What is strange here is that you didn't get a substitute for Borg.
> I did[0].  Are you deliberately building Borg locally?
> 
> Kind regards,
> 
> T G-R
> 
> [0]: For
> 
>   /gnu/store/qplgyx9h65k509l02bfc4cz9dlid00ck-borg-1.1.9.drv
> → /gnu/store/5fazf5kwalvv3r4bin74fzgxp8hmfdha-borg-1.1.9
> 
> which is a graft of
> 
>   /gnu/store/pgib6ji5792y739jwv90zk679jqxji1q-borg-1.1.9