bug#41116: Guix deploy fails with new version of Herd

bug#41116: Guix deploy fails with new version of Herd

[Alex Sassmannshausen via Bug reports for GNU Guix]

Hello,

I maintain a number of servers using Guix deploy.  It seems that the
recent upgrade to Herd in Guix, and specifically commit
4c0cc7bed3de2c0e2d3a6e95b88693941e839eec might have introduced a bug.

From my testing, guix deploy currently consistently fails with:
-----------------8<----------------------------->8-------------------
ice-9/boot-9.scm:1667:16: In procedure raise-exception:
ERROR:
  1. &inferior-exception:
      arguments: (srfi-34 #<inferior-object #<condition &action-exception-error [service: root action: eval key: keyword-argument-error args: ("#<procedure 7fe24816e240 at shepherd/service.scm:903:4 (command #:key user group directory environment-variables pid-file pid-file-timeout log-file) | (program . program-args)>" "Unrecognized keyword" () (#:file-creation-mask))] 7eff2bd7be00>>)
      inferior: #f
      stack: ()
-----------------8<----------------------------->8-------------------

A workaround is to build the system configuration locally on the target
server, then to reconfigure.  It will still error at the same place, but
at this point, after restarting the server, the new version of Herd will
be running and both deploy and reconfigure will work.

I don't know what a good solution to this could be, but it may be
something we need to consider in future development of Herd.

Best wishes,

Alex

bug#41116: A naive proposal for a solution

[Alex Sassmannshausen via Bug reports for GNU Guix]

Upon thinking further about this it seems to me the problem is caused by
guix deploy attempting to restart services as well as it can during
deployment.  When this fails deployment fails.

guix system reconfigure on the other hand does not do this (afaik).  As
a result it can complete.

Once reconfigure is completed a reboot switches to the new system
version and is then thus able to restart the services.

If all this is correct, then the long-discussed guix deploy feature of
service restart policies would resolve this issue elegantly:
When a similar herd upgrade in future looms, a switch away from "restart
running services" to "no restart services" or "reboot after deployment"
would avoid this currently hard-coded failure mode.

Food for thought perhaps, if my understanding is anywhere close to
right, that is.

Alex

bug#41116: Guix deploy fails with new version of Herd

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 1845 bytes --]

Hi Alex,

Alex Sassmannshausen via Bug reports for GNU Guix <bug-guix@gnu.org>
writes:

> Hello,
>
> I maintain a number of servers using Guix deploy.  It seems that the
> recent upgrade to Herd in Guix, and specifically commit
> 4c0cc7bed3de2c0e2d3a6e95b88693941e839eec might have introduced a bug.
>
> From my testing, guix deploy currently consistently fails with:
> -----------------8<----------------------------->8-------------------
> ice-9/boot-9.scm:1667:16: In procedure raise-exception:
> ERROR:
>   1. &inferior-exception:
>       arguments: (srfi-34 #<inferior-object #<condition &action-exception-error [service: root action: eval key: keyword-argument-error args: ("#<procedure 7fe24816e240 at shepherd/service.scm:903:4 (command #:key user group directory environment-variables pid-file pid-file-timeout log-file) | (program . program-args)>" "Unrecognized keyword" () (#:file-creation-mask))] 7eff2bd7be00>>)
>       inferior: #f
>       stack: ()
> -----------------8<----------------------------->8-------------------
>
> A workaround is to build the system configuration locally on the target
> server, then to reconfigure.  It will still error at the same place, but
> at this point, after restarting the server, the new version of Herd will
> be running and both deploy and reconfigure will work.
>
> I don't know what a good solution to this could be, but it may be
> something we need to consider in future development of Herd.

This issue has been reported by a number of users on IRC.  I think the
problem is that the the #:file-creation-mask keyword requires support
from the running Shepherd, which may not have it yet.  I think we should
revert commit 4c0cc7bed3de2c0e2d3a6e95b88693941e839eec until we find a
smooth upgrade path.  Can you try it and push if that fixes guix deploy?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#41116: Guix deploy fails with new version of Herd

[Alex Sassmannshausen via Bug reports for GNU Guix]

Hi Marius,

Marius Bakke <mbakke@fastmail.com> writes:

> Hi Alex,
>
> [...]
>
> This issue has been reported by a number of users on IRC.  I think the
> problem is that the the #:file-creation-mask keyword requires support
> from the running Shepherd, which may not have it yet.  I think we should
> revert commit 4c0cc7bed3de2c0e2d3a6e95b88693941e839eec until we find a
> smooth upgrade path.  Can you try it and push if that fixes guix deploy?

I believe Ludovic has now done this.  I will test and close this bug
if it is now working.

Cheers,

Alex

bug#41116: Guix deploy fails with new version of Herd

[Ludovic Courtès]

Hello Alex & Marius,

Marius Bakke <mbakke@fastmail.com> skribis:

> Alex Sassmannshausen via Bug reports for GNU Guix <bug-guix@gnu.org>
> writes:
>
>> Hello,
>>
>> I maintain a number of servers using Guix deploy.  It seems that the
>> recent upgrade to Herd in Guix, and specifically commit
>> 4c0cc7bed3de2c0e2d3a6e95b88693941e839eec might have introduced a bug.
>>
>> From my testing, guix deploy currently consistently fails with:
>> -----------------8<----------------------------->8-------------------
>> ice-9/boot-9.scm:1667:16: In procedure raise-exception:
>> ERROR:
>>   1. &inferior-exception:
>>       arguments: (srfi-34 #<inferior-object #<condition &action-exception-error [service: root action: eval key: keyword-argument-error args: ("#<procedure 7fe24816e240 at shepherd/service.scm:903:4 (command #:key user group directory environment-variables pid-file pid-file-timeout log-file) | (program . program-args)>" "Unrecognized keyword" () (#:file-creation-mask))] 7eff2bd7be00>>)
>>       inferior: #f
>>       stack: ()
>> -----------------8<----------------------------->8-------------------
>>
>> A workaround is to build the system configuration locally on the target
>> server, then to reconfigure.  It will still error at the same place, but
>> at this point, after restarting the server, the new version of Herd will
>> be running and both deploy and reconfigure will work.
>>
>> I don't know what a good solution to this could be, but it may be
>> something we need to consider in future development of Herd.
>
> This issue has been reported by a number of users on IRC.  I think the
> problem is that the the #:file-creation-mask keyword requires support
> from the running Shepherd, which may not have it yet.  I think we should
> revert commit 4c0cc7bed3de2c0e2d3a6e95b88693941e839eec until we find a
> smooth upgrade path.  Can you try it and push if that fixes guix deploy?

I’ve reverted the patch in 5aa4d2dcf2f4f8786358feb45338893ed08a4cd9.

Diego: I guess we can reinstate the patch “later”, once Shepherd 0.8 can
be considered widespread.

More importantly, we should handle service reload failures more
gracefully, as proposed in <https://issues.guix.gnu.org/issue/30706>,
for both ‘reconfigure’ and ‘deploy’.

Thanks,
Ludo’.

bug#41116: Guix deploy fails with new version of Herd

[Diego Nicola Barbato]

Hey,

Ludovic Courtès <ludo@gnu.org> writes:

> Hello Alex & Marius,
>
> Marius Bakke <mbakke@fastmail.com> skribis:
>
>> Alex Sassmannshausen via Bug reports for GNU Guix <bug-guix@gnu.org>
>> writes:
>>
>>> Hello,
>>>
>>> I maintain a number of servers using Guix deploy.  It seems that the
>>> recent upgrade to Herd in Guix, and specifically commit
>>> 4c0cc7bed3de2c0e2d3a6e95b88693941e839eec might have introduced a bug.
>>>
>>> From my testing, guix deploy currently consistently fails with:
>>> -----------------8<----------------------------->8-------------------
>>> ice-9/boot-9.scm:1667:16: In procedure raise-exception:
>>> ERROR:
>>>   1. &inferior-exception:
>>>       arguments: (srfi-34 #<inferior-object #<condition &action-exception-error [service: root action: eval key: keyword-argument-error args: ("#<procedure 7fe24816e240 at shepherd/service.scm:903:4 (command #:key user group directory environment-variables pid-file pid-file-timeout log-file) | (program . program-args)>" "Unrecognized keyword" () (#:file-creation-mask))] 7eff2bd7be00>>)
>>>       inferior: #f
>>>       stack: ()
>>> -----------------8<----------------------------->8-------------------
>>>
>>> A workaround is to build the system configuration locally on the target
>>> server, then to reconfigure.  It will still error at the same place, but
>>> at this point, after restarting the server, the new version of Herd will
>>> be running and both deploy and reconfigure will work.
>>>
>>> I don't know what a good solution to this could be, but it may be
>>> something we need to consider in future development of Herd.
>>
>> This issue has been reported by a number of users on IRC.  I think the
>> problem is that the the #:file-creation-mask keyword requires support
>> from the running Shepherd, which may not have it yet.  I think we should
>> revert commit 4c0cc7bed3de2c0e2d3a6e95b88693941e839eec until we find a
>> smooth upgrade path.  Can you try it and push if that fixes guix deploy?
>
> I’ve reverted the patch in 5aa4d2dcf2f4f8786358feb45338893ed08a4cd9.
>
> Diego: I guess we can reinstate the patch “later”, once Shepherd 0.8 can
> be considered widespread.

I'm sorry I broke reconfigure and deploy.  I didn't consider testing
upgrading from before Shepherd 0.8 to after my change and I didn't even
think of deploy.  Going forth I'll leave messing with core functionality
to the pros.

> More importantly, we should handle service reload failures more
> gracefully, as proposed in <https://issues.guix.gnu.org/issue/30706>,
> for both ‘reconfigure’ and ‘deploy’.

Regards,

Diego

bug#41116: Guix deploy fails with new version of Herd

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 2988 bytes --]

Diego Nicola Barbato <dnbarbato@posteo.de> writes:

> Hey,
>
> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Hello Alex & Marius,
>>
>> Marius Bakke <mbakke@fastmail.com> skribis:
>>
>>> Alex Sassmannshausen via Bug reports for GNU Guix <bug-guix@gnu.org>
>>> writes:
>>>
>>>> Hello,
>>>>
>>>> I maintain a number of servers using Guix deploy.  It seems that the
>>>> recent upgrade to Herd in Guix, and specifically commit
>>>> 4c0cc7bed3de2c0e2d3a6e95b88693941e839eec might have introduced a bug.
>>>>
>>>> From my testing, guix deploy currently consistently fails with:
>>>> -----------------8<----------------------------->8-------------------
>>>> ice-9/boot-9.scm:1667:16: In procedure raise-exception:
>>>> ERROR:
>>>>   1. &inferior-exception:
>>>>       arguments: (srfi-34 #<inferior-object #<condition &action-exception-error [service: root action: eval key: keyword-argument-error args: ("#<procedure 7fe24816e240 at shepherd/service.scm:903:4 (command #:key user group directory environment-variables pid-file pid-file-timeout log-file) | (program . program-args)>" "Unrecognized keyword" () (#:file-creation-mask))] 7eff2bd7be00>>)
>>>>       inferior: #f
>>>>       stack: ()
>>>> -----------------8<----------------------------->8-------------------
>>>>
>>>> A workaround is to build the system configuration locally on the target
>>>> server, then to reconfigure.  It will still error at the same place, but
>>>> at this point, after restarting the server, the new version of Herd will
>>>> be running and both deploy and reconfigure will work.
>>>>
>>>> I don't know what a good solution to this could be, but it may be
>>>> something we need to consider in future development of Herd.
>>>
>>> This issue has been reported by a number of users on IRC.  I think the
>>> problem is that the the #:file-creation-mask keyword requires support
>>> from the running Shepherd, which may not have it yet.  I think we should
>>> revert commit 4c0cc7bed3de2c0e2d3a6e95b88693941e839eec until we find a
>>> smooth upgrade path.  Can you try it and push if that fixes guix deploy?
>>
>> I’ve reverted the patch in 5aa4d2dcf2f4f8786358feb45338893ed08a4cd9.
>>
>> Diego: I guess we can reinstate the patch “later”, once Shepherd 0.8 can
>> be considered widespread.
>
> I'm sorry I broke reconfigure and deploy.  I didn't consider testing
> upgrading from before Shepherd 0.8 to after my change and I didn't even
> think of deploy.  Going forth I'll leave messing with core functionality
> to the pros.

Mistakes happen, don't worry about it.

One thing that would be really useful and can prevent such situations in
the future is to have a "system test" that tries to run reconfigure from
the latest released version of Guix (currently 1.1.0).

There are already a few Shepherd tests in gnu/tests/base.scm and
gnu/tests/reconfigure.scm that can be used as inspiration.

Food for thought, patches welcome, etc.  :-)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#41116: Guix deploy fails with new version of Herd

[Ludovic Courtès]

Hi,

Marius Bakke <mbakke@fastmail.com> skribis:

> Diego Nicola Barbato <dnbarbato@posteo.de> writes:

[...]

>>> I’ve reverted the patch in 5aa4d2dcf2f4f8786358feb45338893ed08a4cd9.
>>>
>>> Diego: I guess we can reinstate the patch “later”, once Shepherd 0.8 can
>>> be considered widespread.
>>
>> I'm sorry I broke reconfigure and deploy.  I didn't consider testing
>> upgrading from before Shepherd 0.8 to after my change and I didn't even
>> think of deploy.  Going forth I'll leave messing with core functionality
>> to the pros.
>
> Mistakes happen, don't worry about it.

Yup!  Plus, the person who reviewed the patch, undoubtedly an equally
nice person, didn’t notice the issue either.  :-)

> One thing that would be really useful and can prevent such situations in
> the future is to have a "system test" that tries to run reconfigure from
> the latest released version of Guix (currently 1.1.0).
>
> There are already a few Shepherd tests in gnu/tests/base.scm and
> gnu/tests/reconfigure.scm that can be used as inspiration.

Yes.

I was also wondering if it would make sense for services to somehow
state the major+minor version they’re targeting.

Ludo’.