Debugging an mcron job

Debugging an mcron job

[Jack Hill]

[-- Attachment #1: Type: text/plain, Size: 1591 bytes --]

Hi Guix,

I'm looking for help or tips on debugging an mcron job. My goal is to run 
a site-specific telemetry script that I have packaged [0] in my channel. 
The script collects some information and sends it to a remote server. I am 
trying to run the script as an mcron job via a service I created in my 
channel [1].

[0] https://gitlab.oit.duke.edu/jackhill/guix-at-duke/blob/0ed0a8e7530ef32b9e268d43b36944e38defa5f1/guix-at-duke/packages/endpoint-management.scm#L98-140
[1] https://gitlab.oit.duke.edu/jackhill/guix-at-duke/blob/0ed0a8e7530ef32b9e268d43b36944e38defa5f1/guix-at-duke/services/endpoint-management.scm

So far, this looks good, and after pulling my channel, reconfiguring, and 
restarting mcron, I can see my job in herd schedule mcron:

"""
# herd schedule mcron
Tue Mar 12 00:51:00 2019 -0400
/gnu/store/61692zjpkhy3jqv4a7jk1k8a14k6yn8k-planisphere-report-0-0.1b606af/bin/planisphere-report
…
"""

However, this doesn't appear to work because after the appointed time, the 
remote server is not updated. If I run the planisphere-report command by 
hand, the remote server is updated.

If I replace the planisphere-report command in my job with 
"/run/current-system/profile/bin/touch /tmp/test-file", test-file is 
indeed touched, so I know that the jobs are running.

Being fairly new to Guix, and extremely new to mcron, I don't know how to 
debug further. Are jobs run in such a way that they don't have network 
access? How can I see any errors or output?

Best,
Jack

P.S. I'm happy to accept any comments about how to improve the code in my 
channel as well ☺.

Re: Debugging an mcron job

[Jack Hill]

On Mon, 11 Mar 2019, Jack Hill wrote:

> Hi Guix,
>
> I'm looking for help or tips on debugging an mcron job.

I haven't yet solved this, but have some done some more (potentially 
interesting) troubleshooting.

The problem only seems to present itself, when mcron is run from Shepherd. 
For testing purposes, I changed the schedule to run every minute. This is 
how mcron is being called from Shepherd:

# ps -ef|grep mcron
root     32463     1  0 15:11 ?        00:00:00 /gnu/store/mamwayq00mqs85kgs6ibww7xw6dy776s-mcron-1.1.1/bin/mcron /gnu/store/0h36hing715c57k9xz5f6fgm77krq4p7-mcron-job

If I then run the same thing from a root shell:

# /gnu/store/mamwayq00mqs85kgs6ibww7xw6dy776s-mcron-1.1.1/bin/mcron /gnu/store/0h36hing715c57k9xz5f6fgm77krq4p7-mcron-job
;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
;;;       or pass the --no-auto-compile argument to disable.
;;; compiling /gnu/store/0h36hing715c57k9xz5f6fgm77krq4p7-mcron-job
;;; compiled /root/.cache/guile/ccache/2.2-LE-8-3.A/gnu/store/0h36hing715c57k9xz5f6fgm77krq4p7-mcron-job.go

it works. What could be causing this?

The contents of the job file are:

(job "* * * * *" "/gnu/store/61692zjpkhy3jqv4a7jk1k8a14k6yn8k-planisphere-report-0-0.1b606af/bin/planisphere-report")

Best,
Jack

Re: Debugging an mcron job

[Jack Hill]

I've figured out what was going wrong with my job (by redirecting the 
output to a file). It couldn't verify the remote host's certificate:

urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726)>

Now to figure out why. I have nss-certs in my system packages.

Best,
Jack

Re: Debugging an mcron job

[Jack Hill]

On Tue, 12 Mar 2019, Jack Hill wrote:

> I've figured out what was going wrong with my job (by redirecting the output 
> to a file). It couldn't verify the remote host's certificate:
>
> urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate 
> verify failed (_ssl.c:726)>
>
> Now to figure out why. I have nss-certs in my system packages.

I've solved this for my service by adding SSL_CERT_DIR to the environment:

https://gitlab.oit.duke.edu/jackhill/guix-at-duke/blob/a9674ec7b0decac94745c91df886817e2fc8a91b/guix-at-duke/services/endpoint-management.scm

Is that a reasonable solution?

Best,
Jack

Re: Debugging an mcron job

[Ludovic Courtès]

Hello,

Jack Hill <jackhill@jackhill.us> skribis:

> On Tue, 12 Mar 2019, Jack Hill wrote:
>
>> I've figured out what was going wrong with my job (by redirecting
>> the output to a file). It couldn't verify the remote host's
>> certificate:
>>
>> urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED]
>> certificate verify failed (_ssl.c:726)>
>>
>> Now to figure out why. I have nss-certs in my system packages.
>
> I've solved this for my service by adding SSL_CERT_DIR to the environment:
>
> https://gitlab.oit.duke.edu/jackhill/guix-at-duke/blob/a9674ec7b0decac94745c91df886817e2fc8a91b/guix-at-duke/services/endpoint-management.scm
>
> Is that a reasonable solution?

I’m (very) late to the party but what you did looks like the right thing
to me.

Ludo’.

Re: Debugging an mcron job

[Ludovic Courtès]

Hello,

Jack Hill <jackhill@jackhill.us> skribis:

> On Tue, 12 Mar 2019, Jack Hill wrote:
>
>> I've figured out what was going wrong with my job (by redirecting
>> the output to a file). It couldn't verify the remote host's
>> certificate:
>>
>> urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED]
>> certificate verify failed (_ssl.c:726)>
>>
>> Now to figure out why. I have nss-certs in my system packages.
>
> I've solved this for my service by adding SSL_CERT_DIR to the environment:
>
> https://gitlab.oit.duke.edu/jackhill/guix-at-duke/blob/a9674ec7b0decac94745c91df886817e2fc8a91b/guix-at-duke/services/endpoint-management.scm
>
> Is that a reasonable solution?

I’m (very) late to the party but what you did looks like the right thing
to me.

Ludo’.

PS: Good to see you’re building interesting stuff with this!

Re: Debugging an mcron job

[Jack Hill]

[-- Attachment #1: Type: text/plain, Size: 650 bytes --]

On Mon, 1 Apr 2019, Ludovic Courtès wrote:

> Jack Hill <jackhill@jackhill.us> skribis:
>> I've solved this for my service by adding SSL_CERT_DIR to the environment:
>>
>> https://gitlab.oit.duke.edu/jackhill/guix-at-duke/blob/a9674ec7b0decac94745c91df886817e2fc8a91b/guix-at-duke/services/endpoint-management.scm
>>
>> Is that a reasonable solution?
>
> I’m (very) late to the party but what you did looks like the right thing
> to me.

No worries, I still find the sanity check helpful.

> PS: Good to see you’re building interesting stuff with this!

Me too :) I've really enjoyed working with the Guix and mcron building 
blocks.

Best,
Jack