System monitoring

System monitoring

[Pjotr Prins]

I was looking to deploy Nagios on our servers, but I am discouraged by
its architecture. I would like something minimalistic that can run
anywhere (including small routers).

System monitoring has a number of important components, but they could
all be simple and written in guile:

  1. a (small) monitoring daemon (say for monitoring a web end point,
     temperature or disk space)

  These would be run by shepherd and submit events to a message queue
  somewhere on the monitoring server (2)

  2. queue handler

  The queue handler sits on the monitoring server and drops messages
  into a database

  3. notification handler(s)

  Reads the database and sends out alerts

  4. curses and web-based monitors

  These tools just fetch data from the database and handle aggregation

I envisage rather simple tooling. 

I am raising this here to see if anyone has come up with similar or
partial solution(s). And to see who would be interested in such a
project.

How do we monitor the Guix servers right now?

Pj.

Re: System monitoring

[Nicolò Balzarotti]

Hello everybody!

Pjotr Prins <pjotr.public12@thebird.nl> writes:

> I was looking to deploy Nagios on our servers, but I am discouraged by
> its architecture. I would like something minimalistic that can run
> anywhere (including small routers).
>
I think zabbix should work, but I've never used it.  On the surface, it
seems to have a steep learning curve, but this is just my impression.

I manage a couple of personal servers and I wanted to do the same.  My
first approach was with julia, with the server (written using Mux.jl)
that takes data (as JSON objects) from "clients" and saves it to the
disk (and for warnings/error send it via matrix).  Clients were simple
programs, again written in julia, with a cron-like scheduler (Sched.jl)
to run them at fixed time.  So basically the idea was like yours.  My
main problem was with the data format. Since I wanted to store free disk
space, cpu usage, temperature from multiple sensors, I was not sure on
how to represent them in a nice way (and I never solved the problem, I
had separate metrics on separate logs).

> System monitoring has a number of important components, but they could
> all be simple and written in guile:
>
>   1. a (small) monitoring daemon (say for monitoring a web end point,
>      temperature or disk space)
>
>   These would be run by shepherd and submit events to a message queue
>   somewhere on the monitoring server (2)
>
>   2. queue handler
>
>   The queue handler sits on the monitoring server and drops messages
>   into a database
>
>   3. notification handler(s)
>
>   Reads the database and sends out alerts
>
>   4. curses and web-based monitors
>
>   These tools just fetch data from the database and handle aggregation
>
> I envisage rather simple tooling. 
>
> I am raising this here to see if anyone has come up with similar or
> partial solution(s). And to see who would be interested in such a
> project.
>

It was nothing serious and julia is not available (or at least a pain to
compile) on arm, so I agree that we can write things with guile.
I'll have more spare time at the end of February, but I'm willed to
help!

As a side note, maybe we should check how prometheous+graphana work.  I
read some blog post were it seems they work quite well.


> How do we monitor the Guix servers right now?
>
> Pj.

Thanks! Nicolò

Re: System monitoring

[Pjotr Prins]

On Sun, Dec 29, 2019 at 09:05:40PM +0100, Nicolò Balzarotti wrote:
> I think zabbix should work, but I've never used it.  On the surface, it
> seems to have a steep learning curve, but this is just my impression.

The problem with these systems is that they target (complex)
deployments that have people watching these systems. 

What I need is much simpler - I don't want to watch systems, but I
need a cursory idea of health of say 20-40 machines out there. I also
want something that can notify me if things go really wrong. For
example when backups fail. These are not massive requirements - just
something flexible! I used to have scripts for that that would
mail/text me. But that was all a bit ad hoc and I got tired of
maintaining them and I got tired of repeating notifications ;)

What would be really cool is to be able to use logic programming. It
would allow questions like:

  What services showed interruptions in the last month on low RAM
  machines that also ran guix < 1.0 and a specific version of nginx.

This would mean storing state of machines in a database that gets
updated by messages. It means a good message broker. It means that
every time you write a monitoring service, you'll have to write a
receiver to turn it into a datastructure something like miniKanren can
solve. Key is to make *creating* such small reporter/receiver tools
really easy.

Visualisations are less important - though I am sure some people enjoy
creating those.

I.e., what I have in mind is a different type of systems monitor: a
minimalistic system that is hackable and can work out of the box for
Guix systems and are really easy to extend.

I think if we can prototype something in the coming months it would
make a great GSoC project to build out functionality.

Pj.

Re: System monitoring

[Arun Isaac]

[-- Attachment #1: Type: text/plain, Size: 401 bytes --]


> I was looking to deploy Nagios on our servers, but I am discouraged by
> its architecture. I would like something minimalistic that can run
> anywhere (including small routers).

I have been looking for minimalistic monitoring systems too and it would
be great to have one written in Guile. I can't commit any labour at the
moment, but would be very happy to try it out if you write such a
system.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: System monitoring

[Gábor Boskovits]

One lightweight soultion on the client would be to provide a
prometheus exporter, implementing the spec here:
https://prometheus.io/docs/instrumenting/exposition_formats/
That would need a prometheus server to scrape it, but would be very lightweight.
Zabbix can be used also, Data can be pushed for example by zabbix_sender:
https://www.zabbix.com/documentation/2.4/manpages/zabbix_sender
Wdyt?

Arun Isaac <arunisaac@systemreboot.net> ezt írta (időpont: 2019. dec.
30., H, 16:36):
>
>
> > I was looking to deploy Nagios on our servers, but I am discouraged by
> > its architecture. I would like something minimalistic that can run
> > anywhere (including small routers).
>
> I have been looking for minimalistic monitoring systems too and it would
> be great to have one written in Guile. I can't commit any labour at the
> moment, but would be very happy to try it out if you write such a
> system.



-- 
OpenPGP Key Fingerprint: 7988:3B9F:7D6A:4DBF:3719:0367:2506:A96C:CF63:0B21

Re: System monitoring

[Pjotr Prins]

On Sun, Jan 05, 2020 at 08:39:03PM +0100, Gábor Boskovits wrote:
> One lightweight soultion on the client would be to provide a
> prometheus exporter, implementing the spec here:
> https://prometheus.io/docs/instrumenting/exposition_formats/
> That would need a prometheus server to scrape it, but would be very lightweight.
> Zabbix can be used also, Data can be pushed for example by zabbix_sender:
> https://www.zabbix.com/documentation/2.4/manpages/zabbix_sender
> Wdyt?

My thinking currently is to write something from scratch. Basically
system monitoring is about handling messages. I want something that is
distributed and does not depend on single servers. Also I want 2-way
communication AND broadcasting :). If we focus on getting messaging
right it will be trivial to tie it to existing engines like Zabbix.

Other requirements are that the base daemon(s) should be written in
guile, but the filters can be in any programming language. To
popularize a system we'll need to support python, for example. With
network interfaces that should not be too hard.

Note that a messaging system is generic. It would allow me to E-mail
or text messages to trigger on meetings in my diary. One of the features
of Google calender I appreciate. The idea is really to be able to
write applications that push messages in on one end, and pick them
up on the other to send out as notifications, or log them in a
database for statistics.

If I have time I'll do some prototyping this month. We should have a
working group during the Guix days!

Pj.