High polling frequency of public-inbox-watch

High polling frequency of public-inbox-watch

[Jan Kiszka]

Hi,

I'm running public-inbox for some months very successfully on my server. Great work!

On thing though: I realized that the public-inbox-watch service is scanning the
configured maildirs with a sleep delay of just 2 seconds, and that causes a CPU
load of 20-30% on my box. Can this be configured to a more moderate rate? I
would have patched this already, but I'm not seeing in the code where these 2
seconds may come from.

Thanks,
Jan

Re: High polling frequency of public-inbox-watch

[Eric Wong]

Jan Kiszka <jan.kiszka@web.de> wrote:
> Hi,
> 
> I'm running public-inbox for some months very successfully on my server. Great work!

Good to know :>

> On thing though: I realized that the public-inbox-watch service is scanning the
> configured maildirs with a sleep delay of just 2 seconds, and that causes a CPU
> load of 20-30% on my box. Can this be configured to a more moderate rate? I
> would have patched this already, but I'm not seeing in the code where these 2
> seconds may come from.

I guess it's from Filesys::Notify::Simple.  Which OS are you using?

It should be able to use inotify / kqueue depending on whether
you're on Linux or FreeBSD (I haven't tested other BSDs).  It
uses Linux::Inotify2 on Linux which AFAIK all deb-based distros
will pull in, and Filesys::Notify::KQueue on KQueue-supported
systems.

Re: High polling frequency of public-inbox-watch

[Jan Kiszka]

On 11.08.19 18:51, Eric Wong wrote:
> Jan Kiszka <jan.kiszka@web.de> wrote:
>> Hi,
>>
>> I'm running public-inbox for some months very successfully on my server. Great work!
>
> Good to know :>
>
>> On thing though: I realized that the public-inbox-watch service is scanning the
>> configured maildirs with a sleep delay of just 2 seconds, and that causes a CPU
>> load of 20-30% on my box. Can this be configured to a more moderate rate? I
>> would have patched this already, but I'm not seeing in the code where these 2
>> seconds may come from.
>
> I guess it's from Filesys::Notify::Simple.  Which OS are you using?
>

Debian Linux, Stretch.

> It should be able to use inotify / kqueue depending on whether
> you're on Linux or FreeBSD (I haven't tested other BSDs).  It
> uses Linux::Inotify2 on Linux which AFAIK all deb-based distros
> will pull in, and Filesys::Notify::KQueue on KQueue-supported
> systems.
>

I was expecting inotify usage as well, but I'm not seeing that in strace, rather
a long *stat series, and then nanosleep(2)... Hah!

     apt install liblinux-inotify2-perl

solved it. Did I miss a note in the readme?

Jan

Re: High polling frequency of public-inbox-watch

[Eric Wong]

Jan Kiszka <jan.kiszka@web.de> wrote:
> On 11.08.19 18:51, Eric Wong wrote:
> > Linux::Inotify2 on Linux which AFAIK all deb-based distros
> > will pull in,

Oops, I was wrong on that :x

> I was expecting inotify usage as well, but I'm not seeing that in strace, rather
> a long *stat series, and then nanosleep(2)... Hah!
> 
>     apt install liblinux-inotify2-perl
> 
> solved it. Did I miss a note in the readme?

Nope, but it should probably be noted as an optional dependency
in INSTALL, along with KQueue dependencies for *BSDs.

[PATCH] INSTALL: recommend inotify|kqueue modules for -watch

[Eric Wong]

Jan Kiszka reported high polling frequency when using -watch.

It turns out OS-specific packages for Filesys::Notify::Simple do
not pull in interfaces to use kqueue or inotify, which are
required to perform power-efficient event-based wakeups on
Maildir writes.

Fix the name of the Filesys::Notify::Simple for FreeBSD while
we're at it.

Link: https://public-inbox.org/meta/c85803c6-6d77-a300-491a-9f310dd284c1@web.de/
---
 INSTALL | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/INSTALL b/INSTALL
index f7e5999e..aad52c7b 100644
--- a/INSTALL
+++ b/INSTALL
@@ -130,10 +130,17 @@ above, so there is no need to explicitly install them:
                                     typically installed alongside Perl5)
 
 - Filesys::Notify::Simple          deb: libfilesys-notify-simple-perl
-                                   pkg: pkg-Filesys-Notify-Simple
+                                   pkg: p5-Filesys-Notify-Simple
                                    rpm: perl-Filesys-Notify-Simple
                                    (for public-inbox-watch, pulled in by Plack)
 
+- Linux::Inotify                   deb: liblinux-inotify2-perl
+                                   rpm: perl-Linux-Inotify2
+                                   (for public-inbox-watch on Linux)
+
+- Filesys::Notify::KQueue          pkg: p5-Filesys-Notify-KQueue
+                                   (for public-inbox-watch on FreeBSD)
+
 - IO::Compress::Gzip               deb: perl-modules (or libio-compress-perl)
                                    pkg: perl5
                                    rpm: perl-IO-Compress