The benchmark of Artanis: guile server, Fibers, and Ragnarok

The benchmark of Artanis: guile server, Fibers, and Ragnarok

[Nala Ginrut]

This is not a serious benchmark, since Artanis is still not  in the
stage for optimizing.

The test case is from the first test of Techempower, the json serialization:

https://www.techempower.com/benchmarks/#section=data-r13&hw=ph&test=json

According to Techempower, the response should look like:
--------------------------------------------
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 28
Server: Example
Date: Wed, 17 Apr 2013 12:00:00 GMT

{"message":"Hello, World!"}
---------------------------------------------


Linux system config:
-----------------/etc/sysctl.conf----------------
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 1200
net.ipv4.ip_local_port_range = 10000 65000
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_max_tw_buckets = 5000
net.core.wmem_default = 8388608
net.core.rmem_default = 8388608
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_timestamps = 0
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
----------------------------------------------------------


The Artanis code for test:
-------------code--------------
(get "/json" #:mime 'json
  (lambda (rc)
     (:mime rc "{\"message\":\"Hello, World!\"}")))
--------------end---------------

The benchmark tool is wrk:
-----------------------------------
sudo apt-get install wrk
-----------------------------------


The platform:
--------------------
Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
16G RAM
Debian9
GNU Artanis-0.2.5
Guile-2.2.3
Fibers  (commit b86405a2)
--------------------

The command:
-----------
wrk -c 100 -t 4 -d 10 http://localhost:3000/json
-----------
It means 100 connections keep open, 4 threads, 10 seconds.


======
1. Guile builtin server

----------run--------------
art work -s guile
-----------------------------

-c 100
--------
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    81.66ms  158.75ms   1.98s    96.69%
    Req/Sec   419.24     66.16   777.00     79.75%
  16704 requests in 10.01s, 2.88MB read
  Socket errors: connect 0, read 0, write 0, timeout 10
Requests/sec:   1669.45
Transfer/sec:    295.09KB
-----------------

-c 1000
----------------
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   203.02ms  121.22ms   1.90s    59.75%
    Req/Sec   428.52    115.43   780.00     70.25%
  17076 requests in 10.06s, 2.95MB read
  Socket errors: connect 0, read 0, write 0, timeout 39
Requests/sec:   1697.54
Transfer/sec:    300.05KB
-----------------


==========
Fibers

----------run---------
art work -s fibers
------------------------

-c 100
----------------
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    61.13ms    3.19ms  70.30ms   88.85%
    Req/Sec   409.62     79.96   505.00     55.25%
  16321 requests in 10.01s, 2.82MB read
Requests/sec:   1629.75
Transfer/sec:    288.07KB
----------------

-c 1000
--------------------
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   613.76ms   79.96ms 666.43ms   95.09%
    Req/Sec   395.84    121.35   707.00     68.75%
  15772 requests in 10.06s, 2.72MB read
Requests/sec:   1567.32
Transfer/sec:    277.04KB
--------------------


===============
Ragnarok
(server.multi requires Linux-3.9+)

-----------run----------
art work -s ragnarok
--------------------------

-c 100
----------------
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    95.32ms  106.54ms 546.36ms   83.09%
    Req/Sec   391.62    338.22     1.58k    76.96%
  15444 requests in 10.01s, 2.67MB read
Requests/sec:   1542.98
Transfer/sec:    272.73KB
----------------

-c 100 (enable multi-server, 4 instances)
----------------
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    26.26ms   24.61ms 149.54ms   54.36%
    Req/Sec     1.12k   166.39     1.82k    66.50%
  44749 requests in 10.01s, 7.72MB read
Requests/sec:   4468.86
Transfer/sec:    789.91KB
----------------

-c 1000
---------------
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   284.85ms  515.70ms   1.97s    82.14%
    Req/Sec   411.60    363.25     1.68k    74.80%
  15838 requests in 10.06s, 2.73MB read
  Socket errors: connect 0, read 0, write 0, timeout 1556
Requests/sec:   1574.00
Transfer/sec:    278.22KB
---------------

-c 1000 (enable multi-server, 4 instances)
----------------
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   209.78ms  154.24ms 559.29ms   53.19%
    Req/Sec     1.18k   365.76     2.56k    70.00%
  47131 requests in 10.06s, 8.14MB read
Requests/sec:   4684.08
Transfer/sec:    827.95KB
----------------



Seems there're lot of things need to be optimized. But in the
anticipated future, I'll only focus on stability and more features.

And please don't forget that the Guile compiler/VM has large room to
optimize, we can hope for the future.

If anyone wants to write a server-core with libuv or libev, feel free
to do it, I'll provide support if any necessary.

Happy hacking!

Re: The benchmark of Artanis: guile server, Fibers, and Ragnarok

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 3830 bytes --]

Thank you for sharing your numbers! I’ll focus on the largest change I see:

Nala Ginrut <nalaginrut@gmail.com> writes:
> -c 1000
> art work -s guile
> ----------------
>   Thread Stats   Avg      Stdev     Max   +/- Stdev
>     Latency   203.02ms  121.22ms   1.90s    59.75%

> art work -s fibers
> --------------------
>   Thread Stats   Avg      Stdev     Max   +/- Stdev
>     Latency   613.76ms   79.96ms 666.43ms   95.09%

> art work -s ragnarok; single instance
> --------------------
>   Thread Stats   Avg      Stdev     Max   +/- Stdev
>     Latency   284.85ms  515.70ms   1.97s    82.14%

You can see the systems cope differently with overload.

Ragnarok and pristine Guile both let specific requests starve, while
fibers accepts higher average latency to avoid high maximum latency.

Is this repeatable? If yes, then you can see the difference in
scheduling here: With fibers none of the 1000 requests has to wait more
than a second, while with pristine guile and with ragnarok some requests
can stall everything. If you have a lot of resources being loaded to
display a page, the maximum latency is the effective page load delay.

I left out the 4-instance ragnarok test, because its coping with latency
in the face of overload is not comparable, since it is less highly
overloaded (and that’s the feature which struck me while reading).

And anyway: These are already pretty good numbers. They don’t achieve
the level of static file serving with massive caching (in my tests
lighttpd could get more than a factor 2 increase over (fibers web
server)), but it’s already on a level where it could support around 500
active users on a single instance running on consumer hardware.

As reference the following are results when I test plain guile fibers
vs. lighttpd (the "it works" page) against my home server in my local
network:

Guile (fibers web server)
-------------------------

wrk -c 100 -t 4 -d 10
  4 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   138.76ms   46.83ms 318.43ms   80.22%
    Req/Sec   183.58     35.46   261.00     75.27%
  7401 requests in 10.00s, 664.93KB read
Requests/sec:    740.02
Transfer/sec:     66.49KB

wrk -c 1000 -t 4 -d 10
  4 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.45s   685.52ms   4.08s    84.22%
    Req/Sec   144.65     41.90   248.00     72.22%
  5768 requests in 10.03s, 518.22KB read
  Socket errors: connect 0, read 33, write 0, timeout 472
Requests/sec:    575.29
Transfer/sec:     51.69KB

(for the 1000 connections, I had to use ulimit -n 24000 as root to avoid
a "too many open files" error)


Lighttpd
--------

wrk -c 100 -t 4 -d 10
  4 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    65.60ms   43.02ms 416.93ms   78.83%
    Req/Sec   372.27     49.61   523.00     73.79%
  14755 requests in 10.00s, 4.96MB read
Requests/sec:   1475.31
Transfer/sec:    508.31KB

wrk -c 1000 -t 4 -d 10 http://d6.gnutella2.info
  4 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   129.14ms  145.20ms   1.56s    88.33%
    Req/Sec   372.59    140.95   728.00     60.26%
  14830 requests in 10.01s, 4.99MB read
  Socket errors: connect 0, read 4, write 0, timeout 3074
Requests/sec:   1481.89
Transfer/sec:    510.59KB


The Guile web server is just this:

    (import (fibers web server))
    (define (handler request body) (values '((content-type . (text/plain))) "Hello World!\n"))
    (run-server handler #:addr INADDR_ANY #:port 1234)


What I also see is that Artanis seems to have low overhead. How do the
numbers change with more complex pages?


Best wishes,
Arne

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1076 bytes --]

Re: The benchmark of Artanis: guile server, Fibers, and Ragnarok

[Nala Ginrut]

 hi Arne!
Thanks for the reply!


On Sat, May 12, 2018 at 4:00 AM, Arne Babenhauserheide <arne_bab@web.de> wrote:

> Ragnarok and pristine Guile both let specific requests starve, while
> fibers accepts higher average latency to avoid high maximum latency.
>
> Is this repeatable? If yes, then you can see the difference in
> scheduling here: With fibers none of the 1000 requests has to wait more
> than a second, while with pristine guile and with ragnarok some requests
> can stall everything. If you have a lot of resources being loaded to
> display a page, the maximum latency is the effective page load delay.

Yes, Ragnarok is not preempt-able yet, so it may delay too long when a
big request stall.
There're only 4 situations for a task could be scheduled:
1. I/O blocking
2. The socket buffer is full (users may tweak it)
3. Resources are insufficient to allocate (listening sockets, DB
connection pool, etc...)
4. Developers call (break-task) explicitly in the handler

I would like to make it preempt-able, but I still don't get the skill
how to do it from outside of a delimited-continuation.
And I would like to implement a better scheduler, for now it's just
simple FIFO. But I need to know how many size left
when suspend-able ports were blocking. It seems there's not interface
for me to get that size. Maybe there should be
a patch for it.


> I left out the 4-instance ragnarok test, because its coping with latency
> in the face of overload is not comparable, since it is less highly
> overloaded (and that’s the feature which struck me while reading).
>
> And anyway: These are already pretty good numbers. They don’t achieve
> the level of static file serving with massive caching (in my tests
> lighttpd could get more than a factor 2 increase over (fibers web
> server)), but it’s already on a level where it could support around 500
> active users on a single instance running on consumer hardware.

I have to mention that Django of Python got 700 req/s in the same test
under my same condition.
But it's trivial since Django is not good at performance but the
security and full-featured web stuffs.
I think the best choice is that to use Nginx for reverse-proxy, since
Nginx handles static files that may get 300,000 req/s though-out.
No one can compete with it for static files handling.

> What I also see is that Artanis seems to have low overhead. How do the
> numbers change with more complex pages?

I have to mention that even I have modified the code to do real json
serialization from assoc-list, the test result still remains.
But OK it's the credit of the author of guile-json ;-)
For more complex pages and DB based dynamic pages, I've tested before
roughly, it's not bad. And I have many ideas to optimize it. So
no hurry to test the current things.

I'm going to submit Artanis to Techempower for full tests, there'll be
more con-vincible test result. But before that, I have to finish all
the
features in my TODO, and make it more stable to avoid crash. The
version 0.2.5 is already very stable by eliminated many exceptions.
But I still need
more users to test it and feed back.

Best regards.

Re: The benchmark of Artanis: guile server, Fibers, and Ragnarok

[Nala Ginrut]

To whom may care, with our latest Guile-2.9.1, the requests/sec has
increased around 19.6% compared to the same test under this topic.
Nice job!

On Sat, May 12, 2018 at 11:43 PM Nala Ginrut <nalaginrut@gmail.com> wrote:
>
>  hi Arne!
> Thanks for the reply!
>
>
> On Sat, May 12, 2018 at 4:00 AM, Arne Babenhauserheide <arne_bab@web.de> wrote:
>
> > Ragnarok and pristine Guile both let specific requests starve, while
> > fibers accepts higher average latency to avoid high maximum latency.
> >
> > Is this repeatable? If yes, then you can see the difference in
> > scheduling here: With fibers none of the 1000 requests has to wait more
> > than a second, while with pristine guile and with ragnarok some requests
> > can stall everything. If you have a lot of resources being loaded to
> > display a page, the maximum latency is the effective page load delay.
>
> Yes, Ragnarok is not preempt-able yet, so it may delay too long when a
> big request stall.
> There're only 4 situations for a task could be scheduled:
> 1. I/O blocking
> 2. The socket buffer is full (users may tweak it)
> 3. Resources are insufficient to allocate (listening sockets, DB
> connection pool, etc...)
> 4. Developers call (break-task) explicitly in the handler
>
> I would like to make it preempt-able, but I still don't get the skill
> how to do it from outside of a delimited-continuation.
> And I would like to implement a better scheduler, for now it's just
> simple FIFO. But I need to know how many size left
> when suspend-able ports were blocking. It seems there's not interface
> for me to get that size. Maybe there should be
> a patch for it.
>
>
> > I left out the 4-instance ragnarok test, because its coping with latency
> > in the face of overload is not comparable, since it is less highly
> > overloaded (and that’s the feature which struck me while reading).
> >
> > And anyway: These are already pretty good numbers. They don’t achieve
> > the level of static file serving with massive caching (in my tests
> > lighttpd could get more than a factor 2 increase over (fibers web
> > server)), but it’s already on a level where it could support around 500
> > active users on a single instance running on consumer hardware.
>
> I have to mention that Django of Python got 700 req/s in the same test
> under my same condition.
> But it's trivial since Django is not good at performance but the
> security and full-featured web stuffs.
> I think the best choice is that to use Nginx for reverse-proxy, since
> Nginx handles static files that may get 300,000 req/s though-out.
> No one can compete with it for static files handling.
>
> > What I also see is that Artanis seems to have low overhead. How do the
> > numbers change with more complex pages?
>
> I have to mention that even I have modified the code to do real json
> serialization from assoc-list, the test result still remains.
> But OK it's the credit of the author of guile-json ;-)
> For more complex pages and DB based dynamic pages, I've tested before
> roughly, it's not bad. And I have many ideas to optimize it. So
> no hurry to test the current things.
>
> I'm going to submit Artanis to Techempower for full tests, there'll be
> more con-vincible test result. But before that, I have to finish all
> the
> features in my TODO, and make it more stable to avoid crash. The
> version 0.2.5 is already very stable by eliminated many exceptions.
> But I still need
> more users to test it and feed back.
>
> Best regards.

Re: The benchmark of Artanis: guile server, Fibers, and Ragnarok

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 332 bytes --]


Nala Ginrut <nalaginrut@gmail.com> writes:

> To whom may care, with our latest Guile-2.9.1, the requests/sec has
> increased around 19.6% compared to the same test under this topic.
> Nice job!

That’s great! Thank you for checking!

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1076 bytes --]