[ENet-discuss] ENet scalability benchmark data

[Espen Overaae]

I've been running some benchmarks to see how many clients I could get
a single server process to serve, and ran into some interesting
bottlenecks.

I call enet_host_service 1000 times per second. I have 4 channels.
Each connected client sends a small packet every 100 ms, and gets
various amounts of data in return every 100 ms.

First the no connections and no traffic scenario:
With max peers at a low number like 64, cpu usage is 0%
With max peers at about 2 k, cpu usage is 2-3%
With max peers at about 8 k, cpu usage is about 9%, with most of it
being spent in this function:
72.74% - enet_protocol_dispatch_incoming_commands
Dropping the polling rate to 100 Hz reduces cpu usage to 1% for 8K max peers.


When I connect a bunch of peers and start sending data all over the place:

With 8 k max peers and 100 Hz polling rate, the server stays
responsive until about 2 k clients, and uses about 90% cpu.
Profiling shows nearly 25% of this time is spent in ENet:
12.83% - enet_protocol_dispatch_incoming_commands
9.31% - enet_protocol_send_outgoing_commands

With 8 k max peers and 1 kHz polling rate, the server is more
responsive all over, but still only handles about 2 k clients, and cpu
usage rises to about 150% (the server is multithreaded and running on
a quad-core).
Profiling shows more than 50% of this time is spent in ENet, which
translates to about 80% cpu usage for the thread servicing ENet.
The big culprits are, according to gprof:
27.35% - enet_protocol_dispatch_incoming_commands
26.32% - enet_protocol_send_outgoing_commands.

Creating two server processes with 2 k max peers each and a 1 kHz
polling rate, allows me to connect a total of 3.5 k clients spread
over the two processes before the servers become unresponsive. CPU use
with two server processes is about 150% for each process, 40%
system(kernel - I guess this is the time spent inside system calls)
time, and only 5% idle (the remaining 55% probably spent in the client
processes and other background processes).
Profiler still shows about 50% time spent in ENet:
29.43% - enet_protocol_dispatch_incoming_commands
18.00% - enet_protocol_send_outgoing_commands


These numbers do not show how much time is spent in system calls and
how much is spent in actual enet code, they only show the grand total
of time spent within the functions listed and all subfunctions. I
assume much of it is spent in ENet. Looking at the ENet code, I assume
increasing the number of channels would increase the cpu time spent in
ENet. The total throughput in these tests has been a few megabits per
second, most of it unreliable. Responsiveness is simply measured by
the percieved time it takes to connect a batch of 500 new clients and
seeing how many of them fail to connect.

The server processes did some computations on the data transmitted.
Previously I've done essentially nothing with them and the profiler
showed enet to use a greater share of the cpu time, but the total
number of clients I could connect remained fairly constant. Even with
4 server processes and no computations, the servers got unresponsive
when the total client number approached 4 k.

To get useful profiling information from multiple threads, I used
this: http://sam.zoy.org/writings/programming/gprof.html


Espen Overaae