[ENet-discuss] Bandwidth/latency optimization

Wed Dec 4 00:16:40 PST 2013

My experiences as for Cube/Sauerbraten:

1. packet size

Do not hand ENet lots of tiny packets. This is bad. Use the ENet packets as
batches, i.e. take a bunch of your little user-level with their own
specialized little headers (in Sauerbraten these headers are only 1 byte
per packet!). However, keep these batch sizes underneath the ENet MTU with
some small reserved space for ENet headers, i.e. peer->mtu - 100 should be
good for limiting.

The reason this became important for Sauerbraten is we were sending *lots*
of these, so ENet packet headers would have become a significant fraction
of the total size, since the payloads of these packets are small. That
Sauerbraten's internal protocol can get this down to 1 byte sub-header
essentially gets rid of that issue. These are then batched into
peer->mtu-100 ENet packets as described above to avoid triggering ENet's
transparent fragmentation system.

2. compression

Pack your data intelligently in the first place, and you won't get much if
any benefit from compression. At least Sauerbraten's packets get at most
only 5-10% more out of ENet's range coder because we encode, say, integers
using an extension scheme that uses as few bytes as necessary (not unlike
how, say, UTF8 works). So, just be as smart as possible on the encoding up
front, and you can mostly avoid compression and save a lot of the CPU time
that you would have normally spent on that.

Failing that, others have been happy enough to stick with ENet's built-in
range coder.

3. channels

Nothing wrong with a 2 channel setup, and is almost what I use for
Sauerbraten. I send bulk movement data on a channel that mostly is only
unreliable. Occasionally reliable stuff gets sequenced into that, but very
rarely (only for special events like teleports or jumppads). The other
channel I use for reliable stuff like gun shots, chat, etc. There are other
schemes like how Quake 3 works that can be used to send even that
'reliable' data as unreliable, but it wasn't terribly worth it for
Sauerbraten's case.

I also have another channel or two I use for file transfers and things that
I don't want stalling the flow of everything else for long durations.

4. server-cycle

Sauerbraten just because of its insane movement speed uses 33ms (~30 Hz).
Whereas for variants with slightly saner move speed, like Red
Eclipse/Tesseract/AssaultCube, I have preferred to settle on 40ms (25 Hz).
What matters most for this is the quality of your physics interpolation and
also that you do a lot of client-side prediction rather than waiting on
round-trips, then these small changes in rate don't matter quite as much,
but higher fidelity of input to the interpolation always helps.

On Tue, Dec 3, 2013 at 11:45 PM, Jérémy Richert
<jeremy.richert1 at gmail.com>wrote:

> Hi all,
>
> I am currently developing a multiplayer FPS, and I am wondering how to
> tune ENet to have a network engine as optimized as possible on a bandwidth
> usage and a latency point of view.
> I would like to have your thougts on the following aspects:
>
> ----------------
> 1. Packet size
> ----------------
>
> I was wondering whether it was better to send large packets or to divide
> them into small packets.
>
> Pros of the large packets :
> - Less overhead due to the protocols (8 bytes for UDP, 20 bytes for IPv4,
> 10 for ENet)
>
> Pros of the small packets :
> - When a packet is lost, less data is lost
> - When a reliable packet is lost, resending it requires less bandwidth
> - Lower latency
>
> From what I have read, most people agree that it is recommended to send
> small packets to avoid packet splitting. This means that the application
> has to ensure that the size of the data sent does not exceed the MTU. Also,
> as the MTU depends on the router, some people recommended to have packets
> that will never be splitted, i.e. <= 576 bytes.
> What is your experience on this point?
> For now I have capped the data size to 1500, but I am thinking of reducing
> it for a better latency. Does anyone know the typical packet loss rate?
>
> ----------------
> 2. Compression
> ----------------
>
> Has anyone used compression in a network engine? If yes, which compression
> algorithm? What was the average gain?
> I have read that John Carmack used the Huffman compression in the Quake 3
> network engine because it was well suited for network data compression, but
> I still need to find some time to implement it in my program and do some
> tests.
>
> ----------------
> 3. Channels
> ----------------
>
> What is your network channel policy? How many channels do you use? What do
> you send on each channel?
> At the moment I have 2 channels: one for sending unreliable data (a lot),
> another one for reliable data.
> I have chosen this organization to avoid blocking the unreliable data
> while waiting for an ACK for the reliable data. I am thinking of adding
> another channel to send high priority reliable data, but I am not sure of
> the benefit as I already group the reliable data before sending them to
> limit the blocking. It may be useful if the packet loss rate is too high.
>
> ----------------
> 4. Server cycle
> ----------------
>
> I know this aspect depends on the game type, but I would be interested in
> knowing how your applications work on this.
> On my side, based on Valve's introduction to network concepts (+ some
> readings on the UT and Quake network engines), I have decided to implement
> a 50-ms cycle on the server side. This means that the servers only updates
> the simulation and notifies the clients each 50 ms. Meanwhile, it only
> reads the network messages to empty the network event buffer and to handle
> the reliable packet sending.
> On the client side, I have introduced a voluntary delay of 70 ms, which
> goes unnoticed on a user point of view, but helps a lot for overall
> fluidity as the client will almost always have the following world update
> (unless a packet is lost).
> I will soon try to increase the server cycle time to 60 ms to gain 10-20%
> of bandwidth. I also plan to separate the display delay of the player (it
> will stay to 70 ms) and the rest of the world (increased to 100 ms).
> However, I am afraid of the impact on reactivity.
>
> Does anyone have a similar architecture? What is your experience on
> timings?
> If not, has anyone already developed a FPS or a game very dependant on the
> network speed? If yes, what are your advices?
>
> ----------------
> 5. Other
> ----------------
>
> If anyone has some useful advices on how to improve the network engine of
> a game/application, I would be pleased to hear it (well, to read it at
> least).
>
>
> Thanks in advance for all your experience sharing.
>
> Best regards,
> Jeremy Richert
>
> _______________________________________________
> ENet-discuss mailing list
> ENet-discuss at cubik.org
> http://lists.cubik.org/mailman/listinfo/enet-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cubik.org/pipermail/enet-discuss/attachments/20131204/44524423/attachment-0001.html>