[ENet-discuss] ENet-discuss Digest, Vol 126, Issue 1

Jérémy Richert jeremy.richert1 at gmail.com
Thu Dec 5 10:24:27 PST 2013


Thanks for your quick and detailed answers!
I will follow your advices and lower the server cycle, it indeed seems to
be a good way to have a better overall reactivity.
As for compression, it may be interesting only for sending maps, but I will
put this at the end of the network to-do list :)

Just one additional question: do you know approximately the bandwidth usage
on the server side for the games you have work on?
My target is to be below 50 kB/s upload for 8 players and below 100 kB/s
for 16 players, to be able to host such a server on a "classic" Internet
line (in France at least, I do not know the max upload rate for other
countries...)

Best regards,
Jeremy Richert



On Wed, Dec 4, 2013 at 9:16 AM, <enet-discuss-request at cubik.org> wrote:

> Send ENet-discuss mailing list submissions to
>         enet-discuss at cubik.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://lists.cubik.org/mailman/listinfo/enet-discuss
> or, via email, send a message with subject or body 'help' to
>         enet-discuss-request at cubik.org
>
> You can reach the person managing the list at
>         enet-discuss-owner at cubik.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of ENet-discuss digest..."
>
> Today's Topics:
>
>    1.  Bandwidth/latency optimization (J?r?my Richert)
>    2. Re:  Bandwidth/latency optimization (Blair Holloway)
>    3. Re:  Bandwidth/latency optimization (Lee Salzman)
>
>
> ---------- Forwarded message ----------
> From: "Jérémy Richert" <jeremy.richert1 at gmail.com>
> To: enet-discuss at cubik.org
> Cc:
> Date: Tue, 3 Dec 2013 22:45:49 +0100
> Subject: [ENet-discuss] Bandwidth/latency optimization
> Hi all,
>
> I am currently developing a multiplayer FPS, and I am wondering how to
> tune ENet to have a network engine as optimized as possible on a bandwidth
> usage and a latency point of view.
> I would like to have your thougts on the following aspects:
>
> ----------------
> 1. Packet size
> ----------------
>
> I was wondering whether it was better to send large packets or to divide
> them into small packets.
>
> Pros of the large packets :
> - Less overhead due to the protocols (8 bytes for UDP, 20 bytes for IPv4,
> 10 for ENet)
>
> Pros of the small packets :
> - When a packet is lost, less data is lost
> - When a reliable packet is lost, resending it requires less bandwidth
> - Lower latency
>
> From what I have read, most people agree that it is recommended to send
> small packets to avoid packet splitting. This means that the application
> has to ensure that the size of the data sent does not exceed the MTU. Also,
> as the MTU depends on the router, some people recommended to have packets
> that will never be splitted, i.e. <= 576 bytes.
> What is your experience on this point?
> For now I have capped the data size to 1500, but I am thinking of reducing
> it for a better latency. Does anyone know the typical packet loss rate?
>
> ----------------
> 2. Compression
> ----------------
>
> Has anyone used compression in a network engine? If yes, which compression
> algorithm? What was the average gain?
> I have read that John Carmack used the Huffman compression in the Quake 3
> network engine because it was well suited for network data compression, but
> I still need to find some time to implement it in my program and do some
> tests.
>
> ----------------
> 3. Channels
> ----------------
>
> What is your network channel policy? How many channels do you use? What do
> you send on each channel?
> At the moment I have 2 channels: one for sending unreliable data (a lot),
> another one for reliable data.
> I have chosen this organization to avoid blocking the unreliable data
> while waiting for an ACK for the reliable data. I am thinking of adding
> another channel to send high priority reliable data, but I am not sure of
> the benefit as I already group the reliable data before sending them to
> limit the blocking. It may be useful if the packet loss rate is too high.
>
> ----------------
> 4. Server cycle
> ----------------
>
> I know this aspect depends on the game type, but I would be interested in
> knowing how your applications work on this.
> On my side, based on Valve's introduction to network concepts (+ some
> readings on the UT and Quake network engines), I have decided to implement
> a 50-ms cycle on the server side. This means that the servers only updates
> the simulation and notifies the clients each 50 ms. Meanwhile, it only
> reads the network messages to empty the network event buffer and to handle
> the reliable packet sending.
> On the client side, I have introduced a voluntary delay of 70 ms, which
> goes unnoticed on a user point of view, but helps a lot for overall
> fluidity as the client will almost always have the following world update
> (unless a packet is lost).
> I will soon try to increase the server cycle time to 60 ms to gain 10-20%
> of bandwidth. I also plan to separate the display delay of the player (it
> will stay to 70 ms) and the rest of the world (increased to 100 ms).
> However, I am afraid of the impact on reactivity.
>
> Does anyone have a similar architecture? What is your experience on
> timings?
> If not, has anyone already developed a FPS or a game very dependant on the
> network speed? If yes, what are your advices?
>
> ----------------
> 5. Other
> ----------------
>
> If anyone has some useful advices on how to improve the network engine of
> a game/application, I would be pleased to hear it (well, to read it at
> least).
>
>
> Thanks in advance for all your experience sharing.
>
> Best regards,
> Jeremy Richert
>
>
> ---------- Forwarded message ----------
> From: Blair Holloway <thynameisblair at chaosandcode.com>
> To: Discussion of the ENet library <enet-discuss at cubik.org>
> Cc:
> Date: Tue, 3 Dec 2013 15:16:54 -0800
> Subject: Re: [ENet-discuss] Bandwidth/latency optimization
> Here are my thoughts:
>
> 1) Packet size
>
> Smaller is nearly universally better, especially if any of your data are
> reliable, as your packets may quickly be eaten up by resent data over poor
> connections.
>
> A second advantage of smaller data sizes is that you can generally have a
> higher number of maximum players, or more stuff happening in your game.
>
> 2) Compression
>
> Of all the games I've worked on, we only ever used bit-packing or a range
> encoder to compress the data; we never used a more general-purpose
> compression algorithm.
>
> One exception was a game whose packets often contained a lot of repeated
> string data. We generated a dictionary at runtime using previously seen
> strings, and used indices into that dictionary during future transmissions.
>
> In general, the best "compression" we implemented was always "remove the
> need for X to be sent". :)
>
> 3) Channels
>
> Can you envision a need up-front for high-priority reliable data? I've
> never encountered such a case myself, but I *have* found cases where
> certain unreliable data need to be prioritized ahead of others. Sometimes
> this can be implemented as simply as "put the high priority stuff in the
> packet first; continue until pre-set limit reached".
>
> 4) Server cycle
>
> The only thing I can really say here is that the smaller simulation time
> step your server takes, the lesser your latency will be. Increasing the
> timestep will save you bandwidth, but it can make your game seem laggier.
> For an FPS, you probably want to aim for an even higher packet rate -- 30
> packets per second wouldn't be unusual.
>
> Cheers,
>
> - Blair
>
>
>
> On Tue, Dec 3, 2013 at 1:45 PM, Jérémy Richert <jeremy.richert1 at gmail.com>wrote:
>
>> Hi all,
>>
>> I am currently developing a multiplayer FPS, and I am wondering how to
>> tune ENet to have a network engine as optimized as possible on a bandwidth
>> usage and a latency point of view.
>> I would like to have your thougts on the following aspects:
>>
>> ----------------
>> 1. Packet size
>> ----------------
>>
>> I was wondering whether it was better to send large packets or to divide
>> them into small packets.
>>
>> Pros of the large packets :
>> - Less overhead due to the protocols (8 bytes for UDP, 20 bytes for IPv4,
>> 10 for ENet)
>>
>> Pros of the small packets :
>> - When a packet is lost, less data is lost
>> - When a reliable packet is lost, resending it requires less bandwidth
>> - Lower latency
>>
>> From what I have read, most people agree that it is recommended to send
>> small packets to avoid packet splitting. This means that the application
>> has to ensure that the size of the data sent does not exceed the MTU. Also,
>> as the MTU depends on the router, some people recommended to have packets
>> that will never be splitted, i.e. <= 576 bytes.
>> What is your experience on this point?
>> For now I have capped the data size to 1500, but I am thinking of
>> reducing it for a better latency. Does anyone know the typical packet loss
>> rate?
>>
>> ----------------
>> 2. Compression
>> ----------------
>>
>> Has anyone used compression in a network engine? If yes, which
>> compression algorithm? What was the average gain?
>> I have read that John Carmack used the Huffman compression in the Quake 3
>> network engine because it was well suited for network data compression, but
>> I still need to find some time to implement it in my program and do some
>> tests.
>>
>> ----------------
>> 3. Channels
>> ----------------
>>
>> What is your network channel policy? How many channels do you use? What
>> do you send on each channel?
>> At the moment I have 2 channels: one for sending unreliable data (a lot),
>> another one for reliable data.
>> I have chosen this organization to avoid blocking the unreliable data
>> while waiting for an ACK for the reliable data. I am thinking of adding
>> another channel to send high priority reliable data, but I am not sure of
>> the benefit as I already group the reliable data before sending them to
>> limit the blocking. It may be useful if the packet loss rate is too high.
>>
>> ----------------
>> 4. Server cycle
>> ----------------
>>
>> I know this aspect depends on the game type, but I would be interested in
>> knowing how your applications work on this.
>> On my side, based on Valve's introduction to network concepts (+ some
>> readings on the UT and Quake network engines), I have decided to implement
>> a 50-ms cycle on the server side. This means that the servers only updates
>> the simulation and notifies the clients each 50 ms. Meanwhile, it only
>> reads the network messages to empty the network event buffer and to handle
>> the reliable packet sending.
>> On the client side, I have introduced a voluntary delay of 70 ms, which
>> goes unnoticed on a user point of view, but helps a lot for overall
>> fluidity as the client will almost always have the following world update
>> (unless a packet is lost).
>> I will soon try to increase the server cycle time to 60 ms to gain 10-20%
>> of bandwidth. I also plan to separate the display delay of the player (it
>> will stay to 70 ms) and the rest of the world (increased to 100 ms).
>> However, I am afraid of the impact on reactivity.
>>
>> Does anyone have a similar architecture? What is your experience on
>> timings?
>> If not, has anyone already developed a FPS or a game very dependant on
>> the network speed? If yes, what are your advices?
>>
>> ----------------
>> 5. Other
>> ----------------
>>
>> If anyone has some useful advices on how to improve the network engine of
>> a game/application, I would be pleased to hear it (well, to read it at
>> least).
>>
>>
>> Thanks in advance for all your experience sharing.
>>
>> Best regards,
>> Jeremy Richert
>>
>> _______________________________________________
>> ENet-discuss mailing list
>> ENet-discuss at cubik.org
>> http://lists.cubik.org/mailman/listinfo/enet-discuss
>>
>>
>
>
> ---------- Forwarded message ----------
> From: Lee Salzman <lsalzman at gmail.com>
> To: Discussion of the ENet library <enet-discuss at cubik.org>
> Cc:
> Date: Wed, 4 Dec 2013 10:16:40 +0200
> Subject: Re: [ENet-discuss] Bandwidth/latency optimization
> My experiences as for Cube/Sauerbraten:
>
> 1. packet size
>
> Do not hand ENet lots of tiny packets. This is bad. Use the ENet packets
> as batches, i.e. take a bunch of your little user-level with their own
> specialized little headers (in Sauerbraten these headers are only 1 byte
> per packet!). However, keep these batch sizes underneath the ENet MTU with
> some small reserved space for ENet headers, i.e. peer->mtu - 100 should be
> good for limiting.
>
> The reason this became important for Sauerbraten is we were sending *lots*
> of these, so ENet packet headers would have become a significant fraction
> of the total size, since the payloads of these packets are small. That
> Sauerbraten's internal protocol can get this down to 1 byte sub-header
> essentially gets rid of that issue. These are then batched into
> peer->mtu-100 ENet packets as described above to avoid triggering ENet's
> transparent fragmentation system.
>
> 2. compression
>
> Pack your data intelligently in the first place, and you won't get much if
> any benefit from compression. At least Sauerbraten's packets get at most
> only 5-10% more out of ENet's range coder because we encode, say, integers
> using an extension scheme that uses as few bytes as necessary (not unlike
> how, say, UTF8 works). So, just be as smart as possible on the encoding up
> front, and you can mostly avoid compression and save a lot of the CPU time
> that you would have normally spent on that.
>
> Failing that, others have been happy enough to stick with ENet's built-in
> range coder.
>
> 3. channels
>
> Nothing wrong with a 2 channel setup, and is almost what I use for
> Sauerbraten. I send bulk movement data on a channel that mostly is only
> unreliable. Occasionally reliable stuff gets sequenced into that, but very
> rarely (only for special events like teleports or jumppads). The other
> channel I use for reliable stuff like gun shots, chat, etc. There are other
> schemes like how Quake 3 works that can be used to send even that
> 'reliable' data as unreliable, but it wasn't terribly worth it for
> Sauerbraten's case.
>
> I also have another channel or two I use for file transfers and things
> that I don't want stalling the flow of everything else for long durations.
>
> 4. server-cycle
>
> Sauerbraten just because of its insane movement speed uses 33ms (~30 Hz).
> Whereas for variants with slightly saner move speed, like Red
> Eclipse/Tesseract/AssaultCube, I have preferred to settle on 40ms (25 Hz).
> What matters most for this is the quality of your physics interpolation and
> also that you do a lot of client-side prediction rather than waiting on
> round-trips, then these small changes in rate don't matter quite as much,
> but higher fidelity of input to the interpolation always helps.
>
>
>
>
> On Tue, Dec 3, 2013 at 11:45 PM, Jérémy Richert <jeremy.richert1 at gmail.com
> > wrote:
>
>> Hi all,
>>
>> I am currently developing a multiplayer FPS, and I am wondering how to
>> tune ENet to have a network engine as optimized as possible on a bandwidth
>> usage and a latency point of view.
>> I would like to have your thougts on the following aspects:
>>
>> ----------------
>> 1. Packet size
>> ----------------
>>
>> I was wondering whether it was better to send large packets or to divide
>> them into small packets.
>>
>> Pros of the large packets :
>> - Less overhead due to the protocols (8 bytes for UDP, 20 bytes for IPv4,
>> 10 for ENet)
>>
>> Pros of the small packets :
>> - When a packet is lost, less data is lost
>> - When a reliable packet is lost, resending it requires less bandwidth
>> - Lower latency
>>
>> From what I have read, most people agree that it is recommended to send
>> small packets to avoid packet splitting. This means that the application
>> has to ensure that the size of the data sent does not exceed the MTU. Also,
>> as the MTU depends on the router, some people recommended to have packets
>> that will never be splitted, i.e. <= 576 bytes.
>> What is your experience on this point?
>> For now I have capped the data size to 1500, but I am thinking of
>> reducing it for a better latency. Does anyone know the typical packet loss
>> rate?
>>
>> ----------------
>> 2. Compression
>> ----------------
>>
>> Has anyone used compression in a network engine? If yes, which
>> compression algorithm? What was the average gain?
>> I have read that John Carmack used the Huffman compression in the Quake 3
>> network engine because it was well suited for network data compression, but
>> I still need to find some time to implement it in my program and do some
>> tests.
>>
>> ----------------
>> 3. Channels
>> ----------------
>>
>> What is your network channel policy? How many channels do you use? What
>> do you send on each channel?
>> At the moment I have 2 channels: one for sending unreliable data (a lot),
>> another one for reliable data.
>> I have chosen this organization to avoid blocking the unreliable data
>> while waiting for an ACK for the reliable data. I am thinking of adding
>> another channel to send high priority reliable data, but I am not sure of
>> the benefit as I already group the reliable data before sending them to
>> limit the blocking. It may be useful if the packet loss rate is too high.
>>
>> ----------------
>> 4. Server cycle
>> ----------------
>>
>> I know this aspect depends on the game type, but I would be interested in
>> knowing how your applications work on this.
>> On my side, based on Valve's introduction to network concepts (+ some
>> readings on the UT and Quake network engines), I have decided to implement
>> a 50-ms cycle on the server side. This means that the servers only updates
>> the simulation and notifies the clients each 50 ms. Meanwhile, it only
>> reads the network messages to empty the network event buffer and to handle
>> the reliable packet sending.
>> On the client side, I have introduced a voluntary delay of 70 ms, which
>> goes unnoticed on a user point of view, but helps a lot for overall
>> fluidity as the client will almost always have the following world update
>> (unless a packet is lost).
>> I will soon try to increase the server cycle time to 60 ms to gain 10-20%
>> of bandwidth. I also plan to separate the display delay of the player (it
>> will stay to 70 ms) and the rest of the world (increased to 100 ms).
>> However, I am afraid of the impact on reactivity.
>>
>> Does anyone have a similar architecture? What is your experience on
>> timings?
>> If not, has anyone already developed a FPS or a game very dependant on
>> the network speed? If yes, what are your advices?
>>
>> ----------------
>> 5. Other
>> ----------------
>>
>> If anyone has some useful advices on how to improve the network engine of
>> a game/application, I would be pleased to hear it (well, to read it at
>> least).
>>
>>
>> Thanks in advance for all your experience sharing.
>>
>> Best regards,
>> Jeremy Richert
>>
>> _______________________________________________
>> ENet-discuss mailing list
>> ENet-discuss at cubik.org
>> http://lists.cubik.org/mailman/listinfo/enet-discuss
>>
>>
>
> _______________________________________________
> ENet-discuss mailing list
> ENet-discuss at cubik.org
> http://lists.cubik.org/mailman/listinfo/enet-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cubik.org/pipermail/enet-discuss/attachments/20131205/c9510b2d/attachment-0001.html>


More information about the ENet-discuss mailing list