[ENet-discuss] Peer starving problem....

[Lee Salzman]

There is an in-flight window for reliable data, meaning that for any 
given peer, only some fixed amount of data can be going over the network 
at any given time. So even if you stuff 512KB into one peer, it won't 
send it all out in one call to enet_protocol_send_outgoing_commands().

However, this window is per-peer, so if you have 50 peers, you will 
multiply the amount of data sent in 
enet_protocol_send_outgoing_commands() and likewise the amount of time 
that function takes to execute. While it is stalled in there sending it 
can't acknowledge any connections, hence why peers are timing out.

There are a few feasible ways around this. The first is to just try 
making the send buffer in the socket a lot larger by setting the 
SO_SNDBUF socket option (something I should probably be doing anyway). 
However, if you just keep pumping data at an overwhelming rate that 
would still probably not do much.

The other way is to limit the total amount of data 
enet_protocol_send_outgoing_commands() is allowed to send out in one 
call, so that even if it overflows the send buffer it won't wait very 
long at all.

The whole throttling mechanism currently in place for reliable data, as 
I have said before, is kinda screwy and really does need to be rewritten 
in the future, but I'm still working on that.

Lee

Nono BEZMan wrote:
> This is what I feared... enet does cope well with
> over-the-bandwidth tranfers. I thought it would do
> some flow control with the ack/retry system, but it
> does not seem to do it fairly.
> 
> I also tried to put a bandwidth limit in
> enet_host_create, but it did not do anything. The
> throtle_bandwidth() function is called, but it does
> not seem to slow down the whole connection.
> 
> Again, what I do not get is why sending 512KB on ONE
> peer (which means having to resend out-of-order
> packets) works fine where sending 1KB (<MTU) on 50
> peers does not. The bandwidth used is exactly the
> same!
> 
> Packet loss shold be the same in both cases (except if
> something weird is going on with the rtt on some
> peers, which does not happen when you only have one
> peer). The 50-peer case should even be easier on
> enet!!!
> 
> I am really looking for solutions here, as I do think
> it is a tweak to enet that would really be worth it.
> So much work has already been done in it, and we are
> close to a perfect solution here... except for that
> starvation issue.
> 
> Thanx again.
>