[ENet-discuss] Large data packets over ENet

[Tim Auld]

Hi Jacco,

> > - Send your data unreliably.  With the massive bandwidth the overhead of
> > reliability will not be worth while.
>
> I agree. The only reason I liked reliability was the fact that I needed to
> get my app working in the first place. :) The way it's build right now
> allows for packet loss without too much problems, so I'll definitely
> switch to unreliable transfers.

I was thinking, you might be able to incorporate a custom reliability
protocol that wouldn't incur significant latency.  e.g., send requests for
missing messages as soon as they're detected and have them sent again
immediately.  All this would still be in UDP.  Nothing fancy...  may not
even be a problem.  If you compress the entire frame section and then send
in fragments, would the remainder be corrupt if you lost a fragment?

> > - Increase socket send and receive buffers greatly.  The defaults will
not
> > be enough (8KB is typical I think).
>
> Do you mean ENet send and receive buffers, or UDP/WinSock buffers? How do
> I set them?

I mean the socket buffers (UDP/Winsock if you like).  You do this with the
following code:

int size = bytes;
setsockopt(m_Socket, SOL_SOCKET, SO_RCVBUF, (char *)&size, sizeof(size));
setsockopt(m_Socket, SOL_SOCKET, SO_SNDBUF, (char *)&size, sizeof(size));

Don't know if ENet provides an API for this, but you could hack it in
yourself.  Maybe ENet has its own internal buffers that could make a
difference in performance as well.

> > - Pipeline several frames.
>
> That's being done already. The master sends two frames to each slave
> (together with two numbers indicating the first and last line to be
> rendered by that slave) and sends a new one as soon as the results for the
> first frame come in. That way, the slave always has one frame ready so it
> can keep rendering while the master processes the calculated pixels.

Sounds good.  Keep those slaves busy.

> > Issue
> > as many frames as you can to the servers available.
>
> Can't send too much ahead, the ultimate goal is interactivity. :) So if I
> can get away with one or two frames sent ahead, that would be ideal. It
> would still give considerable lag at 25fps though.

You're right, but I think you should be able to get away with 3 or 4 frames
before it becomes noticeable.  A lot of video cards perform triple buffering
now.  It's worth experimenting with anyway, if it makes a difference.

> > Careful coordination may be
> > necessary to prevent servers from trying to send at the same time.  I
> > think this will also lead to fewer collisions on the ethernet.
>
> I didn't even know that this was an issue. That might explain some of the
> problems with 3 or more clients I was having when experimenting with
> UDP/WinSock. Perhaps a slave should just send a 'data ready' signal to the
> master first, then the master sends 'all clear, send data' to the slave,
> and finally the slave sends the actual data, while other slaves wait for
> their 'all clear' command. While waiting, each slave is of course
> rendering the next frame. :) That should help.

Maybe it's not an issue.  I haven't worked with such high bandwidth before.
Keep it in mind.  A system as you propose was what I was thinking of.

> > - Consider looking at delta compression of frames.  If there is not much
> > change between frames, it could be possible to just send the client the
> > changes since the last one.
>
> Yes. Someone suggested that on GameDev.net: XOR two frames, compress the
> result using LZO or whatever and send that data. The XOR'ed frame will
> compress much better than a raw frame. BTW it's also an option (although
> not ideal) to switch to 16bit color. Right now I'm sending full 32bit
> color.

Switching to 16bit will definitely make a huge difference.  It's a classic
quality/performance trade off.  I think 16bit at 40fps would be much more
impressive than 32bit at 25fps.  Your XOR method sounds promising.  Remember
most video codecs operate with delta compression, but they are usually
lossy, which you might not be happy with.  Perhaps streaming a full
resolution video stream from each slave and interleaving the frames is an
option.  How fast is LZO, out of curiosity?  I use Huffman for game data,
but the implementation is dog slow.  I have to optimise it.

> > - Make sure the processes have as high priority as possible.  If I took
> > focus away from my test application, the bandwidth throughput would drop
> > dramatically.
>
> I didn't have that problem with UDP/WinSock, but I did experience it with
> both ENet and RakNet. I found that carefull tweaking of sleep's is
> neccessary to keep masters and slaves from stalling the entire system,
> even when there's nothing going on, especially when testing on one machine
> (slave and master running in separate processes).

Hmm, maybe ENet and RakNet read the sockets in a separate thread?  I can't
remember, but that would explain the need to sleep the main thread, although
proper use of thread priorities and blocking should save that need.  My test
program doesn't use a thread, but simply reads the socket once per frame.
If your socket buffers are large enough, doing this shouldn't be a problem.
You would have to buffer the data from the thread anyway, until the main
thread wants it.  I think John Carmack agrees with me that threads aren't
necessary for high performance.

> > - Render a low resolution frame at the client.  Just a spitball idea,
but
> > in case a server does not deliver the full resolution frame in time, the
> > client could display a lower resolution frame instead.  This would keep
> > the animation smooth.
>
> Or render odd scalines first, then even scanlines. The second half could
> be dropped if not calculated in time.

Yeah, something like that.  If the master buffers a frame or two before
displaying, variation in delivery time won't matter too much though.

> > I don't know if ENet is suitable for this application.  Someone with
more
> > experience with ENet may be able to comment.  In any case, it shouldn't
be
> > too much work to write some network code and create your own custom
> > protocol.  This would give you maximum control and performance.
>
> I agree. This is all very new to me and I had major problems setting up
> anything that worked at all. So ENet helped me quite a bit to get the
> first results. From there, I can head for unreliable transfer, and once
> that's working, I'll drop ENet and go back to UDP/WinSock, but this time
> armed with proper knowledge. :)

I believe ENet provides unreliable transfer.  I'm just not sure the
architecture is geared towards high throughput.

Tim