[ENet-discuss] enet comments

[Brian Hook]

Hi everyone,

As I've somewhat hinted/promised to Lee and Bruce, I'm doing a
writeup on some comments about enet and open source libraries in
general.  Take them with a grain of salt -- enet has obviously proven
its worth, so these comments are mostly of the form "wouldn't it be
nice if?" and are also somewhat colored by my own experiences writing
networking code.  Some of the comments are, admittedly, nit-picky,
but I feel they're still relevant.  In addition, making enet more
approachable may not be a goal, so some of the stuff I outline is
specifically for the "here's how to make enet more popular" task.

The first minor point is that enet defines its own uint32, etc.  
There is a potential for a name clash here with other packages.  I've
recommended using posh.h (www.poshlib.org) to help streamline this
and make it portable to a larger variety of platforms.

The second minor point is that the way the files are named, e.g.
win32.c, can cause some conflicts with other files named win32.c.  
Specifically, for people that have build systems that dump object
files into a repository like /tmp/o, you can get a win32.c that
clobbers another win32.c.  While this can be fixed by building enet
as a library, some people prefer monolithic build systems for
whatever reason, and it would be trivial to rename the files as
en_win32.c, etc.  

The third minor point is that enet requires enet/include in the
global header search path, which is another minor but aggravating
point when trying to integrate open source.  It would be nice if that
was restructured so that everything just sat in a single directory
and enet could be included via #include "enet.h" and the application
could include it with an absolute or relative path as required.

The above come together on "how do I use this stuff?"  The sign of a
popular and easy to use open source library is when you can just open
up a directory and add all the files to a project or makefile, and it
just compiles and builds.  lua, ijg, ogg vorbis, zlib, etc. all
behave this way to some extent.

Those are minor points, but I thought I'd at least mention them.

Slightly more major points follow.

First, enet calls malloc()/free() directly instead of allowing the
user to specify the memory management routine.  Some open source
libraries allow the user to override these and similar routines by
user defined callbacks.

Even with a user-specified malloc()/free(), I'm concerned that it may
be open to fragmenting the user's heap.  Compaction and minimizing
fragmentation are extremely important when dealing with large numbers
of clients.  My own library keeps a ring buffer per client/peer for
resends, and each client has its own separate buffer.  This is
somewhat wasteful (vs. a global pool), but it's a lot more
predictable and a single bad client won't affect the entire pool.  
Fragmentation is a non-concern for my own stuff.

Second, I'm not sure if enet is re-entrant and/or thread safe.  My
instinct is that it is not since there is no context to pass around,
but I haven't looked at the code closely enough to see if this is an
issue or not.

Third, I'm unclear how well enet can scale up to hundreds or
thousands of clients.  The technological hurdles for each order of
magnitude change, and enet obviously works on the lower end, but it
might be nice (although maybe outside its scope) to ensure that it
doesn't fall apart with 1000 clients.  I haven't looked enough at its
implementation to really judge.

Fourth, I would highly recommend making a quick overview document
that describes the API, a quick start tutorial, and then some
technical discussion to address some of the questions I've brought up
and other issues, such as working with NATs, etc.

Okay, that covers that.  

Now I'll talk about my own networking library and how I've done
stuff, mostly to spark discussion for architecting this type of
thing.  I don't think I've done things the right way, I've just done
them the most intuitive way to me.  In addition, my stuff isn't
battle tested, so a lot of this is theoretical.

Instead of treating everyone like peers, my library has the concept
of "listen sockets" and "send sockets".  Clients are an abstraction
used to validate incoming packets.

The general structure is still connectionless.  A server grabs a
packet, and if it's a login packet (server specified) then it
validates and possibly adds that client to its authorized client
list.

One of the design goals of my library was a bad idea, in hindsight,
and that was to support being a raw abstraction on sockets -- I
wanted people to be able to use it to write a client that works with
someone else's protocol.  I think this added needless complexity to
my package, so I may eschew that.  The biggest problem is that I was
computing CRCs as a way of flagging that it was a compatible packet
type, since it had no other way of telling (magic cookies could end
up with spurious, but eventually ignored, packets).

I mean, maybe it's not a HORRIBLE idea, since it would be nice to
still have an abstraction on top of BSD/Winsock, and I could see a
lot of apps that would rather manage all the packet information
themselves, but it did add some extra complexity.  The extra packet
header information can actually turn out to be a lot.  I've talked to
people that have worked on very large networked games, and one of the
things they often mention is that a few extra bits or flags really
make a difference.  In one case, someone measured his traffic as
consisting primarily of blank UDP headers, and even THAT was hurting
his bandwidth, so he had to change things around to minimize those
without causing zombies.

Maybe a smarter thing would have been to have two libraries, a pure
unreliable socket library and then a higher level connection manager
on top.  Hmmm, that does sound better.

The general system works as follows (for reliable data).

Each client connects to the server.  The client has a client
identifier (based on a hash of its two low order bytes in IP) which
is used to differentiate between clients behind a NAT.

The header is examined, and then a look up is performed in a hash
table of clients.  A linear search search is then performed at that
entry, matching IP against the incoming IP, and then finally matching
client ID vs. client ID.

The most recent valid packet's address is then stored off so that I
have a valid IP/port combination for sends.  I do NOT use the
client's IP + port at connect time, since this can change.  Also, the
server does not have to send to a specified port on the client, since
with NAT this would require forwarding to each machine, which is a
hassle for many users.  Doing a sendto() on the most recent
recvfrom() address is a bit of a pain, but at least solves the
firewall headache (although excessively paranoid firewalls, like
corporate ones, might still be a concern).

>From there, I use a pretty standard sequence number structure for
reliable packets.  Reliable packets are resent periodically by
calling a refresh function, which can be done per client or done per
socket.  I do not require multiple threads, however since all data is
localized into a context structure, it's entirely possible to lock
access to this and run in a separate thread successfully.

Currently the app is responsible for detecting zombie connections or
overflow conditions and manually removing a client in that situation.
I wanted the app to manage connections because there are various DoS
attacks that can be people can do depending on the network
implementation.

For example, if a protocol sends a "connect packet" and then the
server allocates space for that client without a reasonable time
frame to dispose of that client, then you can easily flood a server
with multiple connect packets and varying session/client id values.  
In a manner of minutes a server could think it has several thousand
clients attempting to connect, causing overflow conditions or just
really bad performance.

So I pushed this up to the server's side and made it look for login
packets and, when authorized, THEN creating a client and pushing it
back down into the network library.  I'm not sure if this is the best
way.

I don't like my mismatch in orthogonality there, but that's an issue
of practicality really because my stuff is really architected around
a client/server system and not a potential peer-to-peer system (enet
seems very well suited for peer-to-peer).

Originally I was going to support having encryption/per-packet
compression hooks, but I've decided that's really outside its domain,
so I will make that an app thing.

My "utility" library provides services like delta
compression/decompression, which I think can be generally useful for
a lot of people, but isn't exactly 100% necessary by any means
either.

Brian