[ENet-discuss] Peer starving problem....
David L. Koenig
dave at rancidmeat.com
Mon May 29 16:41:23 PDT 2006
Did you try testing this on a LAN as well as testing it over DSL?
If you're sure that you've got 60KBps(480 Kbps) up, then perhaps it's
not a bandwidth issue, but it does seem to exhibit the classic signs;
it works at first, then the clients start to choke off after a little while.
Here are a couple takes on that theory. Let's assume you were limited
to 128Kbps. You can't really test for 50 clients over the same line,
each with 512KB over a 128Kbps(16KBs) upload. You will certainly run
into bandwidth problems. Even if you were to send 1KBps, you can run
into issues with that number of connections. Here's the math behind
that: You start with 128Kbps. You take 1KB(kilobyte), which is
8Kb(kilobits) since every byte is eight bits. So, for 50 connections
you're looking at 50 * 8Kb = 400Kb. So, you're about 272 Kbytes or 34KB
over, if your upload is capped at 128Kbps. Let's say you were capped at
384Kbps, which is the case with my cable modem. 50 clients would put my
connection over by 16Kbits or 2KBytes. This is not including the packet
and enet overhead or anything else that might be going over the wire at
the time. So 50 connections might work for a few seconds, but eventually
your clients are going to start fail due to bandwidth issues. My guess
is that your clients/servers actually work just fine, but your test is
breaking down.
Feel free to check my math. :)
My previous point with the 1500 bytes packets is that enet has to break
those packets down into smaller chunks which will end up taking longer
than the theoretical time 512KB would take to transfer. Certainly this
functionality was put into enet, to account for cases where you want to
send more than the MTU limit.
-dave
Nono BEZMan wrote:
> Here's the test code I am using (windows based, sorry
> ;-)
>
> - You can modify the following values: nbThreads,
> PacketSize, ServerListenPort, ServerAddress,
> EchoFullPacketSize.
>
> - To start a server use the "-s" command line option.
> To start a client, do not use any command line option.
>
> SOURCE CODE:
> -------------------------------------------------------
>
> // EnetTest.cpp : Defines the entry point for the
> console application.
> //
>
> #include "stdafx.h"
> #include "enet/enet.h"
> #include <Windows.h>
>
> #include <hash_map>
>
> // TEST CONFIG - MODIFY VALUES HERE
> //
> // Nb Threads = peers to use for this test
> static const int nbThreads = 50;
> // Size of packets sent by client to server
> static const unsigned long PacketSize = 1*1024;
> // Server will listen on this port
> static const enet_uint16 ServerListenPort = 55555;
> // Address of server
> static const char *ServerAddress = "192.168.1.12";
> // If true the server sends back a packet of same size
> as received, else send back 1 byte only
> static const bool EchoFullPacketSize = true;
> // Max Random time between to consecutive sends
> static const DWORD SendSleepTime = 0;
> //
>
> typedef stdext::hash_map<enet_uint32, HANDLE>
> SendEventMap;
> typedef stdext::hash_map<enet_uint32,
> HANDLE>::iterator SendEventIterator;
> typedef std::pair<enet_uint32, HANDLE>
> SendEventPair;
>
> static int nbThreadsAlive = 0;
> static HANDLE GlobalStart = CreateEvent(NULL, TRUE,
> FALSE, NULL);
> static SendEventMap SendEvents;
> static bool IsSender = false;
> static DWORD TransferTime = 0;
> static DWORD TransferSize = 0;
> CRITICAL_SECTION sendSyncSection;
>
>
> DWORD WINAPI ThreadProcessService (void *param)
> {
> if (param == NULL)
> return -1;
>
> printf("Entering ThreadProcessService\n");
>
> ENetHost *pHost = (ENetHost *)param;
>
> do
> {
> ENetEvent event;
> EnterCriticalSection(&sendSyncSection);
> int enetServciceRes = enet_host_service (pHost,
> &event, 20);
> LeaveCriticalSection(&sendSyncSection);
>
> if (enetServciceRes < 0)
> {
> printf("Exiting ThreadProcessService\n");
> return -1;
> }
> else if ((enetServciceRes == 0) || (event.type ==
> ENET_EVENT_TYPE_NONE))
> {
> //printf("Looping in ThreadProcessService\n");
> continue;
> }
>
> switch (event.type)
> {
> case ENET_EVENT_TYPE_CONNECT:
> {
> nbThreadsAlive++;
> printf("Got Connection [NbAlive: %d, PeerID :
> %d]\n", nbThreadsAlive, event.peer->incomingPeerID);
> }
> break;
>
> case ENET_EVENT_TYPE_RECEIVE:
> {
> // Got confirmation. Print out response time
> TransferSize += PacketSize;
>
> printf("Packet Received [NbAlive: %d, Size: %.2f
> KBs, Rate: %.2f KB/sec, PeerID: %u]\n",
> nbThreadsAlive, event.packet->dataLength /
> 1024.0,
> 1000.0 * TransferSize / 1024.0 / (GetTickCount()
> - TransferTime),
> event.peer->incomingPeerID);
>
> if (IsSender == false)
> {
> EnterCriticalSection(&sendSyncSection);
> // Echo back "ack"
> ENetPacket *packet = enet_packet_create(
> NULL, (EchoFullPacketSize == true ?
> event.packet->dataLength : 1),
> ENET_PACKET_FLAG_RELIABLE);
> enet_peer_send(event.peer, 0, packet);
> //printf("Sent Ack packet [Size =
> %d]\n",packet->dataLength;
> LeaveCriticalSection(&sendSyncSection);
> }
> else
> {
> // Wake up our thread
> SendEventIterator eventIt =
> SendEvents.find(event.peer->incomingPeerID);
> if (eventIt != SendEvents.end())
> {
> SetEvent(eventIt->second);
> }
> else
> {
> printf("ThreadProcessService::Could not find
> SendEvent [PeerID: %u]\n",
> event.peer->incomingPeerID);
> }
> }
>
> enet_packet_destroy(event.packet);
> }
> break;
>
> case ENET_EVENT_TYPE_DISCONNECT:
> {
> nbThreadsAlive--;
> printf("Got Disconnection [NbAlive: %d, PeerID :
> %d]\n", nbThreadsAlive, event.peer->incomingPeerID);
> }
> break;
>
> default:
>
> break;
> }
> } while (true);
>
> return 0;
> }
>
> DWORD WINAPI ThreadSend (void *param)
> {
> if (param == NULL)
> {
> printf("ThreadSend::Peer is NULL - Exiting");
> return 0;
> }
>
> srand((unsigned)time(NULL));
>
> ENetPeer *pPeer = (ENetPeer *)param;
> SendEventIterator eventIt =
> SendEvents.find(pPeer->incomingPeerID);
> if (eventIt == SendEvents.end())
> {
> printf("ThreadSend::Could not find SendEvent
> [PeerID: %u]\n", pPeer->incomingPeerID);
> }
>
> HANDLE waitEvents[2];
> waitEvents[0] = eventIt->second;
> waitEvents[1] = GlobalStart;
>
> while (true)
> {
> WaitForMultipleObjects(2, waitEvents, FALSE,
> INFINITE);
>
> DWORD sleepTime = SendSleepTime * rand() / RAND_MAX;
> //printf("ThreadSend::EventSet - Sleeping for %u
> ms\n", sleepTime);
> Sleep(sleepTime);
>
> EnterCriticalSection(&sendSyncSection);
>
> ENetPacket *packet = enet_packet_create(NULL,
> PacketSize, ENET_PACKET_FLAG_RELIABLE);
>
> enet_peer_send(pPeer, 0, packet);
> //printf("Sent packet [PeerID: %u, Size = %u, Res =
> %d]\n", pPeer->incomingPeerID, PacketSize);
>
> LeaveCriticalSection(&sendSyncSection);
> }
>
> return 0;
> }
>
> int _tmain(int argc, _TCHAR* argv[])
> {
> srand((unsigned)time(NULL));
>
> enet_initialize();
>
> InitializeCriticalSection(&sendSyncSection);
>
> if ((argc > 1) && (strncmp ("-s", argv[1], 2) == 0))
> {
> IsSender = true;
> printf("Sender Mode\n");
> }
> else
> {
> printf("Receiver Mode\n");
> }
>
> ENetAddress localAddress;
> localAddress.host = ENET_HOST_ANY;
> if (IsSender == true)
> localAddress.port = 0;
> else
> localAddress.port = ServerListenPort;
>
> //ENetHost *pHost = enet_host_create(&localAddress,
> 500, 0, 0, 0, ENET_PEER_PING_INTERVAL);
> ENetHost *pHost = enet_host_create(&localAddress,
> 500, 0, 0);
>
> if (pHost != NULL)
> {
> // DO not care about threadid, handles...etc.. test
> prog only. no cleanup here ;-)
> DWORD threadID;
> ::CreateThread (NULL, 0, ThreadProcessService,
> pHost, 0, &threadID);
> if (IsSender == true)
> {
> for (int i = 0; i < nbThreads; i++)
> {
> ENetAddress remoteAddress;
> enet_address_set_host(&remoteAddress,
> ServerAddress);
> remoteAddress.port = ServerListenPort;
> printf("Starting Connection [Index: %d]\n", i);
> //ENetPeer *pPeer = enet_host_connect (pHost,
> &remoteAddress, 1, NULL);
> ENetPeer *pPeer = enet_host_connect (pHost,
> &remoteAddress, 1);
>
> SendEvents.insert(SendEventPair(pPeer->incomingPeerID,
> CreateEvent(NULL, FALSE, FALSE, NULL)));
> ::CreateThread (NULL, 0, ThreadSend, pPeer,0,
> &threadID);
> }
> }
> }
> else
> {
> printf("enet_host_create FAILED");
> }
>
> if (IsSender == true)
> {
> Sleep(5000);
> printf("Setting GlobalStart Event\n");
> PulseEvent(GlobalStart);
> TransferTime = GetTickCount();
> }
>
> getchar();
>
> enet_deinitialize();
>
> DeleteCriticalSection(&sendSyncSection);
>
> return 0;
> }
>
>
>
> --- Steve Williams <stevewilliams at kromestudios.com>
> wrote:
>
>
>> I think it's worse than that. He was talking
>> kilobytes(KB), not
>> kilobits(Kb). At the end of his post, he states
>> that they are maxing
>> out their outbound connection at 50KB. I believe
>> this will probably be
>> a 512Kbps connection. So you're probably looking at
>> 15-20 seconds to
>> send a 512KB packet per peer.
>>
>> --
>> Sly
>>
>>
>> David L. Koenig wrote:
>>
>>
>>> It sounds to me like the host is running out of
>>>
>> outgoing bandwidth.
>>
>>> Most DSL connections give you about 128Kbps
>>>
>> outgoing, 768Kbps to 3Mbps
>>
>>> incoming. 512k is 4 times the outgoing max. So,
>>>
>> under the best
>>
>>> conditions you can hope to have the packet go
>>>
>> through to the/from the
>>
>>> server in four seconds. That will never happen as
>>>
>> enet is going to have
>>
>>> to break up the packets into chunks of around 1500
>>>
>> bytes which is a
>>
>>> standard MTU limit, so you lose some bytes to
>>>
>> overhead. It's so backed
>>
>>> up with outgoing packets that it can't possibly
>>>
>> send them all within the
>>
>>> connection timeout limit. You want to try to limit
>>>
>> your packet size to
>>
>>> 1500 including any overhead (UDP header overhead,
>>>
>> enet overhead).
>>
>>> Exactly what sort of data is it that you're trying
>>>
>> to send? Why must
>>
>>> the packet be so large? Do you know the
>>>
>> upload/download speed provided
>>
>>> to your server?
>>>
>>> -dave
>>>
>>> Nono BEZMan wrote:
>>>
>>>
>>>
>>>> Hi all,
>>>>
>>>> I have used enet for a while now, and we just came
>>>> across this issue which we could not resolve:
>>>>
>>>> - Setup: we have one client talking to one server
>>>> (enet_host_service every 20 ms) on a DSL
>>>>
>> connection.
>>
>>>> - Test1: client send packets of 512KB to server
>>>>
>> back
>>
>>>> to back, using only one peer. Everything goes
>>>>
>> fine.
>>
>>>> - Test2: client sends packets of 512KB on more
>>>>
>> than
>>
>>>> one peer (2-3 in general). All peers are
>>>>
>> disconnected
>>
>>>> after one minute or so, but one which goes on once
>>>>
>> it
>>
>>>> "killed" the other ones.
>>>>
>>>> We have also tried to send 4kB (or 2KB) packets
>>>>
>> with
>>
>>>> 50 peers at the same time.... at the end most
>>>>
>> "starved
>>
>>>> to disconnection", except for a few ones (like a
>>>> dozen) which then went on happily forever. The
>>>>
>> bigger
>>
>>>> the packet, the less peer at the end of the fight.
>>>> With 1K packets we ended up with ~20 peers, with
>>>>
>> 2KB
>>
>>>> about 12 peers and with 4KB we had ~6 peers still
>>>> going at the end.
>>>>
>>>> We even tried to randomize the order in which the
>>>> peers where serviced in send_outgoing_commands to
>>>>
>> no
>>
>>>> avail.
>>>>
>>>> It is very puzzling to me, as enet being UDP
>>>>
>> based, I
>>
>>>> do not see why peers starve to death after some
>>>>
>> point.
>>
>>>> Sending 512KB packets back to back (and maxing our
>>>> upload connection at 50kKB) works fine until we
>>>>
>> add a
>>
>>>> second peer....
>>>>
>>>> We tried Enet 1.0 (latest CVS) as well as previous
>>>> versions of Enet.
>>>>
>>>> Anybody would have any ideas/pointer on how to
>>>>
>> resolve
>>
>>>> that issue?
>>>>
>>>> Thank you,
>>>>
>>>> __________________________________________________
>>>> Do You Yahoo!?
>>>> Tired of spam? Yahoo! Mail has the best spam
>>>>
>> protection around
>>
>>>> http://mail.yahoo.com
>>>> _______________________________________________
>>>> ENet-discuss mailing list
>>>> ENet-discuss at cubik.org
>>>>
>>> http://lists.cubik.org/mailman/listinfo/enet-discuss
>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>> _______________________________________________
>>> ENet-discuss mailing list
>>> ENet-discuss at cubik.org
>>>
>> http://lists.cubik.org/mailman/listinfo/enet-discuss
>>
>>>
>>>
>>>
>> --
>> Sly
>>
>>
>>
>> This message and its attachments may contain legally
>> privileged or confidential information. This message
>> is intended for the use of the individual or entity
>> to which it is addressed. If you are not the
>> addressee indicated in this message, or the employee
>> or agent responsible for delivering the message to
>> the intended recipient, you may not copy or deliver
>> this message or its attachments to anyone. Rather,
>> you should permanently delete this message and its
>> attachments and kindly notify the sender by reply
>> e-mail. Any content of this message and its
>> attachments, which does not relate to the official
>> business of the sending company must be taken not to
>> have been sent or endorsed by the sending company or
>> any of its related entities. No warranty is made
>> that the e-mail or attachment(s) are free from
>> computer virus or other defect.
>> _______________________________________________
>> ENet-discuss mailing list
>> ENet-discuss at cubik.org
>> http://lists.cubik.org/mailman/listinfo/enet-discuss
>>
>>
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam? Yahoo! Mail has the best spam
> protection around
> http://mail.yahoo.com
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam? Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
> _______________________________________________
> ENet-discuss mailing list
> ENet-discuss at cubik.org
> http://lists.cubik.org/mailman/listinfo/enet-discuss
>
>
>
>
More information about the ENet-discuss
mailing list