[Networking] General TCP Questions

[xalcon]

Hi folks,
in the past I've developed a few applications using TCP Sockets and I always had a situation in mind, which might cause problems.
TCP is a stream based protocol. So, when I want to use packets, I have to create some kind of generic packet handler.
Lets say I my generic packet structure looks like this (Pseudocode)

struct GenericPacket
{
    uint16 opcode;
    uint16 packetSize;
    byte payload[packetSize];
};

I never had any problems with this, even with bigger packets (size > 1500 bytes) but recently I tried to create a websocket server in c# using the autobahn test suite.
In the suite is a test were packets are send in chops

Case 1.1.8: Send text message with payload of length 65536. Sent out data in chops of 997 octets.

In a worst case scenario, I would receive a packet with size X while only getting a few bytes (because of lag, packet loss, etc), so I have to wait until I receive the remaining bytes.

Now my questions: Is this the general case in network programming? Do I have take into account, that packets might come in chunked? How do I handle those scenarios properly? Or is this a problem which doesn't realy exist?

In my websocket server I've read all data from the socket into a byte buffer, so I can peek into the data more easily. But I'd never seen such stuff in any open source projects on the web. Every project I've looked into never used any kind of data buffer to make sure they get always the whole packet.

[Cromon]

Hello xalcon

Yes, thats the general case. There is no guarantee on how much data you receive on a call to recv. Various factors influence that amount, for example this one: Maximum transmission unit - Wikipedia, the free encyclopedia

So in a generic read function would look somewhat like this:

template<typename T>
T read() {
    T ret;
    byte* tPtr = reinterpret_cast<byte*>(&ret);
    int bytesRead = 0;
    while(bytesRead < sizeof(T)) {
        int numRead = recv(mSocket, tPtr + bytesRead, sizeof(T) - bytesRead, 0);
        if(numRead <= 0) errorHandler();
        bytesRead += numRead;
    }

    return ret;
}

void readBytes(std::vector<byte>& data) {
    int bytesRead = 0;
    while(bytesRead < data.size()) {
        int numRead = recv(mSocket, data.data() + bytesRead, data.size() - bytesRead);
        if(numRead <= 0) { errorHandler(); }
        bytesRead += numRead;
    }
}

GenericPacket readPacket() {
    GenericPacket ret = { 0 };
    ret.opcode = read<uint16>();
    ret.packetSize = read<uint16>();
    ret.payload.resize(ret.packetSize);
    readBytes(ret.payload);
    return ret;
}

Greetings
Cromon