(This is all using the v0.27.5.629505 client)
I am trying to reverse the ESO enc/decrypt. So far I have come to:
sub_B303C0(void *Dst, void *Src, size_t Size)
Which writes to Dst the decrypted data, and generates a new key for the next packet.
The part that alludes me is this loop:
.text:00B30340 loc_B30340: ; CODE XREF: sub_B30320+32j
.text:00B30340 mov bl, [eax]
.text:00B30342 lea esi, [edx+eax]
.text:00B30345 mov cl, [esi+edi] ;encrypted data
.text:00B30348 xor bl, cl
.text:00B3034A mov [esi], bl
.text:00B3034C mov [eax], cl
.text:00B3034E inc eax
.text:00B3034F dec [ebp+arg_C]
.text:00B30352 jnz short loc_B30340
.text:00B30354 pop edi
.text:00B30355 pop esi
.text:00B30356 pop ebx
[eax] is a 16 byte buffer located in the Awesomium.dll memory area. It is clearly the key used to xor the data. After each received packet, the key is changed (im guessing sending packets uses a key in a different location). My problem is to figure out how this key is generated in the first place, and what its based off. Tracing it by write bps, you land in a huge function with a bunch of XMM stuff, that writes to different parts of the key. Then again another function. The instruction trace of this function is 1.6k lines.
I looked for docs on Awesomium but it does not seem to support any native encryption methods. My best guess the initial key is generated off a session id, but the problem is the next key generation. To RE all the XMM stuff seems crazy, this must be using some kind of standard alogrithem.
(this is not xor the current packet by the last packet, the key goes through 1 or 2 long XMM functions depending how large the received size is)
Anyone working on this / can help me out? Or give some advice.