Analysis of the CASC filesystem

[akspa420]

One important thing we missing now is ability to find encoding/root files in data.XXX without extracting everything. Pretty sure there's should be a way to find data index and offset by MD5 content hash using data from idx or index files.

Do you think it's possible/probable that the list file is in the index files, but are encrypted using a key provided by the game exe? It would make sense to lock away the index and make it near impossible for us to decode it - as it would perform an additional way to check the game files consistency. Just my 2 cents on the topic - I don't know much about how Storm worked, let alone CASC.
Talking a bit out of my ass here, but I'm thinking that upon execution of the game executable, the index files provide a listfile that is encrypted and decrypted upon load of the game's engine. I haven't the *working* tools or knowledge to see if this is the case or not, so I'm just throwing ideas out there... and now I wonder if anyone has tried to just extract a "(listfile)" from the game... (gonna go try this )

[TOM_RUS]

Do you think it's possible/probable that the list file is in the index files, but are encrypted using a key provided by the game exe? It would make sense to lock away the index and make it near impossible for us to decode it - as it would perform an additional way to check the game files consistency. Just my 2 cents on the topic - I don't know much about how Storm worked, let alone CASC.
Talking a bit out of my ass here, but I'm thinking that upon execution of the game executable, the index files provide a listfile that is encrypted and decrypted upon load of the game's engine. I haven't the *working* tools or knowledge to see if this is the case or not, so I'm just throwing ideas out there... and now I wonder if anyone has tried to just extract a "(listfile)" from the game... (gonna go try this )

There's no listfile. There's no need in it.

[caali]

One important thing we missing now is ability to find encoding/root files in data.XXX without extracting everything. Pretty sure there's should be a way to find data index and offset by MD5 content hash using data from idx or index files.

Well, as of now, the only places where I've seen these files are either as first files of data.000 or first files of data.NNN (N being the highest data file number available). I haven't reverse engineered the code locating those files, but it could really be possible that they simply linearly search through the data.XXX files until they find them. While in the worst case this would take relatively long, usually it's pretty short because - as mentioned before - they are the first files of data.000... I'll take a look at this when I've some spare time.

[raeef]

god help us

[CatsNimo]

if there is a way to extracting everything then maybe we can find the listfile

There is no listfile. Please read.

Well, as of now, the only places where I've seen these files are either as first files of data.000 or first files of data.NNN (N being the highest data file number available). I haven't reverse engineered the code locating those files, but it could really be possible that they simply linearly search through the data.XXX files until they find them. While in the worst case this would take relatively long, usually it's pretty short because - as mentioned before - they are the first files of data.000... I'll take a look at this when I've some spare time.

They just use the index files because the md5 of the root, encoding, download, and install files is obvious (since the game checks build/cdn configs every time it is run, it always knows the newest root md5s).

I'm not sure why would you want even want to find a local version of root or encoding considering how it is freely available on the Blizzard FTP as an independent file. Parsing the indexes for its MD5 is a waste of time when it is such a fast download.

[TOM_RUS]

There is no listfile. Please read.



They just use the index files because the md5 of the root, encoding, download, and install files is obvious (since the game checks build/cdn configs every time it is run, it always knows the newest root md5s).

I'm not sure why would you want even want to find a local version of root or encoding considering how it is freely available on the Blizzard FTP as an independent file. Parsing the indexes for its MD5 is a waste of time when it is such a fast download.

Downloading such huge files may take minutes, while searching local copy should take less than 1 second.

[TOM_RUS]

One important thing we missing now is ability to find encoding/root files in data.XXX without extracting everything. Pretty sure there's should be a way to find data index and offset by MD5 content hash using data from idx or index files.

This is solved now.

[akriso]

is this meen tnat now we can extract all files?

[TOM_RUS]

is this meen tnat now we can extract all files?

Only if you have all names.

[akriso]

are we have some way to generate it fully?

[raeef]

you can find some in the Filedata.dbc

[CatsNimo]

is this meen tnat now we can extract all files?

Only if you have all names.

Technically you don't need names to do what he asked. You can extract all 750,000+ files unnamed.

There's just no point because without names, you don't have context, and there's no reasonable or timely way to sort through such a giant mess.

[raeef]

you can find some of the names from the filedata.dbc I found some usefull info

[ioctl]

The encoding file is more complex than what's described here. Records appear to be variable-length, with the first byte being a count of how many file keys follow the md5. The gaps between blocks aren't always the same size. Sometimes they are 30 bytes, sometimes as little as 14 -- not sure why yet. There's also a whole lot of data (like, 100% more) after all of the hash table blocks.

[ioctl]

Also, looks to me like if there are two hashes for the encoding table in the config file, then the first is the md5 of the contents, and the second is the file key (which you can use to fetch it before you have an encoding table =)