Analysis of the CASC filesystem

[TOM_RUS]

Yes, there's multiple content keys per MD5 in encoding file: https://github.com/tomrus88/CASCExpl...andler.cs#L164.

And some additional unknown data after MD5/content keys block.

[ioctl]

Hm. One of my data.xxx files has a record with a 64 byte header instead of 30 =/

[adraffy]

Hello, first time poster. I was wondering if there has been any more updates on this topic?

I'm the developer for Catus, a feral druid simulator: http://fluiddruid.net/forum/viewtopic.php?f=3&t=4574
I also host a Blizzard-compatible JSON item API that exposes more of the game data: Raffy's Blizzard JSON Item API (v43) Build 17898

In the past, I've been getting my item data from the simulationcraft project, but eventually I was made aware about the MPQ format (and how to extract these files myself), and after getting everything implemented with MPQ support, Blizzard changes their data format :X

I've implemented a bunch of the CASC parsing code in Java with the help of this thread. I've been testing my code against the HotS game data.

Inside some of the BLTE containers, there appear to be MDNX stores that contain MAR data. Has anyone figured out how parse the MAR data? justMaku has done some work on this topic: MARHandler (https://github.com/justMaku/blte/blo.../MARHandler.cs)

Additionally, has anyone figured out the purpose of the *.index files? Their filenames appear to match the values for "archives" and "archive-group" in the CDN config file.

[TOM_RUS]

Additionally, has anyone figured out the purpose of the *.index files? Their filenames appear to match the values for "archives" and "archive-group" in the CDN config file.

They are similar to idx files, but contain file index/size/offsets for CDN storage instead of local data storage.

[k1ngk1ll]

They are similar to idx files, but contain file index/size/offsets for CDN storage instead of local data storage.

Could you please reply or read my pn? Thank you. :-)

[adraffy]

They are similar to idx files, but contain file index/size/offsets for CDN storage instead of local data storage.

Have you parsed these files? Do you know their format?

For a bunch of files that I'm interested in accessing, the .idx files do not contain the appropriate index/offset/size information (even though BlizzHash to ContentMD5 to FileKey exists). I'm assuming that these files were updated in a patch, so the latest index/offset/size is somewhere else.

I've scanned these files, directly searching for a missing FileKey, and it shows up both the file listed as "archive-group" and in one of the files in "archives."

Example:
Path = DBFilesClient\Item-sparse.db2
BlizzHash = be879c2390022273
ContentMD5 = 22f4de41dc7d73164bbad3c57a90bda5
FileKey = 76926fe5a35f4aba15506a05bce5e768 (34115324 bytes)
KeyPrefix = 76926fe5a35f4aba15 (does not have an Idx entry)

[adraffy]

For the 'archive" matching ".Index" files, the format is something like the following:

byte[16] :: header (unknown)

(int32, int32, byte[16]) :: entry1 (size?, offset?, md5)
(int32, int32, byte[16]) :: entry2
(int32, int32, byte[16]) :: entry...

eventually you will encounter a byte[16] = all zeros (unknown what the two int values mean in this case)
(this seems to happen every 169)

byte[16] :: another header? (unknown)

(... repeat of above entry list ...)

eventually, you will get back-to-back byte[16] = all zeros, without any entries
rest of the file are 0's until you reach the bottom, which contains some unknown stuff

When I parse "a719df9e1428b314bd006b52d5628669.index":

Initial Header:
.0 . 5 224 .95 156 . 3 165 .60 109 133 227 206 100 214 146 .84

Entry List Example:
. . . . . . . . . . . . . . . . .0 . 1 . 2 . 3 . 4 . 6 . 7 . 8 . 9 .10 .11 .12 .13 .14 .15 .16
.0 . 1 .91 184 .11 .73 105 232 . 0 .43 103 .83 233 199 .67 221 166 .36 203 .50 168 254 .45 .68
.0 . 0 .55 205 . 9 245 .95 .79 . 0 .71 148 239 195 118 143 .29 204 .65 215 . 4 156 237 102 113 .
.0 . 0 . 0 162 . 3 .80 .83 .25 . 0 .95 .19 182 .44 .68 .20 . 1 .95 126 .65 202 . 4 133 .25 189 .
.0 . 0 . 2 110 . 3 213 185 .82 . 0 102 .32 .52 .37 184 .31 215 .57 113 114 .87 .92 .87 .62 249 .
.0 . 0 .19 213 .11 .79 131 247 . 0 111 .76 178 155 239 .14 .99 . 6 139 .45 .34 145 .64 117 131 .
.0 . 0 .22 217 .11 .79 176 237 . 0 114 237 .24 .89 116 .66 121 .25 . 8 160 .83 .27 233 199 146 .
.0 . 0 206 167 .11 .74 197 160 . 0 147 . 8 .63 169 130 .76 191 147 179 215 160 . 5 .44 201 117 .
.

Parsed Dump:
. . . 0 . . . .89016 . .189360616 002b6753e9c743dda624cb32a8fe2d44 <40>
. . . 1 . . . .14285 . .167075663 004794efc3768f1dcc41d7049ced6671 <64>
. . . 2 . . . . .162 . . 55595801 005f13b62c4414015f7e41ca048519bd <88>
. . . 3 . . . . .622 . . 64338258 0066203425b81fd7397172575c573ef9 <112>
. . . 4 . . . . 5077 . .189760503 006f4cb29bef0e63068b2d2291407583 <136>
. . . 5 . . . . 5849 . .189772013 0072ed18597442791908a0531be9c792 <160>
. . . 6 . . . .52903 . .189449632 0093083fa9824cbf93b3d7a0052cc975 <184>
...
. . 163 . . .3211746 . .201617630 0e7ea69b135a1b2dfd706150a779e8c6 <3952>
. . 164 . . . . 7329 . .189615032 0eb9e1bd82d8aca793901f37eef0c4fa <3976>
. . 165 . . . . 1814 . .169241212 0ebc3b61f709c293a6e10ba85d0df542 <4000>
. . 166 . . . .36432 . .163152218 0eee0a7aab374d944c82b1b0898d79e6 <4024>
. . 167 . . . .77989 . .190944135 0ef4eabc406e55532666e2a23f0010cd <4048>
. . 168 . . . .25303 . .166861606 0efec5c85618945f0263f106315a353c <4072>
. . 169 . . . . . . . . . . . . . . . . .<< all zeros >>
. . 170 . . . .14371 . .110488054 0f3171ee1c1739f33d4d504d08faf85b <4136>
. . 171 . . . . .676 . . 73025833 0f76c78fb64661cb285f52d660b6f73f <4160>
. . 172 . . . .36303 . .163673117 0f7bbc7752a28ef10437bc5bf2b51447 <4184>
. . 173 . . . . 8136 . .166015402 0f80a9baa95489bb8ce0ed5c42b32f57 <4208>
. . 174 . . . . .129 . .137620418 0fb7215efb283e9ed71772877a5f4939 <4232>
. . 175 . . . .32415 . .164225650 0fc0cbf32f42db3074e9d802ac1c88e2 <4256>

[TOM_RUS]

Have you parsed these files? Do you know their format?

Look at my CASCExplorer.

For a bunch of files that I'm interested in accessing, the .idx files do not contain the appropriate index/offset/size information (even though BlizzHash to ContentMD5 to FileKey exists). I'm assuming that these files were updated in a patch, so the latest index/offset/size is somewhere else.

Have you started wow client at least once after downloading a patch? idx files are updated on startup.

[adraffy]

Ah thanks, I had no idea I could pull the data files directly from the CDN (and that they support the Range header.)

[Ladik001]

https://github.com/ladislav-zezula/CascLib

Initial commit of the new CascLib. Can read both Heroes and WoW storages. I'll review the published information and make adjustments if necessary. Thanks for the document.