[Diablo3] MPQ game files and *.gam file info

[Muaziz]

Quick MPQ Introduction
The MPQ file format was developed by Blizzard and is used by all their games. It is essentially an archive format that supports various compression techniques. It is optimized for read operations and uses hashing for quick file indexing.

The MPQ files are also used for patching the client game files. When a patch is released, it will include one or more MPQ files which contain only the updated files. The patch MPQ can also contain references to files that are to be deleted. When a file is patched, it can either be replaced in its entirety (if the changes are significant), or, more commonly, the MPQ will contain a BSDIFF of the file. Each file patch entry contains the MD5 of the file before the patch. This allows the patching mechanism to ensure that it does not attempt to patch a file that is not at the correct version level.

When the game client is loaded, it loads the needed game files from the appropriate "core" MPQ, and then applies that patches one at a time in chronological order. The MPQs do not store a version of the patched file! It's possible that the patched files are stored in the cache directory.

The tool that I've been using for working with MPQ files is Ladik's MPQEditor. (Not sure about the site policies for external linking of private sites, so I didn't include a link, but you can google easily enough.) I've had no trouble extracting and patching Diablo 3 game files with this tool.

Diablo 3 MPQ Files
In the Diablo 3 beta client, the core MPQ files are located in the following directory (for Windows clients):
[D3_Install_Dir]\Data_D3\PC\MPQs\

This directory includes the following MPQ files:
base-Win.mpq -- This MPQ contains the core executable files. Unlike other MPQs, the contents of this MPQ are extracted to the [D3_Install_Dir]. This MPQ includes the "Diablo III.exe", and various DLLs including the "bnet\battle.net.dll".
ClientData.mpq -- This MPQ contains most of the game graphics related files. I think the files in this MPQ are probably platform specific, but I haven't spent any time looking into the details of this MPQ.
CoreData.mpq -- This MPQ contains most of the interesting game client data. This includes all the affix information, item information, monster information, treasure classes, etc.
enUS_Audio.mpq -- As the name implies, this MPQ contains all the enUS localized sound files (*.snd and *.ogg). All sound files with English dialog are included here.
enUS_Cutscene.mpq -- This MPQ contains the enUS localized cut-scenes for the five character classes.
enUS_Text.mpq -- This MPQ contains all the enUS localized strings that appear as game text anywhere in the game. This includes everything from localized affix and item names to NPC conversations.
HLSLShaders.mpq -- This MPQ contains shaders.
Sound.mpq -- This MPQ contains non-localized music and sounds.
Texture.mpq -- This MPQ contains all of the game textures. These textures are stored in a Blizzard specific archive format. Each *.tex file can contain multiple textures. Necrolis' D3TexConv program (he includes the source in the download) can be used to extract the images from the *.tex files.

Diablo 3 MPQ Patch Files
THE MPQ patch files are located in one of three subdirectories under the MPQs directory:
base -- For all base files.
enUS -- For all enUS localized files.
Win -- For all Windows platform specific files.

An MPQ patch file has a name like "d3-update-base-7338.MPQ" where "7338" is the version number. As a reminder, to patch the core MPQ file, you must apply all of the patch files in sequence.

All files in an MPQ have a 0x10 byte file header. The first 4 bytes are always the "magic number": EF BE AD DE. Not sure if the next 4 bytes identify a version number or the file type, but this DWord is identical for all files of the same type for a given version. The last two DWords in this header always seem to be 0x00000000.

MPQ GameBalance *.gam Files
The "CoreData.mpq" file contains the "GameBalance" directory. This is where many interesting gam ("*.gam") files are located including "AffixList.gam" and "Items_*.gam". There's a lot of interesting game information located in these files.

The file type id / version number for *.gam files was 0x0000071C in the core MPQ, but is now 0x0000075F as of version 7318 or 7338.

MPQ GameBalance *.gam Header
The gam files have a header size (excluding the 0x10 byte MPQ header) of 0x3A8 as of version 7318 or 7338 (this was 0x398 in older versions).

I haven't figured out the complete structure of the gam header, but here is what I have so far (make sure you add 0x10 bytes for the MPQ header if you are looking at absolute file offsets):
DWord fileId; // +0x00 (unique file id)
DWord unknown1[2]; // +0x04
DWord startOffsetIndex; // +0x0C (?? see below)
DWord magicNumber1; // +0x10 (fixed value: 0x30303030)
DWord magicNumber2; // +0x14 (fixed value: 0x6962672E)
DWord unknown2[248]; // +0x18 (0xF8 bytes that always seem to be 0x00)
DWord resourceName[256]; // +0x110 (filename of the resource from which this file was generated)
DWord unknown3; // +0x210 (the value of this DWord is different for each file)
DWord unknown4[388]; // +0x214 (0x184 bytes that always seem to be 0x00 with the exception of the data start and end offsets -- see below)

The "unknown4" block contains all 0x00s with the exception of two DWords (back to back) that contain the data portion start and end offsets. For a given version of gam files, the start offset is always the same. For version 7338, this is 0x000003A8. The location of this offset in the "unknown4" block is variable and appears to depend on the "startOffsetIndex" at 0x0C. This value seems to be an index that determines the location of the two DWord offsets. For a given value of the "startOffsetIndex", the location of the offset DWords is always the same. Here are some examples from the core version; the first value is that of 0x0C, and the second value is the absolute file offset where the data start and end offset DWords are located:
0x01 --> 0x228
0x02 --> 0x238
0x03 --> 0x248
0x04 --> 0x348
0x05 --> 0x268
0x07 --> 0x288
0x08 --> 0x278
0x0A --> 0x298

While at first it appears that the "startOffsetIndex" directly corresponds to the offset location, there are several outlier values (e.g., 0x04 --> 0x348) that prevent this from being the case.

The way I have been processing these files is to read all the bytes starting at 0x214 and looking for the first non-zero DWord; this is the location of the data start offset. This approach seems to work fine although I'd like to find something a little more elegant.

The "unknown3" value at 0x210 is completely different for each gam file. I cannot find any kind of pattern and I have no clue what this value is.

MPQ GameBalance *.gam Data
Following the gam header is the data portion. Each record in the data portion has a fixed record size. As far as I can tell, this record size is not specified in the header, nor is the total number of records. However, it's easy enough to figure out the record sizes when looking at the hex dumps of the file.

Some gam files also have a variable data section located at the end of the records. To determine if a given gam file has a variable data section, simply add the MPQ file header (0x10 bytes) with the gam file header (0x3A8) with the data end offset. If this is the same as the gam file size, then there is no variable data.

Two of the files that have variable data are the "AffixList.gam" and "Items_Armor.gam" (as do most of the other "Item_*.gam" files). Gam files that contain variable data will contain pairs of DWords that specify how to read the variable: the offset of the variable data (from the start of the data portion) and the number of bytes of the variable data.

For the "AffixList.gam" file, each record can have up to 4 references to variable data. Each piece of variable data corresponds to an affix modcode (more on this later, probably in a separate post). The structure for the modcodes is as follows (0x18 in size):
DWord modcode; // +0x00
DWord modparam; // +0x04 (used for things like elemental damage, otherwise 0xFFFFFFFF)
DWord unknown1; // +0x08 (currently 0 for everything)
DWord unknown2; // +0x0C (currently 0 for everything)
DWord varDataOffset; // +0x10 (note that this is the offset from the start of the data section)
DWord varDataNumBytes; // +0x14 (number of bytes of the variable data)

The four modcode structures are located towards the end of each record at the following offsets:
modcode1: 0x1C0 (offset at 0x1D0, size at 0x1D4)
modcode2: 0x1A8 (offset at 0x1B8, size at 0x1BC)
modcode3: 0x190 (offset at 0x1A0, size at 0x1A4)
modcode4: 0x178 (offset at 0x188, size at 0x18C)

The size of the variable data varies based on the modcode. The possible values are 0 (in the case of no modcode), 0x0C, 0x1C, 0x28,0x34, and 0x40. In the core version, here are some of the modcodes with their corresponding sizes:
0x0C: Block, GoldPickUpRadius, HitMana, HitLife
0x1C: Experience, Resists, Attack, Prec, Def, Vit, Will, AllStats, Life, MinD, MaxD, Thorns, DR, KillMana, KillLife
0x28: Damage, CriticalD, Defense, Life, Gold, MF, Haste, Cast, LifeS, Regen, Run
0x34: Inferior, Superior
0x40: ItemCost

In the next post, I will dive into the details of the "AffixList.gam" file. After that, I will dive into the "Item_*.gam", and then the "\TreasureClass\*.trs" files.

[Dullface]

Good work so far! I look forward to your future posts, even though I don't understand much of it. Makes me wish I did!
Hopefully it'll lead to something awesome, keep it up brotato!

[R3d-Dr4gon]

Thanks for this great post.

Do you know if the Items.gam file contains all ingame items? (would make sense...)
I'm currently looking for a itemlist, including itemstats. But i couldn't find them yet.
There are several websites which already have datamined all currently implemented items... (for example Files - Diablo 3 Dev and Diablo III Database - )

[UnknOwned]

+rep on that one.

[Muaziz]

Do you know if the Items.gam file contains all ingame items? (would make sense...)
I'm currently looking for a itemlist, including itemstats. But i couldn't find them yet.
There are several websites which already have datamined all currently implemented items...

All of the Item types exist in the Items_Other, Items_Armor, and Items_Weapon .gam files. However, these files do not seem to have the base stats such as armor, weapon dmg, and weapon speed. They do have things like the iLvl and cLvl.

As for the sites that you referenced, all they have are things like iLvl and gold (which is incorrect btw). They also manually linked the images from the texture files. The Taiwanese site does have some good data correlation with salvage data. The salvage data is actually coded as a Treasure Class. This will be the discussion of an upcoming post. I've already decoded all the salvage stuff, but monster TCs are a lot more complex.

The Items_Legendary*.gam files (which I will start working on soon) have the names and affixes for all Legendary items. Unfortunately, I have yet to decipher the range for the mods of a given affix. I am not even sure that this information is available in the MPQ (it could come from somewhere else and be bound at runtime, or it could exist only on the server). My next detailed post is going to focus on the AffixList.gam file.

If you want to see the list of all base items (not including Legendaries), you can see it here. I've posted most of the useful information from the StringList (localized enUS) files. You can also view all changes ever made to these files. That feature will become more interesting to determine what changes are included in a given patch.

[Muaziz]

Quick update. I just took a look at the diffs from Patch 7447 and it looks like a number of files have been deleted from the game client. These include: ItemDropTable.gam, LootDistribution.gam, and sadly, all of the Treasure Class data.

The Monster (*.mon) files were not updated so it looks like that game client still has the ids for the TCs that a mob will drop. However, one would hope that the server has the authoritative TC data and that it is not relying on the client to send it the TC ids. After all, you always have to remember Rule # 1 of Client/Server programming: "Never trust the client".

Well, that really sucks. I was wondering why all those files were in the client to begin with, but it would have been nice to have access to this treasure trove of item generation data. Note that the TCs in the beta didn't include any information past the Skeleton King, so it will be impossible to data mine Inferno loot drops unless the server code gets leaked (not likely).

[Muaziz]

After some feedback from chippydip, I have made some updates to the MPQ section. I have also completely reworked that AffixList.gam page. The latter now contains a lot more detail about item group hashes (and how to compute them), as well as the mechanism for decoding the variable data (thanks to chippydip).

[Feah]

Hey Muaziz,

Do you know if it's possible to be able to somehow use English voice and text MPQE's to replace my Korean MPQE's?

I have a Korean version of Diablo III in which the language cannot be changed (bought this without knowing unfortunately).
Is it as simple as copy and paste or would the files need to be adjusted? If so is it possible?

Thanks in advance,
Kami

[BitHacker]

Muaziz,

Great post. I understood all of it. Even understanding all of it. I don't know how this is going to help me find the structures in memory yet. I'm on my quest to find data in memory today.

-Bit_Hacker