[Mpq][C#] Geometry parser/viewer

[Bananenbrot]

Hey folks,

I'm currently at parsing Mpq data hopefully for the last time in the near future...
I plan to implement my own meshing algorithm in C# (which I will not release to the public for now). Note, that this is ~2 times slower than what I had hacked together in C++ (iirc, who knows), but it is definitely cleaner, easier to extend and stable in regards to changes. Today I prepared for 5.0 (actually hadn't to change anything, only CrystalMpq needed an update), so it will run smoothely for next few months. Also packaged up is a sample WPF project featuring tile selection and viewing.
I even managed to get running WMO water and MH2O liquids based on ExistsTable with one annoying exception... In Stormwind [30, 48] are some liquid tiles around [46] which I couldn't fix. See WoWDev if you want to know why.

It targets .Net 4.5 and is compiled with Visual Studio 11, so don't blame me if you don't get it to run.
I will ignore requests like "I don't get it to work, make a tutorial plz", but answer every specific question which might be related to a crash caused by my failures.

See this mainly as a reference implementation... the viewer is just for demonstration purposes. On the next occassion, I might populate the bug tracker with things I already found but find to annoying to fix.

Some hints to get started: The Scene class carries SceneObjects with all parsed geometry, which can even carry flags (like it is the case for lava and slime).

github: http://github.com/Bananenbrot/BananaMpq
binaries: http://github.com/downloads/Bananenb.../BananaMpq.zip

Edit: You will need the english locale if you don't want it to blast the second you start it. Every semi-competent programmer should be able to fix that in MpqFileReader.cs though.
Edit2: Ok, just adjusted the locale "searching" to be more forgiving. You should not need to fix it anymore.
Edit3: Yummy pictures

[Empted]

Good one, i have to say. Code is very informative by itself. Though, i think doodad sets are not implemented for WMO, or implemented in wrong way (don't see any MODS chunk declaration in your code) You should only refer to doodads that are listed in doodad set. There is index in related MODF header and by that index you should get start in doodad array and total doodad count in the set.

            DoodadSet choosenSet=dSets[modf.doodadSet];
            for(int i=(int)choosenSet.firstinstanceindex;i<choosenSet.firstinstanceindex+choosenSet.numDoodads;i++)
            {

But still i think you know that, just forgot to code

[Bananenbrot]

Yeah, I know that. But because I reference every doodad definition of a WMO anyway, I can't have possibly missed one.
I actually don't have a clue what doodad sets do apart from grouping up doodads...

[Empted]

It's crucial to refer only to those doodads that are listed in doodad set. For example you got some doors that are not there in WoW. So navmeshes can be wrong. Sets not just group doodads, they describe which of them are actually real. Those sets make it viable to get WMO look different, like normal house and abandoned house with mess around. If you merge all those, you won't get true wmo shape.
Still i don't claim anything. Just marked that building navmesh from that geometry can be wrong.

[Bananenbrot]

Ahh... So that makes sense. Will update it asap.
Edit: Done
Edit2: Now also using only the referenced doodads...

[Viano]

Thanks for that

[redcatH]

......Powerful！

[reliasn]

I'm bumping this thread because after a 3-week "research" about WoW geometry and navmesh-related stuff, this tool is the only one I found that works just fine with 5.4.7 (build 17956). Despite the gitHub version being quite old, you can still parse and view the tiles of the game. I just want to point out the issues that I ran into and the fixes:
1 - I checked-out the latest github version and it wasn't even running. Then I noticed that CrystalMPQ has a method to automatically find the WoW installation, which wasn't working for me. So I checked-out the latest modified CrystalMPQ version (https://github.com/Bananenbrot/crystalmpq) and modified the file WoWInstallation.cs to make it return the path of my WoW installation.
2 - After compiling, I grabbed the DLLs and put them in trunk\Dependencies of BananaMPQ. I then ran the project and it finally loaded everything just fine.
3 - I read this thread (http://www.ownedcore.com/forums/worl...ng-issues.html ([NavMesh][MPQ] Geometry parsing issues)) where Bananenbrot reported 2 issues. The first one was just a mistake, nothing to worry about. The second one required some corrections in the code, which were posted by Bananenbrot in the thread, but it took me a while to make everything compile just fine. So I'm posting here the new MapChunk.cs file that corrects the issues with "holes":

using System.Collections.Generic;
using System.Linq;
using System.Reflection;
using System;
using BananaMpq.Layer.Chunks;
using BananaMpq.Visualization;
using SharpDX;

namespace BananaMpq.Layer.AdtRelated
{
    public class MapChunk : Chunk, IChunkCollector
    {
        public const float ChunkWidth = Adt.AdtWidth / 16.0f;
        public const int TilesPerChunk = 8;
        public const float TileSize = ChunkWidth / TilesPerChunk;
        public float[,] HeightMap { get; private set; }
        public uint Flags { get; private set; }
        public byte[] Holes { get; private set; }
        public int X { get; private set; }
        public int Y { get; private set; }
        private BoundingBox _bounds;
        public BoundingBox Bounds
        {
            get { return _bounds; }
        }

        public ChunkLiquid Liquid { get; set; }
        public int[] WmoReferences { get; private set; }
        public int[] DoodadReferences { get; private set; }

        internal unsafe MapChunk(ChunkHeader* header)
            : base(header)
        {
            header->ValidateMagic("MCNK");

            var cur = (byte*)header + sizeof(ChunkHeader);
            ParseMcnkHeader(cur);

            Chunks = ChunkCollector.CreateChunks(cur + sizeof(McnkHeader), cur + header->Size, CreateChunks);
        }

        public bool HasHole(int col, int row)
        {
            return ((Holes[row] >> col) & 1) == 1;
        }

        private static byte[] TransformToHighRes(ushort holes)
        {
            var ret = new byte[8];
            for (int i = 0; i < 8; i++)
            {
                for (int j = 0; j < 8; j++)
                {
                    int holeIdxL = (i / 2) * 4 + (j / 2);
                    if (((holes >> holeIdxL) & 1) == 1)
                        ret[i] |= (byte)(1 << j);
                }
            }
            return ret;
        }

        private unsafe void ParseMcnkHeader(byte* cur)
        {
            //var mcnk = (McnkHeader*)cur;
            McnkHeader* mcnk = (McnkHeader*)cur;

            _bounds = new BoundingBox(
                mcnk->Position + new Vector3(-ChunkWidth, -ChunkWidth, 0.0f),
                mcnk->Position
            );

            //Holes = mcnk->Holes;
            Flags = (uint)mcnk->Flags;
            Holes = mcnk->Flags.HasFlag(McnkHeaderFlags.HighResHoleMap) ? mcnk->HighResHoles : TransformToHighRes(mcnk->Holes);
            X = mcnk->X;
            Y = mcnk->Y;
        }

        private unsafe Chunk CreateChunks(ChunkHeader* header)
        {
            switch (header->Magic)
            {
                case "MCVT":
                    var mcvt = new McvtChunk(header);
                    HeightMap = mcvt.HeightMap;
                    _bounds.Maximum.Z += HeightMap.Cast<float>().Max();
                    return mcvt;
                case "MCRD":
                    var mcrd = new OffsetChunk(header);
                    DoodadReferences = mcrd.Offsets;
                    return mcrd;
                case "MCRW":
                    var mcrw = new OffsetChunk(header);
                    WmoReferences = mcrw.Offsets;
                    return mcrw;
                default:
                    return new Chunk(header);
            }
        }

        internal unsafe void ParseOptionalData(ChunkHeader* header)
        {
            header->ValidateMagic("MCNK");

            var cur = (byte*)header + sizeof(ChunkHeader);
            Chunks = Chunks.Concat(ChunkCollector.CreateChunks(cur, cur + header->Size, CreateChunks));
        }

        #region Implementation of IChunkCollector

        public IEnumerable<Chunk> Chunks { get; private set; }

        #endregion

#pragma warning disable 169, 649

        [Flags]
        public enum McnkHeaderFlags : uint
        {
            HighResHoleMap = 0x10000
        }

        public struct McnkHeader
        {
            public McnkHeaderFlags Flags;
            public int X;
            public int Y;
            int nLayers;
            int nDoodadRefs;
            uint offMcvt; // 0x14
            uint offMcnr; // 0x18
            int offMcly;
            int offMcrf;
            int offMcal;
            int sizeAlpha;
            int offMcsh;
            int sizeShadow;
            int areaId;
            int nMapObjRefs;
            public ushort Holes;
            ushort HolesAlign;
            unsafe fixed short lowQualityTexturingMap[8];
            int predTex;
            int noEffectDoodad;
            int offMcse;
            int nSoundEmitters;
            int offMclq;
            int sizeLiquid;
            public Vector3 Position;
            int offMccv;
            unsafe fixed int pad[2];
            public byte[] HighResHoles { get { return BitConverter.GetBytes(offMcvt + ((ulong)offMcnr << 32)); } } // 0x14
        }
#pragma warning restore 169, 649

        public override IEnumerable<PropertyInfo> VisualizableProperties
        {
            get
            {
                var type = typeof(MapChunk);
                return base.VisualizableProperties.Concat(new[]
                {
                    type.GetProperty("Bounds"),
                    type.GetProperty("X"),
                    type.GetProperty("Y"),
                    type.GetProperty("Flags"),
                    type.GetProperty("Holes"),
                    type.GetProperty("HeightMap"),
                    type.GetProperty("DoodadReferences"),
                    type.GetProperty("WmoReferences"),
                });
            }
        }
    }
}

4 - After doing this, I recompiled all the projects, including the Dumper. And yeah, the issue was perfectly fixed:

5 - To open MOP tiles, add this to the function MapToFolder(WowContinent continent) in MpqFilePath.cs:

case WowContinent.Pandaria:
        return "HawaiiMainLand";

Bananen probably has an updated version of his parser/viewer, but I don't know if he plans to commit it to his Github or not. It seems that this tool currently ignores _obj1.adt's and I honestly have no idea if it will end up missing certain WMO's or Doodads.

Other than that, I just want to thank the developer for his amazing work and for leaving the source opened. I read WoWDev, checked-out wowmapper's (outdated and doesn't seem to support MOP) and meshReader's code, but I was still struggling with all these crazy chunks. Now, with this working version, I guess I can try once again to understand how all the information is parsed. Thanks

PS: C++ boy here... maybe that's why it took me so long to get everything working >.<

[xalcon]

Obj1 and tex1 are just lower detail versions of obj0 and tex0 respectivly. They are Not needed for pathfinding

[Bananenbrot]

snip

I actually have a working and less buggy system within my NavMesh Generator right now, but haven't yet come up with the time needed to seperate out the parsing code again.
Can't say if I will have the motivation/time anytime soon, but I'm surprised you could get it to work.

As for the WoWInstallation: You could open a pull request for a new static At(string path) method or something like that. Or I'll do when I modify it next time.

[hardcpp]

Huumm i can't get this work with visual studio 2012, always crash on CrystalMpq.WoW.WoWInstallation.Find() anyone can help me ?

[xalcon]

Huumm i can't get this work with visual studio 2012, always crash on CrystalMpq.WoW.WoWInstallation.Find() anyone can help me ?

1 - I checked-out the latest github version and it wasn't even running. Then I noticed that CrystalMPQ has a method to automatically find the WoW installation, which wasn't working for me. So I checked-out the latest modified CrystalMPQ version (https://github.com/Bananenbrot/crystalmpq) and modified the file WoWInstallation.cs to make it return the path of my WoW installation.

[the message you have entered is to short, bla bla bla]

[hardcpp]

x) i didnt noticed that, thks

[Geraldo29]

I live in Nevada.

[Bananenbrot]

As I'm on a fresh installation right now, I recall that CrystalMpq can't find your (my) installation because of a missing registry key.
You just have to start wow(-64).exe as administrator once to generate the appropriate keys and your Find() magic can go.

Edit: Also with version 1.7.3 of crystal mpq I just released, there is now a method AssumeAt() on WoWInstallation which does what you (and I on my mac's windows vm) want.