[C#] Enigma.D3

[bitterbee]

Sweet update, I hope to get some time for D3 projects again soon so I can test it out. I guess I'll make some time once the new patch lands

[enigma32]

rev 19

• Multi-threaded applications are now supported. Make sure to set Engine.Current in new threads or last created engine will be used with Helper classes.
• Added Enigma.D3.Sno library. This replaces the "AutoGenerated" folder (which is deleted now).
• Added LCG hash to HashUtils. Apply this to a GameBalance name to get the ID.

Hi Enigma. Great framework u got there. I'm still trying to get access to navigation data. After initial success all went down to shithole after Scene class update xD Here is what I got (based on information from this thread):

You should probably be able to extract what you need using this update

// loading SNO data directly from a file (offset by 0x10 due to header):
var file = new FileMemory(@"...\GameBalance\Items_Legendary.gam");
var gbItemsLegendary = MemoryObject.Create<GameBalance>(file, 0x10);


// loading a StringList from memory, with known ID:
var def = Engine.Current.SnoGroupsByCode[(int)Enums.SnoGroupId.StringList]
    .x10_Container
    .Where(a => a.x08_SnoGroupId == Enums.SnoGroupId.StringList)
    .First(a => unchecked((uint)a.x00_Id == 0x81b909c8));
var stringList = def == null ? null : MemoryObject.Create<StringList>(Engine.Current.Memory, def.x10_SnoItem.Address);
// note that SnoGroupId is checked since the container could be "Global" which contains different types.

Helper methods will come (case reading from memory looks confusing), but in the meanwhile write your own.

The following SNO types might give some problems:
HelpCodes, LootDistribution, Material, Monster and QuestRange. Reason being that different types have the same name and I'm auto-generating the code based on that information, so those will be marked as "known" as soon as seeing it for the first time. I've only added special rule for "Items" in GameBalance, where one should in fact better be named "ItemTypes".

[CrEEzz]

Thx, I'm working on it. Meanwhile I think there is minor bug in ProcessMemory.cs, inside ValidateNumberOfBytesRead WriteLine is missing one argument, suppose it should look like this:

[Conditional("DEBUG")]
		private void ValidateNumberOfBytesRead(int address, int numberOfBytesRead, int numberOfBytesExpected)
		{
			if (numberOfBytesRead != numberOfBytesExpected)
			{
				Debug.WriteLine("{0}({1:X8}): numberOfBytesRead ({2}) != size ({3})",
					"ProcessMemory.Read", address, numberOfBytesRead, numberOfBytesExpected);
			}
		}

[enigma32]

Ah, you're absolutely correct! Guess I never had a partial read
Thanks for pointing that out!

[CrEEzz]

Ok. Its much better now. However as I understand I must connect Sno.Scene with actual Scene somehow. Because some of the Sno.Scenes extracted by following code:

foreach (SnoDefinition<MemoryObject> sno_def in Engine.Current.SnoGroupsByCode[(int)SnoGroupId.Scene].x10_Container)
{
        if (sno_def.x08_SnoGroupId != SnoGroupId.Scene)
            continue;

        Enigma.D3.Sno.Scene sno_scene = sno_def.x10_SnoItemAs<Enigma.D3.Sno.Scene>();
        ...
}

are invalid (its internals look like released memory or sth). I used to connect them using SnoSceneId which is available in Scene.x0E8_SceneSnoId however is now missing in Sno.Scene (it used to be at x08E, but there is now SerializeData). Any ideas appreciated. Thanks Enigma!

[enigma32]

SNO structures are used when dealing with MPQs, messages and streaming content (during patching). The Scene SNO is not the same as the Scene structure that is used to handle actual game logic.

To get the ID from a Scene SNO you have to add a field at 0x00. It's not included in the metadata D3 has (at least not what I've found) that's why it's not in the generated files. I assume the ID is there for all SNO structures, besides for GameBalance which contains multiple items (get GBID by LCG hashing the name.ToLower()).

The SNO ID should also be in the SnoDefinition, x00_Id.

[CrEEzz]

Thanks, it worked! However ocassionaly there is weird problem occuring (all below based on debug version). I get:

ProcessMemory.Read(01CE4AAC): numberOfBytesRead (4) != size (0)

on any memory read performed, example callstack:

>	Enigma.D3.dll!Enigma.ProcessMemory.ReadBytes(int address = 30296748, byte[] buffer = {byte[8]}, int offset = 0, int count = 4) Line 104	C#
 	Enigma.D3.dll!Enigma.MemoryBase.ReadBytes(int address = 30296748, int count = 4) Line 76 + 0x35 bytes	C#
 	Enigma.D3.dll!Enigma.MemoryBase.Read<int>(int address = 30296748) Line 40 + 0x33 bytes	C#
 	Enigma.D3.dll!Enigma.MemoryObject.Field<int>(int offset = 4) Line 147 + 0x3e bytes	C#
 	Enigma.D3.dll!Enigma.D3.LocalData.x04_IsNotInGame.get() Line 17 + 0x10 bytes	C#
 	NavMeshViewer.exe!NavMeshViewer.NavMeshViewer.IsLocalActorValid(Enigma.D3.Engine engine = {Enigma.D3.Engine}) Line 115 + 0x4b bytes	C#

What is astounding is that bytesToRead changes its value from 4 to 0 during Win32.ReadProcessMemory call O_o Well there are no data breakpoints in C# :/ so finding instruction changing its value is more challenging... So I added:

Debug.WriteLine("{0}({1:X8}): last error: {2}", "ProcessMemory.Read", address, Marshal.GetLastWin32Error());

just before ValidateNumberOfBytesRead(address, numberOfBytesRead, bytesToRead); (when reading failed)

and quess what... last error code is 0 meaning ERROR_SUCCESS :confused: I'm investigating further, but any ideas appreciated Enigma

EDIT: I guess I found the reason. I assume it was concurrent calls to framework API from timer thread (when previous call didn't finish).

[CrEEzz]

I would like to summarize findings (including others contribution - thx a lot guys). Scene (not SNO one) contains:

public float x0FC_MeshMinX { get { return Field<float>(0x0FC); } }
public float x100_MeshMinY { get { return Field<float>(0x100); } }
public float x174_MeshMaxX { get { return Field<float>(0x174); } }
public float x178_MeshMaxY { get { return Field<float>(0x178); } }

Temporary code (not optimized from updates point of view) to obtain navigation data:

float player_x = ActorHelper.GetLocalActor().x0A4_WorldPosX;
float player_y = ActorHelper.GetLocalActor().x0A8_WorldPosY;

System.IO.StreamWriter file = null;

if (dump_to_file)
	file = new System.IO.StreamWriter("nav_data.txt");

foreach (SnoDefinition<MemoryObject> sno_def in Engine.Current.SnoGroupsByCode[(int)SnoGroupId.Scene].x10_Container)
{
	if (sno_def.x08_SnoGroupId != SnoGroupId.Scene)
		continue;

	Enigma.D3.Sno.Scene sno_scene = sno_def.x10_SnoItemAs<Enigma.D3.Sno.Scene>();

	foreach (Scene scene in Engine.Current.ObjectManager.x798_Storage.x1BC_Scenes)
	{
		if (sno_scene.x00_Id == scene.x0E8_SceneSnoId)
		{
			if (dump_to_file)
				file.WriteLine("s {0} {1} {2} {3}", scene.x0FC_MeshMinX, scene.x100_MeshMinY, scene.x174_MeshMaxX, scene.x178_MeshMaxY);

			Enigma.D3.Sno.Scene.NavCell[] nav_cells = sno_scene.x180_NavZoneDefinition.x08_NavCells;

			if (nav_cells == null)
				continue;

			foreach (Enigma.D3.Sno.Scene.NavCell nav_cell in nav_cells)
			{
				// skip not walkable
				if ((nav_cell.x18 & 1) == 0)
					continue;
					
				float min_x = nav_cell.x00_Vector3D.x00 + scene.x0FC_MeshMinX;
				float min_y = nav_cell.x00_Vector3D.x04 + scene.x100_MeshMinY;

				float max_x = nav_cell.x0C_Vector3D.x00 + scene.x0FC_MeshMinX;
				float max_y = nav_cell.x0C_Vector3D.x04 + scene.x100_MeshMinY;

				if (dump_to_file)
					file.WriteLine("n {0} {1} {2} {3}", min_x, min_y, max_x, max_y);
			}

			break;
		}
	}
}

if (file != null)
	file.Close();

Looks promising this is the result in Bastion Keep:



However... (there always must be something xD) when I change location and try to obtain data i start getting ERROR_PARTIAL_COPY when accessing Scene fields so I assume there are now scenes in ObjectsManager storage which are no longer valid, but match SNO scenes. Now I'm looking for a method to identify whenever Scene was released but is still on the list.

Second problems is sometimes when I try to get anything from framework I get errors 1008 (ERROR_NO_TOKEN - An attempt was made to reference a token that does not exist.) on following code being first thing called after Engine.Create (and this time there no timers/threads involved - this is called from main thread only). Here is the code:

namespace NavMeshViewer
{
    public partial class NavMeshViewer : Form
    {
        public NavMeshViewer()
        {
            InitializeComponent();

            Engine.Create();
            ActorHelper.GetLocalActor();
        }
    }
}

[enigma32]

EDIT: I guess I found the reason. I assume it was concurrent calls to framework API from timer thread (when previous call didn't finish).

I'm not sure I quite followed everything there. The call stack looked fine and the only way I see "count" could be changed is if it doesn't really change, but another thread entering the same method and one not realizing there are different contexts. I did some research on ReadProcessMemory and what I understand is that "numberOfBytesRead" could never be different from the requested byte count, unless the method also returns with an error (like partial read). So theoretically it's impossible that ValidateNumberOfBytesRead has anything to complain about, since there should be a Win32Exception thrown if there was a partial read. If a buffer was resized during reading however, things could get tricky, but I can't find a scenario where this could happen.

I would like to summarize findings (including others contribution - thx a lot guys). Scene (not SNO one) contains:

...

Second problems is sometimes when I try to get anything from framework I get errors 1008 (ERROR_NO_TOKEN - An attempt was made to reference a token that does not exist.) on following code being first thing called after Engine.Create (and this time there no timers/threads involved - this is called from main thread only). Here is the code:

Thanks for sharing, always appreciated Do you by any chance run the application as a different user than what you run D3 as? Try run as administrator and see if that helps, and also try without debugger. I'm using the process handle directly from the Process instance, but I guess it might not have all the required privileges under some circumstances. At least I know for sure you cannot read memory from another user process unless you're running as admin, but the exact error I'm not sure of.

[bastiflew]

At least I know for sure you cannot read memory from another user process unless you're running as admin, but the exact error I'm not sure of.

It's true, especially on Windows 8.
The best way to force administrator privileges, is to add a manifest file to the executable project (right click on projet, add new item, application manifest file)
Edit the manifest and replace this line :

<requestedExecutionLevel level="asInvoker" uiAccess="false" />

to this line :

<requestedExecutionLevel level="requireAdministrator" uiAccess="false" />

Hope this help.

[CrEEzz]

Ok. Release version of Enigma.D3 seems to be working fine. Now, I'm getting close to first release of Nav.D3. I generate actual navmesh from soup of AABB rectangles and I got simple A* pathing and some caching. But I obviously still have trouble with accessing all scenes. I tried getting them from:

ObjectManager.x954_Scenes

and

ObjectManager.x798_Storage.x1BC_Scenes

(BTW they seems to be the same list). But there are still holes visible on screenshot below:



I suppose there are two options:
1) there is second list of scenes in ObjectManager or Storage that I do not know about
2) missing scenes are not visible because there is no Sno.Scene with matching their sno id

@bastiflew you are using navmesh navigation in your bot, perhaps you can give me a hint?

EDIT: I have verified option 2 and this is not the case. For each scene there is matching sno.scene. Additionally there are no scenes with null empty NavCell list.

[bastiflew]

I don't try to fetch scene with enigma's framework, but there's some rules about scenes :

- If there's no navcell in a SNOScene, i put a unique walkable navcell. (but not always )
- compute sub scene after all other scenes
- the scene list is 0 index-based, but the scene count is the max index in fact. (so if the count is 10, you have 11 scenes)

[CrEEzz]

@bastiflew I see, but can u confirm that there is only one list of scenes? Where are you getting them from, ObjectManager or Storage?

EDIT: I assume you have offline generated navmesh for each scene SNO id. And then at runtime you get list of currently used SNO ids and merge their navmeshes using recast. If I'm correct, there might be list of SNO ids of active scenes somewhere.

EDIT 2: Ok, I have found some disturbing fact. I iterate over 14 scenes from ObjectManager.x954_Scenes and there are only 11 scenes with distinct SceneSnoId. Scenes with the same SceneSnoId have different MeshMinX/Y meaning they are in fact different scenes. This means matching Scene with Sno.Scene using only SceneSnoId is not sufficient.

EDIT 3: Is SceneSnoId unique for each type of map tile? It seems so. I implemented cache which skipped already processed Scenes based on its SceneSnoId. Now I see that was a mistake, and my missing scenes were discarded because had same SceneSnoId (navmesh layout) as already processed Scenes. So for my cache to work I need to distinguish Scenes by some Id which remains unchanged after Scene has been reloaded (obviously D3 keeps only those Scenes that are close enough to player).

EDIT 4: Yeah after caching Scenes by x000_Id everything works like it was supposed so. No missing Scenes anymore.

[axlrose]

Hi Creezz,

1.) Both lists are the same instance (0x798+0x1BC=0x954).
2.) Don't forget, that SnoSceneId may be references multiple times (that's why you find multiple references to Sno.Scenes in Storage with different offsets).
3.) D3 unloads scenes that are far away. You'll have to cache everything needed.

BR

[CrEEzz]

Hi axlrose, You are absolutely right, now everything seems to work just fine. If my tests go well I will release lib in a few days. Below is Dahlgur Oasis navmesh with path leading through waypoints taken from one of the DemonBuddy scripts for farming this location xD