Optimization - Invalidating Objects

[Tanaris4]

Pretty quick question I believe. I'm basically trying to optimize the number of reads my bot is doing. So I'm trying to determine what things I can "cache". You would think I could associate a GUID and an entry ID with a given base object address.

But my question is, can I? Eventually an object will be released, and another could be created with the same base address, so this could create a problem.

This is what I would like to do:

1. When updating my internal object list, compare GUIDs of the objects I'm adding to my list. That way I can prevent the above issue
2. When doing random things like trying to calculate my average item durability, or the total number of mobs, or if a mob is nearby, I can use a cached entry ID or a cached GUID, vs. reading it from the baseAddress everytime

Does that sound about right? I currently invalidate an object my removing it from my internal list if it's not found after 5 new scans (I traverse the linked list every second to scan for new/old wow objects). So in theory an old object is removed after 5 seconds. Which means I could potentially have a lag time of 5 seconds when reading things like average durability, items the player is wearing, etc...

Does that sound about right? I just want to make sure before I make a pretty sizeable change to how my bot "consumes" information.

Thanks :-)

[Seifer]

Personally I wouldn't cache anything. For the sole reason I prefer the accuracy over the performance, and I don't feel my performance is lowering significantly by doing these amounts of reads.

What I don't understand is; are you currently using lists or dictionaries to store objects that you have read previously? And if so, are you flushing those every five minutes?

[Tanaris4]

This is how my object reading works. Every second I start by looking at the first object, then jumping 0x34 to the next until I'm @ the end of the list.

During this scan, I separate the objects into 5 separate arrays. Players, Items, Mobs, GameObjects, and DynamicObjects.

I think pass these arrays into each respective controller. Each controller class (i.e. PlayersController) will check it's list of the objects against the base addresses it was just passed, and it will verify the GUID is the same or even if an object still exists.

That is overall how the object list is managed.

Now for everything outside of this, I would simply like to read the cached GUID and entry ID. Things like health, location, obviously need to be real time.

[EmilyStrange]

Is it causing a performance issue on your particular machine?

I ask this, because until you profile this and have concrete data on an actual performance problem, I wouldn't worry about it caching. The idea of caching is "nice," and I worried about this myself when I started my project, but I have yet to have any performance issues.

To clearly state: I do absolutely no caching of anything in my multiboxing application. Based on the minimal amount of non-scientific code profiling, i.e. bringing up task manager under Windows, my application uses less than 1% of the CPU. Five clients on the same computer, all running at high resolution, with the graphics turned way up on all game clients, with the main multiboxing AI running at around 5Hz, and each client AI with a half-dozen threads each and it barely registers on the CPU. My code is literally creating and destroying thousands of objects every second.

My application is written in C#, and I recall that you are on a Mac, so yours is most likely written in Objective-C, meaning your code, has way less housekeeping overhead than mine.

I admit I have a reasonably good computer to run the game on, but when my MP3 audio player takes up more CPU than my AI, I cannot imagine I have a problem (yet).

However, to directly address your question:
I would create a Dictionary object to cache your game objects.
1. Each time a request is made for a GUID, look in the Dictionary first.
2. If the GUID of the game object exists in the dictionary, perform a quick read of the object's GUID from the game client memory, if they match, assume your cached game object is still valid (even if the data in your cached object contains is not current). You're done with your game object look-up.
3. If the GUID of the game object exists in the dictionary, but the GUID in the game client memory at that position is not identical, assume the game object no longer exists, and proceed with step 4.
4. If the game object does not exist in the dictionary, read the information form the game client memory, create a cached version of it in the dictionary.

This is of course predicated on the assumption that when a game object in the game client memory is removed, the game client memory is almost immediately recycled/compacted/disposed/invalidated. There is still the possibility for a race condition here, so write your code by taking the appropriate defensive steps.

Apoc actually made a post on using EnumVisibleObjects, with a similar caching strategy that recycled allocated game objects to prevent thrashing/rapid allocation & deallocation. You might want to do a search for that.

[Tanaris4]

Thanks for the tip on EnumVisibleObjects, I'll definitely look into that.

And yea I had been noticing some performance issues on Pocket Gnome, in a less populated zone the bot generally does around 1500-2000 reads per second if i'm just mining a zone (while ensuring no mobs or players are around the nodes). This used to be around 3000 before I found the pointer to the first object in the list.

But if I were to add an inventory check every couple seconds (to look for full inventory), I'm realizing it's highly inefficient as it has to scan EVERY item in the item list (and therefore look up the GUID again), so I was looking for some optimizations here. And if it made sense, port it to the other objects as well.

On my macbook pro it's not too bad, but then again I have 4GB and a 2.93 Intel Core Duo, so my machine isn't really taxed + I can run 2 bots fairly well, 3 then I have problems.

But it can spike to around 8000-9000 memory reads/second if I'm in a large PvP battle, so i'm looking to make optimizations where I can. (you can throttle it, say, tell it to only update information on your player every .5 seconds instead of every .1, so there are some options, but I'm still looking to optimize a little more).

I wish mine used 1% of memory, it usually hovers in between 10-35% unfortunately. I did add a feature to monitor which classes are doing the most memory operations to help optimize (I've been on an optimization kick lately):


Edit: The Controller class is what updated the object list every second, so roughly 600 operations per second just to do that, does that sound about right?

Edit 2: If it wasn't obvious, I use no injection, memory reads only. We figured out how to do injection, but I lack the knowledge to decipher the functions in IDA so it didn't really seem like a good idea We don't have an info thread for the mac binary heh

[Flowerew]

Though I haven't read a single line of that thread, I have to say THAT SCREEN LOOKS SEXY!

[Tanaris4]

I like it too :-) But I can't take credit for the UI, Gnome Overlord created the app, then he open-sourced it + I took charge heh (was my first venture into OS X programming and bot coding)

[Apoc]

I haven't looked at the PG bot code for a while, so take my post with a grain of salt.

There are certain things you can cache, without any drawbacks.

GUIDs being the main one.

If your framework is setup the same way as WoW's (eg; base class Object, with inheritors), just cache the GUID on a per-Object basis. GUIDs are unique, regardless of shapeshift, etc, so it's perfectly safe to cache that.

I wouldn't suggest caching anything else to be completely honest. If you need to 'cache' certain things (like the player name, before doing a giant compare loop), do it within the method you're working in. Since WoW is quite dynamic, you really can't cache much.

Also (if you read my thread about better object management), you should only be deallocating the objects once every n frames (10 in my case). You can easily 'invalidate' an object by setting its base pointer to 0, which I do every frame.

Lastly; our bot currently does roughly 10-11k reads/sec. (Mind you; that's also 60 'pulses' if I have vsynch on, it can easily go well above that when I turn vsynch off.) I suggest you optimize your read code first of all. Injection will definitely be the 'best' route, but you can get away with a totally passive bot doing enormous amounts of reads without issue. (We used to use ReadProcMem, and that was only marginally slower than using unsafe code, and directly dereferencing addresses. [RPM was only to allow us to read a set of bytes at a time, rather than having to go through the Marshaler any time we wanted a byte.])

Lastly; figuring out the function prototypes are simple. They should be roughly the same in both Windows and Mac versions. (Exceptions aside obviously.)

If you want, you can upload the binary, and I'll see about 'transferring' info from my windows IDB to the mac IDB. May be able to do a 'dual dump' each patch. Even though that'd be a giant pain in the ass.

Honestly; you're better off just following the code flow from the windows binary, and just copying it through the mac binary. They're 99% similar as far as code structure goes. (They'd be 100% similar if not for the whole different OS, stuff)

WoW actually does a really good job abstracting away any OS specific stuff. (They have funcs called OsReadFile, OsGetFile, OsOpenFile, etc, which are OS specific. Very cleanly done too.)

[RoKFenris]

What I'm doing to avoid bogging down my radar with excessive reads is having a private addon update an array with timestamps and status lists for things that are available from the Lua API, change infrequently, and would require a hefty amount of memory reads. For example, I keep a list of every single mob/object/player I need to find for quests and achievements, and only update that on the addon when it changes, as determined from events. My C++ radar usually only reads the timestamp for the last change, and only reads the whole list if the timestamp has changed.

(I've also limited my radar to reading 10 times per second, tops, or once a second if I'm not yet logged.)

BTW, besides performance reasons, I use caching for better data persistence on my radar. Things vanish too soon (try flying on an epic mount and looking back), so I built a 10-sec delay from the object vanishing from the client and it disappearing from my radar (with a distance based exception in order for things I collect/kill to vanish immediately).

[L33ch]

Though I haven't read a single line of that thread, I have to say THAT SCREEN LOOKS SEXY!

QFT!

That screen's got me new idea's for my ui!