Let's Learn concepts: Endscene, hooks. Sufferin' succotash! Quick into the codecave!

[Megamike55]

This post was born out of my own desire to learn about these concepts. It's always a good idea to try teaching someone else what you have just learned to help solidify it in your own head, so here you go. I wanted to learn concepts, so hopefully this post will help you learn these important concepts as well. (Will update this post based on any feedback) (Total Edit Count:6):

1. [Edit2] First off: "End Scene" is the DirectX API function that the game must call when all its drawing is complete. (Clarified: A DirectX API "Present" is responsible for actually flipping the back buffer to front) Easy enough to understand. Clearly a call to this function exists in WOW, and so long as World of Warcraft is rendering in DirectX mode (DX9), it can be useful to us.
   
2. [Edit2] "hook" or "detour" are not the same thing, but are related. Let's focus on "hook". A definition of a Function Hook (only one of many kinds of hooks, but quite common) is given on wikipedia, as well as examples of other types:
   

   Function hooking is implemented by changing the very first few code instructions of the target function to jump to an injected [piece of] code. (Hooking - Wikipedia, the free encyclopedia)

3. [Edit1] A Hook can (if desired) block original code from running, however in the case of World of Warcraft, that would not be productive (with an Endscene hook)
   
4. [Edit1] Before the hook returns to the Endscene function, it can perform any number of calls to other engine functions, or even draw more to the back buffer. This is what allows us to, for example, draw a bot's current planned movement (way-points, whatever).
   
5. [Edit1] An EndScene hook will be run on the game's render thread, since that is where the call to EndScene originates. In World of Warcraft, the render thread is the same as the game thread.
   
6. [Edit2] Another great thing to use is DLL Injection. Here is a great definition and four ways to inject DLLs given by Wikipedia:
   

   DLL injection is a technique used to run code within the address space of another process by forcing it to load a dynamic-link library. (DLL injection - Wikipedia, the free encyclopedia)

   Using DLL Injection, we can run code in WoW Address space which does not need to be run every frame. Such as LUA commands. When injecting the DLL, one common way is to create a new thread in the target process to call LoadLibrary (see the wikipedia page for details).
7. [Edit3] Some people like to use a term "Code Cave" when talking about hooks or DLL injection. The "Code Cave" is a place in the target process's memory which is unused, that we can replace with our own custom code to be run. (see Xarg0's comment further down the page for an example of when a Code Cave is quite useful). Also See this source, and for an in-depth discussion of code caves on the assembly level this source
8. [Edit1] VMT = Virtual Method Table. I Think it's quite likely that many programmers (I am not exempt from the following generalization, either), after learning how to Use a language, never really understand how it actually works. All that lovely inheritance, overloading, virtual functions, etc. has a lot going on "behind the scenes" so to speak. If you don't know what a VMT is, it's high time to learn: http://en.wikipedia.org/wiki/Virtual_method_table.
   For making a hook to any D3D function, knowing about its Device VMT is quite necessary. If you just take a look at the D3D API, you can go through it, pick a function to hook, and grab the pointer from D3D's device VMT (You will know its index in the device VMT - same order as in d3d?.h - e.g. d3d9.h) source
   Here is a post on this forum about hooking D3D11: http://www.mmowned.com/forums/world-...1-hooking.html (the "problem" comes in that there is no more "EndScene")

I am sure there are more points to add to make this clearer. Any feedback is appreciated. If this post helped you or you'd like more information on related topics, drop a line.

Useful Links / Sources:

1. assembly - What is a code cave, and is there any legitimate use for one?
2. The Beginners Guide to Codecaves - CodeProject
3. "Dll Injection using SetWindowsHookEx() Method" Cached Page by Google (original has been lost)
4. C# – Screen capture with Direct3D 9 API Hooks
5. DLL injection - Wikipedia, the free encyclopedia
6. Hooking - Wikipedia, the free encyclopedia
7. Best Practices for Creating DLLs

[Azzie2k8]

2) "hook" or "detour" are the same thing, and saying "hook endscene" or "detour endscene" means to go to the address in the target game's code which calls the DX endscene function and replace the function call there with a call to your own function.

I am not a pro myself but as far as I understand it a detour does not replace the function but writes a jump to your function to the first bytes of the hooked function.

3) The function you create must at some point call the DX Endscene function.

In general this is not true as far as I can tell. Or let's say it depends on the behaviour you want to achieve. If you do not call the original EndScene I think wow will crash but in general it is not necessary to call the original function and sometimes it is also intended that the original function isn't call to prevent the client from doing things. But with EndScene you are probably right that it would be a good idea

4) Before it calls the Endscene function, your function can perform any number of calls to other engine functions, or even draw more to the back buffer. This is what allows us to, for example, run protected LUA functions.

I think this is true as long as you do it right and don't mess up the stack and so on before calling the original function. But for this you do not need the EndScene hook but just the injeted Dll or whatever. The hook only makes sure that your code gets executed over and over again and not just once.

I am sorry if something was wrong so please correct me. I am not a pro myself. Just trying my best to help.

[suicidity]

A hook and a detour are NOT the same, a detour is a type of hook you can do. A "detour" usually refers to the method of creating a relative jmp to one of your own functions. Your function will usually jmp to a "trampoline".

A "hook" is any method of "calling" (used loosely) your own function when a specific part of code gets executed.

Just clarifying

[adaephon]

Also just clarifying: EndScene doesn't swap the buffers - Present does that (on DX9). Although you're right in that DirectX mode is required, if it's been set to use DX11 (new in Cata) an EndScene hook won't work.

EndScene is a convenient method to hook as it allows you to drawing (as you've mentioned), as in DX9 draw calls must come between IDirect3DDevice9::BeginScene and IDirect3DDevice9::EndScene. It's also convenient because it is guaranteed to run every frame and thus gives you a regular update interval. It also allows you to run code on WoW's render thread, which happens to be it's main thread, which removes most threading issues caused by accessing game objects/state/code from a thread they weren't created from. However, just because WoW's render thread is its main thread, there is no hard and fast requirement that the render thread and program main thread have to be the same (although this is probably the most common scenario).

So I suppose this was a round about way of saying you don't have to hook EndScene - you can hook other methods to do the same things (BeginScene, Present both give you thread affinity and regular updates, and would be just as appropriate for running Lua code, although might not be as useful for drawing to the device for instance).

[Xarg0]

The reason why you aren't running your stuff in DllMain is pretty simple, LoadLibrary aquires the loader lock and it isn't freed until DllMain is finished, only basic initalisation is to be done in DllMain.
Infact, creating a thread from DllMain is a bad idea, you really should read the ms paper on DllMain best practises.

Also, your definition of code cave is wrong, it originally meant places in a modules .text or .code section (some compilers use different names, basically they're executable sections) where there's no data and no code, so you could just aquire write access to those memory regions and write your code into them.
They're often used in trainers to get around using dll injection, or as a way to circumvent inmodule checks, for example wow checks if a function passed to FrameScript_Register(or what it was called again) is in the scope of the wow module, to circumvent that you could write a jmp to your actual function into a code cave, or even the whole function, if you find one that's large enough.

[amadmonk]

Note that there are ways to implement function hooks which do not require modifying the code of the target at all, and as such are not susceptible to any currently-known scan by Warden. I'm not going to go into these (although they're not rocket science; a little system-level debugging should give you the ideas you need) since I'd like to keep my "touchless hooks" Warden free for as long as possible

So saying that a hook involves modifying the target function isn't correct. In fact, one time-tested way of hooking is import/export table modification, although this doesn't apply as much in WoW (except when developing counter-Warden strategies).

[Megamike55]

I feel that the original post is complete at this point. Thank you to everyone for your help and input. Hopefully this will help someone else,.. just simply making it helped me.

I am considering doing a few similar posts like: "Lets Learn Concepts: Object Manager" since there seems to be no post (edit: no good posts) on this forum with such a direct focus on basic concepts that could be directly linkable to all the newbie chinese people that have started posting questions like "How do I find my target"

What do you think?

[suicidity]

Only problem I see with it is that the Object Manager has been covered so much, in my opinion, that it would be ridiculous not to understand how it works.. But then again, it could be of some help to a newbie.

Maybe I just see it useless because I expect that someone have a certain level of knowledge about programming before working with anything related to WoW, a level of knowledge that would automatically give you an image of what is going on.

[namreeb]

I say if he wants to spend his time on that then more power to him. I'm not sure about the object manager but I know there are several things which are allegedly well documented here, but they are documented over such a span of time that much of the information is out of date (and self-contradictory, unclear, etc.).

[Megamike55]

I take back what I just said. I forgot about the following thread, someone just resurrected it a minute ago. http://www.mmowned.com/forums/world-...e-objects.html