Mumble & Injection

[PyGuy]

I know how to inject a DLL using VirtualAllocEx and CreateRemoteThread. The trouble with this is that in order to hook EndScene, you need to know where the target application stores its pointer to the D3D9 device. While this isn't too big an issue for WoW, it can cause trouble in some other apps.

Mumble takes a different approach with its overlay. (Mumble is a open source competitor to Ventrilo.) It hooks Direct3DCreate9 and Direct3DCreate9Ex and then after the device is created, it hooks a number of vtable functions off the device, including reference counting and Present. Mumble can then use the Present hook to draw the user that's currently speaking, who's in channel, etc. over top of any D3D9 application.

All the source is available for browsing here. I'd like to do something similar, but I'm having trouble figuring out how its actually injecting the mumble_ol.dll (which contains the overlay code) into other processes. I THOUGHT it was using a windows hook:

extern "C" __declspec(dllexport) void __cdecl RemoveHooks() {
	DWORD dwWaitResult = WaitForSingleObject(hHookMutex, 1000L);
	if (dwWaitResult == WAIT_OBJECT_0) {
		if (sd->bHooked) {
			if (hhookWnd) {
				UnhookWindowsHookEx(hhookWnd);
				hhookWnd = NULL;
			}
			sd->bHooked = false;
		}
		ReleaseMutex(hHookMutex);
	}
}

extern "C" __declspec(dllexport) void __cdecl InstallHooks() {
	DWORD dwWaitResult = WaitForSingleObject(hHookMutex, 1000L);
	if (dwWaitResult == WAIT_OBJECT_0) {
		if (! sd->bHooked) {
			GetModuleHandleEx(GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS | GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT, (char *) &InstallHooks, &hSelf);
			if (hSelf == NULL) {
				ods("Lib: Failed to find myself");
			} else {
				hhookWnd = SetWindowsHookEx(WH_CBT, CallWndProc, hSelf, 0);
				if (hhookWnd == NULL)
					ods("Lib: Failed to insert WNDProc hook");
			}

			sd->bHooked = true;
		}
		ReleaseMutex(hHookMutex);
	}
}

The trouble is that CallWndProc doesn't DO anything except continue the chain:

static LRESULT CALLBACK CallWndProc(int nCode, WPARAM wParam, LPARAM lParam) {
	return CallNextHookEx(hhookWnd, nCode, wParam, lParam);
}

I've confirmed in other files that InstallHooks() and RemoveHooks() are called at startup and shutdown, but I can't find any place where this would result in code being run in another address space. I expected that CallWndProc would look for a HCBT_CREATEWND event and then use GetWindowThreadProcessId on the window handle to get the process ID and perform normal injection stuff from there, but it doesn't do that.

I did find the following function that is called from the main setup routine:

OverlayPrivateWin::OverlayPrivateWin(QObject *p) : OverlayPrivate(p) {
	QString path=QString::fromLatin1("%1/mumble_ol.dll").arg(qApp->applicationDirPath());

	qlOverlay = new QLibrary(this);
	hpInstall = NULL;
	hpRemove = NULL;

	qlOverlay->setFileName(path);
	if (! qlOverlay->load()) {
		QMessageBox::critical(NULL, QLatin1String("Mumble"), tr("Failed to load overlay library. This means either that:\n"
		                      "- the library (mumble_ol.dll) wasn't found in the directory you ran Mumble from\n"
		                      "- you're on an OS earlier than WinXP SP2"), QMessageBox::Ok, QMessageBox::NoButton);
		qWarning("Overlay failure");
		return;
	}

	GetOverlayMagicVersionProc gompvp = (GetOverlayMagicVersionProc)qlOverlay->resolve("GetOverlayMagicVersion");
	if (! gompvp)
		return;

	if (gompvp() != OVERLAY_MAGIC_NUMBER)
		return;

	hpInstall = (HooksProc)qlOverlay->resolve("InstallHooks");
	hpRemove = (HooksProc)qlOverlay->resolve("RemoveHooks");
	PrepProc pp = (PrepProc) qlOverlay->resolve("PrepareD3D9");
	PrepDXGIProc pdxgi = (PrepDXGIProc) qlOverlay->resolve("PrepareDXGI");

	if (pp)
		pp();

	if (pdxgi)
		pdxgi();
}

To me, it looks like its loading the mumble_ol.dll into the main mumble's address space and then installing the various hooks there. PrepareD3D9 is the function that installs hooks for Direct3DCreate9 and Direct3DCreate9Ex.

The only thing I can think of is that installing the Direct3DCreate9 and Direct3DCreate9Ex hooks against the D3D9.dll in its own application, this allows the hooks to be executed in other processes as well, but that really doesn't make sense to me. Does anyone have any insights here?

[PyGuy]

I found the answer... the action of installing a global hook via SetWindowsHookEx causes every application to load the DLL where SetWindowsHookEx is called from. The mumble_ol.dll has a DllMain function which will get called on load. This, in turn, hooks LoadLibrary and LoadLibraryA to determine when D3D9.dll is loaded and then hooks Direct3DCreate9 / Direct3DCreate9Ex.