GetOpenFileName and CreateRemoteThread mixing [anecdote]
I'm not sure this warrants an entire new topic, but since it caused me a few painful hours of tearing my hair off, I thought if it could save someone else that pain, it ought to be worth it. Rather than being a question or concern from my part, this is more of an anecdote/find/discussion, I guess.
I thought a few words of how this came about might be appropriate, please feel free to skip if you're not interested.
Anyway, for the last few days I've been writing a WoW bot. And, having reversed, built the code and tested it, everything was working smoothly. My bot was running around killing stuff, looting, etc. The stuff simple bots do. The only thing left was adding a GUI and some threading. So, with pain-stricken memories that I'm sure anyone who's done GUI with Windows API and is not a guru of said API could relate to, I went to dig up old Mr. Petzold.
Having found the old man for reference I spent the next one or two hours coding the GUI which included a profile creator, loading/saving, process selecting, etc. Everything was progressing smoothly. The one absolute moment of profound dumbness was when I thought: "hey, these functions are obsolete, but at least I know how to use them, let's ignore the advice of the talented people as msdn.com and be a retard", and went ahead and used GetOpenFileName and SaveOpenFileName for open/save dialogs.
So, following the GUI adding my bot kept crashing WoW.exe. Not understanding anything I went ahead and debugged my client process, my DLL and WoW itself, and found that data flow was looking swimmingly. The one thing that caused the crash, though, was any of my CreateRemoteThread calls. Finding that out I first thought the threading might be broken at some point (it was the only thing new apart from the GUI, and I did not even consider the GUI crashing my Remote debugging), and thread by thread made the process single-threaded again, only to find out it was still crashing.
As a last resort, by process of elimination (read: removing line by line until it no longer crashes), I finally found the place where the CreateRemoteThread crashes were stemming from. To my surprise if I commented out GetOpenFileName and just used a pre-defined path for the loaded profile, it was no longer crashing. So, filled with doubt, I went ahead and read up on the charming IFileOpenDialog interface, implemented it and... got rid of the crashes, to my absolute surprise. Five hours wasted one might think, but at least I learned to take the "superseded by..." comment at msdn.com more seriously in the future.
Actual Problem (TL;DR)
For some, to me, incomprehensive reason calling GetOpenFileName on Windows 7 64 bit (maybe on other platforms too) prior to a CreateRemoteThread made the debugged process (WoW.exe in this case) go blow itself up. I still do not know why. Has anyone else had this problem? Or perhaps some really smart person knows why?
In closing all I can advice is, if CreateRemoteThread is blowing your process up, try removing GetOpenFileName and use the much less explosive IFileOpenDialog interface.
(P.S. forgot tagging my post, sorry)
Last edited by shikyo; 02-04-2012 at 12:57 PM.
Kynox's Sister's Pimp
Totally off topic, but this book may be more appropriate for learning about Win32 threading than Petzold's (though I haven't looked at Petzold's in a while so I'm not sure how comprehensive/accurate it is):
Windows® via C/C++, Fifth Edition
It has about 6 chapters on the various threading and asynch APIs in Windows. Covers the basics, as well as some of the more advanced topics.
Fyi, by using IFileOpenDialogue your project is restricted to running on Vista+. Personally I think XP should be left to die and developers shouldn't waste time supporting it, but you may disagree.
Last edited by Cypher; 02-03-2012 at 05:02 PM.