As documented, the M2 format supports a very limited number of effects that are inherent to it.
For the purpose of this document, I'll assume you have a basic notion of what I'm talking about, but to sum it up: Each object can have multiple textures. These textures can then be applied using some special flags... besides the standard model, it can emit light using a self-lit (which is called unlit, probably because the object ignores the environment light), unfogged, two-sided and with disabled z-buffer, and can be blended as, for the sake of this document, opaque, additive (add) and subtractive (fade) modes. The texture can also be applied as an environment map.
More information on these can be found here - M2/WotLK - WoW.Dev Wiki
WoW engine doesn't support, as far as I'm aware, any external shaders. I'm not even sure the game uses a real light so I suspect even if you were to, say, add normal maps AND normal-map code to the models, they wouldn't work. This means that any effect has to be done with the limited combination of m2 effects.
Now, I personally don't believe this is bad. First, I'm a firm believer in "art from adversity". Too much fanciness and things start to get out of control. When you have a limited pool of resources, you'll have to do your best with it, which is an art per se. Second, it fits WoW art style perfectly. You don't need much more, although you could do it with some (Starcraft 2 for example has a much better engine and uses a very similar style for the texture paintings).
In any case, the purpose of this document is to show you some effects that can be achieved with the texture modes present in the game. I call these materials and effects Faux Shaders, since they're basically simulations of something the engine can't really do.
Self-Illumination
Diffuse - Texture 1, Default, Opaque, Normal
Luminance Map - Texture 2, Unlit, Add, Normal
This is the most basic effect you can achieve. Some models ignore a self-illumination map to conserve resources (Sulfuras has only one map, set to Unlit), but that probably hasn't been the case lately. I abused this effect for my Magmadeep model - The handle and the blade have three luminance maps altogether (one for the static magma, two for the animation). The skull also shares the static map, so that the eyes are always glowing.
You'll need to be careful if you're going with two maps. Remember that the information will overlap. So unless you want to blow over, you'll need to be conservative, preferably using a black color in the diffuse map where it's going to be illuminated by the second map.
Specular
Diffuse - Texture 1, Default, Opaque, Normal
Specular - Texture 2, Default, Add, Normal
This is an effect that consist of two channels - an opaque lit channel, and another lit channel above, set to Add. Because both channels are lit (do not have the "Unlit" flag turned on), they're essentially different from a self-illumination map.
The difference between this method and painting the effect straight in the main texture is that you can get a really strong reflection (hence #FFFFFF) without causing any loss of information in the texture itself - that is, in dark areas, you'll be able to see the details in the texture that wouldn't be possible otherwise if you had painted white in it. You also obviously don't lose as much saturation information, although you will lose some unless you color the secondary map (since any color gaining grayscale additive information is also losing saturation...)
Because your effective texture has now effectively 27-bits of information instead of 24 (Unfortunately, you cannot multiply the information, only add... furthermore you get a ceiling of 512 variations to your color information, or 9 bits per channel (2^9). Not effective considering we're using two maps, but that's the best we can do), you can break the limit of light defined by the ambient light, which is how the effects are achieved. For example, the specular reflection of the sun in the car is orders of magnitude stronger than the actual light being reflected indirectely everywhere else by the sun. By going above the 8-bits border we can add this kind of information.
An effect similar to specular is achieved here by controlling the area of influence in the secondary map (which should be low-key, grayscale), which would in-game raise the pixels above a certain light threshold to #FFFFFF (white). The second map needs to be really dark and black almost everywhere, except in the places you want to show visual cue that a specular reflection is happening. This is NOT like a standard specular map! Although some aspects remain, it's still a FAKE effect and you'll need to actually paint the places you want it to happen instead of just painting how strong the effect is. Of course, you can (and should) do the later too.
The downside is that the WoW shader is Lambertian in essence and the only non-lambertian modes are environment maps or unlit. This means we don't have what truly makes a specular reflection a... reflection, relevant information from the viewing angle. That means you'll need to control very tightly the area of influence in the specular map. This is why it's "faux", you're still not really creating a real reflection. Metallic objects have a very tight specular reflection, which means you'll have to choose in your object the area of influence, and that "painted" area of influence will be the only place the object will be really reflecting the light. The rest of the object will have to be dark, else you're effectively making a translucent object. Even then, be very careful not to leave many "slightly lit" areas. If you do that, the model will look a lot like Tin Foil, although if that's your intention, go for it.
Some metallic objects feature strong colored reflections. If that's the case, be sure to saturate the specular map.
Also I'm not much a fan of Bloom, but turning Bloom on makes this effect really, really good.
Rough Metal Material
Diffuse - Texture 1, Default, Opaque, Normal
Rough Metal Shader- Texture 2, Default, Fade, Environment
This is one of my favorite effects and the one that motivated me to start messing with this. A characteristic of unpolished metal is that they often feature high-contrast specular, with low-key dark areas.
As such, you set up this shader by making an environment mat that is dark on the borders and white in the center.
In order for this effect to look cool though, you'll have to mix it with a specular effect. So consider metal to be one of the most resource-intensive materials, but the final result is worth it.
When setting up Faux Shaders, if you want to add an additional specular effect, be warned that there will be information loss if you set it between Diffuse and Rough Metal. The reason is because the "fade" blending mode (which is a subtraction...) will not consider any light above 255, and since double diffuse effects have a 9-bit range, some information will become white, then get subtracted, which means the original information will be gone and you'll get an ugly gray.
To fix this, you will want to put any specular effect after the Metal, and adjust it accordingly.
Fresnel Glow
Diffuse - Texture 1, Default, Opaque, Normal
Glow - Texture 2, Default, Add, Environment
When I first got the first environment map I immediately thought of Fresnel effects. I originally conceived that they could be done by doubling the object and setting the top layer as a transparent, additive object... I'm 99% sure I've already seen some like this in WoW.
However, Using environment maps make it much, much easier. Simply make an environment map that is entirely black with a strong white border, and you have instant glow.
Translucence
Diffuse - Texture 1, Default, Opaque, Normal
Translucence - Texture 2, Default, Add, Environment
This Faux translucence is actually one of my favorite effects, but you need to be extra careful. It's basically a modified fresnel effect, except you play with the colors and make a much softer transition from bright to dark.
In this example, I more or less turned the skull into a flesh lump with blood inside, resulting in a similar effect to sub-surface scattering from our body (specially in the ear, fingers, etc), or a candle wax.
The REAL issue arises from the fact apparently subtraction (fade) isn't applied the same way addition is. Addition takes in consideration lighting if the object doesn't have unlit, however fade doesn't (according to my own tests at least). This means that the Translucent effect will essentially be a self-lumination map.
The reason this is an issue is because the way to "fix" this situation would be to make a third map fully white and add it as Fade after Translucence. This would re-apply the standard shading back, however that doesn't work. This is also the reason the fresnel glow can't be used in any way other than a glow. Because of this severe limitation, I recommend using translucence for static objects, say, a candle.
The closest I can get to it is by changing that map from white to a bright gray. However, you still lose information, specially regarding to saturation.
Maybe playing with two geosets and alpha channels could fix this, but I have yet to test. However, if we find a way to subtract the lighting information instead of adding, I can say this would open doors to an array of effects that would be beyond everything we have right now, including Oren-Nayar surfaces.
Edit: Got it! Here we go:
Additive Ambient-Lit Environment
Ambient - Texture 1, Default, Opaque, Normal
Environment Map - Texture 2, Default, Fade, Environment
Diffuse Map - Texture 3, Default, Add, Normal
The solution was to make a similar system to the Rough Metal Material, except the first layer is actually an ambient-light "catcher" - a completely white texture.
Then you add the environment map as usual. Remember that although the environment map will be set to Fade (subtraction), you still treat it as if it's an additive map. The result will be an environment map that is affected by ambient light.
You can then add the diffuse map in a third layer and start playing with it.
How can this effect be used?
Basically, this effect allows you to create a faux version of any surface that uses a non-uniform light distribution model. This includes:
> Fresnel reflections
> Glow effects that are influenced by light.
> Surfaces such as velvet and other cloth.
> Other rough-surface models such as Oren-Nayar
I'll make a new custom model in the next few days showing a bit of these effects.
Conclusion:
These are the effects I've made so far. I'll keep the thread updated with more as I test. Discussion is encouraged.