If I'm not mistaken, you should be able to just look for the primary colours first and determine the gem from that.
The only gems with the colour green are all dexterity and the only with blue are intelligence. It's only red that is being used on other gems (and only on Vaal gems)
So perhaps something like:
look for colour green -> If present = dex-gem
look for colour blue -> if present = int-gem
else = str-gem
Don't know if that is the fastest way to do it, but it should be fairly easy and fast, since you should not have to change much from the pixel-searching-code..
fiw7
ps.. thank you for all the work with the sorter so far