Ok simple little trick, on the quest in Schnott's Landing, "Make Yourself Useful" just get on a flying mount and talk to the workers from the air, if they become hostile, the cannot target you and simple disappear, leaving you free and clear to move onto the next worker.