Enemy behaviour in my current (as yet unnamed) game hasn't come on very far - just a single enemy with a traditional hard-coded finite state machine (done the old-school way, with a big switch statement). Which was initially fine as it let me get some of the more important low-level details into place, but now I'm looking at adding more enemies it's not looking so hot so something more elaborate is called for.
Ai Game Dev has been in my bookmarks for a while now, and provides a lot of interesting reading. The approach F.E.A.R. takes towards it's AI is particularly interesting and probably something which would work well, but is a little beyond me at the moment. Instead what's caught my eye is behaviour trees. In particular it seems to solve a problem that I've been having - how to write specific modules of behaviour (like a specific enemy attack) in a way that they can be reused and rearranged rather than having an explicit "next" behaviour.
I'm not sure I entirely understand how it's all going to fit together with some of the higher level gameplay interactions, but it's a promising direction. I think I shall leave my current FSM enemy as it is and code up the next enemy as a behaviour tree (or possibly do the same one again) and see what the resulting code is like.
Since I havn't really done a scrolling beat-em-up before, I'm expecting to take a few wrong turns with the AI before I find something that works. If anyone has any experience to share then feel free to leave a comment.