Those are some good questions. I'll try to answer them the best I can, however, they are very complex problems.
I was wondering in the shaiya bots, why are some things read directly from memory and some read via pixel colors on the screen? For example targets health, were you unable to read it reliably directly from memory?
Yes, it is sometimes difficult to find a pointer-chain (for lack of better word) that will always for everybody reliably. It's relatively easy to find a pointer to a monster's HP, but it can be difficult to find a pointer to the targeted monster's HP. When you do find it, you restart the client and check to make sure it still works. Often, it does not. This means you may need a pointer to a pointer to a pointer to a pointer to an address (or something to that effect). It can be VERY time consuming, and just not worth the effort. Reading from the HDC (Handle to Device Context) is not exactly the best method, but it's something that can be fallen back upon in certain situations, if all else fails.
I primarily use autoit script for automation and have just started working on shaiya, but I'm curious as to what advantages lua holds?
Lua holds many things that make it a preferable scripting language. It is simple, has loose types (with the ability to provide strict type checking, if need be), is embeddable, expandable, and very, very fast. It can be compiled to a byte-code*, or it can be run straight from source. Lua can also be used as a stand-alone programming language, similar to Perl, and is *NOT* designed just for macroing (However, MicroMacro makes use of Lua to provide this, obviously); but rather has a set of functions and libraries that make it versatile. Lua also provides garbage collection. The main thing about MicroMacro is that it is lite; it packs a lot of power for it's size. It also provides a secure environment; including very secure encryption for network traffic, and a protected runtime environment to prevent any sort of crashing. And one of the recent additions is true multi-threading.
Auto-It, on the other hand, is not open-source, nor does it run as efficiently as Lua. It didn't seem as easy to use as Lua, either. That's not to say that it doesn't have it's advantages, though. I do like how it allows you to create a graphical user interface, and am currently planning a plugin to do just this in MicroMacro.
*Lua's byte-code may not be endian-compatible under all architectures.
So far the first thing I have been working on in my bot is setting x,y,z bounds for the botting area, so that if my character ever leaves this area he will attempt to return and if he can't the script will stop. From what I have read, when GMs think someone is botting they send them to a jail area to watch the characters actions. This should effectively stop the script if that is the case.
This should work. Usually, it's the kill stealing that gets you caught, so that should be your #1 concern. This, however, is always a welcome addition. It's probably safe to just check on two axis'. Typically, X and Z are what you would check, as Y is typically height. In some games, like World of Warcraft, they mixed it around; X and Y represent the horizontal plane, and Z is height. Anyways, you can just use the distance formula, and if you are beyond such a radius, you can rotate the camera towards the center (start point) and move towards it. If you are unable to reach that point (or are really really far away for some reason), stop the bot with stopPE().