The only effective way I have found to target on ps3 is to point the camera at the ground and then cycle through the mobs to target. That way you don't select something far away from you.