This seems to be an issue that happens if you don't have "Automatically face target when using action" clicked on, based on further investigation.