As you can see by the discussion taking place, there are many facets involved in the current design of not being able to invite while engaged. We checked in with the dev. team and were informed that because we have not designed content centered around the need to swap party members, we do not plan on changing this. However, depending on the content we add in the future, there is a possibility that we may change our policy or implement specific exceptions.
There will always be "what about [xyz] instance!" where being unable to invite is a hindrance, but overall the system was designed for specific purposes. We do appreciate the comments and feedback presented, and I'll be sure to pass these along as well.