I would argue that more retainers, definitely cross the line, since it not only boils down to more inventory space but also into more ventures per hour.
That being said: it's a minor thing, as long as it stays that way I can ignore it.
I played for 10 years, TBC to Legion (Legion only extremely casually, I was raid active from TBC till end of WOD).
Your question is irrelevant because I am talking systems level. WoW, even the old wow before they eliminated the 400ms position polling, runs circles around FF-XIV in terms of responsiveness to player input.
Also, I am not talking about extreme cases like Eureka. Lag is expected there and you will have lag in WoW too if you put that many people on say... OOOooondasta (my fav world boss

).
I am talking about 8 souls vs a savage raid boss.