Thats not temporarily though, its conditionaly broken (as the issue is always there). In most cases it works well because the longest wait time is generaly a good indication (as the longest waiting person is often prioritized, its unlikely to exceed that value by a lot), but if there isnt anyone, it shouldnt use that measurement anymore since its plain incorrect. In such case it should either resolve to the last time the duty was started, or indicate it simply cannot estimate (forming party is normaly my indication for that, as otherwise you would get a waiting position).
I think it should in these cases resolve to the previous run and check how long the queue was there. For day/night it might still vary a lot. But if it then already states 2h, if it ends up being 3h it at least makes sense (and the inverse then also works). Even if the queue isnt full, if it states it takes generaly 2h to fill, either i will be willing to wait (not), or i will wait until its closer the estimated time it normaly would start, and then queue in the hopes for more people. But yeah, this very issue is the reason why im not going to queue for any alliance raid outside of peak hours (which then can still easily take 30-60minutes of queue, while it states 5min average...).
If they realy want to take it further, they could make it so you can double queue until it reaches a remaining estimated wait time of for example 30mins (normal dungeons generaly wont take this long). It then will at least be able to indicate a better number of people that are waiting, and since there is a 30min treshold, if the duty would start, it doesnt matter as much that its 15min delayed because someone was already in a duty. 15mins in an already 2h long wait is only a small amount. And this would at least promote more people to queue, which is helpful in general.
Queues are one of the biggest problems in MMO's, so any frustration removed here would be a major QoL improvement.