There's currently nothing that increases (or decreases) mining/ harvesting speed. It should be completely static between all players/ levels/ gear. Sometimes there seems to be a bit of a hiccup in the connection that delays me, but not often. I play with a controller and tend to just tap 'X' so I'm always mining as soon as possible. I don't know what might make others appear faster or what might be slowing you down, but neither should happen.