Hmm odd I get latency issues like you descride when using a controller on wireless but wired is fine.

Are you using any software for the controller?