Wait, you can read Kanji but cannot speak the language? How much Kanji can you read? There's thousands, and a few hundred required for basic communication, and 2000 (Last I checked) for high school proficiency. Did you perhaps mean hiragana/katakana?
Personally, I know about 30 Kanji and hiragana/katakana. I know maybe 100-200 vocabulary words, and so far have managed to wrap my head around a few particles like wa/ga, to/ya, ne/yo, and ni. Many of them still confuse me.
Or, maybe you learned Chinese? Many of the Kanji have different meanings in that respect, though. I... think. Not too too sure.