Learning the "Forbidden Language" that is Japanese!

Yes, for the past couple of years I've been desperately trying to learn this wonderful/horrible language. No matter how hard I study or how hard I try, I still can't even seem to hold a basic conversation.
Even if I use all the stuff I know, I can probably sit there and talk for a while, but when they respond to me, I can't understand 90% of what they're saying. So so frustrating. Even when they try to speak slowly and use simple words I still can't understand a lot.
I've been thinking about it a lot and I think I have found my problem(s).

The first and most prominent is that I simply don't know enough words. I've tested myself and actually spent a few weeks creating an extensive vocab list of every word I know or had ever learned in Japanese I could think of. I mainly used my textbooks and class notes for this since that is where I learned about 95% of the Japanese I know.

At best I know around 700 words.

According to a guide I found for the "Japanese Language Proficiency Test"
at least 800 words are needed for "simple conversations"
(which is about where I am now)
and at least 1,500 to be able to take part in daily conversation.
So basically take all the work and hours I've put in over the years and double it!
Taking a whole lot longer then I originally thought to be fluent.
Just how do some people know 30-40 languages or more?
Going on three is giving me a huge headache and I'm probably one of the most advanced students in my class!
Whats also frustrating is that according to just about everyone I talk to, Japanese or American. Once I know this "Tokyo standard" of Japanese, most Japanese people will "probably" understand me. But If I ever stray to far from Tokyo and lets say go to Hokkaido or Okinawa, my spoken Japanese will be virtually useless.

There are more problems of course as well, for instance. The Japanese seem to have this weird thing about other people learning their language. Especially in places farther away from Tokyo without soo many tourists and foreigners.
Speaking bad or broken Japanese will (once again from what both fluent Americans AND native Japanese people have told me themselves!)
hurt their ears and if they know English they will respond to you in English. No matter how bad their English is or how good your Japanese may really be. They simply just don't want to be put in that "awkward" situation.
Very hard idea for me to grasp being a Puerto Rican living in New York. I don't care if someone sounds like an American or not. If they making an attempt to try and learn and speak English, I will do my best to help them and be patient. Even if I know they speak Spanish and they know I speak Spanish. I won't just give up immediately and speak Spanish to them. Only if they ask or seem to really be in some type of trouble or emergency.

ahh, so what do you guys think?
Agree or Disagree with anything I've said?
Ever tried learning a foreign language and had a tough tough or easy time?