How do the INers go about talking "their" talk?

It is already 11 months since I've been in Germany but I'm still far from being fluent in communicating with the natives in German. Professors and students at the university speak English since it is an international program. My colleagues at work have been super-nice to me to use English (which often means getting outta the comfort zone for them). So I manage to get around without any hassles really but in hindsight it is not such a good thing. I feel it is just not 'OK' to not be well-acquainted with the native language. I feel like there is so much out there I'm missing out on. No doubt I'm so much better at it now than the day I arrived here and I'm getting better every single day but the process is just not fast enough. I hold B-1 certificate but trust me, attending language classes and getting through the written exams doesn't necessarily help you improve your fluency in speech. I have tried taking an initiative to talk to my colleagues in German. It goes well for first few minutes and then they'd say something which I totally don't get, mostly because of the lack of my vocabulary knowledge or sometimes because of their accent or the colloquial elements like phrases or idioms they introduce. On the other hand, interestingly enough I found communicating in German with the expats from other countries, say Americans, Brazilians or Brits in my German classes much easier. But again, there is a tendency to use English as soon as the classes were over.
I'm sure most of the fellow INers have had similar experiences in respective countries they live in. So I'd love to hear from you all about such experiences and how have you gone about coping with it...