How fast would you expect your Japanese to improve if you came to Japan with a N2?

Thinking of moving to Japan next year and hoping to have N2 by then as well. If I did my best to avoid English as much as possible did my best to only speak Japanese, how fast would/should my Japanese improve? Mostly thinking conversational wise but other aspects too.