Foot wear – sandals acceptable?

I've heard you're required to take off shoes in lots of places in Japan. I wouldn't want to be barefoot so should I avoid wearing sandals?