Building a Part-Time Life in Japan While Working as a U.S.-Based Physician

My wife and I just returned from our fourth trip to Japan, and we’ve been thinking seriously about whether we could make a part-time life there work. With the ultimate goal of scaling to full-time life in the distant future. Specifically, I’m curious about the logistics of living in Japan for part of the year while continuing to work a locums gig (or permanent contract) as a physician in the U.S. to support that lifestyle.

Some key questions I’m hoping to learn more about:

1. Visas/Residency – For someone who doesn’t plan to practice medicine in Japan but wants to live there part of the year, what visa options are realistic? I know Long Term tourist stays have strict limits, so I’m curious if anyone has navigated other paths that make part-time living sustainable. Or does it make the most sense to stay just under the 90 day limit on a Tourist Visa a couple of times per year? Will doing that annually become an issue with immigration?
2. Locums + Japan Model – Has anyone here structured their work in the U.S. (locums or otherwise) around spending several months a year in Japan? I’d love to hear how you managed the financial and tax side of it; understanding that each person’s situation is unique and we’ll need to consult a tax specialist.
3. Practicalities – If you live in Japan part-time, what have been the biggest challenges compared to visiting as a tourist? Things like banking, and health insurance at a day-to-day level. Re: language, I’ve tested at N5 and feel prepared to test at N4. Re: housing, we're planing to buy a home. I understand that without permanent residency, we likely wouldn’t be able to get a loan, so are prepared to pay cash for a property.
4. Resources – Are there government resources that are especially helpful for Americans trying to set up this kind of arrangement?

I’d be really grateful for any insights or experiences you all can share. Even anecdotes about what worked (or didn’t) would be helpful. Even if you aren’t a physician, but live part-time in Japan, I would still love to hear your experience!