Buying a secondary residence (besso) as your first property whilst renting in Tokyo – Smart thinking outside the box or bad move for middle class?

I (M30) have long given up on the pipe dream of owning nice big house with garage and garden for my family (wife+newborn) in the middle of Tokyo.

Instead, is buying/building a house for half the priceand triple land size, outside of the city as a seconday residence, whilst keeping a primary rented residence within 23区 a big brain move or am I completely misjudging the situation?

For a long time I've been talking with friends and colleagues about the idea of some day owning a house outside Tokyo City (Hachioji Outskirts, Chiba, Tsukuba), when "the timing is right". Ofcourse the time is never right and waiting until retirement means my kids will have long have become independent adults we will have completely missed the opportunity to spend time as a family together living "the life" we're all too familiar with from tv and (perhaps western) idealism: The big house with dogs, kids, a garage, a garden to do whatever you want, play sports, build a mini Sauna etc.

I'm well aware that this is quite a luxury in Japan and not achievable for most people.

Or is it?

Why could I not buy 200-300 squarmeters of land in a rural area for 10M Yen, build a decently sized house (80-100m²⁾ for 30-40M Yen, enough for spending weekends, vacation and the occasional remote work, whilst keeping a 2-3LDK apartment in 23区?

My income is 13M, when my wife is working (currently parental leave) our househould income would top 20M (before tax). Holding down a 10-15man Loan + 20Man rented Apartment per month is doable with this income, as Im currently have around 30-40man to freely invest each month.

Naturally, my investments and portfolio would take a hit, but the goal would be one of intrinsic values anyway.

I don't know any young family who is doing this, am I overseeing something big here?

Couple points that I am aware of:

• Mortgages for non primary homes are more expensive and dont count as 住宅ローン and can thus not be deducted from taxable income.
• Down payment might be higher
• Getting a 住宅ローン might be more difficult in the future or impossible whilst having the another loan.

Couple question that most people might ask:

Why not just move outside of Tokyo completely?
-> Work

You're, on a average, only spending 2 days a week in this house, why would you spend all that money for only "partial usage"?!?
-> Yes, potentially two days every week for the next 20 years I can relax, enjoy closer nature, play with my kids and a more spacious environment, do DIY projects, drill holes wherever I want etc.

Land/House prices so far outside might not appreciate or loose value!
-> As long as the value doesnt suddenly go to zero in the next decades, I would be totally fine with even slight depreciation, the value I'm getting comes from somewhere else.

I am mostly worried that I am ignorant of some risks, that could drastically impact my financial life in Tokyo for my family, like suddenly spiking maintenance costs, taxes, or other problems with leaving a home "alone" for most the time.

One point Id like to finish on is that I feel like browsing for land/houses is so much relaxed further out. especially as a second home. 25+ min from a train station!? Don't care, we will always come by car and dont require proximity as convenience. So many deal breakers for primary residences just don't apply when considering a secondary residence.

Any thoughts on this would be much appreciated, I'd really love to hear you opinions and don't mind being told off, Id rather get some fair criticism than regretting (semi)-ruining my family financially.