Will Leaving Japan Shortly After Receiving Work Visa Raise Immigration Issues?

I'm moving to Japan for work in April/May and currently contracting for the company while waiting for my visa to be approved. Once my visa comes through, I'll transition to a full-time employee and physically move there.

My Japanese partner will be on a Working Holiday Visa in Australia, and since my job is fully remote (with company approval to work from anywhere in the world), I’d like to spend 3-4 months with them there before returning to Japan long-term.

My plan is:

Move to Japan for 1-2 months (register residence, set up bank etc.) after my visa is issued.
Travel to Australia for 3-4 months.
Return to Japan.

Would this be viewed negatively from an immigration perspective? Would it raise any red flags when I re-enter Japan?

Going straight to Aus before moving to Japan isn't an option as I need to collect company equipement etc. so I can work remotely.