First time travel to Japan – early April or mid May?

Planning to celebrate a milestone birthday in Japan. Birthday is in May but thinking if its better to go in April to see cherry blossoms? We wanted to avoid crowds so May seemed perfect cause it would be less crowded and we would be in Japan on the big day. But as i am researching more, April seems like a better time.
We would avoid golden week rush if we go in mid May but would most likely miss the blossoms. Any thoughts? Temperature wise or costs of total travel? Please help!

Been at the portal to book tickets thrice but just cant make up my mind 🙁

by Libfittrip