Want to upgrade my wardrobe from Uniqlo and H&M to a bit more elegant – where do I start?

I (M35) am a notorious customer of Uniqlo and H&M and am at a point in my life where I would like to upgrade my wardrobe a bit. I don't place much value on brands as a status symbol, so while I don't mind paying a bit more for some nice clothes, I also don't want to go overboard just for the sake of showing off money.

I don't really know where to start. Does anybody have some recommendations for good shops for men's clothes in Tokyo? For now, I'm open to all styles to just see what's out there and what I might like.

Edit: Wow, this already got a lot more responses than I anticipated. Lots of helpful advice & recommendations! Since some people talked about it: I don't have a specific style in mind yet and was hoping to also get some inspiration through this post, which I definitely got. Will read through it all, do some research and check out some shops. Thank you!

by gachigachi_