This is less of a Japan issue and more of a psychology issue, but I post here because people in other subreddits are less likely to share a similar environment.
I haven't had much experience with Japanese people themselves, only now transitioning into a working environment, but even the rare interactions with Japanese people have been rather uncomfortable. Most my social interactions so far have been with foreigners in Japanese language classes. I come from a country barely anybody knows about, it's a good and a bad thing, but not the primary reason for my lack of success. I just don't expect anyone's gonna be interested in approaching me first, so I go out of my way to be friendly with everyone. My goal is to find people who can consistently be friends with me and hang out, have long conversations over any topic, though having shared interests would help a lot. I come up to people, trying to be friends. Some reject me immediately, some are open to a conversation, but get tired by the end of it. Some people I have hung out with many times after classes, but at some point they decide to actively avoid me for reason I could tell. Am I being aggressively friendly? If anybody actually told me so, I would probably believe it, but I don't have any alternative. I see people becoming friends quickly and hang out, I want that too, but if I don't approach anybody, nobody's gonna approach me. And those who do approach me, I always respond to. But then they stop. Something worth mentioning is that I'm super comfortable with Japanese, but the classmates are rarely so, they speak English/other languages, while I respond in Japanese. Though attending events with more Japanese people / Japanese-speaking people resulted in similar experiences.
So it does seem to be that my personality is unattractive. Nothing new, it has been the case since elementary school in my country. Throughout the years this lack of success has led me to develop hubris and other negative traits, but at least my previous social failures had made it clear to me what I should work on fixing. I did fix those issues. I'm always friendly, curious, encouraging, even with people I know aren't going to stay around. I'm proud of my progress, but since people still hate me wherever I go, I'm waiting for someone to confront me about it so that I could fix more.
So my life has become constant self-improvement. I figured out my plans for building a future in Japan. I no longer even need people to succeed in life. But it does lead to more and more miscommunications and delusions the more I succeed at work and fail at socializing. I don't know how to joke or be lighthearted. I know a lot of things and say them to keep up the conversation, but I also ask a lot of the things I don't know because it in theory helps establish trust gets people to open up.
Edit: to the people who say my Japanese isn't good.
What do you suggest? I speak fluent English, but it's not my preferred language. If people told me that the only way to make friends in school is to speak English, I would rather accept that I'm not gonna make friends here and move on. I can't prove to any of you how good my Japanese is. But I'm comfortable with it. My Japan experience has been closely tied with my Japanese usage. I like improving every day, which would be more difficult if I had to speak English to socialize. And it sounds like you are suggesting not to go to Japanese speaking events because my Japanese isn't good and hangout with English speaking people more. Sounds rather self-defeating, so I'll have to pass on that.
by poshamoeba