Verbally harassed by a Japanese grandpa

I (f17) was verbally harassed by a Japanese grandpa today on the streets. For some context, My mom’s side of the family is Japanese and lives in Tokyo. I’ve been visiting there every summer since I was born (Im living here for college atm).
Long story short, I was minding my own business as I was about to cross the street. When I was in the middle of the street was when the walk signed turned red. Any normal person (I assume) wouldn’t care. Keep in mind it was like a foot long street as it was a Japanese alleyway. The fact that there’s even a walk signal is funny considering how tiny the street is. Anyways, this grandpa looked at me with absolute disgust and starting shouting at me. I was so caught off guard and looked behind as he started yelling “how dare you cross the street when it turned red!”. He also called me a “hentai onna” which I think he said because I was wearing shorts (it’s 97 degrees 😭). I walked away ignoring him when he told me “to get out of x city”. Ironically my family has lived in this specific city in Tokyo before the grandpa was even born. I also have a Japanese passport whether he likes it or not 🤷‍♀️ The funny thing is, a few minutes later a Japanese woman jaywalked a busy street right after.
I needed to get my justice someway somehow so I flipped the dude off. I’ve heard of Japanese grandpas being mean, but this was such a new experience for me.

Edit: Whether the crosswalk was short or not, I should’ve been patient (I was trying to catch the bus)! Since most people I’ve seen in my neighborhood and my family members treat that specific crosswalk as if it’s just another sidewalk, I thought to do the same. I assumed it was fine since the light technically didn’t change until I was in the middle to end of the crosswalk. Learned my lesson!!