Using your phone on the rush hour trains

So this topic came up at work recently. I work at a company with mostly Japanese people, and I was listening to some co workers talk about their morning commute. One of the things they were complaining about was that they really hate when people are on their phones on the morning train. This made me a bit self conscious, as I use my phone every morning on the train (along with many others). Is this something that's typically seen as rude?

For background, I've been living here for about 4 years, and I'm decently tall so I'm typically above everyone else but I don't rest my arms on anyone while I'm doing it. I'm also a guy, so I have a backpack that I wear on my chest during the rush hour trains like everyone else. I thought that since that space is already there, there's not really a difference if I'm on my phone since I am usually kind of hugging my backpack with one arm anyways. Thoughts?

Update: Just to clarify, they specifically meant just using your phone out in front of you on the rush hour train. Not talking or whatever, just having it out and reading the news or reddit or whatever on the train. Not talking about any kind of particularly offensive content here, just the fact that the phone is out and you're holding it in front of you. Not even about the brightness or anything, just the space it takes to hold your phone.

by Regular-Slip-889