Is Gaijin a slur?

My sister and I were in Japan not too long ago and we were at a small restaurant in Kanazawa. The place was completely empty but the menu looked nice so we went in. We tried ordering in Japanese but admittedly, we’re terrible and kept apologising as we stumbled through the conversation. At the end of our order our waiter went into the kitchen and we heard him say “Gaijin, Gaijin” to his coworker. It instantly made us feel bad.
Like “man, I have to deal with these foreigners 🙄”
I had some friends say he might not have meant it in a bad way but I don’t know, considering the previous context I wouldn’t be surprised if it was. What are your thoughts?

by Traditional-Suit5503