What part of Japanese grammar did you find hardest to grasp?

In my case, it took an embarrassing amount of time to understand the passive tense and distinguish it from the potential tense in a sentence.