How do I correctly book tickets for the Fuji Excursion Limited Express train coming from the JR West region?

Fellow Japan travellers, It's my family's first trip to Japan and I need urgent help from someone with first hand experience… navigating that complicated train booking system.

We'll arrive in Kyoto and go to Fuji Kawaguchiko before ending the trip in Tokyo. We've figured out the way to Kawaguchiko (a Shinkansen to Mishima, then a bus). We'd like to take the Fuji Excursion Limited Express train to Shinjuku… but I can't figure out how / where to pick up the tickets and information about it is all over the place.

Since the train is operated by JR East, I understood that I can't pick up tickets in Kyoto (JR West) or Kawaguchiko itself (some private train operator). I also can't use the e-ticket option on eki-net.com and link the online ticket to an IC card, since our toddler won't have one… Have I overlooked any option here? Are you really only expected to take the Express train if you have been to Tokyo / the JR east are before?

The official documentation by JR East also only lists JR East stations as pickup point…

by mrmisterFi