What’s the origin of 四の五の言う?

I’m already familiar with its meaning but I’m curious as to why theres numbers incorporated in the expression —where does it originate from?

Manga is ながされて藍蘭島 btw