I occasionally play at teaching Helen the cardinal numbers one to five, not in any organised fashion but every so often when she seems alert and curious and there are no more obviously interesting things for her to play with — during long car trips, baths, when I'm tired of building block towers for her to knock down, and so forth. I don't expect a 1.5 year old to do arithmetic, but it seems to me that understanding "two" and "three" is not necessarily any harder than being able to distinguish ducks from other kinds of birds (which she is now not too bad at).
Some people suggest that children learn counting, and thus ordinal numbers, first, but it seems to me that, at least for small numbers (up to five, or perhaps seven?) cardinals should be much easier to learn.
For those to whom the terms are unfamiliar, cardinal numbers represent the number of items in a collection of objects, while ordinal numbers are used to put objects into a sequence or to count them. If you present three blocks to a child and say "three", you're using the cardinal number three. If you count blocks "one", "two", "three" as you take them out of a box, you're using ordinal numbers.
The reason cardinals seem more straightforward to me is that they are simple, independent associations: it's not necessary to understand "two" before understanding "three", and learning the one won't interfere with the other. If you count one, two, three as you (say) put blocks into a box, in contrast, all the number words are associated with the identical same action — putting a block into the box — and the "three" will only have the right context if the "two" was comprehended. Obviously this only holds for small cardinals, within the human ability to comprehend directly, possibly no more than five and I suspect at most seven. The only way to get to ten is by counting (or possibly by comprehension of two groups of five or something like that).
For more on this topic, maths teacher Christopher Danielson explores some of the ways the ordinal/cardinal distinction affects learning. In contrast to my thinking above, he suggests "it could be said that most children learn ordinal numbers first", but that may be because of parental obsession with "one, two, three, ..." (which like the obsession with "a, b, c, d, e, ..." isn't necessarily a good thing). There's other good stuff on his web site.
Addendum: courtesy of Claire Bowern, I learn that the term I'm looking for is subitizing, "the rapid, accurate, and confident judgments of number performed for small numbers of items". See "Do young children acquire number words through subitizing or counting?" (Cognitive Development): "As a whole, subitizing appears to be the developmental pathway for acquiring the meaning of the first few number words, since it allows the child to grasp the whole and the elements at the same time."