Baby Itch

Three of my friends from high school are now expecting, Leslie, Carrie and Maria.
Maria with her 1st child, Leslie with her second, and Carrie with her third.
Lately I have been getting the baby itch. (Anna is almost 29 months old).
Brian and I were blessed to have a boy and a girl. Several people ask us if we are done, being that we have one boy and one girl. I wonder why that means we should be done. Maybe they are trying to tell us, 3 is more than we can handle; or we aren't the worlds best parents.
I have stuck with the information that we are not sure if we are done having children. I enjoy my nights that I can sleep through with waking to feed a baby and the fact my children are old enough to listen to me (it doesn't mean they do, but they can).
How do you know when you are done having children?