But size does matter.
Yes, in matters of pitch and tone the size of the vocal folds and the vocal tract do indeed matter. This is possibly the greatest factor in determining the vocal range, timbre, tessitura (area of greatest comfort) and possibly the location of transition points. But why does this happen?
Perhaps the easiest way of describing the whys and hows is to look at the mechanics of musical instruments: particularly stringed instruments. If one were to compare the lowest string of a double bass which is pitched at standard concert pitch E1 to the top string of a violin which is pitched at E5, visually one sees a difference in length and thickness of the string, with the bass string being more than 3 times longer than a violin’s string.
When plucked, or other wise played, the bass string vibrates at approximately 41 times per second, which then creates the sound of an E1 to our ears(and brain). In contrast, the smaller thinner violin string vibrates at 659 per second and creates the tone of an E5 with in our brain. Other easy comparisons within the musical instrument world is comparing the size of the soprano instruments such as a trumpet to its bass counterpart such as a tuba.
Similarly, the size of a person can determine the size of their vocal folds, and therefore their voice characteristics. Just as people have arms and legs which are longer or shorter than other humans, the size of the vocal folds (also known as vocal cords) varies. A person who is approximately 5 feet tall will generally speaking have a small set of vocal folds than a person who is 6 feet tall. Smaller vocal folds generally vibrate at a faster rate than larger vocal folds.
Here is where humans get interesting. Unlike a musical instrument, humans have more variations than just size. Males have disproportionately larger vocal folds and laryngeal cartilages than females. This difference is a whopping 60% of “effective vocal fold length” (http://www.ncvs.org/ncvs/tutorials/voiceprod/tutorial/influence.html). This means that if we take a human female of five feet tall and a human male of the exact same height, and assume a vocal fold length of 1 centimeter for the female, we can expect the male to have a vocal fold length of 1.6 centimeters. A rather significant difference. In terms of speaking we can expect a female to have an average* range of 180-224 hertz or vibrations per second with a speaking range of 145-275 hertz. Looking on a piano this means the average speaking pitch for females is somewhere between F3(F below middle C) to A3, with excursions from as low as D3 to as high as C4# (C# just above middle C). The corresponding averages for males are as follows: an average speaking pitch of 107 -132 vibrations per second, which roughly translates to A2 to C3, with excursions in conversations ranging from as low as 80(E2) to 165 (E3). (www.shelaghdavies.com)
Size of your vocal folds do matter in what pitch will be produced by the person. This difference can be seen and heard not only by noticing that larger people do tend to have lower pitched voices than smaller people, but also in the significant difference in the size of one's vocal folds depending on sex.
*As always, studies focus on averages - averages mean that 50% of subjects are below the average and 50% are above the average. So yes, there are many people I have met who speak above or below these averages. Anecdotal evidence will often demonstrate both sides of the scientifically determined average numbers.