The problem here is that the underlying and unknown correlations in the sample (aka people) is that they are NOT independent. The larger sample sizes are a ...

The problem here is that the underlying and unknown correlations in the sample (aka people) is that they are NOT independent. The larger sample sizes are a ... sort-of "proxy" ... to overwhelm underlying latent correlations. The whole thing is actually a subtle sort-of generalization of the "Prosecutor's Fallacy". So skepticism with the small sample sizes is absolutely warranted, unless some strong evidence is shown indicating mechanism-based independence.