5 ms·

The article didn’t share the survey method and the sample size is 1000. I couldn’t take it seriously

by zocoi 6y ago

dhosek 6y ago
A sample size of 1000 is huge. https://www.scientificamerican.com/article/howcan-a-poll-of-only-100/ https://www.scientificamerican.com/article/howcan-a-poll-of-...
- 60secz 6y ago
  If random
- whatshisface 6y ago
  It could be huge or tiny depending on the size of the mean and standard deviation of the effect you're looking for in comparison to the mean and sdev of unrelated variations.
  - notafraudster 6y ago
    Given that the QOI is a single multinomial proportion, an unweighted[1] random sample of the population has a maximum[2] ~3.1% margin of error on either side. You could argue the QOI is something else (the sum or difference, or even the ratio estimator reported in the title). Margins of error on ratio estimators are quite a bit larger because of numerical properties of ratios. Since there is interest, I ran a quick simulation[3] to derive the CI of the ratio estimate above ("5x"). By Monte Carlo and using standard assumptions, the confidence interval on the "5X" quantity of interest is 4.51 - 6.82x calculated via the quantile method.[4] The confidence interval is not appreciably smaller in magnitude if you double the sample size, we're well past the point of diminishing returns. [1] Weighting would induce a design effect, but here I would suggest that the real problem is a poorly defined population and sample frame, not sample error. This is the real issue here: we have zero reason to believe these 1,000 people are a random sample of any interesting population. [2] For a parameter \theta = 0.5 -- uncertainly decreases as the statistic becomes further from 0.5. But classical MOEs overreport like this. [3] R Code on Pastebin: https://pastebin.com/raw/UjHg7qnR https://pastebin.com/raw/UjHg7qnR [4] Similar results come from taking the numerical sd of the sampling distribution and calculating a NACI about the mean, I just did the quantile trick because it's less code.
mhh__ 6y ago
Most election polls (in the UK at least) aren't much bigger and can be very very accurate. Usual caveats apply obviously.

The article didn’t share the survey method and the sample size is 1000. I couldn’t take it seriously

by zocoi 6y ago

The article didn’t share the survey method and the sample size is 1000. I couldn’t take it seriously

A sample size of 1000 is huge. https://www.scientificamerican.com/article/howcan-a-poll-of-only-100/ https://www.scientificamerican.com/article/howcan-a-poll-of-...

60secz 6y ago
If random
whatshisface 6y ago
It could be huge or tiny depending on the size of the mean and standard deviation of the effect you're looking for in comparison to the mean and sdev of unrelated variations.
- notafraudster 6y ago
  Given that the QOI is a single multinomial proportion, an unweighted[1] random sample of the population has a maximum[2] ~3.1% margin of error on either side. You could argue the QOI is something else (the sum or difference, or even the ratio estimator reported in the title). Margins of error on ratio estimators are quite a bit larger because of numerical properties of ratios. Since there is interest, I ran a quick simulation[3] to derive the CI of the ratio estimate above ("5x"). By Monte Carlo and using standard assumptions, the confidence interval on the "5X" quantity of interest is 4.51 - 6.82x calculated via the quantile method.[4] The confidence interval is not appreciably smaller in magnitude if you double the sample size, we're well past the point of diminishing returns. [1] Weighting would induce a design effect, but here I would suggest that the real problem is a poorly defined population and sample frame, not sample error. This is the real issue here: we have zero reason to believe these 1,000 people are a random sample of any interesting population. [2] For a parameter \theta = 0.5 -- uncertainly decreases as the statistic becomes further from 0.5. But classical MOEs overreport like this. [3] R Code on Pastebin: https://pastebin.com/raw/UjHg7qnR https://pastebin.com/raw/UjHg7qnR [4] Similar results come from taking the numerical sd of the sampling distribution and calculating a NACI about the mean, I just did the quantile trick because it's less code.

mhh__ 6y ago

Most election polls (in the UK at least) aren't much bigger and can be very very accurate. Usual caveats apply obviously.