4 ms·
1. GPT out of the box was pretty biased (e.g. gender distribution). We fine-tuned on representative survey data to ameliorate this bias so we get Census-level e
by timshell 3y ago
1. GPT out of the box was pretty biased (e.g. gender distribution). We fine-tuned on representative survey data to ameliorate this bias so we get Census-level estimates for conditions such as gender [a] and work status [b].
2. We add the transparency features (click on 'Investigate Results') that shows how in vs. out-of-distribution the target question is. For out-of-distribution, we suggest people run traditional surveys.
More broadly, I think your point is really interesting when it comes to qualitative data. That is one reason we haven't generated qualitative survey data, but a lot of potential customers have already started to ask for it.
----
[a] https://roundtable.ai/sandbox/baa3d5f25236b91f1608c9f606b315ac8e8c2532 https://roundtable.ai/sandbox/baa3d5f25236b91f1608c9f606b315...
[b] https://roundtable.ai/sandbox/7a9ee27872eb29087be2386ccd19f7ec1a5ad5fe https://roundtable.ai/sandbox/7a9ee27872eb29087be2386ccd19f7...
- Gerardo1 3y agoHow can you be reasonably sure that that work sufficiently addresses the bias? What metric(s) are you using to measure bias in general, and what do those metric(s) look like before and after your tuning?
- timshell 3y agoTo respond to Edits - that's a great example, thank you. One of the limitations of surveys more broadly is you're asking for people's opinions, which of course does not correspond to reality. So, what we're simulating is how we estimate a representative U.S. population to answer the question "Which race is most likely to commit a crime?" as opposed to what the actual answer is. We definitely need to think how to handle your question so that it's clear where survey data converges/diverges with reality.