6 ms·

I once had a statistics professor (teaching a programming class using, IIRC, SAS) in college who worked as a data analyst in the federal government, possibly th

by wahern 17d ago

I once had a statistics professor (teaching a programming class using, IIRC, SAS) in college who worked as a data analyst in the federal government, possibly the Labor Department. Part of his job was working with polling and form data. After class he once tried to explain a paper he wrote arguing for an additional null-like type in the formal analytics system in his domain. I was young and stupid (took it as an easy programming course; was not skilled at abstract math) and impatient (wanted to meet up with friends at a pub) so didn't follow along well enough to even properly remember his argument, let alone understand it. But I've always distinctly recalled that he was arguing for a 4th, or possibly 5th type, yet whenever I survey the literature I've only found at most such 2 types. But I've probably been looking in the wrong places. For context, NULL/missing was a type. N/A was a type (i.e. a question irrelevant to someone). I can't remember the 3rd, and of course not the additional (4th or 5th) type he was proposing. If anyone knows a good resource that describes this problem domain, that would be much appreciated. I've been haunted for over 20 years by the regret of not fully appreciating what he was trying to explain to me, both because I could've learned something that day, but also because I almost immediately felt like an a*hole for not showing enough interest in and respect for a piece of scholarly work he was clearly proud of and wanted to share with me. EDIT: I think the third pre-existing type was for a situation where someone abstains from answering a question like, "Sex: male/female", because they consider themselves neither. So something like an Other type. In retrospect this was presumably related to set theory, and he felt there was a gap in the pre-existing formal models.

Someone 17d ago
> For context, NULL/missing was a type. N/A was a type (i.e. a question irrelevant to someone). I can't remember the 3rd, and of course not the additional (4th or 5th) type he was proposing “Don’t know”: this has a value, but we do not know it could be one. Also, within N/A, one _could_ discriminate between “not available yet”, “will never be available”, “was available once, but was lost”, etc. Depending on the domain, others could be - “Can’t tell”: this has a value, but you are not allowed not know it (unlikely, as you likely also shouldn’t be allowed that the value exists) - incomputable: this has a value, but it isn’t possible to know it I think generic systems shouldn’t try to capture such domain specific things, but allow for implementing them. SQL shouldn’t even have null, but have enums and product types on top of which you could create it and functions supporting it.
- adzm 17d ago
  This reminds me of the sixteen or so different flavors of null used in HL7! https://terminology.hl7.org/en/CodeSystem-v3-NullFlavor.html https://terminology.hl7.org/en/CodeSystem-v3-NullFlavor.html though in practice I've only seen a handful of these. Notably: NI No Information - the default, it is unknown and why it is unknown is also unknown UNK Unknown - the most common other than NI ASKU Asked but Unknown NASK Not Asked - we don't know because we didn't ask MSK Masked - hidden due to privacy or legal etc NA Not Applicable NAV Not Available - for example patient might be unconscious or doctor might be unreachable, temporary
- 1718627440 17d ago
  How is “not available yet” different from "Don't know"? How is incomputable different from “will never be available”? > SQL shouldn’t even have null But it needs to, because it needs to represent half of a result record not existing, i.e. on a LEFT JOIN.
  - rmunn 17d ago
    "Not available yet", at least to me, carries the implication that it will, not just might, become available at some point. Probably soon enough that it's worth asking the question again later. Whereas "don't know" suggests that it's not worth asking the question again, because you're probably just going to get "don't know" as the answer. (At least, if "don't know" and "not available yet" are both presented as options).
    - tyrabound 17d ago
      A more accurate description is “May become available”, it might be possible that it could become available and we believe it could within a short order, however it is also not assured or guaranteed that it will become available when we think, and it may also never become available at all for any number of reasons as simple as someone not following up and adding newly ascertained information or that the only possible source of the information has become permanently unavailable for many other reasons of their own.
- 17d ago
rramadass 17d ago
[flagged]
mr_toad 17d ago
> But I've always distinctly recalled that he was arguing for a 4th, or possibly 5th type, yet whenever I survey the literature I've only found at most such 2 types. But I've probably been looking in the wrong places. It’s common for surveys to encode various null values. ‘Refused to answer’, ‘Did not know’, ‘Not applicable’, ‘Illegible response’ etc, although there is no standard way to do this. The SAS language allows you to use multiple custom missing values, but this is not widely known and I’ve never seen anyone do it.
pseudohadamard 16d ago
Oh that's easy. The 4th is "misplaced/lost", and the 5th is your kids pointing fingers at each other and saying "he/she did it, it wasn't me". Finally, the 6th is FILE_NOT_FOUND, coincidentally also the third boolean value after TRUE and FALSE.

by wahern 17d ago

Someone 17d ago

> For context, NULL/missing was a type. N/A was a type (i.e. a question irrelevant to someone). I can't remember the 3rd, and of course not the additional (4th or 5th) type he was proposing “Don’t know”: this has a value, but we do not know it could be one. Also, within N/A, one _could_ discriminate between “not available yet”, “will never be available”, “was available once, but was lost”, etc. Depending on the domain, others could be - “Can’t tell”: this has a value, but you are not allowed not know it (unlikely, as you likely also shouldn’t be allowed that the value exists) - incomputable: this has a value, but it isn’t possible to know it I think generic systems shouldn’t try to capture such domain specific things, but allow for implementing them. SQL shouldn’t even have null, but have enums and product types on top of which you could create it and functions supporting it.

adzm 17d ago
This reminds me of the sixteen or so different flavors of null used in HL7! https://terminology.hl7.org/en/CodeSystem-v3-NullFlavor.html https://terminology.hl7.org/en/CodeSystem-v3-NullFlavor.html though in practice I've only seen a handful of these. Notably: NI No Information - the default, it is unknown and why it is unknown is also unknown UNK Unknown - the most common other than NI ASKU Asked but Unknown NASK Not Asked - we don't know because we didn't ask MSK Masked - hidden due to privacy or legal etc NA Not Applicable NAV Not Available - for example patient might be unconscious or doctor might be unreachable, temporary
1718627440 17d ago
How is “not available yet” different from "Don't know"? How is incomputable different from “will never be available”? > SQL shouldn’t even have null But it needs to, because it needs to represent half of a result record not existing, i.e. on a LEFT JOIN.
- rmunn 17d ago
  "Not available yet", at least to me, carries the implication that it will, not just might, become available at some point. Probably soon enough that it's worth asking the question again later. Whereas "don't know" suggests that it's not worth asking the question again, because you're probably just going to get "don't know" as the answer. (At least, if "don't know" and "not available yet" are both presented as options).
  - tyrabound 17d ago
    A more accurate description is “May become available”, it might be possible that it could become available and we believe it could within a short order, however it is also not assured or guaranteed that it will become available when we think, and it may also never become available at all for any number of reasons as simple as someone not following up and adding newly ascertained information or that the only possible source of the information has become permanently unavailable for many other reasons of their own.
17d ago

rramadass 17d ago

[flagged]

mr_toad 17d ago

> But I've always distinctly recalled that he was arguing for a 4th, or possibly 5th type, yet whenever I survey the literature I've only found at most such 2 types. But I've probably been looking in the wrong places. It’s common for surveys to encode various null values. ‘Refused to answer’, ‘Did not know’, ‘Not applicable’, ‘Illegible response’ etc, although there is no standard way to do this. The SAS language allows you to use multiple custom missing values, but this is not widely known and I’ve never seen anyone do it.

pseudohadamard 16d ago

Oh that's easy. The 4th is "misplaced/lost", and the 5th is your kids pointing fingers at each other and saying "he/she did it, it wasn't me". Finally, the 6th is FILE_NOT_FOUND, coincidentally also the third boolean value after TRUE and FALSE.