5 ms·

Any organization with secrets worth keeping must try to detect spies and repel infiltrators. It may reduce disastrous false negatives at the cost of an increas

Any organization with secrets worth keeping must try to detect spies and repel infiltrators. It may reduce disastrous false negatives at the cost of an increase in false positives. In the 1940s, the Soviet Union had hundreds of agents in the executive branch of the US government. KGB records opened in the last 20 years largely vindicate McCarthy (Edit: in that the threat was real; infiltration had happened; most of his targets were Soviet agents, Communists, or associated with Communists; and security risks had to be removed from positions in which they could do harm for the good of freedom-loving people everywhere).

CamperBob2 12y ago
Are you referring to the Venona intercepts? Those were from WWII, not the 1950s, weren't they?
ska 12y ago
In this, as in all classification schemes, there is a real risk of falling into a "dropping FN will raise FP" category error. The core issue may be be that your algorithm is bad, and you can find another with bother better FN and better FP.
chrisbennet 12y ago
Edit: Someone is wrong on the Internet and that person is me. ;-) When I quoted the 3,370 number I missed the "(end-of-fiscal-year count, excluding Postal Service, in thousands) [ChrisIsWrong] The number of employees in the executive branch in the 1940's peaked at 3,370. [1] So by "hundreds of agents in the executive branch" do you mean that say, 10% of the executive branch were communist agents? [/ChrisIsWrong] My Dad remembers McCarthy and his famous speech but doesn't know that he was discredited or even what the term "McCartheism" means. [1] http://www.opm.gov/policy-data-oversight/data-analysis-documentation/federal-employment-reports/historical-tables/executive-branch-civilian-employment-since-1940/ http://www.opm.gov/policy-data-oversight/data-analysis-docum...
- aurelian 12y ago
  According to your link, the Department of Justice, which included the FBI, had fewer than 20 employees. Edit: Thanks for acknowledging the mistake. I made the same one here.
- twoodfin 12y ago
  That number doesn't pass the smell test. Today's Treasury Department alone employs more than 115,000 people[1]. The figures are "in thousands". The Executive Branch at the time had over 3.3 million employees. [1] http://en.wikipedia.org/wiki/United_States_Department_of_the_Treasury http://en.wikipedia.org/wiki/United_States_Department_of_the...
lotsofmangos 12y ago
I don't see that evidence of spies in government vindicates MCCarthy. It might do if he had restricted himself to the threat of spies in the government, rather than instituting a campaign against communists and homosexuals throughout much of US society. Spies in the executive does not justify Hollywood blacklists, the censoring of library books resulting in book burnings, attacks on the clergy, attacks on academics, the 10000 people forced into unemployment by the loyalty review boards or the repeated burglaries against lawyers who would represent those who were targeted.
- a3n 12y ago
  Yeah, it was just a vehicle for McCarthy. It could just as easily have been the scourge of crabgrass.
jjoonathan 12y ago
According to http://en.wikipedia.org/wiki/Joseph_McCarthy http://en.wikipedia.org/wiki/Joseph_McCarthy > historian John Earl Haynes concluded that, of 159 people identified on lists used or referenced by McCarthy, evidence was substantial that nine had aided Soviet espionage efforts. 9/159. A 94% false positive rate.
- aurelian 12y ago
  Why don't you quote the next sentence? "He suggested that a majority of those on the lists could legitimately have been considered security risks, but that a substantial minority could not."
  - jjoonathan 12y ago
    Because it's nearly tautological: everyone on that list got there for some reason. The fact that a majority of those reasons weren't entirely made up isn't an indication of their quality as predictors of espionage. Since we now have access to the ground truth we can compute the actual false positive rate. The great-grandparent post suggests that such an examination would vindicate McCarthy, but I have trouble sympathizing with this view. A 94% false-positive rate (96% if you use McCarthy's original figure) is bad no matter how you slice it.
    - aurelian 12y ago
      McCarthy identified several agents actively engaged in espionage. KGB records identify hundreds more that he missed. McCarthy identified many more people in sensitive positions who were security risks, many with Communist associations. Even if those people were not actively engaged in espionage, they should not have held sensitive positions. KGB/NKVD archives reveal they had hundreds of agents in the executive branch in the 1940s. McCarthy spurred the removal of Soviet moles, Communist sympathizers, and other security risks from sensitive positions. In this, he did the US a great service, as regrettable as false accusations are. Edit: In response to your questions, it's as if you didn't read what I just wrote. Regarding the Soviet records of infiltration, several books have been written on these records and their revelations.

CamperBob2 12y ago

Are you referring to the Venona intercepts? Those were from WWII, not the 1950s, weren't they?

ska 12y ago

In this, as in all classification schemes, there is a real risk of falling into a "dropping FN will raise FP" category error. The core issue may be be that your algorithm is bad, and you can find another with bother better FN and better FP.

chrisbennet 12y ago

Edit: Someone is wrong on the Internet and that person is me. ;-) When I quoted the 3,370 number I missed the "(end-of-fiscal-year count, excluding Postal Service, in thousands) [ChrisIsWrong] The number of employees in the executive branch in the 1940's peaked at 3,370. [1] So by "hundreds of agents in the executive branch" do you mean that say, 10% of the executive branch were communist agents? [/ChrisIsWrong] My Dad remembers McCarthy and his famous speech but doesn't know that he was discredited or even what the term "McCartheism" means. [1] http://www.opm.gov/policy-data-oversight/data-analysis-documentation/federal-employment-reports/historical-tables/executive-branch-civilian-employment-since-1940/ http://www.opm.gov/policy-data-oversight/data-analysis-docum...

aurelian 12y ago
According to your link, the Department of Justice, which included the FBI, had fewer than 20 employees. Edit: Thanks for acknowledging the mistake. I made the same one here.
twoodfin 12y ago
That number doesn't pass the smell test. Today's Treasury Department alone employs more than 115,000 people[1]. The figures are "in thousands". The Executive Branch at the time had over 3.3 million employees. [1] http://en.wikipedia.org/wiki/United_States_Department_of_the_Treasury http://en.wikipedia.org/wiki/United_States_Department_of_the...

lotsofmangos 12y ago

I don't see that evidence of spies in government vindicates MCCarthy. It might do if he had restricted himself to the threat of spies in the government, rather than instituting a campaign against communists and homosexuals throughout much of US society. Spies in the executive does not justify Hollywood blacklists, the censoring of library books resulting in book burnings, attacks on the clergy, attacks on academics, the 10000 people forced into unemployment by the loyalty review boards or the repeated burglaries against lawyers who would represent those who were targeted.

a3n 12y ago
Yeah, it was just a vehicle for McCarthy. It could just as easily have been the scourge of crabgrass.

jjoonathan 12y ago

According to http://en.wikipedia.org/wiki/Joseph_McCarthy http://en.wikipedia.org/wiki/Joseph_McCarthy > historian John Earl Haynes concluded that, of 159 people identified on lists used or referenced by McCarthy, evidence was substantial that nine had aided Soviet espionage efforts. 9/159. A 94% false positive rate.

aurelian 12y ago
Why don't you quote the next sentence? "He suggested that a majority of those on the lists could legitimately have been considered security risks, but that a substantial minority could not."
- jjoonathan 12y ago
  Because it's nearly tautological: everyone on that list got there for some reason. The fact that a majority of those reasons weren't entirely made up isn't an indication of their quality as predictors of espionage. Since we now have access to the ground truth we can compute the actual false positive rate. The great-grandparent post suggests that such an examination would vindicate McCarthy, but I have trouble sympathizing with this view. A 94% false-positive rate (96% if you use McCarthy's original figure) is bad no matter how you slice it.
  - aurelian 12y ago
    McCarthy identified several agents actively engaged in espionage. KGB records identify hundreds more that he missed. McCarthy identified many more people in sensitive positions who were security risks, many with Communist associations. Even if those people were not actively engaged in espionage, they should not have held sensitive positions. KGB/NKVD archives reveal they had hundreds of agents in the executive branch in the 1940s. McCarthy spurred the removal of Soviet moles, Communist sympathizers, and other security risks from sensitive positions. In this, he did the US a great service, as regrettable as false accusations are. Edit: In response to your questions, it's as if you didn't read what I just wrote. Regarding the Soviet records of infiltration, several books have been written on these records and their revelations.