I was born and bred in the land of the Bard and yet I also mislabelled roughly the same 30% that they did. In my opinion that was mostly caused by the lack of c

I was born and bred in the land of the Bard and yet I also mislabelled roughly the same 30% that they did. In my opinion that was mostly caused by the lack of context (eg, 'Traps') As an example of the above, I assumed the 'traps' one meant "his mouth is so big that it shuts out the sun" (aka an insult) since to 'Shut your Trap' means to shut your mouth/ stop talking. Once there was a body-building context, I worked out that it was a reference to a person's trapezoid muscles and therefore the sentiment was (most likely) Positive rather than the Negative/Confrontational/Sarcasm label that I would have first assigned it. There are similar examples but that gives a rough idea about why context is important for sentiment data-set labelling. But - #1 In a Mechanical-Turk setup - who has time to scan through paragraphs and #2 How far back to you go to get the full picture? I don't think you can so why not do it by hiring a temp for an in-house two week gig? Cheaper and you can directly monitor their performance. Win-Win