Providing a reference for your joke doesn't really change much. I don't think you were downvoted because anyone thought your comment need citations, or because

Providing a reference for your joke doesn't really change much. I don't think you were downvoted because anyone thought your comment need citations, or because they weren't sure if it was actually supposed to be a joke. Perhaps it was just a bit trite, and/or irrelevant.