You can try to find a dataset that contains the equal number of positive and negative documents (sentences, etc.) and use it as the validation set. I.e. to tune

You can try to find a dataset that contains the equal number of positive and negative documents (sentences, etc.) and use it as the validation set. I.e. to tune your hyperparameters on it. In the simple case your hyperparameter can be α in sentiment = (α * #positive_matches - #negative_matches) / (document_word_count)