Reading Time: 5 minutes

Machine learning and NLP research – in the domains of social sciences generally and psychology specifically – requires certain human resources. Most of the time, the experiment subjects in these domains are, well, human. Sometimes, the subjects label the relevant research data themselves, such as when they answer multiple-choice questions. This is called self-assessment. At other times, an expert or another external observer needs to label the data. Either case demands time and effort from people.

Here are three main problems we can encounter when we talk about insufficient data in such machine learning experiments:

1. There are not enough people participate in the experiment
2. There is not enough content or sufficient data gathered from the participants. For example, if we’ve had an interview with many people but too many of them weren’t too responsive to survey questions.
3. There is not enough tagging or labeling for the learning process to achieve sufficient training. For instance, if the experiment is hamstrung by reliance on a single expert who can conduct proper tagging.

Leveraging the power of supervised and unsupervised learning together

Potential solutions to the third problem are particularly intriguing for data scientists. They might afford us the chance to merge the two general approaches taken in machine learning: supervised and unsupervised learning.

The supervised approach for data classification is composed of two stages:

Training time – A learning algorithm (or training algorithm/process) examines data that has already been labeled. The output of this stage is known as a classifier
Inference time – That classifier is then used to determine the labels of unseen and unlabeled data, with some error probability (a good training process will help guarantee this error will be as minimal as possible)

Consider one example of such a learning scenario: determining sentiment polarity of a short text, a subtask of sentiment analysis.

Let’s say you have a database of 40,000 tweets from Twitter that express either positive or negative sentiment. We let someone label each tweet as either positive or negative, and then run the training algorithm (See Muhammad et al., 2013; Baziotis et al., 2017) on them to generate the classifier. The classifier is a statistical model that when fed to a computer program enables it to try and determine if a new tweet is either positive or negative.

The number of tweets that we can tag for training is limited. Therefore, we know that some words would not be seen by the training process or would not be found in all the desired contexts that would allow the classifier to generalize well to unseen tweets.

This is where unsupervised learning comes in.

An unsupervised algorithm is one that should supply us with insights from unlabeled data. And, in the case of Twitter, such data can be found for free.

Word2Vec

The unsupervised algorithm we will use is called word2vec. Like in the supervised case, the algorithm also contains a training phase. But unlike in the supervised case, the output is not a classifier but a lookup table called embedding. This table assigns every word in the dictionary a vector – a list of 300 numbers that represents the semantic meaning of the word. If two words in the dictionary are semantically close, the numbers in the list should be approximate to each other. That way, we can use this algorithm to infer the semantic similarity between words.

If you’re familiar with vector geometry you can use the following figure to illustrate how the position in vector space corresponds to semantic role of the word:

Visualization of learned vectors (Tensorflow.org)

Visualization of learned vectors for word2vec (Tensorflow.org); also see https://www.distilled.net/word2vec-examples/

Instead of letting the learning algorithm train on sentences, we let it train on sequences of word vectors. That way, if the classifier for example were to encounter the sentence, “Your lecture was exquisite,” it would likely assign it the same sentiment as “Your lecture was excellent.” That would be the case even if the word “exquisite” were not seen at all in the training stage, or were seen in another context. This is because the word vectors of “excellent” and “exquisite” are similar.

Now for the big question: How does the unsupervised training of word2vec work? If there’s no one to label the dataset (millions of unlabeled tweets) used for the unsupervised training process, how does the algorithm know to give similar vector values to similar words? The answer is that it works like the way our mind works, by inferring the semantic meaning of the word from its context. If the words “excellent” and “exquisite” were seen in similar contexts on Twitter, then word2vec would assign them a similar semantic vector.

We gave an example of how taking a larger, unlabeled set, can help give us a way to represent words, so with this new representation we can generate models which generalize better for unseen instances. Here, unsupervised learning induced for us a preprocessing stage, or a feature extraction stage, that transformed the data into a meaningful numeric representation before then feeding that data into a supervised learning algorithm.

You are invited to explore other ways to combine supervised and unsupervised learning for partially labeled data, like semi-supervised learning (which we might talk about in future posts).

Works Cited

Rosenthal, S., Farra, N., & Nakov P. (2017). SemEval-2017 Task 4: Sentiment Analysis in Twitter. In Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017) (pp. 502–518). Vancouver, Canada, August 3-4, 2017. © 2017 Association for Computational Linguistics. http://www.aclweb.org/anthology/S17-2088

Mohammad, S.M., Kiritchenko, S., & Zhu, X. (2013). NRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets. In Proceedings of the seventh international workshop on Semantic Evaluation Exercises (SemEval-2013). June 2013, Atlanta, USA. https://arxiv.org/abs/1308.6242

Baziotis, C.; Pelekis, N., & Doulkeridis C. (2017). DataStories at SemEval-2017 Task 4: Deep LSTM with Attention for Message-level and Topic-based Sentiment Analysis. In Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017) (pp. 747–754). Vancouver, Canada, August 3-4, 2017. © 2017 Association for Computational Linguistics. http://www.aclweb.org/anthology/S17-2126

Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems (pp. 3111-3119).

CONTRIBUTORS

ABOUT US