Korean Sarcasm Detector

Period2023.03 – 2023.06
AffiliationUNIST Design Department
CollaboratorsSolo Project
Project TypeCoursework
TopicsCommunication, AI

1 Overview

I built a model that classifies Korean sentences as sarcastic or non-sarcastic, along with a web demo where anyone can try it.


2 Context

We often say the opposite of what we mean to express frustration, disapproval, or humor. People can usually tell whether a remark is praise or a dig, but could a model do the same? I was curious how well it could distinguish between these ambiguous expressions.

When I started this project, Kaggle had more than 500 projects related to sarcasm detection, but most focused on English, and examples involving Korean were rare. So I decided to build a model that classifies Korean sentences as sarcastic or non-sarcastic, then make a webpage where anyone could enter a sentence and see the result.


3 Approach

Exploring the dataset

I used a Korean sarcasm dataset shared publicly on GitHub, made up of tweets paired with labels indicating whether they were sarcastic. It came with 8,999 training samples and 301 test samples.

I removed one entry with a missing value and explored the data. Since the two sets differed so much in size, I combined them and split them again at a 3:1 ratio, keeping the proportions of sarcastic and non-sarcastic sentences consistent across both sets.

Many sentences were around 25 characters long, though the average length was about 50 characters. After checking that the length distributions were similar in the training and evaluation sets, I set the maximum input length to 50.

Preparing Korean text for classification

I prepared the text in stages, taking into account the characteristics of tweets and Korean sentences.

  1. Removed the @USER markers remaining in the tweets.
  2. Removed all characters other than Hangul.
  3. Removed the original spacing and reapplied spacing using PyKoSpacing.
  4. Corrected spelling using py-hanspell.
  5. Tokenized sentences into morphemes using the Okt morphological analyzer.

A Korean word can contain several morphemes, so I needed to break words down into these smaller units. After looking at several analyzers, I chose Okt because it could handle new sentences and offered a suitable balance of speed and performance.

I assigned a number to each morpheme so the model could process it. The training data contained about 19,325 unique terms. Words that appeared only once made up about 60.11% of the vocabulary, but accounted for about 12.38% of all word occurrences. I kept these rare words because they might still offer clues to sarcasm.

Because about 99.28% of the data had a length of 50 or less, I set the maximum input length to 50 and padded shorter sequences with zeros.

Where traditional classifiers struggled with context

I first tried five classification models.

  • Naive Bayes
  • Support Vector Machine
  • Logistic Regression
  • Random Forest
  • Decision Tree

Using Random Forest, I also built a feature that showed whether an entered sentence was sarcastic, along with its probability. I grouped the results into four levels based on that probability: Likely Sarcastic, Possibly Sarcastic, Unclear, and Unlikely Sarcastic.

The model did reasonably well with non-sarcastic sentences such as “오늘은 월요일이다” (“Today is Monday”) and “피자 맛있겠다” (“That pizza looks good”), but struggled to recognize sarcasm in sentences like “잘도 가만히 있네” (“You’re so good at staying still”). These received a low probability of sarcasm, or similar probabilities for both classes.

KcELECTRA

I changed my approach and decided to fine-tune a language model trained on a large body of text using the Korean sarcasm dataset.

I chose KcELECTRA as the base model, a Korean ELECTRA model trained on about 89 million sentences, including Naver News comments and replies. Since it was trained to handle user-generated and noisy text, I thought it would also suit the informal, messy language found on Twitter.

After fine-tuning, the model reached 84% accuracy. I tried the sentences it had struggled with earlier, and the sarcastic examples received sarcasm probabilities of 70% or higher. The results were also clearer than those from the model I had used initially.


4 Outcome

I built an input page and a results page so people could try the model themselves. After entering a sentence, they can see whether it is classified as sarcastic and the probabilities of it being sarcastic or non-sarcastic. I used an orange background for results classified as sarcastic and included the dataset and model accuracy on the page.


5 Reflection

This was my first project putting natural language processing and machine learning into practice. Many of the concepts were unfamiliar at first, but by looking for models I needed and adapting them to my purpose, I was able to build a classifier for a new problem.

Comparing accuracy alone wasn’t enough when choosing a model. I also needed to understand how it worked and whether it suited the data and the problem I was working with. The project gave me a chance to think about how to present the model’s judgments to users, too.