Social Media Topic Modeling: Malaysian
Socio-Economic
Sentiment Analysis
An end-to-end NLP and topic modeling pipeline that processes Malay-dominant social media data using Malaya preprocessing and BERTopic.
#NaturalLanguageProcessing
#TextAnalytics
#SentimentAnalysis #Python
#BERTopic #Malaya
CHALLENGE
Strategic policy analysts and corporate communications teams needs real-time data to evaluate public reactions to policy shifts (e.g., emergency withdrawals) re the Employees Provident Fund (EPF) also known in Malay as KWSP or Kumpulan Wang Simpanan Pekerja.
SOLUTION
To process the massive volume of data, a specialized text intelligence stack was engineered to clean, analyze, and cluster unstructured social media feeds. First, colloquial preprocessing was performed using the Malaya library to normalize Malay slang. Next, a sentiment scoping phase bucketized the dataset via lexicon scores, and finally, a thematic clustering layer deployed BERTopic to compress data points into distinct, human-interpretable semantic topics.
VALUE
-
Provides empirical proof of public anxiety; revealing that 69% of the community views emergency withdrawals as merely a short-term patch rather than a sustainable retirement solution.
-
Identifies exact event triggers to help teams design proactive, data-backed educational campaigns.
Limitations
-
Vector Embedding Context Bottlenecks
Multilingual sentence-transformer backends possess strict token length restrictions.
Recommendation | Refine the initial cleaning function to strip consecutive emoji sequences, truncate repetitive trailing punctuation patterns, and segregate block hashtags from the main semantic body text prior to running embedding generation.
-
Equal Data Row Weighting (Missing Structural Weights)
It ignores standard engagement metadata metrics such as retweet counts, follower reach, shares, and account authority weights.
Recommendation | Incorporate metadata metrics as data-weighting coefficients.