Social Media Listening: Malaysian
Socio-Economic
Sentiment Analysis
CHALLENGE
Public sentiment during national transitions is highly fragmented and noisy. Government agencies need structured, real-time feedback on policy transitions rather than manual, time-consuming forum audits.
SOLUTION
Automating the categorization of unstructured Facebook and Twitter commentary into actionable national topics.
VALUE
-
It provides government agencies with a real-time "temperature check" on how the public reacts to major national policies.
-
By segmenting and analyzing the pillar, policymakers can pinpoint exactly when and where grocery inflation complaints are spiking on social media.
-
Social media is a massive, untapped goldmine of public opinion. This pipeline converts thousands of chaotic, disorganized posts into a clean, queryable relational database ready to power interactive BI Dashboards (like Tableau or Power BI).
Limitations
-
Cross-Linguistic Sentiment Drift & Neutrality Bias
Using TextBlob (built purely on English lexicons) to analyze Malay text causes unrecognized vocabulary to default to a neutral 0.0 polarity score.
Recommendation | Replace lexicon-based scoring with a dedicated Bahasa Malaysia NLP library or a pretrained multilingual deep learning model. This transitions the pipeline to true machine learning, enabling contextual sentiment capture of code-mixed local dialects (Manglish/Malay).
-
Static, Rule-Based Topic Modeling
Sorting posts into categories relies strictly on hardcoded Regular Expressions (Regex) keywords. The pipeline is blind to emerging, unprogrammed public concerns or shifts in slang terms.
Recommendation | Implement a hybrid architecture using BERTopic alongside rule-based constraints. While Regex maintains precision on target policy topics, BERTopic leverages sentence embeddings in the background to dynamically cluster and auto-discover unexpected, trending conversation pillars.
-
Morphological Preprocessing Mismatch
Applying English tokenizers and stemmers fails on Malay grammar, which relies on complex affixes.
Recommendation | Standardize preprocessing using Malaya's native Malay stemmer and stopword list. This properly resolves morphological roots and filters noise according to local linguistic structures, ensuring correct token normalization before modeling.
Built a targeted, rule-based NLP pipeline to parse unstructured social media commentary, segmenting public opinion into key socio-economic pillars during major government transitions.