Predicting Social Welfare Eligibility using Machine Learning
A machine learning classification and statistical validation pipeline designed to predict government welfare eligibility in Malaysia by engineering demographic and geographic predictors.
#MachineLearning #SupervisedClassification #FeatureEngineering #PredictiveModeling #StatisticalHypothesisTesting #Python
#Scikit-Learn #Pandas
CHALLENGE
Welfare programs process millions of citizen applications, but identifying high-priority aid recipients at scale remains a manual, error-prone, and bottlenecked task.
SOLUTION
Built a rule-based tool to extract each person's age, gender, and birth state directly from their Malaysian ID card (MyKad). To handle the massive dataset efficiently, the exact column formats are defined during the data loading process to prevent system warnings and save computer memory. Finally, the data is on trained three different machine learning models and used a mathematical check called a paired t-test to prove which model was truly the most accurate.
VALUE
-
Automating demographic feature extraction directly from memorized MyKad strings eliminates the operational cost, manual errors, and labor hours of standard survey-driven data verification.
-
By accurately mapping household income against calculated age and geographic vulnerability thresholds, government bodies ensure resources reach high-need demographics.
-
Using paired t-test validation proves to regulatory auditors and stakeholders that the final targeting model is mathematically sound, protecting the agency from bias or random error accusations.
Limitations
-
Data Misreporting Bias
The target variable rests heavily on self-reported household income, which is highly vulnerable to under-reporting in regions dominated by informal cash economies.
Recommendation | Since welfare recipients usually represent a minority of the total dataset, implement SMOTE (Synthetic Minority Over-sampling Technique) or apply class weights (class_weight='balanced') to prevent classifiers from ignoring needy cases.
-
Scalability
Training non-parametric classifiers like Support Vector Machines (SVM) scales quadratically with dataset size, causing Out-Of-Memory (OOM) crashes on standard computer hardware when processing huge datasets.
Recommendation | Shift data cleaning and pipeline tasks from Pandas to out-of-core tools like Polars, Dask, or PySpark to parallelize demographic processing.