The Challenge: The Scale vs. Cost Dilemma
Our client, a national non-profit, was managing the enrollment of over 100,000 citizens into a government welfare scheme.
Each applicant was issued a reference number. To track the application—whether it was Pending, Approved, or Rejected—volunteers had to manually check a central government portal.
Doing this manually for 1 Lakh (100,000) records daily was impossible. Hosting a heavy scraper on AWS/Azure would have drained the non-profit's limited budget.
The Engineering: Extreme Optimization on the Edge
We needed a solution that cost almost nothing to run but handled enterprise-grade volume. We turned to Edge Computing.
- Hardware: Instead of a cloud server, we deployed the solution on a single Raspberry Pi device.
- The Code: We wrote a lightweight Python & Bash script stripped of all heavy libraries. It used raw HTTP requests and Regular Expressions (Regex) to parse HTML responses in milliseconds.
- Orchestration: A custom Cron scheduler woke the device every 24 hours to perform the sync cycle during low-traffic hours.
The Outcome
The system successfully refreshed 100,000+ database records every night.
By optimizing the code to run on a $35 micro-computer, we saved the client thousands of dollars in annual cloud computing costs while ensuring field volunteers always had the latest status on their mobile apps.