Development of a Real-time Airfare Price Index for India through Automated Web Scraping of Airline and Online Travel Aggregator Portals for Augmentation of the Consumer Price Index (CPI).
MoSPI · Travel & Tourism · Software
The validation against a published official series is a rare and strong position, but the thirty-day back-test is a calendar requirement and the compliance tension is real, so start collecting immediately and make your missing-fare methodology the thing you defend rather than the scraper.
What it actually is
India's official inflation number includes air travel, but the fares feeding into it are still collected by hand from a few ticketing offices, while over ninety percent of tickets are now bought online at prices that swing wildly within a single day. So the published figure does not reflect what travellers actually pay. The ask is a system that collects real online fares automatically every day and turns them into a proper price index.
What to build
An end-to-end fare index pipeline: a scheduled collection engine gathering quotes from the airlines and aggregators the statement names, handling JavaScript-rendered pages, session management and rate limiting within the ethical and terms-of-service constraints the statement insists on; a fixed basket of representative city pairs chosen on passenger traffic data and sampled at the five advance-purchase windows specified from next-day out to forty-five days; a cleaning pipeline separating base fare from taxes, development fee and convenience charges, removing outliers and handling sold-out and cancelled flights as missing rather than as zero; an index construction module aggregating quotes to a daily, weekly and monthly index with defensible route weights; a dashboard showing trends, sector heatmaps and lead-time elasticity curves; and an API for statistical consumers, validated by at least thirty days of back-testing against published monthly average fares.
Smallest thing that wins the room
Show thirty days of your daily index beside the published monthly average fare for the same sectors, with your series tracking it, and then open the lead-time elasticity curve showing the same seat costing three times more at T+1 than at T+45.
How crowded this one gets
A guess, projected from the 2025 statements — the last year where both the submission counts and the winners were published.
Quieter than 13% of the 226 · #196 of 226 by expected field · reaches the 500 cap
Busier than most. Expect several teams to arrive at the same obvious solution.
Why: central ministry statements sat below the average.
This is a guess, not a fact
Nobody has published 2026’s numbers yet. This is an analysed estimate from last year’s pattern, so please do not take it as the truth — check the live counter on the SIH portal before you decide anything. The range covers the middle half of likely outcomes, so one statement in two lands outside it. Entry closes at 500 ideas per statement, so no range goes past that — a statement that reaches the cap fills and shuts rather than drawing an unlimited crowd. The model reads only three things a team can see before choosing — software or hardware, the theme, and what kind of body posted it — and those explain about a quarter of the variation in last year’s field sizes (R² 0.25 on held-out statements). Trust the band more than the number, and the ordering more than either. It cannot see how good your idea is, which is the part that actually decides it.
The scores
The number is the shorthand. The line under it is the reason.
Acceptance potential
3/5Real public validation data, a sharply specified brief and an official statistics domain that almost no team will enter all help, but the collection layer's compliance tension is structural rather than solvable, and thirty days of back-tested results is a calendar requirement that a team starting late simply cannot satisfy.
Feasibility
3/5The validation target is genuinely public since monthly average fare data is published, and the cleaning and index construction are ordinary work, but the collection layer sits in real tension with itself — the statement demands robots.txt and terms-of-service compliance from sites whose terms generally prohibit automated fare collection, and those sites deploy active anti-bot defences precisely to enforce that.
Innovation scope
3/5The pipeline shape, the city pairs, the advance-purchase windows and the deliverables are all specified, but the index methodology itself — how you weight routes, handle missing quotes and treat a fare that is unavailable rather than expensive — is genuine statistical design and is where the substance of this submission lives.
Clarity
5/5Exceptionally specific for a statistics statement — it names the airlines, the aggregators, example city pairs, all five advance-purchase windows, the fare components to separate, the anti-bot problems to handle and a concrete thirty-day back-test requirement against a named public benchmark.
Effort
HeavyA resilient multi-source collection engine, a cleaning pipeline, an index construction module, a dashboard, an API and thirty days of actual back-tested collection is five components plus a calendar-time dependency, and scrapers against defended sites need continuous maintenance rather than one-time construction.
Demo-ability
MediumThe back-test overlay is genuinely persuasive because it validates against an external published series, and the lead-time elasticity curve is a striking visual, but the product is charts and an API rather than anything interactive.
In its favour
- Green flag: Published monthly average fare data gives you an external validation target, so you can show your index tracks an independent official series rather than merely asserting it is correct — very few statements offer that
- Green flag: The index methodology is the real intellectual content and it is genuinely open: how you weight routes and how you treat an unavailable fare are decisions a statistician will interrogate and reward
- Green flag: Official statistics is a domain essentially no student team enters, so the field for this will be very thin
- Green flag: The lead-time elasticity curve is a compelling and unfamiliar visual that makes an abstract index tangible in a single chart
Against it
- Red flag: The statement demands robots.txt and terms-of-service compliance while asking you to scrape sites whose terms typically prohibit exactly this — read the actual terms of your sources and be prepared to say which you excluded and why, because a judge from a statistics ministry will care about the legality more than the throughput
- Red flag: Thirty days of back-tested results is a calendar dependency, not an engineering one, so collection has to start on day one or the requirement is unmeetable regardless of how good the code is
- Red flag: A sold-out or unavailable flight is not a high price and treating it as one silently biases the index upward — how you handle missingness is the single most consequential methodological choice here
- Red flag: Anti-bot defences change without warning and a collection engine that worked last week can silently return empty results, so build monitoring that fails loudly rather than a scraper that quietly stops
What you will be writing
- Playwright headless collection with session handling
- fare component parsing into base, tax and fee
- Laspeyres or chained index construction with route weights
- outlier and sold-out quote handling in cleaning pipeline
- TimescaleDB fare quote store
- FastAPI index endpoints with dashboard frontend
- Official statistics and price indices
- Web data collection
- Aviation pricing
Prior art to read before you start
automated price collection for index construction · dynamic airfare trend analysis · back-testing against official published series
Analysed by Claude Opus. Every score above is a judgment call with its reasoning attached — kindly cross-check this against the official statement on the SIH portal before your team commits to it.