Universal Log Pre-processing Framework
National Technical Research Organisation (NTRO) · Miscellaneous · Software
The problem is real but the category is solved by mature tools — make the no-code extensibility mechanism your genuine contribution, because a straight log-normaliser is a reimplementation of Logstash a judge already knows.
What it actually is
Enterprises collect logs from everything — servers, firewalls, cloud services, containers, IoT — in a dozen different formats, and security teams waste huge effort writing custom parsers before the data is usable. The ask is a framework that ingests logs in any format, parses and normalises them into one unified schema while keeping the raw original, and scales to billions of events a day.
What to build
A log pre-processing framework that ingests events from diverse sources and formats — Syslog, JSON, XML, CSV, CEF, LEEF, proprietary schemas — parses and normalises them into a unified event schema aligned to a standard like the Elastic Common Schema or OCSF while preserving the complete raw event for forensic and compliance use, with an extensible parser mechanism so new sources can be added without code changes, field extraction and enrichment, and an architecture designed to scale to high-volume Big Data ingestion feeding SIEM, data-lake or ML platforms downstream.
Smallest thing that wins the room
Ingest three genuinely different log formats — a firewall CEF line, a JSON cloud audit event and a raw syslog line — and show all three normalised into the same unified schema with common fields aligned, the raw original preserved alongside each, and a new source format added through configuration rather than code.
How crowded this one gets
A guess, projected from the 2025 statements — the last year where both the submission counts and the winners were published.
Quieter than 38% of the 226 · #141 of 226 by expected field
A normal-sized field. Your idea has to be good, not miraculous.
Why: defence, intelligence and space bodies drew small fields.
This is a guess, not a fact
Nobody has published 2026’s numbers yet. This is an analysed estimate from last year’s pattern, so please do not take it as the truth — check the live counter on the SIH portal before you decide anything. The range covers the middle half of likely outcomes, so one statement in two lands outside it. Entry closes at 500 ideas per statement, so no range goes past that — a statement that reaches the cap fills and shuts rather than drawing an unlimited crowd. The model reads only three things a team can see before choosing — software or hardware, the theme, and what kind of body posted it — and those explain about a quarter of the variation in last year’s field sizes (R² 0.25 on held-out statements). Trust the band more than the number, and the ordering more than either. It cannot see how good your idea is, which is the part that actually decides it.
The scores
The number is the shorthand. The line under it is the reason.
Acceptance potential
3/5The problem is real and the field is thin because log plumbing is unglamorous, but mature open-source tools already do most of this, so your novelty must be the no-code extensibility or ML-assisted format inference — without one of those it reads as reimplementing Logstash.
Feasibility
4/5Log parsing and normalisation is well-trodden with mature tooling to learn from — Vector, Logstash, Fluentd — and public standard schemas like ECS and OCSF give you a normalisation target, so the framework is buildable, with the extensibility mechanism the genuinely interesting design problem.
Innovation scope
3/5Log normalisation is a solved category with established products, so your room is in the extensibility mechanism — adding a source without writing a parser — and in any ML-assisted format inference, rather than in the concept.
Clarity
4/5The description names the source diversity, the format list, the raw-preservation requirement, the unified-schema goal and the scale target, so the deliverable is well defined even though it does not name which standard schema to adopt.
Effort
HeavyThe ingestion layer, the parsing and normalisation engine, the extensible parser mechanism and the schema mapping are focused pieces, with the extensibility and scale claims adding real design work.
Demo-ability
MediumThree formats collapsing into one schema is a clear result, but log normalisation is infrastructure and the value is legible mainly to a security-operations audience rather than a general judge.
In its favour
- Green flag: Standard schemas like ECS and OCSF give you an authoritative normalisation target rather than an invented one
- Green flag: The no-code extensibility mechanism is a genuine differentiator over hand-written parsers and is the part worth investing in
- Green flag: Preserving the raw event alongside the normalised form is a concrete forensic requirement that is cheap to satisfy and shows domain awareness
- Green flag: The unglamorous infrastructure nature genuinely thins the field
Against it
- Red flag: Vector, Logstash and Fluentd already do log normalisation well, so a straight reimplementation has no novelty and a security judge will ask what yours adds
- Red flag: The billions-of-events-per-day scale claim is easy to assert and hard to demonstrate on a laptop, so be careful not to overclaim throughput
- Red flag: ML-assisted format inference sounds impressive but is unreliable on adversarially messy logs, so it can undercut the reliability a log pipeline needs
- Red flag: Without the extensibility differentiator this is a solved-problem reimplementation
What you will be writing
- Elastic Common Schema / OCSF normalisation target
- Grok / regex + Vector-style parsing pipeline
- Extensible parser plugin mechanism
- ML-assisted log format inference (optional)
- Kafka high-throughput ingestion
- Raw-event preservation store
- Log management
- Security data engineering
- SIEM pre-processing
Prior art to read before you start
universal log normalisation · extensible parsing framework · unified event schema mapping
Analysed by Claude Opus. Every score above is a judgment call with its reasoning attached — kindly cross-check this against the official statement on the SIH portal before your team commits to it.