Universal Log Pre-processing Framework

The receipts 🧾

Every red flag on this statement, in full. These are the four places it bites.

1. Exhibit A

   Vector, Logstash and Fluentd already do log normalisation well, so a straight reimplementation has no novelty and a security judge will ask what yours adds

2. It gets worse

   The billions-of-events-per-day scale claim is easy to assert and hard to demonstrate on a laptop, so be careful not to overclaim throughput

3. Still reading?

   ML-assisted format inference sounds impressive but is unreliable on adversarially messy logs, so it can undercut the reliability a log pipeline needs

4. And the finisher

   Without the extensibility differentiator this is a solved-problem reimplementation

The damage report 📉

Every score this statement earned, and what each one actually costs you.

• Feasibility

  4/5

  Actually buildable, which on this slate is rarer than it sounds. Do not squander it on scope.

  Log parsing and normalisation is well-trodden with mature tooling to learn from — Vector, Logstash, Fluentd — and public standard schemas like ECS and OCSF give you a normalisation target, so the framework is buildable, with the extensibility mechanism the genuinely interesting design problem.

• Innovation scope

  3/5

  Mildly interesting. The novelty will not carry the room; the build has to.

  Log normalisation is a solved category with established products, so your room is in the extensibility mechanism — adding a source without writing a parser — and in any ML-assisted format inference, rather than in the concept.

• Clarity

  4/5

  The ask is unambiguous, which quietly removes your favourite excuse.

  The description names the source diversity, the format list, the raw-preservation requirement, the unified-schema goal and the scale target, so the deliverable is well defined even though it does not name which standard schema to adopt.

• Acceptance potential

  3/5

  Middle of the pack. This statement will not win the room for you — you will have to.

  The problem is real and the field is thin because log plumbing is unglamorous, but mature open-source tools already do most of this, so your novelty must be the no-code extensibility or ML-assisted format inference — without one of those it reads as reimplementing Logstash.

• Effort

  Heavy

  Heavy. Somebody on this team is not sleeping in week three. Pick who, on purpose.

  The ingestion layer, the parsing and normalisation engine, the extensible parser mechanism and the schema mapping are focused pieces, with the extensibility and scale claims adding real design work.

• Demo-ability

  Medium

  Demoable, if you rehearse it. Nobody rehearses it.

  Three formats collapsing into one schema is a clear result, but log normalisation is infrastructure and the value is legible mainly to a security-operations audience rather than a general judge.

• Data

  None supplied

  No dataset comes with this one, so every accuracy figure you quote is a number about labels you invented.

  Nothing is provided with the statement. You are sourcing, cleaning and labelling it yourself, and that work is invisible in the demo but very visible in the questions.

The demo they will have already seen 🪞

Somewhere around 150–340 teams are heading here, and the description is doing the choosing for most of them. They will read the same brief, reach the same architecture, and build a version of the same demo you are planning. Being correct is the floor. If your five minutes could be swapped with the team before you and nobody in the room would notice, you have not picked badly — you have built predictably, which costs exactly the same and hurts more.

What survives 🛡️

The ground worth standing on when the questions start.

• Standard schemas like ECS and OCSF give you an authoritative normalisation target rather than an invented one
• The no-code extensibility mechanism is a genuine differentiator over hand-written parsers and is the part worth investing in
• Preserving the raw event alongside the normalised form is a concrete forensic requirement that is cheap to satisfy and shows domain awareness