On-device Visual Perception for Light-weight Browser Agents
Indian Space Research Organisation(ISRO) · Miscellaneous · Software
A genuinely interesting privacy-first architecture that browser APIs now make real, but reliable redaction is the whole game — one missed field breaks the premise, so make recall on sensitive elements your headline metric rather than showing it redacting only the obvious cases.
Data: Any open-source data; evaluation use cases provided at finale
What it actually is
AI agents that can see your screen could automate complex workflows, but sending your screen to a server leaks sensitive data. The ask is a browser agent that runs a lightweight vision model locally to read the screen, and before sending anything to a server for heavier reasoning, automatically detects and redacts the sensitive parts — blurring faces, masking passwords and PII — so only anonymised data ever leaves the machine.
What to build
A privacy-preserving browser vision agent with a client-side architecture where a local Vision Transformer or equivalent runs in the browser via WebGPU and WebAssembly, reads the current screen or DOM state, and makes or proposes actions; when heavier server reasoning is needed, it first detects and redacts sensitive elements — blacking out password fields, masking PII, blurring faces — using DOM-tag awareness and visual detection, so only sanitised, unidentifiable data is transmitted, with the server aware of the redaction scheme so it can still reason over the anonymised structure.
Smallest thing that wins the room
Load a page containing a password field, a name and an email, and show the local model detecting and redacting all three in the browser before any network call, then show the server receiving only the sanitised screen and still being able to act on the page structure.
How crowded this one gets
A guess, projected from the 2025 statements — the last year where both the submission counts and the winners were published.
Quieter than 35% of the 226 · #147 of 226 by expected field
A normal-sized field. Your idea has to be good, not miraculous.
Why: defence, intelligence and space bodies drew small fields.
This is a guess, not a fact
Nobody has published 2026’s numbers yet. This is an analysed estimate from last year’s pattern, so please do not take it as the truth — check the live counter on the SIH portal before you decide anything. The range covers the middle half of likely outcomes, so one statement in two lands outside it. Entry closes at 500 ideas per statement, so no range goes past that — a statement that reaches the cap fills and shuts rather than drawing an unlimited crowd. The model reads only three things a team can see before choosing — software or hardware, the theme, and what kind of body posted it — and those explain about a quarter of the variation in last year’s field sizes (R² 0.25 on held-out statements). Trust the band more than the number, and the ordering more than either. It cannot see how good your idea is, which is the part that actually decides it.
The scores
The number is the shorthand. The line under it is the reason.
Acceptance potential
3/5The privacy-first framing is current and the in-browser architecture is genuinely interesting, but reliable redaction is the crux and one missed PII field breaks the whole premise, browser resource limits constrain the local model, and the finale-only evaluation means you cannot tune to the test — a demo that redacts the obvious fields but misses edge cases undercuts the core claim.
Feasibility
3/5Transformers.js and ONNX Runtime Web genuinely run vision models in the browser via WebGPU, and DOM-based PII detection is tractable, but running a capable ViT client-side within browser resource limits while reliably catching every sensitive element is hard, and the redaction must be near-perfect because one missed PII field defeats the entire privacy premise.
Innovation scope
4/5The split architecture — local perception plus guaranteed client-side redaction before any server call — is a genuinely interesting and current design problem, and how you combine DOM tags with visual detection to catch sensitive elements reliably is open.
Clarity
4/5The architecture, the client-side technologies, the redaction requirement and the local-versus-server split are described clearly, though the evaluation use cases are only revealed at the finale so the exact test is unknown.
Effort
MassiveIn-browser model inference, reliable multi-modal PII detection and redaction, the local agent logic and the server-side coordination is a demanding full-stack build against real resource constraints.
Demo-ability
EasyWatching sensitive fields get redacted in the browser before anything is transmitted, with the network payload shown to contain only sanitised data, is a concrete and convincing privacy demonstration.
In its favour
- Green flag: Transformers.js and ONNX Runtime Web genuinely enable in-browser model inference, so the client-side architecture is real rather than aspirational
- Green flag: The redact-before-transmit demo is concrete and directly proves the privacy claim by showing the sanitised network payload
- Green flag: DOM-tag awareness gives you structured, reliable signals for detecting sensitive fields alongside the visual model
- Green flag: The privacy-first agent framing is current and resonant, and few teams will attempt in-browser vision inference
Against it
- Red flag: The privacy premise demands near-perfect redaction, and one missed PII field defeats the entire point, so recall on sensitive elements is the make-or-break metric
- Red flag: Browser resource limits constrain how capable the local model can be, forcing a genuine accuracy-versus-footprint trade-off
- Red flag: Evaluation use cases are revealed only at the finale, so you cannot tune to the test and must generalise
- Red flag: Visual PII detection misses text rendered in images or canvas that DOM tags do not cover, which is exactly where redaction quietly fails
What you will be writing
- Transformers.js / ONNX Runtime Web in-browser inference
- WebGPU + WebAssembly acceleration
- Lightweight Vision Transformer (MobileViT-class)
- DOM-tag + visual PII detection
- Client-side redaction (blur / mask) pipeline
- Local agent with server coordination protocol
- Privacy-preserving AI
- Browser agents
- On-device inference
Prior art to read before you start
client-side visual redaction · in-browser model inference · privacy-preserving agent architecture
Analysed by Claude Opus. Every score above is a judgment call with its reasoning attached — kindly cross-check this against the official statement on the SIH portal before your team commits to it.