Video analytics and metadata systems

Video analytics reports observations from a model operating in a particular scene. Retain the model, confidence and scene context when applying downstream rules.

Overview#

Analytics output is an observation produced by a model under particular scene and configuration conditions, not ground truth. Preserve input/model/context and uncertainty so a downstream rule can't turn a probabilistic label into an unqualified fact.

Pipeline#

video/sensor input -> preprocessing/ROI -> model/inference -> tracking
 -> rule/threshold -> metadata/event -> review/workflow/automation

Canonical records should identify camera/source and media time, object/track/event IDs, class/attributes, confidence/quality where defined, region/rule, model/product/version, configuration/calibration, source frame/clip reference, processing time/location, and transformation/mapping version.

Standards and interoperability#

ONVIF Profile M standardizes analytics metadata/event interfaces including generic objects and conditional vehicle, plate, face/body, geolocation, rules, and MQTT event handling. Conditional support must be checked per conformant product; Profile M isn't a promise of a specific model's accuracy.

IEC 62676-6:2026 establishes performance testing and grading scope for real time intelligent video analytics, including core/complex capabilities and operating stress. The normative test methods are paid; public catalogue scope does not validate a site.

Evaluation#

Define target population, scene, weather/illumination, camera/view, object sizes/speeds/occlusion, threshold, dwell/rule, and consequence. Measure at the operational decision level:

• true/false positive and negative outcomes;
• precision/recall or detection probability and nuisance alarm rate;
• latency, duplicate/split/merged tracks, missed transitions;
• performance by relevant environmental and demographic groups;
• operator workload and escalation outcome;
• unavailable/uncertain periods, not only scored frames.

Aggregate accuracy can conceal a harmful subgroup or rare condition failure. NIST's Face Recognition Vendor Test demographic report demonstrates why demographic effects must be evaluated for facial recognition; it doesn't evaluate a local deployment automatically.

Model/configuration lifecycle#

Inventory model and runtime versions, training/provenance information available from supplier, camera/input contract, thresholds/regions, license, firmware/hardware acceleration, and rollback. Rebaseline after camera move, lens/illumination change, scene change, model/update, codec/profile change, or seasonal conditions.

Monitor input quality, inference availability, event rate distribution, confidence drift, operator disposition, ground truth sample process, and configuration changes. Avoid self reinforcing labels in which prior model output becomes unreviewed training truth.

Security, privacy, and automation#

• Minimize faces, plates, biometric embeddings, trajectories, demographics, occupancy, and cross camera tracking; define purpose, access, retention, and subject processes.
• Authenticate metadata source and preserve media linkage; reject stale/replayed events and tenant/site confusion.
• Bound images/metadata and untrusted model/plugin inputs.
• Separate analytics administration from video viewing and actuation.
• Require human or separately assured policy for high consequence action. Never unlock, dispatch, deny service, or declare identity solely from an unvalidated generic detection.

Failure semantics#

Represent not-detected, no-object, analytics-unavailable, input-unusable, rule-disabled, and unknown separately. Silence from an analytics service isn't a normal scene.

Return to Video surveillance systems.