Skip to content

Sophia

Sophia builds a digital twin of a place from camera, radar and 3D scans, one that humans and AI understand together. The AI sets restricted zones in it on its own, and every change is verified. It runs in the browser and on the Meta Quest 3.

Sophia websiteDemonstrator runs locally
Sophia: digital twin of a place with detected people and vehicles

What it is about

Cameras deliver images, robots each build their own map, and AI models think in text. What is missing is a shared, persistent picture of the place that everyone can look into and everyone can update.

Sophia builds that picture: camera, radar and 3D scans become a digital twin of a place that humans and AI understand and use together.

How it works

A camera with object detection, radar and the LiDAR scan from the iPhone come together over MQTT. The twin runs in the browser and on the Meta Quest 3.

The AI works through tools inside the twin: it checks lines of sight, routes and sensor coverage and sets restricted zones itself. Every change is verified in the twin before it is reported.

What was hard

Two sensors, one person

A camera and a low-cost radar see the same person in two different ways. Sophia merges both into one shared track.

The AI adds, it does not overrule

A vision-language model rates scenes in a fixed format and is checked against the measured sensor traces. It adds to the evidence but never overrules it. The models themselves are measured too: their answers are scored against values from the twin, such as straight-line distance against walkable route.

Staying repeatable

The model is interchangeable and runs through a provider or locally. Recordings can be replayed, so that the same moment gives the same result twice.

Key facts

Status
demonstrator in a real building
Runs on
browser and Meta Quest 3
Built
solo, since June 2026
Tech
Python, MQTT, YOLO, OpenCV, COLMAP, three.js