01 / The Problem & Hard Constraints

The Problem & Hard Constraints

Translating physical environments into tactile feedback requires processing high-resolution camera streams and 3D depth maps (LiDAR/Time-of-Flight) in real-time.

• Memory Bandwidth & Thermal Throttling: Extracting 60 frames per second (FPS) of raw 4K video and depth data from the camera sensor into the application layer causes massive memory copying, leading to CPU thermal throttling and battery drain within minutes.
• Thread Blocking: Triggering native haptic motors (Taptic Engine) via standard OS channels often blocks the main execution thread, causing the spatial mapping to desync from the user’s physical movement.
• Sensor Noise: Raw depth data is highly volatile. Translating flickering pixels directly into vibrations causes overwhelming, chaotic haptic feedback for the user.

02 / Architecture & Core Design Decisions

Architecture & Core Design Decisions

• Zero-Copy Buffer Pipeline (Native FFI): Bypassed the standard Flutter camera plugins. Built a C++ native layer that uses Direct Memory Access (DMA) to read the camera’s CVPixelBuffer / ImageReader memory pointers directly. The data is processed in C++ without ever being serialized or copied into the Dart VM.
• Spatial Downsampling & Grid Matrix: Instead of processing 12 million pixels per frame, the C++ layer applies a fast pooling algorithm, downsampling the 3D point cloud into a localized 9-zone collision grid, reducing matrix math overhead by 99%.
• Asynchronous Actuation Queue: Haptic vibration commands (varying by intensity and frequency) are dispatched to a background thread queue, completely decoupling the device’s physical motor actuation from the 60Hz visual processing loop.

03 / Deep Technical Challenges & Solutions

Deep Technical Challenges & Solutions

The Challenge: Jitter and False Positives in Depth Translation. Translating raw LiDAR depth values directly to haptic intensity meant that walking past a window (which absorbs/scatters infrared light) caused sudden, violent vibration spikes, disorienting the user.

The Solution: Temporal Smoothing & Kalman Filtering. I implemented a lightweight 1D Kalman filter and a sliding-window temporal average over the depth matrix. Before a haptic signal is fired, the engine evaluates the current frame against the previous 4 frames. This mathematical smoothing eliminated infrared scattering artifacts, resulting in a smooth, radar-like tactile pulse that accurately reflects solid geometry.