Teaching a phone to see: building Seelai

For our thesis we built Seelai, an assistant for partially sighted users. Point the phone, and it tells you what's in front of you. If you need help, it connects you to a caretaker on a video call.

The first rule was that it had to work without internet. Someone crossing a room can't wait for a server. So detection runs on the phone: a YOLOv8n model, converted to TensorFlow Lite, inside a Flutter app.

Ten thousand photos

No public dataset had the ten classes we needed, so we made one: more than 10,000 images, labelled by hand in Label Studio and Roboflow, trained on Google Colab and Kaggle GPUs. Most of the work wasn't the model. It was the labelling, and going back to fix the labels the model kept proving wrong.

The final model reached 95.80% mAP@50 while running at about 30 FPS on the device. On top of detection we added ML Kit to read text aloud, face recognition for familiar people, and WebRTC calls to a caretaker, with Firebase and Node.js behind them.

Seelai is the project I'm proudest of, because the people it's for would actually notice if it failed.