When you point your camera at a plate of food, a lot has to happen in the couple of seconds before your macros show up. This post is a plain-English walkthrough of what the model is actually doing, what it's good at, and the specific places it still misses — so you know when to trust the estimate and when to adjust.
The pipeline in three steps
1. Identify the foods. The image is passed to a vision model that labels what's on the plate. "Grilled chicken breast. Jasmine rice. Steamed broccoli." It can pick out three to five items on a single plate reliably; dense mixed dishes (like a stir-fry) compress into fewer labels.
2. Estimate portion size. This is the hard part. The model looks at relative area, depth cues from the plate rim, and typical serving size for the food in question. It converts pixels to grams — which, on a photo taken from above, with no known reference, is fundamentally a guess. The model errs on the side of "average serving," and you can edit the portion before you save.
3. Compute macros. Once the food and portion are locked in, macros are a database lookup: grams → protein, carbs, fat, calories. This step is boring and accurate. All of the interesting error lives in step 2.
What it's genuinely good at
- Single plates, from above, in reasonable light. The "phone over the dinner table" shot is the format the model is trained on most.
- Common restaurant and home-cooked dishes. Chicken bowls, pasta plates, burgers, salads — things that look like what they are.
- Multiple items per plate. The detection will typically catch 3–5 separate foods without you having to tap-select.
- Side-by-side portions. Two tacos look like two tacos; it counts them.
Where it trips up
- Dense ambiguous food. A ground-beef chili and a lentil stew look very similar through a lens. Manual search is faster there.
- Liquid measurements. Oils, dressings, and sauces have wildly different caloric density per volume. If a dish is swimming in something, bump the portion up before you save.
- Empty-calorie density. A croissant and a bagel are roughly the same shape; their macros aren't. The model is aware of this class of ambiguity but can't resolve it from a photo alone.
- Unusual angles. Low-angle shots and close-ups lose the size reference. Top-down is best.
What we do, step by step
- Resize and normalize. We resize the photo to a standard resolution so the model sees a consistent input.
- Detection pass. The image goes to OpenAI's vision API with instructions to return structured output — food names and portions in grams. Only the image is sent: no name, email, or diary history travels with it, and per OpenAI's API terms it is not used to train their models.
- Uncertainty check. If an item comes back with low confidence or zero calories, the app flags it before you save so you can review it — or swap it for a manual search entry. A wrong log is worse than one you had to type.
- Your correction is final. When you adjust a portion or a food name before saving, that is what lands in your diary — the estimate is a starting point, not the record.
The trade-off in one sentence
Photo logging turns 90 seconds of typing into five seconds of aiming, at the cost of some portion-estimation slop you can fix by editing the portion before you save. For most meals on most days, that's the right trade.
When to skip the camera
- You're logging something you eat identically every day — make it a custom food and one-tap it in.
- The food is packaged with a nutrition label — searching the database (or saving it once as a custom food) will always be more accurate.
- You know the exact weight you cooked — enter it manually.
The camera is an accelerator for the messy middle, not a replacement for the other inputs. Use whichever is fastest for the meal in front of you.