Nutrition Label Agent
Habits · LangChain, LangGraph, Gemini Vision, Tavily, FastAPI, Pydantic
Problem to Solve
This microservice lets Habits app users upload a photo of a nutrition label (packaging or nutrition facts panel) and get in seconds: product name and main ingredients, NOVA classification (1–4), whether it is ultra-processed, a health score (1–10), a brief analysis, and—if ultra-processed—a suggested healthier alternative via web search.
Users can make better food choices and replace ultra-processed products with better options. The agent is implemented as a LangGraph workflow with vision (Gemini) and search (Tavily), exposed via FastAPI.
1. How does it work?
The user uploads an image from the Nutrition section; the frontend sends it to the
microservice via POST /analyze-label. The agent runs a LangGraph graph with analyzer, optional
searcher, and finalizer nodes.
| Step | Component | Description |
|---|---|---|
| 1 | Analyzer | Image sent to Gemini (vision); JSON with product, NOVA, ingredients; validated with Pydantic
(AnalysisResult) |
| 2 | Decision | If ultra-processed → Searcher; else → Finalizer |
| 3 | Searcher | Tavily searches healthier alternatives; Gemini extracts one suggested food name |
| 4 | Finalizer | Builds final report (product, NOVA, score, analysis, alternative, warnings); validated with Pydantic
(NutritionalResponse) |
The flow is orchestrated by LangGraph (nodes and conditional edges); LangChain connects to Gemini and Tavily. The API returns the JSON to the frontend for display.
2. Implementation
The service is built with FastAPI, LangGraph for the agent flow, Gemini Vision for label analysis, Tavily for alternatives search, and Pydantic for validation.
2.1 API
FastAPI with a single analysis endpoint (POST /analyze-label) and health check.
The image is sent as multipart/form-data, converted to base64, and injected into the graph's
initial state.
2.2 Graph
Defined in graph.py with LangGraph (StateGraph): nodes
analyzer, searcher, finalizer; entry at analyzer;
conditional edge based on is_ultra_processed; state passed from node to node (image, analysis,
search results, final report).
2.3 Nodes
In nodes.py: analyzer_node calls Gemini (multimodal: prompt +
base64 image), parses JSON and validates with Pydantic; searcher_node uses
Tavily (LangChain) and optionally Gemini to extract the alternative's name;
finalizer_node builds the report and validates it with Pydantic.
2.4 Models & Deployment
In models.py, Pydantic defines AnalysisResult and
NutritionalResponse. Standalone Python service (Docker), configured with
GOOGLE_API_KEY and TAVILY_API_KEY; frontend uses
VITE_LABEL_ANALYZER_API_URL.
Key Concepts
LangGraph
Orchestrates the agent as a state graph with conditional edges (analyzer → searcher or finalizer).
Gemini Vision
Multimodal model that reads the label image and returns structured JSON (product, NOVA, ingredients).
Tavily
Web search used to find healthier alternatives when the product is classified as ultra-processed.
Pydantic
Validates analyzer output (AnalysisResult) and API response
(NutritionalResponse).