Email Agents From Scratch
Build an email assistant with triage, approval checkpoints, evaluation, and persistent memory.
The evidence describes human review for high-impact tool calls such as sending email and scheduling meetings, while the shown workflow permissions are mostly read-only, supporting partial credit for confirmation and external-effect control. The README identifies the principal OpenAI, LangSmith, Gmail, tracing, and persistent-memory data paths, and the license and copyright notice provide source attribution. Deductions apply because Gmail scopes, least-privilege design, transmission and retention of email or memory content, sensitive-data sanitation, and rollback or recovery for completed actions are not documented. Dependencies use broad lower bounds and beta workflow actions without a lockfile, vulnerability scanning, or mitigation evidence. Human review is described but cannot be checked against implementation code in the supplied material. Unknown publisher identity is treated neutrally and earns no brand-based credit.
The README, project configuration, and tests identify Python requirements, installation paths, external services, and test entry points, but support only limited static reliability credit. Deductions follow from inconsistent package naming, a conftest default selecting the memory implementation while the runner permits only the basic implementation, and explicit exclusion of HITL and memory variants because the Question tool and resume command can cause an indefinite loop. Operation depends on external APIs, credentials, and LangSmith without offline or degradation guidance. Failure messages are mainly limited to unknown test implementations and notebook execution failures; product-level runtime recovery and actionable errors are not demonstrated.
The material addresses learners and email-assistant builders through four stages—basic agent, evaluation, human intervention, and memory—and distinguishes mock tools, Gmail integration, deployment, and future LangMem work. This gives reasonable scenario and boundary coverage. Deductions apply because production scale, privacy-constrained environments, model substitution, and concrete migration beyond email are not addressed. Trigger behavior is described only as triage and review of “specific” tool calls, without classification thresholds, allowlists, approval policies, or false-trigger handling. Python 3.11–3.13, uv/pip, and environment-variable guidance support ordinary environments, but mandatory external-service dependencies limit full environment fit.
The README has strong information architecture, staged navigation, notebook and code pointers, environment setup, two installation paths, test commands, and Gmail deployment guidance. The full MIT license agrees with the metadata. Deductions apply because the project name, README-stated installed package name, and import name are inconsistent or unstable. Examples are plentiful, but there is no substantive FAQ or troubleshooting section. Some testing limitations and future work are disclosed, though important limitations reside mainly in test-code comments. Only version 0.1.0 is shown, with no changelog, release policy, or compatibility commitment. A copyright owner is named, but verified maintainers, support routes, and update responsibility are not established.
As an instructional repository, the staged notebooks, companion modules, evaluation material, human approval, and memory design provide a useful artifact with clear marginal value over a single minimal agent example. Deductions apply because the supplied evidence does not show a final output contract, production deployment outcome, or usable complete HITL and memory implementation; the test runner covers only the basic assistant. OpenAI, LangSmith, Gmail, and possible hosted deployment introduce credential, service, and runtime costs, but no estimates, rate-limit guidance, resource requirements, or lower-cost alternatives are provided.
The README maps major claims to specific notebooks, source paths, tests, and configuration, while pyproject and test files corroborate the Python range, dependencies, test approach, and implementation names. Future extensions are clearly labeled, and test comments candidly separate testable from currently untestable variants. Deductions apply because the central agent source, Gmail-tool instructions, notebook bodies, and actual test results are absent from the supplied evidence. Claims such as deployability, automated-suite coverage, and HITL or memory behavior are therefore traceable only to references, not fully inspectable here, and no independent source corroboration is present.
- The HITL and memory variants are not covered by the current runner; test comments explicitly record that the Question tool can make the resume flow loop indefinitely.
- Before connecting a real Gmail account, verify OAuth scopes, per-action approval coverage for sending and calendar changes, and compensation or reversal procedures for completed actions.
- Email content, feedback, long-term memories, and LangSmith traces may contain sensitive data; minimization, redaction, retention, and deletion procedures are not documented.
- Dependencies are not precisely pinned, and GitHub workflows use beta third-party actions; lock versions and perform supply-chain and vulnerability review before deployment.
- Do not treat README statements about testing or deployment as verified outcomes; this assessment did not execute code, notebooks, or external APIs.
What does this agent do, and when should you use it?
This repository is a staged developer guide and reference implementation that culminates in an ambient email assistant connected to the Gmail API. It pairs four notebooks with Python implementations under `src/email_assistant`, progressing through basic agent construction, evaluation, human-in-the-loop control, and memory. The assistant combines an email-triage step with an agent that handles responses, while selected actions such as sending email or scheduling meetings can be submitted for user review. Its memory-enabled implementation persists feedback and preferences with LangGraph Store, and the complete Gmail variant lives in `email_assistant_hitl_memory_gmail.py`. The repository also demonstrates evaluation with Pytest, the LangSmith `evaluate` API, and LLM-as-a-judge; it mentions deployment to LangGraph Platform, although the supplied material does not include the deployment commands.
The base implementation in src/email_assistant/email_assistant.py combines email triage with an agent responsible for handling the response. notebooks/evaluation.ipynb evaluates the assistant against eval/email_dataset.py using Pytest and the LangSmith evaluate API, covering response quality, tool calls, and triage decisions; response evaluation includes LLM-as-a-judge. src/email_assistant/email_assistant_hitl.py adds review checkpoints through Agent Inbox for selected calls such as sending an email or scheduling a meeting. src/email_assistant/email_assistant_hitl_memory.py uses LangGraph Store to persist memories so the assistant can learn from feedback and adapt to preferences. The notebooks operate with mock email and calendar tools, while src/email_assistant/email_assistant_hitl_memory_gmail.py contains the Gmail API implementation. Test traces and metrics are logged to the LangSmith project selected by LANGSMITH_PROJECT.
- A Python developer learning LangGraph can follow the notebooks to study chat models, tool calling, nodes, edges, memory, and the distinction between agents and workflows.
- An engineering team prototyping email automation can use the triage, response-handling, and Gmail integration code as a reference implementation.
- A team that must supervise consequential actions can study the Agent Inbox flow for reviewing email-sending and meeting-scheduling calls.
- A developer building a preference-aware assistant can examine how the project persists user feedback and preferences with LangGraph Store.
- An evaluation engineer can adapt the email dataset, Pytest suite, LangSmith
evaluatecalls, and checks for responses, tool use, and triage decisions.
What are this agent's strengths and limitations?
- The four-stage progression pairs notebooks with corresponding Python implementations, connecting conceptual instruction to inspectable code.
- Human oversight is implemented through Agent Inbox for concrete high-impact calls such as sending email and scheduling meetings.
- The memory variant persists feedback and preferences with LangGraph Store instead of limiting memory to one conversation.
- Evaluation covers response quality, tool calls, and triage decisions through Pytest, LangSmith
evaluate, and LLM-as-a-judge. - Mock email and calendar tools support early experimentation, while a separate implementation demonstrates real Gmail API integration.
- The documented setup depends on Python 3.11+, an OpenAI API key, and a LangSmith API key, tying execution and evaluation to named services.
- Gmail support requires separate Google API credentials, while the supplied material omits the exact authorization, permission, and deployment steps.
- The listed test implementations include only
email_assistant; the source does not establish equivalent automated coverage for every HITL, memory, or Gmail variant. - No single interactive launch command, production runtime configuration, or failure-recovery behavior is documented in the supplied material.
- LangMem-based background memory management and memory lookup tools are identified only as future extensions, not current capabilities.
How do you install or deploy this agent?
Python 3.11 or later is required. In the repository root, run cp .env.example .env, then configure LANGSMITH_API_KEY=your_langsmith_api_key, LANGSMITH_TRACING=true, LANGSMITH_PROJECT="interrupt-workshop", and OPENAI_API_KEY=your_openai_api_key. The recommended installation is pip install uv, followed by uv sync --extra dev and source .venv/bin/activate. The documented pip alternative is python3 -m venv .venv, source .venv/bin/activate, python3 -m pip install --upgrade pip, and pip install -e .. The editable installation is required for the notebooks; the distribution is named interrupt_workshop and its import name is email_assistant. Gmail use additionally requires Google API credentials configured according to src/email_assistant/tools/gmail/README.md, but those credential steps are not included in the supplied material.
How do you use this agent?
Work through notebooks/agent.ipynb, notebooks/evaluation.ipynb, notebooks/hitl.ipynb, and notebooks/memory.ipynb; use notebooks/langgraph_101.ipynb first if an introduction to the underlying concepts is needed. Installed code can be imported with from email_assistant import .... The base, human-review, memory, and complete Gmail implementations are in email_assistant.py, email_assistant_hitl.py, email_assistant_hitl_memory.py, and email_assistant_hitl_memory_gmail.py, respectively. Run the automated suite with python tests/run_all_tests.py. Validate all notebooks with python tests/test_notebooks.py or pytest tests/test_notebooks.py -v. The supplied material does not document one complete command for launching an interactive assistant or deploying it to LangGraph Platform, so a copyable first production invocation cannot be established from this source alone.
How does this agent compare with similar options?
Compared with a fixed workflow, the repository combines a defined email-triage step with an agent that handles responses; langgraph_101.ipynb explicitly introduces agents versus workflows. It does not provide an evidence-based comparison with another named email assistant or agent framework.
FAQ
Can I try it without connecting a real Gmail account?
Which actions can require human approval?
Which external credentials are required?
How is assistant quality tested?
evaluate API to assess responses, tool calls, and triage decisions. Response evaluation includes LLM-as-a-judge.