Kedro
A toolbox for production-ready data science, enabling reproducible, maintainable, and modular pipelines.
- Source repo
- kedro-org/kedro
- Stars
- ★ 11k
- Last updated
- 3d ago
- License
- NOASSERTION
- Primary language
- Python
- FA score
- 53/100 · Major gaps
At a glance
- How it runs
- Works with
- Universal · cross-platform
- Cost
- Free, no paid service needed
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- Data engineers needing to build reproducible, maintainable data processing pipelines with multiple data sources and sinks.
- Not a fit if
- Solo users who want to stay in notebooks or one-off scripts
- Users who need a GUI and avoid the terminal and Python virtual environments
- Tiny scripts where the project template and layered abstraction are overkill
- Source review
- 53/100 · Major gaps
What does this agent do, and when should you use it?
Kedro is an open-source Python framework hosted by the LF AI & Data Foundation that applies software engineering best practices to data science. It provides a standard project template based on Cookiecutter Data Science, a Data Catalog for saving/loading data across many file formats and file systems (including local, cloud, and HDFS) with versioning, and a pipeline abstraction that automatically resolves dependencies between Python functions, with visualization via Kedro-Viz. It promotes coding standards (pytest, Sphinx, ruff, logging) and supports flexible deployment on platforms like Argo, Prefect, Kubeflow, AWS Batch, and Databricks. Kedro helps teams build maintainable, reproducible data pipelines from development to production.
Kedro uses a CLI (e.g., kedro new to scaffold a project) and a Python API to build data pipelines. Users define nodes (pure Python functions) and a Data Catalog (lightweight connectors) to declaratively assemble pipelines. During execution, Kedro resolves dependencies automatically, runs nodes in order, and loads/saves data to and from local filesystems, cloud object stores, HDFS, and more, with optional versioning. It supports test-driven development (pytest), documentation generation (Sphinx), linting (ruff), and logging. It provides pipeline visualization through Kedro-Viz and can deploy to single or distributed machines, including Argo, Prefect, Kubeflow, AWS Batch, and Databricks.
- Data engineers needing to build reproducible, maintainable data processing pipelines with multiple data sources and sinks.
- Machine learning engineers who want to modularize experiment code, reuse data loading/preprocessing, and manage versions.
- Teams that need a standardized project structure to facilitate collaboration between members with varying software engineering expertise.
- Organizations that need to deploy data pipelines to distributed platforms like Kubeflow or AWS Batch for production.
- Beginners looking to follow best practices for organizing data science projects and avoid messy code.
How do you install or deploy this agent?
Install via pip: uv pip install kedro, or via conda: conda install -c conda-forge kedro. Requires Python 3.10 or higher. For the latest development version, install from source: uv pip install git+https://github.com/kedro-org/kedro@main. Full installation instructions are in the Get Started guide.
How do you use this agent?
After installation, run kedro new to create a new project, following prompts for project name etc. The project structure includes src, conf, data directories. Define data catalog entries (in conf/base/catalog.yml) and node functions (in src/<package>/nodes), then assemble pipelines in pipeline.py using the Python API. Run kedro run to execute the pipeline. Refer to the official spaceflights tutorial for a hands-on guide and use Kedro-Viz to visualize.
What are this agent's strengths and limitations?
- Provides a standardized project template to bootstart projects consistently.
- Built-in Data Catalog and versioning supports multiple storage backends (local, cloud, HDFS).
- Automatic dependency resolution and visualization help understand data flow.
- Hosted by LF AI & Data Foundation, with an active community and Slack support.
- Supports multiple deployment platforms, enabling scaling to distributed environments.
- Steep learning curve due to layered abstraction (nodes, catalog, pipelines).
- May be heavyweight for simple small projects, adding setup overhead.
- Some advanced features (e.g., specific dataset plugins) require extra installation and maintenance.
- Migrating existing scripts or notebooks requires refactoring into Kedro's structure.
How does this agent compare with similar options?
Compared to general-purpose workflow orchestration tools like Airflow, Kedro focuses more on data science project structure and engineering, offering a project template and data catalog.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| Kedro This agent | 53 · Major gaps | CLIFree | ★ 11k | 3d ago | Python | — |
| NeMo Data Designer | 49 · Major gaps | Library / SDKFree + model costs | ★ 2.3k | 2d ago | Python | OpenAI API · Claude API |
| OpenBB Open Data Platform | 37 · Major gaps | Library / SDKFreemium | ★ 74k | today | Python | — |
| Wisp Science | 71 · Some gaps | Desktop appFree + model costs | ★ 1k | today | Rust | Codex · Claude Code · OpenAI API · Claude API |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence shows a security policy (SECURITY.md) and dependency scanning (detect-secrets), but least privilege principle or user confirmation mechanisms are not explicit. Data flow transparency is limited, and sensitive data handling is not detailed. External effects (e.g., telemetry) exist but user consent is not explicit. Rollback mechanisms are not mentioned. Source attribution is clear (Apache-2.0 license). Deductions: lack of detailed evidence for permission controls, user confirmation, and sensitive data handling.
The project has extensive tests (unit, e2e) and CI configuration, indicating good self-consistency. Dependency versions are constrained, but no guarantee of dependency availability. Failure messages are not detailed in documentation. Deduction: insufficient evidence for failure message handling.
Target audience is clear (data scientists, engineers), with diverse scenarios (tutorials, deployment). Capability boundaries are described in docs, but trigger precision (e.g., CLI commands) is not detailed. Environment fit is good (supports multiple Python versions and deployment platforms). Deduction: insufficient evidence for trigger precision.
Information architecture is clear (README, docs, API reference). Installation notes are detailed. Naming is stable (versioned). Examples and FAQ are abundant. Known limitations are not explicitly listed. License is complete (Apache-2.0). Versioning changelog exists (via releases). Maintenance responsibility is clear (product team and contributors). Deduction: known limitations not explicit.
Output usability is high (docs, tutorials, visualization). Marginal value is evident (addresses shortcomings of notebooks and scripts). Cost-benefit is reasonable (open-source free). Deduction: none significant.
Claims are supported by docs and code, but lack cross-source corroboration. Fact-inference separation is not clear. Deductions: insufficient evidence for cross-source corroboration and fact-inference separation.
- Publisher identity is unverified; treat as unknown and do not infer safety from brand.
- Static review cannot verify runtime behavior; all conclusions are based on file evidence with low confidence.
- Dependencies include telemetry (kedro-telemetry); verify data collection and user consent mechanisms.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
Release notes excerpt
Major features and improvements - Added an optional params query parameter to GET /snapshot for resolving ${runtime_params:...} interpolation per request, using the same format as kedro run --params. - Graduated llm_context_node, LLMContextNode, LLMContext and tool from experimental to stable. They no longer emit a KedroExperimentalWarning. Breaking changes to the API …View on GitHub ↗ -
Release notes excerpt
Major features and improvements - Added dataset validation: datasets can declare a validator in their catalog entry, and the DataCatalog enforces it on every load and save. - Failures raise DataValidationError (a DatasetError subclass), reporting every failed check at once. - Pandera is the built-in adapter, installed with the new kedro[pandera-pandas] or kedro[pandera-polars] extras; custom validator classes and functions also work. - The DATASET_VALIDATION setting and the KEDRO_DATASET_VALIDATION environment variable switch validation off. …View on GitHub ↗ -
Release notes excerpt
Major features and improvements * Added a new HTTP Server layer which uses the KedroServiceSession to execute pipelines from HTTP requests. It offers the following endpoints: * /run to run pipelines with optional runtime parameters. * /health to check the server's health status. * /snapshot to retrieve the current project snapshot. * Added a new CLI command kedro server start to run the server. * Added selective pipeline loading: when --pipelines is specified, only the requested pipeline modules are imported. Bug fixes and other changes …View on GitHub ↗