-
Notifications
You must be signed in to change notification settings - Fork 4
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Update README.md - Update code sample - Update docs links --------- Co-authored-by: Siddhant Sadangi <[email protected]>
- Loading branch information
1 parent
3269d07
commit f6282dc
Showing
1 changed file
with
41 additions
and
29 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| # Neptune + Kedro Integration | ||
| # Neptune + Kedro integration | ||
|
|
||
| Kedro plugin for experiment tracking and metadata management. It lets you browse, filter and sort runs in a nice UI, visualize node/pipeline metadata, and compare pipelines. | ||
|
|
||
|
|
@@ -10,61 +10,73 @@ Kedro plugin for experiment tracking and metadata management. It lets you browse | |
| * and do whatever else you would expect from a modern ML metadata store | ||
|
|
||
|  | ||
| *Kedro pipeline metadata in custom dashboard in the Neptune UI* | ||
| *Kedro pipeline metadata in custom dashboard in the Neptune web app* | ||
|
|
||
| Note: Kedro-Neptune plugin supports distributed pipeline execution and works in Kedro setups that use orchestrators like Airflow or Kubeflow. | ||
| Note: The Kedro-Neptune plugin supports distributed pipeline execution and works in Kedro setups that use orchestrators like Airflow or Kubeflow. | ||
|
|
||
| ## Resources | ||
|
|
||
| * [Documentation](https://docs.neptune.ai/integrations-and-supported-tools/automation-pipelines/kedro) | ||
| * [Code example on GitHub](https://github.com/neptune-ai/examples/blob/main/integrations-and-supported-tools/kedro/scripts/kedro_neptune_quickstart) | ||
| * [Runs logged in the Neptune app](https://app.neptune.ai/o/common/org/kedro-integration/e/KED-632/dashboard/Basic-pipeline-metadata-42874940-da74-4cdc-94a4-315a7cdfbfa8) | ||
| * How to [Compare Kedro pipelines](https://docs.neptune.ai/integrations-and-supported-tools/automation-pipelines/kedro/compare-kedro-pipelines) | ||
| * How to [Compare results between Kedro nodes](https://docs.neptune.ai/integrations-and-supported-tools/automation-pipelines/kedro/compare-results-between-kedro-nodes) | ||
| * How to [Display Kedro node metadata and outputs](https://docs.neptune.ai/integrations-and-supported-tools/automation-pipelines/kedro/display-kedro-node-metadata-and-outputs) | ||
| * [Documentation](https://docs.neptune.ai/integrations/kedro) | ||
| * [Code example on GitHub](https://github.com/neptune-ai/examples/tree/main/integrations-and-supported-tools/kedro/scripts/kedro-neptune-quickstart) | ||
| * [Example run logged in the Neptune app](https://app.neptune.ai/o/common/org/kedro-integration/e/KED-1563/dashboard/Basic-pipeline-metadata-42874940-da74-4cdc-94a4-315a7cdfbfa8) | ||
| * How to [Compare Kedro pipelines](https://docs.neptune.ai/integrations/kedro_comparing_pipelines/) | ||
| * How to [Compare results between Kedro nodes](https://docs.neptune.ai/integrations/kedro_comparing_nodes/) | ||
| * How to [Display Kedro node metadata and outputs](https://docs.neptune.ai/integrations/kedro_displaying_node_outputs/) | ||
|
|
||
| ## Example | ||
|
|
||
| ```python | ||
| # On the command line: | ||
| pip install neptune-client kedro kedro-neptune | ||
| On the command line: | ||
|
|
||
| ``` | ||
| pip install kedro neptune[kedro] | ||
| kedro new --starter=pandas-iris | ||
| ``` | ||
|
|
||
| # In your Kedro project directory: | ||
| In your Kedro project directory: | ||
|
|
||
| ``` | ||
| kedro neptune init | ||
| ``` | ||
|
|
||
| In a pipeline node, in `nodes.py`: | ||
|
|
||
| ```python | ||
| # In a pipeline node, in nodes.py: | ||
| import neptune.new as neptune | ||
| import neptune | ||
|
|
||
| # Add a Neptune run handler to the report_accuracy() function | ||
| # and log metrics to neptune_run | ||
| def report_accuracy(predictions: np.ndarray, test_y: pd.DataFrame, | ||
| neptune_run: neptune.run.Handler) -> None: | ||
| target = np.argmax(test_y.to_numpy(), axis=1) | ||
| accuracy = np.sum(predictions == target) / target.shape[0] | ||
|
|
||
| def report_accuracy( | ||
| y_pred: pd.Series, | ||
| y_test: pd.Series, | ||
| neptune_run: neptune.handler.Handler, | ||
| ) -> None: | ||
| accuracy = (y_pred == y_test).sum() / len(y_test) | ||
| logger = logging.getLogger(__name__) | ||
| logger.info("Model has accuracy of %.3f on test data.", accuracy) | ||
|
|
||
| # Log metrics to the Neptune run | ||
| neptune_run["nodes/report/accuracy"] = accuracy * 100 | ||
|
|
||
| # Add the Neptune run handler to the Kedro pipeline | ||
| node( | ||
| report_accuracy, | ||
| ["example_predictions", "example_test_y", "neptune_run"], | ||
| None, | ||
| name="report") | ||
| ``` | ||
| ```python | ||
| # On the command line, run the Kedro pipeline | ||
| kedro run | ||
| func=report_accuracy, | ||
| inputs=["y_pred", "y_test", "neptune_run"], | ||
| outputs=None, | ||
| name="report_accuracy", | ||
| ) | ||
| ``` | ||
|
|
||
| On the command line, run the Kedro pipeline: | ||
|
|
||
| ``` | ||
| kedro run | ||
| ``` | ||
|
|
||
| ## Support | ||
|
|
||
| If you got stuck or simply want to talk to us, here are your options: | ||
|
|
||
| * Check our [FAQ page](https://docs.neptune.ai/getting-started/getting-help#frequently-asked-questions) | ||
| * Check our [FAQ page](https://docs.neptune.ai/getting_help) | ||
| * You can submit bug reports, feature requests, or contributions directly to the repository. | ||
| * Chat! When in the Neptune application click on the blue message icon in the bottom-right corner and send a message. A real person will talk to you ASAP (typically very ASAP), | ||
| * You can just shoot us an email at [email protected] | ||