This notebook is a follow up from the previous notebook and youtube video in which we explored the overall evaluation approach and a RAG system's overall accuracy.
In this notebook we will take a closer look at specific RAG evaluation metrics and explore how different RAG components impact these RAG evaluation metrics. If you prefer more of a walkthrough and more background information, please refer to the upcoming youtube video or to the corresponding blog post here.
If you want to deploy the entire solution with IaC, log into your AWS management console. Then click the Launch Stack button below to deploy the required resources.
To avoid incurring additional charges to your account, stop and delete all of the resources by deleting the CloudFormation template at the end of this tutorial.
Amazon Bedrock offers a choice of high-performing foundation models (FMs) from leading AI companies like AI21 Labs, Anthropic, Cohere, Meta, Stability AI, and Amazon. In the Amazon Bedrock console, you can see the descriptions for these foundation models by the Base models link in the navigation pane.
You need to request access to these foundation models before using them. In the Amazon Bedrock console, do the following:
Go to the Amazon Bedrock console. In the navigation pane, select Model access. In the Model access page, you should see a list of base models. Click on the Manage model access button to edit your model access. Select the check box next to all of the following models to get started.
- Titan
- Claude
- Cohere
Click on the Save changes button and it may take several minutes to save the changes. This also brings you back to the Model access page. Models will show as Access granted on the Model access page under the Access status column, if access is granted.
To access VSCode Editor hosted in SageMaker, go to Amazon SageMaker console, select Domains in the navigation menu, and click on the newly created domain "SageMakerDomain".
Launch SageMaker Studio by selecting "Studio" from the dropdown button on the right.
Once in SageMaker Studio, first select CodeEditor in the top left, and then click on the "open" link as shown below.
In CodeEditor, open a terminal and run the following command: git clone https://github.com/fhuthmacher/aws-rag-system-eval
Review dev-rageval.env file and make updates as needed.
In CodeEditor, open Jupyter notebook RAG-System-Eval-DeepDive-mlflow.ipynb and run all cells