10_3_Appendix_Empirical_Performance_Eval.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluating AI Models: Code, Human, and Model-Based Grading\n",
    "\n",
    "In this notebook, we'll delve into a trio of widely-used techniques for assessing the effectiveness of AI models, like Claude v3:\n",
    "\n",
    "1. Code-based grading\n",
    "2. Human grading\n",
    "3. Model-based grading\n",
    "\n",
    "We'll illustrate each approach through examples and examine their respective advantages and limitations, when gauging AI performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code-Based Grading Example: Sentiment Analysis\n",
    "\n",
    "In this example, we'll evaluate Claude's ability to classify the sentiment of movie reviews as positive or negative. We can use code to check if the model's output matches the expected sentiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Import python's built-in regular expression library\n",
    "import re\n",
    "\n",
    "# Import boto3 and json\n",
    "import boto3\n",
    "import json\n",
    "from botocore.exceptions import ClientError\n",
    "# Store the model name and AWS region for later use\n",
    "\n",
    "%store -r modelId\n",
    "%store -r region"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Function to build the input prompt for sentiment analysis\n",
    "def build_input_prompt(review):\n",
    "    user_content = f\"\"\"Classify the sentiment of the following movie review as either 'positive' or 'negative' provide only one of those two choices:\n",
    "    <review>{review}</review>\"\"\"\n",
    "    return [{\"role\": \"user\", \"content\": [{\"text\": user_content}]}]\n",
    "\n",
    "# Define the evaluation data\n",
    "eval = [\n",
    "    {\n",
    "        \"review\": \"This movie was amazing! The acting was superb and the plot kept me engaged from start to finish.\",\n",
    "        \"golden_answer\": \"positive\"\n",
    "    },\n",
    "    {\n",
    "        \"review\": \"I was thoroughly disappointed by this film. The pacing was slow and the characters were one-dimensional.\",\n",
    "        \"golden_answer\": \"negative\"\n",
    "    }\n",
    "]\n",
    "\n",
    "# Function to get completions from the model\n",
    "bedrock_client = boto3.client(service_name='bedrock-runtime', region_name=region)\n",
    "\n",
    "def get_completion(messages, system_prompt=None):\n",
    "    inference_config = {\n",
    "        \"temperature\": 0.5,\n",
    "        \"maxTokens\": 200\n",
    "    }\n",
    "    additional_model_fields = {\n",
    "        \"top_p\": 1\n",
    "    }\n",
    "    converse_api_params = {\n",
    "        \"modelId\": modelId,\n",
    "        \"messages\": messages,\n",
    "        \"inferenceConfig\": inference_config,\n",
    "        \"additionalModelRequestFields\": additional_model_fields\n",
    "    }\n",
    "    if system_prompt:\n",
    "        converse_api_params[\"system\"] = [{\"text\": system_prompt}]\n",
    "    try:\n",
    "        response = bedrock_client.converse(**converse_api_params)\n",
    "        text_content = response['output']['message']['content'][0]['text']\n",
    "        return text_content\n",
    "\n",
    "    except ClientError as err:\n",
    "        message = err.response['Error']['Message']\n",
    "        print(f\"A client error occured: {message}\")\n",
    "\n",
    "# Get completions for each input\n",
    "outputs = [get_completion(build_input_prompt(item[\"review\"])) for item in eval]\n",
    "\n",
    "# Print the outputs and golden answers\n",
    "for output, question in zip(outputs, eval):\n",
    "    print(f\"Review: {question['review']}\\nGolden Answer: {question['golden_answer']}\\nOutput: {output}\\n\")\n",
    "\n",
    "# Function to grade the completions\n",
    "def grade_completion(output, golden_answer):\n",
    "    return output.lower() == golden_answer.lower()\n",
    "\n",
    "# Grade the completions and print the accuracy\n",
    "grades = [grade_completion(output, item[\"golden_answer\"]) for output, item in zip(outputs, eval)]\n",
    "print(f\"Accuracy: {sum(grades) / len(grades) * 100}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Human Grading Example: Essay Scoring\n",
    "\n",
    "Some tasks, like scoring essays, are difficult to evaluate with code alone. In this case, we can provide guidelines for human graders to assess the model's output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Function to build the input prompt for essay generation\n",
    "def build_input_prompt(topic):\n",
    "    user_content = f\"\"\"Write a short essay discussing the following topic:\n",
    "    <topic>{topic}</topic>\"\"\"\n",
    "    return [{\"role\": \"user\", \"content\": [{\"text\": user_content}]}]\n",
    "\n",
    "# Define the evaluation data\n",
    "eval = [\n",
    "    {\n",
    "        \"topic\": \"The importance of education in personal development and societal progress\",\n",
    "        \"golden_answer\": \"A high-scoring essay should have a clear thesis, well-structured paragraphs, and persuasive examples discussing how education contributes to individual growth and broader societal advancement.\"\n",
    "    }\n",
    "]\n",
    "\n",
    "# Get completions for each input\n",
    "outputs = [get_completion(build_input_prompt(item[\"topic\"])) for item in eval]\n",
    "\n",
    "# Print the outputs and golden answers\n",
    "for output, item in zip(outputs, eval):\n",
    "    print(f\"Topic: {item['topic']}\\n\\nGrading Rubric:\\n {item['golden_answer']}\\n\\nModel Output:\\n{output}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model-Based Grading Examples\n",
    "\n",
    "We can use Claude to grade its own outputs by providing the model's response and a grading rubric. This allows us to automate the evaluation of tasks that would typically require human judgment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 1: Summarization\n",
    "\n",
    "In this example, we'll use Claude to assess the quality of a summary it generated. This can be useful when you need to evaluate the model's ability to capture key information from a longer text concisely and accurately. By providing a rubric that outlines the essential points that should be covered, we can automate the grading process and quickly assess the model's performance on summarization tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Function to build the input prompt for summarization\n",
    "def build_input_prompt(text):\n",
    "    user_content = f\"\"\"Please summarize the main points of the following text:\n",
    "    <text>{text}</text>\"\"\"\n",
    "    return [{\"role\": \"user\", \"content\": [{\"text\": user_content}]}]\n",
    "\n",
    "# Function to build the grader prompt for assessing summary quality\n",
    "def build_grader_prompt(output, rubric):\n",
    "    user_content = f\"\"\"Assess the quality of the following summary based on this rubric:\n",
    "    <rubric>{rubric}</rubric>\n",
    "    <summary>{output}</summary>\n",
    "    Provide a score from 1-5, where 1 is poor and 5 is excellent.\"\"\"\n",
    "    return [{\"role\": \"user\", \"content\": [{\"text\": user_content}]}]\n",
    "\n",
    "# Define the evaluation data\n",
    "eval = [\n",
    "    {\n",
    "        \"text\": \"The Magna Carta, signed in 1215, was a pivotal document in English history. It limited the powers of the monarchy and established the principle that everyone, including the king, was subject to the law. This laid the foundation for constitutional governance and the rule of law in England and influenced legal systems worldwide.\",\n",
    "        \"golden_answer\": \"A high-quality summary should concisely capture the key points: 1) The Magna Carta's significance in English history, 2) Its role in limiting monarchical power, 3) Establishing the principle of rule of law, and 4) Its influence on legal systems around the world.\"\n",
    "    }\n",
    "]\n",
    "\n",
    "# Get completions for each input\n",
    "outputs = [get_completion(build_input_prompt(item[\"text\"])) for item in eval]\n",
    "\n",
    "# Grade the completions\n",
    "grades = [get_completion(build_grader_prompt(output, item[\"golden_answer\"])) for output, item in zip(outputs, eval)]\n",
    "\n",
    "# Print the summary quality score\n",
    "print(f\"Summary quality score: {grades[0]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2: Fact-Checking\n",
    "\n",
    "In this example, we'll use Claude to fact-check a claim and then assess the accuracy of its fact-checking. This can be useful when you need to evaluate the model's ability to distinguish between accurate and inaccurate information. By providing a rubric that outlines the key points that should be covered in a correct fact-check, we can automate the grading process and quickly assess the model's performance on fact-checking tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Function to build the input prompt for fact-checking\n",
    "def build_input_prompt(claim):\n",
    "    user_content = f\"\"\"Determine if the following claim is true or false:\n",
    "    <claim>{claim}</claim>\"\"\"\n",
    "    return [{\"role\": \"user\", \"content\": [{\"text\": user_content}]}]\n",
    "\n",
    "# Function to build the grader prompt for assessing fact-check accuracy\n",
    "def build_grader_prompt(output, rubric):\n",
    "    user_content = f\"\"\"Based on the following rubric, assess whether the fact-check is correct:\n",
    "    <rubric>{rubric}</rubric>\n",
    "    <fact-check>{output}</fact-check>\"\"\"\n",
    "    return [{\"role\": \"user\", \"content\": [{\"text\": user_content}]}]\n",
    "\n",
    "# Define the evaluation data\n",
    "eval = [\n",
    "    {\n",
    "        \"claim\": \"The Great Wall of China is visible from space.\",\n",
    "        \"golden_answer\": \"A correct fact-check should state that this claim is false. While the Great Wall is an impressive structure, it is not visible from space with the naked eye.\"\n",
    "    }\n",
    "]\n",
    "\n",
    "# Get completions for each input\n",
    "outputs = [get_completion(build_input_prompt(item[\"claim\"])) for item in eval]\n",
    "\n",
    "grades = []\n",
    "for output, item in zip(outputs, eval):\n",
    "    # Print the claim, fact-check, and rubric\n",
    "    print(f\"Claim: {item['claim']}\\n\")\n",
    "    print(f\"Fact-check: {output}]\\n\")\n",
    "    print(f\"Rubric: {item['golden_answer']}\\n\")\n",
    "    \n",
    "    # Grade the fact-check\n",
    "    grader_prompt = build_grader_prompt(output, item[\"golden_answer\"])\n",
    "    grade = get_completion(grader_prompt)\n",
    "    grades.append(\"correct\" in grade.lower())\n",
    "\n",
    "# Print the fact-checking accuracy\n",
    "accuracy = sum(grades) / len(grades)\n",
    "print(f\"Fact-checking accuracy: {accuracy * 100}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 3: Tone Analysis\n",
    "\n",
    "In this example, we'll use Claude to analyze the tone of a given text and then assess the accuracy of its analysis. This can be useful when you need to evaluate the model's ability to identify and interpret the emotional content and attitudes expressed in a piece of text. By providing a rubric that outlines the key aspects of tone that should be identified, we can automate the grading process and quickly assess the model's performance on tone analysis tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Function to build the input prompt for tone analysis\n",
    "def build_input_prompt(text):\n",
    "    user_content = f\"\"\"Analyze the tone of the following text:\n",
    "    <text>{text}</text>\"\"\"\n",
    "    return [{\"role\": \"user\", \"content\": [{\"text\": user_content}]}]\n",
    "\n",
    "# Function to build the grader prompt for assessing tone analysis accuracy\n",
    "def build_grader_prompt(output, rubric):\n",
    "    user_content = f\"\"\"Assess the accuracy of the following tone analysis based on this rubric:\n",
    "    <rubric>{rubric}</rubric>\n",
    "    <tone-analysis>{output}</tone-analysis>\"\"\"\n",
    "    return [{\"role\": \"user\", \"content\": [{\"text\": user_content}]}]\n",
    "\n",
    "# Define the evaluation data\n",
    "eval = [\n",
    "    {\n",
    "        \"text\": \"I can't believe they canceled the event at the last minute. This is completely unacceptable and unprofessional!\",\n",
    "        \"golden_answer\": \"The tone analysis should identify the text as expressing frustration, anger, and disappointment. Key words like 'can't believe', 'last minute', 'unacceptable', and 'unprofessional' indicate strong negative emotions.\"\n",
    "    }\n",
    "]\n",
    "\n",
    "# Get completions for each input\n",
    "outputs = [get_completion(build_input_prompt(item[\"text\"])) for item in eval]\n",
    "\n",
    "# Grade the completions\n",
    "grades = [get_completion(build_grader_prompt(output, item[\"golden_answer\"])) for output, item in zip(outputs, eval)]\n",
    "\n",
    "# Print the tone analysis quality\n",
    "print(f\"Tone analysis quality: {grades[0]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These examples demonstrate how code-based, human, and model-based grading can be used to evaluate AI models like Claude on various tasks. The choice of evaluation method depends on the nature of the task and the resources available. Model-based grading offers a promising approach for automating the assessment of complex tasks that would otherwise require human judgment."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "",
   "name": ""
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}