diff --git a/versioned_docs/version-2.0/tutorials/optimize_classifier.mdx b/versioned_docs/version-2.0/tutorials/optimize_classifier.mdx index 4d40267c..41966a44 100644 --- a/versioned_docs/version-2.0/tutorials/optimize_classifier.mdx +++ b/versioned_docs/version-2.0/tutorials/optimize_classifier.mdx @@ -140,6 +140,9 @@ as the gold-truth output of the datapoint, it will use the correction. ![Optimization Negative](./static/class-optimization-neg.png) +Note that these automations only start working for future runs. +So they will not pick up the runs we previously kicked off above - we will need redo those. + ## Update the application We can now update our code to pull down the dataset we are sending runs to. diff --git a/versioned_docs/version-2.0/tutorials/personlization.mdx b/versioned_docs/version-2.0/tutorials/personlization.mdx new file mode 100644 index 00000000..772702d3 --- /dev/null +++ b/versioned_docs/version-2.0/tutorials/personlization.mdx @@ -0,0 +1,275 @@ +--- +sidebar_label: User-level personalization +sidebar_position: 4 +--- + +# User-level personalization + +This tutorial walks through how to use LangSmith to help build an LLM powered application that is personalized for each user. +Personalizing an application for each user can make the experience feel more unique and "correct" for each user, +increasing adoption and retention. +There are many forms of personalization. +Here we focus on one that uses feedback on previous interactions to improve and personalize its responses over time. +It is very similar to the [optimization](./optimize_classifier) tutorial, but rather than optimizing a system globally it does it for each user individually. + +## The objective + +In this example, we will build a bot that generates tweets based on a topic. +It will take in a topic and generate a tweet. +Then, we will start to collect user feedback and use that to shape how this bot performs. + +## Getting started + +To get started, we will first set it up so that we send all traces to a specific project. +We can do this by setting an environment variable: + +```python +import os +os.environ["LANGCHAIN_PROJECT"] = "tweeter" +``` + +We can then create our initial application. +This will be a really simple function that just takes in a topic and generates a tweet. + +```python +import openai +from langsmith import traceable, Client +import uuid + +client = openai.Client() +ls_client = Client() + + +prompt_template = """Write a tweet about the following topic: + +Topic: {topic}""" + +@traceable( + run_type="chain", + name="Tweeter", +) +def tweeter( + topic: str +): + return client.chat.completions.create( + model="gpt-3.5-turbo", + temperature=0, + messages=[ + { + "role": "user", + "content": prompt_template.format( + topic=topic, + ) + } + ], +``` + +We can then start to interact with it. +When interacting with it, we will generate the LangSmith run id ahead of time and pass that into this function. +We do this so we can attach feedback later on. +We will also pass in a user ID so we can know which generations came from which user. + +Here's how we can invoke the application: + +```python +run_id = uuid.uuid4() +tweeter( + "nba finals", + langsmith_extra={"run_id": run_id, "metadata": {"user": "foo"}} +) +``` + +Here's how we can attach feedback after. +We can collect feedback in two forms. + +First, we can collect "positive" feedback - this is for examples that the model got right. + +```python +run_id = uuid.uuid4() + +tweeter( + "nba finals", + langsmith_extra={"run_id": run_id, "metadata": {"user": "foo"}} +) + +ls_client.create_feedback( + run_id, + key="user-score", + score=1.0, +) +``` + +Next, we can focus on collecting feedback that corresponds to a "correction" to the generation. +These are super valuable pieces of feedback to collect, because they identify places where the bot could improve. + +```python +run_id = uuid.uuid4() +tweeter( + "nba finals", + langsmith_extra={"run_id": run_id, "metadata": {"user": "foo"}} +) + +ls_client.create_feedback( + run_id, + key="correction", + correction="nba finals... what a snooze fest... nuggets too good, no chance they lose" +) +``` + +## Set up automations + +We can now set up automations to move examples with feedback of some form into a dataset. +We will set up two automations, one for positive feedback and the other for negative feedback. + +The first will take all runs with positive feedback and automatically add them to a dataset. +The logic behind this is that any run with positive feedback we can use as a good example in future iterations. +Let's create a dataset called `tweeter` to add this data to. + +![Optimization Positive](./static/tweeter-pos.png) + +The second will take all runs with a correction and use a webhook to add them to a dataset. +When creating this webhook, we will select the option to "Use Corrections". +This option will make it so that when creating a dataset from a run, rather than using the output of the run +as the gold-truth output of the datapoint, it will use the correction. + +![Optimization Negative](./static/tweeter-neg.png) + + +Note that these automations only start working for future runs. +So they will not pick up the runs we previously kicked off above - we will need redo those. +After kicking them off and leaving feedback, we will need to wait a few minutes for the automations to run (they run every five minutes). + +## Update the application + +We can now update our code to pull down the dataset we are sending runs to. +Once we pull it down, we can create a string with the examples in it. +We can then put this string as part of the prompt! + +```python + +### NEW CODE ### + +# Import a helper that will let us get information about the metadata of a run +from langsmith.run_helpers import get_current_run_tree + +# Initialize the LangSmith Client so we can use to get the dataset +ls_client = Client() + +# Create a function that will take in a list of examples and format them into a string +def create_example_string(examples): + final_strings = [] + for e in examples: + final_strings.append(f"Topic: {e.inputs['topic']}\n> {e.outputs['output']}") + return "\n\n".join(final_strings) + +### NEW CODE ### + + +client = openai.Client() + + +prompt_template = """Write a tweet about the following topic. + +Make it in a similar style & tone as the following examples: + +{examples} + +Begin! + +Topic: {text} +>""" + +@traceable( + run_type="chain", + name="Tweeter", +) +def tweeter( + topic: str, +): + # Get the current run tree, this will let us access metadata of the run + rt = get_current_run_tree() + # Construct some metadata to filter on, in this case the user ID + metadata_to_filter = {"user": rt.metadata['user']} + # We can now pull down the examples from the dataset + # We do this inside the function so it always get the most up-to-date examples, + # But this can be done outside and cached for speed if desired + examples = list(ls_client.list_examples(dataset_name="tweeter", metadata=metadata_to_filter)) # <- New Code + example_string = create_example_string(examples) + + return client.chat.completions.create( + model="gpt-3.5-turbo", + temperature=0, + messages=[ + { + "role": "user", + "content": prompt_template.format( + text=topic, + examples=example_string, + ) + } + ], + ).choices[0].message.content +``` + +If now run the application with the same user ID as before, we can see that it learns from the example and generates a tweet more in that style. + +```python +run_id = uuid.uuid4() +tweeter( + "nhl finals", + langsmith_extra={"run_id": run_id, "metadata": {"user": "foo"}} +) +``` + +We can also pass in a different user ID and see that the tweet it generates is back to the original style. + +```python +run_id = uuid.uuid4() +tweeter( + "nhl finals", + langsmith_extra={"run_id": run_id, "metadata": {"user": "bar"}} +) +``` + +We could then make another correction, this time for that different user: + +```python +ls_client.create_feedback( + run_id, + key="correction", + correction="OMG THESE NHL FINALS ARE THE BESTTTTT" +) +``` + +If we wait a few minutes for the automations to run, we'll see that we can now generate pretty distinctive tweets for each user. +Let's take a look at some examples: + +A brand new user (generic tweet style): + +```python +run_id = uuid.uuid4() +tweeter( + "superbowl", + langsmith_extra={"run_id": run_id, "metadata": {"user": "baz"}} +) +``` + +The first user (`foo`) which has more of a chill, all lowercase style: + +```python +run_id = uuid.uuid4() +tweeter( + "superbowl", + langsmith_extra={"run_id": run_id, "metadata": {"user": "foo"}} +) +``` + +The second user (`bar`) which has more of an excited, all caps style: + +```python +run_id = uuid.uuid4() +tweeter( + "superbowl", + langsmith_extra={"run_id": run_id, "metadata": {"user": "bar"}} +) +``` \ No newline at end of file diff --git a/versioned_docs/version-2.0/tutorials/static/class-optimization-neg.png b/versioned_docs/version-2.0/tutorials/static/class-optimization-neg.png index 144d869d..d5c65490 100644 Binary files a/versioned_docs/version-2.0/tutorials/static/class-optimization-neg.png and b/versioned_docs/version-2.0/tutorials/static/class-optimization-neg.png differ diff --git a/versioned_docs/version-2.0/tutorials/static/class-optimization-pos.png b/versioned_docs/version-2.0/tutorials/static/class-optimization-pos.png index d5c65490..144d869d 100644 Binary files a/versioned_docs/version-2.0/tutorials/static/class-optimization-pos.png and b/versioned_docs/version-2.0/tutorials/static/class-optimization-pos.png differ diff --git a/versioned_docs/version-2.0/tutorials/static/tweeter-neg.png b/versioned_docs/version-2.0/tutorials/static/tweeter-neg.png new file mode 100644 index 00000000..af6292fd Binary files /dev/null and b/versioned_docs/version-2.0/tutorials/static/tweeter-neg.png differ diff --git a/versioned_docs/version-2.0/tutorials/static/tweeter-pos.png b/versioned_docs/version-2.0/tutorials/static/tweeter-pos.png new file mode 100644 index 00000000..8f6020aa Binary files /dev/null and b/versioned_docs/version-2.0/tutorials/static/tweeter-pos.png differ