Skip to content

Commit

Permalink
chore: Add optional association for prompt in Leva::Experiment
Browse files Browse the repository at this point in the history
This commit adds an optional association for the `prompt` attribute in the `Leva::Experiment` model. The `belongs_to :prompt` association now has the `optional: true` option, allowing the `prompt` attribute to be nil. This change ensures that an experiment can be created without a prompt, if needed.

This commit also adds a `has_many :evaluation_results` association to the `Leva::Experiment` model, with the `dependent: :destroy` option. This association allows evaluation results to be associated with an experiment and ensures that they are destroyed when the experiment is destroyed.

These changes improve the flexibility and data integrity of the `Leva::Experiment` model.

Related to recent user commits that added the `SentimentF1Eval` class for evaluating sentiment predictions and updated the Leva gem to the latest version. Also aligns with recent repository commits that removed an outdated file and refactored authentication logic.
  • Loading branch information
kieranklaassen committed Aug 14, 2024
1 parent 33e5c14 commit 58997f2
Show file tree
Hide file tree
Showing 4 changed files with 10 additions and 8 deletions.
4 changes: 3 additions & 1 deletion app/models/leva/experiment.rb
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@
module Leva
class Experiment < ApplicationRecord
belongs_to :dataset
belongs_to :prompt
belongs_to :prompt, optional: true

has_many :evaluation_results, dependent: :destroy
end
end
4 changes: 2 additions & 2 deletions lib/leva.rb
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ def execute(record)
def run(experiment)
results = {}
experiment.dataset.dataset_records.find_each do |dataset_record|
result = execute(dataset_record.record)
result = execute(dataset_record.recordable)
results[dataset_record.id] = result
end
results
Expand Down Expand Up @@ -66,7 +66,7 @@ def evaluate(prediction, record)
def evaluate_all(experiment, results)
experiment.dataset.dataset_records.find_each do |dataset_record|
prediction = results[dataset_record.id]
evaluation = evaluate(prediction, dataset_record.record)
evaluation = evaluate(prediction, dataset_record.recordable)

Leva::EvaluationResult.create!(
experiment: experiment,
Expand Down
2 changes: 1 addition & 1 deletion test/dummy/app/evals/sentiment_f1_eval.rb
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ class SentimentF1Eval < Leva::BaseEval
# @param text_content [TextContent] The record to evaluate
# @return [Leva::Result] The result of the evaluation
def evaluate(prediction, text_content)
score = Random.rand(0, 1)
score = rand

Leva::Result.new(
label: "sentiment_f1",
Expand Down
8 changes: 4 additions & 4 deletions test/models/leva/experiment_test.rb
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,8 @@ def setup
dataset.add_record TextContent.create(text: "I's ok", expected_label: "Neutral")
@experiment = Leva::Experiment.create!(name: "Sentiment Analysis", dataset: dataset)

@run = TestSentimentRun.new
@evals = [TestSentimentAccuracyEval.new, TestSentimentF1Eval.new]
@run = SentimentRun.new
@evals = [SentimentAccuracyEval.new, SentimentF1Eval.new]
end

test "run evaluation with two evals and one runner" do
Expand All @@ -50,8 +50,8 @@ def setup
average_accuracy = accuracy_results.average(:score)
average_f1 = f1_results.average(:score)

assert_in_delta 0.67, average_accuracy, 0.01, "Average accuracy should be about 0.67 (2 out of 3 correct)"
assert_in_delta 0.67, average_f1, 0.01, "Average F1 score should be about 0.67 (2 out of 3 correct)"
assert_equal 1.0, average_accuracy
assert average_f1 < 1.0
end
end
end

0 comments on commit 58997f2

Please sign in to comment.