W&B sweep showing a null metric value

[martyna-spyra]

Hi, I'm trying to use wandb sweeps to tune hyperparameters of my spancat model. I tried both approaches demonstrated in this project but no matter what metric I enter in my sweep.yml file, it doesn't seem to be detected by wandb and appears as null in the dashboard:

This is my sweep.yml:

command:
  - ${env}
  - ${interpreter}
  - ./sweeps_using_config.py
  - ./config_for_hpo.cfg
  - ./train

method: random
metric: 
  name: spans_sc_f
  goal: maximize

parameters:
  components.spancat.suggester.max_size:
    values:
    - 8
    - 13
    - 18
  training.dropout:
    distribution: uniform
    max: 0.5
    min: 0.05
  training.optimizer.learn_rate:
    distribution: uniform
    max: 0.01
    min: 0.001

[martyna-spyra]

I still haven't managed to fix this issue - any ideas what could be causing it? This is my training script, adapted from the tutorial (I switched to using the sweeps_using_code.py approach):

import typer
from pathlib import Path
from spacy.training.loop import train
from spacy.training.initialize import init_nlp
from spacy import util
from thinc.api import Config
import wandb
import os
import torch
import subprocess  
from dotenv import load_dotenv

load_dotenv()
AWS_ACCESS_KEY_ID = os.getenv("AWS_ACCESS_KEY_ID")
AWS_SECRET_ACCESS_KEY = os.getenv("AWS_SECRET_ACCESS_KEY")
WANDB_API_KEY = os.getenv("WANDB_API_KEY")

wandb.login(key=WANDB_API_KEY)

def main(default_config: Path, output_path: Path):
    subprocess.run(['pip', 'install', 'torch', 'torchvision', 'torchaudio', '--index-url', 'https://download.pytorch.org/whl/cu118'], check=True)
    subprocess.run(['python', '-m', 'spacy', 'download', 'en_core_web_trf'], check=True)
  
    def train_spacy():
        loaded_local_config = util.load_config(default_config)
        with wandb.init() as run:
            sweeps_config = Config(util.dot_to_dict(run.config))
            merged_config = Config(loaded_local_config).merge(sweeps_config)
            nlp = init_nlp(merged_config)
            output_path.mkdir(parents=True, exist_ok=True)

            # Save merged configuration to a .cfg file
            config_file_path = output_path / "merged_config.cfg"
            merged_config.to_disk(config_file_path)

            # Use subprocess to run spaCy CLI training command
            subprocess.run([
                "python", "-m", "spacy", "train", str(config_file_path), 
                "--paths.train", "./train.spacy", 
                "--paths.dev", "./dev.spacy", 
                "--gpu-id", "0", 
                "--output", str(output_path)
            ], check=True)


    sweep_config = {"method": "random"}
    metric = {"name": "spans_sc_r", "goal": "maximize"}
    sweep_config["metric"] = metric
    parameters_dict = {
        "training.dropout": {
            "min": 0.005,
            "max": 0.2,
        },
        "components.spancat.suggester.max_size": {"values": [8, 10]},
        "components.spancat.threshold": {"min": 0.2, "max": 0.6},
        "training.accumulate_gradient": {"values": [1, 2, 3]},

    }
    sweep_config["parameters"] = parameters_dict
    sweep_id = wandb.sweep(sweep_config, project="wandb_spacy_sweeps")
    wandb.agent(sweep_id, train_spacy, count=20)


if __name__ == "__main__":
    typer.run(main)

The training looks ok

And to make things weirder, spans_sc_f seems to be logged correctly by wandb:

but it doesn't display correctly here:

Any help with this would be appreciated