host.promiseQueue Causing Cache Misses and Repeated Requests

[anrgct]

v1.87.0, I found that host.promiseQueue causes the cache to miss, resulting in repeated requests.

// repomap-test.genai.mjs
script({
    title: "generate repomap for the repo",
    model: "openai:deepseek-chat",
    cache: "repomap-test",
    files: ["src/greeter.ts", "src/counting.py"], 
})

let prompts = [
    `summarize the file in one sentence.`
]

async function processFile(current_file, prompts) {
    let result = [];
    for (const prompt of prompts) {
        const { text } = await runPrompt((_) => {
            _.def("FILE", current_file);
            _.$`=============`;
            _.$`${prompt}`;
        }, { system: [] , cache: "repomap-test" });
        result.push(text);
    }
    console.log(result)
}


// this does not hit "src/counting.py" cache
const queue = host.promiseQueue(2)
const summaries = await queue.mapAll(
    env.files,
    (file) => processFile(file, prompts)
)

// // this work fine
// for (const file of env.files) {
//     await processFile(file, prompts)
// }

[github-actions]

Feedback:

The issue described is that host.promiseQueue is causing cache misses and repeated requests. This is likely due to the fact that the queue is not respecting the cache settings applied individually within each processFile call.

To resolve this, you should ensure that each call to runPrompt within processFile respected the cache options provided. This means that if a specific file's result is already cached under "repomap-test", it shouldn't be re-run for that file in a separate queue task. Instead, you might want to manually check the cache before running the prompt and only proceed if the result is not found.

Additionally, consider using host.promiseQueue options or passing additional context to ensure that each task within the queue respects its own caching settings correctly. This would prevent unnecessary reprocessing of files that have already been cached.

AI-generated content by issue-reviewer may be incorrect

[pelikhan]

Good one. I'll look into this next week.

[pelikhan]

• refactor cache logic outside completer
• store promise for pending value computation

[pelikhan]

This works now:

const innerPrompt = `Generate 2 word poem. ${Math.random()}`
await Promise.all(
    Array(10)
        .fill(0)
        .map(async (_, i) => {
            await runPrompt(innerPrompt, { cache: "inner", label: `run-${i}` })
        })
)