Accessing document vectors for msmarco-v1-passage prebuilt index

[michaelmilleryoder]

Hello! I'm trying to view the TF-IDF or BM25 vectors for the msmarco-v1-passage prebuilt index. But when I try running

from pyserini.index.lucene import LuceneIndexReader

index_reader = LuceneIndexReader.from_prebuilt_index('msmarco-v1-passage')

doc_id = '1000021' # a known document ID
tf = index_reader.get_document_vector(doc_id)

I get the following error:

JavaException: JVM exception occurred: Document vector not stored! io.anserini.index.NotStoredException

I also get the same error when trying to dump out all document vectors with:

index_reader.dump_documents_BM25('msmarco_v1_passage_documents_bm25_dump.jsonl')

Neither of these errors occur with the robust04 prebuilt index for me, which is what the documentation uses: https://github.com/castorini/pyserini/blob/master/docs/usage-indexreader.md#how-do-i-dump-out-bm25-vectors-for-every-document

Do you know what could be causing this error? Apologies if this is not an issue with pyserini and I'm just missing what to do.

[lintool]

Ah, you need the version of the prebuilt index that stores the docvectors. Try msmarco-v1-passage-full: https://github.com/castorini/pyserini/blob/master/pyserini/resources/index-metadata/lucene-inverted.msmarco-v1-passage.20221004.252b5e.README.md

Which is linked from this page: https://github.com/castorini/pyserini/blob/master/docs/prebuilt-indexes.md

[michaelmilleryoder]

That works, thank you!