huggingface_hub removed cached_download

[awdemos]

text-embedding package uses huggingface_hub which has removed support for cached_download in the latest versions. A tested version of this continues to work is 0.25.2 and this version should either be pinned or have the backend python script reworked to remove the usage of cached_download.

The README.md in text-embeddings could have the pip install command updated to be pip install 'huggingface_hub[cli,hf_transfer]==0.25.2'

See this separate un-related GitHub issue where this topic is discussed at length easydiffusion/easydiffusion#1851

[justinthelaw]

We do pin huggingface_hub in the pyproject.toml:

leapfrogai/packages/text-embeddings/pyproject.toml

Lines 1 to 39 in 1562db1

	[project]
	name = "lfai-text-embeddings"
	description = "A LeapfrogAI API-compatible embeddings library wrapper for text-based embedding generation."
	# x-release-please-start-version
	version = "0.14.0"
	# x-release-please-end
	dependencies = [
	"InstructorEmbedding == 1.0.1",
	"torch == 2.2.0",
	"numpy ==1.26.4",
	"tqdm ==4.66.5",
	"sentence-transformers == 2.2.2",
	"transformers == 4.38.0",
	"leapfrogai-sdk",
	]
	requires-python = "~=3.11"
	readme = "README.md"
	[project.optional-dependencies]
	dev = ["huggingface_hub ==0.25.1"]
	[tool.pip-tools]
	generate-hashes = true
	[tool.setuptools]
	packages = ["leapfrogai_sdk"]
	package-dir = { "" = "../../src" }
	[tool.pytest.ini_options]
	addopts = ["--import-mode=importlib"]
	[tool.ruff]
	target-version = "py311"
	[build-system]
	requires = ["setuptools"]
	build-backend = "setuptools.build_meta"

but there is a mis-mtach in the README document within the text-embeddings package's directory:

https://github.com/defenseunicorns/leapfrogai/blob/main/packages/text-embeddings/README.md#deployment

We also should have mentioned the dev dependencies in text-embeddings within the DEVELOPMENT.md, but only mention vllm and whisper since they cover everything text-embeddings usually needs for package creation:

https://github.com/defenseunicorns/leapfrogai/blob/main/docs/DEVELOPMENT.md#bundle-development

If you open a PR with the docs changes, I can approve it and merge it in.