Wikidata <> authors integration: first steps

[RayBB]

CLOSED IN FAVOR OF #9130

Document with the goal of this project

v0 goals:

• to start pulling wikidata into the database and showing something basic
• Upstream code class Author —> Add get_wikidata
• Should Fetch from wikidata based on parameter. Otherwise only check wikidata_cache table
• Author edit/view templates will call this
  • Render_wikidata_infobox method for rending the wikidata author box

Next steps:

• Decide on a bulk import method
• Enrich the infobox
• Use wikidata for author autocomplete

Technical

Trying to keep it as simple as possible.
The wikidata method is on the authors model because works/editions store wikidata IDs differently so we'll have to handle that when we get there. My current coals is just authors.

Testing

Add a wikidata ID to an author and then you'll start seeing this "short description" field showing up on the side.

Screenshot

Aug 30 demo video (slightly outdated), see below

wd_demo.mp4

September 2 screenshot (most recent)

Stakeholders

@cdrini

[davidscotson]

It would be really neat if this was added to the autocomplete, which currently pulls in the author's name, date-of-birth/death, genres and top work.

On the Wikidata side this is mostly designed to be used to differentiate between two identically named items, so it's an ideal use case.

It might also be worth looking at the Wikidata-powered infoboxes on Wikipedia to see what kind of info they surface.

https://en.wikipedia.org/wiki/Template:Infobox_writer/Wikidata

[tfmorris]

Is there an issue associated with this PR? Making better use of Wikidata data is definitely a good idea, but I'm not sure description is the best place to start. In addition to the I18N issues, these are also mostly machine generated from templates so you're going to end up with lots of " author (birth-death)" which doesn't really add a lot of value.

Also, given the volumes of data involved, it's probably more appropriate to use the data dumps than be hitting their API.

[hornc]

What is the usecase behind this? Is there to be a bulk QID fetching UI or something?
Individual author lookups on an individual UI could be done with links or simple requests with a review feature.
I'm not sure what a Wikidata table is for.

[RayBB]

@hornc
use case: this is an MVP to start integrating data from wikidata.
There will be many next steps to improving the reader experience and using the data for search.
Also I made this to lay out some ideas:
https://github.com/internetarchive/openlibrary/wiki/Wikidata-Integration

[cdrini]

Nice! A few small things, but I think the main things are:

The split between the two files is a little confusing, especially since we have WikidataEntities, WikidataEntity, WikidataRow and the distinction between these is a little confusing. I think we want:

@dataclass
class WikidataEntity:
    id: str
    data: dict
    updated: datetime

    # In general we prefer function to have verb names
    def get_description(self, locale: str) -> str | None


def get_entity(id: str, cache_only = True) -> WikidataEntity | None
    
def _get_from_web(id: str) -> WikidataEntity
def _get_from_cache(id: str) -> WikidataEntity | None
def _add_to_cache(id: str) -> None

That will I think keep all the code easily in one place, and since I doubt our wikidata caching will ever really grow to be more complicated than these, I think it'll keep things a touch tidier!

[cdrini]

Put the css for this in a new less in static/css/wikidatabox.less.

Then import that CSS from page-user.less .

(I'm sorry, this flow is not great 😅 )

[cdrini]

We can probably inline this function, and the timedelta helpers might be a little clearer here!

Suggested change

	if entity and seconds_since(entity.updated) < ttl:
	if entity and (datetime.now() - entity.updated) < timedelta(days=30):

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 15.96%. Comparing base (45ed081) to head (9594dd1).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #8236   +/-   ##
=======================================
  Coverage   15.96%   15.96%           
=======================================
  Files          89       89           
  Lines        4710     4710           
  Branches      821      821           
=======================================
  Hits          752      752           
  Misses       3449     3449           
  Partials      509      509           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[RayBB]

@cdrini I think I've addressed all your concerns and the code is looking a lot cleaner.
Ready for your next round of feedback!

[cdrini]

This looks good to me! Would be great if we can merge those two classes + one other comment.

Next step for me is to create this table on prod so we can deploy this.

Next step for the project is to begin working on bulk import. We can make tweaks to how/where the data is displayed at any point, but getting the bulk data import sorted is likely the next most impactful step.

[cdrini]

Nice! Logic looks good; a few style things. I created the wikidata table and will toss this up on testing now since the logic is already 👍

[RayBB]

I am leaving this code here now for testing.openlibrary.org to be easier to verify this PR on.

I can either remove it before we merge this or there is already a commit to remove it in #9130

[RayBB]

Closing in favor of #9130