Kobotoolbox: New API signatures

[josephjclark]

For background see #603

New API

Optimised for getting form submisisons

getForms() // no options
geSubmissions(formId, options) // options is { query, limit, start }
getDeployment(formId) // no options

Notes:

• getSubmissions supports {start, limit, query}, see https://eu.kobotoolbox.org/api/v2/assets/<uuid>/data/
• make sure to include ?format=json in the query URLs. The accept header might work too?
• getSubmissions calls /assets//data. Technically you can pass any asset id and it'll get the data.
• getForms MUST add the asset_type filter: ?q=asset_type:survey.
• rename params to options
• remove callbacks
• See Improve kobotoolbox adaptor: functions, documentation and examples #603 for docs comments - lets make sure the docs are nice and clean

Old API

For the record the current adaptor is not so different after all:

getForms(params, callback) // gets all asset
getSubmissions(params, callback) gets submissions for a given asset (form) - ie asset/id/data
getDeploymentInfo(params, callback) gets deployment info, ie asset/id/submission

Pagination

pagination is really hard in openfn if the adpator doesn't support it. We provide zero looping capability.

So we should automatically handle pagination to handle all data, unless offset or limit is provided (in which case we assume users will handle their own pagination)

[josephjclark]

Understanding the kobo API

kobo's v2 API is called KPI

See source at: https://github.com/kobotoolbox/kpi

• /api/v2/assets?format=json - "list of all assets, including forms/projects, questions, blocks, templates, collections". Can we filter asset by type?
• /api/v2/assets/<asset-id>/data/ - access submitted data for an asset. there are other endpoints like assets/id/deployments

it acutally looks like the v2 JSON API mirrors the app APIs. So maybe the way to go is to sign into kobo and make a list of the APIs there

• In the app I have a Project
• And I create a form
• Can I can see the Form, and Data (for submissions)
• And I can see what's deployed and what's not

but I don't actually see any of that stuff in the URL

Asset query syntax: https://github.com/kobotoolbox/kpi/blob/main/kpi/utils/query_parser/query_parser.py

OH LOOK WHAT IS THIS GORGEOUSNESS: https://eu.kobotoolbox.org/api/v2/assets/

[josephjclark]

Ok based on the above, here's a summary of the kobo API:

GET /assets?asset_type=survey // get a list of assets: block, question, survey, template
POST /api/v2/assets/ // create or clone a form
GET /api/v2/assets/{uid}/counts // daily submission counts
GET /api/v2/assets/{uid}/counts/?days=7 // submission counts for last n days
GET /api/v2/assets/{uid}/data/ // submissions, basically
GET api/v2/assets/{uid}/deployment/
POST api/v2/assets/{uid}/deployment/
GET /api/v2/assets/{uid}/reports/ // Returns the submission data for all deployments of a survey. This data is grouped by answers, and does not show the data for individual submissions.

I don't anticipate much need to create or deploy forms. More likely we want to get submissions and maybe stats

[josephjclark]

So I'm thinking maybe we just have

get(resource, options) // where resource is 'form', 'data', (maybe 'submissions?'), count, deployment, report

The question is; how complicated will options be? We don't have a nice page to link to to explain resources and options.

Also, actually, for most of those things formId is needed.

So maybe the current API is right, and we need:

getForms() // Note that this MUST add the asset_type filter!
getData(formId, options)
getDeployments(formId, options)

And the other stuff you have to use generic http get.

The primary benefits of this structure are:

• kobo will never have many resource types so it makes sense to support them directly
• We can clearly document options and examples clearly for the different resource types

[josephjclark]

@aleksa-krolls this is ready to roll out next sprint!

I went through a bit of a process here: in the end the new API is just a cleaned up variant of the existing one. Still a breaking release but it's tweaks, rather than a rethink :)

[martalovescoffee]

Thanks @josephjclark ! Will review and come back to you :)

[aleksa-krolls]

hey @hunterachieng moving this into your Ready backlog for this sprint, as it looks like you'll have time and it builds on the other kobo issues

cc @martalovescoffee

[aleksa-krolls]

also @hunterachieng please remember to add an estimate before starting work, and schedule time to discuss with Joe as needed - thanks!

[hunterachieng]

@aleksa-krolls this is noted. Thank you