Scheduled Action Failure - Ingest - 📁 Open Data Routine Updates

[github-actions]

Failed action

[fvankrieken]

This is a funny one. A bunch of datasets seem to have reordered rows, which since we compare with pandas.DataFrame.equals are not considered equal (as opposed to say SELECT * ... EXCEPT in sql, which has its own limitations but would be okay with a different order).

I think short term, it might be useful to add some sort of logging to test/display whether dataframes are equal other than order, but keep throwing errors in these cases. Order in the exported data seems to mirror order in the data explorer in socrata, though that doesn't mean that they guarantee order. I think we monitor and address issue in future if this seems like an actual limitation of the api