[SPIKE] Investigate deleting all files at the end of a read job

[Aryex]

Description

Currently, we cleanup exported files at 2 points

1. When a reader is closing. At this point, the reader will delete files that have all their portions read.
2. When the application is closing. It will delete all non-external data folders created.

However, if the user has a long-running Spark application (Jupiter notebook) and Spark doesn't read all of the exported files, like with df.show() which only reads 20 rows by default, then the unread files will not be deleted.

If we are able to know when a read job has finished, it would greatly simplify our clean procedure.

Reason: Improves user experience by removing superfluous files