Defer as.Date() coercion to R level

[MichaelChirico]

Closes #6105

As seen in the tests, this might be a breaking change if any downstreams depend on the output being specifically Date (and not IDate). I am not sure it's possible -- inherits(., "Date") is still TRUE, and relevant methods should just back up to Date methods if not available for IDate. I lean towards just going ahead with this change unless revdeps finds anything concerning, WDYT @jangorecki?

cc also @HughParsonage who may have some extra insights.

---

One alternative I looked into that won't work is adding "Date" as an alias for "IDate" in the type hierarchy:

data.table/src/freadR.c

Lines 27 to 29 in d19bfef

	static int typeSxp[NUT] = {NILSXP, LGLSXP, LGLSXP, LGLSXP, LGLSXP, LGLSXP, INTSXP, REALSXP, REALSXP, REALSXP, REALSXP, INTSXP, REALSXP, STRSXP, REALSXP, STRSXP };
	static char typeRName[NUT][10]={"NULL", "logical", "logical", "logical", "logical", "logical", "integer", "integer64", "double", "double", "double", "IDate", "POSIXct", "character", "numeric", "CLASS" };
	static int typeEnum[NUT] = {CT_DROP, CT_EMPTY, CT_BOOL8_N, CT_BOOL8_U, CT_BOOL8_T, CT_BOOL8_L, CT_INT32, CT_INT64, CT_FLOAT64, CT_FLOAT64_HEX, CT_FLOAT64_EXT, CT_ISO8601_DATE, CT_ISO8601_TIME, CT_STRING, CT_FLOAT64, CT_STRING};

The problem is that there are files where our rudimentary date parser doesn't detect the date, but as.Date() does, meaning fread.c returns a string --> the hierarchy is broken as there's an attempt to cast string->int32. See in particular this test:

https://github.com/Rdatatable/data.table/blob/d19bfef7026f25bb2d36c17879187d09bcb2b2c3/inst/tests/tests.Rraw#L11043

[MichaelChirico]

• Defer as.Date() coercion to R level #6107 👈
• Refactor colClasses handling for readability #6106
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @MichaelChirico and the rest of your teammates on Graphite

[github-actions]

Generated via commit 92c8f08

Download link for the artifact containing the test results: ↓ atime-results.zip

Time taken to finish the standard R installation steps: 11 minutes and 23 seconds

Time taken to run atime::atime_pkg on the tests: 6 minutes and 27 seconds

[jangorecki]

Yes, let's see revdeps

[MichaelChirico]

@jangorecki I think we should run fread(DT) once here in the setup because of cacheing, right? Or do we need to run in a subprocess? Here the benchmark is really about what happens after the fread.c code is run, only care about differences emerging (1) in freadR.c and/or (2) fread.R post-processing. WDYT?

[MichaelChirico]

(regardless, we see substantial improvement in all 3 cases already)

[HughParsonage]

What is the consequence of a dangling else here? (As in, if the new condition evaluates to false, lines 339-340 would previously be evaluated.) I think a comment for this line would be useful here.

[MichaelChirico]

Thanks for the call-out, this is something I wasn't understanding well myself.

The key is we just skip updating type[i] for that case. We might consider rewriting this then as:

type[i] = (type[i] == CT_ISO8601_DATE && tt == char_Date) ? type[i] : typeEnum[w-1];

The "trouble" comes in the next line, because w==NUT in the relevant case here (i.e., "Date" is not a known class for C-side coercion).

So with the simple fix, we're back to R-side as.Date() coercion.

As noted, we might prefer that (retaining fully back-compatible Date output columns), but it'll require a corresponding change in the colClasses=list() branch.

So it's down again to whether the result of this PR should be "cols parsed as IDate and requested as Date are returned as IDate, fully avoiding any coercion" or "we now skip the middle step in IDate->char->Date coercion; for maximum efficiency, request IDate instead of Date".

[MichaelChirico]

My recollection is this was ready to submit besides the issue of tidying up the atime part (deferred to follow-up now), and we held off on the potential breaking change near release; submitting now.