Code

Managed checkpoint 0.4.3#568

Code

Open

mjohns-databricks wants to merge 60 commits intomaindatabrickslabs/mosaic:mainfrom managed_checkpoint_0.4.3databrickslabs/mosaic:managed_checkpoint_0.4.3Copy head branch name to clipboard

+14,420-6,620

Commits

Commits on May 28, 2024

• added python bindings for rst_avg, rst_max, rst_median, rst_min, and rst_pixelcount.

  mjohns-databricks

  committed
  e6cdbbe

  Copy full SHA for e6cdbbe

Commits on Jun 6, 2024

• Move to managed or manual local file cleanup.

  mjohns-databricks

  committed
  a03ab78

  Copy full SHA for a03ab78

Commits on Jun 8, 2024

• CleanUpManager.scala now a long-running thread that manages tmp file cleanup based on configured policies, e.g. file age of 30 minutes.

  mjohns-databricks

  committed
  7281f6d

  Copy full SHA for 7281f6d

Commits on Jun 9, 2024

• estimated memsize for non-file backed rasters and safer subdivide handling

  mjohns-databricks

  committed
  137dda7

  Copy full SHA for 137dda7

• zip install for docker testing, additional file cleanup centralization

  mjohns-databricks

  committed
  80a046e

  Copy full SHA for 80a046e

Commits on Jun 12, 2024

• PAM file cleanup in tests.

  mjohns-databricks

  committed
  09d4240

  Copy full SHA for 09d4240

• fix scalastyle errors

  mjohns-databricks

  committed
  7efab04

  Copy full SHA for 7efab04

Commits on Jun 14, 2024

• UDF Example for GDAL translate and warp; RST_Write function tests and docs.

  mjohns-databricks

  committed
  cd939b6

  Copy full SHA for cd939b6

Commits on Jun 19, 2024

• raster_to_grid performance, gdal reader, test cleanup.

  mjohns-databricks

  committed
  d4a8f96

  Copy full SHA for d4a8f96

• for github testing, adjusted to 30 minute cleanup.

  mjohns-databricks

  committed
  f9f0ad2

  Copy full SHA for f9f0ad2

• MosaicRasterGDAL is now able to hydrate dataset.

  mjohns-databricks

  committed
  0199047

  Copy full SHA for 0199047

Commits on Jun 20, 2024

• removed tile wrapper functions to raster. improved driver detection.

  mjohns-databricks

  committed
  f366d1a

  Copy full SHA for f366d1a

Commits on Jul 1, 2024

• compiles only commit (fully expect tests to fail).

  mjohns-databricks

  committed
  b548e35

  Copy full SHA for b548e35

Commits on Jul 10, 2024

• local tests passing (rc1)

  mjohns-databricks

  committed
  07972ab

  Copy full SHA for 07972ab

• removed a local path that was used for debugging.

  mjohns-databricks

  committed
  b4a24b6

  Copy full SHA for b4a24b6

• separating artefact uploads in github actions, commenting out r build (for now).

  mjohns-databricks

  committed
  a11746c

  Copy full SHA for a11746c

• fix r tests, reverted accidental find/replace for 'path' to 'rawPath'.

  mjohns-databricks

  committed
  356b87e

  Copy full SHA for 356b87e

• github build blocks cleanup manager thread, adding action to cleanup tmp files.

  mjohns-databricks

  committed
  c972224

  Copy full SHA for c972224

• actions checkout must be first

  mjohns-databricks

  committed
  94897fe

  Copy full SHA for 94897fe

• cleanup only after scala

  mjohns-databricks

  committed
  71c2597

  Copy full SHA for 71c2597

• add cleanup cmds to 'scala_build' action.

  mjohns-databricks

  committed
  fb42c59

  Copy full SHA for fb42c59

• add missing sudo to bash cmds.

  mjohns-databricks

  committed
  1c95eff

  Copy full SHA for 1c95eff

• fix r RST_Clip test (num params changed); also scalastyle cleanup.

  mjohns-databricks

  committed
  1ebae94

  Copy full SHA for 1ebae94

Commits on Jul 11, 2024

• r test use "as_path" instead of "in_memory" read strategy.

  mjohns-databricks

  committed
  61328bd

  Copy full SHA for 61328bd

• reverted previous adjustments to r tests, needed to handle metadata for "GDAL_MOSAIC_BAND_INDEX".

  mjohns-databricks

  committed
  16ee8a8

  Copy full SHA for 16ee8a8

• reverted wrong driverName for R test; sudo aware delete (for github builds).

  mjohns-databricks

  committed
  a9187a9

  Copy full SHA for a9187a9

• test if user is 'root' to know if 'sudo' required for certain commands.

  mjohns-databricks

  committed
  e7b6708

  Copy full SHA for e7b6708

• additional file cleanup for current MosaicContext session tmp dir during testing afterAll.

  mjohns-databricks

  committed
  1882e1b

  Copy full SHA for 1882e1b

Commits on Jul 12, 2024

• [WIP] post-PR - revert a find/replace issue, 'raster' -> 'tile' in strings and text, may break build.

  mjohns-databricks

  committed
  966fba1

  Copy full SHA for 966fba1

Commits on Jul 26, 2024

• [WIP] post-PR - additional revert a find/replace issue, 'raster' -> 'tile' in strings and text.

  mjohns-databricks

  committed
  dafae43

  Copy full SHA for dafae43

• [WIP] post-PR - R bindings revert a find/replace issue, 'raster' -> 'tile' in strings and text.

  mjohns-databricks

  committed
  d56ef84

  Copy full SHA for d56ef84

• [WIP] post-PR - additional R bindings revert a find/replace issue, 'raster' -> 'tile' in strings and text.

  mjohns-databricks

  committed
  899191f

  Copy full SHA for 899191f

Commits on Aug 2, 2024

• retile_on_read improvements, doc cleanup, added doc script

  mjohns-databricks

  committed
  8422968

  Copy full SHA for 8422968

Commits on Aug 9, 2024

• interim work post PR - for subdatasets raster_to_grid needs more + rst_tessellate

  mjohns-databricks

  committed
  e7da172

  Copy full SHA for e7da172

• cleanup age limit to 5 minutes for testing to overcome "No space left on device" on remote build

  mjohns-databricks

  committed
  6b7ca39

  Copy full SHA for 6b7ca39

Commits on Aug 15, 2024

• srid defaults to WGS84 again, additional subdataset, interim testing output cleanup.

  mjohns-databricks

  committed
  9c9f1d1

  Copy full SHA for 9c9f1d1

• streamlined RasterAsGridReaderTest

  mjohns-databricks

  committed
  e8d3aea

  Copy full SHA for e8d3aea

• added "limitTessellate" option for testing in RasterAsGridReader

  mjohns-databricks

  committed
  c240d5d

  Copy full SHA for c240d5d

• fixed scalastyle error and warnings

  mjohns-databricks

  committed
  608f5ef

  Copy full SHA for 608f5ef

• cleanup python tests

  mjohns-databricks

  committed
  f5bcae3

  Copy full SHA for f5bcae3

Commits on Aug 19, 2024

• SubdivideOnRead setting subdataset prior to splitting

  mjohns-databricks

  committed
  782de41

  Copy full SHA for 782de41

Commits on Aug 22, 2024

• table options for raster_to_grid

  mjohns-databricks

  committed
  d2dc159

  Copy full SHA for d2dc159

• raster_to_grid option key fix for tables

  mjohns-databricks

  committed
  218f0c7

  Copy full SHA for 218f0c7

• raster_to_grid adjusted columns and finalize tile to fuse for gdal read.

  mjohns-databricks

  committed
  fa4b9cd

  Copy full SHA for fa4b9cd

• raster_to_grid removed trailing "," on select expression (scalastyle error)

  mjohns-databricks

  committed
  2e53b32

  Copy full SHA for 2e53b32

• raster_to_grid added "overwriteSchema" to table write options; modified "finalTableFuse" handling.

  mjohns-databricks

  committed
  7341b5e

  Copy full SHA for 7341b5e

Commits on Aug 23, 2024

• fixed checkpointing for "gdal" reader; "raster_to_grid" added "deltaFileRecords" and "stepTessellate" options.

  mjohns-databricks

  committed
  1774a09

  Copy full SHA for 1774a09

• ran black and isort on python.

  mjohns-databricks

  committed
  044e1b7

  Copy full SHA for 044e1b7

• "raster_to_grid" geo-scientific handling adjustments.

  mjohns-databricks

  committed
  c5bac6d

  Copy full SHA for c5bac6d

Commits on Aug 30, 2024

• "raster_to_grid" [iterim] "toTif" option.

  mjohns-databricks

  committed
  83268eb

  Copy full SHA for 83268eb

Commits on Sep 3, 2024

• "raster_to_grid" cleanup.

  mjohns-databricks

  committed
  de3d9e9

  Copy full SHA for de3d9e9

• python tests adjusted to checkpoint = true as default.

  mjohns-databricks

  committed
  c4bf4d1

  Copy full SHA for c4bf4d1

• r tests adjusted to "as_path" strategy to match checkpoint = true as default.

  mjohns-databricks

  committed
  f987f70

  Copy full SHA for f987f70

Commits on Sep 6, 2024

• Additional ExprConfig handling. Specified "in_memory" for R raster testing.

  mjohns-databricks

  committed
  da2a777

  Copy full SHA for da2a777

• clean-up local tmp files immediately on finalize to fuse for StringType or after writing to BinaryType.

  mjohns-databricks

  committed
  9edbf11

  Copy full SHA for 9edbf11

• added starts with "/tmp" check on local file delete.

  mjohns-databricks

  committed
  f4a5ffc

  Copy full SHA for f4a5ffc

Commits on Sep 7, 2024

• added more checks prior to local file delete.

  mjohns-databricks

  committed
  fb0a557

  Copy full SHA for fb0a557

Commits on Sep 9, 2024

• Reduced size of raster_to_grid tests.

  mjohns-databricks

  committed
  90968e0

  Copy full SHA for 90968e0

Commits on Sep 10, 2024

• tile finalizes to fuse; 'raster' field set only for BinaryType.

  mjohns-databricks

  committed
  8d9830c

  Copy full SHA for 8d9830c

• accidental keystroke in sparklyr test.

  mjohns-databricks

  committed
  ca96125

  Copy full SHA for ca96125