Terrain Storage

[TokisanGames]

Juan wrote:

Terrain Storage

The terrain can be stored directly as high resolution material index regions + mipmaps, so only what is needed is streamed-in. Foliage indices can also be saved as well as decoration indices or LODs for different draw distances.

--

My interpretation:

• a heightmap as an image, with mipmaps? Can we generate mipmaps on 16-bit/32-bit images and load the mipmap only for the section we need?
• foliage/decoration indices - splat map
• LODs - build LODs into the heightmap?? Seems unnecessary, or may be sufficient with the mipmaps. Or LODs on the detail splatmaps? Also unnecessary. I'm thinking LODs on the terrain mesh is fine. Foliage can disappear based on distance.

---

HTerrain

HTerrain allows the user to specify a directory, then stores all maps in there and handles all file management for the user. I adopted this approach. It contains files like:

• height.res
• color.png
• splat[1234].png
• detail[1-9].png

They were imported as Textures and connected to shaders. The maps were not saved inside of shaders or materials. They were converted to Images for processing or saving. He made his own custom resource loader that handled files.

New system

• I like the workflow of offering a directory for the user to specify, then the plugin setting up the data it needs. Or connecting to an existing data dir.
• For height, .res with compression for 16/32bit float works well. For RGB(A) is .res better than .png? We can compare. We should have a format table (like I do in TerrainMap) and load any understandable format and convert to what we need, so that we can upgrade from HTerrain.
• I'd rather keep maps as separate files, and not embedded in a single material resource file, in case of a corruption problem from an unstable beta Godot, alpha plugin, and crashes of unknown origin.
• Do we import as Texture or Image? We need a Map that contains both for processing on either side. It probably makes more sense to import as Texture, though I did not do that on my current system.

What are your thoughts? What other aspects should we discuss on? How do you interpret Juan's statements?

[outobugi]

in case of a corruption problem from an unstable beta Godot, alpha plugin, and crashes of unknown origin

That's very good point that I did not think of! We should save them as binary (Image, .res or custom extension), so the editor does not re-import them and the loading/saving time stays fast. I stand by using binary and nothing can change that!

Do we import as Texture or Image?

Image. Because the maps are only data and not visuals. Or a custom resource loader that imports them as binary, which would be an Image either way.

I like the workflow of offering a directory for the user to specify

Like the way you have in your editor? It seemed very ugly. Sorry. Instead we should use the TerrainStorage resource to define the data directory.

How do you interpret Juan's statements?

Badly. I barely understood. I'm still processing those.

What TerrainStorage?

TerrainStorage holds layer materials, height, quad density, clipmap levels and any other data that is needed for generating everything.
TextureArrays are not saved because they can changed (e.g modding) and take unnecessary space. (Meshes aren't saved either)
The layer materials hold textures, UV scale, anti-tile toggle and other properties which the "master" material uses. This is the aspect I most like about my terrain. It gives better interface to control the visuals. Initially I wanted to use the default Godot materials but they do not emit "changed" signal and that is required for the master material to update.

[TokisanGames]

Though a discussion for the materials thread, the witcher allows painting w/ different UV scaling, as well as slope with auto scaling. I'm not convinced the index material is the way to go. But for the purpose of Storage, the bottom line is just that it needs to house maps and provide data. I see it as a fairly dumb class.

[outobugi]

TerrainStorage

Generated from a menu because needs to be saved to disk.

Should be exposed, it will be easier to find where terrain's data is saved.
Only holds data needed for the terrain and generates anything that is missing. Properties should be read only. Empty looking resource is not ideal.

Edit: also has each Layer Material and automatically combines to a TextureArray and other data arrays. Layer Material list should be exposed.

TerrainMaterial

Created by the user, because terrain acts as a mesh.

Terrain material property should accept ShaderMaterial as well, for those who want to make their own.
Terrain will pass data from the storage to the material. Material itself doesn't save any images or textures.
Material is a property of the terrain and not the storage. This allows to keep the editor functional even if the material is custom.

[outobugi]

This is much more complex problem than I thought. Fuck.

The material needs to know when storage data changes and is pretty much tied to the storage, so re-using it with another storage might cause problems.

I would propose that the storage is a property of material.
But wait, there is a fucking problem as well! The storage should also hold data for e.g grass, material doesn't care about that.

[TokisanGames]

Generated from a menu because needs to be saved to disk.

Yes

The material needs to know when storage data changes

Godot already reports externally changed files. We can have a changed flag on our edits, as I do in my terrain. Then storage updates() the material.

and is pretty much tied to the storage, so re-using it with another storage might cause problems.

Shouldn't be an issue. Should be able to work independently and be swappable.

Storage can have an array of maps, as I and zylann do. It can detect when any one of them changes and update the appropriate class that needs it. Either it hands out data to whomever needs it, or other classes retreive it from them. Material doesn't need the grass map so does not get it, nor ask for it. NP. Only the particle/MMI gets the foliage map from Storage.

[TokisanGames]

• Do all heightmaps have to be placed at a location via code or the paint brush? Does that mean API users have to know the location? Does the user have to know the exact grid coordinates or does it snap?

• In the current system users will have to pay for at least a 1k heightmap and control map. What if they only want a 64b or 256b map? I have multiple levels in my game that range from 256-2048, and vram is precious due to so many meshes.

• We need a public API that allows loading a whole image. I should be able to simply pass in an up to 16k image to a function. If it wants to store as 1k slices, then fine, but there needs to be a slicer function.

• What about repeating? Are we going to force the player who wants a terrain that extends to the horizon to sculpt and store an 8k map? I have some 256b maps on small levels, I only have to sculpt and pay for 256^2. If the player goes up on the ridge, they just see the landscape extends off in the distance. I think we need an option to repeat. It would also alleviate part of Maps w/ content don't display properly #17.

• On my terrain system I thought about doing these:

  • 1 heightmap up to 16k, infinitely repeating (done)
  • Option to turn off repeating and fade to a noise map, so it's infinite, but your sculpted features aren't noticeably repeating.
  • Option to turn off repeating, fade to noise, and slope downward, so your heightmap is an island.

• I think we should start with these modes:

  • Non repeating, manual painting everything. As you sculpt outward from the center, it automatically creates new maps to accomodate. (I think that's what you are doing now, but not 100% sure.)
  • Repeat what is there. If there are two heightmaps wide, one deep, then repeat 2x1 everywhere. (or we could require it be square).
    These would be independent of the data that is loaded. If I load in a 4k^2 map that gets sliced into 16x1k maps, then that block of 16 gets repeated.

[outobugi]

Do all heightmaps have to be placed at a location via code or the paint brush? Does that mean API users have to know the location? Does the user have to know the exact grid coordinates or does it snap?

Via tool or code. User only needs to know the general area for the tile and it will be snapped to grid automatically.

I'll address rest of these later.