HDDS-11404. Make RocksDB all parameters configurable.

[weimingdiit]

What changes were proposed in this pull request?

Make RocksDB all parameters configurable.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-11404

How was this patch tested?

unit tests and local test environment, and i will provide some test results from my local test environment

Additional notes:

1. We can use different ini files to customize the rocksDB configuration of corresponding services (such as OM, SCM, DN).
   For example, you can create a file such as om.db.ini to configure OM's rocksDB, scm.db.ini to configure SCM's rocksDB, and container.db.ini to configure DN's rocksDB
2. OM, SCM, and DN were tested separately, and the results were in line with expectations

for OM

om.db.ini file:

OPTION file:

for SCM

scm.db.ini file:

OPTION file:

for DN

container.db.ini file:

OPTION file:

[weimingdiit]

@jojochuang @szetszwo @adoroszlai Could you help review this PR? Thanks.

[szetszwo]

@weimingdiit , just have a quick comment inlined.

[szetszwo]

Could we move the new classes to hadoop-hdds/managed-rocksdb/... instead of adding allowedImport?

[weimingdiit]

@szetszwo Thanks for your comments, I have updated the code to fully support configuring all rocksDB parameters via ini files

[kerneltime]

@weimingdiit instead of wrapping every config via Ozone, why not let Ozone specify the RockDB config file https://github.com/facebook/rocksdb/blob/5b40c0c074f16dae6e1d60a3c5d7aa3e7d63fdc7/include/rocksdb/utilities/options_util.h#L56
This way, we do not need to keep managing RocksDB Config value additions or deprecation.

[errose28]

@kerneltime I agree. I made the same suggestion in #7117 with a pointer to the RocksDB docs. The approach taken here is not sustainable to maintain and also less intuitive than having Ozone work with RocksDB's config file exactly how RocksDB documents it.

This DBProfile abstraction is old and not really used as far as I know. It might be better to just get rid of it. We can instead directly provide example RocksDB config files for different use cases. This will be much easier to follow than hiding the tuning inside the Java code.

[weimingdiit]

@errose28 @kerneltime This is a very good idea. I looked at the relevant code and felt that directly supporting INI files might be a better choice. Thank you very much for your suggestion. I will modify the code to fully support INI files.
By the way, I think this issue #7117 should be closed

[sumitagrawl]

@weimingdiit Do we have mechanism to control service specific usecase, like if OM, SCM and DN have different configuration, like WAL manual flush may need disable only for OM ?

[sadanand48]

Do we have mechanism to control service specific usecase, like if OM, SCM and DN have different configuration, like WAL manual flush may need disable only for OM ?

Probably we could have configuration files for each Role . By default all of these can point to a base config but also provide an option to add overrides for each role. Similar to ozone.metadata.dirs where each process can define its own metadata dir but if not defined , we can fallback to the common config

[weimingdiit]

Do we have mechanism to control service specific usecase, like if OM, SCM and DN have different configuration, like WAL manual flush may need disable only for OM ?

Probably we could have configuration files for each Role . By default all of these can point to a base config but also provide an option to add overrides for each role. Similar to ozone.metadata.dirs where each process can define its own metadata dir but if not defined , we can fallback to the common config

@sumitagrawl @sadanand48 Yes, this may be our ultimate goal.

[weimingdiit]

@errose28 @szetszwo Hi, could you help review the pr?