You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
1. Data read and write operations should consider the characteristics of the disk, for example, the disk is based on 512-byte sectors as its storage unit, and read data in batches of 64k; the file system will eliminate the cache according to certain rules Pages in memory etc. If the read and write operations take these contents into account, I believe that the current TPS can be higher;
-----------------------------------
I understand the reason for this problem, the problem I think is that the data alignment needs to be considered when storing data to disk, so that the head can read the data with as few accesses as possible. For example, the data b in the picture, when it unaligned stored, two sectors need to be accessed, but the head only needs to access and read one sector after aligned storage:
The performance difference of this problem may not be obvious under a single access, but under the reading of massive messages, the performance will be essentially improved from quantitative to qualitative changes, especially since TubeMQ internally reads data randomly, it is more necessary to consider this .
After the modification, a new problem introduced based on this modification needs our attention, that is, the storage space used after the system modification will increase under the same data amount. As shown in the picture, because the data is aligned, for small packet data (below the sector size ) a, will occupy more space. However, considering that the overall performance of the disk is unchanged, and the disk space can reach a very high capacity, this waste is considered acceptable to me
For the optimization of this problem, do you have any suggest ? If not, I am going to claim this modification and optimize it according to this idea.
1. Data read and write operations should consider the characteristics of the disk, for example, the disk is based on 512-byte sectors as its storage unit, and read data in batches of 64k; the file system will eliminate the cache according to certain rules Pages in memory etc. If the read and write operations take these contents into account, I believe that the current TPS can be higher;
-----------------------------------
I understand the reason for this problem, the problem I think is that the data alignment needs to be considered when storing data to disk, so that the head can read the data with as few accesses as possible. For example, the data b in the picture, when it unaligned stored, two sectors need to be accessed, but the head only needs to access and read one sector after aligned storage:
The performance difference of this problem may not be obvious under a single access, but under the reading of massive messages, the performance will be essentially improved from quantitative to qualitative changes, especially since TubeMQ internally reads data randomly, it is more necessary to consider this .
After the modification, a new problem introduced based on this modification needs our attention, that is, the storage space used after the system modification will increase under the same data amount. As shown in the picture, because the data is aligned, for small packet data (below the sector size ) a, will occupy more space. However, considering that the overall performance of the disk is unchanged, and the disk space can reach a very high capacity, this waste is considered acceptable to me
For the optimization of this problem, do you have any suggest ? If not, I am going to claim this modification and optimize it according to this idea.
JIRA link - [INLONG-120] created by gosonzhangThe text was updated successfully, but these errors were encountered: