Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[INLONG-120] Aligned disk data storage #120

Open
dockerzhang opened this issue Jul 28, 2021 · 3 comments
Open

[INLONG-120] Aligned disk data storage #120

dockerzhang opened this issue Jul 28, 2021 · 3 comments

Comments

@dockerzhang
Copy link
Owner

1. Data read and write operations should consider the characteristics of the disk, for example, the disk is based on 512-byte sectors as its storage unit, and read data in batches of 64k; the file system will eliminate the cache according to certain rules Pages in memory etc. If the read and write operations take these contents into account, I believe that the current TPS can be higher;

-----------------------------------

I understand the reason for this problem, the problem I think is that the data alignment needs to be considered when storing data to disk, so that the head can read the data with as few accesses as possible. For example, the data b in the picture, when it unaligned stored, two sectors need to be accessed, but the head only needs to access and read one sector after aligned storage:

 
The performance difference of this problem may not be obvious under a single access, but under the reading of massive messages, the performance will be essentially improved from quantitative to qualitative changes, especially since TubeMQ internally reads data randomly, it is more necessary to consider this .

After the modification, a new problem introduced based on this modification needs our attention, that is, the storage space used after the system modification will increase under the same data amount. As shown in the picture, because the data is aligned, for small packet data (below the sector size ) a, will occupy more space. However, considering that the overall performance of the disk is unchanged, and the disk space can reach a very high capacity, this waste is considered acceptable to me

For the optimization of this problem, do you have any suggest ? If not, I am going to claim this modification and optimize it according to this idea.

JIRA link - [INLONG-120] created by gosonzhang
@github-actions
Copy link

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

@github-actions
Copy link

github-actions bot commented Jul 9, 2022

This issue is stale because it has been open 60 days with no activity.

@github-actions github-actions bot added the stale label Jul 9, 2022
@github-actions
Copy link

github-actions bot commented Sep 7, 2022

This issue is stale because it has been open for 60 days with no activity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant