Flushes hash cache data file's mmap after done writing #2567
Conversation
|
Backports to the beta branch are to be avoided unless absolutely necessary for fixing bugs, security issues, and perf regressions. Changes intended for backport should be structured such that a minimum effective diff can be committed separately from any refactoring, plumbing, cleanup, etc that are not strictly necessary to achieve the goal. Any of the latter should go only into master and ride the normal stabilization schedule. Exceptions include CI/metrics changes, CLI improvements and documentation updates on a case by case basis. |
There was a problem hiding this comment.
lgtm.
a symptom this may cause is the cache hash data file may be the right size, but at some point will become zeros and stop containing correct data. The data is sorted, just effectively 'truncated', except it is replaced by zeros (zero pubkey, zero lamports, zero hash).
|
Yah. I think the files start off as all zeroes and of the right size. If they do not get flushed fully, they'll still have zeroes. And since the hash of the cache file itself is based on the account storage files, we'll never try to regenerate an already-bad cache file. |
(cherry picked from commit 65f6466)
…t of #2567) (#2575) Flushes hash cache data file's mmap after done writing (#2567) (cherry picked from commit 65f6466) Co-authored-by: Brooks <[email protected]>
Problem
We've been seen intermittent issues with the accounts hash calculation where the cache files have large chunks of zeroes. This is very elusive to observe in the wild.
Right now, when saving a new cache data file we:
We never flush after writing all the entries though!
When the reader used to also mmap, I think we handed the same mmap from writer to reader, so we had the full memory resident in the same process and there was no issue. Now, however, the reader does not use the same mmap (or no mmap at all). So if the file is opened and the OS hasn't flushed the mmap back to the file, it is possible the reader does not see the data.
Summary of Changes
Flush the mmap after writing. Also add a stat for how long the flushing takes.