overlayfs: upperdir not freed by resync #60
Labels
Comments
|
@Heiko-L |
Ideally, this should be it but while running tests with clearing the upper directory. There seems to be some issues of data losses due to the internal caching mechanism of overlayFS. It doesn't update the content of the lower directory deterministically as the logic dictates that the lower directory is supposed to be read-only as per overlayFS. So if we try to copy all the content to lower directory and free the upper directory, there seems to be some inconsistencies in the directory mounted with overlayFS. I hope I come across some solution by which I'll be able to implement it but as of now am afraid i can't do much. |
|
Hi, |
|
No, I haven't implemented this so the data loss must not be due to this. The data loss can always occur while using ASD depending on what kind of data loss you're talking about. Think of ASD as a volatile memory with a timeout for syncing back stuff ( by default it is set to one hour ). Suppose you were working on something for the period of one hour and some power cut off occurred before the sync back ( not a proper shutdown ). Now if this occurs the data that was in the volatile memory wouldn't sync back to the disk. However, asd tried to keep the last synced backup intact and is focused on that part. I had created a lot of checks to try to minimize this data loss and all to keep the data intact from the last backup. Have you lost all your data or is it just the volatile data that am talking about? Also, are you on the latest version of ASD or some older version? |
|
My current version of ASD is 6.0.0-1. The data loss might have ocured with the previous version, I'm not sure when the Archlinux repo was last updated. My setup involves one folder (scratch) where I do most of my development stuff. In the past few years I've had several unclean shutdowns (partly Out of Memory Exceptions) and have learned to recover my data (minus the last hour of course). Everything else was commented out. Over the last week(s), I've been working on some project. I did not turn off my computer but I have suspended it serval times. Then, I did a system update (including new kernel if that is relevant) and restarted the computer. When I looked into the scratch folder, the data there was older. At least a weeks worth of code was gone. So naturally, I've looked for a crash recovery snapshot. There were two snapshots (.scratch-backup_asd-crashrecovery-20220626_015926.tar.zstd and .scratch-backup_asd-crashrecovery-20220726_105434.tar.zstd). The older snapshot is too old to be relevant, so I've looked into the newer one. The code in there is quite a bit older than what I've had before the restart. Next, I've looked into the journal to see if asd has reported any errors. It seems to me it did but I'm not sure how to interpret them. First part where an error was shownand Second part where an error was thrownThank you for taking time to respond and look into this. ASD is a wonderfull piece of software :) |
|
I think I can guess what happened, thought I don't know why the I/O errors are occurring in Though once I got to know about it I had updated @graysky2 about this change and mentioned him to update the arch scripts to stop asd before any upgrade ( regardless of the version numbers ) though I can still see the old scripts only available in the repo. |
|
Wow. That's quite a comprehensive analysis. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
To reproduce:
Maybe I don't understand it, but shouldn't everything be copied from the upper to the lower dir after the timer expired so that the overlayfs is "flushed" for lower memory consumption?
The text was updated successfully, but these errors were encountered: