feat(backup): support export and import of backups on Web, iOS and Android [WPB-10575] #3228
Conversation
vitorhugods
requested review from
sbakhtiarov,
MohamadJaara,
Garzas,
saleniuk and
aweiss-dev
echoes-hq
bot
added
the
echoes: product-roadmap/feature
Test Results3 367 tests 3 260 ✅ 5m 21s ⏱️ Results for commit 72cdd83. ♻️ This comment has been updated with latest results. |
Datadog ReportBranch report: ✅ 0 Failed, 3260 Passed, 107 Skipped, 36.59s Total Time |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## epic/multiplatform-backup #3228 +/- ##
============================================================
Coverage ? 54.02%
============================================================
Files ? 1246
Lines ? 36152
Branches ? 3656
============================================================
Hits ? 19532
Misses ? 15209
Partials ? 1411 Continue to review full report in Codecov by Sentry.
|
| private var persistedConversationsChunks = 0 | ||
| private var persistedMessagesChunks = 0 | ||
|
|
||
| private val backupInfo by lazy { |
There was a problem hiding this comment.
Would having this as a lazy cause it to always have the same time if i create 2 backups after each other
not sure why this have to be from here instead of passing those info to the create backup function
There was a problem hiding this comment.
I wasn't really considering the possibility that the same instance of MPBackupExporter could be used for multiple backups.
I was thinking of saying bye bye to it after using it, no reason to keep it around and reuse it.
| internal suspend fun initializeLibSodiumIfNeeded() { | ||
| if (!LibsodiumInitializer.isInitialized()) { | ||
| LibsodiumInitializer.initialize() | ||
| } |
There was a problem hiding this comment.
should this function be thread safe or it will never not be called multiple times
|
PR Submission Checklist for internal contributors
The PR Title
SQPIT-764The PR Description
What's new in this PR?
Issues
While creating a unified Backup format for iOS, Web and Android, we realised that encryption and compression logic must also be unified.
So, this PR aims to bring the whole solution together:
The whole flow is joined together here.
Solutions
Whilst encryption/decryption and serialisation/deserialisation were already implemented, some changes had to be done in these parts to join the whole flow together. However, the logic implemented in previous PRs stay the same.
Pagination
One small change to the current logic is that backup data is now paginated.
As the application exports data, multiple files (pages) are created and memory is released.
Except on Web, where the pages are created but they stay always in memory.
During importing, the process is reversed, and the application can read one file at a time.
Compression
There is no multiplatform zip library for Kotlin
So I had to come up with some solution to bridge this gap.
JS / Web / Browser
The current Web client is using JSZip for zipping/unzipping files. It was possible to embed this solution within the library as well using the NPM dependency and some glue / Kotlin bindings + magic to solve dynamic types.
iOS
I wasn't able to add compression directly to the library. Our current Kotlin version has some bugs when generating Kotlin bindings for objc and therefore I couldn't use ZipArchive to implement it.
My solution is to just delegate this responsibility to the consumer.
iOS app will need to provide a
zipperandunzipperin order to use the library.Android
For the sake of reducing code, I left it exactly like iOS. So
kalium:logicwill need to provide azipperandunzipperas well.Next Steps
More fields
We need to add more fields to the messages, users, conversations, etc. in order to make sure it all works.
File Peeking
I'm raising a smaller PR after this one that will provide "file peeking" capabilities.
This is: read the header of a backup archive and be able to say if self user created it, and if it is encrypted or not.
This way the clients can ask for a password if needed, and abort the importing if self user isn't the one that created the backup.
Polishing
There are some possible improvements in terms of performance, and API will probably change a bit as we gather feedback from clients.
Publishing
The goal is to publish a npm library for web, and a spm (?) artifact for iOS
Testing
For testing code that uses compression, it was straight forward on JS: just use JSZip as normal.
But, for Android/iOS, because I left it as a responsibility of callers, I faked it completely.
Instead of actually zipping stuff, all the FakeZip does is:
Then, when unzipping, the reverse process happens:
Test Coverage
PR Post Merge Checklist for internal contributors
References
feat(conversation-list): Sort conversations by most emojis in the title #SQPIT-764.