Increasing the layer limit beyond 256 layers

[gorbak25]

Hi!
Is there a fundamental limitation behind the current 256 layer limit or will overlaybd just work fine after increasing it to 10k? In your paper it's stated the performance should not decrease with increasing layer count, do you have a benchmark backing up that claim?

overlaybd/src/overlaybd/lsmt/file.cpp

Line 1599 in 2bcbd0b

if (n > MAX_STACK_LAYERS) {

[BigVan]

will overlaybd just work fine after increasing it to 10k

The maximum number of stacked layers supported by overlaybd is allowed to exceed 256.
We have implemented a nesting mechanism like dolls, which can be found at this link.BigVan@ccadb26

     /* n + m > 256 */
     auto lsmt_ro_file_0 = open_files_ro( files0, n );
     auto lsmt_ro_file_0 = open_files_ro( files1, m );

     IFile* files[2] = { lsmt_ro_file_0, lsmt_ro_file_1 };
     auto dolls_file = open_files_ro( files, 2 );    

In your paper it's stated the performance should not decrease with increasing layer count, do you have a benchmark backing up that claim

In our paper, we tested hundreds of production images of Alibaba, we found no correlation between the effective number of segments of a merged view image index and the depth of image layers. Therefore the increase in the number of layers does not affect the index lookup.
You can use 'IMemoryIndex::size()' to checkout the number of segments in your image, like:

   auto file = stack_files(...);
   auto n_segs = file->index()->size(); 

[lihuiba]

Is there a fundamental limitation

No. The limit is only for memory-saving. And the blob format allows arbitrary number of layers.

We use unsigned char for the layers in current implementation, which is enough in container image scenario. If you need 10k, you can modify the code to use uint32_t instead.

Index query performance is log(n), where n is the number of segments in merged index.