optimized pagination?

[testbird]

It may be benificial to join/depend on and use the meteor-pages package for its features.
https://github.com/alethes/meteor-pages

[julpod]

Before I started this projects, I have done a little research and also found meteor-pages. After tested it and as far as I've understood, the package is great but did not accomplish my project needs since the its scope is much bigger.

I think that trying to merge the two packages would be a little messy since both try to solve same user cases but with differents approachs.

The main goal of this project is to be a solid tool for Meteor's developers that could be easily integrated with other packages (like iron route) but independent at the same time.

If you consider that some particular feature of meteor-pages could be useful to be included in this packages, please let me know or do a pull request and I will look at it enthusiastically.

Thanks!

[testbird]

Hey Julian! Thanks for your kind response. Good to hear you know meteor-pages already. I didn't think merging would be desired, maybe just using it for pagination if the implementation is good, but you're right, I better look into it some more.

A glitch I see with the f-c demo is that when the switching pages a longer list flickers up before the next (short) page is shown.

And from the f-c readme I couldn't tell which features listed at meteor-pages are coverd, e.g. preloading of next increments, and only adding to the local collection cache (no re-subs mode). Maybe you could add such a list for f-c as well?

[julpod]

You are right, that is something that I've noticed and I think that this happend because invalidation is performed after the cursor is back to the subscriber so at a first glance, i consider that this could be fixed doing invalidation before subscribing. This will cause a visual document cleanup effect before the new ones are loaded. Let me know your thoughts.

Thanks.

[testbird]

Unfortunately, I know almost nothing about meteor rendering, and user "aldeed" has pointed me to upcoming changes: https://github.com/meteor/meteor/wiki/New-Template-Engine-Preview

Let us move our paging related thoughts from "how does it work?" issue here as well.

Thoughts for proper differentiation of scoping and pagination on server client:

stages

From the entire db, the server "scopes" down the data to sync with the client (from throusands of documents to hundreds).
The client can sort and filter the local documents cache as the user requests.
Finally, pagination may bring the number of listed documents down to tenths.

usage

Desktop client browsers should have absolutely no problem working with hundreds of documents in the (minimongo) cache.

Using the cache is fast. Re-subscribing to changed publications (scope changes) should be avoided, because this invalidates and retransmitts all documents. Hopefully, there is a way to broaden a subsribed scope (adding additional documens) without invalidating and re-subscriging. (The meteor-pages description seems to describe that? Start syncing only the first increment, load next in background, keep all previous when switching to next page and loading the next+1 in the backgound.)

Clients can subscribe to multiple "scope" publications (differnent filter and sort order), they are then able to switch between sorting and filtering locally, returning up-to date correct results without any re-subs.

Maintaining a multi-scope cache will also allow the client to keep working when Meteor.status.connected is false.

Depending on the client platform/resources there is a limit to the cache size when performance starts to degrade. Above that limit the client may:

• Reduce the number of synced documents from currently not required scopes to a minimum, to avoid completely unsubsribing from not required scopes.
• If the cache can not be kept within limits, because the request is to show (paginate to) documents above that limit, the client may set up a special pub/sub into a separate, specific local collection, that skips most of the previous documents (and will not contain any documents from other scopes).

implementation

As multiple scopes get merged into the same synced cache (merge box on server and minimogo on client), correct pagination requires to keep all previous documents in the cache (to be able to select the ones to show). This comes naturally for "infinite scroll", classic pagination will just have to skip the previous entries in the cache.

These comments seems to hint to some new build in features though: tmeasday/meteor-paginated-subscription#3

[testbird]

A good pattern may be to let the router waitOn the minimum size of the required scope, and preload the minimum sizes of all other basic app scopes in the background after page rendering.

[testbird]

The distinction, that I think is important, is between scoping (server code) and pagination (client code).

Pagination should be done by client code, except when the amount of documents synced according to all subscribed scopes can not be handled efficiently on the client anymore.

Usually, one chooses the published scopes so that they don't reveal only a limited set of info from the database and don't overwhelm the client (publish only recent documents, fields relevant to the user, etc.), and pagination can be handled by the client without any re-subs and document re-transmissions (even offline).

Nevertheless, if the data that the client is supposed to sort, filter and look at (paginate through) becomes too much to be handled in the client database, the client should be able to request a small separate sync subscription (scope), and store it as a separate client only collection (cache).

That separate local collection (with another name than the collection on the server) then really just contains the data that is rendered (optionally plus the previous/next documents of some neigboring increments). Thus the client can reliably use skip and limit locally to paginate through this separately cached and synced server scope (result of server's sort, filter, skip and limit). However, the client now always has to trigger a server query as well, to move the scope.

[julpod]

Sound interesting. I've read several times all your posts and I think that
I don't fully understand or i can't imagine yet how this could be
implemented correctly. I agree that requesting the server on every query
change should be highly improved. The issue here is to design well the
"how" and following your comments, there is something I find strange. Let's
check some questions:

Pagination should be done by client code, except when the amount of documents synced according to all subscribed scopes can not be handled efficiently on the client anymore.

How do you imagine a client implementing paging when minimongo does not
have the knowledge of all documents? (that's the reason why I've added a
new custom collection to each instance). If you only consider paging
(without filters applied) that would be alright since the resulset order
should be always the same but, if you have some filters applied, the skip
and limit document range would vary depending the filter criteria and for
that you need to perform a mongo (not minimongo) query against all
documents.

Usually, one chooses the published scopes so that they don't reveal only a limited set of info from the database and don't overwhelm the client (publish only recent documents, fields relevant to the user, etc.), and pagination can be handled by the client without any re-subs and document re-transmissions (even offline).

I think that here I don't understand what is written. one chooses the published scopes so that they don't reveal only a limited set of info from the database and don't overwhelm the client. I think that one chooses the
published scopes so that the do reveal only a limited set of info.
Otherwise, on a large app, minimongo should storage a very long set of
documents that not always is needed to render the current template, right?
Am I missing something?

`Nevertheless, if the data that the client is supposed to sort, filter and
look at (paginate through) becomes too much to be handled in the client
database, the client should be able to request a small separate sync
subscription (scope), and store it as a separate client only collection
(cache).

That separate local collection (with another name than the collection on
the server) then really just contains the data that is rendered (optionally
plus the previous/next documents of some neigboring increments). Thus the
client can reliably use skip and limit locally to paginate through this
separately cached and synced server scope (result of server's sort, filter,
skip and limit). However, the client now always has to trigger a server
query as well, to move the scope.`

Well, as far as I know, the package does that. Once you set an FC instance,
the Deps.autorun, manage 2 subscriptions: 1) for the mongo db query (that
is manged with self.query object) and 2) a dynamic created collection to
manage record count over the same query object. Both of these have their
own publishers on server and only transmits Mongodb results to minimongo.
So for example, if you query {filterA: 123, filterB: {$regex:'lorem', 'i'},
{skip:100, limit:10},{sort:{created_at:-1}} the publisher will send only 10
records to minimongo based on sort values.

So to be clear, once the user do something (paging, sorting, filter) the FC
client instance invalidates the computation for self.query and also Meteor
invalidates the FC instance subscriber. All of this happens if the query
has some changed property (if you are trying to query two times the same
query object, none invalidation never happens).

Could be very probable that I'm missing some concept to fully understand
your thoughts. Please let me know if it's the case. I would like to
understand more deeper these ideas to figure out some cool solution.

Thanks a lot!

On Tue, Jan 28, 2014 at 9:34 AM, testbird [email protected] wrote:

The distinction, that I think is important, is between scoping (server
code) and pagination (client code).

Pagination should be done by client code, except when the amount of
documents synced according to all subscribed scopes can not be handled
efficiently on the client anymore.

Usually, one chooses the published scopes so that they don't reveal only a
limited set of info from the database and don't overwhelm the client
(publish only recent documents, fields relevant to the user, etc.), and
pagination can be handled by the client without any re-subs and document
re-transmissions (even offline).

Nevertheless, if the data that the client is supposed to sort, filter and
look at (paginate through) becomes too much to be handled in the client
database, the client should be able to request a small separate sync
subscription (scope), and store it as a separate client only collection
(cache).

That separate local collection (with another name than the collection on
the server) then really just contains the data that is rendered (optionally
plus the previous/next documents of some neigboring increments). Thus the
client can reliably use skip and limit locally to paginate through this
separately cached and synced server scope (result of server's sort, filter,
skip and limit). However, the client now always has to trigger a server
query as well, to move the scope.

Reply to this email directly or view it on GitHubhttps://github.com//issues/9#issuecomment-33474280
.

toOit
Web consulting & development
www.tooit.com

_Julián Esteban [email protected]* :: +*54 9 11 3468-0401

[testbird]

Hi!

How do you imagine a client implementing paging when minimongo does not have the knowledge of all documents?

Until meteor provides a solution for that (maybe something like dynamically adding ranks and tagging documents with scope skips/offsetts), I think scope publications will just have to sync all documents up to a limit to the client (for every desired sort order without further filtering). Then, I believe, all sort, filter, and pagination results based on minimongo should be ok.

This (my) usecase of course requires that one is able to scope down all the data on the server, to a set that consists of all the data the client may require (in the context/template it is currently in), and that the whole set can be handled on the client. (Loading time can still be snappy by waiting only on a small subscription that just syncs what is necessary to render the requested url, and then letting the the full scope subscriptions sync in the background.)

I think that here I don't understand what is written. `one chooses the published scopes so that they don't reveal only a limited set

Oh shame on me, that was an ugly typo. Please scratch my "don't" there as well.

However, the client now always has to trigger a server query as well, to move the scope.

Well, as far as I know, the package does that.

Yes, thank you so much again for your explanations, I think now I kind of understand "how filter-collections works".

Maybe our conversation can lead to filter-collections supporting both use cases: To easily render and switch the filter, sort, and pagination features on local minimongo datasets, as well as on separately requested server side queries if necessary.

Hopefully, my other comments do also make a little more sence to you now?

---

By the way:

All of this happens if the query has some changed property (if you are trying to query two times the same query object, none invalidation never happens).

I had an idea for another optimization. If, instead of stoping the subscription and creating a new one, you add the new one, and only close the old subscription after the new one is ready, you may avoid retransmitting the documents that are part of both subscriptions.

[testbird]

To provide an example for the scopes that I believe should work with limited minimongo data subsets.

Imagine there are a lot of posts and you publish the 20 most recent, the 20 most popular, and the latest posts from the 20 most active authors. All these end up in (get synced into) the client's minimongo cache. (Skips are not allowed.)

Isn't it then so that all filtering and paginating along those sort ranks should also give the correct results on the client (only up to the 20th position of course.)?