Join: Delayed primary stream

[farbodahm]

Currently Streame doesn't support delayed primary stream; It means if we first receive event from the Streaming topic and then receive event from the Table topic, it will not be written to output. So we need some kind of retry mechanism.

Here is what Photon mentions:

Delayed primary stream: A click can only be joined if the corresponding query event is available in the logs datacenter. Logically, the query event always occurs before the corresponding click. However, the servers generating click events and query events are distributed throughout the world (to minimize end-user latency), and click and query logs are shipped independently to logs datacenters. The volume of query logs is orders of magnitude more than the volume of click logs, thus, it is not uncommon for a subset of query logs to be delayed relative to the corresponding click logs. Pho- ton needs to be able to join whenever the query is available. This makes Photon different from standard RDBMS where a foreign key must always exist in the primary table.