Fault tolerance (data)

[t-saideep]

http://www.cs.berkeley.edu/~matei/papers/2011/tr_spark.pdf

http://www.cs.berkeley.edu/~matei/papers/2010/hotcloud_spark.pdf

They talk about a specific model which lets them recompute intermediate results if there is data loss. We can use this as an alternate to saving intermediate data for blocks which can be recomputed. For example File-crawler can resend files if any intermediate node fails.