Public Beta 0.11 Release Planning

[jhellerstein]

Deliverables

• Hydroflow+ as primary DSL (@shadaj)
  • [ x ] 1.0 of language/API
  • docs
  • examples
• Initial release of stdlib (@jhellerstein)
  • move HF+ examples
  • flesh out initial list of targets
    • Anna KVS
      • Gossip
    • 2PC
    • Voting
    • Paxos
    • PBFT
    • Collaborative editor
    • Leader election?
    • Membership?
    • Load Balancer?
• Lattices/Algebra crate (@conor-23, @jhellerstein)
  • CRDTs for consistency
  • Move/benchmark RGA (collab editing tree)
  • Z-sets
• Hydroflow cleanup (@jhellerstein)
  • operator pruning and naming consistency
  • finish docs
  • re-add poll_futures() feat(hydroflow): Added poll_futures and poll_futures_async operators. #1143 revert: "feat(hydroflow): Added poll_futures and poll_futures_async operators. (#1143)", fix #1183 #1216 (@MingweiSamuel)
• Flo syntax/optimizations (@MingweiSamuel)
  • 1. Flo semantics/syntax (nested iteration) feat(hydroflow): implement flo loop syntax parsing #1507 feat(hydroflow_lang): flo basic graph structure checks #1513
  • 2. Implement this in the current scheduler
  • 3. Pipeline types, operator type negotiation (Implement this in the codegen/operatorinfo types)
  • 4+. Runtime optimizations beyond in/out trees
    • Graph is DAG within an iteration
    • Later: eliminate operator resumability (in many cases). All operators become fully “stateless”
    • Laziness short-circuiting
    • Borrowing tee
• KVS service (@rohitkulshreshtha)
  • benchmarking replicated HF Anna 1.0
  • scale-out/elasticity support (Anna 2.0)
    • metrics in HF Anna
    • read replicas
      • serializing HF data structures (collections, lattices, etc) to NW and disk
        • HF+ currently uses bincode
    • (re-)partitioning in HF??

Post 1.0

• HydroDeploy 2.0 / k8s integration
• unordered semantics support for HF+
  • unordered collections for HF+
  • unordered networking for HF+ clusters (union)
  • FT model for clusters in HF+
  • CALM-style consistency guaranteed by HF+ types