Added a broadcast via spark & switched to a single long-lived spark context #51
Conversation
| @@ -54,9 +56,17 @@ class VeloxApplication extends Application[VeloxConfiguration] with Logging { | |||
|
|
|||
| // this assumes that etcd is running on each velox server | |||
| val etcdClient = new EtcdClient(conf.hostname, 4001, conf.hostname, new DispatchUtil) | |||
| logWarning("Starting spark context") | |||
|
Overall this looks good. I like having the long-lived |
| private val cachedValues: mutable.Map[Version, T] = mutable.Map() | ||
|
|
||
| override def put(value: T, version: Version): Unit = this.synchronized { | ||
| sc.parallelize(Seq(value)).saveAsObjectFile(s"$path/$version") |
There was a problem hiding this comment.
What happens if these methods get called concurrently and they aren't synchronized?
There was a problem hiding this comment.
Nevermind, dumb question. They could both be writing to the same place at the same time.
|
LGTM |
Added a broadcast via spark & switched to a single long-lived spark context
Relates to (but doesn't fully cover) issues #48 #49 and #46