Skip to content
This repository has been archived by the owner on Jul 15, 2020. It is now read-only.

Refactor Pzsvc-Exec to use CF Tasks for Auto-Scalability #205

Open
ssaniev opened this issue Feb 26, 2018 · 0 comments
Open

Refactor Pzsvc-Exec to use CF Tasks for Auto-Scalability #205

ssaniev opened this issue Feb 26, 2018 · 0 comments

Comments

@ssaniev
Copy link
Contributor

ssaniev commented Feb 26, 2018

As a BF User, I need the BF application to scale so that I can reliably use the product even while many other users are using it; so that I may accomplish my mission in a timely manner when required.
Currently, Beachfront Pzsvc-ndwi-py (running Pzsvc-exec) scales linearly on a manual basis. That is, the application only scales when a developer or administrator updates the number of instances of the application. This is difficult to automate because the algorithm does not uniformly use system resources, so it can be hard to automatically detect when scaling is needed.
In order to implement proper auto-scalability, it is instead recommended to use Cloud Foundry Tasks. cf run-tasks. This can be used to create one-off containers in PCF, tied to the host application, that can run a single task before that container is reaped. Using this system, we can create individual tasks for each run of the detection algorithm, that will scale uniformly and linearly with available PCF resources. Using Tasks, we will not be required to manage any state, or perform any polling of resources. This makes it an incredibly desirable solution for BF Scaling.
Pzsvc-Exec needs to be reworked to use CF Tasks. Code refactoring should be kept to a minimum so as not to break any existing code. This is not intended to be a major refactor - just a rework to use CF Tasks.
The design will have the Taskworker (now renamed to Dispatcher) be the single process that runs on the Pzsvc-ndwi-py application. This process will poll Piazza for work and when found, will create a task that runs the Pzsvc-exec application (now renamed to Worker). This task will run, isolated, and send results to Piazza; at which point CF will clean it up. This allows us to have a single application for the algorithm that will scale with tasks. 

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant