Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spike upgrading comps algorithm with taichi #229

Open
dfsnow opened this issue Apr 16, 2024 · 1 comment · May be fixed by #236
Open

Spike upgrading comps algorithm with taichi #229

dfsnow opened this issue Apr 16, 2024 · 1 comment · May be fixed by #236
Assignees
@jeancochrane @dfsnow

Comments

@dfsnow
Copy link
Member

dfsnow commented Apr 16, 2024

The comps algorithm currently takes a long time to run. It takes 12 hours even with a reduced search space. However, the algorithm is highly parallelizable. It's possible that using some simple GPU programming could massively speed up the algorithm and make it tenable to run for every model. Specifically, I'm thinking that a quick spike using Python's taichi could be a worthwhile side adventure if someone has a spare day/needs a break from other work.
@jeancochrane jeancochrane linked a pull request Apr 29, 2024 that will close this issue
Open
@jeancochrane jeancochrane linked a pull request Apr 29, 2024 that will close this issue
[Do not merge] Spike upgrading comps algorithm with taichi #236
Open
@jeancochrane
Copy link
Contributor

I did a bunch of investigation into taichi in #236. TL;DR:

  • Taichi makes the code more complicated than numba and does not appear to be significantly faster, even with GPU support
  • We could speed up the comps pipeline 2x by simply using a bigger instance type (c5.24xlarge)
  • I am not going to update the pipeline to use a new instance type right now, for a couple of reasons:
    • Performance improvement work is still ongoing, and we may end up improving the algorithm and changing the instance type shortly anyway
    • We may not want to use the same instance type for comps and model training, since their resource requirements are different; we will sort out this design, including the specific instance types for our compute environments, when we work on Simplify build-and-run-batch-run design actions#16 later this year

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jeancochrane jeancochrane

@dfsnow dfsnow

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Do not merge] Spike upgrading comps algorithm with taichi ccao-data/model-res-avm
2 participants
@jeancochrane @dfsnow