Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Module import takes a long while #284

Closed
3 tasks done
hombit opened this issue Jun 4, 2024 · 2 comments
Closed
3 tasks done

Module import takes a long while #284

hombit opened this issue Jun 4, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@hombit
Copy link
Contributor

hombit commented Jun 4, 2024

Bug report

import hipscat takes ~5s on my laptop (Apple M2Pro) and ~8-9s on Bridges2 (AMD Epyc 7742). In some cases it takes up to two minutes to import it on Bridges2.

cProfile shows that it may be connected to numba compilation and dynamic library loading, but further investigation is needed.

Part of the call graph for the 2-minute import case (Bridges2)
image

Part of the call graph for a "normal" 5s import case (laptop). Times do not count, because 5s is the wall time, while time shows 200% CPU usage
image

Before submitting
Please check the following:

  • I have described the situation in which the bug arose, including what code was executed, information about my environment, and any applicable data others will need to reproduce the problem.
  • I have included available evidence of the unexpected behavior (including error messages, screenshots, and/or plots) as well as a descriprion of what I expected instead.
  • If I have a solution in mind, I have provided an explanation and/or pseudocode and/or task list.
@hombit hombit added the bug Something isn't working label Jun 4, 2024
@smcguire-cmu
Copy link
Contributor

This should be closed by #395 , can you test it?

@hombit
Copy link
Contributor Author

hombit commented Oct 25, 2024

It is 1.5-2.5 faster now. I haven't profiled it yet

@nevencaplar nevencaplar moved this to In Progress in HATS / LSDB Oct 26, 2024
@github-project-automation github-project-automation bot moved this from In Progress to Done in HATS / LSDB Oct 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: Done
Development

No branches or pull requests

3 participants