Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement engine="thread" in ChunkRecordingExecutor #3526

Open
wants to merge 27 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
e0ef39b
proof of concept of chunkexecutor with thread
samuelgarcia Nov 8, 2024
a28c33d
for progress_bar the for res in results need to be inside the with
samuelgarcia Nov 8, 2024
67b055b
wip
samuelgarcia Nov 8, 2024
cecb211
wip
samuelgarcia Nov 12, 2024
4365321
Move worker_index to job_tools.py
samuelgarcia Nov 20, 2024
e929820
change max_threads_per_process to max_threads_per_worker
samuelgarcia Nov 20, 2024
d4a6e95
implement get_best_job_kwargs()
samuelgarcia Nov 20, 2024
423801b
oups
samuelgarcia Nov 20, 2024
1736b65
oups
samuelgarcia Nov 20, 2024
f0ec139
Feedback from Zach and Alessio better test for waveforms_tools
samuelgarcia Nov 22, 2024
cc8b4c4
fix tests
samuelgarcia Nov 22, 2024
5cf6ace
Merge branch 'main' of github.com:SpikeInterface/spikeinterface into …
samuelgarcia Nov 22, 2024
c16ca72
oups
samuelgarcia Nov 22, 2024
155ab31
merci zach
samuelgarcia Nov 25, 2024
dd385e2
merge with main
samuelgarcia Dec 20, 2024
6ecbb01
quick benchmark
samuelgarcia Jan 8, 2025
9329243
Pierre suggestion
samuelgarcia Jan 8, 2025
6daf18d
Merge branch 'main' into parralel_with_thread
samuelgarcia Jan 8, 2025
61f8509
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jan 8, 2025
cf42955
Merge branch 'main' of github.com:SpikeInterface/spikeinterface into …
samuelgarcia Jan 21, 2025
6b50837
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jan 21, 2025
f4c2480
Remove the glonals warnings for job_kwargs when n_jobs=1.
samuelgarcia Jan 21, 2025
4a8e622
Merge branch 'parralel_with_thread' of github.com:samuelgarcia/spikei…
samuelgarcia Jan 21, 2025
760538b
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jan 21, 2025
90c4d13
Merge branch 'main' of github.com:SpikeInterface/spikeinterface into …
samuelgarcia Jan 21, 2025
87ed340
Merge branch 'parralel_with_thread' of github.com:samuelgarcia/spikei…
samuelgarcia Jan 21, 2025
8df1c17
Merge branch 'main' into parralel_with_thread
alejoe91 Jan 22, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc/get_started/quickstart.rst
Original file line number Diff line number Diff line change
Expand Up @@ -287,7 +287,7 @@ available parameters are dictionaries and can be accessed with:
'detect_threshold': 5,
'freq_max': 5000.0,
'freq_min': 400.0,
'max_threads_per_process': 1,
'max_threads_per_worker': 1,
'mp_context': None,
'n_jobs': 20,
'nested_params': None,
Expand Down
2 changes: 1 addition & 1 deletion src/spikeinterface/core/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@
write_python,
normal_pdf,
)
from .job_tools import ensure_n_jobs, ensure_chunk_size, ChunkRecordingExecutor, split_job_kwargs, fix_job_kwargs
from .job_tools import get_best_job_kwargs, ensure_n_jobs, ensure_chunk_size, ChunkRecordingExecutor, split_job_kwargs, fix_job_kwargs
from .recording_tools import (
write_binary_recording,
write_to_h5_dataset_format,
Expand Down
2 changes: 1 addition & 1 deletion src/spikeinterface/core/globals.py
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ def is_set_global_dataset_folder() -> bool:


########################################
_default_job_kwargs = dict(n_jobs=1, chunk_duration="1s", progress_bar=True, mp_context=None, max_threads_per_process=1)
_default_job_kwargs = dict(pool_engine="thread", n_jobs=1, chunk_duration="1s", progress_bar=True, mp_context=None, max_threads_per_worker=1)

global global_job_kwargs
global_job_kwargs = _default_job_kwargs.copy()
Expand Down
Loading