frankie_edits_pickled_data

[frankiepe]

Description

Bypasses the need to load in the large json files by loading the Trace objects in from pickle files (tests/test_data/tr_before.pkl and tests/test_data/tr_after.pkl)

Supports #21

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Testing

• Testing is done automatically and codecov shows test coverage
• This cannot be tested automatically

Documentation checklist

• I have updated all documentation in the code where necessary.
• I have checked spelling in all (new) comments and documentation.
• I have added a note to RELEASE.md if relevant (new feature, breaking change, or notable bug fix).

[frankiepe]

Having issues with the pickled Trace class (and dependencies)

[kwabenantim]

Pickling the Trace object alone will probably not be sufficient because subsequent operations still depend on the original trace data:

======================================================================
ERROR: test_run_qc (tests.test_herg_qc.TestHergQC.test_run_qc)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/.../pcpostprocess/tests/test_herg_qc.py", line 57, in test_run_qc
    before = tr_before.get_trace_sweeps(sweeps)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.../pcpostprocess/venv/lib/python3.12/site-packages/syncropatch_export/trace.py", line 169, in get_trace_sweeps
    with open(os.path.join(self.filepath, trace_file), 'r') as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'tests/test_data/13112023_MW2_FF/staircaseramp (2)_2kHz_15.01.07/Tracedata_000.dat'

A first step might be to remove unused traces from the test data as suggested in #37. Currently, only the 180M of trace data highlighted below is used in tests:

[456M]  13112023_MW2_FF
├── [ 46M]  StaircaseInStaircaseramp (2)_2kHz_15.01.51
├── [ 46M]  StaircaseInStaircaseramp (2)_2kHz_15.12.17
├── [1.7K]  export_config.py
├── [ 91M]  staircaseramp (2)_2kHz_15.01.07    <-- trace before
├── [ 91M]  staircaseramp (2)_2kHz_15.06.53
├── [ 91M]  staircaseramp (2)_2kHz_15.11.33    <-- trace after
└── [ 91M]  staircaseramp (2)_2kHz_15.17.19

Pickling the output from Trace.get_trace_sweeps() doesn't seem to reduce the data size:

├── [180M]  trace_sweeps_after.pkl
├── [180M]  trace_sweeps_before.pkl

If pickling won't reduce test data size, it's probably a good idea to auto download the test data instead, as #35 suggests.