diff --git a/docs/conf.py b/docs/conf.py index a6d6197..7ef8306 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -64,9 +64,9 @@ # built documents. # # The short X.Y version. -version = '0.4' +version = '0.5' # The full version, including alpha/beta/rc tags. -release = '0.4' +release = '0.5' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. diff --git a/docs/contributing.rst b/docs/contributing.rst index fbbd730..a2cb74b 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -57,8 +57,8 @@ There are several ways you can get involved: `this one `__. -Contributions Acknowledgement ------------------------------ +Contributions Acknowledgment +---------------------------- Any contributions to this documentation will be listed in the front page, just below the authors. @@ -76,3 +76,46 @@ Contributor Code of Conduct The Photon-HDF5 team subscribes to the Contributor Covenant, version 1.0.0, available from http://contributor-covenant.org/version/1/0/0/. + +.. _compatibility: + +Maintaining Compatibility +------------------------- + +Maintaining compatibility between versions is important. + +Compatibility can be broadly definined in two ways + +1. **Backward compatibility** means that files from previous version can be read and used by + newer versions. +2. **Forward compatibility** means that files from newer versions can be read by older versions. + +Backward compatibility is generally much easier to maintian, as it largely means not removing +anything from the format or changing something once it is established. +Forward compatibility on the other hand is more complicated, as new features cannot be read by +older versions, basically by definition. So the principle is that an older version should be +able to interpret any file of a newer version that does *not* contain/use the new feature. +Additionally, the new feature should be added in a way that older versions can still read +the data that does not involve the new feature. + +In Photon-HDF5, new versions must be backwards compatible, and foward compatibility should be +maintained as much as possible. For this we have 5 main principles for maintaining compatibility: + +#. New fields should always be optional/conditionally mandatory (i.e. mandatory only when a new + feature is used in the particular experiment) with minor version updates, major version + updates may make a new field mandatory. +#. The data type (options) of a field will not change from version to version. +#. Fields cannot be removed. +#. Whether or not a field is required must be implemented in a way consistent with previous + versions, meaning: + + a. Any field introduced as mandatory will necessarily be mandatory in all future versions + b. For conditionally mandatory fields, if a set of conditions requires a field to be + mandatory in a previous version, it will also be mandatory under those conditions + in future versions. + c. In cases where new features (usually in another field) are added, then the new + implementation should keep the field mandatory in all cases where, ignoring the new feature. +#. Validators should be considered version specific, they cannot validate photon-HDF5 files of + versions newer that what they were designed for. However, we can implement a + "permissive/strict" option (or other similar name) determining whether or not the validator + checks for unknown fields, and whether or not to throw an error or simply warn. \ No newline at end of file diff --git a/docs/generic.rst b/docs/generic.rst index c81a3b0..35c954b 100644 --- a/docs/generic.rst +++ b/docs/generic.rst @@ -6,6 +6,9 @@ For measurement types which cannot be described by any other it is possible to use the "generic" type and use the ``/setup`` fields to describe the exact configuration. +The "generic" type is the encouraged type for most experiments +going forward, as it supports arbitrary configurations. + Examples -------- @@ -25,8 +28,8 @@ A setup detecting two polarizations in the detection path will be defined by:: and will also specify the detector used for each polarizations:: /photon_data/measurement_specs/detectors_specs/ - polarization_ch1 = 0 - polarization_ch2 = 1 + polarization_ch1 = [0, ] + polarization_ch2 = [1, ] (where the values 0 and 1 are only examples). In addition ``/setup/lifetime`` will indicate whether the measurement has TCSPC data or not. Finally, the optional field @@ -46,8 +49,8 @@ in two using a non-polarizing beam splitter will be defined by:: and the measurement_specs will contain:: /photon_data/measurement_specs/detectors_specs/ - split_ch1 = 0 - split_ch2 = 1 + split_ch1 = [0, ] + split_ch2 = [1, ] (where the values 0 and 1 are only examples). In addition ``/setup/lifetime`` will indicate whether the measurement has TCSPC data or not. Finally, the optional @@ -74,8 +77,8 @@ as in the "smFRET" ``measurement_type``. In particular:: and:: /photon_data/measurement_specs/detectors_specs/ - spectral_ch1 = 0 - spectral_ch2 = 1 + spectral_ch1 = [0, ] + spectral_ch2 = [1, ] TCSPC measurements ^^^^^^^^^^^^^^^^^^ diff --git a/docs/intro.rst b/docs/intro.rst index d4ede9d..7f5d2fe 100644 --- a/docs/intro.rst +++ b/docs/intro.rst @@ -12,7 +12,9 @@ as a standard container format for a broad range of experiments involving confocal microscopy. Examples are confocal smFRET experiments performed with a single or multiple excitation spots. Both `μs-ALEX `_ -and ns-ALEX data are supported. +and `PIE `_ +(aka `ns-ALEX `_)data +are supported. What problems are we trying to solve? @@ -47,7 +49,8 @@ The main design principles are: We aim to define a format that has a minimal set of specifications and therefore is easy to implement. At the same time, it is important that the format can be -expanded to accommodate new use cases while maintaining backward compatibility. +expanded to accommodate new use cases while maintaining backward compatibility +(see :ref:`compatibility`. To achieve simplicity, the only required file characteristics are a general file layout and the presence of a few basic attributes and parameters. diff --git a/docs/limitations.rst b/docs/limitations.rst index 034fbf0..8a35bea 100644 --- a/docs/limitations.rst +++ b/docs/limitations.rst @@ -9,7 +9,7 @@ please contact us. Timestamps with rollover ------------------------ -In Photon-HDF5 timestamps are always signed 64 bit integers. Thanks to +In Photon-HDF5 timestamps are usually signed 64 bit integers. Thanks to compression, there is no size penalty compared to 32 bit integers. Most timestamping hardware produce a timestamp with 24 or 32 bits and a rollover flag in order to compute the full "unwrapped" timestamp. diff --git a/docs/phdata.rst b/docs/phdata.rst index 2fef3e1..2207a28 100644 --- a/docs/phdata.rst +++ b/docs/phdata.rst @@ -19,7 +19,7 @@ A screen-shot of a typical Photon-HDF5 file opened in HDFView is shown here: :alt: HDView screenshot showing a Photon-HDF5 file The previous figure shows the 5 main groups contained in a Photon-HDF5 file. -Of these, :ref:`/photon_data ` and :ref:`/setup ` +Of these, :ref:`/photon_data/` and :ref:`/setup/` contains the raw data and all the information needed for the analysis. A schematic overview is shown in the next figure: @@ -36,16 +36,26 @@ analysis: A brief description of these 3 metadata groups follows: -- :ref:`/identity `: +- :ref:`/identity/`: Information about the data file. -- :ref:`/provenance `: +- :ref:`/provenance/`: Information about the original data file (when the Photon-HDF5 file has been converted from another format). -- :ref:`/sample `: +- :ref:`/sample/`: Description of the measured sample. +Finally, the group :ref:`/user/` may be included to store +additional data not defined in the Photon-HDF5 spec. +Unlike all other groups, the contents of the ``/user/`` group are not +rigidly defined. However, guidelines are provided for how data should +be stored and what should be included. + +.. note:: + Since the format of the :ref:`/user/` group is not rigid, it will be ignored + by official readers. + .. _root_params: @@ -60,8 +70,8 @@ The root node ("/") in a Photon-HDF5 file contains the following fields: In addition, the root node has the following attributes which distinguish a Photon-HDF5 file from other HDF5 files: -- **format_name**: must contain the string "Photon-HDF5" -- **format_version**: (string) the Photon-HDF5 format version. +- **/format_name**: (string) must contain the string "Photon-HDF5" +- **/format_version**: (string) the Photon-HDF5 format version. .. _photon_data_group: @@ -69,39 +79,56 @@ distinguish a Photon-HDF5 file from other HDF5 files: Photon-data group ----------------- -This section describes the layout and fields in the **/photon_data** group. -Note that only the kind of data is specified (i.e. scalar, +This section describes the layout and fields in the ``/photon_data/`` group. +Note that only the kind of data is specified (i.e. scalar (float or integer), integer array, float array), but no data type size is mandated. For arrays, the most commonly used data-type is indicated. +.. note:: + In multi-spot measurements each spot is represented in a separate group + named sequentially ``photon_data0``, ``photon_data1`` ... + + See :ref:`multi_spot` for more details. + Mandatory fields: -- **timestamps**: (array) photon timestamps. Typical data-type int64. +- **timestamps**: (integer array) photon timestamps. Typical data-type int64. - **timestamps_specs/** - **timestamps_unit**: (float) timestamp units in *seconds*. Optional if there is only 1 detector, otherwise mandatory: -- **detectors**: (array of integers) :ref:`detector_ids` for each timestamp. - Typical data-type uint8. +- **detectors**: (integer array) :ref:`detector IDs` for each timestamp. + Must be the same length as ``timestamps``. Typical data-type uint8. When the dataset contains TCSPC or nanotime information (i.e. arrival time of each photon with respect to a laser pulse), the following fields must be present: -- **nanotimes**:(array of integers) TCSPC nanotimes. Conventionally the time +- **nanotimes**: (integer array) TCSPC nanotimes. Conventionally the time axis direction is the "natural" direction, i.e. lifetime decays look - correctly oriented in time. For more details see :ref:`nanotimes_time_axis`. + correctly oriented in time. Must be the same length as ``timestamps`` + For more details see :ref:`nanotimes_time_axis`. Typical data-type uint16. - **nanotimes_specs/** - **tcspc_unit**: (float) TAC/TDC bin size (in seconds). + In official photon-HDF5 readers, this field is used for determining resolution of nanotimes. + Other fields in ``nanotimes_specs`` are ignored if not needed. - **tcspc_num_bins**: (integer) number of TAC/TDC bins. - - **tcspc_range**:(float) (optional) full-scale range of the TAC/TDC (in seconds). + - **tcspc_range**:(float) (*Optional*) full-scale range of the TAC/TDC (in seconds). This a derived field equal to ``tcspc_unit * tcspc_num_bins``. -Finally, if the data come from a simulation, ``/photon_data`` may contain: +.. note:: -- **particles**: (array of integers) a particle ID (integer) for each + In the unusual case where detectors in a single spot do not share the same + settings, then ``nanotimes_specs`` should be omitted and the data + exclusively stored in the equivalent fields in ``/setup/detectors/`` where + per-detector settings can be stored. Doing so will mean readers of ` which supports -a large combination of setup configurations. In this case, values in -``/setup/`` will determine mandatory ``measurement_specs`` fields. -If you feel that a new "specific" measurement type is needed -for your application we have :doc:`instructions ` +an arbitrary range of setup configurations. In this case, values +in :ref:`/setup/ ` will determine the +mandatory fields in the :ref:`measurement_specs/ ` group. +The "generic" measurement type is recommended to be used in most cases +as it covers all other types possible within the photon-HDF5 spec. + +If you feel that a new "specific" measurement type would be especially +useful to add for your application we have :doc:`instructions ` to propose a new one. -An advantage of a "specific" measurement type compared to "generic", -is that additional fields can be made mandatory. -The following ``measurement_specs`` fields are present in specific -measurement types. +The following ``measurement_specs`` fields are used to specify key values for +determining to which photon stream a given photon belongs when using +alternated/interleaved excitation (both continuous wave and pulsed excitation). -For μs-ALEX, 2, 3 or N colors: +For continuous wave excitation with any number of colors: -- **alex_period**: (integer or float) duration of one complete excitation - alternation period expressed in timestamp units. The alternation period - is equal to ``alex_period * timestamps_unit``. - -For ns-ALEX (or lifetime with no alternation): - -- **laser_repetition_rate**: (float) excitation laser pulse repetition rate - in *Hertz*. - -For 2-color (or more) μs-ALEX and ns-ALEX (optional): +- **alex_period**: (integer or float) [μs-ALEX only] duration of one complete + excitation alternation period expressed in *timestamp units*. The alternation + period is equal to ``alex_period * timestamps_unit``. - **alex_offset**: (scalar) [μs-ALEX only] Time offset (in timestamps units) to be applied to the timestamps array before computing the μs-ALEX histogram. It is assumed that the μs-ALEX alternation histogram is the histogram of (``timestamps`` - ``alex_offset``) **MOD** ``alex_period``. -- **alex_excitation_period1**: (array with an even-number of integer - elements, normally 2) start and stop values identifying the excitation - periods for the **first** wavelength in ``/setup/excitation_wavelengths`` - (which is the shortest wavelength). +When ``photon_data/nanotimes`` is present the following is mandatory: + +- **laser_repetition_rate**: (float) excitation laser pulse repetition rate + in *Hertz*. + +For alternated excitation experiments (e.g. μs-ALEX and ns-ALEX), the following +fields are used to specify the alternation period: + +- **alex_excitation_period1**: (integer array with an even-number of + elements, normally 2) start and stop value(s) identifying the excitation + period(s) for the **first** wavelength in ``/setup/excitation_wavelengths`` + (which is the shortest wavelength). Indexing of start and stop value(s) + starts from 0, and the period is defined as [start, stop). In smFRET experiments with 2-colors excitation this field defines the *donor excitation period*. See also :ref:`wavelengths_order` and note below. -- **alex_excitation_period2**: (array with an even-number of integer +- **alex_excitation_period2**: (integer array with an even-number of elements, normally 2) start and stop values identifying the excitation periods for the **second** wavelength in ``/setup/excitation_wavelengths``. In smFRET experiments with 2-colors excitation this field defines the @@ -179,7 +212,7 @@ For 2-color (or more) μs-ALEX and ns-ALEX (optional): For 3 (or more) colors alternated or interleaved excitation: -- **alex_excitation_period3**: (array with an even-number of integer +- **alex_excitation_period3**: (integer array with an even-number of elements, normally 2) start and stop values identifying the excitation periods for the **third** wavelength in ``/setup/excitation_wavelengths``. See also :ref:`wavelengths_order` and note below. @@ -189,11 +222,17 @@ For 3 (or more) colors alternated or interleaved excitation: .. note:: For μs-ALEX, both *alex_excitation_period1* and *alex_excitation_period2* - are 2-element arrays and are expressed in *timestamps_units*. + are 2-element arrays and are expressed in *timestamps units* as defined in + ``timestamps_specs/timestamps_unit``. For ns-ALEX (also known as PIE), they are arrays with an even-number of elements, comprising as many start-stop nanotime pairs as there are excitation periods within the TAC/TDC range. - In this case the values are expressed in *nanotimes_units*. + In this case the values are expressed in *nanotimes units* as defined in + ``nanotimes_specs/tcspc_unit``. If their is a single pair of start-stop + times, then the array should be 1D, but if multiple start-stop pairs are + specified, it is preferable to specify each pair in a separate row, thus + resulting in a 2D array of shape Nx2. If the array is 1D official readers + will tread ever consecutive pair of points as a separate start-stop pair. For more details see :ref:`alex_period_def`. @@ -202,10 +241,11 @@ For 3 (or more) colors alternated or interleaved excitation: Detectors specs """"""""""""""" -Within **measurement_specs**, the **detectors_specs/** sub-group -contains all the :ref:`pixel ID `--detection channel -associations, i.e. spectral bands, polarizations or -:ref:`beam-split channels `. +Within ``measurement_specs/``, the ``detectors_specs/`` sub-group +contains all the :ref:`detector ID`--event +associations, i.e. spectral bands, +polarizations, :ref:`beam-split channels`, +and non-photon events. When a measurement records more than 1 spectral band, the fields: @@ -213,18 +253,21 @@ When a measurement records more than 1 spectral band, the fields: - **spectral_ch2** - etc... -specify which detector pixel is employed in each spectral band. When the measurement -records only 1 spectral band these fields may be omitted. The spectral bands -are strictly ordered for increasing wavelengths. For example, for 2-color -smFRET measurements ``spectral_ch1`` and ``spectral_ch2`` represent the +specify which detector is employed in each spectral band. When the +measurement records only 1 spectral band these fields may be omitted. The +spectral bands are strictly ordered for increasing wavelengths +(see :ref:`wavelength_order`). +For example, for 2-color smFRET measurements +``spectral_ch1`` and ``spectral_ch2`` represent the *donor* and *acceptor* channel respectively. -If a measurement records more than 1 polarization states, the fields: +If a measurement records more than 1 polarization state, the fields: - **polarization_ch1** - **polarization_ch2** +- etc... -specify which detector pixel is used for each polarization. When the measurement +specify which detector ID is used for each polarization. When the measurement records only one polarization, these fields may be omitted. When the detection light is split into 2 channels using a non-polarizing @@ -232,18 +275,48 @@ beam-splitter the fields: - **split_ch1** - **split_ch2** +- etc.. + +specify which detector ID is used in each of the "beam-split" channels. -specify which detector pixel is used in each of the "beam-split" channels. +If :ref:`non-photon IDs ` are also recorded, then the fields (*new in v0.5*): -All previous fields are arrays containing one or more :ref:`detector_ids`. -For example, a 2-color smFRET measurement will have only one value in -``spectral_ch1`` (donor) and one value in ``spectral_ch2`` -(acceptor). A 2-color smFRET measurement with polarization +- **non_photon_id1** +- **non_photon_id2** +- etc... + +are used to categorize each non-photon id. Currently there are no specifications +for the type of markers. These should be explained in +the :ref:`/user/experimental_settings` group for human readable +understanding enabling *custom* code to be written to process these detector IDs. +All official software for v0.5 will ignore data points specified as non-photon IDs. +Future releases may specify sorts of measurements that will have defined uses +for particular non-photon IDs. + + +.. note:: + + Use of :ref:`non-photon IDs ` is discouraged as their purpose + is defined in the :ref:`/user/experimental_settings` group, + and thus lack an internal definition. Additionally, the use of non-photon ids + may cause poorly implemented readers designed for ` +(Detector IDs for all ``spectral_chX``, ``polarization_ch1`` and ``split_chX``, +and :ref:`non-photon IDs ` for ``non_photon_chX``). +If a given experiment type does not involve a given field type, that field should +be omitted. For example, a 2-color smFRET measurement without polarization or split +channels (2 detectors) will have only one value in ``spectral_ch1`` (donor) and one +value in ``spectral_ch2`` (acceptor). A 2-color smFRET measurement with polarization (4 detectors) will have 2 values in each of the ``spectral_chX`` and ``polarization_chX`` fields (where X=1 or 2). -For a multispot smFRET measurement, in each ``photon_dataN`` group, -there will be ``spectral_chX`` fields containing the donor/acceptor -pixels used in that spot (see :ref:`multi_spot`). +For a multispot smFRET measurement, in each ``/photon_dataX/`` group, +there will be ``spectral_chX`` fields containing the donor/acceptor detector IDs +(pixels) used in that spot (see :ref:`multi_spot`). .. _setup_group: @@ -251,17 +324,19 @@ pixels used in that spot (see :ref:`multi_spot`). Setup group ----------- -The **/setup** group contains information about the measurement setup. +The ``/setup/`` group contains information about the measurement setup. This group can be **absent** in some files, an example being a file containing only detector dark counts, for which the following fields do not necessarily have a meaning. When setup is present, the following 9 fields are mandatory: - **num_spectral_ch**: (integer) number of distinct detection spectral - channels. For example, in a 2-color smFRET experiment there are 2 + channels. For example, in a simple 2-color smFRET experiment there are 2 detection spectral channels (donor and acceptor), therefore its value is 2. When there is a single detection channel or all channels detect the same spectral band, its value is 1. + There should be equivalent number of ``spectral_chX`` fields in each + ``/photon_data/measurement_specs/detectors_specs`` group. - **num_polarization_ch**: (integer) number of distinct detection polarization channels. For example, in polarization anisotropy measurements, its value @@ -269,24 +344,32 @@ When setup is present, the following 9 fields are mandatory: When there is a single detection channel or all channels detect the same polarization (including when no polarization selection is performed) its value is 1. + There should be equivalent number of ``polarization_chX`` fields in each + ``/photon_data/measurement_specs/detectors_specs`` group. - **num_split_ch**: (integer) number of distinct detection channels detecting the same spectral band **and** polarization state. For example, when a non-polarizing beam-splitter is employed in the detection path, its value is 2. When no splitting is performed, its value is 1. + There should be equivalent number of ``split_chX`` fields in each + ``/photon_data/measurement_specs/detectors_specs/`` group. - **num_spots**: (integer) the number of excitation (or detection) "spots" in the sample. This field is 1 for all the measurements using a single confocal excitation volume. + If field is greater than 1 there should be equivalent number of + ``/photon_dataX/`` groups (see :ref:`multi_spot`) -- **num_pixels**: (integer) total number of detector pixels. For example, +- **num_pixels**: (integer) total number of detector ids. For example, for a single-spot 2-color smFRET measurement using 2 single-pixel SPADs as detectors this field is 2. This field is normally equal to ``num_pixels = num_spectral_ch * num_split_ch * num_polarization_ch * num_spot``. This equation is not valid when the optical setup has non-symmetric branches, for example if the emission path is split in two spectral bands and only one of the two is further split in two polarizations. + This only counts actual detectors that produces photon IDs, and does not + include :ref:`non-photon IDs `. - **excitation_cw**: (array of booleans) for each excitation source, this field indicates whether excitation is continuous wave (CW), *True*, @@ -295,20 +378,20 @@ When setup is present, the following 9 fields are mandatory: ``excitation_wavelengths`` and is in increasing order of wavelengths. - **lifetime**: (boolean) *True* (or 1) if the measurements includes a - *nanotimes* array of (usually sub-ns resolution) photon arrival times with - respect to a laser pulse (as in TCSPC measurements). + ``photon_data/nanotimes`` array of (usually sub-ns resolution) photon + arrival times with respect to a laser pulse (as in TCSPC measurements). - **modulated_excitation**: (boolean) *True* (or 1) if there is any form of excitation modulation either in the wavelength space (as in μs-ALEX or PAX) or in the polarization space. This field is also *True* for - pulse-interleaved excitation (PIE) or ns-ALEX measurements. + pulse-interleaved excitation (PIE/ns-ALEX) measurements. - **excitation_alternated**: (array of booleans) *New in version 0.5.* For each excitation source, this field indicates whether the excitation is alternated (True) or not alternated (False). In ALEX measurement all sources are alternated. In PAX measurements only one of the two sources is alternated. - In measurements with pulsed interleaved excitation (PIE), this field + In measurements with pulsed-interleaved excitation (PIE/ns-ALEX) , this field should contain all False. .. _detectors_group: @@ -316,33 +399,44 @@ When setup is present, the following 9 fields are mandatory: Detectors group ^^^^^^^^^^^^^^^^ -*New in version 0.5.* The group **/setup/detectors** -contains arrays with one element per detector's pixel. The allowed fields are: +*New in version 0.5.* The group ``/setup/detectors/`` +contains arrays with one element per detector (detector ID). +This group may be **absent** *if* the :ref:`/photon_data/` group(s) do **not** contain ``detectors`` arrays. +The allowed fields are: - - **id** (array of int): *Mandatory.* Number used by in ``/photon_data/detectors`` to - identify the pixel. - - **id_hardware** (array of int): *Optional.* Detector numbers as used by the + - **id** (integer array): *Mandatory.* Number used by in + ``/photon_data/detectors`` to identify the :ref:`detector ID `. + This include both normal photon IDs and non-photon IDs. + - **id_hardware** (integer array): *Optional.* Detector numbers as used by the acquisition hardware if different from ``id``. - - **label** (array of string): *Optional.* A human-readable label for each detector. - - **counts** (array of int): *Optional.* Number of timestamps counted by each detector. + - **spot** (integer array): *Multispot only, mandatory.* The spot number each + detector ID is used in. If present, must be of the same length as ``id``. + - **label** (array of string): *Optional.* A human-readable label for each detector ID, including non-photon IDs. + If present, must be of the same length as ``id``. - **module** (array of string): *Multispot only, optional.* Name of the module each - pixel belongs to. - - **position** (2-D array of int): *Multispot only, optional.* Columns are x,y - positions of each pixel in the array. - - **dcr** (array of float): *Optional.* Dark counting rate in Hz for each pixel. - - **afterpulsing** (array of float): *Optional.* Afterpulsing probability - for each pixel. - - **spot** (array of int): *Multispot only, mandatory.* The spot number each pixel is - used in. - - **tcspc_unit:** (array of float) array of TAC/TDC bin size (in seconds). + pixel belongs to. If present, must be of the same length as ``id``. + - **position** (2-D integer array): *Multispot only, optional.* Columns are x,y + positions of each spot in the array. Must be of the length specified + in ``num_spots``. + - **tcspc_unit:** (float array) array of TAC/TDC bin size (in seconds). Present only if ``/setup/lifetime`` is True and if TCSPC info is different - for each pixel. - - **tcspc_num_bins:** (integer) array of number of TAC/TDC bins. Present + for each detector. If present, must be of the same length as ``id``. + This value should be 0 for positions corresponding to marker IDs + - **tcspc_num_bins:** (integer array) array of number of TAC/TDC bins. Present only if ``/setup/lifetime`` is True and if TCSPC info is different - for each pixel. + for each detector. If present, must be of the same length as ``id``. + This value should be 0 for positions corresponding to 0 for marker ids. + - **tcspc_offset:** (integer array) *Optional* Offset per record ID (as in ``id``) to add to + nanotime to account for speed of light delays between detectors. When using pulsed + interleaved excitation, the nanotimes window for a given detector is defined by + ``/photon_data/measurement_specs/alex_excitation_periodX + tcspc_offset``. Should be 0 + for at least 1 :ref:`photon ID`, and default 0 for :ref:`non-photon IDs`, + for which value is irrelevant. + - **counts** (integer array): *Optional.* Number of timestamps counted by each + record ID. If present, must be of the same length as ``id``. + -For more info see -:ref:`Group /setup/detectors `. +For more info see :ref:`Group /setup/detectors`. .. _optional_setup_fields: @@ -356,41 +450,41 @@ information is irrelevant or not available. The following fields are arrays, one element per excitation source, in the order of increasing wavelengths: -- **excitation_wavelengths**: (array of floats) list of excitation wavelengths +- **excitation_wavelengths**: (float array) list of excitation wavelengths (center wavelength if broad-band) in increasing order (unit: *meter*). -- **laser_repetition_rates**: (array of floats) *New in version 0.5.* +- **laser_repetition_rates**: (float array) *New in version 0.5.* The laser repetition rate in Hz for each excitation source. This field is mandatory only if there is at least one pulsed excitation source. When there are both CW and pulsed lasers, the CW lasers will have a value of 0 in this field. -- **excitation_polarizations**: (arrays of floats) list of polarization +- **excitation_polarizations**: (float arrays) list of polarization angles (in degrees) for each excitation source. -- **excitation_input_powers**: (array of floats) excitation power in *Watts* +- **excitation_input_powers**: (float array) excitation power in *Watts* for each excitation source. This is the excitation power entering the optical system. -- **excitation_intensity**: (array of floats) excitation intensity in the +- **excitation_intensity**: (float array) excitation intensity in the sample for each excitation source (units: *W/m²*). In the case of confocal excitation this is the peak PSF intensity. The following fields are also arrays: -- **detection_wavelengths**: (arrays of floats) reference wavelengths (in +- **detection_wavelengths**: (float array) reference wavelengths (in *meters*) for each detection spectral band. This array is ordered in increasing order of wavelengths. The first element refers to ``detectors_specs/spectral_ch1``, the second to ``detectors_specs/spectral_ch2`` and so on. -- **detection_polarizations**: (arrays of floats) polarization angles +- **detection_polarizations**: (float array) polarization angles for each detection polarization band. The first element refers to ``detectors_specs/polarization_ch1``, the second to ``detectors_specs/polarization_ch2`` and so on. This field is not relevant if no polarization selection is performed. -- **detection_split_ch_ratios**: (array of floats) power fraction detected +- **detection_split_ch_ratios**: (float array) power fraction detected by each "beam-split" channel (i.e. independent detection channels obtained through a non-polarizing beam splitter). For 2 beam-split channels that receive the same power this array should be ``[0.5, 0.5]``. @@ -399,7 +493,6 @@ The following fields are also arrays: This field is not relevant when no polarization- and spectral-insensitive splitting is performed. - .. _sample_group: Sample group @@ -424,7 +517,7 @@ The **identity/** group contains information about the specific Photon-HDF5 file. The following fields are mandatory (and automatically added by -`phconvert `__): +`phconvert `_): - **creation_time**: (string) the Photon-HDF5 file creation time with the following format: "YYYY-MM-DD HH:MM:SS". @@ -483,6 +576,87 @@ Also, if some information is not available the relative field may be omitted. - **software**: (string) Software used to save the original data file. - **software_version**: (string) Version of the software used to save the original data file. +.. _user_group: + +User group +---------- + +The ``/user/`` group is reserved for user defined fields. +There are no specific definitions for the groups contained within ``/user/``. +Nor will standard readers of *photon-HDF5* files use any fields withing this group. + +This group is intended to be used to store data not covered by the official photon-HDF5 +specification. While there is not required format for this group, some suggestions are +provided, most notably the :ref:`experimental_settings/ ` group. + +If the data is being converted from a format like .ptu from picoquant or .spc from +Beckr&Hickl, some metadata stored in the file is not included in the official photon-HDF5 +specification. This metadata should be stored in a group named by the company manufacturing +the original data file (i.e. ``picoquant`` for .ptu files, `b`ecker_hickl`` for .spc files). + +The ``user`` group can also be used to store results of various type of analysis. +Such data should be stored in a sub-group named according to the software used +for such analysis. +For instance, FRETBursts, if directed to do so, will save results of burst +analysis under the group ``/user/FRETBursts``. + +.. _exp_settings: + +Experimental Settings User Group +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +It is often desirable to store additional information and/or explanation of the experiment. +These fields should be values that are not know *a priori* before the measurement, or +which may have ambiguous definitions or unclear provenance. +Such information is best stored in a ``/user/experimental_settings/`` group. All information in this +group should be stored in a way that is self-explanatory, with human-readable details included. + +If :ref:`non-photon IDs` are included in +``/photon_data/detectors``, ``/user/experimental_settings/`` should contain the +group ``non_photon_id/`` with strings named ``id1``, ``id2`` ... which contain +human-readable explanations of the role of each :ref:`non-photon IDs` +group of detectors (numbered identically. Additional explanation can be included as +a human readable string named ``non_photon_id_explanation``. + + +FRET correction factors can be stored within ``/user/experimental_settings/``, +each should contain 1 element per `photon_dataN`` group: + +- **gamma** (float or float array) The gamma correction factor, correcting for differences + in detection efficiency and fluorophore brightness between donor and acceptor. +- **beta** (float or float array) The beta correction factor, correcting for differences + in donor and acceptor excitation. +- **direct_excitation** (float or float array) the direct excitation fraction, accounting for + the fraction of acceptor directly excited by donor laser. +- **leakage** (float or float array) the leakage factor accounting for the +- **fret_provenance** (string) a human readable string defining how the FRET correction + factors were determined and a doi to the definition used for the correction factors. + We suggest the following ``https://doi.org/10.1038/s41592-018-0085-0`` + (`Hellenkamp et. al. 2018 `_) + +.. note:: + If any FRET correction factor is stored, the ``fret_provenance`` field + should also be included so that users will know how to apply the supplied + correction factors correctly. + +Additional fields are suggested for storing information about each detector's +(ie instruments that generate photon IDs) background and afterpulsing characteristics: + +- **dcr** (float array): *Optional.* Dark counting rate in Hz for each pixel. + If present, must be of the same length as ``id``. + This value should be 0 for positions corresponding to 0 for marker IDs. +- **dcr_provenance** (string) Human readable description for how the dcr was determined. +- **afterpulsing** (float array): *Optional.* Afterpulsing probability + for each record ID. If present, must be of the same length as ``id``. + This value should be 0 for positions corresponding to 0 for non-photon IDs. +- **afterpulsing_provenance** (string) Human readable description for how the + afterpulsing was determined. + +In general, the method for determining any value in ``/user/experimental_settings/`` should +be specified as a string in a ``[value]_provenance`` group. + +See for an example of how to use the */user/experimental_settings/** +group. + .. _glossary: @@ -491,19 +665,46 @@ Additional notes and definitions .. _detector_ids: -Detector pixel IDs -^^^^^^^^^^^^^^^^^^ +Detector IDs +^^^^^^^^^^^^ + +A *detector ID* is the "name" for each type of event recorded in ``/photon_data/detectors``. +These are divided into two broad types: "photon IDs" and "non-photon IDs". +In previous versions only photon IDs were handled, and were also referred to as pixel IDs; -A *detector pixel ID* (or simply *pixel ID*) is the "name" of each pixels and -is usually a single integer. Pixels are normally numbered incrementally, but not necessarily so. -In other words, a file containing data taken with 2 single-point (pixel) detectors could have +A "non-photon ID" is any other type of event (e.g. sync signal, FLIM marker) recorded +in the original source file/by the TCSPC card. + +Detectors IDs are usually a single integer. +Detectors are normally numbered incrementally, but are not required to be. +In other words, a file containing data taken with 2 single-point detectors could have the first detector labeled "4" and the second detector labeled "6". -In some cases (when using detector arrays) the pixel ID +In some cases (when using detector arrays) the record ID can be a *n*-tuple of integers. This allow to specify, for each pixel, the module number and the X, Y location, for example. Therefore, an -array of pixel IDs can be either a 1-D column array or a 2-D array. -In either cases, each row identifies a pixel. +array of record IDs can be either a 1-D column array or a 2-D array. +In either case, each row identifies a record. + +.. _non_photon_ids: + +Non-Photon IDs +"""""""""""""" + +If the acquisition software also records events such as sync signals or markers indicating +a change in position of a piezo or galvo scanner, these events should also be assigned a unique +pixel ID. These events are all considered markers. To distinguish between detector (i.e. real photons) +and marker IDs, the record IDs of marker events must be recorded in one of the ``non_photon_ch1``, +``non_photon_ch2`` ... groups within ``/photon_data/measurement_specs/detectors_specs`` group. +In photon-HDF5 v0.5 no additional information is officially recorded, and thus it is recommended +to store a description of the meaning of each type of marker ID in the ``user/experimental_settings`` +group. See :ref:`user_group` to understand how to use this group. + + +.. note:: + Inclusion of non-photon ids is discouraged for ` for each beam-split channel (see :ref:`detectors_specs_group`). Beam split channels can receive same or different (depending on whether the beam splitter is 50-50). The fractional power of each beam split channel -can be saved in the field `detection_split_ch_ratios` in the -:ref:`setup_group`. +can be saved in the field ``detection_split_ch_ratios`` in +the :ref:`/setup/ group`. .. _wavelengths_order: @@ -533,7 +734,7 @@ longest wavelength. This ordering is strictly followed and removes any ambiguity in defining first, second, etc... wavelength or spectral band. For examples, for μs-ALEX and ns-ALEX (or PIE) the excitation wavelengths -(in ``/setup/excitation_wavelenths``) are ordered as +(in ``/setup/excitation_wavelengths``) are ordered as 1. *donor excitation wavelength*, 2. *acceptor excitation wavelength* @@ -542,7 +743,7 @@ Similarly, the donor (or acceptor) excitation period range is defined by ``/photon_data/measurement_specs/alex_excitation_period1`` (or ``/photon_data/measurement_specs/alex_excitation_period2``). -Finally the donor (or acceptor) :ref:`detector_ids` number is defined in +Finally the donor (or acceptor) :ref:`detector IDs` number is defined in ``/photon_data/measurement_specs/detectors_specs/spectral_ch1`` (or ``/photon_data/measurement_specs/detectors_specs/spectral_ch2``). @@ -555,7 +756,7 @@ Definition of alternation periods Note for μs-ALEX """""""""""""""" -The fields *alex_offset*, *alex_excitation_period1* and *alex_excitation_period2* +The fields ``alex_offset``, ``alex_excitation_period1`` and ``alex_excitation_period2`` define the excitation period for each excitation source. The alternation histogram is the histogram of the following quantity: @@ -589,12 +790,14 @@ Each *measurement_type* has an associated set of mandatory fields which must be present to ensure that all information needed to unambiguously interpret the data is present. For example, for a 2-color smFRET measurement, a software package creating -a file should check that -the association between detector-pixel and donor or acceptor channel +a file should check that the association between detector ID (previously called pixel) and donor or acceptor channel is present. -The list of supported measurement types is reported in -:ref:`measurement_specs_group`. +The list of supported measurement types is reported in :ref:`measurement_specs_group`. + +.. toctree:: + + generic .. _nanotimes_time_axis: @@ -610,36 +813,34 @@ configuration, the resulting raw TCSPC histogram looks inverted along the time a with the nanotimes of photons emitted shortly after a laser pulse being larger than the nanotimes of photons emitted much later. -By convention, the Photon-HDF5 format requires nanotimes to be properly oriented. In other words, -when a `nanotimes` time axis inversion is needed, this correction needs to be performed before -the data is saved into a Photon-HDF5 file. As a corollary, TCSPC histograms computed directly from -`nanotimes` from Photon-HDF5 files are always properly oriented, regardless of -the way the nanotimes were acquired. +By convention, the Photon-HDF5 format requires nanotimes to be properly oriented. +In other words, when a ``/photon_data/nanotimes`` time axis inversion is needed, +this correction needs to be performed before the data is saved into a Photon-HDF5 file. +As a corollary, TCSPC histograms computed directly from ``nanotimes`` from Photon-HDF5 +files are always properly oriented, regardless of the way the nanotimes were acquired. .. _setup_detectors_group: -Group /setup/detectors -^^^^^^^^^^^^^^^^^^^^^^ +Group /setup/detectors/ +^^^^^^^^^^^^^^^^^^^^^^^ + This group is new in version 0.5 and contains fields which are arrays, one -element per detector -(see :ref:`definition `). +element per detector ID (both photon and non-photon) (see :ref:`definition`). The only mandatory field is ``id`` which contains all detectors IDs as they appear in ``/photon_data/detectors``. -Within each spot, IDs appear in ``/setup/id`` in increasing order. -All values which appears in -``/photon_data/detectors`` need to be listed here. This includes non-standard -detectors (e.g. a monitor channel to monitor the input power) or "markers" -saved by the acquisition hardware (for example PicoQuant TCSPC -hardware can save makers for synchronization). However, special -detector IDs used for overflow correction must be removed before -saving a Photon-HDF5 file. - -In TCSPC measurements where each pixel has different TCSPC bin width, -the ``/setup/detectors`` group allows to save per-pixel TCSPC info. +Within each spot, IDs appear in ``/setup/detectors/id`` in increasing order. +All values which appears in ``/photon_data/detectors`` need to be listed here. +Note that while v0.5 now has soft-support for non-photon IDs, supporting such events +as markers for FLIM, sync signals from the TCSPC card, signals for monitoring laser +power etc. overflow markers such as those present in many Picoquant formats must +be removed. + +In TCSPC measurements where each detector (pixel) has different TCSPC bin width, +the ``/setup/detectors`` group allows to save per-photon detector TCSPC info. In this case the the ``nanotimes_specs`` group is not present in ``photon_data`` and the group -``/setup/detectors`` will contain per-pixel TCSPC info:: +``/setup/detectors`` will contain per-photon detector TCSPC info:: /setup/detectors/tcspc_units /setup/detectors/tcspc_num_bins @@ -671,3 +872,4 @@ group. In version 0.5 and above the ``/photon_dataNN/detectors`` arrays need to contain detectors identifier which are unique across all the spots. In version 0.4 the same identifiers (e.g. 0 and 1) were allowed in different spots. + diff --git a/docs/reading.rst b/docs/reading.rst index cb3d4fb..88b0a25 100644 --- a/docs/reading.rst +++ b/docs/reading.rst @@ -39,8 +39,8 @@ This scheme is implemented in the burst analysis program (see `code `_ for full details). - Read Photon-HDF5 files - '''''''''''''''''''''' +Read Photon-HDF5 files +'''''''''''''''''''''' 1. Get the Photon-HDF5 version from ``format_version`` root-node attribute (see :ref:`root_params`). The version must be ``'0.4'`` or greater. diff --git a/docs/whats_new_ver05.rst b/docs/whats_new_ver05.rst index fb5ea14..1ed625b 100644 --- a/docs/whats_new_ver05.rst +++ b/docs/whats_new_ver05.rst @@ -3,6 +3,10 @@ What changed in version 0.5 =========================== +- A new measurement_type "generic" was added. In a "generic" measurement, the + specifications are determined by the fields in the ``/setup`` group. + For details see :doc:`generic`. + - A new field: ``/setup/excitation_alternated`` (array of booleans), was added, which is True for all excitation sources which are intensity-modulated. This field allows distinguishing between ALEX and PAX measurements (i.e. @@ -11,10 +15,12 @@ What changed in version 0.5 - A New group: ``/setup/detectors`` containing arrays of per-pixel information, was added. See :ref:`setup_detectors_group`. -- Support was added for cases where TCSPC specifications change for each pixel, by addition of +- Introduction of limited support for markers :ref:`marker_ids`, allowing sync, markers for FLIM, + sync signals etc. to be included. Currently the meaning of these ids is left to the user. In + upcoming release of v0.6, support for basic raster-scan FLIM will be introduced, where markers + can optionally be used to specify the beginning of pixels/lines/frames. + +- Support was added for cases where TCSPC specifications change for each pixel, by addition of dedicated fields in ``/setup/detectors``. See :ref:`Group /setup/detectors `. -- A new measurement_type "generic" was added. In a "generic" measurement, the - specifications are determined by the fields in the ``/setup`` group. - For details see :doc:`generic`.