Data Processing models#

Data processing models are used to process data. Result retrieved with run_mode() will show datatree.DataTree structure containing two groups:

Bucket group, containing the Scene, Photon, Charge, Pixel, Signal and Image, if initialized in the pipeline.

Data group, containing processed data for each data processing model used in the YAML configuration file.

Processed data from models in the used in the YAML configuration file can be also accessed directly via detector.data.

Create and Store a detector#

The models Save detector and Load detector can be used respectively to create and to store a Detector to/from a file.

These models can be used when you want to store or to inject a Detector into the current Pipeline.

Save detector#

This model saves the current Detector into a file. Accepted file formats are .h5, .hdf5, .hdf and .asdf.

- name: save_detector
  func: pyxel.models.save_detector
  enabled: true
  arguments:
    filename: my_detector.h5

pyxel.models.save_detector(detector, filename)[source]: Save the current detector into a file.

Load detector#

This model loads a Detector from a file and injects it in the current pipeline. Accepted file formats are .h5, .hdf5, .hdf and .asdf.

- name: load_detector
  func: pyxel.models.load_detector
  enabled: true
  arguments:
    filename: my_detector.h5

pyxel.models.load_detector(detector, filename)[source]

Load a new detector from a file.

Raises:: TypeError – If the loaded detector has not the same type of the current detector.

Statistics#

The model Statistics can be used to do simple statistics computations, giving the var, mean, min, max and count of the data buckets photon, pixel, signal and image of the detector. The calculated statistics can then be accessed via detector.data.statistics.

data_processing:
- name: statistics
  func: pyxel.models.data_processing.statistics
  enabled: true

Note

You can find examples of this model in these Jupyter Notebooks from Pyxel Data:

pyxel.models.data_processing.statistics(detector, data_structure='all')[source]#

Compute basic statistics.

Parameters:

detector (Detector) – Pyxel Detector object.
data_structure (Literal) – Keyword to choose data structure. Can be any from: (“pixel”, “photon”, “signal”, “image”, “all”). Default is “all” and computes the statistics on “pixel”, “photon”, “signal” and “image”.

Notes

For more information, you can find examples here:

Extract ROI#

Extracts the source data of the final pixel array and output in the form of an xarray dataset. The models makes use of the Photutils library and configured it into a library of stand-alone functions and classes.

The Photutils library is a useful post-processing tool capable of calculating statistics of a given array.

data_processing:
  - name: source_extractor
    func: pyxel.models.data_processing.source_extractor
    arguments:
      thresh: 80
      minarea: 5
    enabled: true

Note

You can find an example of this model used in this Jupyter Notebook Extract Region of Interest(s) from Pyxel Data.

pyxel.models.data_processing.source_extractor(detector, array_type='pixel', thresh=50, minarea=5)[source]#

Extract the roi data converts it to xarray dataset and saves the information to the final result.

A warning is generated if the processed data_array is empty.

Parameters:

array_type
detector (Detector) – Pyxel Detector object.
thresh (int) – Threshold pixel value above which information from the image array is extracted
minarea (int) – Minimum area of pixels required that are above the threshold for the extractor to extract information

Raises:

ValueError – If parameter ‘array_type’ is not ‘pixel’,’signal’,’image’,photon’ or ‘charge’

Notes

For more information, you can find an example here: Extract Region of Interest(s).

There is code within Pyxel capable of harnessing some data, such as background subtraction and imaging a given 2D given numpy array.

pyxel.models.data_processing.plot_roi(data, roi)[source]#

Plot the input data on a graph and overlays ellipses over the roi’s found by the extract function.

Parameters:

data (np.ndarray) – 2D image array
roi (np.ndarray / xarray.Dataset) – structured numpy array or xarray dataset of extracted data

Mean-variance#

Compute a Mean-Variance 1D array that represents the relationship between the mean signal of a detector and its variance.

This is particularly useful for analyzing the statistical properties of image data, such as determining the consistency of pixel values in a detector.

This model takes detector data (e.g., pixel, photon, image, or signal) and computes the mean and variance of the specified data structure. The results are stored within the detector’s internal .data tree for further analysis or visualization.

YAML configuration example:

Below is an example of how to configure the Mean-Variance model in the Pyxel YAML configuration file:

data_processing:
  - name: mean_variance
    func: pyxel.models.data_processing.mean_variance
    enabled: true
    arguments:
      data_structure: image  # Options: 'pixel', 'photon', 'image', 'signal'

Hint

>>> import pyxel
>>> config = pyxel.load("configuration.yaml")

>>> data_tree = pyxel.run_mode(
...     mode=config.running_mode,
...     detector=config.detector,
...     pipeline=config.pipeline,
... )

>>> data_tree["/data/mean_variance/image/variance"]
<xarray.DataTree 'image'>
Group: /data/mean_variance/image
    Dimensions:      (pipeline_idx: 100)
    Coordinates:
      * pipeline_idx (pipeline_idx) int64 0 1 ... 98 99
    Data variables:
        mean         (pipeline_idx) float64 5.723e+03 1.144e+04 ... 5.238e+04 5.238e+04
        variance     (pipeline_idx) float64 3.238e+06 1.294e+07 2.91e+07 ... 4.03e+05 3.778e+05

>>> (
...     data_tree["/data/mean_variance/image"]
...     .to_dataset()
...     .plot.scatter(x="mean", y="variance", xscale="log", yscale="log")
... )

Note

You can find an example of this model used in this Jupyter Notebook Data processing models: Observation mode from Pyxel Data.

pyxel.models.data_processing.mean_variance(detector, data_structure='image', name=None)[source]#

Compute the mean and variance and store the result it in ‘.data’ bucket.

Parameters:

detector (Detector)
data_structure ('pixel', 'photon', 'image' or 'signal', optional. Default: 'image') – Data bucket to use to compute the mean and variance.
name (str, optional) – Name to use for the result.

Examples

>>> import pyxel
>>> config = pyxel.load("exposure_mode.yaml")

Run exposure mode with ‘mean-variance’ model

>>> data_tree = pyxel.run_mode(
...     mode=config.exposure,
...     detector=config.detector,
...     pipeline=config.pipeline,
... )

Get results

>>> data_tree["/data/mean_variance"]
<xarray.DataTree 'data'>
Group: /data
└── Group: /data/mean_variance
    └── Group: /data/mean_variance/image
            Dimensions:      (pipeline_idx: 100)
            Coordinates:
              * pipeline_idx (pipeline_idx) int64 0 1 ... 98 99
            Data variables:
                mean         (pipeline_idx) float64 5.723e+03 1.144e+04 ... 5.238e+04 5.238e+04
                variance     (pipeline_idx) float64 3.238e+06 1.294e+07 2.91e+07 ... 4.03e+05 3.778e+05

>>> mean_variance = data_tree["/data/mean_variance/image"]
>>> mean_variance
<xarray.DataTree 'image'>
Group: /data/mean_variance/image
    Dimensions:      (pipeline_idx: 100)
    Coordinates:
      * pipeline_idx (pipeline_idx) int64 0 1 ... 98 99
    Data variables:
        mean         (pipeline_idx) float64 5.723e+03 1.144e+04 ... 5.238e+04 5.238e+04
        variance     (pipeline_idx) float64 3.238e+06 1.294e+07 2.91e+07 ... 4.03e+05 3.778e+05

Display mean-variance plot

>>> (
...     data_tree["/data/mean_variance/image"]
...     .to_dataset()
...     .plot.scatter(x="mean", y="variance", xscale="log", yscale="log")
... )
.. figure:: _static/mean_variance_plot.png
    :scale: 70%
    :alt: Mean-Variance plot
    :align: center

Linear regression#

Compute a linear regression along readout time.

data_processing:
  - name: linear_regression
    func: pyxel.models.data_processing.linear_regression
    enabled: true
    arguments:
      data_structure: image

Note

You can find an example of this model used in this Jupyter Notebook Data processing models: Observation mode from Pyxel Data.

pyxel.models.data_processing.linear_regression(detector, data_structure='image', name=None)[source]#

Compute a linear regression along ‘readout_time’ and store it in ‘.data’ bucket.

Parameters:

detector (Detector)
data_structure ('pixel', 'photon', 'image' or 'signal') – Data bucket to use for the linear regression.
name (str, optional) – Name to use for the result.

Raises:

ValueError – If less than 2 timing steps is used.

Examples

>>> import pyxel
>>> config = pyxel.load("exposure_mode.yaml")
>>> linear_regression(detector=detector)

Run exposure mode with ‘data_processing/linear_regression’ model

>>> data_tree = pyxel.run_mode(
...     mode=config.exposure,
...     detector=config.detector,
...     pipeline=config.pipeline,
... )

Get results

>>> data_tree["/data/linear_regression"]
DataTree('linear_regression', parent="data")
└── DataTree('image')
        Dimensions:        (y: 100, x: 100)
        Coordinates:
          * y              (y) int64 0 1 2 3 4 5 6 7 8 9 ... 91 92 93 94 95 96 97 98 99
          * x              (x) int64 0 1 2 3 4 5 6 7 8 9 ... 91 92 93 94 95 96 97 98 99
        Data variables:
            slope          (y, x) float64 396.8 396.2 396.7 397.0 ... 396.1 396.2 396.7
            intercept      (y, x) float64 8.363e+03 8.356e+03 ... 8.364e+03 8.36e+03
            r2             (y, x) float64 0.9313 0.9312 0.9316 ... 0.9309 0.931 0.9313
            slope_std      (y, x) float64 14.94 14.93 14.91 14.91 ... 14.96 14.96 14.94
            intercept_std  (y, x) float64 846.4 845.8 844.5 844.5 ... 847.7 847.6 846.1

Display slope as a 2D map

>>> data_tree["/data/linear_regression/image/slope"].plot(robust=True)

Notes

For more information, you can find an example here: Extract Region of Interest(s).

Remove Cosmic Rays#

Removes cosmic rays from the pixel array using LACosmic package.

data_processing:
  - name: remove_cosmic_rays
    func: pyxel.models.data_processing.remove_cosmic_rays
    enabled: true
    arguments:
      contrast: 1.0
      cr_threshold: 50.0
      neighbor_threshold: 50.0
      effective_gain: 1.0
      readnoise: 0.0

pyxel.models.data_processing.remove_cosmic_rays(detector, contrast=1.0, cr_threshold=50.0, neighbor_threshold=50.0, effective_gain=1.0, readnoise=0.0)[source]#

Extract the roi data converts it to xarray dataset and saves the information to the final result.

Refer to lacosmic documentation for parameter descriptions. https://lacosmic.readthedocs.io/en/stable/api/lacosmic.lacosmic.html#lacosmic.lacosmic

Parameters:

detector (Detector) – Pyxel Detector object.
contrast (float) – Contrast threshold between the Laplacian image and the fine-structure image.
cr_threshold (float) – The Laplacian signal-to-noise ratio threshold for cosmic-ray detection.
neighbor_threshold (float) – The Laplacian signal-to-noise ratio threshold for detection of cosmic rays in pixels neighboring the initially-identified cosmic rays.
effective_gain (float) – Ratio of counts (e.g., electrons or photons) to the units of data.
readnoise (float) – The read noise (in electrons) in the input data.

Signal-to-noise ratio#

The model Signal-to-noise ratio can be used to get the signal-to-noise-ratio (SNR) along the time for of the data buckets photon, pixel, signal and image of the detector. The data_structure “signal” is the one selected by default.

data_processing:
- name: snr
  func: pyxel.models.data_processing.signal_to_noise_ratio
  enabled: true
  arguments:
    data_structure: "signal"

Note

You can find an example of this model used in this Jupyter Notebook Data processing models: Observation mode from Pyxel Data.

pyxel.models.data_processing.signal_to_noise_ratio(detector, data_structure='signal')[source]#

Get signal-to-noise-ratio (SNR) for given data structure.

Parameters:

detector (Detector) – Pyxel Detector object.
data_structure (Literal) – Keyword to choose data structure. Can be any from: (“pixel”, “photon”, “signal”, “image”, “all”). Default is “signal” and computes the SNR on “signal”.

Notes

For more information, you can find an example here: Extract Region of Interest(s).

Data Processing models

Contents

Data Processing models#

Create and Store a detector#

Save detector#

Load detector#

Statistics#

Extract ROI#

Mean-variance#

Linear regression#

Remove Cosmic Rays#

Signal-to-noise ratio#