Skip to content
Snippets Groups Projects
Commit 494e0b1f authored by Danilo Ferreira de Lima's avatar Danilo Ferreira de Lima
Browse files

Bumped the version and added devpi documentation.

parent 9949e757
No related branches found
No related tags found
1 merge request!15Bumped the version, added devpi documentation, removed old code.
# pes_to_spec # Virtual spectrometer
The aim of this project is to read low-resolution photon spectrometer measurements and predict, with an uncertainty band, a high-resolution invasive The aim of this project is to read low-resolution photon spectrometer measurements and predict, with an uncertainty band, a high-resolution invasive
photon spectrometer result, in such a way that one may continue to collect low-resolution data without stopping the beam, but obtaining nevertheless, high-resolution photon spectrometer result, in such a way that one may continue to collect low-resolution data without stopping the beam, but obtaining nevertheless, high-resolution
...@@ -10,17 +10,32 @@ The concept is to collect both results simultaneously during a training phase an ...@@ -10,17 +10,32 @@ The concept is to collect both results simultaneously during a training phase an
One may install it simply with `pip install pes_to_spec`. One may install it simply with `pip install pes_to_spec`.
While the dependencies should be automatically used, using the Intel-optimized `numpy` and `scipy` packages is recommended, as they are much faster.
This has been tested with `scipy==1.7.3`, but it should work with any version of scipy and numpy.
```
# install the optimized numpy and scipy versions
# skip dependencies
pip install --force-reinstall -i https://pypi.anaconda.org/intel/simple --no-dependencies numpy scipy
# now install dependencies
pip install numpy scipy
# install PyTorch
pip install torch --index-url https://download.pytorch.org/whl/cpu
```
## Usage ## Usage
The API may be used as follows: The API may be used as follows:
``` ```
from pes_to_spec.model import Model from pes_to_spec.model import Model
from itertools import product
# this is the main object holding all # this is the main object holding all
# information needed for training and prediction # information needed for training and prediction
# the default parameters should be sufficient in most times # the default parameters should be sufficient in most times
model = Model() # if some channels are dead, you may want to specify the active channels
channels = [f"channel_{i}_{l}" for i, l in product(range(1, 5), ["A", "B", "C", "D"])]
model = Model(channels=channels)
# this trains the model # this trains the model
# low_resolution_raw_data is a dictionary with keys "channel_[1-4]_[A-D]" and values set to 2D-shaped # low_resolution_raw_data is a dictionary with keys "channel_[1-4]_[A-D]" and values set to 2D-shaped
...@@ -29,7 +44,8 @@ model = Model() ...@@ -29,7 +44,8 @@ model = Model()
# high_resolution_intensity and high_resolution_photon_energy are estimates from the high-resolution invasive spectrometer # high_resolution_intensity and high_resolution_photon_energy are estimates from the high-resolution invasive spectrometer
model.fit(low_resolution_raw_data, model.fit(low_resolution_raw_data,
high_resolution_intensity, high_resolution_intensity,
high_resolution_photon_energy) high_resolution_photon_energy,
pulse_energy=beam_intensity)
# save it for later usage: # save it for later usage:
model.save("model.joblib") model.save("model.joblib")
...@@ -42,7 +58,7 @@ model = Model.load("model.joblib") ...@@ -42,7 +58,7 @@ model = Model.load("model.joblib")
# as before, the low_resolution_raw_data refers to a dictionary mapping the channel name # as before, the low_resolution_raw_data refers to a dictionary mapping the channel name
# in the format "channel_[1-4]_[A-D]" to the 2D numpy array with shape (number_of_train_IDs, features) # in the format "channel_[1-4]_[A-D]" to the 2D numpy array with shape (number_of_train_IDs, features)
# all names and shapes must match the format in training, except for the number_of_train_IDs, which may vary # all names and shapes must match the format in training, except for the number_of_train_IDs, which may vary
interpolated_high_resolution_data = model.predict(low_resolution_raw_data) interpolated_high_resolution_data = model.predict(low_resolution_raw_data, pulse_energy=beam_intensity)
# extra useful functions for debugging: # extra useful functions for debugging:
...@@ -100,16 +116,3 @@ low_resolution_raw_data = {ch: run['SA3_XTD10_PES/ADC/1:network', ...@@ -100,16 +116,3 @@ low_resolution_raw_data = {ch: run['SA3_XTD10_PES/ADC/1:network',
After installation, the command-line tool `offline_analysis` may be used to test the tool in recorded datasets. After installation, the command-line tool `offline_analysis` may be used to test the tool in recorded datasets.
## Exploration initial tests
A first draft and explorative code can be seen in the `exploration` directory.
1. inv_train.py -> Train a model on the specific RUN.
Thish will save the pca model and fit model in experiments/YOUR_DIR/checkpoints and the data in
experiments/YOUR_DIR/data.
2. inv_eval.py -> Use to evaluate the trained model
3. inv_inference -> Use trained model to do inference on new data point
4. data_drift_check.py -> Check data drift between two datasets
# Updating wheel in DA server
This tool is packaged and made available through the DA server.
The following commands can be used to update the tool in the DA server:
```
# if not done before:
devpi use https://devpi.exfldadev01.desy.de/
devpi login <username>
devpi push --index euxfel/internal <my_file.whl>
```
# Installing it from the DA server
To install this tool from the DA server, use the following index URL:
```
pip install --index-url "https://devpi.exfldadev01.desy.de/euxfel/internal" pes_to_spec==X.Y.Z
```
...@@ -2,4 +2,4 @@ ...@@ -2,4 +2,4 @@
Estimate high-resolution photon spectrometer data from low-resolution non-invasive measurements. Estimate high-resolution photon spectrometer data from low-resolution non-invasive measurements.
""" """
VERSION = "0.2.7" VERSION = "0.2.8"
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment