Network API¶

All network operations are handled by the pollyxt_pipelines.scc_access module and specifically, the SCC class. The code is based the work of Ioannis Binietoglou on SCC Access.

To programmatically access SCC, begin by importing pollyxt_pipelines.scc_access and creating a SCC_Credentials object.

from pollyxt_pipelines.config import Config
import pollyxt_pipelines.scc_access

# TODO remove dependency on Config
credentials = SCC_Credentials(Config())

Afterwards, you can access SCC as follows:

# As a context
with scc_access.scc_session(credentials) as scc:
  # Do stuff with scc
  # ...
  scc.upload_file(...)

# As a class
scc = scc_access.SCC(credentials)
scc.login()
# ...

The context manager handles login/logout automatically. If you use the class yourself, you should call login() before attempting to call any other method.

API¶

Tools for communicating with the SCC backend

Author: Thanasis Georgiou <ageorgiou@noa.gr>

Based on scc-access by Iannis Binietoglou <i.binietoglou@impworks.gr>: https://repositories.imaa.cnr.it/public/scc_access

class pollyxt_pipelines.scc_access.SCC(credentials)[source]¶

Bases: object

Represents a session with SCC. Before making any calls, the user should login using login()!

It’s recommended to use the scc_session() context manager, which handles logging in and out.

delete_measurement(measurement_id)[source]¶

Deletes a measurement from SCC

Parameters:: measurement_id (str) – Which measurement to delete

download_file(url, path)[source]¶

Downloads a single file from SCC to the given path

Parameters:

url (str) – Which URL to download the file from
path (Path) – Where to store the downloaded file

download_products(measurement_id, download_path, hirelpp=True, cloudmask=True, elpp=True, optical=True, elic=True)[source]¶

Downloads products for a given measurement (ID) to the given path. This function is a generator, yielding the filename of each downloaded file.

Parameters:

measurement_id (str) – Which measurement to download products for
download_path (Path) – Where to store the downloaded products
hirelpp – Whether to download HiRELPP files
cloudmask – Whether to download Cloudmask files
elpp – Whether to download ELPP files
optical – Whether to download optical (ELDA or ELDEC) files
elic – Whether to download ELIC files

get_anchillary(file_id, file_type)[source]¶

Uses the SCC API to fetch information about anchillary files.

Parameters:

file_id (str) – File ID to lookup
file_type (str) – What kind of file to lookup (‘sounding’, ‘overlap’ or ‘lidarratio’)

Return type:

Optional[APIObject]

Returns:

The API response about the file

get_lidar_consants(date_start, date_end, location, page=1)[source]¶

Fetches the Lidar constants from SCC

Parameters:

date_start (date) – First day of results
date_end (date) – Last day of results
location (Optional[Location]) – Optionally, filter results by a location
page – Which page to return (starts from 1, default value is 1)

Return type:

Tuple[int, List[Measurement]]

Returns:

The number of pages and the list of measurements

get_measurement(measurement_id)[source]¶

Fetches information about one measurement from SCC.

Parameters:: measurement_id (str) – Which measurement to lookup
Return type:: Optional[Measurement]
Returns:: The measurement if it exists, None otherwise

login()[source]¶

This function starts a session with the SCC backend, storing the authentication cookies so they can be used by the rest of the methods. Remember to call logout()!

logout()[source]¶: Logout of SCC

query_measurements(date_start, date_end, location, page=1)[source]¶

Searches SCC for uploaded measurements

Parameters:

date_start (date) – First day of results
date_end (date) – Last day of results
location (Optional[Location]) – Optionally, filter results by a location
page – Which page to return (starts from 1, default value is 1)

Return type:

Tuple[int, List[Measurement]]

Returns:

The number of pages and the list of measurements

rerun_processing(measurement_id)[source]¶

Asks SCC to re-run processing routines for a given measurement ID

Parameters:: measurement_id (str) – Which measurement to re-run

upload_file(filename, system_id, rs_filename=None, ov_filename=None, lr_filename=None)[source]¶

Uploads a file to SCC, together with the auxilary files. There is no return value, but it will throw for potential errors.

Parameters:

filename (Path) – Path to the SCC netCDF file
system_id (str) – SCC Lidar System ID for the system that made the measurement
rs_filename (Optional[Path]) – Path to the radiosonde netCDF file
ov_filename (Optional[Path]) – Path to the overlap netCDF file
lr_filename (Optional[Path]) – Path to the lidar ratio netCDF file

class pollyxt_pipelines.scc_access.SCC_Credentials(config)[source]¶

Bases: object

Contains all required credentials to authenticate with SCC

pollyxt_pipelines.scc_access.scc_session(credentials)[source]¶

An SCC session as a context, to use with with:

Example:

with scc_access(credentials) as scc:
    # Use scc
    # ...

Exceptions¶

Exceptions that might occur while working with SCC

exception pollyxt_pipelines.scc_access.exceptions.MeasurementNotFound(measurement_id)[source]¶

Bases: Exception

Raised when the requested measurement does not exist in the SCC database

exception pollyxt_pipelines.scc_access.exceptions.PageNotAccessible(page_url, status_code)[source]¶

Bases: Exception

Raised when a page cannot be accessed

exception pollyxt_pipelines.scc_access.exceptions.SCCError(errors)[source]¶

Bases: Exception

Raised when an error message from SCC is parsed from the page body

exception pollyxt_pipelines.scc_access.exceptions.UnexpectedResponse(message)[source]¶

Bases: Exception

Raised when the response is not OK and we don’t have a concrete reason for it

exception pollyxt_pipelines.scc_access.exceptions.WrongCredentialsException[source]¶

Bases: Exception

Raised when login() fails due to wrong credentials

Container types¶

Various container classes and utility functions for handling SCC responses

class pollyxt_pipelines.scc_access.types.APIObject(response_body)[source]¶

Bases: object

SCC generic API response object

The only objects fetched from the API are Anchillary files, so this class doesn’t do much.

class pollyxt_pipelines.scc_access.types.LidarConstant(measurement_id, channel_id, system_id, product_id, detection_wavelength, lidar_constant, lidar_constant_stat_err, profile_start_time, profile_end_time, calibration_window_bottom, calibration_window_top, creation_date, elda_version)[source]¶

Bases: object

static from_table_row(tr)[source]¶: Create a measurement object from a table row. Table rows are formatted like in https://scc.imaa.cnr.it/admin/database/lidarconstant/

class pollyxt_pipelines.scc_access.types.Measurement(id, station_code, date_start, date_end, date_creation, date_updated, is_uploaded, hirelpp, cloudmask, elpp, elda, eldec, elic, elquick, is_queued, is_processing)[source]¶

Bases: object

static from_table_row(tr)[source]¶: Create a measurement object from a table row. Table rows are formatted like in https://scc.imaa.cnr.it/admin/database/measurements/

class pollyxt_pipelines.scc_access.types.ProductStatus(value)[source]¶

Bases: Enum

Represents the status of a product (ie ELDA) on SCC

to_emoji()[source]¶

Converts the given status to emoji.

Contains color tags for use with the rich library.

pollyxt_pipelines.scc_access.types.scc_bool(node)[source]¶

Convert a table cell to a bool

Return type:: bool

pollyxt_pipelines.scc_access.types.scc_date(tag)[source]¶

Convert a table cell to a date

Return type:: datetime

pollyxt_pipelines.scc_access.types.scc_product_status(node)[source]¶

Convert a table cell to a Product. Preserves all states (OK, NO_RUN, ERROR).

Return type:: Product

← Manage SCC Measurements