Network API

All network operations are handled by the pollyxt_pipelines.scc_access module and specifically, the SCC class. The code is based the work of Ioannis Binietoglou on SCC Access.

To programmatically access SCC, begin by importing pollyxt_pipelines.scc_access and creating a SCC_Credentials object.

from pollyxt_pipelines.config import Config
import pollyxt_pipelines.scc_access

# TODO remove dependency on Config
credentials = SCC_Credentials(Config())

Afterwards, you can access SCC as follows:

# As a context
with scc_access.scc_session(credentials) as scc:
  # Do stuff with scc
  # ...
  scc.upload_file(...)

# As a class
scc = scc_access.SCC(credentials)
scc.login()
# ...

The context manager handles login/logout automatically. If you use the class yourself, you should call login() before attempting to call any other method.

API

Tools for communicating with the SCC backend

Author: Thanasis Georgiou <ageorgiou@noa.gr>

Based on scc-access by Iannis Binietoglou <i.binietoglou@impworks.gr>: https://repositories.imaa.cnr.it/public/scc_access

class pollyxt_pipelines.scc_access.SCC(credentials)

Bases: object

Represents a session with SCC. Before making any calls, the user should login using login()!

It’s recommended to use the scc_session() context manager, which handles logging in and out.

delete_measurement(measurement_id)

Deletes a measurement from SCC

Parameters:

measurement_id (str) – Which measurement to delete

download_file(url, path)

Downloads a single file from SCC to the given path

Parameters:

• url (str) – Which URL to download the file from

• path (Path) – Where to store the downloaded file

download_products(measurement_id, download_path, hirelpp=True, cloudmask=True, elpp=True, optical=True, elic=True)

Downloads products for a given measurement (ID) to the given path. This function is a generator, yielding the filename of each downloaded file.

Parameters:

• measurement_id (str) – Which measurement to download products for

• download_path (Path) – Where to store the downloaded products

• hirelpp – Whether to download HiRELPP files

• cloudmask – Whether to download Cloudmask files

• elpp – Whether to download ELPP files

• optical – Whether to download optical (ELDA or ELDEC) files

• elic – Whether to download ELIC files

get_anchillary(file_id, file_type)

Uses the SCC API to fetch information about anchillary files.

Parameters:

• file_id (str) – File ID to lookup

• file_type (str) – What kind of file to lookup (‘sounding’, ‘overlap’ or ‘lidarratio’)

Return type:

Optional[APIObject]

Returns:

The API response about the file

get_lidar_consants(date_start, date_end, location, page=1)

Fetches the Lidar constants from SCC

Parameters:

• date_start (date) – First day of results

• date_end (date) – Last day of results

• location (Optional[Location]) – Optionally, filter results by a location

• page – Which page to return (starts from 1, default value is 1)

Return type:

Tuple[int, List[Measurement]]

Returns:

The number of pages and the list of measurements

get_measurement(measurement_id)

Fetches information about one measurement from SCC.

Parameters:

measurement_id (str) – Which measurement to lookup

Return type:

Optional[Measurement]

Returns:

The measurement if it exists, None otherwise

login()

Login to SCC

This function starts a session with the SCC backend, storing the authentication cookies so they can be used by the rest of the methods. Remember to call logout()!

logout()

Logout of SCC

query_measurements(date_start, date_end, location, page=1)

Searches SCC for uploaded measurements

Parameters:

• date_start (date) – First day of results

• date_end (date) – Last day of results

• location (Optional[Location]) – Optionally, filter results by a location

• page – Which page to return (starts from 1, default value is 1)

Return type:

Tuple[int, List[Measurement]]

Returns:

The number of pages and the list of measurements

rerun_processing(measurement_id)

Asks SCC to re-run processing routines for a given measurement ID

Parameters:

measurement_id (str) – Which measurement to re-run

upload_file(filename, system_id, rs_filename=None, ov_filename=None, lr_filename=None)

Uploads a file to SCC, together with the auxilary files. There is no return value, but it will throw for potential errors.

Parameters:

• filename (Path) – Path to the SCC netCDF file

• system_id (str) – SCC Lidar System ID for the system that made the measurement

• rs_filename (Optional[Path]) – Path to the radiosonde netCDF file

• ov_filename (Optional[Path]) – Path to the overlap netCDF file

• lr_filename (Optional[Path]) – Path to the lidar ratio netCDF file

class pollyxt_pipelines.scc_access.SCC_Credentials(config)

Bases: object

Contains all required credentials to authenticate with SCC

pollyxt_pipelines.scc_access.scc_session(credentials)

An SCC session as a context, to use with with:

Example:

with scc_access(credentials) as scc:
    # Use scc
    # ...

Exceptions

Exceptions that might occur while working with SCC

exception pollyxt_pipelines.scc_access.exceptions.MeasurementNotFound(measurement_id)

Bases: Exception

Raised when the requested measurement does not exist in the SCC database

exception pollyxt_pipelines.scc_access.exceptions.PageNotAccessible(page_url, status_code)

Bases: Exception

Raised when a page cannot be accessed

exception pollyxt_pipelines.scc_access.exceptions.SCCError(errors)

Bases: Exception

Raised when an error message from SCC is parsed from the page body

exception pollyxt_pipelines.scc_access.exceptions.UnexpectedResponse(message, body=None)

Bases: Exception

Raised when the response is not OK and we don’t have a concrete reason for it

exception pollyxt_pipelines.scc_access.exceptions.WrongCredentialsException

Bases: Exception

Raised when login() fails due to wrong credentials

Container types

Various container classes and utility functions for handling SCC responses

class pollyxt_pipelines.scc_access.types.APIObject(response_body)

Bases: object

SCC generic API response object

The only objects fetched from the API are Anchillary files, so this class doesn’t do much.

class pollyxt_pipelines.scc_access.types.LidarConstant(measurement_id, channel_id, system_id, product_id, detection_wavelength, lidar_constant, lidar_constant_stat_err, profile_start_time, profile_end_time, calibration_window_bottom, calibration_window_top, creation_date, elda_version)

Bases: object

static from_table_row(tr)

Create a measurement object from a table row. Table rows are formatted like in https://scc.imaa.cnr.it/admin/database/lidarconstant/

class pollyxt_pipelines.scc_access.types.Measurement(id, station_code, date_start, date_end, date_creation, date_updated, is_uploaded, hirelpp, cloudmask, elpp, elda, eldec, elic, elquick, is_queued, is_processing)

Bases: object

static from_table_row(tr)

Create a measurement object from a table row. Table rows are formatted like in https://scc.imaa.cnr.it/admin/database/measurements/

class pollyxt_pipelines.scc_access.types.ProductStatus(*values)

Bases: Enum

Represents the status of a product (ie ELDA) on SCC

to_emoji()

Converts the given status to emoji.

Contains color tags for use with the rich library.

pollyxt_pipelines.scc_access.types.scc_bool(node)

Convert a table cell to a bool

Return type:

bool

pollyxt_pipelines.scc_access.types.scc_date(tag)

Convert a table cell to a date

Return type:

datetime

pollyxt_pipelines.scc_access.types.scc_product_status(node)

Convert a table cell to a Product. Preserves all states (OK, NO_RUN, ERROR).

Return type:

Product