OGC Earth Observation Applications Pilot: CRIM Engineering Report

The OGC API - Processes path is obtained by appending /ems/api and /ades/api to the EMS and ADES endpoint URLs, respectively, while processes can be found at /ems/processes and /ades/processes.

To facilitate research platform software re-use, CANARIE requires that all platforms developed under the Research Software program, like PAVICS, support the CANARIE Platform and Service Registry and Monitoring System. For each endpoint and as depicted in the next figure, the /canarie route provides the registry and monitoring API that fully describe the details of the platform, how to use it, and how to obtain assistance. CANARIE’s platform monitoring service also measures the availability and usage of research platforms.

For the pilot, CRIM provided resources from its private cloud. These cloud resources have been provisioned and used continuously in Testbed-13, 14, 15 and 16. The basic provisioned hardware is composed of twelve OpenStack VM m4.large (2 VCPU, 8GB RAM, 200GB DISK) and three volumes of 2TB on spinning-disk. These resources have been extended in the pilot to accommodate applications with large memory footprints. Virtual machines are deployed in a public OpenStack tenant. Its IT department supports daily operations, while R&D personnel manages a large part of its application space. CRIM, being part of Réseau d’informations scientifiques du Québec (RISQ) and CANARIE network, has access to a pan-Canadian high-speed network dedicated to research.

The Canadian Forest Service (CFS) presented or supported several experiments in OGC’s innovation program, in part by contributing resources on the Pacific Boreal Cloud (PBC), a high-performance cloud infrastructure at the Pacific Forestry Centre in Victoria. The center provided virtual machines, storage, networking, as well as EO data, to enable numerous experiments such as biomass estimation using point clouds, cloudless mosaicking, tree species recognition or lake-river discrimination.

In OGC Testbed-13, CFS expressed interest in extracting polarimetric parameters from Radarsat-2 SQW data using a combination of OGC Web services and Cloud environments. The requirements for such EO processing are well inline with the remote sensing application presented in this pilot in Section 7. In the architecture envisioned at that moment, resources are only used when necessary, thus reducing overhead costs of maintaining expensive servers or computing power. Assuming successful research and development, CFS could consider processing larger regions in Canada, and possibly for the National Forest Inventory Plots across the country. Later, in Testbed-14, CRIM assessed the feasibility of installing its EMS on the PBC, but did not proceed at the time. CRIM also accessed the PBC to ensure that application packages could be deployed by an ADES hosted on the cloud. Finally, the Distributed Access Control System (DACS) security solution was tested and briefly evaluated. For this pilot, one key expectation of Pacific Forestry Service and sponsors is the deployment of an ADES/EMS pair and successful execution of a remote sensing application.

NRCan is responsible for the operation of the EODMS. EODMS provides an archiving and discovery system for the Government of Canada’s EO data (e.g. satellite imagery). As EODMS facilitates federal and public access to crucial geospatial information, it represents a core component of the CGDI.

New developments in the EO domain present challenges to the ongoing viability of the current EODMS architecture. Traditionally, EO users have identified appropriate imagery in repositories, downloaded the required information and finally applied necessary processing on local workstations. With massively increasing availability of EO data, the EO community is moving to an “exploitation platform” approach. Here, users and their applications are brought close to the physical location of EO data, helping to minimize data transfer between repositories and applications. To fully enable EO exploitation platforms, an interoperable, open standards-based architecture transformation is required.

With two Weaver platforms now deployed inside the NRCan’s EO data value stream, one upstream with EODMS and LEVEL 0, 1 data and one downstream in the PBC with LEVEL 2 value-add data, NRCan will be looking to invest in two main avenues:

The following data interfaces were used in the system, in the pilot or in previous initiatives.

Most of CRIM’s applications adopt a fan-in design, where an array of files is reduced to a single output. CRIM public application packages repository can be found on GitHub. The AP contains the CWL descriptors that refers to the appropriate execution unit, in this case Docker images. The images are built on demand, and stored in a private Docker registry. The table below shows the source code location and the resulting Docker images. Images published in the /ogc-public can be accessed, pulled and executed by anonymous users. Access to all other images requires appropriate credentials by end users or the ADES.

CRIM employs JSON bindings for body of both the deploy and execute requests, as described in WPS-T 2.0 with REST/JSON bindings. Translation from XML to JSON, and vice-versa, is a matter of changing a library. With respect to Python programming, use of JSON files are considered a best practice. JSON files are supported natively by Python standard library. Additionally, their structure is very similar. Participants would benefit from selecting and maintaining common JSON parsing libraries and services.

In this pilot, CRIM did not seek to improve its implementation of billing and quoting presented in Testbed-14. As stated in the ADES & EMS Results and Best Practices ER, the complexity of quoting a workflow execution is very high. As the number of steps increase, the deterministic behavior of a workflow rapidly decrease. For example, assuming adequate description by application developers, it might be possible to infer estimates of time required for a single application based on input data volume alone, for a specific type of machine. If the data output of that first application is meant to be an input for a second application, the subsequent estimates might vary wildly due to mounting uncertainty on data volume. Also, as the workflow gets larger, the parameter space grows, adding even more uncertainty.

Experiments with multiple sites indicate that EMS is a good practice, as it acts as a proxy to the ADES. As such, it provides more flexibility in security schemes. Technically, nothing would preclude from a service or user to call ADES API directly. By allowing registration of pre-deployed WPS and API routes, the EMS can also act as a federated process catalog. An EMS can take into account services, applications and processes provided by several ADES, facilitating its role of orchestrator of distributed workflows.

Data from different missions, captured at different azimuth or time, need to be co-registered to a common reference before analysis. This registration takes into account the local topography of the terrain to compensate for visual aberrations and missing information from the sources. The purpose of a stacker is therefore to generate a set of analysis-ready data from heterogenous observational sources. The resulting file, a multi-band image, exhibits uniform spatial sampling projected on the same location. Akin to datacubes, the data can then be considered analysis-ready.

The application’s reader and writer operations are compatible with the file types presented in the table below.

The main software packaged in the application is SNAP. Below are the main components used.

For this application, CRIM also created a new toolbox, CRIMTBX, that can be easily added to the current SNAP modules. At the moment of writing, the source code of this toolbox is not yet released as open source software, but is available on demand. Below are the operations provided by the CRIMTBX toolbox and used by the application.

Previous versions of this application were introduced in Testbed-13 [6] and Testbed-14 [2]. The following references provide public information about components, techniques, algorithms and information relevant to this application.

The application’s interface has been kept intentionally simple, with a minimal number of parameters. While this led to simpler packaging, it causes limitations when using multiresolution products as reference images. In this implementation, only the metadata of the first band is used as a reference. In case of products with multiple resolutions, for example Sentinel2 at 10-20-60 meters, it is not possible to use a 10m resolution because these are not the first band.

The purpose of this application is to determine the most likely class for each pixel of a Sentinel-2 image. The figure below shows annotations that were used to train the model.

Once trained, the model output classes or labels for each pixel of an input image.

As the model was trained with Sentinel-2 L2A data, the application only supports this data source. Using any other EO data would produce very noisy outputs.

Below are the main components used in this application.

The source code for the application package is not yet open source, but is available on demand. It can be found on CRIM private code repository. Compiled Docker images from this code can be found in at CRIM private Docker registry.

Previous versions of this application were introduced in Testbed-14 [4] to demonstrate classification of Very High Resolution imagery. It was further developed in Testbed-15 [5] to train and execute a lake-river discrimination model.

Additionally, the following references provide public information about components, techniques, algorithms and background information relevant to this application.

The built Docker image can be called as follows:

This will print the help usage message and argument descriptions by default. Additionally, thelper can be called directly using an explicit override of the executed command as follows:

This second command will result in exactly the same result as calling the ogc-thelper-base image or the original geo-enabled thelper Docker image.

Using cwltool, the Application Package can be tested with the following command:

This call is similar to how an ADES would actually call the process from a WPS-T REST call. As it is a manual test, the process-job file needs to be modified to provide the path to valid input files. Running the image with cwltool will pull the tagged remote Docker image as required, and then call it with the specified job arguments. If not already generated on the remote repository, remote tag can be created locally by calling the following command, which will build and tag the latest version for future push.

In this case, the Docker image also contains the JSON body of process description and matching deployment CWL definitions in order to generate a compatible AP.

Climate indices are not measured in the field or calculated by climate models. They are calculated or derived from climate variables such as temperature and precipitation, often through statistics or metrics related to individual grid cells. An exhaustive list of climate indices offered by Finch can be found in Annex B. Usually, indices can be more easily attributed to specific themes, sectors or domains than raw variables.

The processes have been tested with the datasets presented in the table below. All climate data is stored in NetCDF files adhering to the CF Convention. In all cases, the main variables used in the pilot are temperature and precipitations.

The CMIP6 test data was acquired from ESGF CMIP6 Search, using keywords tasmax and scenariomip. The main use case considered is the simple access and data reduction through a subsetting process. Below is a screen capture of the UI of the search feature. The data can be downloaded at tasmax_day_HadGEM3-GC31-LL_ssp245_r2i1p1f3_gn_20150101-20201230.

Climate indices are computed from climate model outputs using the specialized xclim Python library. Xclim is deployed as a WPS endpoint using Finch, part of the Birdhouse framework. Additional base climate services and processes are also found in PAVICS, a climate research platform built on top of Birdhouse.

As with any other "birds" of the Birdhouse climate framework, Finch is meant to be deployed as an autonomous WPS 1.0 endpoint served by PyWPS. An EMS implemented with Weaver can then register the external endpoint. Once registered, climate services can be discovered and executed using OGC API - Processes, or chained into workflows. The services are considered as applications in the sense that they are provided with appropriate descriptors allowing use in workflows and remote execution. Nevertheless, there is no deployment per se, the closest operation being registration in an EMS.

PAVICS users tend to access to climate data through notebooks. The Birdy native Python client is built on OWSLIB. It allows the dynamic generation of a module whose functions look and feel like native python objects but actually execute remote processes. Scientists and developers can thus blend online WPS functionalities in ordinary scripts, offloading large computation tasks to external dedicated computing resources.

Previous versions of this application were introduced in Testbed-14 activities supported by U.S. Department of Energy’s (DOE) Office of Biological and Environmental Research (BER) [1]. The latest architectural updates can be found in ESGF Future Architecture report. The report notes the following activities as relevant to ESGF.

The following references provide public information about components, techniques, algorithms and contextual information relevant to this application.

One technical objective of the climate services experiments is evaluation of OpenSearch as data acquisition and preparation step of a workflow. This use case is similar to Testbed-14 OpenSearch results.

This service can be found here, while the code is on GitHub. The description document lists the search facets that it supports. This OpenSearch repository expose ESA Climate Change Initiative datasets. OpenSearch uses a drill down approach, where the search is done at the top-level with collections (datasets), then is followed down to granules (individual files). Below are a few sample requests:

CRIM notes the following practices pertaining to the template of an AP:

The following remarks apply to the WPS interface of deployed applications:

For remote sensing apps, most dependencies (operations, libraries) are in Java as SNAP is the main component. Most problems were related to packaging of SNAP since there is not official ESA Docker image. Each participant has his own flavor which leads to inconsistent functionalities and expectations. Also, because each developer is left on his own to package SNAP, there have always been problems related to file/dir permissions, as most developers testing locally have close to root access while server CWL execution does not.

For machine learning apps, the main dependency is PyTorch. A Python helper library called thelper is exposing the application’s entry point. It allows the definition of training/inference of a given model over specified datasets, but providing metadata, chain of transformations and other properties via a YAML configuration. It helps repeatability and customization of the operations and tests without modifications to the code. Without this kind of parametrization, each researcher is left to himself to develop his custom ML workflow and scripts, which leads to many assumptions left undocumented or implementation choices hard to comprehend by following researchers/developers. Fully-parametrized applications helps guide researchers toward a somewhat common development mentality, which provides easier collaboration or extension opportunities over preexisting algorithms. Additional dependencies are pulled from pip and Conda at build time of the Docker image.

For climate services, Birdhouse framework provides individual WPS servers called birds. Each bird runs PyWPS and contains several related processes, such as computation of climate indices. In the framework, dependencies are also managed by combination of pip and Conda. The reuse of existing WPS processes accelerated development of functional application (compared to starting from scratch), but also highlighted the need to support older generation WPS. Although OGC API - Processes offer new functionalities and formats, sometimes these preexisting processes still need to be employed for a longer period. This led to the creation of WPS-1 and WPS-2 "converters" to new WPS-REST bindings.

The reader can also refer to external documentation found at PAVICS WPS 1 and 2 and PAVICS ESGF access. These process definitions allow a common ADES interface, but offered for different kinds of applications such as AP or remote pre-existing, pre-deployed WPS. This also forced consideration of ADES and AP implementations, deployment mechanism and features offered as not exclusively and completely manageable as local Docker image.

In a first scenario, the developed application is managed by the same institution that offer the application package. In that case, it is good practice for the developer to only provide a command-line interface to his executable. The developer is in charge of packaging then create a Docker image where the run command is mapped to the command-line interface. As described previously, the CWL then defines the Execution Unit to the Docker image. An advantage of this practice is that the application can be tested locally easily, without having to redeploy each time. The local execution is also similar to the distant execution on an ADES.

To test basic cases of ADES-EMS deployments, CRIM used an EO application calling a local script, instead of pulling a Docker image. This test app runs a script that extracts and fetches NetCDF files from a JSON file containing an array of URLs, and provides them on the output directory.

In the case of applications, a Docker image is available in a registry. At Deploy time, the process is created on ADES and rendered visible. At Execute time, the image is pulled by the ADES and run, spawning a Docker container on a virtual machine.

For workflows, each step is registered as a process in the EMS. For each step, a Deploy request is sent to the appropriate ADES. Currently, and as it was the case for Testbed-14, the EMS explicitly links different mission data to specific ADES. We recommend that the EMS API provides a way to specify the target ADES. Ultimately, the EMS role could be extended to offer orchestration and optimization rules against a federated data and processing infrastructure.

Applications could benefit from a formal description of their hardware requirements, for example GPU, multicore, memory. Inversely, the ADES would benefit from describing its capabilities and offerings. This way, at Deploy time, application requirements can be checked against ADES capabilities. Amongst advanced platform capabilities to potentially expose to the application user, we note the distributed execution environment, for example Kubernetes or Amazon Elastic BeanStalk.

CRIM favors use of CWL in the Common Architecture for the following reasons: