I. Abstract
Platforms for the exploitation of Earth Observation (EO) data have been developed by public and private companies in order to foster the usage of EO data and expand the market of Earth Observation-derived information. A fundamental principle of the platform operations concept is to move the EO data processing service’s user to the data and tools, as opposed to downloading, replicating, and exploiting data ‘at home’. In this scope, previous OGC activities initiated the development of an architecture to allow the ad-hoc deployment and execution of applications close to the physical location of the source data with the goal to minimize data transfer between data repositories and application processes.
This document defines the Best Practice to package and deploy Earth Observation Applications in an Exploitation Platform. The document is targeting the implementation, packaging and deployment of EO Applications in support of collaborative work processes between developers and platform owners.
The Best Practice includes recommendations for the application design patterns, package encoding, container and data interfaces for data stage-in and stage-out strategies focusing on three main viewpoints: Application, Package and Platform.
II. Keywords
The following are keywords to be used by search engines and document catalogues.
ogcdoc, OGC document, EO, Application Package, Container, CWL, STAC
III. Preface
Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.
Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.
IV. Security Considerations
This document defines Best Practice to package and deploy an Application on a Platform that implies a trust relationship between the Application developer, the Application integrator, the Platform and the consumer.
No security considerations have been made for this Best Practice.
V. Submitting Organizations
The following organizations submitted this Document to the Open Geospatial Consortium (OGC):
- European Space Agency
- Terradue
- CRIM
- CubeWerx Inc.
- 52°North GmbH
- SatCen
- Telespazio VEGA UK
- RHEA Group
- Pixalytics
- Solenix
- West University of Timisoara
VI. Submitters
All questions regarding this submission should be directed to the editor or the submitters:
Name | Representing |
Pedro Gonçalves (editor) | Terradue |
Fabrice Brito | Terradue |
Tom Landry | CRIM |
Francis Charette-Migneault | CRIM |
Richard Conway | Telespazio VEGA UK |
Adrian Luna | European Union Satellite Centre |
Omar Barrilero | European Union Satellite Centre |
Panagiotis (Peter) A. Vretanos | CubeWerx Inc. |
Cristiano Lopes | European Space Agency (ESA) |
Antonio Romeo | RHEA Group |
Paulo Sacramento | Solenix |
Samantha Lavender | Pixalytics |
Marian Neagul | West University of Timisoara |
OGC Best Practice for Earth Observation Application Package
1. Scope
This document defines the Best Practice to package and deploy Earth Observation Applications in an Exploitation Platform.
The document is targeting the implementation, packaging and deployment of EO Applications in support of collaborative work processes between developers and platform owners. It supports developers that want to adapt and package their existing algorithms written in a specific language to be deployed in Earth Observation Exploitation Platforms and exposed through a Web Service endpoint, OGC API — Processes.
The Best Practice includes application design patterns, package encoding, container and data interfaces for data stage-in and stage-out strategies.
Section 6 introduces the information material about Earth Observation (EO) Platform architecture targeting the deployment and execution of EO Applications in distributed Cloud Platforms. The section provides an overview of EO applications design patterns, package and data interfaces.
Sections 7, 8 and 9 present the normative material defining the best practices to implement an EO Application, to package an EO Application and to deploy the packaged EO Application.
2. Conformance
This document defines the Best Practice for Earth Observation Application Packages targeting three standardization targets:
-
Application — defines the best practices when implementing an EO Application
-
Package — defines the best practices when packaging an EO Application
-
Platform — defines the best practices when deploying a packaged EO Application in a platform
For the three standardization targets listed above, this Best Practice focuses on Earth Observation Applications that require the staging, input, and/or output of EO Products. The Best Practice also discusses the implications for the Application Package and hosting Platform.
In order to conform to this OGC Best Practice, an application developer shall choose to implement the following conformance classes:
In order to conform to this OGC Best Practice, and according to the Application Conformance Class, the application integrator shall implement the following conformance classes:
In order to conform to this OGC Best Practice, and according to the Application Package Conformance Class, a platform shall choose to implement the following conformance classes:
Conformance with this Best Practice shall be checked using all the relevant tests specified in Annex A (normative) of this document. The framework, concepts, and methodology for testing, and the criteria to be achieved to claim conformance are specified in the OGC Compliance Testing Policies and Procedures and the OGC Compliance Testing web site.
All requirements-classes and conformance-classes described in this document are owned by the documents(s) identified.
3. Normative references
The following documents are referred to in the text in such a way that some or all of their content constitutes requirements of this document. For dated references, only the edition cited applies. For undated references, the latest edition of the referenced document (including any amendments) applies.
Arliss Whiteside Jim Greenwood : OGC 06-121r9, OGC Web Service Common Implementation Specification. Open Geospatial Consortium (2010). https://portal.ogc.org/files/?artifact_id=38867
OGC API — Processes — Part 1: Core Standard, 2021. https://docs.ogc.org/is/18-062r2/18-062r2.html
Commonwl.org: Common Workflow Language Specifications, https://w3id.org/cwl/
Radiant Earth Foundation: SpatioTemporal Asset Catalog specification, https://stacspec.org
4. Terms and definitions
This document uses the terms defined in OGC Policy Directive 49, which is based on the ISO/IEC Directives, Part 2, Rules for the structure and drafting of International Standards. In particular, the word “shall” (not “must”) is the verb form used to indicate a requirement to be strictly followed to conform to this document and OGC documents do not use the equivalent phrases in the ISO/IEC Directives, Part 2.
This document also uses terms defined in the OGC Standard for Modular specifications (OGC 08-131r3), also known as the ‘ModSpec’. The definitions of terms such as standard, specification, requirement, and conformance test are provided in the ModSpec.
For the purposes of this document, the following additional terms and definitions apply.
This document uses the terms defined in Sub-clause 5.3 of OGC 06-121r9, which is based on the ISO/IEC Directives, Part 2, Rules for the structure and drafting of International Standards. In particular, the word “shall” (not “must”) is the verb form used to indicate a requirement to be strictly followed to conform to this standard.
For the purposes of this document, the following additional terms and definitions apply.
4.1. Application
A self-contained set of operations to be performed, typically to achieve a desired data manipulation, written in a specific language (e.g. Python, R, Java, C++, C#, IDL).
4.2. Application Package
A platform independent and self-contained representation of an Application, providing executables, metadata and dependencies such that it can be deployed to and executed within an Exploitation Platform.
4.3. Compute Platform
The Platform providing the computational resources for the execution of the Application.
4.4. Container
A container is a standard unit of software that packages up code and all its dependencies so that includes everything needed to run an application: code, runtime, system tools, system libraries and settings.
4.5. Exploitation Platform
An on-line system made of products, services and tools for exploitation of data.
4.6. Spatiotemporal Asset
Any file that represents information about the earth captured in a certain space and time.
4.7. GeoTIFF
A public domain metadata standard that allows georeferencing information to be embedded within a TIFF file. The potential additional information includes map projection, coordinate systems, ellipsoids, datums, and everything else necessary to establish the exact spatial reference for the file.
4.8. HDF5
The Hierarchical Data Format version 5 (HDF5), is an open source file format that supports large, complex, heterogeneous data. HDF5 uses a “file directory” like structure that allows you to organize data within the file in many different structured ways, as you might do with files on your computer
4.9. JPEG2000
An image compression standard and coding system
4.10. SAFE
SAFE, the Standard Archive Format for Europe, is designed to act as a standard format for archiving and conveying Earth observation data within the European Space Agency (ESA) archiving facilities and, potentially, within the archiving facilities of cooperating agencies.
4.11. Processing Result
The Products produced as output of a Processing Service execution.
4.12. Processing Service
A non-interactive data processing provided as a service by a platform that has a well defined set of input data types, input parameterization, producing Processing Results with a well defined output data type.
4.13. Processing Software
A set of predefined functions that interact to achieve a result. For the exploitation platform, it comprises interfaces to derive data products from input data, conducted by a hosted processing service execution.
4.14. Products
Earth Observation data (commercial and non-commercial) and value-added data. It is assumed that the Exploitation Platform provides the data access mechanisms for an existing supply of Earth Observation Products.
5. Conventions
This section provides details and examples for any conventions used in the document. Examples of conventions are symbols, abbreviations, use of XML schema, or special notes regarding how to read the document.
5.1. Identifiers
The normative provisions in this document are denoted by the URI
http://www.opengis.net/spec/eoap-bp/1.0
All requirements and conformance tests that appear in this document are denoted by partial URIs which are relative to this base.
5.2. Abbreviated terms
-
API Application Programming Interface
-
COG Cloud Optimized GeoTIFF
-
CWL Common Workflow Language
-
EO Earth Observation
-
EP Exploitation Platform
-
GDAL Geospatial Data Abstraction Library
-
IW Interferometric Wide
-
JSON JavaScript Object Notation
-
OGC Open Geospatial Consortium
-
OS Operating System
-
OWS OGC Web Services
-
REST Representational State Transfer
-
SAFE Standard Archive Format for Europe
-
SLC Single Look Complex
-
SNAP Sentinel Application Platform toolbox
-
STAC SpatioTemporal Asset Catalog
-
TBD To Be Determined
-
TIFF Tagged Image File Format
-
URL Uniform Resource Locator
-
YAML YAML Ain’t Markup Language
6. Components Overview
6.1. Introduction
In recent years, Platforms for the Exploitation of Earth Observation (EO) data have been developed by public and private companies in order to foster the usage of EO data and expand the market of Earth Observation-derived information. The domain is composed of platform providers, service providers who use the platform to deliver a service to their users, and data providers. The availability of free and open data (e.g. Copernicus Sentinel), together with the availability of affordable computing resources, creates an opportunity for the wide adoption and use of EO data in a growing number of fields in our society.
An EO exploitation platform is a collaborative virtual work environment providing the mechanisms to deliver EO data and the tools, processors, and ICT resources required to work with them, through one coherent set of interfaces. A fundamental principle of the platform operations concept is to move the EO data processing service’s user to the data and tools, as opposed to downloading, replicating, and exploiting data ‘at home’. Furthermore, platforms offer an environment which takes care of all data processing orchestration tasks and the availability of scalable computational resources offered by the Cloud shortens the time to market of applications.
In this scope, OGC Testbeds 13-16 initiated the drafting of an architecture to allow the deployment and execution of externally developed applications on Earth Observation (EO) data and processing platforms. During the OGC Innovation Program initiative OGC Earth Observation Applications Pilot, conducted between December 2019 and July 2020 (OGC 20-042, OGC 20-073), the participants explored and assessed the existent draft specifications, which addressed both application description and discovery, APIs for deployment, execution, and result access, as well as specifications for service chaining and workflow building. This document summarizes their findings and defines a Best Practice to package and deploy Earth Observation Applications in an Exploitation Platform.
In summary, the architecture as defined by the OGC Earth Observation Applications Pilot targets the following requirements:
-
Decouple application developers from exploitation platform operators and from application consumers
-
Allow application developers to make their applications available on any number of platforms with minimal modifications
-
Allow application developers to focus on application development by minimizing platform specific particularities
-
Enable exploitation platforms to virtually support any type of packaged EO application
The figure below provides a high-level view of the interactions between the actors and the architecture.