OGC Testbed-16: Full Motion Video to Moving Features Engineering Report

The first phase of the algorithm is the initialization or "seeding" of tracks where new detections are used to instantiate tracks.

VMTIs from the first frame or VMTIs that cannot be matched to an existing track each begin a new track. The VMTIs from the following frames are loaded. Each track looks for VMTIs within a given radius of the last known position. The new VMTIs that are matched to existing tracks are used to extend them. VMTIs that don’t match to any existing track start a new track

Existing tracks are expanded by matching detections to the track’s predicted location based on speed and bearing.

Based on the direction and speed of the moving object, their predicted location is computed. The algorithm looks for the next track location within a given radius of the predicted location and expands the track. If no VMTI is found within the radius, the track is finalized after some time.

During track expansion, detections may come close to the predicted location of several tracks.

In the situation above, both VMTIs fall into the radius of the predicted location of two tracks. This is resolved by matching the VMTIs to the predicted locations in order to minimize the overall predicted to true location distance.

Detections may be missing because the target was obscured, because the camera looks away or because the detection algorithm fails. Missing detections may be accounted for by keeping tracks alive for a certain amount of time in the hope of matching it with a detection at a later time.

This solution works well as long as the tracked object keeps the same speed and direction. Increasing the radius within which to search for matching detections helps to account for acceleration and heading changes but runs the risk of mixing up the tracks. This solution works well once the search radius and keep-alive time are optimized for a certain type of moving object.

False positives are detections that do not actually correspond to a moving object. They may be due to many things such as video glitches that the detection algorithm interprets as an actual object. Removing outliers is tempting but an outlier is not necessarily a false detection and it may help expand tracks with few detections. Another approach is to filter out unusable tracks after the tracking algorithm is done. These are tracks made up of just a few detections which could not be matched with any existing tracks and therefore generated tracks of their own.

The image above shows the long tracks on the right that follow the roads and the short jittery tracks made up of false positives on the left.

The raw tracks can be jittery and lack resolution. A simple method to smooth and interpolate the tracks can be used. In this implementation, the location of a track at a given time is calculated based on a weighted average of the forces pulling the track to the next known true locations.

The algorithm can be made probabilistic by considering information such as class, color, intensity and target confidence level that are sometimes present in MISB 0903 VMTIs. This information would weigh-in to resolve some ambiguous situations.

The speed and acceleration of the vehicle may be provided inside the VTracker LDS packages or they may be calculated during tracking. The result may be encoded as temporal properties with stepwise, linear or spline interpolation

The VObject LDS uses an ontology to describe the set of allowed classes or type values encoded as Web Ontology Language (OWL). The MISB 0903 element is made up of a URI link to the ontology and a class. Once a parser is available, encoding the temporal property may be trivial. However, the class may also describe enumerations, unions, intersections, complements and property restrictions. Rather than forcing this information into an ill-suited format, the class may be encoded as full text in the temporal property’s values while the URI to the ontology can be encoded as the property’s Unit of Measure (uom).

While the use of a json Property rather than a Temporal Property for this field would be better suited in many cases, this approach could not account for a one to one mapping between MISB 0903 and OGC Moving Features.

MISB 0903 adopted the OGC Geography Markup Language (GML) and the VFeature LDS is based on the OGC Observations and Measurements (O&M) Encoding Standard and related schemas.

The VFeature LDS is made up of a URI to a schema and a Feature. A Feature is represented as an OGC GML document structured according to the schema. The GML feature contains one or more values observed for a VMTI.

While the GML data may be parsed, nested data will be difficult to encode in the OGC Moving Features format because the values of a temporal property cannot be nested. According to the Moving Features standard, a temporal property must be a flat array of Strings or numbers.

While the properties field (rather than temporal properties) allows nested values, the temporal aspect of the data would have to be implemented as a parallel mechanism not described by the schema.

The data can instead be encoded in full text as values and the URI to the schema can be encoded as the property’s uom.

The VTracker LDS, if present and populated, groups detections into tracks. This achieves the same task as the Tracking Algorithm although the technique used and results may be different. The VTracker LDS packages may provide Speed and Acceleration information as well as a tracker specific confidence value.

Other values from this LDS concern information that was already covered.

The VChip LDS defines an image or a link to an image with the corresponding Multipurpose Internet Mail Extension (MIME) type. If present, the information can be encoded as a temporal property. For the case where actual images are provided, they may be encoded as byte strings.

Parsing the MPEG-2 Transport Stream (TS) sample files used in this project demonstrated the complexity of the process. The steps to access metadata content were:

Established World Wide Web Consortium (W3C) formats such as Timed Text Markup Language (TTML) and Web Video Text Tracks (WebVTT) synchronize data with video using out-of-band metadata which is stored in a discrete linked file. This approach makes access faster and simpler by eliminating the complex video stream structure and offering a common format that is not dependent on the video codec or container format.

If video metadata were exported to a common out-of-band format, such as WebVMT, metadata content could be accessed more quickly and efficiently. This reduces development time and file bandwidth and also allows metadata parsing to be agnostic of the media encoding format. WebVMT has the added advantage of being designed for the web so metadata can be easily exposed through HTML DataCue. This enables access through web browser APIs, such as JavaScript, and by web crawlers so users can rapidly identify key sections of metadata and video footage using online search engines.

The following gives a general guide for mapping UAS Datalink LS packets.

In general, mapping the Frame Center (MISB 0601, tags 23 & 24) to the WebVMT map center is a sensible choice. There are three possible mappings to WebVMT map zoom using MISB target width [1], field of view and range [2], and corner offsets [3]. Object trajectories, such as the Sensor Location (MISB 0601, tags 13 & 14), can be mapped to a WebVMT path and in certain use cases mapping the Frame Center to a WebVMT path can be insightful.

Any set of Target Locations (MISB 0903, VTarget Pack tags 10, 11 & 17) which form an object trajectory can be mapped to a WebVMT path, and multiple paths can be discriminated by using the WebVMT path identifier. Care should be taken not to overload the user when using WebVMT data with multiple paths, either by limiting the number of paths shown or by limiting the length of the path history displayed.

These videos show footage captured from a commercial Unmanned Aerial Vehicle (UAV), colloquially referred to as a drone, with a camera that can face directly downward or look forward in the direction of travel.

In both cases, Frame Center (MISB 0601, tags 23 & 24) and Target Width (MISB 0601, tag 22) control the map center and zoom respectively, and Sensor Location (MISB 0601, tags 13 & 14) is mapped to a WebVMT path shown as a blue line in the figures below.

When the camera is facing downward, the front of the path remains aligned with the map center and accurately follows the path of the river, as expected, which demonstrates that the location details are correctly exported to WebVMT.

When the camera is facing forward, the front of the path lags behind the map center and is correctly positioned between the two parallel roads seen in the footage. Once again, this is consistent with the video content and demonstrates that location is accurately exported.

Files for these demos are available from the OGC Portal at Innovation Program/OGC Testbed-16/Full Motion Video/Demos/20200827_WebVMT which can be displayed at the WebVMT website as described in the Readme file. OGC Portal access is available to OGC Members only.

This footage shows a haulage truck travelling along a highway filmed from a circling reconnaissance aircraft. Though the source video file contains no MISB 0903 data to track this target, the Frame Center (MISB 0601, tags 23 & 24) has been exported to a WebVMT path to best represent this from the information available.

The accuracy of the exported location data is demonstrated as the truck passes under a bridge, showing that the truck and path are well synchronized.

A crosshair overlay baked into the video footage indicates the position of the frame center. Once again, the accuracy is demonstrated as the location of the frame center drifts on to the opposite carriageway and across to the truck stop exit road before recentering on the target truck, and this is correctly reflected in the exported WebVMT path.

Files for this demo are available from the OGC Portal at Innovation Program/OGC Testbed-16/Full Motion Video/Demos/20200903_WebVMT which can be displayed at the WebVMT website as described in the Readme file. Note that the MapboxGL display option may not be available as this was a pre-release prototype feature at the time of publication of this ER.

Video footage evidence can be submitted by the public with accurate location, for instance to report dangerous drivers using dashcam footage or to report forensic evidence of boats smuggling goods/people spotted from pleasure craft or the shore using smartphone footage. See also the WebVMT Police Evidence use case;

Benefits: Video evidence with accurate location in machine-readable format can be sourced from the public.

Provide a common format for sharing body-worn video footage for police and rescue services, which is compatible with disparate streams from vehicle dashcams and aerial video from drones and helicopters that can be easily aggregated for situational awareness or during pursuit. See also the WebVMT Area Monitoring use case;

Benefits: Geotagged video can be aggregated and shared in a common format for live pursuit/situational awareness.

Underground features can be inspected using a remote camera, either integrated into a drilling rig or mounted on a remotely operated vehicle in a borehole or pipe. Location is calculated using inertial navigation so video frames can be tagged with geospatial information to enable features visible to the camera (and other sensor data) to be accurately mapped. This technique is used in the oil and gas industries, and can also offer valuable insight for inspection of underground assets such as pipes and cables.

Benefits: Valuable assets in inaccessible locations can be inspected cheaply and accurately with searchable access.

WebVMT’s out-of-band design is well-suited for moving objects and sensor data to decouple metadata access from the current video playback time. This approach avoids the inherent overheads of in-band design and enabling seamless integration with web browsers and search engines to make video metadata easily accessible to the online community. Exporting MISB metadata to WebVMT has demonstrated the value of exposing geospatial metadata in a common format, including:

SensorHub is a Compusult product that harnesses the OGC SensorThings API to shuttle data from various Internet of Things (IoT) sources to a common destination dashboard. The SensorHub application consists mainly of drivers, which take either fixed files or periodically polled source URLs as inputs and generate SensorThings sensors as output.

Sensors can be viewed and manipulated via the Dashboard Editor. Users can see the current observation values of multiple widgets or navigate through the historically recorded values of a single one. Front-end widgets that depict similar units of measure will be displayed in a consistent manner, even if the originating data sources are in different formats.

SensorHub also supports logic to create rule-based alerts, which, via broadcast protocols such as Message Queuing Telemetry Transport (MQTT), can be transmitted using the Tasking component of the SensorThings API.

The Sensing side of the OGC SensorThings API deals with creating, housing, and broadcasting hierarchically structured observations and is the primary format into which the end products of all SensorHub inputs must be converted. SensorHub components are depicted in the below diagram:

Structurally, the MovingFeatures package is typical of a SensorHub driver package. Each of these accepts files of a certain type (or set of types) as an input, converts the files to output OGC SensorThings entities, and performs the associated input/output (I/O). User interaction with the main class, MovingFeaturesDriver, is done via a MovingFeaturesConfig class and associated Jakarta Server Pages (.jsp) document, wherein the user can manage a list of file paths.

This driver supports inputs in either Full-Motion Video formats (e.g., .mp4, .h264, .mpeg4) or existing MovingFeatures .json formats, since video files will be converted to MovingFeatures JSON as part of the process of converting them to OGC SensorThings. The workflow for both file types is depicted in the below Unified Modeling Language (UML) diagram:

The simplified diagram above gives an idea of how the application handles a Video File. The MISB metadata is accumulated and decoded to a common data model. The data model contains the telemetry information and VMTIs. Then, the data model is augmented with tracks obtained from the tracker.

A UI widget allows the user to change the playback time. The Video Orchestrator sends a message to the data model that the current time has changed updating the different visual layers.

The layers show the individual VMTI detections, the tracks, the platform and field of view, the draped video and a separate window for an un-draped video.

The application also allows connecting to a video stream. The difference is the absence of a playback widget and the tracking that occurs in real time.

Drones have limited hardware but are asked, in real-time, to encode video, detect moving objects and calculate their location relative to the telemetry of the drone and the elevation model they have in memory. As a result, the data is not always high quality.

Several factors can affect the accuracy of detections. The algorithm that detects moving objects from the video is subject to errors, the elevation model of the earth directly impacts the conversion from pixel to longitude/latitude and finally, the telemetry of the drone and camera may be inaccurate.

More generally, this results in inaccurate locations, false positives and missing detections.

To some extent, the quality of the data can be improved with post-processing.

Testing of the MovingFeatures package was limited, largely due to a shortage of available sample data that is compliant with MISB and other North Atlantic Treaty Organization (NATO) standards.

The difficulty of obtaining NATO standards–compliant FMV data files has, for both demonstrators, been the greatest shortcoming of the FMV-to-Moving Features initiative and its associated ability to be tested. Nonetheless, a varied collection of data files from two main sources were obtained.

One of the demonstrator’s data sources is an archive of datasets provided by a public Google Drive folder. The data source comprised eight files containing Unmanned Air System (UAS) full-motion data recorded in various locations. Specific files and their locations were:

Although the videos’ streaming formats differ, the metadata bytes’ structure is identical, being comprised of UAS data with a top-level UAS universal key. As such, the MovingFeatures package can accept and parse these data sources in the same way.

Eight sample files were contributed by the United States Department of Defense’s Joint System Integration Laboratory (JSIL). These files depict moving objects on various military bases. The eight specific files and the locations covered are:

As in the other sources, the top-level data sets in these files are all UAS and most use exclusively those keys defined in MISB 0601. Of these source files, only S05.ts and S06.ts contain VMTI data, and thus only they can be used as inputs to test certain use-cases of the WebVMT application. Moreover, even their VMTI sets are compliant with only the deprecated MISB 0903.3 standard and not the current 0903.5 revision.

S05_VTracks.ts and S06_VTrack.ts contain VTrack local sets as descendants of the top-level UAS ones. As such, they permit writing and testing of the MovingFeatures workflow that parses data out of this dataset type. They are also otherwise equivalent to S05.ts and S06.ts, meaning VTrack observations can be compared against the observations from these. The MISB 0903 in these files contains missing data, inaccurate detections and inaccurate telemetry to re-create a realistic situation.

Testing of the MovingFeatures package has thus far consisted mainly of ad-hoc unit-tests that arose as new modules were written. Feedback from the tests is simply available via Compusult’s SensorHub through its built-in Apache Log4j logging system as well as the results conveyed to the dashboards.

Unit tests conducted on the MovingFeatures package, in chronological order, included the following.

OGC O&M was extensively reviewed but not implemented through a working encoder and decoder of moving tracks. A more thorough investigation of OGC O&M is needed to conclude on that standard’s usability as an interoperability format for tracks exported from MISB 0903 and post-processed for various use-cases.

WebVMT work in Testbed-16 FMV produced successful results which could be developed further and identified new topics worth investigating in the future.

A Deep Learning approach for point clouds can be applied to the VMTI tracking problem. In theory, a network like PointNet++ or similar can segment the detections into tracks and handle the problem of missing detections and false positives. Such a tracking algorithm can also be trained with varied datasets so that the tracking algorithm does not need to be tuned to the type of object it is tracking. Moreover, a point-cloud deep-learning network can be used as a classifier to identify abnormal behaviors such as drunk-driving or sea piracy.

Such an algorithm can also play a role in organizing a library of videos by identifying videos with similar content.

Training these algorithms requires considerable data and building a database of tracks with a classification schema would be invaluable.

Using a fleet of autonomous sensors to monitor the volume and velocity of traffic as it flows through city streets. The collected data is the moving feature representation of all moving vehicles. Collection will go through motion imagery sensors capable of generating VMTI. Note that only the VMTI needs to be collected. The motion imagery can be discarded once the motion data has been extracted.

The result is a collection of moving features which represent the actual traffic during a period of time. This would allow detailed modeling of traffic behavior within an urban setting. The model would include not just the aggregate properties, but also the behavior of individual vehicles within a flow. Similar capabilities have been discussed in the context of urban planning. See https://portal.ogc.org/files/?artifact_id=82917 for one example.

This use case focuses on detecting a small UAV in video streams, estimating the position of that UAV and then generating MovingFeatures data with the position and trajectories. Multiple Video streams would be needed to triangulate the position of the object.

A variety of algorithms are being developed to control the movement of drone swarms. Drones within the flock adapt their location based on the observations that are being made by the collective of sensors on the UAVs and this may call for a new standard.