Developers

Explore example code snippets

Interacting with the Data Flow Index (DFI)

DFI customers can integrate with the DFI web service with their choice of technology such as Python, Javascript, Rust and Java. The simplest way to connect is via the Python SDK where you can integrate with analytics tools such as Jupyter Notebooks.

The DFI Web API is at https://api.dataflowindex.io/docs/api/

Data Sets

The solution is designed for analysing moving entities that use location-based data such as vehicles, GPS trackers and smart phones.

Think of a dataset as a large table with Entity ID, its location (latitude, longitude, altitude), a date/timestamp and an optional payload of 8 integer fields.

A record is a single observation of an entity so there may be several records for each entity as it changes location over time.

‍

‍

What can you do with the DFI Web API?

Ingest and add records from datalakes such as AWS S3
Run spatiotemporal queries such as points-in-polygon on your data including entity ID, latitude and longitude, and date/time
Remove records
Filter up to 8 custom attributes to quickly find records of interest
Store and retrieve unlimited attributes that your data may have

‍

While there are different types of spatiotemporal queries supported, the solution is particularly suited to complex event detection and queries such as Polygons.

‍

About Polygon Queries

Polygons are typically used to analyse moving objects such as phones, vehicles and trackers, making them suitable for retail, logistics, delivery and law enforcement applications.

Queries can be large (>100MB) and computationally intensive. Historically, they have been time-consuming, complex and expensive to run in real time or at scale.

Draw a perimeter around a shape (polygon) to see what entities are located within the polygon over a given time. These queries (count, history, unique entities) are extremely powerful and allow you to ask questions such as:

What unique entities were in the area at a certain time?

What is the historic movement of those entities?

Alert me when certain entities enter or exit the location (geofences)

What other entities were co-located or in proximity at the same time?

A world of spatiotemporal data

The example queries below were run against a synthetic data set of 1.6 million vehicles in London with historic movements loaded in time order to simulate a real-time feed of 92.4 billion data points. The data was stored on an AWS single server with 24 CPUs, 192GB of RAM and 2 x 7.5TB and NVMe SSD.

Set up

Simply install the Python package and connect to the DFI:

from dfi import Client
token = <your personal DFI token>
dataset = "london_traffic"
dfi = Client(token, dataset, "hhtps://api.dataflowindex.io")

Copied!

Examples

What Next?

Apply for your own API token at https://www.generalsystem.com/test-drive

Follow our tutorials https://github.com/thegeneralsystem/dfipy-examples
‍
If you are not using Python, the DFI Web API is at https://api.dataflowindex.io/docs/api/ and we also provide tutorials to show how to call the API methods https://github.com/thegeneralsystem/quickstart-guide/tree/main