Explore example code snippets
Interacting with the Data Flow Index (DFI)
The DFI Web API is at https://api.dataflowindex.io/docs/api/
The solution is designed for analysing moving entities that use location-based data such as vehicles, GPS trackers and smart phones.
Think of a dataset as a large table with Entity ID, its location (latitude, longitude, altitude), a date/timestamp and an optional payload of 8 integer fields.
A record is a single observation of an entity so there may be several records for each entity as it changes location over time.
What can you do with the DFI Web API?
- Ingest and add records from datalakes such as AWS S3
- Run spatiotemporal queries such as points-in-polygon on your data including entity ID, latitude and longitude, and date/time
- Remove records
- Filter up to 8 custom attributes to quickly find records of interest
- Store and retrieve unlimited attributes that your data may have
While there are different types of spatiotemporal queries supported, the solution is particularly suited to complex event detection and queries such as Polygons.
About Polygon Queries
Polygons are typically used to analyse moving objects such as phones, vehicles and trackers, making them suitable for retail, logistics, delivery and law enforcement applications.
Queries can be large (>100MB) and computationally intensive. Historically, they have been time-consuming, complex and expensive to run in real time or at scale.
Draw a perimeter around a shape (polygon) to see what entities are located within the polygon over a given time. These queries (count, history, unique entities) are extremely powerful and allow you to ask questions such as:
A world of spatiotemporal data
The example queries below were run against a synthetic data set of 1.6 million vehicles in London with historic movements loaded in time order to simulate a real-time feed of 92.4 billion data points. The data was stored on an AWS single server with 24 CPUs, 192GB of RAM and 2 x 7.5TB and NVMe SSD.
Simply install the Python package and connect to the DFI:
from dfi import Client
token = <your personal DFI token>
dataset = "london_traffic"
dfi = Client(token, dataset, "hhtps://api.dataflowindex.io")
Apply for your own API token at https://www.generalsystem.com/test-drive
Follow our tutorials https://github.com/thegeneralsystem/dfipy-examples
If you are not using Python, the DFI Web API is at https://api.dataflowindex.io/docs/api/ and we also provide tutorials to show how to call the API methods https://github.com/thegeneralsystem/quickstart-guide/tree/main