Thursday 27 March 2025
The pursuit of better data management has been a driving force in various industries, and transportation research is no exception. A recent study delves into the intricacies of managing sensor data collected during field testing of instrumented pavement sections on I-65 and I-69 Greenfield districts.
For researchers, managing this vast amount of data can be a daunting task. With millions of sensor samples recorded daily, it’s crucial to have a system that can efficiently store, process, and analyze the data in real-time. The study proposes a comprehensive end-to-end solution for collecting embedded sensor data remotely using solar-powered equipment.
The researchers utilized a Campbell Scientific data acquisition system (DAQ) with built-in support for MQTT data transmission. This enabled seamless integration with a centralized NATS server, which received the data payloads via wildcard subscription on all sensors’ topics. The study demonstrates how this architecture can reduce latency between measurement and use in transportation research decisions.
For historical datasets, summarizing and metric data are suitable for storage in relational databases like Oracle, while unprocessed data is more suited to blob storage solutions. Researchers primarily prefer sensor metrics over raw sensor data, reducing complexity by applying analytical methods before storing the data.
The study also highlights the importance of data visualization tools like ArcGIS Online, which allows researchers to create customizable dashboards and share them with others. This feature enables collaborative supervision and broad-scale inspection of the sensor data stream.
In this context, TimescaleDB, an open-source database optimized for time-series data, proved to be an ideal storage solution. It efficiently organizes incoming data streams by partitioning both spatially and temporally, allowing for complex analytical queries.
The real-time system architecture proposed in the study seamlessly integrates Avena, a modern framework that enables edge computing on the data stream. This reduces latency between measurement and use, enabling researchers to monitor sensors 24/7 and produce useful metrics in real-time.
While the limitations of the Campbell DAQ hardware motivated further exploration into introducing Avena within the roadside DAQ, the study demonstrates the potential for improved data flow. The comprehensive end-to-end solution proposed here offers holistic solutions for static and real-time data management in transportation research studies, encouraging researchers to adopt these proposals into various projects.
The study’s findings have significant implications for the future of transportation research, enabling more efficient data collection, processing, and analysis.
Cite this article: “Optimizing Data Management in Transportation Research: A Comprehensive End-to-End Solution”, The Science Archive, 2025.
Sensor Data Management, Pavement Testing, Instrumented Sections, I-65, I-69 Greenfield Districts, Campbell Scientific Daq, Mqtt Data Transmission, Nats Server, Timescaledb, Edge Computing, Avena Framework
