The Evolution of Field Data Collection

Feb 20, 2020

By Sean King


No-Code Tools


Streamlining the flow of data from the field to the final report is vital for any organization that conducts on site-projects, inspections, or monitoring as a core part of their operations. Common use-cases include Environmental Consulting, Engineering, and Construction projects initiated by private industry or public utilities. This article will discuss the evolution of field data management from the era of paper forms to current digital best practices and the future of 24/7 condition monitoring leveraging the Industrial Internet of Things (IIoT).

The Past

Mountains of paper, errors and frustration

Veteran environmental professionals, civil engineers, and industrial inspectors will remember how data collection used to be. Preparing paper forms, locating existing records, hand-drawing site diagrams, and having photographs developed… Then physically returning it all to the office, only to wait further for the data to be transcribed, analyzed, and typed up into a final hard copy report.

And don't forget how difficult it could be to decipher handwritten paper forms filled out in inclement weather or by staff with 'less than perfect' penmanship… Preventable human errors that could cost valuable time and lead to significant project risk.

Thankfully over the last two decades, things have evolved. Improvements in mobile computing have led to a revolution in mobile data collection that has allowed scientists, engineers, and technicians to gradually dedicate more and more time towards developing insights and analysis, and less time to mundane data entry and administrative tasks.

The Present

Smart forms – no more paper!

The rise of mobile devices has made it possible to reliably collect and manage data at the source. In recent years, many vendors have offered simple mobile form solutions to replace paper in the field. Although limited, these initial simple form solutions were an important step in the evolution of data collection as they allowed basic records to be extracted directly into spreadsheets that were relatively easy to manipulate with reduced potential for human error.

As technology has advanced, organizations across different industries now leverage more robust data collection apps. These apps include a mix of smart and simple fields, providing the ability to collect and archive rich data types, including images, audio, GPS locations, and barcode scanning.

Sophisticated workflows can be configured to include on-device calculations that ensure data accuracy, automated email notifications and reporting that improve communication between the field and office, and offline functionality to guarantee data is not lost when working in remote or restricted areas.

Centralized database - from the field to the cloud!

Leading data collection solutions are powered by centralized, structured relational databases. SQL databases are increasingly hosted in the cloud, allowing organizations to avoid having to incur the cost of hosting data centers.

A structured approach to data management provides two major benefits over the paper-based and simple spreadsheet driven data collection processes of the past.

Firstly, data collected on smartphones can be submitted instantly to the database using an internet connection which completely removes delays and errors associated with retrieving, transcribing, and storing multiple paper forms or versions of a spreadsheet file.

The second major benefit of centralized databases is real-time two-way communication, sending and receiving data between the field and office, which allows organizations to implement sophisticated workflow capabilities. Data collection requirements may change depending on a variety of factors. As a result, organizations can now create custom apps that guide information through a business process from beginning to end and throughout the entire enterprise.

This two-way communication empowers field users to see data in real-time to make better and more timely decisions. For example, a technician performing data sampling can now access related records on-site conditions in the field instantly, providing an important source of context.

No more rifling through past paper forms, in the field or the office, to find important information! Every record is available in an instant.

Mapping & Integration - closing the loop, more convenient data collection

In the past, a lack of viable tools for tying structured data to maps made it difficult for organizations to leverage data visualizations based on geographic locations.

Hand-drawn diagrams would show the location of sample sites points of interest during inspections. Collecting and maintaining the relevant data to create up-to-date digital maps proved to be prohibitive for even the most well-resourced organizations.

Modern advances in Geographic Information Systems (GIS) data collection and management practices have made it possible for organizations to embrace mapping as part of their workflows.

Coordinates captured by smartphone or commercial receivers can be rendered on basemaps connected to a centralized database make creating and editing records from a geospatial perspective viable for organizations of any size or technical ability. This means that the old process of drawing diagrams or manually recording coordinates can be completely discarded.

Advances in system integration capabilities have also aided field data collection, allowing different applications and devices to more easily communicate and share data. For example, if you require GNSS location coordinates, using either a commercial device or mobile device functionality, these readings can be easily incorporated into your database and then shared with external software such as ERP or CRM solutions to create one unified system driven by geospatial data.

The Future

IIoT and Automation - feed your software robots with seamless data streams

Moving forward, the rise of the Industrial Internet of Things (IIoT) promises to bring field data collection into a new automated era.

From paper and pen to smart forms powered by smartphones, IIoT will allow businesses to continually monitor conditions in real-time, 24/7.

This continuous stream of data will be collected by internet-enabled devices featuring sensors and on-board processors to instantly relay readings to the central cloud database, where automated software robots will be able to process and integrate data according to business logic.

For organizations that require frequent or high-volume data collection, IIoT will revolutionize their workflows and allow for previously unimaginable levels of automation and accuracy.

About Flowfinity

To discover how your firm can optimize your data collection workflows, contact one of our experts.

« Make Better Business Decisions with Dashboard Maps

4 Ways to Defeat Your Application Development Backlog with No-Code Tools »