Synchronizing Big Data on Mobile Devices
Sept 23, 2013
Dealing with big data on mobile devices requires big thought. When considering a mobile enterprise application solution, it’s important to carefully define data usage and update requirements upfront, or face sky high cellular data costs, overtaxed servers, and slow performance.
One of Flowfinity’s customers, a large CPG manufacturer, sells their products at 600,000 retail locations which they survey using Flowfinity mobile enterprise apps. This customer wanted to know how to efficiently synchronize the list of these locations in their Flowfinity database with the master list of retail stores that is managed in another system.
Maintaining and synchronizing reference lists like this is a typical problem that business application managers face every day. The “Import Merge” operation in Flowfinity performs an intelligent comparison of records that are being imported to the ones that are already present in the database and only applies the updates that are necessary.
Since only up to 1000 of the customer’s location records change every month, we decided to give it a try and set up a test environment to see how quickly their 150MB CSV data file could be processed to update the master retail store list in the Flowfinity database.
For the test we used a virtual machine with 4 CPU cores running both SQL Server Standard Edition and Flowfinity on the same machine. The system uses low voltage Intel CPUs which means each core is running at only 2.26GHz.
After the initial dataset was loaded into a clean database we pointed the “Import Merge” function at a file that had 100 records modified relative to the original version plus it had another 100 records added to the file to represent newly opened retail locations.
As expected, the result was that only the updated and added records were updated in the Flowfinity database. What surprised us was how quickly the system processed 150MB of CSV data to produce those results. It only took 2 minutes for the whole operation to complete.
This type of intelligent update for quick list synchronization with minimal load on the database is great for the server, and makes an even bigger impact on the smartphones and tablets running mobile app software.
Typically large reference lists are divided into geographical regions and then synchronized into offline storage on each user’s mobile device, based on the region that they belong to. Even though each device only stores the retail location list for their region, there could still be thousands of records on each device.
Flowfinity automatically updates the records on each device as soon as updates are applied to the server database. A straight upload of 600,000 records with 1000 connected smartphones and tablets would have resulted in massive record synchronization effort by the server. With “Import Merge”, however, only a few records need to be updated on each device when the retail location list is uploaded.
The difference is that Flowfinity’s device synchronization engine and “Import Merge” function are combined automatically and are triggered with a simple CSV file upload. All the complexities of efficiently and reliably updating records on 1000 devices are hidden from the system managers and the device users.
Build an code-free, enterprise-class mobile solution today.