Read Martin Lew's Article in the NCTA Coal Transporter: Leveraging Big Data Within The Rail Industry

Technology Junction (Transparent)

It is rare to attend a transportation conference or seminar these days without hearing catchy tech buzzwords such as “Internet of Things (IoT),” “Artificial Intelligence (AI),” “Autonomous Vehicles,” and/or “Big Data.” Are these just trendy technology concepts, or is there real substance that will have a material impact on the transportation and supply chain industries? Each of these technologies has one common denominator: The underlying use of data that drives their respective functionalities. We are now entering the next phase of our modern industrialized society which many experts refer to as a “data-driven economy.” Information will be transformative to social and business interactions in ways that we are only seeing mere glimmers of today.

Big Data

Big Data is a term that many use, but few truly understand. There is no question that technology touches all of our lives in ways that we do not realize, and it is important to understand the basic definition of Big Data. Put simply, Big Data is larger, more complex data sets from sources that are so voluminous and dynamic that traditional data processing software is not able to manage it. All data has intrinsic value, but it is not useful until that value is discovered and applied. By developing ways to better understand this data, we will gain greater insight into real-time operational intelligence that leverages event-based information in order to increase the reliability, safety, and efficiency of the problem that is trying to be solved.

So, how can Big Data be valuable to the rail industry? There is a wide range of issues that Big Data can address, but there are two primary areas where it can be most useful: Predictive maintenance, and predictive operations and analytics.

Predictive Maintenance

In an industry where fleet reliability is critical for increasing efficiency and reducing costs, predictive maintenance represents the greatest opportunity for reducing mechanical failures and unplanned repairs. With rail fleets operating for 35-50 years and maintenance costs accounting for 40-50% of overall costs, the main objective of rail operators is to efficiently increase fleet reliability. For components that are highly regulated, such as those that can lead to train failure (e.g. brakes), planned preventative maintenance is conducted based on time or usage. For many of the other components, unplanned reactive repair (e.g. fix when broken) is the maintenance strategy. However, both of these strategies have limitations and cannot prevent certain failures from occurring.

In order for the rail industry to increase safety and become more operationally efficient, it must transition from time-based and reactive maintenance schedules to predictive-based maintenance. This will ensure that locomotives and railcars are only taken out of service for maintenance when necessary. Mathematical models, in conjunction with sensor data, are used to predict when parts are likely to fail so they can be replaced before unplanned downtime occurs. Sensor data can provide a real-time overview of the rail fleet, monitoring the condition of parts at all times. Predictive-based maintenance alone can lead to an overall reduction of at least 10-15% in maintenance costs. Additionally, predictive maintenance alerts operators before components fail by using multi-faceted data analysis enabled by sensors to create machine-learning algorithms that raise red flags. Sensor technology and data analytics are changing the maintenance paradigm from time and usage-based maintenance to predictive maintenance driven by the use of Big Data.

Predictive Operations & Analytics

The next best thing to foreseeing the future is knowing what trends are occurring and adjusting operations to optimize these trends. This concept is known as predictive operations and analytics, which provides organizations with the tools necessary to make reasonable data-based decisions. Until now, shippers have had no choice but to rely on historical data to create these forecasts. However, with advanced technologies such as Big Data, shippers are now provided with real-time insight that allows for better allocation of resources and optimization across the entire supply chain. For example, one of the major keys to on-time delivery and optimal load factors is crew and equipment routing. Railroads are no different than trucking companies and airlines which need to understand the implications of changing weather conditions, missed connections, crew work time-limits, and maintenance schedules. Big Data is changing the way rail logistics and operations are being managed.

Big Data Challenges

While Big Data has tremendous promise, it is not without challenges. One of the major hurdles? Big Data is BIG. Even with new technologies being developed for data management and storage, data volumes are doubling in size every two years. Organizations are struggling to keep up with the sheer size of their data, and they are constantly trying to find ways to securely and cost-effectively store their data. But storing data is just the beginning -- companies must also figure out how to organize and manage their data in ways that allow for meaningful analysis. Data scientists spend 50 to 80 percent of their time curating and preparing data before it can actually be used. Keeping up with the advances in Big Data technology is an ongoing challenge for the application of Big Data.


In order for rail to remain a leader in the transportation industry, it must continue to adopt technological innovation. Despite its challenges, the application of Big Data across rail networks and supply chains is crucial for future growth and efficiency. When thinking about the world’s largest tech companies, most of their value comes from data. They are constantly analyzing data to increase efficiency and develop better solutions for their customers. The rail industry should not be any different. In order to create a safer and more operationally efficient mode of transportation, it is incumbent upon the rail industry to leverage the vast amount of data that is collected and analyzed. Implementing procedures that utilize Big Data will be the key to remaining competitive in the data-driven economy of the future.