OOIDA: CSA needs more than a mathematical overhaul

By Jami Jones, Land Line managing editor | 9/28/2017

No matter what changes are made to the Compliance, Safety, Accountability safety measurement program, OOIDA says that the end goal of reducing crashes has not been achieved and must take priority going forward.

The National Academies of Science completed a congressionally mandated study of the safety measurement system and released the results earlier this year. In short, the study concluded that the safety measurement program is defendable on paper but falls short on execution.

The study by the National Academies of Science validates a number of concerns and criticisms of the program articulated by the Owner-Operator Independent Drivers Association.

The study was prompted by Congress in the Fixing America’s Surface Transportation Act, or FAST Act. The highway bill, signed into law in 2015, recommended that the Federal Motor Carrier Safety Administration fund a study by the National Academies of Sciences, Engineering and Medicine. The purpose of the study was to evaluate the Safety Measurement System.

The SMS serves as the backbone of the Compliance, Safety Accountability, or CSA for short, safety measurement program. The FMCSA uses the CSA program in an attempt to predict which motor carriers are more likely to be involved in crashes. As part of the program, motor carriers are ranked in six Behavior Analysis and Safety Improvement Categories, which are categories of similar regulations. Motor carriers with high rankings are considered at risk and are subject to enforcement interventions by the agency.

Following release of the study’s results in June, FMCSA initiated a public comment period to get input from industry stakeholders and the public on the study’s recommendation.

OOIDA filed its comments on Wednesday, Sept. 27.

“Most of the discussion concerning CSA has centered primarily on the accuracy of the data and the methodology behind the safety ratings. While these factors are important, OOIDA fears that they have distracted the agency and others from properly evaluating the actual performance and effectiveness of CSA,” OOIDA’s comments state.

“Since the inception of the CSA and SMS programs in 2010, we have seen a steady uptick in truck-related crashes, injuries and fatalities of 39 percent, opposed to a 27 percent decrease in total crashes between 2005 and 2009.”

OOIDA says that FMCSA officials have tried to explain that increase in crashes away by saying more trucks were on the road and traveling more miles.

“What they fail to mention is that the number of vehicle miles traveled (VMT) prior to CSA was 2.5 percent higher, and that the number of registered trucks and buses was also greater, yet with a much lower fatality and injury rate per 100 million VMT,” OOIDA states.

The Association is concerned that the National Academies of Science review of the program neglected crash rates and focused too specifically on the inner workings of the program.

“The reality of increased fatalities and injuries far outweigh statistical algorithms and ‘scientific’ predictions. OOIDA is hopeful that FMCSA, by working with industry stakeholders, can find a way to implement the NAS recommendations in a manner that will improve safety and reduce fatalities and crashes,” OOIDA comments state.

The math
Getting deep into the mathematical and data weeds of the CSA program, the National Academies Study ultimately recommended a new algorithm.

As it stands, the program attempts to draw linear relationships between violations and crashes, even though FMCSA is unable to prove these relationships. One major problem statisticians face with the CSA program is that ultimately crashes are rare in the grand scheme of things. That makes finding a correlation between violations and the actual causation of crashes difficult.

To counter this problem, the National Academies of Science recommends an Item Response Theory model for the CSA program.

While the model can be complicated, in plain terms this form of predictive mathematics can account for factors such as congestion, demographics, roadway type, seasonal impacts, weather, etc.

It could also address one of the chief criticisms about CSA, inconsistencies in enforcement around the country, by improving data quality so it is uniform and standardized.

“While it is critical that the IRT model proportionately represents the diversity of the trucking industry rather than focusing on large motor carriers, OOIDA is concerned with the concept put forward by the NAS that the model could generate a safety score for a carrier with only one violation. OOIDA fears that this could even increase the inaccuracy of SMS,” according to OOIDA comments.

To that end, OOIDA appeals to FMCSA officials to understand fully the nuances of the IRT model before implementing it. The Association went so far in acknowledging the complexities of the model to encourage FMCSA to form a standing committee with the National Academies of Science to oversee and provide advice.

The data
Data quality, consistency and standardization have plagued the CSA program since almost the beginning.

OOIDA strongly supports the adoption of Model Minimum Uniform Crash Criteria. 

“Without consistent and standardized data, the IRT model will be plagued with the same lack of data integrity that the current CSA and SMS programs suffer with now,” according to the OOIDA.

The hurdle the agency faces on data is not just in information provided by the states but also from motor carriers themselves. OOIDA supports the NAS suggestion that the agency should collaborate with states to gain access to mileage reported via the International Fuel Tax Agreement and the International Registration Plan.

To spur voluntary reporting to the program, OOIDA also suggests that FMCSA, along with the National Highway Traffic Safety Administration and the Federal Highway Administration, should encourage states to increase compliance with an adopted Model Minimum Uniform Crash Criteria through grants, partnerships and guidance.

In short, tying some funding to participation would increase participation, OOIDA contends.

Finally, OOIDA went one step beyond conventional data currently used in the CSA program. The Association advocated using driver turnover, driver pay and driver experience as metrics the system could use to determine the “safety culture” of motor carriers.

The Association referred to both a J.B. Hunt study in 1995 and research by Michael Belzer in 2002 that point to driver-related factors having an effect on company’s crash rates.

Keep it in house
While CSA has had its problems, motor carriers faced adverse public reaction because the scores were publicly available. They have since been pulled from public view, and OOIDA would like to see it stay that way.

“These ratings fail to give accurate and adequate information for the purpose of evaluating a carrier’s safety program. The perception of shippers and brokers based upon the present data that has been shown to be inadequate and misleading is unconscionable,” according to OOIDA.

Copyright © OOIDA