9.8 C
Vancouver
Thursday, March 28, 2024

GAO Knocks CSA Evaluation Data

A provision in a Senate report requires the Government Accountability Office to periodically assess the Federal Motor Carrier Safety Administration’s implementation of its Compliance, Safety, Accountability program. This report examines the extent to which FMCSA has implemented CSA interventions, evaluated the effectiveness and efficiency of CSA interventions, and monitored progress toward achieving outcomes.

GAO reviewed FMCSA data and documentation on all eight CSA intervention types from fiscal years 2010–2015, including FMCSA’s strategic planning documents, guidance and program evaluations. GAO interviewed industry stakeholders and FMCSA officials in headquarters, in each of FMCSA’s service centers and in eight states selected for their participation in FMCSA’s CSA pilot test, location and program size, among other factors.

What GAO found

In July 2010, the Federal Motor Carrier Safety Administration chose to delay nationwide implementation of two of the eight interventions that used to address motor carrier safety concerns under its CSA program. This delay is linked to continuing delays in developing software needed to support the two interventions, offsite investigations and the use of cooperative safety plans. The software under development is intended to help FMCSA overcome some of the information challenges it faces due to its reliance on legacy information systems. FMCSA estimates that the software development project will be completed by April 2017.

FMCSA has conducted evaluations of the effectiveness and efficiency outcomes it established for the CSA program. However, GAO identified several limitations in FMCSA’s approaches that impact the usefulness of the evaluations, as described below.

Intervention effectiveness: FMCSA has developed a statistical model to annually evaluate the combined effectiveness of interventions. Although the model has some key strengths, such as accounting for a broad range of external factors, GAO identified a number of design and methodology limitations that reduce the usefulness of its results. For example, the model does not include an assessment of individual intervention types. Without this type of specific information, FMCSA is hampered in its ability to identify the circumstances under which different types of interventions are effective. Similarly, these types of limitations affect FMCSA’s ability to accurately draw conclusions about intervention effectiveness across all intervention types.

Intervention efficiency: To assess the efficiency of CSA interventions, FMCSA has relied on a study that it sponsored and that was published in 2011. This study estimated the average cost of conducting interventions in four states from October 2008 through May 2009. However, FMCSA has not taken steps to update its cost estimates for interventions since the 2011 evaluation, despite changes since that time in the resources needed to conduct CSA interventions, nor has it taken steps to develop additional information that is representative of the costs in other states. Without current cost estimates that are representative of all states, FMCSA cannot appropriately assess the efficiency of its interventions.

FMCSA has taken some actions to improve the effectiveness and efficiency of CSA interventions, but lacks measures to monitor progress. In April 2014, FMCSA established a working group to assess CSA interventions and make recommendations for improvement. As of April 2016, the group had made 20 recommendations, of which 12 had been implemented. However, GAO found that while FMCSA has established some performance measures for its effectiveness outcome that are appropriate, it has not established similar measures for its efficiency outcome. FMCSA headquarters officials told GAO that effectiveness and efficiency are complementary outcomes that FMCSA strives to balance. Without a complete set of measures for both outcomes, FMCSA lacks benchmarks needed to regularly measure progress to achieve these outcomes.

Recommendations

GAO recommends that FMCSA evaluate the effectiveness of individual intervention types, update cost estimates so that they are current and representative of all states, and establish complete performance measures.

The Department of Transportation concurred with all of GAO’s recommendations.