Highlights • Present a fundamental challenge in prescriptive analytics modeling regarding fair comparison of decision quality. • Propose three solutions that involve using sufficient historical data, constructing new test sets, and generating synthetic data to address the fundamental challenge. • Use four practical examples in freight transport to demonstrate the fundamental challenge and the solutions.

Abstract Prescriptive analytics, in which some parameters are predicted using statistical or machine learning models and then input into an optimization model, is often used to prescribe recommended solutions to freight transportation problems. The effectiveness of the optimal decision prescribed by prescriptive analytics is typically evaluated through a comparison with the results of the current decision model using predicted data. However, such comparisons are often flawed because of insufficient and uncertain data. We use four freight transport examples to illustrate this fundamental challenge in prescriptive analytics modeling. Furthermore, we propose three solutions to fully or partially overcome this challenge and fairly compare the optimal decisions generated by prescriptive analytics and the current approach. The three solutions involve using sufficient historical data, constructing new test sets, and generating synthetic data. We show how these solutions address the challenges in the four examples and are suitable for different problems considering data availability. The proposed solutions allow for a more comprehensive, accurate, and fair comparison of the optimal decisions to validate those generated by prescriptive analytics. This improves the effectiveness of the prescriptive analytics paradigm and can promote its application in freight transport and other disciplines.