The Issue of Trust
The use of models is fundamental to decision-making in Operational Research (OR), in application areas that range widely across all aspects of an organisation’s activities. For an OR practitioner, as an expert modeller working with a client, the epistemic basis for trusting the findings generated from models tends to come down to a question of model validation and verification. What this means in practice is that the client needs to be satisfied, to trust, that the conceptual representation of the problem is valid and that the computer model is a verified representation of the conceptual model.
Models based on techniques such as Discrete-Event Simulation (DES) are frequently used to make predictions about the future behaviour based on new assumptions and configurations. For example, answering questions such as ‘what would happen to performance if we were to modify the resources available and/or process flows?’ The development of simulation models, their parameterisation using data obtained during the study, and validation are difficult and time-consuming activities.
Despite the obvious benefits of modelling and simulation studies, evidence suggests that there is patchy uptake. The considerable technical complexity of conducting a simulation study adds conceptual distance between the expert modeller and the commissioners or owners of a study and make the problem of establishing trust harder. Of course, all practitioners want to see their studies translate into practical application; for example, delivering operational improvements in the client organisation or seeing policy recommendations implemented. Approaches do exist for reducing this conceptual distance with the aim of improving uptake. Drawing on Problem Structuring Methods (PSMs), hybrid techniques combine elements of a PSM with the simulation study in a multi-methodology to help bridge the conceptual divide. A good example of this approach is PartiSim, developed by Tako and Kotiadis (2015) for facilitated simulation modelling in healthcare.
In ‘Facets of Trust’, Harper, Mustafee and Yearworth (2021) looked at the practice issues facing the expert modeller when conducting a modelling and simulation study, particularly the intangible factors that mediate trust; such as interpersonal relationships, the credibility of the simulation practitioner, and facilitation skills, rather than the technical details of specific methodologies. Their literature review into this question surfaced the dynamic tensions[1] that operate in the space of the tripartite set of relationships that exist between the expert modeller, the simulation model, and the stakeholders.
This implies that the greater the risk to stakeholders arising from decisions informed by the modelling and simulation study the greater the level of trust is required in order to get the findings of the study adopted. Although this relationship is not (yet) an empirically tested proposition, it does resonate well with some of the meaning that has been associated with the term stakeholder as it has evolved over time (Reed, 2022). Literally, what is, or could be, at stake in the decision-making informed by the modelling and simulation study?
In Facets of Trust, the authors identify a number of concerns that exist in each of the three parts of the trust relationship that lead to the overall model. For the expert modeller/stakeholder relationship, practitioners are interested in questions of problem formulation, conceptual modelling, and methods selection. This is where PSMs come into their own as they focus on the process of problem formulation in the presence of diverse and conflicting worldviews on the problem at hand. Between the expert modeller and the model, practitioners not only have to address the more familiar questions about validation and verification, but also take care about documentation, reproducibility and the replicability of the study. And then between the stakeholder and the model, there are questions of model credibility and aspects of facilitation and social learning. On the issue of model credibility and the learning potential from ‘wrong models’ see recent work by Tsioptsias, Tako and Robinson (2022).
It is the expert modeller/stakeholder and stakeholder/model relationships that deserve more of our attention since the trust issues involved here are less commensurate with the particularly technical nature of Discrete-Event Simulation and the software packages that are used (e.g. Simul8, AnyLogic…). The notion of a calculus of trust and risk invites questions of measurement. If there is a trade-off between trust and risk, how much effort should be placed in building trust in order to affect the likelihood for the take-up of recommendations from the modelling and simulation study? How can trust be evaluated in order to calibrate this effort? Is there a level of trust that would transcend the purely technical considerations? This latter question has been explored by Tully, White and Yearworth (2019) in the context of attaching value to PSMs during the pre-contractual phase of a client/expert engagement i.e., this work posits the idea that trust can be established contractually such that questions of method and what the models are actually showing become secondary considerations in the relationship and therefore less important to making decisions about implementation[2].
Trust between an expert and a client or stakeholder group is clearly a broader question of interest to management and organisation scholars than just the specific concerns of OR practitioners attempting to get the findings from their modelling and simulation studies implemented; and the research methods of such scholarship are applicable to the questions posed here. Ormerod, Yearworth and White (2022) make the case that practice theories are an important place to look to gain further insight into how OR practitioners approach their work and could lead to a better understanding of the issues around trust that arise in the relationship between an expert modeller and their clients or stakeholders. At a practical level the Facets of Trust article suggests pertinent questions for a critical reflective practitioner to consider in order to improve the likelihood of adoption of recommendations from their studies.
References
Harper, A., Mustafee, N., & Yearworth, M. (2021). Facets of trust in simulation studies. European Journal of Operational Research, 289(1), 197-213. https://doi.org/10.1016/j.ejor.2020.06.043
Ormerod, R., Yearworth, M., & White, L. (2022). Understanding participant actions in OR interventions using practice theories: a research agenda. European Journal of Operational Research, 306(2), 810-827. https://doi.org/10.1016/j.ejor.2022.08.030
Reed, M. (2022). Should we banish the word “stakeholder”? https://www.fasttrackimpact.com/post/why-we-shouldn-t-banish-the-word-stakeholder
Tako, A., & Kotiadis, K. (2015). PartiSim : A multi-methodology framework to support facilitated simulation modelling in healthcare. European Journal of Operational Research, 244(2), 555-564. https://doi.org/10.1016/j.ejor.2015.01.046
Tsioptsias, N., Tako, A., & Robinson, S. (2022). Are “wrong” models useful? A qualitative study of discrete event simulation modeller stories. Journal of Simulation, 17(5), 594-606. https://doi.org/10.1080/17477778.2022.2108736
Tully, P., White, L., & Yearworth, M. (2019). The Value Paradox of Problem Structuring Methods. Systems Research and Behavioral Science, 36(4), 424-444. https://doi.org/10.1002/sres.2557
[1]A trust/risk calculus?
[2] What this work is really saying, between the lines, is that management consultancies can trade on the trust of their brand, not necessarily their methods. We’re not questioning their methods, just that they don’t figure in deciding what counts as success. If a leading consultancy said ‘do this’ on the basis of a simulation study, would the client have the nerve not to? Deciding to hire the consultancy is a bigger decision than implementing their recommendations; perhaps the only really difficult decision that needs to be made. In this context the trust relationship is doing some very heavy lifting.