Investigating Uncertainty in Postoperative Bleeding Management: Design Principles for Decision Support

Decision-making under uncertainty is a difficult and unavoidable challenge in clinical contexts. Technologies such as probabilistic programming languages (PPLs) allow their users to explicitly model and reason with uncertainty. By taking a user-centric approach to the deployment of these technologies, we believe there is an opportunity to involve clinicians in the modelling process. In this paper, we present a field study of decisions taken to manage postoperative bleeding. From analysis of the findings, we outline three central themes that emerge and discuss implications for design, developing a set of evaluative design principles to assess a PPL-based tool in this context. These include visualising zones of optimal intervention, surfacing relative risk trade-offs between teams, and accessing specialist views within a holistic picture. These findings provide a structure for critically exploring PPL-based tools to support clinical reasoning under uncertainty.

Content

Author and article information

Contributors

Diana Robinson

Luke Church

Alan F Blackwell

Alain Vuylsteke

Kenton O’Hara

Martin Besser

Conference

Publication date: July 2022

Publication date (Print): July 2022

Pages: 1-10

Affiliations

[0001]University of Cambridge

15 JJ Thomson Ave

Cambridge CB3 0FD

UK

[0002]Royal Papworth Hospital

Papworth Road, Trumpington

Cambridge CB2 0AY

UK

[0003]Microsoft Research

21 Station Road

Cambridge CB1 2FB

UK

[0004]Addenbrooke’s Hospital

Hills Road

Cambridge CB2 0QQ

UK

Article

DOI: 10.14236/ewic/HCI2022.25

SO-VID: b03726bb-0da8-499e-9376-7f25effb257a

License:

This work is licensed under a Creative Commons Attribution 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

Conference name: 35th International BCS Human-Computer Interaction Conference

Conference acronym: HCI2022

Conference number: 35

Conference location: Keele, Staffordshire

Conference date: July 11th to 13th, 2022

Conference sponsor: Electronic Workshops in Computing (eWiC)

Conference theme: Towards a Human-Centred Digital Society

History

Product

1477-9358 BCS Learning & Development

Self URI (article page): https://www.scienceopen.com/hosted-document?doi=10.14236/ewic/HCI2022.25

Self URI (journal page): https://ewic.bcs.org/

REFERENCES

Berg, M. (1992). The construction of medical disposals medical sociology and medical problem solving in clinical practice. Sociol. Health Illn. 14(2), 151–180.
Blackwell, A. F., T. Kohn, M. Erwig, A. G. B. Baydin, L. Church, J. Geddes, A. Gordon, M. Gorinova, B. Gram-Hansen, N. Lawrence, V. K. Mansinghka, B. Paige, T. Petriceck, D. Robinson, A. Sarkar, and O. Strickson (2019, August). Usability of probabilistic programming languages. In M. Marasoiu, L. Church, and L. Marshall (Eds.), PPIG 2019 - 30th Annual Workshop, pp. 53–68.
Borghouts, J., A. D. Gordon, A. Sarkar, and N. Toronto (2019). End-user probabilistic programming. In Quantitative Evaluation of Systems, Lecture notes in computer science, pp. 3–24. Cham: Springer International Publishing.
Cai, C. J., E. Reif, N. Hegde, J. Hipp, B. Kim, D. Smilkov, M. Wattenberg, F. Viegas, G. S. Corrado, M. C. Stumpe, and M. Terry (2019, May). Human-Centered tools for coping with imperfect algorithms during medical Decision-Making. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19).
Cai, C. J., S. Winter, D. Steiner, L. Wilcox, and M. Terry (2019, November). “hello AI”: Uncovering the onboarding needs of medical practitioners for Human-AI collaborative Decision-Making. Proc. ACM Hum.-Comput. Interact. 3(CSCW), 1–24.
Christensen, M. C., F. Dziewior, A. Kempel, and C. von Heymann (2012, February). Increased chest tube drainage is independently associated with adverse outcome after cardiac surgery. J. Cardiothorac. Vasc. Anesth. 26(1), 46–51.
Cooke, S. and J.-F. Lemay (2017, June). Transforming medical assessment: Integrating uncertainty into the evaluation of clinical reasoning in medical education. Acad. Med. 92(6), 746–751.
Dunlop, M. and R. M. Schwartzstein (2020, September). Reducing diagnostic error in the intensive care unit. engaging uncertainty when teaching clinical reasoning. ATS Sch 1(4), 364–371.
Garg, A. X., N. K. J. Adhikari, H. McDonald, M. P. Rosas-Arellano, P. J. Devereaux, J. Beyene, J. Sam, and R. B. Haynes (2005, March). Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 293(10), 1223–1238.
Geddes, J., O. Strickson, and T. Counsell. Nocell: Probabilistic programming for spreadsheet experts. https://www.turing.ac.uk/research/research_projects/nocell-probabilistic-programming-spreadsheet_experts. Accessed: 2021-9-7.
Ghahramani, Z. (2015, May). Probabilistic machine learning and artificial intelligence. Nature 521(7553), 452–459.
Goodwin, D. (2009, February). Acting in Anaesthesia: Ethnographic Encounters with Patients, Practitioners and Medical Technologies. Cambridge University Press.
Gorinova, M. I., A. Sarkar, A. F. Blackwell, and D. Syme (2016, May). A live, Multiple-Representation probabilistic programming environment for novices. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pp. 2533–2537. New York, NY, USA: Association for Computing Machinery.
Harish, V., F. Morgado, A. D. Stern, and S. Das (2021, January). Artificial intelligence and clinical decision making: The new nature of medical uncertainty. Acad. Med. 96(1), 31–36.
Hartswood, M., R. Procter, M. Rouncefield, R. Slack, J. Soutter, and A. Voss (2003). ’repairing’ the machine: A case study of the evaluation of Computer-Aided detection tools in breast screening. ECSCW 2003: Proceedings of the Eighth European Conference on Computer Supported Cooperative Work, 1–20.
Johnson, R., K. O’Hara, A. Sellen, C. Cousins, and A. Criminisi (2011, May). Exploring the potential for touchless interaction in image-guided interventional radiology. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’11, New York, NY, USA, pp. 3323–3332. Association for Computing Machinery.
Karkouti, K., D. N. Wijeysundera, T. M. Yau, W. S. Beattie, E. Abdelnaem, S. A. McCluskey, M. Ghannam, E. Yeo, G. Djaiani, and J. Karski (2004, October). The independent association of massive blood loss with mortality in cardiac surgery. Transfusion 44(10), 1453–1462.
Kawamoto, K., C. A. Houlihan, E. A. Balas, and D. F. Lobach (2005, April). Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 330(7494), 765.
Khajouei, R. and M. W. M. Jaspers (2010). The impact of CPOE medication systems’ design aspects on usability, workflow and medication orders: a systematic review. Methods Inf. Med. 49(1), 3–19.
Kl¨uber, S., F. Maas, D. Schraudt, G. Hermann, O. Happel, and T. Grundgeiger (2020, July). Experience matters: Design and evaluation of an anesthesia support tool guided by user experience theory. In Proceedings of the 2020 ACM Designing Interactive Systems Conference, DIS ’20, New York, NY, USA, pp. 1523–1535. Association for Computing Machinery.
Ledley, R. S. and L. B. Lusted (1959, October). Probability, logic and medical diagnosis. Science 130(3380), 892–930.
Mansinghka, V., R. Tibbetts, J. Baxter, P. Shafto, and B. Eaves (2015, December). BayesDB: A probabilistic programming system for querying the probable implications of data.
Martin, O. (2018, December). Bayesian Analysis with Python: Introduction to statistical modeling and probabilistic programming using PyMC3 and ArviZ, 2nd Edition. Packt Publishing Ltd.
Miles, M. B., A. Michael Huberman, and J. Saldana (2019, January). Qualitative Data Analysis: A Methods Sourcebook. SAGE Publications.
Morrison, C., K. Huckvale, B. Corish, J. Dorn, P. Kontschieder, K. O’Hara, Team, ASSESS MS, A. Criminisi, and A. Sellen (2016, July). Assessing multiple sclerosis with kinect: Designing computer vision systems for Real-World use. Human–Computer Interaction 31(3-4), 191–226.
Nardi, B. A. (1993). A Small Matter of Programming: Perspectives on End User Computing. MIT Press.
Petrou, A., P. Tzimas, and S. Siminelakis (2016, July). Massive bleeding in cardiac surgery. definitions, predictors and challenges. Hippokratia 20(3), 179–186.
Presanis, A., C. Jackson, D. De Angelis, P. Kirwan, A. Brown, A. Miltz, R. Harris, C. Chau, S. Migchelsen, H. Mohammed, K. Davison, S. Croxford, S. Desai, K. Lowndes, V. Delpech, and N. Gill (2019, August). Estimating the prevalence of HIV infection in england using bayesian evidence synthesis. StanCon 2019.
Ranucci, M., E. Baryshnikova, V. Pistuddi, L. Menicanti, A. Frigiola, and Surgical and Clinical Outcome REsearch (SCORE) Group (2017, February). The effectiveness of 10 years of interventions to control postoperative bleeding in adult cardiac surgery. Interact. Cardiovasc. Thorac. Surg. 24(2), 196–202.
Sakrejda, K. and E. Novik (2019, August). A Decision-Theoretic journey from early clinical data to late stage efficacy using hierarchical joint models. StanCon 2019.
Schaekermann, M., G. Beaton, E. Sanoubari, A. Lim, K. Larson, and E. Law (2020, April). Ambiguityaware AI assistants for medical data analysis. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–14. New York, NY, USA: Association for Computing Machinery.
Sendak, M., M. C. Elish, M. Gao, J. Futoma, W. Ratliff, M. Nichols, A. Bedoya, S. Balu, and C. O’Brien (2020). “the human body is a black box” supporting clinical decision-making with deep learning. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 99–109.
Sendak, M. P., W. Ratliff, D. Sarro, E. Alderton, J. Futoma, M. Gao, M. Nichols, M. Revoir, F. Yashar, C. Miller, K. Kester, S. Sandhu, K. Corey, N. Brajer, C. Tan, A. Lin, T. Brown, S. Engelbosch, K. Anstrom, M. C. Elish, K. Heller, R. Donohoe, J. Theiling, E. Poon, S. Balu, A. Bedoya, and C. O’Brien (2020, July). Real-World integration of a sepsis deep learning technology into routine clinical care: Implementation study. JMIR Med Inform 8(7), e15182.
Shiffman, R. N., Y. Liaw, C. A. Brandt, and G. J. Corb (1999, March). Computer-based guideline implementation systems: a systematic review of functionality and effectiveness. J. Am. Med. Inform. Assoc. 6(2), 104–114.
Simonsen, J. and T. Robertson (2012). Routledge international handbook of participatory design. Routledge.
Simpkin, A. L. and R. M. Schwartzstein (2016, November). Tolerating uncertainty - the next medical revolution? N. Engl. J. Med. 375(18), 1713–1715.
Swiers, R. (2019, August). Handling missing data, censored values and measurement error in machine learning models using multiple imputation for early stage drug discovery. StanCon 2019.
Taka, E., S. Stein, and J. H. Williamson (2020). Increasing interpretability of bayesian probabilistic programming models through interactive representations. Frontiers in Computer Science 2, 52.
von Heymann, C. and C. Boer (Eds.) (2019). Patient Blood Management in Cardiac Surgery. Springer, Cham.
Vuylsteke, A., C. Pagel, C. Gerrard, B. Reddy, S. Nashef, P. Aldam, and M. Utley (2011, June). The papworth bleeding risk score: a stratification scheme for identifying cardiac surgery patients at risk of excessive early postoperative bleeding. Eur. J. Cardiothorac. Surg. 39(6), 924–930.
Wang, T. D., C. Plaisant, A. J. Quinn, R. Stanchak, S. Murphy, and B. Shneiderman (2008, April). Aligning temporal data by sentinel events: discovering patterns in electronic health records. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’08, New York, NY, USA, pp. 457–466. Association for Computing Machinery.
Yang, Q., A. Steinfeld, and J. Zimmerman (2019, May). Unremarkable AI: Fitting intelligent decision support into critical, clinical Decision-Making processes. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Number Paper 238 in CHI ’19, New York, NY, USA, pp. 1–11. Association for Computing Machinery.
Yang, Q., J. Zimmerman, and A. Steinfeld. Review of medical decision support tools: Emerging opportunity for interaction design. https://www.researchgate.net/profile/Qian_Yang29/publication/282121061_Review_of_Medical_Decision_Support_Tools_Emerging_Opportunity_for_Interaction_Design/links/56035cf608ae08d4f1716009.pdf. Accessed: 2020-9-2.
Yang, Q., J. Zimmerman, A. Steinfeld, L. Carey, and J. F. Antaki (2016, May). Investigating the heart pump implant decision process: Opportunities for decision support tools to help. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI ’16, New York, NY, USA, pp. 4477–4488. Association for Computing Machinery.

Comments

Comment on this article

[1] Berg, M. (1992). The construction of medical disposals medical sociology and medical problem solving in clinical practice. Sociol. Health Illn. 14(2), 151–180.

[2] Blackwell, A. F., T. Kohn, M. Erwig, A. G. B. Baydin, L. Church, J. Geddes, A. Gordon, M. Gorinova, B. Gram-Hansen, N. Lawrence, V. K. Mansinghka, B. Paige, T. Petriceck, D. Robinson, A. Sarkar, and O. Strickson (2019, August). Usability of probabilistic programming languages. In M. Marasoiu, L. Church, and L. Marshall (Eds.), PPIG 2019 - 30th Annual Workshop, pp. 53–68.

[3] Borghouts, J., A. D. Gordon, A. Sarkar, and N. Toronto (2019). End-user probabilistic programming. In Quantitative Evaluation of Systems, Lecture notes in computer science, pp. 3–24. Cham: Springer International Publishing.

[4] Cai, C. J., E. Reif, N. Hegde, J. Hipp, B. Kim, D. Smilkov, M. Wattenberg, F. Viegas, G. S. Corrado, M. C. Stumpe, and M. Terry (2019, May). Human-Centered tools for coping with imperfect algorithms during medical Decision-Making. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19).

[5] Cai, C. J., S. Winter, D. Steiner, L. Wilcox, and M. Terry (2019, November). “hello AI”: Uncovering the onboarding needs of medical practitioners for Human-AI collaborative Decision-Making. Proc. ACM Hum.-Comput. Interact. 3(CSCW), 1–24.

[6] Christensen, M. C., F. Dziewior, A. Kempel, and C. von Heymann (2012, February). Increased chest tube drainage is independently associated with adverse outcome after cardiac surgery. J. Cardiothorac. Vasc. Anesth. 26(1), 46–51.

[7] Cooke, S. and J.-F. Lemay (2017, June). Transforming medical assessment: Integrating uncertainty into the evaluation of clinical reasoning in medical education. Acad. Med. 92(6), 746–751.

[8] Dunlop, M. and R. M. Schwartzstein (2020, September). Reducing diagnostic error in the intensive care unit. engaging uncertainty when teaching clinical reasoning. ATS Sch 1(4), 364–371.

[9] Garg, A. X., N. K. J. Adhikari, H. McDonald, M. P. Rosas-Arellano, P. J. Devereaux, J. Beyene, J. Sam, and R. B. Haynes (2005, March). Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 293(10), 1223–1238.

[10] Geddes, J., O. Strickson, and T. Counsell. Nocell: Probabilistic programming for spreadsheet experts. https://www.turing.ac.uk/research/research_projects/nocell-probabilistic-programming-spreadsheet_experts. Accessed: 2021-9-7.

[11] Ghahramani, Z. (2015, May). Probabilistic machine learning and artificial intelligence. Nature 521(7553), 452–459.

[12] Goodwin, D. (2009, February). Acting in Anaesthesia: Ethnographic Encounters with Patients, Practitioners and Medical Technologies. Cambridge University Press.

[13] Gorinova, M. I., A. Sarkar, A. F. Blackwell, and D. Syme (2016, May). A live, Multiple-Representation probabilistic programming environment for novices. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pp. 2533–2537. New York, NY, USA: Association for Computing Machinery.

[14] Harish, V., F. Morgado, A. D. Stern, and S. Das (2021, January). Artificial intelligence and clinical decision making: The new nature of medical uncertainty. Acad. Med. 96(1), 31–36.

[15] Hartswood, M., R. Procter, M. Rouncefield, R. Slack, J. Soutter, and A. Voss (2003). ’repairing’ the machine: A case study of the evaluation of Computer-Aided detection tools in breast screening. ECSCW 2003: Proceedings of the Eighth European Conference on Computer Supported Cooperative Work, 1–20.

[16] Johnson, R., K. O’Hara, A. Sellen, C. Cousins, and A. Criminisi (2011, May). Exploring the potential for touchless interaction in image-guided interventional radiology. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’11, New York, NY, USA, pp. 3323–3332. Association for Computing Machinery.

[17] Karkouti, K., D. N. Wijeysundera, T. M. Yau, W. S. Beattie, E. Abdelnaem, S. A. McCluskey, M. Ghannam, E. Yeo, G. Djaiani, and J. Karski (2004, October). The independent association of massive blood loss with mortality in cardiac surgery. Transfusion 44(10), 1453–1462.

[18] Kawamoto, K., C. A. Houlihan, E. A. Balas, and D. F. Lobach (2005, April). Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 330(7494), 765.

[19] Khajouei, R. and M. W. M. Jaspers (2010). The impact of CPOE medication systems’ design aspects on usability, workflow and medication orders: a systematic review. Methods Inf. Med. 49(1), 3–19.

[20] Kl¨uber, S., F. Maas, D. Schraudt, G. Hermann, O. Happel, and T. Grundgeiger (2020, July). Experience matters: Design and evaluation of an anesthesia support tool guided by user experience theory. In Proceedings of the 2020 ACM Designing Interactive Systems Conference, DIS ’20, New York, NY, USA, pp. 1523–1535. Association for Computing Machinery.

[21] Ledley, R. S. and L. B. Lusted (1959, October). Probability, logic and medical diagnosis. Science 130(3380), 892–930.

[22] Mansinghka, V., R. Tibbetts, J. Baxter, P. Shafto, and B. Eaves (2015, December). BayesDB: A probabilistic programming system for querying the probable implications of data.

[23] Martin, O. (2018, December). Bayesian Analysis with Python: Introduction to statistical modeling and probabilistic programming using PyMC3 and ArviZ, 2nd Edition. Packt Publishing Ltd.

[24] Miles, M. B., A. Michael Huberman, and J. Saldana (2019, January). Qualitative Data Analysis: A Methods Sourcebook. SAGE Publications.

[25] Morrison, C., K. Huckvale, B. Corish, J. Dorn, P. Kontschieder, K. O’Hara, Team, ASSESS MS, A. Criminisi, and A. Sellen (2016, July). Assessing multiple sclerosis with kinect: Designing computer vision systems for Real-World use. Human–Computer Interaction 31(3-4), 191–226.

[26] Nardi, B. A. (1993). A Small Matter of Programming: Perspectives on End User Computing. MIT Press.

[27] Petrou, A., P. Tzimas, and S. Siminelakis (2016, July). Massive bleeding in cardiac surgery. definitions, predictors and challenges. Hippokratia 20(3), 179–186.

[28] Presanis, A., C. Jackson, D. De Angelis, P. Kirwan, A. Brown, A. Miltz, R. Harris, C. Chau, S. Migchelsen, H. Mohammed, K. Davison, S. Croxford, S. Desai, K. Lowndes, V. Delpech, and N. Gill (2019, August). Estimating the prevalence of HIV infection in england using bayesian evidence synthesis. StanCon 2019.

[29] Ranucci, M., E. Baryshnikova, V. Pistuddi, L. Menicanti, A. Frigiola, and Surgical and Clinical Outcome REsearch (SCORE) Group (2017, February). The effectiveness of 10 years of interventions to control postoperative bleeding in adult cardiac surgery. Interact. Cardiovasc. Thorac. Surg. 24(2), 196–202.

[30] Sakrejda, K. and E. Novik (2019, August). A Decision-Theoretic journey from early clinical data to late stage efficacy using hierarchical joint models. StanCon 2019.

[31] Schaekermann, M., G. Beaton, E. Sanoubari, A. Lim, K. Larson, and E. Law (2020, April). Ambiguityaware AI assistants for medical data analysis. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–14. New York, NY, USA: Association for Computing Machinery.

[32] Sendak, M., M. C. Elish, M. Gao, J. Futoma, W. Ratliff, M. Nichols, A. Bedoya, S. Balu, and C. O’Brien (2020). “the human body is a black box” supporting clinical decision-making with deep learning. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 99–109.

[33] Sendak, M. P., W. Ratliff, D. Sarro, E. Alderton, J. Futoma, M. Gao, M. Nichols, M. Revoir, F. Yashar, C. Miller, K. Kester, S. Sandhu, K. Corey, N. Brajer, C. Tan, A. Lin, T. Brown, S. Engelbosch, K. Anstrom, M. C. Elish, K. Heller, R. Donohoe, J. Theiling, E. Poon, S. Balu, A. Bedoya, and C. O’Brien (2020, July). Real-World integration of a sepsis deep learning technology into routine clinical care: Implementation study. JMIR Med Inform 8(7), e15182.

[34] Shiffman, R. N., Y. Liaw, C. A. Brandt, and G. J. Corb (1999, March). Computer-based guideline implementation systems: a systematic review of functionality and effectiveness. J. Am. Med. Inform. Assoc. 6(2), 104–114.

[35] Simonsen, J. and T. Robertson (2012). Routledge international handbook of participatory design. Routledge.

[36] Simpkin, A. L. and R. M. Schwartzstein (2016, November). Tolerating uncertainty - the next medical revolution? N. Engl. J. Med. 375(18), 1713–1715.

[37] Swiers, R. (2019, August). Handling missing data, censored values and measurement error in machine learning models using multiple imputation for early stage drug discovery. StanCon 2019.

[38] Taka, E., S. Stein, and J. H. Williamson (2020). Increasing interpretability of bayesian probabilistic programming models through interactive representations. Frontiers in Computer Science 2, 52.

[39] von Heymann, C. and C. Boer (Eds.) (2019). Patient Blood Management in Cardiac Surgery. Springer, Cham.

[40] Vuylsteke, A., C. Pagel, C. Gerrard, B. Reddy, S. Nashef, P. Aldam, and M. Utley (2011, June). The papworth bleeding risk score: a stratification scheme for identifying cardiac surgery patients at risk of excessive early postoperative bleeding. Eur. J. Cardiothorac. Surg. 39(6), 924–930.

[41] Wang, T. D., C. Plaisant, A. J. Quinn, R. Stanchak, S. Murphy, and B. Shneiderman (2008, April). Aligning temporal data by sentinel events: discovering patterns in electronic health records. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’08, New York, NY, USA, pp. 457–466. Association for Computing Machinery.

[42] Yang, Q., A. Steinfeld, and J. Zimmerman (2019, May). Unremarkable AI: Fitting intelligent decision support into critical, clinical Decision-Making processes. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Number Paper 238 in CHI ’19, New York, NY, USA, pp. 1–11. Association for Computing Machinery.

[43] Yang, Q., J. Zimmerman, and A. Steinfeld. Review of medical decision support tools: Emerging opportunity for interaction design. https://www.researchgate.net/profile/Qian_Yang29/publication/282121061_Review_of_Medical_Decision_Support_Tools_Emerging_Opportunity_for_Interaction_Design/links/56035cf608ae08d4f1716009.pdf. Accessed: 2020-9-2.

[44] Yang, Q., J. Zimmerman, A. Steinfeld, L. Carey, and J. F. Antaki (2016, May). Investigating the heart pump implant decision process: Opportunities for decision support tools to help. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI ’16, New York, NY, USA, pp. 4477–4488. Association for Computing Machinery.

Celebrating 65 years of The Computer Journal - free-to-read perspectives - bcs.org/tcj65