Review
Are videoconferenced mental and behavioral health services just as good as in-person? A meta-analysis of a fast-growing practice

https://doi.org/10.1016/j.cpr.2020.101944Get rights and content

Highlights

  • Being physically present with a client does not appear essential to generating therapeutic outcomes.

  • There are few meaningful differences in intervention or assessment outcomes across remote and in-person deliveries.

  • Medical settings may produce more favorable outcomes through the use of videoconferencing than other settings.

  • More rigorous research designs that focus on a wider range of client demographics and clinical concerns are lacking.

Abstract

The use of videoconferencing technologies (VCT) is on the rise given its potential to close the gap between mental health care need and availability. Yet, little is known about the effectiveness of these services compared to those delivered in-person. A series of meta-analyses were conducted using 57 empirical studies (43 examining intervention outcomes; 14 examining assessment reliability) published over the past two decades that included a variety of populations and clinical settings. Using conventional and HLM3 meta-analytical approaches, VCT consistently produced treatment effects that were largely equivalent to in-person delivered interventions across 281 individual outcomes and 4336 clients, with female clients and those treated in medical facilities tending to respond more favorably to VCT than in-person. Results of an HLM3 model suggested assessments conducted using VCT did not appear to lead to differential decisions compared to those conducted in-person across 83 individual outcomes and 332 clients/examinees. Although aggregate findings support the use of VCT as a viable alternative to in-person service delivery of mental healthcare, several limitations in the current literature base were revealed. Most concerning was the relatively limited number of randomized controlled trials and the inconsistent (and often incomplete) reporting of methodological features and results. Recommendations for reporting the findings of telemental health research are provided.

Introduction

Although various forms of remote and mobile services have begun to infiltrate the practice of psychology and psychiatry, the use of videoconferencing technology (VCT) has increased rapidly over the past decade (and exponentially in the past few months alone), with trends in the use of these systems expected to continue well into the new decade (Norcross, Pfund, & Prochaska, 2013). Specifically, VCT uses “real-time” audiovisual monitors/screens to connect agencies or clients in need of services to providers who can render such services (Ax et al., 2007). In fact, the November 2017 issue of The Monitor on Psychology listed the integration of technology into psychological practice as a top 10 trend in the field. Others have hailed remote technologies including VCT as the “key to solving mental healthcare access problems in the twenty-first century” (Frueh, 2015, p. 304) and a “modern answer to mental health” (Matthews, 2017).

The use of remote healthcare proliferated over a relatively brief period of time. For example, in 1991, there were only four telemedicine networks across the United States; merely five years later, there were approximately 160 (Miller, Clark, Veltkamp, Burton, & Swope, 2008). Among psychologists, the use of VCT increased from 2% in 2000 to 10% in 2008 (APA Psychology Health Service Provider Survey, 2008). In a more recent survey, nearly 40% of behavioral health providers in the U.S. reported using VCT as adjunctive to in-person services, and almost 45% had used VCT independently (Gershkovich, Herbert, Forman, & Glassman, 2016). The use of such technology is a popular response to rising health care costs and the need to increase access to care with qualified professionals and specialists, especially in rural communities (Nordal, 2015). While much of the research to date is based on U.S. practices and clientele, the fast-growing use of VCT can be observed internationally (see e.g., De Las Cuevas, Arredondo, Cabrera, Sulzenbacher, & Meise, 2006; Modai et al., 2006; Yoon et al., 2018). In particular, Australia (Australia's National Digital Health Strategy, 2018) and the United Kingdom (Digital Health & Care Scotland, 2018; United Kingdom National Health Service, 2019) have also been at the forefront of virtual mental health technologies. In Canada, one study estimated reductions associated with telepsychiatric services at an average of $50 per visit (O'Reilly et al., 2007).

In the U.S., the high demand for mental health services and increased acceptance of virtual modalities has resulted in legislative efforts to make it easier for telehealth technologies to reach across state lines. Established in 2014, the Interstate Medical Licensure Compact (IMLC) allows qualifying physicians to practice remotely in up to 29 states that are part of the agreement without obtaining licensure in those states. Following suit, the Psychology Interjurisdictional Compact (PSYPACT; Lerman, Kim, Ozinal, & Thompson, 2018) was introduced to similarly allow licensed psychologists in participating states to provide remote services for specific and limited purposes in other participating states. At the time of this publication, 12 states had signed onto PSYPACT and 12 others had pending PSYPACT legislation. Regardless, nearly all state professional boards have set some parameters for inter-jurisdictional remote mental and behavioral health practice (Lerman et al., 2018). In keeping up with the surgency of remote services, various medical and mental health organizations, including the American Psychological Association (2013), the American Telemedicine Association (2013), and the American Psychiatric Association (2018) have also established guidelines for the ethical and secure practice of mental health delivered over a distance. Given the current global health crisis, it would be remiss not to acknowledge the near overnight shift in practitioners turning to virtual services, frantically needing to acquire adequate training and infrastructure to connect with clients. Accompanying this shift have been expansions to Medicare coverage such that, for the duration of the COVID-19 emergency, real-time telehealth visits can be billed at the same rate as in-person visits (Centers for Medicare and Medicaid Services, 2020), as well as the passing of a congressional bill to establish home-based telemental health care (S. 3917, 2020) that make mental health services more accessible. Similar changes were also observed globally as other countries such as China (Liu et al., 2020), Australia (Zhou et al., 2020), and France (Ohannessian, Duong, & Odone, 2020) sought to expand and encourage the use of telemedicine during the COVID-19 emergency. This pandemic is likely to forever alter the way clinicians view and deliver mental healthcare.

VCT is not a new technology. In fact, elements of VCT were first introduced in medical settings across the U.S. for the purpose of diagnosis, patient care, and training over 30 years ago (Miller et al., 2008). As technology has progressed, an increasing number of studies have examined the efficacy of this modality compared to in-person intervention and assessment of mental and behavioral health concerns. The majority of existing studies have examined the success of VCT in the treatment of specific disorders, such as anxiety disorders, substance use, depression, and eating disorders (e.g., Benavides-Vaello, Strode, & Sheeran, 2013; Osenbach, O'Brien, Mishkind, & Smolenski, 2013; Rees & Maclaine, 2015; Sproch & Anderson, 2019). Other studies have focused on using VCT with special populations for whom the gap between service need and availability is especially wide or for specific types of services (e.g., neuropsychological assessment; see Brearly et al., 2017). The most common population for which VCT has been studied appears to be veterans (Gros, Yoder, Tuerk, Lozano, & Acierno, 2011; Luxton, Nelson, & Maheu, 2016), though several studies have also focused on forensic clients and inmates (see Batastini, McDonald, & Morgan, 2013 for a review). In general, current research on VCT-delivered mental and behavioral health services has produced promising results, supporting the idea that these services are largely as effective as in-person. These findings appear to be good news considering the growing expectation that remote services, and VCT in particular, will help combat the mental health care crisis that is projected to cost the U.S. nearly $750 billion dollars in 2021 by increasing access to affordable psychological and psychiatric care, reducing wait times, and offering better continuity of care (LaRock, 2019).

Despite the growing use of and support for VCT, along with other forms of remote mental health care (e.g., online support groups, mobile apps), there remains comparatively little empirical evidence on its effectiveness. Although most studies that examine video-based services are promising, the aggregate effects are not fully understood. To date, multiple meta-analyses have examined the overall efficacy of telemental health within specific populations and diagnostic presentations (e.g., Batastini & Morgan, 2016; Brearly et al., 2017; Larson, Rosen, & Wilson, 2018; Osenbach et al., 2013; Sloan, Gallagher, Feinstein, Lee, & Pruneau, 2011). Another looked specifically at objective psychiatric assessments, finding no differences in accuracy or satisfaction across modalities (Hyler, Gangure, & Batchelder, 2005). In fact, satisfaction and therapeutic alliance have been broadly examined in various countries, including Canada (Germain, Marchand, Bouchard, Guay, & Drouin, 2010), England (Manchanda & McLaren, 1998), Australia (Stubbings, 2012) and Scotland (Simpson, Bell, Knox, & Mitchell, 2005). The only known meta-analysis comparing therapy and assessment outcomes across VCT and in-person providers (Drago, Winding, & Antypa, 2016) focused strictly on psychiatric services. Importantly, given differing educational and training models of each discipline, psychological and psychiatric services can differ quite dramatically with regard to therapeutic orientation, the client-doctor relationship, duration of services, and structure of services, among other variables. Therefore, meta-analyses that do not include literature related to psychological approaches are limited in scope and do not fully capture the range or intensity of services that are now routinely provided through VCT. Other systematic reviews of the effectiveness of telemental health also exist that attempt to take a broader look at the practice (e.g., Hilty et al., 2013; Langarizadeh et al., 2017; Rees & Maclaine, 2015; Salmoiraghi & Hussain, 2015); however, these only provide descriptive statistics and more of a narrative overview rather than a statistically controlled comparison between traditional in-person and VCT services.

The current meta-analysis adds a more generalist perspective to the literature base that has not yet been captured empirically. That is, our primary aim was to answer the basic question of whether VCT is better, worse, or relatively equivalent to in-person mental health services. Taking a more comprehensive, aggregate approach is needed for several reasons. First, it allows for an additive, bigger picture interpretation; can we say VCT is equally effective in an overall sense and for specific clinical purposes or clients? Second, knowing the overall effects of VCT can serve as point of comparison to better contextualize effect sizes produced by narrowly focused meta-analyses; that is, how do aggregate effects for specific clinical purposes or clients measure up to those for VCT in general? Third, working with a larger collection of studies offers an opportunity for more robust comparisons of effects across different variables of interest than what can be performed within smaller-scale meta-analyses; for example, are outcomes for certain disorder categories stronger or weaker compared to others? The ability to examine these comparisons likewise provides a meaningful context within which to interpret findings from other analyses of specialized client groups. Fourth, and related to comparability, because this meta-analysis captures studies published before the dramatic shifts associated with COVID-19, it can serve as a historical baseline for future meta-analyses that are sure to follow from the inevitable spike in empirical studies that will emerge during and post-COVID-19. Finally, we view the present study as a wholistic effort to reveal what is out there and to what extent and, conversely, uncover the types of clinical services and populations that are less represented in the literature. While extant systematic reviews can help with this delineation, they lack accompanying statistical insights about the actual potential of VCT to engender relatively equivalent (or distinct) outcomes to in-person.

Several factors contribute to the uniqueness of the present set of meta-analyses. First, we applied comprehensive inclusionary criteria such that studies addressing a wide range of mental health disorders, client populations, interventions, and assessment types are represented. Second, we were concerned only with studies that compare VCT services to an in-person control; pre-post studies examining only VCT were excluded to clearly and directly answer the question of whether VCT produces outcomes that are comparable to in-person rather than whether VCT is an improvement over the absence of services. Further, in examining outcomes explicitly associated with therapeutic interventions, we apply multiple contemporary meta-analytic strategies and examine several moderator analyses that further explore what works and for whom. Overall, we hypothesized that meta-analytic results would support VCT as an equally effective treatment and assessment modality compared to in-person services by yielding small and/or non-significant effect size estimates when comparing relevant mental health outcomes by modality type.

Perhaps equally important to the meta-analytic findings themselves, the process of compiling empirical work products uncovered several limitations and flaws in the available literature that we argue must be discussed and addressed in future evaluation efforts. Following the presentation of our primary findings, we emphasize the need for more controlled treatment outcome and assessment reliability studies and offer initial reporting recommendations for future researchers. As will be detailed later, a large number of studies were excluded for lack of a comparison group, because they did not focus directly on service efficacy (e.g., instead asking about service satisfaction or acceptability), or because useful data was unknown or reported in a limited manner. While some guidelines exist regarding the empirical study of telemedicine practices (Krupinski & Bernard, 2014) and there are general manuscript reporting guidelines (e.g., Centers for Medicare and Medicaid Services, 2020), standardized recommendations for reporting findings specifically from telemental health research to scientific consumers, service providers, legislators, and other stakeholders is lacking. One available set of recommendations focuses exclusively on reporting VCT studies for depression interventions (Abel, Glover, Brandt, & Godleski, 2017). While many of these recommendations may apply to this line of research generally, they may also miss important factors or considerations highlighted by work in other domains. Ensuring the systematic and consistent reporting of such findings will not only deepen our understanding of VCT as an alternative service modality but will also increase our confidence in embracing VCT as the new norm in mental healthcare.

Section snippets

Study inclusion

To be included in this meta-analysis, studies had to (1) be published or available in English, (2) evaluate mental health-related outcomes (e.g., symptom reduction, hospitalizations) following a mental health service (i.e., psychiatric consultations, psychotherapy/counseling, assessment), (3) use a telecommunication service delivery system that transmitted live audio and visual information simultaneously, (4) use a between-groups comparison design, and (5) report sufficient information to allow

Intervention studies

All intervention outcome measures (for VCT and in-person) were converted to Hedges' g standardized effect sizes. Most of the intervention outcomes (n = 256 or 91.1%) were reported as means and corresponding standard deviations (SDs) or standard errors (SEs). These outcomes were converted to Hedges' g indices using the pooled SDs of the pre-measures, when available. The method of standardization based on pre-SDs is preferable to one based on the post-SDs measures because of a possible subject by

Study and sample characteristics

Forty-three individual studies published between the years of 2002 to 2019 met all inclusionary criteria and were included in the final analyses examining intervention outcomes. Seven studies were published in 2016; five in 2017; four in 2015 and between one and three studies were published in other years. A total of 281 individual outcomes were included across these 43 studies. Methodological features and sample characteristics of the analyzed intervention studies are presented in Table 1,

Discussion

Advances in technology have virtually transformed the way people communicate and interact with each other. It is not surprising, then, that the mental health sector is embracing video communication systems in an effort to keep up with existing social norms and take advantage of the opportunity to reach clients who may otherwise have limited options for local specialty providers, face increased costs and lost wages for travel to distant clinics, or go without services altogether. Beyond this,

Final conclusions

There is no question that remote delivery mechanisms are hitting a stride in mental and behavioral health service industries. But, are VCT services just as good as those delivered in-person? Thus far, the answer points to “yes,” as the available evidence suggests VCT does not grossly impede clinical outcomes in an overall general sense. Yet, there is more work to be done. Stronger research designs, greater inclusivity regarding client demographics and service settings, purposeful examination of

Role of funding sources

This project was supported by the Federal Office of Rural Health Policy (FORHP), Health resources and Services Administration (HRSA), U.S. Department of Health and Human Services (HHS) under cooperative agreement award number 6 U66RH31459–02-03. Dr. Paprzycki was also partially supported by the National Institute of General Medical Sciences of the National Institutes of Health under award number 5 U54 GM115428. The information, conclusions, and opinions expressed are those of the authors and no

Contributors

Ashley B. Batastini lead, supervised, and designed the study; to include development of the codebook, training and supervising coders, and overseeing reliability procedures and primary data analysis. Dr. Batastini also wrote significant portions of the manuscript and served as primary editor of all sections of the manuscript. Peter Paprzycki was the primary statistician on this project; he conducted the majority of statistical analyses and wrote all respective sections of the manuscript

Funding

This project was supported by the Federal Office of Rural Health Policy (FORHP), Health resources and Services Administration (HRSA), U.S. Department of Health and Human Services (HHS) under cooperative agreement award number 6 U66RH31459–02-03. Dr. Paprzycki was also partially supported by the National Institute of General Medical Sciences of the National Institutes of Health under award number 5 U54 GM115428. The information, conclusions, and opinions expressed are those of the authors and no

Declaration of Competing Interest

None.

Acknowledgements

We would like to thank the following individuals for their assistance with coding, data extraction, and manuscript preparation: Christopher M. King, Ph.D., J.D., Stephanie Van-Horn, Ph.D., Lauren Coaker, M.S., Michael Lester, M.S., Alexandra Repke, M.A., Alexandra Teller, M.A., Riley Davis, M.A., Nathan Ross, Ph.D., Ja'Nisha Robinson, M.S., Madison Pike, B.A., Kendall Klumpp, B.A., Abigail Armstrong, B.A., and Olivia Miller, M.S.

Ashley B. Batastini received her PhD in counseling psychology at Texas Tech University. She is currently assistant professor in the Department of Counseling, Educational Psychology, and Research at the University of Memphis. Dr. Batastini was affiliated with the University of Southern Mississippi when this work was completed. Her professional interests include development of and research on mental health interventions for criminal justice-involved populations and best practices in forensic

References (134)

  • T.W. Miller et al.

    Teleconferencing model for forensic consultation, court testimony, and continuing education

    Behavioral Sciences & the Law

    (2008)
  • J.E. Mitchell et al.

    A randomized trial comparing the efficacy of cognitive-behavioral therapy for bulimia nervosa delivered via telemedicine versus face-to-face

    Behaviour Research and Therapy

    (2008)
  • E.A. Abel et al.

    Recommendations for the reporting of telemental health (TMH) literature based on a systematic review of clinical video teleconferencing (CVT) and depression

    Journal of Technology in Behavioral Science

    (2017)
  • R. Acierno et al.

    Behavioral activation and therapeutic exposure for post traumatic exposure for posttraumatic stress disorder: A noninferiority trial of treatment delivered in person versus home-based telehealth

    Depression and Anxiety

    (2016)
  • American Psychiatric Association. (2018). Annual Meeting | AJMC. (n.d.). Retrieved February 25, 2020, from...
  • American Psychological Association

    Guidelines for psychological evaluations in child protection matters

    The American Psychologist

    (2013)
  • American Telemedicine Association

    Practice guidelines for video-based online mental Health services

    (2013)
  • D.J. Antonacci et al.

    Empirical evidence on the use and effectiveness of telepsychiatry via videoconferencing: Implications for forensic and correctional psychiatry

    Behavioral Sciences & the Law

    (2008)
  • M. Assink et al.

    Fitting three-level meta-analytic models in R: A step-by-step tutorial

    The Quantitative Methods for Psychology

    (2016)
  • Australia'’s National Digital Health Strategy

    Safe, seamless and secure: Australia's national digital health strategy

  • R.K. Ax et al.

    Innovations in correctional assessment and treatment

    Criminal Justice and Behavior

    (2007)
  • A. Backhaus et al.

    Videoconferencing psychotherapy: A systematic review

    Psychological Services

    (2012)
  • A.B. Batastini et al.

    Connecting the disconnected: Preliminary results and lessons learned from a telepsychology initiative with special management inmates

    Psychological Services

    (2016)
  • A.B. Batastini et al.

    Perceptions and use of videoconferencing in forensic mental health assessments: A survey of evaluators and legal personnel

    Psychology, Crime & Law

    (2019)
  • S. Benavides-Vaello et al.

    Using technology in the delivery of mental health and substance abuse treatment in rural communities: A review

    The Journal of Behavioral Health Services & Research

    (2013)
  • C.A. Blevins et al.

    The posttraumatic stress disorder checklist for DSM-5 (PCL-5): Development and initial psychometric evaluation

    Journal of Traumatic Stress

    (2015)
  • M. Borenstein

    Effect sizes for continuous data

  • M. Borenstein et al.

    Comprehensive meta-analysis (version 3.3.070) [computer software]

    (2014)
  • S. Bouchard et al.

    Delivering cognitive-behavior therapy for panic disorder with agoraphobia in videoconference

    Telemedicine Journal and E-Health

    (2004)
  • M.J. Bovin et al.

    Psychometric properties of the PTSD checklist for diagnostic and statistical manual of mental disorders-fifth edition (PCL-5) in veterans

    Psychological Assessment

    (2016)
  • T.W. Brearly et al.

    Neuropsychological test administration by videoconference: A systematic review and meta-analysis

    Neuropsychology Review

    (2017)
  • L.F. Campbell et al.

    A telepsychology casebook: Using technology ethically and effectively in your professional practice

    American Psychological Association.

    (2018)
  • L.E. Carlson et al.

    Telehealth-delivered group smoking cessation for rural and urban participantsL feasibility and cessation rates

    Addictive Behaviors

    (2011)
  • Centers for Medicare and Medicaid Services

    Medicare telemedicine health care provider fact sheet

  • J.E. Chapman et al.

    Comparing face-to-face and videoconference completion of the Montreal Cognitive Assessment (MoCA) in community-based survivors of stroke

    Journal of Telemedicine and Telecare

    (2019)
  • M. Cheung

    A guide to conducting a meta-analysis with non-independent effect sizes

    Neuropsychology Review

    (2019)
  • M.W. Cheung

    Meta-analysis: A structural equation modeling approach

    (2019)
  • M.W.L. Cheung

    Modeling dependent effect sizes with three-level meta-analyses: A structural equation modeling approach

    Psychological Methods

    (2014)
  • M.W.L. Cheung

    metaSEM: An R package for meta-analysis using structural equation modeling

    Frontiers in Psychology

    (2015)
  • N.G. Choi et al.

    Six-month postintervention depression and disability outcomes of in-home telehealth problem-solving therapy for depressed, low-income homebound older adults

    Depression and Anxiety

    (2014)
  • J.S. Comer et al.

    Internet-delivered, family-based treatment for early-onset OCD: A pilot randomized trial

    Journal of Consulting and Clinical Psychology

    (2017)
  • J.S. Comer et al.

    Remotely delivering real-time parent training to the home: An initial randomized trial of internet-delivered Parent-Child Interaction Therapy (I-PCIT)

    Journal of Consulting and Clinical Psychology

    (2017)
  • S.X. Day et al.

    Psychotherapy using distance technology: A comparison of face-to-face, video, and audio treatment

    Journal of Counseling Psychology

    (2002)
  • C. De la Cuevas et al.

    Randomized clinical trial of telepsychiatry through videoconference versus face-to-face conventional psychiatric treatment

    Telemedicine and e-health

    (2006)
  • M. Dekhtyar et al.

    Videoconference administration of the Western aphasia battery-revised: Feasibility and validity

    American Journal of Speech-Language Pathology

    (2019)
  • Digital Health & Care Scotland

    Scotland's digital health and care strategy: Enabling, connecting and empowering

  • R. Elford et al.

    A randomized, controlled trial of child psychiatric assessments conducted using videoconferencing

    Journal of Telemedicine and Telecare

    (2000)
  • D. Farabee et al.

    An experimental comparison of telepsychiatry and conventional psychiatry for parolees

    Psychiatric Services

    (2016)
  • J.C. Fortney et al.

    A budget impact analysis of telemedicine-based collaborative care for depression

    Medical Care

    (2011)
  • J.C. Fortney et al.

    A randomized trial of telemedicine-based collaborative care for depression

    Journal of General Internal Medicine

    (2007)
  • Cited by (75)

    View all citing articles on Scopus

    Ashley B. Batastini received her PhD in counseling psychology at Texas Tech University. She is currently assistant professor in the Department of Counseling, Educational Psychology, and Research at the University of Memphis. Dr. Batastini was affiliated with the University of Southern Mississippi when this work was completed. Her professional interests include development of and research on mental health interventions for criminal justice-involved populations and best practices in forensic mental health assessment.

    Peter Paprzycki received his Ph.D. in Research, Evaluation, Statistics and Measurement at the University of Toledo. He is a part-time Professor at the University of Toledo, and a statistical consultant at Mississippi Center for Clinical and Translational Research (MCCTR) and Grant Fundamentals, LCC. Dr. Paprzycki is a member of the Networking Urban Resources with Teachers and University to enRich Early Childhood Science (NURTURES), a National Science Foundation funded Mathematics and Science Partnership (MSP) program.

    Ashley C. T. Jones received her master's degree at Arizona State University. She is currently a doctoral student at the University of Southern Mississippi. Her professional interests include innovative technologies in correctional treatment, program evaluation, and research on jury decision-making.

    Nina MacLean received her PhD in counseling psychology at Texas Tech University. She is currently a consulting forensic examiner at the Center for Forensic Psychiatry. Her professional interests include professional issues and best practices in forensic mental health evaluations.

    1

    Dr. Batastini was primarily affiliated with the University of Southern Mississippi at the time this study was completed

    View full text