ReviewAre videoconferenced mental and behavioral health services just as good as in-person? A meta-analysis of a fast-growing practice
Introduction
Although various forms of remote and mobile services have begun to infiltrate the practice of psychology and psychiatry, the use of videoconferencing technology (VCT) has increased rapidly over the past decade (and exponentially in the past few months alone), with trends in the use of these systems expected to continue well into the new decade (Norcross, Pfund, & Prochaska, 2013). Specifically, VCT uses “real-time” audiovisual monitors/screens to connect agencies or clients in need of services to providers who can render such services (Ax et al., 2007). In fact, the November 2017 issue of The Monitor on Psychology listed the integration of technology into psychological practice as a top 10 trend in the field. Others have hailed remote technologies including VCT as the “key to solving mental healthcare access problems in the twenty-first century” (Frueh, 2015, p. 304) and a “modern answer to mental health” (Matthews, 2017).
The use of remote healthcare proliferated over a relatively brief period of time. For example, in 1991, there were only four telemedicine networks across the United States; merely five years later, there were approximately 160 (Miller, Clark, Veltkamp, Burton, & Swope, 2008). Among psychologists, the use of VCT increased from 2% in 2000 to 10% in 2008 (APA Psychology Health Service Provider Survey, 2008). In a more recent survey, nearly 40% of behavioral health providers in the U.S. reported using VCT as adjunctive to in-person services, and almost 45% had used VCT independently (Gershkovich, Herbert, Forman, & Glassman, 2016). The use of such technology is a popular response to rising health care costs and the need to increase access to care with qualified professionals and specialists, especially in rural communities (Nordal, 2015). While much of the research to date is based on U.S. practices and clientele, the fast-growing use of VCT can be observed internationally (see e.g., De Las Cuevas, Arredondo, Cabrera, Sulzenbacher, & Meise, 2006; Modai et al., 2006; Yoon et al., 2018). In particular, Australia (Australia's National Digital Health Strategy, 2018) and the United Kingdom (Digital Health & Care Scotland, 2018; United Kingdom National Health Service, 2019) have also been at the forefront of virtual mental health technologies. In Canada, one study estimated reductions associated with telepsychiatric services at an average of $50 per visit (O'Reilly et al., 2007).
In the U.S., the high demand for mental health services and increased acceptance of virtual modalities has resulted in legislative efforts to make it easier for telehealth technologies to reach across state lines. Established in 2014, the Interstate Medical Licensure Compact (IMLC) allows qualifying physicians to practice remotely in up to 29 states that are part of the agreement without obtaining licensure in those states. Following suit, the Psychology Interjurisdictional Compact (PSYPACT; Lerman, Kim, Ozinal, & Thompson, 2018) was introduced to similarly allow licensed psychologists in participating states to provide remote services for specific and limited purposes in other participating states. At the time of this publication, 12 states had signed onto PSYPACT and 12 others had pending PSYPACT legislation. Regardless, nearly all state professional boards have set some parameters for inter-jurisdictional remote mental and behavioral health practice (Lerman et al., 2018). In keeping up with the surgency of remote services, various medical and mental health organizations, including the American Psychological Association (2013), the American Telemedicine Association (2013), and the American Psychiatric Association (2018) have also established guidelines for the ethical and secure practice of mental health delivered over a distance. Given the current global health crisis, it would be remiss not to acknowledge the near overnight shift in practitioners turning to virtual services, frantically needing to acquire adequate training and infrastructure to connect with clients. Accompanying this shift have been expansions to Medicare coverage such that, for the duration of the COVID-19 emergency, real-time telehealth visits can be billed at the same rate as in-person visits (Centers for Medicare and Medicaid Services, 2020), as well as the passing of a congressional bill to establish home-based telemental health care (S. 3917, 2020) that make mental health services more accessible. Similar changes were also observed globally as other countries such as China (Liu et al., 2020), Australia (Zhou et al., 2020), and France (Ohannessian, Duong, & Odone, 2020) sought to expand and encourage the use of telemedicine during the COVID-19 emergency. This pandemic is likely to forever alter the way clinicians view and deliver mental healthcare.
VCT is not a new technology. In fact, elements of VCT were first introduced in medical settings across the U.S. for the purpose of diagnosis, patient care, and training over 30 years ago (Miller et al., 2008). As technology has progressed, an increasing number of studies have examined the efficacy of this modality compared to in-person intervention and assessment of mental and behavioral health concerns. The majority of existing studies have examined the success of VCT in the treatment of specific disorders, such as anxiety disorders, substance use, depression, and eating disorders (e.g., Benavides-Vaello, Strode, & Sheeran, 2013; Osenbach, O'Brien, Mishkind, & Smolenski, 2013; Rees & Maclaine, 2015; Sproch & Anderson, 2019). Other studies have focused on using VCT with special populations for whom the gap between service need and availability is especially wide or for specific types of services (e.g., neuropsychological assessment; see Brearly et al., 2017). The most common population for which VCT has been studied appears to be veterans (Gros, Yoder, Tuerk, Lozano, & Acierno, 2011; Luxton, Nelson, & Maheu, 2016), though several studies have also focused on forensic clients and inmates (see Batastini, McDonald, & Morgan, 2013 for a review). In general, current research on VCT-delivered mental and behavioral health services has produced promising results, supporting the idea that these services are largely as effective as in-person. These findings appear to be good news considering the growing expectation that remote services, and VCT in particular, will help combat the mental health care crisis that is projected to cost the U.S. nearly $750 billion dollars in 2021 by increasing access to affordable psychological and psychiatric care, reducing wait times, and offering better continuity of care (LaRock, 2019).
Despite the growing use of and support for VCT, along with other forms of remote mental health care (e.g., online support groups, mobile apps), there remains comparatively little empirical evidence on its effectiveness. Although most studies that examine video-based services are promising, the aggregate effects are not fully understood. To date, multiple meta-analyses have examined the overall efficacy of telemental health within specific populations and diagnostic presentations (e.g., Batastini & Morgan, 2016; Brearly et al., 2017; Larson, Rosen, & Wilson, 2018; Osenbach et al., 2013; Sloan, Gallagher, Feinstein, Lee, & Pruneau, 2011). Another looked specifically at objective psychiatric assessments, finding no differences in accuracy or satisfaction across modalities (Hyler, Gangure, & Batchelder, 2005). In fact, satisfaction and therapeutic alliance have been broadly examined in various countries, including Canada (Germain, Marchand, Bouchard, Guay, & Drouin, 2010), England (Manchanda & McLaren, 1998), Australia (Stubbings, 2012) and Scotland (Simpson, Bell, Knox, & Mitchell, 2005). The only known meta-analysis comparing therapy and assessment outcomes across VCT and in-person providers (Drago, Winding, & Antypa, 2016) focused strictly on psychiatric services. Importantly, given differing educational and training models of each discipline, psychological and psychiatric services can differ quite dramatically with regard to therapeutic orientation, the client-doctor relationship, duration of services, and structure of services, among other variables. Therefore, meta-analyses that do not include literature related to psychological approaches are limited in scope and do not fully capture the range or intensity of services that are now routinely provided through VCT. Other systematic reviews of the effectiveness of telemental health also exist that attempt to take a broader look at the practice (e.g., Hilty et al., 2013; Langarizadeh et al., 2017; Rees & Maclaine, 2015; Salmoiraghi & Hussain, 2015); however, these only provide descriptive statistics and more of a narrative overview rather than a statistically controlled comparison between traditional in-person and VCT services.
The current meta-analysis adds a more generalist perspective to the literature base that has not yet been captured empirically. That is, our primary aim was to answer the basic question of whether VCT is better, worse, or relatively equivalent to in-person mental health services. Taking a more comprehensive, aggregate approach is needed for several reasons. First, it allows for an additive, bigger picture interpretation; can we say VCT is equally effective in an overall sense and for specific clinical purposes or clients? Second, knowing the overall effects of VCT can serve as point of comparison to better contextualize effect sizes produced by narrowly focused meta-analyses; that is, how do aggregate effects for specific clinical purposes or clients measure up to those for VCT in general? Third, working with a larger collection of studies offers an opportunity for more robust comparisons of effects across different variables of interest than what can be performed within smaller-scale meta-analyses; for example, are outcomes for certain disorder categories stronger or weaker compared to others? The ability to examine these comparisons likewise provides a meaningful context within which to interpret findings from other analyses of specialized client groups. Fourth, and related to comparability, because this meta-analysis captures studies published before the dramatic shifts associated with COVID-19, it can serve as a historical baseline for future meta-analyses that are sure to follow from the inevitable spike in empirical studies that will emerge during and post-COVID-19. Finally, we view the present study as a wholistic effort to reveal what is out there and to what extent and, conversely, uncover the types of clinical services and populations that are less represented in the literature. While extant systematic reviews can help with this delineation, they lack accompanying statistical insights about the actual potential of VCT to engender relatively equivalent (or distinct) outcomes to in-person.
Several factors contribute to the uniqueness of the present set of meta-analyses. First, we applied comprehensive inclusionary criteria such that studies addressing a wide range of mental health disorders, client populations, interventions, and assessment types are represented. Second, we were concerned only with studies that compare VCT services to an in-person control; pre-post studies examining only VCT were excluded to clearly and directly answer the question of whether VCT produces outcomes that are comparable to in-person rather than whether VCT is an improvement over the absence of services. Further, in examining outcomes explicitly associated with therapeutic interventions, we apply multiple contemporary meta-analytic strategies and examine several moderator analyses that further explore what works and for whom. Overall, we hypothesized that meta-analytic results would support VCT as an equally effective treatment and assessment modality compared to in-person services by yielding small and/or non-significant effect size estimates when comparing relevant mental health outcomes by modality type.
Perhaps equally important to the meta-analytic findings themselves, the process of compiling empirical work products uncovered several limitations and flaws in the available literature that we argue must be discussed and addressed in future evaluation efforts. Following the presentation of our primary findings, we emphasize the need for more controlled treatment outcome and assessment reliability studies and offer initial reporting recommendations for future researchers. As will be detailed later, a large number of studies were excluded for lack of a comparison group, because they did not focus directly on service efficacy (e.g., instead asking about service satisfaction or acceptability), or because useful data was unknown or reported in a limited manner. While some guidelines exist regarding the empirical study of telemedicine practices (Krupinski & Bernard, 2014) and there are general manuscript reporting guidelines (e.g., Centers for Medicare and Medicaid Services, 2020), standardized recommendations for reporting findings specifically from telemental health research to scientific consumers, service providers, legislators, and other stakeholders is lacking. One available set of recommendations focuses exclusively on reporting VCT studies for depression interventions (Abel, Glover, Brandt, & Godleski, 2017). While many of these recommendations may apply to this line of research generally, they may also miss important factors or considerations highlighted by work in other domains. Ensuring the systematic and consistent reporting of such findings will not only deepen our understanding of VCT as an alternative service modality but will also increase our confidence in embracing VCT as the new norm in mental healthcare.
Section snippets
Study inclusion
To be included in this meta-analysis, studies had to (1) be published or available in English, (2) evaluate mental health-related outcomes (e.g., symptom reduction, hospitalizations) following a mental health service (i.e., psychiatric consultations, psychotherapy/counseling, assessment), (3) use a telecommunication service delivery system that transmitted live audio and visual information simultaneously, (4) use a between-groups comparison design, and (5) report sufficient information to allow
Intervention studies
All intervention outcome measures (for VCT and in-person) were converted to Hedges' g standardized effect sizes. Most of the intervention outcomes (n = 256 or 91.1%) were reported as means and corresponding standard deviations (SDs) or standard errors (SEs). These outcomes were converted to Hedges' g indices using the pooled SDs of the pre-measures, when available. The method of standardization based on pre-SDs is preferable to one based on the post-SDs measures because of a possible subject by
Study and sample characteristics
Forty-three individual studies published between the years of 2002 to 2019 met all inclusionary criteria and were included in the final analyses examining intervention outcomes. Seven studies were published in 2016; five in 2017; four in 2015 and between one and three studies were published in other years. A total of 281 individual outcomes were included across these 43 studies. Methodological features and sample characteristics of the analyzed intervention studies are presented in Table 1,
Discussion
Advances in technology have virtually transformed the way people communicate and interact with each other. It is not surprising, then, that the mental health sector is embracing video communication systems in an effort to keep up with existing social norms and take advantage of the opportunity to reach clients who may otherwise have limited options for local specialty providers, face increased costs and lost wages for travel to distant clinics, or go without services altogether. Beyond this,
Final conclusions
There is no question that remote delivery mechanisms are hitting a stride in mental and behavioral health service industries. But, are VCT services just as good as those delivered in-person? Thus far, the answer points to “yes,” as the available evidence suggests VCT does not grossly impede clinical outcomes in an overall general sense. Yet, there is more work to be done. Stronger research designs, greater inclusivity regarding client demographics and service settings, purposeful examination of
Role of funding sources
This project was supported by the Federal Office of Rural Health Policy (FORHP), Health resources and Services Administration (HRSA), U.S. Department of Health and Human Services (HHS) under cooperative agreement award number 6 U66RH31459–02-03. Dr. Paprzycki was also partially supported by the National Institute of General Medical Sciences of the National Institutes of Health under award number 5 U54 GM115428. The information, conclusions, and opinions expressed are those of the authors and no
Contributors
Ashley B. Batastini lead, supervised, and designed the study; to include development of the codebook, training and supervising coders, and overseeing reliability procedures and primary data analysis. Dr. Batastini also wrote significant portions of the manuscript and served as primary editor of all sections of the manuscript. Peter Paprzycki was the primary statistician on this project; he conducted the majority of statistical analyses and wrote all respective sections of the manuscript
Funding
This project was supported by the Federal Office of Rural Health Policy (FORHP), Health resources and Services Administration (HRSA), U.S. Department of Health and Human Services (HHS) under cooperative agreement award number 6 U66RH31459–02-03. Dr. Paprzycki was also partially supported by the National Institute of General Medical Sciences of the National Institutes of Health under award number 5 U54 GM115428. The information, conclusions, and opinions expressed are those of the authors and no
Declaration of Competing Interest
None.
Acknowledgements
We would like to thank the following individuals for their assistance with coding, data extraction, and manuscript preparation: Christopher M. King, Ph.D., J.D., Stephanie Van-Horn, Ph.D., Lauren Coaker, M.S., Michael Lester, M.S., Alexandra Repke, M.A., Alexandra Teller, M.A., Riley Davis, M.A., Nathan Ross, Ph.D., Ja'Nisha Robinson, M.S., Madison Pike, B.A., Kendall Klumpp, B.A., Abigail Armstrong, B.A., and Olivia Miller, M.S.
Ashley B. Batastini received her PhD in counseling psychology at Texas Tech University. She is currently assistant professor in the Department of Counseling, Educational Psychology, and Research at the University of Memphis. Dr. Batastini was affiliated with the University of Southern Mississippi when this work was completed. Her professional interests include development of and research on mental health interventions for criminal justice-involved populations and best practices in forensic
References (134)
- et al.
Videoteleconferencing in forensic and correctional practice. Telemental health: Clinical, technical, and administrative foundations for evidence-based practice
(2013) - et al.
Videoconferencing in psychiatry, a meta-analysis of assessment and treatment
European Psychiatry
(2016) - et al.
Guided internet-based self-help intervention for social anxiety disorder with videoconferenced therapist support
Cognitive and Behavioral Practice
(2016) - et al.
Exposure therapy for PTSD delivered to veterans via telehealth: Predictors of treatment completion and outcome and comparison to treatment delivered in person
Behavior Therapy
(2011) - et al.
Telehealth versus in-person acceptance and commitment therapy for chronic pain: A randomized noninferiority trial
The Journal of Pain
(2017) - et al.
A randomized pilot trial comparing videoconference versus face-to-face delivery of behavior therapy for childhood tic disorders
Behaviour Research and Therapy
(2012) - et al.
Gender differences in mediated communication: Women connect more than do men
Computers in Human Behavior
(2013) - et al.
A randomized trial of web-based videoconferencing for substance abuse counseling
Journal of Substance Abuse Treatment
(2014) - et al.
A comparison of face-to-face and remote assessment of inter-rater reliability on the Hamilton depression rating scale via videoconferencing
Psychiatry Research
(2008) - et al.
Online mental health services in China during the COVID-19 outbreak
Lancet Psychiatry
(2020)
Teleconferencing model for forensic consultation, court testimony, and continuing education
Behavioral Sciences & the Law
A randomized trial comparing the efficacy of cognitive-behavioral therapy for bulimia nervosa delivered via telemedicine versus face-to-face
Behaviour Research and Therapy
Recommendations for the reporting of telemental health (TMH) literature based on a systematic review of clinical video teleconferencing (CVT) and depression
Journal of Technology in Behavioral Science
Behavioral activation and therapeutic exposure for post traumatic exposure for posttraumatic stress disorder: A noninferiority trial of treatment delivered in person versus home-based telehealth
Depression and Anxiety
Guidelines for psychological evaluations in child protection matters
The American Psychologist
Practice guidelines for video-based online mental Health services
Empirical evidence on the use and effectiveness of telepsychiatry via videoconferencing: Implications for forensic and correctional psychiatry
Behavioral Sciences & the Law
Fitting three-level meta-analytic models in R: A step-by-step tutorial
The Quantitative Methods for Psychology
Safe, seamless and secure: Australia's national digital health strategy
Innovations in correctional assessment and treatment
Criminal Justice and Behavior
Videoconferencing psychotherapy: A systematic review
Psychological Services
Connecting the disconnected: Preliminary results and lessons learned from a telepsychology initiative with special management inmates
Psychological Services
Perceptions and use of videoconferencing in forensic mental health assessments: A survey of evaluators and legal personnel
Psychology, Crime & Law
Using technology in the delivery of mental health and substance abuse treatment in rural communities: A review
The Journal of Behavioral Health Services & Research
The posttraumatic stress disorder checklist for DSM-5 (PCL-5): Development and initial psychometric evaluation
Journal of Traumatic Stress
Effect sizes for continuous data
Comprehensive meta-analysis (version 3.3.070) [computer software]
Delivering cognitive-behavior therapy for panic disorder with agoraphobia in videoconference
Telemedicine Journal and E-Health
Psychometric properties of the PTSD checklist for diagnostic and statistical manual of mental disorders-fifth edition (PCL-5) in veterans
Psychological Assessment
Neuropsychological test administration by videoconference: A systematic review and meta-analysis
Neuropsychology Review
A telepsychology casebook: Using technology ethically and effectively in your professional practice
American Psychological Association.
Telehealth-delivered group smoking cessation for rural and urban participantsL feasibility and cessation rates
Addictive Behaviors
Medicare telemedicine health care provider fact sheet
Comparing face-to-face and videoconference completion of the Montreal Cognitive Assessment (MoCA) in community-based survivors of stroke
Journal of Telemedicine and Telecare
A guide to conducting a meta-analysis with non-independent effect sizes
Neuropsychology Review
Meta-analysis: A structural equation modeling approach
Modeling dependent effect sizes with three-level meta-analyses: A structural equation modeling approach
Psychological Methods
metaSEM: An R package for meta-analysis using structural equation modeling
Frontiers in Psychology
Six-month postintervention depression and disability outcomes of in-home telehealth problem-solving therapy for depressed, low-income homebound older adults
Depression and Anxiety
Internet-delivered, family-based treatment for early-onset OCD: A pilot randomized trial
Journal of Consulting and Clinical Psychology
Remotely delivering real-time parent training to the home: An initial randomized trial of internet-delivered Parent-Child Interaction Therapy (I-PCIT)
Journal of Consulting and Clinical Psychology
Psychotherapy using distance technology: A comparison of face-to-face, video, and audio treatment
Journal of Counseling Psychology
Randomized clinical trial of telepsychiatry through videoconference versus face-to-face conventional psychiatric treatment
Telemedicine and e-health
Videoconference administration of the Western aphasia battery-revised: Feasibility and validity
American Journal of Speech-Language Pathology
Scotland's digital health and care strategy: Enabling, connecting and empowering
A randomized, controlled trial of child psychiatric assessments conducted using videoconferencing
Journal of Telemedicine and Telecare
An experimental comparison of telepsychiatry and conventional psychiatry for parolees
Psychiatric Services
A budget impact analysis of telemedicine-based collaborative care for depression
Medical Care
A randomized trial of telemedicine-based collaborative care for depression
Journal of General Internal Medicine
Cited by (75)
Trust in higher guidance and suicidality in Afghan students under the Taliban regime{fr}(Confianza en la orientación superior y suicidalidad en estudiantes afganos bajo el régimen Talibán)
2023, International Journal of Clinical and Health PsychologyTherapist perspectives on telehealth-based virtual reality exposure therapy
2024, Virtual RealityOnline versus in-person delivery of cognitive behaviour therapy for obsessive compulsive disorder: An examination of effectiveness
2024, Clinical Psychology and PsychotherapyRemote blended treatment for individuals with suicidal ideation: A single-arm proof-of-concept trial
2024, Clinical Psychology and Psychotherapy
Ashley B. Batastini received her PhD in counseling psychology at Texas Tech University. She is currently assistant professor in the Department of Counseling, Educational Psychology, and Research at the University of Memphis. Dr. Batastini was affiliated with the University of Southern Mississippi when this work was completed. Her professional interests include development of and research on mental health interventions for criminal justice-involved populations and best practices in forensic mental health assessment.
Peter Paprzycki received his Ph.D. in Research, Evaluation, Statistics and Measurement at the University of Toledo. He is a part-time Professor at the University of Toledo, and a statistical consultant at Mississippi Center for Clinical and Translational Research (MCCTR) and Grant Fundamentals, LCC. Dr. Paprzycki is a member of the Networking Urban Resources with Teachers and University to enRich Early Childhood Science (NURTURES), a National Science Foundation funded Mathematics and Science Partnership (MSP) program.
Ashley C. T. Jones received her master's degree at Arizona State University. She is currently a doctoral student at the University of Southern Mississippi. Her professional interests include innovative technologies in correctional treatment, program evaluation, and research on jury decision-making.
Nina MacLean received her PhD in counseling psychology at Texas Tech University. She is currently a consulting forensic examiner at the Center for Forensic Psychiatry. Her professional interests include professional issues and best practices in forensic mental health evaluations.
- 1
Dr. Batastini was primarily affiliated with the University of Southern Mississippi at the time this study was completed