The California collaborative network to promote data driven care and improve outcomes in early psychosis (EPI-CAL) project: rati

Objectives

The main aims of the study are to (1) determine the feasibility of implementing an LHCN across a diverse, decentralized network of early psychosis programs, (2) determine if the implementation of an LHCN leads to an increase in the delivery of measurement-based care, and to what extent, and (3) determine if the implementation of measurement-based care is associated with significant improvements in key outcomes.

Study setting and eligibility criteria

Recruitment

Recruitment is only open to service users and their support persons from participating EP programs in California that have joined the EPI-CAL network (Supplementary Table S1). Any service user who is eligible for the EP program, along with their support persons, is eligible to participate in the study and can be registered in the Beehive application by program staff. Data generated from survey completion on the Beehive application is clinically relevant, meant to be used in clinical care, and integrated into a clinic’s workflow. Therefore, EPI-CAL study staff support service user registration and enrollment in Beehive. The research team works closely with each participating EP clinic to ensure adequate integration of Beehive in clinical care in addition to demonstrating the utility of data collected through Beehive to ensure adequate recruitment of EP program participants. Use of Beehive by service users, support persons, and EP program staff does not require written informed consent, but rather a completed end-user license agreement (EULA). Trained EP program staff introduce Beehive to participants who are shown a video explaining the purpose of the study and how their data will be used. Participants are then presented with the EULA to read and make data sharing choices. Participants are not able to use Beehive until they have completed their EULA. Participants can change their data sharing permissions at any time during participation in EPI-CAL. If a service user does not opt in to sharing data for research purposes, the research team does not use data about that service user for research purposes even if the data are derived from clinic staff or support persons. The institutional review board of the University of California, Davis, reviewed and approved the study (1403828-21, see Ethics Section).

Interventions

Outcome measures and instruments

The first aim of the project is to determine the feasibility of implementing an LHCN across a diverse, decentralized group of EP programs. To determine if the LHCN has been effectively implemented (Aim 1), we record the total number of eligible service users who enroll in the EP program during the study period, the number that successfully complete the EULA [21], the number that agree for their data to be used to support research activities, and the number of service users and support persons that complete at least one survey. We expect that 70% of eligible EP program participants and 50% of their available support persons across the network will enroll and complete baseline surveys based on prior studies within an EP population [22]. To determine this, EP programs will be asked to provide our team with the total program census number annually, which is compared to service users enrolled in Beehive. Service users must have completed their EULA to be considered enrolled. Data on the number of available support persons is available in Beehive, and we can assess whether a primary support person (PSP) has completed enrollment.

We will measure survey completion of any of the surveys available in Beehive’s CAB in order to further assess implementation success. The CAB includes validated measures for both service users and their primary support person to complete. The initial proposed CAB was developed by selecting relevant measures from the PhenX toolkit [23], the Mental Health Block Grant (MHBG) minimum dataset, Mental Health Services Act (MHSA) demographic reporting requirements, and existing program evaluation measures. From there, the final measures and domains were reviewed and refined in focus groups with service users, family members, and providers conducted by our team [24], and the EPINET workgroup. The EPI-CAL CAB overlaps significantly with the EPINET CAB [25] (https://nationalepinet.org/core-assessment-battery-cab/) but differs in some domains and administration methods. See Table 1 for a comprehensive list of outcomes assessed by the CAB. In addition to the EPI-CAL CAB, service users will also be able to complete cognitive testing through Beehive annually.

We also conduct semi-structured qualitative interviews with service users and providers to assess barriers and facilitators to implementing the LHCN examining service user-, provider- and program-level barriers to enrollment and completion. Purposive sampling will be used to recruit participants across clinics where Beehive adoption and implementation has been either high or low, and with service users who have and have not received measurement-based care. Service user participants will be recruited either through clinician referral or by the research team directly by contacting individuals who had previously given permission to be contacted for future research opportunities.

To determine if implementation of the LHCN leads to an increase in the delivery of measurement-based care (Aim 2), providers will complete self-report questionnaires in the pre- and post-implementation period of the project. To examine adoption of a new technology in the EP program, we will compare providers with respect to their self-reported use of data to determine treatment choices at two timepoints: prior to Beehive implementation, and then after training in and use of Beehive. Pre- Beehive implementation surveys include Treatment Alliance and Use of Data in Care Planning in reference to specific service users. Providers will also complete the Comfort with Technology survey. The sampling frame for each site will consist of surveys about service users of currently enrolled EP providers, restricted to service users with at least six months under treatment and who had been seen in the preceding month. Beehive training materials will then be implemented consistently across participating EP programs; implementation efforts will highlight the utility of data to identify treatment goals and metrics of improvement during treatment planning and provide guidance on service user-centered ways to collaboratively review data and monitor progress during care. Then, in post-implementation, the same set of surveys are administered to EP programs who have implemented Beehive in their program for at least a year. The survey sampling strategy used in the pre-implementation period to select clinician-service user pairs will be repeated after Beehive has been implemented in the clinic for a full year, to ensure a valid pre/post comparison on this outcome. Due to expected turnover from the clinician side and discharge/exit from the program on the service user side, we will not be able to sample the same group from the pre-implementation period. However, there will most likely be some representation in the post-implementation period from respondents who participated in the first phase of surveys. Therefore, these are not completely independent samples, nor are they completely repeated samples.

To determine the extent to which providers utilize the Beehive platform to deliver measurement-based care (Aim 2), we adopt two different approaches. First, we will examine whether a service user’s treatment team lead reviews completed surveys in Beehive, which is recorded by the application. During service user registration in Beehive, EP providers designate the service user’s treatment team lead, typically their primary clinician. Once a survey respondent completes a survey in Beehive, the data are immediately available to view in the Beehive dashboard. The Beehive survey reports will include a variable that shows whether or not each survey has been viewed by the service user’s treatment team lead (binary yes/no). Research staff will use this data to determine the degree to which providers are actively viewing data collected in Beehive.

To explore how Beehive data are used by those who frequently utilize the application, two types of in-app queries were developed: urgent clinical issues and data-use questions. Urgent clinical issues are a type of notification (in-app and email) in Beehive that encourages EP staff to review service user data. These notifications trigger if, during registration or survey completion, a service user endorses risk-to-self or risk-to-others on the Modified Colorado Symptom Index (MCSI) [26], the intent to stop taking their medication [27], or lack of a permanent address. Beehive displays urgent clinical issues on a dashboard widget and to resolve them, users must indicate how they used the data in care. Additionally, an in-app query is presented to the service user’s treatment team lead every ten visits to the service user’s data page. Data-use questions assess (1) if the data was reviewed during a session with the service user or support persons and, if yes, (2) how the data was used as part of care.

We will then examine whether providers’ implementation of measurement-based care is associated with significant improvements in key outcomes (Aim 3). To do this, we will compare adjusted between-groups mean differences in baseline to 12-month change in surveys available in the Beehive CAB, including the MCSI. Groups will be defined by clinician metrics from Beehive described above and assessed during this 12-month period. The MCSI is a 14-item, self-report scale designed to assess the frequency of psychiatric symptoms related to psychosis, mood, desire to hurt oneself and others, cognition and forgetfulness. Each item is scored on a 0–4 Likert-style scale and added together to give a score between 0 and 56, with higher scores indicating greater emotional distress. Reduction in score over time indicates clinical improvement.

Data collection timeline

Participants, support persons, and EP program staff complete CAB clinical outcomes surveys at treatment baseline and every six months throughout treatment in the EP program. EP program staff also enter data at the time of the service user’s discharge from the EP program. Survey windows and due dates are all tied to the survey baseline date that program staff enter for each service user. For our purposes, the survey baseline date is associated with a service user’s start of care in their program, not when they are enrolled in Beehive. Though different programs may use a different visit during the early phase of engagement in clinical care as the anchor for the survey baseline date, all the service users within any given clinic use the same type of clinical visit as the baseline benchmark.

Beehive survey windows span several months each to facilitate the completion of the CAB. The baseline window in Beehive starts at the survey baseline date and ends 4 months later. Subsequent follow-up windows are due every six months from the survey baseline date. The follow-up windows open two months before the due date and close 4 months after the due date (Fig. 3). Programs are encouraged to complete surveys as close to the due date as possible, but flexibility is given to accommodate delays in clinical visits or unexpected barriers to completion of the survey (i.e., service user crisis or hospitalization). Data collection continues every six months throughout a service user’s treatment.

Sample size

Our estimated sample size is based on each EP program’s estimate of new clinical intakes in their program per year. We estimate that we will be able to systematically collect outcomes data on approximately 500 FEP service users and 300 additional CHR service users over the course of the study. Our target sample size for the barriers and facilitator interviews is 30 providers and 30 service users, or at the point where the main themes meet saturation. There are also additional participants in EPI-CAL from various focus groups, qualitative interviews, surveys outside of Beehive, and fidelity assessments.

Data collection methods

EPI-CAL researchers consulted the Guidelines for Data Acquisition, Quality and Curation for Observational Research Designs (DAQCORD) [52] on the design and implementation of this protocol to achieve a high-quality data set. Whenever possible, DAQCORD indicators were implemented through Beehive. We will highlight our implementation of several DAQCORD indicators below. See Supplementary Table S2 for the full list of DAQCORD indicators and how they have been implemented in this study.

Regarding the DAQCORD indicator of internationalization, Beehive and the EPI-CAL CAB will be localized into 12 threshold languages to meet the requirements of operating in California Medicaid (Medi-Cal) clinics. The languages will include English, Spanish, Vietnamese, Mandarin, Cantonese, Farsi, Armenian, Hmong, Khmer (Cambodia), Korean, Russian, and Tagalog. All measure translations go through either back-translation or translation by committee.

The majority of survey data collected for EPI-CAL is collected directly from service users and their support persons through Beehive and is described in the outcomes section. We designed Beehive to facilitate service users and support persons completing these surveys independently, though we encourage clinical staff to support individuals as needed (e.g., answering questions about survey content). There are some outcomes that require clinical staff to input data into Beehive (e.g., diagnosis, service use) and several outcomes of interest (e.g., symptoms, functioning, suicide risk) are collected from multiple respondent types.

Regarding the DAQCORD indicator to test data collectors and provide them with feedback regarding the accuracy of their performance across all relevant study domains, EPI-CAL clinical psychologists oversee the training of program staff who will complete the COMPASS-10 [48] and Global Functioning: Social [44] and Role [43] scales. Individuals either attend the live training or can watch a recording of it. Program staff must then provide a rating for 10 vignettes and must score within 1 point of the gold standard rating for each vignette to pass. We provide feedback for incorrect answers and allow program staff to retake the test with a different set of vignettes if needed.

Data management

For data collected outside of Beehive, including program census information collected to assess the feasibility of implementing an LHCN, data are collected from programs and entered into an Excel document and stored on a HIPAA-compliant cloud storage.

Regarding the EPI-CAL CAB, since participants enter survey data directly into Beehive, data entry by research staff is not required. Most data cannot be edited or modified after it is entered. For example, after the first survey has been completed about a service user, the survey baseline date can no longer be modified by the user. Any requests to change this field must be approved and implemented by the research team so that they can ensure all data are automatically assigned to the appropriate time point by the application. Additionally, surveys are in a read-only state after they have been completed by any type of survey respondent (service user, primary support person, and clinical team).

The database is hosted on Amazon Web Services (AWS) cloud. Amazon’s Simple Storage Service (Amazon S3) is used for storing content, logs, and backup data. This allows versioning, secured storage, and retrieval. Server-side data encryption is managed using AWS S3. Report functionality in Beehive allows data to be saved as a .csv file. The research team can access data across all programs but have access to only a limited data set. The only protected health information (PHI) that the research team has access to is a service user’s zip code, month and year of birth, and dates of service. When data is exported from Beehive by the research team, it is stored according to UC Davis policies for storage of PHI.

Because the data collection is happening in the context of regular clinical service, EP programs also have access to the data from their own program. Access to data is restricted based on user permissions managed by the EP programs. When programs download data from Beehive, it is their responsibility to treat the data as sensitive data and protect that data according to federal and state regulations and institutional requirements.

Information about all variables is documented in a data dictionary. The data dictionary documents: (a) variable name, (b) date of implementation for each translation, (c) a description of the data stored under the variable name, (d) the type and format of data (e.g., numeric, date MM/DD/YYYY), (e) implemented data validation, (f) response options when relevant, (g) missing data code, (h) validly skipped data code, (i) scoring information, and (j) details about skip logic when relevant. The data dictionary is updated whenever changes are made to the database, for example when new variables or new translations of existing variables are added.

Data validation is built into the data collection process in Beehive. Whenever possible, data collection has been designed to avoid free text fields so that data validation can be implemented. Some examples of data validation implemented in Beehive data collection are (a) minimum and maximum values for numeric text entry data, (b) formats for dates and zip codes, etc., (c) restriction on certain response options that are part of select-multiple questions (e.g., cannot select “none of the above” and select any other option).

Data analysis

Prior to analysis, we will complete descriptive summaries for all data collected in Beehive, including service user and clinician demographics, survey completion for each survey at each timepoint, and survey scores for quantitative measures. The distribution and completeness of each analysis variable will be examined to determine appropriateness of different statistical methods. The availability of within-person longitudinal data will be reviewed to determine whether longitudinal or cross-sectional approaches are most appropriate. Descriptive summaries will be generated for each clinic individually, as well as network wide.

To address Aim 1, we will examine whether we achieved adequate enrollment in Beehive by using descriptive statistics to see if at least 70% of eligible participants and 50% of their available support persons across the network were enrolled and completed at least one survey timepoint. To approximate the number of total service users eligible for enrollment, we will pull the total census number from each program annually. Eligible service user participants are defined as those who are determined eligible to receive care at each program. Available support persons are defined and identified by the service user. Service users must have completed their EULA to be considered enrolled. For the analysis, we only consider individuals who have agreed to share data with the University of California, Davis (UCD) as “enrolled”, but service users can decline this option and still use their data within their program for clinical purposes. Just like service users, primary support persons are not considered enrolled unless they have agreed to share data with UCD. Service users and support persons can make different choices regarding their data sharing permissions, i.e., a service user can decline to share their data for research purposes while a support person can opt-in. For the feasibility analysis, we will only examine what proportion of enrolled service users also have an enrolled PSP, acknowledging that there may be more enrolled PSPs whose corresponding service user opted out of data sharing. Survey data analysis procedures for clustered data (treating EP programs as clusters) will summarize characteristics of enrolled service users who complete enrollment and at least one survey. Enrollment rates (with 95% confidence interval) will be computed for (1) all eligible service users and (2) potentially available support persons. For the latter, we will report, for the denominator of eligible service users with available support persons, what proportion of those service users had at least one support person complete a baseline or 6-month survey assessment.

To assess Aim 2, the adoption of Measurement-Based Care (MBC), we will first examine the degree of use of data in care between the pre- and post-implementation periods of the project. Before Beehive implementation in each EP program, providers complete pre-implementation surveys about their demographic information (age, sex, race, ethnicity) and professional characteristics (years of education, degree type) and complete questionnaires on their (1) beliefs about the utility of data in care planning and (2) skills in discussing data with service users. Compared to the pre-implementation period, we hypothesize that providers will report a change in the use of data to determine treatment choices after training and using the app for at least one year. Separate models will be fit for each of the primary and alternative operationalization of Beehive clinician-usage metrics as the exposure variable of interest. Adoption of data in care will also be measured by examination of whether a service user’s treatment team lead examined completed surveys from service users. To determine the degree to which providers are actively viewing data collected in Beehive, research staff will review the Beehive survey reports variable that shows whether or not each survey has been viewed by the service user’s treatment team lead (binary yes/no). We hypothesize that EPI-CAL treatment team leads will have viewed service user data collected through Beehive in at least 50% of cases. Lastly, we will examine whether the clinician reported that Beehive data impacted the treatment plan as assessed by the in-app queries periodically presented to EP provider users.

Through the qualitative work that was completed in the first phase of this project [24], a variety of key outcomes were identified by our program, service users, and support person workgroups. Psychiatric symptoms, quality of life, and functioning were prioritized as key outcomes by all types of respondents and our analysis will center on these domains. When examining group-level differences, it is important to note that there is not a “Beehive” and “not Beehive” group of service users; all service users are assigned to the Beehive group and thus no analysis can examine the effect of Beehive use in treatment compared to a typical control group. Instead, to assess the impact of utilizing MBC in an EP LHCN (Aim 3), we will analyze a dataset consisting of one record per service user per follow-up assessment timepoint and outcomes expressed as within-person change scores from baseline (for continuous measures) or as count or binary outcomes. For count or binary outcome data, the corresponding baseline value of the outcome will be included as a person-level covariate, when appropriate. Outcomes will be measured by the MCSI, personal wellbeing index (PWI), and functioning measures (Global Functioning Social and Role Scales (GF: S and GF: R) or Mental Illness Research, Education, and Clinical Center (MIRECC) version of the Global Assessment of Functioning (GAF) scale) for each of the six-monthly assessment timepoints during the first 24 months. Continuous outcomes will be transformed into within-person change scores from the baseline assessment for each follow-up assessment. Data are structured hierarchically; there is nesting of measurements from service users, who are nested within clinicians within EP programs. Therefore, for continuous, binary, and count outcomes, generalized linear mixed models will be used to estimate the adjusted effects of exposures on the key outcomes of interest, adjusting for a parsimonious set of other clinician- and service user-level covariates. Random effects will be specified for sites, with additional effects specified for clinician and service users’ contribution to the model fit, according to the Schwarz Information Criterion.

A key operationalization of the exposure indicator will be based on a composite indicator for any review of the service users’ completed surveys. In particular, this variable will be scored a 1 for a given service user in a given follow-up assessment if the treatment team lead reviewed the service users’ completed survey data. The comparison groups are defined by clinician metrics from Beehive aggregated over the 6-, 12- and 18-month assessment period, and the primary analysis is based on a composite indicator for any review of the service users’ survey data by the treatment team lead. We will also assess timepoint-specific changes in psychotic symptom severity for each of the half-yearly assessment timepoints during the first 24 months, with the primary analysis based on a time-varying indicator for any endorsement of “impact on treatment plan” on the in-app queries as a time-varying independent variable.

Data monitoring

Research staff monitor the enrollment of eligible participants on a weekly basis and discuss progress with EP programs at regular meetings to facilitate enrollment of all potential participants. Our internal data management team monitor data as it is collected by sites in Beehive, such as service user enrollment and completion of the MCSI, to ensure data quality. We also periodically monitor datasets exported from Beehive for statistical analyses, including evaluating individual survey data, for missing or incomplete data. We provide feedback to sites, such as following up about missing diagnoses, if applicable. We conduct an ongoing analysis of the enrollment of service users and support persons solely to ensure that the study is appropriately reaching the target population.