Clinical Operations as a distinct corporate function and department has evolved into a standard corporate model at sponsor and CRO companies, with all the trappings: Titles (e.g. Director of ClinOps), standard roles, its own language, a bevy of software applications, and dedicated trade events. As with any corporate function, it needs a set of metrics to provide a way to measure performance. And while the adoption of clinical trial performance metrics — KPIs — has been uneven, ClinOps professionals today tend to identify a common set of indicators.
The Need for Metrics Isn’t New
In Applied Clinical Trials, way back in 2016, the issue of clinical study KPIs was already coming to focus. In an article titled “Finally, Standardized KPIs are Front and Center,” the author Kenneth A. Getz identified the following as some of the most important things that should be measured in a critical trial:
- Proportion of final databases locked on time
- Mean number of protocol amendments post-protocol approval
- Proportion of studies completing patient enrollment on time
- Mean number of protocol deviations per study volunteer
- Proportion of studies with investigative sites activated on time
- Mean number of times databases are unlocked per study
- Proportion of vendors with critical findings following an audit
That list is a fine start, but they are a little narrowly focused on technical study factors — rather than operational team factors — within each study.
ClinOps leaders at sponsors and CROS need measures that measure their organization, not just the studies they are responsible for. That may start with a simple Red/Yellow/Green indicator for each study, but a larger set of indicators have the potential to signal trends and issues across studies.
A List of Clinical Operations Metrics
Based on a completely unscientific survey of ClinOps leaders working with Agatha, I have aggregated a list of what they feel are the most important measures for their organization as well as the studies they are running. Each metric listed includes the actual item measured in (parentheses). Many of these are being incorporated into Agatha’s dashboard capabilities in our ClinOps applications suite.
Core ClinOps Metrics
These are measurements of the core ClinOps organization, intended to capture the capacity and performance of the team. In most cases, the best approach is to track them monthly and show them as column charts to illustrate trends over time.
- Number of Studies Underway (Studies)
- Studies Completed Last 12 Months (Studies)
- Number of Countries in Each Study (Countries)
- Number of Sites in Each Study (Sites)
- Number of Patients in Each Study/Phase (Patients)
- Level of Outsourcing (% of work completed in house vs. outsourced)
- Number of Vendors utilized per Study (Vendors)
- Total ClinOps Team Members (People)
- Total FTE Count (People)
- Employee versus Contractor (People)
Quality Metrics
Quality metrics are perhaps the most important set of measurements because they highlight the team’s ability to execute trials accurately and effectively.
For each study:
- Number of CAPAs created and resolved (CAPAs)
- Number of AEs/SAEs per Site/Country/Study
- Number of database queries created and resolved (Queries)
- Number of Days to data entry per Site/Study (Days)
- Number of Monitoring Visits completed and reported per CMP (Monitoring Plan)
- Number of planned and unplanned SOP deviations (Deviations)
- Number of Audits completed at Site and Study Levels (Audits)
- Number of Audits with major findings (Findings)
- Number of Major Findings resolved and time to resolution (Findings)
- Patient Population Diversity (Patients)
Financial Metrics
Money matters, and measuring the cost and use of dollars is a critical measure for the ClinOps team. While sometimes confidential, you can use percentages for the whole team to help convey the importance of staying on budget.
For each study:
- Total Study Budget (Dollars)
- Budget by Category (Dollars)
- Internal Budget (Dollars)
- Contractor/Vendor Budget (Dollars)
- Budget Consumed (Dollars)
- Budget Remaining (Dollars)
For CROs:
- Total Invoicing (Dollars)
- Payment and Collections (Dollars)
- Aging Report (Vendors)
TMF Metrics
As the central repository of study documentation, the Trial Master File can be used as a proxy for the state of the overall study. The “What’s Missing” measure is probably the most common measure across all the ClinOps leaders I talked to.
For each study:
- Total TMF Items Expected (Documents)
- Total TMF Documents Collected (%)
- Total TMF Documents Approved (%)
- Missing TMF Documents (%)
- TMF Completeness (%)
- Average Approval Time per Document (Days)
- Items Awaiting Approval (Documents)
- Artifacts QC’ed (represented as a % of total artifacts uploaded)
- Artifacts QC’ed returned with errors (%) vs. no errors (%)
Cycle Times
Cycle times are tricky because every study has its own dynamics that drive the calendar. Using an approach that compares plan to actual is therefore important when looking at the cycle times of core study activities.
For each study:
- Study and/or Site level IRB Submit to IRB Approval (Days)
- Study Approval To First Enrollment (Days)
- Total Time (Days) Per Phase
- Study End to Submission (Days)
- Time LPLV to Database Lock (DBL)
- Time DBL to final CSR (Days)
As you look through this list you might find some clinical operation metrics missing. Let me know what I have missed. What do you feel are the most important things to measure in your operation?