Eliane Boucher LinkedIn Posts
How do you plan your evidence generation strategy?
A few weeks ago, Katherine Grill, PhD asked about different types of evidence and I shared my approach. She suggested I present it during a conference, but I'm not doing much conference travel these days - so I thought I'd share some of it on LinkedIn instead. Every day this week, I'll be posting about Digital Health Evidence Generation Strategy 101!
Today I'm starting with Step 1: Patient-Centered Research.
Qualitative research often gets a bad wrap among scientists, but I view this as the foundation for any digital health evidence generation strategy. Here's why:
📣 You need to hear from your users. Evidence generation aside - you don't want to invest in costly research before knowing that people will actually use your product. You want to identify ways to optimize your product before you proceed with any other steps in your evidence generation strategy.
💸 It's cost-effective. Of all the types of (prospective) research you'll do, this is probably the quickest and cheapest. Don't get me wrong - it'll be a major time investment as you collect and analyze data, but these can often be done in 1-2 months max.
📰 You can leverage the insights for marketing (good quotes are like gold), white papers, and yes - even peer-reviewed publications. The latter is something companies often forget, but if you plan your patient-centered research properly, it is 100% publishable and can help to show the market you invest in being knowledge experts and listening to patients and users. We did this and published a qualitative paper on how people with loneliness viewed our products and how they were coping with lockdown measures during the pandemic.
But here's the rub: if you want to make the most of this kind of research, it's important that you have trained qualitative researchers involved. There's a lot of lackluster patient-centered research done because people aren't trained in these methods (and you might be surprised that not all scientists are well versed in qualitative research - in fact, few are).
Where is patient-centered research in your strategy?
hashtag#digitalhealth hashtag#dtx hashtag#patientengagement hashtag#research hashtag#strategy
So you've gathered some patient and user feedback - and you feel ready to test your product's impact - what's next?
In digital health, the gold standard for evidence generation is often on randomized controlled trials - and for good reason (I'll get there later this week). But I've said it before, and I'll say it again: RCTs are expensive, resource-intensive, and they take a long time. You don't want to invest in an RCT unless you're confident it will be a success.
🎯 You need to de-risk your RCTs. But how do you do that?
If you have a wellness product, and you have time, my suggestion would be to start with releasing your product as a Direct to Consumer product. This might be your full product or an abbreviated version you're comfortable releasing into the wild. The advantage here is that you now have a product that is collecting data for you, and from users that may be more similar to real-world users than typical research participants.
Then you can take one of two approaches: (1) if you've been thoughtful about data collection within your app, you can conduct retrospective analyses of real-world user data. Or (2) you can invite new signups to participate in a study and compensate them (that compensation might even be free access to the full product) and they can fill out outcome measures outside of the app.
This is also a great opportunity to build in repeated opportunities for patient-centered research.
Can't release your product into the wild? No problem - you can still de-risk your costly trials by running pilot studies. Don't jump straight into the deep end - run a single-arm study to estimate the impact of your product, or a small version of your RCT. This will help you estimate the effects, and an added bonus is you'll also be able to use this information to help power those RCTs.
You might view this as delaying your "gold standard" evidence, or as a waste of money. But just like patient-centered research, if you're thoughtful about this approach, these smaller tests are also publishable and will help communicate early findings to your clients. By the time you're ready to launch your RCT, you'll already have a series of publications to garner confidence in the market.
I said what I said: RCTs are a necessary part of your evidence generation strategy - but they're not sufficient.
I mentioned yesterday that RCTs are the gold-standard for evidence generation for good reason. If you want to say your product is "clinically proven" (nothing is ever proven - but that's a conversation for another day), you can't do that without some kind of controlled study. It doesn't matter how creative you get with other research, it doesn't replace demonstrating the efficacy of your product above and beyond a control group - and better yet, an active control group.
What's more - the market will ask you about it. The market doesn't understand all of the nuances of research and different types of evidence - but you know what they all seem to know? That RCTs matter. Almost every RFP I've helped to respond to asks us how many RCTs we've done, and what we've found.
So while it's costly, and it takes time - every good evidence generation strategy should include plans for an RCT.
But it doesn't stop there - even with a successful RCT, that isn't sufficient evidence for your product. Why?
RCT participants are fundamentally different from real patients and users.
🤓 People who sign up for RCTs tend to differ from the general population - they're more motivated and interested in science and research.
🤑 We're typically compensating participants in RCTs - and even if we don't compensate them for using the intervention, we're increasing their overall commitment nevertheless.
💬 Study coordinators are usually in touch with RCT participants throughout the trial - increasing the likelihood that they stay engaged with your product.
The result is that engagement rates may be up to 4x higher in RCTs than they are in the real world (Baumel et al., 2019)!
So you need to balance your RCT, which has the benefits of allowing you to talk about causality and rule out placebo effects, with real-world data, which allows you to talk about how your product works with actual users. Because if your RCT effects are beautiful, but it doesn't represent how people are engaging with your product in the wild, your clients aren't going to be impressed when you start reporting on outcomes.
Your RCT may help land the deal, but your real-world evidence is going to help you RETAIN that deal.
I've got two more posts this week for this series - so if you've got questions you'd like me to weigh in on, let me know in the comments!
As much as it pains me to say it...
Your evidence probably won't get you very far, unless you can also demonstrate ROI.
Many of us working in digital health may have been motivated about doing good in the world - and I'd like to believe our clients are too - but the truth is that that pesky question is lurking in all sales conversations: 💸 what's the ROI?
Because of this, I've seen some digital health companies jump to ROI first - before they establish any other evidence. It's a gamble that might pay off - but it's not a gamble I'd suggest taking.
Here's the way I see it:
🏆 Maximizing ROI is your ultimate goal
📈 Your ROI is probably driven by having an actual impact on the condition(s) your interventions are targeting ➡ so maximizing outcomes is key.
👩💻 Maximizing outcomes isn't just about increasing the effect size, but maximizing how many people benefit from your solution. So that means optimizing engagement - and specifically how many users engage with your product in a beneficial way.
See what I did there? It's back to the three previous steps! Use Step 1 to help maximize engagement. Steps 2 and 3 help to assess and maximize your outcomes. And now I'd say you're ready to invest in HEOR work, feeling good that the ROI you get at the end of it will be impressive.
I realise this is a lot to ask - who has years to wait to share ROI with their prospective clients? There are things you can do in the interim, especially if you have a good HEOR strategist on board. If you're thoughtful, you may be able to use results from earlier studies to model out potential cost savings. And if you've laid the appropriate groundwork to de-risk your RCT, you can combine the RCT with the HEOR work, reducing timelines and associated costs of separating these steps.
hashtag#digitalhealth hashtag#dtx hashtag#research hashtag#roi hashtag#strategy
Are you treating research as an expense, or as an investment in generating future revenue?
As my last tip for this week - I want to push back against the typical view in business that research is an expense. If you're treating it as an expense, it's much easier to cut from your budget when looking to trim costs.
But you're probably still sending your commercial team to conferences, paying for ads, and other "expenses" you view as contributing to revenue generation. And there's the rub - people don't tend to view research as an important part of revenue generation, reducing its perceived value.
Here's why it's important to reconsider how you view research in your digital health company:
⏲ Research takes time, and there aren't any shortcuts. That means that each time you cut that study from your budget - you're delaying when you'll have that data and be able to make those claims. And once you hear from the market that you're coming up short, it's too late (especially if your competitors have continued to invest in their research).
🔬 The market is changing - I've gone from having conversations with clients who are relatively unfamiliar with research, to having to answer difficult questions from clinicians, scientists, and even biostatisticians. Long gone are the days of people taking your claims at face value - more and more, your clients have their own experts to scrutinize your research findings.
💰 The market DOES care about research. Look at all the arguments why payers aren't reimbursing digital therapeutics - it always boils down to lack of compelling evidence.
Research may be less directly related to revenue generation than the conferences you're attending, or the ads you're running, but don't let that distance fool you - how rigorous and compelling your evidence is is playing a role in how successful you are in the market.
Invest in your research. Invest in having credible scientists who can represent you well in the market.
And remember this isn't a one-time investment either. Every researcher worth their salt knows that research becomes outdated and requires replication - and that's especially true in the tech world where our products are continuously evolving. Plan for an ongoing and active program of research that builds in all the strategies I discussed this week and you'll never run out of material you can use to optimize your product and support your commercial strategy!
