Kowatsch et al. (2019) A Design and Evaluation Framework for Digital Health Interventions Study Guide
A Design and Evaluation Framework for Digital Health Interventions
Tobias Kowatsch, Lena Otto, Samira Harperink, Amanda Cotti & Hannes Schlieter · it – Information Technology, 61(5–6):253–263 · 2019
Despite a mature tradition of evidence-based medicine, systematic guidelines for designing and evaluating digital health interventions remain scarce. The DEDHI framework closes this gap by mapping 331 evaluation criteria and 98 implementation barriers across four iterative life cycle phases — from early research prototype to large-scale market deployment.
Research Question
How should digital health interventions (DHIs) be systematically designed, evaluated, and implemented across their full life cycle, and what evaluation criteria and implementation barriers are relevant at each stage?
The Problem: A Guidance Gap
Existing design and evaluation frameworks for health interventions — including the MRC Framework, MOST, and IDEAS — share a common shortcoming: none provide phase-specific guidance on which evaluation criteria to apply or which implementation barriers to anticipate as a DHI matures from prototype to product. Technology-related aspects such as maturity, scalability, and security are particularly underserved.
The MOST Foundation
The framework is built on an extended version of the Multiphase Optimization Strategy (MOST) developed by Collins et al. MOST was selected because it describes DHI development rigorously and iteratively, explicitly addresses just-in-time adaptive interventions (JITAIs) and micro-randomized trials, and focuses on behavioral health at the individual level. A fourth implementation phase — absent from the original MOST — was added by drawing on DHI life cycle models and the MRC Framework.
Scope and Contribution
DEDHI integrates research from three disciplines: behavioral medicine (behavior change components), medical informatics (clinical applications), and information systems (technology acceptance and barriers). The resulting framework is intended for both researchers and practitioners, and applies to all DHI types: mobile apps, web-based platforms, and hybrid interventions that combine digital coaching with human health professionals.
The DEDHI Framework at a Glance
| Phase | Goal | Technical Maturity | Key Evaluation Criteria | Key Barriers |
|---|---|---|---|---|
| 1. Preparation | Define conceptual and technological foundation; feasibility and acceptability study; select optimization criterion | Research prototype with basic functionality | Ease of use, adherence, personalization, safety, privacy & security | Individual characteristics, usability, expectations, planning, funding, regulatory issues |
| 2. Optimization | Run optimization trials (e.g. micro-randomized trial); identify best DHI configuration | Elaborated prototype with full functionality for component-level testing | Effectiveness (components), perceived benefit, content quality, aesthetics, adherence, service quality | Social support, outcome expectations, usability, funding for equipment, cost, integration |
| 3. Evaluation | Confirm effectiveness in randomized controlled trial vs. control condition | Elaborated prototype with full functionality for RCT | Effectiveness, perceived benefit, adherence, personalization, safety, privacy & security, accountability | Funding, guidelines, methodology (missing proof of cost-effectiveness) |
| 4. Implementation | Develop market-grade product; monitor reach, impact, side effects; update content and technology | Product-grade DHI with long-term operational readiness | Adherence, personalization, content quality, ethics, service quality, accountability | Individual resources, interoperability, human technical support, regional infrastructure, reimbursement, culture |
Top Evaluation Criteria Categories (from 331 total)
| Category | Count | % of Total | Example |
|---|---|---|---|
| Ease of Use | 87 | 26.3% | Using common interaction paradigms to minimize effort |
| Content Quality | 41 | 12.4% | Real-time, location-based, accurate health information |
| Privacy & Security | 41 | 12.4% | GDPR compliance; encrypted data transmission |
| Accountability | 39 | 11.8% | Author details accessible within the DHI |
| Adherence | 27 | 8.2% | Ratio of actual to intended usage per week |
| Aesthetics | 19 | 5.7% | Consistent use of colors, figures, and fonts |
| Effectiveness | 17 | 5.1% | Significant reduction in clinical outcome measure |
| Ethics | 5 | 1.5% | Design for diverse cultural backgrounds and disabilities |
| Safety | 3 | 0.9% | Interaction limits to prevent addiction behavior |
■ Framework Concepts ■ Research Methods ■ Barriers & Risks ■ Evaluation Criteria
Framework Concepts
Research Methods
Barriers & Risks
Evaluation Criteria
Study Design at a Glance
DEDHI is a conceptual framework paper combining two inputs: a new systematic literature review on DHI evaluation criteria, and a pre-existing review of implementation barriers (Otto & Harst, 2019). Both sets of findings were then mapped deductively to an extended version of MOST using qualitative content analysis. The paper is theoretical — the framework was not empirically validated in the field.
Literature Search: Evaluation Criteria
- Starting point: Nouri et al. (2018) systematic review of mHealth quality criteria, updated and broadened to cover all DHI types (mobile, web-based, hybrid)
- Three parallel searches: (1) backward search in Nouri et al. sources from 2000–2016; (2) updated Nouri search extended to DHIs from Dec 2016–May 2019; (3) extended search of socio-technical databases (ACM DL, IEEE Explore) and A/B-ranked digital health journals
- Search term structure: (evaluation OR criteria OR scoring) AND (intervention OR app OR therapy) AND (health OR clinical) AND (digital OR mHealth OR smartphone) — applied to title and abstract
- Inclusion criteria: original, peer-reviewed, English-language works describing a tool with evaluation criteria for DHIs; systematic reviews excluded but their source references screened
- Result: 2,616 initial results → 36 included records → 331 evaluation criteria extracted
Consolidation into Categories
- Two authors independently reviewed all 331 extracted criteria and consolidated them into inductive categories using qualitative content analysis (Mayring, 2000)
- Disagreements resolved through discussion; a third author consulted when consensus could not be reached bilaterally
- Result: 13 categories ranging from Ease of Use (87 criteria, 26.3%) to Safety (3 criteria, 0.9%)
- Implementation barriers: a pre-existing set of 98 barriers from Otto & Harst (2019) was similarly consolidated into 26 inductive categories with 106 assignments (some barriers span multiple categories)
Mapping to DEDHI Phases
- Evaluation criteria and barrier categories were each mapped to one or more of the four DEDHI life cycle phases using deductive qualitative content analysis
- Mapping conducted independently by at least two scientists; inconsistencies resolved through discussion until consensus
- Most criteria and barriers map to a single phase; exceptions include funding, cost, and several individual characteristic barriers, which span all four phases
- Some barriers (missing cooperation incentives, unclear responsibilities, disease-specific constraints) could not be aligned to any phase — they represent structural framing conditions outside the DHI development process
Limitations
- Not empirically validated: DEDHI was developed inductively from literature and has not been applied or revised through real-world DHI development cycles
- Scientific literature only: country-specific regulatory frameworks are incorporated only to the extent they appear in peer-reviewed outlets — practical legal requirements may differ substantially
- Stakeholder-blind: the framework does not distinguish between research teams (focused on preparation/optimization) and commercial companies (focused on implementation), whose documentation and testing requirements diverge significantly
- Subjective methodology: both literature search and content analysis involve researcher judgment; mitigated by multi-author independent coding and consensus procedures
Collins et al. — The Multiphase Optimization Strategy (MOST)
Why it matters to DEDHI
MOST is the direct precursor to DEDHI’s structure. Collins et al. proposed a rigorous iterative approach to behavioral intervention development with explicit preparation, optimization, and evaluation phases. DEDHI adopts these three phases wholesale, adding only a fourth implementation phase absent from MOST.
MOST’s distinguishing feature is its explicit treatment of optimization: rather than testing a whole intervention against a control, it isolates and evaluates individual intervention components. This component-level logic justifies the micro-randomized trial as the recommended design for DEDHI Phase 2. The MOST framework also embraces JITAIs, which require technology-intensive adaptive delivery — a core reason it was selected over more clinically traditional frameworks like the MRC.
Nouri et al. — Criteria for assessing quality of mHealth apps
Why it matters to DEDHI
Nouri et al. provided the primary starting point for the DEDHI evaluation criteria literature search. Their systematic review identified quality criteria specifically for mobile health applications. DEDHI extended this work in three directions: broadening scope from mHealth only to all DHI types (including web-based and hybrid); updating the search to May 2019; and adding socio-technical databases (ACM DL, IEEE Explore) not covered by Nouri.
The backward search from Nouri’s reference list (Search Strategy 1 in DEDHI) was applied to sources back to 2000, establishing the historical baseline for the criteria analysis.
Nahum-Shani et al. — JITAIs in Mobile Health
Why it matters to DEDHI
Nahum-Shani et al.’s work on JITAIs is cited as one of the primary reasons MOST was chosen as DEDHI’s lifecycle model. JITAIs represent a clinically important and technology-dependent class of DHI: they deliver tailored support at precisely the right moment based on real-time sensing data and predictive models. This requires technological infrastructure and evaluation designs — such as the micro-randomized trial — that conventional frameworks do not address.
The 2015 paper established the conceptual pragmatic framework for JITAIs; the 2018 paper elaborated the key design components. Both inform DEDHI’s Phase 2 (Optimization) methodology.
Campbell et al. — MRC Framework for Complex Interventions
Why it matters to DEDHI
The Medical Research Council Framework for complex interventions is one of the most widely cited frameworks in health intervention design. It provided the conceptual basis for DEDHI’s fourth (implementation) phase, which MOST does not address. The MRC framework’s emphasis on monitoring reach, impact, and long-term side effects in real-world settings directly informed the goals and evaluation criteria of Phase 4.
Alongside Broens et al.’s telemedicine life cycle model, the MRC framework is credited for DEDHI’s explicit inclusion of a post-RCT implementation phase — the phase most relevant to practitioners and commercial DHI companies.
Broens et al. — Determinants of Successful Telemedicine Implementations
Why it matters to DEDHI
Broens et al. proposed a four-layered DHI life cycle model distinguishing prototypes, small-scale pilots, large-scale pilots, and operational products — linking specific success determinants to each stage. This model provided DEDHI with its phase-specific view of technical maturity and the idea that implementation success factors differ meaningfully across stages.
The Broens model is particularly important for DEDHI’s Phase 4, informing how a commercially viable, maintained DHI product should be monitored and updated over time.
Klasnja et al. — Microrandomized Trials
Why it matters to DEDHI
The micro-randomized trial (MRT) is the recommended evaluation design for DEDHI’s Optimization Phase. Unlike conventional RCTs that compare whole interventions, MRTs randomize at each individual decision point — enabling DHI developers to isolate the causal effect of specific components delivered in real-world context.
Klasnja et al.’s paper introduced MRTs as a tool specifically designed for JITAIs, making it a natural methodological complement to the Nahum-Shani JITAI design framework and to MOST’s component-selection logic.
The DEDHI framework integrates two parallel literature streams. The first, from behavioral medicine and clinical trial methodology (Collins MOST, Nahum-Shani JITAIs, Klasnja MRTs, Campbell MRC), shapes the life cycle phases and their evaluation designs. The second, from information systems and technology management (Broens life cycle, Nouri mHealth criteria, Otto & Harst barriers), contributes the technology maturity lens and the pragmatic barrier taxonomy. The framework’s core innovation is the deliberate synthesis of these two streams — rarely combined in prior work.
Effective and scalable digital health interventions require more than good ideas — they require a structured, phase-sensitive approach that aligns evaluation criteria and implementation barriers to where the DHI actually is in its development journey. The DEDHI framework provides this scaffolding, from first prototype to market-deployed product.
-
Phase Matters: One Framework Does Not Fit All
Applying a full clinical evaluation battery to a Phase 1 research prototype wastes resources; deploying a barely-tested prototype at scale risks patient harm. DEDHI’s key contribution is that the relevant evaluation criteria and addressable barriers differ meaningfully at each of the four phases — and knowing which ones apply when is actionable guidance that prior frameworks lacked.
-
Optimize Before You Confirm
The Optimization Phase (Phase 2) is the framework’s most operationally novel contribution. Using micro-randomized trials to select effective components before an RCT avoids the costly mistake of testing an under-optimized intervention. This logic — borrowed from Collins’ MOST — is especially important for adaptive, JITAI-based DHIs where component interactions are complex.
-
Effectiveness is Underrepresented in the Literature
Despite being the primary objective of health interventions, effectiveness ranks only 8th among the 13 evaluation criteria categories (5.1% of all criteria). Ethics and safety are even more neglected. This signals a systemic bias in the mHealth evaluation literature toward usability and interface aesthetics at the expense of clinical outcomes — a gap DEDHI explicitly calls out.
-
Implementation is a Phase, Not an Afterthought
The addition of Phase 4 is not a minor extension — it reflects a substantive argument that long-term DHI effectiveness requires ongoing monitoring of reach, impact, and side effects; regular content and technology updates; and navigation of barriers (interoperability, reimbursement, regional infrastructure) that only surface at scale. Researchers who stop at the RCT miss everything that determines whether a DHI actually helps patients in practice.
-
Researcher and Practitioner Goals Diverge
A research team funded by a national foundation cares most about Phases 1–2 and publishable results. A commercial DHI company needs to reach Phase 4 rapidly and meet hard regulatory requirements from day one. DEDHI does not yet distinguish these stakeholder perspectives — a limitation the authors acknowledge and flag for future development of the framework.
-
Criteria and Barriers Show Universal Reach
The evaluation criteria and implementation barriers in DEDHI originate from studies conducted across the United States, Europe, Australia, and Africa. The authors interpret this geographic breadth as evidence of the framework’s universality — the same core challenges (usability, funding, regulatory issues, adherence) appear regardless of country-specific context, lending DEDHI cross-national applicability.