Framework for the development and evaluation of complex interventions: gap analysis, workshop and consultation-informed update

Background The Medical Research Council published the second edition of its framework in 2006 on developing and evaluating complex interventions. Since then, there have been considerable developments in the field of complex intervention research. The objective of this project was to update the framework in the light of these developments. The framework aims to help research teams prioritise research questions and design, and conduct research with an appropriate choice of methods, rather than to provide detailed guidance on the use of specific methods. Methods There were four stages to the update: (1) gap analysis to identify developments in the methods and practice since the previous framework was published; (2) an expert workshop of 36 participants to discuss the topics identified in the gap analysis; (3) an open consultation process to seek comments on a first draft of the new framework; and (4) findings from the previous stages were used to redraft the framework, and final expert review was obtained. The process was overseen by a Scientific Advisory Group representing the range of relevant National Institute for Health Research and Medical Research Council research investments. Results Key changes to the previous framework include (1) an updated definition of complex interventions, highlighting the dynamic relationship between the intervention and its context; (2) an emphasis on the use of diverse research perspectives: efficacy, effectiveness, theory-based and systems perspectives; (3) a focus on the usefulness of evidence as the basis for determining research perspective and questions; (4) an increased focus on interventions developed outside research teams, for example changes in policy or health services delivery; and (5) the identification of six ‘core elements’ that should guide all phases of complex intervention research: consider context; develop, refine and test programme theory; engage stakeholders; identify key uncertainties; refine the intervention; and economic considerations. We divide the research process into four phases: development, feasibility, evaluation and implementation. For each phase we provide a concise summary of recent developments, key points to address and signposts to further reading. We also present case studies to illustrate the points being made throughout. Limitations The framework aims to help research teams prioritise research questions and design and conduct research with an appropriate choice of methods, rather than to provide detailed guidance on the use of specific methods. In many of the areas of innovation that we highlight, such as the use of systems approaches, there are still only a few practical examples. We refer to more specific and detailed guidance where available and note where promising approaches require further development. Conclusions This new framework incorporates developments in complex intervention research published since the previous edition was written in 2006. As well as taking account of established practice and recent refinements, we draw attention to new approaches and place greater emphasis on economic considerations in complex intervention research. We have introduced a new emphasis on the importance of context and the value of understanding interventions as ‘events in systems’ that produce effects through interactions with features of the contexts in which they are implemented. The framework adopts a pluralist approach, encouraging researchers and research funders to adopt diverse research perspectives and to select research questions and methods pragmatically, with the aim of providing evidence that is useful to decision-makers. Future work We call for further work to develop relevant methods and provide examples in practice. The use of this framework should be monitored and the move should be made to a more fluid resource in the future, for example a web-based format that can be frequently updated to incorporate new material and links to emerging resources. Funding This project was jointly funded by the Medical Research Council (MRC) and the National Institute for Health Research (Department of Health and Social Care 73514).


Appendix 2 Updating the Medical Research
Council guidance on complex interventions: an outline proposal for developing an integrated body of guidance

Background: the need for updaƟng and integraƟng the MRC guidance
The MRC guidance on evaluaƟng complex intervenƟons (CIs) was first published in 2000, and updated in 2008. Over this period, interest in complex intervenƟons has grown rapidly, and more specialised guidance has been published or is being developed in several related areas, including intervenƟon development, intervenƟon descripƟon, process evaluaƟon, natural experimental approaches, dealing with intervenƟon context, group-delivered intervenƟons, realist trials, surgical trials, exploratory trials, trial management, disability research and knowledge translaƟon [see

References for examples]. A number of other areas that were idenƟfied as gaps in the 2008
guidance, such as research priority-seƫng and the applicaƟon of complex systems science to health intervenƟons, have also aƩracted interest, but are not yet covered by accessible guidance for producers or users of evidence. Both the 2000 and 2008 core guidance documents conƟnue to be highly cited (figure), but given the pace and extent of methodological development, there is a strong case for updaƟng the core guidance, linking it with related developments, and also addressing some of the remaining weaknesses and gaps in the exisƟng guidance.

Aim & objecƟve:
To idenƟfy and summarise aspects of the 2006 MRC complex intervenƟon guidance that require updaƟng, with the aim of using this gap analysis as a starƟng point for discussion (through workshops and authorship group) to achieve consensus on focus points and resulƟng updates for the new guidance.
Source: Web of Science. The '2000 Guidance' refers to Campbell et al. (2000) and the '2008 Guidance' refers to Craig et al. (2008).

Proposed outputs
The 2000  a) A downloadable pdf version of the core guidance. b) A journal arƟcle (ideally published simultaneously in a number of journals, as the CONSORT statements are) which would describe the importance and need for the guidance as well as outlining its content, also referring potenƟal users to the detailed online pdf version and signposƟng resource.
In addiƟon, we will conƟnue discussions about creaƟng an online resource comprising the updated core guidance with links to signpost other related source documents (e.g. the MRC's process or natural experiments guidance) or to brief summary statements prepared specifically, with links to other useful published resources already available online. Gap analysis: based on scoping reviews to find publicaƟons that idenƟfy gaps and weaknesses in the exisƟng guidance, or that provide more detailed guidance on specific topics. This will take the form of horizon scanning and further literature review: a brief scoping review will be followed by discussion at the ScienƟfic Advisory Group (SAG) meeting (24/11/17). Our iniƟal search (google scholar, forward/backward citaƟons) will focus on: (a) New approaches/progress since previous guidance; (b) CriƟcisms of exisƟng guidance; (c) Other gaps. Prior to the SAG meeƟng we will develop a brief list of topics to address in the updated guidance. We will present these topics to the SAG (by sending a summary in advance and by presentaƟon of an overview at the meeƟng). Following discussion at the SAG meeting we will finalise a list of topics that we will explore in more depth. Expert workshop: The findings from the gap analysis will inform the agenda for an expert workshop to be held in early 2018. Each of the 'topics/themes' idenƟfied for update should be represented (by an expert) at the workshop i.e. we will invite experts based on these themes. The aim of the expert workshop is to achieve consensus on topics that should be newly covered or updated by the new guidance and as a basis for the project team, along with the rest of the authorship group, to produce the updates and addiƟons. Follow-up consultaƟon (email and a consensus meeting) will be used to achieve consensus on the details of the updated guidance.

(b) WriƟng group:
The wriƟng group will meet on a more regular basis (tbc), be engaged in the progress of the guidance and contribute to the wriƟng process. Members of the wriƟng group will be acknowledged as authors on the final guidance. The project team will be members of the wriƟng group, with addiƟonal authors idenƟfied as appropriate, e.g. to fill gaps in expertise. Q5. Figure 3. This figure relates to the concept of using different research perspectives (as noted in Q3). Does Q7. Signposting to other guidance: We are keen to signpost to further guidance where appropriate. If we have missed any specific guidance that may be helpful to readers can you please note it here, thank you. [TEXT BOX HERE] Q8. Clarity: Was anything in the guidance particularly unclear (please elaborate)? [TEXT BOX HERE] Q9. Case studies: We are collating case studies to include in the guidance. We would welcome suggestions in the following areas: l Modifying interventions to improve their intervention design and/or evaluation design l The use of programme theory throughout the research process l Involvement of stakeholders throughout the research process l Economic considerations throughout the research process l The exploration and use of context throughout the research process l Addressing uncertainties throughout the research process l Complexity informed approaches to intervention research Q10. Other feedback: If you have other comments that may be useful as we finalise the guidance, please provide brief details [TEXT BOX HERE] ¢ Version 6.1 of the Cochrane Handbook includes new chapters on intervention complexity, qualitative evidence, network meta-analysis and synthesising findings using alternative methods to meta-analysis. 344 ¢ PRISMA-CI is an extension of PRISMA, the accepted standard for encouraging consistency and transparency in systematic reviews. 354 PRISMA-CI incorporates issues specific to complex interventions, alongside an 'explanation and elaboration' guideline to support its use. 355

Phase-specific considerations
Developing interventionshave you used a formal framework (such as INDEX) to guide development of the intervention?
Identifying interventionsfor policy and practice interventions, have you performed an evaluability assessment to determine whether or not and how an evaluation should be undertaken?
Feasibilityhave you defined and used clear progression criteria to guide decisions about whether to proceed to an evaluation study?
Evaluationhave you chosen an appropriate study design to answer the research questions and provide robust evidence to inform decision-making about further intervention refinement, evaluation or implementation?
Implementationhave constraints and enablers of implementation been considered at all phases, from intervention development, through feasibility and effectiveness testing, to large-scale roll-out?