Involving carer advisors in evidence synthesis to improve carers’ mental health during end-of-life home care: co-production during COVID-19 remote working

Background: Family carers play a central role in supporting people at the end of life, but often suffer detrimental impacts on their own mental health as a result. This project conducted evidence synthesis of

• NIHR RfPB PB-PG-0614-34070: Peer support to maintain psychological wellbeing in people with advanced cancer: a feasibility study for a randomised controlled trial (CI Walshe) £249,767, 36 months, start November 2015.

Christine Rowland
Funded post • NIHR Greater Manchester Research Design Service: part-time funded front-line advisor.Start December 2019.

Danielle Harris
Funded post • NIHR ARC NIHR200174: Research Assistant NIHR Applied Research Collaboration Greater Manchester Healthy Ageing Theme.

Fellowships
• NIHR Career Development Fellowship, NIHR-CDF-2012-05-218: Trajectories in advanced chronic obstructive pulmonary disease (COPD): determining a new paradigm for the care and support of patients and their informal carers at the end of life (Living with Breathlessness study).£601,925, 48 months, start December 2012.

Grants
• NIHR ARC EoE NIHR200177: NIHR Applied Research Collaboration East of England (CI Jones), £9,000,000 (£225,103 palliative and end-of-life care), 60 months, start October 2019.Introduction P atient and public involvement (PPI) in research has been defined as 'research being carried out "with" or "by" members of the public rather than "to", "about" or "for" them'. 1 The importance of involving patients, carers and members of the public in health research has long been recognised. 2][4] This brings different knowledge and expertise which can lead to greater insights than would be possible without PPI.Further, there is an ethical argument that people have a right to have a say in research that is likely to affect them and their care. 2,4tient and public involvement in systematic reviews is important to ensure their relevance and meaning both to the people affected by a health condition or context and to people using the reviews to inform health policy and practice. 5Benefits again include insights from first-hand experience, researchers and PPI members learning from each other, and improved relevance, communication and dissemination of findings. 6PPI members may also gain new skills and confidence. 7,8is paper reports on PPI work with a carer Review Advisory Panel (RAP) in a metasynthesis of research on what might lead to, or prevent, poor mental health (e.g.anxiety, depression, distress) in carers who look after a family member or friend who is at the end of life (EOL).Stage 1 of the project involved separate syntheses of the relevant qualitative, 9 observational quantitative 10 and intervention literature, integration of these reviews within a common framework, 11 and creation of draft project outputs.Stage 2 involved one-off workshops and focus groups with key stakeholders (carers, patients, practitioners, commissioners, policy-makers) to help tailor contents and presentation of outputs to each stakeholder group.
The carer RAP provided PPI throughout stages 1 and 2. The RAP PPI was a secondary focus of the project, albeit a very important one where continuous, meaningful involvement was sought.The RAP planned role in stage 1 included a review of project evidence and findings to see if they made sense to carers, how they could be meaningfully presented and combined into a common framework, and whether there were gaps to be addressed in further research; in stage 2 they advised on consultation and dissemination.However, the role evolved over time as the RAP took more control over their contributions to produce carer-led recommendations from project findings.These recommendations became an important output in their own right alongside review findings.
Review Advisory Panel involvement was therefore initially limited to the roles of 'contributing', where there is indirect influence over the review process, and 'influencing', where there is direct influence on the review but without direct control over decisions or the process. 5Alternatively, the role could be summarised as 'consultation'. 1,2However, the aim was to truly listen and be responsive to carers' views, even if this took the project in new directions.The project therefore provided what Knowles et al. 3 refer to as 'space to talk' and 'space to change'.'Space to talk' meant enabling shared dialogue and recognising carer and researcher expertise as equally valuable.'Space to change' allowed for adaptation in response to carer feedback, both to the study and the way of working together. 33]12 This evolution took place within the constraints imposed by the COVID pandemic, where involvement had to be adapted from face-to-face to remote working.This paper aims to add to the limited literature on PPI in systematic reviews by reporting lessons learnt for creating space for genuine co-production within a tightly defined literature review project and achieving meaningful PPI in the context of remote working.It addresses a recognised need for more transparent and critical reporting of PPI in research to progress beyond separate case studies INTRODUCTION and capture the impacts of PPI, improve the evidence base, encourage innovation and enable robust appraisal. 4,8The Guidance for Reporting Involvement of Patients and the Public (GRIPP2) reporting checklists have been developed to help address this. 13As PPI was a secondary focus of the project, our reporting is guided by the GRIPP2 Short Form. 13e aims of the project overall were to help improve mental health among carers during EOL through evidence synthesis of factors related to carer mental health and translation of findings into accessible information for key stakeholders, to better target future efforts to improve carers' mental health.The aims of PPI were to ensure study findings and outputs were relevant and accessible to carers themselves and reflected a carer perspective.
The aims of this paper are to present principles and components that facilitated successful PPI in an evidence synthesis project to help inform PPI in similar projects. 13This includes presentation of: • methods, in the form of the principles and processes applied to foster good PPI in the project • results, in the form of outcomes from PPI in the study, particularly carer recommendations initiated and produced by the RAP • reflections by carers and researchers on what worked well and less well • discussion, including critical reflection on principles, remote working, PPI resourcing and limitations.
Both open and targeted recruitment strategies were employed 5 to ensure wide reach.The project was advertised widely through Twitter with links to project webpage information.Further, local carer support groups were identified and were e-mailed flyers and further information.In a second recruitment drive, KB followed up with additional e-mails and phone calls, and located and contacted additional relevant groups.Finally, potential candidates known to JF were contacted directly. 1,5 help ascertain eligibility and ensure equitable access and diversity in the composition of the RAP, all candidates completed a brief application form, through e-mail, paper document or online survey.The project aimed to recruit six RAP members who varied in terms of age, sex, ethnicity, experiences of caring for patients with different conditions, and experience of PPI. 1 A selection process protocol was set up to shortlist applicants and conduct interviews to identify the final RAP membership.However, as there was only a small number of applicants (eight), KB conducted informal telephone conversations with all to check their understanding of the project and ability to commit to regular meetings, and to gain information on their demographics, background of caring and PPI.This information was anonymised, tabulated and mapped to the recruitment criteria to identify the candidates who represented the widest range of carer experiences; for example, five of the candidates were men, of whom three were selected to ensure gender balance.The resulting group had balanced numbers of males and females, ethnic minority community representation, and included carers of people with cancer or non-cancer diagnoses, with long-term or shorter-term experience of caregiving, and ranging from extensive PPI experience to none.
Pre-RAP preparation and relationship-building was facilitated by KB and TS through individual faceto-face meetings with the selected RAP members to explain what was required of the role in more detail 1,5,14 and give carers an opportunity to tell their story.This proved extremely beneficial in building researcher-carer relationships, so carers felt comfortable and able to contribute when attending their first RAP meeting.Unfortunately, the carer from the ethnic minority community stepped back before the first RAP meeting due to personal circumstances, leaving a total of five RAP members.A second carer stepped back towards the end of the project, as the role evolved to be more demanding, but was still able to make a substantial contribution.

Meeting principles, formats and processes
The RAP met nine times in total.Table 1 provides an outline of the meetings and their content.
Below we outline the principles that guided the meetings (see Box 1 for summary). Copyright

Starting-up principles
The first face-to-face meeting (3 hours) was essential in laying the foundations for the RAP collaboration and building trust.It focused on the core principles of (1) relationship-building, (2) agreeing how to work together, (3) clarifying expectations and (4) training. 1,5,14First, time and space to socialise and share experiences were factored in at the beginning, during and at the end of the meeting.Second, identifying ways of working together was the first core agenda item, with both researchers and carers contributing to generating a list.This included keeping personal information confidential, treating others with respect and establishing that 'there are no silly questions' (see Box 2 for the agreed list).Third, the group discussed what the RAP carers could expect from the project itself (its content and aims), and what they could expect from the research team.This included a good working relationship based on mutual trust and respect, clear information on what the team were asking of them, feedback on how their contribution was making a difference, and a contact outside meetings for queries and support.Then, to help define carers' role within the project, the researchers presented general principles of PPI and asked carers to discuss informally (during the break) what they expected to gain from the project.The group then talked through what it was hoped the RAP could contribute to the project, including helping to make sense of the evidence, communicating findings to others and identifying problems with the project and potential solutions.Finally, introductory training in systematic reviews and qualitative and quantitative research was provided.A 'jargon buster' list was started to define academic terms that carers were unsure of. 1,14X 2 Ground rules for ways of working 1.Any personal information is confidential and will not be shared outside the group 2. There are no silly questions 3. Everyone is valued 4. Observe personal boundaries 5. Everyone will be respected when talking 6. Deal with differences of opinion or conflicts in a calm and professional way 7.No racist, homophobic or sexist comments 8.All jargon will be explained (unfamiliar words to be added to a 'jargon buster' dictionary) 9.The agenda is used only as a guide to allow discussions to flow NIHR Journals Library www.journalslibrary.nihr.ac.uk

Ongoing meeting principles
To ensure carer voices were heard and valued, the project focused on the composition and chairing of the RAP meetings and on providing space to follow the carers' agenda.It also followed recommended principles for appropriate payment, training, preparation and communication. 1,5

Review Advisory Panel meeting composition: carer majority and researcher commitment
Review Advisory Panel meetings comprised five or six carer members, including the Chair, and three researchers.The aim was that carers should be regarded as partners and should feel free to express their views and feel valued.To facilitate this, carers formed the majority group and researchers the minority group at meetings, so carers could more easily support each other and express their views.However, to highlight the importance of the RAP to the project, there was substantial researcher representation, with the Principal Investigator always one of the researchers present.

The Review Advisory Panel Chair should be a carer
It was considered crucial to have a carer chair the RAP, rather than a researcher.This gave a signal that carers' views were at the fore and that their input was valued equally to that of the researchers.However, it was also important to ensure this carer had chairing experience and felt confident and comfortable in the role, and to help ensure meetings were run efficiently and effectively.Consequently, JF was asked whether she was willing to take on the chairing, given her familiarity with the project as the carer representative on the project Research Management Group (RMG) and her previous PPI experience.JF agreed, but requested the role be properly defined so there was clarity regarding the expectations of her role as RAP Chair and how this would be combined with her RMG role.A member of the research team liaised with the Chair before meetings to review each meeting's aims and how best to conduct them and to ensure the Chair was up to date with project developments.During meetings, the Chair ensured carers' views remained at the fore and that all carer members were encouraged to participate in discussions. 1

Space to follow carers' agenda
Although all meetings had a clear agenda, this served as a guide rather than a fixed structure for meetings, to encourage RAP members to express their perspectives freely and allow discussion to flow productively.This facilitated the principles of 'space to talk' and 'space to change'. 3 If carers began to explore aspects outside the scheduled agenda or materials, the meeting would follow these discussions through and then consider how they informed the project as a whole.This might only involve changes to project materials, but sometimes it had implications for future PPI work, which was then flagged and negotiated (see Results).Having a clear agenda, however, helped ensure core scheduled items were still returned to and addressed in the course of the meetings.

Proper and prompt payment
Project policy was to pay RAP members for time spent at the RAP meetings and time preparing for meetings, including analysis. 1,5,15However, RAP members decided themselves whether they wanted payment or not, and whether in money or voucher form.Members were also reimbursed for any travel costs.The initial aim was to pay participants in cash at the end of each meeting, so there would be no delay in payment. 3However, COVID and a move to online working meant this was only possible for the first and only face-to-face meeting.Payment thereafter was via bank transfer managed through the university claim form system.
Initially, higher pay rates were applied for meeting attendance than for meeting preparation.However, as RAP members were set tasks prior to meetings, or at least were required to engage closely with meeting materials, the effort involved in meeting preparation became more similar to that of the meetings themselves.The project therefore changed to a flat, high rate for all hours spent and an added a £5 fee per online meeting to cover members' internet connection. 15opyright © 2023 Grande et al.This work was produced by Grande et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care.This is an Open Access publication distributed under the terms of the Creative Commons Attribution CC BY 4.0 licence, which permits unrestricted use, distribution, reproduction and adaptation in any medium and for any purpose provided that it is properly attributed.See: https://creativecommons.org/licenses/by/4.0/.For attribution the title, original author(s), the publication source -NIHR Journals Library, and the DOI of the publication must be cited.

Enabling Review Advisory Panel members to prepare
Before each meeting, it was important that RAP members knew what they were being asked to do and what the meeting agenda and contents were, and that they had time to consider the meeting materials. 1,7Meeting documents and slides were therefore sent well in advance, in each RAP member's preferred format (electronic or paper), allowing them at least a week to review materials and prepare.

Ongoing training
Training was an ongoing component of meetings. 1,6,7The training included general introductions to health research, systematic reviews and qualitative and quantitative research, accompanied by more in-depth coverage of qualitative reviews, qualitative analysis techniques and coding of data with illustrative template(s), and similarly, quantitative review processes and presentation of numerical findings.Meetings covered both new materials and repetition of training related to previous meetings as required, to ensure members were familiar with or 'refreshed' regarding methods and concepts to be discussed (see Table 1 for contents of each meeting).Templates with examples were provided for tasks.Questions were always welcomed and efforts were made to avoid jargon; 7 any new terms were added to the list of 'jargon buster' definitions.

Ongoing communication
The project followed the following principles for good communication: 5,7 (1) maintaining proactive contact with RAP members by e-mail so they felt supported, rather than wait for them to come to researchers with questions; (2) ensuring RAP feedback had been properly understood, by writing up and circulating feedback to ensure it captured what carers wanted to say; (3) providing an alternative channel for raising concerns, by encouraging RAP members to contact the Chair after meetings regarding any concerns or suggestions for improvement, which were fed back anonymously to researchers, as appropriate; (4) ensuring RAP members saw how their input made a difference, 6 through meeting updates, participation in stage 2 stakeholder meetings and a post-project meeting.

Adaptations in response to COVID
After the first face-to-face meeting, the project had to adapt to online meetings due to COVID restrictions.This was a challenge for some carers who had not used Zoom™ (Zoom Video Communications, San Jose, CA, USA) before or who had to borrow a computer to join the group.KB tested Zoom with each member before the first online meeting to ensure everyone would be able to link up; however, one member was unable to activate their camera.As Zoom meetings can be more taxing than those conducted face to face, the meetings had to become shorter (2 hours maximum, including a break) and were therefore also more structured, with more emphasis on pre-meeting preparation.However, flexibility to let meetings follow the flow of the group discussion was retained, and the group continued to work well.

Review Advisory Panel involvement as planned
The RAP made the following contributions set out in the original project aims: reviewing the search strategy; contributing to qualitative review analysis; reviewing findings and helping shape the presentation of findings; considering how findings may fit together within a comprehensive framework; supporting the stage 2 stakeholder consultation; and reviewing final project products (dissemination materials and reporting).Some further details are outlined below.
Importantly, the RAP ensured the content of the synthesis was meaningful.They were central in ensuring that the qualitative literature synthesis identified factors associated with carer mental health and grouped them into themes that were meaningful from carers' perspective. 9The main themes were 'patient condition', 'impact of caring responsibilities', 'finances', 'relationships', 'carer internal processes' and 'support'.Findings from the subsequent quantitative observational and intervention syntheses were then mapped onto these themes (with an added theme of 'contextual factors').The RAP reviewed whether the subthemes to these main themes made sense to carers and whether anything was missing.One clear gap identified by the RAP was the lack of consideration of the impact of financial factors on carer well-being within the quantitative literature. 10Additional missing components were: identifying as a 'carer', loss of own identity and limited consideration of ethnicity.Further, while the quantitative literature labelled some coping strategies as 'dysfunctional' or maladaptive, RAP members considered these as potentially helpful strategies in some circumstances, in the moment and in the context of an individual's caregiving situation.Rather than labelling coping strategies as 'dysfunctional' or 'functional', it was felt that these should be judgement-free and simply presented as coping strategies found to be associated with 'better', 'worse' or 'no change' to carer mental health within the literature. 10e RAP also advised on the presentation format of the findings.This included presentation of numerical data using colour-coded columns to highlight the number of investigations showing that a factor was associated with worse carer mental health (red) or better carer mental health (green), or showed no association (neutral colour). 10Further, RAP members advised on how the total body of evidence could be organised into different levels of detail to enable stakeholders to first gain an overview of findings and then find the detailed content relevant to them (further detail will be reported elsewhere).
Finally, the RAP contributed to stage 2, where 'products' from syntheses were presented to other stakeholders (additional carers, patients, practitioners, commissioners, policy-makers) for feedback.The RAP advised on how to involve additional carers.Further, five RAP members, including the Chair, wanted to be involved in the stage 2 workshops and discussion groups, either as active participants or as observers.Four participated in the four online workshops with carers, practitioners and commissioners/ policy-makers as participant (the Chair) or observers, and four RAP members each participated in one of two online discussion groups with GPs and carers.Further details about the stage 2 consultation and products will be reported elsewhere (https://www.arc-gm.nihr.ac.uk/carer-project-).Three RAP members agreed to feed back on the qualitative synthesis report, 9 and all members contributed to the current paper through participation in reflection on their RAP involvement and paper review.

Changes to the project plan
The 'space to talk' within the project introduced perspectives beyond the original plan which were then accommodated through 'space to change'. 3 This led to greater RAP involvement in the qualitative review analysis, using a simplified framework to organise findings, and most importantly, the addition of carer recommendations to the project 'products'.The RAP's wish to be involved in the analysis and to develop recommendations expanded the number of meetings from the planned six meetings to nine in total.The added resources required to cover additional RAP hours and project time were aided by cost savings in moving from face-to-face to online activities.We outline project changes further below.

Whole-group engagement in qualitative synthesis
The initial plan, introduced in meeting 1, was for the qualitative data extraction and analysis to be conducted by KB and one or two carer co-analysts.However, RAP members decided that they would all like to contribute to data extraction and/or analysis as a group.Three members of the group agreed to conduct data extraction, and then the group as a whole discussed the themes that emerged from the data.
To accommodate a change to whole-group analysis, KB delivered training on qualitative research methods in meeting 2 and recapped the systematic review training.The RAP members were given three papers (10% of the total) and asked to consider the results of the papers, identify sections that described things that affected carers' mental health, and write notes on the themes and subthemes within these.They were also given questions to help them think about themes and a template for recording thoughts on themes. 7,9All received the same papers, selected for maximum variation and clarity, and completed data extraction in their own time.KB collated the findings, and these were then discussed at meeting 3. The fourth RAP meeting continued to focus on the qualitative metasynthesis.KB presented the themes and subthemes she had identified from the three papers, and the group discussed similarities and differences between the themes identified by the carers, how the findings related to the carers' experiences, and what themes they believed were missing (see Bayliss et al.). 9

Simplified framework for organising findings
The project originally aimed to use existing psychological models of factors affecting carers' mental health [16][17][18] as a framework for the final organisation and presentation of findings for stakeholders in stage 2.However, work with the RAP on a potential updated model suggested that this may not aid presentation, and that a simple presentation of themes would be as helpful (and less confusing).In particular, RAP members felt the perspective on caregiving from models was too linear and did not reflect the dynamic nature of caregiving, the continuous interaction between themes, and the relationship and interaction between patient and carer.Further, they felt this perspective implied that negative mental health outcomes were inevitable despite strategies to improve mental health, and a more positive picture was required.What was important to communicate to other stakeholders was the themes themselves and evidence that they were likely to affect carer mental health.As the models were not seen as helpful in communication of findings, the project thereafter focused on presenting findings as meaningfully as possible for each theme, rather than pursing the models further.

Development of carer recommendations
The most important development was that the RAP decided they wanted to produce carer recommendations from the project.Originally, the project aim was solely to synthesise and present findings in ways that were meaningful to carers, and then use stakeholder engagement in stage 2 to help make the presentation of findings more accessible and useful to different stakeholder groups.Then it would be for stakeholders themselves to consider what actions may follow from findings (e.g. for practitioners, what direct support may be given in consultations; for commissioners, what services were important to fund; for policy-makers, what policy changes may help; for carer organisations, how best to target support of their carers).However, RAP members felt this left a notable gap in the project.They believed it was important to consider what findings told us about the support carers needed and for the RAP to produce recommendations on how to support carers that could be presented to other stakeholders alongside the findings themselves.
To enable this, a seventh RAP meeting in January 2021 included discussion of how project findings could be used to develop recommendations for improving carers' mental health, and some initial recommendations emerged at this meeting.Next, an additional February meeting was held to develop further recommendations.Before this meeting, RAP members requested a recap of total project findings and reflected on what actions could be taken within each theme to improve mental health for carers.Questions to consider included: what factors could or may be changed, what would be considered as priorities for change, how services could be changed (gold standard/minimum standard), and how factors that cannot be changed could help to identify carers who need added support.The RAP feedback, from both the January and February meetings, was synthesised (by TS) and reviewed by RAP members and the research team via e-mail.
Table 2 presents the final RAP-developed recommendations to support carers to maintain or improve their mental health.These are divided into 'overarching principles' which refer to recommendations that extended across all seven project themes, and 'theme-specific' recommendations that seemed to fit with individual themes from the research evidence (see Report Supplementary Material 1 for full details; further summaries and resources can be found on the project website: https://arc-gm.nihr.ac.uk/ the-carer-project-outputs).
The RAP highlighted that these recommendations should be seen within the context of key considerations for carers overall (see Report Supplementary Material 1).Firstly, that recommendations were based on making it as easy as possible for the carer to fulfil their role and 'removing obstacles which might stop them being a better carer', including removing or reducing unnecessary demands on their time to allow them more time to look after the person cared for.The RAP noted that it may have significant impact on a carer's mental health if they felt they 'might not be offering the optimum care'.Secondly, the RAP stressed the importance of carers feeling empowered, having their voices heard, and having proper engagement with services as active participants within their caregiving role.Further, recommendations focused on overcoming common challenges experienced by EOL carers, which may help to support the mental health of the majority; however, it should be recognised that experiences of individual carers will vary and that 'one size does not fit all'.The RAP also recognised the need to be pragmatic in their recommendations based on the limits of what services can provide.Recommendations therefore focused on what may realistically be changed, rather than representing an ideal 'gold standard'.The RAP mainly focused on developing recommendations that spanned themes and reflected the whole of the EOL caregiving journey (including post bereavement), as trying to fit recommendations to the seven specific themes from the research evidence was often difficult.F eedback was sought from RAP members and researchers to gain insights into their perspectives on RAP involvement.An independent research project examining RAP involvement was not part of the original project protocol, but it was considered important to gain feedback from those involved to reflect on lessons learnt to inform PPI in future reviews.As such, this should be considered a collective reflective piece, rather than a research-based evaluation, as the RAP members and researchers who provided the feedback are also the authors of this paper.

Method
Feedback was collected from RAP members and researchers via e-mail, using open-ended questions and written responses.Box 3 shows the questions asked.These were distributed to all RAP members and Core Research Team (CRT) members.Respondents were: all final RAP members, including the Chair (n = 5); CRT members who worked directly with the RAP (n = 4/5); and CRT members not directly involved with the RAP (n = 2/4).In the findings reported below, each respondent is allocated the ID codes RAP1-5, CRT1-4 and CRT 5-6, respectively.

BOX 3 Reflection questions
Questions for RAP members: 1. Briefly, what is your prior experience of patient and public involvement (PPI)? 2. What were your expectations of being involved in this project?3. How did the experience of being involved match up to your expectations?4. What did you gain? 5. What did you give? 6.What worked well? 7. What didn't work so well?8. What would you say to other people, like yourself, about working on a similar project?9. What were the challenges of completing this project during the COVID-19 pandemic/how did you cope with it?Questions for CRT members who worked directly with the RAP: 1.Briefly, what is your prior experience of patient and public involvement and engagement (PPIE)? 2. What were your expectations for working with a carer RAP on this project?3. What have been the gains for this project of the RAP involvement?4. What have you gained or learnt from the RAP involvement personally or in your work? 5. Were there any challenges that you faced or had to solve of working with a RAP in general?6.What were the challenges of working with a RAP during the COVID-19 pandemic/how did you solve it?
Questions for the rest of the CRT: 1. What have been benefits for this project that have resulted from working with a carer RAP? 2. Have there been any challenges for this project when involving a carer RAP? KB and GG initially reviewed the collated materials from RAP and researcher respondents.Based on this, it was agreed that the material could broadly be organised into a framework of what made PPI work well or not so well.GG then worked through materials, writing freehand notes on the contents and then considered how the contents fitted the framework.Comments were next indexed according to the broad themes of what made the RAP involvement work and what were challenges.All materials relevant to a theme were collated for further in-depth scrutiny of any subthemes and how carer and researcher views compared or contrasted.The comments and resulting themes were reviewed in full by CR, and revisions made.Finally, all researchers and RAP members who had commented reviewed the resulting draft document to assess whether this had captured their perspectives.Thereafter, some further RAP feedback was incorporated in the final text and final adjustments made to subthemes.

What made patient and public involvement work well
Reflections mainly focused on what made the PPI work well: a shared sense of gains to the project from PPI; personal gain; mutual commitment and respect; and meaningful bridging between academic and lay perspectives.

Gains to the project from patient and public involvement
Review Advisory Panel members and researchers seemed to have a shared sense of the purpose of the RAP involvement, which is likely to have helped them work together effectively.
All the RAP members hoped that their involvement could contribute insights into carers' experience and make a difference in supporting other carers.

To share what I learned from being an end-of-life carer for my late wife with a view to improving carer experiences. RAP1 I thought my experiences may prove of some use for other carers. RAP2
Some also indicated the value of PPI in 'shaking up' the agenda or focus of researchers and clinicians.

… to be involved in research which had the potential to affect real people's lives rather than just project research papers for clinicians and academics. RAP5
We will never find the right answers until we find out what the problems really are, not what professionals think they might or should be.

RAP3
The researchers' comments mirrored RAP members' perspectives.Firstly, in the value of having carers' experience informing the project: … the RAP would be able to offer rich insights and perspectives of the carer experience and offer context to research findings using their lived experience.CRT4 Secondly, how carers' involvement would help the project achieve its aim of making a difference to carers: It is important that the findings make sense from a carers' perspective and resonate with their experience.

If the project loses touch with carers' reality, it is less likely to achieve its aims. CRT2
Finally, how the RAP may be able to challenge existing research and provide new insights: … some of the key findings from the RAP group, such as financial difficulties and stressors, were interesting to see considering many of the trials and observational studies had not assessed these outcomes.CRT6 I expected it to be a valuable contribution that would take us out of some of our academic ways of thinking and give us new insights.CRT2 … the recommendations were not on anyone's radar at the start of the project, I don't think, but are such a valuable output.CRT3 Researchers also noted the positives of being challenged by the RAP within the project: It was also a nice way to be held accountable, too, in some ways.CRT3

Personal gain
Comments also reflected a sense of personal gain from PPI involvement, which may have sustained the commitment to PPI by both parties.
All the RAP members stressed learning as a personal gain.This included learning about other carers' experiences:

I also realised the various dynamics at play in caring, the varying levels of intensity that I suppose must exist … RAP2
A deeper understanding of the issues experienced across different carers.RAP4 However, learning about research or technical aspects was also mentioned:

Gained some helpful insights into the preparation of a research paper. RAP5 … I have a little more understanding of how qualitative research is conducted and hope to use this as a base to improve my knowledge in the future. RAP3
Review Advisory Panel members also mentioned gaining from increased self-awareness, meeting people or having a sense of purpose through the COVID pandemic: … I realised all the anxiety, stress, anger and guilt I had encountered over those seven years… RAP2 Met some interesting people.RAP5 I felt useful during a time that most people were finding challenging.RAP3 While researchers said less about personal gain from PPI, they noted a gain in learning, in terms of both understanding carers: I have gained a greater understanding of how it feels to care for those at the end of life that cannot be gained by reading papers.CRT1 NIHR Journals Library www.journalslibrary.nihr.ac.uk

CARERS' AND RESEARCHERS' REFLECTIONS ON REVIEW ADVISORY PANEL INVOLVEMENT
and gaining insights into the overall benefits of PPI: As mostly a quantitative researcher, I have become more aware of the importance of PPI … CRT6

Mutual commitment and respect
Comments indicated that both researchers and RAP members were committed to the project and that this commitment was acknowledged by both parties: … We had a good group of engaged individuals who were determined to see their work through.RAP5 The amount of work the researchers did was amazing and I can only praise them for their professionalism.RAP2 In working together, there was clear mutual respect expressed by both groups: … we were lucky to have a very constructive, reflective and positive [RAP] group where everybody contributed and everyone was focused on the aims of the project, to make sure it would benefit carers.CRT2 It was a great (research) project; the team were so thoughtful and accommodating, and I think we have produced a high-quality product.RAP4 An integral component of respect for the RAP was that their views were taken seriously and that they had meaningful involvement: I felt we were involved quite significantly throughout the process and our opinion was given due consideration.RAP3 Payment was noted by RAP members as an important, tangible signal that RAP perspectives mattered: Being paid for your time and effort is for me a marker that consultation is being taken seriously and is not just a box-ticking exercise.I was very impressed that the university gave us the option of being taxed or not -that was very accommodating of them.RAP3

Bridging between academic and lay perspectives
Core to meaningful PPI involvement is bridging between academic and lay perspectives.This involved communicating research perspectives to carers, ensuring carers were able to comment meaningfully and have their perspectives heard, and negotiation and compromise between the two perspectives.
Enabling lay members to engage meaningfully with 'academic' materials can often pose a challenge and requires effort by both parties: I was slightly worried about how well they would be able to work with the large volumes of synthesis data we would be producing -it was going to be a difficult balance between maintaining a high standard of reporting true to the findings while making it accessible to a lay person.CRT3 Copyright © 2023 Grande et al.This work was produced by Grande et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care.This is an Open Access publication distributed under the terms of the Creative Commons Attribution CC BY 4.0 licence, which permits unrestricted use, distribution, reproduction and adaptation in any medium and for any purpose provided that it is properly attributed.See: https://creativecommons.org/licenses/by/4.0/.For attribution the title, original author(s), the publication source -NIHR Journals Library, and the DOI of the publication must be cited.

… The arrangement of the themes was quite complicated and it took me a few meetings to get the hang of it. This was clearly the bulk of the work, and the researchers had no doubt spent many hours working out how to best present the evidence, but it was a lot of information to assimilate at once … RAP3
The investment in training, time and guidance was an important means to overcome this: Developing training that would be accessible and delivered in plain English, identifying support and training needs from the start ….Providing enough time and resources for the carers to be meaningfully involved … [and for analysis] Provide an example of how to feed back on a thematic analysis using a template, as lay members can vary greatly in how they interpret a task and provide feedback.CRT1 Practical details also mattered in ensuring the RAP had time and opportunity to prepare feedback: We were given copies of slides and documents in plenty of time for the next meeting, in digital and paper form, so we always had an opportunity to study them at leisure.Paper documents are important if you need to refer to them during a meeting, and printing them off yourself can be quite expensive if they are very colourful … RAP3 Further, it was central that RAP members felt able to express their perspectives to researchers during meetings.Crucial to this was a feeling that carers and researchers were equal partners.
I think the project worked because of the mutual respect between the RAP and researchers, people were listened to and ideas were debated and modified if necessary.RAP2 Obviously carers were usually in larger numbers in the RAP meetings, but I believed that irrespective of that, each individual (whether they be a researcher or carer) were on an equal par … RAP5 This may have been facilitated by having a carer as chair: To help the carers feel more comfortable to express their views, we had a lay Chair.CRT1 However, core to making this work was a willingness by both parties to value different forms of knowledge and engage in negotiation and compromise in straddling the academic and lay worlds.Again, the role of the Chair may have formed an important part of bridging between groups: I think she probably made a big difference to the project because of her contacts, her previous experience of this sort of work, and her outgoing personality, and wonder whether all projects need someone like her to straddle the Core Research Team and the public patient involvement and to lead the PPI.CRT5

What made patient and public involvement work less well
The main issue voiced was the impact of COVID on communication, in both the move to online working and ongoing communication.Management of PPI outside the RAP context may also have been improved.

COVID impact on communication
Verbal, direct group communication was essential to the project.However, moving from face-to-face to online Zoom meetings due to COVID posed a serious challenge to this communication.In addition to challenges of coping with the technology, online working led to reduced informality, flexibility and personal connection: … Zoom meetings had to replace classroom meetings during the pandemic.I thought this was a loss to the project's dynamic; nothing can replace personal interaction, a thought or an idea could arise over a coffee or an informal chat at break time, it's these little yet important things that can't be done over Zoom.RAP2 … it's not quite the same as meeting in person where you can gain more insight about a person and build a stronger connection.CRT4 Some of the above may have been ameliorated by utilising additional Zoom options: Maybe more ongoing 'social' type meetings of some sort in breakout rooms might have been useful to cement further the RAP together.CRT3 However, these would have required more time commitment, and managing the technology for ordinary Zoom meetings was already challenging for some RAP members.Further, the reduction in detection of non-verbal communication was an important online challenge that would still be hard to mitigate: … it is hard to judge atmosphere remotely.When you are talking about very sensitive matters it is harder to show empathy/sympathy from a distance.RAP5 … it's hard to get the balance right and make sure that everyone gets the chance to contribute … as there are less apparent cues which you can pick up on from people's body language to signal when someone is wanting to jump in and say something ….I think also given the sensitive nature of the subject of this research, working remotely makes it harder to read if people are struggling.CRT4 COVID-induced home working also meant research team members were less able to follow up RAP communication outside meetings as proactively as they would under normal circumstances due to considerable stresses in juggling work and family roles:

This was extremely stressful as I felt torn between two very different roles of mother and researcher. It also affected how much time I could dedicate to regularly contacting carers. CRT1
Being able to have a first meeting face-to-face made a big difference in mitigating some of the challenges of effective PPI through distance working, online meetings and e-mails, particularly when the issue was a sensitive one: Building relationships and regular contact with RAP members are central to a successful working relationship.Luckily we were able to meet face to face, one to one, and as a group before the lockdown.This helped to build rapport.CRT1

Management of patient and public involvement outside the Review Advisory Panel context
While the role of the RAP Chair worked well in the context of the RAP itself, the RAP Chair was also a participant in RMG meetings.Here the Chair's role on the RMG may have been better defined, as there was an element of her perhaps feeling like an onlooker listening to specialists.
… I always felt a bit on the outside.My main purpose was supposedly to provide PPI/PPE/lay input but this felt like an add-on on occasions, and we did have a member(s) on the [RMG] who were experts in PPI/ PPE who probably felt more expert in this area … RAP5 This highlights the importance of clarifying and negotiating how PPI fits in across the processes and management structure of the project as a whole.

Discussion
T his paper seeks to add to the limited literature on PPI in evidence synthesis by reporting principles and components that appeared to facilitate successful PPI, to inform future projects. 4,8This includes an evolution from PPI consultation to co-production, whereas most systematic reviews remain at the level of consultation. 2Further, it illustrates how meaningful co-production can be achieved during remote working, indicating the viability of hybrid ways or working.However, it also highlights the time and resources required for meaningful PPI.

Principles
The project ensured it created a strong, viable carer RAP whose perspectives could be properly expressed: firstly, by investing considerable time on recruitment processes to ensure an eligible, diverse and balanced group; 1,5,14 secondly, through the composition of RAP meetings, with carers in the majority for mutual support; and thirdly, through a Chair who combined carer and chairing experience, providing both an understanding of the carer perspective and the confidence to bring it to the fore, while nurturing 'novice' PPI members.Being an 'expert PPI member' also helped the Chair facilitate bridging of researcher and carer perspectives.
Further, the project signalled the value placed on PPI from the outset: firstly, through prompt and appropriate payment, 1,3 which demonstrated the tangible value ascribed to PPI (RAP feedback noted this as important); secondly, through strong and senior researcher representation at RAP meetings; and finally, by facilitating responsiveness to RAP input by providing 'space to talk' outside the agenda structure, and 'space to change', 1,5 which led to substantial project changes.Additional principles of ongoing training, proper preparation and ongoing communication 1,6,14 further ensured the RAP could contribute actively.This enabled meaningful PPI and movement from mere consultation to the co-production of RAP carer recommendations based on findings (see Table 2).These were presented to the broader stakeholder groups in stage 2 alongside evidence synthesis findings, and perceived to be valuable in their own right (stage 2 will be reported in a future report).
Joint researcher and carer reflection indicated components underpinning this successful PPI.Firstly, there was a shared sense of the purpose and gains from RAP involvement.This may have been facilitated by individual pre-project meetings and 'foundations' laid in meeting 1.Secondly, there was mutual commitment and respect, possibly facilitated by early signalling of how carers were valued, clarification of project aims, and agreement of how to work together.Finally, there was meaningful bridging between academic and lay perspectives, based on sound principles of training and preparation, creation of a sense of equal partnership, and willingness by both parties to negotiate and compromise.Remote working enforced by COVID was perceived to hamper meeting communication, but there was sufficient resilience in the carer-researcher collaboration to overcome this.
The project applied principles drawn from existing literature. 1,3,5-7,14,15Additionally, it carefully considered participant composition at meetings and use of the Chair to foster meaningful involvement, which has been given less attention, although Buck et al. 14 recommended using larger PPI groups, rather than single members.We cannot know to what extent successful project PPI was down to principles employed, as opposed to the characteristics of the participants (researchers and carers).However, it may be telling that the PPI input on the RMG worked less well, where some of the same participants were involved (Chair and research team), but there was less preparation and prior definition of roles and only one PPI member.Nevertheless, it is not a given that success would follow from mechanistic application of PPI principles in the absence of positive, constructive attitudes and a willingness to compromise and change.
NIHR Journals Library www.journalslibrary.nihr.ac.ukDISCUSSION Knowles et al. 3 concluded that relational openness was more important to build trust for successful PPI than any particular methods used.However, the above principles are probably an important bedrock on which such openness and trust can be built.

Hybrid working
Our project indicates that meaningful co-production can be achieved through a hybrid between faceto-face and remote working.COVID provided an enforced 'opportunity' to test how project PPI could be maintained through online video meetings.Virtual meetings led to reduction in informality, flexibility, personal connection and non-verbal communication, compared to face-to-face communication.However, early face-to-face groundwork, including relationship-building, agreement of how to work together and understanding of each other's roles, appeared to enable productive collaboration to continue remotely.
Online working also made meetings logistically easier to arrange and to attend, particularly for those more geographically distant (although the meeting preparation required from researchers and carers may actually have increased).This may have facilitated the consistently solid attendance across meetings.Research indicates that remote working increases inclusiveness in terms of geography, 7,19 and also race, ethnicity, education, income, 19 and accessibility for those affected by ill health or caregiving. 20reviously, remote working may often have been limited to electronic communication through online forums 20 or e-mail, 7 with tele-and videoconferencing being relatively costly, limiting opportunity for dynamic interaction.However, online virtual meetings have now become a more viable, less costly option through greater access to online video meeting platforms, laptops and mobiles.While face-toface working is still preferred 7,14 and probably necessary for relationship-building, further research is warranted on how and when remote working provides a viable alternative.

Resourcing meaningful patient and public involvement
Adequate resources need to be built into PPI budgets from the outset to achieve meaningful PPI. 2,5This includes proper investment in recruitment and participant preparation 2 and ample time and resources for meetings to permit dynamic interaction and collaboration.In particular, allowing for 'space to talk' and 'space to change' 3 to achieve genuine co-production has resource implications.Accommodating change in our project not only required researchers' willingness to be flexible and make adaptations and RAP members' willingness to take on extra work and involvement, it also required proper resourcing: first, to enable payment for additional RAP hours to accommodate changes, and second, to allow extended and protected project time for researchers to facilitate communication and develop or adapt materials.Here, COVID in fact helped facilitate more 'space' to make changes, as the time and costs saved by moving to virtual meetings (on travel, venues, catering) provided more flexibility to incorporate required adaptations.However, projects cannot generally rely on such serendipity.The ability to work iteratively and make changes in response to PPI input is central to co-production.Such flexibility therefore needs to be built into projects.However, structures of academic funding can currently make this difficult, 3 and more flexible funding models for PPI may need to be considered.
Going forward, future projects must budget properly for PPI participants' time and ability to participate to allow meaningful engagement.This includes time for preparation, travel, meeting hours and review, and reimbursement for travel, subsistence and/or internet connectivity.However, a major PPI cost that may be overlooked in budgets is adequate researcher time for PPI in the project schedule, including time for proper meeting preparation (including procedures, documents and training), responding to meeting feedback (including production of new/amended project materials, procedures and schedules), and ongoing communication and engagement (including gaining PPI review and feedback for amendments).Preparing for PPI (before PPI engagement formally commences) is also resource-intensive, but proved crucial to our project.However, this may pose a dilemma, as it should ideally take place before the start of the project proper, at which point funding may not be available.For example, we adopted a broad carer RAP recruitment strategy before project start, but it yielded few candidates, and would probably have been more effective had there been more resources for this; for example, to speak to carer group leads and present to group members.For longer-term longitudinal projects, it would also be beneficial to recruit and cost for a larger PPI group, to allow for potential dropout along the way.Additionally, building in 'space to change' would require PPI budgets that are realistic about costs, but where there is flexibility regarding details of how money should be spent, with allowance for viring between budget headings.Finally, meaningful PPI engagement for funded projects may be further aided by easier access to flexible PPI funds for preparing PPI before project start or accommodating unexpected, added work arising out of PPI feedback.

Limitations
A strength was the joint reflection of both researchers and carers on PPI in the project.However, it would have been beneficial to evaluate researcher and RAP experiences of PPI at each project stage rather than just at the end.This would have provided richer insights and enabled ongoing project improvements.However, the time and capacity for evaluation were limited within the tight timetable.The reflections from the RAP and researchers at the end of the project did provide some insight into their respective perspectives, but did not permit in-depth exploration.Further, this was a small set of respondents whose identities were known to analysts and could be guessed at by others.While group members (both RAP and researchers) were used to having open and frank discussion, this may still have made some respondents hold back on any negative feedback.The evaluation did not include a RAP member who stepped back towards the end of the project and who may also have had more mixed views of the experience.Finally, the analysis was conducted and written up by researchers who also had contributed responses, and reviewed by the group who had contributed responses as a whole.Although honest feedback and objective analysis was attempted, the conclusions may therefore be biased towards existing views.Researcher and RAP views therefore remain a piece of self-reflection, rather than an independent investigation.

Dissemination
Carer RAP recommendations and PPI work have been reported to stakeholder groups through project meetings, workshops and focus groups and will be disseminated via posters and leaflets, podcasts, webinars and the project website.Awareness of the findings will be raised via Twitter and through stakeholder networks of NIHR ARC GM and co-applicants.The current paper, reports for additional project components, and all project materials will be available through the project website https://www.arc-gm.nihr.ac.uk/carer-project-.

Equality, diversity and inclusion
Regarding the research itself, the evidence synthesis project ensured that carers' own perspectives were highlighted from the synthesis by making the qualitative review of carers' own experiences the central review, which then guided the synthesis of the quantitative reviews.Further, all reviews considered the ethnic composition of reviewed studies, and all reports presented any findings on ethnicity and noted the limited representation of ethnic minorities in the literature.The evidence synthesis was limited to Organisation for Economic Co-operation and Development (OECD) countries to enable comparison with countries whose health-care systems were likely to be comparable with the UK, and limited to publications in English and Scandinavian due to resource constraints, and this is noted as a limitation of NIHR Journals Library www.journalslibrary.nihr.ac.ukDISCUSSION the project.The work with the carer RAP and the stage 2 stakeholder consultation sought to ensure that project findings were presented in formats accessible to all carer groups and stakeholders.
Regarding the research team and other involvement, the project was shaped and guided by carers themselves, as described in this report.First, by having a carer co-applicant help develop the project; and second, by recruiting a strong and active carer RAP who helped shape the search strategy, conduct the analysis, interpret the results, and shape the presentation of project materials and dissemination, and whose carer recommendations became a key project outcome.Further, in the project's stage 2 stakeholder consultation, carers constituted nearly one-third of the meeting participants.Although there was no ethnic minority representation within the co-applicant team, the project was initially able to bring on board a carer advisor from an ethnic minority community group who was involved in RAP pre-meeting preparations, but had to withdraw before the first full RAP meeting proper.However, in the project's stage 2 stakeholder consultation one-tenth of the meeting participants were from ethnic minorities.
Copyright © 2023 Grande et al.This work was produced by Grande et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care.This is an Open Access publication distributed under the terms of the Creative Commons Attribution CC BY 4.0 licence, which permits unrestricted use, distribution, reproduction and adaptation in any medium and for any purpose provided that it is properly attributed.See: https://creativecommons.org/licenses/by/4.0/.For attribution the title, original author(s), the publication source -NIHR Journals Library, and the DOI of the publication must be cited.

Conclusion
T his paper outlines the principles and components that appeared to facilitate successful, meaningful PPI in this project, to help address the lack of information on what leads to meaningful PPI and inform PPI in other evidence synthesis projects. 5,8Important elements were the emphasis placed on preparation work, RAP meeting composition and enabling 'space to talk' and 'space to change'.This allowed RAP carer recommendations to emerge as a valuable and substantial project output.The recommendations highlight the importance of: awareness raising of and about carers; providing a carer 'road map'; bespoke carer support; accounting for carers' contributions in patient assessments; ensuring co-ordination; and dealing with practical matters, particularly finance.Importantly, the recommendations should have relevance for carers in a range of caregiving contexts (e.g.encompassing caregiving for short-term conditions with late diagnosis and short prognosis, through to long-term conditions such as dementia) and highlight the need for bespoke support based on holistic assessment of support needs.
While we have described principles that appeared to enable successful co-production in this project, these require testing in future work, including the importance of early relationship-building, group composition and chair selection.As online meetings may enable greater inclusiveness and flexibility, it would be important to investigate further the preconditions for successful online working, and which elements of face-to-face working may be replicated online and which cannot, and under what circumstances.Finally, the time and resources for meaningful PPI should not be underestimated, and flexible funding models should be considered to enable iterative and responsive working to allow genuine co-production.
Three Schools Mental Health' Senior Fellowship.£250,350, 30 months, start January 2020.Member of NIHR RfPB Northwest Funding Panel 2016-18 and renewed membership for a second term between 2018 and 2020.• Invited member of the Scrutiny Sub-committee for the NIHR RfPB/RfSC Mental Health Research Call in Northern England 2021.• Holder of 4-year tenure on Subcommittee A Stage 2 Programme Grants from 2022.

TABLE 1
List of carer RAP meetings

TABLE 2
Summary of recommendations from the carer RAP (see Report Supplementary Material 1 for full details)

TABLE 2 Summary
of recommendations from the carer RAP (see Report Supplementary Material 1 for full details) (continued) Copyright © 2023 Grande et al.This work was produced by Grande et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care.This is an Open Access publication distributed under the terms of the Creative Commons Attribution CC BY 4.0 licence, which permits unrestricted use, distribution, reproduction and adaptation in any medium and for any purpose provided that it is properly attributed.See: https://creativecommons.org/licenses/by/4.0/.For attribution the title, original author(s), the publication source -NIHR Journals Library, and the DOI of the publication must be cited.13

TABLE 2 Summary
of recommendations from the carer RAP (see Report Supplementary Material 1 for full details) (continued)