Comprehensive Research Report on Understanding Website Audiences
This private and confidential research report by Mace & Menter delves into understanding the audience of the NAO website through stakeholder engagement, project objectives, and recommendations for better serving user needs and enhancing impact.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Understanding our website audiences Research report 26 April 2021 | Private & Confidential 1 Mace & Menter - Private and Confidential
Contents 1. Introduction 5. Executive summary 40 mins 2. Objectives 6. Findings by theme 20 mins 3. Approach 4. Understanding your audiences 7. Recommendations 8. Next steps 30 mins Q&A 2 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Introduction Mace & Menter - Private and Confidential
1. Introduction 1.1. Background 1.2. Stakeholder engagement 4 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Background The NAO website is a key channel to reach your audiences. This is an opportunity to invest in developing the technology, design and content of the website so that it better supports your new strategy, enabling the NAO to provide more accessible insight and increase its impact. NAO commissioned Mace & Menter to run a discovery project to better understand its website audiences and how it might better serve their needs. This will support and evidence your business case for investment. 5 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Stakeholder engagement A key part of the discovery project is to gain a clear understanding of the expectations from key NAO internal stakeholders. So far we have conducted in-depth interviews with 12 NAO stakeholders from across the organisation (knowledge, VfM, ERT and PRT). We will continue to further engage with stakeholders during the concepting phase of the project. We welcome any further engagement after this presentation for follow up. 6 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Objectives Mace & Menter - Private and Confidential
2. Objectives 2.1. Project objectives 2.2. Scope of work 2.3. Where we are 8 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Project objectives Provide evidence to support the business case for redeveloping the website Understand your key website audiences and their needs from the NAO Understand how well the website is currently supporting their needs Uncover the unmet needs of your website users Validate concepts of how the website could better support unmet needs Validate the technical feasibility of our recommendations 9 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Scope of work In scope Deep website-related focus on Parliamentarians and Civil Servants groups (who we were able speak to) Snapshot of the website experience of a broad range of audiences Not in scope Comprehensive research study into all types of Civil Service user groups Comprehensive research study into all Parliamentarian user groups 10 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Where we are so far Done Understood organisational needs and existing research Conducted qualitative user research to deeply understand needs of key audiences Surveyed all key NAO audiences to understand broader needs * Distilled our research into clear insights and recommendations Next Work with you to prioritise recommendations as hypotheses that we can test Work with you to create website concepts to test with our two user groups Report with user feedback and design recommendations from concept testing Review the technical implications and feasibility of the recommendations * Survey findings to be presented separately 11 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Approach Mace & Menter - Private and Confidential
3. Approach 3.1. Approach 3.2. Methodology 3.3. Recruitment 13 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Approach Mixed method research approach A critical part of our work is to gain a deep understanding of your key audiences We do this in a number of ways combining both qualitative and quantitative research methods to ensure that we balance the need for a deep of understanding of your core audience groups whilst also gaining the breadth to cover broader audiences. 14 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Methodology Quantitative research Analytics - We reviewed all of your relevant analytics data to help inform our understanding of current website behaviour Surveys* - We have surveyed a broad spectrum of your audience groups to increase the breadth and depth of our insight Response - promoted via the website, social media and sent out to all key NAO audience groups mailing lists, we have so far received: 842 responses * Survey findings to be presented separately 15 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Methodology Qualitative research We conducted a total of 17 semi-structured 1 hour depth interviews with: X7 Parliamentarians X10 Civil servants 16 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Understanding our audiences Persona summary Mace & Menter - Private and Confidential
Persona summary Please note: We won t have time to go through the detailed persona groups today, however here s an overview to help give you some context when we are referring to specific types of users. We will share the complete presentation in due course. 18 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Civil Servants - Persona summary Collaborators Specialists Collaborators are civil servants who work with the NAO to: Specialists are civil servants who: who work in strategic roles driving policy development, project/programme delivery or in operations Support its financial & VfM audits or Provide subject matter expertise and advice to help inform reports and investigations. 19 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Civil Servants - Persona summary Collaborators Specialists Audit Collaborator Specialist: Policy / Operations / Delivery Shared interest Partner Collaborator Report Collaborator Who they might be Director level SCS1 (treasury) official Who they might be Director or grade 6 working in finance / audit / risk assurance Who they might be Directors / specialists in govt depts or from a public body Who they might be Senior (grade 6): Head of Policy / Delivery / Operations or public scrutiny body specialists What they do Scrutinise public spending & value for money / attend PAC committee meetings What they do Manage and audit financial accounts and risk assurance What they do Lead and manage specialist govt depts or public bodies What they do Devise processes & ways of working to support policy development, delivery planning, business case appraisals often during the inception phase. NAO related goals Work with NAO to support VFM & financial audits of their departments + sign off NAO related goals Offer subject matter expertise Advise on report focus areas Supply knowledge / data NAO related goals Understand NAOs remit and how best to work together NAO related goals Understand NAOs value proposition Best practice guidance on ways of working Cross-govt learnings & case studies 20 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Parliamentarians - Persona summary Commons Staff MPs Staff Committee Specialist Scrutiny Unit Person* Clerk Political Advisor Library Researcher What they do Answer private enquires from Members of Parliament Who they might be Economist, statistician, financial analyst, policy expert Who they might be Staff of a Select Committee such as PAC, Treasury, DCMS Who they might be Chief of Staff, Senior Political Advisor Who they might be Legal specialist, statistician, financial analysts, economist, policy specialist What they do Support a MP in their day-to-day tasks, including preparing for briefings, debates and committee meetings What they do Support the day-to-day running of the committee and its members NAO related goals Reference NAO reports in their findings What they do Provide specialist help to select committees, writing briefs & reports What they do Providing scrutiny of public expenditure and pre legislative draft bills NAO related goals Looking at particular topics, themes or constituency interests that match those of the MP they support NAO related goals Direct members to NAO reports and recommendations NAO related goals Reference NAO reports in their own reports NAO related goals Use NAO reports to scrutinise public spending * Did not interview in this study 21 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Executive summary Mace & Menter - Private and Confidential
4. Executive summary Broad unmet need for department agnostic theme and issue based knowledge on the website Finding reports is the primary motivation to visit the website, however once found, users were offered no compelling reasons to stay on the website and would leave Finding recent & specific reports was seen as fairly easy, however finding thematic content was difficult Civil service specialists were the group that felt their needs were least supported by the NAO Clear unmet need from all civil service specialists for cross-government best practice guides and case studies to show them how to improve ways of working in their departments MPs staff are very time poor, having to find and distill information quickly, they did not want more general newsletters, they wanted concise information tailored to be relevant to the interests of their MP & constituents Civil service auditees wanted clearer guidance and transparency about the audit process 23 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Findings Mace & Menter - Private and Confidential
5. Findings 5.1. All users 5.4. Personas 5.2. Civil servants 5.5. Experience maps 5.3. Parliamentarians 25 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Findings All users Mace & Menter - Private and Confidential
All users Overall website experience What we heard Quotes Participants across all user groups experienced issues with the website. Those whose need was only to find specific recent reports, were the most satisfied, whereas those looking for broader information were much less satisfied. Took me a long time to find a report Participant 2 Search function isn't that great Participant 10 The design of the website was seen as dated, busy and text heavy . Home page looks quite busy... difficult for me to quickly understand if there s anything relevant to me Participant 11 Parliamentarian users were more likely to have issues with the findability of content, whereas Civil Servants (specialists in particular) were unsatisfied with the lack of content for them. I find the website very text heavy, text is also quite small, compared to GOV.UK where it is a bit more spaced out and inviting. Participant 15 Some Parliamentarian users felt that the website performed adequately, however this view was to an extent informed by their low expectation from public sector websites as a whole. 27 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
All users Once users find a report they have little reason to stay What we heard Quotes Finding reports was most likely motivation to visit the website, however once found - participants were offered no compelling reasons to stay on the website and would leave. For many participants, the website was merely a destination to find publications and awareness of other types of content was very low. There is this treasure trove of insight that nobody knows about Participant 17 People think the NAO just does VfM and Audit reports, they do so much more and people should be able to access it Participant 16 What this means There was clear unmet need to surface and join up more related and diverse content (not just reports) giving your users a reason to stay on the website, supporting onward journeys and more engagement into NAOs depth and breadth of knowledge. 28 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
All users Ease of finding reports was dependent on recall and recency What we heard Quotes Using the website to find recent and specific reports where the participant remembered the report name was seen as being fairly easy. However, finding reports based on topics or where the report name was not fully remembered, was difficult. Many participants felt that searching via Google was easier than using NAOs site search. Search function isn't that great. Not sure what category it is in... have used filters...hard to know how to filter Participant 7 It is often difficult to know which type of report is which Participant 15 What this means The website search and filter features does not adequately support users finding topical information and easily being able to narrow down and find what they are looking for. Tend to use Google as it s much easier Participant 2 29 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
All users Need for department agnostic themes & issue based knowledge What we heard Quotes There was a clear unmet need for department agnostic themes and issue based knowledge on the website from both parliamentarians and civil servants. Being more of a government resource would be helpful... themes from audits and best practice (cross-government) with examples of positive case studies Participant 13 What this means If it was more agnostic of department or reports and more about a particular problem I would use it more Participant 17 MP s staff wanted to understand what knowledge the NAO had on specific issues important to their MP and constituents. Be good to see a more informal blogs, e.g. BoE s Bank Underground good to have themes such as 'procurement' Participant 2 Civil servant specialists wanted to find thematic information to help support their roles such as guidance on procurement or working from home during the pandemic. 30 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
All users PDF was the favoured report format but there was openness for change What we heard Quotes Participants broadly preferred to read publications (digitally) as PDFs. Those who (pre-pandemic) would mainly read printed reports were more likely to view PDFs as the closest equivalent and most favoured this format, however there was a willingness from all participants to move content to an HTML format - as long as there still was an option to download as a PDF. I much prefer it to be in PDF, I hate HTML but it wouldn't put me off if it was in HTML as long as I could still download the PDF Participant 9 As long as you have option to print out the report even from HTML format it will be fine Participant 6 What this means HTML should be for engagement with bullet points of an overview with the ability to use chevrons to go further with a link to the original [PDF] if you want to go deep [cites Erskine May website] Participant 4 Switching to an HTML first publishing model won t put off users from visiting the NAO website and opens a significant opportunity to make your content more accessible, findable, and reusable. 31 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
All users Digital behaviour shifts and workarounds during the pandemic What we heard Quotes Many participants missed being able to print reports (driven particularly by a need to make notes and highlight important areas) some changed their view and behaviour (preferring digital for shorter documents or when needing to search) and found digital workarounds for annotating documents such as using OneNote and Office 365. A year ago, it would of been a printed doc as it s easier to annotate... but not always the case, if I quickly want to understand something now I would prefer to look at it online Participant 14 Found my own way to classify reports by putting them in OneNote, I also make my own notes in OneNote Participant 14 What this means One of the key motivations to print reports is the ability to then make notes. There is an opportunity to explore how digital note-taking can help shift more users away from print and PDFs. I like printing stuff off and annotating on paper but ok with PDFs and adding annotations Participant 9 32 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Findings Civil servants Mace & Menter - Private and Confidential
Civil servants Specialists were the least supported group What we heard Quotes Civil service specialists were the user group that felt least supported by the NAO and its website. Nobody knows what it is they are doing so no one will look to them for advice, even on their website, it doesn't tell us what they do. Participant 14 There was a strong unmet need for guidance to support common tasks such as writing business cases, conducting appraisals, developing policy and procurement good practice. Does not tell me how to solve a problem Participant 17 All participants stated that the NAO could do more to better support their needs. There was also some lack of clarity of NAOs role, remit and offer to them. NAO should be involved with policy development, but wonder if it is their role to do this. Maybe not. Participant 16 What this means Their unmet needs represent the largest gap and opportunity to improve the website to better support a currently underserved group. 34 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Civil servants Unmet need for cross-government good practice guidance What we heard Quotes All civil service specialists felt that the NAO should do more to support their unmet needs for department-agnostic best practice guidance and case studies to help them solve common cross- government problems. So much goes wrong across government and repeat problems that the NAO knows about that they could help civil servants improve... by linking civil servants to guidance and examples of where this has happened before. Participant 17 What this means Should be something on the website that says on are you looking to develop a new policy - here s some guidance with an overview about why this work is important, steps involved and then provide guidance. Participant 16 There is a clear opportunity to better leverage NAO s (dept /govt agnostic ) knowledge and learnings to provide this good practice guidance and help them solve problems. 35 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Civil servants Auditees want more clarity about the audit process What we heard Quotes Civil service auditees (people working for departments being auditing by the NAO) wanted (either for themselves or new/junior staff) clearer guidance and transparency around the audit process, specifically what to expect, what is expected from them and case studies showing them how audits work. We want to know more about the process of audits, so we can make sure we can have the right people helping the NAO Participant 5 Would be good to share best practice across govt for audits what the NAO expect, what evidence requirements, case studies etc this would make the process more transparent Participant 12 What this means Auditees would value practical guidance on the website detailing the audit process to help give them reassurance. 36 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Civil servants Untapped treasure trove of insight What we heard Quote There was a perception from participants that the NAO had a wealth of insights and knowledge, particularly around cross-government learnings and guidance that either existed but they couldn t find or that the NAO was not making available on the website. There is this treasure trove of insight that nobody knows about because I do collaboration with them outside of audits I know they have this wealth of insights and advice. What this means Participant 17 This is an opportunity to increase awareness of the breadth and depth of NAOs knowledge by doing more to surface and highlight this information on the website. 37 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Civil servants Information sources - Civil Servants We asked participants during the interviews where they look for information to support their role beyond the NAO. Civil Servant specialists gave the following examples which offered them guidance around good working practice, something which they felt was missing from the NAO. Private sector Harvard Business Review Government and Public sector IPA Thought Leadership and trends in ways of working Lean.org Government s centre of expertise for infrastructure and major projects The Green Book Organisational best practice and ways of working HM Treasury guidance on how to appraise and evaluate policies, projects and programmes Supreme audit institutions New Zealand ALB 38 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Findings Parliamentarians Mace & Menter - Private and Confidential
Parliamentarians New and infrequent users faced a steep learning curve What we heard Quote You do have to learn (the website) and spend time with it - there is a learning curve... I mainly use it if I need to refer someone (colleague) to a publication or something coming up. My role is about funnelling information to people. Participant 1 New and infrequent users faced a steep learning curve to find information they were looking for and often gave up. They would look to colleagues more familiar with the website or get in touch directly with NAO staff if they had an existing contact. What this means Took me a long time to find a report [on the website] Participant 2 There is a need to make website more user friendly and intuitive to new and less frequent users. Most frequent, long time users of the website have become accustomed to its limitations as expert users , however they also accept there are areas to improve. 40 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Parliamentarians Not always clear how the NAO website can support their role What we heard Quote First time [I went to the NAO website] was when I was being inducted into my current role. My predecessor showed me the pages that would be most relevant for me. Participant 1 It wasn t always clear to Commons Staff how the NAO and its website can help them in their role. Specifically around NAO s proposition to them. For new staff members coming through this is further exacerbated due to high-levels of staff turnover. What this means I signed up to an induction with NAO to go through what they do, but could not attend. They tend to reach out to me and initiated on their side to say if I need any help or any docs. Participant 9 It may not be feasible for the NAO to actively initiate direct support to new staff. However, this support could be provided by dedicated Commons Staff induction pages or videos on the NAO website. Didn't even know they had a website! Participant 4 This will enable them to better self-serve and clearly understand NAOs proposition for them and the ways it can support their role. 41 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Parliamentarians Little time or opportunity to read reports in depth What we heard Quotes There are frequent scenarios where reports need to be distilled on someone else s behalf so that they understand the key points. This could be ahead of a committee hearing, parliamentary debate or part of a media engagement around a particular event I support [committee member in their] Committee role. I try to read the reports before to help shape the questions, but that wasn't sustainable. Participant 10 What this means The chair on the committee goes through the reports and creates bullet points for everyone else. No one else on the committee goes near the NAO report. Better to make it more concise. Participant 4 There is a strong need to publish a summarised version of the report that will allow someone to quickly understand the context, key findings and any recommendations that were made. This will allow users to quickly determine if a report is relevant to a particular need and have time to form a political angle on the findings 42 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Parliamentarians MP s staff key to understanding and distilling reports What we heard Quote Due to MPs being very time poor, there is a dependency on their staff to prepare them for their day-to-day work. They rarely have time to read reports so will often ask their staff to brief them on the key findings and recommendations I used to go to the NAO briefings with [MP] in person in our office so we can chat. But now if you don't get an invite you assume you can t go, it is all on Teams. I do not directly deal with NAO now What this means Participant 10 Before the pandemic, some MP s staff could arrive at an in person report briefing within parliament. Since restrictions are in place, these briefings are often held virtually on Microsoft Teams and consequently staff members are often not invited to attend 43 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Parliamentarians Keep me updated but only with what's relevant to me What we saw Quote Due to very specific topical interests from advisors and commons staff such as committee specialists , there was a strong need to be notified of any new reports or developments so that they can remain fully informed. I would appreciate direct marketing from NAO to be reminded or notified of any work they have done on the specific interests I have What this means Participant 6 Although blanket emails was seen as a potential burden, there is evidence that if the content was targeted to a relevant topical or political interest then it would be of great use. This would mean reduce the dependency on having one-to-one relationships with NAO contacts to be kept updated. 44 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Information sources - Parliamentarians We asked participants during the interviews where they look for information to support their role beyond the NAO. They gave the following examples as sources of valuable information. Private sector Chatham House Royal United Services Institute Government and Public sector Erskine May on Parliament website Institute for Government Office of Budget Responsibility Office for National Statistics World Health Organization 45 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Experience Maps Mace & Menter - Private and Confidential
Experience maps Experience maps are a visual representation of an end-to-end experience that a person goes through when trying to accomplish a goal. Speaking to a range of users helps us to understand what users do, think and feel over time and may experience throughout their journeys. Consolidating these into a single map helps us to understand: How users currently accomplish a goal How things work (or don t) Interdependencies - for example, between different departments or services Pain points and where things are broken Opportunities to improve 47 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Experience Maps - Parliamentary Committee These will be made available during the project 48 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Experience Maps - Financial audit / VfM report These will be made available during the project 49 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential
Recommendations Mace & Menter - Private and Confidential
