Comprehensive Research Report on Understanding Website Audiences

Understanding our
website audiences
26 April 2021  
|
  Private & Confidential
1
Research report
Contents
1.
Introduction
2.
Objectives
3.
Approach
4.
Understanding
your audiences
2
5.
 
Executive summary
6. 
 
Findings by theme
7.
 
Recommendations
8.
 
Next steps
30 mins
20 mins
40 mins
 
Q&A
Introduction
1. 
Introduction
1.1.
 
 
Background
1.2.
 
 
Stakeholder engagement
4
5
The NAO website is a key channel to reach your audiences.
This is an opportunity to invest in developing the technology, design and content of
the website so that it better supports your new strategy, enabling the NAO to
provide more accessible insight and increase its impact. 
      
NAO commissioned Mace & Menter to run a discovery project to better understand
its website audiences and how it might better serve their needs.
This will support and evidence your business case for investment.
 
Background
6
Stakeholder engagement
A key part of the discovery project is to gain a clear understanding of the
expectations from key NAO internal stakeholders.
So far we have conducted in-depth interviews with 12 NAO stakeholders
from across the organisation (knowledge, VfM, ERT and PRT). 
We will continue to further engage with stakeholders during the concepting
phase of the project.
We welcome any further engagement after this presentation for follow up.
Objectives
2. 
Objectives
2.1.
 
 
Project objectives
2.2.
 
 
Scope of work
2.3.
 
 
Where we are
8
9
Provide evidence to support the business case for redeveloping the website
Understand your key website audiences and their needs from the NAO
Understand how well the website is currently supporting their needs
Uncover the unmet needs of your website users
Validate concepts of how the website could better support unmet needs
Validate the technical feasibility of our recommendations
Project objectives
10
In scope
Deep website-related focus on Parliamentarians and Civil Servants groups (who we
were able speak to)
Snapshot of the website experience of a broad range of audiences
Not in scope
Comprehensive research study into all types of Civil Service user groups
Comprehensive research study into all Parliamentarian user groups
Scope of work
11
Done
Understood organisational needs and existing research
Conducted qualitative user research to deeply understand needs of key audiences
Surveyed all key NAO audiences to understand broader needs *
Distilled our research into clear insights and recommendations
Next
Work with you to prioritise recommendations as hypotheses that we can test 
Work with you to create website concepts to test with our two user groups
Report with user feedback and design recommendations from concept testing
Review the technical implications and feasibility of the recommendations
Where we are so far
* Survey findings to be presented separately
Approach
3. 
Approach
13
3.1.
 
 
Approach
3.2.
 
 
Methodology
3.3.
 
 
Recruitment
14
Mixed method research approach
A critical part of our work is to gain a deep understanding of your key
audiences
We do this in a number of ways combining both 
qualitative
 and 
quantitative
research methods to ensure that we balance the need for a deep of
understanding of your core audience groups whilst also gaining the breadth to
cover broader audiences.
Approach
15
Quantitative research
Analytics - 
We reviewed all of your relevant analytics data to help inform our understanding
of current website behaviour
Surveys*
 
- We have surveyed a broad spectrum of your audience groups to increase the
breadth and depth of our insight
Response
 - promoted via the website, social media and sent out to all key NAO
audience groups mailing lists, we have so far received:
842 responses
Methodology
* Survey findings to be presented separately
16
Qualitative research
We conducted a total of 17 semi-structured 1 hour depth interviews with:
X7 Parliamentarians 
X10 Civil servants
Methodology
Understanding our audiences
Persona summary
Persona summary
Please note:
We won’t have time to go through the detailed persona groups today, however here’s an
overview to help give you some context when we are referring to specific types of users.
We will share the complete presentation in due course.
18
Civil Servants - Persona summary
19
Collaborators
Specialists
Collaborators are civil servants who 
work with the NAO to:
 
Support its financial & VfM audits 
or
Provide subject matter expertise
and advice to help inform reports
and investigations.
Specialists are civil servants who
:
 
who work in strategic roles
driving policy development,
project/programme delivery or
in operations
Civil Servants - Persona summary
20
Audit Collaborator
Specialist
: Policy / Operations / Delivery
Report Collaborator
Collaborators
Specialists
Who they might be
Director or grade 6 working in
finance / audit / risk assurance
What they do
Manage and audit financial
accounts and risk assurance
NAO related goals
Work with NAO to support
VFM & financial audits of their
departments + sign off
Who they might be
Directors / specialists in govt
depts or from a public body
What they do
Lead and manage specialist
govt depts or public bodies
NAO related goals
Offer subject matter expertise
Advise on report focus areas
Supply knowledge / data
Who they might be
Senior (grade 6): Head of Policy / Delivery /
Operations or public scrutiny body specialists
What they do
Devise processes & ways of working to support
policy development, delivery planning, business
case appraisals often during the inception phase.
NAO related goals
Understand NAOs value proposition
Best practice guidance on ways of working
Cross-govt learnings & case studies
Shared interest
Partner Collaborator
Who they might be
Director level SCS1
(treasury) official
What they do
Scrutinise public spending &
value for money / attend
PAC committee meetings
NAO related goals
Understand NAOs remit and
how best to work together
21
Library Researcher
Political Advisor
Committee
Specialist
Commons Staff
MPs Staff
Clerk
Parliamentarians - Persona summary
What they do
Answer private enquires from
Members of Parliament 
NAO related goals
Reference NAO reports in
their findings
Who they might be
Economist, statistician,
financial analyst, policy
expert
What they do
Provide specialist help
to select committees,
writing briefs & reports
NAO related goals
Reference NAO reports
in their own reports
Who they might be
Staff of a Select Committee such
as PAC, Treasury, DCMS
What they do
Support the day-to-day
running of the committee and
its members
NAO related goals
Direct members to NAO
reports and recommendations
Who they might be
Chief of Staff, Senior Political Advisor
What they do
Support a MP in their day-to-day tasks, including
preparing for briefings, debates and committee
meetings
NAO related goals
Looking at particular topics, themes or
constituency interests that match those of the MP
they support
Scrutiny Unit
Person*
Who they might be
Legal specialist,
statistician, financial
analysts, economist,
policy specialist
What they do
Providing scrutiny of
public expenditure and
pre‐legislative draft bills
NAO related goals
Use NAO reports to
scrutinise public
spending
* Did not interview in this study
Executive summary
4. 
Executive summary
Broad unmet need for department agnostic theme and issue based knowledge on the website
Finding reports is the primary motivation to visit the website, however once found, users were offered no
compelling reasons to stay on the website and would leave 
Finding recent & specific reports was seen as fairly easy, however finding thematic content was difficult 
Civil service ‘specialists’ were the group that felt their needs were least supported by the NAO
Clear unmet need from all civil service ‘specialists’ for cross-government best practice guides and case studies
to show them how to improve ways of working in their departments
MPs staff are very time poor, having to find and distill information quickly, they did not want more general
newsletters, they wanted concise information tailored to be relevant to the interests of their MP & constituents
Civil service ‘auditees’ wanted clearer guidance and transparency about the audit process
23
Findings
5. 
Findings
5.1.
 
 
All users
5.2.
 
 
Civil servants
5.3.
 
 
Parliamentarians
25
5.4.
 
 
Personas
5.5.
 
 
Experience maps
All users
Findings
Overall website experience
Quotes
Took me a long time to find a report
 
Participant 2
Search function isn't that great
Participant 10
Home page looks quite busy... difficult for
me to quickly understand if there’s anything
relevant to me
Participant 11
“I find the website very text heavy, text is
also quite small, compared to GOV.UK
where it is a bit more spaced out and
inviting.”
Participant 15
 
What we heard
Participants across all user groups experienced issues with the website.
Those whose need was only to find specific recent reports, were the most
satisfied, whereas those looking for broader information were much less
satisfied.
The design of the website was seen as dated, ‘busy’ and ‘text heavy’.
Parliamentarian users were more likely to have issues with the findability of
content, whereas Civil Servants (specialists in particular) were unsatisfied
with the lack of content for them.
Some Parliamentarian users felt that the website performed adequately,
however this view was to an extent informed by their low expectation from
public sector websites as a whole.
27
All users
Once users find a report they have little reason to stay
Quotes
There is this treasure trove of
insight that nobody knows about
Participant 17 
People think the NAO just does
VfM and Audit reports, they do so
much more and people should be
able to access it
Participant 16 
What we heard
Finding reports was most likely motivation to visit the website,
however once found - participants were offered no compelling
reasons to stay on the website and would leave.
For many participants, the website was merely a destination to find
publications and awareness of other types of content was very low.
What this means
There was clear unmet need to surface and join up more related and
diverse content (not just reports) giving your users a reason to stay on
the website, supporting onward journeys and more engagement into
NAOs depth and breadth of knowledge.
28
All users
Ease of finding reports was dependent on recall and recency
Quotes
Search function isn't that great.
Not sure what category it is in...
have used filters...hard to know
how to filter
Participant 7 
It is often difficult to know which
type of report is which
Participant 15 
Tend to use Google as it’s much
easier
Participant 2
What we heard
Using the website to find recent and specific reports where the
participant remembered the report name was seen as being fairly
easy. However, finding reports based on topics or where the report
name was not fully remembered, was difficult. Many participants felt
that searching via Google was easier than using NAOs site search.
What this means
The website search and filter features does not adequately support
users finding topical information and easily being able to narrow
down and find what they are looking for.
29
All users
Need for department agnostic themes & issue based knowledge
Quotes
Being more of a government resource
would be helpful... themes from audits and
best practice (cross-government) with
examples of positive case studies
’’ 
Participant 13
If it was more agnostic of department or
reports and more about a particular
problem I would use it more
Participant 17 
Be good to see a more informal blogs, e.g.
BoE’s Bank Underground… good to have
themes such as 'procurement'
Participant 2 
What we heard
There was a clear unmet need for department agnostic themes and
issue based knowledge on the website from both parliamentarians
and civil servants.
What this means
MP’s staff wanted to understand what knowledge the NAO had on
specific issues important to their MP and constituents.
Civil servant ‘specialists’ wanted to find thematic information to help
support their roles such as guidance on procurement or working from
home during the pandemic.
30
All users
PDF was the favoured report format but there was openness for change
Quotes
I much prefer it to be in PDF, I hate HTML
but it wouldn't put me off if it was in HTML
as long as I could still download the PDF
Participant 9  
As long as you have option to print out the
report even from HTML format it will be
fine
Participant 6
HTML should be for engagement … with
bullet points of an overview with the ability
to use chevrons to go further … with a link
to the original [PDF] if you want to go
‘deep
” [cites Erskine May 
website
]
Participant 4
What we heard
Participants broadly preferred to read publications (digitally) as PDFs.
Those who (pre-pandemic) would mainly read printed reports were
more likely to view PDFs as the closest equivalent and most favoured
this format, however there was a willingness from all participants to
move content to an HTML format - as long as there still was an option
to download as a PDF.
What this means
Switching to an HTML first publishing model won’t put off users from
visiting the NAO website and opens a significant opportunity to make
your content more accessible, findable, and reusable.
31
All users
Digital behaviour shifts and workarounds during the pandemic
Quotes
A year ago, it would of been a printed
doc… as it’s easier to annotate... but not
always the case, if I quickly want to
understand something now I would prefer
to look at it online
 
Participant 14
Found my own way to classify reports by
putting them in OneNote, I also make my
own notes in OneNote
Participant 14 
I like printing stuff off and annotating on
paper but ok with PDFs and adding
annotations
Participant 9
What we heard
Many participants missed being able to print reports (driven
particularly by a need to make notes and highlight important areas)
some changed their view and behaviour (preferring digital for shorter
documents or when needing to search) and found digital
workarounds for annotating documents such as using OneNote and
Office 365. 
What this means
One of the key motivations to print reports is the ability to then make
notes. There is an opportunity to explore how digital note-taking can
help shift more users away from print and PDFs.
32
All users
 
Civil servants
Findings
Specialists were the least supported group
Quotes
Nobody knows what it is they are
doing so no one will look to them for
advice, even on their website, it doesn't
tell us what they do.
Participant 14
Does not tell me how to solve a
problem
Participant 17
NAO should be involved with policy
development, but wonder if it is their
role to do this. Maybe not.
Participant 16
What we heard
Civil service ‘specialists’ were the user group that felt least supported by the
NAO and its website.  
There was a strong unmet need for guidance to support common tasks such
as writing business cases, conducting appraisals, developing policy and
procurement good practice.
All participants stated that the NAO could do more to better support their
needs. There was also some lack of clarity of NAOs role, remit and offer to
them.
What this means
Their unmet needs represent the largest gap and opportunity to improve
the website to better support a currently underserved group.
34
Civil servants
Unmet need for cross-government good practice guidance
What we heard
All civil service ‘specialists’ felt that the NAO should do more to
support their unmet needs for department-agnostic best practice
guidance and case studies to help them solve common cross-
government problems.
What this means
There is a clear opportunity to better leverage NAO’s 
(dept /govt
agnostic ) 
knowledge and learnings to provide this good practice
guidance and help them solve problems.
35
Quotes
So much goes wrong across government
and repeat problems that the NAO knows
about that they could help civil servants
improve... by linking civil servants to
guidance and examples of where this has
happened before.
Participant 17
Should be something on the website that
says on ‘are you looking to develop a new
policy - here’s some guidance’ with an
overview about why this work is important,
steps involved and then provide guidance.
Participant 16
Civil servants
Auditees want more clarity about the audit process
Quotes
We want to know more about the process
of audits, so we can make sure we can have
the right people helping the NAO
Participant 5
Would be good to share best practice
across govt for audits… what the NAO
expect, what evidence requirements,
case studies etc… this would make the
process more transparent
Participant 12
What we heard
Civil service ‘auditees’ (people working for departments being
auditing by the NAO) wanted (either for themselves or new/junior
staff) clearer guidance and transparency around the audit process,
specifically what to expect, what is expected from them and case
studies showing them how audits work.   
What this means
‘Auditees’ would value practical guidance on the website detailing the
audit process to help give them reassurance.
36
Civil servants
Untapped ‘treasure trove of insight’
Quote
There is this treasure trove of
insight that nobody knows about
because I do collaboration with
them outside of audits I know they
have this wealth of insights and
advice.
Participant 17
What we heard
There was a perception from participants that the NAO had a ‘wealth of
insights’ and knowledge, particularly around cross-government learnings
and guidance that either existed but they couldn’t find or that the NAO was
not making available on the website.
What this means
This is an opportunity to increase awareness of the breadth and depth of
NAOs knowledge by doing more to surface and highlight this information
on the website.
37
Civil servants
Information sources - Civil Servants
38
Private sector
Harvard Business Review
Thought Leadership and trends in ways of working
Lean.org
Organisational best practice and ways of working
Supreme audit institutions  
New Zealand ALB
Government and Public sector
IPA
Government’s centre of expertise for infrastructure and major projects
The Green Book
HM Treasury guidance on how to appraise and evaluate policies, projects and
programmes
We asked participants during the interviews where they look for information to support their role beyond the NAO.
Civil Servant ‘specialists’  gave the following examples which offered them guidance around good working practice,
something which they felt was missing from the NAO.
Civil servants
Findings
Parliamentarians
New and infrequent users faced a steep learning curve
What we heard
New and infrequent users faced a steep learning curve
 to find information
they were looking for and often gave up. They would look to colleagues
more familiar with the website or get in touch directly with NAO staff if they
had an existing contact.
What this means
There is a need to make website more user friendly and intuitive to new and
less frequent users. 
Most frequent, long time users of the website have become accustomed to
its limitations as ‘expert users’, however they also accept there are areas to
improve.
40
Quote
You do have to learn (the website) and
spend time with it - there is a learning
curve...  I mainly use it if I need to refer
someone (colleague) to a publication or
something coming up. My role is about
funnelling information to people. 
Participant 1 
“Took me a long time to find a
report [on the website]”
Participant 2
Parliamentarians
Not always clear how the NAO website can support their role
What we heard
It wasn’t always clear to Commons Staff how the NAO and its website can
help them in their role. Specifically around NAO’s proposition to them.
For new staff members coming through this is further exacerbated due to
high-levels of staff turnover. 
What this means
It may not be feasible for the NAO to actively initiate direct support to new
staff. However, this support could be provided by dedicated Commons Staff
induction pages or videos on the NAO website.
This will enable them to better ‘self-serve’ and clearly understand NAOs
proposition for them and the ways it can support their role.
41
Quote
First time [I went to the NAO website] was
when I was being inducted into my current
role. My predecessor showed me the pages
that would be most relevant for me.
Participant 1 
I signed up to an induction with NAO to go
through what they do, but could not attend.
They tend to reach out to me and initiated
on their side to say if I need any help or any
docs.
Participant 9
Didn't even know they had a website!
 
Participant 4
Parliamentarians
Little time or opportunity to read reports in depth
Quotes
“I support [committee member in their]
Committee role. I try to read the reports
before to help shape the questions, but
that wasn't sustainable.”
Participant 10 
“The chair on the committee goes
through the reports and creates bullet
points for everyone else. No one else
on the committee goes near the NAO
report. Better to make it more concise.”
Participant 4 
What we heard
There are frequent scenarios where reports need to be distilled on someone
else’s behalf so that they understand the key points. This could be ahead of
a committee hearing, parliamentary debate or part of a media engagement
around a particular event
What this means
There is a strong need to publish a summarised version of the report that will
allow someone to quickly understand the context, key findings and any
recommendations that were made. This will allow users to quickly determine
if a report is relevant to a particular need and have time to form a political
angle on the findings
42
Parliamentarians
MP’s staff key to understanding and distilling reports
Quote
“I used to go to the NAO briefings
with [MP] in person in our office so
we can chat. But now if you don't
get an invite you assume you can’t
go, it is all on Teams. I do not
directly deal with NAO now”
Participant 10
What we heard
Due to MPs being very time poor, there is a dependency on their staff to
prepare them for their day-to-day work. They rarely have time to read
reports so will often ask their staff to brief them on the key findings and
recommendations
What this means
Before the pandemic, some MP’s staff could arrive at an in person report
briefing within parliament. Since restrictions are in place, these briefings are
often held virtually on Microsoft Teams and consequently staff members are
often not invited to attend
43
Parliamentarians
Keep me updated but only with what's relevant to me
Quote
“I w
ould appreciate direct marketing
from NAO to be reminded or notified of
any work they have done on the
specific interests I have
Participant 6
What we saw
Due to very specific topical interests from advisors and commons staff such
as ‘committee specialists’
, there was a strong need to be notified of any
new reports or developments so that they can remain fully informed.
What this means
Although blanket emails was seen as a potential burden, there is evidence
that if the content was targeted to a relevant topical or political interest then
it would be of great use. This would mean reduce the dependency on having
one-to-one relationships with NAO contacts to be kept updated.
44
Parliamentarians
Information sources - Parliamentarians
45
Private sector
Chatham House
Royal United Services Institute
Government and Public sector
Erskine May on Parliament website
Institute for Government
Office of Budget Responsibility
Office for National Statistics
World Health Organization
We asked participants during the interviews where they look for information to support their role beyond the NAO.
They gave the following examples as sources of valuable information.
Experience Maps
47
Experience maps
Experience maps are a visual representation of an end-to-end experience that a person
goes through when trying to accomplish a goal.
Speaking to a range of users helps us to understand what users do, think and feel over time and may
experience throughout their journeys. 
Consolidating these into a single map helps us to understand:
How users currently accomplish a goal
How things work (or don’t)
Interdependencies - for example, between different departments or services
Pain points and where things are broken
Opportunities to improve
Experience Maps - Parliamentary Committee
48
These will be made available during the project
Experience Maps - Financial audit / VfM report
49
These will be made available during the project
Recommendations
Clearer signposting for key user groups to 
sections tailored to their needs
51
All users
The problem to solve
Many participants looking for role and task specific information were not able to find this
on the website (e.g. auditees looking for guidance on the audit process or MP’s staff
looking to easily understand what the NAO can offer them and how they can access
this).
What we recommend
Clearer and more visible calls-to-action directing specific users to a section of the
website with information relevant to their role and links to common tasks.
E.g. ‘Guidance for civil servants’ > ‘Good practice guides’ > ‘When developing policy’
‘Support for MPs and their staff’  >  ‘Beginners guide to how the NAO supports you’
Making it easier to find specific information
52
All users
The problem to solve
Whilst users were broadly satisfied with being able to browse or search for specific
recent publications, particularly where they recall the report name, however they were
less able to find publications where only the broad report theme was searched for due
to the lack of relevance from the results and ability to refine the search.
What we recommend
Using a faceted search approach, allows 
users to see and select multiple (facets) to
narrow the number of search results to get to most relevant results quickly.
Add richer content taxonomy via metadata (e.g. tags for ‘themes’ and ‘issues’) to help
improve the relevance of results.
Making it easier to find thematic information
53
All users
The problem to solve
Users have an unmet need to find more thematically related information to give them the full
breadth and depth of NAO’s knowledge around a specific topic.
What we recommend
Promote the findability and discoverability of thematic content.
Users who have specific interests will benefit from having clear visibility of the 
content
(such as reports, articles and data visualisations) related to their themes of interest.
Having more related content links will also help the site to drive deeper website
engagement.
Making it easier to discover relevant information
54
All users
The problem to solve
The main reason many users currently come to the website is to find a report but then they
leave without discovering anything else that could be useful to them.  
What we recommend
We need to give website users more compelling reasons to stay on the website (after finding
a report) by surfacing more contextually relevant information.
Surfacing all relevant content available (not just publications) including increasing awareness
of related support and guidance knowledge and role specific sections of the website.
Making text easier to read
55
All users
The problem to solve
The readability of the text on the website was seen as a pain point, being difficult to
read and as result, hard to find what you are looking for and to scan and digest
information.
What we recommend
Improve website typography, by increasing text size, line height and spacing across
all areas of the website to improve the clarity and legibility of headlines, summaries
and links and the readability of content.
Keep me updated but only when its relevant to me
56
The problem to solve
Participants mentioned how time poor they were (particularly MPs staff) and bemoaned the lack of relevant
information of the communications received from the NAO. They wanted more information, but it had to be
specifically relevant to their role and interests.
What we recommend
There is an opportunity to offer more tailored ways of keeping users updated based on their specific interests  
Receive updates (based on my role, role-specific tasks and specified topics of interest) when:
A new report published
A new report is being worked on (in progress)
Updates to the status of actions/recommendations of a report that I have read 
New insight, guidance or case study based on my area of role and role specific tasks
All users
Improve report content findability, reuse & accessibility
57
All users
The problem to solve
PDF documents can make your content 
harder to find, use and maintain
 and do not work
well with assistive technologies like screen readers a lot of the time.
What we recommend
Transition from publishing documents (such as reports) exclusively in PDF format and instead
begin publishing documents as accessible ‘HTML gateway pages’.
Gateway pages are html web pages that offer the key points and critical information from a
PDF, but still offer users the option to download the full PDF without having to open it in a
browser window.
Make reports more useful  
1 of 2
58
The problem to solve
Reports and other PDF publications are currently a digital replication of their printed form.
They offer little (apart from search) to leverage the benefits of digital technology to make them more
useful. 
Participants reported frustrations with being able to use publications within their work, copying and
pasting text and graphics was difficult. 
It was also difficult to share or embed NAO content, such as data visualisations in their own online
content or reports.
All users
Make reports more useful  
2 of 2
59
What we recommend
A key benefit of transitioning to a HTML publishing approach, is being able to make the content
within publications more useful in the following ways:
Being ‘bite-sized’ and digestible, with structured HTML content, we can help users to quickly
find the most valuable sections and choose to see only what's of interest to them
Easy to find, by more report content being indexed within site search and connected to other
type of relevant content
Usable, embeddable and shareable when users are using NAO content in their own work
Dynamic and visually engaging by embedding more real time data and visualisations
All users
Make reports more annotatable
60
The problem to solve
A key user motivation to print NAO publications is being able to then add notes.
During the pandemic, users have found digital workarounds to simulate this, such as annotating documents in digital
note taking apps like OneNote.
What we recommend
There is an opportunity to explore how digital technology can help shift more users away from print and PDFs.
This could be a hypothesis to test, e.g. 
we believe 
that if we publish a few reports in digital formats that support
annotation and ease of reuse (such as G Suite and Office 365) 
this will result in
 more users choosing to use digital.
We know this is true
, if users choose to access these reports in these formats instead of PDFs.
All users
Recommendations
Civil servants
Better communicate your offer to civil servants
62
Civil servants
The problem to solve
All groups of civil servants (collaborators and specialists) were not completely clear around
NAOs value proposition or support to them. 
Collaborators wanted a clearer sense of how the NAO works with them whereas specialists were either
completely unaware of NAO’s offer or unclear what their statutory remit covered.
What we recommend
Clear and visible call-to-action on the home page and primary site navigation to guide civil servants to a
section dedicated to funneling different user groups to role and task-specific guidance, along with simply
and concisely explained NAOs value proposition to them .
Give ‘specialists’ the guidance to solve common problems
63
The problem to solve
There was a clear unmet need for guidance to support ‘specialists’ with common tasks such as writing
business cases, conducting appraisals, developing policy and procurement good practice. 
How we might improve
We could use a guided navigation approach, which asks users to ‘self identify’ their role, area of work and
what problem they need help with, for example:
I am a [
Civil Servant
]
 working in [ 
specialism
 e.g. policy ]
Looking for help with… [ 
problem to solve
 e.g. developing new policy ]
Then take them to a ‘resource hub’ with good practice guides, case studies, reusable frameworks along with articles
to any relevant NAO insight and thought leadership.
Civil servants
Give auditees clear guidance around the audit process
64
The problem to solve
Civil service ‘auditees’ (people working in departments being audited) wanted (either for themselves or
new/junior staff) clearer guidance and transparency around the audit process to give them greater clarity
and reassurance to help allay their concerns.
How we might improve
Guide ‘auditees’ via clearer signposting to a ‘collaborator resource hub’, clearly explaining the audit process,
the relationship with NAO, what is expected from them and case studies demonstrating good audit practice
along with FAQ’s, points of contact and links to relevant resources.
Civil servants
Recommendations
Parliamentarians
Share your cross-gov departmental knowledge
66
What we recommend
The NAO is seen to be in a privileged and unique position of having insights on topics
and themes that span multiple departments. Parliamentarians working on a specific
topic would benefit from knowing these themes exist and the recommendations that
surround them
How we might do this
Curate dedicated theme pages that are common across government departments that
communicate the findings, recommendations and case studies.
Parliamentarians
Help me to quickly understand a report
67
What we recommend
It is a common theme that MPs need get up-to-speed quickly on a particular topic or report.
For example, a committee hearing has been called or a topical event is being covered in the
media that requires a MP’s reaction. This need is currently being facilitated by the MP’s staff
briefing them with short lead times. 
How we might do this
This could be achieved by providing MPs with an intuitive destination page for topical
coverage and shortened versions of reports (with key findings and recommendations) on
your website. By having these shortened pages on your website, it will make finding and
sharing key report information easier than by using traditional PDF documents.
Parliamentarians
Allow me to attend your report briefings
68
What we recommend
There is a MP’s need to allow their staff members to attend NAO report briefings. This allows the
staff members to familiarise themselves with the findings and also help support the MPs
preparations. We recommend by allowing details of the schedule of briefings being made available
will allow staff members to attend. 
How we might do this
By having a calendar of Parliamentary briefings available on your website will allow associated MP
staff members to ‘request to attend’ a particular briefing. This would be most beneficial to virtual
briefings where currently invitations are sent to a closed list which often doesn’t include the staff
members.
Parliamentarians
Slide Note

Welcome and thank you for joining us today.

The purpose of this session is to present you with our findings from the

‘user research’ portion of the website discovery project.

Embed
Share

This private and confidential research report by Mace & Menter delves into understanding the audience of the NAO website through stakeholder engagement, project objectives, and recommendations for better serving user needs and enhancing impact.


Uploaded on Aug 09, 2024 | 0 Views


Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript


  1. Understanding our website audiences Research report 26 April 2021 | Private & Confidential 1 Mace & Menter - Private and Confidential

  2. Contents 1. Introduction 5. Executive summary 40 mins 2. Objectives 6. Findings by theme 20 mins 3. Approach 4. Understanding your audiences 7. Recommendations 8. Next steps 30 mins Q&A 2 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  3. Introduction Mace & Menter - Private and Confidential

  4. 1. Introduction 1.1. Background 1.2. Stakeholder engagement 4 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  5. Background The NAO website is a key channel to reach your audiences. This is an opportunity to invest in developing the technology, design and content of the website so that it better supports your new strategy, enabling the NAO to provide more accessible insight and increase its impact. NAO commissioned Mace & Menter to run a discovery project to better understand its website audiences and how it might better serve their needs. This will support and evidence your business case for investment. 5 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  6. Stakeholder engagement A key part of the discovery project is to gain a clear understanding of the expectations from key NAO internal stakeholders. So far we have conducted in-depth interviews with 12 NAO stakeholders from across the organisation (knowledge, VfM, ERT and PRT). We will continue to further engage with stakeholders during the concepting phase of the project. We welcome any further engagement after this presentation for follow up. 6 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  7. Objectives Mace & Menter - Private and Confidential

  8. 2. Objectives 2.1. Project objectives 2.2. Scope of work 2.3. Where we are 8 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  9. Project objectives Provide evidence to support the business case for redeveloping the website Understand your key website audiences and their needs from the NAO Understand how well the website is currently supporting their needs Uncover the unmet needs of your website users Validate concepts of how the website could better support unmet needs Validate the technical feasibility of our recommendations 9 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  10. Scope of work In scope Deep website-related focus on Parliamentarians and Civil Servants groups (who we were able speak to) Snapshot of the website experience of a broad range of audiences Not in scope Comprehensive research study into all types of Civil Service user groups Comprehensive research study into all Parliamentarian user groups 10 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  11. Where we are so far Done Understood organisational needs and existing research Conducted qualitative user research to deeply understand needs of key audiences Surveyed all key NAO audiences to understand broader needs * Distilled our research into clear insights and recommendations Next Work with you to prioritise recommendations as hypotheses that we can test Work with you to create website concepts to test with our two user groups Report with user feedback and design recommendations from concept testing Review the technical implications and feasibility of the recommendations * Survey findings to be presented separately 11 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  12. Approach Mace & Menter - Private and Confidential

  13. 3. Approach 3.1. Approach 3.2. Methodology 3.3. Recruitment 13 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  14. Approach Mixed method research approach A critical part of our work is to gain a deep understanding of your key audiences We do this in a number of ways combining both qualitative and quantitative research methods to ensure that we balance the need for a deep of understanding of your core audience groups whilst also gaining the breadth to cover broader audiences. 14 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  15. Methodology Quantitative research Analytics - We reviewed all of your relevant analytics data to help inform our understanding of current website behaviour Surveys* - We have surveyed a broad spectrum of your audience groups to increase the breadth and depth of our insight Response - promoted via the website, social media and sent out to all key NAO audience groups mailing lists, we have so far received: 842 responses * Survey findings to be presented separately 15 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  16. Methodology Qualitative research We conducted a total of 17 semi-structured 1 hour depth interviews with: X7 Parliamentarians X10 Civil servants 16 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  17. Understanding our audiences Persona summary Mace & Menter - Private and Confidential

  18. Persona summary Please note: We won t have time to go through the detailed persona groups today, however here s an overview to help give you some context when we are referring to specific types of users. We will share the complete presentation in due course. 18 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  19. Civil Servants - Persona summary Collaborators Specialists Collaborators are civil servants who work with the NAO to: Specialists are civil servants who: who work in strategic roles driving policy development, project/programme delivery or in operations Support its financial & VfM audits or Provide subject matter expertise and advice to help inform reports and investigations. 19 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  20. Civil Servants - Persona summary Collaborators Specialists Audit Collaborator Specialist: Policy / Operations / Delivery Shared interest Partner Collaborator Report Collaborator Who they might be Director level SCS1 (treasury) official Who they might be Director or grade 6 working in finance / audit / risk assurance Who they might be Directors / specialists in govt depts or from a public body Who they might be Senior (grade 6): Head of Policy / Delivery / Operations or public scrutiny body specialists What they do Scrutinise public spending & value for money / attend PAC committee meetings What they do Manage and audit financial accounts and risk assurance What they do Lead and manage specialist govt depts or public bodies What they do Devise processes & ways of working to support policy development, delivery planning, business case appraisals often during the inception phase. NAO related goals Work with NAO to support VFM & financial audits of their departments + sign off NAO related goals Offer subject matter expertise Advise on report focus areas Supply knowledge / data NAO related goals Understand NAOs remit and how best to work together NAO related goals Understand NAOs value proposition Best practice guidance on ways of working Cross-govt learnings & case studies 20 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  21. Parliamentarians - Persona summary Commons Staff MPs Staff Committee Specialist Scrutiny Unit Person* Clerk Political Advisor Library Researcher What they do Answer private enquires from Members of Parliament Who they might be Economist, statistician, financial analyst, policy expert Who they might be Staff of a Select Committee such as PAC, Treasury, DCMS Who they might be Chief of Staff, Senior Political Advisor Who they might be Legal specialist, statistician, financial analysts, economist, policy specialist What they do Support a MP in their day-to-day tasks, including preparing for briefings, debates and committee meetings What they do Support the day-to-day running of the committee and its members NAO related goals Reference NAO reports in their findings What they do Provide specialist help to select committees, writing briefs & reports What they do Providing scrutiny of public expenditure and pre legislative draft bills NAO related goals Looking at particular topics, themes or constituency interests that match those of the MP they support NAO related goals Direct members to NAO reports and recommendations NAO related goals Reference NAO reports in their own reports NAO related goals Use NAO reports to scrutinise public spending * Did not interview in this study 21 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  22. Executive summary Mace & Menter - Private and Confidential

  23. 4. Executive summary Broad unmet need for department agnostic theme and issue based knowledge on the website Finding reports is the primary motivation to visit the website, however once found, users were offered no compelling reasons to stay on the website and would leave Finding recent & specific reports was seen as fairly easy, however finding thematic content was difficult Civil service specialists were the group that felt their needs were least supported by the NAO Clear unmet need from all civil service specialists for cross-government best practice guides and case studies to show them how to improve ways of working in their departments MPs staff are very time poor, having to find and distill information quickly, they did not want more general newsletters, they wanted concise information tailored to be relevant to the interests of their MP & constituents Civil service auditees wanted clearer guidance and transparency about the audit process 23 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  24. Findings Mace & Menter - Private and Confidential

  25. 5. Findings 5.1. All users 5.4. Personas 5.2. Civil servants 5.5. Experience maps 5.3. Parliamentarians 25 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  26. Findings All users Mace & Menter - Private and Confidential

  27. All users Overall website experience What we heard Quotes Participants across all user groups experienced issues with the website. Those whose need was only to find specific recent reports, were the most satisfied, whereas those looking for broader information were much less satisfied. Took me a long time to find a report Participant 2 Search function isn't that great Participant 10 The design of the website was seen as dated, busy and text heavy . Home page looks quite busy... difficult for me to quickly understand if there s anything relevant to me Participant 11 Parliamentarian users were more likely to have issues with the findability of content, whereas Civil Servants (specialists in particular) were unsatisfied with the lack of content for them. I find the website very text heavy, text is also quite small, compared to GOV.UK where it is a bit more spaced out and inviting. Participant 15 Some Parliamentarian users felt that the website performed adequately, however this view was to an extent informed by their low expectation from public sector websites as a whole. 27 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  28. All users Once users find a report they have little reason to stay What we heard Quotes Finding reports was most likely motivation to visit the website, however once found - participants were offered no compelling reasons to stay on the website and would leave. For many participants, the website was merely a destination to find publications and awareness of other types of content was very low. There is this treasure trove of insight that nobody knows about Participant 17 People think the NAO just does VfM and Audit reports, they do so much more and people should be able to access it Participant 16 What this means There was clear unmet need to surface and join up more related and diverse content (not just reports) giving your users a reason to stay on the website, supporting onward journeys and more engagement into NAOs depth and breadth of knowledge. 28 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  29. All users Ease of finding reports was dependent on recall and recency What we heard Quotes Using the website to find recent and specific reports where the participant remembered the report name was seen as being fairly easy. However, finding reports based on topics or where the report name was not fully remembered, was difficult. Many participants felt that searching via Google was easier than using NAOs site search. Search function isn't that great. Not sure what category it is in... have used filters...hard to know how to filter Participant 7 It is often difficult to know which type of report is which Participant 15 What this means The website search and filter features does not adequately support users finding topical information and easily being able to narrow down and find what they are looking for. Tend to use Google as it s much easier Participant 2 29 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  30. All users Need for department agnostic themes & issue based knowledge What we heard Quotes There was a clear unmet need for department agnostic themes and issue based knowledge on the website from both parliamentarians and civil servants. Being more of a government resource would be helpful... themes from audits and best practice (cross-government) with examples of positive case studies Participant 13 What this means If it was more agnostic of department or reports and more about a particular problem I would use it more Participant 17 MP s staff wanted to understand what knowledge the NAO had on specific issues important to their MP and constituents. Be good to see a more informal blogs, e.g. BoE s Bank Underground good to have themes such as 'procurement' Participant 2 Civil servant specialists wanted to find thematic information to help support their roles such as guidance on procurement or working from home during the pandemic. 30 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  31. All users PDF was the favoured report format but there was openness for change What we heard Quotes Participants broadly preferred to read publications (digitally) as PDFs. Those who (pre-pandemic) would mainly read printed reports were more likely to view PDFs as the closest equivalent and most favoured this format, however there was a willingness from all participants to move content to an HTML format - as long as there still was an option to download as a PDF. I much prefer it to be in PDF, I hate HTML but it wouldn't put me off if it was in HTML as long as I could still download the PDF Participant 9 As long as you have option to print out the report even from HTML format it will be fine Participant 6 What this means HTML should be for engagement with bullet points of an overview with the ability to use chevrons to go further with a link to the original [PDF] if you want to go deep [cites Erskine May website] Participant 4 Switching to an HTML first publishing model won t put off users from visiting the NAO website and opens a significant opportunity to make your content more accessible, findable, and reusable. 31 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  32. All users Digital behaviour shifts and workarounds during the pandemic What we heard Quotes Many participants missed being able to print reports (driven particularly by a need to make notes and highlight important areas) some changed their view and behaviour (preferring digital for shorter documents or when needing to search) and found digital workarounds for annotating documents such as using OneNote and Office 365. A year ago, it would of been a printed doc as it s easier to annotate... but not always the case, if I quickly want to understand something now I would prefer to look at it online Participant 14 Found my own way to classify reports by putting them in OneNote, I also make my own notes in OneNote Participant 14 What this means One of the key motivations to print reports is the ability to then make notes. There is an opportunity to explore how digital note-taking can help shift more users away from print and PDFs. I like printing stuff off and annotating on paper but ok with PDFs and adding annotations Participant 9 32 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  33. Findings Civil servants Mace & Menter - Private and Confidential

  34. Civil servants Specialists were the least supported group What we heard Quotes Civil service specialists were the user group that felt least supported by the NAO and its website. Nobody knows what it is they are doing so no one will look to them for advice, even on their website, it doesn't tell us what they do. Participant 14 There was a strong unmet need for guidance to support common tasks such as writing business cases, conducting appraisals, developing policy and procurement good practice. Does not tell me how to solve a problem Participant 17 All participants stated that the NAO could do more to better support their needs. There was also some lack of clarity of NAOs role, remit and offer to them. NAO should be involved with policy development, but wonder if it is their role to do this. Maybe not. Participant 16 What this means Their unmet needs represent the largest gap and opportunity to improve the website to better support a currently underserved group. 34 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  35. Civil servants Unmet need for cross-government good practice guidance What we heard Quotes All civil service specialists felt that the NAO should do more to support their unmet needs for department-agnostic best practice guidance and case studies to help them solve common cross- government problems. So much goes wrong across government and repeat problems that the NAO knows about that they could help civil servants improve... by linking civil servants to guidance and examples of where this has happened before. Participant 17 What this means Should be something on the website that says on are you looking to develop a new policy - here s some guidance with an overview about why this work is important, steps involved and then provide guidance. Participant 16 There is a clear opportunity to better leverage NAO s (dept /govt agnostic ) knowledge and learnings to provide this good practice guidance and help them solve problems. 35 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  36. Civil servants Auditees want more clarity about the audit process What we heard Quotes Civil service auditees (people working for departments being auditing by the NAO) wanted (either for themselves or new/junior staff) clearer guidance and transparency around the audit process, specifically what to expect, what is expected from them and case studies showing them how audits work. We want to know more about the process of audits, so we can make sure we can have the right people helping the NAO Participant 5 Would be good to share best practice across govt for audits what the NAO expect, what evidence requirements, case studies etc this would make the process more transparent Participant 12 What this means Auditees would value practical guidance on the website detailing the audit process to help give them reassurance. 36 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  37. Civil servants Untapped treasure trove of insight What we heard Quote There was a perception from participants that the NAO had a wealth of insights and knowledge, particularly around cross-government learnings and guidance that either existed but they couldn t find or that the NAO was not making available on the website. There is this treasure trove of insight that nobody knows about because I do collaboration with them outside of audits I know they have this wealth of insights and advice. What this means Participant 17 This is an opportunity to increase awareness of the breadth and depth of NAOs knowledge by doing more to surface and highlight this information on the website. 37 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  38. Civil servants Information sources - Civil Servants We asked participants during the interviews where they look for information to support their role beyond the NAO. Civil Servant specialists gave the following examples which offered them guidance around good working practice, something which they felt was missing from the NAO. Private sector Harvard Business Review Government and Public sector IPA Thought Leadership and trends in ways of working Lean.org Government s centre of expertise for infrastructure and major projects The Green Book Organisational best practice and ways of working HM Treasury guidance on how to appraise and evaluate policies, projects and programmes Supreme audit institutions New Zealand ALB 38 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  39. Findings Parliamentarians Mace & Menter - Private and Confidential

  40. Parliamentarians New and infrequent users faced a steep learning curve What we heard Quote You do have to learn (the website) and spend time with it - there is a learning curve... I mainly use it if I need to refer someone (colleague) to a publication or something coming up. My role is about funnelling information to people. Participant 1 New and infrequent users faced a steep learning curve to find information they were looking for and often gave up. They would look to colleagues more familiar with the website or get in touch directly with NAO staff if they had an existing contact. What this means Took me a long time to find a report [on the website] Participant 2 There is a need to make website more user friendly and intuitive to new and less frequent users. Most frequent, long time users of the website have become accustomed to its limitations as expert users , however they also accept there are areas to improve. 40 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  41. Parliamentarians Not always clear how the NAO website can support their role What we heard Quote First time [I went to the NAO website] was when I was being inducted into my current role. My predecessor showed me the pages that would be most relevant for me. Participant 1 It wasn t always clear to Commons Staff how the NAO and its website can help them in their role. Specifically around NAO s proposition to them. For new staff members coming through this is further exacerbated due to high-levels of staff turnover. What this means I signed up to an induction with NAO to go through what they do, but could not attend. They tend to reach out to me and initiated on their side to say if I need any help or any docs. Participant 9 It may not be feasible for the NAO to actively initiate direct support to new staff. However, this support could be provided by dedicated Commons Staff induction pages or videos on the NAO website. Didn't even know they had a website! Participant 4 This will enable them to better self-serve and clearly understand NAOs proposition for them and the ways it can support their role. 41 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  42. Parliamentarians Little time or opportunity to read reports in depth What we heard Quotes There are frequent scenarios where reports need to be distilled on someone else s behalf so that they understand the key points. This could be ahead of a committee hearing, parliamentary debate or part of a media engagement around a particular event I support [committee member in their] Committee role. I try to read the reports before to help shape the questions, but that wasn't sustainable. Participant 10 What this means The chair on the committee goes through the reports and creates bullet points for everyone else. No one else on the committee goes near the NAO report. Better to make it more concise. Participant 4 There is a strong need to publish a summarised version of the report that will allow someone to quickly understand the context, key findings and any recommendations that were made. This will allow users to quickly determine if a report is relevant to a particular need and have time to form a political angle on the findings 42 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  43. Parliamentarians MP s staff key to understanding and distilling reports What we heard Quote Due to MPs being very time poor, there is a dependency on their staff to prepare them for their day-to-day work. They rarely have time to read reports so will often ask their staff to brief them on the key findings and recommendations I used to go to the NAO briefings with [MP] in person in our office so we can chat. But now if you don't get an invite you assume you can t go, it is all on Teams. I do not directly deal with NAO now What this means Participant 10 Before the pandemic, some MP s staff could arrive at an in person report briefing within parliament. Since restrictions are in place, these briefings are often held virtually on Microsoft Teams and consequently staff members are often not invited to attend 43 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  44. Parliamentarians Keep me updated but only with what's relevant to me What we saw Quote Due to very specific topical interests from advisors and commons staff such as committee specialists , there was a strong need to be notified of any new reports or developments so that they can remain fully informed. I would appreciate direct marketing from NAO to be reminded or notified of any work they have done on the specific interests I have What this means Participant 6 Although blanket emails was seen as a potential burden, there is evidence that if the content was targeted to a relevant topical or political interest then it would be of great use. This would mean reduce the dependency on having one-to-one relationships with NAO contacts to be kept updated. 44 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  45. Information sources - Parliamentarians We asked participants during the interviews where they look for information to support their role beyond the NAO. They gave the following examples as sources of valuable information. Private sector Chatham House Royal United Services Institute Government and Public sector Erskine May on Parliament website Institute for Government Office of Budget Responsibility Office for National Statistics World Health Organization 45 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  46. Experience Maps Mace & Menter - Private and Confidential

  47. Experience maps Experience maps are a visual representation of an end-to-end experience that a person goes through when trying to accomplish a goal. Speaking to a range of users helps us to understand what users do, think and feel over time and may experience throughout their journeys. Consolidating these into a single map helps us to understand: How users currently accomplish a goal How things work (or don t) Interdependencies - for example, between different departments or services Pain points and where things are broken Opportunities to improve 47 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  48. Experience Maps - Parliamentary Committee These will be made available during the project 48 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  49. Experience Maps - Financial audit / VfM report These will be made available during the project 49 Mace & Menter - Private and Confidential Mace & Menter - Private and Confidential

  50. Recommendations Mace & Menter - Private and Confidential

Related


More Related Content

giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#