Delivering the Census 2021 digital service

• Census 2021 exceeded expectations, with 97% of households across England and Wales taking part to make sure they are counted when it comes to decision-making on local services such as GP surgeries, school places and hospital beds.

• For the first time, this was a digital-first census with our online electronic questionnaire (eQ) receiving over 22 million submitted census responses from February to June 2021.

• Over just a few months, the Census 2021 website (including the eQ) recorded over 19 million user sessions and over 2 billion pageviews; as these counts were recorded through web analytic cookies, the total number of user sessions and pageviews is likely to have been considerably higher than this.

• Use of cloud architecture allowed us to scale up to meet the very high demand experienced on Census Day, where we were receiving just under half a million census submissions per hour at the peak.

• The success of the Census 2021 digital service shows that large government digital services can be securely delivered in-house using cloud architecture and Agile development.

With 88.9% of household responses to Census 2021 in England and Wales being completed online, our target of 75% online completions was easily outstripped. In total we received over 22 million online submissions during the three months the electronic questionnaire (eQ) was live (from 22 February to 25 June 2021). To achieve this, we built a highly scalable, secure survey data collection tool.

In this article we set out how and when respondents used the Census 2021 website and the electronic questionnaire, including the length of time it took them to complete and the types of devices they used. The technical elements section at the end of this article sets out some of the more technical details of how we delivered a successful digital service for Census 2021 in England and Wales, and Northern Ireland, including how we helped to keep the census secure.

Through telling the story of this digital-first census, what we have learnt from delivering it and the online behaviours of people using the digital service, this article provides insights that may help inform future large government digital service delivery.

How the website supported a digital-first census

Our published article, sets out how we designed a digital-first census that ensured everyone could take part in Census 2021, either online, on paper or through other channels.

During the census data collection period, the Census 2021 website served as the host for the secure electronic questionnaire (eQ), through which households and individuals across England and Wales, and Northern Ireland were able to complete their census. In addition, it provided a wide range of information to support citizens with questions about how to complete the census, enabled people to request paper questionnaires or online access codes, and hosted a comprehensive set of materials for local authorities, community groups and others to use to explain and promote Census 2021.

All households, communal establishments and, on request, individuals were provided with an online access code, either through a stand-alone census letter or on the front of a paper questionnaire. Across England and Wales, 9 out of 10 households received the letter. The remaining households, where the take-up of the online option was expected to be low but willingness to take part without further prompts (reminder letters or field visits) was high, received the paper questionnaire. Both paper questionnaires and online access codes were also available via the Census 2021 website and our contact centre. The vast majority of recipients chose to respond online.

For respondents in Wales, it was possible to complete the census in Welsh both online and on paper. The Census 2021 website was fully bilingual and hosted a Welsh language version of the eQ for Wales; users were able to toggle between English and Welsh while using the website and eQ.

Volume of access to the Census 2021 website

With all households invited to complete the census online - and nearly 90% of those that responded doing so (for England and Wales, and Northern Ireland) - the website and eQ received an extremely large number of visitors and pageviews.

Between March and the end of June 2021, 19.4 million user sessions and 2.1 billion pageviews were recorded. These are only those visits where users agreed to selected Google Analytics website usage cookies, which we estimate to be around 65% of the overall number of sessions, so the total page views for the Census 2021 website are likely to have been much higher, potentially around 3.3 billion.

When did people use the service?

Use of the Census 2021 website peaked on Census Day, 21 March 2021, corresponding with the peak of online census submissions.

Figure 1 shows this peak, with over 1.8 million unique visitors recorded to the English language start the census page on Census Day. Showing the recorded visitors from 22 February, when the electronic questionnaire went live, to 22 April, the chart shows the pattern of daily use of the website over the period when most households completed the census. As with completions, it shows a gradual increase during March and a drop-off after Census Day, with some spikes in activity during the "follow-up" period when households received reminder letters and visits from census field staff.

It is also useful to recognise the patterns of use across the day, which can help managers of digital products plan for future service and support design in the future. Showing the number of census submissions we received over the peak weekend, Figure 2 demonstrates a clear morning and evening peak on 20 and 21 March, but also continued high levels of usage throughout both days. The usage in the very early morning and late at night highlights the need for the 24-hour support we provided over that weekend and during the weekends either side.

Figure 3 shows the pattern of use each day of the week over the census period. These volumes, both for Census Day and the wider period, contrast with the evening peak observed in the 2011 Census (see Figures 2 and 3 in Providing the online census (PDF, 199.1KB)). The 2021 usage pattern may partly be explained by more respondents working from home because of the coronavirus (COVID-19) pandemic.

There is more information on how we managed this peak in the technical elements section, ensuring that respondents were able to get very fast response times and avoid latency as we scaled up and down to meet this.

How long did it take to complete the census?

The average completion time for the household electronic questionnaire (eQ) was 23 minutes 0 seconds (as measured in a session that reached the completion page and discounting sessions over 60 minutes and under one minute as outliers). This is slightly longer than for the 2019 Census Rehearsal, where the average completion time was 21 minutes 13 seconds.

This average completion time aligns with our expectation, reflected in the guidance in the census letter, that the census would take around 10 minutes per person to complete - the most common household size in England and Wales in 2011 being two people. The distribution of the lengths of sessions that reached the completed census page is plotted in Figure 4. This shows a peak with the most common duration (the mode) being 16 to 17 minutes, indicating large proportions being able to complete the eQ quickly.

Not everyone completed the census online in one go. Google Analytics figures show 3.8 million uses of the save and return capability in the electronic questionnaire; this figure includes only those who had chosen to enable non-essential cookies.

How people completed the census online: devices and browsers

We built the Census 2021 website to work across most browser platforms, this was to ensure that it was accessible to citizens who were not using the latest technology but who still needed to be able to access the website and the eQ. With the vast majority of households responding online, data on the devices and platforms used give us an excellent insight into how people access services online. It is helpful to understand the types of browsers and devices in use to inform priorities for digital service design, usability, and discoverability.

Through the period 1 March to 18 April 2021, where most of the use of the website and eQ was focused, the breakdown of devices used was:

• desktop (including laptop computers): 34.6%

• mobile: 56.4%

• tablet: 8.9%

These proportions are broadly in line with the proportions recorded for access to UK government websites in the period leading up to the census, though with slightly more tablet and less mobile phone use for the census.

Figure 5 shows the proportions of users who used each device type on each day during the census period. The proportion of users accessing the census using desktop computers (which includes laptop computers) peaked in mid-March when the initial contact letter was sent to households and again on Census Day. There is also an almost weekly cycle where desktops are more popular at the beginning of the week and then taper off to be replaced by mobiles and tablets. The exact drivers of this are unclear.

Figure 6 shows the proportions of citizens with the most popular browsers during the peak period of census collection; again, these figures represent those users who accepted cookies. Notable is the dip in the proportion of users recorded using the two most popular browsers - Chrome and Safari - over the census weekend itself, 20 to 21 March.

Table 2 shows the most browser types and device categories used by at least 1 million of our users who accepted non-essential cookies between 1 March and 30 June 2021 (a full table can be found in the technical elements section). As some browsers do not allow Google website usage cookies (for example, DuckDuckGo), and therefore do not feature in the list, we do not know their prevalence.

For Census 2021, we successfully delivered a secure, live, national and digital-first array of online services that were able to cope with nearly 9 in 10 households completing the census online in a short period of time. Doing this represented a massive technical challenge for the Office for National Statistics (ONS), which we undertook successfully.

It was designed, built and delivered in-house by ONS teams reflecting the efficacy and maturity of our technical capabilities (there is more detail on our use of Agile, continuous integration and how we secured this in the technical elements section). The high online completion rate both exceeded our expectations and demonstrates positive advances in digital inclusion. From this we are better able to understand the willingness of citizens to interact with national government services online and how they do so.

We have also demonstrated that it is possible to deliver high volume live services handling sensitive information reliably and securely using cloud infrastructure and we hope our experiences will be useful across other government departments, helping to continue this positive trend.

Managing the Census Day usage peak

The peak period on that day was between 10:30am and 11:30am. The electronic questionnaire (eQ) experienced very high volumes of interactions, making it one of the most used national digital services so far developed. The usage statistics for this peak hour include:

• 108,670,923 HTTP GET/POST requests (peaking at 30,186 requests per second)

• 427,838 Census submissions

• 126 clicks per submission (average)

• 11.429 seconds think time per click/page

• 23 minute average user journey time to submission

In order to manage this volume we had 468 instances of the eQs application in use, each with one Docker container. We opted not to use auto scaling (because of the latency this would have created and the impact on users) so we had a total of 885 containers provisioned with a further 225 reserved in case we needed them.

During the whole census period, the eQ product served 5.8 billion HTTP requests. We had a total of 1,097 HTTP errors, equating to a 0.00001862% error rate. On Census Day, there were no HTTP errors.

This expected Census Day peak carried live service risks that had to be managed. From a digital service best practice perspective, a flatter usage curve would be preferred in order to stabilise demand on system resources and avoid peaks. However, as the Census Day peak was predictable, as well as a better business outcome, we were able to build for it. We did this using flexible architecture, cloud-based infrastructure and excellent relationships with our cloud provider to build in massive scalability.

User-led, Agile and dev ops approach to delivery

Census 2021 is an excellent reference case for Agile delivery that can be shared across government. For several reasons, in-house Agile development was well suited to deliver the census; chief amongst these was the need to understand the implications of operating a live, scalable service at the same time as we were building it.

In digital best practice, it would be possible to run a live service and iterate over months and years to improve that service. However that was not an option for Census 2021, where the suite of technologies (website, eQ, integrations and so on) was needed only for a discrete period and it was important for the eQ to be provided consistently throughout that period.

Although in-live iteration was not an option for some parts of the census technologies, the the Office for National Statistics (ONS) ran the large-scale 2017 Census Test and 2019 Census rehearsal, which enabled us to test the basic solution. We also undertook a vast amount of user research on the products we were building to get feedback on usability and performance, making extensive use of face-to-face research with a broad range of user types.

We continued these investigations even after the first coronavirus (COVID-19) pandemic lockdown by switching to remote research, using screen-sharing software and phone calls. During the lockdown period that coincided with our main build, between March and October 2020, 135 user research sessions were carried out remotely, including accessibility testing to ensure the service was built in an inclusive manner. In addition, we had 104 participants who took part in unmoderated research, such as card sorting and surveys.

Making technical changes during live operations

Rapid improvement when we identified a need was a critical design principle because we were able to change some parts of the census while live and in response to what we were learning from watching user behaviour. To enable this, we used continuous integration along with disaggregated, microservice architecture to allow the quick implementation of change. During the first four months of the census service being live, we delivered 376 releases to live across the eight core products (as shown in Figure 7). We did this successfully without impacting statistical rigour, users or back end systems.

Keeping census secure

The safety of the information provided through the service has been our top priority in Census 2021 . As with any national critical service, designing against any potential cybersecurity attacks is extremely important to ensure service integrity and citizen confidence. We secured the Census 2021 online collection and support sites against two main kinds of attacks.

Type 1: Data exfiltration attacks - potential loss of citizen online response data:

• attempted hacking of our Google infrastructure

• attempted hacking of our data pipelines to the ONS ("man in the middle" attacks)

• spoofing of our brand or service - which could redirect responses

Type 2: Service disruption attacks and events:

• distributed denial of service (DDOS) attacks or targeted resource attacks

• greater than expected legitimate usage damaging the site performance

• speculative use of census resources for unintended purposes

• defacing the census website

Type 1 attacks - data exfiltration

We secured against Type 1 attacks by working closely with the National Cyber Security Centre (NCSC) and building robust architecture, which encrypted data both at rest and in transit. We also used service design principles to ensure that users did not accidentally share their data and had a way to fill in the census separately from the rest of their household if required.

In live operations our Security and Information Management Team operated round the clock surveillance through a dedicated Security Operations Centre. This supported phishing site detection where we could use the Netcraft platform to initiate takedowns of any sites attempting to scam the public into thinking they were being followed up for non-completion of the census (see Figure 8). Unfortunately, the trend of spoofing government services looks set to continue, so detecting fake sites and managing site takedowns is an activity that needs to be built into the support of key government digital services to maintain public trust. Our Communications Team was also active in getting the message out that the ONS would never need credit card details and educating the public in how to recognise a scam.

Type 2 attacks - service disruption

Type 2 attacks were prevented by having a robust network infrastructure making use of Google's Global Load Balancers as well as Google Cloud Armour . This gave us a highly scalable infrastructure, which was able to muscle its way out of a DDOS attack, as well as a way for us to quickly block suspicious traffic as it was identified.

We also used the Cloud Armour Managed Protection Plus service with its AI tooling to help us spot suspicious traffic and create rule sets to block it, and the Google Cloud Content Delivery Network to cache our more static web pages, effectively making the main route into the census questionnaire extremely robust (we tested these static pages to 1.8 million requests per hour without service degradation).

To protect the elements of functionality that made up the census services, we put limits on how they were used based on IP addresses, physical addresses (for letter requests) and mobile phone numbers (for SMS requests) to prevent excessive usage. Each functional element was individually secured so that it could only be called by an authorised application, ensuring that our resources, for example, address checking, were only being used by us.