Rotterdam

1. Enablers

1.1 - Skills

Does the local authority have internally any position such as service designers OR user researchers OR user experience experts?

Yes

If yes, provide the evidence, including the approximate number of such positions

25 The employees assigned in such positions include the UX lab staff and experts (3 UX experts and 2 UX/service designers), WCAG experts/testers. We have multiple people assigned as designers within DEvOps teams etc. CV's or organization schedules available when desired

Has the local authority provided training on service design or user research to its employees in the last three years?

Yes

If yes, provide the evidence, including the approximate number of people trained

Strictly speaking this is done under municipal funding and direction by a separate partner: www.rotterdam.hetinformatiepunt.nl. the number mentioned was 3292 in 2021, numbers 2022 not yet available.

Has the local authority provided training on ICT to its employees in the last three years?

Yes

If yes, provide the evidence, including the approximate number of people trained

Hundreds (via innovation network training, specialists training, plus we have the 20 person designer pool in Rotterdam)

Has the local authority provided training on ICT to citizens in the last three years?

Yes

If yes, provide the evidence, including the approximate number of people trained

Thousands (on privacy, security. MS apps, etc) (feb 2023: still applicable: mandatory training apply for numerous aspects of ICT such as security and privacy aspects, and numerous applications)

1.2 - Strategies

Does the local authority have a digital strategy less than three years old?

Yes

If yes, provide the evidence

https://www.rotterdam.nl/wonen-leven/rotterdam-digitaal/ update: still applicable 2023

Does the local authority have a position of Chief Digital Officer or equivalent?

Yes

If yes, provide the evidence

https://nl.linkedin.com/in/basboorsma?challengeId=AQEaQKoZQATU6AAAAYBlppLu4NgpiuYidlgC5jzgG8H_6zUx0CPcAw0tPF7kZzM-buxUF6bavKqV-zEozvEvoiTnBH1eQSh8VQ&submissionId=8a3c6dd2-106f-e916-e6f1-4b5367a2e343 (Update 2023: still applicable)

Does the local authority have formal service standard (like for instance the UK Government Digital Service standards)?

Yes

If yes, provide the evidence

Update 2023: Standards have changed from service standards to "citizens satisfaction". Internally though, may quality related items are measured, monitored and reported, such as: waiting times, response time website/social media, call-back time, etc. The reason for this change in approach lies in the fact that there is only limited correlation for service standards with citizen satisfaction, our main goal.

Does the local authority have design guidelines valid across departments, including for instance standards or protocols for simple language?

Yes

If yes, provide the evidence

Specifically for Simple Language, a whole team is working on this (NB, this is an intranet page): https://rio.rotterdam.nl/Project/Duidelijketaal. furthermore we have a WCAG compliant cross-departmental guideline for (digital-) design (Rotterdam DesignSystem RODS as presented in Rotterdam to UCC community, see ppt provided. Read also: https://www.rotterdam.nl/toegankelijkheid

Does the local authority have in place formal methods to monitor and enforce such service standards and design guidelines?

Yes

If yes, provide evidence of monitoring methods

There are different dashboards applicable to this item, all used to monitor and govern our standards, for instance: Net Promoter Score System Usability Scale powerBI dashboard site improve etc... Dashboards are only accessible from within our network. Screenshots available on request

Are service design or user experience mentioned in digital strategy or some other strategy level document?

Yes

If yes, provide evidence

https://www.watdoetdegemeente.rotterdam.nl/apps/voorjaarsnota/programmas/bestuur-en-dienstverlenin/burgerzaken/Programma-Dienstverlening-2019-2022.pdf We sent the NEW programme as a pdf-file to UCC. The programme is called De Rotterdamse Dienstverlening, which roughly translates into “The public services of the City of Rotterdam”. In this policy paper, we don't literally mention the words service design and/ or user experience. This is intentional. Because these words clearly aren't common language for our citizens, nor widely used within our organization (yet). In the pdf-file, we highlighted and translated the relevant parts in which policy refers to service design and/ or user experience. The city Counsil has yet to approve this strategy, it is sceduled for debate soon. This comment also applies to the current programme although we chose to highlight the passages in the NEW programme rather then the old one because it better reflects our commitment the upcoming 4 years

1.3 - Ecosystem

Does the local authority provide APIs to other administrations and to private companies?

Yes

If yes, how many external organisations (public and private) access the API monthly on average?

Yes we do. Data and API’s are provided through the municipal open data platform, which can be accessed here: https://rotterdam.dataplatform.nl/#/data An example of a provided API: https://rotterdam.dataplatform.nl/#/data/6a58bf9f-62b4-4d15-aed1-e3368c46418e?tabName=download_tab&tabId=3

Does the local authority use standardised services modules, provided at national or European level (e.g. CEF building blocks, national payment service or eID)?

Yes

If yes, provide evidence of standardised services modules

https://dloket.rotterdam.nl/loket/burger/digid-akkoord?dest=%2Fburger%2Fstartform%3Fid%3DDV_VHZ&auth=DIG1 national standards for payment is used (Ideal for Netherlands https://www.ideal.nl/en/) and DigiD for identification as a national standard: https://www.digid.nl/en and many others)

Has the local authority carried out within the last two years innovative forms of procurement, such as pre-commercial procurement, hackathons, other forms of collaboration with SMEs and startups?

Yes

If yes, provide evidence of collaborations carried out

Innovative procurement is done on average in every 10 EU-procurement procedures. Recent examples are what we procured for our digital twin. evidence available on request, these are internal documents

2. User-Centricity Performance

2.1 - Co-creation

Does the local authority habitually (as standard practice on every new digital service) use service co-design / user research sessions in developing its services before their launch?

Yes

If yes, provide evidence on the type of service sessions

This can be divided in 1. commissioned service design (privately owned studios), 2. internally commissioned research e.g. by the UX lab, and 3. initiatives by individual project managers in cooperation with service design experts. 1. in 2022 approximately 280.000 euro has been spent by Rotterdam on service design assignments. The number might be much higher due to the fact that not all procurement's contain the words "design" or "servicedesign". 2. approximately 25 assignments per year, for instance www.rotterdam.nl website testing with citizens, chat010 chatwidget co-creation and testing process 3. An example https://www.linkedin.com/posts/bjorndirkse_videocalls-design-contextmapping-activity-7027197720100491264-eSZX?utm_source=share&utm_medium=member_desktop Before the launch of the new website, more than 10.000 citizens participated in tests of the beta version, resulting in valuable feedback. It seems that Rotterdam is the first in the Netherlands to have chosen this co-creation strategy for the main website. Press coverage: https://openrotterdam.nl/nieuwe-website-gemeente-rotterdam-voor-en-door-rotterdammers/

Does the local authority habitually carry out users' research sessions on a regular basis (at least once a year) AFTER the launch of a service?

Yes

If yes, provide the evidence, including the approximate number of sessions or people of citizens that have engaged in service co-design practices in the past two years

Of course our public services use the reviews and responses of the public to make iterations and changes to existing services. One of the ways this is done is by the UX lab. The UX lab tested about 15 applications in 2022, among them some internal, but also multiple external applications such as the website. At least 7 citizens are involved per assignment. documentation available. The UX lab's turnaround in 2022 was around 156k, in 2023 we expect to double this. The UX lab provides testing, co-creation and other forms of design related activities both before as after launch of a

Does the local authority habitually use specific service co-design / user research sessions with disadvantaged communities (e.g. minorities, elderly, disabled people, etc.)?

Yes

If yes, provide evidence of research sessions with disadvantaged communities

Disadvantaged citizens are taken into account when developing public services in which it is required to do so. We work with for instance https://www.lezenenschrijven.nl/ and https://www.meerotterdamrijnmond.nl/. We have articles and pictures of such processes available on request. In 2022 700 municipal employees have been trained in using simple language to provide better service and communication for citizens with limited language skills. evidence available on request.

Does the local authority habitually use web analytics and other service data to improve digital services (e.g. completion rates and most frequent errors)?

Yes

If yes, provide evidence of web analytics examples employed

We use dashboards (Cognos) to show the number of calls, the topics, the service level norms, etc We use siteimprove to monitor errors, dead links, behavioural mapping etc. super cool stuff. screenshots available on demand.

Does the local authority habitually plan for releasing regular (at least once a year) new releases for existing digital services (not including technical updates automatically provided by software provider)?

Yes

If yes, provide evidence

Multiple feedback loops, such as customer signal framework, UX testing etc leed to a constant process of iteration and improvement of services. Is it habitual: absolutely. planned and regular: we prefer to work agile and devops, which means we do whats needed, when needed, and (citizens) needs driven, which doesn't allow for a strict schedule or plan. We can deliver evidence of customer signal framework analysis on request, and more info on this customer feedback and agile improvement loops. For example: the Digital Counter (launched in 2020) will be updated in march to version 7.0.

2.2 - Supply of online services

Does the local authority provide the majority of services fully online (out of total services provided that could potentially be digitalised)?

Yes

If yes, provide evidence of the percentage of service provided online

https://dloket.rotterdam.nl/loket/burger/digid-akkoord?dest=%2Fburger%2Fstartform%3Fid%3DDV_VHZ&auth=DIG1 After logging in you can access all relevant sources. Another platform is: https://www.rotterdam.nl/ due to national policy (Plasterk 2017) it is mandatory for local authorities to provide public service digitally/online where possible. A "maturity" scan on Rotterdam products (406 in total) shows that only 25 of those are physical service only. 75 products are (also) digitally available on level 1, which means "application by email or PDF form" and the others (306) are fully digital if desired by the citizen. This leads to a percentage of almost 94% is available digitally, 75.4% is available fully digital. Evidence available on request from internal dashboard.

Has the local authority already fulfilled the requirements of the <a target="_blank" href="https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=uriserv:OJ.L_.2018.295.01.0001.01.ENG&toc=OJ:L:2018:295:TOC">Single Digital Gateway</a> (deadline end of 2022)?

Yes

If yes, provide evidence of how many and what requirements of the Single Digital Gateway have been fulfilled

We are compliant to phase 1, phase 2 is work in progress. https://www.waarstaatjegemeente.nl/Dashboard/dashboard/landelijke-voorzieningen This dutch national website website works like an app, the url stays unchanged for any selection. If “Rotterdam” is selected on the site, it provides a dashboard of where every municipality in the Netherlands stands on different legislation and compliance, among them are the SDG guidelines (bottom right). The dashboard is curated by the VNG, the Dutch organization for municipalities, and provides audited data. Screenshot sent to UCC for evidence

Does the local authority provide habitually to citizens the possibility to check online the progress status of the services they request?

Yes

If yes, provide evidence

Citizens have access to their own portal, https://www.rotterdam.nl/loket. Behind the security wall (inlog with DigiD, dutch digital ID) they find the progress of their service request. screenshots available on demand

2.3 - Useability

Does the local authority habitually carry out useability assessment of its online services, using standard tools such as System Useability Scale (SUS)?

Yes

If yes, provide evidence of usability assessments carried out

SUS is standard methodology in almost all UX research processes. The Digital Counter measures SUS for most of the videocalls by automatically sending survey links to the citizens who used the service. Evidence available on request through our dashboards serviced by Expoints.

Do all the websites or websites sections of the local authority have consistent design and look&feel?

Yes

If yes, explain how (e.g. use of colour palette, consistent text size and fonts used, etc)

For all digital content and applications the use of Rotterdam Digital Design System is mandatory for all creators, both internal and external. https://www.rotterdam.nl/huisstijl provides all guidelines for look and feel (online and offline as well as AV for instance) with more specific on UX: https://www.figma.com/file/ZWSC4gCrOXRUR9UX3aoZ8x/Gemeente-Rotterdam-Design-System?node-id=413%3A13001

Does the local authority provide the possibility to citizens to have live audio/video interaction (such as videoconference or single telephone number)?

Yes

If yes, provide evidence on what type of interaction

Digital Counter (Digitale Balie) is the standard for public service through video interaction: https://www.rotterdam.nl/videobellen Video calls, access to relevant digital processes including the right authentication methods etc. The Digital Counter is the most advanced videoplatform for local governements available, and invented and developed by the municipality. The SUS score rates about 78, NPS 35 avarage. 1 central phone number/point of entry: 14010 https://www.rotterdam.nl/bellen

Are the local authority web services in line with <a href="https://www.w3.org/TR/WCAG21/" target="_blank" >accessibility guidelines (WCAG)</a>?

Yes

If yes, provide examples of WCAG guidelines compliance with web services

Most web services are, evidence: https://www.rotterdam.nl/toegankelijkheidsverklaringen. some (mostly newer or innovative) applications are under continuous testing and auditing to require the necessary certification.

2.4 - Security and privacy

Are the users able to use national eID, as a means of authentication for online services requiring authentication?

Yes

If yes, provide evidence

for instance see https://dloket.rotterdam.nl/loket/burger/digid-akkoord?dest=%2Fburger%2Fstartform%3Fid%3DDV_VHZ&auth=DIG1 where citizens can register their change of address themselves. the website asks for EID (DigiD in NL), as is for all online services.

Has the local authority put in place measures to ensure citizens' control over the data held about them (such as seeing who has access to the data and for what reason, correcting data, etc.)?

Yes

If yes, provide evidence of the measures put in place to ensure citizens' control over the data held about them

See https://www.rotterdam.nl/uw-gegevens

Has the local authority in place documents on measures, practices or procedures on ICT security?

Yes

If yes, provide evidence of what documents

We have a CISO, who is in charge of making sure our security is both as locked as possible, as well as usuable as possible at the same time (balancing act). We take a lot and serious care of our security (not just technical, social as well). In the Demand Supply procedures Security is by default one of the checks before development or procurement starts. Development is security and privacy by design. The development department has standards for security checks, penetration tetst and procurement of external expertise through onboarding bootcamps etc. lots of documentation available on demand, internal documentation.

2.5 - Citizens redress and feedback mechanisms

Does the local authority provide online mechanisms for both citizens and businesses to complain and seek change to a decision?

Yes

If yes, provide evidence on what online mechanisms for complaints have been put in place

Online you can leave feedback: https://www.rotterdam.nl/klacht-indienen, or one can call us (14010) anytime and you can use an app to complaint about public spaces https://www.rotterdam.nl/meldr. Text mining technology scans all qualitative feedback on the complaint-channels, which is reported through the appropriate channels. The website mentions many ways to complain or give feedback https://www.rotterdam.nl/zoeken?zoekopdracht=klachten and theres a feedback button for website feedback and on all applications. A complaint will be coordinated by a central complaints-coordinator (1FTE), and can be referred to a departmental coordinator, available in every department.

Has the local authority put in place structured means for users to provide feedback?

Yes

If yes, provide evidence on how users can provide feedback

Among other means, the most evident example is the feedback button on websites and applications. for instance see www.rotterdam.nl (allow cookies)

2.6 - Interoperability

Does the local authority encourage the use of standards in IT development and procurement?

Yes

If yes, provide evidence

Most important standard is the common ground strategy which Rotterdam supports: https://commonground.nl/ Common Ground is a national ambition to get to a common architecture to make sharing, integration and connectability easier. Rotterdam also supports the national policy to encourage open source software development. This plays an important role in for instance the development of the Digital Twin: https://www.rotterdam.nl/digitale-stad, but also participation in chatbot GEM https://commonground.nl/groups/view/978adf62-97f2-4955-87cc-7fee0cd238e0/team-virtuele-assistent-gem

Has the local authority policies in place to encourage the use of open source solutions?

Yes

If yes, provide evidence

see above

3. Outcome

3.1 - Adoption

Considering only services that are available online, are the majority of transactions carried out online?

Yes

If yes, provide evidence of the percentage of transactions carried out online

Considering only services available online, then transactions of those carried out online as well will be close to 100%. For evidence also see reported number of digital services. more evidence (from internal dashboards) available on request

Does the local authority publish data on usage of online services (such as a dashboard with metrics on transactions) on a regular basis?

Yes

If yes, provide evidence of what kind of data on usage are provided

We use dashboards (Cognos) to show the number of online services etc. This is an internal dashboard. We also have the open data platform mentioned earlier: https://rotterdam.dataplatform.nl/#/data

3.3 - Satisfaction

Does the local authority measure the citizens’ level of satisfaction with regards to the services’ provision?

Yes

If yes, provide some evidence including the use of standard methods (such as Net Promoter Score)

Nps is used on many channels, but not all digital services. The burden for citizens needs to be minimal, so we selectively measure. There is a lot of data on NPS measurements on for instance phone, Digital Counter, Website and other services available on requests. This is available from within our network. Evidence available on request

Is the share of satisfied users above 80%?

Yes

If yes, provide the numbers

In 2022 we measured the satisfaction through NPS on different channels. Reporting over 2022 is still in progress and has not been published, but the number is 84,6%, measured on channels phone and counter/physical visits. Chat and video calls are excluded. From 2023 we will measure the percentage of not-satisfied citizens, as a governement target. This means the definitions might be deviating in the future. Evidence (data) is available on request, from internal dashboards and datasets.