Artificial Intelligence and Machine Learning

Technical framework 

3.1 Technical & Functional requirements 

Test requirements of AI/ML models and algorithms within LDT: 

Purpose: 

• The city/users of AI/ML models and algorithms should have a clear understanding of the goal to be achieved with an AI/ML model (i.e., what does the city want it to predict?). 
• The objectives of the city/users of AI/ML models and algorithms should align with the available data for training, validation, and testing (i.e., the data should be able to support the desired predictions). 
• The AI/ML models/algorithms should be in alignment with the available data and goals of the city in order to be able to generate trustworthy and understandable outputs. 

Data: 

• AI/ML models and algorithms should always use local data from the corresponding city for training, validating, and testing purposes whenever possible in order to best represent the local reality the city aims to explore and make predictions. 
• The AI/ML solution should use diverse training data that is representative of the local context it is meant to represent and that is regularly assessed for potential biases. 

Self-learning: AI/ML models and algorithms should be capable of learning and adapting based on real-time data inputs in order to support continuous improvement of the models and algorithms. 

Human oversight: AI system should incorporate solutions that enable human oversight to maintain human autonomy and reduce potentially negative impacts (e.g., via a combination of Human-in-the-loop (HIL), Human-on-the-loop (HOL), Human-in-command (HIC), or other approaches recognised and accepted as benchmarks) 

Explainability: 

• AI solution should allow users to understand why a given model has generated a particular output or decision (and what combination of input factors contributed to that) whenever possible. 
• AI/ML models should be designed whenever possible with interpretability constraints (e.g., provide a better understanding of how predictions are made by making it clear how variables are jointly related to form the final prediction of a model). 
• Black box models/algorithms should not be prioritised if technically equivalent and accurate interpretable models exist for the same purposes.1

Traceability:  

• AI solution should provide traceability features so users can track its predictions and processes whenever possible. 
• AI solution should facilitate the traceability and auditability of its models/algorithms, helping users to understand the outcomes of the models and how changes can be made (if necessary). 

Transparency: 

• An AI solution should clearly and proactively communicate its capabilities and limitations to users, enabling them to set realistic expectations for its use. 
• An AI/ML solution should communicate transparently to users that they are interacting with or using output from an AI system/model. 
• AI/ML processes and models should be made transparent and with sufficient documentation in order to allow users to understand which (and how) data is collected and processed by the models. 

Fairness: 

• The development and use of AI models/algorithms must obey fairness principles (i.e., AI-based solutions should aim to ensure equal and just distribution of benefits and costs related to the purpose of the AI models, and that groups and individuals do not experience unfair bias, discrimination, or stigmatisation – AI systems should neither cause nor exacerbate harm2 ) 
• AI solution should integrate fairness metrics into the models in order to assess the impact of predictions on different subgroups (e.g., Equal Opportunity, Disparate Impact, Demographic Parity, or other fairness metrics recognised and accepted as benchmarks) 
• AI solution should allow the application of constraints in order to minimise potential disparities in outcomes across demographic groups (or other groups of interest) during training and application. 
• AI systems/models should incorporate proportionality principles regarding means and ends between potential competing interests related to the models (e.g., data collection should be limited to what is strictly necessary, respecting individuals' rights and privacy, and approaches that have the least impact on fundamental rights and ethical norms should be prioritised whenever possible). 

Robustness:  

• AI/ML solution should be capable of handling exceptional conditions, such as input abnormalities or potential malicious attacks, while minimising the risk of causing unintentional harm as a result of its models. 
• AI system should implement robust security measures to ensure the safety of the data, models, and supporting infrastructure against potential malicious attacks (e.g., AES-256 encryption, robust Role-Based Access Control (RBAC), use of federated learning, Adversarial Detection Mechanisms, or other measures recognised and accepted as benchmarks) 
• AI systems should include security measures that allow for the activation of fallback plans in the event of a problem or malfunction (e.g., switching from statistical models to rule-based models, blocking decisions until human intervention is performed)2. 

Performance: 

• AI/ML models should be able to provide robust performance measures in order to assess the models' ability to make predictions or classify data (e.g., accuracy, precision, recall, log loss, confusion matrix, or others recognised and accepted as benchmarks according to the type of models being used) 
• AI/ML models should be able to provide likelihood/confidence levels for cases in which there is uncertainty associated with model results. 
• Privacy: AI systems should implement solutions that are fully compliant with applicable policies and legislations throughout a system’s entire lifecycle (GDPR, AI Act, EU ePrivacy Regulation (under discussions), or other applicable policies and legislations). 

Please consider that these requirements can be integrated within any template, except for network infrastructure and IoT devices templates, considering the necessary adaptations. 

4.2 Minimum Interoperability Mechanisms (MIMs)  

While embedding procurement templates to guarantee the successful adoption of AI/ML technologies across digital infrastructures, cities should also consider the Minimum Interoperability Mechanisms (MIMs) to enable sufficient interoperability for data, systems, and services, specifically in smart city solutions. More specifically, cities can refer to MIM5: Fair and Transparent Artificial Intelligence3  produced by OASC. At present, due to the 2024/25 MIMs redesigning framework, there is an ongoing transition towards a more comprehensive approach, which will focus on "Interoperable AI". For more specific information related to this topic, you can consult Guideline on Compliance with MIMs, standards, specifications and certifications.

4.3 Evaluation and Award criteria  

This section of the Guideline aims to provide insights and possible tender development options, addressing any gaps in specific technical and functional requirements for AI & ML solutions.  

Hence, for the evaluation and award criteria identification, selection and development, please refer to the technical and functional requirements mentioned above in section 3.1., as the minimum technical and functional requirements, de facto serving as Technical Specifications, to be then included in the chosen template among the available ones. 

Alternatively, if the most appropriate template is chosen from the available options, specific technical/functional requirements may not need to be provided. In such cases, additional discretionary criteria can be included in the tender evaluation & award criteria section (that is, for example: “Does the offer provide any AI & ML solutions for the given application as an add-on” – if a given tenderer provides AI & ML solutions, it will be assigned an additional rewarding score based on the proposal’s evaluation carried out by the CA itself).  

Legal framework 

While embedding procurement templates to ensure the successful adoption of AI/ML technologies, cities must also comply with the EU legal framework. This involves adhering to relevant EU laws, regulations, and guidelines to safeguard the ethical, transparent, and lawful integration of AI/ML in digital infrastructures. 

Of further assistance to cities and communities could be the consulting of Guideline on the EU Legal Framework on digital products and services relevant to LDTs, which offers an overview of compliance requirements for public procurement processes involving AI/ML technologies. 

Additionally, cities can refer to the EU model contractual AI clauses, available on Living-in.EU, to ensure standardisation and legal alignment when drafting contracts for AI solutions. These resources provide practical tools for navigating the legal landscape and fostering trust and transparency in AI deployment.