Project Title

Comparative Study on Best Practices to Detect and Avoid Harmful Biases in Artificial Intelligence Systems 

Project Year

2021   

Project Number

DESG 05 2021A 

Project Session

Session 2   

Project Type

Standard 

Project Status

Project in Implementation   
View Budget TableView Budget Table
|
PrintPrint

Project No.

DESG 05 2021A 

Project Title

Comparative Study on Best Practices to Detect and Avoid Harmful Biases in Artificial Intelligence Systems 

Project Status

Project in Implementation 

Publication (if any)

 

Fund Account

APEC Support Fund 

Sub-fund

ASF: Digital Innovation 

Project Year

2021 

Project Session

Session 2 

APEC Funding

50,000 

Co-funding Amount

Total Project Value

50,000 

Sponsoring Forum

Digital Economy Steering Group (DESG) 

Topics

Digital Economy 

Committee

Committee on Trade and Investment (CTI) 

Other Fora Involved

 

Other Non-APEC Stakeholders Involved

 

Proposing Economy(ies)

Chile 

Co-Sponsoring Economies

New Zealand; Russia; Singapore; Chinese Taipei 

Expected Start Date

01/10/2021 

Expected Completion Date

31/12/2022 

Project Proponent Name 1

Ignacio Acuña Frías (wef 10 June 2022) 

Job Title 1

Advisor 

Organization 1

Ministry of Science, Technology, Knowledge, and Innovation 

Postal Address 1

Not Applicable 

Telephone 1

Not Applicable 

Fax 1

Not Applicable 

Email 1

iacuna@minciencia.gob.cl  

Project Proponent Name 2

Antonia Moreno 

Job Title 2

Future and Social Adoption of Technology Unit 

Organization 2

Ministry of Economy, Development and Tourism 

Postal Address 2

Not Applicable 

Telephone 2

Not Applicable 

Fax 2

Not Applicable 

Email 2

amoreno@economia.cl 

Declaration

Ignacio Acuña Frías and Antonia Moreno 

Project Summary

One of the most essential considerations of developing and implementing Artificial Intelligence (“AI”) systems is how to balance the development and advancement of the technology with fundamental rights, such as, non-discrimination, privacy, and the protection of personal data. The study will conclude with the publication of a report that will provide (i) a comparative overview of the current status of the APEC economies with regard to implementing policy, regulatory and technical best practices; (ii) an outline of certain limitations of privacy and data protection laws and/or regulations in addressing bias; (iii) an examination of which other  types/forms of frameworks/levers need to be brought in order to prevent or avoid harmful biases in AI in connection with privacy and data protection; and (iv) a set of recommendations to the competent authorities of APEC economies, with a view to establishing guidelines for developing AI systems free of harmful biases.

Relevance

Relevance – Region: Artificial Intelligence (“AI”) as a general-purpose technology - understood as a set of computer techniques that enable a machine (e.g. a computer or telephone) to perform tasks that typically require intelligence, such as reasoning or learning -, is shaping how societies develop while being shaped by societies themselves. This phenomenon is not limited to economic development: AI has the potential to impact all aspects of our lives and may entail larger societal challenges such as climate changes and the preservation of democratic institutions.

One of the most essential considerations of developing and implementing AI systems is how to balance the development of the technology with fundamental rights and liberties, such as, non-discrimination, privacy, and the protection of personal data. The use of AI systems may impact life in a wide range of areas, including healthcare, education, law enforcement, work, and social responsibility. Identifying and attempting to resolve any tensions between ethical considerations and business objectives that arise from the use of AI in society (e.g., privacy vs. security; focalization vs. nondiscrimination) need consideration due to the potential that AI has to amplify biases (implicit, unconscious or intentional bias) and undermine laws that protect them (e.g., such as data protection and privacy). The direct participants in this kind of problems are Officials from Data, Science and Technology Agencies; Officials from Transparency Oversight Bodies; Executives from Big Data Companies; and Academia Experts.

This debate is entering a new phase and economies are exploring how to best regulate AI since its increasing embedment in companies´ products, services, processes, and decision-making represents several ethical and regulatory challenges (e.g., increases the potential scale for bias). Furthermore the development of AI involves a wide range of uses  which need to be taken into account in the face of the rapid pace of technological developments (e.g., inadequacies of existing legislation in APEC economies to combat AI). As such, this project will contribute to the systematization of the needs and best practices for developing and implementing AI systems while protecting fundamental rights such as, the protection of personal data and the prevention of harmful discrimination due to the development of biased systems. To achieve this objective, the technical team will need to define the most prominent AI systems and/or use case studies for the preparation of a set of recommendations for the development of unbiased AI systems. 

Relevance – Eligibility and Fund Priorities: The project will support capacity building needs for APEC economies through conducting research on the best practices to elaborate a set of recommendations for developing unbiased AI systems by executing a comparative study to identify policy, regulatory and technical best practices for detecting, avoiding and addressing harmful biases in AI systems—both the algorithm, outcomes, and other components such as data, biased models, societal changes, consumer interaction, developers biases—   during  its whole life cycle—from ideation and design to deployment, monitoring and disassembly—, as well as mitigating the risks of affecting privacy and data protection rights.

This project meet´s the three points of the eligibility criteria of the ASF Digital Innovation Sub-Fund, related to promoting the innovation and adoption of enabling technologies and services; facilitating the free flow of information and data for the development of the Internet and Digital Economy, while respecting applicable domestic laws and regulations; and enhancing trust and security in the use of ICTs.

AI systems can be developed by technology companies, universities, and entrepreneurs to solve various problems associated with the public sector, the productive sector, academia, or civil society. Given that the raw material of these systems is advanced human capital, technological infrastructure, and data, and that biases can arise from any of these, it is relevant that this be analyzed, and a recommendation can be given to policy makers on how to avoid these harmful biases. 

Relevance – Capacity Building: This project will build capacities of APEC member economies to address the risks and challenges presented by AI systems, by

1) Prioritizing specific use case studies or sectors in which AI may have a significant impact on fundamental rights such as data privacy or show bias in the training data that could be detrimental for society.

2) Taking stock of related policies and practices in the APEC region; learning from global implementations to drive success of the best practices to detect and avoid harmful biases in AI systems within APEC economies.

3) Raising awareness on how AI can affect life in a wide range of areas, including healthcare, education, law enforcement, work, and social responsibility among APEC economies. This should specifically address harmful biases and personal data breaches.

4) Developing recommendations for the use case studies or sectors prioritized, specifically on:

a. Detecting and avoiding harmful biases in AI algorithms and in other components associated with AI systems, such as the databases that feed them; the workforce building the algorithms; or the stakeholders engaged in their development.

b. Identifying privacy and data protection issues associated with the development and use of AI; and

c. Mitigating risks of harmful bias, including through levers such as privacy and personal data protection.

Objectives

The overall objective of this project is to collect best practices and develop recommendations for assisting APEC economies to address AI systems’ potential for harmful bias and risks associated with personal privacy issues.

Alignment

Alignment - APEC:  This project will contribute to two focuses of the APEC Internet and Digital Economy Roadmap: (1) Enhancing trust and security in the use of ICTs, and (2) Facilitating the free flow of information and data for the development of the Internet and Digital Economy, while respecting applicable domestic laws and regulations. To develop a knowledge framework/lever for APEC economies on how emerging technologies affect through implicit biases and data management (e.g., collection, ingesting and manipulation of data) is crucial to enhancing trust and security in the use of ICTs, and it is important to understand the flow of information and regulations that this kind of data needs. 

Alignment – Forum: This project is aligned with the APEC Internet and Digital Economy Roadmap (AIDER), in the following focus areas: 1. Development of Digital Infrastructure; 4. Development of holistic government policy frameworks for the Internet and Digital Economy; and 7. Enhancing trust and security in the use of ICTs. AI systems are key factors in relation to the development of better and efficient digital infrastructure. In this sense, to generate a study to compare best practices to detect and avoid harmful biases AI systems will improve the development of holistic government policy frameworks for the Internet and Digital Economy. Finally, it is important to understand the harmful biases in AI systems to enhance trust and security in the use of ICTs.

The international interest in the development of the AI and the challenges it presents, makes it a priority for DESG Data Privacy Subgroup (DPS)—to take a leading role in the discussion on AI, mainly related to detect and avoid harmful biases.

TILF/ASF Justification

Not Applicable.

Beneficiaries and Outputs

Outputs: 

1) Research Report: The report will include an international benchmark of different practices which will be carried out to identify how different economies detect and address harmful biases in AI systems (including data privacy issues).. This international benchmark will involve conducting desktop research and interviews with APEC economies and will additionally highlight case studies of best practice or where success has been proven and innovative legislative or regulatory approaches. This international benchmark will cover a wide range of areas of application setting priorities for the use cases or sectors where harmful biases and data privacy issues in AI systems can have a significant impact. This report will be the basis for the generation of the discussion tables that will be held during the online workshop.

The report will be 20-30 pages long and will have the following structure:

1.  Executive Summary

2. Overview of AI Systems and risks associated with harmful biases and data privacy issues

3. Main sectors and use cases relevant to address harmful biases and data privacy issues

4. Benchmark of practices in APEC economies

5. Case studies of best practice or where success has been proven

6. Key trends and findings

7. Conclusions

8. References

2) Online Workshop: A two-day online workshop will be held. Each day, APEC Economies will meet for approximately 4 hours to work in roundtables in which the Research Report will be discussed and a set of recommendations on how to address harmful biases in AI systems (including data privacy issues) developed.

The workshop will invite experts and policy officers from APEC economies. We expect between 20 to 50 attendants each day, to work in 3-6 breakout rooms.

The tentative agenda for both days will be as follows:

Day 1:

a) Presentation of the Research Report (60 minutes).

b) Queries arising from the presentation of the Report (30 minutes).

c) 30 minutes break

d) Breakout Rooms to discuss in roundtables the main sectors and use cases relevant to addressing harmful biases and data privacy issues (90 minutes). A key topic to discuss will be gender disparities and biases.

d)  Main conclusions of the day (30 minutes)

Day 2:

a)  Summary of the discussion in Day 1 (30 minutes)

b) Presentation of use cases of best practice  (4 cases, 15 minutes each)

c) 30 minutes break

d) Breakout rooms to discuss in roundtables the development of a set of recommendations to address harmful biases in the use of AI systems (90 minutes). A specific area where recommendations will be discussed is gender disparities and biases.

e) Wrap up and conclusions (30 minutes)

3) Recommendations Report: A final report will be generated, providing a a set of recommendations to the competent authorities of APEC economies, with a view to establishing best practices for developing AI systems free of harmful biases.This report will sum-up the discussions held during the online workshop and provide recommendations based on the research report. These recommendations will be 10-20 pages long and the structure of the document will be the following:

a. Summary of the key findings in the research report and online workshop.

b. Key recommendations based on successful case studies.

c. Key challenges for future discussion.

Outcomes: 

The research report will provide an up-to-date standard of the measures currently being applied, as well as provide economies with evidence for decision-making on harmful bias in AI. A survey will be circulated among the economies to measure its usefulness, completeness, and relevance. 

The online workshop will discuss best practices to foster an ecosystem of public policy generators associated with AI harmful bias issues. This will enable institutions to share experiences and discuss key recommendations on best practices associated with the area. We will measure assistance and generate a post-event survey to measure knowledge gain.

The recommendations report will provide recommendations, based on APEC economies, for issues associated with bias in AI and key challenges for future discussion. A final survey will be circulated to measure relevance and applicability of the recommendations, and see if stakeholders would continue discussing the key challenges proposed in future initiatives.

Beneficiaries: 

Primary beneficiaries

a)  Officials from agencies related to Data, Science and Technology;

b)  Officials from Transparency Oversight Bodies;

c)  Policy officials related to technology governance issues;

d)  Policy officials involved in digital government initiatives;

e) Executives from Companies using AI systems in sectors where harmful biases are relevant;  

f)  Academia Experts working on topics related to AI and biases.

g)  NGOs that work on topics related to harmful biases in AI.

The skills required are related to policy-making in technology governance issues, development and implementation of AI systems, and data management issues.

Secondary beneficiaries

Governments from APEC economies that are dealing with AI systems being implemented in governments and/or developing regulation related to AI. These beneficiaries range from government officials to elected authorities and firm executives

In addition to APEC economies, relevant international bodies will be invited such as the OECD, GPAI, UN, and OEA.

Dissemination

The documents generated as output 1 and 3 of this Proposal,  (research and recommendations report) will be uploaded to the database associated with APEC meetings. Likewise, it will be highly recommended to the participating economies and interest groups to disseminate these documents through their websites. Since this project is part of the Action Plan of Chile's Artificial Intelligence Policy, a press release will also be made to disseminate the results, together with the participating Ministries from Chile. Finally, the recommendations report will be sent to be submitted as an APEC publication following the Standard Guidelines on Drafting Project Proposals.

APEC economies will be invited to hold joint online events to present the results of the study, as well as to connect with others interested in the subject. This will reinforce further progress on bias issues, as well as engage other economies to discuss these issues. To do this, interested economies should manifest their interest and organize online events, however, the PO can assist such events with contacts and inputs. These events are not part of the outputs of the project.

Gender

Equal participation of men and women will be encouraged throughout each stage of the study. Likewise, considering gender equity is fundamental, and one of the biases that can be harmful when generating AI algorithms. It is for this reason that a gender axis will be generated in the previous documentary research, and it will also be a topic to be addressed in the online workshop, as well as in the final project policy paper. The project is committed to collecting sex-disaggregated data for all project participants and speakers.     

Targets

Female Participants (%)

50

Female Speakers/Experts (%)

50

Referring to the Guide on Gender Criteria for APEC Project Proposals in the Guidebook on APEC Projects, please tick the pillar or pillars that this project supports, in promoting women’s economic empowerment:

5 Pillars (you may tick more than one)

1

Access to Capital and Assets

2

Access to Markets

3

Skills, Capacity Building, and Health

X

4

Leadership, Voice and Agency

X

5

Innovation and Technology

X


Work Plan

Time

Task

Deliverables

Mar – Apr 2022

Project start.  The Project Overseer (PO) of the study starts the RFP to hire the Contractor of the project.  The PO starts a first draft document research and stock-take.

RFP

Signed Contract

Apr 2022

Submission APEC Project Monitoring Report.

APEC Project Monitoring Report

May 2022

The Contractor will continue the document research and stock-take, according to the work did by the PO

Jun – Jul 2022

The Contractor will coordinate the document research and will have a first draft to share with APEC Economies.

The Contractor and PO will start the preparation of the Online Workshop, sharing invitation to the event and asking for the collaboration between APEC Economies, NGOs or Academia Experts who wants to participate in the event. In  June, the Contractor will send the invitation of each participant to the online workshop.

The PO will send the General Information Circular. 

Final draft of the Research Report

List of attendees

Program of the event

General Information Circular

Aug 2022

The Contractor will create the different groups for the online workshop. In August, the 2-days online workshop will take place.

Systematization about the outputs in each of the roundtables

Following the APEC Project Event Recording Policy, the event will be recorded with the purpose of revisiting the discussion and making it available for people who could not attend or want to revisit it. Full and unedited recordings will not be disseminated.

Aug 2022

The Contractor and PO will prepare the draft Project Policy Paper, based on the document research and the outputs of each roundtable.

First draft of the Recommendations Report

Sep 2022

The Recommendations Report will be finalized for APEC revision.

Submission of the final Draft of the Recommendations Report

Nov 2022

The draft Recommendations Report will be reviewed by the Secretariat and shared to each APEC Economy and participant to send feedback. The Contractor and PO will make revisions to the draft based on the comments by the Secretariat and economies, if necessary.

Endorsement of the Recommendations Report

Feb 2023

Submission of the APEC Project Completion Report and all supporting documents to the Secretariat

APEC Project Completion Report

6-12 months after the Project Completion Date

Participation in the Longer-Term Evaluation of the APEC Projects

Risks

The greatest risk present in this project is related to the limited time to carry it out. Due to the COVID-19 pandemic, these types of events are held online, so issues associated with travel or agendas are becoming easier to organize. Also, the use of tools to share documents and work on them online. Regarding the issue to be addressed, it is necessary that the study be done quickly, since APEC Economies are using mostly AI tools every day, which can be biased by the data used, which can cause harm to those who are directed its potential benefits. In this context, generating a survey work and recommendations within a period of 6 months means that we have, in a short time, an updated pulse of the main problems associated with risky biases in AI, as well as how they can be overcome in the fourth industrial revolution that we are currently experiencing.

Monitoring and Evaluation

Evaluation Focus

Indicators

Target Goals

Evaluation Method

Reporting Method

Outputs

1.   Research Report

1.    Contracted executed

30 of April, 2022.

Certification by PO

Contract

2 Number of economies researched.

6

Certification by PO

Email to the secretariat of the Research Report

3.      Number of experts interviewed.

16

Certification by PO

Email to the secretariat of the Research Report

4.    Sectors analysed

5

Certification by PO

Email to the secretariat of the Research Report

5.    Number of Case Studies

3

Certification by PO

Email to the secretariat of the Research Report

6.    Research Report completed

30 of June, 2022.

Certification by PO

Email to the secretariat of the Research Report

7.    Number of pages

20-30

Certification by the PO

Email to the secretariat of the Research Report

2.   Workshop

1.    No. of participants (excl. speakers/ experts)

30

Event Attendance List

Completion Report

2.    % of participating men/women (excl. speakers/experts)

50/50

Event Attendance List

Completion Report

3.    No. of speakers/
experts engaged

5

Event Attendance List

Completion Report

4.    % of speakers/
experts (men/women)

50/50

Event Attendance List

Completion Report

5.    No. of attending economies

12

Event Attendance List

Completion Report

6.    Content developed and deployed

31 August 2022

Certification by PO

Completion Report

3.   Recommendations Report

1.    No. of pages

10-20

Certification by PO

Email to the Secretariat

2.    Submission to the Secretariat

30 Sept 2022

Submission to the Secretariat

Email to the Secretariat

Out-comes

1.   Knowledge about up-to-date standards of the measures being applied regarding harmful biases in AI systems.

1.    Perception of usefulness,

70%

Research report survey

Report to the Working Group

2.      Perception of Relevance.

70%

Research report survey

Report to the Working Group

3.      Perception of completeness.

70%

Research report survey

Report to the Working Group

2-     Knowledge gain and creation about best practices in AI regarding harmful biases.

1.   % of participants report substantial knowledge increase

75%

Post event survey.

Completion Report

2.    Reach of the discussion.

30 participants.

Event Attendance List.

Completion Report

3.    Recommendations on how to implement best practices in issues related to harmful biases in AI systems.

1.   No. of recommendations made

10

Included in the Recommendations Report.

Report to the Working Group

The Longer-term Evaluation of APEC Projects (LTEAP)


Linkages

Many international organizations have shown interest in working on this issue. The OECD has its "Going Digital"[1] platforms and the Artificial Intelligence Policy Observatory[2], in addition to establishing the first global principles for the development and use of AI technologies in 2019. The G-20 later incorporated these OECD AI principles into their framework. The G-20 Digital Economy Task Force (DETF) has worked together with the OECD in the development of the G20 Artificial Intelligence Principles[3]. Recently, in October 2020, the European Parliament adopted three resolutions on AI, covering ethics aspects, civil liability, and intellectual property, and in April 2021, EU released the EU AI Act too. The international interest in the development of this technology and the challenges it presents, makes it a priority for APEC –particularly for its DESG-DPS—to take a leading role in the discussion on AI.



[1] OECD (2016), Going Digital, Artificial Intelligence Project. http://www.oecd.org/going-digital/ai/

[2] OECD (2019), OECD AI Policy Observatory. https://www.oecd.ai/

[3] G20 (2019), G20 AI Principles. https://www.g20-insights.org/wp-content/uploads/2019/07/G20-Japan-AI-Principles.pdf 

Sustainability

The application and support for this project was included within the priority actions associated with the National Artificial Intelligence Policy of the Chilean Government, which was launched in October 2021. The objective of the project is to start and advance the discussion about harmful biases in AI and how this affects the development of algorithms. With the information gathered, it is expected to be able to work with the various APEC Economies to deepen on the various recommendations, and that these are put into practice. Likewise, it is expected to continue deepening bilateral relations on these topics, and to continue working on them according to what the different economies decide. Finally, it is hoped that the economies will work together to provide regulations that will benefit citizens in this area.

Even though it is not within the reach of this project, it will be recommended to Chilean APEC delegates to monitor the implementation of the recommendations developed in the Recommendations Report.

Project Overseers

This project will be managed by the Ministry of Science, Technology, Knowledge and Innovation, and the Ministry of Economy, Development and Tourism. In the case of the Ministry of Science, Carlos Ávila will manage the project. Carlos is the Coordinator of the National AI Policy. He is algo MSc. In Engineering and Industrial Engineer from the Pontificia Universidad Católica de Chile. In the case of the Ministry of Economy, Manuela Mujica will manage the project. Manuela is part of the Future and Social Adoption of Technology Unit.

Cost Efficiency

Not Applicable.

Drawdown Timetable

Not Applicable.

Direct Labour

No.

Specific Tasks

No. of Hours

1.             

Desktop Research: benchmark of APEC economies best practices.

75

2.             

Interviews to experts for best practices.

25

3.             

Desktop Research: AI applications and sectors analysis to prioritize.

50

4.             

Desktop Research: Identify case studies and best practices.

25

5.             

Desktop Research: Literature Review

75

6.             

Draft Research Report

150

7.             

Summary of Key findings in Workshop

25

8.             

Drafting of Recommendations Report

100

9.             

Design of the methodology for the discussion in the Workshop

25


No.

Specific Tasks

No. of Hours

1.             

Drafting the run sheets of the workshop for PO and Speakers

10

2.             

Preparing and testing virtual Platform for the event.

5

3.             

Trouble shooting and facilitating the flow of virtual events during the event – such as providing ‘housekeeping’ rules on muting, chat box and camera use; recording the event etc. This includes pre and after event margins.

10

4.             

Coordinate invitations to the event for experts and public.

60

5.             

Drafting the Agenda based on the Research Report

10

6.             

Drafting invitations, answering requirements, facilitating nomination and registration process for the workshop.

60

7.             

Moderation of the roundtables during the Workshop

40

8.             

Note taking during the discussion

10

9.             

Other administrative tasks

50




Waivers

Not Applicable.

Are there any supporting document attached?

No 
Attachments
Version: 6.0 
Created at 02/03/2022 22:06  by Lucy Phua 
Last modified at 10/06/2022 10:40  by Lucy Phua 
Version HistoryVersion History

Project No.

Project Title

Project Status

Publication (if any)

Fund Account

Sub-fund

Project Year

Project Session

APEC Funding

Co-funding Amount

Total Project Value

Sponsoring Forum

Topics

Committee

Other Fora Involved

Other Non-APEC Stakeholders Involved

Proposing Economy(ies)

Co-Sponsoring Economies

Expected Start Date

Expected Completion Date

Project Proponent Name 1

Job Title 1

Organization 1

Postal Address 1

Telephone 1

Fax 1

Email 1

Project Proponent Name 2

Job Title 2

Organization 2

Postal Address 2

Telephone 2

Fax 2

Email 2

Declaration

Project Summary

Relevance

Objectives

Alignment

TILF/ASF Justification

Beneficiaries and Outputs

Dissemination

Gender

Work Plan

Risks

Monitoring and Evaluation

Linkages

Sustainability

Project Overseers

Cost Efficiency

Drawdown Timetable

Direct Labour

Waivers

Are there any supporting document attached?

hdFldAdmin

Project Number

Previous Fora

Secretariat Comments

Reprogramming Notes

Consolidated QAF

Endorsement By Fora

PD Sign Off

Batch

Forum Priority

Committee Ranking Category

Committee Priority

PDM Priority

Priority Within Funding Category

Monitoring Report Received

Completion Report Received

PMU Field 1

PMU Field 2

PMU Field 3

On Behalf Of

Proposal Status

Originating Sub-Forum

Approval Status
Attachments
Content Type: Standard Proposal
Version:
Created at by
Last modified at by
Go Search