Future FG-AI4H Processes Discussion and Updates
FGAI4H-E-030-A1
Geneva, 30 May – 1 June 2019
Onboarding
1.
Creating document helping with ITU online registration
•
FG to create an onboarding document for external experts that includes all
the steps needed to register and participate
•
In a prominent location on the website
2.
Create documents for guideline to participate in topic groups
•
Document for data donation
•
Document for “AI for health” description / update the call for use cases
New work areas
3.
Soliciting new health topics (that meet FG requirements)
•
Sameer Pujari (WHO) will be drafting a document describing the
scope in health of the Focus Group.
Consider:
a)
Malaria
b)
HIV/AIDS
c)
Pharmaceuticals
4. Creating (ToR for) new WG on Ethics
•
WG to create a document on generic ethical considerations and
requirements for handling data with in the Focus Group work
•
WG to provide an ethical assessment and suggestions on every “AI for
health” topic
•
Data sourcing and labelling
•
AI-based prediction quality / uncertainty / explainability
•
Deployment
•
WG would deliver statements to be included in TDDs
•
WG to hold meetings (in person and online) and workshops
•
The experts that convene the WG to reach out to the community to share
best practice
•
To consider the data handling policy
5. Specifying expectations for
(training/testing) data quality (1/3)
•
Consider:
•
Bias, incompatibility, ethics, legalities, etc.
•
Reproducibility of data generation
•
Statistical characteristics of data that are relevant (summary statistics)
•
Data homogeneity issues and data completeness
•
Data annotation
5. Specifying expectations for
(training/testing) data quality (2/3)
•
To establish a WG that comes up with documentation and software tools that can be used to
assess the quality of data
•
Chair: Pat Baird (Philips), Vice-Chair Luis Oala (Fraunhofer HHI)
•
ToRs WG on Data AI solutions quality [to be refined by chair]
•
Determine metrics / characteristics for assessing data quality
•
Documentation and software tools that can be used to assess the quality of data
•
Study available public datasets
•
Liaise with WG-RC
•
Provide recommendations on data collection for the health topics
•
To consider the data handling policy
•
Adaptation of AI solutions
•
Selection of performance measurements/metrics
•
Methods for quantifying robustness and uncertainty
•
Creating ToRs on testing robustness
•
Methods for explainability and generalizability
•
Value of workshops
5. Specifying expectations for
(training/testing) data quality (3/3)
•
ToRs WG on Data handling
[to be refined by chair]
•
To draft ToR based on the below -> via AAP (applies to all ToRs)
•
Data transfer
•
Data encryption
•
Data processing
•
Benchmarking
•
Submitted sollutions handling
•
Data usage / right to use / manipulate the data
•
Data spliting
•
Data aggregation
•
Data provinance
•
Normalisation
•
Data lifecycle
•
Data ownership
•
Data storage
•
Review B-105
•
Chair: Marc Lecoultre, Vice-Chair: Ferah Kherif (CHUV)
6.
Nominating experts (rapporteurs,
reviewers)
Consider:
a)
Identifying experts in the fields of health, AI, and statistics, social sciences, ethics / governance
b)
Recruiting experts (advertising, certificates of appreciation)
•
Advertising in journals, conferences, social media, etc.
•
Deploy our network of partners, and extend to a worldwide representation
•
Seek equal representation of gender, race, geography, seniority
•
Develop a description of what is expected from the experts
•
Application form and portal
c)
Vetting experts
•
Conflict of Interest statements
•
Resume / CV
•
References
•
Setup a Review Board, that will be doing the vetting
•
Process for assignment to TGs
7. Facilitating communication
Consider:
•
Access to equipment/technology for online cooperation/virtual meetings
•
Defining need for documentation
•
Modernizing FG (and TG) website(s)
•
Circulating relevant documents over FG mailing list
8. Conflicting Terminology document
•
Initial thoughts: E-029
•
An initial collection will be provided by Pat Baird (Philips) for
meeting F
9. Specifying expectations for AI solutions
•
Consider:
a)
adaptation of AI solutions
b)
selection of performance measurements/metrics
c)
methods for quantifying robustness and uncertainty
d)
creating (ToR for new) ad-hoc group on testing robustness
e)
methods for explainability and generalizability
f)
value of workshops
10. Addressing issues with benchmarking
software (AIcrowd)
•
accessing back-end of program
•
finding work-around for high-dimensional data
The discussions and updates from the FG-AI4H meeting in Geneva on May 30 to June 1, 2019. It covers topics such as creating onboarding documents for ITU online registration, soliciting new health topics for the Focus Group, creating a new Working Group on Ethics, specifying expectations for data quality in training and testing, and more.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
FGAI4H-E-030-A1 Geneva, 30 May 1 June 2019 Source: Chairman FG-AI4H Title: Future FG processes discussion Purpose: Discussion Thomas Wiegand Fraunhofer HHI, Germany Monique Kuglitsch Fraunhofer HHI, Germany E-mail: thomas.wiegand@hhi.fraunhofer.de Contacts: E-mail: monique.kuglitsch@hhi.fraunhofer.de Abstract: This PPT contains the information in E-030 plus the updates discussed during the FG-AI4H meeting. NOTE Further updates are expected in the ToR of the various WGs created during the meeting.
Onboarding 1. Creating document helping with ITU online registration FG to create an onboarding document for external experts that includes all the steps needed to register and participate In a prominent location on the website Create documents for guideline to participate in topic groups Document for data donation Document for AI for health description / update the call for use cases 2.
New work areas 3. Sameer Pujari (WHO) will be drafting a document describing the scope in health of the Focus Group. Soliciting new health topics (that meet FG requirements) Consider: a) b) HIV/AIDS c) Pharmaceuticals Malaria
4. Creating (ToR for) new WG on Ethics WG to create a document on generic ethical considerations and requirements for handling data with in the Focus Group work WG to provide an ethical assessment and suggestions on every AI for health topic Data sourcing and labelling AI-based prediction quality / uncertainty / explainability Deployment WG would deliver statements to be included in TDDs WG to hold meetings (in person and online) and workshops The experts that convene the WG to reach out to the community to share best practice To consider the data handling policy
5. Specifying expectations for (training/testing) data quality (1/3) Consider: Bias, incompatibility, ethics, legalities, etc. Reproducibility of data generation Statistical characteristics of data that are relevant (summary statistics) Data homogeneity issues and data completeness Data annotation
5. Specifying expectations for (training/testing) data quality (2/3) To establish a WG that comes up with documentation and software tools that can be used to assess the quality of data Chair: Pat Baird (Philips), Vice-Chair Luis Oala (Fraunhofer HHI) ToRs WG on Data AI solutions quality [to be refined by chair] Determine metrics / characteristics for assessing data quality Documentation and software tools that can be used to assess the quality of data Study available public datasets Liaise with WG-RC Provide recommendations on data collection for the health topics To consider the data handling policy Adaptation of AI solutions Selection of performance measurements/metrics Methods for quantifying robustness and uncertainty Creating ToRs on testing robustness Methods for explainability and generalizability Value of workshops
5. Specifying expectations for (training/testing) data quality (3/3) ToRs WG on Data handling [to be refined by chair] To draft ToR based on the below -> via AAP (applies to all ToRs) Data transfer Data encryption Data processing Benchmarking Submitted sollutions handling Data usage / right to use / manipulate the data Data spliting Data aggregation Data provinance Normalisation Data lifecycle Data ownership Data storage Review B-105 Chair: Marc Lecoultre, Vice-Chair: Ferah Kherif (CHUV)
6. Nominating experts (rapporteurs, reviewers) Consider: a) b) Identifying experts in the fields of health, AI, and statistics, social sciences, ethics / governance Recruiting experts (advertising, certificates of appreciation) Advertising in journals, conferences, social media, etc. Deploy our network of partners, and extend to a worldwide representation Seek equal representation of gender, race, geography, seniority Develop a description of what is expected from the experts Application form and portal Vetting experts Conflict of Interest statements Resume / CV References Setup a Review Board, that will be doing the vetting Process for assignment to TGs c)
7. Facilitating communication Consider: Access to equipment/technology for online cooperation/virtual meetings Defining need for documentation Modernizing FG (and TG) website(s) Circulating relevant documents over FG mailing list
8. Conflicting Terminology document Initial thoughts: E-029 An initial collection will be provided by Pat Baird (Philips) for meeting F
9. Specifying expectations for AI solutions Consider: a) adaptation of AI solutions b) selection of performance measurements/metrics c) methods for quantifying robustness and uncertainty d) creating (ToR for new) ad-hoc group on testing robustness e) methods for explainability and generalizability f) value of workshops
10. Addressing issues with benchmarking software (AIcrowd) accessing back-end of program finding work-around for high-dimensional data
