Community Meeting Highlights and Latest Features Update

The community meeting on January 19, 2023, showcased collaborations among Infrastructure Developers, Committers, Contributors, and Algorithm Developers working on federated analysis algorithms using vantage6. The meeting agenda covered recent releases, new features like multi-factor authentication, expanded socket functionality, improved logging client, and task creation in the UI. The event emphasized sharing developments, aligning efforts, and enhancing security and user experience in the vantage6 platform.

• Community Meeting
• Collaborations
• Algorithm Development
• New Features
• Security

ken_qua Follow

Uploaded on Sep 28, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.

Presentation Transcript

1. Community meeting 19 January 2023

2. The community Infrastructure Developers Committers & contributors Algorithm Developers Creating federated analysis algorithms Users Use vantage6 for privacy preserving analysis

3. Aim Collaboration opportunities Finding common interests by sharing what you are working on Align development efforts Avoid doing things twice Bug reports Please submit these in our issue tracker: https://github.com/vantage6/vantage6/issues

4. Agenda 10:00 Short introduction 10:10 Recently released 10:20 Infrastructure development 10:55 Short break 11:00 Algorithm development 11:45 Miscellaneous

5. Recently released 3.5.0 3.6.1 (4 releases)

6. Feature: Multi-factor authentication Multi-factor authentication enhances security. We are implementing the option that server admin requires users to scan a QR code with an authenticator app and then authenticate via a temporary 6-digit code on their phone. Contributors: Bart van Beusekom (IKNL), Frank Martin (IKNL) [Feature Request] MFA Issue #355

7. Feature: Expand socket functionality Nodes are connected to servers via aSocketIO connection. Now, we intend to add users to these channels so they can get status updates (node online, task finished) and stop tasks remotely. Note: if you re using the dockerized UI, v3.6+ requires different environment variables. Contributors: Bart van Beusekom (IKNL), Frank Martin (IKNL) [Feature Request] Extend socket functionality [Feature Request] Kill algorithms remotely

8. Feature: Improved logging client The Python client only has `verbose=True` to enable logging. From v3.6 onwards, the logging level can be set to one of: debug, info, warning, error, or critical. Contributors: Bart van Beusekom, IKNL; Frank Martin, IKNL [Feature Request] Clearer error messages in the Python client

9. Feature: Create task in UI You can now create tasks in the UI. But only in non-encrypted collaborations (for now). Note that you can repeat tasks that you ve sent before more easily Contributors: Bart van Beusekom, IKNL; Frank Martin, IKNL [Feature Request] Algorithm metadata and UI generation

10. Feature: multi- database wrapper The wrappers written so far pass a pandas dataframe to the algorithm. This new wrapper instead passes a dictionary of all databases and their file paths or URIs to the algorithm. This allows an algorithm to use multiple databases Contributors: Bart van Beusekom, IKNL; Anja van Gestel, IKNL [Feature Request] Multi database wrapper

11. Feature: vnode set-api-key A new command `vnode set-api-key` will allow you to change the API key in your configuration file. Then, you do not need to open your configuration file to do this manually. Contributors: Bart van Beusekom, IKNL; Frank Martin, IKNL [Feature Request] vnode set-api-key to put API key in config file

12. Change: documentation Redesigning the documentation: as much as possible in one location and generated from the vantage6 source code. Also add technical documentation. Note: work on technical documentation is ongoing Contributors: Bart van Beusekom (IKNL), Frank Martin (IKNL) [Feature Request] Sphinx documentation generation

13. Other changes 10 bugfixes 6 14 features changes For more details on minor features and resolved bugs, see https://docs.vantage6.ai/en/main/release_notes.html

14. Infrastructure development Upcoming features and changes

15. Feature: Whitelisting of data-sources Currently it is not possible to attach algorithm containers to external data sources. Only data sources reachable in the internal Docker network are accessible by the algorithm container. Contributors: Frank Martin (IKNL), Djura Smits (e-science center), Reggie Cushing (e- science center), Bart van Beusekom (IKNL) Estimate: Q1-3 2023 [Feature request] Generalize proxy to allow connections to white-listed URLs Issue #162 vantage6/vantage6 (github.com)

16. Feature: Whitelisting of data-sources Create an SSH tunnel from the internal network to an external resource that contains the data. Contributors: Frank Martin (IKNL), Bart van Beusekom (IKNL) Estimate: Q1 2023 [Feature Request] Kubernetes deployment option Issue #248 vantage6/vantage6 (github.com)

17. Feature: Support Kubernetes API The infrastructure is now dependent on the Docker API (mainly the node). For security and scalability, it would be better to use Kubernetes to manage containers. This also opens up the possibility to use other container tools (e.g. Podman or Singulairity) Contributors: Frank Martin (IKNL), Bart van Beusekom (IKNL) Estimate: Q1 2023 [Feature Request] Kubernetes deployment option Issue #248 vantage6/vantage6 (github.com)

18. Algorithm client update The algorithm (container) client has a different interface from the user Python client. We aim to make them similar, so also `client.task.create()` from that client. Additionally, we want to add functions to facilitate VPN communication Contributors: Frank Martin (IKNL), Bart van Beusekom (IKNL) Estimate: Q1 2023 [Feature Request] Update the ContainerClient

19. UI as desktop application The UI is currently a web application. Because it is in the browser, access to the file system is difficult. Some actions, such as sending encrypted tasks, are difficult without the file system. We plan to wrap the Angular app, probably in Electronjs. Contributors: Bart van Beusekom, IKNL; Frank Martin, IKNL Estimate: Q1-Q2 2023 [Feature Request] UI as desktop application

20. Work towards version 4.0 Several breaking changes need to be made, such as a Python version upgrade, redesign of encryption module, improved API pagination, improved algorithm naming and several database changes. Contributors: Bart van Beusekom, IKNL; Frank Martin, IKNL Estimate: Q2 2023 Project board 4.0.0

21. Algorithm development

22. Algorithm: IKNL package Algorithms that are used by IKNL bundled and version controlled. Summary GLM CoxPH Contributors: Frank Martin (IKNL) Estimate: Q2-4 2023 Summary Discussion #353 vantage6/vantage6 (github.com)

23. Algorithm: GLMM (horizontal) GLMM on horizontally partitioned hierarchical or longitudinal datasets using AGHQ for the numerical integration of the marginal likelihood. Writing manuscript detailing mathematics and dive into example of application; Poisson-GLMM for some data generated through the Poisson mixed effects process showing results of federated, centralized and purpose-built Poisson GLMM's. Contributors: Hasan Alradhi (IKNL), Hadrien Charvat (University of Tokyo) Status: Algorithm done (*), manuscript in progress Estimate: Q1 2023 (*) Need to Dockerize for full release https://github.com/iknl/vantage6- algorithms/tree/glmm

24. Algorithm: privacy preserving n-party scalar product protocol An algorithm to calculate the scalar product protocol among n-parties. This protocol can be used to count individuals fulfilling specific criteria. Which can be used for statistics such as informationgain or maximum likelyhood. This is a library and not limited to vantage6. For examples of use within vantag6 see health-ri demo, VertiBayes, Federated Bayesian Ensembles, or Verticox Can deal with a hybrid split. Contributors: Florian van Daalen (Maastricht University) Estimate: Finished, accepted for publication Library: https://github.com/MaastrichtU-CDS/n-scalar-product- protocol Vantage6 example implementation: https://github.com/CARRIER-project/health-ri-demo paper: https://arxiv.org/abs/2112.09436

25. Algorithm: VertiBayes An algorithm to train a Bayesian Network in a Vertically Split setting, as well as to validate the network using AUC in a privacy preserving manner. This algorithm can deal with a hybrid split as well Contributors: Florian van Daalen (Maastricht University) Estimate: Finished, working on publication Independent library: https://github.com/MaastrichtU-CDS/vertibayes Vantage6 wrapper code: https://github.com/MaastrichtU-CDS/vertibayes_vantage6 Paper: https://arxiv.org/abs/2210.17228

26. Algorithm: Federated Bayesian Ensembles An algorithm to train an ensemble of Bayesian Networks where the ensemble specificly takes advantage of the naturally occurring splits in the data. This algorithm can deal with a hybrid split. Contributors: Florian van Daalen (Maastricht University) Estimate: Currently validating implementation https://github.com/MaastrichtU- CDS/bayesianEnsemble

27. Algorithm: Verticox+ An improvement of the original verticox. The original algorithm is intended for learning cox proportional Hazard models in a vertically split setting Verticox+ will improve the privacy guarantees. Can function in a hybrid split. Contributors: Djura Smits (e-science center) Florian van Daalen (Maastricht University) Status: Algorithm done, currently figuring out encryption, starting validation V1.0 will be released soon! https://github.com/CARRIER-project/verticox Original paper: https://ieeexplore.ieee.org/abstract/document/90 76318

28. Miscellaneous Questions Finding common interests by sharing what you are working on Feedback Finding common interests by sharing what you are working on Ideas Finding common interests by sharing what you are working on