Community Meeting Highlights and Latest Features Update
The community meeting on January 19, 2023, showcased collaborations among Infrastructure Developers, Committers, Contributors, and Algorithm Developers working on federated analysis algorithms using vantage6. The meeting agenda covered recent releases, new features like multi-factor authentication, expanded socket functionality, improved logging client, and task creation in the UI. The event emphasized sharing developments, aligning efforts, and enhancing security and user experience in the vantage6 platform.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Community meeting 19 January 2023
The community Infrastructure Developers Committers & contributors Algorithm Developers Creating federated analysis algorithms Users Use vantage6 for privacy preserving analysis
Aim Collaboration opportunities Finding common interests by sharing what you are working on Align development efforts Avoid doing things twice Bug reports Please submit these in our issue tracker: https://github.com/vantage6/vantage6/issues
Agenda 10:00 Short introduction 10:10 Recently released 10:20 Infrastructure development 10:55 Short break 11:00 Algorithm development 11:45 Miscellaneous
Recently released 3.5.0 3.6.1 (4 releases)
Feature: Multi-factor authentication Multi-factor authentication enhances security. We are implementing the option that server admin requires users to scan a QR code with an authenticator app and then authenticate via a temporary 6-digit code on their phone. Contributors: Bart van Beusekom (IKNL), Frank Martin (IKNL) [Feature Request] MFA Issue #355
Feature: Expand socket functionality Nodes are connected to servers via aSocketIO connection. Now, we intend to add users to these channels so they can get status updates (node online, task finished) and stop tasks remotely. Note: if you re using the dockerized UI, v3.6+ requires different environment variables. Contributors: Bart van Beusekom (IKNL), Frank Martin (IKNL) [Feature Request] Extend socket functionality [Feature Request] Kill algorithms remotely
Feature: Improved logging client The Python client only has `verbose=True` to enable logging. From v3.6 onwards, the logging level can be set to one of: debug, info, warning, error, or critical. Contributors: Bart van Beusekom, IKNL; Frank Martin, IKNL [Feature Request] Clearer error messages in the Python client
Feature: Create task in UI You can now create tasks in the UI. But only in non-encrypted collaborations (for now). Note that you can repeat tasks that you ve sent before more easily Contributors: Bart van Beusekom, IKNL; Frank Martin, IKNL [Feature Request] Algorithm metadata and UI generation
Feature: multi- database wrapper The wrappers written so far pass a pandas dataframe to the algorithm. This new wrapper instead passes a dictionary of all databases and their file paths or URIs to the algorithm. This allows an algorithm to use multiple databases Contributors: Bart van Beusekom, IKNL; Anja van Gestel, IKNL [Feature Request] Multi database wrapper
Feature: vnode set-api-key A new command `vnode set-api-key` will allow you to change the API key in your configuration file. Then, you do not need to open your configuration file to do this manually. Contributors: Bart van Beusekom, IKNL; Frank Martin, IKNL [Feature Request] vnode set-api-key to put API key in config file
Change: documentation Redesigning the documentation: as much as possible in one location and generated from the vantage6 source code. Also add technical documentation. Note: work on technical documentation is ongoing Contributors: Bart van Beusekom (IKNL), Frank Martin (IKNL) [Feature Request] Sphinx documentation generation
Other changes 10 bugfixes 6 14 features changes For more details on minor features and resolved bugs, see https://docs.vantage6.ai/en/main/release_notes.html
Infrastructure development Upcoming features and changes
Feature: Whitelisting of data-sources Currently it is not possible to attach algorithm containers to external data sources. Only data sources reachable in the internal Docker network are accessible by the algorithm container. Contributors: Frank Martin (IKNL), Djura Smits (e-science center), Reggie Cushing (e- science center), Bart van Beusekom (IKNL) Estimate: Q1-3 2023 [Feature request] Generalize proxy to allow connections to white-listed URLs Issue #162 vantage6/vantage6 (github.com)
Feature: Whitelisting of data-sources Create an SSH tunnel from the internal network to an external resource that contains the data. Contributors: Frank Martin (IKNL), Bart van Beusekom (IKNL) Estimate: Q1 2023 [Feature Request] Kubernetes deployment option Issue #248 vantage6/vantage6 (github.com)
Feature: Support Kubernetes API The infrastructure is now dependent on the Docker API (mainly the node). For security and scalability, it would be better to use Kubernetes to manage containers. This also opens up the possibility to use other container tools (e.g. Podman or Singulairity) Contributors: Frank Martin (IKNL), Bart van Beusekom (IKNL) Estimate: Q1 2023 [Feature Request] Kubernetes deployment option Issue #248 vantage6/vantage6 (github.com)
Algorithm client update The algorithm (container) client has a different interface from the user Python client. We aim to make them similar, so also `client.task.create()` from that client. Additionally, we want to add functions to facilitate VPN communication Contributors: Frank Martin (IKNL), Bart van Beusekom (IKNL) Estimate: Q1 2023 [Feature Request] Update the ContainerClient
UI as desktop application The UI is currently a web application. Because it is in the browser, access to the file system is difficult. Some actions, such as sending encrypted tasks, are difficult without the file system. We plan to wrap the Angular app, probably in Electronjs. Contributors: Bart van Beusekom, IKNL; Frank Martin, IKNL Estimate: Q1-Q2 2023 [Feature Request] UI as desktop application
Work towards version 4.0 Several breaking changes need to be made, such as a Python version upgrade, redesign of encryption module, improved API pagination, improved algorithm naming and several database changes. Contributors: Bart van Beusekom, IKNL; Frank Martin, IKNL Estimate: Q2 2023 Project board 4.0.0
Algorithm: IKNL package Algorithms that are used by IKNL bundled and version controlled. Summary GLM CoxPH Contributors: Frank Martin (IKNL) Estimate: Q2-4 2023 Summary Discussion #353 vantage6/vantage6 (github.com)
Algorithm: GLMM (horizontal) GLMM on horizontally partitioned hierarchical or longitudinal datasets using AGHQ for the numerical integration of the marginal likelihood. Writing manuscript detailing mathematics and dive into example of application; Poisson-GLMM for some data generated through the Poisson mixed effects process showing results of federated, centralized and purpose-built Poisson GLMM's. Contributors: Hasan Alradhi (IKNL), Hadrien Charvat (University of Tokyo) Status: Algorithm done (*), manuscript in progress Estimate: Q1 2023 (*) Need to Dockerize for full release https://github.com/iknl/vantage6- algorithms/tree/glmm
Algorithm: privacy preserving n-party scalar product protocol An algorithm to calculate the scalar product protocol among n-parties. This protocol can be used to count individuals fulfilling specific criteria. Which can be used for statistics such as informationgain or maximum likelyhood. This is a library and not limited to vantage6. For examples of use within vantag6 see health-ri demo, VertiBayes, Federated Bayesian Ensembles, or Verticox Can deal with a hybrid split. Contributors: Florian van Daalen (Maastricht University) Estimate: Finished, accepted for publication Library: https://github.com/MaastrichtU-CDS/n-scalar-product- protocol Vantage6 example implementation: https://github.com/CARRIER-project/health-ri-demo paper: https://arxiv.org/abs/2112.09436
Algorithm: VertiBayes An algorithm to train a Bayesian Network in a Vertically Split setting, as well as to validate the network using AUC in a privacy preserving manner. This algorithm can deal with a hybrid split as well Contributors: Florian van Daalen (Maastricht University) Estimate: Finished, working on publication Independent library: https://github.com/MaastrichtU-CDS/vertibayes Vantage6 wrapper code: https://github.com/MaastrichtU-CDS/vertibayes_vantage6 Paper: https://arxiv.org/abs/2210.17228
Algorithm: Federated Bayesian Ensembles An algorithm to train an ensemble of Bayesian Networks where the ensemble specificly takes advantage of the naturally occurring splits in the data. This algorithm can deal with a hybrid split. Contributors: Florian van Daalen (Maastricht University) Estimate: Currently validating implementation https://github.com/MaastrichtU- CDS/bayesianEnsemble
Algorithm: Verticox+ An improvement of the original verticox. The original algorithm is intended for learning cox proportional Hazard models in a vertically split setting Verticox+ will improve the privacy guarantees. Can function in a hybrid split. Contributors: Djura Smits (e-science center) Florian van Daalen (Maastricht University) Status: Algorithm done, currently figuring out encryption, starting validation V1.0 will be released soon! https://github.com/CARRIER-project/verticox Original paper: https://ieeexplore.ieee.org/abstract/document/90 76318
Miscellaneous Questions Finding common interests by sharing what you are working on Feedback Finding common interests by sharing what you are working on Ideas Finding common interests by sharing what you are working on
