Vantage6 Community Meeting Highlights & Updates

The Vantage6 community meeting on May 25, 2023, focused on collaboration opportunities, recent releases, upcoming features, and changes in the platform. Key topics included infrastructure and algorithm development, bug reports, and the Whitelisting of data sources feature request. The meeting also covered the release of version 3.8.2 and upcoming version 3.9.0, with improvements and bug fixes. Attendees were encouraged to align development efforts and avoid duplication of work. For more details, visit the release notes page.

• Vantage6
• Community Meeting
• Collaboration
• Algorithm Development
• Infrastructure
• Bug Reports

kendell Follow

Uploaded on Sep 21, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.

Presentation Transcript

1. Community meeting 25thof May 2023

2. The community Infrastructure Developers Committers & contributors Algorithm Developers Creating federated analysis algorithms Users Use vantage6 for privacy preserving analysis

3. Aim Collaboration opportunities Finding common interests by sharing what you are working on Align development efforts Avoid doing things twice Bug reports Please submit these in our issue tracker: https://github.com/vantage6/vantage6/issues

4. Agenda 11:00 Short introduction 10:05 Recently released 10:20 Infrastructure development 10:35 Algorithm development 10:50 Miscellaneous

5. Recently released 3.8.2 3.8.8 (7 releases)

6. Other changes 15 bugfixes 1 1 0 changes features security patches For more details on minor features and resolved bugs, see https://docs.vantage6.ai/en/main/release_notes.html

7. About to be released 3.9.0

8. Feature: Whitelisting of data-sources Currently external data sources can only be reached over SSH. We also want to access them via a Squid proxy server. Contributors: Frank Martin (IKNL), Djura Smits (e-science center), Reggie Cushing (e- science center), Bart van Beusekom (IKNL) [Feature request] Generalize proxy to allow connections to white-listed URLs Issue #162 vantage6/vantage6 (github.com)

9. Change: Node proxy logs to separate log file Node proxy server logs were included in node logs and on the console. Now the proxy logs are written to a separate file as they were confusing. Also, algorithm input that is printed is limited in length (in case it is very long). Contributors: Bart van Beusekom (IKNL), Frank Martin (IKNL) [Change] Log messages from master task that is waiting [Change] Limit number of characters printed to logs for task input

10. Feature: getting VPN addresses of children/tasks New utility function in AlgorithmClient to get IP address + port of all subtasks or the parent task. client.vpn.get_parent_address() client.vpn.get_child_addresses() Contributors: Bart van Beusekom, Frank Martin (IKNL) [Feature] get algorithm addresses children or parent

11. Feature: Enable GPU access by algorithm container Algorithm containers are configured to be able to access GPUs by setting node configuration Contributors: Alexander Harms (ErasmusMC) [Feature Request] Enable GPU access by algorithm container

12. Infrastructure development Upcoming features and changes

13. Work towards version 4.0 Several breaking changes need to be made now to improve further development. This includes improving the performance of the server (pagination, API output), improved algorithm dev and a new API as alternative to file mounts. Contributors: Bart van Beusekom, IKNL; Frank Martin, IKNL Estimate: Q3 2023 Project board 4.0.0

14. [Feature] Algorithm decorators (4.0) You can decorate algorithm functions to indicate they get an AlgorithmClient or one or more datasets. This removes the distinction between the master functions and RPC functions we had until now Contributors: Bart van Beusekom, IKNL; Frank Martin, IKNL Estimate: Q3 2023 [Change] Algorithm decorators

15. [Feature] API performance (4.0) Vantage6 API requests will be paginated by default. User Interface should be much faster as tables in there will be paginated as well. Additionally, what is returned is shorter. For instance, requesting an organization will no longer give a link for each task, but one link to get all tasks of that organization. Contributors: Bart van Beusekom, IKNL; Frank Martin, IKNL Estimate: Q3 2023 [Feature Request] Extend API to enable pagination in UI

16. [Change] Remove environments in config files (4.0) One could specify environments in node/server configuration files. In practice, these were very rarely used (it was easier to create a separate dev configuration). The environments will therefore be removed. Contributors: Bart van Beusekom, IKNL; Frank Martin, IKNL Estimate: Q3 2023 [Feature Request] Get rid of environments in config files

17. [Change] Rename result to run (4.0) client.result.get() When you run an algorithm container, the data was stored in result . This leads to confusing fields such as result.input, result.started_at and of course result.result. client.run.get() AND client.result.get() The name run makes more sense here Contributors: Bart van Beusekom, IKNL; Frank Martin, IKNL Estimate: Q3 2023 [Feature Request] Get rid of environments in config files

18. [Change] Split vantage6-client into two packages (4.0) The Python client package also contains the algorithm (wrapper) tools. These may be separated in a different package. This would also reduce dependencies required for the client. Contributors: Bart van Beusekom, IKNL; Frank Martin, IKNL Estimate: Q3 2023 [Change Request] Split vantage6-client into two packages

19. [Feature] OMOP support Connect algorithms to an OMOP data- source. The first step is to extract a default data set from OMOP, in a later stage the user should be able to specify the data to be extracted. Contributors: Frank Martin, IKNL; Estimate: Q3 2023

20. [Feature] Basic Algorithm preprocessing For example: Data transformation Feature Engineering Stratification Discretization Filtering Contributors: Frank Martin, IKNL; Estimate: Q3 2023

21. Feature: Researcher UI The User Interface is currently basically a management tool around the API. We would like to extend it to include a researcher perspective that facilitates sending tasks and showing graphs based on algorithm metadata. Contributors: Bart van Beusekom, IKNL; Frank Martin, IKNL Estimate: Q4 2023

22. Feature: documentation The documentation currently contains few details on how to use optional features. We are working on pages that describe what a feature is and how to use it. Contributors: Bart van Beusekom, IKNL; Frank Martin, IKNL Estimate: ongoing, complete Q3 2023

23. Feature: dummy- collaboration Adding developer feature, dummy- collaboration to setup a local network (single server instance accompanied with nodes) in order to test vantage6 infrastructure. Contributors: Hasan Alradhi1, Frank Martin1, Bart van Beusekom1 Estimate: ongoing (documentation) 1: IKNL

24. Feature: Algorithm Store It is difficult to get an overview of existing vantage6 algorithms, and not much more information is available except for the docker image. The algorithm store will be a new User Interface that lists available algorithms with metadata such as who developed/reviewed them, the underlying code, etc. Contributors: Bart van Beusekom, IKNL; Frank Martin, IKNL; Ananya Choudhury, Maastro Estimate: 2024

25. Algorithm development

26. Feature: documentation We are working on documentation for an IKNL package containing several algorithms. Contributors: Frank Martin, IKNL Estimate: ongoing IKNL algorithm repository

27. GLMM Paper status: Being reviewed by co-authors Update to algorithm: Adding multiple Random effects. Contributors: Hasan Alradhi1, Gijs Geleijnse1 and Hadrien Charvat2 Estimate: ongoing 1: IKNL, 2: Juntendo University vtg.glmm

28. Cox ZPH About: An extension of the CoxPH algorithm already implemented within vantage6 on horizontally partitioned data. This extension tests the Proportionality Hazards (PH) assumption after performing Cox regression. Also plots the Schoenfeld residuals. Algorithm status: Being restructured to fit within v6 framework. Contributors: Matteo Cellemare1, Hasan Alradhi2, Frank Martin2 Estimate: End of the month 1: Georgetown University, 2: IKNL

29. Miscellaneous

30. Recording this meeting What does the community think? A. Community meeting should not be recorded B. Recording should only be distributed within a small group C. Recording should be distributed publically

31. Miscellaneous Questions Finding common interests by sharing what you are working on Feedback Finding common interests by sharing what you are working on Ideas Finding common interests by sharing what you are working on