Vantage6 Community Meeting Highlights & Updates
The Vantage6 community meeting on May 25, 2023, focused on collaboration opportunities, recent releases, upcoming features, and changes in the platform. Key topics included infrastructure and algorithm development, bug reports, and the Whitelisting of data sources feature request. The meeting also covered the release of version 3.8.2 and upcoming version 3.9.0, with improvements and bug fixes. Attendees were encouraged to align development efforts and avoid duplication of work. For more details, visit the release notes page.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Community meeting 25thof May 2023
The community Infrastructure Developers Committers & contributors Algorithm Developers Creating federated analysis algorithms Users Use vantage6 for privacy preserving analysis
Aim Collaboration opportunities Finding common interests by sharing what you are working on Align development efforts Avoid doing things twice Bug reports Please submit these in our issue tracker: https://github.com/vantage6/vantage6/issues
Agenda 11:00 Short introduction 10:05 Recently released 10:20 Infrastructure development 10:35 Algorithm development 10:50 Miscellaneous
Recently released 3.8.2 3.8.8 (7 releases)
Other changes 15 bugfixes 1 1 0 changes features security patches For more details on minor features and resolved bugs, see https://docs.vantage6.ai/en/main/release_notes.html
About to be released 3.9.0
Feature: Whitelisting of data-sources Currently external data sources can only be reached over SSH. We also want to access them via a Squid proxy server. Contributors: Frank Martin (IKNL), Djura Smits (e-science center), Reggie Cushing (e- science center), Bart van Beusekom (IKNL) [Feature request] Generalize proxy to allow connections to white-listed URLs Issue #162 vantage6/vantage6 (github.com)
Change: Node proxy logs to separate log file Node proxy server logs were included in node logs and on the console. Now the proxy logs are written to a separate file as they were confusing. Also, algorithm input that is printed is limited in length (in case it is very long). Contributors: Bart van Beusekom (IKNL), Frank Martin (IKNL) [Change] Log messages from master task that is waiting [Change] Limit number of characters printed to logs for task input
Feature: getting VPN addresses of children/tasks New utility function in AlgorithmClient to get IP address + port of all subtasks or the parent task. client.vpn.get_parent_address() client.vpn.get_child_addresses() Contributors: Bart van Beusekom, Frank Martin (IKNL) [Feature] get algorithm addresses children or parent
Feature: Enable GPU access by algorithm container Algorithm containers are configured to be able to access GPUs by setting node configuration Contributors: Alexander Harms (ErasmusMC) [Feature Request] Enable GPU access by algorithm container
Infrastructure development Upcoming features and changes
Work towards version 4.0 Several breaking changes need to be made now to improve further development. This includes improving the performance of the server (pagination, API output), improved algorithm dev and a new API as alternative to file mounts. Contributors: Bart van Beusekom, IKNL; Frank Martin, IKNL Estimate: Q3 2023 Project board 4.0.0
[Feature] Algorithm decorators (4.0) You can decorate algorithm functions to indicate they get an AlgorithmClient or one or more datasets. This removes the distinction between the master functions and RPC functions we had until now Contributors: Bart van Beusekom, IKNL; Frank Martin, IKNL Estimate: Q3 2023 [Change] Algorithm decorators
[Feature] API performance (4.0) Vantage6 API requests will be paginated by default. User Interface should be much faster as tables in there will be paginated as well. Additionally, what is returned is shorter. For instance, requesting an organization will no longer give a link for each task, but one link to get all tasks of that organization. Contributors: Bart van Beusekom, IKNL; Frank Martin, IKNL Estimate: Q3 2023 [Feature Request] Extend API to enable pagination in UI
[Change] Remove environments in config files (4.0) One could specify environments in node/server configuration files. In practice, these were very rarely used (it was easier to create a separate dev configuration). The environments will therefore be removed. Contributors: Bart van Beusekom, IKNL; Frank Martin, IKNL Estimate: Q3 2023 [Feature Request] Get rid of environments in config files
[Change] Rename result to run (4.0) client.result.get() When you run an algorithm container, the data was stored in result . This leads to confusing fields such as result.input, result.started_at and of course result.result. client.run.get() AND client.result.get() The name run makes more sense here Contributors: Bart van Beusekom, IKNL; Frank Martin, IKNL Estimate: Q3 2023 [Feature Request] Get rid of environments in config files
[Change] Split vantage6-client into two packages (4.0) The Python client package also contains the algorithm (wrapper) tools. These may be separated in a different package. This would also reduce dependencies required for the client. Contributors: Bart van Beusekom, IKNL; Frank Martin, IKNL Estimate: Q3 2023 [Change Request] Split vantage6-client into two packages
[Feature] OMOP support Connect algorithms to an OMOP data- source. The first step is to extract a default data set from OMOP, in a later stage the user should be able to specify the data to be extracted. Contributors: Frank Martin, IKNL; Estimate: Q3 2023
[Feature] Basic Algorithm preprocessing For example: Data transformation Feature Engineering Stratification Discretization Filtering Contributors: Frank Martin, IKNL; Estimate: Q3 2023
Feature: Researcher UI The User Interface is currently basically a management tool around the API. We would like to extend it to include a researcher perspective that facilitates sending tasks and showing graphs based on algorithm metadata. Contributors: Bart van Beusekom, IKNL; Frank Martin, IKNL Estimate: Q4 2023
Feature: documentation The documentation currently contains few details on how to use optional features. We are working on pages that describe what a feature is and how to use it. Contributors: Bart van Beusekom, IKNL; Frank Martin, IKNL Estimate: ongoing, complete Q3 2023
Feature: dummy- collaboration Adding developer feature, dummy- collaboration to setup a local network (single server instance accompanied with nodes) in order to test vantage6 infrastructure. Contributors: Hasan Alradhi1, Frank Martin1, Bart van Beusekom1 Estimate: ongoing (documentation) 1: IKNL
Feature: Algorithm Store It is difficult to get an overview of existing vantage6 algorithms, and not much more information is available except for the docker image. The algorithm store will be a new User Interface that lists available algorithms with metadata such as who developed/reviewed them, the underlying code, etc. Contributors: Bart van Beusekom, IKNL; Frank Martin, IKNL; Ananya Choudhury, Maastro Estimate: 2024
Feature: documentation We are working on documentation for an IKNL package containing several algorithms. Contributors: Frank Martin, IKNL Estimate: ongoing IKNL algorithm repository
GLMM Paper status: Being reviewed by co-authors Update to algorithm: Adding multiple Random effects. Contributors: Hasan Alradhi1, Gijs Geleijnse1 and Hadrien Charvat2 Estimate: ongoing 1: IKNL, 2: Juntendo University vtg.glmm
Cox ZPH About: An extension of the CoxPH algorithm already implemented within vantage6 on horizontally partitioned data. This extension tests the Proportionality Hazards (PH) assumption after performing Cox regression. Also plots the Schoenfeld residuals. Algorithm status: Being restructured to fit within v6 framework. Contributors: Matteo Cellemare1, Hasan Alradhi2, Frank Martin2 Estimate: End of the month 1: Georgetown University, 2: IKNL
Recording this meeting What does the community think? A. Community meeting should not be recorded B. Recording should only be distributed within a small group C. Recording should be distributed publically
Miscellaneous Questions Finding common interests by sharing what you are working on Feedback Finding common interests by sharing what you are working on Ideas Finding common interests by sharing what you are working on
