Overview of PaNOSC Workshop API Endpoints & Data Structures
This project discusses the API endpoints and data structures from the PaNOSC Workshop, focusing on reviewing use cases, required search results/data, actors involved, and the design of endpoints reflecting proposal/experiment processes. It covers detailed discussions on data models, instrument setups, schedules, metadata, and access to different data types within the scientific investigation context. The API endpoint propositions adhere to a strict REST API interpretation for efficient data retrieval related to proposals, schedules, datasets, and data files, ensuring proper validation by the server for secure access.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
PaNOSC WP3 ILL Workshop API Endpoints & Returned Data Structure 18thSeptember, 2019 This project has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No. 823852
Summary Review of API Use Cases Review of required search results/data Data model discussions API Endpoint propositions Returned data structure 2
Review of Use Cases Actors Open data user: FAIR exposure of metadata from each site WP4 Data Analysis Portal: access primarily to restricted data Search requirements General search by metadata type/keyword Search for data for specific proposal or experiment (my data) Metadata from data files and proposals Experimental date ranges Domain specific extensions Chemical Abstracts Service (CAS) for molecules Synonyms (eg Small Angle Diffraction => SANS & SAXS) Ontologies (eg Q-ranges such as small angle, very small angle, ) 3
Review of required results/data All available metadata Instrument setup and experimental state Paths to acquisition data Proposal details Team Schedules Instruments Title and abstract Associated data Log books DOIs Statistics (e.g. who has accessed my data) 4
Data model discussions ILL & ESRF discussions based on their processes and data catalogues Design endpoints that reflect proposal/experiment process Ease mapping to data catalogues Proposal Details of scientific investigation Scientific planning: dates and instruments n Schedules for a Proposal Schedule Data collected under specific conditions (e.g. sample) n Datasets for a Schedule Dataset Data acquisition, e.g. raw data n Datafiles in a Dataset Datafile Is this description/process suitable for all sites? 5
API Endpoint propositions Strict REST API interpretation for data model (i.e. nested approach) /proposals Returns all proposals /proposals/{proposalId} Returns specific proposal including associated schedules /proposals/{proposalId}/schedules Return schedule data for a specific proposal /proposals/{proposalId}/schedules/{scheduleId}/datasets Returns datasets of a specific schedule /proposals/{proposalId}/schedules/{scheduleId}/datasets/{datasetId}/files Returns data files of a dataset /proposals/{proposalId}/schedules/{scheduleId}/datasets/{datasetId}/status Returns status (ONLINE, RESTORING, ARCHIVED) of a dataset Implies obtaining datasets/files requires knowledge of proposal and schedule IDs Also requires validation by the server that scheduleId corresponds to given proposalId Complicates general search for data 6
API Endpoint propositions Flattened REST API Remove necessity to provide proposalId and scheduleId to obtain datasets Uniform endpoint structure Simplify general search for data Remove /dataset/{id}/status endpoint: Status added to /dataset/{id} returned data 7
API Endpoint propositions Flattened REST API /proposals Returns all proposals (paginated) /proposals/{proposalId} Returns a specific proposal including associated schedules /schedules Returns all schedules (paginated) /schedules/{scheduleId} Returns a specific schedule /schedules/{scheduleId}/datasets Returns paginated datasets of a specific schedule (including status information) /datasets Returns all datasets (paginated), allows for general search of data /datasets/{datasetId} Returns a specific dataset /datasets/{datasetId}/files Returns paginated data files of a dataset 8
API Endpoint propositions Additional endpoints /instruments Returns information about instruments /info Returns information on the API implementation at the facility and includes a list of searchable keywords Used for portal integration Propose list of instruments to a user Determine capabilities of a site s implementation of the API 9
API Endpoint: Open points Differentiating between open data and personal data e.g. authenticated user request to /proposals Returns all of my proposals and public proposals? Returns only my proposals? Do we differentiate via a query parameter? Do we differentiate via a different endpoint? Any other endpoints to add? Elements useful to a client application (like the /instruments endpoint) 10
PaNOSC WP3 ILL Workshop Returned Data Structure 18thSeptember, 2019 This project has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No. 823852 11
Summary Data model details API Conventions Meta and Pagination JSON examples Open points 12
Returned data structure Current status Proposal Id Name Title Summary DOI Start date End date Release date Schedule Id DOI Start date End date Dataset Id Name Owner details License Embargo end date Size Path Technique Sample name Chemical formula Wavelength File Id Name Path File size Instrument Id Name Description Member Role Parameter Name Value Person Laboratory Id Name Address City Country Id Name Surname Orcid ID Researcher ID Publication Name 13
Returned data structure API Conventions: meta and pagination Non-paginated: { "meta": { "institute": "string", }, "data": [ ] } Paginated: { "meta": { "institute": "string", "page": 0, "pageSize": 0, "totalPages": 0 }, "data": [ ], "links": { "self": "string", "prev": "string", "next": "string", "first": "string", "last": "string" } } 14
Returned data structure: Proposal { "id": "string", "title": "string", "summary": "string", "doi": "string", "startDate": "string", "endDate": "string", "releaseDate": "string", "name": "string", "parameters": [PARAMETER OBJECT (name, value)], "members": [{ "person": PERSON OBJECT, "affiliation": LABORATORY OBJECT, "role": "string" }], "schedules": [SCHEDULE OBJECT] } 15
Returned data structure: Schedule { "id": "string", "proposal": { "name": "string", "id": 0, "link": "string" }, "startDate": "string", "endDate": "string", "doi": "string", "instrument": INSTRUMENT OBJECT (id, name, description) } 16
Returned data structure: Dataset { "id": "string", "name": "string", "owner": "string", "contactEmail": "string", "orcidOfOwner": "string", "license": "string", "embargoEndDate": "string", "startDate": "string", "endDate": "string", "schedule":SCHEDULE OBJECT, "parameters": [PARAMETER OBJECT (name, value)] "size": MEASUREMENT OBJECT (value, units) "path": "string", "technique": "string", "sampleName": "string", "chemicalFormula": "string", "wavelength": MEASUREMENT OBJECT (value, units) } Question: can we add status here too (ONLINE, RESTORING, ARCHIVED)? 17
Open points Should child elements return details of parents? e.g. Schedule data includes Proposal data { "id": "string", "proposal": { "name": "string", "id": 0, "link": "string" }, "startDate": "string", "endDate": "string", "doi": "string", "instrument": { "name": "string", "description": "string" } } Keep only minimal information (i.e. id, link)? Can specific Dataset metadata be included in parameters e.g. Wavelength, Technique All parameters are optional? Dataset owner details (name, orcid, email) Do datasets have owners can we replace these with a Person element? 18
Open points Optional and mandatory data What is mandatory? Return null if not available/implemented Well defined parameter names? sampleTemperature sample_temperature sample.temperature Add units to parameters Currently have Measurement object: can we integrate this into parameter data? Use meta attribute in parameters for additional information? e.g. since units may be optional, add this to the meta object e.g. type of value (float, integer, string) 19
