Challenges in Compliance Framework for TRUST Certification Landscape
The document discusses challenges in compliance frameworks for TRUST certification landscape, highlighting issues such as unstructured assessments, semantic ambiguity, and lack of clear definitions affecting TRUST, FAIR, and other principles. It emphasizes the need for structured evaluation mechanisms and tools to ensure compliance with open science expectations.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Compliance Framework Discussion Document Part I A Formalised LOD Graph of the TRUST Certification Landscape 4 November 2021 IG - RDA/WDS Certification of Digital Repositories:TRUST Principles and challenges on implementation Wim Hugo
A Diverse Landscape Community Expectations TRUST FAIR OAIS- RM GO-FAIR Many .. CTS ISO 16363 F-UJI, .. nestor (CARE )
Typical Challenges One can distill the following typical challenges and non-idealities respect of compliance: Unstructured assessment of important portfolios of principles, such as TRUST, FAIR, and other areas of compliance assessment and monitoring. Semantic ambiguity in respect of concepts (principles, criteria, benchmarks, metrics, indicators, best practices, maturity, recommendations, standards, ) and the relationship between these concepts. Not all criteria, benchmarks, and best practices are specified at the same level of detail or granularity, and moreover, metrics for levels of maturity associated with these criteria can apply at many levels of detail. No clear definition of the differences and overlaps between sets of principles (e.g. TRUST and FAIR) and semantic alignment between them. Multiple evaluation mechanisms and tools for the same criteria are emerging. The same criteria can be duplicated in more than one assessment approach, often at different levels of maturity. The same assessment approach can also identify different levels of maturity. Open Science expectations require the principles of TRUST, FAIR, and similar initiatives to apply across all research outputs, not only data.
Ordering the Landscape Community Expectations DATA Criteria TRUST FAIR OAIS- RM Best Practice/ Recommendations GO-FAIR Many .. CTS Metrics ISO 16363 F-UJI, .. nestor Data (CARE )
Ordering the Landscape Also: But also: Mandatory Mandatory (Policy, Legislation) Good Systems Practice Good Systems Practice Club Membership Motivation: Community Expectations Why would one consider compliance? Criteria: Who has made the effort to structure and unpack the scope and implications of compliance? Elaboration: What does compliance entail in practice? Implementation: How will I verify that I am compliant? Are there any tools to assist with measurement or standardisation? DATA Criteria TRUST FAIR OAIS- RM Best Practice/ Recommendations GO-FAIR Many .. CTS Metrics ISO 16363 F-UJI, .. Implementation: How? nestor Elaboration: What? Criteria: Who? Motivations: Why? (CARE )
Community Expectations DATA Criteria TRUST FAIR OAIS- RM Best Practice/ Recommendations GO-FAIR ... Many .. CTS Metrics ISO 16363 Protocols F-UJI, .. nestor Code Semantics Data (CARE )
Characteristics of a Solution # Design Element Description 1 Conceptual alignment Reduce and eliminate the large diversity of opinions and definitions about principles, their criteria and implementation practices, how they are measured 2 Simple, using existing standards Any solution should be based on existing web and data infrastructure standards and not require any new standards, but rather develop recommendations in respect of the semantics of compliance encoding, recording, and measurement based on existing standards. 3 Federated in practice, conceptually a single entity It is highly unlikely that all compliance information for a specific object will ever be recorded, preserved, and published by a single source: in practice, such information will be scattered in many locations and services, and the best possible solution will limit the complexity of the federated information space by standardising its encoding, implementation, and vocabulary. 4 Machine and human readable, machine actionable Solutions need to consider from the start that navigation and application of the compliance information for an object, service, subject (nodes), or collections of these will be complex, potentially involve many thousands or even millions of records, and may not result in a unanimous assessment of the compliance characteristics of the node in question. With this in mind, machine actionability - both in terms of aggregation and subsetting, as well as analysis and potential ML and AI applications, is a design imperative. 5 Precision and flexibility Map and define relations between sets of principles, criteria, metrics, and doing so flexibly - for example, allowing nested criteria with metrics and levels of maturity coupled to any level of detail, as required. Unambiguously define best practice, mandatory or optional recommendations, guidelines, and so on. 6 Reduced Complexity Improve understanding about principles, criteria, and metrics, etc. and on how these can be applied in practice.
Characteristics of a Solution # Design Element Description 7 Parsimony Minimise the set of applicable criteria and their formulations, standards, metrics, and maturity definitions, and/ or reduce duplication and complexity. 8 Relational nature of compliance assessments Compliance information should be seen as a property or properties of a relation between a digital object or service and a context - in this case, a compliance measurement event. Compliance information cannot and should not be seen as part of the metadata associated with a digital object in a one-to-one relation, since it can be evaluated by multiple tools, mechanisms and institutions, against several divergent or competing criteria, with varying levels of assertion, and so on. 9 Universally applicable in Open Science Avoid duplication of compliance principles, implementations, criteria, measures, and metrics across different research outputs. 10 Accommodate humans and subjective measurements Make the certification ecosystem machine-readable and actionable where feasible, recognising that some measurements rely heavily on human assessment - sometimes on site. One should accommodate the likely increased reliance on AI and ML to assist evaluation of complex or qualitative metrics. 11 Standardised measures and reporting Map institutional/ repository/ object compliance onto a formalised structure, and agree on mechanisms for evaluation and disambiguation of multiple, possibly divergent assessments of the same node or object.
Goals Conceptual Model: Provide unambiguous and clear definitions of the following: Motivations for compliance measurement: Principles, Regulations, Policies, Clubs, Patterns, ... Criteria, Authorities, and frameworks for that: OAIS-RM, ISO, CTS, nestor, Benchmarks and Detailed Criteria: elements of compliance expressed as best practices, recommendations, Standards and what they mean Metrics and indicators, and how they are assessed by humans and machines Aggregations of measurements and metrics: badges, endorsements, Define these and their relationships as a graph Encoding: Provide a specification and a POC Capturing nodes and relations in the graph; Capturing compliance assessments; Querying compliance status; Needs vocabularies Tests/ Use Cases Validation and Refinement via RDA WG
Implementations Conceptual Model Motivations Criteria Elaborations
Specifications The specification should be simple, and based on the following considerations: 1. Implementation as a graph, without specifying the implementation technology but the range of API services that must be available (syntax) and the schema and semantic standards that apply to them. For example, it must be possible to use any appropriate technology to store the graph (SKOS, Graph databases, relational databases, RDF triplestores, ) as long as the API schema and semantics are defined. 2. Providing a vocabulary or ontology that is used to unambiguously label nodes and relations in the graph. 3. Provide a practical mechanism for any digital object in the web to expose its compliance information to interested parties via Signposting.
CoreTrustSeal and TRUST Mapping CoreTrustSeal has developed a set of criteria reflecting community expectations through a process of engagement, and it is aligned to some extents with an existing framework (OAIS-RM). It expresses compliance by way of notional maturity (level of compliance) with benchmarks that are tested qualitatively. It provides extended guidance to repositories to assist with benchmarking prior to evaluation. TRUST is based on assessment of community expectations by experts that have categorised these into principles. The principles map in various degrees of exactness to the criteria identified by CoreTrustSeal.
Opportunities for R&D # Use Case Name and Description Execution or Implementation Status 1 FAIR and FAIR Recommendations, Best Practices, and Benchmarks from FAIR s FAIR Project DANS will be doing this work as part of the FAIR s FAIR project in the second half of 2021. The output will include prototypes of the compliance graph, candidate conceptual models and supporting vocabularies, and an API service. Funded and under way Deliverable: Mar 2022 2 CoreTrustSeal Application Management CoreTrustSeal will look at mechanisms to transform its certification applications to a graph-based compliance knowledge base, assisted by DANS. Funded and under way Deliverable: Mar 2022 3 PID Policy Compliance - EOSC Consortium of EU-based institutions are responding to the INFRA-EOSC Calls and will include PID-policy compliance use case(s) into the proposal Proposal submitted Commencement mid-2022 4 FAIR Implementation Profiles The ODISSEI project is considering community consensus work using FAIR Implementation Profiles as a guideline. This work can use compliance graphs as a mechanism for encoding in a more flexible alternative to the matrix-based schema initially defined for FIP. Proposal submitted
Thank You! 10 November 2021 BoF (OA Sponsored) - The future of "trustworthiness" and reliability of repositories and services: Trust, FAIR, CARE Contributions from Dawei Lin, Jon Petters, John Westbrook, Bob Downs, Barbara Sierman, and additional members of the TRUST Implementation WG Wim Hugo partly funded by the FAIR s FAIR project - European Union s Horizon 2020 project call - Grant agreement 831558
