Enterprise Data Architecture Using MS .NET Blazor and Azure
Explore the structure of record storage and data architecture in an enterprise environment, utilizing technologies such as MS .NET Blazor, Azure, and GreatIdeaz TrellisPark. Learn about concepts like Record definition, Data-Agnostic Storage, CQRS, CRUD functions, optimization of query operations, scalability, and sustainability impact.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Record Storage Enterprise Architecture Data Architecture, with examples using MS .NET Blazor, Azure and GreatIdeaz trellispark
Overview What is a Record? What is Data Agnostic Record Storage? What is Command and Query Responsibility Segregation (CQRS)? Common Create, Read, Update and Delete (CRUD) functions How can we further optimize query functions? How can we scale this? How does this impact sustainability?
What is a record? A Record is defined a specific Instance of a Concept (or Entity) A Concept defines a class of things we store information about Example: Vehicle is a Concept Concepts will have one or more data elements (attributes . Columns) Example: Vehicles have: Make, Model, Year, Seating Capacity, An Instance is a specific single example of a Concept Example: My vehicle has: Make=Toyota, Model=RAV4, In relational databases we store Records in Tables with data elements in Columns Every Concept gets its own Table and Stored Procedures Relational database design enforces relationships between concepts
What is Data Agnostic Record Storage? Hybrid approach combining document and relational database principles Records are stored as documents using a data agnostic XML or JSON format Allows for both traditional simple and complex database field types Reduces maintenance of tables, columns and relationships Only need one Instances table that includes metadata describing the Record Unique Record Identifier Type of data being stored Key relationships between Records (Parent-Child Hierarchy) The record data (XML/JSON)
What is Command and Query Responsibility Segregation (CQRS)? An architecture pattern that separates read and update operations for a data store Improves performance, scalability and security In Data Agnostic Services and trellispark Command is simply SaveInstance or DeleteInstance May trigger updates to read only lookup tables (minimum set for optimal performance) Query is optimized search over the indexed lookup tables Access to a specific record is simply ReadInstance The Command side gives the benefits of a document database The Query side fully utilizes the relational properties of the database
trellispark Code Generation for CQRS T-SQL Trigger on the Instances table for Insert, Update, Delete Based on ConceptGUID, may optionally call T-SQL stored procedures: Call T-SQL Delete/Save Stored Procedures to maintain first-level lookup table Also provides a great hook for: Calling custom T-SQL to execute when Instances are: Created, Updated or Deleted Adding outbound data flows for synchronization to other solutions (ESL) Fully optimized solutions may require multi-level read only lookup tables Used to optimize queries that require multiple complex joins Works best where there are lots of reads and relatively few updates
What about record relationships? Most common relationship is a simple Parent Child hierarchy ParentGUID contains the InstanceGUID of the parent ParentGUID of E is D Hierarchy contains the full path to the record Hierarchy of E is /A/C/D/ Can be represented in the Instance Metadata for fast search Relationships across the data model can be stored in the XML fragment The XML of D may contain an element that references B
What about record version history? All Instance records have an incrementing data version InsertedOn and InsertedBy fields All Instance record updates trigger MaintainLookups Copy all new record versions to InstanceHistory Can purge history records for selected concepts if not required
How can we secure individual records? Row-level Security is a Privacy pre-requisite Users people who need data Groups of users to simplify security Security is granted to a set of Users/Groups A user must be granted access to see the record data Access cascades down the hierarchy Access can be overridden at any point within the hierarchy Easy to add additional security features as required: Read audit if required for privacy Break the glass overrides
What happens if we delete a record? Can extend record storage to use a virtual deletion strategy Can validate delete requests down the entire descendent hierarchy Persists the Audit Trail even when the user tries to remove the records Can automatically purge deleted records based on Concept
How can we further optimize query functions? Easy to create and maintain Reporting databases Offload the query, reporting and analytics tasks to separate databases / servers Record Storage closely integrate with Data in Motion services for Outbound flows Using the Instances trigger we could also: Track Instance Changes into a separate table and pull/push to decoupled reporting Send copies of new Instance records to message queues or ESBs Run reports/analytics without impacting core performance Run complex queries and return a list of matching InstanceGUIDs
How can we scale? Easy to scalable horizontally Partition Instance Tables Across file groups/disks Across databases/servers Implementation is hidden Look up tables and indexes anywhere
How does this impact sustainability? Add or remove fields its just XML! Add or remove whole Concepts Restructure how Concepts inter-relate Reuse existing Concepts in multiple places Add new functionality to further extend CRUD+ Change underlying database storage technology if more effective and efficient solutions become available The efficiency of the underlying architecture as means you need less infrastructure to host your record storage
