Evolution of DevOps in Testing Journey at American Family Insurance
American Family Insurance embarked on a transformation journey towards a DevOps culture, revolutionizing their release and testing processes. From a traditional setup lacking coordination to a structured approach emphasizing automation, performance, and collaboration, the journey showcased significant improvements in deployment stability, testing efficiency, and team communication.
Uploaded on Sep 21, 2024 | 0 Views
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
DEVOPS; A TESTER S BEST FRIEND
DEVOPS; A TESTER S BEST FRIEND TOM JANIK AMERICAN FAMILY INSURANCE DEVOPS MANAGER RELEASE MANAGEMENT KEN ROBERTSON AMERICAN FAMILY INSURANCE SQA TEST MANAGER SPECIALIZED TESTING OBJECTIVE OF SESSION: WALK THROUGH THE AMERICAN FAMILY JOURNEY TO BECOMING A COMPANY THAT OPERATES IN A CULTURE BASED IN DEVOPS
IN THE BEGINNING: RELEASE FLOW DEVELOPMENT LEG DEV INT SIT LEG DEV INT QA RELEASE LEG DEV INT QA PERF PROD Deploys done by development teams, could happen at any time No stability requirements, no triage of deployment / environment issues 4 year development cycle using Amgile SDLC (selective Agile) Development Leg (flow), SIT Leg , 2 month effort in Release Leg
IN THE BEGINNING: SIT TESTING LOCAL/DEV Development and Unit Testing DEV/INT Functional Testing SIT QA Test Data Creation TDC TDC Elevation n Elevation 1 Elevation 2 Manual testing Testing led by development teams, lacked integration coordination 2 year SIT cycle with multiple code elevations No formalized automation or performance testing No shared test data, test scenarios, or documented financial testing Limited communication between development and testing throw it over the wall mentality
FORMING: THE DEVOPS REVOLUTION DEVELOPMENT LEG 1 INT DEV QA DEVELOPMENT LEG 2 INT DEV QA DEVELOPMENT LEG 3 INT DEV QA RELEASE LEG DEV INT QA PERF PROD D: Release Coordination, Automation, Performance, and Manual Testing Teams O: Change Management, Build and Deploy, Software Configuration Management, Infrastructure Coordination Teams Specific deployment windows, DENT environments, Floating QA environment Branching and merging strategies begin, targeted deploys to PERF (testing left) 3 parallel development ( Legs ) streams, Monthly maintenance release Leg Press ESC to continue
FORMING: SIT TESTING CYCLES LOCAL/DEV Development and Unit Testing Functional Testing DEV/INT/DENT AIT (Application Integration Testing) INT Phase 1 Phase 2 Phase 3 QA HIST Test Data Creation SIT (System Integration Testing) QA Cycle 1 Cycle 2 Cycle n... Introduction of DENT and QA HIST tiers Point to point integration testing coordinated by development teams Application integration coordination being introduced 6 month SIT duration 3 months financial infrastructure testing and 3 months for financials Documented financial testing in collaboration with controllers
ACTIVITY: DEVOPS WHAT S IMPORTANT TO YOU? Using the index card in front of you write down 3 things that are most important to you about DevOps. (1:00 Minute) Gather in groups of #-# at one of the posters on the wall. (1:00 Minute) Come up with something common between everyone in the group and write that on top of the poster as your group name. (1:00 Minute) Come up with one common thing from everyone s index cards and draw (no words) that one common thing on the rest on the space on the poster. (1:00 Minute)
STORMING: A CULTURE CHANGE TO DEVOPS DevOps is the practice of operations and development engineers participating together in the entire service lifecycle: from design, through the development process, and then into production support Specifically DevOps is characterized by: Operations staff making use many of the same techniques as developers for their systems work Developers incorporating more operational concerns in their planning and coding (designing for operationalization) DevOps can be a team, a department, an effort, a movement, but most importantly it is a culture!
KEY TENETS OF DEVOPS Operate Plan FEEDBACK Theory of Constraints ( The Goal ) Release Architect The 3 Ways ( The Phoenix Project ): Systems thinking Test Dev Amplified feedback loops Continuous experimentation and learning Lean principles of manufacturing ( TPS: Toyota Production System ) People, then process, then tools (Men, Methods, Machines, Measurements) Collaborative development (Scrum, Agile, Kanban) Environment provisioning, quicker access to new code for testing Continuous deployment, integration, testing, delivery, feedback, and improvement
THE NEW DEVOPS CULTURE Automation Testing Specialty Testing Quality Assurance Release Management Release Automation Data Team Services Team DevOps Environment Management Application Reliability
Planning (Enterprise) Value Stream Agile Project Tracking Small Batches Increased Frequency Speed to Defect Resolution Minimal Viable Product Build Quality Into the Process Mapping Architecture, Design & Development Developer Commit Application Instrumentation Loosely Coupled Architectures Micro services Branching by Abstraction /Toggling Design for Operations Incl. Graceful Degradation Standardization The Automated Delivery Pipeline (DevOps) Continuous Integration Environment Management Deployment Orchestration Automated Testing Production Deployment Feedback Mechanisms Layered Testing (GUI, API, Backend) Full System Testing (Functional, UX Verification, Regression, Security, Performance, Compatibility, Acceptance) Environment Sanity Tests Application Resiliency Testing (Chaos Monkey, etc.) Non-Functional Sanity Tests Infrastructure Resiliency Testing Version Build Automation Basic Test (incl. static code quality, security, performance, and code coverage) Package Deploy to Software Asset Repository Governance Checks Feedback Mechanism Infrastructure Integrations Baselined / Modelled Environments Test Data Propagation Automatic Configuration Infrastructure-as- Code Automation On-Premises/ Cloud/Hybrid Virtualization Loosely Coupled Architectures Automated Deployment Development Integration Acceptance Performance Production On-Demand Environment Provisioning Deployment Phased Rollout Rollback Plans APM & Analytics Communication Dashboards/ Scorecards Trends Reports Notifications ALDM Monitor/Measure Everything Continuous Process Improvement Common Metrics Measuring Quality Measuring Efficiency Visible Status Principles Lean Process Debt Removal of Technical Debt Removal of Continuous Experimentation Continuous Improvement Trust Dev in Ops Ops in Dev Common Goals based on Customer Satisfaction Collaboration / Break Collective Blameless Postmortem Openness & Visibility Never Done Culture (Foundational) Down Silos Ownership
Planning (Enterprise) Value Stream Agile Project Tracking Small Batches Increased Frequency Speed to Defect Resolution Minimal Viable Product Build Quality Into the Process Mapping Architecture, Design & Development Developer Commit Application Instrumentation Loosely Coupled Architectures Micro services Branching by Abstraction /Toggling Design for Operations Incl. Graceful Degradation Standardization The Automated Delivery Pipeline (DevOps) Continuous Integration Environment Management Deployment Orchestration Automated Testing Production Deployment Feedback Mechanisms Layered Testing (GUI, API, Backend) Full System Testing (Functional, UX Verification, Regression, Security, Performance, Compatibility, Acceptance) Environment Sanity Tests Application Resiliency Testing (Chaos Monkey, etc.) Non-Functional Sanity Tests Infrastructure Resiliency Testing Version Build Automation Basic Test (incl. static code quality, security, performance, and code coverage) Package Deploy to Software Asset Repository Governance Checks Feedback Mechanism Infrastructure Integrations Baselined / Modelled Environments Test Data Propagation Automatic Configuration Infrastructure-as- Code Automation On-Premises/ Cloud/Hybrid Virtualization Loosely Coupled Architectures Automated Deployment Development Integration Acceptance Performance Production On-Demand Environment Provisioning Deployment Phased Rollout Rollback Plans APM & Analytics Communication Dashboards/ Scorecards Trends Reports Notifications ALDM Monitor/Measure Everything Continuous Process Improvement Common Metrics Measuring Quality Measuring Efficiency Visible Status Principles Lean Process Debt Removal of Technical Debt Removal of Continuous Experimentation Continuous Improvement Trust Dev in Ops Ops in Dev Common Goals based on Customer Satisfaction Collaboration / Break Collective Blameless Postmortem Openness & Visibility Never Done Culture (Foundational) Down Silos Ownership
NORMING: WHO DOES WHAT? The Automated Delivery Pipeline (DevOps) Continuous Integration Environment Management Deployment Orchestration Automated Testing Production Deployment Feedback Mechanisms Layered Testing (GUI, API, Backend) Full System Testing (Functional, UX Verification, Regression, Security, Performance, Compatibility, Acceptance) Environment Sanity Tests Application Resiliency Testing (Chaos Monkey, etc.) Non-Functional Sanity Tests Infrastructure Resiliency Testing Version Build Automation Basic Test (incl. static code quality, security, performance, and code coverage) Package Deploy to Software Asset Repository Governance Checks Feedback Mechanism Infrastructure Integrations Baselined / Modelled Environments Test Data Propagation Automatic Configuration Infrastructure-as- Code Automation On-Premises/ Cloud/Hybrid Virtualization Loosely Coupled Architectures Automated Deployment Development Integration Acceptance Performance Production On-Demand Environment Provisioning Deployment Phased Rollout Rollback Plans APM & Analytics Communication Dashboards/ Scorecards Trends Reports Notifications ALDM Monitor/Measure Everything Continuous Process Improvement Common Metrics Measuring Quality Measuring Efficiency Visible Status ENVIRONMENT MANAGEMENT RELEASE AUTOMATION RELEASE MANAGEMENT APPLICATION RELIABILITY QUALITY ASSURANCE
NORMING DEV INT QA DEV INT QA DEV INT QA QA DEV INT Release management, Better defined schedules 2 week release activity cycle, Bi-monthly maintenance releases Overlapping release cycles, Same branching strategy as development legs
NORMING: QUARTERLY RELEASES 6 Months Summer CTP Code Freeze Prod Performance Testing P2 Release Development and Functional Testing SIT Testing Data Creation Functional Code Freeze Final Code Freeze m DEV INT QA 2 2 Weeks 6 Months
NORMING: FULL RELEASE SCHEDULE DEVELOPMENT LEG 1 6 Months Summer Summer Winter DEVELOPMENT LEG 2 Spring Spring Fall Spring RELEASE LEG m m m m m m m m m m m m M M m m m m m M m m m m m M m m m m m M 2
NORMING: TESTING LOCAL/DEV Development and Unit Testing Functional Testing DEV/INT/DENT INT Functional Integration Testing Phase 1 Phase 2 Phase 3 TDC QA HIST SIT QA Cycle 2 Cycle 1 Cycle 3 Program level test lead for coordination of functional and integration testing Phased / cycle approach to functional integration and SIT testing 10 week SIT duration with data creation critical path constraint removed Introduction of forced month ends for financial testing
TESTING BENFITS FROM DEVOPS Continuous integration builds Gated deployments based on testing results Automated unit testing Automated smoke testing Automatic rollback for failed deploys Environment stability Quicker, higher quality code delivery Ability to feature toggle and/or deploy dormant code Automated deployment notification and release notes Better merging strategies ease regression testing requirements Exposure into development process
NORMING: TOOLS Maven for configuration GIT for code management CA Harvest for build and deploy JUnit / GUnit for continuous integration testing Jenkins for automated deployments CA Dev Test (formerly Lisa) for automated testing Word for requirements (moving toward TFS) Excel for testing plan (moving toward TFS) Excel release plans Puppet for IaaS CA Service Desk for change management and configuration item repository Artifactory for Enterprise Data Management Solution (EDMS)
QUESTIONS? THANK YOU TOOL APPENDIX FOLLOWS
APPENDIX A: TOOL RESEARCH Tool AccuRev (Micro Focus Software) Agile Software Factory (Grid Dynamics) Amazon ECS Ansible (Red Hat) Apache Ant Category Configuration Management Application Release Automation Container Management Configuration Management, Application Release Automation Yes Build Management Trial No No No Site borland.com griddynamics.com aws.amazon.com ansible.com ant.apache.org Yes Name Ansible Artifactory Asgard AWS Awsbox Description An agent less orchestration engine that can automate application and systems. JFrog s open source binary repository management product with support for Apache Maven artifacts. A netflix web interface tool for AWS Cloud Management and application deployment. A cloud computing platform offering ready to use web services which are loosely coupled. A featherweight DYI PaaS on top of awscloud EC2 for deploying node apps. http://www.devopsbookmarks.com/ https://xebialabs.com/periodic-table-of-devops-tools/
