Automated Testing and Baseline Management Overview

Dive into the world of automated testing with a focus on simulating user actions, UI checkpoints, and test suite execution. Explore the eyes system for visual checkpoints and result matching. Learn about writing application and test code, handling baseline management, and resolving mismatches efficiently.

• Automation
• Testing
• UI Checkpoints
• Baseline Management
• Test Suite

try_ras Follow

Uploaded on Oct 10, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

Presentation Transcript

1. Test Suite 2.1: Simulate User actions Application Under Test Simulate UI Checkpoint Simulate UI Checkpoint 2.2: Take screenshot Simulate UI Checkpoint : : Eyes System 3: Match results Match results Metadata 1: Start Test Suite 4: Review and Save Results Eyes Test Manager Tester F0001-V02

2. Execution Environment Test Program 2.1: Simulate user actions 2.1: Simulate user actions Application Under Test Driver (e.g. Selenium: Test Suite 2.2a: Get screenshot Simulate user actions Checkpoint image image 2.2a: Get screenshot Simulate user actions Checkpoint 2.2: Visual checkpoint 2.2c: Check screenshot Simulate user actions Checkpoint : : image Eyes Server Eyes SDK 3a: Test results 3: Test results 1: Start test suite execution 4: Get results, update baseline 4: Review test results, update baseline Eyes Test Manager F0001-V04

3. Test Program Execution Environment 2.1: Simulate user actions Test Suite 2.1: Simulate user actions Application Under Test Driver e.g. Selenium 2.2b: Send screenshot Simulate user actions Checkpoint image 2.2a: Get screenshot image Simulate user actions Checkpoint 2.2: Visual checkpoint 2.2c: Check screenshot Simulate user actions Checkpoint : : Eyes Server image 3: Test results 3a: Test results Eyes SDK 4: Get results, update baseline 1: Start test suite execution 4: Review test results, update baseline Eyes Test Manager

4. Write application code Write test code Baseline Baseline Exists ? Reference images regions bugs Change test code Change application code rerun test Yes No mismatches ? passed SAVE Fail/Pass Update baseline image Add ignore/floating regions Bug fixes Bug fixes SDK Ignore details Match Level Review and Resolve Compare captured and baseline images View Similar Differences Group, Sort, Filter by attributes View Match Level influence Mail & Chat with Team members Issues Remarks Chat Assign

5. Img A Img B Img C Img D Img E Img A Img B Img C Img D Img E Img A Img E Img G Img C1 Img D Img F Img A Img B Img C Img D Img E Img A Img C1 Img D Img F Img E Img G same missing diff same new same new

6. Test Suite Test 1 checkPoint 1a checkPoint 1b : Test 2 checkPoint 2a : Test 3. : Test N

7. Write Test Eyes Results Baseline First Test Run Inspect Initial Results Change Test Software Change Application Software Eyes Results Test Run View and resolve results Baseline Update Baseline

8. Baseline <empty> visualTest(){ simulate UI state 1 check( check 1 ) simulate UI state 2 check( check 2 ) simulate UI state 3 check( check 3 ) ) Result: Baseline Image 1 new new Image 1 Image 2 Image 1 Image 1 Image 3 new Image 1 Image 1 Image 1 2) Run test 1st time 1) Write app & test code 3) Review results 4) Baseline created Baseline Image 1 1F new feature visualTest() { simulate UI state 1F check( check 1 ) simulate UI state 2 check( check 2 ) simulate UI state 3B check( check 3 ) ) Image 2 Result: Baseline Image 1F Image 3 diff == Image 1F Image 2 Image 2 Image 1F Image 3 diff Image3B Image 2 Image 3B 3B : bug ! 5) Update app & test code 6) Differences detected 7) Review results 8) Save baseline

9. Baseline Step Test Batch End of 1st test run: + this is the image Thumbnail Captured in The test No further action required updated = No further action required After 2nd test run this is the image Thumbnail Captured in The test = this is the image Thumbnail Captured in The test Update database Pass step Don t update database Fail step + this is the image Thumbnail Captured in The test Report an Issue Fail step - this is the image Thumbnail Captured in The test

10. Test() { e = Eyes( ) e.open(name, ) e.check( point 1 ) e.check( point 2 ) e.check( point 3 ) e.close( point 4 ) ) Baseline ID = name + OS + browser + viewport Image 1 Image 2 Image 3 Image 4

11. Test() { e = Eyes( ) e.open(name, ) e.check( ) //check point 1 e.check( ) //check point 2 e.check( ) //check point 3 e.check( ) //check point 4 e.check( ) //check point 5 e.check( ) //check point 6 result = e.close( ) ) Baseline ID = name + OS + browser + viewport Actual images Baseline image and Metadata Image 1 Image 1 Image 2 Image 2 Image 3 Image 3 Image 4 Image 4 new name or environment Image 5 Image 5 Image 6 Image 6 Test() { e = Eyes( ) e.open(name, ) Baseline ID = name + OS + browser + viewport same name and environment Baseline image Baseline image and Metadata Actual images Image 1 Image 1 Image 1 e.check( ); //check point 1 same e.check( ); //check point 2 differs e.check( ); //check point 3 differs e.check( ); //check point 99 new e.check(); //check point 5 differs e.check( ) //check point 6 same e.close( ) //same ) bug Image 2A Image 2 Image 2 Image 3A Image 3 Image 3A missing Image 4 Image 99 Image 99 ignore Image 5A Image 5 Image 5A Image 6 Image 6 F0002 ver 1

12. Test version 1 before user confirmation actual expected match go to Applitools.com, click contact , wait form displayed name name Email Email new eyes.checkWindow() No expected image OK Enter form Name, click OK John Doe name name Email new eyes.checkWindow( click on OK ) No expected image OK wait for thanks dialog new Thank You eyes.checkWindow( thank you dialog ) No expected image OK F0002a-V02

13. Test version 1 after user confirmation actual expected match go to Applitools.com, click contact , wait form displayed name Email name Email passed eyes.checkWindow() OK OK Enter form Name, click OK John Doe John Doe name Email name Email passed eyes.checkWindow( click on OK ) OK OK wait for thanks dialog passed Thank You Thank You eyes.checkWindow( thank you dialog ) OK OK F0002b-V02

14. Test version 2 before user confirmation actual expected match go to Applitools.com, click contact , wait form displayed name Email Different name Email eyes.checkWindow() OK cancel OK cancel enter form Name, click OK John Buck Different name Email John Doe name Email eyes.checkWindow( click on OK ) OK cancel OK cancel wait warning Dialog shown New eyes.checkWindow( warning dialog ) ! Email missing OK No expected image No expected image Enter email and press OK John Buck JB@email.com cancel name Email New eyes.checkWindow( click on OK ) No expected image OK //wait thanks Dialog shown Missing No actual image Thank You //eyes.checkWindow( thank you dialog ) OK //click OK on Dialog , wait for thanks dialog New No expected image eyes.checkWindow( main window ) F0002c-V02

15. Test version 2 before user confirmation actual expected match go to Applitools.com, click contact , wait form displayed name Email name Email No diffs eyes.checkWindow() OK cancel OK cancel enter form Name, click OK John Doe John Doe name Email name Email No diffs eyes.checkWindow( click on OK ) cancel OK OK cancel wait warning Dialog shown No diffs ! Email missing OK eyes.checkWindow( warning dialog ) ! Email missing OK Enter email and press OK John Doe JD@email.com cancel name Email New No expected image eyes.checkWindow( click on OK ) //wait thanks Dialog shown OK //eyes.checkWindow( thank you dialog ) //click OK on Dialog , wait for thanks dialog wait Dialog closed No diffs eyes.checkWindow( main window ) F0002d-V02

16. Test version 2 before user confirmation actual expected match go to Applitools.com, click contact , wait form displayed name Email name Email passed eyes.checkWindow() OK cancel OK cancel enter form Name, click OK John Doe John Doe name Email name Email passed eyes.checkWindow( click on OK ) cancel OK OK cancel wait warning Dialog shown passed ! Email missing OK eyes.checkWindow( warning dialog ) ! Email missing OK Enter email and press OK John Doe JD@email.com cancel John Doe JD@email.com cancel name Email name Email passed eyes.checkWindow( click on OK ) OK OK //wait thanks Dialog shown //eyes.checkWindow( thank you dialog ) //click OK on Dialog , wait for thanks dialog wait Dialog closed passed eyes.checkWindow( main window ) F0002e-V02

17. Test (application Name, Test Name) Baseline revision N Image Image Execution Environment Region Functionality - ignore - Floating Auto adjust - Paste (??) Env Name Image Image O/S Image Image - App Platform Image Image Viewport Size Team coordination - Remarks - Issues - Chat Branch Eyes Baseline revision N - 1 Baseline revision N - 2 Result Baseline revision N - 3

18. S D K 1) Checkpoint images Test Results Test Manager Matching Engine 2) Baseline images, Matching parameters Baseline 3) Save test results Save parameters

19. Repository Commit comment Hash

20. <OS, Browser,viewportSize> Test:<app_name, test_name> Baseline<viewportSize, OS, Browser> Eyes Server Send UI event Eyes->check() Updated baselines - Selected images - Annotations Test result img1 imgA img2 imgB imgC img4

21. <OS, Browser,viewportSize> Test:<app_name, test_name> Send UI event Eyes->check() Baseline<viewportSize, OS, Browser><Branch> Eyes Server Test result img1 imgA img2 imgB steps Updated baselines - Selected images - Annotations imgC img4

22. <OS, Browser,viewportSize> Test:<app_name, test_name> Baseline<viewportSize, OS, Browser><Branch> Send UI event Eyes Server Eyes->check() <1024x768, windows10, Chrome> imgA <1024x768, windows10, Chrome> imgA <1024x768, windows10, Firefox> imgB <600x900, windows10, Chrome> imgB <600x900, windows10, Firefox> <600x900, OS X, Chrome> imgC Test result imgC imgC <600x900, windows10, Chrome> <1024x768, OS X, Chrome> <600x900, OS X, Chrome> <600x900, Android, Chrome > imgD imgD img1 imgA imgD <1024x768, OS X, Chrome> imgE imgE img2 imgB <600x900, OS X, Firefox > steps imgE Updated baselines - Selected images - Annotations imgF <600x900, Android, Chrome > imgC imgF <1024x768, OS X, Firefox > img4 imgG <1024x768, Android, Firefox > imgK Branch Beta Branch Release imgH

23. baselines <1024x768, windows10, Chrome> <OS, Browser,viewportSize> Baseline<viewportSize, OS, Browser> imgA Test:<app_name, test_name> <1024x768, windows10, Firefox> imgB <600x900, windows10, Firefox> imgC Send UI event <600x900, windows10, Chrome> <600x900, OS X, Chrome> imgD <1024x768, OS X, Chrome> Eyes->check() imgE <600x900, OS X, Firefox > imgF <600x900, Android, Chrome > <1024x768, OS X, Firefox > <1024x768, Android, Firefox > imgG imgH imgK Test result img1 imgA img2 imgB imgC img4

24. App 1 App 2 Test 3 Test 1 Test 2 Test 4 Test 1 Windows, Firefox, 768x1024 Branch Beta Img a1 Baselines Windows, Firefox, 1024x768 Windows, Firefox, 1024x768 Windows, Firefox, 1024x768 Windows, Firefox, 1024x768 Windows, Firefox, 1024x768 Img 2a Rev 1245 annotations Rev 2356 Img a3 Latest Windows, Firefox, 768X1024 Windows, Firefox, 768X1024 Windows, Firefox, 768X1024 Windows, Chrome, 768X1024 Windows, Chrome, 768X1024 Img 1b Results Img 2b Windows, chrome, 1900x1024 Windows, chrome, 1900x102 4 Windows, chrome, 1900x102 4 Results 1 Img b3 Results 2 Results 3 img 4b Mac OS, chrome, 1900x1024 Mac OS, chrome, 1900x1024 Mac OS, chrome, 1900x1024 Branch Release Windows, chrome, 1900x1024 Windows, chrome, 1900x102 4 Windows, chrome, 1900x102 4 Windows, Chrome, 768X1024 Windows, Chrome, 768X1024

25. Code Eyes 0) System has main branch code and baseline. Main Branch M.1 Main Branch EM.1 Test M 1) Start feature B, branch code & baseline. Test B1 that branch worked OK no changes yet. Feature B.1 Baseline EB.1 Test B1 2) Test B2, code change B.2 , no baseline change EB.1, test B2 no regression. Feature B.2 Test B2 3) Change code B.3, change baseline EB.2, test B3 new/changed steps and annotations Baseline EB.2 Feature B.3 Test B3 4) Start feature A branch code & baseline. Test A1 that branch worked OK no changes yet. Feature A.1 Baseline EA.1 Test A1 5) Feature B complete, trivial merge code and baseline and test B4 Compare and auto merge Main Branch M.2 Main Branch EM.2 Test MB Feature A.2 6) Feature A code A.2 and baseline EA.2 changes . Test A2 Baseline EA.2 Test A2 7) Feature A ready, merge M+B code to A code, compare Baseline, add M+B annotations, rerun test. Compare Feature A.3 Baseline EA.3 Test A3 Merge Main Branch M.3 8) Main code and baseline contains MAB . Check regression because of merge Main Branch EM.2 Test MBA

26. Code Eyes Main Branch E Main Branch M 0) System has main branch code and baseline. Test M 1) Start feature B, branch code & baseline. Test B1 that branch worked OK no changes yet. Feature B.1 Baseline EB.1 Test B1 2) Test B2, code change B.2 , no baseline change EB.1, test B2 no regression. Feature B.2 Test B2 3) Change code B.3, change baseline EB.2, test B3 new/changed steps and annotations Baseline EB.2 Feature B.3 Test B3 4) Start feature A branch code & baseline. Test A1 that branch worked OK no changes yet. Feature A.1 Baseline EA.1 Test A1 5) Feature B complete, trivial merge code and baseline and test B4 Compare and auto merge Main Branch MB Main Branch EMB Test MB Feature A.2 6) Feature A code A.2 and baseline EA.2 changes . Test A2 Baseline EA.2 Test A2 7) Feature A ready, merge M+B code to A code, compare Baseline, add M+B annotations, rerun test. Compare Feature A.3 Baseline EA.3 Test A3 Merge Main Branch MBA 8) Main code and baseline contains MAB . Check regression because of merge Main Branch EMBA Test MBA

27. For team T1 { For Slack { {workspace/channel {<app a1, all runs>, <app a2, unresolved>, .} { .other workspaces/channels} } For Email { { A@a.com, B@a.com,...} <app a2, all runs>} { other email address, .} } } For team T2 { ..} - Batch(B, {user, termNotification=true; - <test t1, app a1, results>, - <test t2, app a1, results>, - <test t3, app a2, results>} Eyes Server Email Notify Slack Notify Find notification list for users team Find notification list for users team Test Suite Test Manager (Admin) termNotification = true Notify this application ? Notify this application ? Notify this type of result ? Notify this type of result ? Sent to Slack workspace/channel Sent to email list {<app a1, all runs>, <app a2, unresolved>, <app a2, all runs> Slack Email

28. runner = new VisualGridRunner()or new ClassicRunner() config = new Configuration() Set up the test suite eyes = new Eyes(runner) driver = create web driver Set up before the test webdriver = eyes.open(driver) eyes.setConfiguration(config) The test webdriver.do something eyes.check() eyes.closeAsync() driver.quit() Wrap up after test Wrap up after test suite results = runner.getAllTestResults(false)

29. Outer region up Inner region right left down