Overview of OPNFV Test Strategy and Project Interactions
Testing remains a key challenge for OPNFV, with projects managing unit, functional, security, and performance testing. Specific test projects have been validated by the TSC, covering test cases, performance analysis, and more. The interactions between projects like Releng, DB, and test projects are integral to integrating testing into the CI pipeline and providing result reporting and analytics for OPNFV solutions.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Test strategy Testing is still a key challenge for OPNFV All the projects must manage their test strategy (unit, fonctional, security, performance) Several specific test projects have been validated by TSC -Define testcases -Perform tests not covered by a single project -Create tooling -Study Performance end to end
R1 In R1: -BGS managed its sanity tests -Functest created and managed 4 test suites -Octopus was in charge to run Functest at the end of a fresh installation -Pharos was in charge of providing documentation on hardware and testbeds
R2 -More testcases (vIMS, ?) -Better coverage -Detect missing test domains -Analytics system
Overview of the different projects Releng DBFor Result reporting/storage/analy tics Traffic profile straorea Releng Test projects Functest QTIP Transformers VPerf Yardstick Testbed Pharos Note: to be discussed Common could be managed by Releng In R1
Project Interactions Artifact Releng DB Test projects Tests are integrated in CI through Releng scripts Releng provides Scripting/automation/DB for result reporting/storage/analytics Octopus Pharos exposes - available labs (lab topology, tooling,..) - history of test campaigns Tests are run on Pharos testbed(s) Releng Pharos (federation of testbeds) Releng provides - scripts to triggers build/deploy OPNFV solution on pharos labs - Scripting/automation/DB for installation build/deploy reporting/storage/analytics Pharos generates community dashboard (aggregation of info of build/deploy and tests)
Pharos Is not a test project Is a testbed management project Develop a tool to dynamically retrieve configuration (hardware, network, tooling) Manage history of test campaigns Create Guides HW Requirements Best practices Create a tool to manage testbed resource Create a Dashboard (describing the available testbeds, their capabilities, )
Releng Project Scope R1R1 R2 deliverables Releng (Fatih Degirmenci) Common functions -Collection of test profiles/payloads,.. - Collection of results + DB + Analytics - Test Dashboard - Analytics tooling through API for any test project - traffic profile definition and collection Part of Octopus, help creating CI jobs
Test projects Project Scope R1 R2 deliverables Functest (Morgan Richomme) Common functions for testing project -definition of default configuration for tools -definition of format for output files - coverage - Test Dashboard - Tempest + rally suite - ODL suite - vPing - 4 suites integrated in CI - Prescription (format, configuration,..) - Test Dashboard (API for Pharos) - Test coverage - testcase search engine - vIMS testcase - vSBC? -new testcases in upstream suite -Move from Tempest smoke to full? - new functional tests - integration of upstream suites (R1) - update of existing suites Vperf (Mike Lynch) Vswitch perf testing Dev & tests done but not integrated in CI for v1 New tests Integration in CI Yardstick (Ana Cunha) In -VM testing Tooling Test suites - Methodology for Infrastructure verification from VNF perspective, aligned with ETSI-NFV TST001 - Generic test cases for Compute, Storage and Networking areas - OPNFV Infrastructure verification from VTC and SFC perspective - Mobility Traffic Profile as Test Stimuli Qtip (Wenjing Chu) Transformers (Michael Wiegers)
Test projects Test System Scope Functional component/subsystem/en d-to-end R1 Functest - Tempest - ODL R2 deliverables Functest - Tempest - ODL - VIMS? - System testing from a user (i.e. VNF) perspective Performance Functest - VPing Yardstick OPNFV components Vperf VPerf OPNFV system level testing Security Functest (Rally) Qtip? Functest (Rally) ?
Test Coverage by component Component OPNFV project External tool OpenStack Functest Tempest, Rally ODL Functest/ODL Robot KVM Yardstick OVF Yardstick VNF Functest (vPing, vIMS, vXXX) Yardstick Spirent/Ixia tooling Ceph Functest?, Yardstick
Test Coverage by domain Domain OPNFV project Comments Security Installation BGS OAM ETSI domains -vEPC -vIMS -Mobile base stations -Fixed Access Network -vHGW -vCDNs -Network Functions Virtualisation IaaS -Virtual Network Function aaS -Virtual Network PaaS -VNF Forwarding Graphs Functest Yardstick Is it possible to automate/integrate some of the 36 ETSI PoCs into OPNFV? OpenStack Telco WG testcases https://wiki.openstack.org/wiki/T elcoWorkingGroup#Use_Case_ Definition
Common needs: Big data / analytics => Releng Problem: Thanks to CI and test projects we will get lots of results logs coming from different testbeds (pharos federation including different hardware, configuration), the idea is to perform some analytics on these tool sets to study the influence of hardware on performance The tool could include: -API for collecting the results of all the tests (from the artifact) -A way to associate a toolset and a configuration (description needed from Pharos) -A NoSQL DB to store the results -Analytic tooling -Analytics script -Display and share results
Reference Testcases => Releng + Functest Problem Each OPNFV subproject is responsible for its test strategy. Specific test projects have been created. It is difficult to get the big picture of the testcases by checking project by project. List of testcases shall be automatically generated. It shall be possible to search somewhere if testcase has already been run, in which project. It would also be interesting to have an history of the testcases (testcase X from project Y is run oin testbed Z1,Z2, .) The task will thus mean: -Automation of production of a document collecting all the testcases -Keywords + search engine system: goal know if testcase already considered, which suite, how to automate it,.. -Template for test case documentation
