Layers of Testing
The purpose of testing is to verify expected functionality and find potential defects. Testing enhances the confidence in delivering qualified products and also brings the possibility of agile iteration. It is safe to say that testing determines the product development progress.
As you may already know, in networking, there are 7 layers of OSI model and 4 layers of TCP model. In development, we also have different patterns such as MTV, MVC, MVP, and MVVM. These technologies have become more mature featuring high cohesion, low coupling, and division of responsibilities, modules, and layers. They have helped us build architectures and standards.
Likewise, we also have different types of testing — UI testing, API testing, and unit testing. Testing is a way to balance output and cost rather than a new type of technology.
You can see from the pyramid of testing shown in the above image that the higher layer of testing entails higher cost, stricter environment requirements, longer implementation period, and more careful maintenance. However, the higher the layer of testing gets, the closer it is to the end-user scenario. In the book How Google Tests Software, Google suggested a 70/20/10 split: 70% unit testing, 20% API testing, and 10% UI testing.
This blog shares the idea on how to use the KubeSphere DevOps System to run automated tests for applications driven by Kubernetes.
What Is KubeSphere DevOps
Based on Jenkins, KubeSphere provides a one-stop DevOps system for the use cases of Kubernetes and containers, including rich CI/CD pipeline building and plug-in management functions, Binary-to-Image (B2I), and Source-to-Image (S2I) for DevOps in Kubernetes. Code dependency caching support is also provided for pipelines, S2I, and B2I, as well as other functions such as code quality management and pipeline logging.
The built-in DevOps system of KubeSphere smoothly combines application development and automated release with the container platform. It also supports the integration of third-party private image registries and code repositories, which brings improved CI/CD functions under private scenarios and provides end-to-end user experiences.
Nevertheless, it is rarely known to users that KubeSphere DevOps can also be used to build automated testing systems, so that automated unit testing, API testing, and UI testing can have great convenience and the work of testers can be more efficient.
Unit Testing
Unit testing has a high running frequency that it should be triggered with every code commit. Due to few dependencies, unit testing usually only requires one container runtime environment.
The following is an example of running a unit test in a pipeline using golang:latest.
pipeline {
agent {
node {
label 'go'
}
}
stages {
stage('testing') {
steps {
container('go') {
sh '''
git clone https://github.com/etcd-io/etcd.git
cd etcd
make test
'''
}
}
}
}
}
You can see the pipeline logs as below:
As for other languages and frameworks, unit testing can also be easily run on Kubernetes by installing certain packages and Mock-related services. As such, we should focus more on how to write useful unit tests instead of being obsessed with runtimes or unit test plans.
API Testing
If a team is just getting started with its automated testing, automated API testing would be a great starting point.
A unit test is mainly written by R&D folks. In a rapid iteration process, the experienced R&D folks will not forget to write unit tests. Testing plays a much more important role in the faster process of refactoring and making changes. The unit testing is overlooked if no one cares to write any, and it is quite difficult to propel something that is neglected by the practitioners and has limited benefits to show to the managerial folks.
Besides, automated UI testing is usually replaced by manual testing. Meanwhile, the maintenance of automated UI testing brings high costs, which means the automated UI testing in the process of rapid iteration should be performed reasonably.
API tests are more preferable if team members are more familiar with APIs as they run tests with complete API documents and materials as reference, especially in an architecture with separate frontend and backend code.
The following is an example of running an automated API test in a pipeline using Postman.
pipeline {
agent {
kubernetes {
label 'apitest'
yaml '''apiVersion: v1
kind: Pod
spec:
containers:
- name: newman
image: postman/newman_alpine33
command: [\'cat\']
tty: true
volumeMounts:
- name: dockersock
mountPath: /var/run/docker.sock
- name: dockerbin
mountPath: /usr/bin/docker
volumes:
- name: dockersock
hostPath:
path: /var/run/docker.sock
- name: dockerbin
hostPath:
path: /usr/bin/docker
'''
defaultContainer 'newman'
}
}
parameters {
string(name: 'HOST', defaultValue: '10.10.10.10', description: '')
string(name: 'PORT', defaultValue: '8000', description: '')
string(name: 'USERNAME', defaultValue: 'admin', description: '')
string(name: 'PASSWORD', defaultValue: 'password', description: '')
}
stages {
stage('testing') {
steps {
sh '''
apk add --no-cache bash git openssh
git clone https://yourdomain.com/ns/ks-api-test.git
cd ks-api-test
sed -i "s/__HOST__/$HOST/g" postman_environment.json
sed -i "s/__PORT__/$PORT/g" postman_environment.json
sed -i "s/__USERNAME__/$USERNAME/g" postman_environment.json
sed -i "s/__PASSWORD__/$PASSWORD/g" postman_environment.json
npm install -g newman-reporter-htmlextra
newman run iam/postman_collection.json -e postman_environment.json -r htmlextra
'''
}
}
}
post {
always {
archiveArtifacts 'ks-api-test/newman/*'
}
}
}
The artifact will be archived after the running process:
The framework of automated API testing is easy to establish with the implementation of the following functions:
Interface Request
Response Assertion
Request Orchestration
Report Generation
Keep in mind that you need to select the solution that best suits your team based on your API tests and delivery habits. For example, you can develop your own solution or just use the existing tools. The above solution chooses Postman and Newman for the reason that the team generally uses Postman for API testing.
What’s left is how to organize the testing accross the team. The team can either separately submit files to the same repository, or use the charged version of Postman to share data for centralized testing.
UI Testing
The high cost of automated UI testing can be attributed to the following:
Testing cases are difficult to maintain as the front-end styles and product logic varies.
A stable operating environment is difficult to provide, and various timeouts and dirty reads will lead to a high failure rate.
The automated UI testing here uses the Robotframework I’m familiar with, and keywords are used for the automated testing.
The following is an example of running an automated UI test in a pipeline using Robotframework.
pipeline {
agent {
kubernetes {
label 'robotframework'
yaml '''apiVersion: v1
kind: Pod
spec:
containers:
- name: robotframework
image: shaowenchen/docker-robotframework:latest
tty: true
volumeMounts:
- name: dockersock
mountPath: /var/run/docker.sock
- name: dockerbin
mountPath: /usr/bin/docker
volumes:
- name: dockersock
hostPath:
path: /var/run/docker.sock
- name: dockerbin
hostPath:
path: /usr/bin/docker
'''
defaultContainer 'robotframework'
}
}
parameters {
string(name: 'HOST', defaultValue: '10.10.10.10', description: '')
string(name: 'PORT', defaultValue: '8080', description: '')
string(name: 'USERNAME', defaultValue: 'admin', description: '')
string(name: 'PASSWORD', defaultValue: 'password', description: '')
}
stages {
stage('testing') {
steps {
sh '''
curl -s -L https://raw.githubusercontent.com/shaowenchen/scripts/master/kubesphere/preinstall.sh | bash
git clone https://yourdomain.com/ns/ks-ui-test.git
cd ks-ui-test
sed -i "s/__USERNAME__/$USERNAME/g" tests/common.robot
sed -i "s/__PASSWORD__/$PASSWORD/g" tests/common.robot
echo "\nTestEnv http://$HOST:$PORT" >> tests/api.robot
echo "\nTestEnv http://$HOST:$PORT" >> tests/devops.robot
./start.sh'''
}
}
}
post {
always {
sh 'tar cvf report-$BUILD_NUMBER.tar ks-ui-test/tests/report'
archiveArtifacts '*.tar'
}
}
}
You can see the pipeline logs as below:
Download the archived artifact and decompress it to view the testing report:
About KubeSphere
KubeSphere is an open-source container platform based on Kubernetes with applications at its core. It provides full-stack IT automated operation and streamlined DevOps workflows.
KubeSphere has been adopted by thousands of enterprises across the globe, such as Aqara, Sina, Benlai, China Taiping, Huaxia Bank, Sinopharm, WeBank, Geko Cloud, VNG Corporation and Radore. KubeSphere offers wizard interfaces and various enterprise-grade features for operation and maintenance, including Kubernetes resource management, DevOps (CI/CD), application lifecycle management, service mesh, multi-tenant management, monitoring, logging, alerting, notification, storage and network management, and GPU support. With KubeSphere, enterprises are able to quickly establish a strong and feature-rich container platform.
For more information, please visit https://kubesphere.io
.gif)
0 Comments