FaSilET² full end-to-end testing solution presented at OW2con'19, June 12-13, 2019, Paris.

Confidentiel SILICOM
© 2017
© Silicom 2019
1
Modifiez le style du titre
© Silicom 2019
SILICOM
FaSilET² : End To End Tester
Multifunction Tool for Developers and QA

© 2017
© Silicom 2019
2
§ Framework and tool for developers and quality assurance
§ Interact with code in production conditions
§ No need to insert test-only code or stubs
§ Act like a human tester
FaSilET² : Main concepts

© 2017
© Silicom 2019
3
§ Actuators :
Scripts launched in a pre-established order defined in a config file.
Simulate human interactions through GUI or CLI and catch their returns.
May be launched in parallel to simulate multiple users at the same time, using
LXC containers.
Examples :
§ Web interface interaction simulation using Selenium
§ Writing into files using ssh or telnet
§ Database modification
§ Physical buttons action simulation using /proc or /sys
§ Etc.
FaSilET² : Vocabulary

© 2017
© Silicom 2019
4
§ Validators :
Get information on changes resulting from actuators activities
Zero to many validators for every actuator
Launched in parallel in LXC container prior to the corresponding actuator
Store validation results until needed
Return status PASSED, FAILED, TIMEOUT

© 2017
© Silicom 2019
5
§ Validators :
Exemples :
§ Reading local or remote logs
§ Database queries
§ Email reception
§ Etc.

© 2017
© Silicom 2019
6
§ Monitors :
Monitor one or more machine components during actuators execution.
Launched in parallel in LXC containers prior to test session start.
Stopped after the end of the whole process
Timestamped logs to compare with actuator activity
Exemples :
§ CPU usage
§ RAM usage
§ Disk space
§ Etc.

© 2017
© Silicom 2019
7
FaSilET² is currently a little better than a proof of concept.
Core is ready
Some modules are at production level
Validators :
§ Logs reader using ssh connections
§ Databases reader
Actuators :
§ Web (Selenium)
FaSilET² : Current development status

© 2017
© Silicom 2019
8
Current version
FaSilET² : Current architecture
Validator
1
Validator
n
GUI or system
under
tests
Logs
server
L
X
C
Action
GUI returns
Other
(mail, etc.)
Results
Centralization
(JSON file)
Results
analysis
Actuators
Validator
2
Exécute
Database
Monitor
1
Monitor
n
L
X
C
L
X
C

© 2017
© Silicom 2019
9
Dynamic operation in list
FaSilET² : Architecture
Actuator
parsing
Containers
creation for
validators
and
monitoring
Validators
and
monitors
started
Actuator
started
Waiting for
validators
Getting and
storing
results
Next
actuator
Other
actuator in
list
?
Et2.pl
JSON
export

© 2017
© Silicom 2019
10
§ Scripts based
Operation based on Perl scripts
Use abstraction modules
For exemple :
§ ET2::Web::Selenium
§ ET2::Utils::Report
§ ET2::Validators::SSHLogs
§ Etc.
FaSilET² : Operation

© 2017
© Silicom 2019
11
§ Directory layout
Testing scripts must be put into directories like this :
test
├── actuators
│ ├── activate_equipment.pl
│ ├── badlogin.pl
│ ├── finish.pl
│ ├── login.pl
│ ├── ntp.pl
│ └── share.pl
├── et2.conf
├── monitors
│ └── CPU1.pl
└── validators
├── Log1.pl
├── Log2.pl
└── LogNTP.pl
And the main script named et2.pl is called with the top directory as parameter
Execution order of actuators is given in configuration file et2.conf. This file is a simple “ini” format
one.
FaSilET² : Operation

© 2017
© Silicom 2019
12
§ Results
Test results are stored in two JSON files :
§ Results from actuators
§ Results from validators
A graphical analysis solution development is planned.
Results are stamped with UIDs
UID format : <actuator_number>-<action_number>-<start_ts>-<stop_ts>
FaSilET² : Running

© 2017
© Silicom 2019
13
§ Actuator result example
{
"actuators" : {
"login" : {
"results" : {
"wait_for_text" : {
"main" : "SUCCESS",
"description" : "Waiting for error message",
"UID" : "0001-0001-20190425132501-20190425132512"
}
}
},
"ntp" : {
"results" : {
"change_ntp_server" : {
"main" : "SUCCESS",
"description" : "Checking NTP server is the one entered previously",
"comment" : "NTP server read : 0.debian.pool.ntp.org",
"UID" : "0002-0001-20190425132602-20190425132623"
}
}
},
…
FaSilET² : Running

© 2017
© Silicom 2019
14
§ Validator result example{
"maintitle" : "SG4000",
"maindescription" : "SG4000 end to end tests",
"actuators" : {
"ntp" : {
"description" : "Change NTP server address",
"results" : {
"LogNTP" : {
"status" : "PASSED",
"data" : [],
"params_ERROR" : null,
"params_OK" : "server 0.debian.pool.ntp.org",
"UID" : "0002-0001-20190425132400-20190425132730",
"name" : "LogNTP"
}
},
"status" : {
"LogNTP" : "ok"
}
},
"login" : {
"status" : {
"Log1" : "ok"
},
"description" : "Good login tests",
"results" : {
"Log1" : {
"status" : "PASSED",
"params_OK" : "WARN admin_vpn0: connection",
"name" : "Log1",
"UID" : "0001-0001-20190425132412-20190425132842",
"data" : [
"2019/03/25 15:44:15 WARN admin_vpn0: connection "
],
"params_ERROR" : null
}
}
},
…
FaSilET² : Running

© 2017
© Silicom 2019
15
FaSilET² : Screenshot

© 2017
© Silicom 2019
16
§ Providing a VM image (OVA), LXC or LXD template
§ Add network scanning modules (nmap, wireshark…)
§ Add IA modules
§ GUI to generate actuator and validator scripts
§ Other ways to use this framework :
§ Cyber security (attack and defense)
§ Quality Assurance
§ …
FaSilET² : Near future
Improvements

© 2017
© Silicom 2019
17
Modifiez le style du titre
© Silicom 2019
Thank you
Any questions ?
Sources available on gitlab : https://gitlab.com/silicom/fasilet

© 2017
© Silicom 2019
18
Actuator example :
FaSilET² : Scripts exemple
#!/usr/bin/perl
use 5.20.0;
use File::Basename;
my $dirname = dirname(__FILE__);
require "$dirname/share.pl";
…
use ET2::Web::Selenium;
use ET2::Utils::Report;
our @VALIDATORS = ( 'Log1' );
…
if(!defined($sel))
{
$sel = ET2::Web::Selenium->new(
host => "https://$host",
driver => "firefox",
encoding => 'latin1',
);
$sel->setname("Sel1");
$sel->start();
}
if(!defined($report))
{
$report = ET2::Utils::Report->new(
title => 'VPN tests',
description => 'End to end tests for VPN project'
);
}

© 2017
© Silicom 2019
19
FaSilET² : Scripts example
$sel->goto('/');
$sel->write(name => "username", value =>
"admin_vpn0");
$sel->write(name => "password", value =>
"admin_vpn1");
$sel->clickbtn(class_name => "btn");
$ret = $sel->waitforpage(
timeout => 2,
url =>
'https://10.10.2.4/#/configuration/localnetwork/
ipconfig'
);
if($ret > 0)
{
say “Login : TIMEOUT”;
}
else
{
$rettitle = $sel->gettitle();
$returl = $sel->geturl();
if(($returl =~ /config$/) && ($rettitle =~ /^Param/))
{
say “Login : TEST SUCCESSFUL";
}
else
{
say “Login : TEST FAILED";
}
}

© 2017
© Silicom 2019
20
Validator example :
#!/usr/bin/perl
use 5.20.0;
…
use ET2::Validator::SSHLogs;
sub run
{
my $vlog = new ET2::Validators::SSHLogs(
host => $host,
port => 10022,
key => '/home/cedric/Developpement/keys/ssh.key',
logfile => '/var/log/diags.log',
timeout => '0..20',
autostart => 1,
user => "root"
);

© 2017
© Silicom 2019
21
return undef if(!defined($vlog));
$vlog->setregexok("WARNsw+s: connection");
$vlog->setname("Log1");
$vlog->tail();
return $vlog->getresults();
}
run();

© 2017
© Silicom 2019
22
FaSilET² : Why such a name ?
• ET² stands for End To End Testing = ETET = (ET) ²
• Sil stands for Silicom
• Fa(r) is a long, long way to run… because we can do many
things with it, mainly a name one can easily remember J

FaSilET² full end-to-end testing solution presented at OW2con'19, June 12-13, 2019, Paris.

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à FaSilET² full end-to-end testing solution presented at OW2con'19, June 12-13, 2019, Paris.

Similaire à FaSilET² full end-to-end testing solution presented at OW2con'19, June 12-13, 2019, Paris. (20)

Plus de OW2

Plus de OW2 (20)

Dernier

Dernier (20)

FaSilET² full end-to-end testing solution presented at OW2con'19, June 12-13, 2019, Paris.