4. CESE
The structure of the approach
Model
List of
abstract
actions
ITestMonkey
List of abstract
input data
provider
TestMonkeyImpl ITestDataProvider methods
Interface
System Under Test TestDataProviderImpl
Model is agnostic to the test execution technology
ITestMonkey interface hides the test execution framework
TestMonkeyImpl uses interfaces of the test execution framework
5. CESE
Tools Infrastructure
• Modeling: Yed Graphical Editor from yWorks
• Model traversal and test generation: Jumbl- Uni. Tennessee
• Test Execution:
– Junit (Java)
– CuTest (C)
– Selenium (Web)
– UISpec (Java Swing)
– Sikuli (Image-based testing of legacy systems)
• Glue scripts:
– Conversion of Yed models to Jumbl models
– Preparing a test suite from generated test cases
– Generation of system-specific build files (e.g., makefiles)
– Helper scripts to clean-up generated files
6. CESE
Test Generation @ GMSEC …
• State-of-the-practice: Test cases are hand-crafted
• New initiative started to evaluate the feasibility of
the FAST approach
• Modeled a portion of the GMSEC Software Bus
based on existing test cases and documentation
• Automatically generated test cases
• Found a few problems (already fixed now)
7. CESE
Hand-crafted test case (snippet)
public static void main( String args[] ) {
Status result = new Status();
Connection conn = new Connection();
ConnectionConfig cfg = new ConnectionConfig( args );
// Create the connection
result = ConnectionFactory.Create( cfg, conn );
checkError( false, result, "Creating the connection object" );
// Disconnect
result = conn.Disconnect();
checkError( true, result, "Disconnecting before connection is established" );
// Connect
result = conn.Connect();
checkError( false, result, "Establishing the connection to the middleware" );
} //..main()
8. CESE
Manually developed test cases –
source of Inspiration
• We reviewed existing Java test cases
• Found that the tester has used certain
permutations of API-usage
• Also, both good and “evil” cases are considered
• We used these test cases as a source of
reference for building API usage models
9. CESE
Test Generation @ GMSEC …
APIs of the module under test
public interface IConnection
{
public Status Connect();
public Status Disconnect();
…
}
12. CESE
Sample APIs
/******************************************************************************
** Directory API
******************************************************************************/
// Makes a new directory
int32 OS_mkdir (const char *path, uint32 access);
// Opens a directory for searching
os_dirp_t OS_opendir (const char *path);
// Closes an open directory
int32 OS_closedir(os_dirp_t directory);
// Removes an empty directory from the file system.
int32 OS_rmdir (const char *path);
14. CESE
API doc of open directory
/*--------------------------------------------------------------------------------------
Name: OS_mkdir
Purpose: makes a directory specified by path.
Returns: OS_FS_ERR_INVALID_POINTER if path is NULL
OS_FS_ERR_PATH_TOO_LONG if the path is too long to be stored locally
OS_FS_ERR_PATH_INVALID if path cannot be parsed
OS_FS_ERROR if the OS call fails
OS_FS_SUCCESS if success
Note: The access parameter is currently unused.
---------------------------------------------------------------------------------------*/
int32 OS_mkdir (const char *path, uint32 access);
17. CESE
Sample generated Test in CuTest
void Testosal_Filesystem_min_2(CuTest* tc)
{
status = makeFilesystemValid();
CuAssertIntEquals_Msg(tc,"Filesystem could not be created",
OS_FS_SUCCESS, status);
status = mountFilesystemValid();
CuAssertIntEquals_Msg(tc,"Filesystem could not be mounted",
OS_FS_SUCCESS, status);
pointer = openDirectoryValid();
CuAssertTrue(tc, pointer != NULL);
…
status = removeFilesystemValid();
CuAssertIntEquals_Msg(tc,"Filesystem could not be removed”, status);
}
18. CESE
Issues found using this method
• File-descriptors after removing file-system:
• After somewhat long tests we would run out of
file-descriptors
• This would even happen with a newly created
file-system
• OSAL does not remove file-descriptors for files
open when the file-system is removed
• Unable to create and open files
• Some wrong error codes returned