Антон Семенченко, опыт в IT более 10 лет, работает в компании ISSoft, специализируется в разработке и автоматизированном тестировании ПО плюс менеджмент\продажи. C++ Architect, Automation Practice Lead, PM, Group Manager «Agile ValueTeam, учимся понимать Scrum». IT секция. Agile отделение. Для всех уровней подготовки. «Как эффективно продавать Automation Service». IT секция. Продажи. «Как эффективно организовать Автоматизацию, если у вас недостаточно времени, ресурсов и денег». Development секция. Отделение тестирования.

1. Coherent Solutions, Inc. Confidential Page 1 of 3
Coherent Solutions – QA Automation Questionnaire
The purpose is to provide delivery manages, sales people and QA leads a structured tool to collect
information about QA automation opportunities for the purpose of (a) qualifying the opportunity (i.e. is
it real); (b) scoping the opportunity to determine parameters for proposals; (c) demonstrating
Coherent’s automation expertise by asking intelligent questions; and (d) providing
requirements/constraints used to staff and initiate the team’s activities.
1) What are your QA automation goals, vision, high level needs? Which factors influenced your
decision to consider automation?
2) What type of benefits do you expect to get out of automation? Specific cost savings in reducing
manual labor? Better quality by providing broader test coverage, faster releases by improving
regression testing efficiency?
3) What types of Automation are you interested in pursuing?
 UI
o Smoke testing
o Regression testing
o Functional testing
o Other testing
 Performance testing
 Domain specific (database testing)
 Application development business process specific (like Behavior Driven
Development and Testing)
 Code driven testing
o Web Services / APIs
 Smoke testing
 Regression testing
 Functional testing
 Other testing
 Performance testing
 Domain specific (database testing)
 Application development business process specific (like Behavior Driven
Development and Testing)
4) Is your application stable (how much do you change the application UI and APIs) enough for the
efficient Automation? How often will your application be changed?
Do you have an established process for managing changes to the existing application? If yes, is it
well documented and closely followed? How often do you see exceptions to the process?”
 UI changes often trigger changes in automation tests. The degree to which changes drive
automation maintenance effort depends on the automation solution’s architecture, the
automation tools used, and the implementation technology of the underlying application.
Therefore, if an application’s UI will be changed frequently, this becomes an important
consideration in selecting an appropriate automation technology that can be resilient to those
changes

2. Coherent Solutions, Inc. QA Automation Questionnaire
Coherent Solutions, Inc. Confidential Page 2 of 3
 When backend Interfaces are changed, code driven automation tests will need to be modified
appropriately.
 An unstable application will increase the maintnenace effort required for the automation suite
thereby increasing its costs and reducing ROI.
5) What Automation experience do you have internally in your organization? Do you have existing
tools, standards, etc. in place or do you need a vendor to develop those?
6) How you envision execution of the Automation suite?
 Do you envision running Automation manually on demand?
 Do you envision running Automation as a part of Continuous Integration (CI)? If yes, do you have
any requirements/constraints defined for how that would work?
 Do you envision running Automation as a part of daily development flow? If yes, do you have
any requirements/constraints defined for how that would work
7) Do you have requirements/constraints on the automation solution’s architecture? For example
7.1) Develop a separate Automation solution for each Application layer (UI, High level backend,
Database Layer) and run (use) each layer separately?
7.2) Integrate all Automation layers via continuous integration?
7.3) Use or develop your own one entry point solution for all Automation layers and use it as a part
of every-day development flow?
Notes:
7.1) This is the first phase of Automation and it is the most effort intensive/time consuming phase.
7.2) This is the next phase of Automation solution’s maturation. It introduces and additional set of
effort/costs:
o Special tools
o Additional hardware (CI servers)
o Administration effort/labor (CI servers)
o possibly Automation solution performance improvement
7.3) This is the highest level of Automation solution maturity, however, because it builts on the
prior phases( 7.1 and 7.)2, it will cost much less than the first phase. If one can accomplish
this level of maturity the automation solution will change the development process
dramatically, significantly increasing development performance and decreasing new
developers training time.
8) Can your database be used directly in the Automation solution to validate results for UI & database?
9) Do you have confirguration management practices and presense of development / test / QA /
integration environments (for the full application stack including database) and recommendations
around where the automation should be done. Do you have or need a test database for
Automation?
10) What are your performance requirements for Automation (time, hardware)? Do you want to run
Automation remotely? How frequently do you want to run All Automation tests?
11) Do you have any requirements/constraints for Automation tools (open source, vendor supported,
etc.)?

3. Coherent Solutions, Inc. QA Automation Questionnaire
Coherent Solutions, Inc. Confidential Page 3 of 3
Dictionary
 UI testing (frontend) - access, identify, and manipulate UI elements in the same way as end user
(the highest level of Automation).
 Code driven/ Service / API testing (backend)
o Web Servicies (backend) testing
o Services /daemons (backend) testing
o Shared libraries [dlls or shared objects] (backend) testing
- the public (usually) interfaces to classes, modules or libraries (i.e. code internals) are tested
with a variety of input arguments to validate that the results that are returned are correct.
 Smoke testing (frontend or backend) - is any type of “quick and dirty” software testing that
proves that the software will not crash outright (i.e. the device doesn’t start smoking when you
simply plug it it). Passing smoke testing means the system is ready for more robust testing.
Smoke tests are first tests made after assembly or repairs to a system, to provide some
assurance that the system under test will not catastrophically fail.
 Regression testing (frontend or backend) - is any type of software testing that seeks to uncover
software bugs, or regressions, in the unchanged parts of a system after changes,
(enhancements, patches, configuration changes, etc.), have been made to them or other parts
of the system.
 Functional testing (frontend or backend) - is a QA black box testing process that bases its test
cases on the specifications of the software component under test.
 CI (Continuous Integration) - the practice of merging all developer working copies of code with
a shared mainline of code several times a day. CI was originally intended to be used in
combination with automated tests (first strictly with unit tests, but best practice has evolved to
ideally include the execution of all tests against all environments, at least periodically). There are
a set of industry standard tools that support CI and we use some flavor of CI (often without
corresponding automation) on almost every development project we do.
Principles of CI
o Maintain a code repository
o Automate the build
o Make the build self-testing
o Everyone commits to the baseline every day
o Every commit (to baseline) should be built
o Keep the build fast
o Test in a clone of the production environment
o Make it easy to get the latest deliverables
o Everyone can see the results of the latest build
o Automate deployment