Advanced techniques for testing and updating WordPress core and plugins to avoid regressions Updating WordPress core and plugins is an important and often recurring maintenance task, that many often neglect due to the inherent risk of regressions and potential downtime. At Seravo.com we update hundreds of enterprise grade WordPress sites in a production-proof way using automated testing with RSpec and Phantom.js. In this talk I will show you how we do it and what are the open source tools anybody else can use as well to test their own sites before and after updates. Presentation delivered at WordCamp Copenhagen 2017 https://2017.copenhagen.wordcamp.org/session/advanced-techniques-for-testing-and-updating-wordpress-core-and-plugins-to-avoid-regressions/

1. AVOIDING
REGRESSIONS
Advanced techniques for testing and
updating
WordPress core
and plugins
WordCamp Copenhagen 2017
Otto Kekäläinen
Seravo.com
@ottokekalainen

2. ● Linux and open source advocate
● Contributed to WordPress Core,
translations, Linux, Docker,
Nginx, Redis, MariaDB…
● CEO, sysadmin and developer at
Seravo.com – WordPress
hosting and upkeep
Otto Kekäläinen

3. Enterprise grade
hosting and upkeep
for WordPress

4. WHY UPDATE?
1. Security bugs
2. Other bugs
3. New features

5. WHY NOT TO UPDATE?
1. New security bugs
2. New other bugs
3. Old features

6. Example case: Mossack Fonseca aka Panama papers
● The site www.mossfon.com was running WordPress
● Unauthorized access of WP lead to unauthorized access of MS Exchange
email server on internal network and other sites at *.mossfon.com
● The intruders most likely came through an old and insecure version of the
Revolution Slider plugin.
○ Well known vulnerability, WordPress.org even has a patch as a separate plugin
(https://wordpress.org/plugins/patch-for-revolution-slider/) as Revolution Slider itself is not
available at WordPress.org.

7. Example case: Mossack Fonseca aka Panama papers
● Case analysis at
https://www.wordfence.com/blog/2016/04/mossack-fonseca-breach-vulner
able-slider-revolution/

8. WP PLUGIN REVIEW GUIDELINES FOR
CAPITALISTS*
If the logo is red and
name contains revolution,
don’t install it on your system!
* a small dose of parody can’t hurt?

9. You must keep your
WordPress site secure.

10. THE PROBLEM:
WHY AREN’T
EVERYBODY
UPDATING THEIR
WORDPRESS AND
PLUGINS?

11. BECAUSE
OF THIS:

12. UPDATES IN WORDPRESS
● WordPress core minor version updates (4.7.4 -> 4.7.5): security
● WordPress major version updates (3.9 -> 4.0, 4.6 -> 4.7): features
● WordPress plugin updates can contain anything
● There is just one WordPress.org update channel
○ No separate security updates channel like in Linux distros
● Plugins and themes from other places than WordPress.org might
have automatic update channel
○ No guarantee: worst case scenario is that there are no update
notifications and you need to do everything about updates
manually

13. THE PROBLEM IS THE PLUGINS.

14. SOLUTION:
ROLL-BACK BAD UPDATES?

15. YOU HAVE NIGHTLY OFF-SITE
BACKUPS, RIGHT?

16. FILES VS. DATABASE
Updates install new files, and they might
upgrade the data format in the database to
become backwards incompatible.
Reverting by putting the old files in place might
not work because of the database contents!
cp -ra /data/backups/wordpress /wordpress
wp db import /data/backups/db/site.sql

17. ROLL-BACKS IN PRODUCTION
ARE BAD
1. Downtime between bad update and
roll-back
2. Lost database contents
(WooCommerce orders, anybody?)
3. If the site broke so badly that you
could not access WP-admin, was that a
bad or actually a good thing?

18. INTRODUCING SHADOW UPDATES
1. Make an identical copy of the
production site (same URLs etc)
that is not visible to the public
2. Update the shadow
3. Test the shadow
4. Only if tests pass, run the same
updates in production

19. REGRESSION TESTING WORDPRESS
Open source tools
● RSpec – test runner
● Capybara – navigate the site virtually (headlessly)
● PhantomJS – headless browser
● GraphicsMagic – visual comparison
Tests part of our project template:
https://github.com/Seravo/wordpress/tree/master/tests/rspec
Docs: https://seravo.com/docs/tests/integration-tests/

20. INTERGRATION TEST EXAMPLE 1/2
before do
visit WP.siteurl('/wp-login.php')
end
it "There's a login form" do
expect(page).to have_id "wp-submit"
end

21. INTERGRATION TEST EXAMPLE 2/2
if WP.user?
it "Logged in to WordPress Dashboard" do
within("#loginform") do
fill_in 'log', :with => WP.user.username
fill_in 'pwd', :with => WP.user.password
end
click_button 'wp-submit'
# Should obtain cookies and be able to visit /wp-admin
expect(page).to have_id "wpadminbar"
end
end

22. VISUAL REGRESSION TESTS
$ gm compare -highlight-style assign
-highlight-color purple -file diff.png *.png

23. VISUAL REGRESSION TESTS
$ gm compare -verbose -metric mse *.png
Image Difference (MeanSquaredError):
Normalized Absolute
============ ==========
Red: 0.0319159868 8.1
Green: 0.0251841368 6.4
Blue: 0.0278537225 7.1
Opacity: 0.0000000000 0.0
Total: 0.0212384615 5.4

24. Where do you draw the line
between acceptable changes
and failures/regressions?

25. AUTOMATING UPDATES:
90 % BY ROBOTS
10 % BY HUMANS

26. What could WordPress
plugin nd theme developers
do to avoid regressions?

27. TRAVIS-CI.ORG SIMPLE EXAMPLE
https://github.com/Seravo/seravo-plugin/blob/master/.travis.yml
sudo: false
language: php
php:
- 5.6
- 7.0
- nightly
script:
- find -name '*.php' -exec php -d
error_reporting=32767 -l {} ;

28. TRAVIS-CI.ORG IN ACTION

29. TRAVIS-CI CHECKING EVERY COMMIT

30. ..AND PULL REQUESTS!

31. NOTIFICATON EMAILS THAT
CAN’T GO UNNOTICED

32. TRAVIS-CI.ORG BIG EXAMPLE 1/3
https://github.com/Seravo/wordpress/blob/master/.travis.yml
[...]
env:
- WP_TEST_URL=http://localhost:12000 WP_TEST_USER=test WP_TEST_USER_PASS=test
DB_USER=root DB_PASSWORD='' DB_NAME=test
matrix:
allow_failures:
- php: nightly
before_install:
- rvm install 2.2.5
before_script:
# Install composer packages before trying to activate themes or plugins
- composer install
# Create database
- mysqladmin create $DB_NAME --user="$DB_USER" --password="$DB_PASSWORD"

33. TRAVIS-CI.ORG BIG EXAMPLE 2/3
https://github.com/Seravo/wordpress/blob/master/.travis.yml
before_script:
...
# Install router so that we don't need nginx/php-fpm
- curl -s
https://raw.githubusercontent.com/Seravo/wordpress-test-template/master/lib/router.ph
p > htdocs/router.php
# Start php server on background
- cd htdocs && php -S 0.0.0.0:12000 router.php &
# Install WordPress with wp-cli
- curl -O https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar
- php wp-cli.phar core install --url=$WP_TEST_URL --title='Test'
--admin_user=$WP_TEST_USER --admin_password=$WP_TEST_USER_PASS
--admin_email="$WP_TEST_USER@wordpress.dev" --path=htdocs/wordpress

34. TRAVIS-CI.ORG BIG EXAMPLE 3/3
https://github.com/Seravo/wordpress/blob/master/.travis.yml
# Activate all plugins
- php wp-cli.phar plugin activate --all --path=htdocs/wordpress
# test webserver
- curl -i http://localhost:12000
# Install packages for gulp
- npm install
# Test gulp and compile assets
- gulp
# Install gems for rspec tests
- gem install bundler
- bundle install --gemfile=tests/rspec/Gemfile
script:
- cd tests/rspec && bundle exec rspec *.rb

35. My META contributions tomorrow?
● Increased git use at WordPress.org
● Plugin CI and QA infra for WordPress.org
Any core devs around tomorrow?

36. TAK FOR
JERES
TID! SERAVO.COM
facebook.com/Seravocom
Twitter: @Seravo @ottokekalainen