WordPress testing for hosting companies

Last Revised: January 26, 2022

Hosting companies can host hundreds of websites with WordPress, and that’s why it’s important that their configuration is as compatible as possible with the software.

To verify this compatibility, the WordPress Hosting Community team makes available a series of PHPUnit tests with which to check the operation of WordPress in any environment.

Requirements

The objective of these tests is to check and validate that the next WordPress updates work correctly in your environment, so the ideal is to have hosting as close as possible to the one you offer your customers.

This means that we will need, at least:

A database (its contents will be created and deleted). You can use MySQL, MariaDB or any supported system.
A folder to host the software (not public).
A temporary folder where content will be generated and deleted.

In addition to this, extra, the following elements will be needed:

npm (to use Node.js)
Composer 2.x
Git

Installing on Ubuntu (example)

npm on Ubuntu

apt-get -y install npm

Composer 2.x on Ubuntu

cd
curl -sS https://getcomposer.org/installer -o composer-setup.php
php composer-setup.php --install-dir=/usr/local/bin --filename=composer

Git on Ubuntu

apt-get -y install git

Installing phpunit-test-runner

The phpunit-test-runner is a small piece of PHPUnit tests specifically designed for hosting companies. There is a whole documentation about this tool. Also, if you want, you can make your test results appear on the WordPress Host Test Results page.

The tool can be run manually or through an automated system. To see the operation and purpose of this document we will do the tests manually, and at the end we will add an automation system.

Runner Installation

The first thing we will do is download and synchronize the tool.

cd /home/wordpressbot/
git clone https://github.com/WordPress/phpunit-test-runner.git wordpress-test
cd /home/wordpressbot/wordpress-test/

The next step will be to configure the environment. To do this we will first make a copy of the sample file and then configure it.

cp .env.default .env
vim .env

The content (in summary form) can be something like this:

export WPT_PREPARE_DIR=/tmp/wp-test-runner
export WPT_TEST_DIR=/tmp/wp-test-runner
export WPT_REPORT_API_KEY=
export WPT_REPORT_URL=
export WPT_DB_NAME=wordpressdatabase
export WPT_DB_USER=wordpressusername
export WPT_DB_PASSWORD=wordpresspassword
export WPT_DB_HOST=localhost
export WPT_TABLE_PREFIX=${WPT_TABLE_PREFIX-wptests_}
export WPT_PHP_EXECUTABLE=${WPT_PHP_EXECUTABLE-php}
export WPT_PHPUNIT_CMD=
export WPT_RM_TEST_DIR_CMD=
export WPT_SSH_CONNECT=
export WPT_SSH_OPTIONS=
export WPT_SSH_PRIVATE_KEY_BASE64=
export WPT_DEBUG=

We will configure the folder where the download tests of the WordPress software and the database accesses will be done to be able to prepare the tests.

For this example, we will use a system test folder such /tmp/wp-test-runneras . You can use a temporary folder or an existing folder for that same user, for example. We will configure this complete route in:

export WPT_PREPARE_DIR=/tmp/wp-test-runner
export WPT_TEST_DIR=/tmp/wp-test-runner

The next thing will be to include the configuration of the database that we have previously created. For example, we can create it with this command:

CREATE DATABASE wordpressdatabase CHARACTER SET = utf8mb4 COLLATE = utf8mb4_general_ci;
GRANT ALL ON wordpressdatabase.* TO 'wordpressusername'@'localhost' IDENTIFIED BY 'wordpresspassword';
FLUSH PRIVILEGES;

This database should not be used for anything other than the use of these tests.

Subsequently, we will configure the data in the configuration file:

export WPT_DB_NAME=wordpressdatabase
export WPT_DB_USER=wordpressusername
export WPT_DB_PASSWORD=wordpresspassword
export WPT_DB_HOST=localhost

This should be enough for the execution of the tests.

Now that we have the configuration ready, we can do the first test to validate that everything works correctly.

Testing the test

Before doing the tests we will update everything necessary to the latest version and activate the configuration of the environment.

cd /home/wordpressbot/wordpress-test/
git pull
npm update
source .env

We will enter the folder where we have the testing software and update it by downloading the latest version. We will also update the npm in case there are updates, and finally, we will load the configuration.

From here, we only have to run the 4 files that do the set of tests.

The first of these is the system that verifies that the environment is ready to go.

php prepare.php

The second of them is the one that performs the tests itself.

php test.php

Once the tests are finished, the report can be generated. If you have set up a username and password it will be sent to WordPress.org.

php report.php

This system generates two files in which is the information of what will be sent as a response to the tests that are being analyzed. You can see the contents of the files easily.

cat /tmp/wp-test-runner/env.json
cat /tmp/wp-test-runner/junit.xml

The first of them includes the information of all the necessary elements for WordPress, such as PHP and its extensions and versions, and the database, with its version.

The second of the files is a summary of all the tests launched with their results and that, if they are sent to WordPress.org, those that have failed will be shown on the results page, in order to work on them, whether they are elements to be solved by the hosting, as if they are errors that affect development and the Core.

And, finally, once you have the system finished, we only have to clean the files and the database to be able to perform a next test at another time.

php cleanup.php

Creating your bot for WordPress.org

If you want the test results to appear on the WordPress.org page , you can create a user for it.

The first thing will be to create a user in WordPress.org. If your company is called, for example, Somos Hosting, S.L., you can call your user something like somoshostingbot. Please note that the associated email account should be checked frequently, as emails will arrive regarding the possible operation of the tests.

Create an issue on the test page asking for the bot to be included on the results page as “Test Reporter request”, indicating the email account you have used with that user.

When the user has been created in the system you will be given access to a password so that you can configure that the system sends the information automatically. You will have to enter Users -> Your Profile and there generate the application password. Later you can modify the constant of the environment with something similar to export WPT_REPORT_API_KEY='somoshostingbot:ABCD 1234 abcd 4567 EFGH efgh'.

Automating the bot

If you’re sending the information to WordPress.org you might want to automate the system to send reports when there’s news.

In this case we will use a series of scripts and systems for this automation, with the aim of reducing the number of executions, and only launch them when necessary.

Surely there are more optimized ways to perform this system, but this is the one I have tried and it works.

Have a copy of WordPress

cd /home/wordpressbot/
git clone https://github.com/WordPress/wordpress-develop.git wordpress-develop
cd /home/wordpressbot/wordpress-develop/
git pull
git log -n1 --format=format:"%H" > /home/wordpressbot/wordpress-develop.log

Inside our user’s folder we will make a copy of the WordPress repository that is used for development (and we will do a test to validate that it is downloaded and updated correctly).

Later (the last line) will generate a file in which the identifier of the last commit made will be included.

Control files

We will create two control files.

echo '' > /home/wordpressbot/wordpress-tested.log
echo 0 > /home/wordpressbot/wordpress-running.log

The first line creates a file without content in which we will save the version of the last test that has been launched.

The second line will save the binary state (0 or 1) to know if a test is running or not, as a traffic light.

Installing the runner

This is not necessary if you have done it before.

cd /home/wordpressbot/
git clone https://github.com/WordPress/phpunit-test-runner.git wordpress-test
cd /home/wordpressbot/wordpress-test/
git pull

We download the PHPUnit test runner, and validate that the updates are done well.

We configure the environment. To do this we will first make a copy of the sample file and then configure it.

cp .env.default .env
vim .env

The content (in summary form) can be something like this:

export WPT_PREPARE_DIR=/home/wordpressbot/wp-test-runner
export WPT_TEST_DIR=/home/wordpressbot/wp-test-runner
export WPT_REPORT_API_KEY=somoshostingbot:ABCD1234abcd4567EFGHefgh
export WPT_REPORT_URL=
export WPT_DB_NAME=wordpressdatabase
export WPT_DB_USER=wordpressusername
export WPT_DB_PASSWORD=wordpresspassword
export WPT_DB_HOST=localhost
export WPT_TABLE_PREFIX=${WPT_TABLE_PREFIX-wptests_}
export WPT_PHP_EXECUTABLE=${WPT_PHP_EXECUTABLE-php}
export WPT_PHPUNIT_CMD=
export WPT_RM_TEST_DIR_CMD=
export WPT_SSH_CONNECT=
export WPT_SSH_OPTIONS=
export WPT_SSH_PRIVATE_KEY_BASE64=
export WPT_DEBUG=

Unlike in the previous case, the configuration of the test path is also on the same user. In this way, when the tests are launched there will be 3 folders:

/home/wordpressbot/wordpress-develop/: A copy of the WordPress development Git.
/home/wordpressbot/wordpress-test/: A copy of the Git of the test tool.
/home/wordpressbot/wp-test-runner/: WordPress and files where the tests will be carried out.

The script that runs everything

cat > /home/wordpressbot/wordpressbot.sh << EOF
#!/bin/bash
echo ''
echo '[*] Launching the WordPress testing tool for hosting.'
echo ''
echo '[+] Detecting changes.'
echo ''
cd /home/wordpressbot/wordpress-develop/
git pull
git log -n1 --format=format:"%H" > /home/wordpressbot/wordpress-develop.log
estado=`cat /home/wordpressbot/wordpress-running.log`  
if [ "$estado" -eq 0 ]; then
  echo ''
  echo '[+] Preparing test.'
  echo ''
  cd /home/wordpressbot/wpsabot-test/
  git pull
  npm update
  file1="/home/wordpressbot/wordpress-develop.log"
  file2="/home/wordpressbot/wordpress-tested.log"
  if cmp -s "$file1" "$file2"; then
    echo ''
    echo 'Nothing to do. The last test is the same.'
    echo ''
  else
    echo ''
    echo '[+] Starting test.'
    echo ''
    echo 1 > /home/wordpressbot/wordpress-running.log
    cp /home/wordpressbot/wordpress-develop.log /home/wordpressbot/wordpress-tested.log
    cd /home/wordpressbot/wpsabot-test/
    source .env
    php prepare.php
    php test.php
    php report.php
    cat /home/wordpressbot/wp-test-runner/env.json
    php cleanup.php
    echo 0 > /home/wordpressbot/wordpress-running.log
  fi
else
  echo ''
  echo '[+] Test already in progress.'
  echo ''
fi
echo ''
echo '[*] The End.'
echo ''
EOF

First we will launch the file where we are going to save the script.

cat > /home/wordpressbot/wordpressbot.sh << EOF

We will download the latest wordpress development update. We will save, in the log file, the identifier of the last commit that has been executed.

cd /home/wordpressbot/wordpress-develop/
git pull
git log -n1 --format=format:"%H" > /home/wordpressbot/wordpress-develop.log

We’ll see if a test is running right now or not. In case a test is currently running, we won’t do anything (and wait for it to finish).

estado=`cat /home/wordpressbot/wordpress-running.log`  
if [ "$estado" -eq 0 ]; then
...
else
  echo ''
  echo '[+] Test already in progress.'
  echo ''
fi

If there is no test running, we will update the latest version of the test software.

  echo ''
  echo '[+] Preparing test.'
  echo ''
  cd /home/wordpressbot/wpsabot-test/
  git pull
  npm update

We will check if there are any updates in the development software with respect to the last test.

  file1="/home/wordpressbot/wordpress-develop.log"
  file2="/home/wordpressbot/wordpress-tested.log"
  if cmp -s "$file1" "$file2"; then
  ...
  fi

If the last commit of the development software is the same one on which we already did a test, we will not release it again.

    echo ''
    echo 'Nothing to do. The last test is the same.'
    echo ''

If the test is new, then we will have to do the test. The first thing will be to set the configuration files. We will tell the system that we are running a test, and we will store the identifier of the test we are going to perform (commit).

    echo ''
    echo '[+] Starting test.'
    echo ''
    echo 1 > /home/wordpressbot/wordpress-running.log
    cp /home/wordpressbot/wordpress-develop.log /home/wordpressbot/wordpress-tested.log

We will do the test.

    cd /home/wordpressbot/wpsabot-test/
    source .env
    php prepare.php
    php test.php
    php report.php
    cat /home/wordpressbot/wp-test-runner/env.json
    php cleanup.php

And finally, we will tell the system that we have finished the test.

    echo 0 > /home/wordpressbot/wordpress-running.log

And the test will be over.

echo ''
echo '[*] The End.'
echo ''

Now that we have the script, the first thing will be to launch it manually at least once to check that it works correctly.

bash /home/wordpressbot/wordpressbot.sh

Scheduling the tests (cron)

Now that we have a script that does it all, we can program a cron that, every minute, runs the entire system.

As the script has a traffic light system so as not to run more than once in parallel, it can be launched very frequently.

* * * * * bash /webs/wpsabot/wpsabot.sh >/dev/null 2>&1

Full tests

If you want to do a full test on a setup from scratch, you have a few options:

Try runner on WordPress with Ubuntu 18 and PHP 7

Improving configuration

Do not forget that the objective of this tool is the verification that our environment and infrastructure is the ooptima for WordPress to work, so, following the example of before, we could make several improvements such as the installation of the extension of bcmath, , gd, libsodium, mod_xml mcryptand imagick or utilities such as ghostscript and imagemagick.

The goal? That there are no errors and have the green light for the perfect configuration.

WordPress Community Hosting Team

If you are a company that is somehow in the day to day of offering services related to WordPress I recommend that you remain attentive to the publications of the WordPress Community Hosting team, in addition to closely following the conversations of the Slack channel #hosting-community (see the meeting schedules of the team), where you can be in contact with those responsible for other hosting companies.

About this document

This document is regulated by the EUPL v1.2 license, published in WP SysAdmin and created by Javier Casares. Please, if you use this content in your website, your presentation or any material you distribute, remember to mention this site or its author, and having to put the material you create under EUPL license.