Monday, November 01, 2010

Acceptance Test Driven Development ATDD with Selenium

The User Interface defines how the application works. Automated testing from the user interface is therefore a must-do. And since Front End Engineers are in charge of developing the user interface the best investment you can do for Quality Assurance ( QA ) is to have Front End Engineers responsible for testing the application.

No need to mention how important is to have automated tests as part of any SDLC. Unfortunately for many projects it is prohibited to do any kind of Test Driven Development (TDD). Whatever the reason is (and yes, there are reasons that are sometimes beyond the IT team control) still Tests are necessary.

The tests you cannot forget about are those resulting from existing bugs (Test Driven Bug Fix or TDBF). When you correct something you should write a test that asserts that part of the application works as expected. The reason is that you want to be sure the very same problem does not hit you back. It is embarrassing to hear from Business side "This problem is happening again", period.

So you might not be able to afford TDD but at least you should be able to provide Development with Automated Tests (Let us call it DAT as these days we are supposed to remember abbreviations for almost anything :-)

While U-TDD (Unit TDD) is in reality invisible for the stake holders A-TDD (Acceptance TDD) is not.

User Stories have proven to be an effective mechanism to allow both, business and developers talk the same language. Plain English constructs "Given, When, Then" are a good starting point to build automated acceptance tests. There are multiple testing frameworks supporting code assertions for user stories. Selenium is one that I think is leading the race.

Your Test team (also called QA) can use any language regardless of the language being used for your real website. The reality is that small and medium size companies commonly cannot afford paying for dedicated people for this matter (Even though this is the ideal way to go either for Agile or Waterfall approaches). But as an experienced software architect you know that separation of concerns is the most important principle of successful enterprise software development so even if you decide to use the same language and keep it maintained by the same team of programmers you should:

1. Convince stake holders they must agree on User Stories.
2. Put the acceptance tests that assert the user stories in an acceptance tests project (Do not use your application project to host acceptance tests).
3. Externalize any configurations as URLs and credentials.
4. Schedule (at least daily) automated acceptance tests. Continuum Integration is a must-have. I have built it myself in the past but do not waste your time there are really good tools in the market for this.

The development team must be focused on what the customer really wants. Your customer (stake holders) must be in control to evaluate if the software you are building meets the requirements. There is then, as I have written before, a need for business people to write effective user stories.

Here is how the User Story 1 should be written by our stake holders. Note that we could divide this User Story in more than just one but for the purpose of showing the use of differenct methods in the final Java class I prefer to include all related scenarios:
User Story 1:
------------
1.1 
Given a user is not logged in 
When any page is hit
Then the user should be asked to authenticate

1.2 
Given a user with admin role is logged in 
When an admin page is hit
Then the user should see the page

1.3 
Given a user with just regular user role is logged in 
When an admin page is hit
Then the user should see an error page

In the Java world I prefer Maven as building tool. Below you can find what a POM file will look like to use Selenium 2 for Acceptance tests.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.nestorurquiza</groupId>
    <artifactId>acceptance-tests</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>acceptance-tests</name>
    <packaging>jar</packaging>
    <description>Acceptance tests for my web application</description>
    <properties>
        <maven.compiler.source>1.6</maven.compiler.source>
        <maven.compiler.target>1.6</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>
    
    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.8.1</version>
            <scope>test</scope>
        </dependency>
        <dependency>
           <groupId>net.sourceforge.htmlunit</groupId>
           <artifactId>htmlunit</artifactId>
           <version>2.8</version>
           <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>selenium-java</groupId>
            <artifactId>selenium-java</artifactId>
            <version>2.0a6</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
    <build>
        <defaultGoal>install</defaultGoal>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>${maven.compiler.source}</source>
                    <target>${maven.compiler.target}</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-site-plugin</artifactId>
                <configuration>
                    <source>${maven.compiler.source}</source>
                    <target>${maven.compiler.target}</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <configuration>
                    <source>${maven.compiler.source}</source>
                    <target>${maven.compiler.target}</target>
                    <archive>
                        <addMavenDescriptor>false</addMavenDescriptor>
                    </archive>
                </configuration>
            </plugin>
            <plugin>
                <artifactId>maven-release-plugin</artifactId>
                <version>2.0-beta-7</version>
                <configuration>
                    <tagBase>
                        http://my.svn.repo/acceptance-tests/tags/
                    </tagBase>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-eclipse-plugin</artifactId>
                <version>2.7</version>
                <configuration>
                    <ajdtVersion>2.0</ajdtVersion>
                    <wtpversion>2.0</wtpversion>
                </configuration>
            </plugin>
        </plugins>
    </build>
    <distributionManagement>
        <repository>
            <id>central</id>
            <name>libs-releases-local</name>
            <url>
                http://my.artifacts.repo/libs-releases-local
            </url>
            <uniqueVersion>false</uniqueVersion>
        </repository>
    </distributionManagement>
    <scm>
        <connection>scm:svn:http://my.svn.repo/acceptance-tests/tags/trunk/</connection>
    </scm>
</project>

Below is an example of a properties file to hold the target application URL, the WebDriver to use (HTMLUnit driver in this case) and user names and passwords to test:
#acceptanceEnvironment.propeties
baseUrl=http://localhost:8080
webDriverClass=org.openqa.selenium.htmlunit.HtmlUnitDriver
adminUserName=admin@nestorurquiza.com
adminUserPassword=test
regularUserName=user@nestorurquiza.com
regularUserPassword=test

Here is a class you can use to get the properties from the file above in you acceptance tests:
package com.nestorurquiza.acceptance;

import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.util.Properties;

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.htmlunit.HtmlUnitDriver;

/**
 * A VM argument providing the properties path parameter is needed.
 * for example: -DacceptanceEnvironment.properties=/Users/nestor/projects/config/acceptanceEnvironment.properties
 * 
 * Here is an example of such a property file
 * 
* #acceptanceEnvironment.propeties
 * baseUrl=http://localhost:8080
 * webDriverClass=org.openqa.selenium.htmlunit.HtmlUnitDriver
 * adminUserName=admin@nestorurquiza.com
 * adminUserPassword=test
 * regularUserName=user@nestorurquiza.com
 * regularUserPassword=test
 * 
* */ public class EnvironmentProperties { private static final String ACCEPTANCE_ENVIRONMENT_PROPERTIES = "acceptanceEnvironment.properties"; private static final EnvironmentProperties INSTANCE = new EnvironmentProperties(); private static Properties properties = null; private EnvironmentProperties() { } public static Properties getInstance() throws FileNotFoundException { if(properties == null) { properties = new Properties(); //Use -DacceptanceEnvironment.properties=/Users/nestor/projects/config/acceptanceEnvironment.properties String acceptanceEnvironmentPropertiesPath = System.getProperty(ACCEPTANCE_ENVIRONMENT_PROPERTIES); if(acceptanceEnvironmentPropertiesPath != null && acceptanceEnvironmentPropertiesPath.trim().length() > 0){ InputStream in = new FileInputStream(acceptanceEnvironmentPropertiesPath); if(in != null) { try { properties.load(in); in.close(); } catch (IOException e) { e.printStackTrace(); } } } } return properties; } public static WebDriver getWebDriver() throws FileNotFoundException { //The simplest not environment specific Driver WebDriver webDriver = new HtmlUnitDriver(); Class c; try { c = Class.forName(EnvironmentProperties.getInstance().getProperty("webDriverClass")); webDriver = (WebDriver) c.newInstance(); } catch (ClassNotFoundException e) { // TODO Auto-generated catch block e.printStackTrace(); } catch (InstantiationException e) { // TODO Auto-generated catch block e.printStackTrace(); } catch (IllegalAccessException e) { // TODO Auto-generated catch block e.printStackTrace(); } return webDriver; } public static String getBaseUrl() throws FileNotFoundException { return EnvironmentProperties.getInstance().getProperty("baseUrl"); } public static String getAdminUserName() throws FileNotFoundException { return EnvironmentProperties.getInstance().getProperty("adminUserName"); } public static String getAdminUserPassword() throws FileNotFoundException { return EnvironmentProperties.getInstance().getProperty("adminUserPassword"); } public static String getRegularUserName() throws FileNotFoundException { return EnvironmentProperties.getInstance().getProperty("regularUserName"); } public static String getRegularUserPassword() throws FileNotFoundException { return EnvironmentProperties.getInstance().getProperty("regularUserPassword"); } }

Finally here is a sample acceptance test that makes several assertions, all of them included in our User Story 1. Note that the test below asserts a site protected with Spring Security:
package com.nestorurquiza.acceptance;

import static org.hamcrest.CoreMatchers.*;
import static org.junit.Assert.assertThat;

import java.io.FileNotFoundException;

import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;


/**
 * Verifying behavior of UserStory1.
 */
public class UserStory1Test {

    private WebDriver driver = null;
    
    @Before
    public void aTearUpMethodForEachTest() throws FileNotFoundException {
        driver = EnvironmentProperties.getWebDriver();
    }

    @After
    public void aTearDownMethodForEachTest() {
        driver.close();
    }

    @Test
    public void givenNotLoggedInWhenAnyPageIsHitThenShouldBeAskedToAuthenticate() throws FileNotFoundException {

        String url = EnvironmentProperties.getBaseUrl() + "/client/list";
        driver.get(url);
        assertThat(driver.findElement(By.name("loginForm")).getAttribute("action"), is("j_spring_security_check"));
    }
    
    @Test
    public void givenLoggedInAndAdminUserRoleWhenAnAdminPageIsHitThenShouldSeeThePage() throws FileNotFoundException {
        String url = EnvironmentProperties.getBaseUrl() + "/";
        driver.get(url);
        WebElement loginForm = driver.findElement(By.name("loginForm"));
        WebElement userName = loginForm.findElement(By.name("j_username"));
        userName.sendKeys(EnvironmentProperties.getAdminUserName());
        WebElement password = loginForm.findElement(By.name("j_password"));
        password.sendKeys(EnvironmentProperties.getAdminUserPassword());
        loginForm.submit();
        url = EnvironmentProperties.getBaseUrl() + "/client/list";
        driver.get(url);
        
        //Not supported by HtmlUnit
        //assertThat(driver.findElement(By.cssSelector(".form-navigation")), is(notNullValue()));
        
        assertThat(driver.findElement(By.className("form-navigation")), is(notNullValue()));
    }
    
    @Test
    public void givenLoggedInAndRegularUserRoleWhenAnAdminPageIsHitThenShouldSeeAnErrorPage() throws FileNotFoundException {
        String url = EnvironmentProperties.getBaseUrl() + "/";
        driver.get(url);
        WebElement loginForm = driver.findElement(By.name("loginForm"));
        WebElement userName = loginForm.findElement(By.name("j_username"));
        userName.sendKeys(EnvironmentProperties.getRegularUserName());
        WebElement password = loginForm.findElement(By.name("j_password"));
        password.sendKeys(EnvironmentProperties.getRegularUserPassword());
        loginForm.submit();
        url = EnvironmentProperties.getBaseUrl() + "/client/list";
        driver.get(url);
        
        //Not supported by HtmlUnit
        //assertThat(driver.findElement(By.cssSelector(".form-navigation")), is(nullValue()));
        
        assertThat(driver.findElement(By.className("errors")), is(notNullValue()));
    } 
}

Below is how to run our Acceptance Tests from Maven:
mvn test -DargLine="-DacceptanceEnvironment.properties=/Users/nestor/projects/config/acceptanceEnvironment.properties"

To run the Acceptance Tests from an IDE you will need to configure a VM argument like
-DacceptanceEnvironment.properties=/Users/nestor/projects/config/acceptanceEnvironment.properties

You can of course change the driver so you use different browsers. As the available drivers[1] support javascript from a front end perspective you can come up with very complete test assertions that later can be scheduled so you keep checking the website functionality in a regular basis.

I have to say though that to test Front End development you better wait for the preference of your UI Engineer. Java is most likely not his preferred language.

While I here provide guidance to ATDD from a Java perspective I still think that given the fact that ATDD is closer to front end development or User experience (UX) it should be better to study other possibilities in the case you have the luxury of having a UX engineer on board.

Bibliography

1. http://seleniumhq.org/docs/09_webdriver.html
2. http://thinkinginsoftware.blogspot.com/2010/08/writing-agile-specifications.html

14 comments:

Jazzie Casas said...

Java and to some degree .Net are the main choices because they have been consistently pegged as the “safe” choice to go with for mid-level project managers in the corporate world. No one was ever fired for choosing Java or Microsoft.

However, there are many large distributed applications these days that run primarily with technologies like Python, PHP, et al. Even companies like Google and Yahoo are heavily invested in these technologies. Java may be the main choice for enterprise development now, but it’s days are numbered as the only stalwart option to go with.

Let’s face it, many of these so called “enterprise applications” could easily have been written much faster and with less overhead using technologies like Python, PHP, et al.


uml training

Nestor Urquiza said...

I think any high level language (PHP, Python, Java, .NET) are a bad choice for acceptance tests when you want them to be managed by the UI/UX engineer.

I do believe that UAT is a better fit for a front end developer than a back end developer (In the case there is no QA engineer).

As selenium tests are stored in plain html the front end developer can either improve them through the firefox plugin or directly in code. The tests can be run from Selenium Remote Control (command line) for simple automation.

Selenium plus tools like dynaTrace can speed up the front end testing experience. Read this to see how actually some performance analysis could be achieved as well.

None of the above methods require a high level programming language. I really can't wait to have a front end developer on board (just waiting for the budget ;-)

nishit said...

Hi Nestor,

it is good to use selenium but using TestNG along with selenium gives you additional advantage in terms of reporting...

Nestor Urquiza said...

Thanks for the suggestion Nishit. I do not want to start just another TestNG vs. JUnit discussion here but I am sure there are features that you will need at some point which ultimately will decide a switch between one framework to the other.

So far what I need is cover by just using JUnit. As I apply YAGNI and KISS principles in each of my decisions I prefer to push that moment to the future.

As I stated I do think Selenium tests are a better fit for Back end agnostic people so I would prefer to actually push that concern to a front end developer that in any case should be in charge of browser compatibility for example.

milez said...

Hi,

it looks like there is no support for php and selenium 2 atm, right?

i am thinking about bridging the gap between product owner test developer:
the product owner wants to specify an acceptance test in a more natural language, but the test developer needs a formal specification.

i also think that test should be written by frontend engineers, who usually dont code in java.

jan

Nestor Urquiza said...

Actually there is PHP support http://seleniumhq.org/about/platforms.html#programming-languages

But if you ask me a front end developer is better off even PHP. There is a Selenium RC client for NodeJS I am planning to use when I get my UI/UX Engineer: https://github.com/LearnBoost/soda

heway10 said...

I'm tasked with setting up the infrastructure for a project that will introduce my company to (ahem) modern development practices. Besides demonstrating Bootstrap (minds blown), showing my cowrokers that jQuery can do more than hide() and show(), identifying a (PHP) framework, deciding between SVN and Git (I'm leaning towards Git), I'm researching best practices for integrating testing into our development activities. So, here I am.

You make this assertion:
"Do not use your application project to host acceptance tests".

Could you elaborate on that? By "project", do you mean SVN project? Or are you saying "never store your code in a way that could result in your tests being released into production"? Or...something else?
Thanks!

heway10 said...

I'm tasked with setting up the infrastructure for a project that will introduce my company to (ahem) modern development practices. Besides demonstrating Bootstrap (minds blown), showing my cowrokers that jQuery can do more than hide() and show(), identifying a (PHP) framework, deciding between SVN and Git (I'm leaning towards Git), I'm researching best practices for integrating testing into our development activities. So, here I am.

You make this assertion:
"Do not use your application project to host acceptance tests".

Could you elaborate on that? By "project", do you mean SVN project? Or are you saying "never store your code in a way that could result in your tests being released into production"? Or...something else?
Thanks!

Nestor Urquiza said...

@heway10 That is exactly what I meant. Terminology is always overlapping so to make it clear:
1. Code needed to run the application should not be hosted in the same project as UAT.
2. In fact code needed to run an application is probably hosted in several different projects just because the application is just too big to manage with just one project and/or because you have different teams taking care of different areas (Separation Of Concerns)

Divide and Conquer, you know.

Best,
-Nestor

Jose said...

I agree with the post Nestor. My favorite point of all was actually in one of your comments. In my opinion if you have a dedicated frontend engineer writing HTML/CSS/JS all day, you should definitely use something like https://github.com/LearnBoost/soda to do testing, it will be much easier for the developer to jump from Javascript to Javascript to write tests, than Javascript to Java.

Now if using a frontend framework, definitely consider Angular, one of its many advantages is that they provide an E2E (End to End) testing platform out of the box, in which all you have to write is Javascript. Check out the DSL: http://docs.angularjs.org/guide/dev_guide.e2e-testing - you'll see its similar to the User Stories format.

Obviously with the above you wouldn't be able to fully test the user stories above (non-logged in user getting the login form, etc) but hey not sure that's something the frontend should worry about (whether sessions are getting authenticated correctly) so once again separation of concerns is the best way.

And last one, a frontend engineer should never rely on automated tests for their testing. One of the biggest reasons is that the code can assert everything is working correctly (because the DOM says so) but as we all know different browsers will often render different outputs, and I feel that as a frontend developer it is your job to test exactly what the customer is going to be doing according to the user story, that way when the alert you thought was coming up is halfway off the screen it can fixed before going into production.

Just my .2 cents, great post!

Nestor Urquiza said...

@Jose thank you for your input. Automated tests for sure cannot replace individuals 100%. They can do it for a high percentage though, so high that I think it is totally worth it. I still believe that UAT is front end concern though and in the particular scenario of user not being logged that is simply achievable if you run the test without providing authentication. Angular Rocks!

Nestor Urquiza said...

@Jose, in addition Selenium (or any wrapper around it like soda) allows to spawn browsers where the tests are run. So going that route it will be asserting on the FE behalf if a given page is actually rendering in Firefox as expected, just to give an example.

codenuance said...

Great post!
it makes me think about how ATDD relates to BDD ( behavior driven development.)

Nestor Urquiza said...

@codenuance (Should I use your real name? :) I believe BDD (I have used JBehave in the past) is great and definitely can be used to provide ATDD. The important think is that if you go with JBehave for example you should provide a user interaction centric approach. I removed the U on purpose from User Acceptance Test Driven Development because I thought it was redundant (Who accepts the software is not the user of such software?) but that is what I mean by developing well tested software that relies uniquely in how it really works. How you call it ultimately does not matter, someone might say they are end to end tests, others will stay away from a particular name stating they are just large tests ( see They are simply large tests ( http://googletesting.blogspot.com/2010/12/test-sizes.html ) if we are talking about test size ) but what ultimately matters is the importance the user centric acceptance test has. Perception is definitely reality. Cheers! - Nestor

Followers