Robot FrameWork: The Ultimate Guide to Running Your Tests

This blog post is dedicated to the Robot Framework — an open-source test automation framework for acceptance testing and acceptance test-driven development (ATDD). This blog post will explain how to create your project and Selenium tests in Robot and provide tips and best practices. In the end, I will show you how to run Robot in Taurus.

But first, to understand why Robot Framework is a good solution for that, we need to understand what acceptance testing is. Acceptance testing determines whether a system answers the acceptance criteria, defined by users needs and requirements. When we execute acceptance testing, we verify the whole system as a single unit on a high level, to understand if the system under test can be used by the end users.

Usually, test engineers act as system users by executing steps and scenarios that come from requirements and business processes, by forming a set of predefined keywords. This approach to testing, which is based on a set of keywords that can be re-used across all tests, is called keyword-driven testing. Robot Framework utilizes this approach in a very good fashion. It doesn't limit you to any approach or format with keywords: if you wish to use keywords on a high level, then it's ok. You wish to manipulate variables and do programming with keywords? Please, go ahead. You like Gherkin syntax and want to use it to describe your tests? Feel free to do it.

Additionally, Robot Framework has a rich ecosystem of internal and external libraries that consist of many useful keywords that you can re-use in your ecosystem. But don't worry, if you didn't find a keyword you need, you can develop a new library itself. You can develop tests with the Robot Framework on Java and Python. In the blog post, we will use Python since the framework itself is implemented on Python and there are more external libraries on Python.

For more information about Robot Framework and the ecosystem, see the official site. You can find plenty more documentation there, together with demo projects, a list of available test libraries and other tools, and so on.

Sounds like a great framework, doesn't it? Let's have a look at how to use it. You can follow the test source code usage of the Robot Selenium library here. You can find the test version with custom selenium keywords here.

First, we need to follow the requirements:

1. Install Python 2.7.14 or above. You can download it from here. There are two major versions of Python: 2.7.14 and 3.6.4. Code snippets in the blog post will be given for version 2.7.14. If there is any difference for version 3.6.4, a note will be made with appropriate changes to version 3.6.4. It's up to the reader to choose which version to install.

2. Install the python package manager (pip). It can be downloaded from its download page. All further installations in the blog post will make use of pip, so it's highly recommended to install it.

3. Download a development environment. The PyCharm Community edition will be used in the blog post. You can download it from the Pycharm website. You can use any IDE of your choice since code snippets are not IDE dependent.

Now, let's create our project.

1. Create the project in PyCharm IDE with File -> New Project.

2. Specify the location for the project (the last part of the path will be the project's name)

When developing a Python application, it's good practice to isolate its dependencies from others. With this, you can be sure that you are using the right version of the library in case there are multiple versions of it in your  PYTHON_PATH . (The PYTHON_PATH   variable is used in Python applications to find any dependency that is declared via import statement in python modules).

3. To do this, you need to install the virtual environment.

• Install the Virtual Environment tool with the command  pip install virtualenv  in the prompt.
• In the project root directory, create the environment with virtualenvvenv  in the prompt where venv  is the environment name.

4. If the environment is activated, your prompt will be prefixed with the environment's name as below:

5. If the environment has been activated successfully, install the Robot Framework with pip:

Once we are done with the prerequisites, let's choose an application that we are going to automate tests for. Check out a demo version of a Simple Travel Agency that allows you to search and book a flight.

We will create a simple test case that allows searching for any flight. This will be a UI test; hence, we need a tool that will help us manipulate with elements on a web page. Selenium WebDriver will suit this purpose as it's open-source, stable, and easy to use. Selenium WebDriver gives us as test developers a simple programming interface to interact with graphical elements on web pages. The only thing you need to do is to choose the appropriate WebDriver depending on the browser you would like to use.

In general, any test forms a test case. In the Robot Framework, test cases are created in test cases files that can have any extension from the following: .html, .xhtml, .htm, .tsv, .txt, .rst, .rest or .robot

Any test cases file forms a test suite file, regardless of the number of tests it has. Sometimes you need to group test suite files into higher-level suites. This can be achieved by creating directories. You can create a tree of directories with any depth by forming a logical structure. To do that, you should follow the application business logic or any other grouping criteria.

A test case is composed of keywords. We can use keywords created us (they are named as user keywords), or import keywords from Robot Framework libraries. It's up to a test developer to choose which one to use, but keep in mind the following recommendation: in a test case try to avoid any programming and low-level keywords by using high-level user keywords instead. It will make such a test case easy to read and easy to understand.

How should you apply this recommendation? Consider the following issue: when automating UI tests with Selenium and the Robot Framework, you need to choose between already developed Selenium keywords or your own high-level business keywords to interact with web elements. The choice depends on answers to the following questions:

1. Will other people, besides the test developers, implement new tests and keywords?
2. Is the business logic simple and do you not need to use object-related models?

If the answer to those two questions is yes, then feel free to use keywords that were already developed. But if you need to implement complex business logic and you can't do it without a business model, then encapsulate manipulation with web pages in internal classes and create keywords for business steps instead.

In the blog post, we will use the Selenium2Library from the Robot Framework as there will not be any complex business logic — any flow can be implemented with the available keywords.

Have a look at the test below:

To book a flight we need to perform the following steps:

1. Open a the Travel Agency Demo web page.
2. Choose any departure city from a drop-down list.
3. Choose any destination city from a drop-down list.
4. Click on "Find Flights" button.
5. Verify if there are flights found on Flights search result page.

To implement the test, the steps above need to be converted to keywords. If we tried to implement a test case without user defined keywords, our test would look like the following (consider that we are using SeleniumLibrary from the Robot Framework).

search_flights.robot:

*** Settings ***
Library  SeleniumLibrary

*** Test Cases ***
The user can search for flights
    [Tags]    search_flights
    Open browser    http://blazedemo.com/   Chrome
    Select From List By Value   xpath://select[@name='fromPort']  Paris
    Select From List by Value   xpath://select[@name='toPort']    London
    Click Button    css:input[type='submit']
    @{flights}=  Get WebElements    css:table[class='table']>tbody tr
    Should Not Be Empty     ${flights}
    Close All Browsers

To be able to use keywords from external libraries (like SeleniumLibrary), we need to import them. This should be done in the "Settings" section of the code in Robot with the setting "Library." The "Settings" section is used not only for importing external libraries and resources, but also for defining metadata for test suites and test cases. We will come back to the "Settings" section later in the blog post.

Underneath the "Settings" section, there is "Test Cases" section where you should add all the test cases within a test cases file. As you can see, we have used keywords from SeleniumLibrary to open the browser and to select appropriate values from departure and destination drop-down lists.

In the Robot Framework, any keyword can accept any number of arguments. In our case, for example, the keyword "Open browser" accepts two arguments: the URL to open and the browser this URL should be opened in. The keyword "Select From List by Value" accepts the selector of the web element as the first argument, and accepts the value to select as the second argument.

The Tags section is used to assign a logical group to a test case. Furthermore, it can be used to execute tests only with the specified tag. I will explain how to achieve this later in the blog post.

There is one more thing that needs to be explained in more detail here. The result of the  Get WebElements keyword is assigned to the variable @flights . You can use the variable to describe keyword arguments, to store a value in a test suite or to save the results of keyword execution. The names of variables are case-insensitive, and as well as names of keywords, and spaces and underscores are ignored. The test case can be re-written with the usage of variables in the following way:

*** Settings ***
Library  SeleniumLibrary

*** Variables ***
${URL}          http://blazedemo.com/
${BROWSER}      Chrome

*** Test Cases ***
The user can search for flights
    Open browser    ${URL}   ${BROWSER}

Such styling adds more readability to the test case. Variables can be scalar (with the $ prefix), lists (with the @ prefix, dictionaries (with the & prefix) and environment (with the % prefix).

After saving the available flights as web elements in the list  flights , we use the built-in keyword Should Not Be Empty to verify there is at least one flight found. As a last step we close all opened browsers.

Now, it's high time to make some improvements using Robot Framework's capabilities. If you want to add more test cases to the test suite, you will notice that anytime you want to interact with the page, you need to open it. And, after test passed, you need to close the browser. With the help of the "Settings" section that was mentioned previously, you can manage these tear up and tear down operations.

Updated test case:

As you can see, we have moved open browser and closing all browsers to suite setup and suite teardown respectively.

As another improvement, we can develop custom keywords to select departure and destination cities and to find flights.

All keywords we are intending to use should be created in the "Keywords" section. You can notice that we don't have any test cases in the file. Such files are considered as resource files and they have slightly different properties than test cases files: in the "Settings" section you cannot use metadata settings such as Suite Setup, Suite TearDown or Tags settings. You can only use the import settings (Library, Resource, Variables) and Documentation.

Going back to the created keywords: In "Select Departure City" and "Select Destination City" keywords we have an Arguments  property, which means these keywords require arguments. A keyword can have any number of arguments but it's advised not to overload it with too many. If a keyword has many arguments it's harder to understand what it does and it becomes prone to errors.

Our test cases file search_flights.robot was changed to reflect the modifications:

*** Settings ***
Resource        search_flights_keywords.robot
Suite Setup     Open Home Page
Suite Teardown  Close Browsers


*** Test Cases ***
The user can search for flights
   [Tags]   search_flights
    Select Departure City   Paris
    Select Destination City    London
    Search For Flights
    There are available Flights

Resources files are loaded via the setting "Resource." Now our test looks more elegant and it is easy to understand what the test steps are.

The next thing we can do is to add a setup suite and a teardown suite to the test suite initialization file. As mentioned previously in the blog post, a test cases file forms a test suite. But test suite files can be put to a higher-level test suite by creating directories. On its own, a directory cannot have the setup and teardown information, but the Robot Framework has initialization files for that. The name of the initialization file should be __init__.ext, where ext should be one of the supported formats.

Let's create a directory for our test suite and name it search_flights.

Our initialization file will be search_flights/__init__.robot (don't confuse it with python __init__.py)

*** Settings ***
Suite Setup     Open Home Page
Suite Teardown  Close Browsers
Resource        search_flights_keywords.robot

Like resource files, initialization files cannot have test cases and not all settings are supported. Variables and keywords that were created or imported will not be available in the lower level suites. Use initialization files to provide common tags, documentation and setup/teardown operations.

With the initialization file, our test case looks even simpler:

*** Settings ***
Resource        search_flights_keywords.robot


*** Test Cases ***
The user can search for flights
    Select Departure City   Paris
    Select Destination City    London
    Search For Flights
    There are available Flights

It's a good practice to keep test cases as simple as possible. It will make them more stable and easy to modify when needed.

Ok, we saw how to create a test case using existing keywords that are grouped into user-defined keywords. But what if we need to save data to a database when selecting the departure city? Or to log information to a file? Those are cases when we need to form a keyword from a function or method of the class. The Robot Framework allows us to do that.

Let's modify our test to use keywords formed from python class methods.

1. First, we need to install the Selenium library with pip:

2. Download the Chrome WebDriver from the Selenium download page.

3. Let's create a Python class that will manage the WebDriver session:

The WebDriverManager class stores the created driver session in the __driver   variable and returns it when the get_web_driver   method is called.

4. Since web pages elements are not loaded simultaneously, we need to wait until the element becomes visible on the page. In Selenium, this can be achieved through the WebDriverWait  class. We will create the class Web that will encapsulate the usage of the WebDriverWait   class instance by exposing simple methods to search for web elements on the page by xpath.

5. Now, we need a class that will expose methods to interact with elements on the Flights search page.

This class uses a Web instance to access web elements on the web page and has a method to select values from departure and destination drop-down lists, clicking on "Find Flights" button, and getting found flights.

This class also has the methods open()   and close()  , which are self-explanatory.

Let's have a look at our modified test case:

*** Settings ***
Library         pages//SearchFlightPage.py  Chrome
Suite Setup     Open
Suite Teardown  Close

*** Test Cases ***
The user can search for flights
    Select Departure City   Paris
    Select Destination City    London
    @{flights}=     Get Found Flights
    Should Not Be Empty     ${flights}

We are using SearchFlightsPage.py library to import class methods as keywords.

6. There is still one thing we can improve here. As you see we are doing a little programming by saving a list of flights into a variable and then passing it further to Should Not Be Empty   keyword.

We can bypass this by saving a test variable using the keyword Set Test Variable   in the resource file.

Have a look at search_flights_keywords.robot:

*** Settings ***
Library         pages//SearchFlightPage.py  Chrome


*** Keywords ***
Open search page
    Open

Close pages
    Close

Select departure
    [Arguments]  ${city}

    select departure city  ${city}

select destination
    [Arguments]  ${city}
    select destination city  ${city}

Search Flights
    search for flights
    @{flights}=     Get Found Flights
    set test variable  ${flights}

Flights are found
    Should Not Be Empty     ${flights}

In the Search Flights   keyword we are saving search results into the test variable  ${flights} . Later, in Flights are found keyword we read this variable.

Our modified test:

*** Settings ***
Resource         search_flights_keywords.robot
Suite Setup     Open search page
Suite Teardown  Close pages

*** Test Cases ***
The user can search for flights
    Select Departure   Paris
    Select Destination    London
    Search Flights
    Flights are found

The choice to use an already developed keyword or to create your own is always up to a test developer. Try to keep things as simple as possible: if you don't need additional behavior that can be implemented only in python modules or classes, use already developed keywords — it will save your time.

To execute robot tests in your prompt, type: robot path/to/tests.

'path/to/tests' should be a name of a suite file or a suite directory.

You can select which tests to be executed from a suite by filtering by tags. Any robot test or suite can have a tag assigned. To execute tests with an assigned tag, type the following in your prompt:

Robot -include [tag]    path/to/tests

You can also create your test scripts with Taurus or upload existing ones, in a much easier way. In Taurus, you can run your test in loops to get transaction time statistics. By using the command —cloud you can delegate the test to the cloud through BlazeMeter, getting an online report and without having to have your own infrastructure.

Create and execute a configuration for your Robot test in Taurus and you will get a detailed result of how your application works.

Consider the following configuration to check if our "Find Flights" scenario will work for 1 min:

execution:
- executor: selenium
  runner: robot
  hold-for: 1m
  scenario:
    script: search_flights/

reporting:
  - final-stats
  - blazemeter

This will start Taurus and execute our scenario for 1 min:

BlazeMeter will soon have exciting updates about GUI testing! Sign up here to get information. For stress testing, request a live BlazeMeter demo with one of our performance engineers.