Как пишутся автотесты на питоне

Продолжаем погружаться в работу тестировщика, он же — QA, quality assurance engineer. Его задача — проверить код на наличие ошибок и работу программы в разных условиях. 

Мы уже писали о том, что ещё делают тестировщики и какие инструменты для этого используют:

Сегодня мы попробуем написать автотесты — чаще всего именно этим занимаются тестировщики на работе.

Что такое автотесты

Автотесты — это когда одна программа проверяет работу другой программы. Работает это примерно так:

Чтобы всё было наглядно, покажем работу автотестов на реальном коде.

Исходная программа

Допустим, мы пишем интерактивную текстовую игру — в ней всё оформляется текстом, и развитие игры зависит от ответов пользователя. Мы сделали отдельный модуль, который делает четыре вещи:

# Собираем приветствие
def hello(name):
    # делаем первую букву имени большой
    out = name.title()
    # формируем приветствие
    out = 'Привет, ' + out + '.'
    # возвращаем его как результат работы функции
    return out

Эта функция хранится в файле hello_function.py — так мы разбиваем программу на модули, каждый из которых делает что-то своё. 

Напишем начало основной программы, которая запрашивает имя, формирует приветствие и добавляет к нему стартовую фразу:

# импортируем функцию из другого файла
from hello_function import hello
# объясняем, что нужно сделать пользователю
print("Введите имя, чтобы начать игру")

# спрашиваем имя
name = input("Как вас зовут: ")
# обрабатываем имя и формируем приветствие
result = hello(name)
# добавляем вторую строку
print(result + " nДобро пожаловать в «Код»!")

Сохраним это в новом файле start.py и запустим его:

Вроде работает, но хорошо бы проверить, а всегда ли приветствие будет формироваться правильно? Можно сделать вручную, а можно написать автотест.

Пишем автотест

Первое, что нам нужно сделать, — подключить стандартный модуль для автотестов unittest. Есть модули покруче, но для наших проектов стандартного хватит с запасом. Также получаем доступ к функции hello() из файла hello_function.py — работу именно этой функции мы будем проверять автотестом.

# подключаем модуль для автотестов
import unittest
# импортируем функцию из другого файла
from hello_function import hello

А теперь самое важное: нам нужно объявить класс и функцию, внутри которой и будет находиться наш тест. Причём название функции должно начинаться с test_, чтобы она выполнялась автоматически.

Внутри функции делаем такое:

Этими действиями мы как будто вызываем ту функцию и смотрим, получилось ли то, что нам нужно, или нет. При этом нам не нужно обрабатывать результаты тестов — за нас это сделает модуль unittest. 

Для запуска тестов добавляем в конец кода стандартный вызов. Читайте комментарии, чтобы лучше вникнуть в код:

# подключаем модуль для автотестов
import unittest
# импортируем функцию из другого файла
from hello_function import hello

# объявляем класс с тестом
class HelloTestCase(unittest.TestCase):
    # функция, которая проверит, как формируется приветствие
   def test_hello(self):
        # отправляем тестовую строку в функцию
        result = hello("миша")
        # задаём ожидаемый результат
        self.assertEqual(result, "Привет, Миша.")

# запускаем тестирование
if __name__ == '__main__':
    unittest.main() 

После запуска мы увидим такое. Ответ «OK» означает, что наш тест сработал и завершился без ошибок:

Ещё такие тесты позволяют найти ошибки в самом коде. Допустим, разработчик забыл добавить принудительный перевод большой буквы в имени, и тогда тест не пройдёт. Если получится, система даже подсветит, что именно не совпало в результате, — в нашем случае это первая буква имени.

Что дальше

Мы написали самый простой тест и всего с одним условием. При этом мы не проверили тестами работу основной программы — в реальном проекте это тоже нужно было бы сделать. Чтобы попрактиковаться, мы сделаем тесты для одного из наших старых проектов на Python. Заодно проверим, нет ли там каких ошибок, которые мы не заметили.

Автор: Энди Найт (Andy Knight)
Оригинал статьи
Перевод: Ольга Алифанова

Часть 1

Теперь, когда мы знаем, зачем нам Web UI-тестирование и какими должны быть наши цели, настроим проект тест-автоматизации на Python, используя pytest!

Почему Python?

Python – один из самых популярных из доступных языков программирования. На нем пишут бэкэнды сайтов, проводят анализ данных, создают скрипты для администрирования системы… Его синтаксис чист, читабелен и элегантен, и отлично подходит как для экспертов, так и для начинающих. К тому же все мыслимое и немыслимое всего лишь в одном импорте от вас. Логично, что Python также отлично подходит для тест-автоматизации. Его лаконичность позволяет больше фокусироваться на тесте, и меньше – на коде. Тестировщики без особых навыков программирования, как правило, изучают Python быстрее, нежели другие языки вроде Java или C#. Python прекрасен для быстрого погружения в автоматизацию!

Что такое pytest?

В сердце любого проекта функциональной тест-автоматизации лежит «корневой» тест-фреймворк. Фреймворк имеет дело со структурой тест-кейсов, выполнением тестов, и отчетности о пройденных и упавших проверках. Это основа, к которой добавляются дополнительные пакеты и код (вроде Selenium WebDriver).

pytest – один из лучших тест-фреймворков Python. Он прост, масштабируем, и «Пайтоничен». Тест-кейсы пишутся как функции, а не классы. Падения тест-утверждений выводятся в отчете с реальными значения ми. При помощи плагинов можно добавить покрытие кода, красивые отчеты, и параллельный запуск. pytest может интегрироваться с другими фреймворками вроде Django и Flask. Согласно опросу разработчиков Python-2018, pytest – самый популярный тест-фреймворк для Python.

Приступим

Давайте создадим наш первый тест-проект на Python! Если вы еще этого не сделали, пожалуйста, скачайте и установите Python 3. Затем создайте новую директорию для проекта:

1. $ mkdir python-webui-testing
2. $ cd python-webui-testing

Создавая новый проект Python, я всегда создаю виртуальное окружение для его зависимостей. В этом случае у проектов на одной машине не будет конфликтующих версий пакетов. Я пользуюсь pipenv, потому что он упрощает дело. Для глобальной установки pipenv введите:

1. $ pip install pipenv

Затем установите pytest для вашего нового проекта:

1. $ pipenv install pytest —dev

Pipenv добавит в ваш проект два новых файла: Pipfile и Pipfile.lock. Pipfile определяет требования проекта, а Pipfile.lock «блокирует» явные версии, которыми проект будет пользоваться. Опция “–dev” в команде означает, что пакет pytest будет использоваться для разработки, а не для деплоя.

Первый тест

По традиции большинство проектов размещают все тесты в директории tests/. Давайте следовать традиции:

1. $ mkdir tests
2. $ cd tests

Создайте модуль Python с названием test_math.py для нашего первого теста, и добавьте следующий код:

1. def test_addition():
2. assert 1 + 1 == 2

Тестам, написанным при помощи pytest, обычно не требуется много кода. Эти две строчки – полнофункциональный тест-кейс! Тест-кейс записан как функция, а не как класс. Для такого базового теста импорты не нужны. Нативный оператор контроля Python используется вместо кастомных контрольных вызовов.

Запуск тестов

Давайте запустим наш новый тест. Смените директорию на корневую директорию проекта, и вызовите модуль pytest:

1. $ cd ..
2. $ pipenv run python -m pytest
3. ============================= test session starts ==============================
4. platform darwin — Python 3.7.3, pytest-4.5.0, py-1.8.0, pluggy-0.12.0
5. rootdir: /Users/andylpk247/Programming/automation-panda/python-webui-testing
6. collected 1 item
8. tests/test_math.py . [100%]
10. =========================== 1 passed in 0.02 seconds ===========================

Наш первый тест пройден!

Как pytest обнаружил наш тест? По имени: pytest будет искать тест-функции с именем test_* в модулях по имени test_*.py. Интересно, что pytest не требует файла __init__.py в тест-директориях.

Упавшие тесты

Что будет, если тест упадет? Давайте добавим другой тест с багом, чтобы проверить это:

1. def test_subtraction():
2. diff = 1 — 1
3. assert diff == 1

Теперь при запуске Pytest мы увидим вот что:

1. $ pipenv run python -m pytest
2. ============================= test session starts ==============================
3. platform darwin — Python 3.7.3, pytest-4.5.0, py-1.8.0, pluggy-0.12.0
4. rootdir: /Users/andylpk247/Programming/automation-panda/python-webui-testing
5. collected 2 items
7. tests/test_math.py .F [100%]
9. =================================== FAILURES ===================================
10. _______________________________ test_subtraction _______________________________
12. def test_subtraction():
13. diff = 1 — 1
14. > assert diff == 1
15. E assert 0 == 1
17. tests/test_math.py:13: AssertionError
18. ====================== 1 failed, 1 passed in 0.08 seconds ======================

Тест test_subtraction упал с «F» вместо «.». Более того, pytest выводит сообщения трассировки, показывающие упавшее утверждение вместе с модулем и номером строки. Отметим, что реальные значения каждого выражения в утверждении тоже отображаются: diff оценивается как 0, что явно не равно 1. Круто! Эта самодиагностика утверждений очень помогает разбираться в падениях тестов.

Исправим баг:

1. def test_subtraction():
2. diff = 1 — 1
3. assert diff == 0

И перезапустим эти тесты:

1. $ pipenv run python -m pytest
2. ============================= test session starts ==============================
3. platform darwin — Python 3.7.3, pytest-4.5.0, py-1.8.0, pluggy-0.12.0
4. rootdir: /Users/andylpk247/Programming/automation-panda/python-webui-testing
5. collected 2 items
7. tests/test_math.py .. [100%]
9. =========================== 2 passed in 0.02 seconds ===========================

Мы снова на верном пути.

Параметризованные тесты

Что, если нам нужно запустить тест-процедуру с разными сочетаниями ввода? У pytest для этого есть декоратор! Составим тест для умножения с параметризованным вводом:

1. import pytest
3. @pytest.mark.parametrize(
4. «a,b,expected»,
5. [(0, 5, 0), (1, 5, 5), (2, 5, 10), (-3, 5, -15), (-4, -5, 20)])
6. def test_multiplication(a, b, expected):
7. assert a * b == expected

На этот раз нужно импортировать модуль pytest. Декоратор @pytest.mark.parametrize будет заменять наборы значений для аргументов тест-функции, запуская функцию по разу для каждого набора. Повторный запуск тестов покажет больше пройденных тестов:

1. $ pipenv run python -m pytest
2. ============================= test session starts ==============================
3. platform darwin — Python 3.7.3, pytest-4.5.0, py-1.8.0, pluggy-0.12.0
4. rootdir: /Users/andylpk247/Programming/automation-panda/python-webui-testing
5. collected 7 items
7. tests/test_math.py ……. [100%]
9. =========================== 7 passed in 0.03 seconds ===========================

Супер! Параметры – отличный способ реализовать тестирование, управляемое через данные.

Sweet! Parameters are a great way to do data-driven testing.

Верификация исключений

Pytest относится к необрабатываемым исключениям, как к падениям теста. На самом деле оператор контроля просто выдает исключение для регистрации падения. Что, если нам нужно убедиться, что выдается правильное исключение? Используйте pytest.raises с нужным типом исключения – например, так:

1. def test_divide_by_zero():
2. with pytest.raises(ZeroDivisionError):
3. 1 / 0

Rerun the tests to make sure all is well:

1. $ pipenv run python -m pytest
2. ============================= test session starts ==============================
3. platform darwin — Python 3.7.3, pytest-4.5.0, py-1.8.0, pluggy-0.12.0
4. rootdir: /Users/andylpk247/Programming/automation-panda/python-webui-testing
5. collected 8 items
7. tests/test_math.py …….. [100%]
9. =========================== 8 passed in 0.04 seconds ===========================

Отлично. Математика все еще работает!

Больше информации

В этой главе мы рассказали об основах, но pytest умеет и многое другое. Изучите официальный сайт pytest, чтобы узнать о более продвинутых фичах фреймворка. Далее мы разберемся, как использовать фикстуры и новые плагины.

Software testing is the process in which a developer ensures that the actual output of the software matches with the desired output by providing some test inputs to the software. Software testing is an important step because if performed properly, it can help the developer to find bugs in the software in very less amount of time. Software testing can be divided into two classes, Manual testing and Automated testing. Automated testing is the execution of your tests using a script instead of a human. In this article, we’ll discuss some of the methods of automated software testing with Python. Let’s write a simple application over which we will perform all the tests. 

Note: For more information about the function __repr__(), refer this article. Now that we have our software ready, let’s have a look at the directory structure of our project folder and after that, we’ll start testing our software.

---Software_Testing
   |--- __init__.py (to initialize the directory as python package)
   |--- app.py (our software)
   |--- tests (folder to keep all test files)
           |--- __init__.py

The ‘unittest’ module

One of the major problems with manual testing is that it requires time and effort. In manual testing, we test the application over some input, if it fails, either we note it down or we debug the application for that particular test input, and then we repeat the process. With unittest, all the test inputs can be provided at once and then you can test your application. In the end, you get a detailed report with all the failed test cases clearly specified, if any. The unittest module has both a built-in testing framework and a test runner. A testing framework is a set of rules which must be followed while writing test cases, while a test runner is a tool which executes these tests with a bunch of settings, and collects the results. Installation: unittest is available at PyPI and can be installed with the following command –

pip install unittest

Use: We write the tests in a Python module (.py). To run our tests, we simply execute the test module using any IDE or terminal. Now, let’s write some tests for our small software discussed above using the unittest module.

1. Create a file named tests.py in the folder named “tests”.
2. In tests.py import unittest.
3. Create a class named TestClass which inherits from the class unittest.TestCase. Rule 1: All the tests are written as the methods of a class, which must inherit from the class unittest.TestCase.
4. Create a test method as shown below. Rule 2: Name of each and every test method should start with “test” otherwise it’ll be skipped by the test runner. 

1. Rule 3: We use special assertEqual() statements instead of the built-in assert statements available in Python. The first argument of assertEqual() is the actual output, the second argument is the desired output and the third argument is the error message which would be displayed in case the two values differ from each other (test fails).
2. To run the tests we just defined, we need to call the method unittest.main(), add the following lines in the “tests.py” module. 

1. Because of these lines, as soon as you run execute the script “test.py”, the function unittest.main() would be called and all the tests will be executed.

Finally the “tests.py” module should resemble the code given below. 

Having written our test cases let us now test our application for any bugs. To test your application you simply need to execute the test file “tests.py” using the command prompt or any IDE of your choice. The output should be something like this.

.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

In the first line, a .(dot) represents a successful test while an ‘F’ would represent a failed test case. The OK message, in the end, tells us that all the tests were passed successfully. Let’s add a few more tests in “tests.py” and retest our application. 

.F.F
======================================================================
FAIL: test_area_negative (__main__.TestSum)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests_unittest.py", line 11, in test_area_negative
    self.assertEqual(sq.area(), -1, f'Area is shown {sq.area()} rather than -1 for negative side length')
AssertionError: 9 != -1 : Area is shown 9 rather than -1 for negative side length

======================================================================
FAIL: test_perimeter_negative (__main__.TestSum)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests_unittest.py", line 19, in test_perimeter_negative
    self.assertEqual(sq.perimeter(), -1, f'Perimeter is {sq.perimeter()} rather than -1 for negative side length')
AssertionError: -24 != -1 : Perimeter is -24 rather than -1 for negative side length

----------------------------------------------------------------------
Ran 4 tests in 0.001s

FAILED (failures=2)

A few things to note in the above test report are –

• The first line represents that test 1 and test 3 executed successfully while test 2 and test 4 failed
• Each failed test case is described in the report, the first line of the description contains the name of the failed test case and the last line contains the error message we defined for that test case.
• At the end of the report you can see the number of failed tests, if no test fails the report will end with OK

Note: For further knowledge you can read the complete documentation of unittest.

The “nose2” module

The purpose of nose2 is to extend unittest to make testing easier. nose2 is compatible with tests written using the unittest testing framework and can be used as a replacement of the unittest test runner. Installation: nose2 can be installed from PyPI using the command,

pip install nose2

Use: nose2 does not have any testing framework and is merely a test runner which is compatible with the unittest testing framework. Therefore we’ll the run same tests we wrote above (for unittest) using nose2. To run the tests we use the following command in the project source directory (“Software_Testing” in our case),

nose2

In nose2 terminology all the python modules (.py) with name starting from “test” (i.e. test_file.py, test_1.py) are considered as test files. On execution, nose2 will look for all test files in all the sub-directories which lie under one or more of the following categories,

• which are python packages (contain “__init__.py”).
• whose name starts with “test” after being lowercased, i.e. TestFiles, tests.
• which are named either “src” or “lib”.

nose2 first loads all the test files present in the project and then the tests are executed. Thus, with nose2 we get the freedom to split our tests among various test files in different folders and execute them at once, which is very useful when dealing with large number of tests. Let’s now learn about different customisation options provided by nose2 which can help us during the testing process.

1. Changing the search directory – If we want to change the directory in which nose2 searches for test files, we can do that using the command line arguments -s or –start-dir as,

nose2 -s DIR_ADD DIR_NAME

1. here, DIR_NAME is the directory in which we want to search for the test files and, DIR_ADD is the address of the parent directory of DIR_NAME relative to the project source directory (i.e. use “./” if test directory is in the project source directory itself). This is extremely useful when you want to test only one feature of your application at a time.
2. Running specific test cases – Using nose2 we can also run a specific test at a time by using the command line arguments -s and –start-dir as,

nose2 -s DIR_ADD DIR_NAME.TEST_FILE.TEST_CLASS.TEST_NAME

• TEST_NAME: name of the test method.
• TEST_CLASS: class in which the test method is defined.
• TEST_FILE: name of the test file in which the test case is defined i.e. test.py.
• DIR_NAME: directory in which the test file exists.
• DIR_ADD: address of the parent directory of DIR_NAME relative to the project source.

1. Running tests in a single module – nose2 can also be used like unittest by calling the function nose2.main() just like we called unittest.main() in previous examples.

The “pytest” module

pytest is the most popular testing framework for python. Using pytest you can test anything from basic python scripts to databases, APIs and UIs. Though pytest is mainly used for API testing, in this article we’ll cover only the basics of pytest. Installation: You can install pytest from PyPI using the command,

pip install pytest

Use: The pytest test runner is called using the following command in project source,

py.test

Unlike nose2, pytest looks for test files in all the locations inside the project directory. Any file with name starting with “test_” or ending with “_test” is considered a test file in the pytest terminology. Let’s create a file “test_file1.py” in the folder “tests” as our test file. Creating test methods: pytest supports the test methods written in the unittest framework, but the pytest framework provides easier syntax to write tests. See the code below to understand the test method syntax of the pytest framework. 

Note: similar to unittest, pytest requires all test names to start with “test”. Unlike unittest, pytest uses the default python assert statements which make it further easier to use. Note that, now the “tests” folder contains two files namely, “tests.py” (written in unittest framework) and “test_file1.py” (written in pytest framework). Now let’s run the pytest test runner.

py.test

You’ll get a similar report as obtained by using unittest.

============================= test session starts ==============================
platform linux -- Python 3.6.7, pytest-4.4.1, py-1.8.0, pluggy-0.9.0
rootdir: /home/manthan/articles/Software_testing_in_Python
collected 6 items                                                              

tests/test_file1.py .F                                                   [ 33%]
tests/test_file2.py .F.F                                                 [100%]

=================================== FAILURES ===================================

The percentages on the right side of the report show the percentage of tests that have been completed at that moment, i.e. 2 out of the 6 test cases were completed at the end of the “test_file1.py”. Here are a few more basic customisations that come with pytest.

1. Running specific test files: To run only a specific test file, use the command,

py.test <filename>

1. Substring matching: Suppose we want to test only the area() method of our Square class, we can do this using substring matching as follows,

py.test -k "area"

1. With this command pytest will execute only those tests which have the string “area” in their names, i.e. “test_file1_area()”, “test_area()” etc.
2. Marking: As a substitute to substring matching, marking is another method using which we can run a specific set of tests. In this method we put a mark on the tests we want to run. Observe the code example given below, 

1. In the above code example test_file1_area() is marked with tag “area”. All the test methods which have been marked with some tag can be executed by using the command,

py.test -m <tag_name>

1. Parallel Processing: If you have a large number of tests then pytest can be customised to run these test methods in parallel. For that you need to install pytest-xdist which can be installed using the command,

pip install pytest-xdist

1. Now you can use the following command to execute your tests faster using multiprocessing,

py.test -n 4

1. With this command pytest assigns 4 workers to perform the tests in parallel, you can change this number as per your needs. If your tests are thread-safe, you can also use multithreading to speed up the testing process. For that you need to install pytest-parallel (using pip). To run your tests in multithreading use the command,

pytest --workers 4

Software testing is the process in which a developer ensures that the actual output of the software matches with the desired output by providing some test inputs to the software. Software testing is an important step because if performed properly, it can help the developer to find bugs in the software in very less amount of time. Software testing can be divided into two classes, Manual testing and Automated testing. Automated testing is the execution of your tests using a script instead of a human. In this article, we’ll discuss some of the methods of automated software testing with Python. Let’s write a simple application over which we will perform all the tests. 

Note: For more information about the function __repr__(), refer this article. Now that we have our software ready, let’s have a look at the directory structure of our project folder and after that, we’ll start testing our software.

---Software_Testing
   |--- __init__.py (to initialize the directory as python package)
   |--- app.py (our software)
   |--- tests (folder to keep all test files)
           |--- __init__.py

The ‘unittest’ module

One of the major problems with manual testing is that it requires time and effort. In manual testing, we test the application over some input, if it fails, either we note it down or we debug the application for that particular test input, and then we repeat the process. With unittest, all the test inputs can be provided at once and then you can test your application. In the end, you get a detailed report with all the failed test cases clearly specified, if any. The unittest module has both a built-in testing framework and a test runner. A testing framework is a set of rules which must be followed while writing test cases, while a test runner is a tool which executes these tests with a bunch of settings, and collects the results. Installation: unittest is available at PyPI and can be installed with the following command –

pip install unittest

Use: We write the tests in a Python module (.py). To run our tests, we simply execute the test module using any IDE or terminal. Now, let’s write some tests for our small software discussed above using the unittest module.

1. Create a file named tests.py in the folder named “tests”.
2. In tests.py import unittest.
3. Create a class named TestClass which inherits from the class unittest.TestCase. Rule 1: All the tests are written as the methods of a class, which must inherit from the class unittest.TestCase.
4. Create a test method as shown below. Rule 2: Name of each and every test method should start with “test” otherwise it’ll be skipped by the test runner. 

1. Rule 3: We use special assertEqual() statements instead of the built-in assert statements available in Python. The first argument of assertEqual() is the actual output, the second argument is the desired output and the third argument is the error message which would be displayed in case the two values differ from each other (test fails).
2. To run the tests we just defined, we need to call the method unittest.main(), add the following lines in the “tests.py” module. 

1. Because of these lines, as soon as you run execute the script “test.py”, the function unittest.main() would be called and all the tests will be executed.

Finally the “tests.py” module should resemble the code given below. 

Having written our test cases let us now test our application for any bugs. To test your application you simply need to execute the test file “tests.py” using the command prompt or any IDE of your choice. The output should be something like this.

.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

In the first line, a .(dot) represents a successful test while an ‘F’ would represent a failed test case. The OK message, in the end, tells us that all the tests were passed successfully. Let’s add a few more tests in “tests.py” and retest our application. 

.F.F
======================================================================
FAIL: test_area_negative (__main__.TestSum)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests_unittest.py", line 11, in test_area_negative
    self.assertEqual(sq.area(), -1, f'Area is shown {sq.area()} rather than -1 for negative side length')
AssertionError: 9 != -1 : Area is shown 9 rather than -1 for negative side length

======================================================================
FAIL: test_perimeter_negative (__main__.TestSum)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests_unittest.py", line 19, in test_perimeter_negative
    self.assertEqual(sq.perimeter(), -1, f'Perimeter is {sq.perimeter()} rather than -1 for negative side length')
AssertionError: -24 != -1 : Perimeter is -24 rather than -1 for negative side length

----------------------------------------------------------------------
Ran 4 tests in 0.001s

FAILED (failures=2)

A few things to note in the above test report are –

• The first line represents that test 1 and test 3 executed successfully while test 2 and test 4 failed
• Each failed test case is described in the report, the first line of the description contains the name of the failed test case and the last line contains the error message we defined for that test case.
• At the end of the report you can see the number of failed tests, if no test fails the report will end with OK

Note: For further knowledge you can read the complete documentation of unittest.

The “nose2” module

The purpose of nose2 is to extend unittest to make testing easier. nose2 is compatible with tests written using the unittest testing framework and can be used as a replacement of the unittest test runner. Installation: nose2 can be installed from PyPI using the command,

pip install nose2

Use: nose2 does not have any testing framework and is merely a test runner which is compatible with the unittest testing framework. Therefore we’ll the run same tests we wrote above (for unittest) using nose2. To run the tests we use the following command in the project source directory (“Software_Testing” in our case),

nose2

In nose2 terminology all the python modules (.py) with name starting from “test” (i.e. test_file.py, test_1.py) are considered as test files. On execution, nose2 will look for all test files in all the sub-directories which lie under one or more of the following categories,

• which are python packages (contain “__init__.py”).
• whose name starts with “test” after being lowercased, i.e. TestFiles, tests.
• which are named either “src” or “lib”.

nose2 first loads all the test files present in the project and then the tests are executed. Thus, with nose2 we get the freedom to split our tests among various test files in different folders and execute them at once, which is very useful when dealing with large number of tests. Let’s now learn about different customisation options provided by nose2 which can help us during the testing process.

1. Changing the search directory – If we want to change the directory in which nose2 searches for test files, we can do that using the command line arguments -s or –start-dir as,

nose2 -s DIR_ADD DIR_NAME

1. here, DIR_NAME is the directory in which we want to search for the test files and, DIR_ADD is the address of the parent directory of DIR_NAME relative to the project source directory (i.e. use “./” if test directory is in the project source directory itself). This is extremely useful when you want to test only one feature of your application at a time.
2. Running specific test cases – Using nose2 we can also run a specific test at a time by using the command line arguments -s and –start-dir as,

nose2 -s DIR_ADD DIR_NAME.TEST_FILE.TEST_CLASS.TEST_NAME

• TEST_NAME: name of the test method.
• TEST_CLASS: class in which the test method is defined.
• TEST_FILE: name of the test file in which the test case is defined i.e. test.py.
• DIR_NAME: directory in which the test file exists.
• DIR_ADD: address of the parent directory of DIR_NAME relative to the project source.

1. Running tests in a single module – nose2 can also be used like unittest by calling the function nose2.main() just like we called unittest.main() in previous examples.

The “pytest” module

pytest is the most popular testing framework for python. Using pytest you can test anything from basic python scripts to databases, APIs and UIs. Though pytest is mainly used for API testing, in this article we’ll cover only the basics of pytest. Installation: You can install pytest from PyPI using the command,

pip install pytest

Use: The pytest test runner is called using the following command in project source,

py.test

Unlike nose2, pytest looks for test files in all the locations inside the project directory. Any file with name starting with “test_” or ending with “_test” is considered a test file in the pytest terminology. Let’s create a file “test_file1.py” in the folder “tests” as our test file. Creating test methods: pytest supports the test methods written in the unittest framework, but the pytest framework provides easier syntax to write tests. See the code below to understand the test method syntax of the pytest framework. 

Note: similar to unittest, pytest requires all test names to start with “test”. Unlike unittest, pytest uses the default python assert statements which make it further easier to use. Note that, now the “tests” folder contains two files namely, “tests.py” (written in unittest framework) and “test_file1.py” (written in pytest framework). Now let’s run the pytest test runner.

py.test

You’ll get a similar report as obtained by using unittest.

============================= test session starts ==============================
platform linux -- Python 3.6.7, pytest-4.4.1, py-1.8.0, pluggy-0.9.0
rootdir: /home/manthan/articles/Software_testing_in_Python
collected 6 items                                                              

tests/test_file1.py .F                                                   [ 33%]
tests/test_file2.py .F.F                                                 [100%]

=================================== FAILURES ===================================

The percentages on the right side of the report show the percentage of tests that have been completed at that moment, i.e. 2 out of the 6 test cases were completed at the end of the “test_file1.py”. Here are a few more basic customisations that come with pytest.

1. Running specific test files: To run only a specific test file, use the command,

py.test <filename>

1. Substring matching: Suppose we want to test only the area() method of our Square class, we can do this using substring matching as follows,

py.test -k "area"

1. With this command pytest will execute only those tests which have the string “area” in their names, i.e. “test_file1_area()”, “test_area()” etc.
2. Marking: As a substitute to substring matching, marking is another method using which we can run a specific set of tests. In this method we put a mark on the tests we want to run. Observe the code example given below, 

1. In the above code example test_file1_area() is marked with tag “area”. All the test methods which have been marked with some tag can be executed by using the command,

py.test -m <tag_name>

1. Parallel Processing: If you have a large number of tests then pytest can be customised to run these test methods in parallel. For that you need to install pytest-xdist which can be installed using the command,

pip install pytest-xdist

1. Now you can use the following command to execute your tests faster using multiprocessing,

py.test -n 4

1. With this command pytest assigns 4 workers to perform the tests in parallel, you can change this number as per your needs. If your tests are thread-safe, you can also use multithreading to speed up the testing process. For that you need to install pytest-parallel (using pip). To run your tests in multithreading use the command,

pytest --workers 4