Thursday, April 23, 2020

Cypress basics: how do cypress tests work?

Introduction

As I wrote earlier in a blogpost now we have cypress.io test framework integrated into Collabora Online codebase. Since we are testing an online, Javascript-based application that means we are working in an asynchronous environment. So triggering a user event via this framework, does not mean that this event is actually finished at the same time when the related cypress method is called. On the other hand, we try to cover real user scenarios, which are usually a sequence of events, as the user doing things on the UI step-by-step. So the question is, how to write sequential tests in an asynchronous environment. Fortunately, cypress has the answer to that.

The problem of asynchronicity

First, let's see an example of simulating user events in the cypress test framework. The following code triggers a selection of the text by simulating CTRL+A shortcut:

// Select all text
cy.get('textarea.clipboard')
 .type('{ctrl}a')

With calling this method we trigger a keyboard input, which will be handled by the client code (loleaflet). The client JS code sends a message to the server about the keyboard input. On the server-side, we will execute the shortcut, which results in a text selection. Then the server will send back the new text selection to the client-side, which will display it for the user. However the cy.get().type() call won't wait for that, it just triggers the event on the client-side and moves on.

Why is it a problem? Let's extend this code with another event:

// Select all text
cy.get('textarea.clipboard')
 .type('{ctrl}a')

// Apply bold font
cy.get('#tb_editbar_item_bold')
 .click();

Let's assume that this '#tb_editbar_item_bold' item exists. It's a toolbar button which applies bold font on the selected text. The problem here is that the selection is not finished yet when we already triggered the font change. Triggering these events does not mean that they will be executed in the specified order. Both events are handled in an asynchronous way, so if the first event (e.g. typing) takes more time to execute, then it will be finished later. Which means the application won't apply bold font on anything, because text selection is not there yet.

Event-indicator method

What is the solution here? What we need is the event-indicator method. It's very simple. After every event, we trigger in the test code, we should use an indicator that the event is actually processed. Let's see how we can fix the previous scenario:

// Select all text
cy.get('textarea.clipboard')
 .type('{ctrl}a')

// Wait for the text selection to appear
cy.get('.leaflet-marker-icon')
 .should('exist');

// Apply bold font
cy.get('#tb_editbar_item_bold')
 .click();

The '.leaflet-marker-icon' DOM element is part of the text selection markup in the application, so we can use that as an indicator. In cypress, every cy.get().should() call has an implicit retry feature. It tries to get the related DOM element, again and again, until it appears in the DOM (or until a given timeout). Also, it tries to check the assumption described by the should() method, again and again, until the DOM properties meet with this assumption (or until a given timeout). So this specific cy.get().should() call will wait until the selection appears on the client-side. After that, it's safe to apply bold font because we have the selection to apply on. It's a best practice to use an indicator after every event, so you can make sure that the events are processed in the right order.

Bad pracitice

Someone might think of using some kind of waiting after a simulated event, so the application has time to finish the processing. Like bellow:

// Select all text
cy.get('textarea.clipboard')
 .type('{ctrl}a')

// Wait for the text selection to appear (DONT USE THIS)
cy.wait(500);

// Apply bold font
cy.get('#tb_editbar_item_bold')
 .click();

I think it's easy to see why this is not effective. If this constant time is too small, then this test will fail on slower machines or on the same machine when it's overloaded. On the other side, if this constant is too big, then the test will be slow. So if we use a big enough value to make the tests passing on all machines, then these tests will be as slow as they would be on the slowest machine.

Using an indicator makes the test framework more effective. Running these tests will be faster on faster machines since it waits only until the indicator is found in the DOM, which is fast in this case. It also works on slower machines since we have a relatively big timeout for cypress commands (6000 ms), so it's not a problem if it takes time to apply the selection. All in all, we should try to minimize the usage of cy.wait() method in our test code.

One more thing: then() is not an indicator

It's good to know, that not every cypress command can be used as an indicator. For example, we often use then() method, which yields the selected DOM element, so we can work with that item in the test code. An example of this is the following, where we get a specific <p> item and try to check whether it has the specified text content:

cy.get('#copy-paste-container p')
 .then(function(item) {
  expect(item).to.have.lengthOf(1);
  expect(item[0].innerText).to.have.string('\u00a0');
 });

This method call will retry getting the <p> element until it appears in the DOM. However, it won't retry the assumptions specified with expect methods. It will check these expectations only once, when it gets the <p> item first. If we would like to wait for the <p> item to match with the assumptions, then we should use should() method instead.

cy.get('#copy-paste-container p')
 .should('contain.text', '\u00a0');

This method will retry to find a <p> item meet both the selector('#copy-paste-container p') and the assumption. In general, it's always better to use a should() method where it possible. However sometimes then() is also needed. We can use that safely if we use an indicator before it and so we can make sure we check the DOM element at the right time.

In the example below, we have a cy.get('.blinking-cursor') call. If we know that before the text selection we did not have the blinking cursor, then this cy.get() call is a good indicator so we can use then() after that safely.

// Select all text
cy.get('textarea.clipboard')
 .type('{ctrl}a')

// Use blinking cursor position
cy.get('.blinking-cursor')
 .then(function(cursors) {
  var posX = cursors[0].getBoundingClientRect().left;
  ... // Do something with this position
 })

Summary

I hope this short description of the event-indicator method and cypress retry feature is useful to understand how these cypress tests work. This simple rule, to always use indicators, can make things much easier when somebody writes new tests or modifies the existing ones. Sometimes the application does not provide us an obvious indicator and we need to be creative to find one.

Friday, March 27, 2020

New integration test framework in Collabora Online.

Introduction

At Collabora, we invest a lot of hard work to make LibreOffice's features available in an online environment. Recently we greatly improved the Collabora Online mobile UI, so it's more smooth to use it from a mobile device. While putting more and more work into the software, trying to support more and more different platforms, we need also to spend time improving the test frameworks we use for automatic testing. These test frameworks make sure that while we enrich the software with new features, the software remains stable during the continuous development process.

End-to-end testing in the browser

One step on this road was the integration of cypress.io test framework into Collabora Online code. cypress.io is an end-to-end test run in the browser, so any online application can be tested with it. It mainly allows us to simulate user interaction with the UI and check the event's results via the website's DOM elements. That allows us to simulate user stories and catch real-life issues in the software, so our quality measurement matches the actual user experience.

When I investigated the different alternatives for browser testing I also checked the Selenium test framework. I didn't spend more than two days on that, but I had an impression that Selenium is kind of "old" software, which tries to support many configurations, many language bindings which makes it hard to use and also makes it stuck in the current state, where it is. While cypress.io is a newer test framework, which seems more focused. It is easier to integrate into our build system and easier to use, which is a big plus because it's not enough to integrate a test framework, but developers need to learn how to use it too. I saw one advantage of Selenium: the better browser support. It supports all the main browsers (Chrome, Mozilla Firefox, Safari, Internet Explorer), while cypress.io mainly supports only Chrome, but it improves on this area. Now it has a beta Mozilla Firefox support. So finally I decided to use cypress.io and I'm happy I did that because it nicely works.

cypress.io in Collabora Online

So cypress.io is now integrated into the Collabora Online code and we already have 150 tests mainly for mobile UI. As we submit most of our work to upstream, these tests are also available in the community version of the software. It's integrated into the software's GNU make based build system, so a make check will run these tests automatically. This is also part of the continuous integration system, so we can catch any regression instantly, before it actually hits the code. It's recommended to all developers of the online code to get familiar with the test framework, so it will be easier to understand if a test failure indicates an issue in their proposed patch. There are a set of useful notes in the source code, in the readme file: [source_dir]/cypress_test/README. Next to that, I try to add some good advice in the following paragraphs, how to investigate if any cypress test is failing on your patch.

How to check a test failure?


Interactive test runner

When you run make check the cypress tests are run in headless mode, so you can't see what happens on the UI, while the tests are running. If you see a test failure, the easiest way to understand what happens is to check it in the interactive test runner. To do that you can call make run-mobile or make run-desktop depending on what kind of test you are interested in. In interactive mode, you'll get a window, where you can select a test suite (*_spec.js file) and run that test suite only.

After you select a test suite you'll see the tests running in the browser. It's fast so probably you can't follow all the steps, but after the tests are finished you can select the steps and check screenshot for every step, so you can follow the state of the application. This way you can see how the application gets to a failure state.

Can't reproduce a failure in the interactive test runner

Sometimes, it happens that a test failure is reproducible only in headless mode. There are more options, that you can do in this case. First, you can check a screenshot taken at the point when the test failed. This screenshot is added automatically into a separate folder:

[source_dir]/cypress_test/cypress/screenshots/

This screenshot shows only the failure state, which might not be enough. You can also use the cypress command log to write out important information into the console during a test run. You can do that using the cy.log() method, called from the JS test code (this is not equivalent to console.log() method). In the case of test failure, these logs are dumped on the console output. These logs are also available here:

[source_dir]/cypress_test/cypress/logs/

A third option is to enable video recording. With video recording, the cypress test framework will generate a video of the test run, where you can see the same thing that you would see in the interactive test runner. To enable video recording you need to remove "video" : false, line from [source_dir]/cypress_test/cypress.json file. After that, running make check will record videos for all test suites you are running and put them under videos folder:

[source_dir]/cypress_test/cypress/videos/

How to run only one test?

To run one test suite you can use the spec option:

make check-mobile spec=writer/apply_font_spec.js

This spec option can be used with check-mobile and check-desktop rules, depending on what kind of test you intend to run. This is the headless run, but in the interactive test runner, you also can do that by selecting one test suite from the list. With these options, you can run a test suite, but a test suite means more tests. If you would like to run only one test you need to combine a test suite run, with using only() method. You need to open the test suite file and add only() to the definition of the specific test case:

- it('Apply font name.', function() {
+ it.only('Apply font name.', function() {

So both the headless build and the interactive test runner will ignore any other tests in the same test suite. It's useful when somebody investigates why a specific test fails.

Summary

So that's it for now. I hope these pieces of information are useful for getting familiar with the new test framework. Fortunately, the cypress.io test framework provides us nice tools to write new tests and check test failures. I'm still working on the test framework to customize it to our online environment. Hopefully, using this test framework will improve software quality in the long term.