Skip to main content

Tests

Want help writing your test steps? Check out the Action Builder prototype.

Tests tell Doc Detective what actions to perform, how, and where. Tests are made of three sets of components:

  • Test specification: The highest-level component, test specifications (or specs) are collection of tests that should run together. Contexts defined in the spec are shared by all tests in the spec. Test specifications are equivalent to test suites in other testing frameworks.

  • Test: A test to run within a spec. Each test has a name, and a set of steps to perform. Tests are equivalent to test cases in other testing frameworks.

  • Steps: A step is a single action to perform within a test. Each individual step acts as an assertion that the step completes as expected. Steps are equivalent to assertions in other testing frameworks.

    Each step has an action, which is a command that tells Doc Detective what to do. Actions can have additional properties that further define the action.

    • checkLink: Check if a URL returns an acceptable status code from a GET request.
    • find: Check if an element exists with the specified selector.
    • goTo: Navigate to a specified URL.
    • httpRequest: Perform a generic HTTP request, for example to an API.
    • runShell: Perform a native shell command.
    • saveScreenshot: Take a screenshot in PNG format.
    • setVariables: Load environment variables from a .env file.
    • startRecording and stopRecording: Capture a video of test execution.
    • typeKeys: Type keys. To type special keys, begin and end the string with $ and use the special key’s enum. For example, to type the Escape key, enter $ESCAPE$.
    • wait: Pause before performing the next action.

Define a test​

You can define test specs in multiple ways:

  • Standalone JSON files make it easy to see the entirety of the spec and keep it in a separate, machine-readable format.
  • Inline JSON allows you to define tests directly in your documentation source files.
  • Detected tests are automatically generated by Doc Detective based on your documentation source files and your configuration.

Standalone JSON​

Test specs in standalone JSON files use the following basic structure:

{
  "id": "spec-id",
  "description": "Spec Description",
  "tests": [
    {
      "id": "test-id",
      "description": "Test Description",
      "steps": [
        {
          "id": "step-id",
          "description": "Step Description",
          "action": "action-name"
          // Additional step properties
        }
      ]
    }
  ]
}

For comprehensive options, see specification.

Here's an example test for performing a Google search and saving a screenshot of the results:

{
  "tests": [
    {
      "steps": [
        {
          "action": "goTo",
          "url": "https://www.google.com"
        },
        {
          "action": "find",
          "selector": "[title=Search]",
          "click": true
        },
        {
          "action": "typeKeys",
          "keys": ["American Shorthair kittens", "$ENTER$"]
        },
        {
          "action": "wait",
          "duration": 5000
        },
        {
          "action": "saveScreenshot",
          "path": "search-results.png"
        }
      ]
    }
  ]
}

Inline JSON​

You can define tests directly in your documentation source files using inline JSON. Inline JSON is useful for small, simple tests that you want to keep close to the content they test.

Inline tests are also excellent for maintaining tests because they are easy to update and keep in sync with the content they test as the content changes.

Inline tests depend on your config's fileType definitions. This page uses the default Markdown configuration, but you should update your config to match your documentation source files.

Inline tests use specially formatted comments with JSON objects to declare the tests and steps. Doc Detective reads the input file, line by line, and extracts the tests and steps from the comments. You can declare multiple tests and steps in a single file, using your config's test start, test end, step, and ignore patterns.

All test and step comments must be on a single line. Doc Detective doesn't support multi-line comments.

When you declare a test, you can specify any of the test properties available in the standalone JSON format, though steps is normally omitted. See test.

If you declare a new test without closing the previous test, Doc Detective automatically closes the previous test before starting the new one.

When you declare a step, you can specify any of the properties of the action you want to perform. See Actions.

If you declare a step without declaring a test, Doc Detective automatically creates a test to contain the step.

Here's an example of an inline test for performing a Google search and saving a screenshot of the results:

[comment]: # 'test {"id": "kitten-search"}'

To search for American Shorthair kittens,

1. Go to [Google](https://www.google.com).

   [comment]: # 'step {"action":"goTo", "url":"https://www.google.com"}'

2. In the search bar, enter "American Shorthair kittens", then press Enter.

   [comment]: # 'step { "action": "find", "selector": "[title=Search]", "click": true }'
   [comment]: # 'step { "action": "typeKeys", "keys": ["American Shorthair kittens", "$ENTER$"] }'
   [comment]: # 'step { "action": "wait", "duration": 5000 }'

![Search results](search-results.png)

[comment]: # 'step { "action": "saveScreenshot", "path": "search-results.png" }'
[comment]: # "test end"

Detected tests​

Doc Detective can automatically generate tests based on your documentation source files and your fileTypes configuration. Detected tests are useful for large, complex test suites that you want to keep in sync with your documentation. Test detection works by setting runTests.detectSteps to true and defining markup patterns and associated actions in the fileTypes.markup array in your config, which Doc Detective uses to extract steps from your doc source files. You can define multiple test patterns in your config to extract different types of tests from your documentation.

Detected tests are useful for keeping your tests in sync with your documentation. When you update your documentation, Doc Detective automatically updates the tests based on the new content. Detected tests are generated automatically. You can't edit detected tests directly, but you can update your config or documentation source files to change the tests.

You can mix detected tests with inline tests to declare steps that might not be covered in your content, such starting or stopping a recording.

For example, markup for Markdown files might look like this:

{
  ...
  "runTests": {
    "detectSteps": true
  },
  "fileTypes": [
    {
      ...
      "markup": [
        {
          "name": "Hyperlink",
          "regex": ["(?<!!)\\[.+?\\]\\(.+?\\)"],
          "actions": ["checkLink"]
        },
        {
          "name": "Navigation link",
          "regex": [
            "(?:[Cc]hose|[Oo]pen|[Cc]lick|[Nn]avigate to|[Gg]o to)(?<!!)\\s+\\[.+?\\]\\(.+?\\)"
          ],
          "actions": ["goTo"]
        },
        {
          "name": "Onscreen text",
          "regex": ["\\*\\*(.+?)\\*\\*"],
          "actions": ["find"]
        },
        {
          "name": "Click",
          "regex": ["(?:[Cc]lick|[Pp]ress|[Cc]hoose|[Tt]ap)\\s+\\*\\*(.+?)\\*\\*"],
          "actions": [{
            "action": "find",
            "description": "Click $1",
            "selector": "aria/$1",
            "click": true
          }]
        },
        {
          "name": "Image",
          "regex": ["!\\[.+?\\]\\((.+?)\\)"],
          "actions": [
            {
              "action": "saveScreenshot",
              "path": "$1",
              "directory": "samples",
              "maxVariation": 5,
              "overwrite": "byVariation"
            }
          ]
        }
      ]
      ...
    }
  ],
  ...
}

The regex property defines the regular expression patterns to match, and the actions property defines the actions to perform when the pattern is detected. With the above config, Doc Detective would take the following Markdown and generate tests for every hyperlink, navigation link, onscreen text, and image.

Markdown:

To get started,

1. Go to [Acme Console](https://console.acme.com).
2. Press **Search**.

![Search results](search-results.png)

Detected tests:

{
  "tests": [
    {
      "steps": [
        {
          "action": "goTo",
          "url": "https://console.acme.com"
        },
        {
          "action": "find",
          "selector": "aria/Search",
          "click": true
        },
        {
          "action": "saveScreenshot",
          "path": "search-results.png"
        }
      ]
    }
  ]
}

Default actions​

If you only need to perform simple steps, you can define actions using a shorthand of only specifying the action name. Doc Detective uses default actions, substituting the first capture group (or if there are no capture groups, the entire match) from the regex pattern into the most common location for the action. In the above example, the config generates a goTo action for every Markdown hyperlink preceded by a navigation verb.

The default actions are as follows:

Check that the match returns a valid status code.

{
  "action": "checkLink",
  "url": "$1"
}

goTo​

Open the match as a URL in a browser.

{
  "action": "goTo",
  "url": "$1"
}

find​

Find an element on the current page that has an ARIA label that equals the match.

{
  "action": "find",
  "selector": "aria/$1"
}

saveScreenshot​

Save a screenshot to a path equalling the match.

{
  "action": "saveScreenshot",
  "path": "$1"
}

typeKeys​

Type the match into the current page.

{
  "action": "typeKeys",
  "keys": "$1"
}

httpRequest​

Make an GET request to the match.

{
  "action": "httpRequest",
  "url": "$1"
}

runShell​

Run the match as a shell command.

{
  "action": "runShell",
  "command": "$1"
}

startRecording​

Start recording a video to a path equalling the match.

{
  "action": "startRecording",
  "path": "$1"
}

stopRecording​

Stop recording a video.

{
  "action": "stopRecording"
}

wait​

Wait for the specified duration.

{
  "action": "wait",
  "duration": "$1"
}

setVariables​

Load environment variables from an .env file, where the path is the match.

{
  "action": "setVariables",
  "path": "$1"
}

Run tests​

Doc Detective's runTest command runs your tests. Input files are read from your config's input and runTests.input properties, but you can also specify input files directly in the command with the --input flag.

This example runs all test specs in your config's input and runTest.input parameters:

npx doc-detective runTests

This example runs all test specs in a file named doc-content.md in the samples directory:

npx doc-detective runTests --input ./samples/doc-content.md

Remotely hosted tests​

You can run tests hosted remotely by specifying the URL of the test file with the --input argument. For example, to run tests from a file hosted at https://doc-detective.com/sample.spec.json, run the following command:

npx doc-detective runTests --input https://doc-detective.com/sample.spec.json

These tests run the same way as local tests, but Doc Detective fetches the test file from the specified URL and stores it in a temporary directory. The URL must be accessible to the machine running the tests.

Read the results​

Doc Detective outputs test results to a testResults-<timestamp>.json file in your output or runTests.output directory. You can also specify your output directory with the --output flag:

npx doc-detective runTests --input ./samples/doc-content.md --output ./samples/