Tests

Want a helping hand writing your test steps? Check out the Action Builder prototype.

Tests tell Doc Detective what actions to perform, how, and where. Tests are made of three sets of components:

Test specification: The highest-level component, test specifications (or specs) are collection of tests that should run together. Contexts defined in the spec are shared by all tests in the spec. Test specifications are equivalent to test suites in other testing frameworks.
Test: A test to run within a spec. Each test has a name, and a set of steps to perform. Tests are equivalent to test cases in other testing frameworks.
Steps: A step is a single action to perform within a test. Each individual step acts as an assertion that the step completes as expected. Steps are equivalent to assertions in other testing frameworks.

Each step has an action, which is a command that tells Doc Detective what to do. Actions can have additional properties that further define the action.
- checkLink: Check if a URL returns an acceptable status code from a GET request.
- find: Check if an element exists with the specified selector.
- goTo: Navigate to a specified URL.
- httpRequest: Perform a generic HTTP request, for example to an API.
- runShell: Perform a native shell command.
- saveScreenshot: Take a screenshot in PNG format.
- setVariables: Load environment variables from a .env file.
- startRecording and stopRecording: Capture a video of test execution.
- typeKeys: Type keys. To type special keys, begin and end the string with $ and use the special key’s enum. For example, to type the Escape key, enter $ESCAPE$ .
- wait: Pause before performing the next action.

Define a test

You can define test specs in multiple ways:

Standalone JSON files make it easy to see the entirety of the spec and keep it in a separate, machine-readable format.
Inline JSON allows you to define tests directly in your documentation source files.
Detected tests are automatically generated by Doc Detective based on your documentation source files and your configuration.

Standalone JSON

Test specs in standalone JSON files use the following basic structure:

{
  "id": "spec-id",
  "description": "Spec Description",
  "tests": [
    {
      "id": "test-id",
      "description": "Test Description",
      "steps": [
        {
          "id": "step-id",
          "description": "Step Description",
          "action": "action-name"
          // Additional step properties
        }
      ]
    }
  ]
}

For comprehensive options, see specification.

Here’s an example test for performing a Google search and saving a screenshot of the results:

{
  "tests": [
    {
      "steps": [
        {
          "action": "goTo",
          "url": "https://www.google.com"
        },
        {
          "action": "find",
          "selector": "[title=Search]",
          "click": true
        },
        {
          "action": "typeKeys",
          "keys": ["American Shorthair kittens", "$ENTER$"]
        },
        {
          "action": "wait",
          "duration": 5000
        },
        {
          "action": "saveScreenshot",
          "path": "search-results.png"
        }
      ]
    }
  ]
}

Inline JSON

You can define tests directly in your documentation source files using inline JSON. Inline JSON is useful for small, simple tests that you want to keep close to the content they test.

Inline tests are also excellent for maintaining tests because they are easy to update and keep in sync with the content they test as the content changes.

Inline tests depend on your config’s fileType definitions. This page uses the default Markdown configuration, but you should update your config to match your documentation source files.

Inline tests use specially formatted comments with JSON objects to declare the tests and steps. Doc Detective reads the input file, line by line, and extracts the tests and steps from the comments. You can declare multiple tests and steps in a single file, using your config’s test start, test end, step, and ignore patterns.

All test and step comments must be on a single line. Doc Detective doesn’t support multi-line comments.

When you declare a test, you can specify any of the test properties available in the standalone JSON format, though steps is normally omitted. See test.

If you declare a new test without closing the previous test, Doc Detective automatically closes the previous test before starting the new one.

When you declare a step, you can specify any of the properties of the action you want to perform. See Actions.

If you declare a step without declaring a test, Doc Detective automatically creates a test to contain the step.

Here’s an example of an inline test for performing a Google search and saving a screenshot of the results:

[comment]: # (test {"id": "kitten-search"})

To search for American Shorthair kittens,

1. Go to [Google](https://www.google.com).

   [comment]: # (step {"action":"goTo", "url":"https://www.google.com"})

2. In the search bar, enter "American Shorthair kittens", then press Enter.

   [comment]: # (step { "action": "find", "selector": "[title=Search]", "click": true })
   [comment]: # (step { "action": "typeKeys", "keys": ["American Shorthair kittens", "$ENTER$"] })
   [comment]: # (step { "action": "wait", "duration": 5000 })

![Search results](search-results.png)

[comment]: # (step { "action": "saveScreenshot", "path": "search-results.png" })
[comment]: # (test end)

Detected tests

Doc Detective automatically generates tests based on your documentation source files and your fileTypes configuration. Detected tests are useful for large, complex test suites that you want to keep in sync with your documentation.

Detected tests are generated based on the test patterns you define in your config. Doc Detective reads your documentation source files, looking for test patterns, and extracts the tests and steps from the files.

Detected tests are generated automatically by Doc Detective. You can’t edit detected tests directly, but you can update your documentation source files to change the tests.

You can use inline tests to supplement detected tests and declare steps that might not be covered in your content, such starting or stopping a recording.

Detected tests are useful for keeping your tests in sync with your documentation. When you update your documentation, Doc Detective automatically updates the tests based on the new content.

Run tests

Doc Detective’s runTest command runs your tests. Input files are read from your config’s input and runTests.input properties, but you can also specify input files directly in the command with the --input flag.

This example runs all test specs in your config’s input and runTest.input parameters:

npx doc-detective runTests

This example runs all test specs in a file named doc-content.md in the samples directory:

npx doc-detective runTests --input ./samples/doc-content.md

Read the results

Doc Detective outputs test results to a testResults-<timestamp>.json file in your output or runTests.output directory. You can also specify your output directory with the --output flag:

npx doc-detective runTests --input ./samples/doc-content.md --output ./samples/

Actions