Skip to main content

Tests

Want help writing your test steps? Check out the Action Builder prototype.

Tests tell Doc Detective what actions to perform, how, and where. Tests are made of three sets of components:

  • Test specification: The highest-level component, test specifications (or specs) are collection of tests that should run together. Contexts defined in the spec are shared by all tests in the spec. Test specifications are equivalent to test suites in other testing frameworks.

  • Test: A test to run within a spec. Each test has a name, and a set of steps to perform. Tests are equivalent to test cases in other testing frameworks.

  • Steps: A step is a single action to perform within a test. Each individual step acts as an assertion that the step completes as expected. Steps are roughly equivalent to assertions in other testing frameworks.

    Each step can perform an action, which is a command that tells Doc Detective what to do. Actions can have additional properties that further define the action.

    • checkLink: Check if a URL returns an acceptable status code from a GET request.
    • click: Click or tap an element.
    • find: Check if an element exists with the specified selector.
    • goTo: Navigate to a specified URL.
    • httpRequest: Perform a generic HTTP request, for example to an API.
    • loadVariables: Load environment variables from a .env file.
    • record and stopRecord: Capture a video of test execution.
    • runCode: Assemble and run code.
    • runShell: Perform a native shell command.
    • screenshot: Take a screenshot in PNG format.
    • type: Type keys. To type special keys, begin and end the string with $ and use the special key’s enum. For example, to type the Escape key, enter $ESCAPE$.
    • wait: Pause before performing the next action.

Define a test​

You can define test specs in multiple ways:

  • Standalone JSON files make it easy to see the entirety of the spec and keep it in a separate, machine-readable format.
  • Inline JSON allows you to define tests directly in your documentation source files.
  • Detected tests are automatically generated by Doc Detective based on your documentation source files and your configuration.

Standalone JSON or YAML​

Test specs in standalone JSON or YAML files use the following basic structure:

{
  "specId": "spec-id",
  "description": "Spec Description",
  "tests": [
    {
      "testId": "test-id",
      "description": "Test Description",
      "steps": [
        {
          "stepId": "step-id",
          "description": "Step description. This object also needs additional step properties like `find`, `goTo`, or `httpRequest`."
        }
      ]
    }
  ]
}

For comprehensive options, see specification.

Here's an example test for performing a Google search and saving a screenshot of the results:

{
  "tests": [
    {
      "steps": [
        {
          "goTo": "https://www.google.com"
        },
        {
          "find": {
            "selector": "[title=Search]",
            "click": true
          }
        },
        {
          "type": [
            "American Shorthair kittens",
            "$ENTER$"
          ]
        },
        {
          "wait": 5000
        },
        {
          "screenshot": "search-results.png"
        }
      ]
    }
  ]
}

Inline JSON or YAML​

You can define tests directly in your documentation source files using inline JSON or YAML. Inline tests are useful for small, simple tests that you want to keep close to the content they test.

Inline tests are also excellent for maintaining tests because they are easy to update and keep in sync with the content they test as the content changes.

Inline tests depend on your config's fileType definitions. This page uses the default Markdown configuration, but you should update your config to match your documentation source files.

Inline tests use specially formatted comments with JSON or YAML objects to declare the tests and steps. Doc Detective reads the input file, line by line, and extracts the tests and steps from the comments. You can declare multiple tests and steps in a single file, using your config's test start, test end, step, and ignore patterns.

All test and step comments must be on a single line. Doc Detective doesn't support multi-line comments.

When you declare a test, you can specify any of the test properties available in the standalone JSON format, though steps is normally omitted. See test.

If you declare a new test without closing the previous test, Doc Detective automatically closes the previous test before starting the new one.

When you declare a step, you can specify any of the properties of the action you want to perform. See Actions.

If you declare a step without declaring a test, Doc Detective automatically creates a test to contain the step.

Comment patterns by file type​

Doc Detective supports different comment patterns depending on your file type:

Markdown files (.md, .markdown, .mdx):

  • JSX-style comments: {/* test { "testId": "my-test" } */}
  • HTML comments: <!-- test { "testId": "my-test" } -->
  • Markdown link comments: [comment]: # (test { "testId": "my-test" })
  • Step comments: {/* step { "goTo": "https://example.com" } */}
  • Test end: {/* test end */} or <!-- test end --> or [comment]: # (test end)
  • Ignore blocks: {/* test ignore start */} and {/* test ignore end */}

HTML files (.html, .htm):

  • Test comments: <!-- test { "testId": "my-test" } -->
  • Step comments: <!-- step { "goTo": "https://example.com" } -->
  • Test end: <!-- test end -->
  • Ignore blocks: <!-- test ignore start --> and <!-- test ignore end -->

AsciiDoc files (.adoc, .asciidoc, .asc):

  • Line comments: // (test { "testId": "my-test" })
  • Step comments: // (step { "goTo": "https://example.com" })
  • Test end: // (test end)
  • Ignore blocks: // (test ignore start) and // (test ignore end)

Examples​

Here's an example using JSON syntax in Markdown:

[comment]: # 'test {"testId": "kitten-search"}'

To search for American Shorthair kittens,

1. Go to [Google](https://www.google.com).

   [comment]: # 'step {"goTo": "https://www.google.com"}'

2. In the search bar, enter "American Shorthair kittens", then press Enter.

   [comment]: # 'step { "find": { "selector": "[title=Search]", "click": true } }'
   [comment]: # 'step { "type": ["American Shorthair kittens", "$ENTER$"] }'
   [comment]: # 'step { "wait": 5000 }'

![Search results](search-results.png)

[comment]: # 'step { "screenshot": "search-results.png" }'
[comment]: # "test end"

Here's the same example using YAML syntax in Markdown:

{/_ test testId: kitten-search _/}

To search for American Shorthair kittens,

1. Go to [Google](https://www.google.com).

   {/_ step goTo: https://www.google.com _/}

2. In the search bar, enter "American Shorthair kittens", then press Enter.

   {/_ step find: selector: "[title=Search]" click: true _/} {/_ step type: - "American Shorthair kittens" - "$ENTER$" _/} {/_ step wait: 5000 _/}

![Search results](search-results.png)

{/_ step screenshot: search-results.png _/} {/_ test end _/}

HTML example:

<!-- test { "testId": "navigation-test" } -->
<p>Navigate to our homepage:</p>
<!-- step { "goTo": "https://example.com" } -->

<p>Click the menu button:</p>
<!-- step { "click": "Menu" } -->

<!-- test end -->

AsciiDoc example:

// (test { "testId": "doc-test" })
== Getting Started

Navigate to the application:
// (step { "goTo": "https://app.example.com" })

Click the *Start* button:
// (step { "click": "Start" })

// (test end)

Detected tests​

Doc Detective can automatically generate tests based on your documentation source files and your fileTypes configuration. Detected tests are useful for large, complex test suites that you want to keep in sync with your documentation. Test detection works by setting detectSteps to true and defining markup patterns and associated actions in the fileTypes.markup array in your config, which Doc Detective uses to extract steps from your doc source files. You can define multiple test patterns in your config to extract different types of tests from your documentation.

Detected tests are useful for keeping your tests in sync with your documentation. When you update your documentation, Doc Detective automatically updates the tests based on the new content. Detected tests are generated automatically. You can't edit detected tests directly, but you can update your config or documentation source files to change the tests.

You can mix detected tests with inline tests to declare steps that might not be covered in your content, such starting or stopping a recording.

Doc Detective includes several default markup patterns for Markdown files. Here are all the built-in patterns:

{
  "fileTypes": [
    {
      "name": "markdown",
      "extensions": ["md", "markdown", "mdx"],
      "markup": [
        {
          "name": "checkHyperlink",
          "regex": [
            "(?<!\\!)\\[[^\\]]+\\]\\(\\s*(https?:\\/\\/[^\\s)]+)(?:\\s+\"[^\"]*\")?\\s*\\)"
          ],
          "actions": ["checkLink"]
        },
        {
          "name": "clickOnscreenText",
          "regex": [
            "\\b(?:[Cc]lick|[Tt]ap|[Ll]eft-click|[Cc]hoose|[Ss]elect|[Cc]heck)\\b\\s+\\*\\*((?:(?!\\*\\*).)+)\\*\\*"
          ],
          "actions": ["click"]
        },
        {
          "name": "findOnscreenText",
          "regex": ["\\*\\*((?:(?!\\*\\*).)+)\\*\\*"],
          "actions": ["find"]
        },
        {
          "name": "goToUrl",
          "regex": [
            "\\b(?:[Gg]o\\s+to|[Oo]pen|[Nn]avigate\\s+to|[Vv]isit|[Aa]ccess|[Pp]roceed\\s+to|[Ll]aunch)\\b\\s+\\[[^\\]]+\\]\\(\\s*(https?:\\/\\/[^\\s)]+)(?:\\s+\"[^\"]*\")?\\s*\\)"
          ],
          "actions": ["goTo"]
        },
        {
          "name": "screenshotImage",
          "regex": [
            "!\\[[^\\]]*\\]\\(\\s*([^\\s)]+)(?:\\s+\"[^\"]*\")?\\s*\\)\\s*\\{(?=[^}]*\\.screenshot)[^}]*\\}"
          ],
          "actions": ["screenshot"]
        },
        {
          "name": "typeText",
          "regex": ["\\b(?:press|enter|type)\\b\\s+\"([^\"]+)\""],
          "actions": ["type"]
        },
        {
          "name": "httpRequestFormat",
          "regex": [
            "```(?:http)?\\r?\\n([A-Z]+)\\s+([^\\s]+)(?:\\s+HTTP\\/[\\d.]+)?\\r?\\n((?:[^\\s]+:\\s+[^\\s]+\\r?\\n)*)?(?:\\s+([\\s\\S]*?)\\r?\\n+)?```"
          ],
          "actions": [
            {
              "httpRequest": {
                "method": "$1",
                "url": "$2",
                "request": {
                  "headers": "$3",
                  "body": "$4"
                }
              }
            }
          ]
        },
        {
          "name": "runCode",
          "regex": [
            "```(bash|python|py|javascript|js)(?![^\\r\\n]*testIgnore)[^\\r\\n]*\\r?\\n([\\s\\S]*?)\\r?\\n```"
          ],
          "actions": [
            {
              "unsafe": true,
              "runCode": {
                "language": "$1",
                "code": "$2"
              }
            }
          ]
        }
      ]
    }
  ]
}

Default markup pattern descriptions​

checkHyperlink: Detects standard Markdown links (excluding images) and checks that they return a valid HTTP status code.

  • Pattern: [link text](https://example.com)
  • Action: Performs a GET request to verify the link is accessible

clickOnscreenText: Detects action verbs followed by bold text and clicks on that element.

  • Pattern: Click **Button Name**, Tap **Menu Item**, Select **Option**
  • Action: Clicks on the specified element using its text content

findOnscreenText: Detects any bold text and verifies it exists on the page.

  • Pattern: **Any Bold Text**
  • Action: Searches for the text on the current page

goToUrl: Detects navigation instructions with links and opens the URL.

  • Pattern: Go to [Page Name](https://example.com), Navigate to [Site](https://example.com)
  • Action: Opens the specified URL in the browser

screenshotImage: Detects images with a .screenshot class or attribute and takes a screenshot.

  • Pattern: ![Alt text](image.png){.screenshot}
  • Action: Saves a screenshot to the specified path

typeText: Detects typing instructions with quoted text and types the content.

  • Pattern: Type "Hello World", Enter "username"
  • Action: Types the specified text into the active element

httpRequestFormat: Detects HTTP request code blocks and executes the request.

  • Pattern: HTTP request in code blocks with method, URL, headers, and body
  • Action: Executes the HTTP request with the specified parameters

runCode: Detects code blocks in supported languages and executes the code.

  • Pattern: Code blocks with bash, python, py, javascript, or js language tags (excluding those marked with testIgnore)
  • Action: Executes the code in the specified language (marked as unsafe)

The regex property defines the regular expression patterns to match, and the actions property defines the actions to perform when the pattern is detected. With the default configuration, Doc Detective would take the following Markdown and generate tests automatically:

Markdown:

To get started,

1. Go to [Acme Console](https://console.acme.com).
2. Click **Search**.
3. Type "American Shorthair kittens".

Check out our [documentation](https://docs.example.com) for more information.

![Search results](search-results.png){ .screenshot }

Detected tests:

{
  "tests": [
    {
      "steps": [
        {
          "goTo": "https://console.acme.com"
        },
        {
          "click": "Search"
        },
        {
          "type": "American Shorthair kittens"
        },
        {
          "checkLink": "https://docs.example.com"
        },
        {
          "screenshot": "search-results.png"
        }
      ]
    }
  ]
}

Custom markup patterns​

You can extend or override the default patterns by defining custom markup patterns in your configuration:

{
  "fileTypes": [
    {
      "name": "markdown",
      "extensions": ["md", "markdown", "mdx"],
      "markup": [
        {
          "name": "customAction",
          "regex": ["\\b(?:custom)\\b\\s+\\*\\*(.+?)\\*\\*"],
          "actions": [
            {
              "find": {
                "selector": "aria/$1",
                "timeout": 10000
              }
            }
          ]
        }
      ]
    }
  ]
}

Other file types​

Doc Detective also includes default configurations for HTML and AsciiDoc files, though these only include inline statement patterns for test definitions and don't include markup patterns for detected tests:

HTML files (.html, .htm):

  • Uses HTML comment syntax for inline test statements
  • Example: <!-- test { "testId": "my-test" } -->

AsciiDoc files (.adoc, .asciidoc, .asc):

  • Uses AsciiDoc comment syntax for inline test statements
  • Example: // (test { "testId": "my-test" })

To add detected test functionality to HTML or AsciiDoc files, you would need to define custom markup patterns in your configuration similar to the Markdown examples above.

Default actions​

If you only need to perform simple steps, you can define actions using a shorthand of only specifying the action name. Doc Detective uses default actions, substituting the first capture group (or if there are no capture groups, the entire match) from the regex pattern into the most common location for the action. In the above example, the config generates a goTo action for every Markdown hyperlink preceded by a navigation verb.

The default actions are as follows:

Check that the match returns a valid status code.

{ "checkLink": "$1" }

click​

Click or tap the match.

{ "click": "$1" }

goTo​

Open the match as a URL in a browser.

{ "goTo": "$1" }

find​

Find an element on the current page that has an ARIA label that equals the match.

{ "find": "$1" }

screenshot​

Save a screenshot to a path equalling the match.

{ "screenshot": "$1" }

type​

Type the match into the current page.

{ "type": "$1" }

httpRequest​

Make an GET request to the match.

{ "httpRequest": "$1" }

runShell​

Run the match as a shell command.

{ "runShell": "$1" }

record​

Start recording a video to a path equalling the match.

{ "record": "$1" }

stopRecord​

Stop recording a video.

{ "stopRecord": true }

wait​

Wait for the specified duration.

{ "wait": "$1" }

loadVariables​

Load environment variables from an .env file, where the path is the match.

{ "loadVariables": "$1" }

Run tests​

Doc Detective's doc-detective command runs your tests. Input files are read from your config's input property, but you can also specify input files directly in the command with the --input flag.

This example runs all test specs in your config's input and runTest.input parameters:

npx doc-detective

This example runs all test specs in a file named doc-content.md in the samples directory:

npx doc-detective --input ./samples/doc-content.md

Remotely hosted tests​

You can run tests hosted remotely by specifying the URL of the test file with the --input argument. For example, to run tests from a file hosted at https://doc-detective.com/sample.spec.json, run the following command:

npx doc-detective --input https://doc-detective.com/sample.spec.json

These tests run the same way as local tests, but Doc Detective fetches the test file from the specified URL and stores it in a temporary directory. The URL must be accessible to the machine running the tests.

Read the results​

Doc Detective outputs test results to a testResults-<timestamp>.json file in your output directory. You can also specify your output directory with the --output flag:

npx doc-detective --input ./samples/doc-content.md --output ./samples/