Test runs

A test run turns a script into evidence. The agent wraps a flow in astral.test, splits it into steps, states what should be true, and Astral records what happened.

Test runs use the run_script tool, which runs agent-written JavaScript in a sandbox that calls the other tools. The script group is off by default; the tests group is on by default but only takes effect while script is on. See the script group.

An example

return await astral.test('Login demo', async () => {
  await astral.step('Open the store', () => astral.navigate({url: 'http://localhost:3000/'}));
  await astral.step('Sign in', async () => {
    await astral.type({locator: {role: 'textbox', name: 'Username'}, text: 'demo'});
    await astral.type({locator: {role: 'textbox', name: 'Password'}, text: 'demo', submit: true});
    await astral.expect.visible({role: 'button', name: 'Sign out'});
    await astral.expect.noErrors();
  });
  await astral.step('Pay', async () => {
    await astral.click({locator: {role: 'button', name: 'Pay'}});
    await astral.expect.request({urlContains: '/api/', method: 'POST', status: '2xx'});
  });
}, {continueOnFailure: true, record: 'actions'});

Give run_script a timeoutMs of 60000 to 120000 for a test: evidence takes time.

astral.test and astral.step

astral.test(name, fn, options) options: tabId, continueOnFailure (default false) and record. It does not throw when a step fails: check the status in the result, which also has the totals, a Markdown summary, and the paths of the report.

astral.step(name, fn) groups actions. At its end, and always on failure, Astral saves a screenshot, the URL and title, the requests made during the step (failing ones with headers and bodies redacted), console errors and warnings, and any new error groups. A step fails when its function throws or an expectation in it fails. Without continueOnFailure the test stops at the first failed step.

run_script also accepts test: {name, continueOnFailure, ...} to treat the whole script as one test.

Expectations

A locator is {role, name, text, nth, exact}, a snapshot ref such as "e12", or a text. A failed expectation throws an assertion error with what was expected and what was found, and the report shows it. astral.screenshot({name, fullPage}) inside a test adds a named screenshot to the current step.

Record modes

Markers are drawn in every mode except 'off', disappear after 2.5 seconds and at the end of the run, and never catch clicks. Text typed into a password, one-time-code or card field is never written: captions show •••••• and the value is masked everywhere else in the report. See Platforms for video support.

What a run leaves

Each run writes a folder in Astral's data directory containing:

• report.html: a readable report with a summary, a card per step with its screenshots, the failed assertion, failing requests, console errors, grouped page errors, an actions timeline and a performance section.
• report.json: the same data, plus a Markdown summary written to be pasted into an issue.
• The screenshots as PNG files, and the recording if you asked for one.

The newest 50 runs are kept (configurable in Settings → Developer, where test runs can also be turned off). The agent reads runs back with test_runs_list and test_run_get.

Evidence for tickets

The Markdown summary is meant to be pasted into an issue, next to the screenshots and the report. If your agent is connected to your ticket tool, it can post that evidence there for you. Astral itself does not integrate with any ticket tool: it writes the evidence and leaves the posting to you or your agent.

Test names, steps, screenshot names and everything Astral writes about a run are in English, whatever language you, the ticket or the page use. Only locators keep the page's own text.