Skip to main content
Use OS-level controls to move and click the mouse, type and press keys, scroll, drag, and capture screenshots from a running browser session.

​
Click the mouse

Simulate mouse clicks at specific coordinates. You can select the button, click type (down, up, click), number of clicks, and optional modifier keys to hold. 
import { Kernel } from '@onkernel/sdk';

const kernel = new Kernel();
const kernelBrowser = await kernel.browsers.create();

// Basic left click at (100, 200)
await kernel.browsers.computer.clickMouse(kernelBrowser.session_id, {
  x: 100,
  y: 200,
});

// Double right-click while holding Shift
await kernel.browsers.computer.clickMouse(kernelBrowser.session_id, {
  x: 100,
  y: 200,
  button: 'right',
  click_type: 'click',
  num_clicks: 2,
  hold_keys: ['Shift'],
});

​
Move the mouse

Move the cursor to specific screen coordinates. Optionally hold modifier keys during the move. 
import { Kernel } from '@onkernel/sdk';

const kernel = new Kernel();
const kernelBrowser = await kernel.browsers.create();

await kernel.browsers.computer.moveMouse(kernelBrowser.session_id, {
  x: 500,
  y: 300,
  hold_keys: ['Alt'],
});

​
Take screenshots

Capture a full-screen PNG or a specific region. 
import fs from 'fs';
import { Buffer } from 'buffer';
import { Kernel } from '@onkernel/sdk';

const kernel = new Kernel();
const kernelBrowser = await kernel.browsers.create();

// Full screenshot
{
  const response = await kernel.browsers.computer.captureScreenshot(kernelBrowser.session_id);
  const blob = await response.blob();
  const buffer = Buffer.from(await blob.arrayBuffer());
  fs.writeFileSync('screenshot.png', buffer);
}

// Region screenshot
{
  const response = await kernel.browsers.computer.captureScreenshot(kernelBrowser.session_id, {
    region: { x: 0, y: 0, width: 800, height: 600 },
  });
  const blob = await response.blob();
  const buffer = Buffer.from(await blob.arrayBuffer());
  fs.writeFileSync('region.png', buffer);
}

​
Type text

Type literal text, optionally with a delay in milliseconds between keystrokes. 
import { Kernel } from '@onkernel/sdk';

const kernel = new Kernel();
const kernelBrowser = await kernel.browsers.create();

await kernel.browsers.computer.typeText(kernelBrowser.session_id, {
  text: 'Hello, World!',
});

await kernel.browsers.computer.typeText(kernelBrowser.session_id, {
  text: 'Slow typing...',
  delay: 100,
});

​
Press keys

Press one or more key symbols (including combinations like “Ctrl+t” or “Ctrl+Shift+Tab”). Optionally hold modifiers and/or set a duration to hold keys down. 
import { Kernel } from '@onkernel/sdk';

const kernel = new Kernel();
const kernelBrowser = await kernel.browsers.create();

// Tap a key combination
await kernel.browsers.computer.pressKey(kernelBrowser.session_id, {
  keys: ['Ctrl+t'],
});

// Hold keys for 250ms while also holding Alt
await kernel.browsers.computer.pressKey(kernelBrowser.session_id, {
  keys: ['Ctrl+Shift+Tab'],
  duration: 250,
  hold_keys: ['Alt'],
});

​
Scroll

Scroll the mouse wheel at a specific position. Positive delta_y scrolls down; negative scrolls up. Positive delta_x scrolls right; negative scrolls left. 
import { Kernel } from '@onkernel/sdk';

const kernel = new Kernel();
const kernelBrowser = await kernel.browsers.create();

await kernel.browsers.computer.scroll(kernelBrowser.session_id, {
  x: 300,
  y: 400,
  delta_x: 0,
  delta_y: 120,
});

​
Drag the mouse

Drag by pressing a button, moving along a path of points, then releasing. You can control delay before starting, the granularity and speed of the drag via steps_per_segment and step_delay_ms, and optionally hold modifier keys. 
import { Kernel } from '@onkernel/sdk';

const kernel = new Kernel();
const kernelBrowser = await kernel.browsers.create();

await kernel.browsers.computer.dragMouse(kernelBrowser.session_id, {
  path: [
    [100, 200],
    [150, 220],
    [200, 260],
  ],
  button: 'left',
  delay: 0,
  steps_per_segment: 10,
  step_delay_ms: 50,
  hold_keys: ['Shift'],
});
⌘I