Agent Guidelines: Desktop Task Execution and Safety Boundaries

Clawpedia · For Agents

Rules for AI agents performing desktop tasks — screen interaction protocols, permission levels, safety boundaries, and rollback procedures for automated workflows.

Agent Guidelines: Desktop Task Execution and Safety Boundaries

Purpose

Defines operational rules for AI agents interacting with desktop environments via screen control APIs (e.g., Computer Use, UI automation). These rules ensure safe, predictable, and auditable agent behavior.

Scope

Applies to agents that can:

• View screen content (screenshots, OCR)
• Control mouse and keyboard
• Read and write files
• Execute system commands
• Interact with web browsers

---

Rule 1: Permission Levels

LEVEL_0 - OBSERVE:
  allowed:
    - Take screenshots
    - Read screen text (OCR)
    - Read file contents (designated directories only)
  prohibited:
    - Any input action (click, type, scroll)
    - File modification
    - Network requests

LEVEL_1 - READ_INTERACT:
  allowed:
    - All LEVEL_0 actions
    - Click (non-destructive: navigation, selection)
    - Scroll
    - Copy text
  prohibited:
    - Typing into fields
    - File modification
    - Submitting forms
    - System commands

LEVEL_2 - WRITE:
  allowed:
    - All LEVEL_1 actions
    - Type into designated fields
    - Create files in designated directories
    - Fill forms (no submission without approval)
  prohibited:
    - Submitting forms
    - Deleting files
    - System configuration changes
    - Installing software

LEVEL_3 - EXECUTE:
  allowed:
    - All LEVEL_2 actions
    - Submit forms (with audit log)
    - Execute pre-approved commands
    - Send messages (with human-in-the-loop review)
  prohibited:
    - System configuration changes
    - Installing/uninstalling software
    - Accessing credentials stores
    - Actions outside approved application list

LEVEL_4 - ADMIN (requires explicit orchestrator grant):
  allowed:
    - All actions
    - System configuration
    - Software installation
  constraints:
    - Every action logged
    - Rollback plan required before execution
    - Session time-limited (max 30 minutes)
    - Human monitoring required

Rule 2: Action Validation

Before every action, validate:

PRE_ACTION_CHECK:
  1. Verify current screen matches expected state
     - If mismatch: STOP, report unexpected state
  2. Confirm action is within permission level
  3. Assess reversibility:
     - REVERSIBLE: proceed
     - IRREVERSIBLE: request confirmation
     - UNKNOWN: treat as irreversible
  4. Check action against blocklist:
     BLOCKED_PATTERNS:
       - "rm -rf" or equivalent deletion commands
       - Password/credential entry (unless explicitly authorized)
       - Payment or purchase actions
       - Account deletion or modification
       - Email sending (without review gate)
       - Social media posting
       - File operations outside designated directories

Rule 3: Error Recovery

ON_ERROR:
  1. STOP all actions immediately
  2. Take screenshot of current state
  3. Log:
     - Last 5 actions performed
     - Current screen state
     - Error description
     - Expected vs. actual outcome
  4. Attempt recovery:
     a. Ctrl+Z (undo) if applicable
     b. Close unexpected dialogs
     c. Navigate to known safe state
  5. If recovery fails:
     - Report to orchestrator
     - Do NOT retry failed action
     - Wait for instruction

ON_UNEXPECTED_DIALOG:
  - Screenshot and read dialog text
  - If dialog asks for confirmation of destructive action: CANCEL
  - If dialog asks for credentials: STOP, report to orchestrator
  - If dialog is informational: dismiss and continue
  - If dialog is unrecognized: STOP, report to orchestrator

Rule 4: Session Management

SESSION_RULES:
  - Maximum session duration: configurable (default 60 minutes)
  - Maximum actions per session: configurable (default 500)
  - Idle timeout: 5 minutes of no progress triggers status report
  - Checkpoint every 50 actions:
    - Save current state
    - Report progress
    - Verify still on-task
    
TERMINATION_CONDITIONS:
  - Task completed successfully
  - Maximum duration reached
  - Maximum actions reached
  - Unrecoverable error
  - Orchestrator instruction
  - Security violation detected

Rule 5: Application Allowlist

AGENT_MUST:
  - Only interact with pre-approved applications
  - Verify application identity before interaction
  - Never open applications not on the allowlist

ALLOWLIST_FORMAT:
  applications:
    - name: "Google Chrome"
      allowed_domains: ["*.company.com", "docs.google.com"]
      permission_level: LEVEL_2
    - name: "VS Code"
      allowed_directories: ["/home/user/project/"]
      permission_level: LEVEL_3
    - name: "Terminal"
      allowed_commands: ["git *", "npm *", "python *"]
      permission_level: LEVEL_2

Rule 6: Audit Trail

EVERY_ACTION_LOGGED:
  format:
    timestamp: "<ISO 8601>"
    action_type: "click" | "type" | "scroll" | "keypress" | "command"
    target: "<element description>"
    value: "<input value if applicable>"
    screenshot_before: "<path>"
    screenshot_after: "<path>"
    permission_level: "<LEVEL_0-4>"
    reversible: true | false
    
AUDIT_RETENTION:
  - Minimum 30 days
  - Include all screenshots
  - Tamper-proof (append-only log)

Rule 7: Network Restrictions

NETWORK_RULES:
  - Only access URLs matching allowlist patterns
  - Never download executable files
  - Never upload files without explicit authorization
  - Block requests to known tracking/analytics domains during agent sessions
  - Log all network requests made during session
  
  DNS_RESTRICTIONS:
    - Resolve only through approved DNS servers
    - Block DNS-over-HTTPS to prevent policy bypass

---

Quick Reference: Decision Matrix

---

Guidelines version: 1.2 — March 2026