Step-by-step guide for using the entire platform
This guide walks end users through the full UmamiAI lifecycle: organization setup, agent and workflow configuration, knowledge + tools, run monitoring, approvals, team collaboration, API keys, webhooks, and day-to-day operations.
Start here
If you are brand new to the platform, follow this exact sequence first.
- 1Create or access your account
Use Sign up to create a new account, or Sign in if your admin already invited you. If you received an invite link, open it first to attach your membership to the correct organization.
- 2Complete onboarding
Create your organization (or confirm the invited org), then create your first agent so the platform can route you into the main console experience.
- 3Verify access to key sections
Open Dashboard, Agents, Runs, and Approvals. Owners/Admins should also verify Team, API Keys, and Webhooks are visible and working.
- 4Pick your first success path
For business users: create and run an agent. For technical users: create an API key and test the Platform API. For operations users: configure webhooks and approvals.
Roles and permissions
Understanding access levels helps prevent confusion during setup and approval workflows.
- Manage org settings and members
- Create/revoke API keys and webhooks
- Approve/reject sensitive actions
- Configure agents, tools, and workflows
- Most configuration and operational actions
- Can invite members (org policy dependent)
- Common choice for platform operators
- Typically allowed to manage API/webhooks
- Run agents and workflows
- View resources permitted by org policies
- Can participate in operational review
- Less access to sensitive configuration
Onboarding
The onboarding flow is designed to get you to a usable console quickly.
- 1Step 1 — Organization
Enter a clear organization name. Use a stable naming convention (for example, “Acme Operations” instead of temporary project nicknames) so audits, exports, and team invites remain easy to track later.
- 2Step 2 — First agent
Create a first agent with a simple, testable purpose such as summarization, triage, extraction, or QA. Keep the first version focused; you can add tools and workflow orchestration after the baseline run works.
- 3Step 3 — Ready
You will land on the Dashboard. From here, continue into Agents to refine the agent, or into Team/API Keys if another teammate or external integration needs access.
Dashboard
Use the dashboard as your operational overview before drilling into detailed pages.
- 1Review activity and status cards
Check high-level metrics and recent activity to confirm your org is active and no queue/backlog issue is preventing runs from completing.
- 2Jump to the next action
Use the dashboard links to open Agents, Runs, Workflows, Approvals, or Team depending on your current objective.
- 3Use dashboard as a daily checklist
Many teams begin each day by scanning pending approvals, failed runs, and webhook delivery status before making changes to agents or workflows.
Agents and versions
Agents are versioned execution units. Use versions to keep behavior reproducible while improving prompts and tool access over time.
- 1Open Org → Agents and create a new agent
Give it a business-readable name (for example: “Sales Weekly Summarizer” or “Invoice Triage Agent”).
- 2Add instructions with hard boundaries
Be explicit about what the agent should do, what outputs are expected, what it must not do, and when it should request approval.
- 3Attach tools only when needed
Start with the minimum set of tools. Add additional tools after you have validated the baseline prompt behavior and run outputs.
- 4Save and run a test
Use realistic sample input (not placeholder text) so you can validate formatting, accuracy, and runtime behavior under real constraints.
- Create v1 with a narrow task and no risky external actions.
- Test run output quality and edge cases.
- Create a new version when changing prompts, model settings, or tool access.
- Document what changed (purpose, risk, expected output differences).
- Promote the version only after reviewers confirm expected behavior.
Knowledge (RAG)
Knowledge pages let you manage documents and retrieval inputs used by agents and workflows.
- 1Open Org → Knowledge
Review documents already added to your org. Confirm the source set is correct before relying on retrieval in production agents.
- 2Add or update documents
Upload source material or connect content according to your org process. Use clear titles and ownership tags so operators can identify stale documents quickly.
- 3Validate retrieval relevance
Run a knowledge search or a test run using representative queries. Confirm the returned chunks are accurate, current, and safe to use in downstream outputs.
- 4Retire stale knowledge
Remove, replace, or supersede old documents when business rules change to avoid incorrect answers and accidental policy drift.
Tools
Tools extend agents with structured actions (e.g., HTTP calls, external systems). Treat tool setup like production integration work.
- 1Open Org → Tools and create a tool
Define the tool name and purpose so reviewers understand exactly what action the agent may take using this integration.
- 2Configure endpoint/method/schema
Provide the request details (URL, method, headers, payload schema) and validate the expected inputs/outputs before enabling the tool on agents.
- 3Store secrets securely
Never paste credentials into prompts. Use the platform’s tool secret handling so tokens and keys are encrypted and not shown in normal UI views.
- 4Apply approvals for risky actions
If the tool can mutate external systems, enable approvals so humans review intent before execution completes.
Runs
Runs are your primary execution record for prompts, tool calls, outputs, and debugging.
- 1Open Org → Runs and start a new run
Select the target agent and provide the input payload or prompt. Use realistic production-like examples for testing.
- 2Track run status
Watch the run move through statuses as steps execute. If a tool requires approval, the run may pause until a reviewer acts.
- 3Inspect run steps and outputs
Open the run detail to inspect each step, payloads, tool interactions, and final output for validation or debugging.
- Check if the run is waiting on an approval.
- Review the tool request/response and input payload.
- Confirm the relevant tool secret and endpoint are valid.
- Verify required knowledge documents/chunks exist.
- Re-run with narrower input to isolate the issue.
Workflows
Workflows coordinate multi-step execution. Use them for repeatable, auditable automation that spans prompts and tools.
- 1Open Org → Workflows and create a workflow
Name the workflow by business outcome (for example: “Weekly pipeline digest + anomaly review”) and define a clear owner for operational accountability.
- 2Add steps in execution order
Each step can call an agent or tool. Keep each step focused and clearly named so approvals and logs are easy to interpret later.
- 3Reference earlier outputs
Pass context from previous steps into later steps using template references. This keeps workflows deterministic and reduces manual copying between runs.
- 4Test with realistic payloads
Run the workflow with representative input and validate intermediate outputs, approval pauses, and final artifacts before broader rollout.
Use a template token like this inside later steps to reference the output of an earlier step.
{{step.N}} # Example: {{step.1}}Approvals
Approvals are the human control plane for sensitive or high-impact actions.
- 1Open Org → Approvals
Use filters to focus on pending items, workflow-only items, or agent-run items depending on your triage responsibilities.
- 2Sort by step when reviewing workflows
Use Step ↑ or Step ↓ to review workflow approvals in execution order. This helps reviewers understand context before approving downstream actions.
- 3Open details and verify intent
Inspect the action payload, related run/workflow, and any tool context. Confirm the requested action matches policy and expected business logic.
- 4Approve or reject
Submit the decision. The page preserves your filters/sort settings so you can continue triaging without losing context.
Team and invites
Use the Team page to scale access safely while preserving role boundaries.
- 1Go to Org → Team
Review current members and pending invites before creating a new invite.
- 2Create invite
Enter the teammate email and choose the minimum role they need right now. You can elevate later if required.
- 3Track invite acceptance
Pending invites remain visible so admins can resend, revoke, or confirm whether the user joined.
- Keep only a small set of owners/admins.
- Use member role for most day-to-day operators.
- Review memberships regularly during audits.
- Revoke access immediately when responsibilities change.
API keys
API keys let external systems call your UmamiAI Platform API securely.
- 1Open Org → API Keys
Only owners/admins should create keys. Review existing keys and scopes to avoid issuing duplicates for the same integration.
- 2Create a scoped key
Provide a name that identifies the integration (for example, “crm-sync-prod”), choose the minimum scopes, and create the key.
- 3Copy the plaintext key once
The full key is shown only at creation time. Store it immediately in a secret manager, CI/CD secret store, or runtime environment variable.
- 4Test with the Platform API
Use the docs and developer guide curl examples to verify auth, create a run, fetch results, and confirm your integration path before production rollout.
# Example API auth header
curl -sS "$UMAMIAI_BASE_URL/api/v1/agent-runs/$RUN_ID" \
-H "Authorization: Bearer umami_..."Webhooks
Webhooks notify your systems when events happen in the platform (for example run completion or approvals).
- 1Open Org → Webhooks and add an endpoint
Provide a reachable HTTPS endpoint in your system. Use separate endpoints for staging and production where possible.
- 2Store the signing secret
Copy the webhook signing secret and store it securely. Your receiver will use it to verify each delivery signature.
- 3Implement signature verification
Validate the delivery using the signature header before trusting the payload. Reject requests that fail verification or come from unexpected sources.
- 4Test and monitor deliveries
Trigger a platform event, inspect webhook delivery attempts, and verify your receiver handles retries and idempotency safely.
X-Umami-Event, X-Umami-Delivery, and X-Umami-Signature.Audit and operations
Use the platform’s records to support troubleshooting, governance, and operational review.
- Pending approvals are triaged and not stale
- Failed runs are investigated and re-tested
- Webhook deliveries are succeeding / retried safely
- API keys are scoped correctly and old keys revoked
- Knowledge sources are current and owned
- Document who owns each agent and workflow
- List critical tools and their approval requirements
- Review team membership and least-privilege roles
- Export / capture evidence from runs and approvals
- Capture current integration endpoints and key rotations
Best practices for a production-ready setup
These practices reduce incidents and speed up onboarding for new teammates.
- Start narrow: one agent, one workflow, one clear outcome.
- Version all meaningful changes to prompts/tools.
- Prefer structured outputs where downstream systems depend on parsing.
- Add approvals before enabling destructive or external mutations.
- Use environment-specific webhooks and API keys.
- Rotate keys regularly and when staff changes occur.
- Keep knowledge fresh; retire stale documents explicitly.
- Use pending approval queues as a daily ops ritual.
Troubleshooting and common issues
Use these quick checks before escalating to your platform owner or developer team.
Your role may not have permission. Confirm you are an owner/admin in the current organization and that you are in the correct org (top-level console route often includes the org ID).
Check the Approvals queue first. If no approval is pending, open the run and inspect the last step/tool response for missing inputs, bad secrets, or endpoint failures.
Confirm the endpoint URL is correct, publicly reachable, and returns a success status. Review delivery logs and signature verification logic. Ensure your receiver handles retries idempotently.
Check whether the agent version changed, a tool configuration changed, or knowledge documents were updated. Compare recent runs with known-good runs to isolate the difference.
After your operators understand the platform, use the Platform API documentation and integration guide to connect external systems using scoped API keys and signed webhooks.