Experimently Documentation

Get up and running in under 10 minutes.

• Quick Start—Create your first experiment in minutes
• Core Concepts—Feature flags, experiments, variants, and metrics
• Architecture Overview—How Experimently works under the hood
• FAQ—Common questions answered
• Modules & Profiles—What the core profile includes, what each module adds, and how to run the full profile

Control feature rollouts safely with targeting rules.

• Creating Feature Flags—Set up your first flag
• Targeting & Rules—20+ operators, segment targeting
• Gradual Rollouts—Staged rollout schedules
• Safety Monitoring—Auto-rollback on error spikes

Design, run, and analyze experiments with statistical rigour.

• Running Experiments—End-to-end experiment lifecycle
• Statistical Methods—Frequentist, Bayesian, sequential testing
• CUPED Variance Reduction—Reduce variance with pre-experiment covariates
• Multi-Armed Bandits—Thompson Sampling, UCB1, Epsilon-Greedy
• Split URL Testing—Server-side URL splitting via Lambda@Edge
• Mutual Exclusion Groups—Prevent cross-experiment contamination
• Live Results Streaming—Real-time experiment results via WebSocket — p-values, lift, and significance updated live.

Compare prompt versions, model variants, and agent configs against real business metrics.

• Overview—What LLM evaluation is and why it matters
• Quick Start—Compare GPT-4o vs Claude Sonnet in 5 minutes

Client libraries for 14 languages and frameworks — from web to mobile to server.

• JavaScript SDK—Browser and Node.js
• Python SDK—Server-side Python integration
• Java SDK—Spring Boot auto-configuration, consistent hash bucketing
• React SDK—Hooks, HOC, SSR support via ServerClient
• Go SDK—Native Go client for microservices and CLIs. Zero dependencies, context-aware, goroutine-safe.
• iOS Swift SDK—Native Swift SDK for iOS 14+ and macOS 11+. Async/await API, offline fallback.
• Android Kotlin SDK—Native Kotlin SDK for Android (minSdk 21). Coroutines, OkHttp, Compose examples.
• Flutter SDK—Dart SDK for Flutter (iOS, Android, Web, Desktop). Offline fallback, consistent hashing.
• React Native SDK—useFlag and useExperiment hooks, Provider, AsyncStorage offline support.
• MCP Server—AI-powered experiment design via Model Context Protocol
• OpenFeature Provider—CNCF-standard OpenFeature provider for TypeScript and Python — swap vendors without changing app code
• Edge SDK—Sub-millisecond evaluation for Cloudflare Workers, Vercel Edge, and Deno Deploy — zero Node.js dependencies, pure-JS MD5 consistent hash
• Ruby SDK—Native Ruby gem, zero runtime dependencies, thread-safe Mutex TTL cache, Net::HTTP, consistent MD5 hash.
• PHP SDK—Composer package (ext-json + ext-curl only), PSR-compatible, consistent MD5 hash bucketing.
• .NET SDK—netstandard2.1 + net6.0, System.Text.Json, HttpClient, LRU cache with TTL, xUnit-tested.
• Elixir SDK—Hex package with :httpc + Jason, GenServer-backed ETS cache, OTP-compatible, 93 ExUnit tests.

Full REST API documentation with request/response schemas.

• Authentication—API keys and JWT tokens
• Experiments—CRUD, scheduling, results
• Feature Flags—Flags, rollouts, evaluation
• Compliance & Audit—Audit events, SOC 2 / ISO 27001 reports, export
• Integrations—Jira, Salesforce, GitHub webhooks
• All Endpoints—Complete endpoint reference

Connect Experimently to your existing data stack.

• AWS—ECS, Lambda, CloudFront, DynamoDB
• Data Warehouses—Snowflake, BigQuery, Redshift — warehouse-native analytics
• Databricks—Databricks SQL warehouse connector — read-only analytics via Unity Catalog
• ClickHouse—ClickHouse columnar analytics connector — blazing fast OLAP queries
• MySQL—MySQL / MariaDB connector — parameterised read-only analytics queries
• Jira—Sync experiment status, auto-create issues
• Salesforce—OAuth2 integration, webhook sync
• GitHub—HMAC-signed webhooks, PR linking
• Slack & Email—Alerting and experiment notifications

Security, audit logging, and compliance reports.

• Compliance audit logging—Signed audit events and report export
• ISO 27001—ISO 27001 compliance report export
• Audit Logging—HMAC-SHA256 tamper-proof audit trail
• RBAC—Role-based access control: Admin, Developer, Analyst, Viewer
• API Key Management—Scoped keys, rotation, revocation
• SSO / SAML / OIDC—SAML 2.0, OIDC/OAuth2 for Okta, Azure AD, Google Workspace, GitHub, and OneLogin.
• HIPAA Compliance—PHI encryption (Fernet AES-128-CBC), 6-year audit retention, BAA management, data residency.

Deploy Experimently to your own AWS account.

• AWS CDK Deployment—One-command CDK deploy to ECS Fargate
• Docker Compose—Local development setup
• Environment Variables—Configuration reference
• Database Migrations—Alembic migration guide
• Monitoring—CloudWatch dashboards, Prometheus metrics

Step-by-step walkthroughs for common use cases.

• Your First A/B Test—End-to-end experiment walkthrough
• Bayesian Experimentation—Beta-Binomial posteriors and stopping rules
• Warehouse-Native Analytics—Query Snowflake/BigQuery directly
• Guided Experiment Builder—5-step draft-and-submit API
• Interaction Detection—Detect and handle experiment interactions

Deep dives into the statistical methods used by the platform.

• Power Analysis & Sample Size—Pre-experiment planning: MDE, alpha, power, runtime estimation
• Post-Stratification—Variance reduction using post-experiment stratification
• FDR Correction—Benjamini-Hochberg false discovery rate correction for multiple metrics

Isolate experiments and feature flags per team, project, or product area with role-based access control.

• Workspace Overview—Role hierarchy (OWNER → VIEWER), plan limits, scoped API keys
• Quickstart—Create a workspace, invite your team, and generate a scoped API key