Configuring VictorOps (Splunk On-Call) Integration

StatusDrift integrates with VictorOps (now Splunk On-Call) to route monitoring alerts through your incident management workflows. This guide explains how to configure the VictorOps integration to ensure your on-call teams receive timely notifications.

What This Integration Does

The VictorOps integration creates incidents in your VictorOps timeline when:

• A monitored service goes down or becomes unreachable
• A service recovers and returns to normal operation
• SSL certificates are expiring or have issues
• Performance thresholds are exceeded

Alerts flow through your VictorOps escalation policies, ensuring the right team members are notified based on your on-call schedules.

Prerequisites

Before configuring the VictorOps integration, ensure you have:

• A VictorOps (Splunk On-Call) account with admin or alert admin permissions
• A StatusDrift account with notification channel management access
• At least one routing key configured in VictorOps

Step 1: Get Your VictorOps REST Endpoint URL

First, locate your REST endpoint URL in VictorOps:

1. Log in to your VictorOps account at portal.victorops.com
2. Navigate to Integrations in the top menu
3. Search for and select REST Generic
4. If not already enabled, click Enable Integration
5. Copy the REST Endpoint URL shown on the page
6. Note your Routing Key (or create a new one for StatusDrift alerts)

The endpoint URL format is: https://alert.victorops.com/integrations/generic/20131114/alert/[API_KEY]/[ROUTING_KEY]

Step 2: Add the VictorOps Channel in StatusDrift

Now configure the integration in your StatusDrift dashboard:

1. Log in to your StatusDrift account at app.statusdrift.com
2. Navigate to Notification Channels in the sidebar
3. Click Add Channel
4. Select Webhook as the channel type
5. Enter a descriptive name like “VictorOps Alerts”
6. Paste the complete REST Endpoint URL (including your routing key)
7. Click Save to create the channel

Step 3: Assign to Monitors

Connect the notification channel to your monitors:

1. Go to Monitors in the StatusDrift dashboard
2. Select the monitor you want to configure
3. Scroll to the Notification Channels section
4. Enable the VictorOps channel you created
5. Save your changes

Testing the Integration

Verify your VictorOps integration is working:

1. Go to Notification Channels in StatusDrift
2. Find your VictorOps channel in the list
3. Click the Test button
4. Check your VictorOps timeline for the test incident

Alert Message Types

StatusDrift sends different message types to VictorOps based on the event:

Routing Keys

VictorOps routing keys determine which team or escalation policy receives alerts. Consider creating dedicated routing keys for:

• Production services – Route to your primary on-call rotation
• Staging/Development – Route to development team or lower priority queue
• Critical infrastructure – Route to infrastructure team with aggressive escalation

Troubleshooting

Alerts not appearing in VictorOps

• Verify the REST endpoint URL is complete and includes your API key
• Confirm the routing key exists in your VictorOps configuration
• Check that the REST Generic integration is enabled in VictorOps
• Use the Test button in StatusDrift to send a test alert

Incidents not auto-resolving

• Ensure recovery notifications are enabled on your StatusDrift monitor
• Verify the same routing key is used for both alert and recovery messages
• Check your VictorOps incident rules for any custom resolution settings

Wrong team receiving alerts

• Review the routing key in your webhook URL
• Check the escalation policy associated with that routing key in VictorOps
• Consider creating a dedicated routing key for StatusDrift alerts