If you need to “break glass and pull the plug”, sign in to HQ, navigate to your workspace and click the Declare Incident button.The intent is that freezing your data and disabling sync can de-escalate most crisis situations to provide breathing room to work the problem.
A red alert is a global problem or outage affecting multiple customers. This alerts the engineering and infrastructure team that Provisionr is experiencing problems.This is not for troubleshooting workspace specific issues. Please create a support ticket (low priority) or declare a yellow alert (high priority) if you need help with your workspace.You can see any known problems in our Slack Community in the #anomaly-detected channel where you can stay informed or comment in real time.
A yellow alert is a serious issue with your workspace. By freezing your data, we can mitigate the crisis and start triaging the problem. Yellow alerts are handled during business hours.
A blue alert is a non-urgent issue that you believe requires freezing your workspace in a read-only “safe mode” to mitigate the problem. The Provisionr support team is not involved. If you find an issue that you need help with, you can escalate it to a Yellow alert.
Our business hours are 9:00am to 5:00pm US Central (CDT/CST).Our UTC hours shift with US Daylight Savings Time and are 15:00 to 23:00 UTC from November to March and 14:00 to 22:00 UTC from March to November.
For yellow alerts declared later in the day, we will try to make contact with you before the end of the business day. If we are unable to reach you, we will follow up first thing the next business day.
When you declare an incident, we disable all integration syncs for both incoming identity data and outgoing group and resource member sync. Any users with global.* roles retain their read-write permissions. All other users are downgraded to *.viewer for their respective roles. Their access will be restored when the incident is resolved.This puts your workspace in an effective “safe mode” with mostly read-only state to allow your team to investigate and remediate the problem without worrying about further changes being made. In other words, no one can make things worse while you are trying to fix the problem.
When you declare a red alert or yellow alert, we create a support ticket that automatically emails you to ask for more information about what’s going on.You have the option of choosing whether you want email, private Slack channel, or Zoom support. Feel free to share the Slack channel or Zoom meeting invite link with anyone who needs to be involved.You will receive a calendar booking link for a Zoom meeting in the initial email after you declare the incident. Mitigation Zoom calls are typically scheduled between 12:00pm to 4:00pm CDT/CST. This gives us time to remediate any global outages first thing in the morning and merge fixes for any bugs that we can resolve before our call.Once we start talking with you, we will determine next steps.
We recognize that an incident or outage can happen at any time. We are committed to supporting our customers, however we do not provide 24/7 support. We do not have a night shift or offer an SLA outside of our business hours.We prioritize time with our families outside of business hours. We check our phones occasionally to keep an eye on alerts, so you may see a response if someone is available, however no guarantees.We’re happy to support customers globally, however please expect next business day responses in your local timezone.You can also ask other customers for help in our Slack Community.
We have automation that detects if multiple customers have declared red alerts and will page our team to investigate if there is a global outage.If Provisionr is experiencing a problem, we will be actively posting in our Slack Community in the #anomaly-detected channel where you can stay informed in real time.Email support tends to be delayed an hour or two while we focus on working the problem and providing broadcast updates in Slack.In the future, we will add a feature to HQ for you to add a group email address for incident notifications and configure a webhook with your own Slack workspace channel(s) to broadcast incident and outage notifications.Any global problems discovered overnight will be posted to the Slack #anomaly-detected channel by 9:30am CDT/CST or 15:30 UTC. Our engineers will post preliminary acknowledgement that a problem exists as soon as we find something, regardless of the time of day. This allows our engineers to quickly acknowledge and start working on the fix. Our ability to support individual customer workspaces will be limited until our team comes online during our business hours.
We do not provide workspace specific support outside of our business hours. We are only on call for global outages.When you declare a yellow alert or blue alert, your Workspace is frozen so no syncs will occur. This typically lowers the urgency of most problems. We will be available to support you during our business hours.You also have the self service option to easily roll back (restore) changes made in the last few days that may solve your problem.Expect an email response by 11:00am CDT/CST or 17:00 UTC unless a Red Alert is declared.You will receive a calendar booking link for a Zoom meeting in the initial email after you declare the incident. Mitigation Zoom calls are typically scheduled between 12:00pm to 4:00pm CDT/CST. This gives us time in the morning to remediate any global outages first and merge fixes for any bugs that we can resolve.
We do not monitor our support tickets on weekends, from Friday at 5:00pm CDT/CST to Monday morning. We may check Sunday afternoon to see what we have waiting for us on Monday, so it’s possible you might see a response before Monday (not a guarantee).We are only on call for red alert global outages during daytime hours.On Friday and Saturday nights, we mute our red alert pager notifications to let our team enjoy uninterrupted sleep. If we receive any red alerts overnight, we will respond starting at 9:00am CDT/CST.
