Yesterday from 5:51pm to 6:06pm (all times eastern) the LMS was not responsive to users. The system began to show increased response times at 5:47pm. The response times increased rapidly from under 100mS to 649mS. At 5:48pm we recorded response times above 4,000 mS. At 5:51pm the system was shut down. We rebooted two of the many systems that support the overall LMS and at 6:06pm response times returned and have been maintained at just under 100 mS. Read on for some insights into how we know about and keep track of the system.
The many systems supported by the electrical training ALLIANCE are under constant monitoring by personnel and programmatic support. A team we call the LMS Operations team is tasked with administering this constant monitoring. The group of folks is comprised of our Enterprise Data and Hardware Support team, all Lead Code-Development teams, and staff within the ALLIANCE. Each company represented has a defined list of workers who are to be available for and begin
communications when needed, every moment of every day. These communications and monitors include both full-time maintenance of the system as well as emergency response. We always know who has the Bat Phone or the Bat Spotlight…if you will.
In a case like last night;
The moment the system ever begins to show any semblance of poor performance, support staff are aware and working to identify and clear the issue. The teams are immediately assembled, begin communication protocols and a system of priority level alarm responses are instigated.
Four different companies with up to 5 team members each begin open communication, troubleshooting, identification and resolution of the issue.
Decisions are made as a group on all progress and the moment we have definable information to share, a notice is posted to users of the LMS. These notices may be posted in a number of different ways in order to reach our LMS users. Depending on the specific case of the disturbance, the message will be placed in the location best to serve you. — Last night, that place was on the login page to both the LMS and the Enrollment Application.
Following the outcomes of troubleshooting and resolution, a blog post like this one is written to fully clarify the concern.
We are again sorry for last nights disturbance and will continue to work to see that this concern does not occur again.