Monday 13th November 2017

Control Panel Panel outage, scheduled 6 years ago

3:00 AM EST (-0500 GMT): Panel outage reported.

8:45 AM EST (-0500 GMT): Panel functionality restored.

Root cause: Luna serves as the default panel reverse proxy endpoint. Unless logged in prior to the outage, content would have been served from Luna. All servers participate in nightly updates; once code is signed off from development and pushed to production, panels update with the new code at ~3 AM EST. Sunday's change relocated Apache configuration from per-dir .htaccess to httpd.conf for improved performance. Luna diverged from the codebase thus requiring merging changes manually, which only mandates when incurred. As changes were replayed into the codebase, .htaccess had been removed first - resulting in the 404 errors - prior to manual merger effectively leaving the commit in a broken state.