The system is experiencing issues
Maintenance
Helios kernel upgrade

Helios will be taken down for a kernel upgrade at 12 AM EDT (-0400 GMT). As this is an older platform, anticipate a 5-7 minute outage while the server is rebooted and brought back online.

Luna kernel upgrade

A kernel upgrade is scheduled tonight for Luna to address improvements in OverlayFS that powers filesystem composition for all accounts. The outage will begin at 12:00 AM EDT (-0400 GMT). It is expected to last less than 2 minutes.

Control panel upgrades

Luna, Sol, and Helios will undergo a major internal CP upgrade on Sunday, September 25 beginning at noon. Expect intermittent outages as the upgrade rolls out. Following this rollout the following servers will no longer receive CP updates and are considered EOL:

  • Apollo
  • Aleph
  • Augend
  • Image

Clients on these platforms are encouraged to migrate off within the next 6-24 months before they are decommissioned. A new platform will be released near the end of this year with significant hardware improvements over the latest release (Luna).

12:07 PM Update: Pushing back to 5 PM EDT...

4:50 PM: Beginning to roll out updates

6:28 PM: Maintenance concluded. CP upgrades completed.

Let's Encrypt server cert bootstrapping

Helios, Sol, and Luna will implement automatic Let's Encrypt bootstrapping for HTTP, email, and FTP services tonight beginning at 11 PM EST (-0500 GMT). Bootstrapping only covers the primary server certificate, not individual sites, which still require a separate SSL addon service. No outage is anticipated beyond a 10-30 second outage for each service that is brought up with the new certificate. Server certificates will automatically renew every 80 days.

10:30 PM EST: maintenance began earlier than anticipated.

11 PM EST: maintenance window concluded.

Sol synthetic filesystem change

Sol will be taken down tonight, Sunday January 8 at 12:00 AM EST (-0500 GMT) to switch out its synthetic filesystem driver from aufs to OverlayFS. OverlayFS is used on newer platforms to provide improved throughput over aufs with minimal lock contention, which has plagued Sol in high I/O situations.

At least 30 minutes of intermittent outages is planned for this window.

Update: Sol still experiencing deadlocks, OverlayFS has been switched out ahead of schedule.

Cabinet Migration

All servers will be physically disconnected and moved to a new location in the data center on Saturday, January 28 beginning at 10 PM EST (-0500 GMT). The migration is expected to last 30-90 minutes.

Past Incidents

Thursday 30th January 2025

No incidents reported

Wednesday 29th January 2025

No incidents reported

Tuesday 28th January 2025

No incidents reported

Monday 27th January 2025

No incidents reported

Sunday 26th January 2025

No incidents reported

Saturday 25th January 2025

No incidents reported

Friday 24th January 2025

No incidents reported