Phaonix Status - Incident history

Service Disruption: Node-1 VPS

Thu, 16 Jan 2025 18:47:35 +0000

Type: Incident

Affected Components:

Jan 16, 18:47:35 GMT+0
Resolved - The issue with our Node-1 VPS has been resolved. The cause was a filesystem inconsistency on the root partition, which required manual intervention. Our team performed a filesystem repair (fsck), and the server is now back online and functioning as expected. To ensure the safety of data and make future recovery easier if required, we also initiated a manual backup of the server during the intervention. This proactive step ensures that, in case of further disk issues, we can transition smoothly and securely. We sincerely apologize for any inconvenience caused and thank you for your patience during this time. If you experience any further issues, please don't hesitate to contact our support team..

Problème avec le fournisseur BGP - Intervention en cours

Tue, 16 Apr 2024 16:35:24 +0000

Type: Incident

Duration: 1 hour and 19 minutes

Affected Components:

Apr 16, 16:35:24 GMT+0
Investigating - Je tiens à vous informer qu'il y a actuellement un problème avec notre fournisseur BGP, ce qui affecte notre connectivité réseau. Pour résoudre ce problème dans les plus brefs délais, nous avons dépêché un technicien sur place pour diagnostiquer et corriger la situation. Nous sommes pleinement engagés à rétablir la connectivité aussi rapidement que possible afin de minimiser les perturbations pour nos services. Nous vous tiendrons informés de l'évolution de la situation dès que de nouvelles informations seront disponibles..

Apr 16, 17:54:13 GMT+0
Resolved - This incident has been resolved..

Infrastructure outage

Thu, 11 Apr 2024 00:17:45 +0000

Type: Incident

Duration: 14 hours and 27 minutes

Affected Components:

Apr 11, 00:17:45 GMT+0
Investigating - We are aware of an issue with our connectivity and are currently investigating..

Apr 11, 00:32:32 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Apr 11, 14:44:50 GMT+0
Resolved - This incident has been resolved..

GTT Planned Work - Reston, VA

Thu, 4 Apr 2024 04:00:00 +0000

Type: Maintenance

Duration: 4 hours

Affected Components:

Apr 4, 04:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 4, 04:00:00 GMT+0
Identified - GTT is planning a scheduled maintenance during the below timeframe. Service interruption is possible. Maximum service interruption expected is 15 minutes. **Start: 2024-04-05 00:00:00 GMT** **End: 2024-04-05 04:00:00 GMT** **Backup Start: 2024-04-09 00:00:00 GMT** **Backup End: 2024-04-09 04:00:00 GMT**.

Apr 4, 08:00:00 GMT+0
Completed - Maintenance has completed successfully.

Outage in VA/DFW

Wed, 7 Feb 2024 04:07:22 +0000

Type: Incident

Duration: 17 hours and 36 minutes

Affected Components:

Feb 7, 04:58:01 GMT+0
Identified - We are continuing to work on a fix for this incident..

Feb 7, 04:07:22 GMT+0
Investigating - We are currently investigating this incident..

Feb 7, 05:22:29 GMT+0
Monitoring - We have located the issue, and implemented a temporary fix for the issue. As of now, traffic previously routing through Ashburn has been diverted to the next closest pop for your user. We are working on a full scale solution, and will have an update ASAP. For now, things should be stable with slightly increased latency being possible. We will also be sharing a full scale RFO in the next few days..

Feb 7, 21:42:59 GMT+0
Resolved - This incident has been resolved..

Feb 7, 04:50:19 GMT+0
Investigating - No updates to share. Still working/investigating the issue..

Management Panel (DCP)

Sun, 12 Nov 2023 14:56:30 +0000

Type: Incident

Duration: 23 hours and 49 minutes

Affected Components:

Nov 12, 14:56:30 GMT+0
Investigating - Currently investigating a potential power supply issue on our ops server, impacting the server management panel. Our tech team is en route to the data center today for a thorough assessment and resolution. Apologies for any inconvenience caused..

Nov 12, 22:01:10 GMT+0
Investigating - The update has been postponed to tomorrow..

Nov 13, 14:45:28 GMT+0
Resolved - This incident has been resolved..

Infrastructure outage

Sat, 11 Nov 2023 02:49:21 +0000

Type: Incident

Duration: 44 minutes

Affected Components:

Nov 11, 03:33:36 GMT+0
Resolved - This incident has been resolved..

Nov 11, 02:49:21 GMT+0
Investigating - A network outage affected our network. Users reported connectivity issues, hindering operations..

Nov 11, 03:23:00 GMT+0
Investigating - Update: Tech is on site..

Network Performance Incident

Wed, 25 Oct 2023 04:17:13 +0000

Type: Incident

Duration: 7 hours and 39 minutes

Affected Components:

Oct 25, 04:17:13 GMT+0
Investigating - We are currently experiencing network performance issues that have resulted in slow connectivity and latency across our services. Our team is actively investigating the issue, and we are working in close collaboration with our vendor, to pinpoint the root cause and find a resolution..

Oct 25, 11:56:12 GMT+0
Resolved - We are pleased to announce that the recent network performance issue has been successfully resolved. The root cause of the incident was identified as a fibercut that affected multiple critical network providers, including TATA, Arelion, and Level3 fiber..

Network Maintenance - Core

Sat, 21 Oct 2023 16:30:00 +0000

Type: Maintenance

Duration: 6 hours

Affected Components: ,

Oct 21, 16:30:01 GMT+0
Identified - Maintenance is now in progress.

Oct 21, 16:30:00 GMT+0
Identified - We will be undergoing maintenance on our core network..

Oct 21, 22:30:00 GMT+0
Completed - Maintenance has completed successfully.

Network Maintenance

Sun, 15 Oct 2023 13:05:00 +0000

Type: Maintenance

Duration: 6 hours and 33 minutes

Affected Components:

Oct 15, 19:38:23 GMT+0
Completed - Maintenance has completed successfully..

Oct 15, 13:05:00 GMT+0
Identified - Scheduled maintenance is currently in progress. We will provide updates as necessary..

Network Maintenance - Upgrade

Fri, 6 Oct 2023 05:30:00 +0000

Type: Maintenance

Duration: 4 hours

Affected Components:

Oct 6, 05:30:00 GMT+0
Identified - We will be performing maintenance in all our locations to upgrade our overall capacity. While no outage is expected, there may be brief periods of higher latency as we connect and let traffic balance across the links..

Oct 6, 09:30:00 GMT+0
Completed - Maintenance has completed successfully.

Oct 6, 05:30:01 GMT+0
Identified - Maintenance is now in progress.

Maintenance Notice - Shared02 Web Hosting Server

Thu, 28 Sep 2023 15:45:00 +0000

Type: Maintenance

Duration: 3 hours and 45 minutes

Affected Components:

Sep 28, 15:46:25 GMT+0
Identified - Maintenance is now in progress..

Sep 28, 15:45:00 GMT+0
Identified - We want to inform you that we will be performing critical maintenance on our Shared02 web hosting server, starting in approximately one hour from now. This maintenance is essential to ensure the continued reliability and security of your hosting environment. During this process, there will be an expected downtime of around 15 minutes for your website. We apologize for any inconvenience this may cause and assure you that our team will work diligently to complete the maintenance as swiftly as possible. Your understanding and patience are greatly appreciated as we strive to provide you with the best hosting experience. If you have any urgent concerns or questions, please do not hesitate to contact our support team..

Sep 28, 19:31:03 GMT+0
Completed - The scheduled maintenance has been completed..

Core Routing Upgrades

Thu, 7 Sep 2023 23:01:03 +0000

Type: Maintenance

Duration: 14 hours and 5 minutes

Affected Components: ,

Sep 8, 13:06:17 GMT+0
Completed - Maintenance has completed successfully..

Sep 7, 23:01:03 GMT+0
Identified - We will be undergoing core routing upgrades across all locations. No outage is anticipated during this time..

Upgrade Preparation

Thu, 7 Sep 2023 01:22:09 +0000

Type: Maintenance

Duration: 8 hours and 47 minutes

Affected Components:

Sep 7, 10:08:42 GMT+0
Completed - The scheduled maintenance has been completed..

Sep 7, 01:24:26 GMT+0
Identified - Scheduled maintenance is currently in progress. We will provide updates as necessary..

Sep 7, 01:22:09 GMT+0
Identified - We will be executing some pre-upgrade preparation steps in order to ensure a smooth core routing upgrade. This will not have any affect on your services..

Firewall Manager outage

Mon, 28 Aug 2023 22:12:59 +0000

Type: Incident

Duration: 3 hours and 37 minutes

Affected Components:

Aug 28, 22:12:59 GMT+0
Investigating - We are aware of an issue with our Firewall Manager and are currently investigating..

Aug 29, 01:49:41 GMT+0
Resolved - This incident has been resolved..

Core Network Maintenance

Thu, 24 Aug 2023 12:00:00 +0000

Type: Maintenance

Duration: 2 hours

Affected Components:

Aug 24, 14:00:00 GMT+0
Completed - Maintenance has completed successfully.

Aug 24, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

Aug 24, 12:00:00 GMT+0
Identified - We will be performing core network maintenance. Little impact is expected..

FiveM Bypass

Wed, 23 Aug 2023 04:55:03 +0000

Type: Incident

Duration: 14 hours and 11 minutes

Affected Components:

Aug 23, 04:55:03 GMT+0
Identified - We want to bring to your attention an important development regarding our security systems and a recent incident involving a new FiveM bypass that has managed to circumvent our DDoS protection measures. Our team has been actively monitoring the situation and has taken immediate steps to address this issue. **Incident Overview:** Our security team has identified a new method used to bypass our DDoS protection protocols in the context of FiveM. **PCAP File and Investigation:** In response to this incident, we have already taken proactive measures. Our security experts have captured a Packet Capture (PCAP) file of the malicious traffic associated with this bypass. A PCAP file is a record of network traffic that captures the data flowing between network devices. This will allow our engineers to perform a thorough analysis of the attack vector, understand its intricacies, and devise an effective countermeasure. **Immediate Action Plan:** Our engineering team will be working diligently on analyzing the PCAP file to gain a comprehensive understanding of the attack vector. The insights gained from this analysis will guide the development of a robust fix that will effectively mitigate the bypass and restore the integrity of our DDoS protection measures. **Expected Timeline:** We understand the urgency of this matter and are committed to resolving it as swiftly as possible. Our engineers will be dedicating their efforts to investigating and implementing a fix throughout the day. We anticipate that the fix will be ready for deployment within the next 24 hours. **Communication and Updates:** We are aware of the importance of clear communication during incidents of this nature. We will provide regular updates as the situation progresses. Rest assured, we are committed to keeping you informed every step of the way. In conclusion, please be assured that we take this matter seriously, and the security of our systems and your data remains our top priority. We appreciate your understanding and patience as we work to resolve this issue. If you have any immediate concerns or questions, please feel free to reach out to our dedicated support team at support@phaonix.com. Thank you for your continued trust and cooperation..

Aug 23, 12:40:48 GMT+0
Identified - We've pinpointed the issue—it largely stems from the cfx.re team. There's a significant vulnerability in their thread handler. The network team is actively developing a solution, set to be implemented later today..

Aug 23, 17:44:39 GMT+0
Monitoring - We have successfully implemented a fix within our XDP FiveM filter. Currently, we are closely monitoring the outcomes to ensure its effectiveness..

Aug 23, 19:06:17 GMT+0
Resolved - After testing our fix on your FiveM filter, we can confirm that this issue has been resolved. We will continue to monitor everything to ensure its stability..

Intermittent Packet Loss When Routing Over NTT

Tue, 22 Aug 2023 21:12:07 +0000

Type: Incident

Duration: 7 days, 7 hours and 33 minutes

Affected Components:

Aug 22, 21:12:07 GMT+0
Identified - We are currently, and have been aware of the reports about intermittent packet loss throughout the day, and mostly during peak times for players routing over NTT (one of our upstreams). NTT has confirmed the issue to be on their end, and have said it's due to a major subsea cable outage causing this issue in numerous locations. They have said their estimated time for resolution will be on the 25th (Friday). We've been in contact with them daily, and will continue to remain in contact until this is resolved. We'll update this page with any major updates. Thanks for your patience and understanding..

Aug 30, 04:45:14 GMT+0
Resolved - This incident has been resolved..