Survival Servers · @SurvivalServers
10th Feb 2017 from TwitLonger
Statement regarding our recent downtime with Conan Exiles
To our Conan Exiles customers and other customers affected by this week's downtime:
I would like to apologize for the extended downtime that brought customer Conan Exiles servers (and other games) down Sunday evening through Wednesday early morning. A large amount of our Conan Exiles customers were brought down with issues like control panel status unknowns, lag, performance issues, and roll backs.
The problem stemmed from our internal backup system that we use to backup ARK, Conan, and Rust servers. The system goes inside a customer's Save folder, zips up the file and uploads it to our cloud. We had been using this method before and implemented it day 1 for Conan customers. Because launch went well, we thought the system was doing it's job. On Sunday Feb 5th, Conan Exiles server files were updated which included an update to SQLite database (3.9.2 to 3.16.2), some memory optimizations, and performance optimizations. With this update, a CPU leak bug became apparent and we now know that it is due to our backup method. When files in Save folders are zipped, a temporary lock is created on those save files. When the Conan server (after the Sunday update) goes to write the save files, it does not notice the file is locked and hangs up causing excessive CPU usage which bogs down the entire machine. The backup system zipping and Conan save simultaneously occuring across multiple machines caused this bug to become widespread across all of our Conan servers.
We've been in contact with the Conan Exiles developers and have let them know of the CPU leaking issue when save files are being accessed by another process on the same server that is online.
** Saves as of Wednesday afternoon and onward should not be affected ** Roll backs can be expected for some Save files prior. It is a case by case basis. We are working on a new method to backup and will have that online in the next couple days. It will not be brought online without thorough testing to make sure it does not lock files within the Save folder during backup.
I do apologize it took us several days to figure out the fix. Our entire team has worked tirelessly this week to battle against the huge influx of support tickets while simultaneously testing differents methods to fix the issue. Each fix could result in the CPU bug after several hours (because our backup system is every 6 hours) which required us to restart entire machines making the process very cumbersome and slow.
The delay in support tickets can be attributed to this widespread issue but I do believe we dropped the ball in that department as well. Earlier this week we hired three new support technicians: Michael, Jesse, and Steven (yes that will be two Stevens now) and we are reworking our entire support cycle to improve response times and be much better geared for anything in the future.
While we come back online, please do not hesitate to contact us via support ticket to be credited for a free week of service for the troubles.
We value your business and hope to rebuild trust.
Sincerely,
Ryan Pennington
CEO, Survival Servers