SmallWeb Ltd · @SmallWebLtd
11th Jan 2020 from TwitLonger
Discontinuation of Shared-LAX1 (09/01/2020)
On the 9th January 2020 our Los Angeles server was disconnected from its network. It was available through local console only and had no connection to the internet. I believed that the network interface scripts were not functioning correctly due to accidental package deletion.
Attempts to restore the deleted packages and correct the network were attempted through recovery mode but were unsuccessful.
The network had been disconnected for several hours now and I made the decision to reinstall the server as I believed it would be the fastest way to get customers back online. The server drives were overwritten and the reinstallation process begun. During this I uploaded backups pulled from our script to an alternative server to verify their contents.
The script we use to automate backup transfers contains a list of commands that transfers backup files in a directory to a defined destination on our remote server. To categorise our hosting servers we use shorthand tags such as "AMS", "LAX" and "MEL".
The line in the script for "LAX" backups was misconfigured to receive backup files of "LUX" and the contents of it's directory contained backup files from our Luxembourg location only. This was a mistake by me when configuring the script for backups and more in-depth testing should have been done.
Because of this mistake it has become clear that additional contingencies and processes need to be put in place and I'll be working towards improving our backup policies to ensure this never happens again.
The script used for backup transfers has now been fully tested for our locations "AMS", "GER (Unreleased)", "LON", "LUX", "MEL" and "NYC". Backups of user data for these locations are locally and remotely stored and have been verified.
Release for "GER" has been postponed until new policies have been put in place. "Cloud server" snapshot backup services have been deployed for future "GER" data.
More snapshot backup services will be deployed with providers where applicable.
Additional backup storage servers will be deployed to increase backup redundancy.
For the customers who are affected by my horrendous mistakes during this, I apologise from deep within. It was never my intention for this to ever happen and for such an abrupt and disappointing end to these services.
You are 100% entitled to a refund. If there is anything I can do for you please reach out.
Michael C
SmallWeb Ltd