Jump to content
KittyBoy

[2019-12-07] Server failure incident and rollback

Recommended Posts

Greetings Everyone,

At around 13:41 2019-12-07 the servers stopped working and was not possible to login and players in-game was stuck, the following is report on what happened and the revert back.

Around a week ago we did few changes on our backup system to reduce a case where the server would freeze for seconds twice a day, however the backup script had a little mistake caused it to backup a lot of unnecessary information and also not delete it as soon as the backups was moved to our backup storage, due to this mistake at around 13:01 the server had it storage fully used by the garbage left on our game server and our database server which is hosted on the same machine was not able to commit the changes to disk, in which after the ram was fully filled with uncommited changes also caused the database server to no longer process the game server requests to add more data and caused it to freeze.

Due to the previous errors also the future backups was not being collected and last viable backup was on 12:31 7th December which is around 30 minutes of when the incident happened and 1 hour from when the servers fully froze and stopped working, we had to revert to this backup to ensure no data corruptions happened during the time the server was going wiggly wiggly.

We sincerely apologize for this and we're now taking further actions to ensure this incident won't happen again.

With this said, enjoy double experience and double drop rate for the coming 24 hours.

Share this post


Link to post
Share on other sites
On 12/26/2019 at 11:05 PM, jhusz said:

why i cant play this server on my android phone?? always said failed to connet to master server?? anyone knows?

 

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...