Life In 19x19
http://www.lifein19x19.com/

We're Back! Extended Downtime Postmortem and Future Plans
http://www.lifein19x19.com/viewtopic.php?f=9&t=14008		 Page 1 of 2
Author:  apetresc [ Sat Feb 18, 2017 4:38 pm ]
Post subject:  We're Back! Extended Downtime Postmortem and Future Plans
Hey everyone! As you've all certainly noticed by now, Life in 19x19 was down for about 11 days. The good news is that we're now back and (hopefully) better than ever!


What Happened

Without going into too much technical detail, there was some sort of quota that was hit on the database server hosted by GoDaddy. Jordus (the admin who owned the actual hosting account) was having trouble getting timely action from GoDaddy, so not much progress was being made. After about a week, Brian Kirby and I offered to move everything to my AWS account, where we would have complete access to both the hardware and software, instead of being at the mercy of GoDaddy's tech support. At first we were scraping together old backups from Kirby's hard drive and what was left of the FTP server (which were over a year old! :( ), but eventually Jordus saved the day and gave us a raw dump of the database just prior to the outage. After some scripting, I had the full site up on an EC2 instance (for phpBB/Apache) and RDS instance (for MySQL).

The Future
Obviously the length of this downtime re-emphasized the need to reduce our bus factor. There were a couple of us with admin access to the board, but only two that could directly connect to the database, only one that controlled the domain, and 0 of us that had root on the physical hardware everything ran on (thanks GoDaddy). Going forward, we're going to be:
  • Posting the phpBB source, with all our theming and modifications, to a public GitHub organization, so anyone can clone it and help develop plugins and themes.
  • Scheduling automated daily backups of the DB. (This is actually already done as of today)
  • Make a sanitized (i.e, PMs and password hashes removed, etc) copy of this daily backup publicly available, as Linus Torvalds himself recommends ;)
  • Get a few more technically-savvy admins access to the AWS account this is running on. More on this soon.
  • We're running on a downright ancient version of phpBB (3.0.8, from November 2010 !!). Once everything's calmed down, I've verified the backup procedures work, and the versioning to GitHub is complete, I'm going to run an upgrade to the latest phpBB 3.2. Security and performance are the main benefits. This should be complete by end of day Monday February 20th.

Are we missing anything? Are there any other points of failure the community wants to see plugged?

Known Issues
Pretty much every feature that existed before the downtime has been restored. The below are the issues that I'm aware of, but decided weren't urgent enough to delay launch any further:
  • The database backup we were operating on had some character encoding issues; as such, you may notice some UTF-8 characters (e.g, accented names like Törmänen, or CJK ones like 古力) in usernames and topics be malformed. If that is the case, please contact an admin/mod and we will take care of it. I've fixed a couple of the glaring ones already, but I'm sure some have escaped me. NOTE: This should not affect actual post text, since that was binary-encoded.
  • The search index is being rebuilt overnight. Until then, search terms won't work for any topics older than today's.
  • For the next ~48 hours, your DNS settings may flip back and forth, leading you back to the old site (i.e, the error page). This will just solve itself within a day or two, as the new address reaches all the corners of the world.
If you notice any other problems not on this list, please reply to this thread with it!


Good luck and have fun, everyone :D
-Adrian
Author:  jeromie [ Sat Feb 18, 2017 8:34 pm ]
Post subject:  Re: We're Back! Extended Downtime Postmortem and Future Plan
Glad the site is back. There are both content and members here that would be sorely missed if the go community were to lose them.

I'm moderately tech-savvy; let me know if there's anything I can do to help.
Author:  Kirby [ Sat Feb 18, 2017 9:15 pm ]
Post subject:  Re: We're Back! Extended Downtime Postmortem and Future Plan
I'd like to extend thanks to Adrian, once again, who offered to host us on his AWS instance, and who also set things up with a pretty quick turnaround.

He received the database backup yesterday, and we are up and running today!

With multiple people having access to the AWS account, along with a daily backup of the database publicly available, we should be able to avoid the problem we had this time around - we won't be bottle-necked in fixing the site if an unexpected area fails.
Author:  bayu [ Sun Feb 19, 2017 2:43 am ]
Post subject:  Re: We're Back! Extended Downtime Postmortem and Future Plan
Thank you all for resolving this!

I've got one point to add to the list:
Hopefully there won't be a next time, but I d appreciate it in case of long downtime if there was, after some delay of course, a more informative error message saying "give us a week or two" or pointing to senseis or reddit where there was some status update.
Author:  ez4u [ Sun Feb 19, 2017 6:02 am ]
Post subject:  Re: We're Back! Extended Downtime Postmortem and Future Plan
Many, many, many thanks to Adrian and Brian (Apetresc and Kirby)!!!
:clap: :clap: :clap: :clap: :clap: :clap:
Author:  apetresc [ Sun Feb 19, 2017 6:37 am ]
Post subject:  Re: We're Back! Extended Downtime Postmortem and Future Plan
Marcel Grünauer wrote:
When logged in, my surname "Grünauer", shows up as "Grünauer" in the upper left and also in the username for this post.

Yup, that's the sort of encoding problem I was referring to in the "Known Issues" part. Thanks for pointing it out, I've fixed it now :)

bayu wrote:
Hopefully there won't be a next time, but I d appreciate it in case of long downtime if there was, after some delay of course, a more informative error message saying "give us a week or two" or pointing to senseis or reddit where there was some status update.

Yeah, for sure. Sometimes it's not possible to put an error message on the lifein19x19.com URL itself, depending on the nature of what the problem is, but at the very least Reddit/SL.
Author:  Gomoto [ Sun Feb 19, 2017 8:28 am ]
Post subject:  Re: We're Back! Extended Downtime Postmortem and Future Plan
Thanks a lot, keep up your outstanding work!
Author:  joellercoaster [ Sun Feb 19, 2017 9:01 am ]
Post subject:  Re: We're Back! Extended Downtime Postmortem and Future Plan
*dances*
Author:  apetresc [ Sun Feb 19, 2017 4:53 pm ]
Post subject:  Re: We're Back! Extended Downtime Postmortem and Future Plan
Progress Report

  • Fixed a serious bug with the [sgf] tag when referencing an attachment rather than an inline SGF. The links had been generated, at that time, with the /forum path, but we no longer use that. So I added a rewrite rule, and they all work now.
  • We have a GitHub organization, and the forum code is up there.
  • The search index has been rebuilt, so your searches should now work!
Author:  Bonobo [ Sun Feb 19, 2017 5:53 pm ]
Post subject:  Re: We're Back! Extended Downtime Postmortem and Future Plan
Wow, SO happy to be here again.

HUGE THANKS to everybody involved ♥

I had reloaded my “view unread posts” tab almost every hour, only to get “The host does not exist.”, but just a minute ago Schachus wrote on the German DGoB forum that the URL without the WWW, i.e. http://lifein19x19.com/, DOES work, while the URL with WWW doesn't.

So:
http://www.lifein19x19.com/ BAD
http://lifein19x19.com/ GOOD
Would be nice to have this resolved, too.

• Also, could we get https?

• And about the “Donate” button … does it point to the correct PayPal acct already?


Thanks folks, you’re cool!
Author:  apetresc [ Sun Feb 19, 2017 6:10 pm ]
Post subject:  Re: We're Back! Extended Downtime Postmortem and Future Plan
Bonobo wrote:
I had reloaded my “view unread posts” tab almost every hour, only to get “The host does not exist.”, but just a minute ago Schachus wrote on the German DGoB forum that the URL without the WWW, i.e. http://lifein19x19.com/, DOES work, while the URL with WWW doesn't.

So:
http://www.lifein19x19.com/ BAD
http://lifein19x19.com/ GOOD
Would be nice to have this resolved, too.

Good catch. I've just added the DNS entry and VirtualHost for www.lifein19x19.com too, it should start working in a few hours, again as DNS propagates. Thanks! :)

Bonobo wrote:
• Also, could we get https?

Yes! Now that Let's Encrypt is giving out free certificates, that's a possibility. I'll add that to the roadmap.

Bonobo wrote:
• And about the “Donate” button … does it point to the correct PayPal acct already?

Nope, haven't sorted that part out at all yet. I guess I could remove the button for now.
Author:  Bonobo [ Sun Feb 19, 2017 6:27 pm ]
Post subject:  Re: We're Back! Extended Downtime Postmortem and Future Plan
apetresc wrote:
[..]

I've just added the DNS entry and VirtualHost for http://www. [..]
Awesome :)

Quote:
Quote:
• [..] https?
[..] I'll add that to the roadmap.
:-)

Quote:
Quote:
• [..] “Donate” button [..]?
[..] I guess I could remove the button for now.
NOOOOOO, totally inacceptable, a new and valid button, please :twisted:
Author:  jeromie [ Sun Feb 19, 2017 6:45 pm ]
Post subject:  Re: We're Back! Extended Downtime Postmortem and Future Plan
Thanks for getting everything running again!
Author:  xiayun [ Sun Feb 19, 2017 10:45 pm ]
Post subject:  Re: We're Back! Extended Downtime Postmortem and Future Plan
Thanks so much for all the efforts to get the site back online!
Author:  daal [ Mon Feb 20, 2017 3:17 am ]
Post subject:  Re: We're Back! Extended Downtime Postmortem and Future Plan
I just want to add my appreciation that Jordus, Brian and Adrian had the will and technical wherewithal to turn a database dump back in to L19 gold. Thanks for your time and work!
Author:  Jhyn [ Mon Feb 20, 2017 6:04 am ]
Post subject:  Re: We're Back! Extended Downtime Postmortem and Future Plan
Thank you all for your hard work and long life to the nineteen.
I blame the outage for my bad tournament results this weekend.
Author:  nukeu666 [ Mon Feb 20, 2017 8:04 am ]
Post subject:  Re: We're Back! Extended Downtime Postmortem and Future Plan
Godaddy is probably the worst well-known host today, congrats on finally moving off them
Easier way to get https is go through cloudflare, only a few dns changes needed, unsure about cost vs traffic though
Author:  Bonobo [ Mon Feb 20, 2017 8:14 am ]
Post subject:  Re: We're Back! Extended Downtime Postmortem and Future Plan
One thing I notice is that pages seem to load a lot faster.
(Or is it the “Safari is snappier now after the OS update” illusion? :D )

I’m even tempted to raise the number of displayed comments on page … you’ll all remember the many problems we had loading with loading pages … would you recommend trying it or should I better not?

<edit>
Oh, and could we have two shrines, one for the folks who managed the site before the resurrection, another for the new admins? I’d want to burn some incense there :-)
</edit>
Author:  Jordus [ Mon Feb 20, 2017 7:57 pm ]
Post subject:  Re: We're Back! Extended Downtime Postmortem and Future Plan
My apologies for the long downtime. :oops: Apetresc gave a good summary of the issue, the root problem was our original hosting service changed terms and put a 1gb size limit on the database. We outgrew that limit. I did not receive any warnings about the database and was not monitoring for this as I was unaware of the service change. The hosting service locked the database down which caused the database error everyone was seeing at the beginning of our downtime. I had made decisions throughout our downtime that I thought would help us get back up faster. Unfortunately it all backfired. I won't go into more details here but the short of it is after $500 and a lot of wasted time things went nowhere. I also was not able to be as attentive to the issue as I would have liked due to demands of the job(Network Engineer w/ on-call schedule falling in the middle of the issue.... :blackeye: ) and family (wife and 3 young children.... help me? ;-) ).

Again, I humbly apologize for the downtime. As a member of the tech community I take it as a personal failing on my part. Despite issues with our hosting I should have had contingencies in place. I've learned a lot since Lifein19x19 first came into existence. There are somethings I think I would have done differently and somethings I know could have been done better as I looked back at my old work during this migration process and thought "What was I doing when I did that?", "Oh wow I was definitely an amateur then.", "OMG what was I thinking?! :shock: :-? :oops: ". As Apetresc pointed out in his posting we now know where we came short and will work to make this not happen again.

Special thanks to Kirby for facilitating the communications with Apetresc to get us back up and running. Without Kirby we still may not have been back up at this time. :salute:

Also special thanks to Apetresc for using his servers and technical expertise to get us back up and running. :batman:

I'm glad that while the site was unavailable for some time, we managed to keep the data intact and prevent the loss of this treasure of data this forum holds like that which we felt from the loss of our predecessor site.

My Humblest Apologies,
Jordus

Long live Lifein19x19!!!!! :white: :bow: :black:
Author:  Bonobo [ Mon Feb 20, 2017 9:57 pm ]
Post subject:  Re: We're Back! Extended Downtime Postmortem and Future Plan
Thank you, Jordus!

I appreciate how all awesomely you all co-operated in reanimating L19 again, and it’s very nice to see you’re still on board; knowing how voluntary work can sometimes “grow over one’s head”, as we say in Germany, I had feared you’d opted out. Everybody else also still there?

This here is one of the friendliest places I’ve ever been on the ’net. Thanks for that also to all former and current team members.

♥

(dammit, can we have font sizes >200 pls? That’s supposed to be a HUGE heart :roll: )
Page 1 of 2 All times are UTC - 8 hours [ DST ]
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/