We're Back! Extended Downtime Postmortem and Future Plans

[Jhyn]

Thank you all for your hard work and long life to the nineteen.
I blame the outage for my bad tournament results this weekend.

[nukeu666]

Godaddy is probably the worst well-known host today, congrats on finally moving off them
Easier way to get https is go through cloudflare, only a few dns changes needed, unsure about cost vs traffic though

[Bonobo]

One thing I notice is that pages seem to load a lot faster.
(Or is it the “Safari is snappier now after the OS update” illusion? )

I’m even tempted to raise the number of displayed comments on page … you’ll all remember the many problems we had loading with loading pages … would you recommend trying it or should I better not?

<edit>
Oh, and could we have two shrines, one for the folks who managed the site before the resurrection, another for the new admins? I’d want to burn some incense there
</edit>

[Jordus]

My apologies for the long downtime. Apetresc gave a good summary of the issue, the root problem was our original hosting service changed terms and put a 1gb size limit on the database. We outgrew that limit. I did not receive any warnings about the database and was not monitoring for this as I was unaware of the service change. The hosting service locked the database down which caused the database error everyone was seeing at the beginning of our downtime. I had made decisions throughout our downtime that I thought would help us get back up faster. Unfortunately it all backfired. I won't go into more details here but the short of it is after $500 and a lot of wasted time things went nowhere. I also was not able to be as attentive to the issue as I would have liked due to demands of the job(Network Engineer w/ on-call schedule falling in the middle of the issue.... ) and family (wife and 3 young children.... help me? ).

Again, I humbly apologize for the downtime. As a member of the tech community I take it as a personal failing on my part. Despite issues with our hosting I should have had contingencies in place. I've learned a lot since Lifein19x19 first came into existence. There are somethings I think I would have done differently and somethings I know could have been done better as I looked back at my old work during this migration process and thought "What was I doing when I did that?", "Oh wow I was definitely an amateur then.", "OMG what was I thinking?! ". As Apetresc pointed out in his posting we now know where we came short and will work to make this not happen again.

Special thanks to Kirby for facilitating the communications with Apetresc to get us back up and running. Without Kirby we still may not have been back up at this time.

Also special thanks to Apetresc for using his servers and technical expertise to get us back up and running.

I'm glad that while the site was unavailable for some time, we managed to keep the data intact and prevent the loss of this treasure of data this forum holds like that which we felt from the loss of our predecessor site.

My Humblest Apologies,
Jordus

Long live Lifein19x19!!!!!

[Bonobo]

Thank you, Jordus!

I appreciate how all awesomely you all co-operated in reanimating L19 again, and it’s very nice to see you’re still on board; knowing how voluntary work can sometimes “grow over one’s head”, as we say in Germany, I had feared you’d opted out. Everybody else also still there?

This here is one of the friendliest places I’ve ever been on the ’net. Thanks for that also to all former and current team members.

♥

(dammit, can we have font sizes >200 pls? That’s supposed to be a HUGE heart )

[schultz]

A big thanks to Adrian and Brian! Thanks for all the work you did getting us back up and running.

And a big thanks go to Jordus too, for all the work you did trying to get us there as well (and all the work you've done in the past)!

This had all the rumblings of godiscussions, which was particularly painful for me as my (attempted) account creation over there happened in the start/middle of that mess (and wasn't "resolved" until this site came up). Glad to see that is not the case, and that we have a very solid sounding plan moving forward.

Being at least a little technically savvy, I would be willing to help out - but I lurk more than actively participate so I'll withhold any binding commitments at this time.

Thanks, again!

[Solomon]

Awesome job guys!

[Drew]

Thank you for adding to GitHub. Please update the first post in this thread when the forum version upgrade is complete.

[Bonobo]

Will you please comment here when the button points to the appropriate PayPal account again? I wouldn’t want to miss the opportunity to do my part to keep L19 alive and FAST

Or is everything still valid? Then please just reply with that information.

Thank you

[Kirby]

AFAIK, admins are still discussing whether a donate button is necessary.

[arnaudgo]

Hello,

I guess you meant "increase the bus factor" and not the opposite

Arnaud

[Bonobo]

AFAIK, admins are still discussing whether a donate button is necessary.

Thanks, Kirby.

TBH, being an L19 addict user for whom L19 feels like an extension of my living room (even though I mostly just read), it would make me feel comfortable if I knew that the L19 crew always had enough money in the L19 pot, that $$ would not be a problem to keep this up. I’m sure there’s a lot of other users who feel similarly.

[goTony]

Well done and welcome back. Just to recho it would be a shame if all the information here was lost. Thanks for all the effort!!!!