It is currently Thu Oct 31, 2024 4:54 pm

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 21 posts ]  Go to page 1, 2  Next
Author Message
Offline
 Post subject: The KGS server keeps crashing
Post #1 Posted: Sat May 22, 2010 9:20 pm 
Lives in gote

Posts: 302
Liked others: 70
Was liked: 8
Rank: DDK
KGS: Sujisan 12 kyu
OGS: Sujisan 13 kyu
What's wrong with the server tonight? It's apparently crashed twice.

Hopefully, it's nothing serious.

_________________
My plan to become an SDK is here.

Top
 Profile  
 
Offline
 Post subject: Re: The KGS server keeps crashing
Post #2 Posted: Sun May 23, 2010 12:09 am 
Lives in gote
User avatar

Posts: 450
Location: Portland, OR USA
Liked others: 257
Was liked: 287
KGS: wms
Not sure what it is. It is a crash that has been there since version 3.0.0; it's a bug that I haven't been able to track down. Usually it hits about once every 60 days, but in the past 24 hours it has hit 4 times instead.

It is possible that whatever causes the bug, somebody has decided to start doing that *A LOT*. But that would be strange, because I didn't think that the bug was caused by anything a user does. It is caused by memory corruption in my low level networking code. This code is extremely tricky, it is heavily multithreaded, written in C (the only part of the server that is), and uses the epoll Linux interface, which explains why there has been a bug that I've known about for 3+ years but haven't been able to fix.

If it keeps hitting...well, that will be useful information, but I'd rather get it another way of course.

Top
 Profile  
 
Offline
 Post subject: Re: The KGS server keeps crashing
Post #3 Posted: Sun May 23, 2010 2:06 am 
Lives with ko
User avatar

Posts: 286
Location: Germany
Liked others: 146
Was liked: 81
Rank: EGF 3-4k - KGS 2-3k
Online playing schedule: A schedule..? When hell freezes over... maybe. ^^;
Well, I can't access KGS at all now (from Germany), the homepage seems down, too.
At first when I tried to login I immediately got the message that the server might be down, after waiting a bit that message took some time to appear and now there is no message at all but I still can't get onto the server, the client seems to just keep on trying and trying and... :cry:

But I guess you're already working on it, wms, so I hope that you'll fix that bug soon and everything will be allright. :)

EDIT: Please ignore the message below.. :D

EDIT 2: Well, I guess it was a bit too early to triumph.. ^^; It crashed again.

_________________
Image


Last edited by Ember on Sun May 23, 2010 2:19 am, edited 1 time in total.
Top
 Profile  
 
Offline
 Post subject: Re: The KGS server keeps crashing
Post #4 Posted: Sun May 23, 2010 2:16 am 
Lives in gote
User avatar

Posts: 450
Location: Portland, OR USA
Liked others: 257
Was liked: 287
KGS: wms
Yeah, tried a reboot to see if that helped. It looks like "no."

The server hasn't had an upgrade for months. So something new changed outside the server that made this bug show up a lot more often. No idea what.

Top
 Profile  
 
Offline
 Post subject: Re: The KGS server keeps crashing
Post #5 Posted: Sun May 23, 2010 4:03 am 
Gosei
User avatar

Posts: 1449
Liked others: 1562
Was liked: 140
Rank: KGS 6k
GD Posts: 892
There were some authentication problems with the desktop version a while back. Are you using that? either way, a screenshot or the details of the popup might help.

_________________
a1h1 [1d]: You just need to curse the gods and defend.
Good Go = Shape.
Associação Portuguesa de Go

Top
 Profile  
 
Offline
 Post subject: Re: The KGS server keeps crashing
Post #6 Posted: Sun May 23, 2010 5:17 am 
Lives in gote
User avatar

Posts: 429
Location: Sweden
Liked others: 101
Was liked: 73
Rank: SDK
KGS: CarlJung
Helel wrote:
...swedish text in picture...


Ha ha, visste inte att du var svensk.

_________________
FusekiLibrary, an opening library.
SGF converter tools: Wbaduk NGF to SGF | 440 go problems | Fuseki made easy | Tesuji made easy | Elementary training & Dan level testing | Dan Tutor Shortcut To Dan

Top
 Profile  
 
Offline
 Post subject: Re: The KGS server keeps crashing
Post #7 Posted: Sun May 23, 2010 5:43 am 
Gosei

Posts: 1348
Location: Finland
Liked others: 49
Was liked: 129
Rank: FGA 7k GoR 1297
Since my line is quite slow, I'd really appreciate if the pictures weren't several megabytes...

_________________
Offending ad removed

Top
 Profile  
 
Offline
 Post subject: Re: The KGS server keeps crashing
Post #8 Posted: Sun May 23, 2010 6:46 am 
Honinbo
User avatar

Posts: 8859
Location: Santa Barbara, CA
Liked others: 349
Was liked: 2076
GD Posts: 312
Helel wrote:
I only did it to annoy you. :twisted: It will not become a habit. ;-)
Helel, you can also try:
(1) Alt+"Prnt Scrn" to grab only the error window.
(2) Paste it in an editor like Irfanview -- http://www.irfanview.com/
(3) Reduce the screenshot size.
(4) Save as jpeg at 80% quality (see attached).
This usually reduces the file size by over 90%.
Or, you can TYPE all the error messages by hand. ;-)


Attachments:
xxx.jpg
xxx.jpg [ 61.49 KiB | Viewed 14059 times ]
xxx1.jpg
xxx1.jpg [ 55.81 KiB | Viewed 14059 times ]
Top
 Profile  
 
Offline
 Post subject: Re: The KGS server keeps crashing
Post #9 Posted: Sun May 23, 2010 6:57 am 
Gosei

Posts: 1348
Location: Finland
Liked others: 49
Was liked: 129
Rank: FGA 7k GoR 1297
Helel wrote:
Svenska talas ju till och med av skåningar och annat slödder. :twisted:

Ja, även nästan tio procent av finnar talar svenska som modersmål (jag är dock inte en av dom).

_________________
Offending ad removed

Top
 Profile  
 
Offline
 Post subject: Re: The KGS server keeps crashing
Post #10 Posted: Mon May 24, 2010 12:19 pm 
Lives in gote

Posts: 302
Liked others: 70
Was liked: 8
Rank: DDK
KGS: Sujisan 12 kyu
OGS: Sujisan 13 kyu
wms wrote:
Not sure what it is. It is a crash that has been there since version 3.0.0; it's a bug that I haven't been able to track down. Usually it hits about once every 60 days, but in the past 24 hours it has hit 4 times instead.

It is possible that whatever causes the bug, somebody has decided to start doing that *A LOT*. But that would be strange, because I didn't think that the bug was caused by anything a user does. It is caused by memory corruption in my low level networking code. This code is extremely tricky, it is heavily multithreaded, written in C (the only part of the server that is), and uses the epoll Linux interface, which explains why there has been a bug that I've known about for 3+ years but haven't been able to fix.

If it keeps hitting...well, that will be useful information, but I'd rather get it another way of course.


Hmmm...Interesting. Hopefully, you can find it and fix it.

On a lighter note, there's two quotes that I thought of.

1. "If debugging is the process of removing bugs from a program, then programming is the process in which bugs are introduced to the program."

2. "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it."

@WMS: In what way would you like to receive the information?

_________________
My plan to become an SDK is here.

Top
 Profile  
 
Offline
 Post subject: Re: The KGS server keeps crashing
Post #11 Posted: Mon May 24, 2010 12:38 pm 
Lives in gote
User avatar

Posts: 450
Location: Portland, OR USA
Liked others: 257
Was liked: 287
KGS: wms
When there's a bug, it's best if the bug shows up when I test so that I can fix it there.

But this bug happens so rarely, and I don't know how to make it happen, so I only get info about it from crashes...and very, very little info even then. All I've been able to pin down is that variables are getting utter nonsense in them. A boolean, for example, will have a number in it instead of a 0 or 1. I suspect that I'm walking past the end of an array in the network code, or something like that, but I can't figure out where that could be happening.

Top
 Profile  
 
Offline
 Post subject: Re: The KGS server keeps crashing
Post #12 Posted: Tue May 25, 2010 1:44 am 
Lives in gote
User avatar

Posts: 429
Location: Sweden
Liked others: 101
Was liked: 73
Rank: SDK
KGS: CarlJung
Helel wrote:
wms wrote:
When there's a bug, it's best if the bug shows up when I test so that I can fix it there.

But this bug happens so rarely, and I don't know how to make it happen, so I only get info about it from crashes...and very, very little info even then. All I've been able to pin down is that variables are getting utter nonsense in them. A boolean, for example, will have a number in it instead of a 0 or 1. I suspect that I'm walking past the end of an array in the network code, or something like that, but I can't figure out where that could be happening.


:twisted: :evil: :twisted: :evil: :twisted: :evil: :twisted: :evil: :twisted: :evil: :twisted:
Ever heard of open source code...
:twisted: :evil: :twisted: :evil: :twisted: :evil: :twisted: :evil: :twisted: :evil: :twisted:

Have fun debugging! :D


That wouldn't really change the nature of the bug, and there would still only be wms who has access to the server where the problem occurs. More eyes on the problem, yes, but that's it.

_________________
FusekiLibrary, an opening library.
SGF converter tools: Wbaduk NGF to SGF | 440 go problems | Fuseki made easy | Tesuji made easy | Elementary training & Dan level testing | Dan Tutor Shortcut To Dan

Top
 Profile  
 
Offline
 Post subject: Re: The KGS server keeps crashing
Post #13 Posted: Tue May 25, 2010 3:26 am 
Lives in gote
User avatar

Posts: 429
Location: Sweden
Liked others: 101
Was liked: 73
Rank: SDK
KGS: CarlJung
Helel wrote:
Ahh, so the bug is in no way related to anything wms has coded. My bad. :oops:


It's quite possible that it is, it remains to be seen. But even so, open sourceing the code wouldn't make it any easier to debug. It's on the server the error occurs, and you can't give everyone access to it to tinker away at their hearts content.

_________________
FusekiLibrary, an opening library.
SGF converter tools: Wbaduk NGF to SGF | 440 go problems | Fuseki made easy | Tesuji made easy | Elementary training & Dan level testing | Dan Tutor Shortcut To Dan

Top
 Profile  
 
Offline
 Post subject: Re: The KGS server keeps crashing
Post #14 Posted: Tue May 25, 2010 3:30 am 
Lives in gote
User avatar

Posts: 429
Location: Sweden
Liked others: 101
Was liked: 73
Rank: SDK
KGS: CarlJung
wms,

I'm sure you have a test server. Have the error ever occurred on that one? Can't we fill it with a few thousand weakbots/randombots that all play blitz in order to simulate some load? I have 10MBit upload that mostly sits idle and a quite powerful computer. I'm sure others have similar setups. That could potentially be a way forward.

_________________
FusekiLibrary, an opening library.
SGF converter tools: Wbaduk NGF to SGF | 440 go problems | Fuseki made easy | Tesuji made easy | Elementary training & Dan level testing | Dan Tutor Shortcut To Dan

Top
 Profile  
 
Offline
 Post subject: Re: The KGS server keeps crashing
Post #15 Posted: Tue May 25, 2010 3:36 am 
Gosei

Posts: 1348
Location: Finland
Liked others: 49
Was liked: 129
Rank: FGA 7k GoR 1297
CarlJung wrote:
Helel wrote:
Ahh, so the bug is in no way related to anything wms has coded. My bad. :oops:


It's quite possible that it is, it remains to be seen. But even so, open sourceing the code wouldn't make it any easier to debug. It's on the server the error occurs, and you can't give everyone access to it to tinker away at their hearts content.

What I remember from my merry days of coding and debugging C/C++ (which is what I suspect the code in question is), this kind of bug isn't often caught by debugging when the actual error occurs. The problematic code may have been executed well in advance. If the culprit is something wms wrote, then it might help to have several people look at it. Of course, if the bug may be virtually anywhere, it won't probably help unless it can be narrowed down. Of course one theoretical possibility is to run everything in debugger with watches guarding the memory that will eventually be overwritten, but that is (or at least used to be) much too slow. Perhaps debugging tools have improved since I coded for living.

_________________
Offending ad removed

Top
 Profile  
 
Offline
 Post subject: Re: The KGS server keeps crashing
Post #16 Posted: Tue May 25, 2010 8:29 am 
Lives in sente
User avatar

Posts: 918
Liked others: 395
Was liked: 163
Rank: German 2 dan
Perhaps there is a hardware error.

_________________
A good system naturally covers all corner cases without further effort.

Top
 Profile  
 
Offline
 Post subject: Re: The KGS server keeps crashing
Post #17 Posted: Tue May 25, 2010 9:27 am 
Lives in gote
User avatar

Posts: 450
Location: Portland, OR USA
Liked others: 257
Was liked: 287
KGS: wms
I had a test server (and will have one again eventually), but no, the bug has never appeared there. Not enough load I suspect.

And yes, open sourcing the code, somebody might spot the bug. I have never denied that there are some advantages to open sourcing KGS; although I still believe that for me, I prefer to keep it closed.

Top
 Profile  
 
Offline
 Post subject: Re: The KGS server keeps crashing
Post #18 Posted: Tue May 25, 2010 10:13 am 
Beginner
User avatar

Posts: 5
Location: Vienna,Austria
Liked others: 0
Was liked: 0
Rank: EGF 9 kyu
My ideas :
- malicious denial of service attack with deliberately wrong communication packets :twisted:
- rogue KGS client with reverse engineered protocol engine ( more likely ) :roll:

Sounds like something in the communication you assume never to happen happens
( length information incorrect or something...)

Well, it maybe something completley different. :oops:
I keep my fingers crossed...

Top
 Profile  
 
Offline
 Post subject: Re: The KGS server keeps crashing
Post #19 Posted: Tue May 25, 2010 4:29 pm 
Lives in gote
User avatar

Posts: 399
Location: Shanghai, China
Liked others: 5
Was liked: 35
Rank: EGF 2d KGS 3d
GD Posts: 353
KGS: freegame
Quote:
rogue KGS client with reverse engineered protocol engine

I don't think there are many of those. specially after some recent updates.

I had one that I sometimes used to watch really busy games (like tournament demo's) (It had some very useful features for that.)
But It does not work anymore because of some relatively recent updates.
It just tells me the client is "extremely out of date" :cry: and fails to connect.

I did not create it myself, so I have no knowledge about how it worked.

I haven't been on KGS much lately. I haven't had noticed any sudden disconnects due to KGS going down.

_________________
Laurens
Go school: freegame's Teaching School
Author of: The Next Move a book for 15-3kyu players.

Top
 Profile  
 
Offline
 Post subject: Re: The KGS server keeps crashing
Post #20 Posted: Tue May 25, 2010 6:49 pm 
Gosei
User avatar

Posts: 1758
Liked others: 378
Was liked: 375
Rank: 4d
Clients really shouldn't be able to crash servers. Given how infrequently the error occurs, it seems rather unlikely to me. Besides, this "certain anonymous client" we all know about has been around for a long time, but doesn't even work currently on KGS (at least the version that's available publicly doesn't).

_________________
We don't know who we are; we don't know where we are.
Each of us woke up one moment and here we were in the darkness.
We're nameless things with no memory; no knowledge of what went before,
No understanding of what is now, no knowledge of what will be.

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 21 posts ]  Go to page 1, 2  Next

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group