+- Cuberite Forum (https://forum.cuberite.org)
+-- Forum: Cuberite (https://forum.cuberite.org/forum-4.html)
+--- Forum: Discussion (https://forum.cuberite.org/forum-5.html)
+--- Thread: How to report crashes - Linux (/thread-631.html)
How to report crashes - Linux - xoft - 11-21-2012
A Windows version of this guide has been around for quite some time and has proven useful. So it's about time a Linux version was written up. However, since I'm not so much skilled in Linux debugging, the guide won't be as polished.
Here we go:
If Cuberite crashes on you or stops responding, please take the time to do as much from this guide as possible - this will enable us developers to fix the issue as soon as possible for you.
Start by creating an empty folder (issue folder), where you'll be saving various files to include with the report.
The log
The log contains bits of the server's history that can come in handy for hunting a bug. It can show the number of players connected, the actions they were doing, helper objects that Cuberite created in memory for them etc.
Cuberite creates the log file in the "logs" subfolder of the folder it is run from, the file is called "LOG_<numbers>.txt". The highest numbers are the latest log. Move that file to your issue-folder.
Core dump
If your system has core-dumps enabled, it creates a "core dump" file, which is basically a snapshot of the program state at the time of the crash. This file is extremely helpful for developers to track down the crash.
The system creates the core dump file in the folder in which Cuberite is run from, and the file is usually called "core.<pid>" or "core-mcserver-<pid>". Move that file to your issue-folder.
For the core dump to be useful, you need to also copy the server executable to the issue-folder. The core cannot be opened without it.
If the process stopped responding, but didn't crash, you can use a system program to produce the core dump. First execute "ps uxaw | grep Cuberite", this will list running processes with Cuberite in their name:
Code:
pi@raspi ~/mc-server/Server $ ps uxaw | grep Cuberite
pi 22960 85.5 22.1 199316 102960 pts/0 Tl 07:47 1:37 ./Cuberite
pi 22978 0.0 0.1 2428 796 pts/0 S+ 07:48 0:00 grep --color=auto CuberiteCode:
pi@raspi ~/mc-server/Server $ gcore 22960
Program received signal SIGTTIN, Stopped (tty input).
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
[New Thread 0xad6ff460 (LWP 22974)]
[New Thread 0xadeff460 (LWP 22973)]
[New Thread 0xae88c460 (LWP 22972)]
[New Thread 0xb309c460 (LWP 22969)]
[New Thread 0xb389c460 (LWP 22968)]
[New Thread 0xb409c460 (LWP 22967)]
[New Thread 0xb4d31460 (LWP 22966)]
[New Thread 0xb5531460 (LWP 22965)]
[New Thread 0xb64ab460 (LWP 22964)]
[New Thread 0xb6cab460 (LWP 22963)]
0xb6d46230 in nanosleep () from /lib/arm-linux-gnueabihf/libc.so.6
Saved corefile core.22960
[1]+ Stopped ./CuberiteReproduce
Can you make Cuberite crash the same way? Is the crash specific to an action a player was doing (e. g. opening a chest that another player already opened)? Restart the server and try to re-do the few last steps you did when the server crashed. If it crashes again, don't forget to store the crashdump and logfile! If possible, make a subfolder in your issue-folder so that the developer can pair the log file to the crashdump file).
Get latest development version
Make a backup of your "Server" folder. Download the newest sources and compile it into the copy, and try running it. Can you still reproduce the crash? Store the crashdump file, the executable and the log file.
Live Analysis (Optional, but beneficial)
This step is only for crashes. If you know your way around Linux a bit more, you can perform the initial live analysis to help the devs out:
1, Fire up gdb, in the same folder as the Cuberite binary, by running the following command:
gdb ./Cuberite
2, gdb starts, loads everything and finally displays a "(gdb)" prompt.
3, issue the following 3 commands to make gdb save all its output to a file:
set logging file crash.txt
set logging on
show logging
This will copy all the onscreen output into a file called crash.txt.
4, issue a "run" command; gdb will run the server
5, wait for the server to crash; when it does, gdb displays the crash info and a "(gdb)" prompt again.
6, issue a "info threads" command; gdb will print out list of threads with their numbers
7, issue a "thread apply all bt" command; gdb will print out the stack trace for each thread
8, Quit gdb using the "q" command, it will ask whether to kill inferior, answer yes.
9, Copy the crash.txt file, into the issue folder; also consider copying it to the thread in which you'll report the crash directly
Post-mortem analysis (Optional, but beneficial)
This step is similar to Live analysis, but is instead performed on the already-crashed server, if you have the core dump:
1, Fire up gdb, in the same folder as the Cuberite binary, by running the following command:
gdb ./Cuberite ./<core-file>
2, gdb starts, loads everything and finally displays a "(gdb)" prompt.
3, issue the following 3 commands to make gdb save all its output to a file:
set logging on
set logging file crash.txt
show logging
This will copy all the onscreen output into a file called crash.txt.
(the rest is the same as in Live Analysis from step 5)
4, issue a "info threads" command; gdb will print out list of threads with their numbers
5, issue a "thread apply all bt" command; gdb will print out the stack trace for each thread
6, Save all the output into the issue folder; also consider copying it to the thread in which you'll report the crash directly
Hung analysis (Optional, but beneficial)
This step is only for when the server stops responding. It is similar to Live analysis, but it's done on the process while it's still hung:
1, Find the PID of the server process, using the "ps uxaw | grep Cuberite" command (PID is in the second column)
2, Fire up gdb, in the same folder as the Cuberite binary, by running the following command:
gdb --pid=<PID>
3, gdb starts, loads everything and finally displays a "(gdb)" prompt.
4, issue the following 3 commands to make gdb save all its output to a file:
set logging on
set logging file crash.txt
show logging
This will copy all the onscreen output into a file called crash.txt.
(the rest is the same as in Live Analysis from step 5)
5, issue a "info threads" command; gdb will print out list of threads with their numbers
6, issue a "thread apply all bt" command; gdb will print out the stack trace for each thread
7, Save all the output into the issue folder; also consider copying it to the thread in which you'll report the crash directly
Plugins
If you're using any plugins besides the standard Core, try disabling them (temporarily, in the settings.ini file). Does that fix the problem? If a plugin is the source of the problem, you might want to report that to the plugin's author as well. If you report such a crash to Cuberite developers, a list of used plugins is a bare minimum; you cannot go wrong if you actually copy the plugins into the issue-folder, as Cuberite developers may not have the plugins or have different versions.
Settings
Chances are that the devs are using different server settings and therefore unable to reproduce the bug. Copy your settings.ini and possibly world.ini to the issue folder, so that the developers can test with your exact setup.
Report the crash
Zip up your issue folder and upload it to a web hosting service, such as Mediafire. Create a new post in the issue tracker (https://github.com/cuberite/cuberite/issues) and put a link to the zip file in it, together with description. Consider providing as much information as possible. Posts "It crashed" won't help much, posts "Rev 415 crashed when two players opened the same chest" will be much better. Things you might want to include in the post:
- version of Cuberite used (GIT revision). Note that "current" or "latest" doesn't help at all, since it changes almost daily.
- OS version (Ubuntu / RedHat / CentOS / ..) and architecture (x86 / x64 / ARM / ...)
- location of your Server folder
- number of players on the server
- the steps you've taken in reproducing / testing
Provide more information upon request
The developer may need some additional information from you. They will reply to your bug report, so read the replies.
Be ready to re-test
Eventually, a developer will try to fix your crash. When we do, we'll ask you to re-test the newest dev version.
Additional notes
* Crashdumps are only useful if the developer has the executable that produced the crashdump. You need to include it in the report!
* If you don't believe in your English skills, try anyway, post both English and your native language description. Cuberite developers are from all around the world and might be able to understand your native language better than a google-translated English
I will update this guide with new information when necessary. Feel free to comment on it
RE: How to report crashes - Linux - xoft - 03-28-2013
I just found a great timesaver, you can produce a callstack for all threads with a single command: "thread apply all bt"
Updated the main post to reflect this.
RE: How to report crashes - Linux - xoft - 06-13-2013
Updated with info on how to create core dumps for hung processes
RE: How to report crashes - Linux - tigerw - 03-04-2014
I have an idea for an update:
Core files never seem to generate for me no matter what settings I have a VPS on, so perhaps add that you can run the server under gdb, and when it crashes, issue the generate-core-file command to get a coredump?
RE: How to report crashes - Linux - Woazboat - 05-10-2015
"set logging file crash.txt" needs to be run before "set logging on". Otherwise the log file for the current session won't be updated and the default gdb.txt file will be used instead.
RE: How to report crashes - Linux - sphinxc0re - 11-06-2015
Someone should update this post
RE: How to report crashes - Linux - Schwertspize - 11-06-2015
why not just leave the set logging file out? just use gdb.txt?
RE: How to report crashes - Linux - xoft - 11-06-2015
Strange, it worked for me the way I wrote it. I changed the sequence. And without setting the log file, gdb didn't create any logfile for me, so it needs to stay.