digital-domain.net
Some basic GDB tips
Examine a core file with GDB
$ gdb program corefile
(gdb) bt
(gdb) bt full
(gdb) frame n
(gdb) info locals
That will give a good indication of where the program died.
Look at all threads
(gdb) thread apply all bt full
Examine variables
(gdb) p var
(gdb) p *var
(gdb) p *a_struct->member
You can also describe a variable, e.g to see the definition of a structure
(gdb) ptype some_struct
addr2line
When there is no core dump available you might still be able to get a good idea
where it crashed. If you see lines like the following in dmesg then you may be
in luck
c-icap[21154]: segfault at 1 ip 00007fb6704a83fe sp 00007fb66fa03a60
error 6 in srv_url_check.so[7fb6704a0000+1a000]
You can take the ip value or sometimes the ip - load address of the
module/library that it crashed in and pass that to addr2line which
will hopefully show you the source line that caused the crash. Here's an
example using the above segfault
By taking the ip address from above and subtracting the srv_url_check.so load
address (we need to do this because it crashed in a loadable module) i.e
0x7fb6704a83fe - 0x7fb6704a0000 = 0x7efb
we can use that result in addr2line, i.e
$ addr2line -e /usr/local/lib/c_icap/srv_url_check.so -fCi 0x7efb
which gave
find_by_ip
/opt/c-icap-modules-git/services/url_check/user_hash.c:94
From there it was a simple matter of checking the source where the problem
became obvious.