Some basic GDB tips

  Examine a core file with GDB

$ gdb program corefile
(gdb) bt
(gdb) bt full
(gdb) frame n
(gdb) info locals

That will give a good indication of where the program died.

Look at all threads

(gdb) thread apply all bt full

  Examine variables

(gdb) p var
(gdb) p *var
(gdb) p *a_struct->member

You can also describe a variable, e.g to see the definition of a structure

(gdb) ptype some_struct


When there is no core dump available you might still be able to get a good idea where it crashed. If you see lines like the following in dmesg then you may be in luck
c-icap[21154]: segfault at 1 ip 00007fb6704a83fe sp 00007fb66fa03a60
error 6 in[7fb6704a0000+1a000]
You can take the ip value or sometimes the ip - load address of the module/library that it crashed in and pass that to addr2line which will hopefully show you the source line that caused the crash. Here's an example using the above segfault

By taking the ip address from above and subtracting the load address (we need to do this because it crashed in a loadable module) i.e

0x7fb6704a83fe - 0x7fb6704a0000 = 0x7efb
we can use that result in addr2line, i.e
$ addr2line -e /usr/local/lib/c_icap/ -fCi 0x7efb
which gave
From there it was a simple matter of checking the source where the problem became obvious.