$ gdb program corefile (gdb) bt
(gdb) bt full
(gdb) frame n
(gdb) info localsThat will give a good indication of where the program died.
Look at all threads
(gdb) thread apply all bt full
(gdb) p var (gdb) p *var (gdb) p *a_struct->memberYou can also describe a variable, e.g to see the definition of a structure
(gdb) ptype some_struct
c-icap: segfault at 1 ip 00007fb6704a83fe sp 00007fb66fa03a60 error 6 in srv_url_check.so[7fb6704a0000+1a000]You can take the ip value or sometimes the ip - load address of the module/library that it crashed in and pass that to addr2line which will hopefully show you the source line that caused the crash. Here's an example using the above segfault
By taking the ip address from above and subtracting the srv_url_check.so load address (we need to do this because it crashed in a loadable module) i.e
0x7fb6704a83fe - 0x7fb6704a0000 = 0x7efbwe can use that result in addr2line, i.e
$ addr2line -e /usr/local/lib/c_icap/srv_url_check.so -fCi 0x7efbwhich gave
find_by_ip /opt/c-icap-modules-git/services/url_check/user_hash.c:94From there it was a simple matter of checking the source where the problem became obvious.