need help to debug SIGSEGV in ssl3_get

On 24 September 2014 10:03, Michael Menge <

Post by Michael Menge
Hi,
Last week i asked on openssl-user Mailinglist about an SIGSEGV
in Cyrus-Imapd 2.4.17 which seems to be received in ssl3_get_message
or a function called by ssl3_get_message, but received no reply.
I asked on the cyrus mailinglists as well, but the developer have no idea
what could cause this. Can you help me to debug this further?
Below is the BT of one of the core dumps
Regards
Michael Menge

What OS/platform is this, and what version of OpenSSL?

Matt

Andy Polyakov

2014-09-24 10:16:32 UTC

Post by Michael Menge
Last week i asked on openssl-user Mailinglist about an SIGSEGV
in Cyrus-Imapd 2.4.17 which seems to be received in ssl3_get_message
or a function called by ssl3_get_message, but received no reply.
I asked on the cyrus mailinglists as well, but the developer have no idea
what could cause this. Can you help me to debug this further?
Below is the BT of one of the core dumps
Regards
Michael Menge
What OS/platform is this, and what version of OpenSSL?

Also, run 'disass ssl3_get_message' at debugger prompt, advance to
vicinity of address provided in back-trace, 0x00007fe5a839334f in
provided example, and send that page. I mean it's lesser point to send
whole disass output, limit it to ~20 lines.
______________________________________________________________________
OpenSSL Project http://www.openssl.org
Development Mailing List openssl-***@openssl.org
Automated List Manager ***@openssl.org

Michael Menge

2014-09-24 11:43:51 UTC

0x00007fe5a839331d <+1069>: callq *%r10
0x00007fe5a8393320 <+1072>: jmpq 0x7fe5a8392f4c <ssl3_get_message+92>
0x00007fe5a8393325 <+1077>: movslq 0x60(%rbx),%r8
0x00007fe5a8393329 <+1081>: mov 0xa0(%rbx),%rdx
0x00007fe5a8393330 <+1088>: mov %rbx,%r9
0x00007fe5a8393333 <+1091>: mov 0x50(%rbx),%rbp
0x00007fe5a8393337 <+1095>: mov (%rbx),%esi
0x00007fe5a8393339 <+1097>: xor %edi,%edi
0x00007fe5a839333b <+1099>: mov 0x8(%rbp),%rcx
0x00007fe5a839333f <+1103>: add $0x4,%r8
0x00007fe5a8393343 <+1107>: mov %rdx,(%rsp)
0x00007fe5a8393347 <+1111>: mov $0x16,%edx
0x00007fe5a839334c <+1116>: callq *%r10
0x00007fe5a839334f <+1119>: jmpq 0x7fe5a83930e4 <ssl3_get_message+500>
0x00007fe5a8393354 <+1124>: lea 0xfc35(%rip),%rcx #
0x7fe5a83a2f90
0x00007fe5a839335b <+1131>: mov $0x1ef,%r8d
0x00007fe5a8393361 <+1137>: mov $0x7,%edx
0x00007fe5a8393366 <+1142>: mov $0x8e,%esi
0x00007fe5a839336b <+1147>: mov $0x14,%edi
0x00007fe5a8393370 <+1152>: callq 0x7fe5a8372938 <***@plt>
0x00007fe5a8393375 <+1157>: jmpq 0x7fe5a8393282 <ssl3_get_message+914>

--------------------------------------------------------------------------------
M.Menge Tel.: (49) 7071/29-70316
UniversitÃ€t TÃŒbingen Fax.: (49) 7071/29-5912
Zentrum fÃŒr Datenverarbeitung mail:
***@zdv.uni-tuebingen.de
WÃ€chterstraÃe 76
72074 TÃŒbingen

Andy Polyakov

2014-09-24 12:17:14 UTC

Post by Matt Caswell
What OS/platform is this, and what version of OpenSSL?

The challenge here is to map this to source code. I was hoping that call
would be direct, i.e. not *%r10, which would make the task easier. Could
you 'disass 0x7fe5a83930e4' and see if there are direct calls nearby
that location. You see a direct call below. Another option is to make
your libssl.so binary for download somewhere. It's probably better.
Could you? [Feel free to post link to me personally].
______________________________________________________________________
OpenSSL Project http://www.openssl.org
Development Mailing List openssl-***@openssl.org
Automated List Manager ***@openssl.org

Michael Menge

2014-09-24 12:54:05 UTC

Post by Matt Caswell
What OS/platform is this, and what version of OpenSSL?

I don't know if it is nearby, but the closest one is

0x00007fe5a83930e4 <+500>: movl $0x1,(%r14)
0x00007fe5a83930eb <+507>: movslq 0x60(%rbx),%rax
0x00007fe5a83930ef <+511>: jmpq 0x7fe5a8392f88 <ssl3_get_message+152>
0x00007fe5a83930f4 <+516>: nopl 0x0(%rax)
0x00007fe5a83930f8 <+520>: movzbl %al,%r10d
0x00007fe5a83930fc <+524>: cmp %ebp,%r10d
0x00007fe5a83930ff <+527>: jne 0x7fe5a839324f <ssl3_get_message+863>
0x00007fe5a8393105 <+533>: movzbl (%r12),%r11d
0x00007fe5a839310a <+538>: mov 0x80(%rbx),%r10
0x00007fe5a8393111 <+545>: lea 0x1(%r12),%r8
0x00007fe5a8393116 <+550>: mov %r11d,0x3b8(%r10)
0x00007fe5a839311d <+557>: movzbl 0x1(%r12),%r9d
0x00007fe5a8393123 <+563>: movzbl 0x2(%r8),%ebp
0x00007fe5a8393128 <+568>: movzbl 0x1(%r8),%eax
0x00007fe5a839312d <+573>: shl $0x10,%r9
0x00007fe5a8393131 <+577>: or %r9,%rbp
0x00007fe5a8393134 <+580>: shl $0x8,%rax
0x00007fe5a8393138 <+584>: or %rax,%rbp
0x00007fe5a839313b <+587>: cmp 0x10(%rsp),%rbp
0x00007fe5a8393140 <+592>: ja 0x7fe5a8393295 <ssl3_get_message+933>
0x00007fe5a8393146 <+598>: test %rbp,%rbp
0x00007fe5a8393149 <+601>: je 0x7fe5a839315f <ssl3_get_message+623>
0x00007fe5a839314b <+603>: mov 0x50(%rbx),%rdi
0x00007fe5a839314f <+607>: lea 0x4(%rbp),%esi

Post by Andy Polyakov
Another option is to make
your libssl.so binary for download somewhere. It's probably better.
Could you? [Feel free to post link to me personally].

I will send you the link

--------------------------------------------------------------------------------
M.Menge Tel.: (49) 7071/29-70316
UniversitÃ€t TÃŒbingen Fax.: (49) 7071/29-5912
Zentrum fÃŒr Datenverarbeitung mail:
***@zdv.uni-tuebingen.de
WÃ€chterstraÃe 76
72074 TÃŒbingen

Michael Menge

2014-09-24 11:13:51 UTC

Post by Matt Caswell
On 24 September 2014 10:03, Michael Menge <

What OS/platform is this, and what version of OpenSSL?

SUSE Linux Enterprise Server 11 SP3 x86_64

libopenssl0_9_8-0.9.8j-0.62.1
openssl-0.9.8j-0.62.1

--------------------------------------------------------------------------------
M.Menge Tel.: (49) 7071/29-70316
UniversitÃ€t TÃŒbingen Fax.: (49) 7071/29-5912
Zentrum fÃŒr Datenverarbeitung mail:
***@zdv.uni-tuebingen.de
WÃ€chterstraÃe 76
72074 TÃŒbingen

Marcus Meissner

2014-09-24 12:56:45 UTC

Post by Matt Caswell
On 24 September 2014 10:03, Michael Menge <

What OS/platform is this, and what version of OpenSSL?

SUSE Linux Enterprise Server 11 SP3 x86_64
libopenssl0_9_8-0.9.8j-0.62.1
openssl-0.9.8j-0.62.1

in that case it crashes here:

if (s->msg_callback)
s->msg_callback(0, s->version, SSL3_RT_HANDSHAKE, s->init_buf->data, (size_t)s->init_num + 4, s, s->msg_callback_arg);

Ciao, Marcus
______________________________________________________________________
OpenSSL Project http://www.openssl.org
Development Mailing List openssl-***@openssl.org
Automated List Manager ***@openssl.org

Michael Menge

2014-09-25 08:11:17 UTC

Marcus and Andy,

thank you very much for your help so far.
I have a few more questions, see below.

Post by Marcus Meissner

Post by Matt Caswell
On 24 September 2014 10:03, Michael Menge <

What OS/platform is this, and what version of OpenSSL?

SUSE Linux Enterprise Server 11 SP3 x86_64
libopenssl0_9_8-0.9.8j-0.62.1
openssl-0.9.8j-0.62.1

if (s->msg_callback)
s->msg_callback(0, s->version, SSL3_RT_HANDSHAKE,
s->init_buf->data, (size_t)s->init_num + 4, s, s->msg_callback_arg);

So is the pointer to the callback wrong, or is the SIGSEGV in the
called function?

I searched the cyrus code for _set_msg_callback to find where the function
is registered. But could not find it, which other ssl function could be used
to register the msg_callback? Is registering the msg_callback mandatory?

In which cases would trigger calling the callback? I ask because not
all processes receive a SIGSEGV, and I want to figure out if it will
SIGSEGV every time msg_callback is called, or only in some cases.

Regards

Michael

--------------------------------------------------------------------------------
M.Menge Tel.: (49) 7071/29-70316
UniversitÃ€t TÃŒbingen Fax.: (49) 7071/29-5912
Zentrum fÃŒr Datenverarbeitung mail:
***@zdv.uni-tuebingen.de
WÃ€chterstraÃe 76
72074 TÃŒbingen

Andy Polyakov

2014-09-25 08:46:06 UTC

Post by Marcus Meissner
if (s->msg_callback)
s->msg_callback(0, s->version, SSL3_RT_HANDSHAKE,
s->init_buf->data, (size_t)s->init_num + 4, s, s->msg_callback_arg);

So is the pointer to the callback wrong, or is the SIGSEGV in the
called function?

What happens if you type just 'disass' at debugger prompt. Question is
if you see meaningful code at point of failure, at 0x80e130 in original
example. If you see meaningful instruction with reference to memory,
issue even 'info reg'. If you don't see meaningful code, then it's
likely that pointer to callback is wrong. In which case 'print $r10'
would print address of failure. $r10 is because we already established
that it was called with call *%r10.

Basically what is baffling about it is that *if* SEGV is in called
function, then stack back-trace would be more meaningful. Normally you
see meaningless back-traces when you are in assembly subroutine (because
assembly doesn't provide stack unwinding information) or when something
went terribly wrong. Assembly is excluded here...

If it's callback that crashes, then key question who sets it. Meaningful
code at point failure without meaningful back-trace should mean that
callback is static and only some function in its vicinity that could
have set it. 0x80e130 is not impossible value for application code
segment. Yet, it doesn't exclude possibility of something terribly wrong
that going wrong.

Post by Michael Menge
I searched the cyrus code for _set_msg_callback to find where the function
is registered. But could not find it, which other ssl function could be used
to register the msg_callback? Is registering the msg_callback mandatory?

No.

Post by Michael Menge
In which cases would trigger calling the callback?

It says above, if callback is set, then it's called. You probably mean
"trigger setting the callback". There are no such cases in libssl, not
that I know of...

Post by Michael Menge
I ask because not
all processes receive a SIGSEGV, and I want to figure out if it will
SIGSEGV every time msg_callback is called, or only in some cases.

Examine several core dumps...
______________________________________________________________________
OpenSSL Project http://www.openssl.org
Development Mailing List openssl-***@openssl.org
Automated List Manager ***@openssl.org

Michael Menge

2014-09-25 09:24:36 UTC

Post by Marcus Meissner
if (s->msg_callback)
s->msg_callback(0, s->version, SSL3_RT_HANDSHAKE,
s->init_buf->data, (size_t)s->init_num + 4, s, s->msg_callback_arg);

So is the pointer to the callback wrong, or is the SIGSEGV in the
called function?

(gdb) disass
No function contains program counter for selected frame.
(gdb) disass 0x000000000080e130
No function contains specified address.
(gdb) print $r10
$1 = 8446256

8446256 = 0x000000000080e130

so the pointer was wrong in the first place,
got changed or the function is not there anymore

Post by Andy Polyakov
Basically what is baffling about it is that *if* SEGV is in called
function, then stack back-trace would be more meaningful. Normally you
see meaningless back-traces when you are in assembly subroutine (because
assembly doesn't provide stack unwinding information) or when something
went terribly wrong. Assembly is excluded here...
If it's callback that crashes, then key question who sets it. Meaningful
code at point failure without meaningful back-trace should mean that
callback is static and only some function in its vicinity that could
have set it. 0x80e130 is not impossible value for application code
segment. Yet, it doesn't exclude possibility of something terribly wrong
that going wrong.

No.

Post by Michael Menge
In which cases would trigger calling the callback?

It says above, if callback is set, then it's called. You probably mean
"trigger setting the callback". There are no such cases in libssl, not
that I know of...

I was thinking of something like a signal call back which is only called
if the signal was received.

So the next step is to find the place where the callback is registerd
by cyrus and if it is changed.

--------------------------------------------------------------------------------
M.Menge Tel.: (49) 7071/29-70316
UniversitÃ€t TÃŒbingen Fax.: (49) 7071/29-5912
Zentrum fÃŒr Datenverarbeitung mail:
***@zdv.uni-tuebingen.de
WÃ€chterstraÃe 76
72074 TÃŒbingen

Martin Simmons

2014-09-25 10:32:57 UTC

Post by Marcus Meissner
if (s->msg_callback)
s->msg_callback(0, s->version, SSL3_RT_HANDSHAKE,
s->init_buf->data, (size_t)s->init_num + 4, s, s->msg_callback_arg);

So is the pointer to the callback wrong, or is the SIGSEGV in the
called function?

(gdb) disass
No function contains program counter for selected frame.
(gdb) disass 0x000000000080e130
No function contains specified address.
(gdb) print $r10
$1 = 8446256
8446256 = 0x000000000080e130
so the pointer was wrong in the first place,
got changed or the function is not there anymore

I suggest you try 'x/16i 0x80e130' as well, because disass can report "No
function contains specified address." for an address that has no symbol
information, even if it contains code.

You could also try 'info target' to see if gdb associates that address with
anything.

__Martin
______________________________________________________________________
OpenSSL Project http://www.openssl.org
Development Mailing List openssl-***@openssl.org
Automated List Manager ***@openssl.org

Michael Menge

2014-09-25 11:32:45 UTC

Post by Martin Simmons

Post by Marcus Meissner
if (s->msg_callback)
s->msg_callback(0, s->version, SSL3_RT_HANDSHAKE,
s->init_buf->data, (size_t)s->init_num + 4, s, s->msg_callback_arg);

So is the pointer to the callback wrong, or is the SIGSEGV in the
called function?

(gdb) disass
No function contains program counter for selected frame.
(gdb) disass 0x000000000080e130
No function contains specified address.
(gdb) print $r10
$1 = 8446256
8446256 = 0x000000000080e130
so the pointer was wrong in the first place,
got changed or the function is not there anymore

I suggest you try 'x/16i 0x80e130' as well, because disass can report "No
function contains specified address." for an address that has no symbol
information, even if it contains code.

(gdb) x/16i 0x80e130
=> 0x80e130: loopne 0x80e138
0x80e132: addl $0x0,(%rax)
0x80e138: rclb 0x7e(%rdx)
0x80e13b: add %al,(%rax)
0x80e13d: add %al,(%rax)
0x80e13f: add %dh,%al
0x80e141: push %rdx
0x80e142: jle 0x80e144
0x80e144: add %al,(%rax)
0x80e146: add %al,(%rax)
0x80e148: add %eax,(%rax)
0x80e14a: add %al,(%rax)
0x80e14c: add (%rax),%eax
0x80e14e: add %al,(%rax)
0x80e150: orb $0x0,0x0(%rax,%rax,4)
0x80e158: add %al,(%rax)

Post by Martin Simmons
You could also try 'info target' to see if gdb associates that address with
anything.

(gdb) info target
[...]
0x00000000007a3000 - 0x0000000000825000 is load4
[...]

--------------------------------------------------------------------------------
M.Menge Tel.: (49) 7071/29-70316
UniversitÃ€t TÃŒbingen Fax.: (49) 7071/29-5912
Zentrum fÃŒr Datenverarbeitung mail:
***@zdv.uni-tuebingen.de
WÃ€chterstraÃe 76
72074 TÃŒbingen

Andy Polyakov

2014-09-25 12:41:56 UTC

Post by Martin Simmons

Post by Marcus Meissner
if (s->msg_callback)
s->msg_callback(0, s->version, SSL3_RT_HANDSHAKE,
s->init_buf->data, (size_t)s->init_num + 4, s, s->msg_callback_arg);

So is the pointer to the callback wrong, or is the SIGSEGV in the
called function?

What happens if you type just 'disass' at debugger prompt. Question is
if you see meaningful code at point of failure, at 0x80e130 in

original

Post by Andy Polyakov
example. If you see meaningful instruction with reference to memory,
issue even 'info reg'. If you don't see meaningful code, then it's
likely that pointer to callback is wrong. In which case 'print $r10'
would print address of failure. $r10 is because we already established
that it was called with call *%r10.

(gdb) disass
No function contains program counter for selected frame.
(gdb) disass 0x000000000080e130
No function contains specified address.
(gdb) print $r10
$1 = 8446256
8446256 = 0x000000000080e130
so the pointer was wrong in the first place,
got changed or the function is not there anymore

I suggest you try 'x/16i 0x80e130' as well, because disass can report "No
function contains specified address." for an address that has no symbol
information, even if it contains code.

(gdb) x/16i 0x80e130
=> 0x80e130: loopne 0x80e138
0x80e132: addl $0x0,(%rax)
0x80e138: rclb 0x7e(%rdx)
0x80e13b: add %al,(%rax)
0x80e13d: add %al,(%rax)

This is not meaningful code and therefore crash is more likely to be
caused by corrupted data... Hmmmmm...
______________________________________________________________________
OpenSSL Project http://www.openssl.org
Development Mailing List openssl-***@openssl.org
Automated List Manager ***@openssl.org

Marcus Meissner

2014-09-25 13:38:37 UTC

Post by Martin Simmons