Re: [Fwd: Re: Crash noticed multiple times using libssh 0.3.4]

[Uday Tennety <uday.tennety@xxxxxxxxxxxxxx>]

Hello Aris,
I tried initialization using ssh_init() before calling any of the ssh functions but I still notice the crash. I've attached the relevant log statements. We're actually using one thread to read from the channel and another thread to write. Do you think this might be the reason for the crash? If so, what kind of workarounds do you recommend?

Thanks for your time and help.

using pstack:
------------
-----------------  lwp# 149 / thread# 149  --------------------
 ff2c16e8 _lwp_kill (6, 0, ff2a4d28, ffffffff, ff2e8298, 6) + 8
 ff240158 abort    (2d740, 1, feb00ab0, a8244, ff2eb298, 0) + 110
 feaf67b4 __1cCosFabort6Fb_v_ (1, fedca58c, 1, fedb2000, 1858c, 18400) + 5c
 fec088d8 ???????? (feded4a8, 0, 1, fed6095b, fed67006, fedf2ce8) + 112180
 fe5b91e8 JVM_handle_solaris_signal (b, dec7d670, dec7d3b8, afc00, 0, dec7fa58) + a8c
 ff2c0620 __sighndlr (b, dec7d670, dec7d3b8, fe5b8724, 0, 1) + c
 ff2b5718 call_user_handler (b, ffbffeff, c, 0, fbeddc00, dec7d3b8) + 3b8
 ff351040 memset   (362d00, 4, fbc5d6a0, fbc5d978, 1fe, 362d00) + 140
 fbc37210 ssh_connect (362d00, 362868, e0c13c00, e0c01c00, e0c13c00, 362d00) + 330
 e0b2e9f8 gne_ssh_connect (20202020, 20202020, 20202020, 20202020, 20202020, 20202020) + 84

using dbx:
----------
..............
Reading libgcc_s.so.1
Reading nss_files.so.1
t@149 (l@149) terminated by signal ABRT (Abort)
0xff2c16e8: __lwp_kill+0x0008:  bcc,a,pt  %icc,__lwp_kill+0x18  ! 0xff2c16f8
Current function is ssh_log
   67         memset(indent, ' ', min);
(dbx) where
current thread: t@149
  [1] __lwp_kill(0x0, 0x6, 0x0, 0x6, 0xfc00, 0x0), at 0xff2c16e8
  [2] raise(0x6, 0x0, 0xff2a4d28, 0xffffffff, 0xff2e8298, 0x6), at 0xff25ff38
  [3] abort(0x2d740, 0x1, 0xfeb00ab0, 0xa8244, 0xff2eb298, 0x0), at 0xff240158
  [4] os::abort(0x1, 0xfedca58c, 0x1, 0xfedb2000, 0x1858c, 0x18400), at 0xfeaf67b4
  [5] VMError::report_and_die(0xfeded4a8, 0x0, 0x1, 0xfed6095b, 0xfed67006, 0xfedf2ce8), at 0xfec088d8
  [6] JVM_handle_solaris_signal(0xb, 0xdec7d670, 0xdec7d3b8, 0xafc00, 0x0, 0xdec7fa58), at 0xfe5b91e8
  [7] __sighndlr(0xb, 0xdec7d670, 0xdec7d3b8, 0xfe5b8724, 0x0, 0x1), at 0xff2c0620
  ---- called from signal handler with signal 11 (SIGSEGV) ------
  [8] _memset(0xdec7d758, 0x20202020, 0xffffffd7, 0x17, 0xffffd740, 0xdec80000), at 0xff351040
=>[9] ssh_log(session = 0x20202020, verbosity = 538976288, format = 0x20202020 "<bad address 0x20202020>", ...), line 67 in "log.c"
  [10] ssh_connect(session = 0x20202020), line 510 in "client.c"


Thanks,
Uday.


Uday Tennety wrote:

Thank you all for your inputs. 
Aris, Thanks for your help. I will upload the file today and send you the link. Also, As per your suggestion, I'll try calling ssh_init() function before I call the other functions. I'll also post you on that.

Thanks,
Uday.
  

---

---

Subject:

Re: Crash noticed multiple times using libssh 0.3.4

From:

Aris Adamantiadis <aris@xxxxxxxxxxxx>

Date:

Sat, 16 Jan 2010 11:41:48 +0100

To:

uday.tennety@xxxxxxxxxxxxxx

To:

uday.tennety@xxxxxxxxxxxxxx

Hi Uday,

Could you provide me the backtrace that is located in the java crash log file ? I'm interested to know exactly where in libssh the broken memset happen. Just cut anything you find not relevant and/or confidential.

Please also call the global function once before launching the libssh threads :
int ssh_init(void);

This function initializes all the global structures in libssh, otherwise it's called right under ssh_connect, causing known reentrancy problems.
Next version of libssh will be linked with threading and mutexes libs, but meanwhile the best solution is to call ssh_init() at the begin of your program.

There is no other known reentrancy bugs.

I'm looking forward your logfile.

Regards,

Aris

Uday Tennety a écrit :

Hello,
In our development environment, we noticed that some error in libssh library resulted in crashing our application server. We saw this issue a few times. Our scenario is:
We are using libssh 0.3.4 and noticed the crash when our libssh client was trying to connect to multiple ssh servers running on various network elements. Before we dive deep into the code, I wanted to find out if there is a known threading issue in 0.3.4 with a libssh client trying to connect to multiple ssh servers?  Also, it would really help if someone can point me in the right direction looking at the error. I really appreciate your time and help.

Thanks,
Uday.



Sample Error:
-------------

[func]              leaving function ssh_disconnect line 690 in /home/public/temp/libssh-0.3.4/libssh/client.c
[func]             entering function ssh_cleanup line 91 in /home/public/temp/libssh-0.3.4/libssh/session.c
[func]              entering function channel_free line 724 in /home/public/temp/libssh-0.3.4/libssh/channels.c
[func]              leaving function channel_free line 755 in /home/public/temp/libssh-0.3.4/libssh/channels.c
Current time:Fri Jan 15 16:10:29 PST 2010
Remote host is 10.2.32.95 and remote port is 22[func] entering function ssh_connect line 489 in /home/public/temp/libssh-0.3.4/libssh/client.c
[func]  entering function ssh_connect_host line 184 in /home/public/temp/libssh-0.3.4/libssh/connect.c
[func]  leaving function ssh_connect_host line 249 in /home/public/temp/libssh-0.3.4/libssh/connect.c
[func] leaving function ssh_connect_host line 258 in /home/public/temp/libssh-0.3.4/libssh/connect.c
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0xff351040, pid=5438, tid=149
#
# JRE version: 6.0_16-b01
# Java VM: Java HotSpot(TM) Server VM (14.2-b01 mixed mode solaris-sparc )
# Problematic frame:
# C  [libc_psr.so.1+0x1040]  memset+0x140
#
# An error report file with more information is saved as:
# /home/IEM/Repository/hs_err_pid5438.log
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp