[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: multithreading for a single session



It is possible to use a single session in a multi-threaded context as long as you protect access to the session (and associated channels) using a session mutex. It's very inefficient however, as every channel read operation must timeout in order to release the mutex when the associated channel is inactive so other channels can be processed. This timeout is the source of the performance penalty, it decimates bandwidth. I've seen single threaded applications capable of 30-40 mbs being reduced to 256kps using the multi-threaded approach.

My favored solution these days is to use a boost io_service to peek at the session socket, whenever data needs to be processed, an asynchronous handler is invoked which processes the active channels.

On 14/03/12 13:38, Eddy Valdes wrote:
Jeetu,

I seem to recall somebody telling me that the if you are going to use threads, then you must use a separate session for each thread. The application that used libssh that I wrote and used to maintain had a separate session for each thread and it worked well.

Eddy

On 03/14/2012 09:31 AM, jeetu.golani@xxxxxxxxx wrote:
Hi guys,

Just wanted to know if anyone has had a chance to look into the
possibility of multithreading over a single session which I had
discussed in any earlier mail here (pasted below).

For now for my open source eBrainPool (ebrain.in) project I have gone
ahead with piped openssh server and daemon however this is not ideal
for variety of reasons. I would love to use libssh however the issues
I highlighted earlier prevent me from doing that.

Please let me know how I can help with this :)....thanks again for all
the help everyone here has provided me with in the past :)

Regards,
Jeetu
ebrain.in | Beehive Computing
Share your software and computing resources - discover and run
software from any device around you.

On Wed, Feb 15, 2012 at 2:09 PM, jeetu.golani@xxxxxxxxx
<jeetu.golani@xxxxxxxxx>  wrote:
oops forgot to mention the git link where this code can be seen
;)....git://ebrain.in:/libssh-server.git :)

sorry


On Wed, Feb 15, 2012 at 1:59 PM, jeetu.golani@xxxxxxxxx
<jeetu.golani@xxxxxxxxx>  wrote:
Hi Aris,

I've been continuing work with my libssh server based on suggestions you had made with regards multithreading when we were debugging issues where I
was seeing bad packet length, HMAC errors et al.

I refactored my code as you had suggested (below) such that there would be only one thread per session, although there could be multiple connects i.e. sessions. This worked fantastic for me and I was able to solve the packet corruption issues caused due to a race condition with multiple ssh loops running. I could therefore have different ssh clients connect to the same instance of the server and X forward simple x clients like xcalc, xclock,
xeyes, xgc.

Unfortunately as soon as I tried to forward more complex applications such as libreoffice, chromium etc things weren't working. Further analysis and a
look at the openssh code showed that these applications spawn multiple
connects to the same $DISPLAY. Therefore as I understand it (could be wrong) the server needs to accept and process different event connections on the
same session.

I have refactored the code again to account for this but basically this reintroduces the same race condition with multiple event handling we had seen earlier and while programs like libreoffice, chromium etc more further
along they eventually crash due to bad packet length, mac errors,etc.

I tried to protect the event handling function ssh_event_dopoll with a
mutex hoping this will resolve the issue, however it didn't make any
difference.

I'm at my wits end here and could appreciate advice here. One obvious way is for me to somehow refactor my code in a way that eludes me at the moment to get around this issue. The other could be if libssh itself would handle multithreaded event handling for the same session differently or if I am
somehow using it in a wrong way.

I have created a git repo of my code so anyone can study it. I have not yet put a GPL license header to the code (just realized that ;) ) however I don't care and in fact would be honoured if people would find some use for this code. The code will eventually be part of the eBrainPool (ebrain.in) project which I am working on which itself is GPLv3 - so feel free to use it
in any way you want.

For now I could really use some ideas on how I could either refactor my code to avoid the above issues or somehow if libssh could be modified to
handle multithreading for the same session differently.

Thanks,

Bye for now
Jeetu
ebrain.in | Beehive Computing
Share your software and computing resources - discover and run software
from any device around you.



On Mon, Nov 21, 2011 at 5:42 PM, jeetu.golani@xxxxxxxxx
<jeetu.golani@xxxxxxxxx>  wrote:
Hi Aris,

took me a lot of time and head scratching, but I narrowed it down to a
threading problem. Two of your threads are running ssh loops at the
same
I can completely appreciate the effort you must have gone through to
debug this :)...threading issues can be pesky and since I didn't
understand the internals of libssh it was proving quite difficult for
me to trace this out.....thank you so much Aris :)

I hacked your example so the server_thread() functions leaves early. It
doesn't remove the race but reduces its window. now it works for me,
and I'm
able to run an xterm with it.
I strongly recommend that you refactor your code in a singlethread
design,
and that you move the x11 callbacks back in the server loop.
I just did a cursory test of your code and it definitely runs much
better from what I could see :)....I will take your advice to heed and
try to refactor the code.

Let's discuss what libssh can do better to help you write better code.

I'm a little concerned, I would ultimately need to handle multiple
sessions.....the code you saw dealt with a single session....do you
think this would also trigger similar issues?....what precautions do
you think I need to keep in mind while I attempt this?

Thanks so much Aris, I'm sooooooo glad to finally have the answer to
my question and know that this was a problem in my code....now I can
dedicate my energies in this direction :)

Bye for now



On Mon, Nov 21, 2011 at 3:11 AM, Aris Adamantiadis<aris@xxxxxxxxxxxx>
wrote:
Hi Jeetu,

took me a lot of time and head scratching, but I narrowed it down to a
threading problem. Two of your threads are running ssh loops at the
same
time, and this cause races. The probability that the race causes
problem can
be high, because there's a trigger (data on socket) and both threads
run
almost the same code at the same time.

Thread 1:      message = ssh_message_get(session);
in server_thread()
Thread 2:                 ssh_event_dopoll(event, 1000);
in wait_for_something()

I hacked your example so the server_thread() functions leaves early. It
doesn't remove the race but reduces its window. now it works for me,
and I'm
able to run an xterm with it.
I strongly recommend that you refactor your code in a singlethread
design,
and that you move the x11 callbacks back in the server loop.

Let's discuss what libssh can do better to help you write better code.

Kr,

Aris
Le 09/11/11 12:38, jeetu.golani@xxxxxxxxx a écrit :
Hi Aris,

No probs and I completely understand and appreciate your interest in
looking into this....please let me know how I can help further :)

Bye for now

On Wed, Nov 9, 2011 at 3:32 PM, Aris Adamantiadis<aris@xxxxxxxxxxxx>
  wrote:
Hi Jeetu,

I apologize, still had no time to check it. I'm pretty busy and all
my
free time is swallowed by another project. I hope I can free myself
an
hour or two next week to debug this.

kr,

Aris

Le 3/11/11 19:13, jeetu.golani@xxxxxxxxx a écrit :
Hi Aris,

Just wanted to check in if you've had a chance to try out the libssh
server code I've sent and reproduce the errors I've been seeing?

Thanks so much again for looking into this.

Bye for now

On Sat, Oct 22, 2011 at 2:20 PM, jeetu.golani@xxxxxxxxx
<jeetu.golani@xxxxxxxxx>    wrote:
Hi Aris,

I'm attaching my proof of concept server code
as.....ebpsshd-singlesession.c has compile instructions at the
beginning of the code. You will also need to generate a key.h file
that holds the public key of the user who will be connecting to
this
server - this is tempoarary since as of now I'm not reading this
info
from an authorized_keys or something similar.

Just create a key.h file in the same directory and put something
like :

#define MY_PUB_KEY "[YOUR PUBLIC KEY WITHIN THESE QUOTES]"

Also as of now ebpsshd-singlesession listens in on port 2000. So
ssh
should connect to that port.

I also have a libssh-project-wrapper script that allows me to try
this
out without needing to install the libssh i've built. It basically
has
the content:

#!/bin/sh

export

LD_LIBRARY_PATH=/home/jeetu/utils/libssh/libssh-project/build/src:/home/jeetu/utils/libssh/libssh-project/build/src/threads
./$1

I have been testing this code with simple examples like xeyes and
xcalc. For some reason xcalc shows the problem much sooner than
with
xeyes, maybe because of the volume of data being transmitted to and
fro?

This is proof of concept code with a lot of fiddling with buffer
sizes
as I have been trying to study if any of that makes an impact
however
please point out any way you think this can be improved on :)

I'm sorry to drop this in your lap especially if it turns out it
was
some server side code issue, however I completely appreciate your
help
on this. I would like to squash this bug regardless of where it
lies
i.e. in my code or libssh, unfortunately my understanding of libssh and the ssh protocol is a little limited. However I do not want to
put
all of this load completely in your lap and if you share your
thoughts
and there's something you would like me to look into then please
let
me know.

Thanks,
Jeetu
ebrain.in | Beehive Computing
Discover and run software from any device around you - an open
source
(GPL) project.


On Fri, Oct 21, 2011 at 11:22 PM, jeetu.golani@xxxxxxxxx
<jeetu.golani@xxxxxxxxx>    wrote:
Hi Aris,

I think I'll need a proof-of-concept code to debug this. Would
you
mind
sharing your code, or it's not possible (too much integration
with
existing code).
No problem at all :) The code is an independent unit as of now
since I
wanted to make it work before I integrate it within my open source
project (eBrainPool)

I'll mail this out to you tomm (not on the machine with the code
right
now :) )

Thanks so much for looking into this.....truly appreciate it :)

Bye for now


On Fri, Oct 21, 2011 at 7:26 PM, Aris
Adamantiadis<aris@xxxxxxxxxxxx>
  wrote:
Hi Jeetu,

By seeing your logs, I understand this:
Both side have a hmac error. The openssh client sees it first,
sends
a
disconnect (that works), then there's the error in the libssh
log.

I think I'll need a proof-of-concept code to debug this. Would
you
mind
sharing your code, or it's not possible (too much integration
with
existing code).

If so, do you think I can reproduce the problem by "fixing"
samplesshd
to make new X11 channels connection connect to the local X11 unix
socket ?

Thanks.

Aris


Le 18/10/11 20:34, jeetu.golani@xxxxxxxxx a écrit :
This is not a problem and shouldn't cause invalid MAC errors.
I'm
on
travel so I'll look at the log when i'm back.
Thanks so much Aris :)

On Tue, Oct 18, 2011 at 5:52 PM, Aris
Adamantiadis<aris@xxxxxxxxxxxx>    wrote:
Hi,

This is not a problem and shouldn't cause invalid MAC errors.
I'm
on
travel so I'll look at the log when i'm back.

Aris

Le 18/10/11 12:30, u@xxxxxxxxxxxxx a écrit :
Hi all,

debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/jeetu/.ssh/id_rsa" as a RSA1
public
key
debug2: key_type_from_name: unknown key type '-----BEGIN'


On Tue, Oct 18, 2011 at 03:22:08PM +0500,
jeetu.golani@xxxxxxxxx
wrote:
Hi Aris,

I've attached a log of the libssh server
(log-1-ebpsshd-singlesession-18102011.txt) and the OpenSSH
client
(log-ssh-1-18102011.txt).
Greetings
--
Stefan Kuttler ==*== nc.netbeisser.de









Follow-Ups:
Re: multithreading for a single sessionAris Adamantiadis <aris@xxxxxxxxxxxx>
References:
Re: multithreading for a single session"jeetu.golani@xxxxxxxxx" <jeetu.golani@xxxxxxxxx>
Re: multithreading for a single sessionEddy Valdes <evaldes@xxxxxxxxxxxxxxxxxxxxxx>
Archive administrator: postmaster@lists.cynapses.org