[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: SFTP Write async


Hi Carlos

I already implemented a basic API some time ago, which is used in production. Maybe it can serve you as a starting point. The API is very raw, but it works. It allows you to trigger an arbitrary number of concurrent async SFTP reads and writes and to fetch the results later using the request id. Look for the sftp_async_*() functions:

https://github.com/limes-datentechnik-gmbh/libssh/blob/limes/src/sftp.c

The hard parts have to be implemented by user code which is why I did not submit it to upstream. The user code has to manage lists of in-transit requests, handle write ACKs, implement a suitable read-ahead strategy (depends on read pattern, e.g. lengthy sequential reads vs. heavy seeking), proper seeks (you need to track file offsets from user perspective for working relative seeks and discard already requested blocks). Those parts should be handled by libssh in a final implementation because it's easy to produce memory leaks, lose track of the correct file offset or have an underperforming read-ahead algorithm.

The most difficult aspect is the read-ahead strategy. There is no one size fits all. If you optimize for maximum throughput by using a large read-ahead window, you will waste a lot of bandwidth/time waiting for data you no longer need if you seek a lot because there is no way in SFTP to cancel sent read requests. If you optimize for seek, you will have slower throughput.

The current SFTP API could be re-implemented using the async methods and a configurable per-file read-ahead strategy which would considerably speed up performance. Libssh users that don't write async applications but want reasonable throughput and a simple API would no longer be bound by network latency as is the case with the current implementation.

Regards,
Tilo

Am 02.06.2020 um 20:32 schrieb Carlos Cabanero:
I couldn’t find any async_write method, but some thought was put into it
from this 2014 thread:
https://www.libssh.org/archive/libssh/2014-06/0000014.html. Is this
somewhere? Are we still interested to have it and are there any suggestions?

Thanks!


References:
SFTP Write asyncCarlos Cabanero <carlosecabanero@xxxxxxxxx>
Archive administrator: postmaster@lists.cynapses.org