slow transfer speeds

Flyboy · June 22, 2016 at 3:00 PM

I have a strange issue I've been working on for a while and I can't figure out the problem, so I thought I would try here.
I have 6 conquest servers set up, through 4 different locations.
Whenever I go to transfer images between them, I only ever get 3Mbps on the transfer speeds.
The servers are connected through VPN connections, but on fast lines.
2 of them are on gigabit internet lines in 2 datacenters.

When I test my vpn connections with any other traffic between those locations, I get a steady 250Mbps or higher (tested http, smb, ftp, ...)
I even recreated my vpn's with all kind of different settings and encryption methods to see if any of them are more friendly towards the traffic conquest generates, but no luck.

I also tested by installing a second instance of conquest on one of the servers, transfers between the 2 instances are instant, as you would expect. So it does not seem to be a conquest limitation.

Has anyone experienced this before and might know of something that might be slowing down the connections, or what I could try to speed them up?

Our conquest servers are all running windows, on HP proliant servers if that helps.

dicom config:

Code

# This file contains configuration information for the DICOM server
# Do not edit unless you know what you are doing


[sscscp]
MicroPACS                = sscscp
Edition                  = Personal


# Network configuration: server name and TCP/IP port#
MyACRNema                = OSCONQUEST1
TCPPort                  = 104


# Reference to other files: known dicom servers; database layout; sops
ACRNemaMap               = acrnema.map
kFactorFile              = dicom.sql
SOPClassList             = dgatesop.lst


# Host(ignored), name, username and password for ODBC data source
SQLHost                  = localhost
SQLServer                = xxx
Username                 = xxx
Password                 = xxx
BrowseThroughDBF         = 1
DoubleBackSlashToDB      = 0
UseEscapeStringConstants = 0


# Configure database
TruncateFieldNames       = 10
MaxFieldLength           = 254
MaxFileNameLength        = 255
FixPhilips               = 0
FixKodak                 = 0
KeepAlive                = 0
LargeFileSizeKB          = 4096
PrintSquareLandscape     = 0
UseKpacsDecompression    = 1
ZipTime                  = 05:
UIDPrefix                = 1.2.826.0.1.3680043.2.135.734788.53261698
EnableReadAheadThread    = 1
PatientQuerySortOrder    = 
StudyQuerySortOrder      = 
SeriesQuerySortOrder     = 
ImageQuerySortOrder      = 
EnableComputedFields     = 1
IndexDBF                 = 1
PackDBF                  = 0
LongQueryDBF             = 1000
TCPIPTimeOut             = 300
FailHoldOff              = 60
RetryDelay               = 100
RetryForwardFailed       = 0
ImportExportDragAndDrop  = 0
QueueSize                = 1280
WorkListMode             = 0
WorkListReturnsISO_IR_100 = 1
DebugLevel               = 4
Prefetcher               = 0
LRUSort                  = 
AllowTruncate            = 
DecompressNon16BitsJpeg  = 1
UseBuiltInJPEG           = 1
LossyQuality             = 95
IgnoreOutOfMemoryErrors  = 0
NoDICOMCheck             = 0
PadAEWithZeros           = 0
FileNameSyntax           = 3


# Configuration of compression for incoming images and archival
DroppedFileCompression   = n4
IncomingCompression      = n4
ArchiveCompression       = as


# Names of the database tables
PatientTableName         = DICOMPatients
StudyTableName           = DICOMStudies
SeriesTableName          = DICOMSeries
ImageTableName           = DICOMImages
DMarkTableName           = DICOMAccessUpdates
RegisteredMOPDeviceTable = RegisteredMOPIDs
UIDToMOPIDTable          = UIDToMOPID
UIDToCDRIDTable          = UIDToCDRID


# Banner and host for debug information
PACSName                 = OSCONQUEST1
OperatorConsole          = 127.0.0.1


# Configure email of error messages
MailHost                 = 
MailPort                 = smtp
MailSignon               = 
MailFromName             = 
MailRcptName1            = 
MailCollectTime          = 1
MailWaitTime             = 10


# Configuration of disk(s) to store images
MAGDeviceThreshhold      = 0
MAGDeviceFullThreshHold  = 10240
IgnoreMAGDeviceThreshold = 0
MAGDevices               = 3
MAGDevice0               = F:\PACSIMAGES\
MAGDevice1               = D:\PACSIMAGES\
MAGDevice2               = E:\PACSIMAGES\
NightlyCleanThreshhold   = 0
NightlyMoveThreshhold    = 0

Display More

marcelvanherk · June 22, 2016 at 6:14 PM

Hi,

DICOM uses a lot of acknowledgements during transfer, so you may be hit by a long latency? Can you look at the latency e.g. using ping?

Marcel

Flyboy · June 22, 2016 at 6:54 PM

Our latency is around 20ms between the servers I am currently testing with
The speed I get stuck at is always 3.1Mbps, doesn't matter if they are large images, small images, uncompressed, jpg compression, ...

Any help would be greatly appreciated

marcelvanherk · June 22, 2016 at 10:19 PM

Hi,

the latency could be the issue. I suggest to get a version to you where you can change the maximum sub length from 16384 to something bigger. Can you setup test servers (e.g. on other ports) to investigate this issue? I can then do this in 1.4.19beta. A new dgate(64).exe should suffice.

regards,

Marcel

Flyboy · June 22, 2016 at 10:23 PM

I can set up test servers, no problem.

David

marcelvanherk · June 23, 2016 at 12:13 AM

Hi,

you can pickup the test exe here:

http://ingenium.home.xs4all.nl/dicomserver/dgate64pdu32768.exe

it has a hardcoded pdu size of 32768 instead of 16384. Curious what it will bring.

regards

Marcel

Flyboy · June 23, 2016 at 5:05 AM

Thanks for the quick turnaround of the test exe file
I tested with the new dgate executable and the speed stays at 3.1Mbps
Tested from conquest to conquest
Tested requesting images from kpacs in the second location
Speed is the same in both instances, so it seems like an issue on the sending side on the Conquest server side

Tested with the server serving nki compressed, uncompressed dcm files and sending them both as jpg and uncompressed between servers, no difference there

marcelvanherk · June 23, 2016 at 8:46 AM

Just to check, did you see a change in the log for max pdu length?

Marcel

Flyboy · June 23, 2016 at 1:37 PM

Checked the logs from last night, only pdu length I can find is 16384
Just ran it with the new dgate executable again, even with full debug logging on, this is the only mention of pdu length I get:
6/23/2016 7:29:22 AM [CONQUEST] Application Context : "1.2.840.10008.3.1.1.1", PDU length: 16384

and just went back to double check, it was running the new version:
20160623 07:32:40 DGATE (1.4.19beta, build Wed Jun 22 22:23:34 2016, bits 64) is running as threaded server

Flyboy · June 24, 2016 at 4:35 AM

Just some more info from testing I have done between 2 virtual machines, with simulated latency introduced.
Hopefully this might help in finding some kind of solution.

Test setup:
2 virtual machines on hyper-v
each with a virtual 10gbps network card
same amount of data to be copied: 7 patients with 3.3GB of data in total

Transfer speed without any limits imposed: 450Mbps
Transfer speed with 25ms latency: 100Mbps
Transfer speed with 50ms latency: 60Mbps
Transfer speed with 100ms latency: 50Mbps
Transfer speed with 200ms latency: 8Mbps

marcelvanherk · June 24, 2016 at 5:22 PM

Hi,

I have seen a PDU size change with the new version. Are you sure both sides were new?

Marcel

Flyboy · June 29, 2016 at 7:40 PM

I redid the test and now I see the new PDU size in the logs.
Unfortunately, there is not a noticeable difference in transfer speeds or times.

marcelvanherk · June 30, 2016 at 10:25 AM

Hi,

It was worth a shot. Just got a mail from Bruce about transfer speed. I will implement his change shortly and give it to you for testing.

Marcel

Flyboy · August 11, 2016 at 3:34 PM

Just to keep other interested parties here updated in this issue:
We set up a VPN to a new medical facility we are dealing with.
When they were sending test images to confirm everything is working correctly, they were sending us images at their max. throughput speed, 200MB/sec
This over a 50ms latency line, about the same as I have been testing with internally here.
VPN setup and everything is the same as before.

So this proved to me that Conquest is able to achieve higher speeds at that latency.

Seeing that, I wanted to do some more testing here to see if I could figure out our slowness when sending
I tested sending images from different other programs.
K-PACS, same speed issue
SendToPACS (Java based), was able to send to Conquest at maximum speed.

When checking some logs I found they are sending the files in different packet sizes, but I don't know if that might be causing this or not:
[Blocked Image: http://i.imgur.com/Dt4kyuV.jpg]
[Blocked Image: http://i.imgur.com/pdWeK7f.jpg]

Tested more by tweaking network settings, network cards, vpn tunnels, ...
But I could not get Conquest to go at the correct speeds.
3Mbps seems to be what I get stuck at.

Next I hooked up a new test-lab by vpn to my live environment and sent images across.
Was I surprised when I got all of a sudden the maximum speed out of my connection.
The only difference being the test-servers running windows 2012R2 and my live ones are 2008R2
All settings, hardware, conquest versions and settings are all the same.

I spun up another 2012R2 server in a different location and was able to replicate the way improved speeds. (way improved, being: going from 3Mbps to maximum connection throughput)

Unless this finding makes a lightbulb go off with someone, my next recommendation to my customer is going to be to migrate all their image servers from windows 2008R2 to windows 2012R2

David

marcelvanherk · August 23, 2016 at 8:21 PM

Thanks for that update and an interesting problem.

There may a TCP registry setting that affects network speed at higher latencies.

Marcel

rahuldhoot2 · October 22, 2022 at 8:42 AM

sir,

i have deployed conquest server with default setup at my workplace. The issue i am facing that while send image from CR workstation (tested from other station also- issue remains the same ) ,there is so much buffering. It takes much time ( more than 30 sec ) to send the 1 image. The speed is very slow in send image to conquest.

Kindly suggest.

marcelvanherk · October 22, 2022 at 12:35 PM

Hm,

can you show a debug log of one transfer?

I nowadays see that virus scanners may cause slow down as well.

Marcel

Participate now!