Monday, May 13, 2013

Using Tsunami UDP in AWS- How it helps, Configuration and Comparison


Tsunami UDP is an open source file transfer protocol that helps you with faster files transfers compared to traditional models like FTP and SCP. Tsunami UDP is designed with a fast user-space file transfer protocol that uses TCP control and UDP data for transfer over very high speed long distance networks, it helps you achieve more throughput than possible with TCP over the same networks.

In this article we are going to use Tsunami UDP to transfer files from Amazon EC2 US-East region to other AWS regions. 

Step 1:) Installing Tsunami UDP on Amazon EC2:

Tsunami UDP needs to be compiled from the source and we have used Red Hat Enterprise Linux 6.4 (RHEL 6.4) EC2 instance in the source and target regions.  Since we are using RHEL we have to ensure that firewall disabled on RHEL 6.4 EC2 instances or add rules to iptables.



TsunamiUDP needs some dependency packages and it should be installed from the repository cvs, autoconf, gcc, and automake.

Execute the following commands for Installing Tsunami UDP on Amazon EC2. The following command will install both Tsunami UDP Server and Tsunami UDP Client in the Amazon EC2 instance.


$ cvs -z3 -d:pserver:anonymous@tsunami-udp.cvs.sourceforge.net:/cvsroot/tsunami-udp co -P tsunami-udp
$ cd tsunami-udp
$ ./recompile.sh
$ sudo make install


Tsunami UDP Server and Tsunami UDP Client have options to configure lots of parameters based on the file transfer requirements. For simplicity reasons we are going with default settings.



Step 2:) Open relevant ports in Security Groups for Tsunami UDP:

Tsunami UDP uses the port 46224 on TCP and UDP. We have allowed port 22 for SSH. The following ports illustrated in the below image should be allowed in the AWS security group for Tsunami UDP to transfer files. 



Step 3) Transfer files from the Server Side


On the server side traverse to the directory/path where the files are available for transfer and start the Tsunami UDP Server daemon.
$ cd /path/to/files
For transferring single file:
$ tsunamid file1 
For transferring multiple files:
$ tsunamid file1 file2 …
For all files inside folder:
$ tsunamid *

When the file transfer is completed, you will get to know transfer details as shown below,




Transmission complete.
Server 1 transferred 536870912 bytes in 62.07 seconds (66.0 Mbps)



Step 4) Receive files in the Tsunami UDP Client side

On the client side you can receive/download the files by connecting to Tsunami UDP Server using the Tsunami UDP client.



$ cd /path/to/receive/files
$ tsunami
tsunami> connect ec2-XX-XX-XX-83.compute-1.amazonaws.com 
tsunami> get file1 file2 
tsunami> get * 



When the file transfer is completed, you will get to know transfer details like below


Transfer complete. Flushing to disk and signaling server to stop...
!!!!
PC performance figure : 32 packets dropped (if high this indicates receiving PC overload)
Transfer duration       : 62.06 seconds
Total packet data       : 9440.27 Mbit
Goodput data              : 8760.23 Mbit
File data                       : 4096.00 Mbit
Throughput                  : 152.12 Mbps
Goodput w/ restarts  : 141.16 Mbps
Final file rate               : 66.00 Mbps
Transfer mode             : lossless



Now lets see what we observed as Test results in comparison to SCP

Tsunami UDP vs SCP Comparison Matrix:


The following table illustrates the comparison (time taken) between Tsunami UDP and SCP between Amazon EC2 USA-EAST and AWS Singapore region.

Source : US-EAST  | Target: Singapore

EC2 Instance Type
File Size / Type
Tsunami UDP (time)
SCP (time)
m1.large
5GB Single File Compressed
2m 45s
-
m1.xlarge
5GB Single File Compressed
1m 28s
-
m2.4xlarge
5GB Single File Compressed
51.56s
11m 50s
m2.4xlarge
50GB Single File Compressed
19m 33s
1hr 50min


50 GB Single compressed file when uncompressed translates to 500 GB file size. 

Some Best practices while using Tsunami UDP in AWS:
Point 1) When you are moving larger files between Amazon EC2 regions, disk IO is very critical , so use RAID 0 EBS Volumes in the Amazon EC2 instances for better IO performance
Point 2) Use right EC2 instance size for better performance and lower cost. If you are going to transfer 5 GB compressed file, m1.large will be cheaper whereas if you are going to transfer 500 GB compressed file, m2.4xlarge might be faster and cheaper.
Point 3) Atleast 512 MB memory is needed for Tsunami UDP. Do not use micro, small and other EC2 instances with moderate IO and smaller memory for this purpose.
Point 4) As the instance size increases the IO capacity also increases in Amazon cloud. For faster file transfers in event of DR etc, use larger instance types. Example: In event of DR when RTO is critical, you can transfer 50GB files using m2.4xlarge than m1.large.




1 comment:

jpdave said...

thanks for the great post .... it helped me a lot... but when i try to access the server from my system I am not able to download it.

Need Consulting help ?

Name

Email *

Message *

DISCLAIMER
All posts, comments, views expressed in this blog are my own and does not represent the positions or views of my past, present or future employers. The intention of this blog is to share my experience and views. Content is subject to change without any notice. While I would do my best to quote the original author or copyright owners wherever I reference them, if you find any of the content / images violating copyright, please let me know and I will act upon it immediately. Lastly, I encourage you to share the content of this blog in general with other online communities for non-commercial and educational purposes.

Followers