Bulk data transfer is software-based mechanism designed to move large data file using compression, blocking and buffering methods to optimize transfer times.

Every single second of everyday, data moves from computer to another. Some move over the local area network, other over the wide area network or over the internet.

Data can move from storage to another different physical media. These media can include telephone lines, fiber optics, infra reds, radio signals and many more. But since these media have different capacities and transmission speeds, they need a software to manage the transfer of data from one storage to another. These software can manage bulk data traffic and at the same time perform compression processes to minimize sizes and therefore save bandwidth.

One of the most common ways to transfer bulk of data is by using the File Transfer Protocol (FTP). FTP is a client-server protocol which means that one computer acts as the server and another as the client. The FTP server runs a server software and constantly listens to requests on a network until a client initiates a request for file transfer. When the connection is established, the client can already do many sorts of files manipulation such as downloading, changing files names, creating folders, moving files between folders, and deleting files. An FTP server can services many simultaneous clients depending on the settings.

FTP runs over the TCP layers of the TCP/IP, a suite of protocols that the internet is heavily depending on for communication. FTP listens for connection on port 21.

Despite FTP’s widespread popularity as a means to transfer bulks of data, it is not without its problem. One of the biggest problems with FTP is security. Many FTP clients do not have secure methods of encryption and usernames, passwords, configurations, transferred files may be viewed by anyone on the same network using some tools like packet sniffer. But this can be easily overcome by using SSH files transfer protocol (SFTP) or FTP over SSL (FTPS) which adds TLS or SSL encryptions to the FTP.

A protocol which specifically serves bulk data transfers is the Network Block Transfer (NETBLT). This is a transport level protocol designed to rapidly transfer very large quantity of data over the network. The protocol is structured to provide maximum efficiency and the flow is very controlled for better reliability.

NETBLT works by opening connection between clients and making a buffer which is a series of data aggregates. Because of the large bulk of data, the client will not be required to provide all the data at once. Instead, the data will have to be stored temporarily in the buffer. NETBELF then transfers the buffered data as a packet sequence. 

NETBLT’s buffer is only limited by the amount of memory on the client and the protocol requires a memory size that is as large as possible to improve performance. NETBELF in particular does not keep buffer copies for retransmission.

Another way to transfer bulk data is through peer to peer protocol. The problem with FTP is that the server may be overloaded by many simultaneous client connections and requests which can stress and break server due to heavy workload.

Computers using peer to peer protocol to transfer bulk data are specifically designed for reducing client loads for popular files like multimedia files. Peer to peer uses adaptive technology by employing randomized back off strategy to control loads on server precisely. Computers form a mesh and data gets through the mesh from neighboring computers on the network so load is shared.