Streaming File send with File Inbound/File Connector

Hi all,

I think you are familiar with connector, streaming and VFS transfer. If not please refer my previous blogs. As I mentioned in the previous blog we can do the vfs sending feature by using file connector and file inbound. File connector is quite simple to use compared to the vfs sender for various file operations. Before that we will get some knowledge about inbound.

Brief intro to inbound endpoint

Inbound is a feature in WSO2 ESB. An inbound endpoint is a message entry point that can inject messages directly from the transport layer to the mediation layer, without going through the Axis engine. The following diagram illustrates the inbound endpoint architecture. For proxy, we have to invoke it. but a inbound can be invoked by itself. There are three types of inbounds in WSO2 ESB.

• Listening Inbound Endpoint: A listening inbound endpoint listens on a given port for requests that are coming in. When a request is available it is injected to a given sequence.
• Polling Inbound Endpoint: A polling inbound endpoint polls periodically for data and when data is available the data is injected to a given sequence.
• Event based Inbound Endpoint: An event-based inbound endpoint polls only once to establish a connection with the remote server and then consumes events.

Here we mostly focus on Polling Inbound Endpoint as it deal with file in periodic manner. Polling inbound endpoint available with WSO2 ESB are given below.

• File Inbound Protocol
• JMS Inbound Protocol
• Kafka Inbound Protocol

File Inbound Protocol 

The WSO2 ESB file inbound protocol is a multi-tenant capable alternative to WSO2 ESB VFS transport. In previous post we got the idea about VFS transport. Same functionality can be done through file inbound. we can specify streaming facility in WSO2 ESB 5.0 in file inbound. For Streaming we have to add message builder and formatter in the <ESB_HOME>/repository/conf/axis2/axis2.xml file.

• Add <messageFormatter contentType="application/file"
          class="org.wso2.carbon.relay.ExpandingMessageFormatter"/> under the message formatters.
• Add  <messageBuilder contentType="application/file"
          class="org.apache.axis2.format.BinaryBuilder"/> under the message builders.

Steps 

•  Start ESB 5.0 by going to <ESB_HOME>/bin.
•  You can get file connector from the store (https://store.wso2.com/store/assets/esbconnector/aec1554a-29ea-4dbb-b8c5-5d529a853aa2).
•  Add the connector by going to Connectors->Add in WSO2 ESB management console and browse connector zip file and enable it.
• Go to Service bus-> sequence  in ESB management console and add following lines in source view to specify the connector configuration.
  
  <?xml version="1.0" encoding="UTF-8"?>
  <sequence xmlns="http://ws.apache.org/ns/synapse">
         <property name="OUT_ONLY" value="true"/>
         <property action="remove" name="ClientApiNonBlocking" scope="axis2"/>
         <fileconnector.send>
          <address>file:///xxx/xxx/Desktop/testESB/OutTest</address>
          <append>true</append>
         </fileconnector.send>
      <drop/>
  </sequence>
  
  Here we specify the where do we want to send the file in address. Then save this sequence in registry.
• Then go to Service bus->Inbound Endpoints-> Add Inbound Endpoint. After that specify the name and select Type as File. Then press next. In next window you have to specify sequence. u already saved your sequence in registry. Then you can select configuration registry. Then you will get following window and select your sequence.

 

 In error sequence you can put as fault. In suspend you initially put as false. If suspend is true, then the file inbound will not invoke. Put sequential and coordination as true. Specify the FileURI where you have the file and specify content type. if you want to use streaming specify contentType as application/file. Then click on show advanced options. In that you can specify the streaming as true if you want to send big files. ActionAfterProcess is the one what do you want to do after send the file to that specific location. you can specify MOVE or DELETE. we have to specify this to avoid recursive sending the same file again and again. You can get details of other parameters in here. Then save the inbound.

• When you put a file in the fileURI location, you can see the response file in the address location you specified.

Note: You can use file send method in File connector not only for VFS transport, but also HTTP, JMS, FTP to send files.

Streaming VFS File Transfer using WSO2 ESB

Brief idea for VFS

VFS means Virtual File System. The purpose of a VFS is to allow client applications to access different types of concrete file systems in a uniform way. A VFS can, for example, be used to access local and network storage devices transparently without the client application noticing the difference.

Brief Intro to WSO2 ESB VFS transport with streaming

Through this we can pick up a file from a directory and process it within the ESB. You can get more about WSO2 ESB VFS transport from here. But when we using this VFS configurations for transferring large files (Greater that 500 MB), we get an Out Of Memory exceptions. In this time we can use streaming facility. To enable streaming, we have to add message builder "org.apache.axis2.format.BinaryBuilder" for this. Apart from that  we need to include the property "ClientApiNonBlocking" in the proxy configuration. To use vfs transport facility from WSO2 ESB, there are some steps you have to follow.

1. Enable the VFS transport by editing the <ESB_HOME>/repository/conf/axis2/axis2.xml file and uncomment the VFS listener and the VFS sender as follows:
   

   <transportreceiver name="vfs" class="org.apache.synapse.transport.vfs.VFSTransportListener"/>
   ...
   <transportSender name="vfs" class="org.apache.synapse.transport.vfs.VFSTransportSender"/>

2.  Add message builder and formatter in the <ESB_HOME>/repository/conf/axis2/axis2.xml file.
   
   • Add <messageFormatter contentType="application/file"
             class="org.wso2.carbon.relay.ExpandingMessageFormatter"/> under the message formatters.
   • Add  <messageBuilder contentType="application/file"
             class="org.apache.axis2.format.BinaryBuilder"/> under the message builders.
      
3. You have to add a proxy like below.
   
   <?xml version="1.0" encoding="UTF-8"?>
   <proxy xmlns="http://ws.apache.org/ns/synapse"
          name="FileProxy"
          transports="vfs"
          startOnLoad="true"
          trace="disable">
      <description/>
      <target>
         <inSequence>
            <log level="custom">
               <property name="FileProxy" value="Processing file"/>
            </log>
            <property name="OUT_ONLY" value="true"/>
            <property name="ClientApiNonBlocking"
                      value="true"
                      scope="axis2"
                      action="remove"/>
            <send>
               <endpoint name="FileEpr">
                  <address uri="vfs:file:///home/Desktop/testESB/out"/>
               </endpoint>
            </send>
         </inSequence>
      </target>
      <parameter name="transport.vfs.Streaming">true</parameter>
      <parameter name="transport.PollInterval">20</parameter>   <parametername="transport.vfs.ActionAfterProcess">MOVE</parameter>
      <parameter name="transport.vfs.FileURI">file:///home/Desktop/testESB/in</parameter>
      <parameter name="transport.vfs.MoveAfterProcess">file:///home/Desktop/testESB/original</parameter>
      <parameter name="transport.vfs.MoveAfterFailure">file:////home/Desktop/testESB/failure</parameter>
      <parameter name="transport.vfs.Append">true</parameter>
      <parameter name="transport.vfs.Locking">enable</parameter>
      <parameter name="transport.vfs.FileNamePattern">.*.zip|.*.txt</parameter>
      <parameter name="transport.vfs.ContentType">application/file</parameter>
      <parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
   </proxy>
   
   Here we specify ActionAfterProcess for doing some process after that file send to the specific location. We can specify MOVE/DELETE for this.
   
   We can specify source file to send from transport.vfs.FileURI. Destination of the file where we want to send that file can be specified in <address uri="vfs:file:///home/Desktop/testESB/out"/>
   
   This can be done through file inbound/ file connector. Next blog will cover about it. Until that bye. :)